# Multisensory Experiences for Art Appreciation

Sakshi Arora

► To cite this version:

Sakshi Arora. Multisensory Experiences for Art Appreciation. 18th IFIP Conference on Human-Computer Interaction (INTERACT), Aug 2021, Bari, Italy. pp.657-666, 10.1007/978-3-030-85613-7_42. hal-04292382

HAL Id: hal-04292382

https://inria.hal.science/hal-04292382v1

Submitted on 17 Nov 2023

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons CC BY 4.0 - Attribution - International License
<figure><img src="image_2.png" /></figure>

ifip

International Federation for Information Processing

This document is the original author manuscript of a paper submitted to an IFIP
conference proceedings or other IFIP publication by Springer Nature. As such, there
may be some differences in the official published version of the paper. Such
differences, if any, are usually due to reformatting during preparation for publication or
minor corrections made by the author(s) during final proofreading of the publication
manuscript.
# Multisensory Experiences for Art Appreciation

Sakshi Arora[0000-10002-9501-4792]

¹ Pearl Academy, New Delhi 110027, India
sakshiarora.2698@gmail.com

**Abstract.** This study aims at understanding multisensory interactions, from visual or bimodal to all senses, to design user experiences for art appreciation. Museum experiences are not just an encounter between an eager visiting public who soaks up the knowledge articulated by the curatorial team. It is a multilayered journey that is proprioceptive, sensory, intellectual, aesthetic, and social, leading to learning and reflection [1]. The target users for this study are people who are not well versed with the context of a visual artwork and are unable to interpret it. Based on secondary readings, it is essential to determine what sensory experiences we can design for and how we can meaningfully stimulate such experiences, when interacting with technology, while understanding the limitations that come into play when users need to monitor more than one sense at a time [2]. The research conducted for this study involved evaluation of simple everyday life activities for multisensorial interactions, through both qualitative and quantitative data collection. The observations from these activities and secondary readings of the subject, have further been tested in the context of visual art experiences, through auditory and visual art prototypes. The key finding of the study is that any multisensory experiences should be designed by isolating the secondary sense modalities. The experience through unconscious senses should be induced involuntarily, by associating a narrative with the entire experience. Based on these findings, a set of principles for designing multisensory experiences have been formed. These principles can further be tested in different contexts for designing intuitive immersive experiences.

**Keywords:** Multisensory interfaces, Sensory isolation, Immersive art experiences.

## 1 First Section

### 1.1 Introduction

The senses that we use while interacting with technology are mostly restricted to vision and audition. In order to translate these experiences, be it conscious or unconscious, to HCI, we can design multisensory experiences by evaluating different experiences in real life and implementing those in our designs [2]. These design interfaces should take into account the relationships between the senses and understand what limitations come
into play when users must monitor information from more than one sense simultane-
ously [3].

This study focuses on identifying different principles for designing multisensory interfaces from everyday experiences and applying those in a specific context to build meaningful experiences.

1.2 Research design

The research has been done in the following two phases:

* Informal research, through quick prototyping for everyday activities, has been conducted with 5 individuals. Both qualitative and quantitative data is collected through observation and conversations during the course of the activities. The raw data obtained is supported with visual and auditory references of the process to draw observational insights.

* Testing of audio-visual prototypes of visual artworks, based on the learnings from the first phase. These prototypes were specifically created in the context of art appreciation for people who are not well versed with the context of visual artworks. The responses were collected from 8 users, through formal interviews.

1.3 Designing multisensory experiences by secondary sense isolation

Multisensory experiences should be designed by isolating the secondary sense modalities. Human physical interaction is inherently multisensory [4]. As participants, one may not be aware of the senses that they are engaging with. For instance, when sipping a cup of coffee, it is not just the smell and taste of coffee that constitutes the experience. The involvement of temperature, color, texture or the material of the cup, sounds in the background, feeling and the overall environment, contributes to the experience as well. These may not be obvious to a user when participating in an activity but can alter the user behavior and experience. In this case, the senses of smell and taste are primary or the conscious senses of interaction. While the ones which are less obvious, are the secondary or the unconscious senses. Isolation of senses means enhancing the experience with secondary sense, while eliminating the interaction with primary sense. Playing the sound of crunching of chips, to provide an experience of the activity or the environment, where one usually eats chips, is a simple example of isolation.

In order to understand conscious and unconscious sensory interactions in everyday
life, a research activity was conducted. This was based on the 5 senses graph by Jinsop
Lee and the idea of experimenting and designing intuitive interactions with Hidden
Senses exhibit curated in Milan Design Week, 2018. The idea of designing interactions
to tap two different layers of experience: the immediate experience of actually interact-
ing with the technology and the past, familiar experiences that populate people's mem-
ories was showcased in the Hidden senses exhibit [5]. Jinsop Lee, an industrial de-
signer, proposed his five senses theory where one can grade in the intensity of every
experience with respect to each sense [6]. With these graphs as a tool to collect re-
sponses, we have tested everyday prompts, like crunching chips, reading a book, mak-
ing coffee etc. for the following activities:
**Activity 1.** For each prompt, a user rates the intensity of experience with each sense modality. The data is collected for both before and after the activity. During the activity, the users are asked to observe the senses intentionally.

<figure><img src="image_3.png" /><figcaption>**Fig. 1.** Intensity graphs for before and after an activity. The x-axis represents the five senses, while the intensity of interaction is mapped on a scale of 0-10 on y-axis, respectively.</figcaption></figure>

**Activity 2.** The conscious or the obvious mode of interaction for each prompt is hidden and a user describes the interaction with other i.e. the unconscious senses. The emotional value for each interaction is plotted on a similar graph. For example:

* Smell through visual

* Sound through visuals

* Visual through sound

<figure><img src="image_4.png" /></figure>
**Fig. 2.** This is an emotion graph, which was created when the activity of crunching of chips is identified by the secondary sense of sound. The x-axis represents different emotions, while the y-axis rates the intensity of the emotional experience on a scale of 0-10, respectively.

Based on the above research activity, conducted as a part of this study, we found the
following:

* During an everyday activity, the impact of obvious sensory interactions, which the users relate to with an experience, tend to fade away. On the other hand, the unconscious interactions are contributing to enhance the experience.

* Asking the users to relate to the original experience, based on the unconscious/ unobvious interaction, adds an element of surprise to an existing experience.

<figure><img src="image_5.png" /></figure>

**Fig. 3.** Intensity graphs for Activity 1 for the prompt of making coffee. In this case, the intensity of experience with sound (secondary sense) rises, with decline in the intensity of sight (primary sense).

This showed that the emotional response to a sensory interaction changes with
change in primary sense modality but lies in a similar range for different users. During
the experience, users tend to relate these interactions to a personal experience or the
expected nature of the activity in general.
RESPONSE 1

<figure><img src="image_6.png" /></figure>

<figure><img src="image_7.png" /><figcaption>**Fig. 4.** Response 1 for the prompt of making coffee, accompanied by the emotion graphs.</figcaption></figure>

RESPONSE 2

<figure><img src="image_8.png" /></figure>

<figure><img src="image_9.png" /><figcaption>**Fig. 5.** Response 2 for the prompt of making coffee, accompanied by the emotion graphs.</figcaption></figure>

Of course, there is a possibility that when repeated frequently, experience with the unconscious senses could turn out to be less engaging like the primary sensory interactions in real life. Therefore, when designing multi-sensorial experiences, using the idea of isolation, one time-sensory interactions should be preferred over repetitive interactions.

# 2 Second Section

## 2.1 Narrative based multisensory experiences for art appreciation.

There is a growing international interest in more immersive media (multisensory) experiences [7]. For users, non-linear stories stay with them because they demand careful thinking about all the pieces presented. When one completes that puzzle, they feel a real sense of achievement [8].

The experience through unconscious senses should be induced involuntarily, by associating a narrative with the entire experience, because not only would the narrative help in enhancing the experience by engaging unconscious senses, but also give a direction or flow for the user journey. “Multisensory stimulation, along with interactive components and dynamic displays, has a strong influence on the visitor experience, especially in terms of the visitor’s sense of flow and immersion.” [9]
This is also evident from the testing of visual art prototypes with multiple soundscapes and audio commentaries to describe the artworks, done with the users. To understand the user behavior and experience with visual artworks, created by Amrita Sher Gil in different settings, visual and auditory prototypes have been created. These prototypes add an additional auditory experience to visual artworks by combining soundscapes associated with the elements of the artworks. There are multiple soundscapes associated with each artwork. These soundscapes could be ambient sounds with still or moving visuals and audio commentary. The purpose of these soundscapes could be satisfying the visual experience or creating a disturbing experience overall. Here, sound becomes the primary sensory mode of communication and can alter the visual interpretation or experience of the prototype.

<figure><img src="image_10.png" /><figcaption>Visual_Art_Prototype_1.mp4</figcaption></figure>

**Fig. 6.** This is the first art prototype for Notre Dame (1932), an artwork by Amrita Sher Gil. The auditory input supports the visual cue of church tower blocking the view of the impressionist river and the city. The sound of church bells dominates over the city sounds and the river.

<figure><img src="image_11.png" /><figcaption>Visual_Art_Prototype_2.mp4</figcaption></figure>

**Fig. 7.** This is the second art prototype for Notre Dame (1932), an artwork by Amrita Sher Gil. The soundscape associated with the background, depicts the idea of a quiet city by a river with natural and calming sounds which aligns with the visuals in the artwork. The interpretation of the artwork with both visual and sound, contradicts with the idea of the setting being at a busy location in Paris.

<figure><img src="image_12.png" /><figcaption>Visual_Art_Prototype_3.MOV</figcaption></figure>
**Fig. 8.** This is the final art prototype for Notre Dame (1932), an artwork by Amrita Sher Gil. This version aims to create a disturbing experience by expressing different message through both sensory modalities. The soundscape signifies the setting of a busy tourist spot of the city of Paris, where this artwork was originally created by the artist.

Within this activity, most users could engage with secondary senses effectively, when accompanied with a conversation during the interviews. The main observations from these interviews are as follows:

* A user, who is not well versed with the context of an artwork, finds the visual experience to be minimal and less relatable.

* Ambient soundscapes turned out to be useful in connecting and directing the users in the initial phase of experiencing the artworks, while keeping the scope of interpretation open.

* Audio commentaries on the other hand, provided clearer ideas about the context but were a bit overwhelming or too informative in terms of experience.

<figure><img src="image_13.png" /><figcaption>art_prototype_ambient_soundscape.mp4</figcaption></figure>

**Fig. 9.** This is a art prototype for Hungarian Peasant (1938), an artwork by Amrita Sher Gil. Ambient soundscape is associated with the visual here.

<figure><img src="image_14.png" /><figcaption>audio_commentary.MOV</figcaption></figure>

**Fig. 10.** This is a art prototype for Hungarian Peasant (1938), an artwork by Amrita Sher Gil. Here, an audio commentary is associated with the visual artwork [10].

Based on these, one could mention that audio commentaries are informative and are very common in museum exhibits and may not be treated as multisensory interaction. That may be true, and audio commentary could be added after the immersive multisensory experience with the artwork like those museums. But they should not be a part of the narrative that is created specifically for the purpose of art appreciation, because of their conclusive and informative nature. The multisensory artworks should rather be open to interpretation where every user can have a unique experience with the same artwork.
## 3 Conclusion

The above stated arguments form on the following design principles, which must be followed to design a multisensory art experience:

* Secondary sense should be isolated to design multisensory experience.

* Create a narrative to engage with the unconscious/ inactive senses of a user.

* Background interactions for every sensory modality, should be considered.

* One-time sensory experiences should be preferred over repetitive interactions.

* Experience with either sense, primary or secondary, should be ambient and not over-whelmingly informative.

* Any essential information, which may not be communicated through the ambient sensory experience, can be shared after the multisensorial experience.

Based on these design considerations, I am working on conceptualizing a virtual reality multisensory prototype, with three sets of visual artworks by Amrita Sher Gil. These sets are based on time periods and locations of the creation of these artworks. Soundscapes and smells based on the narrative of an artwork or the context would be an additional element to an existing visual experience. The idea of experiencing an artwork in a VR environment by participating would be more of a learning experience, where everyone has an inclusive and memorable experience, paving the path for art appreciation.

Considering these design principles as an elementary framework, we can design more immersive user experiences in future. These experiences would not only be meaningful for every user, but would also lead to valuable insights and opportunities, as to how multisensory simulations can influence human computer interaction.
References

1. Levant, N., & Leone, A. P. (2014). The Multisensory Museum : Cross-disciplinary Perspectives on Touch, Sound, Smell, Memory.. Plymouth: Rowman & Littlefield. (Obrist, Gatti, Maggioni, Vi, & Velasco, 2017)

2. Obrist, Marianna & Velasco, Carlos & Vi, Chi & Ranasinghe, Nimesha & Israr, Ali & Cheok, Adrian & Spence, Charles & Gopalakrishnakone, Ponnampalam. (2016). Touch, Taste, & Smell User Interfaces: The Future of Multisensory HCI. 3285-3292. 10.1145/2851581.2856462.

3. Obrist, M., Gatti, E., Maggioni, E., Vi, C. T., & Velasco, C. (2017). Multisensory Experience in HCI. *IEE MultiMedia*, 9-12.

4. Pai D.K. (2005) Multisensory Interaction: Real and Virtual. In: Dario P., Chatila R. (eds) Robotics Research. The Eleventh International Symposium. Springer Tracts in Advanced Robotics, vol 15. Springer, Berlin, Heidelberg. https://doi.org/10.1007/11008941_52

5. Sony Design. (2019). Hidden Senses Concept / Stories / Sony Design. [online] Available at: https://www.sony.com/en/SonyInfo/design/stories/hidden_senses/ [Accessed 15 Apr. 2021].

6. Kate Torgovnick May (2013). *The 5 senses showdown: Designer Jinsop Lee grades 6 great experiences | TED Blog*. [online] @tedtalks. Available at: https://blog.ted.com/the-5-senses-showdown-how-to-grade-your-experiences/ [Accessed 15 Apr. 2021].

7. Obrist, Marianna. (2015). Multi-Sensory Media Experiences. 221-221. 10.1145/2745197.2783433.

8. Motionographer. (2015). *No Particular Order: Thoughts on Non-Linear Storytelling*. [online] Available at: https://motionographer.com/2015/10/19/no-particular-order-thoughts-on-non-linear-storytelling/ [Accessed 15 Apr. 2021].

9. M.L. Harvey et al., "The Influence of Museum Exhibit Design on Immersion and Psychological Flow," *Environment and Behavior*, vol. 30, no. 5, 1998, pp. 601-627.

10. Shakeel, T. (2001). Amrita Sher Gil and her commentaries[online] . Available at: http://ir.amu.ac.in/3390/1/T%205867.pdf [Accessed 10 Mar. 2021].