<figure><img src="image_1.png" /></figure>

HAL
open science

# Exploring the Virtuality Continuum Frontiers:
Multisensory and Magical Experiences in Interactive Art

Nefeli Georgakopoulou, Dionysios Zamplaras, Sofia Kourkoulakou, Chu-Yin Chen, François Garnier

<figure><img src="image_2.png" /></figure>

## To cite this version:

Nefeli Georgakopoulou, Dionysios Zamplaras, Sofia Kourkoulakou, Chu-Yin Chen, François Garnier. Exploring the Virtuality Continuum Frontiers: Multisensory and Magical Experiences in Interactive Art. 3rd International Conference on Human Computer Interaction Theory and Applications, VISIGRAPP, Feb 2019, Prague, France. pp.175-182, 10.5220/0007573901750182. hal-04382868

HAL Id: hal-04382868

https://univ-paris8.hal.science/hal-04382868v1

Submitted on 29 Jan 2024

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_3.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Exploring the Virtuality Continuum Frontiers: Multisensory and Magical Experiences in Interactive Art

Nefeli Georgakopoulou<fnref n="1" />, Dionysios Zamplaras<fnref n="1" />, Sofia Kourkoulakou<fnref n="1" />,
Chu-Yin Chen<fnref n="1" /> and François Garnier<fnref n="2" />

**Keywords:** Interaction Design, User Interface Design, Tangible Interfaces, Smart Material Interfaces, Haptics, Active Sensing, Immersive Experience, Multisensory Experience, Mixed Reality, Magical Reality.

**Abstract:** This article is an effort to approach certain aspects of the evolution of user interface design, as well as the design and aesthetics of interactive works of art. Interactive works attempt to connect the viewer with the work and invite him to deploy his senses and his body. But many researchers and artists are concerned by the fact that interface technologies tend to put us out of touch with ourselves, so they try to create interactions that take place in our periphery, drawing our attention rather than demanding it. Taking as a starting point the sense of touch, whose importance has been widely analysed in philosophy, phenomenology and aesthetics, we describe the modalities of active touch and active sensing. We then propose that with the use of physical materials and tangible interfaces in interactive artworks we can achieve more magical experiences that engage us in a multi sensorial way. As an example we are describing the concepts around the mixed reality installation *VitRails*.

## 1 INTRODUCTION

Through the use of different types of human computer interfaces, interactive works seek to create various multimodal interactions. The resulting multi sensorial experiences challenge and defy our perceptual channels. And while the visual channel seems to predominate in our culture, the conception and design of interfaces and connected objects has long embraced the idea of alternative and more natural ways of producing additional and complementary feedback through the involvement of our other perceptual channels.

This research is an effort to analyze and understand the quality of different Human-Computer-Interactions (HCI) aiming in more “calm” technologies and “natural” interactions through the shift from passive to active sensing. We suggest that a change from the pictorial images of graphical user interfaces towards the abstraction of touch and materiality in tangible user interfaces can result in more interesting immersive and magical experiences in art.

In the following sections, we try first to state

some basic ideas around the characteristics of interactive art and the way we perceive it. We then suggest that the active use of our other senses can engage us in multi-sensory immersive experiences. We also propose the enrichment of these experiences with physical materials, as this can create more natural bonds with the physical world, thus adding the layer of magic to the real virtual continuum. We end with the description of the mixed reality installation *VitRails*<fnref n="1" />.

## 2 INTERACTIVE ART & NEW TYPES OF HCI

Art and cultural practices in general have always been related to the tools and media provided. Nowadays, new ways of using existing technology, as well as new technological advances, come to offer new tools and new media at the disposition of artists and creators. With the arrival of the digital era and

<footnote n="1">¹INREV Laboratory, Paris 8 University, Paris, France</footnote>

<footnote n="2">²EnsadLab Spatial Media, PSL Université, Paris, France</footnote>

<footnote>{nefeli.valeria, dzamplaras, sopheeha.k}@gmail.com, chu-yin.chen@univ-paris8.fr, francois.garnier@ensad.fr</footnote>

<footnote n="1">¹ https://collectifcontinuum.wixsite.com/continuum</footnote>
the computers, the borders between different forms of art merge. The image becomes more modulable and easy to manipulate and transform, sometimes even more abstract, challenging anew the notions of time and space, putting the viewer / spectator in the center of action and minimizing the distance between subject and object. As the technological tools in our disposal continue to evolve, the borders between the different forms of expression also become more blurry (Manovich, 2001).

Interactive works are a form of expression that attempt to connect the viewer with the work, involve him and make him participate in the events, to make him interact with his body and his mind. The human body and the way it operates, which in some forms of artistic expression are less important, become now fundamental notions in the evolution of the narrative. Interactive works often reveal the body and the senses during the process of the experience.

Moreover, interactive art conceives the work as “open” to the various manipulations of the visitor, and thus in a certain way it is proposed as “unfinished” by its creator. To the action of the spectator, the work becomes phenomenologically different and same goes for the experience. This is obviously not without asking questions about the limits of the work or the role of interaction, or even the role of the artist himself (Balpe, 2000).

The interaction with our environment, whether real or virtual, becomes possible through the body, through the use and coordination of expressions, movements, or speech. The response of the environment is received at the emotional and mental levels via our five main senses.

From the human perception point of view, our five primary senses constitute our sensory modalities and modes of communication. When it comes to virtual and interactive works, we use the word modality to describe the input or output modes of interaction. This means that an interaction modality can be defined as a particular and concrete form of a communication mode (Appert, 2016). From these notions, we can deduce that an interactive work is multimodal when it has several modalities as input or output. The notion of modalities is of great importance in the fields of interaction and interface design.

Most interface technologies tend to separate the functions of our physical body, bringing us near to a state of disintegration by putting us out of touch with ourselves and the environment around us (McLuhan, 1964). Moreover, the reception of the majority of the information that comes from the digital world is made through devices that we carry on with us, that

tend to remove us from our surroundings and demand our attention. Nevertheless, many researchers and artists are concerned by this fact and try to create more “calm” technologies, where the interactions take place in our periphery, drawing our attention rather than demanding it.

We are now witnessing new opportunities in interactive art especially with the introduction of new materials and interface technologies such as Tangible User Interfaces (Ishii, 2008), Organic User Interfaces (Vertegaal and Poupyrev, 2008), and Reality-Based Interfaces (Jacob et al., 2008) as well as with the new vision called Smart Material Interfaces (Minuto and Nijholt, 2013) - TUI, OUI, RBI and SMI respectively.

All of these interaction styles try to take advantage on users' pre-existing knowledge of everyday life, except of SMIs which are trying to surpass classical patterns of interaction and abandon the “digital feeling” for a more analog and continuous type of interaction through methods that focus on the properties of materials (Minuto and Nijholt, 2013). This can add to interactive art and design a new layer of magic that transcends what we are already used to.

Here we are not going to discuss about magic as the possess of supernatural forces, but rather as the surprising phenomena that are hidden/existing in nature<fnref n="2" />, and with the help of science and technology we can reveal them. We are going to focus on this magical and exciting quality of the combination of techno science with art which can make something seem different from ordinary things. We propose that with the use of physical materials in interactive artworks we can achieve more fluid experiences that engage us in a multi sensorial way. In the next section, we will be discussing the way we actively sense and perceive the interactive works, taking as a starting point haptics and the sense of touch.

## 3 FROM HAPTICS TO ACTIVE SENSING

The word haptics comes from the greek word *haptein/háptô*, meaning touch<fnref n="3" />. In the field of art and aesthetics, this word has been used by the autrichien art historian AloïsRiegl who, in his work “Late

<footnote n="2">² https://www.larousse.fr/dictionnaires/francais/magie/48531</footnote>

<footnote n="3">³ Le Grand Robert. 2008, Dictionnaires Le Robert/Sejer.500/117/200808/1LGR264CF-CD</footnote>
Roman Art Industry" made the distinction between the optical and the tactile modes of representation and suggested an evolution from one pole to the other (Zerner, 1976). To these days, the word haptics is often used as the term to describe the kinesthetic and cutaneous sensory feedback.

Historically, the sense of touch has often been considered essential to our perception of the environment and the objects around us. According to Aristotle, who affirmed that things are tangible prior to being visible or audible, the touching, since it affects the whole body, can be considered as the sense of embodiment "par excellence" (Massie, 2013), to the extent that it places the man closer to the world and the things closer to him (Aristote, 1993).

According to Merleau-Ponty, but also many other theorists, like Varela, it is recognized that action and perception should be considered together when analyzing behavior and cognition (Merleau-Ponty, 1945; Varela et al, 1993).

Gibson, in his work, made the distinction between being touched and the act of touching, which he named active touching. These two notions made a rather obvious, but very important observation, that objects can be identified and distinguished more precisely if a person is allowed to move their hands and fingers.

This concept of exploration is relative to the theories of Merleau-Ponty. Our capacity of understanding and knowing our environment as human beings lies more to a combination of active processes. This should remind us that the interest of an active touching exploration resides, not only in the perceptual experience in itself, but also in the understanding of the way we form the knowledge of our surroundings and our world (Prescott et al, 2011).

Regarding the tactile perception, Loomis and Lederman (1986) attempted an analysis of interpreting information through this modality. According to them, a person can use multiple modes of exploration for each characteristic, when trying to identify an object through touch. They state that there are two fundamental and distinct senses that together provide us with a sense of touch: the cutaneous sense that provides an awareness of the stimulation and the kinesthesis that provides an awareness of the relative positioning of the body. The form that combines these two senses is described as the haptic perception, and it is the mode of exploration and understanding of our environment which interests this research.

In an attempt to generalize these ideas around

active touch to our other senses, many philosophers consider that all the senses function in a similar way to the touch and that they are based on the same principles (Levine and Touboul, 2015). For french philosopher and scientist Réné Descartes, for example, to see is first and foremost to touch. As the blind "sees" with his hands, a metaphor to state that the object in this case is to the hand what the ray of light is to the eye (Descartes, 1666).

Sensing and perceiving is in constant and co-dependent relation with our body and our spirit. Our meeting with the world happens through our bodies, as this is our way of expressing ourselves and our intentions in a visible form, a place where our spirit takes a form, a way to understand our surroundings. This incarnated consciousness is directly linked to our body and the world, because our existence as "bodies" cannot be separated from the existence of our world (Merleau-Ponty, 1945). In that way, a purely personal reality is created through our way of perceiving and the world becomes unique for each human being, outside of any objectivity.

# 4 MATERIALS AS INTERFACES

As discussed above, unfortunately, technologies today only engage a small portion of our senses, deeply neglecting the full range of material qualities we experience in the physical world. (M.Coehlo, 2012). Materials have always played a major role in our lives and the way we communicate. Walls served historically as a visual representation surface, clay was a means of communication, the stick counting device used to record and document numbers and so on.

We are witnessing, through digital technology, a reduction of the hand to a finger. This has also changed the way we touch and communicate with the digital word. Our interaction with it happens in a more discrete manner, we make contact in a more "on and off" way.

Mark Weiser's insight, that computers might be seamlessly integrated into the world, marked an important overture in what might be called a "material turn" (Robles and Wiberg, 2010). The material turn in interaction design is formulated in tangibility. Tangible interactions connect the digital with the physical by reconsidering computation through materiality (Robles and Wiberg, 2010). Thanks to physical computing some artists focused on more tangible interfaces and therefore on the use of materials, either as a display, or as an interface. Interactivity through materials in digital arts is a
characteristic that privileges the sense of touch. The
hand stops being just an extension of our body that
works in an “on and off” way; through the use of
materials in digital artworks we can now sense our
physical world in a different way, sometimes
creating a magical experience.

# 5 MAGICAL REALITY AND THE EXAMPLE OF VITRAILS

## 5.1 Magical Reality

Professor Hiroshi Ishii and his Tangible Media Group have a vision for the future of human-material interaction, in which all digital information has a physical manifestation so that we can interact directly with it. We no longer think of designing the interface, but rather we conceive the interface itself as material.

The use of materials in our digital age offers the public a new and more natural form of interaction that brings the digital world closer to nature; a union that can give rise to a magical reality. Subbotsky (Subbotsky, 2010) describes the term magical reality as a contrast to physical reality in which magical events are forbidden. For Subbotsky magical events may happen, at least in children's play, in fantasy, dreams, and art. He is not interested on how magic is achieved but what effects are experienced as magic. For Rasmussen (Rasmussen et al, 2013) technological artifacts can construct such magical realities.

Hanna Landin (Landin, 2005) investigates our relationship to technology that could be described as magical. She states that by the use of computer-generated images we can create illusions, but also through technology we can create ‘magical’ things; things that we can in theory understand, but somehow transcend common sense. Judith Guez (Guez, 2015) inspired by the effects of magical tricks, creates magical experiences through the mixture of Illusions between the Real and the Virtual (IRV). She uses real objects as interfaces for the interaction with the virtual world. Her research focuses on creating illusions between the real and the virtual and on exploring new artistic forms that challenge the concept of presence.

Andrea Minuto (Minuto and Pittarello, 2015) in a workshop about the relation of smart materials, art and technology in which students interpreted the relation between materiality and digitality in different modalities, noticed that when the digital

part was hidden under the hood, the visitor experienced a “magic” interaction between different materialities. Jongh Hepworth (Hepworth, 2007) points out that users are used to anything happening on their computer monitor, thus it is more interesting to investigate magical experiences in the gray area between virtuality and physicality. Hepworth is describing four ingredients of magical interaction: Surprise (something unexpected), Unordinary (something different from a previous experience), Unnatural (to do something you cannot do, for example using physical objects to select virtual content), Exciting (something you anticipate).

## 5.2 Magical Materiality

As all the above researches are referring mostly to the field of design and not to the domain of arts, we are now going to examine some artworks that use these ingredients by combining digital technology with materiality.

In Eye Catcher, an interactive project from Lin Zhang and Ran Xie, a seemingly inconspicuous frame on a wall, surprisingly comes to life. Combining receptive software and analog mechanics, it reads your facial expression and expresses itself accordingly, in the form of two morphing black liquid blobs. (ferrofluid -- nanoparticles of ferromagnetic material suspended in a carrier liquid).

In lotus 7.0 an interactive project by Dutch design firm studio Roosevelta, an illuminated reflective wall is coming to life without any mechanical support. The self-commissioned project is made from smart foil, lights and custom electronics. This extraordinary interaction is achieved thanks to the smart foil which opens and closes in response to human behavior.

In the example of interactive work of Random International's Rain Room visitors have the opportunity to experience what is seemingly impossible and unnatural: the ability to control rain.

Aerial Tunes is a collaborative, tangible interface, based on balls hovering in mid-air, which can be manipulated individually, or collaboratively to explore and experiment with an ambient soundscape. Aerial Tunes exemplify how systems can be designed to support aesthetic experiences and promote enjoyment and excitement through a seemingly magical and unstable display. (Rasmussen et al., 2013).
## 5.3 Magical Virtuality

In Milgrams virtuality continuum (Milgram and Kishino, 1994) as we move towards a more virtual environment we are being more and more isolated from the real world that is surrounding us. Our relation with the natural environment is vanishing. For instance, nowadays we can visit, without cost and without spending time, many cities through virtual reality.

In 1988 the anthropologist Alfred Gell studied the relation of magic with technology and proposed magic as an ideal technology; a technology that offers to the user the best result with lesser effort. Such technologies are being created very often the last century but they easily lose their magic (Svanaes and Verplank, 2000). We are used in interacting with a device through peripherals such mouse, keyboard, screen, HMD e.t.c but when we can interact with a VR world through physical materials it is quite surprising. A great example of this is Judith Guez's installation "Lab'surd" (le LABoratoire de la SURvirtualité). In this work the spectator is carried away, through the use of a virtual reality headset, in a contemplative and evolutionary experience of "virtual magic". Between presence and wonder, the spectator is progressively transported from a virtual version of his actual surroundings towards a successive series of illusions that transform the space. Through his journey, he is accompanied by a small luminous sphere, one element that he brings with him from the physical world inside the virtual.

In her installation "Tangible Worlds", the artist Stela Speziali also questions the way we interact in VR experiences. She seeks to combine the immateriality of the digital with the sense of touch through a tangible setup in the physical space surrounding the user. In this example, the sense of touch functions as a catalyst, along with visual and auditory stimuli, for full immersion in a parallel universe. Thus, "Tangible Worlds" questions the perceptual correlation of what we perceive from the outside and the inside of VR and their mutual relationship.

In our research we envisage VR as a medium through which we can re-invent our physical surroundings. Magicians' base technique to create surprises is to establish a pattern between cause and effect and then break it. VR immersion pattern of wearing a headset and immediately entering to a new world is one of the routines that we try to break in our study (Kumari et al., 2018).

As we saw earlier, materials are used in several cases to enrich reality with magical experiences. The same can be done with the use of materials in a virtual experience in order to make it more magical. By adding materiality as a new layer between the virtual and the real, as a "magical mirror" that is needed to be passed in order to achieve the immersion, we try to give to this medium another dimension and make the experience more magical because of a surprising, unordinary, unnatural and exciting way of interaction – to cite Hepworth's ingredients - with both the physical and the virtual world.

## 5.4 Case Study: VitRails

Inspired from these fields of research, the installation *VitRails*, created by Nefeli Georgakopoulou, Sofia Kourkoulakou and Dionysios Zamplaras of the collective Continuum, seeks to create a multisensory experience by integrating touch into a work of mixed reality. Interacting in a virtual environment can sometimes by itself constitute a magical experience. When the interactions occur through different and various modalities, this effect can be enhanced. In *VitRails*, the effort of the user to reveal and sense the virtual environment passes by the sense of touch and the physical materials in front of him.

The installation uses a thermochromic interface. The user, wearing a virtual reality helmet, is immersed in a dark world, without horizon or light. The only way to exit exists in another dimension, this of the physical world. In the physical environment of the user there is a painted wall with a color that detects thermal activity. By touching and coming in contact with it, the user can leave his traces on this "magical interface". These traces on the surface become then a temporary window, which gives him access to the virtual world.

In this work, the participant is invited to use his body, his hands, in order to see the virtual world. Once he achieves to access the virtual world, he has only a few seconds to explore the events revealed on the other side. The cracks will start to disappear quickly, while the color of the wall returns to its original state. In *VitRails*, the user is invited to be in constant contact with the physical world around him in order to sense and perceive the virtual. Thus, the spectator lingers on the frontier between the real and the virtual.

On the other side of the wall, the user – through his temporary traces on the magical interface - witnesses glimpses of the virtual environment. This
other universe consists of a room inside an
abandoned school, surrounded by the sea, which is
visible through the rear door of the room, as well as
the windows on the left wall. Outside of one of the
windows of the room the user can also see a small
boat.

<figure><img src="image_4.png" /><figcaption>Figure 1: The VR environment in VitRails.</figcaption></figure>

<figure><img src="image_5.png" /><figcaption>Figure 2: Interacting with VitRails in Laval Virtual 2018.</figcaption></figure>

Inside the room there are two abstract figures,
which narrate fragments of different stories. The
voices of these figures are getting louder as the user
leans on the wall. Through this setup it seems like
the visitor is placed outside of the virtual world and
in order to understand of what he is seeing he must
eavesdrop the whispers of the figures.

The room and the boat are reconstructed as a
point cloud through the use of photogrammetry. In
other words, actual photographs are used in order to
render the virtual space. In a similar way the sea and
the two beings are also rendered as animated
particles. The technique of photogrammetry is used
first of all as an aesthetic choice from the artists.
Moreover, in our case, the points of the cloud may
also be considered as the equivalent of a digital
materialand a link between the physical and the
virtual visual stimuli.

VitRails also implements bone conduction as a
means of accessing audio content. Hearing through
bone conduction occurs when the vibrations are
conducted through the skull of a listener in the inner
ear, bypassing the eardrum. This kind of devices,
called transducers, have been used in art (as well as
increasingly in consumer products) for several

decades. Despite their widespread use, because of
the “unnatural” way of producing sound through
conduction, that gives the impression that the waves
come out of the interior of objects, the transducers
continue to attract interest.

The contact with the surface in *VitRails* also
allows one to access the vibrations coming out of
these parallel worlds. By manipulating a bone
conductor, the user is invited either to push it against
the thermal surface to make it vibrate and emit
sound, or to press it on his skull in order to acquire a
more personal and intimate experience.

<figure><img src="image_6.png" /><figcaption>Figure 3: Using the bone transducer in VitRails.</figcaption></figure>

The multimodal approach suggested in *VitRails* reveals the complex relationships proposed by multisensory experiences. Sometimes, such practices also allow us to consider a certain degree of effects such as "sensory substitution" or "sensory augmentation". Sensory substitution is here explained by the solicitation of one sense to treat the information normally received by one of the other senses. In our case, this occurs by the use of bone conduction, which make one perceive sound through vibrations on his skin and bones. This "haptic" dimension of sound contributes to the magic of the experience.

The sensory augmentation can be described as
the intention to extend the body's ability to sense
aspects of the environment. In *VitRails* the “sensory
augmentation” occurs from the fact that the user is
deprived of his vision in the beginning of the
experience. This forces him to awake his other
senses, in an effort to sense and reveal the “hidden”
aspects of the real virtual continuum presented. This
effect adds a layer to this passage from one
dimension to the other, making it more magical
through these non-ordinary and surprising modalities
of interaction with the virtual.

6 EVALUATION

The installation has been presented to the public two
times so far. The first in France in the Laval Virtual 2018 festival, in the exposition Recto VRso In, which was entitled *MatièreRéelle / MatièreVirtuelle* (3-8/04/2018). The second in Greece, during the Athens Digital Arts Festival, in its 2018 edition entitled *Singulairty Now* (24-27/05/2018).

The unordinary and innovative way of interacting with this VR installation led to a very high rate of participation in both exhibitions. This was also enhanced by the setup during the two exhibitions. The thermochromic interface was presented as a mere black and otherwise empty painting with a black frame hanging inside the gallery, on a black wall. This rather minimal setup seemed to arouse curiosity.

The SMI, as used in *VitRails*, served as provider of full metaphor (Fishkin, 2004). In the mind of the users, the virtual wall was the same with the physical one. According to the users, we managed a good interaction, as leaning on the wall to see through the cracks and hearing what is happening on the other side connected correctly the actions between the two worlds. The fact that the interface was a physical material really surprised the audience. Moreover, it even excited the part of the public that is less interested in VR experiences, because of its physical changes that were visible to the non-participants. The transducer setup also seemed to arouse curiosity to the public because of the unnatural way of handling this small device and their contact with the surface in order to feel the sound vibrations.

Nevertheless, the thermochromic interface was rather fragile, as on one hand it was influenced by the environmental factors and on the other was deteriorating day by day due to the use. The contact with the bare hands of some dozens of visitors per hour is sure to leave permanent traces or even scratches on the surface. The temperature outside as well as inside the gallery, or even the lighting conditions, had an impact on the overall color of the surface, as well as its capacity to effectively respond to contact.

# 7 CONCLUSIONS AND FUTUREWORK

While most VR installations play with vision as the main input, *VitRails* invites the user to multisensory interaction to enrich mixed reality experiences. Through it, the user is part of a continuum between the real world and the virtual world.

In this research, we tried to explore certain aspects of multisensory and magical experiences in interactive art. We think that interactive art should challenge multiple perceptive and sensory channels and that interaction with them should not engage our full attention but rather immerse us in a more natural way.

Existing research often treats the sense of touch as a merely receptive sensation. But we have evoked that active touch and active sensing is the way we acquire our daily experience, the way we explore our surroundings, whether they are physical or virtual. The understandings of these mechanisms would be very beneficial to haptics and human-machine interaction in general, especially for artists, since they should propose new ways of experiencing the virtual.

As interaction interface design evolves towards more tangible solutions and controllable materials, the concept of magicality as the unordinary and unexpected behavior of physical objects becomes more and more fundamental. Through the use of a technique called electrowetting Umapathi et al are creating a graphical display that uses water droplets to communicate information. As he puts it "The larger idea is to provoke surprise and delight, the way only the natural world can."<fnref n="4" /> In the same way, in the work by Sahoo et al., (2018) it is pointed out that the lack of tangibility in digital interfaces leaves the rich sensory capabilities of our hands and bodies particularly under-utilised. On the contrary, adding tangibility through materials and dynamic haptic feedback aims to connect the physical world more directly to our digital content, in the same way that is imagined and highlighted by Ishii's "tangible bits".

Through our research we also suggest that the use of physical materials and the creation of tangible interfaces creates a unique bonding of the virtual environment with our physical dimension, thus resulting in more magical multisensory experiences. If the magical materiality is the human-material interaction augmented by technology, then the magical virtuality is the human-machine interaction augmented by materiality.

# REFERENCES

Appert, D., 2016. *Conception et évaluation de techniques d'interaction non visuelle optimisées pour de la*

<footnote n="4">⁴ https://www.fastcompany.com/90169644/calm-interfaces-are-here-and-theyre-wonderful (visited 05/01/2019)</footnote>
transmission d'information. Interface homme-machine
[cs.HC]. PhdThesis, Université de Toulouse,
Toulouse.

Aristote, 1993. *De l'âme*, GF Flammarion. Paris.

Balpe, J.-P. (Ed.), 2000. *L'art et le numérique*, Les cahiers du numérique, volume 1, Hermès, Paris.

Coehlo, M., 2013. Materializing Interaction, PhD Thesis, MIT.

Descartes, 1666. *La Dioptrique*, GF Flammarion. Paris.

Fishkin, K., 2004. A taxonomy for and analysis of tangible interfaces. *Personal and Ubiquitous Computing* 8. Springer-Verlag. pages 347–358.

Ginosatis, D., Kavvathas, D., Golemi, A., 2012. Microphysics of Hardware or the Gigantism of the Infinitesimally Small. In *Marshall McLuhan and Walter J. Ong International Centenary Symposium. SSRN Electronic Journal*.

Guez, J., 2015. *Illusions entre le réel et le virtuel (IRV) comme nouvelles formes artistiques : présence et émerveillement*, Thèse, Esthétique, Science et Technologie des Arts, Université Paris 8, Paris.

Hepworth, S., 2007. Magical experiences in interaction design, *Proceedings of the 2007 conference on Designing pleasurable products and interfaces*. ACM. pages 108–118.

Ishii, H., 2008. Tangible bits: beyond pixels. *International Conference of Tangible and Embedded Interaction*, TEI '08, pages xv-xxv..

Jacob, R. J., Girouard, A., Hirshfeld, L. M., Horn, M. S., Shaer, O., Solovey, E. T. and Zigelbaum, J., 2008. Reality-based interaction: a framework for post-wimp interfaces. In *Proc. conf. SIGCHI on Human factors in computing systems*, CHI '08, pages 201-210.

Kumari, S., Deterding, S., Kuhn, G., 2018. Why game designers should study magic. *In Proceedings of the 13th International Conference on the Foundations of Digital Games (FDG '18)*. ACM.

Landin, H., 2005. Fragile and magical: materiality of computational technology as design material. *Proceedings of the 4th decennial conference on Critical computing: between sense and sensibility*. ACM. pages 117–120.

Levine, E., Touboul, P., 2015. *Le corps*. GF Flammarion. Paris.

Loomis, J., Lederman, S. J., 1986. Tactual Perception, *Handbook of perception and human performance*, Boff, Kenneth R. (ed.), New York: John Wiley and Sons.

Manovich, L., 2001. *The language of New Media*, Leonardo, MIT Press.

Massie, P., 2013. Touching, Thinking, Being: The sense of touch in Aristotle's De anima and its implications, *Minerva - An Internet Journal of Philosophy*, pages 74-101.

McLuhan, M., 1964. *Understanding Media - The extensions of man*, McGraw-Hill Education, Canada.

Merleau-Ponty, M., 1945. *Phénoménologie de la perception*, Gallimard. Paris.

Milgram, P., Kishino, F., 1994. Augmented Reality: A Class of Displays on the Reality-virtuality Continuum.

*Telemanipulator and Telepresence Technologies.*
SPIE, vol. 2351, pages 42-48.

Minuto, A. and Nijholt, A., 2013. Smart material interfaces as a methodology for interaction: A survey of smis’s state of the art and development. In *Proc. of 2nd Int. Workshop on SMIs: Another Step to a Material Future*, ACM, pages 1–6.

Minuto, A., Pittarello, F., 2015. Smart Materials: When Art Meets Technology, *In: Nijholt A. (eds) More Playful User Interfaces. Gaming Media and Social Effects*. Springer. pages 177-196.

Prescott, T. J., et al., 2011. Active touch sensing, In *Philosophical Transactions of The Royal Society of London, Series B, Biological Sciences* 366(1581):2989-95

Rasmussen, M. K., 2013. «Magical realities in interaction design», *Proceedings of the 7th International Conference on Tangible, Embedded and Embodied Interaction*. ACM. pages 125-128.

Rasmussen, M. K., Grönvall, E., Kinch, S., Petersen, M., 2013. It's alive, it's magic, it's in love with you: opportunities, challenges and open questions for actuated interfaces. *Proceedings of the 25th Australian Computer-Human Interaction Conference: Augmentation, Application, Innovation, Collaboration*. ACM. pages 63-72.

Robles, E., Wiberg, M., 2010. Texturing the "material turn" in interaction design. In *Proceedings of the 4th International Conference on Tangible, Embedded, and Embodied Interaction*. ACM. pages 137-144.

Sahoo, D., Neate, T., Tokuda, Y., Pearson, J., Robinson, S., Subramanian, S., and Jones, M., 2018. Tangible Drops: A Visio-Tactile Display Using Actuated Liquid-Metal Droplets. In *Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems (CHI '18)*. ACM, New York.

Subbotsky, E., 2010. *Magic and the Mind: Mechanisms, Functions, and Development of Magical Thinking and Behavior*, Oxford University Press, Oxford

Svanaes, D., Verplank, W., 2000. In search of metaphors for tangible user interfaces. In *Proceedings of DARE 2000 on Designing augmented reality environments*. ACM, pages 121–129.

Varela, F., Thompson, E., Rosch, E., 1993. *L'inscription corporelle de l'esprit*, éditions du Seuil. Paris.

Vertegaal, R., and Poupyrev, I., 2008. Organic User Interfaces: Introduction to Special Issue. *Communications of the ACM*, 51(6), pages 26-30.

Zerner, H., 1976. *AloïsRiegl: Art, Value, and Historicism*. *Daedalus*, 105(1), 177-188. Retrieved from http://www.jstor.org/stable/20024392