# "Kapow!": Augmenting Contacts with Real and Virtual Objects Using Stylized Visual Effects

Victor Mercado, Jean-Marie Normand, Anatole Lécuyer

► To cite this version:

Victor Mercado, Jean-Marie Normand, Anatole Lécuyer. "Kapow!": Augmenting Contacts with Real and Virtual Objects Using Stylized Visual Effects. 2020 IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct), Nov 2020, Recife, France. IEEE, 21, pp.116-117, 2001, 10.1109/ISMAR-Adjunct51615.2020.00043. hal-03097352

HAL Id: hal-03097352

https://hal.science/hal-03097352

Submitted on 5 Jan 2021

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.
# “Kapow!”: Augmenting Contacts with Real and Virtual Objects
## Using Stylized Visual Effects

Víctor Mercado*

Univ Rennes, INSA, Inria, CNRS, IRISA

Jean-Marie Normand†

École Centrale de Nantes, AAU, Inria Hybrid

Anatole Lécuyer*

Univ Rennes, Inria, CNRS, IRISA

<figure><img src="image_1.png" /><figcaption>Figure 1: The proposed visual effects applied to real objects in Augmented Reality. Our approach explores different ways to augment contact in AR through the use of comics-inspired onomatopoeias and visual representations of physics-inspired effects on surfaces. Left: A “Kapow!” onomatopoeia is combined with a virtual crack to recreate the sensation of breaking a fragile object in AR. Center-Left: Combination of a “Tap!” onomatopoeia and virtual waves trembling around the object give the sensation of shaking. Center-Right: Comics-inspired collision graphics (stars and lightning) applied to virtual objects. Right: Physics-inspired effects (wrinkles and hollows) applied to virtual objects. As a note: the user’s hand depicted in the illustrative setup pictures used a glove for aesthetic purposes rather than as a tracking aid.</figcaption></figure>

## ABSTRACT

In this work we propose a set of stylized visual effects (VFX) meant to improve the sensation of contact with objects in Augmented Reality (AR). Various graphical effects have been conceived, such as virtual cracks, virtual wrinkles, or even virtual onomatopoeias inspired by comics. The VFX are meant to augment the perception of contact, with either real or virtual objects, in terms of material properties or contact location for instance. These VFX can be combined with a pseudo-haptics approach to further increase the range of simulated physical properties of the touched materials. An illustrative setup based on a HoloLens headset was designed, in which our proposed VFX could be explored. The VFX appear each time a contact is detected between the user's finger and one object of the scene. Such VFX-based approach could be introduced in AR applications for which the perception and display of contact information are important.

**Index Terms:** Augmented-Reality—Virtual Environments—Visual Effects—Pseudo-Haptics;

## 1 INTRODUCTION

Augmented Reality (AR) promises to enhance real-life scenarios with virtual objects available for interaction [1]. The notable availability of these systems for the average consumer has brought interest to develop and optimize interactions in AR. Nevertheless, several limitations can be observed concerning the simulation of object contact within AR such as: the lack of tangible elements, the lack of visual feedback for the users to acknowledge the contact with a virtual surface, and the lack of virtual object's tactile information concerning properties such as friction or hardness. Thus, we propose a set of stylized visual effects (VFX) for augmenting collisions and

contacts with either real or virtual surfaces. The visual effects comprehend different strategies for representing collisions and contact in an AR environment, ranging from comics-inspired onomatopoeias to physics-inspired effects. We designed an illustrative setup in AR based on a Microsoft HoloLens headset to contextualize our approach.

In a nutshell, the main contributions of this preliminary work are: a new approach to augment contacts with real or virtual objects in AR using stylized visual effects, a set of VFX appearing in case of collision/contact in AR, and an illustrative setup based on a HoloLens AR headset showcasing our effects in various interaction contexts.

## 2 RELATED WORK

Work related to our own comprises environment stylizing in AR as well as and the use of passive and pseudo haptics.

Introducing realistic elements in AR still represents a challenge due to the many factors that need to be taken into account in the rendering process of the virtual objects, in particular regarding lighting (shading, shadows, etc.), texturing (color mapping), etc. As a consequence, stylization has been considered as an alternative way to represent virtual objects in AR. For example, this approach has been considered to increase the sense of presence in AR by Fischer et al. [4–6], who applied image filters to give a visual style based on cartoons. These effects permit to display every object seen by the user using a style that does not need to consider complex rendering techniques such as lightning estimation, realistic shadows between virtual and real objects or adapting virtual objects color to the real environment in real-time [8].

The emergence of passive haptics in AR research has led to the development of many novel interaction techniques for object manipulation [2, 3]. Pseudo-haptics research has led to further understand the visual sense's dominance over the sense of touch [7] for augmenting contact in AR. When touching real or virtual objects, pseudo-haptic feedback permits to alter the perceived "physical properties" of virtual and real objects' materials or composition, such as fragility and softness.

<footnote>*e-mail: surname.lastname@inria.fr</footnote>

<footnote>†e-mail: jean-marie.normand@ec-nantes.fr</footnote>
## 3 STYLIZED VISUAL EFFECTS FOR IMPROVING CONTACT SENSATION IN AR

We believe that stylizing VFX after contact can lead to a new paradigm that not only permits to save complex rendering processes related to shape deformation but also to increase the sense of presence and the perception of contact in AR environments. Therefore, our approach consists in relying on stylized VFX to augment contacts and collisions in AR. Our paradigm encloses different features that can be used with both real and virtual surfaces. The stylized VFX we propose are inspired by: (i) physical phenomena that can occur when engaging contact with a surface, and (ii) comics-based effects designed to highlight contact information. A description of these VFX families is presented hereby.

**Physics-Inspired Visual Effects:** This family of effects considers designing VFX to represent physical effects that can occur after contact with a surface. These physics-inspired visual effects not only help users acknowledge contact with the surface but also provide deeper insight about the surface's material properties. Four examples of this family were conceived: "Hollow", "Cracks", "Wrinkles", and "Waves" as seen in Fig. 1.

**Comics-Inspired Visual Effects:** This family of effects considers the use of comics-inspired visual effects to augment the contacts and collisions with real or virtual objects in AR. These effects are divided in two categories: "onomatopoeias" and "collision graphics".

Onomatopoeias, just as in comics, are used in our approach to represent in a visual manner the sound that happens once a collision or a contact with the real or the virtual object has occurred in AR. This subset of VFX aims at taking advantage of the wide variety of sound effects generally used in comics and to exploit them to represent different types of collisions or contact in AR both for real and virtual objects. Two canonical examples were considered for representing onomatopoeias: "Kapow!" and "Tap!", see Fig. 1.

Collision graphics are designed to visually represent the impact of the user's finger on a surface in AR. This VFX subset permits to exploit graphic elements from comics that are used to represent a crash or a shock. An important aspect to be considered from these graphics is that they signal the collision point rather than just appearing close to where the contact occurred. Two representative examples were conceived: "Stars" and "Lightning", cf. Fig. 1.

## 4 ILLUSTRATIVE SETUP

An illustrative setup was designed to showcase our approach and the use of our set of VFX when touching both virtual and real objects in AR. A Microsoft HoloLens was used for displaying the virtual objects as well as the VFX in AR. The virtual simulation was created using the Unity engine version 2019.3. Object tracking as well as finger tapping detection were implemented thanks to Microsoft's Mixed Reality Toolkit. Please refer to our accompanying video to see our VFX in action.

Our illustrative setup enables interaction with real and virtual objects in AR. In this case, the VFX are directly applied to the virtual surfaces. We manually designed invisible virtual 3D counterparts of the real objects and aligned them in the AR simulation to display the VFX onto the real objects' shapes. In the case of virtual objects, the VFX were directly applied to the object's mesh.

## 5 DISCUSSION

We proposed a novel approach to improve the sensation of contact with an object in AR through the use of stylized VFX. Such augmentation of contact in AR is expected to enhance the object presence in augmented environments, and to provide supplementary information about: collision detection, contact position, object material, fragility, etc.

In this work, we considered physics-inspired effects as well as comics-inspired VFX to augment contact with both virtual and

<figure><img src="image_2.png" /><figcaption>Figure 2: Pseudo-Haptic feedback applied to the Cracks VFX. By relying on different intensity levels, we can render different contact sensations. Here, the visual crack's size suggests a certain fragility level different for every egg.</figcaption></figure>

real objects. The physics-inspired effects not only help warning about a contact occurrence but also provide potential insights of the objects' material properties. The comics-inspired VFX profit from their design and dynamic elements to catch the user's attention and accentuate the action of contacting a surface. Collision graphics can profit from graphic elements to signal the contact point. Pointing towards the contact area could notably be useful for enhancing distance or depth perception in AR. Interestingly, the presented VFX are not mutually exclusive in their use as seen in Fig. 1. They can be combined to create a higher contact augmentation effect or a more specific sensation depending on the application context.

Previous work in pseudo-haptics suggests that providing specific and modified visual feedback after contact with a virtual surface can lead to experience various kinds of haptic sensations. In theory, pseudo-haptic feedback can enable to display different and varying contact sensations when relying on different visual intensity levels. Within our approach, this could notably be useful to display haptic properties such as: fragility (such as the case of "Cracks" as seen in Figure 2), softness ("Hollows"), friction ("Wrinkles"), or stability ("Waves").

## 6 CONCLUSION

In this paper we proposed a novel approach to increase the sensations of contact in Augmented Reality based on stylized visual effects. A large set of VFX was designed, inspired by physics or comics. These VFX can be combined altogether, and they appear in case of collision or contact with a real or a virtual object in the AR scene. The VFX intensity can be modified in a pseudo-haptic way to increase the range of simulated sensations. An illustrative setup based on a HoloLens headset showcases our multiple effects in various interaction contexts, such as tapping over fragile objects (such as a teapot or eggs) and displaying virtual cracks corresponding to their different fragility levels. Taken together, our stylization approach could be used in various AR applications in which the augmentation of contact information is meaningful.

## ACKNOWLEDGMENTS

The authors would like to thank the French National Research Agency (ANR) for funding the LobbyBot project.

## REFERENCES

[1] R. Azuma, Y. Baillot, R. Behringer, S. Feiner, S. Julier, and B. MacIntyre. Recent advances in augmented reality. *IEEE Computer Graphics and Applications*, 21(6):34–47, 2001. doi: 10.1109/38.963459

[2] B. Bach, R. Sicat, J. Beyer, M. Cordeil, and H. Pfister. The Hologram in My Hand: How Effective is Interactive Exploration of 3D Visualizations in Immersive Tangible Augmented Reality? *IEEE Transactions on Visualization and Computer Graphics*, 24(1):457–467, 2018. doi: 10.1109/TVCG.2017.2745941
[3] D. Chamzas and K. Moustakas. 3D Augmented Reality Tangible User Interface using Commodity Hardware. arXiv:2003.01092 [cs], 2020. arXiv: 2003.01092.

[4] J. Fischer, D. Bartz, and W. Strasser. Reality tooning: fast non-photorealism for augmented video streams. In Proc. *IEEE ISMAR*, pp. 186–187, 2005. doi: 10.1109/ISMAR.2005.50

[5] J. Fischer, D. Bartz, and W. Strasser. Stylized augmented reality for improved immersion. In Proc. *IEEE VR*, pp. 195–232, 2005. doi: 10.1109/VR.2005.1492774

[6] J. Fischer, M. Haller, and B. H. Thomas. Stylized depiction in mixed reality. *IPI Press*, 2008.

[7] A. Lecuyer, S. Coquillart, A. Kheddar, P. Richard, and P. Coiffet. Pseudohaptic feedback: can isometric input devices simulate force feedback? In Proc. *IEEE VR*, pp. 83–90, 2000. doi: 10.1109/VR.2000.840369

[8] S. Wang, K. Cai, J. Lu, X. Liu, and E. Wu. Real-time coherent stylization for augmented reality. *The Visual Computer*, 26(6-8):445–455, 2010. doi: 10.1007/s00371-010-0436-z