# Haptic Rattle: Multi-Modal Rendering of Virtual Objects Inside a Hollow Container

Emilie Hummel, Claudio Pacchierotti, Valérie Gouranton, Ronan Gaugne, Théophane Nicolas, Anatole Lécuyer

► To cite this version:

Emilie Hummel, Claudio Pacchierotti, Valérie Gouranton, Ronan Gaugne, Théophane Nicolas, et al. Haptic Rattle: Multi-Modal Rendering of Virtual Objects Inside a Hollow Container. EuroHaptics 2022 - International conference on Haptics: Science, Technology, Applications, May 2022, Hamburg, Germany. pp.1-8, 10.1007/978-3-031-06249-0_22. hal-03614072

HAL Id: hal-03614072

https://inria.hal.science/hal-03614072

Submitted on 19 Mar 2022

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>
# Haptic Rattle: Multi-Modal Rendering of Virtual Objects Inside a Hollow Container*

Emilie Hummel¹, Claudio Pacchierotti², Valérie Gouranton¹, Ronan Gaugne³, Theophane Nicolas⁴, Anatole Lécuyer⁵

¹ Univ Rennes, INSA Rennes, IRISA, Inria, CNRS – Rennes, France

² CNRS, Univ Rennes, Inria, IRISA – Rennes, France

³ Univ Rennes, Inria, CNRS, IRISA – Rennes, France

⁴ Inrap, UMR Trajectoires – Rennes, France

⁵ Inria, Univ Rennes, CNRS, IRISA – Rennes, France

**Abstract.** The sense of touch plays a strong role in the perception of the properties and characteristics of hollow objects. The action of shaking a hollow container to get an insight of its content is a natural and common interaction. In this paper, we present a multi-modal rendering approach for the simulation of virtual moving objects inside a hollow container, based on the combination of haptic and audio cues generated by voice-coils actuators and high-fidelity headphones, respectively. We conducted a user study. Thirty participants were asked to interact with a target cylindrical hollow object and estimate the number of moving objects inside, relying on haptic feedback only, audio feedback only, or a combination of both. Results indicate that the combination of various senses is important in the perception of the content of a container.

**Keywords:** Haptics · Multi-modal rendering · Interaction.

## 1 Introduction

The action of shaking a hollow container to get an insight of its content is a natural reflex [6]. This way of interacting can be of great value in Virtual and Augmented Reality applications, e.g., for the manipulation of virtual instruments such as maracas or to get insights on the number of incoming messages on a smartphone [11]. When interacting with a hollow container that houses one or multiple objects inside, we take advantage of multiple senses to identify various characteristics of the hidden objects, e.g., their shape, weight, size, material, all of them affecting the way they interact with the hollow object when shacked.

This paper introduces a multi-modal method for the rendering of multiple virtual moving objects inside a hollow container, combining audio and haptic feedback for applications in Virtual and Augmented Reality. Starting from an acceleration-based interaction model of the moving objects with respect to the hollow object, we generate the interactions to be rendered through the audio and/or haptic output systems. We called the proposed approach "Haptic Rattle", and performed a user study to assess this approach.

<footnote>* This project has received funding from the European Union's Horizon 2020 program under grant agreement No 801413; project "H-Reality".</footnote>
## 2 Related Work

The perceptual and interaction aspects of manipulating a hollow container filled with small moving objects have been investigated in the literature. Sekiguchi et al. [7] evaluated the perception of feeling a moving object inside a larger container using two solenoids. They assessed the user's ability to discriminate between four interaction models representing a box with an object inside, with only one model faithfully simulating such interaction. Results showed that users were able to effectively discriminate each interaction model [8]. Tanaka et al. [10] assessed the ability of users to discriminate changes in the inner parameters of their model to render the presence of moving objects in a container, using voice coil motors. Similarly, Yamamoto et al. [12] tested the perception between a real and virtual model for solid and liquid content inside a container, with no significant difference between the two models. On another line, the Gravity Grabber of Minamiza et al. used two motors to actuate the motion of a belt onto the fingertip of the user, providing the sensation of holding a glass with liquid inside [1]. Such approach ignores the proprioceptive/kinesthetic aspects of holding a mass, but still offers a reliable feeling of weight and the ability to discriminate between objects with different levels of liquid [2]. Other works aim at improving the realism of simulating the presence of one or multiple objects inside a container using the combination of a voice coil motor and impact actuators [4], the high-speed change of the rotational inertia [9], physics-based haptic vibrations interface [11] or library [5].

While effective, these works focus on rendering the motion of single objects inside a hollow container. On the other hand, Plaisier and Smeets [6] performed an experiment on the ability to perceive the numerosity of objects inside a container, using a tangible box filled with real spheres. Their experiment assessed the number of spheres the participants could accurately detect using audio and haptic or only audio cues, considering between one and five spheres. Results showed that participants accurately detected up to three spheres but underestimated their numerosity when presented with four or five. Finally, results were significantly more accurate when the participants had audio and haptic feedbacks with respect to when they were provided only with audio feedback.

With respect to the abovementioned works, we present a simplified physic-interaction virtual model that can be easily evaluated at runtime for an arbitrary number of virtual moving hidden objects. To assess the viability of our rendering model, we carry out a user study inspired by the work of Plaisier and Smeets [6].

## 3 Methods

When interacting with a container filled with one or more moving objects, the manipulation of the hollow container provides various feedbacks, such as the feeling of the objects hitting or rolling on the inner surface of the container. If the container is rigid, this feedback has audio and haptic components that are characterized by many factors, such as the magnitude of the container's movement and the materials and size of the moving objects. The user's imparted
movement on the container is essential in this perception as it directly causes the interactions of the moving objects and thus the resulting audiohaptic sensations. To render such interactions, we propose an interactive acceleration-based model combined with a multi-modal audio-haptic rendering.

## 3.1 Apparatus

As a proof-of-concept, we use a 3D-printed hollow cylinder which dimensions allow comfortable grasping, containing two haptic voice coil actuators on each base (HapCoil-One, Actronika, FR) and an IMU MPU6050 with a sampling rate of 1kHz, a 3-axis gyroscope and a 3-axis accelerometer in the middle (see Fig. 1a). The voice coils are connected to a TDA3116D2 dual-channel amplifier, powered by a 5V power supply, and a stereo 24-bit 96kHz MOTU sound card.

<figure><img src="image_2.png" /><figcaption>Fig. 1. (a) Our cylindrical prop, housing an IMU and two voice-coil actuators (b) Experimental setup for the manipulation of the device.</figcaption></figure>

The actuators are used to simulate the interaction of virtual spheres inside the container, while the IMU registers the cylinder accelerations when it is shaken by the user. The haptic actuators are inserted symmetrically inside the cylinder, to avoid favoring one handedness over the other. The IMU is set up in a plane position to minimize the impact of rotational movements.

## 3.2 Acceleration-Based Rendering Model

We chose a simple acceleration-based model to enable the fast simulation of an arbitrary number of moving objects inside the container. It registers the data from the IMU and then provides the acceleration parameters for the multi-modal rendering. The acceleration data is filtered using a low-pass filter at frequency $\omega_0 = 60Hz$ and then integrated to output the movement velocity. When the acceleration is higher than a threshold $a_t = 0.5 \text{ m/s}^2$, the magnitude and direction of the movement are transmitted to the rendering algorithm. The frequency $\omega_0$ and the threshold $a_t$ are tuned based on empirical data coming from pilot
experiments, so as to correctly detect when the user is shaking the cylinder. In the world reference frame, the forces acting on the system are the gravity, the inertia of each element, and the force applied by the user. The force applied on each sphere is induced by the acceleration induced by the cylinder movement. Each sphere follows the simple equation: $\vec{F} + \vec{g} + \vec{i} = m\vec{a}$, with $\vec{F}$ being the acceleration applied to the cylinder by the user, $\vec{g}$ the acceleration gravity, $\vec{i}$ the inertia component, $m$ the mass and $\vec{a}$ the acceleration of the sphere. The gravity and inertial physic interactions are handled through the physics engine of Unity 3D, in which we add the forces induced by the user movements. In the cylinder's reference frame, the sphere follows the movement applied by the user with a small delay, induced by the inertia, which depends on the mass of the object. The inertia has an impact on the realism of the interaction [4], however, the simulated spheres are small objects with a small mass. The induced inertia is therefore small as well. For this reason, to reduce the complexity of the algorithm, we considered inertia negligible.

Vibrations and sounds generated by the interaction of the spheres with the cylinder are highly dependent on the inner properties of the cylinder itself. The vibratory impact of a small object against a surface can be simplified as the combination of a sinusoidal signal and an exponential decay, inspired by the model of [3] (see eq. (1)). The resulting low-frequency signal depends on a decay constant $B$, and the frequency $\omega$. $B$ and $\omega$ depend on the modulus of elasticity of the material, and on the density and geometry of the object,

$$Q(t) = e^{-Bt} \sin(\omega \cdot t) \qquad (1)$$

If we consider objects with a regular shape, this equation allows to tune the interaction for any material by recording repeated impacts and fitting the sinusoidal waveform and decay to match the record. This signal corresponds to the output feedback and only needs to be modulated by the object velocity $\vec{v}$ before being displayed through the audio and/or haptic output systems. In our implementation, we used voice-coil actuators for haptic feedback and hi-fi headphones for audio feedback. The signal is displayed through two audio channels, which are independently controlled depending on the direction/location of the impact. Doing so, when the user shakes the container, the system is able to detect its movement and effectively simulate the presence of an arbitrary number of spheres moving inside. Fig. 2 shows a diagram of the rendering process. The

<figure><img src="image_3.png" /><figcaption>Fig. 2. Diagram of the rendering process: from the data sensed by the IMU to the feedback output.</figcaption></figure>
IMU acceleration data are computed into an estimated velocity vector $\vec{v}$. $f(v)$ is a linear function computing a gain factor from each sphere acceleration, based on $v$ value and the configuration of the spheres inside the container. This factor is used to modulate the audio and haptic signals. Audio signal $S_a(t)$ is then computed based on signal $R(t)$ of recorded impact sounds from the desired objects, e.g. wooden spheres, with multiple frequencies and intensities of impact,

$$S_a(t) = f(v)R(t) \qquad (2)$$

On the other hand, the parameters $(B, \omega)$ are chosen according to a wood material ($B = 154 \text{ s}^{-1}$; $\omega = 67 \text{ rad/s}$) to compute $Q(t)$ in eq. (1), on which is based the haptic signal $S_h(t)$,

$$S_h(t) = f(v)Q(t) \qquad (3)$$

These signals are then duplicated according to the number of spheres $n$ and displayed according to the feedback condition considered, with a randomized delay between two empirical values (0 ms and 100 ms) to give a natural effect of collisions.

# 4 User Study

We conducted a user study to assess the effectiveness of our approach in simulating the presence of an arbitrary number of virtual moving objects inside a hollow container. A video is available at https://youtu.be/cMXTvAOvQtc.

## 4.1 Population, Materials and Setup

Thirty participants volunteered to participate in this experiment, aged between twenty and fifty-five years old. Among the participants were eight women and twenty-two men, five left-handed and twenty-five right-handed.

The setup is that described in Sec. 3.1 and shown in Fig. 1b. Users are asked to sit comfortably in front of a computer screen while manipulating the hollow cylinder. The screen shows the instructions of the experiment and enables the user to answer the related questions. Users also wear a pair of headphones for providing audio feedback when this type of feedback is considered, as well as for masking any external audio cues when dealing with haptics-only conditions, e.g., they can mask the audio cues coming from the vibrating voice-coil actuators.

## 4.2 Experimental Conditions and Hypotheses

The design of our user study was inspired by that of Plaisier and Smeets [6]. Participants were asked to power grasp the cylinder with their dominant hand, as shown in Fig. 1b, shake the cylinder along its main axis (left-right movement) for five seconds, and answer the question: “How many spheres do you feel moving inside the container?”. As the cylinder was shaken, the rendering algorithm described in Sec. 3 provided the user with compelling audio or/haptic feedback to simulate the presence of multiple spheres. We considered three feedback
conditions: combined haptics and audio feedback (H+A), audio-only feedback (A), and haptics-only feedback (H). For each condition, the number of simulated spheres inside the hollow container varies from one to five. Each participant carried out 15 repetitions of the trial per condition and per number of spheres, yielding 15 (repetitions) × 3 (feedback conditions) × 5 (number of simulated spheres) = 225 trials, that were randomized. Following the results of Plaisier and Smeets [6] (see Sec. 2), our hypotheses (HP) are as follows:

**HP1:** Participants can effectively discriminate the presence of up to three spheres inside the container.

**HP2:** Participants underestimate the number of spheres when there are four or five spheres in the container.

**HP3:** Participants show a better recognition score with composite audiohaptic (H+A) feedback than with audio (A) or haptic (H) only feedback.

Our objective is to achieve a performance as close as possible to that which humans achieve when interacting with a hollow container filled with real objects.

## 4.3 Results

The following results are the analysis of the experiment data. We observed a learning effect on the five first trials for each participant and therefore removed the related answers, leaving seventy trials for each condition. Two subjects were considered outliers and removed from this data analysis, as they were unable to follow the experiment indications and did not understand the experimental task. To avoid any bias, we did not provide them with additional information with respect to the other participants. Fig. 3 shows the mean and standard deviation of the user's answer in discriminating the number of spheres inside the container, for each actual rendered number of spheres and conditions.

<figure><img src="image_4.png" /><figcaption>**Fig. 3.** Presented vs. Reported numerosity of the spheres inside the container. Mean and standard deviation are plotted.</figcaption></figure>

We evaluated the error between the user's reported numerosity and the actual rendered one, computed simply as the absolute value of the difference between the actual and reported numerosity. Then, we carried out statistical analysis to see whether there is a difference with respect to the feedback condition (H+A, A, H) or the number of rendered spheres (1, 2, 3, 4, 5). Boxplots of the reported
<figure><img src="image_5.png" /><figcaption>**Fig. 4.** (Left) Presented vs. Reported numerosity of the spheres inside the container, per condition. (Right) Presented numerosity vs. error, per condition.</figcaption></figure>

number of spheres and error per condition and number of presented spheres are reported in Fig. 4. We used non-parametric tests because the user's response is not on a continuous scale. The Kruskal-Wallis rank sum test for error according to the conditions shows a significant difference between at least two conditions ($p < 0.001$, $\chi^2(2)=33.279$). The pairwise Wilcoxon rank sum test with continuity correction confirms these results, with a significant difference between conditions (H) vs. (A), $z = 3.83$ and $p < 0.001$, and (H) vs. (H+A), $z = 5.62$ and $p < 0.001$, but no significant difference between (A) vs. (H+A), $z = 1.83$ and $p > 0.1$.

# 5 Discussion and Conclusion

This paper introduces a multi-modal rendering approach for presenting an arbitrary number of virtual moving objects inside a hollow container, using voice-coil actuators and audio headphones to provide haptic and audio feedback about their interaction, respectively. We carried out a user study to assess the capability of human users to effectively discern the number of virtual spheres moving inside the hollow container, providing them with audiohaptic feedback (H+A), audio feedback only (A), or haptic feedback only (H). Results summarized in Figs. 3 and 4 show that, in general, subjects were quite good at estimating the number of spheres rendered inside the hollow container, proving the effectiveness of the proposed rendering techniques. This is especially true when 1 to 3 spheres were rendered, confirming the results of [6] and **HP1**. As also seen in [6], as the number of rendered spheres augments, users tend to underestimate their number, confirming also **HP2**. Indeed, after 3 rendered spheres the curves shown in Fig. 3 flatten and never reach 5. This phenomenon is stronger for condition (H), and it is very similar in conditions (A) and (H+A). Also, the standard deviation increases as the number of rendered spheres increases. The statistical analysis of the results reveals that performance under (H) is significantly different (worse) than the two other conditions, but no significant difference between (H+A) and (A) was found, partially disproving hypothesis **HP3**.

Asking the users about their strategies revealed a certain similarity. They fall into two main categories: the first one consists of repeatedly and energetically shaking the container and comparing the intensity of the feedback with previous
trials; the second one consists of counting the impacts on one extremity of the
cylinder after a single energetic stroke. These two strategies can be of course
mixed together and carried out one after the other.

After the experiment, participants shared their impressions. While they agreed
that the task was difficult, some of them expressed that they were "impressed by
the realism of the sensation". One participant added that he was "surprised to
actually feel the spheres moving in the middle of the cylinder". About the diffi-
culty of the task, participants mostly referred to conditions (A) and (H). Finally,
even if we do not measure a significant difference between (A) and (H+A), par-
ticipants indicated condition (H+A) as significantly easier. (H+A) was overall
the preferred condition.

As for future work, it would be interesting to consider the inertia of the
simulated spheres, which adds to the realism of the interaction [4]. We could
also study whether a continuous variation of frequency for the audio feedback
has an impact.

Finally, it could be interesting to study a comparison between simulated and
real interactions and the impact of the number of actuators inside.

References

1. Minamizawa, K., Fukamachi, S., Kajimoto, H., Kawakami, N., Tachi, S.: Grav-ity grabber: wearable haptic display to present virtual mass sensation. In: ACM SIGGRAPH 2007 Emerging Technologies, pp. 8–11 (2007)

2. Minamizawa, K., Fukamachi, S., Kawakami, N., Tachi, S.: Interactive representa-tion of virtual object in hand-held box by finger-worn haptic display. In: IEEE Symp. on Haptic Interfaces for Virt. Env. and Teleop. Syst. pp. 367–368 (2008)

3. Okamura, A.M., Dennerlein, J.T., Howe, R.D.: Vibration feedback models for virtual environments. IEEE Int. Conf. on Robotics and Automation 1, 674–679 (1998)

4. Park, C., Park, J., Oh, S., Choi, S.: Realistic haptic rendering of collision effects using multimodal vibrotactile and impact feedback. IEEE WHC pp. 449–454 (2019)

5. Park, G., Choi, S.: A physics-based vibrotactile feedback library for collision events. IEEE Trans. Haptics 10(3), 325–337 (2018)

6. Plaisier, M.A., Smeets, J.B.: How many objects are inside this box? IEEE WHC pp. 240–244 (2017)

7. Sekiguchi, Y., Hirota, K., Hirose, M.: Haptic Interface using Estimation of Box Contents Metaphor. ICAT 203 (2003)

8. Sekiguchi, Y., Matsuoka, S., Hirota, K.: Inertial Force Display to Represent Content Inside the Box. Eurohaptics pp. 81–86 (2010)

9. Shimizu, S., Hashimoto, T., Yoshida, S., Matsumura, R., Narumi, T., Kuzuoka, H.: Unident: Providing impact sensations on handheld objects via high-speed change of the rotational inertia. IEEE VR pp. 11–20 (2021)

10. Tanaka, Y., Hirota, K.: Shaking a Box to Estimate the Property of Content. Eurohaptics pp. 564–576 (2012)

11. Williamson, J., Murray-Smith, R., Hughes, S.: Shoogle: excitatory multimodal in-teraction on mobile devices. SIGCHI Conf. Human Fact. Comp. Syst. pp. 121–124 (2007)

12. Yamamoto, T., Hirota, K.: Recognition of Weight through Shaking Interaction. IEEE WHC pp. 451–456 (2015)