# Can we effectively combine tangibles and ultrasound mid-air haptics? A study of acoustically transparent tangible surfaces

Thomas Howard, Guillaume Gicquel, Claudio Pacchierotti, Maud Marchal

► To cite this version:

Thomas Howard, Guillaume Gicquel, Claudio Pacchierotti, Maud Marchal. Can we effectively combine tangibles and ultrasound mid-air haptics? A study of acoustically transparent tangible surfaces. IEEE Transactions on Haptics (ToH), 2023, 16 (4), pp.477-483. 10.1109/TOH.2023.3267096. hal-04057832v2

HAL Id: hal-04057832

https://inria.hal.science/hal-04057832v2

Submitted on 12 Apr 2023

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>
# Can we effectively combine tangibles and ultrasound mid-air haptics?
## A study of acoustically transparent tangible surfaces

Thomas Howard, Guillaume Gicquel, Claudio Pacchierotti, and Maud Marchal

*Abstract*—We propose to study the combination of acoustically transparent tangible objects (ATTs) and ultrasound mid-air haptic (UMH) feedback to support haptic interactions with digital content. Both these haptic feedback methods have the advantage of leaving users unencumbered, and present uniquely complementary strengths and weaknesses. In this paper, we provide an overview of the design space for haptic interactions covered by this combination, as well as requirements for their technical implementation. Indeed, when imagining the concurrent manipulation of physical objects and delivery of mid-air haptic stimuli, reflection and absorption of sound by the tangibles may impede delivery of the UMH stimuli. To demonstrate the viability of our approach, we study the combination of single ATT surfaces, i.e. the basic building blocks for any tangible object, and UMH stimuli. We investigate attenuation of a focal point focused through various plates of acoustically transparent materials, and run three human subject experiments investigating the impact of acoustically transparent materials on detection thresholds, discrimination of motion, and localization of ultrasound haptic stimuli. Results show that tangible surfaces which do not significantly attenuate ultrasound can be fabricated with relative ease. The perception studies confirm that ATT surfaces do not impede perception of UMH stimulus properties, and thus that both may viably be combined in haptics applications.

*Index Terms*—Human-Computer Interaction, Perception and Psychophysics, Virtual Reality

## I. INTRODUCTION

Tangible haptic props find various applications in human-computer-interaction (HCI). They serve as a physical support for interaction with digital content in tangible user interfaces [1], or constitute physical counterparts to virtual objects in augmented [2] (AR) and virtual reality (VR) [3]. Contrary to most actuator-based haptics, they do not require the user to be equipped in any way [4]. They provide natural affordances for manipulation, leverage sensorimotor behaviors used in our everyday lives, and are uniquely suited to providing the complex distributed multisensory cues that are typical of humans' haptic interactions with their environment [5]. However, the flip-side of these perks is that, contrary to haptic feedback delivered by actuators, tangibles are limited in how much they can adapt to different or changing digital content or virtual objects [6].

Ultrasound mid-air haptic (UMH) devices enable an actuator-based approach to haptic feedback which also does not require the user to be equipped with any special apparatus or device. They are capable of freely delivering vibrotactile stimuli in a 3D volume above the interface [7]. These devices have found applications in various fields, including mid-air

interaction with user interfaces [8] and with virtual environments in AR and VR [9]. UMH stimuli are reconfigurable and adaptable to a wide variety of situations, but are restricted to vibrotactile or light pressure sensations [7]. Thus, applying them to the simulation or replication of many physical interactions with real environments is challenging.

Within approaches to haptic interaction that leave users unencumbered, tangible and ultrasound mid-air haptics appear highly complementary. We therefore investigate the possibility of combining both approaches. In particular, we seek to address the question of how to enable combined UMH and tangible haptic interactions without the manipulated tangible props obstructing the ultrasound stimuli. This leads us to propose and investigate the use of acoustically transparent tangibles (ATTs). Our contributions are:

* An analysis of the design space for physically combining UMH and tangibles in haptic HCI (Sec. III).

* A series of attenuation measurements showing the technical plausibility of focusing ultrasound stimuli through single ATT surfaces (Sec. IV).

* Three perception studies investigating detection thresholds, perception of motion and stimulus localisation for UMH stimuli focused through ATT surfaces (Sec. V).

## II. RELATED WORK

In human-computer interaction, artificial haptic sensations can present information to users, assist in completing tasks, augment or substitute for feedback to the other senses, and make interactions with virtual environments more immersive and realistic [4]. Tangible haptic interactions provide an efficient means for obtaining rich natural multimodal haptic feedback by relying on real physical objects as interfaces for interacting with digital content [3], [1], [2]. However, tangible haptics come with a major drawback: since they rely on real physical objects, the haptic feedback provided cannot easily adapt to evolving digital contents or virtual environments. This has motivated investigations into the combined use of haptic actuators and tangibles. In the field of tangible user interfaces, this has taken the form of actuated and self-configuring interfaces [10]. When interacting with virtual environments, actuators inside tangible props dynamically alter their physical properties [11], or wearable haptics modulate interactions with passive tangible props [12].

These approaches are promising but present two constraints which may prove problematic. Embedding actuators into tangible objects [13] increases object mass, size and power supply requirements, complicating design and usability. The alternative, which consists in combining tangibles with wearable haptics [12], [14] comes with the need for equipping the user, which is contrary to natural haptic interactions with real

T. Howard, G. Gicquel, M. Marchal are with Univ Rennes, INSA, IRISA, Inria, CNRS - France.

M. Marchal is also with IUF, France.

C. Pacchierotti is with CNRS, Univ Rennes, Inria, IRISA - France.

This work was funded under the ANR project "MIMESIS"
environments and may be undesirable. These issues motivate our present investigation into augmenting tangible haptic interactions with ultrasound mid-air haptics, which leaves users unencumbered.

A survey of ultrasound haptic technologies and their applications was presented by Rakkolainen et al. [7]. UMH stimuli usually rely on ultrasound waves being focused towards one or several focal points in the device's workspace, whose position and instantaneous pressure can be changed at frequencies up to the operating frequency of the device's transducers. This enables complex spatio-temporal modulation patterns, creating complex spatial and temporal vibrotactile effects. A formalization of this rendering approach is discussed in the work by Mulot et al. [15].

Early work on UMH used in conjunction with physical objects investigated the impact of thin acoustically transparent surfaces placed between the device and user. The aim was to assess suitability of such interfaces for augmenting interactions with projector screens [16]. The authors analysed the attenuation for a single focal point through 5 different surface materials, showing that a minimum amount of percentage of open space is required to allow ultrasound to pass, but that perforation size is also an important factor. In the field of ultrasound haptics, metamaterial surfaces have also even been used to shape resulting acoustic fields on the output side, allowing passive acoustic focusing, steering and levitation [17]. These works suggest that using ultrasound focused through ATTs should be technically feasible.

More recently, Abe et al. used ultrasound focused onto a physical surface from above to modify its perceived texture properties [18], [19]. They showed that for specially designed resonant films as well as lightweight polystyrene, ultrasound focused on the surface at the level of the contact with the finger can modify perceived roughness. Ohmori et al. later demonstrated this effect in contacts with 3D polystyrene surfaces [20]. This work falls into a specific category of the design space for combined ultrasound mid-air and tangible haptic interaction, which we discuss in Sec. III, and is complementary to the contribution proposed in this paper.

### III. COMBINING TANGIBLE AND UMH FEEDBACK

To combine ultrasound tangible haptics, it is necessary to consider how each of the haptic stimuli are physically arranged within the interaction, as this dictates the possible physical interactions that may occur between the ultrasound waves and the tangible object.

We can distinguish two scenarios. The first are interactions where the stimuli are simply juxtaposed, i.e. they do not interfere with one another. For these, UMH stimuli can only be delivered in unobstructed interaction space (U - white area in Fig. 1). The second are interactions where the stimuli are combined, so interactions between ultrasound and the tangible need to be taken into account. In this case, we can further divide these interactions into those where the haptic stimuli are strictly colocated (i.e. focusing on the object surface S - blue area in Fig. 1), and those where the UMH stimuli are focused through the tangible object (in the obstructed region O - pink area in Fig. 1). In both cases, the tangible

object may be fully acoustically transparent, fully acoustically impermeable or anything in between (including heterogeneous constructions). Using acoustic levitation to actuate tangibles (e.g. [21]) is a scenario which is not discussed here as it does not directly employ ultrasound for haptic feedback.

Tangible objects interact with incident sound waves in three basic ways: reflection, absorption, and transmission [22]. When impinging upon a non-transparent surface, sound waves are either absorbed within said surface, which can in some cases results in perceivable surface vibrations, or be reflected back. Reflection may prevent ultrasound stimuli to be delivered in the O region, but nonetheless this behavior could be applied to the generation of stimuli in the U region, similarly to the way Ariga et al. used a passive acoustic reflector to augment the workspace of a focused ultrasound array [23].

For interactions in the S region, the vibrotactile stimuli can either be overlaid over the tangible, colocating a vibrotactile stimulus with the tangibles' surface, or can be used to actively modulate the tangible's haptic properties, such as in prior work on remote surface friction reduction [18], [19], [20]. In the case of modulation of the tangible's haptic properties, the determining factor is the tangible object surface's acoustic impedance relative to that of air at the frequency of the ultrasound. If the difference in impedance is large, most of the waves will be reflected and the resulting vibration amplitude of the surface will be low. Some specially designed surface can be engineered to have a lower relative impedance at the resonant frequency [18], allowing a tangible surface to be set into vibration. If surface vibration amplitude is sufficient, this can in turn change the perceived haptic properties of the surface.

In the case where the UMH stimuli are overlaid onto the tangible, the determining factor is the tangible object's acoustic transparency, similar to the case of ultrasound focused in the O region. In both cases, the ultrasound must be able to pass through the tangible object to directly act on the user's skin. This interaction scenario is the focus of our subsequent experimental work. Perforated surfaces are known to let sound waves through, a principle which is largely used in the design

<figure><img src="image_2.png" /><figcaption>Fig. 1. Design space for combinations of UMH and tangible haptics</figcaption></figure>
of movie theater screens where the speakers are placed behind the projection screen to ensure synchronisation between sound and image. The structural properties of the perforated surface (percentage of open space, perforation dimensions, material) largely dictate to what extent sound can travel past the surface and to what extent sound is absorbed or reflected back by the surface, i.e. the surface's acoustic transparency. Preliminary experiments on fabric and perforated paper sheets show that percentage of open space as well as hole diameter have a significant impact on acoustics transparency [16]. These results hint at the feasibility of using taut fabric over a solid supporting structure as a possible means for creating ATTs. Alternatives are mechanical fabrication processes (machining, etching or laser cutting) to create porous solid surfaces on hollow objects. Here, as a proof of concept, we 3D print grid surfaces which may then be used as ATT surfaces or assembled into 3D ATT objects, although the latter use-case is beyond the scope of our present work.

## IV. INVESTIGATION OF ATTENUATION

We began by conducting pressure measurements to evaluate the attenuation that could be expected when 3D-printing tangible surfaces or fabricating them from speaker mesh.

1) MATERIALS AND METHODS: We used a Grifema GA 2003 precision scale (0-500g, +/-0.01g) onto which we focused a 200Hz amplitude-modulated focal point, using an Ultraleap Stratos Explore haptic interface mounted upside-down on a height-adjustable frame (See Fig. 2.A). A 3D-printed mount was placed above the scale, obstructing all incident sound waves except through a central 6x6cm cutout section. This cutout served as a receiver for 6x6cm square ATT surface samples whose acoustic transparency we wished to evaluate. The surface samples were held at a height of 50mm above the scale, which was empirically determined to not impede focal point focusing through the cutout.

Based on pilot tests, we empirically chose to consider a set of twelve 0.6mm-thick tangible surface samples with a triangle grid pattern and percentages of open space OS% varying between 90% and 0% (see Fig. 2.B), a square of generic speaker mesh taut across a 3D-printed square frame, and four samples with OS% = 25% whose thickness ranged from 0.6mm to 8mm. The triangle grid pattern was achieved by printing plates without any top or bottom layers, using the triangle infill pattern available in PrusaSlicer 2.5¹, and varying the infill density.

We measured the acoustic radiation force generated on the scale by the focal point 20 times, calculating the mean unattenuated force $F_0 = 1.78mN$. We then similarly measured the force through each ATT sample 20 times in two angular configurations: parallel to the UMH device (blue in Fig 3) and at a 45° angle (green in Fig 3). In the 45° angle, we varied OS% in larger increments, yielding half fewer measurements than in the parallel configuration. This was to assess effects that angles occurring between ATTs and UMH device during manipulations may have on attenuation. All force values $F_i$ were then converted to attenuation values $\alpha = 100.(1 - F_i/F_0)$

<figure><img src="image_3.png" /><figcaption>Fig. 2. (A) Measurement setup for the attenuation assessment. (B) Example set of ATT surface samples with varying percentages of open space (t is the fixed grid line width, w is the variable side length of the triangle openings.</figcaption></figure>

in [%]. For discussion purposes, we subsequently measured the force $F_{th} = 0.63mN$ obtained by an unobstructed focal point at the intensity level corresponding to the detection threshold found in Sec. V-A. From this, we calculated the equivalent attenuation below which we predicted a stimulus would no longer be perceivable $\alpha_{th} = 65\%$ (red line in Fig 3).

2) RESULTS: Fig. 3 shows the resulting attenuation measurements. Normality and homoscedasticity of the data was verified by Shapiro-Wilk and Bartlett tests respectively. We therefore ran Bonferroni-corrected two-sample t-tests to compare $F_i$ to $F_0$ and $F_i$ to $F_{th}$. In the parallel orientation (blue in Fig 3), comparing $F_i$ to $F_0$ revealed significant attenuation for OS% < 60%, the mesh, as well as OS% = 25% for sample thicknesses of 4mm and 8mm. In the 45° orientation (green in Fig 3), comparing $F_i$ to $F_0$ revealed significant attenuation for OS% < 50%, the mesh, as well as OS% = 25% for sample thicknesses of 4mm and 8mm. Comparing $F_i$ to $F_{th}$ revealed significant differences for all conditions except the

<figure><img src="image_4.png" /><figcaption>Fig. 3. Attenuation $\alpha$ of the acoustic radiation force (in [%]) caused by focusing through various ATT plates with different percentages of open space in a triangular grid, through fabric mesh, and through 25% open space ATTs of varying thicknesses. Blue curves and shaded areas represent the mean attenuation and associated 95%-confidence intervals (CI) for ATTs placed parallel to the UMH device. Green curves and shaded areas represent the mean attenuation and associated 95%-CI for ATTs placed at a 45° angle to the face of the UMH device. Negative values indicate amplification of the UMH stimuli, although this may not be a phenomenon that actually occurs, as suggested by the confidence intervals. The red line and shaded area represent $\alpha_{th}$ and the associated 95%-CI.</figcaption></figure>

<footnote n="1">¹Freely available at https://www.prusa3d.com/page/prusaslicer_424/</footnote>
$OS_{\%} = 30\%$ and $OS_{\%} = 10\%$ samples in the 45° orientation. Statistical testing details are provided as supplemental material in Appendix A.

3) **DISCUSSION:** ATT grids with more than 60% open space and thickness below 4mm appeared to produce minimal attenuation, and thus could be optimal for combined ultrasound mid-air and tangible haptics applications. Almost all considered tangible surfaces appeared as though they would not attenuate maximum intensity focal points to the point of making them undetectable. These results held true even for surfaces which were not parallel to the UMH device, suggesting that ultrasound stimulus delivery may be robust to 3D manipulation of ATT surfaces.

## V. PERCEPTION STUDIES

The attenuation observed in Sec. IV may still adversely impact perception of UMH stimulus properties. For this reason, we conducted three experiments on the perception of UMH stimuli through ATT surfaces to investigate to what extent they interfere with the detection of UMH stimuli (Sec. V-A), with localization of UMH stimuli (Sec. V-B) and with perception of UMH stimulus motion (Sec. V-C).

All three experiments made use of the same physical setup shown in Fig. 4.A. The UMH device was fixed in position on a table in front of the subjects. A custom-built fixture supported exchangeable 12x12cm ATT plates at a height of 10cm above the array face. We used a subset of four tangible surfaces from the previous investigation (Mesh, P25t, P70t, P100) with a fixed thickness of 0.6mm. Based on prior measurements, we expect peak focal point pressures through the surfaces to be 1.55 kPa (Mesh), 1.65 kPa (P25t), 0.43 kPa (P70t) and 0 Pa (P100) respectively. Each experiment was divided into 8 blocks, each with a given surface and contact condition. In the contact condition C, subjects placed their hand or finger in contact with the surface, whereas in the non-contact condition NC, subjects placed their hand or finger 5cm above it. The NC condition was omitted for the P100 surface, as it is not acoustically transparent. The C condition is also not defined when no surface is present. For this reason, comparisons across levels of the surface factor were done independently for each level of the contact factor. Additionally, a 2-factor analysis of results (contact x surface) was performed for the Mesh, P25t and P70t surfaces. The order of conditions was counterbalanced across subjects.

The study involved 18 subjects (5F, 13M, ages: 20-46 (M:25,SD:5.9)) and was performed in VR to ensure that visual cues did not affect responses. They wore an HTC Vive Pro VR headset and headphones playing pink noise, while seated behind the table with the ultrasound haptic device. To perform the task, they used their dominant hand, which was tracked using a Vicon Bonita camera system interfaced with Unity3D. In the VR scene, subjects could see their tracked hand as well as instructions for each of the three experimental tasks (see Fig. 4(B,C,D)). They provided responses using an HTC Vive controller held in their non-dominant hand.

### A. Detection thresholds

During each trial, a 1cm diameter spatio-temporally modulated circle (70Hz draw frequency) was continuously rendered

<figure><img src="image_5.png" /><figcaption>Fig. 4. (A) Subject feeling an ultrasound stimulus focused through a tangible surface. The experimenter switched out the surface sample in between trials as per the experiment plan. (B) VR view for the detection threshold experiment: An icon on the right indicated the contact condition so that subjects could ensure correct contact with the tangible surface. A transparent hand indicated the approximate required hand position. Instructions were shown on a panel facing the subject. (C) VR view for the motion discrimination experiment: In addition to the previously described interface elements, the subject could provide their response for the felt motion by selecting one of the 6 possible motion directions, indicated by a green arrow and moving yellow line. (D) VR view for the focal point localization experiment: A timer indicated how much of the 15s allotted to the task had elapsed. Subjects searched for the focal point position in the transparent red plane using their fingertip.</figcaption></figure>

in the middle of the subjects' palm. The virtual environment displayed to the subject is shown in Fig. 4.B.

The experiment followed a method of adjustment protocol, with stimulus intensity (dimensionless value between 0 and 1 proportional to the peak focal point pressure squared) as the adjusted variable. Trials randomly alternated between increasing intensity (starting from 0 Pa) or decreasing intensity trials (starting from the maximum intensity of 1, corresponding to 1.8 kPa). Subjects used the controller to adjust the stimulus intensity in increments of 0.05, until it started to become perceivable (in increasing trials) or stopped being perceivable (in decreasing trials), at which point they confirmed to proceed to the next trial. The before-last value of intensity was recorded as the detection threshold estimate in a trial. Subjects performed 3 ascending and 3 descending trials in both the C and NC conditions for each surface sample. The direction of the trials randomly alternated between ascending and descending within each condition. The mean of these 6 estimates was taken as a subject's detection threshold for said condition.

We hypothesized: (H1) Stimulus detection thresholds will be proportional to attenuation caused by the surface. (H2) For any given surface, stimulus detection thresholds will be similar in the C condition and in the NC condition.

1) **RESULTS AND DISCUSSION:** Fig. 5.A shows the detection threshold distributions across all experimental conditions. Shapiro-Wilk and Bartlett tests showed that the threshold data were mostly not normally distributed. However, they respectively verified normality and homoscedasticity for the log-transformed data used in the following analyses. A 1-
<figure><img src="image_6.png" /><figcaption>Fig. 5. (A) Detection thresholds [dimensionless intensity units]. (B) Localisation errors $\epsilon$ [m] and times $T$ [s]. (C) Precision (pink) and recall (blue) scores (in [%]). Blue dots represent median values, red lines mean values. Horizontal bars show statistically significant differences ($p < 0.01$: brown, **; $p < 0.05$: orange, **; $p < 0.1$: yellow, *)</figcaption></figure>

way ANOVA between surfaces in the NC condition showed no significant effect of surface on detection thresholds. A 1-way ANOVA between surfaces in the C condition showed a significant effect of surface on detection thresholds ($F = 59.81, p < 0.001$). A 2-way ANOVA on the log-transformed data for surfaces *Mesh*, *P25t* and *P70t* showed no significant effect of surface but a significant effect of contact condition ($F = 5.02, p = 0.027$) and no interactions. Post-hoc Bonferroni-corrected t-tests in C revealed a significant difference between thresholds for the *P100* and the *Mesh* ($t = -12.44, p < 0.001$), *P25t* ($t = -12.66, p < 0.001$), and *P70t* ($t = -10.46, p < 0.001$) surfaces respectively. Furthermore, a weak effect of the contact condition was found between thresholds for the *P25t* surface ($t = -2.46, p = 0.076$).

We performed a linear regression analysis to evaluate the relationship between detection thresholds and mean attenuation for the considered surfaces. Results indicate a significant positive relationship between both variables ($R^2 = 0.76, \text{F-stat. vs. constant model: } 342, p < 0.001$), supporting H1.

However, we rejected H2 since contact had a weakly significant effect on detection thresholds for *P25t*. This detrimental effect of contact may depend on the surface properties. Finally, the mean threshold for *P100* is 1.48 kPa, suggesting that high intensity stimuli vibrated the surface at a detectable level.

## B. Stimulus localization

A 250Hz amplitude-modulated focal point was projected at a random location either in the plane of the tangible surface (contact condition C), or 5cm above it (non-contact condition NC). The virtual environment displayed to the subject is shown in Fig. 4.D. Subjects started each trial with their hand in front of the ATT plate holder. They then had to locate the focal point with the tip of their index finger as accurately and as fast as possible. They recorded the position of their fingertip as their estimate of the focal point position using the Vive controller, ending the trial. Subjects performed 9 trials per condition. In each trial, the target locations were randomly selected within a 10x10cm area centered on the ATT holder.

Our hypotheses were: (H3) For a given contact condition, stimuli will be localized with equal accuracy as long as they are perceivable, regardless of the attenuation caused by a given surface. (H4) Stimuli will be localized faster and more accurately when not in contact with a tangible surface.

1) **RESULTS**: Fig. 5.B shows the task execution times $T$ as well as the localization errors $\epsilon$, i.e. the distance between the focal point position estimate and the actual focal point position in the horizontal plane.

Shapiro-Wilk and Bartlett tests verified normality and homoscedasticity of the $\epsilon$ and $T$ data. One-way ANOVAs showed a significant effect of Surface on $\epsilon$ ($F = 3.72, p = 0.015$) and $T$ ($F = 11.39, p < 0.001$) in the C conditions and no effect in the NC conditions. A two-way ANOVA across the *Mesh*, *P25t* and *P70t* data shows no effects on $\epsilon$ but a significant effect of Contact on $T$ ($F = 24.58, p < 0.001$). Post-hoc Bonferroni corrected t-tests showed significant differences in $\epsilon$ between *P100* and *Mesh* ($t = -3.17, p = 0.012$), *P25t* ($t = -2.74, p = 0.038$), and *P70t* ($t = -2.55, p = 0.062$) respectively. We found significant differences in $T$ between *P100* and *Mesh* ($t = -4.48, p < 0.001$), *P25t* ($t = -5.56, p < 0.001$), and *P70t* ($t = -4.57, p < 0.001$) respectively, and significant differences between contact conditions for *Mesh* ($t = 3.06, p = 0.017$), *P25t* ($t = 2.86, p = 0.029$) and *P70t* ($t = 2.66, p = 0.047$).

2) **DISCUSSION**: No significant difference in localization times or errors was observed between surfaces, with the exception of *P100*. So long as they remained perceivable, stimuli were localised with equal ease regardless of the attenuation caused by the tangible surface, supporting H3.

We observed significant differences in localization times within surfaces depending on contact with the surface. H4 was therefore also supported, as subjects tended to localize focal points faster and with equal accuracy when they were not touching the tangible surface. Contact with a tangible therefore likely has a detrimental effect on localization performance.

## C. Perception of motion

A moving spatio-temporally modulated line (length: 10cm, draw frequency 70Hz) was projected either onto the tangible surface (C) or 5cm above it (NC). The line followed one of six possible infinitely looping motion patterns: upward translation, downward translation, left-to-right translation, right-to-left translation, clockwise rotation or anticlockwise rotation. Subjects placed their hand at the location indicated by a transparent hand in the virtual environment. They then reported the motion they felt by selecting from a menu (see Fig. 4.C).
Motions were randomized within each surface and contact condition, with each motion occurring 9 times.

We hypothesized: **(H5)** Stimulus motion will be discriminated equally well for all surfaces, except *P100* for which it will not be possible. **(H6)** Stimulus motion will be discriminated equally well for a given surface, regardless of contact.

**1) RESULTS:** We calculated mean precision and recall scores across stimulus motions, for each condition and subject (see Fig. 5.C). Shapiro-Wilk tests rejected the hypothesis that precision and recall data for each condition were normally distributed. Friedman ANOVAs across all C conditions and across all NC conditions respectively revealed an effect of surface on precision ($\chi^2 = 42.3, p < 0.001$) and on recall ($\chi^2 = 42.2, p < 0.001$) in the C conditions. Two-way aligned-rank-transformed ANOVAs on the data from *Mesh*, *P25t* and *P70t* revealed significant effects of contact on precision ($F = 7.2, p = 0.008$) and on recall ($F = 6.9, p = 0.009$), significant effects of surface on precision ($F = 3.7, p = 0.028$) and on recall ($F = 3.7, p = 0.027$), but no interactions. Post-hoc Bonferroni-corrected t-tests on the aligned-rank-transformed data confirmed a significant difference between *P25t* and *P70t* for precision ($t = 2.57, p = 0.034$) and recall ($t = 2.61, p = 0.031$). Bonferroni-corrected Wilcoxon signed-rank tests confirmed differences between *P100* and *Mesh*, *P25t*, and *P70t* for both precision and recall ($z = 3.8, p < 0.001$ in all cases).

Wilcoxon signed-rank tests on the data for *100* shifted by the chance level indicated a significant difference between chance level and precision ($p = 0.03$) and recall ($p = 0.02$) for this condition.

**2) DISCUSSION:** H5 was only partly supported, as some motion discrimination appeared to take place in the *P100* condition despite the very poor performance, and discrimination performance was significantly better for *P25t* than for *P70t*. We also rejected H6, since contact was shown to significantly worsen motion discrimination performance.

# VI. CONCLUSION

We proposed the combination of tangible objects and ultrasound mid-air haptic (UMH) feedback in 3D interaction with digital content to take advantage of the complementary strengths of both haptic interaction paradigms. We provided an overview of the design space covered by said combination, highlighting the potential for use of acoustically transparent tangible objects (ATTs) to allow focusing of ultrasound stimuli through them. Starting with the simplest possible tangible, a single surface, we experimentally investigated the viability of delivering UMH cues through ATTs, since these may attenuate incident sound waves and disrupt stimulus delivery. First, we investigated attenuation of a focal point focused through various plates of acoustically transparent materials. Second, we performed three human subject experiments investigating the impact of such acoustically transparent material plates on stimulus detection thresholds, discrimination of UMH stimulus motion, and focal point localization. Our work shows that even with widespread processes such as 3D printing, it is possible to design acoustically transparent tangible surfaces that can be effectively used in conjunction with UMH stimuli.

In future work, we will vary additional ATT surface properties, to study the physical interactions between tangible object surfaces and acoustic fields generated by UMH devices. We will then extend the present work to 3D objects composed of multiple acoustically transparent surfaces. Along with investigations into fabrication methods for ATTs, this will allow for fine-tuning the acoustic characteristics of tangible objects. Second, we will investigate this promising novel mixed haptic interaction paradigm in VR interaction scenarios using both tangible surfaces and 3D objects.

# REFERENCES

[1] O. Shaer et al., "Tangible user interfaces: past, present, and future directions," *Found. Trends Hum.-Comput. Interact.*, vol. 3, no. 1–2, pp. 4–137, 2010.

[2] A. Hettiarachchi et al., "Annexing reality: Enabling opportunistic use of everyday objects as tangible proxies in augmented reality," in *Proc. CHI Conf. Hum. Factors Comput. Syst.*, pp. 1957–1967, 2016.

[3] N. Nilsson et al., "Propping up virtual reality with haptic proxies," *IEEE Comput. Graph. Appl.*, vol. 41, no. 5, pp. 104–112, 2021.

[4] H. Culbertson et al., "Haptics: The Present and Future of Artificial Touch Sensation," *Annu. Rev. Control Robot. Auton. Syst.*, vol. 1, no. 1, pp. 385–409, 2018.

[5] S. J. Lederman et al., "Haptic perception: A tutorial," *Atten. Percept. Psychophys.*, vol. 71, no. 7, pp. 1439–1459, 2009.

[6] X. de Tinguy et al., "How different tangible and virtual objects can be while still feeling the same?," in *Proc. IEEE World Haptics Conf.*, pp. 580–585, 2019.

[7] I. Rakkolainen et al., "A survey of mid-air ultrasound haptics and its applications," *IEEE Trans. Haptics*, vol. 14, no. 1, pp. 2–19, 2020.

[8] E. Brown et al., "Augmenting automotive gesture infotainment interfaces through mid-air haptic icon design," in *Ultrasound Mid-Air Haptics for Touchless Interfaces*, pp. 119–145, Springer, 2022.

[9] T. Howard et al., "Ultrasound mid-air tactile feedback for immersive virtual reality interaction," in *Ultrasound Mid-Air Haptics for Touchless Interfaces*, pp. 147–183, Springer, 2022.

[10] I. Poupyrev et al., "Actuation and tangible user interfaces: the vaucanson duck, robots, and shape displays," in *Proc. ACM TEI Conf.*, pp. 205–212, 2007.

[11] A. Zenner and A. Krüger, "Shifty: A weight-shifting dynamic passive haptic proxy to enhance object perception in virtual reality," *IEEE Trans. Vis. Comput. Graph.*, vol. 23, no. 4, pp. 1285–1294, 2017.

[12] S. V. Salazar et al., "Altering the stiffness, friction, and shape perception of tangible objects in virtual reality using wearable haptics," *IEEE Trans. Haptics*, vol. 13, no. 1, pp. 167–174, 2020.

[13] D. Valkov, A. Mantler, and L. Linsen, "Haptic prop: A tangible prop for semi-passive haptic interaction," in *Proc. IEEE VR Conf.*, pp. 1744–1748, 2019.

[14] X. De Tinguy et al., "Weatavix: wearable actuated tangibles for virtual reality experiences," in *Proc. Eurohaptics Conf.*, pp. 262–270, 2020.

[15] L. Mulot et al., "Dolphin: A framework for the design and perceptual evaluation of ultrasound mid-air haptic stimuli," in *Proc. ACM Symp. Appl. Percept.*, pp. 1–10, 2021.

[16] T. Carter et al., "Ultrahaptics: multi-point mid-air haptic feedback for touch surfaces," in *Proc. ACM Symp. UIST*, pp. 505–514, 2013.

[17] G. Memoli et al., "Metamaterial bricks and quantization of meta-surfaces," *Nat. Commun.*, vol. 8, no. 1, pp. 1–8, 2017.

[18] Y. Abe et al., "Remote friction reduction on resonant film surface by airborne ultrasound," *IEEE Trans. Haptics*, vol. 14, no. 2, pp. 260–265, 2021.

[19] Y. Abe et al., "Remote friction reduction on polystyrene foam surface by focused airborne ultrasound," *IEEE Trans. Haptics*, vol. 15, no. 2, pp. 363–371, 2022.

[20] T. Ohmori, Y. Abe, M. Fujiwara, Y. Makino, and H. Shinoda, "Remote friction control on 3-dimensional object made of polystyrene foam using airborne ultrasound focus," in *Proc. CHI Conf. Hum. Fact. Comp. Syst.*, pp. 1–5, 2021.

[21] E. Freeman et al., "Enhancing physical objects with actuated levitating particles," in *Proc. ACM Int. Symp. Pervasive Displays*, pp. 1–7, 2019.

[22] L. E. Kinsler, A. R. Frey, A. B. Coppens, and J. V. Sanders, *Fundamentals of acoustics*. John wiley & sons, 2000.

[23] K. Ariga, M. Fujiwara, Y. Makino, and H. Shinoda, "Midair haptic presentation using concave reflector," in *Proc. Eurohaptics Conf.*, pp. 307–315, Springer, 2020.