# Augmenting the Texture Perception of Tangible Surfaces in Augmented Reality using Vibrotactile Haptics

Erwan Normand, Claudio Pacchierotti, Eric Marchand, Maud Marchal

► To cite this version:

Erwan Normand, Claudio Pacchierotti, Eric Marchand, Maud Marchal. Augmenting the Texture Per-
ception of Tangible Surfaces in Augmented Reality using Vibrotactile Haptics. Eurohaptics conference
2024, Jun 2024, Lille, France. pp.1-15. [hal-04554311v2](https://hal.science/hal-04554311v2)

HAL Id: hal-04554311

https://hal.science/hal-04554311v2

Submitted on 4 Jun 2024

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>
# Augmenting the Texture Perception of Tangible Surfaces in Augmented Reality using Vibrotactile Haptics

Erwan Normand<fnref n="1" />,<fnref n="2" />, Claudio Pacchierotti<fnref n="2" />,<fnref n="3" />, Eric Marchand<fnref n="1" />,<fnref n="3" />, and Maud Marchal<fnref n="3" />,<fnref n="4" />

**Abstract.** Wearable haptic devices are portable and unobtrusive technologies able to provide tactile sensations to the human skin. Their use in Augmented Reality (AR), where visual virtual content is integrated into the real world, has been little explored, especially for generating texture sensations. In this paper, we investigate the perception of simultaneous visual and haptic texture augmentation of real tangible surfaces touched directly with the fingertip in AR, using a wearable vibrotactile haptic device worn on the middle phalanx. When sliding on a tangible surface with an AR visual texture overlay, vibrations are generated based on data-driven texture models and finger speed to augment the haptic roughness perception of the surface. In a user study with twenty participants, we investigate the perception of the combination of nine representative pairs of visuo-haptic texture augmentations. Participants integrated roughness sensations from both visual and haptic modalities well, with haptics predominating the perception, and consistently identified and matched clusters of visual and haptic textures with similar perceived roughness.

**Keywords:** Augmented Reality · Wearable Haptics · Texture · Roughness.

## 1 Introduction

When we look at the surface of an everyday object, we then touch it to confirm or contrast our initial visual impression and to estimate the properties of the object [12]. One of the main characteristics of a textured surface is its roughness, i.e., the microgeometry of the material [17], which is perceived equally well and similarly by both sight and touch [4,3,32]. Many haptic devices and rendering methods have been used to generate realistic virtual rough textures [8]. One of the most common approaches is to reproduce the vibrations that occur when running across a surface, using a vibrotactile device attached to a hand-held tool [9,6] or worn on the finger [2,14]. By providing timely vibrations synchronized with the movement of the tool or the finger moving on a real object, the perceived roughness of the surface can be augmented [6,2]. In that sense,

<footnote n="1">¹ Univ Rennes, Inria, CNRS, IRISA, France
{erwan.normand,eric.marchand}@irisa.fr</footnote>

<footnote n="2">² CNRS, Univ Rennes, Inria, IRISA, France
claudio.pacchierotti@irisa.fr</footnote>

<footnote n="3">³ Univ Rennes, INSA Rennes, IRISA, Inria, CNRS, France
maud.marchal@irisa.fr</footnote>

<footnote n="4">⁴ Institut Universitaire de France (IUF), France</footnote>
# Augmenting the Texture Perception of Tangible Surfaces in Augmented Reality using Vibrotactile Haptics

Erwan Normand<fnref n="1" />,<fnref n="2" />, Claudio Pacchierotti<fnref n="2" />,<fnref n="3" />, Eric Marchand<fnref n="1" />,<fnref n="3" />, and Maud Marchal<fnref n="3" />,<fnref n="4" />

**Abstract.** Wearable haptic devices are portable and unobtrusive technologies able to provide tactile sensations to the human skin. Their use in Augmented Reality (AR), where visual virtual content is integrated into the real world, has been little explored, especially for generating texture sensations. In this paper, we investigate the perception of simultaneous visual and haptic texture augmentation of real tangible surfaces touched directly with the fingertip in AR, using a wearable vibrotactile haptic device worn on the middle phalanx. When sliding on a tangible surface with an AR visual texture overlay, vibrations are generated based on data-driven texture models and finger speed to augment the haptic roughness perception of the surface. In a user study with twenty participants, we investigate the perception of the combination of nine representative pairs of visuo-haptic texture augmentations. Participants integrated roughness sensations from both visual and haptic modalities well, with haptics predominating the perception, and consistently identified and matched clusters of visual and haptic textures with similar perceived roughness.

**Keywords:** Augmented Reality · Wearable Haptics · Texture · Roughness.

## 1 Introduction

When we look at the surface of an everyday object, we then touch it to confirm or contrast our initial visual impression and to estimate the properties of the object [12]. One of the main characteristics of a textured surface is its roughness, i.e., the microgeometry of the material [17], which is perceived equally well and similarly by both sight and touch [4,3,32]. Many haptic devices and rendering methods have been used to generate realistic virtual rough textures [8]. One of the most common approaches is to reproduce the vibrations that occur when running across a surface, using a vibrotactile device attached to a hand-held tool [9,6] or worn on the finger [2,14]. By providing timely vibrations synchronized with the movement of the tool or the finger moving on a real object, the perceived roughness of the surface can be augmented [6,2]. In that sense,

<footnote n="1">¹ Univ Rennes, Inria, CNRS, IRISA, France
{erwan.normand,eric.marchand}@irisa.fr</footnote>

<footnote n="2">² CNRS, Univ Rennes, Inria, IRISA, France
claudio.pacchierotti@irisa.fr</footnote>

<footnote n="3">³ Univ Rennes, INSA Rennes, IRISA, Inria, CNRS, France
maud.marchal@irisa.fr</footnote>

<footnote n="4">⁴ Institut Universitaire de France (IUF), France</footnote>
data-driven haptic textures have been developed as captures and models of real sur-
faces, resulting in the Penn Haptic Texture Toolkit (HaTT) database [7]. While these
virtual haptic textures are perceived as similar to real textures [6], they have been eval-
uated using hand-held tools and not yet in a direct finger contact with the surface con-
text, in particular combined with visual textures in an immersive virtual environment.

Combined with virtual reality (VR), where the user is immersed in a visual virtual environment, wearable haptic devices have also proven to be effective in modifying the visuo-haptic perception of tangible objects touched with the finger, without needing to modify the object [1,2,29]. Worn on the finger, but not directly on the fingertip to keep it free to interact with tangible objects, they have been used to alter perceived stiffness, softness, friction and local deformations [10,29]. However, the use of wearable haptic devices has been little explored in Augmented Reality (AR), where visual virtual content is integrated into the real-world environment, especially for augmenting texture sensations [27,20,22,5,31,13,23]. A key difference in AR compared to VR is that the user can still see the real-world surroundings, including their hands, the augmented tangible objects and the worn haptic devices. One additional issue of current AR systems is their visual display limitations, or virtual content that may not be seen as consistent with the real world [16,19]. These two factors have been shown to influence the perception of haptic stiffness rendering [18,15]. It remains to be investigated whether simultaneous and co-localized visual and haptic texture augmentation of tangible surfaces in AR can be perceived in a coherent and realistic manner, and to what extent each sensory modality would contribute to the overall perception of the augmented texture. Being able to coherently substitute the visuo-haptic texture of an everyday surface directly touched by a finger is an important step towards new AR applications capable of visually and haptically augmenting the real environment of a user in a plausible way.

In this paper, we investigate how users perceive a tangible surface touched with
the index finger when it is augmented with a visuo-haptic roughness texture using
immersive optical see-through AR (OST-AR) and wearable vibrotactile stimuli pro-
vided on the index. In a user study, twenty participants freely explored and evaluated
the coherence, realism and roughness of the combination of nine representative pairs
of visuo-haptic texture augmentations (see Fig. 1, left) from the HaTT database [7].

## 2 User Study

The user study aimed at analyzing the user perception of tangible surfaces when augmented through a visuo-haptic texture using AR and vibrotactile haptic feedback provided on the finger touching the surfaces. Nine representative visuo-haptic texture pairs from the HaTT database [7] were investigated in two tasks: (1) a matching task, where participants had to find the haptic texture that best matched a given visual texture; and (2) a ranking task, where participants had to rank only the haptic textures, only the visual textures, and the visuo-haptic texture pairs according to their perceived roughness. Our objective is to assess which haptic textures were associated with which visual textures, how the roughness of the visual and haptic textures are perceived, and whether the perceived roughness can explain the matches made between them.

**The textures.** The 100 visuo-haptic texture pairs of the HaTT database [7] were preliminary tested and compared using AR and vibrotactile haptic feedback on the
data-driven haptic textures have been developed as captures and models of real sur-
faces, resulting in the Penn Haptic Texture Toolkit (HaTT) database [7]. While these
virtual haptic textures are perceived as similar to real textures [6], they have been eval-
uated using hand-held tools and not yet in a direct finger contact with the surface con-
text, in particular combined with visual textures in an immersive virtual environment.

Combined with virtual reality (VR), where the user is immersed in a visual virtual environment, wearable haptic devices have also proven to be effective in modifying the visuo-haptic perception of tangible objects touched with the finger, without needing to modify the object [1,2,29]. Worn on the finger, but not directly on the fingertip to keep it free to interact with tangible objects, they have been used to alter perceived stiffness, softness, friction and local deformations [10,29]. However, the use of wearable haptic devices has been little explored in Augmented Reality (AR), where visual virtual content is integrated into the real-world environment, especially for augmenting texture sensations [27,20,22,5,31,13,23]. A key difference in AR compared to VR is that the user can still see the real-world surroundings, including their hands, the augmented tangible objects and the worn haptic devices. One additional issue of current AR systems is their visual display limitations, or virtual content that may not be seen as consistent with the real world [16,19]. These two factors have been shown to influence the perception of haptic stiffness rendering [18,15]. It remains to be investigated whether simultaneous and co-localized visual and haptic texture augmentation of tangible surfaces in AR can be perceived in a coherent and realistic manner, and to what extent each sensory modality would contribute to the overall perception of the augmented texture. Being able to coherently substitute the visuo-haptic texture of an everyday surface directly touched by a finger is an important step towards new AR applications capable of visually and haptically augmenting the real environment of a user in a plausible way.

In this paper, we investigate how users perceive a tangible surface touched with
the index finger when it is augmented with a visuo-haptic roughness texture using
immersive optical see-through AR (OST-AR) and wearable vibrotactile stimuli pro-
vided on the index. In a user study, twenty participants freely explored and evaluated
the coherence, realism and roughness of the combination of nine representative pairs
of visuo-haptic texture augmentations (see Fig. 1, left) from the HaTT database [7].

## 2 User Study

The user study aimed at analyzing the user perception of tangible surfaces when augmented through a visuo-haptic texture using AR and vibrotactile haptic feedback provided on the finger touching the surfaces. Nine representative visuo-haptic texture pairs from the HaTT database [7] were investigated in two tasks: (1) a matching task, where participants had to find the haptic texture that best matched a given visual texture; and (2) a ranking task, where participants had to rank only the haptic textures, only the visual textures, and the visuo-haptic texture pairs according to their perceived roughness. Our objective is to assess which haptic textures were associated with which visual textures, how the roughness of the visual and haptic textures are perceived, and whether the perceived roughness can explain the matches made between them.

**The textures.** The 100 visuo-haptic texture pairs of the HaTT database [7] were preliminary tested and compared using AR and vibrotactile haptic feedback on the
<figure><img src="image_1.png" /><figcaption>**Fig. 1.** (Left) The nine visuo-haptic textures used in the user study, selected from the HaTT database [7]. The texture names were never shown, so as to prevent the use of the user's visual or haptic memory of the textures. (Middle) Experimental setup. Participant sat in front of the tangible surfaces, which were augmented with visual textures displayed by the HoloLens 2 AR headset and haptic roughness textures rendered by the vibrotactile haptic device placed on the middle index phalanx. A webcam above the surfaces tracked the finger movements. (Right) First person view of the user study, as seen through the immersive AR headset HoloLens 2. The visual texture overlays are statically displayed on the surfaces, allowing the user to move around to view them from different angles. The haptic roughness texture is generated based on HaTT data-driven texture models and finger speed, and it is rendered on the middle index phalanx as it slides on the considered surface.</figcaption></figure>

finger on a tangible surface. These texture models were chosen as they are visuo-haptic representations of a wide range of real textures that are publicly available online. Nine texture pairs were selected (see Fig. 1, left) to cover various perceived roughness, from rough to smooth, as listed: Metal Mesh, Sandpaper 100, Brick 2, Cork, Sandpaper 320, Velcro Hooks, Plastic Mesh 1, Terra Cotta, Coffee Filter. All these visual and haptic textures are isotropic: their rendering (appearance or roughness) is the same whatever the direction of the movement on the surface, i.e., there are no local deformations (holes, bumps, or breaks).

**Apparatus.** Fig. 1 shows the experimental setup (middle) and the first person view (right) of the user study. Nine 5-cm square cardboards with smooth, white melamine surface, arranged in a 3 × 3 grid, were used as real tangible surfaces to augment. Their poses were estimated with three 2-cm-square AprilTag fiducial markers glued on the surfaces grid. Similarly, a 2-cm-square fiducial marker was glued on top of the vibrotactile actuator to detect the finger pose. Positioned 20 cm above the surfaces, a webcam (StreamCam, Logitech) filmed the markers to track finger movements relative to the surfaces. The visual textures were displayed on the tangible surfaces using the HoloLens 2 OST-AR headset (see Fig. 1, middle and right) within a 43° × 29° field of view at 60 Hz; a set of empirical tests enabled us to choose the best rendering characteristics in terms of transparency and brightness for the visual textures, that were used throughout the user study. When a haptic texture was touched, a 48 kHz audio signal was generated using the corresponding HaTT haptic texture model and the measured tangential speed of the finger, using the rendering procedure described in Culbertson et al. [9]. The normal force on the texture was assumed to be constant at 1.2 N to generate the audio signal from the model, as Culbertson et al. [6], who found that the
<figure><img src="image_2.png" /><figcaption>**Fig. 1.** (Left) The nine visuo-haptic textures used in the user study, selected from the HaTT database [7]. The texture names were never shown, so as to prevent the use of the user's visual or haptic memory of the textures. (Middle) Experimental setup. Participant sat in front of the tangible surfaces, which were augmented with visual textures displayed by the HoloLens 2 AR headset and haptic roughness textures rendered by the vibrotactile haptic device placed on the middle index phalanx. A webcam above the surfaces tracked the finger movements. (Right) First person view of the user study, as seen through the immersive AR headset HoloLens 2. The visual texture overlays are statically displayed on the surfaces, allowing the user to move around to view them from different angles. The haptic roughness texture is generated based on HaTT data-driven texture models and finger speed, and it is rendered on the middle index phalanx as it slides on the considered surface.</figcaption></figure>

finger on a tangible surface. These texture models were chosen as they are visuo-haptic representations of a wide range of real textures that are publicly available online. Nine texture pairs were selected (see Fig. 1, left) to cover various perceived roughness, from rough to smooth, as listed: Metal Mesh, Sandpaper 100, Brick 2, Cork, Sandpaper 320, Velcro Hooks, Plastic Mesh 1, Terra Cotta, Coffee Filter. All these visual and haptic textures are isotropic: their rendering (appearance or roughness) is the same whatever the direction of the movement on the surface, i.e., there are no local deformations (holes, bumps, or breaks).

**Apparatus.** Fig. 1 shows the experimental setup (middle) and the first person view (right) of the user study. Nine 5-cm square cardboards with smooth, white melamine surface, arranged in a 3 × 3 grid, were used as real tangible surfaces to augment. Their poses were estimated with three 2-cm-square AprilTag fiducial markers glued on the surfaces grid. Similarly, a 2-cm-square fiducial marker was glued on top of the vibrotactile actuator to detect the finger pose. Positioned 20 cm above the surfaces, a webcam (StreamCam, Logitech) filmed the markers to track finger movements relative to the surfaces. The visual textures were displayed on the tangible surfaces using the HoloLens 2 OST-AR headset (see Fig. 1, middle and right) within a 43° × 29° field of view at 60 Hz; a set of empirical tests enabled us to choose the best rendering characteristics in terms of transparency and brightness for the visual textures, that were used throughout the user study. When a haptic texture was touched, a 48 kHz audio signal was generated using the corresponding HaTT haptic texture model and the measured tangential speed of the finger, using the rendering procedure described in Culbertson et al. [9]. The normal force on the texture was assumed to be constant at 1.2 N to generate the audio signal from the model, as Culbertson et al. [6], who found that the
HaTT textures can be rendered using only the speed as input without decreasing their perceived realism. An amplifier (XY-502, not branded) converted this audio signal to a current transmitted to the vibrotactile voice-coil actuator (HapCoil-One, Actronika), that was encased in a 3D-printed plastic shell firmly attached to the middle index phalanx of the participant's dominant hand, similarly to previous studies [2,14]. This voice-coil actuator was chosen for its wide frequency range (10 Hz to 1000 Hz) and its relatively low acceleration distortion, as specified by the manufacturer[5]. Overall latency was measured to (46 ± 6) ms, as a result of latency in image acquisition (16±1) ms, fiducial marker detection (8±3) ms, network synchronization (4±1) ms, audio sampling (3±1) ms, and the vibrotactile actuator latency (15 ms, as specified by the manufacturer[5]). This latency was below the 60 ms threshold for vibrotactile feedback [24] and was not noticed by the participants. The user study was held in a quiet room with no windows, with one light source of 800 lm placed 70 cm above the table.

**Procedure and Collected Data.** Participants were first given written instructions about the experimental setup, the tasks, and the procedure of the user study. Then, after having signed an informed consent form, they were asked to seat in front of the table with the experimental setup and to wear the HoloLens 2 AR headset. The experimenter firmly attached the plastic shell encasing the vibrotactile actuator to the middle index phalanx of their dominant hand. As the haptic device generated no audible noise, participants did not wear any noise reduction headphones. A calibration of both the HoloLens 2 and the hand tracking was performed to ensure the correct alignment of the visual and haptic textures on the tangible surfaces. Finally, participants familiarized with the augmented surface in a 2-min training session with textures different from the ones used in the user study.

Participants started with the *matching task*. They were informed that the user study involved nine pairs of corresponding visual and haptic textures that were separated and shuffled. On each trial, the same visual texture was displayed on the nine tangible surfaces, while the nine haptic textures were rendered on only one of the surfaces at a time, *i.e.*, all surfaces were augmented by the same visual texture, but each surface was augmented by a different haptic texture. The placement of the haptic textures was randomized before each trial. Participants were instructed to look closely at the details of the visual textures and explore the haptic textures with a constant pressure and various speeds to find the haptic texture that best matched the visual texture, *i.e.*, choose the surface with the most coherent visual-haptic texture pair. The texture names were never given or shown to prevent the use of visual or haptic memory of the textures, nor a definition of what roughness is was given, so as to let participants complete the task as naturally as possible, similarly to Bergmann Tiest *et al.* [4].

Then, participants performed the *ranking task*, employing the same setup as the matching task and the same 9 textures. In this case, participants were asked to rank the textures according to their perceived roughness. First, they ranked all the haptic textures (without any visual augmentation given), then all the visual textures (without any haptic augmentation given), and finally all the visuo-haptic texture pairs together, being informed that they were the correct matches as per the original HaTT database. The placement of the textures was also randomized before each trial.

<footnote n="5">5 https://www.actronika.com/haptic-solutions</footnote>
HaTT textures can be rendered using only the speed as input without decreasing their perceived realism. An amplifier (XY-502, not branded) converted this audio signal to a current transmitted to the vibrotactile voice-coil actuator (HapCoil-One, Actronika), that was encased in a 3D-printed plastic shell firmly attached to the middle index phalanx of the participant's dominant hand, similarly to previous studies [2,14]. This voice-coil actuator was chosen for its wide frequency range (10 Hz to 1000 Hz) and its relatively low acceleration distortion, as specified by the manufacturer[5]. Overall latency was measured to (46 ± 6) ms, as a result of latency in image acquisition (16 ± 1) ms, fiducial marker detection (8 ± 3) ms, network synchronization (4 ± 1) ms, audio sampling (3 ± 1) ms, and the vibrotactile actuator latency (15 ms, as specified by the manufacturer[5]). This latency was below the 60 ms threshold for vibrotactile feedback [24] and was not noticed by the participants. The user study was held in a quiet room with no windows, with one light source of 800 lm placed 70 cm above the table.

**Procedure and Collected Data.** Participants were first given written instructions about the experimental setup, the tasks, and the procedure of the user study. Then, after having signed an informed consent form, they were asked to seat in front of the table with the experimental setup and to wear the HoloLens 2 AR headset. The experimenter firmly attached the plastic shell encasing the vibrotactile actuator to the middle index phalanx of their dominant hand. As the haptic device generated no audible noise, participants did not wear any noise reduction headphones. A calibration of both the HoloLens 2 and the hand tracking was performed to ensure the correct alignment of the visual and haptic textures on the tangible surfaces. Finally, participants familiarized with the augmented surface in a 2-min training session with textures different from the ones used in the user study.

Participants started with the *matching task*. They were informed that the user study involved nine pairs of corresponding visual and haptic textures that were separated and shuffled. On each trial, the same visual texture was displayed on the nine tangible surfaces, while the nine haptic textures were rendered on only one of the surfaces at a time, i.e., all surfaces were augmented by the same visual texture, but each surface was augmented by a different haptic texture. The placement of the haptic textures was randomized before each trial. Participants were instructed to look closely at the details of the visual textures and explore the haptic textures with a constant pressure and various speeds to find the haptic texture that best matched the visual texture, i.e., choose the surface with the most coherent visual-haptic texture pair. The texture names were never given or shown to prevent the use of visual or haptic memory of the textures, nor a definition of what roughness is was given, so as to let participants complete the task as naturally as possible, similarly to Bergmann Tiest *et al.* [4].

Then, participants performed the *ranking task*, employing the same setup as the matching task and the same 9 textures. In this case, participants were asked to rank the textures according to their perceived roughness. First, they ranked all the haptic textures (without any visual augmentation given), then all the visual textures (without any haptic augmentation given), and finally all the visuo-haptic texture pairs together, being informed that they were the correct matches as per the original HaTT database. The placement of the textures was also randomized before each trial.

<footnote n="5">5 https://www.actronika.com/haptic-solutions</footnote>
After each task, participants answered to the following 7-item Likert scale questions (1=Not at all, 7=Extremely): (Haptic Difficulty) How difficult was it to differentiate the tactile textures? (Visual Difficulty) How difficult was it to differentiate the visual textures? (Textures Match) For the visual-tactile pairs you have chosen, how coherent were the tactile textures with the corresponding visual textures? (Haptic Realism) How realistic were the tactile textures? (Visual Realism) How realistic were the visual textures? (Uncomfort) How uncomfortable was to use the haptic device? In an open question, participants commented also on their strategy for completing the matching task (How did you associate the tactile textures with the visual textures?) and the ranking task (How did you rank the textures?). The user study took on average 1 hour to complete.

**Participants.** Twenty participants took part to the user study (12 males, 7 females, 1 preferred not to say), aged between 20 and 60 years old (M=29.1, SD=9.4). One participant was left-handed, all others were right-handed; they all performed the user study with their dominant hand. All participants had normal or corrected-to-normal vision and none of them had a known hand or finger impairment. They rated their experience with haptics, AR, and VR ("I use it every month or more"); 10 were experienced with haptics, 2 with AR, and 10 with VR. Experiences were correlated between haptics and AR ($r_s$=0.53), haptics and VR ($r_s$=0.61), and AR and VR ($r_s$=0.74); but not with age ($r_s$=-0.06 to $r_s$=-0.05) or gender ($r_s$=0.10 to $r_s$=0.27). Participants were recruited at the university on a voluntary basis. They all signed an informed consent form before the user study.

**Design.** The matching task was a single-factor within-subjects design, *Visual Texture*, with the following levels: Metal Mesh, Sandpaper 100, Brick 2, Cork, Sandpaper 320, Velcro Hooks, Plastic Mesh 1, Terra Cotta, Coffee Filter. To account for learning and fatigue effects, the order of *Visual Texture* was counterbalanced using a balanced 18×18 Latin square design. A total of 20 participants × 9 textures × 3 repetitions = 540 matching trials were collected. The ranking task was a single-factor within-subjects design, *Modality*, with the following levels: Visual, Haptic, Visuo-Haptic. The order of *Modality* was fixed as listed above. A total of 20 participants × 3 modalities = 60 ranking trials were collected.

# 3 Results

## 3.1 Textures Matching

**Confusion Matrix.** Fig. 2 (left) shows the confusion matrix of the matching task with the visual textures and the proportion of haptic texture selected in response, i.e., the proportion of times the corresponding haptic texture was selected in response to the presentation of the corresponding visual texture. A two-sample Pearson Chi-Squared test ($\chi^2(64, N=540)=420, p<0.001$) and Holm-Bonferroni adjusted binomial tests indicated that the following (Visual Texture, Haptic Texture) pairs have proportion selections statistically significantly higher than chance (i.e., 11% each): (Sandpaper 320, Coffee Filter), (Terra Cotta, Coffee Filter), and (Coffee Filter, Coffee Filter) ($p<0.001$ each); (Cork, Sandpaper 320), (Brick 2, Plastic Mesh 1), (Brick 2,
After each task, participants answered to the following 7-item Likert scale questions (1=Not at all, 7=Extremely): (Haptic Difficulty) How difficult was it to differentiate the tactile textures? (Visual Difficulty) How difficult was it to differentiate the visual textures? (Textures Match) For the visual-tactile pairs you have chosen, how coherent were the tactile textures with the corresponding visual textures? (Haptic Realism) How realistic were the tactile textures? (Visual Realism) How realistic were the visual textures? (Uncomfort) How uncomfortable was to use the haptic device? In an open question, participants commented also on their strategy for completing the matching task (How did you associate the tactile textures with the visual textures?) and the ranking task (How did you rank the textures?). The user study took on average 1 hour to complete.

**Participants.** Twenty participants took part to the user study (12 males, 7 females, 1 preferred not to say), aged between 20 and 60 years old (M=29.1, SD=9.4). One participant was left-handed, all others were right-handed; they all performed the user study with their dominant hand. All participants had normal or corrected-to-normal vision and none of them had a known hand or finger impairment. They rated their experience with haptics, AR, and VR ("I use it every month or more"); 10 were experienced with haptics, 2 with AR, and 10 with VR. Experiences were correlated between haptics and AR ($r_s$=0.53), haptics and VR ($r_s$=0.61), and AR and VR ($r_s$=0.74); but not with age ($r_s$=-0.06 to $r_s$=-0.05) or gender ($r_s$=0.10 to $r_s$=0.27). Participants were recruited at the university on a voluntary basis. They all signed an informed consent form before the user study.

**Design.** The matching task was a single-factor within-subjects design, *Visual Texture*, with the following levels: Metal Mesh, Sandpaper 100, Brick 2, Cork, Sandpaper 320, Velcro Hooks, Plastic Mesh 1, Terra Cotta, Coffee Filter. To account for learning and fatigue effects, the order of *Visual Texture* was counterbalanced using a balanced 18×18 Latin square design. A total of 20 participants × 9 textures × 3 repetitions = 540 matching trials were collected. The ranking task was a single-factor within-subjects design, *Modality*, with the following levels: Visual, Haptic, Visuo-Haptic. The order of *Modality* was fixed as listed above. A total of 20 participants × 3 modalities = 60 ranking trials were collected.

# 3 Results

## 3.1 Textures Matching

**Confusion Matrix.** Fig. 2 (left) shows the confusion matrix of the matching task with the visual textures and the proportion of haptic texture selected in response, i.e., the proportion of times the corresponding haptic texture was selected in response to the presentation of the corresponding visual texture. A two-sample Pearson Chi-Squared test ($\chi^2(64, N=540)=420$, p<0.001) and Holm-Bonferroni adjusted binomial tests indicated that the following (Visual Texture, Haptic Texture) pairs have proportion selections statistically significantly higher than chance (i.e., 11% each): (Sandpaper 320, Coffee Filter), (Terra Cotta, Coffee Filter), and (Coffee Filter, Coffee Filter) (p<0.001 each); (Cork, Sandpaper 320), (Brick 2, Plastic Mesh 1), (Brick 2,
<figure><img src="image_2.png" /><figcaption>**Fig. 2.** (Left) Confusion matrix of the matching task, with the presented visual textures as columns and the selected haptic texture in proportion as rows. The number in a cell is the proportion of times the corresponding haptic texture was selected in response to the presentation of the corresponding visual texture. The diagonal represents the expected correct answers. Holm-Bonferroni adjusted binomial test results are marked in bold when the proportion is higher than chance (i.e., more than 11 %, p<0.05). (Right) Means with bootstrap 95 % confidence interval of the three rankings of the haptic textures alone, the visual textures alone, and the visuo-haptic texture pairs. A lower rank means that the texture was considered rougher, a higher rank means smoother.</figcaption></figure>

Sandpaper 320), (Plastic Mesh 1, Sandpaper 320), and (Sandpaper 320, Plastic Mesh 1) (p<0.01); and (Metal Mesh, Cork), (Cork, Velcro Hooks), (Velcro Hooks, Plastic Mesh 1), (Velcro Hooks, Sandpaper 320), and (Coffee Filter, Terra Cotta) (p<0.05 each). Except for one visual texture (Sandpaper 100) and 4 haptic textures (Metal Mesh, Sandpaper 100, Brick 2, and Terra Cotta), all haptic and visual textures were matched statistically significantly higher than chance with at least one visual and haptic texture, respectively. However, many mistakes were made: the expected haptic texture was selected on average only 20 % of the time for five of the visual textures, and even around 5 % for (visual) Sandpaper 100, Brick 2, and Sandpaper 320. Only haptic Coffee Filter was correctly selected 59 % of the time, and was also particularly matched with the visual Sandpaper 320 and Terra Cotta (around 45 % each). Similarly, the haptic textures Sandpaper 320 and Plastic Mesh 1 were also selected for four and three visual textures, respectively (around 25 % each). Additionally, the Spearman correlations between the trials were computed for each participant and only 21 out of 60 were statistically significant (p<0.05), with a mean r<sub>s</sub>=0.52 (CI<sub>95%</sub>[0.43,0.59]).

These results indicate that the participants hesitated between several haptic
textures for a given visual texture, as also reported in several comments, some haptic
textures being more favored while some others were almost not selected at all. Another
explanation could be that the participants had difficulties to estimate the roughness
of the visual textures. Indeed, many participants explained that they tried to identify
or imagine the roughness of a given visual texture then to select the most plausible
haptic texture, in terms of frequency and/or amplitude of vibrations.
<figure><img src="image_3.png" /><figcaption>**Fig. 2.** (Left) Confusion matrix of the matching task, with the presented visual textures as columns and the selected haptic texture in proportion as rows. The number in a cell is the proportion of times the corresponding haptic texture was selected in response to the presentation of the corresponding visual texture. The diagonal represents the expected correct answers. Holm-Bonferroni adjusted binomial test results are marked in bold when the proportion is higher than chance (i.e., more than 11%, p<0.05). (Right) Means with bootstrap 95% confidence interval of the three rankings of the haptic textures alone, the visual textures alone, and the visuo-haptic texture pairs. A lower rank means that the texture was considered rougher, a higher rank means smoother.</figcaption></figure>

Sandpaper 320), (Plastic Mesh 1, Sandpaper 320), and (Sandpaper 320, Plastic Mesh 1) (p<0.01); and (Metal Mesh, Cork), (Cork, Velcro Hooks), (Velcro Hooks, Plastic Mesh 1), (Velcro Hooks, Sandpaper 320), and (Coffee Filter, Terra Cotta) (p<0.05 each). Except for one visual texture (Sandpaper 100) and 4 haptic textures (Metal Mesh, Sandpaper 100, Brick 2, and Terra Cotta), all haptic and visual textures were matched statistically significantly higher than chance with at least one visual and haptic texture, respectively. However, many mistakes were made: the expected haptic texture was selected on average only 20% of the time for five of the visual textures, and even around 5% for (visual) Sandpaper 100, Brick 2, and Sandpaper 320. Only haptic Coffee Filter was correctly selected 59% of the time, and was also particularly matched with the visual Sandpaper 320 and Terra Cotta (around 45% each). Similarly, the haptic textures Sandpaper 320 and Plastic Mesh 1 were also selected for four and three visual textures, respectively (around 25% each). Additionally, the Spearman correlations between the trials were computed for each participant and only 21 out of 60 were statistically significant (p<0.05), with a mean r<sub>s</sub>=0.52 (CI<sub>95%</sub>[0.43,0.59]).

These results indicate that the participants hesitated between several haptic
textures for a given visual texture, as also reported in several comments, some haptic
textures being more favored while some others were almost not selected at all. Another
explanation could be that the participants had difficulties to estimate the roughness
of the visual textures. Indeed, many participants explained that they tried to identify
or imagine the roughness of a given visual texture then to select the most plausible
haptic texture, in terms of frequency and/or amplitude of vibrations.
**Completion Time.** To verify that the difficulty with all the visual textures was the same on the matching task, the *Completion Time* of a trial, i.e., the time between the visual texture display and the haptic texture selection, was analyzed. As the *Completion Time* results were Gamma distributed, they were transformed with a log to approximate a normal distribution. A linear mixed model (LMM) on the log *Completion Time* with the *Visual Texture* as fixed effect and the *Participant* as random intercept was performed. Normality was verified with a QQ-plot of the model residuals. No statistical significant effect of *Visual Texture* was found (F(8,512)=1.9, p=0.06) on *Completion Time* (GM=44s, CI<sub>95%</sub>[42,46]), indicating an equal difficulty and participant behaviour for all the visual textures.

## 3.2 Textures Ranking

Fig. 2 (right) presents the results of the three rankings of the haptic textures alone, the visual textures alone, and the visuo-haptic texture pairs. Almost all the texture pairs in the *Haptic Textures Ranking* results were statistically significantly different ($\chi^2(8,N=20)=146$, p<0.001; p<0.05 for each comparison), except between (Metal Mesh, Sandpaper 100), (Cork, Brick 2), (Cork, Sandpaper 320) (Plastic Mesh 1, Velcro Hooks), and (Plastic Mesh 1, Terra Cotta). Average Kendall's Tau correlations between the participants indicated a very high consensus ($\tau=0.82$, CI<sub>95%</sub>[0.81,0.84]) showing that participants perceived similarly the roughness of the haptic textures. Most of the texture pairs in the *Visual Textures Ranking* results were also statistically significantly different ($\chi^2(8,N=20)=119$, p<0.001; p<0.05 for each comparison), except for the following groups: {Metal Mesh, Cork, Plastic Mesh 1}; {Sandpaper 100, Brick 2, Plastic Mesh 1, Velcro Hooks}; {Cork, Velcro Hooks}; {Sandpaper 320, Terra Cotta}; and {Sandpaper 320, Coffee Filter}. Even though the consensus was high ($\tau=0.61$, CI<sub>95%</sub>[0.58,0.64]), the roughness of the visual textures were more difficult to estimate, in particular for Plastic Mesh 1 and Velcro Hooks. Also, almost all the texture pairs in the *Visuo-Haptic Textures Ranking* results were statistically significantly different ($\chi^2(8,N=20)=140$, p<0.001; p<0.05 for each comparison), except for the following groups: {Sandpaper 100, Cork}; {Cork, Brick 2}; and {Plastic Mesh 1, Velcro Hooks, Sandpaper 320}. The consensus between the participants was also very high $\tau=0.77$, CI<sub>95%</sub>[0.74,0.79]. Finally, calculating the similarity of the three rankings of each participant, the *Visuo-Haptic Textures Ranking* was on average highly similar to the *Haptic Textures Ranking* ($\tau=0.79$, CI<sub>95%</sub>[0.72,0.86]) and moderately to the *Visual Textures Ranking* ($\tau=0.48$, CI<sub>95%</sub>[0.39,0.56]). A Wilcoxon signed-rank test indicated that this difference was statistically significant (W=190, p=0.002). These results indicate, with Fig. 2 (right), that the two haptic and visual modalities were integrated together, the resulting roughness ranking being between the two rankings of the modalities alone, but with haptics predominating.

## 3.3 Perceived Similarity of Visual and Haptic Textures

The high level of agreement between participants on the three haptic, visual and visuo-haptic rankings (see Sec. 3.2), as well as the similarity of the within-participant rankings, suggests that participants perceived the roughness of the textures similarly,
**Completion Time.** To verify that the difficulty with all the visual textures was the same on the matching task, the *Completion Time* of a trial, i.e., the time between the visual texture display and the haptic texture selection, was analyzed. As the *Completion Time* results were Gamma distributed, they were transformed with a log to approximate a normal distribution. A linear mixed model (LMM) on the log *Completion Time* with the *Visual Texture* as fixed effect and the *Participant* as random intercept was performed. Normality was verified with a QQ-plot of the model residuals. No statistical significant effect of *Visual Texture* was found (F(8,512)=1.9, p=0.06) on *Completion Time* (GM=44s, CI<sub>95%</sub>[42,46]), indicating an equal difficulty and participant behaviour for all the visual textures.

## 3.2 Textures Ranking

Fig. 2 (right) presents the results of the three rankings of the haptic textures alone, the visual textures alone, and the visuo-haptic texture pairs. Almost all the texture pairs in the *Haptic Textures Ranking* results were statistically significantly different ($\chi^2(8,N=20)=146$, p<0.001; p<0.05 for each comparison), except between (Metal Mesh, Sandpaper 100), (Cork, Brick 2), (Cork, Sandpaper 320) (Plastic Mesh 1, Velcro Hooks), and (Plastic Mesh 1, Terra Cotta). Average Kendall's Tau correlations between the participants indicated a very high consensus ($\tau=0.82$, CI<sub>95%</sub>[0.81,0.84]) showing that participants perceived similarly the roughness of the haptic textures. Most of the texture pairs in the *Visual Textures Ranking* results were also statistically significantly different ($\chi^2(8,N=20)=119$, p<0.001; p<0.05 for each comparison), except for the following groups: {Metal Mesh, Cork, Plastic Mesh 1}; {Sandpaper 100, Brick 2, Plastic Mesh 1, Velcro Hooks}; {Cork, Velcro Hooks}; {Sandpaper 320, Terra Cotta}; and {Sandpaper 320, Coffee Filter}. Even though the consensus was high ($\tau=0.61$, CI<sub>95%</sub>[0.58,0.64]), the roughness of the visual textures were more difficult to estimate, in particular for Plastic Mesh 1 and Velcro Hooks. Also, almost all the texture pairs in the *Visuo-Haptic Textures Ranking* results were statistically significantly different ($\chi^2(8,N=20)=140$, p<0.001; p<0.05 for each comparison), except for the following groups: {Sandpaper 100, Cork}; {Cork, Brick 2}; and {Plastic Mesh 1, Velcro Hooks, Sandpaper 320}. The consensus between the participants was also very high $\tau=0.77$, CI<sub>95%</sub>[0.74,0.79]. Finally, calculating the similarity of the three rankings of each participant, the *Visuo-Haptic Textures Ranking* was on average highly similar to the *Haptic Textures Ranking* ($\tau=0.79$, CI<sub>95%</sub>[0.72,0.86]) and moderately to the *Visual Textures Ranking* ($\tau=0.48$, CI<sub>95%</sub>[0.39,0.56]). A Wilcoxon signed-rank test indicated that this difference was statistically significant (W=190, p=0.002). These results indicate, with Fig. 2 (right), that the two haptic and visual modalities were integrated together, the resulting roughness ranking being between the two rankings of the modalities alone, but with haptics predominating.

## 3.3 Perceived Similarity of Visual and Haptic Textures

The high level of agreement between participants on the three haptic, visual and visuo-haptic rankings (see Sec. 3.2), as well as the similarity of the within-participant rankings, suggests that participants perceived the roughness of the textures similarly,
<figure><img src="image_3.png" /><figcaption>**Fig. 3.** (Left) Correspondence analysis of the matching task confusion matrix (see Fig. 2, left). The visual textures are represented as blue squares, the haptic textures as red circles. The closer the textures are, the more similar they were judged. The first dimension (horizontal axis) explains 60 % of the variance, the second dimension (vertical axis) explains 30 % of the variance. (Right) Dendrograms of the hierarchical clusterings of the haptic textures (left) and visual textures (right) of the matching task confusion matrix (see Fig. 2, left), using Euclidian distance and Ward's method. The height of the dendrograms represents the distance between the clusters.</figcaption></figure>

but differed in their strategies for matching the haptic and visual textures in the matching task (see Sec. 3.1). To further investigate the perceived similarity of the haptic and visual textures and to identify groups of textures that were perceived as similar on the matching task, a correspondence analysis and a hierarchical clustering were performed on the matching task confusion matrix (see Fig. 2, left).

The correspondence analysis captured 60 % and 29 % of the variance in the first and second dimensions, respectively, with the remaining dimensions each accounting for less than 5 % each. Fig. 3 (left) shows the first two dimensions with the 18 haptic and visual textures. The first dimension was similar to the rankings (see Fig. 2, right), distributing the textures according to their perceived roughness. It seems that the second dimension opposed textures that were perceived as hard with those perceived as softer, as also reported by participants. Stiffness is indeed an important perceptual dimension of a material [25,9].

Fig. 3 (right) shows the dendrograms of the two hierarchical clusterings of the haptic and visual textures, constructed using the Euclidean distance and the Ward's method on squared distance. The four identified haptic texture clusters were: "Roughest" {Metal Mesh, Sandpaper 100, Brick 2, Cork}; "Rougher" {Sandpaper 320, Velcro Hooks}; "Smoother" {Plastic Mesh 1, Terra Cotta}; "Smoothest" {Coffee Filter} (see Fig. 3, top-right). Similar to the haptic ranks (see Fig. 2, right), the clusters could have been named according to their perceived roughness. It also shows that the participants compared and ranked the haptic textures during the matching task to select the one that best matched the given visual texture. The five identified visual
<figure><img src="image_4.png" /><figcaption>**Fig. 3.** (Left) Correspondence analysis of the matching task confusion matrix (see Fig. 2, left). The visual textures are represented as blue squares, the haptic textures as red circles. The closer the textures are, the more similar they were judged. The first dimension (horizontal axis) explains 60 % of the variance, the second dimension (vertical axis) explains 30 % of the variance. (Right) Dendrograms of the hierarchical clusterings of the haptic textures (left) and visual textures (right) of the matching task confusion matrix (see Fig. 2, left), using Euclidian distance and Ward's method. The height of the dendrograms represents the distance between the clusters.</figcaption></figure>

but differed in their strategies for matching the haptic and visual textures in the matching task (see Sec. 3.1). To further investigate the perceived similarity of the haptic and visual textures and to identify groups of textures that were perceived as similar on the matching task, a correspondence analysis and a hierarchical clustering were performed on the matching task confusion matrix (see Fig. 2, left).

The correspondence analysis captured 60 % and 29 % of the variance in the first and second dimensions, respectively, with the remaining dimensions each accounting for less than 5 % each. Fig. 3 (left) shows the first two dimensions with the 18 haptic and visual textures. The first dimension was similar to the rankings (see Fig. 2, right), distributing the textures according to their perceived roughness. It seems that the second dimension opposed textures that were perceived as hard with those perceived as softer, as also reported by participants. Stiffness is indeed an important perceptual dimension of a material [25,9].

Fig. 3 (right) shows the dendrograms of the two hierarchical clusterings of the haptic and visual textures, constructed using the Euclidean distance and the Ward's method on squared distance. The four identified haptic texture clusters were: "Roughest" {Metal Mesh, Sandpaper 100, Brick 2, Cork}; "Rougher" {Sandpaper 320, Velcro Hooks}; "Smoother" {Plastic Mesh 1, Terra Cotta}; "Smoothest" {Coffee Filter} (see Fig. 3, top-right). Similar to the haptic ranks (see Fig. 2, right), the clusters could have been named according to their perceived roughness. It also shows that the participants compared and ranked the haptic textures during the matching task to select the one that best matched the given visual texture. The five identified visual
texture clusters were: "Roughest" {Metal Mesh}; "Rougher" {Sandpaper 100, Brick 2, Velcro Hooks}; "Medium" {Cork, Plastic Mesh 1}; "Smoother" {Sandpaper 320, Terra Cotta}; "Smoothest" {Coffee Filter} (see Fig. 3, bottom-right). They are also easily identifiable on the visual ranking results, which also made it possible to name them.

<figure><img src="image_4.png" /><figcaption>**Fig. 4.** (Left) Confusion matrix of the visual texture clusters with the corresponding haptic texture clusters selected in proportion. (Right) Confusion matrix of the visual texture ranks with the corresponding haptic texture clusters selected in proportion. (Both) Holm-Bonferroni adjusted binomial test results are marked in bold when the proportion is higher than chance (i.e., more than 20 %, p<0.05).</figcaption></figure>

Based on these results, two alternative confusion matrices were constructed. Fig. 4 (left) shows the confusion matrix of the matching task with visual texture clusters and the proportion of haptic texture clusters selected in response. A two-sample Pearson Chi-Squared test ($\chi^2(16, N = 540) = 353, p < 0.001$) and Holm-Bonferroni adjusted binomial tests indicated that the following (Visual Cluster, Haptic Cluster) pairs have proportion selections statistically significantly higher than chance (i.e., 20 % each): (Roughest, Roughest), (Rougher, Rougher), (Medium, Rougher), (Medium, Smoother), (Smoother, Smoother), (Smoother, Smoothest), and (Smoothest, Smoothest) ($p < 0.005$ each). Fig. 4 (right) shows the confusion matrix of the matching task with visual texture ranks and the proportion of haptic texture clusters selected in response. A two-sample Pearson Chi-Squared test ($\chi^2(24, N = 540) = 342, p < 0.001$) and Holm-Bonferroni adjusted binomial tests indicated that the following (Visual Texture Rank, Haptic Cluster) pairs have proportion selections statistically significantly higher than chance: (0, Roughest); (1, Rougher); (2, Rougher); (3, Rougher); (4, Rougher); (5, Smoother); (6, Smoother); (7, Smoothest); and (8, Smoothest) ($p < 0.05$ each). This shows that the participants consistently identified the roughness of each visual texture and selected the corresponding haptic texture cluster.

## 3.4 Questionnaire

Fig. 5 presents the questionnaire results of the matching and ranking tasks. A non-parametric analysis of variance based on the Aligned Rank Transform (ART) was used on the *Difficulty* and *Realism* question results, while the other question results
texture clusters were: "Roughest" {Metal Mesh}; "Rougher" {Sandpaper 100, Brick 2, Velcro Hooks}; "Medium" {Cork, Plastic Mesh 1}; "Smoother" {Sandpaper 320, Terra Cotta}; "Smoothest" {Coffee Filter} (see Fig. 3, bottom-right). They are also easily identifiable on the visual ranking results, which also made it possible to name them.

<figure><img src="image_5.png" /><figcaption>**Fig. 4.** (Left) Confusion matrix of the visual texture clusters with the corresponding haptic texture clusters selected in proportion. (Right) Confusion matrix of the visual texture ranks with the corresponding haptic texture clusters selected in proportion. (Both) Holm-Bonferroni adjusted binomial test results are marked in bold when the proportion is higher than chance (i.e., more than 20 %, p<0.05).</figcaption></figure>

Based on these results, two alternative confusion matrices were constructed. Fig. 4 (left) shows the confusion matrix of the matching task with visual texture clusters and the proportion of haptic texture clusters selected in response. A two-sample Pearson Chi-Squared test ($\chi^2(16, N = 540) = 353, p < 0.001$) and Holm-Bonferroni adjusted binomial tests indicated that the following (Visual Cluster, Haptic Cluster) pairs have proportion selections statistically significantly higher than chance (i.e., 20 % each): (Roughest, Roughest), (Rougher, Rougher), (Medium, Rougher), (Medium, Smoother), (Smoother, Smoother), (Smoother, Smoothest), and (Smoothest, Smoothest) ($p < 0.005$ each). Fig. 4 (right) shows the confusion matrix of the matching task with visual texture ranks and the proportion of haptic texture clusters selected in response. A two-sample Pearson Chi-Squared test ($\chi^2(24, N = 540) = 342, p < 0.001$) and Holm-Bonferroni adjusted binomial tests indicated that the following (Visual Texture Rank, Haptic Cluster) pairs have proportion selections statistically significantly higher than chance: (0, Roughest); (1, Rougher); (2, Rougher); (3, Rougher); (4, Rougher); (5, Smoother); (6, Smoother); (7, Smoothest); and (8, Smoothest) ($p < 0.05$ each). This shows that the participants consistently identified the roughness of each visual texture and selected the corresponding haptic texture cluster.

## 3.4 Questionnaire

Fig. 5 presents the questionnaire results of the matching and ranking tasks. A non-parametric analysis of variance based on the Aligned Rank Transform (ART) was used on the *Difficulty* and *Realism* question results, while the other question results
<figure><img src="image_5.png" /><figcaption>**Fig. 5.** Boxplots of the 7-item Likert scale question results (1=Not at all, 7=Extremely) with Holm-Bonferroni adjusted pairwise Wilcoxon signed-rank tests (*** is p<0.001 and ** is p<0.01), by modality (left) and by task (right). Lower is better for Difficulty and Uncomfortable; higher is better for Realism and Textures Match.</figcaption></figure>

were analyzed using Wilcoxon signed-rank tests. On *Difficulty*, there were statistically significant effects of *Task* (F(1,57) = 13, p < 0.001) and of *Modality* (F(1,57) = 8, p=0.007), but no interaction effect *Task* × *Modality* (F(1,57)=2, *n.s.*). The Ranking task was found easier (M=2.9, SD=1.2) than the Matching task (M=3.9, SD=1.5), and the Haptic textures were found easier to discriminate (M=3.0, SD=1.3) than the Visual ones (M=3.8, SD=1.5). Both haptic and visual textures were judged moderately realistic for both tasks (M=4.2, SD=1.3), with no statistically significant effect of *Task*, *Modality* or their interaction on *Realism*. No statistically significant effects of *Task* on *Textures Match* and *Uncomfort* were found either. The coherence of the texture pairs was considered moderate (M=4.6, SD=1.2) and the haptic device was not felt uncomfortable (M=2.4, SD=1.4).

# 4 Discussion

In this study, we investigated the perception of visuo-haptic texture augmentation of tangible surfaces touched directly with the index fingertip, using visual texture overlays in AR and haptic roughness textures generated by a vibrotactile device worn on the middle index phalanx. The nine evaluated pairs of visuo-haptic textures, taken from the HaTT database [7], are models of real texture captures. Their perception was evaluated in a two-task user study in which participants chose the most coherent combinations of visual and haptic textures (matching task), and ranked all textures according to their perceived roughness (ranking task). The visual textures were displayed statically on the tangible surface, while the haptic textures adapted in real time to the speed of the finger on the surface, giving the impression that the visuo-haptic textures were integrated into the tangible surface. In addition, the interaction with the textures was designed to be as natural as possible, without imposing a specific speed of finger movement, as in similar studies [2,14].
<figure><img src="image_6.png" /><figcaption>**Fig. 5.** Boxplots of the 7-item Likert scale question results (1=Not at all, 7=Extremely) with Holm-Bonferroni adjusted pairwise Wilcoxon signed-rank tests (*** is p<0.001 and ** is p<0.01), by modality (left) and by task (right). Lower is better for Difficulty and Uncomfortable; higher is better for Realism and Textures Match.</figcaption></figure>

were analyzed using Wilcoxon signed-rank tests. On *Difficulty*, there were statistically significant effects of *Task* (F(1,57) = 13, p < 0.001) and of *Modality* (F(1,57) = 8, p=0.007), but no interaction effect *Task* × *Modality* (F(1,57)=2, *n.s.*). The Ranking task was found easier (M=2.9, SD=1.2) than the Matching task (M=3.9, SD=1.5), and the Haptic textures were found easier to discriminate (M=3.0, SD=1.3) than the Visual ones (M=3.8, SD=1.5). Both haptic and visual textures were judged moderately realistic for both tasks (M=4.2, SD=1.3), with no statistically significant effect of *Task*, *Modality* or their interaction on *Realism*. No statistically significant effects of *Task* on *Textures Match* and *Uncomfort* were found either. The coherence of the texture pairs was considered moderate (M=4.6, SD=1.2) and the haptic device was not felt uncomfortable (M=2.4, SD=1.4).

# 4 Discussion

In this study, we investigated the perception of visuo-haptic texture augmentation of tangible surfaces touched directly with the index fingertip, using visual texture overlays in AR and haptic roughness textures generated by a vibrotactile device worn on the middle index phalanx. The nine evaluated pairs of visuo-haptic textures, taken from the HaTT database [7], are models of real texture captures. Their perception was evaluated in a two-task user study in which participants chose the most coherent combinations of visual and haptic textures (matching task), and ranked all textures according to their perceived roughness (ranking task). The visual textures were displayed statically on the tangible surface, while the haptic textures adapted in real time to the speed of the finger on the surface, giving the impression that the visuo-haptic textures were integrated into the tangible surface. In addition, the interaction with the textures was designed to be as natural as possible, without imposing a specific speed of finger movement, as in similar studies [2,14].
In the matching task, participants were not able to effectively match the original visual and haptic texture pairs (see Fig. 2, left), except for the Coffee Filter texture, which was the smoothest both visually and haptically. However, almost all visual textures, except Sandpaper 100, were matched with at least one haptic texture at a level above chance. Similarly, five haptic textures were favored over the others to be matched with the visual textures. Thus, it seems that not all participants perceived visual textures in the same way and that they also hesitated between several haptic textures for a given visual texture. Indeed, the majority of users explained that, based on the roughness, granularity, or imperfections of the visual texture, they matched the haptic texture that seemed most similar or natural to what they imagined. Several strategies were used, as some participants reported using vibration frequency and/or amplitude to match a haptic texture. It should be noted that the task was rather difficult (see Fig. 5), as participants had no prior knowledge of the textures, there were no additional visual cues such as the shape of an object, and the term “roughness” had not been used by the experimenter prior to the ranking task.

The correspondence analysis (see Fig. 3, left) highlighted that participants did indeed match visual and haptic textures primarily on the basis of their perceived roughness (60% of variance), which is in line with previous perception studies on real [3] and virtual [9] textures. The rankings (see Fig. 2, right) confirmed that the participants all perceived the roughness of haptic textures very similarly, but that there was less consensus for visual textures, which is also in line with roughness rankings for real haptic and visual textures [4]. These results made it possible to identify and name groups of textures in the form of clusters, and to construct confusion matrices between these clusters and between visual texture ranks with haptic clusters, showing that participants consistently identified and matched haptic and visual textures (see Fig. 4). Interestingly, 30% of the matching variance was captured with a second dimension, opposing the roughest textures (Metal Mesh, Sandpaper 100), and to a lesser extent the smoothest (Coffee Filter, Sandpaper 320), with all other textures. One hypothesis is that this dimension could be the perceived stiffness of the textures, with Metal Mesh and smooth textures appearing stiffer than the other textures, whose granularity could have been perceived as bumps on the surface that could deform under finger pressure. Stiffness is, with roughness, one of the main characteristics perceived by the vision and touch of real materials [3,32], but also on virtual haptic textures [9,11]. The last visuo-haptic roughness ranking (see Fig. 2, right) showed that both haptic and visual sensory information were well integrated as the resulting roughness ranking was being in between the two individual haptic and visual rankings. Several strategies were reported: some participants first classified visually and then corrected with haptics, others classified haptically and then integrated visuals. While visual sensation did influence perception, as observed in previous haptic AR studies [27,15,13], haptic sensation dominated here. This indicates that participants were more confident and relied more on the haptic roughness perception than on the visual roughness perception when integrating both in one coherent perception. Several participants also described attempting to identify visual and haptic textures using spatial breaks, edges or patterns, that were not observed when these textures were displayed in non-immersive virtual environments with a screen [9,6]. A few participants even reported that they clearly sensed patterns on
In the matching task, participants were not able to effectively match the original visual and haptic texture pairs (see Fig. 2, left), except for the Coffee Filter texture, which was the smoothest both visually and haptically. However, almost all visual textures, except Sandpaper 100, were matched with at least one haptic texture at a level above chance. Similarly, five haptic textures were favored over the others to be matched with the visual textures. Thus, it seems that not all participants perceived visual textures in the same way and that they also hesitated between several haptic textures for a given visual texture. Indeed, the majority of users explained that, based on the roughness, granularity, or imperfections of the visual texture, they matched the haptic texture that seemed most similar or natural to what they imagined. Several strategies were used, as some participants reported using vibration frequency and/or amplitude to match a haptic texture. It should be noted that the task was rather difficult (see Fig. 5), as participants had no prior knowledge of the textures, there were no additional visual cues such as the shape of an object, and the term “roughness” had not been used by the experimenter prior to the ranking task.

The correspondence analysis (see Fig. 3, left) highlighted that participants did indeed match visual and haptic textures primarily on the basis of their perceived roughness (60% of variance), which is in line with previous perception studies on real [3] and virtual [9] textures. The rankings (see Fig. 2, right) confirmed that the participants all perceived the roughness of haptic textures very similarly, but that there was less consensus for visual textures, which is also in line with roughness rankings for real haptic and visual textures [4]. These results made it possible to identify and name groups of textures in the form of clusters, and to construct confusion matrices between these clusters and between visual texture ranks with haptic clusters, showing that participants consistently identified and matched haptic and visual textures (see Fig. 4). Interestingly, 30% of the matching variance was captured with a second dimension, opposing the roughest textures (Metal Mesh, Sandpaper 100), and to a lesser extent the smoothest (Coffee Filter, Sandpaper 320), with all other textures. One hypothesis is that this dimension could be the perceived stiffness of the textures, with Metal Mesh and smooth textures appearing stiffer than the other textures, whose granularity could have been perceived as bumps on the surface that could deform under finger pressure. Stiffness is, with roughness, one of the main characteristics perceived by the vision and touch of real materials [3,32], but also on virtual haptic textures [9,11]. The last visuo-haptic roughness ranking (see Fig. 2, right) showed that both haptic and visual sensory information were well integrated as the resulting roughness ranking was being in between the two individual haptic and visual rankings. Several strategies were reported: some participants first classified visually and then corrected with haptics, others classified haptically and then integrated visuals. While visual sensation did influence perception, as observed in previous haptic AR studies [27,15,13], haptic sensation dominated here. This indicates that participants were more confident and relied more on the haptic roughness perception than on the visual roughness perception when integrating both in one coherent perception. Several participants also described attempting to identify visual and haptic textures using spatial breaks, edges or patterns, that were not observed when these textures were displayed in non-immersive virtual environments with a screen [9,6]. A few participants even reported that they clearly sensed patterns on
haptic textures. However, the visual and haptic textures used were isotropic and homogeneous models of real texture captures, i.e., their rendered roughness was constant and did not depend on the direction of movement but only on the speed of the finger. Overall, the haptic device was judged to be comfortable, and the visual and haptic textures were judged to be fairly realistic and to work well together (see Fig. 5).

These results have of course some limitations as they addressed a small set of visuo-haptic textures augmenting the perception of smooth white tangible surfaces. Indeed, the increase in visuo-haptic texture perception may be limited on surfaces that already have strong visual or haptic patterns [1], or on objects with complex shapes. In addition, the haptic textures used were modelled from the vibrations of a probe sliding over the captured surfaces. The perception of surface roughness with the finger is actually more complex because it involves both the perception of vibrations and the spatial deformation of the skin [17], but also because the sensations generated when exploring a surface depend on factors other than the speed of the finger alone, such as the force of contact, the angle, the posture or the surface of the contact [30], and the integration of these sensory information into one unified perception is not yet fully understood [28]. Another limitation that may have affected the perception of haptic textures is the lack of compensation for the frequency response of the actuator and amplifier [1,9,14]. Finally, the visual textures used were also simple color captures not meant to be used in an immersive virtual environment. However, our objective was not to accurately reproduce real textures, but to alter the perception of simultaneous visual and haptic roughness augmentation of a real surface directly touched by the finger in AR. In addition of these limitations, both visual and haptic texture models should be improved by integrating the rendering of spatially localized breaks, edges or patterns, like real textures [28], and by being adaptable to individual sensitivities, as personalized haptics is a promising approach [21,33]. More generally, a wide range of haptic feedbacks should be integrated to form rich and complete haptic augmentations in AR [20,10,29,23,26].

# 5 Conclusion

<figure><img src="image_6.png" /><figcaption>**Fig. 6.** Illustration of the texture augmentation in AR through an interior design scenario. A user wearing an AR headset and a wearable vibrotactile haptic device worn on their index is applying different virtual visuo-haptic textures to a real wall to compare them visually and by touch.</figcaption></figure>
haptic textures. However, the visual and haptic textures used were isotropic and homogeneous models of real texture captures, i.e., their rendered roughness was constant and did not depend on the direction of movement but only on the speed of the finger. Overall, the haptic device was judged to be comfortable, and the visual and haptic textures were judged to be fairly realistic and to work well together (see Fig. 5).

These results have of course some limitations as they addressed a small set of visuo-haptic textures augmenting the perception of smooth white tangible surfaces. Indeed, the increase in visuo-haptic texture perception may be limited on surfaces that already have strong visual or haptic patterns [1], or on objects with complex shapes. In addition, the haptic textures used were modelled from the vibrations of a probe sliding over the captured surfaces. The perception of surface roughness with the finger is actually more complex because it involves both the perception of vibrations and the spatial deformation of the skin [17], but also because the sensations generated when exploring a surface depend on factors other than the speed of the finger alone, such as the force of contact, the angle, the posture or the surface of the contact [30], and the integration of these sensory information into one unified perception is not yet fully understood [28]. Another limitation that may have affected the perception of haptic textures is the lack of compensation for the frequency response of the actuator and amplifier [1,9,14]. Finally, the visual textures used were also simple color captures not meant to be used in an immersive virtual environment. However, our objective was not to accurately reproduce real textures, but to alter the perception of simultaneous visual and haptic roughness augmentation of a real surface directly touched by the finger in AR. In addition of these limitations, both visual and haptic texture models should be improved by integrating the rendering of spatially localized breaks, edges or patterns, like real textures [28], and by being adaptable to individual sensitivities, as personalized haptics is a promising approach [21,33]. More generally, a wide range of haptic feedbacks should be integrated to form rich and complete haptic augmentations in AR [20,10,29,23,26].

# 5 Conclusion

<figure><img src="image_7.png" /><figcaption>**Fig. 6.** Illustration of the texture augmentation in AR through an interior design scenario. A user wearing an AR headset and a wearable vibrotactile haptic device worn on their index is applying different virtual visuo-haptic textures to a real wall to compare them visually and by touch.</figcaption></figure>
We investigated how users perceived visuo-haptic roughness texture augmentations on tangible surfaces seen in immersive OST-AR and touched directly with the index finger. The haptic roughness texture was rendered using a wearable vibrotactile haptic device worn on the middle phalanx, based on HaTT data-driven models and finger speed. Participants rated the coherence, realism and roughness of the combination of nine representative visuo-haptic texture pairs. The results showed that participants consistently identified and matched clusters of visual and haptic textures with similar perceived roughness. The texture rankings did indeed show that participants perceived the roughness of haptic textures to be very similar, but less so for visual textures, and the haptic roughness perception predominated the final roughness perception ranking of the original visuo-haptic pairs. There are still many improvements to be made to the respective renderings of the haptic and visual textures used in this work to make them more realistic for finger perception and immersive virtual environment contexts. However, these results suggest that AR visual textures that augments tangible surfaces can be enhanced with a set of data-driven vibrotactile haptic textures in a coherent and realistic manner. This paves the way for new AR applications capable of augmenting a real environment with virtual visuo-haptic textures, such as visuo-haptic painting in artistic, object design or interior design contexts. The latter is illustrated in Fig. 6, where a user applies different visuo-haptic textures to a wall to compare them visually and by touch.

**Acknowledgments.** This project has been funded by the European Union (ERC, AD-VHANDTURE, 101088708).

**Disclosure of Interests.** The authors have no competing interests to declare that are relevant to the content of this article.

## References

1. Asano, S., Okamoto, S., Matsuura, Y., Nagano, H., Yamada, Y.: Vibrotactile display approach that modifies roughness sensations of real textures. In: IEEE Int. Symp. Robot Hum. Interact. Commun. pp. 1001–1006 (2012)

2. Asano, S., Okamoto, S., Yamada, Y.: Vibrotactile Stimulation to Increase and Decrease Texture Roughness. IEEE Trans. Hum.-Mach. Syst. 45(3), 393–398 (2015)

3. Baumgartner, E., Wiebel, C.B., Gegenfurtner, K.R.: Visual and Haptic Representations of Material Properties. Multisensory Res. 26(5), 429–455 (2013)

4. Bergmann Tiest, W.M., Kappers, A.M.L.: Haptic and visual perception of roughness. Acta Psychologica 124(2), 177–189 (2007)

5. Chan, S., Tymms, C., Colonnese, N.: Hasti: Haptic and Audio Synthesis for Texture Interactions. In: IEEE World Haptics Conf. pp. 733–738 (2021)

6. Culbertson, H., Kuchenbecker, K.J.: Should haptic texture vibrations respond to user force and speed? In: IEEE World Haptics Conf. pp. 106–112 (2015)

7. Culbertson, H., López Delgado, J.J., Kuchenbecker, K.J.: One hundred data-driven haptic texture models and open-source methods for rendering on 3D objects. In: IEEE Haptics Symp. pp. 319–325 (2014)

8. Culbertson, H., Schorr, S.B., Okamura, A.M.: Haptics: The Present and Future of Artificial Touch Sensation. Annu. Rev. Control Robot. Auton. Syst. 1(1), 385–409 (2018)

9. Culbertson, H., Unwin, J., Kuchenbecker, K.J.: Modeling and Rendering Realistic Textures from Unconstrained Tool-Surface Interactions. IEEE Trans. Haptics 7(3), 381–393 (2014)
We investigated how users perceived visuo-haptic roughness texture augmentations on tangible surfaces seen in immersive OST-AR and touched directly with the index finger. The haptic roughness texture was rendered using a wearable vibrotactile haptic device worn on the middle phalanx, based on HaTT data-driven models and finger speed. Participants rated the coherence, realism and roughness of the combination of nine representative visuo-haptic texture pairs. The results showed that participants consistently identified and matched clusters of visual and haptic textures with similar perceived roughness. The texture rankings did indeed show that participants perceived the roughness of haptic textures to be very similar, but less so for visual textures, and the haptic roughness perception predominated the final roughness perception ranking of the original visuo-haptic pairs. There are still many improvements to be made to the respective renderings of the haptic and visual textures used in this work to make them more realistic for finger perception and immersive virtual environment contexts. However, these results suggest that AR visual textures that augments tangible surfaces can be enhanced with a set of data-driven vibrotactile haptic textures in a coherent and realistic manner. This paves the way for new AR applications capable of augmenting a real environment with virtual visuo-haptic textures, such as visuo-haptic painting in artistic, object design or interior design contexts. The latter is illustrated in Fig. 6, where a user applies different visuo-haptic textures to a wall to compare them visually and by touch.

**Acknowledgments.** This project has been funded by the European Union (ERC, AD-VHANDTURE, 101088708).

**Disclosure of Interests.** The authors have no competing interests to declare that are relevant to the content of this article.

## References

1. Asano, S., Okamoto, S., Matsuura, Y., Nagano, H., Yamada, Y.: Vibrotactile display approach that modifies roughness sensations of real textures. In: IEEE Int. Symp. Robot Hum. Interact. Commun. pp. 1001–1006 (2012)

2. Asano, S., Okamoto, S., Yamada, Y.: Vibrotactile Stimulation to Increase and Decrease Texture Roughness. IEEE Trans. Hum.-Mach. Syst. **45**(3), 393–398 (2015)

3. Baumgartner, E., Wiebel, C.B., Gegenfurtner, K.R.: Visual and Haptic Representations of Material Properties. Multisensory Res. **26**(5), 429–455 (2013)

4. Bergmann Tiest, W.M., Kappers, A.M.L.: Haptic and visual perception of roughness. Acta Psychologica **124**(2), 177–189 (2007)

5. Chan, S., Tymms, C., Colonnese, N.: Hasti: Haptic and Audio Synthesis for Texture Interactions. In: IEEE World Haptics Conf. pp. 733–738 (2021)

6. Culbertson, H., Kuchenbecker, K.J.: Should haptic texture vibrations respond to user force and speed? In: IEEE World Haptics Conf. pp. 106–112 (2015)

7. Culbertson, H., López Delgado, J.J., Kuchenbecker, K.J.: One hundred data-driven haptic texture models and open-source methods for rendering on 3D objects. In: IEEE Haptics Symp. pp. 319–325 (2014)

8. Culbertson, H., Schorr, S.B., Okamura, A.M.: Haptics: The Present and Future of Artificial Touch Sensation. Annu. Rev. Control Robot. Auton. Syst. **1**(1), 385–409 (2018)

9. Culbertson, H., Unwin, J., Kuchenbecker, K.J.: Modeling and Rendering Realistic Textures from Unconstrained Tool-Surface Interactions. IEEE Trans. Haptics **7**(3), 381–393 (2014)
10. de Tinguyl, X., Pacchierotti, C., Marchal, M., Lecuyer, A.: Enhancing the Stiffness Perception of Tangible Objects in Mixed Reality Using Wearable Haptics. In: IEEE Virtual Real. pp. 81–90 (2018)

11. Degraen, D., Zenner, A., Krüger, A.: Enhancing Texture Perception in Virtual Reality Using 3D-Printed Hair Structures. In: CHI Conf. Hum. Factors Comput. Syst. (2019)

12. Ernst, M.O., Banks, M.S.: Humans integrate visual and haptic information in a statistically optimal fashion. Nature 415(6870), 429–433 (2002)

13. Fradin, J., Haliyo, S., Gueorguiev, D.: Humans Struggle to Perceive Congruence in Visuo-Haptic Textures. In: IEEE World Haptics Conf. pp. 128–133 (2023)

14. Friesen, R.F., Vardar, Y.: Perceived realism of virtual textures rendered by a vibrotactile wearable ring display. IEEE Trans. Haptics (2024)

15. Gaffary, Y., Le Gouis, B., Marchal, M., Argelaguet, F., Arnaldi, B., Lecuyer, A.: AR Feels “Softer” than VR: Haptic Perception of Stiffness in Augmented versus Virtual Reality. IEEE Trans. Vis. Comput. Graph. 23(11), 2372–2377 (2017)

16. Kim, K., Billinghurst, M., Bruder, G., Duh, H.B.L., Welch, G.F.: Revisiting Trends in Augmented Reality Research: A Review of the 2nd Decade of ISMAR (2008-2017). IEEE Trans. Vis. Comput. Graph. 24(11), 2947–2962 (2018)

17. Klatzky, R.L., Lederman, S.J., Hamilton, C., Grindley, M., Swendsen, R.H.: Feeling textures through a probe: Effects of probe and surface geometry and exploratory factors. Percept. Psychophys. 65(4), 613–631 (2003)

18. Knorlein, B., Di Luca, M., Harders, M.: Influence of visual and haptic delays on stiffness perception in augmented reality. In: IEEE Int. Symp. Mix. Augment. Real. pp. 49–52 (2009)

19. Macedo, M.C.F., Apolinário, A.L.: Occlusion Handling in Augmented Reality: Past, Present and Future. IEEE Trans. Vis. Comput. Graph. 29(2), 1590–1609 (2023)

20. Maisto, M., Pacchierotti, C., Chinello, F., Salvietti, G., De Luca, A., Prattichizzo, D.: Evaluation of Wearable Haptic Systems for the Fingers in Augmented Reality Applications. IEEE Trans. Haptics 10(4), 511–522 (2017)

21. Malvezzi, M., Chinello, F., Prattichizzo, D., Pacchierotti, C.: Design of Personalized Wearable Haptic Interfaces to Account for Fingertip Size and shape. IEEE Trans. Haptics 14(2), 266–272 (2021)

22. Meli, L., Pacchierotti, C., Salvietti, G., Chinello, F., Maisto, M., De Luca, A., Prattichizzo, D.: Combining Wearable Finger Haptics and Augmented Reality: User Evaluation Using an External Camera and the Microsoft HoloLens. IEEE Robot. Autom. Lett. 3(4), 4297–4304 (2018)

23. Normand, E., Pacchierotti, C., Marchand, E., Marchal, M.: Visuo-Haptic Rendering of the Hand during 3D Manipulation in Augmented Reality. IEEE Trans. Haptics (2024)

24. Okamoto, S., Konyo, M., Saga, S., Tadokoro, S.: Detectability and Perceptual Consequences of Delayed Feedback in a Vibrotactile Texture Display. IEEE Trans. Haptics 2(2), 73–84 (2009)

25. Okamoto, S., Nagano, H., Yamada, Y.: Psychophysical Dimensions of Tactile Perception of Textures. IEEE Trans. Haptics 6(1), 81–93 (2013)

26. Pacchierotti, C., Chinello, F., Koumaditis, K., Luca, M.D., Ofek, E., Georgiou, O.: Haptics in the Metaverse: Haptic feedback for Virtual, Augmented, Mixed, and eXtended Realities. IEEE Trans. Haptics (2024)

27. Punpongsanon, P., Iwai, D., Sato, K.: SoftAR: Visually Manipulating Haptic Softness Perception in Spatial Augmented Reality. IEEE Trans. Vis. Comput. Graph. 21(11), 1279–1288 (2015)

28. Richardson, B.A., Vardar, Y., Wallraven, C., Kuchenbecker, K.J.: Learning to Feel Textures: Predicting Perceptual Similarities From Unconstrained Finger-Surface Interactions. IEEE Trans. Haptics 15(4), 705–717 (2022)
10. de Tinguyl, X., Pacchierotti, C., Marchal, M., Lecuyer, A.: Enhancing the Stiffness Perception of Tangible Objects in Mixed Reality Using Wearable Haptics. In: IEEE Virtual Real. pp. 81–90 (2018)

11. Degraen, D., Zenner, A., Krüger, A.: Enhancing Texture Perception in Virtual Reality Using 3D-Printed Hair Structures. In: CHI Conf. Hum. Factors Comput. Syst. (2019)

12. Ernst, M.O., Banks, M.S.: Humans integrate visual and haptic information in a statistically optimal fashion. Nature 415(6870), 429–433 (2002)

13. Fradin, J., Haliyo, S., Gueorguiev, D.: Humans Struggle to Perceive Congruence in Visuo-Haptic Textures. In: IEEE World Haptics Conf. pp. 128–133 (2023)

14. Friesen, R.F., Vardar, Y.: Perceived realism of virtual textures rendered by a vibrotactile wearable ring display. IEEE Trans. Haptics (2024)

15. Gaffary, Y., Le Gouis, B., Marchal, M., Argelaguet, F., Arnaldi, B., Lecuyer, A.: AR Feels “Softer” than VR: Haptic Perception of Stiffness in Augmented versus Virtual Reality. IEEE Trans. Vis. Comput. Graph. 23(11), 2372–2377 (2017)

16. Kim, K., Billinghurst, M., Bruder, G., Duh, H.B.L., Welch, G.F.: Revisiting Trends in Augmented Reality Research: A Review of the 2nd Decade of ISMAR (2008-2017). IEEE Trans. Vis. Comput. Graph. 24(11), 2947–2962 (2018)

17. Klatzky, R.L., Lederman, S.J., Hamilton, C., Grindley, M., Swendsen, R.H.: Feeling textures through a probe: Effects of probe and surface geometry and exploratory factors. Percept. Psychophys. 65(4), 613–631 (2003)

18. Knorlein, B., Di Luca, M., Harders, M.: Influence of visual and haptic delays on stiffness perception in augmented reality. In: IEEE Int. Symp. Mix. Augment. Real. pp. 49–52 (2009)

19. Macedo, M.C.F., Apolinário, A.L.: Occlusion Handling in Augmented Reality: Past, Present and Future. IEEE Trans. Vis. Comput. Graph. 29(2), 1590–1609 (2023)

20. Maisto, M., Pacchierotti, C., Chinello, F., Salvietti, G., De Luca, A., Prattichizzo, D.: Evaluation of Wearable Haptic Systems for the Fingers in Augmented Reality Applications. IEEE Trans. Haptics 10(4), 511–522 (2017)

21. Malvezzi, M., Chinello, F., Prattichizzo, D., Pacchierotti, C.: Design of Personalized Wearable Haptic Interfaces to Account for Fingertip Size and shape. IEEE Trans. Haptics 14(2), 266–272 (2021)

22. Meli, L., Pacchierotti, C., Salvietti, G., Chinello, F., Maisto, M., De Luca, A., Prattichizzo, D.: Combining Wearable Finger Haptics and Augmented Reality: User Evaluation Using an External Camera and the Microsoft HoloLens. IEEE Robot. Autom. Lett. 3(4), 4297–4304 (2018)

23. Normand, E., Pacchierotti, C., Marchand, E., Marchal, M.: Visuo-Haptic Rendering of the Hand during 3D Manipulation in Augmented Reality. IEEE Trans. Haptics (2024)

24. Okamoto, S., Konyo, M., Saga, S., Tadokoro, S.: Detectability and Perceptual Consequences of Delayed Feedback in a Vibrotactile Texture Display. IEEE Trans. Haptics 2(2), 73–84 (2009)

25. Okamoto, S., Nagano, H., Yamada, Y.: Psychophysical Dimensions of Tactile Perception of Textures. IEEE Trans. Haptics 6(1), 81–93 (2013)

26. Pacchierotti, C., Chinello, F., Koumaditis, K., Luca, M.D., Ofek, E., Georgiou, O.: Haptics in the Metaverse: Haptic feedback for Virtual, Augmented, Mixed, and eXtended Realities. IEEE Trans. Haptics (2024)

27. Punpongsanon, P., Iwai, D., Sato, K.: SoftAR: Visually Manipulating Haptic Softness Perception in Spatial Augmented Reality. IEEE Trans. Vis. Comput. Graph. 21(11), 1279–1288 (2015)

28. Richardson, B.A., Vardar, Y., Wallraven, C., Kuchenbecker, K.J.: Learning to Feel Textures: Predicting Perceptual Similarities From Unconstrained Finger-Surface Interactions. IEEE Trans. Haptics 15(4), 705–717 (2022)
29. Salazar, S.V., Pacchierotti, C., de Tinguyl, X., Maciel, A., Marchal, M.: Altering the Stiffness, Friction, and Shape Perception of Tangible Objects in Virtual Reality Using Wearable Haptics. IEEE Trans. Haptics **13**(1), 167–174 (2020)

30. Schafer, H., Wells, Z., Shao, Y., Visell, Y.: Transfer properties of touch elicited waves: Ef- fect of posture and contact conditions. In: IEEE World Haptics Conf. pp. 546–551 (2017)

31. Teng, S.Y., Li, P., Nith, R., Fonseca, J., Lopes, P.: Touch&Fold: A Foldable Haptic Actuator for Rendering Touch in Mixed Reality. In: CHI Conf. Hum. Factors Comput. Syst. (2021)

32. Vardar, Y., Wallraven, C., Kuchenbecker, K.J.: Fingertip Interaction Metrics Correlate with Visual and Haptic Perception of Real Surfaces. In: IEEE World Haptics Conf. pp. 395–400 (2019)

33. Young, E.M., Gueorguiev, D., Kuchenbecker, K.J., Pacchierotti, C.: Compensating for Fingertip Size to Render Tactile Cues More Accurately. IEEE Trans. Haptics **13**(1), 144–151 (2020)
29. Salazar, S.V., Pacchierotti, C., de Tinguy, X., Maciel, A., Marchal, M.: Altering the Stiffness, Friction, and Shape Perception of Tangible Objects in Virtual Reality Using Wearable Haptics. IEEE Trans. Haptics **13**(1), 167–174 (2020)

30. Schafer, H., Wells, Z., Shao, Y., Visell, Y.: Transfer properties of touch elicited waves: Ef- fect of posture and contact conditions. In: IEEE World Haptics Conf. pp. 546–551 (2017)

31. Teng, S.Y., Li, P., Nith, R., Fonseca, J., Lopes, P.: Touch&Fold: A Foldable Haptic Actuator for Rendering Touch in Mixed Reality. In: CHI Conf. Hum. Factors Comput. Syst. (2021)

32. Vardar, Y., Wallraven, C., Kuchenbecker, K.J.: Fingertip Interaction Metrics Correlate with Visual and Haptic Perception of Real Surfaces. In: IEEE World Haptics Conf. pp. 395–400 (2019)

33. Young, E.M., Gueorguiev, D., Kuchenbecker, K.J., Pacchierotti, C.: Compensating for Fingertip Size to Render Tactile Cues More Accurately. IEEE Trans. Haptics **13**(1), 144–151 (2020)