<figure><img src="image_1.png" /></figure>

HAL
open science

# Humans Struggle to Perceive Congruence in Visuo-Haptic Textures

Jenna Fradin, Sinan Haliyo, David Gueorguiev

► To cite this version:

Jenna Fradin, Sinan Haliyo, David Gueorguiev. Humans Struggle to Perceive Congruence in Visuo-Haptic Textures. IEEE World Haptics conference, Jul 2023, Delft, Netherlands. pp.128-133, 10.1109/WHC56415.2023.10224382. hal-04156797

HAL Id: hal-04156797

https://hal.science/hal-04156797

Submitted on 9 Jul 2023

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_2.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Humans Struggle to Perceive Congruence in Visuo-Haptic Textures

Jenna Fradin

Sinan Haliyo

David Gueorguiev

Institut des Systemes Intelligents et de Robotique

Institut des Systemes Intelligents et de Robotique

Institut des Systemes Intelligents et de Robotique

Sorbonne Université

Sorbonne Université

CNRS, Sorbonne Université

Paris, France

Paris, France

Paris, France

fradin@isir.upmc.fr

sinan.haliyo@isir.upmc.fr

david.gueorguiev@isir.upmc.fr

**Abstract**—Multisensory assessment of objects and textures is ubiquitous in human lives. It has been shown that people can optimally integrate tactile and visual cues to extract sensory cues. It is less clear though how the congruence between tactile and visual cues is perceived when people interact with everyday objects such as fabrics. In this study, we investigate the human accuracy to detect visuo-haptic discrepancy within common fabrics. Participants had to detect the congruent one among two pairs of visuo-haptic textures consisting in a visual texture projected on a semi-mirror and a real texture underneath the visual one, which could be touched but not seen. Overall, participants did not accurately detect congruent visuo-haptic pairs and their performance correlated with the average similarity between the visual and tactile textures. Correlation analysis may suggest a role of the coefficient of kinetic friction to detect the visuo-haptic discrepancy.

***Index Terms***—Multisensory interaction, material perception

## I. INTRODUCTION

When we see or explore a surface or an object, we encode information that enables us to perceive and categorize that surface or object. This information is usually provided by several of our senses simultaneously. Indeed, texture perception is successfully mediated by multiple sensory modalities such as vision and touch [1], [2]. But the question arises as to how information from these two modalities is processed to generate the subjective perception of textures [3]. A number of articles have focused on either visual texture perception [4]–[6] or haptic texture perception [7]–[9] but the effects of multisensory inputs on texture perception are less investigated, although studies have been conducted on artificial stimuli [10] and visuo-haptic illusions [11].

Several perceptual integration models have been proposed to explain this process [12]. One model suggests the summative integration of information between sensory modalities, which means that the final percept is the result of the summation of the visual and haptic information. In another model, which relies on sensory sharing, both visual and haptic inputs contribute independently to the perception process. A well-established model proposed by Ernst and Banks [13] suggests that visuo-haptic integration follows the prediction of a maximum likelihood integrator, which means that it is performed in statistically optimal way. However, these models

focus on the perception of a specific feature and do not predict the sensation of visuo-haptic congruence.

A frequently used strategy to investigate visuo-haptic integration is to create a sensory conflict. To do so, researchers create tasks in which the visual information diverges slightly from haptic information. This approach makes it possible to estimate how the brain solves the challenge of integrating conflicting inputs from distinct sensory channels and unravel important sensory mechanisms like in [14] and [15].

In this study, by creating potential conflicts between tactile and visual properties, we intend to highlight the sensory interaction between visual and haptic cues that shape human visuo-haptic perception of common real-world textures. Up to now, perception of more natural materials has received less attention and few studies have investigated texture perception using material occurring in an everyday context [2] [16] [17]. This may be related to the difficulty of quantifying and modifying in a controlled manner the physical properties of these natural materials. Nevertheless, studying visuo-haptic perception of natural materials and textures is crucial since a major research direction in haptics is the recreation of texture on tactile displays. Indeed, various design methods and actuation principles have been proposed [18], [19] but it is not yet known how much accuracy is needed when the tactile properties of a texture are recreated on a haptic device and superposed on its visual representation. For example, it has been shown that the brain is tolerant to a large amount of redirection of the limbs in virtual reality [20], which enables virtual environments that feel larger or with more objects than their physical size.

Also, humans are very sensitive to tactile properties of materials and previous work has shown that physical surface properties impact inter-modal perceptual judgements [14] [21]. Thus, we also recorded the acceleration and contact force induced during interaction with each of the textures of our panel following the same approach as [16]. These measurements enabled the comparison between the results from our psychophysical experiments and the relevance of common metrics in haptics such as the kinetic friction coefficient and the vibration power.
<figure><img src="image_3.png" /><figcaption>Fig. 1. Texture panel used in the experiments</figcaption></figure>

## II. METHODS

### A. Participants

Two psychophysical experiments and a set of physical measurements were conducted. In the first experiment, seven participants (3 females, 4 males, mean age 24.5 years) performed the task. The second experiment was conducted on sixteen participants (6 females, 10 males, mean age 23.4 years). Three people (1 female, 2 males, mean age 25.3 years) participated in the physical measurements. None of them reported visual disabilities or sensory-motor impairments that could impact on visuo-tactile perception. The experimental procedures were approved by the Ethics Council of Sorbonne University. All participants gave informed consent and received financial compensation for their participation.

### B. Stimuli

We used a panel of texture stimuli which included a variety of everyday fabrics: stretch denim, hucktowel, denim, wool crepe and blue denim (see Fig 1). Each surface is a 10x3 cm rectangle mounted onto a piece of wood by means of double-sided adhesive tape. The thickness of each sample (surface plus wood plate) is approximately 1.5 cm. A picture of each texture was taken and used as visual stimuli in the experiments. To generate still photographs of all textures, we placed individual fabric on a white cloth. The photos were taken at a distance similar to the distance between the participant and the haptic textures on the visuo-haptic workbench. Post-processing images centering of the textures and cropping to a size of 1100 x 290 pixels was done using Processing v3.5.4.

### C. Apparatus

The stimuli were presented in a visuo-haptic workbench (see Fig.2) consisting of a 24" screen (BenQ 3D monitor, 1920 x 1080 pixel), a semi-reflexive mirror (3mm thick semi-mirror Groglass 50/50) and a texture holder (24x2x2 cm). The screen was located nearly 10 cm above the reflective surface and tilted at 45 degrees. Participants watched the visual stimuli reflection of the screen across the semi-reflective surface and manipulated the textures below it, such as the reflected image fell upon the real texture from their point of view. The mirror prevented them from seeing their hands and the real stimuli. A few practice trials were provided so that the participants could get used to the setup and place their hands in the correct position. The position and size of the visual textures corresponded to those of the real ones.

A

<figure><img src="image_4.png" /></figure>

B

<figure><img src="image_5.png" /><figcaption>Fig. 2. **A.** Schematic representation of the visuo-haptic workbench. **B.** Actual experimental setup. Participant viewed the projection (4) of the visual stimuli displayed on the screen (1) across the semi reflective mirror (2). They were asked to touch the haptic stimuli placed on the textures holder below (3).</figcaption></figure>

### D. Procedure

#### Experiment 1.

The first experiment was divided into two parts. In the first part, participants completed a purely haptic discrimination task of a set of textures in the form of a three-alternative forced-choice task. During the haptic experiment, they were seated in front of three textures selected from the panel (see Fig.1) placed on a rigid support behind a black divider, so that they could touch them without seeing them. Two textures (wool crepe and stretch denim) were selected as references based on preliminary experimental results. For each trial, one of the two references was selected. The experimenter then positioned three textures on the support: two of them being the chosen reference and its duplicate, the third being one of the remaining textures of the panel. Each participant performed 85 trials, with an equal number of trials for each reference.

In the second part, only two texture from the initial panel were presented. The participants were asked to compare them and rate the haptic similarity on a 9-point Likert scale ranging from 1 (completely dissimilar) to 9 (completely similar).
A

Reference - Stretch Denim

Reference - Wool Crepe

<figure><img src="image_6.png" /></figure>

B

Reference - Stretch Denim

Reference - Wool Crepe

<figure><img src="image_7.png" /></figure>

Fig. 3. A. Experiment 1: Mean of the success rate across all the participant for each haptic combination for each reference B. Experiment 2: Mean of the success rate across all the participants for each visuo-haptic combination and for each visual reference. Error bars represent the standard deviation. In Fig 3.A, asterisks indicate significance using Tukey's multiple comparison test and in Fig 3.B, they indicate significance of the one sample Wilcoxon test against chance level with \*p<0.05, \*\*p<0.005 and \*\*\*p<0.0005.

All 25 possible pairs were presented. The experience in its totality was carried out in one session lasting 60 minutes for each participant.

## Experiment 2.

The second experiment was also divided into two parts. The first part consisted of a two-alternative forced choice paradigm. In this experiment, participants were standing in front of the visuo-haptic workbench (See Fig.2). For each trial, two surfaces selected from the panel were placed on the texture holder and two visual textures were displayed on the screen. Regarding visual textures, we chose to work only with two textures from our panel: wool crepe and stretch denim, which were already used in the previous experiment. For each trial, one of these two textures was chosen as reference. The position of each surface of the pair was randomly selected for each trial. As for the haptic textures, one matched the visual reference chosen for this trial and the other was chosen randomly among the remaining textures. Participant observed the projection of the visual textures displayed across the semi-reflective mirror and freely touched the haptic textures below. They were asked to find the congruent visuo-haptic texture among the two presented, knowing that only one of the two was matching. There were a total of 85 trials (5 training + 10 repetitions of each possible visuo-haptic pair).

In the second part, participant rated the similarity of the visual and haptic feedback using a 9-point Likert scale. The experimenter placed only one texture on the texture holder and only one visual texture was displayed. All 25 possible combinations were presented. Participant freely explored the haptic texture and verbally gave a similarity rating ranging from 1 (completely dissimilar) to 9 (completely similar). The experience 2 in its totality was carried out in one session lasting between 60 and 90 minutes for each participant.

## Physical measurements.

Each surface was successively placed on a texture holder which was positioned on top of a force sensor (Nano17, ATI Inc.). The sensor was screwed to an aluminium base that was attached to a stationary table. A mono-axial accelerometer (352A24, PCB Piezotronics) was glued to the fingernail of the subject and the cable was taped on the hand. The accelerometer and force data were acquired by a DAQ card (USB-6343, NI Inc).

Each participant slid their index finger on the surface ten times from left to right. The participants were asked to synchronize their sliding speed with the speed of a visual cursor displayed on a screen (7”HDMI Touchscreen, DFRobot) positioned in front of the experimental setup. The sliding speed was fixed at 50 mm/s. The measurements were carried out independently of the behavioural experiments.

# III. RESULTS

## A. Experiment 1

We tested participants' ability to discriminate between different textures using only haptic information. We calculated the success rate for each combination and for each reference across all participants.

To simplify the analysis, we focused on the overall success rate for each combination (see Fig.3.A). An ANOVA statistical analysis and post-hoc testing highlighted significant differences in term of success rate for each combination beside blue denim - denim and hucktowel - wool crepe when the haptic reference was the stretch denim (One-Way ANOVA: F = 45.7, p = 0.001 and Tukey's multiple comparison test). No significant difference was found when wool crepe was used as haptic reference.
## B. Experiment 2

In order to assess participants' capacity to find the congruent visuo-haptic texture, the success rate for each participant and for each visuo-haptic combination was calculated. To simplify the analysis, we focused on the overall success rate for each combination and for each visual reference. An ANOVA statistical analysis showed an impact of the haptic reference on the participants' score when the reference was stretch denim (One-Way ANOVA: F = 2.89, p = 0.043) but not when the reference was wool crepe (One-way ANOVA, F = 0.231, p = 0.874). A Bonferroni-corrected one sample Wilcoxon signed rank test was performed to assess whether detection of discrepancy was above chance-level for the tested visuo-haptic comparisons. Results showed that the discrepancy was significantly recognized only for the hucktowel when the reference stimulus was wool crepe (W = 94, p = 0.043) and the wool crepe when reference was stretch denim (W = 109, p = 0.02). Overall, performance at visuo-haptic detection was almost at chance level within the panel of textures (see Fig.3.B).

We also checked whether there was a significant change in terms of success rate over the time course of the experiment (between the two blocks). However, no significant learning effect was observed (Two-Way ANOVA: F = 2.81, p = 0.095). We then analyzed the similarity ranking obtained in both experiments. We converted the similarity ranking of each participant into normalized dissimilarities by subtracting them from nine and then dividing by eight. To facilitate the analysis, we looked at the average dissimilarity for each combination. A significant correlation was found between the visuo-haptic dissimilarities and the success rate (Pearson correlation, p = 0.012) and between the haptic dissimilarities and the success rate (Pearson correlation, p = 0.016). The results are plotted in Fig.4. These results suggest an influence of haptic differences on the visuo-tactile perception of textures.

## C. Correlation with physical measurements

Following the work from [16], we focused on two different physical metrics: the kinetic friction coefficient and the vibration power during sliding. Previous studies have showed the link between kinetic friction coefficient and the stickiness of the surfaces as well as between vibration power and the perceptual roughness of the surface [22] [23].

We calculated the kinetic friction from the force signals. The signal were first low-pass filtered with a cut-off frequency of 1 kHz to match with human perception, then they were segmented into 10 segments. Each segment represents one finger sliding on the surface. For each segment, the kinetic friction coefficient was calculated by fitting a Coulomb friction model to the tangential and normal forces. The friction coefficient for each surface was obtained by computing the mean friction coefficient across the ten strokes of a given participant, and then by averaging the obtained coefficients across participants.

The vibration power was calculated from the acceleration signals. The signals were detrended, low-pass filtered with cut-off frequency of 1 kHz and segmented into 10 segments. Each

A

<figure><img src="image_8.png" /></figure>

B

<figure><img src="image_9.png" /><figcaption>Fig. 4. A. Correlation between the overall success rate for each visuo-haptic combination and the average visuo-haptic dissimilarities. B. Correlation between the overall success rate for each visuo-haptic combination and the average haptic dissimilarities. The combinations for which reference was the stretch denim are represented with blank circles and the combinations for which reference was the wool crepe are represented with full circles.</figcaption></figure>

segmented signal was band-pass filtered between 20 Hz and 400 Hz, and then the power spectrum was calculated.

<table><caption>TABLE I
KINETIC FRICTION COEFFICIENT AND VIBRATION POWER CALCULATED FOR EACH SURFACE</caption><thead><tr><th>Textures</th><th>Kinetic friction coefficient (Unitless)</th><th>Vibration power (g<sup>2</sup>)</th></tr></thead><tbody><tr><td>Huck towel</td><td>0.512</td><td>8.687 × 10<sup>-4</sup></td></tr><tr><td>Stretch Denim</td><td>0.483</td><td>4.016 × 10<sup>-4</sup></td></tr><tr><td>Blue Denim</td><td>0.490</td><td>5.768 × 10<sup>-4</sup></td></tr><tr><td>Denim</td><td>0.482</td><td>7.312 × 10<sup>-4</sup></td></tr><tr><td>Wool crepe</td><td>0.466</td><td>1.024 × 10<sup>-4</sup></td></tr></tbody></table>

The friction coefficient and the vibration power for each surface are reported in Table I. We compared the computed metrics with the psychophysical data. We calculated the difference in friction coefficient between the reference and compared haptic texture for each pair presented in the Experiment 2 and compared it to the previously calculated success rate (see Fig.5). No significant correlation was found between the success rate and the difference in kinetic fric-
<figure><img src="image_10.png" /><figcaption>Fig. 5. A. Correlation between the average success rate for each combination presented in the experiment 2 and the difference of kinetic friction coefficient. B. Correlation between the overall success rate for each combination presented in the experiment 2 and the difference of vibration power. The combinations for which reference was the stretch denim are represented with blank circles and the combinations for which reference was the wool crepe are represented with full circles.</figcaption></figure>

tion coefficient within a pair of haptic textures (Spearman correlation, $p = 0.054$). Similarly, no correlation was found between the success rate in experiment 2 and the vibration power (Spearman correlation, $p = 0.623$).

## IV. DISCUSSION AND CONCLUSION

In the present study, we investigated how visual and haptic information mediate the subjective perception of visuo-haptic textures. More precisely, we studied the judgements of visuo-haptic congruence by introducing small or large tactile discrepancies between the two modalities. Despite our panel of textures being quite similar, participants could easily sense the tactile differences between textures except when asked to distinguish denim types.

Performance was poorer in the visuo-haptic discrimination task, close to the chance level (see Fig.3). Detecting discrepant from congruent texture information across two sensory modalities does appear to be a complex task for humans. On one hand, these results suggest distinct tactile and visual representations of textures. They are in line with studies that have compared visual and touch recognition with a

bimodal condition without observing bimodal performance to be superior to either single mode condition [24] [25]. On the other hand, the significant correlation between the averaged visuo-haptic dissimilarities and the average success rate (See Fig.4) does suggest that humans have a representation of how the visual and tactile feeling of a texture should relate, which is in line with perception arising from multisensory integration.

The analysis of two metrics that have already been considered for visuo-haptic perception of textures showed no clear impact on congruence perception. Vibration power did not correlate with the capacity to notice the discrepancy while the kinetic friction coefficient showed a marginal correlation. Although the coefficient of kinetic friction are quite close in the panel, the maximum difference being less than 0.05, it is still possible that this cue is used since touch is especially sensitive to frictional cues [26]. Therefore, it is possible that discrepant tactile textures with larger differences in friction will be detected more easily.

Overall, perception of visuo-haptic congruence is a hard task for humans. Thus, it is probably possible to use the tolerance to differences between the haptic component of a texture and its visual representation to efficiently design simple yet effective haptic feedback. This could enable to introduce compelling visuo-haptic feedback in displays without the difficult challenge of rendering the entire complexity of natural haptic interactions.

## V. ACKNOWLEDGEMENT

This study was supported by Sorbonne université and the ANR project Maptics (20-CE33-0013).

## REFERENCES

[1] M. A. Heller, "Visual and tactual texture perception: Intersensory cooperation," *Perception & psychophysics*, vol. 31, pp. 339-344, 1982.

[2] S. Guest and C. Spence, "Tactile dominance in speeded discrimination of textures," *Experimental brain research*, vol. 150, pp. 201-207, 2003.

[3] T. A. Whitaker, C. Simões-Franklin, and F. N. Newell, "Vision and touch: Independent or integrated systems for the perception of texture?" *Brain research*, vol. 1242, pp. 59-72, 2008.

[4] B. Julesz, "Texture and visual perception," *Scientific American*, vol. 212, no. 2, pp. 38-49, 1965.

[5] M. S. Landy and N. Graham, "Visual perception of texture," *The visual neurosciences*, pp. 1106-1118, 2004.

[6] W. Lee and M. Sato, "Visual perception of texture of textiles," *Color Research & Application: Endorsed by Inter-Society Color Council, The Colour Group (Great Britain)*, Canadian Society for Color, Color Science Association of Japan, Dutch Society for the Study of Color, The Swedish Colour Centre Foundation, Colour Society of Australia, Centre Français de la Couleur, vol. 26, no. 6, pp. 469-477, 2001.

[7] M. R. McGee, P. Gray, and S. Brewster, "Haptic perception of virtual roughness," in *CHI'01 Extended Abstracts on Human Factors in Computing Systems*, 2001, pp. 155-156.

[8] M. Hollins and S. J. Bensmaïa, "The coding of roughness." *Canadian Journal of Experimental Psychology/Revue canadienne de psychologie expérimentale*, vol. 61, no. 3, p. 184, 2007.

[9] B. Hughes and G. Jansson, "Texture perception via active touch," *Human Movement Science*, vol. 13, no. 3-4, pp. 301-333, 1994.

[10] T. Konkle, Q. Wang, V. Hayward, and C. I. Moore, "Motion aftereffects transfer between touch and vision," *Current Biology*, vol. 19, no. 9, pp. 745-750, 2009.

[11] P. Abtahi and S. Follmer, "Visuo-haptic illusions for improving the perceived performance of shape displays," in *Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems*, 2018, pp. 1-13.
[12] M. L. Shaw, "Attending to multiple sources of information: I. the integration of information in decision making," *Cognitive Psychology*, vol. 14, no. 3, pp. 353-409, 1982.

[13] M. O. Ernst and M. S. Banks, "Humans integrate visual and haptic information in a statistically optimal fashion," *Nature*, vol. 415, no. 6870, pp. 429-433, 2002.

[14] W. M. B. Tiest and A. M. Kappers, "Haptic and visual perception of roughness," *Acta psychologica*, vol. 124, no. 2, pp. 177-189, 2007.

[15] G. Robles-De-La-Torre and V. Hayward, "Force can overcome object geometry in the perception of shape through active touch," *Nature*, vol. 412, no. 6845, pp. 445-448, 2001.

[16] Y. Vardar, C. Wallraven, and K. J. Kuchenbecker, "Fingertip interaction metrics correlate with visual and haptic perception of real surfaces," in *2019 IEEE World Haptics Conference (WHC)*. IEEE, 2019, pp. 395-400.

[17] E. Baumgartner, C. B. Wiebel, and K. R. Gegenfurtner, "Visual and haptic representations of material properties," *Multisensory research*, vol. 26, no. 5, pp. 429-455, 2013.

[18] C. Basdogan, F. Giraud, V. Levesque, and S. Choi, "A review of surface haptics: Enabling tactile effects on touch surfaces," *IEEE transactions on haptics*, vol. 13, no. 3, pp. 450-470, 2020.

[19] H. Culbertson, J. Unwin, and K. J. Kuchenbecker, "Modeling and rendering realistic textures from unconstrained tool-surface interactions,"

*IEEE transactions on haptics*, vol. 7, no. 3, pp. 381-393, 2014.

[20] F. Lebrun, S. Haliyo, and G. Bailly, "A trajectory model for desktop-scale hand redirection in virtual reality," in *Human-Computer Interaction-INTERACT 2021: 18th IFIP TC 13 International Conference, Bari, Italy, August 30-September 3, 2021, Proceedings, Part V*. Springer, 2021, pp. 105-124.

[21] W. J. Adams, I. S. Kerrigan, and E. W. Graf, "Touch influences perceived gloss," *Scientific reports*, vol. 6, no. 1, pp. 1-12, 2016.

[22] T. Yoshioka, S. J. Bensmaia, J. C. Craig, and S. S. Hsiao, "Texture perception through direct and indirect touch: An analysis of perceptual space for tactile textures in two modes of exploration," *Somatosensory & motor research*, vol. 24, no. 1-2, pp. 53-70, 2007.

[23] H. Culbertson and K. J. Kuchenbecker, "Importance of matching physical friction, hardness, and texture in creating realistic haptic virtual surfaces," *IEEE transactions on haptics*, vol. 10, no. 1, pp. 63-74, 2016.

[24] B. Jones and S. O'Neil, "Combining vision and touch in texture perception," *Perception & Psychophysics*, vol. 37, pp. 66-72, 1985.

[25] I. Brown, "Visual and tactual judgments of surface roughness," *Ergonomics*, vol. 3, no. 1, pp. 51-61, 1960.

[26] D. Gueorguiev, S. Bochereau, A. Mouraux, V. Hayward, and J.-L. Thonnard, "Touch uses frictional cues to discriminate flat materials," *Scientific reports*, vol. 6, no. 1, p. 25553, 2016.