2019 IEEE International Conference on Consumer Electronics(ICCE)

Interaction of Visual and Haptic Impressions
in Visuo-haptic Texture Cognition
Taishi Fujiwara∗ , Yusuke Tani, Atsushi Takemoto, Kensuke Tobitani and Noriko Nagata†
Kwansei Gakuin University, Sanda, Hyogo, Japan
Email: ∗ t.fujiwara@kwansei.ac.jp, † nagata@kwansei.ac.jp

Abstract—The interaction among modalities is useful in thefield of virtual reality and augmented reality. However, the effects
of sensory interaction have yet to be fully elucidated. In thisresearch study, we investigated the interaction between visualand haptic impressions in visuo-haptic texture cognition. We
discovered that the influence of the interaction between visual andhaptic impression occurs when the impression have differenceswith regard to stimulus in the haptic modality.

I. I NTRODUCTION
Recently, multisensory studies have become more common [1]. Especially, in the field of virtual reality and augmented reality, these research studies are in great demand instudies that use multimodal interaction [2]. However, almost
all of them focus on one or certain specific physical properties [3] and differences in materials [4], so the mechanisms ofthe interaction are still not completely understood.
In this research study, we focused on impressions felt fromtexture, and we investigated the interaction between visualand haptic impressions based on the data of our previous
study [5]. The consideration of differences in interaction
among evaluation words will enable us to use effective andefficient sensory interaction.
II. S UBJECTIVE E VALUATION E XPERIMENTS
We conducted subjective evaluation experiments to quantify
impressions felt from texture. We performed experiments forthree conditions: visual, haptic, and visuo-haptic (Fig. 1). Prior
to these experiments, we collected as evaluation words 19
words for the visual and haptic conditions, ensuring representativeness and completeness for the impression space ofthe texture. In the visuo-haptic condition, we used the samewords as those in the visual condition. They were actuallyJapanese words. We used 20 synthetic resin samples (Fig. 2)
with various textures as stimuli.

(a) visual condition;(b) haptic condition;(c) visuo-hapticcondition.

Fig. 1: The three conditions used in the experiments.

Fig. 2: Examples of stimuli.
A. Method
The visual condition had 19 participants (17 men and twowomen) with an average age of 21.45 (SD = 0.75); the haptic
condition had 20 participants (19 men and one woman) withan average age of 21.5 (SD = 0.74); and the visuo-haptic
condition had 20 participants (18 men and two women) with anaverage age of 21.55 (SD = 0.75). They evaluated the strengthof the impression felt from the stimulus on a five-point scale.
B. Results and Discussion
1) Factor Analysis: We scored the rating data from 0 to100, and we contracted three-dimensional data (stimuli × evaluation words × participants) to two-dimensional data (stimuli× evaluation words) by taking the average of each participant.
Then, we performed factor analysis with the data using themaximum likelihood method and the Promax rotation. We
interpreted extracted factors as displayed in TABLE I. The
similarities and differences between the visual and visuohaptic factor structure suggest the superiority of the visual
modality and the influence of the haptic modality on visuohaptic condition.
TABLE I: Interpretation of factors.

2) Two-Way Analysis of Variance (ANOVA): We performed
two-way ANOVA to investigate the difference of impressions,with stimuli and three conditions used in the experiments,
for five evaluation words that all of the conditions shared:“regular,” “sporty,” “uneven,” “progressive,” and “youthful.”
The results are displayed in TABLE II.
For “regular” and “uneven,” the interaction between thestimulus and condition was significant. Therefore we tested
the simple main effects of the condition as part of a post-hocanalysis.
According to the result of the post-hoc test (TABLE III),“regular” has no difference with regard to the stimulus in the

978-1-5386-7910-4/19/$31.00 ©2019 IEEE

TABLE II: Results of two-way ANOVA.

2) Difference among Evaluation Words: We compared themedian values of AIC for each evaluation word to investigatethat whether the interaction between visual and haptic impressions varies depending on evaluation word.
TABLE IV: Comparison of median value of AIC.

* highlights statistical significance.

TABLE III: Results of simple main effect test.

* highlights statistical significance.

haptic condition, in contrast to “uneven.” This suggests that,for participants did not feel difference of “regular” with regard
to stimulus in haptic condition.
III. M ULTIPLE R EGRESSION A NALYSIS
We performed multiple regression analysis to investigate theeffect of the interaction between visual and haptic impressions.
A. Method
We calculated and compared two models (with or withoutinteraction) that predict the visuo-haptic rating based on thevisual and haptic rating. We used data extracted from the same
stimulus for 14 participants (all men) with an average age of21.57 (SD = 0.72) who participated in all of the conditions.
We analyzed five evaluation words common to all conditions.
1) Model without Interaction: The model without interaction is depicted in Equation 1:
V H = α1 V + β1 H

(1)In this equation, V , H, and V H are the scored visual,haptic, and visuo-haptic ratings, respectively; and α1 and β1
are the regression weights for V and H, respectively.
2) Model with Interaction: The model with interaction isdepicted in Equation 2:
V H = α2 V + β2 H + γ(V × H)

(2)

Here, we added the V ×H (the scored visual rating multiplied by the scored haptic rating) to Equation 1 as anindependent variable (γ indicates the regression weight forV ×H).
B. Results and Discussion1) Influence of Interaction: We obtained 100 regression
equations (20 stimuli × 5 evaluation words) for each model. In
28 equations, we confirmed that the model with interaction was
more suitable by comparing it with the Akaike information
criterion (AIC). In these situation, it can be assumed thatthe interaction between visual and haptic impressions haveinfluence on visuo-haptic texture cognition.

According to TABLE IV, contrary to the three evaluation
words of “uneven,” “progressive,” and “youthful,” for thetwo evaluation words of “regular” and “sporty,” Equation 1
was more suitable. Concerning “regular” and “uneven,” whichyielded different results for the simple main effect test, thesuitable model was also different. Both the results of simple
main effect test and the comparison of AIC suggest thesame thing: For evaluation words, like “uneven,” which havedifferences with regard to the stimulus in the haptic condition,interaction occur in the visou-haptic texture cognition.
IV. C ONCLUSION
We investigated the interaction between visual and haptic
impressions felt from texture. We conducted subjective evaluation experiment, and using factor analysis, we confirmed thatboth visual and haptic modality have influence on visuo-haptic
texture cognition. In addition, we performed two-way ANOVA
and multiple regression analysis to investigate differencesof interaction between visual and haptic impressions among
evaluation words. As the result, the influence of the interaction
varies depending on whether feel the differences of the impression with regard to stimulus in haptic modality. Specifically,
when the impression have differences with regard to stimulus
in the haptic modality, the interaction has influence on visuohaptic texture cognition. Detailed research on more diverse
evaluation words is necessary to reproduce the interaction ofthe visual and haptic impressions in the field of virtual reality
and augmented reality.
ACKNOWLEDGMENT
This work was supported in part by the Center of Innovation
Program from Japan Science and Technology Agency, JST.
R EFERENCES[1] R. Martı́n, J. Iseringhausen, M. Weinmann, and M. B. Hullin, “Multimodal perception of material properties,” in Proceedings of the ACM
SIGGRAPH Symposium on Applied Perception, ser. SAP ’15. New York,NY, USA: ACM, 2015, pp. 33–40.
[2] A. Iesaki, A. Somada, A. Kimura, F. Shibata, and H. Tamura, “Psychophysical influence on tactual impression by mixed-reality visualstimulation,” in 2008 IEEE Virtual Reality Conference, March 2008, pp.
265–266.
[3] J. Eck, A. L. Kaas, J. L. Mulders, and R. Goebel, “Roughness perceptionof unfamiliar dot pattern textures,” Acta Psychologica, vol. 143, no. 1,pp. 20–34, May 2013.
[4] W. Fujisaki, N. Goda, I. Motoyoshi, H. Komatsu, and S. Nishida,“Audiovisual integration in the human perception of materials,” Journalof Vision, vol. 14, no. 4, p. 12, Apr. 2014.
[5] T. Fujiwara, A. Takemoto, Y. Tani, K. Tobitani, and N. Nagata, “Theintegration of visual and haptic impressions felt form synthetic resin
texture,” in The 11th IEEE Pacific Visualization Symposium, no. 113,2018.