The Integration of Visual and Haptic Impressions
Felt from Synthetic Resin Texture
Taishi Fujiwara*

Atsushi Takemoto†

Yusuke Tani

Kensuke Tobitani

Noriko Nagata‡

Kwansei Gakuin UniversityA BSTRACT
The ability of texture perception is important to estimate the materials and properties of objects. However, the ways how multimodal information can be applied to texture recognition has yet tobe fully elucidated. In this research study, we modeled the relationship among visual, haptic, and visuo-haptic impressions using
multiple-regressions analysis. Some of the models demonstratedthat the product of scored visual and haptic impressions plays animportant role in multimodal texture perception. This result suggests that a multimodal texture perception can greatly change theimpressions of texture more than we imagined.

(a) visual condition;

2 S UBJECTIVE E VALUATION E XPERIMENTS
We conducted subjective evaluation experiments to investigate differences in texture recognition for various sensory systems. We
performed the experiments for three conditions: visual, haptic, andvisuo-haptic (Fig. 1). We used 20 synthetic resin samples (Fig.
2) with various textures as stimuli. Prior to these experiments, wecollected words to evaluate, via a free-description experiment, theimpressions felt from the stimuli (visual condition only and haptic condition only). Then, we excluded any unfit words based on
the result of goodness-of-fit experiment. Moreover, we conducteda word-similarity rating procedure and used the multidimensional
scaling method and hierarchical cluster analysis to select words thatensured representativeness and completeness. Finally, we selected
19 words as evaluation words for each of two of the conditions:the visual condition and the haptic condition. In the visuo-hapticcondition, we used the same words as those in the visual condition.
They were actually Japanese words.
* e-mail:t.fujiwara@kwansei.ac.jp† e-mail:a.takemoto@kwansei.ac.jp‡ e-mail:nagata@kwansei.ac.jp(c) visuo-haptic condition.

Figure 1: The three conditions used in the experiments.

Index Terms: Human-centered computing—Interaction design—
Interaction design theory, concepts and paradigms
1 I NTRODUCTION
Various studies have been conducted on texture perception related
to a single modality [1]. However, in our lives, we often use multimodal information. Recently, multisensory studies have become
more common, but quantitative analyses of the integration of multimodal information are few in number. Additionally, almost all ofthem focus on one or certain specific physical properties [2, 4] anddifferences in materials [3], so the mechanisms of the integration ofmultimodal impressions are still not completely understood.
In this research study, we focused on impressions regarding thevisual and haptic texture perceptions of synthetic resins, and wemodeled impression structures for the visual, haptic, and visuohaptic modalities. This will enable us to estimate impressions felt
from a texture, and will be helpful for adjusting the texture that havedesire impressions in the field of product design and virtual reality.

(b) haptic condition;

Figure 2: Examples of stimuli.

2.1

MethodThe visual condition had 19 participants (17 men and two women)with an average age of 21.45 (SD = 0.75); the haptic condition had
20 participants (19 men and one woman) with an average age of21.5 (SD = 0.74); the visuo-haptic condition had 20 participants(18 men and two women) with an average age of 21.55 (SD = 0.75).
They evaluated the strength of the impression felt from the stimulus
on a five-point scale.
2.2

Results and Discussion

We scored the rating data from 0 to 100, and we contracted threedimensional data (stimuli×evaluation words×participants) to twodimensional data (stimuli×evaluation words) by taking the averageof each participant. Then, we performed a factor analysis with thedata using the maximum likelihood method and the Promax rotation. Based on the Kaiser-Guttman criterion, three factors each forthe visual and visuo-haptic conditions were extracted. At the sametime, for haptic condition, two factors were extracted (see the lists
in Table 1). We interpreted these factors as shown in Table 2. According to the these interpretation, the similarities and differencesin the visual and visuo-haptic conditions demonstrate the superiority of the visual modality and the influence of the haptic modality
on visuo-haptic condition.
3

M ODELING

We performed a linear multiple-regression analysis to model therelationships among visual, haptic, and visuo-haptic impressions.
3.1

MethodWe calculated a multiple linear regression to predict the visuohaptic rating of a model based on its components’ visual and hapticrating. The regression model is depicted in Equation 1 (wherein V ,H, and V H are the scored visual, haptic, and visuo-haptic ratings;
and α and β are the regression weights for V and H). We useddata extracted from the same stimulus for 14 participants (all men)with an average age of 21.57 (SD = 0.72) who participated in allof the conditions. We analyzed five evaluation words that all of theTable 1: Results of factor analysis.
(a) visual condition;(b) haptic condition;

Figure 3: The most improved model.

impressions. Therefore, we modified the regression equation.
4.1 Method
The modified equation is depicted in Equation 2.
V H = α V + β H + γ (V × H)

(2)

Here, we added the product of V ×H (the scored visual rating
multiplied by the scored haptic rating) to Equation 1 as an independent variable (γ indicate the regression weight for V ×H). Theproduct divided by 100 was used for matching the scale with eachvariable. We compared this model with the unmodified model using
the Akaike information criterion.

(c) visuo-haptic condition.

4.2 Results and Discussion
In 28 models (out of 100), a decrease in the Akaike information
criterion were confirmed. The most improved model is revealed inFig. 3, which depicts a large increase in the coefficient of determination. Moreover, the average absolute value of standardized partial
regression coefficient is 0.71 for the visual terms, 0.73 for the haptic terms, and 0.94 for their interaction. This result suggests that theimpression felt from texture changes is mainly based on visual andhaptic interaction.
5 C ONCLUSION
We have modeled the relationships among the visual, haptic, andvisuo-haptic impressions. The results indicated that the integrationof visual and haptic impressions cannot be expressed with a simple
linear regression model; in addition, the influence of the interaction between the visual and haptic modalities was stronger than weimagined. Hence, we can control the visual impression felt fromtexture by using haptic information. Further consideration will beneeded to reproduce the interaction of the visual and haptic modalities in the field of product design and virtual reality.

Table 2: Interpretation of factors.

conditions shared: “regular,” “sporty,” “uneven,” “progressive,” and“youthful.”
V H = αV + β H(1)ACKNOWLEDGMENTS
This work was supported in part by the Center of Innovation Program from Japan Science and Technology Agency, JST.

3.2 Results and Discussion

R EFERENCES

As a result of the analysis, we obtained 100 models (20 stimuli×5evaluation words). The average absolute value of standardized partial regression coefficient for the visual terms was 0.35, and the average for the haptic terms was 0.25. Therefore, we confirmed thathaptic information is essentially as important as visual information.
However, only seven of the 100 models had high coefficients of determination (over 0.6), and 27 had very low coefficients (less than0.1). This suggests that the relationships among the visual, haptic,
and visuo-haptic modalities cannot be simulated completely with asimple linear regression model.
4 M ODIFIED M ODEL

[1] J. Dorsey and H. Rushmeier. Advanced material appearance modeling.
In Proc. ACM SIGGRAPH Courses, SIGGRAPH ’09, pp. 3:1–3:134.
ACM, New York, 2009. doi: 10.1145/1667239.1667242[2] J. Eck, A. L. Kaas, J. L. Mulders, and R. Goebel. Roughness perceptionof unfamiliar dot pattern textures. Acta Psychologica, 143(1):20–34,
May 2013. doi: 10.1016/j.actpsy.2013.02.002[3] W. Fujisaki, N. Goda, I. Motoyoshi, H. Komatsu, and S. Nishida. Audiovisual integration in the human perception of materials. Journal ofVision, 14(4):12, Apr. 2014. doi: 10.1167/14.4.12[4] A. Iesaki, A. Somada, A. Kimura, F. Shibata, and H. Tamura.
Psychophysical influence on tactual impression by mixed-reality visual stimulation. In Proc. IEEE Virtual Real. Conf., pp. 265–266.
IEEE Computer Society, Los Alamitos, 2008. doi: 10.1109/VR.2008.4480793

In the previous section, it was revealed that a simple linear regression model cannot fully express the integration of visual and haptic