This article has been accepted for publication in IEEE Transactions on Haptics. This is the author's version which has not been fully edited andcontent may change prior to final publication. Citation information: DOI 10.1109/TOH.2023.3269016

JOURNAL OF LATEX CLASS FILES, VOL. XX, NO. X, AUGUST 2022

1

Specifying Visual Parameters for Haptic-visual
Sequential Matching of Material Softness
Yusuke Ujitoko, and Takahiro Kawabe

Abstract—When shopping online, a customer cannot directly
touch the products but may sometimes make judgments about thehaptic properties of a product based only on visual information,
before making a purchase decision. In this scenario, a customer
may be dissatisfied if there is an inconsistency in the judgmentof the product’s haptic properties they made before purchasing,and their actual experience of those haptic properties once
they have received the product. Thus, it is necessary for online
sellers to appropriately optimize visual information for materials
so that perceived softness is consistent between haptic andvisual modalities presented in different locations and at different
moments in time. Focusing on visual indentation depth and speed,
we examined the visual parameters used to sequentially match
haptic and visual softness from haptic and visual information
made available in different locations and at different times.
Participants performed a two-alternative forced choice task todetermine which of two video clips contained an elastic material
with a softness impression most similar to the haptic softnessof an actual material that the participants indented with theirindex finger. Based on a sequence of 25 repeated judgments foreach material, our algorithm optimized each visual parameterbased on a Gaussian process. The optimized visual indentation
depth varied consistently with material compliance, while theoptimized visual indentation speed did not, suggesting thatvisual indentation depth was critical for softness matching. The
optimized visual indentation depth was highly correlated with thehaptic indentation depth. Subjective rating scores for the softness
matching increased significantly after the optimization process.
The results indicate that participants were able to successfully
match the haptic and visual softness of materials by checking
the relationship between indentation depths detected haptically,
and those detected visually.
Index Terms—Material Softness, Sequential Matching, Visual
Softness, Haptic Softness.

I. I NTRODUCTION
Haptic experiences are an essential aspect of purchasing
behaviors. For example, it is known that some individuals
purchase a product only after they have experienced some kindof direct haptic interaction with it [1]. Meanwhile, in somepurchase contexts, purchase decisions need to be made without
direct haptic experience of a product. One representative
example is online shopping. Online shopping volumes havegrown with each passing year [2] and are expected to continue
to do so even after COVID-19 [3]. Although online customers
do not directly touch a product before purchasing, they may
predict certain haptic properties of a product based on visual
information.
One of the haptic properties that can be judged visuallyis softness [4]. Softness is the subjective impression of theY. Ujitoko and T. Kawabe are with NTT Communication Science Laboratories, 3-1, Morinosato Wakamiya, Atsugi, 243-0033, Kanagawa, Japan
Manuscript received XXXXX XX, 2022

physical compressibility and deformability of materials as
described in the introduction of the book [5]. Touching soft
materials gives people a pleasant sensation [6], [7], and soft
materials sometimes invite hedonic touch [8]. To properly
communicate vision-based softness to online customers, it isimportant for online sellers to devise ways to make visual
softness approximate as closely as possible to the actual haptic
softness of their products. Consumers may be dissatisfied if
there is a major discrepancy between their judgment of thesoftness of a product made based on visual information before
buying, and the softness of the product perceived hapticallyonce they have received the actual item after purchasing.
Therefore, online sellers need to properly optimize visual
information relating to product materials so that perceived
softness is matched between haptic and visual modalities.
The present study focuses on the problem of how to achieve
haptic and visual sequential matching of material softness by
optimizing visual information.
Sequential matching here refers to the human behavior ofevaluating the consistency of haptic and visual softness for
haptic-visual stimuli when each is separately presented indifferent locations and at different times. As yet little is known
about how the cognitive system matches material softness
between the senses of touch and sight. In the context ofmulti-modal integration, there have been studies investigating
how humans “integrated” stimuli with different presentation
timings into a single softness perception [9], [10], [11],[12]. For example, Di Luca et al. have shown that a delay
between the presentation of visual and haptic information
during indentation decreased the perceived softness of a virtual
spring [9]. Lecuyer et al. have shown that participants felt avirtual spring on the screen to be softer when the spring wascompressed to a larger extent [10]. Metzger et al have shownthat the haptic softness judgment made by their experimental
participants was sensitive to the sequential order of presented
stimuli [11], suggesting that the judgment depended on theparticipants’ prior experience (or judgment) of softness. Likethese studies regarding stimulus presentation timing, Xiao et
al. have investigated how tactile and visual information ismatched when information for both modalities is given at thesame time [13]. In contrast with previous studies, we wereinterested in situations where, for example, three seconds afteran observer haptically confirms the softness of a cushion bypressing it with her/his finger, the same observer visually confirms the softness of “another” cushion by watching anotherperson pressing it and compare the softness of the cushions.
In the assumed situation, it is unlikely that the perception ofa single object will occur, as has been shown to be the case© 2023 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission.See https://www.ieee.org/publications/rights/index.html for more information.

This article has been accepted for publication in IEEE Transactions on Haptics. This is the author's version which has not been fully edited andcontent may change prior to final publication. Citation information: DOI 10.1109/TOH.2023.3269016

JOURNAL OF LATEX CLASS FILES, VOL. XX, NO. X, AUGUST 2022

in previous studies [9], [10], [11], because it does not seema plausible strategy for the cognitive system to integrate thesignals relating to the different cushions to create a mental
representation of a single object. None of these studies has
investigated how human participants match haptic and visual
softness when these modalities are stimulated sequentially insuch situations.
The purpose of the present study was to clarify which visualparameters in video clips of materials were used for the hapticvisual sequential matching of material softness. Although it
is assumed that people perform softness matching in various
contexts in their daily lives, no previous study has clarified
which information in the haptic and visual modalities wasused for the matching. Once it becomes clear what visualparameters are used for softness matching, we will better
understand the mechanism of this situation. Moreover, theelucidation of the visual parameters used in haptic-visualsequential softness matching has some practical value in thecontext of online shopping. For example, online sellers may
be able to optimize visual information relating to the haptic
softness of their products. By editing videos of material deformation, visual parameters can be easily manipulated. However,no previous studies have focused on the visual parameters usedin the haptic-visual sequential matching of material softness.
By optimizing the visual parameters identified in this study,
online sellers will be able to effectively convey the haptic
softness of their products in their video presentations.
In the present study, we focused on visual parameters which
could be expected to vary when a deformable material was
being indented from its uppermost surface. While there aredifferent perceptual dimensions for visual and haptic softness [14], [15], we focused on the “deformability” dimension,
which corresponds well with material compliance, rather thanother dimensions such as viscosity, granularity, or furriness.
Also, among the different types of materials that could be
rated to be soft and viscous [16], we focused on elastic
materials. Some previous studies have consistently reported
that visual indentation depth was a critical cue for judgmentsof the softness of a deformable material [17], [18], [19],[20]. The visual indentation depth here means the visible
depth of an indentation into an elastic material surface. It
has been demonstrated that a video clip of both a real [17],[19], [20] or a computer-rendered material [18] with a deeper
indentation produced greater impressions of material softness.
In addition, a previous study [19] has also shown that, besides
the visual indentation depth, the visual indentation speed
significantly contributes to the judgment of visual softness.
The visual indentation speed here means the visible speedof an indentation being made into the surface of an elastic
material. The study just cited found that a higher indentation
speed resulted in higher softness rating scores. The authors of
the study also reported that in a comparison between visual
indentation speed and visual indentation depth, the latter had
a greater influence on the judgment of softness.
We regarded visual indentation depth and speed as potential
contributors to the haptic-visual sequential matching of material softness, since they are also haptically discernable whena participant makes an indentation in a real object with their2

own finger. By establishing the relationship between haptic
and visual indentation depths, or haptic and visual indentation
speeds, we expected that participants may be able to achieve
haptic-visual softness matching. It is a focus of our interest to
determine how these visual parameters were actually used forthe haptic-visual sequential matching of material softness.
To clarify this main focus of our study, we conducted anexperiment in which participants performed sequential matching of softness between haptic and visual stimuli. Specifically,
our experimental participants watched a pair of video clips
of a material being indented three seconds either before
or after they themselves haptically indented a real material.
Participants were not allowed to observe the clips during thehaptic indentation, nor were they allowed to haptically performthe indentation during the visual observation of the clips. The
participants judged which of the clips contained a material
whose softness was most similar to the material they were
haptically indenting. The participants’ judgments on the softness similarity were used to optimize the visual parameters —visual indentation depth and speed —, through a human-in-theloop optimization process. This was done by using Bayesian
optimization with a Gaussian process, which updated themodel (that describes the relationship between the participants’
judgments and the visual parameters) iteratively based on
the participants’ judgments. Specifically, in the optimization
procedure, the algorithm took the visual parameters which
the participants chose in the current trial as its input. Next,
the algorithm updated the objective function with a Gaussian
process. Finally, the algorithm searched for and proposed a
new set of visual parameters that would likely induce a preferential choice by the participants in the subsequent trial, on thebasis of a preference learning algorithm featuring exploitation(choosing points near a previously observed optimum) and
exploration (choosing points in areas that have not been well
explored). Thus, as the optimized values were determinedon the basis of the visual parameters that the participantspreferentially chose, it followed that we could conclude thatthe optimized value of the visual parameters was related tothe participants’ preferential judgment for the haptic-visual
softness matching.
II. M ETHOD
A. Participants
Twelve people (7 males and 5 females, all right-handed)with a mean age of 27.4 (SD: 7.0) participated. They were notinformed about the specific purpose of the experiments. They
reported that they had normal or corrected-to-normal visualacuity. They were recruited from outside the laboratory by aJapanese hiring agency and were paid for their participation.
Ethical approval for the present study was obtained from the
ethics committee at Nippon Telegraph and Telephone Corporation (Approval number: R02-002 by NTT Communication
Science Laboratories Ethics Committee). The experiments
were conducted in accordance with the 2008 Declaration of
Helsinki. Written informed consent was obtained from allparticipants.

© 2023 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission.See https://www.ieee.org/publications/rights/index.html for more information.

This article has been accepted for publication in IEEE Transactions on Haptics. This is the author's version which has not been fully edited andcontent may change prior to final publication. Citation information: DOI 10.1109/TOH.2023.3269016

JOURNAL OF LATEX CLASS FILES, VOL. XX, NO. X, AUGUST 2022

3

wall

displaymaterials

90 cm

40 cm

participant

30cm

Fig. 1. Experimental environment. Participants indented the material’s uppersurface without being able to look at it.
TABLE IT WO SIZES OF HOLE , R SQUARED VALUE OF FITTED LINEAR MODEL( BETWEEN FORCE AND DISPLACEMENT ), AND MATERIAL COMPLIANCEFOR EACH MATERIAL .
Material

Diameter offirst hole

Diameter of
second hole

[mm]

[mm]

R squared valueof fitted
linear model

A

0.6

0.5

0.993

0.235

B

1.2

0.65

0.988

0.271

C

1.45

0.5

0.985

0.358

D

1.4

0.95

0.986

0.572

E

0.5

1.8

0.993

0.759

F

1.75

0.8

0.917

2.473

Compliance[mm/N]

B. Apparatus
Figure 1 shows the experimental environment. Participants
were seated comfortably on a chair and placed their rightindex finger on the upper surface of a material positioned
in front of their right hand. To prevent them from receiving
any visual information about the materials, a barrier was
set up between the participants and the materials. A display(VIEWPixx; VPixx Technologies Inc., Canada) with a 1920
× 1200 resolution at 120Hz was positioned in front of theparticipants at a distance of 90 cm.
1) Haptic Stimuli: We attempted to replicate the six cubic
metamaterials introduced in a previous study [21] by using
identical types of material (TangoBlackPlus [22]) and a 3D
printer (Stratasys Obje500). See supplementary Note 1 for how
we selected the specific six materials used for this study fromthe 12 used in the previous study just referenced. Figures 2(a)and (b) respectively show photographs of the materials andtheir force-displacement curves. Each side of the material was
42 mm long and the shape contained 169 cylindrical holes.
There were two sizes of holes in each material, which differed
between the six materials as shown in the first and second
columns in Table I. There was a 3.00 mm gap between theholes. These configurations were identical to those used in aprevious study [21].
To characterize the properties of each material, we performed uniaxial load testing. An increasing force was applied

to the cubes’ upper surface to give a displacement at a constant
speed of 1 mm / 6 seconds, and the corresponding force wasrecorded using a force tester (MCT-2150, A&D Co., Ltd.).
Figure 2(b) shows the measured force-displacement curves.
To check the linearity of the deformation of the six materials, we fitted the data with linear models to regress the force
with displacement for each material. The R squared valuesof the fitness ranged from 0.917 to 0.993 (see the R squared
values for each material in the third column in Table I). Thus,
we regarded the force-displacement relationship as linear anddefined the compliance according to the values in the fourthcolumn of Table I. The compliance value for each material
used for analysis is shown in Figure 2(c).
2) Visual Stimuli: The visual stimuli in our experimentwere video clips that showed the upper surface of an elastic
material being pushed down by an indenter (a cylinder witha diameter of 1.3 cm). The video resolution was 288 x 288
pixels at 29.97 frames per second. We filmed the videos
from a camera position diagonally above the materials so thatthe upper surface pushed by the indenter could be clearly
seen. The horizontal distance between the camera and thematerials was 40 cm and the height of the camera above thematerials was 17 cm. The elevation angle of the camera wasapproximately 23 ◦ . The camera lens was oriented towards thematerial.
The raw video recorded the indenter pressing into thematerials at a constant speed of 1 mm/6 seconds. The raw
videos started from a point when the indenter was stationary
and in contact with the material’s upper surface. They showed
the indenter pushing the material down to a depth of 18 mm
and immediately returning to its starting position (see Figure 3). We exported all the frames from the videos and, by
manipulating which frames were presented in each stimulus
video, we were able to control the visual indentation depth andthe visual indentation speed as they appeared in the videos.
See some example video clips in the Supplementary Videos.
The visual indentation depth refers to the depth of theindentation made by the indenter into the material’s uppersurface. The visual indentation speed refers to the speed atwhich the indenter pressed into the material’s upper surface. In
this experiment, these two visual parameters in the video clips
were optimized on the basis of the participants’ responses. The
presented visual indentation depth could vary from 1.0 mm to18.0 mm. The presented visual indentation speed could varyfrom 2.0 mm/s to 15.0 mm/s. The details of our optimization
algorithm are described in the “II. D. Optimization Algorithm”
section.
3) Tracking Participant’s Finger Position: The participant’s
index finger position was monitored with an optical tracking
system (Optitrack, V120 Trio). The marker for the tracking
system was attached to the nail of the participant’s index finger.
The mean tracking system refresh rate across all trials was
190.4 captures per second.
C. Task
There were two tasks in this experiment: a visual-haptic
task and a haptic-visual task. The tasks differed in the order© 2023 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission.See https://www.ieee.org/publications/rights/index.html for more information.

This article has been accepted for publication in IEEE Transactions on Haptics. This is the author's version which has not been fully edited andcontent may change prior to final publication. Citation information: DOI 10.1109/TOH.2023.3269016

JOURNAL OF LATEX CLASS FILES, VOL. XX, NO. X, AUGUST 2022

4

(a)

(b)

A

B

C

D

E

F

Material ID(c)Fig. 2. (a) Appearance of materials. (b) Force-displacement relationships of materials. (c) Compliance of each material.

of presentation of the visual and haptic stimuli. In the visualhaptic task, participants were presented with the haptic stimuli
first and, three seconds later, with the visual stimuli. In thehaptic-visual task, the order of presentation was reversed. Halfof the participants performed the visual-haptic task for allmaterials first, and the haptic-visual task for all materials
second. The other half of the participants performed thesetasks in reverse order.
The reason why we configured two tasks (the haptic-visualtask and the visual-haptic task) was not to investigate the effectof order but rather to cancel it out.
D. Optimization Algorithm
This study adopted a Bayesian framework with a GaussianProcess [23] to optimize visual parameters. A simple way to
optimize two parameters is by a grid search, a time-consuming
method that allows comprehensive exploration of a twodimensional space. A Bayesian optimization with a Gaussian
process, on the other hand, allows us to specify within asmall number of iterations the visual parameters that maximize
the participants’ judgments of the similarity between haptic
and visual softness. In the optimization process, the algorithm
takes the visual parameters which the participants chose in
the current trial as its input. Next, the algorithm updates theobjective function with a Gaussian process. Finally, the algorithm searches for and proposes a new set of visual parametersthat is likely to induce the participant’s preferential choice inthe next trial on the basis of its preference learning featuresof exploitation (choosing points near a previously observed
optimum) and exploration (choosing points in areas that havenot been well explored) [24]. Thus, as the optimized values
were determined on the basis of the visual parameters thatthe participants preferentially chose, it followed that we could
conclude that the optimized value of the visual parameters
was related to the participant’s preferential judgment for thehaptic-visual softness matching. The objective function wasiteratively updated based on the parameters the participantschose. Note that the algorithm used in our study could
simultaneously optimize both visual parameters (indentation
depth and speed), without being specialized to either oneof them. The optimization procedure was implemented using
preference learning with a Gaussian process in the Python
GPro library [25].
E. Procedure
The experiment was programmed using PsychoPy [26].
In each trial, one material was positioned in front of theparticipant’s right hand so that a smooth surface of the material
was uppermost. The exact same type of material was used forboth the haptic and visual stimuli. For the haptic stimulus
presentation, participants were allowed to compress the top
surface of the material with their index finger once only. To
prevent them accurately assessing the indentation depth oftheir finger press, participants were not allowed to repeatedly
touch the materials.
For the visual stimuli, two video clips were shown in which
each material was indented, and the participants watched them.
One of the clips in each trial was the same clip that theparticipant had selected in the preceding trial. The other clip
was a video newly created by our algorithm on the basis of theclip selected in the preceding trial. In each clip, the material
was indented to a certain depth at a certain speed. The clips
were looped.
After the presentation of the haptic and visual stimuli,
participants were asked to report their judgment as to which of
the two video clips contained a material whose softness wasmost similar to the material they touched with their fingers.
After participants had provided their answers, the algorithm
updated the model and suggested new values of indentation
depth and speed for the video clip to be presented in the© 2023 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission.See https://www.ieee.org/publications/rights/index.html for more information.

This article has been accepted for publication in IEEE Transactions on Haptics. This is the author's version which has not been fully edited andcontent may change prior to final publication. Citation information: DOI 10.1109/TOH.2023.3269016

JOURNAL OF LATEX CLASS FILES, VOL. XX, NO. X, AUGUST 2022

A

B

5

C

D

E

F

Initial
frame

Frame
at 18mm
depth

Last
frame

Low
complianceHighcompliance

Fig. 3. Snapshots of raw videos of each material.

subsequent trial. The software created a new clip based on
the suggested values. As described above, the new clip andthe clip selected by the participant in the current trial were
presented in the subsequent trial.
There were 25 trials in a set. We set the number oftrials at 25 so that the visual indentation depth and speed
would converge within one set. On the basis of a preliminary
examination, we expected that they would converge within
the space of 25 trials. The presented visual indentation depth
could vary from 1.0 mm to 18.0 mm. The presented visual
indentation speed could vary from 2.0 mm/s to 15.0 mm/s.
Based on the author’s preliminary observations, the optimized
values were expected to be in a range up to 10. Thus, theinitial values for indentation depth and speed were randomly
sampled from their variation ranges. One task consisted of six
sets, corresponding to six materials. The presentation orderof the six materials was assigned pseudo-randomly to eachparticipant. There were 2 task types for each participant. Thus,
each participant performed 300 trials (=25 trials × 6 sets × 2
tasks).
Before the 1st trial and after the 5th, 10th, 15th, 20th, and25th trials, participants were also asked to provide subjective
rating scores of the extent to which the softness of the material
in the clip matched the softness of the material they indented.
In the 1st trial, participants rated one of two clips madeby the experimental software. It was determined at random
which clip was rated. In the 5th, 10th, 15th, 20th, and 25thtrials, participants rated the clip they selected in the respective
trials. Participants reported the degree of consistency between
haptic and visual softness as a consequence of the sequential
haptic-visual matching on a seven-point scale with 1 for “donot match at all”, 2 for “do not match”, 3 for “don’t matchvery much”, 4 for “neutral”, 5 for “somewhat match”, 6 for“match”, and 7 for “match perfectly”.
III. R ESULTS
This section is laid out as follows. First, we investigate
whether the visual indentation depth and/or visual indentation
speed after optimization vary with the material compliance.
Second, we examine the relationship of indentation depth
between touch and vision. Finally, we check the improvementin the subjective rating for softness matching arising from theoptimization since it is still unclear whether the evolution
and variation of the optimized visual parameter cause theimprovement in the subjective rating for softness matching.
We report only the statistical results critical to interpreting theexperimental results.
A. Optimization of Visual Parameters
Figures 4(a) and (b) show the evolution of visual indentation
depth and indentation speed. It appears that, while there were
wide differences in the visual indentation depth among thematerials, there were only small differences in the visual
indentation speed.
Generally, it is assumed that the optimized value in thelater trials is more reliable. Thus, we focused on the visualparameters chosen for the last trial (see Figure 4(c)) andinvestigated whether the visual parameters were statisticallydifferent between the materials. If the participant successfully
optimized the visual parameter so that the visual softness
matched the material compliance, the visual parameter should
correspond to the material compliance. Since the materials
used in this experiment had different levels of compliance,© 2023 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission.See https://www.ieee.org/publications/rights/index.html for more information.

This article has been accepted for publication in IEEE Transactions on Haptics. This is the author's version which has not been fully edited andcontent may change prior to final publication. Citation information: DOI 10.1109/TOH.2023.3269016

JOURNAL OF LATEX CLASS FILES, VOL. XX, NO. X, AUGUST 2022

(a)

(c)

6

Material ID

Material ID(b)

Material ID(d)*: p<0.05**: p<0.01***: p<0.001

Cohen’s dFig. 4. (a) Evolution of visual indentation depth. (b) Evolution of visual indentation speed. The shadowed area shows 95 %C.I. (c) Optimized visual indentation
depth and speed after the last trial. (d) Result of multiple comparisons between material pairs for optimized visual indentation depth after the last trial. The
color of the cell denotes Cohen’s d value, and the asterisk denotes the significance of the difference. Bonferroni corrected p-value was used for significancejudgment.

the optimized visual parameter should also differ between
materials if the visual parameter was in fact relevant to thehaptic-visual sequential matching of softness.
1) Optimization of Visual Indentation Depth: We conducteda one-way repeated measures ANOVA using material as a fixedfactor on visual indentation depth. Note that we averaged thedata of the two tasks (visual-haptic and haptic-visual tasks)
per participant. There was a significant main effect of material[F (5, 64) = 27.41, p < 0.001, ηp2 = 0.68]. In post-hoc test, we
conducted multiple comparisons with Bonferroni correction
for the significant main effects of materials. There weresignificant differences only between the following material
pairs: A-D, A-E, A-F, B-D, B-E, B-F, C-D, C-E, C-F, D-E,
and D-F (p < 0.05). See Cohen’s d for each of the differencesin Figure 4(d).
Our results showed that the optimized values of the visual
indentation depth depended on the materials, indicating thatthe optimization was successfully achieved in accordance withthe participants’ judgment for the haptic and visual sequential
matching of material softness.
2) Optimization of Visual Indentation Speed: We also
conducted a one-way repeated measures ANOVA on visual
indentation speed. There was no significant main effect ofmaterial [F (5, 64) = 0.35, p = 0.88, ηp2 = 0.026, observed
power = 0.66]. This showed that the optimized values ofthe visual indentation speed did not depend on the materials,
indicating that the visual indentation speed was not used forthe sequential matching of material softness.

One might suspect that the difference in the optimization
outcomes between visual parameters was due to the specialization of the algorithm to one of the parameters. That is,there was a possibility that the algorithm tried to preferentially
optimize one parameter over the other. To assess this possibility, we checked the value obtained by dividing the standard
deviation of the suggested values by the range of each visualparameter. If the values differed between the two parameters,
we would have been able to conclude that the algorithm treatedthem differently. The value obtained for visual indentation
depth was 0.294 and that for visual indentation speed was
0.290. The values were thus almost identical. These results
indicate that the algorithm treated the two parameters without
specialization to either one of them.
3) Correlation between Visual Indentation Depth and Haptic Indentation Depth: As described in the previous sections,
we found that the visual indentation depth was used for softness matching. It is clear that indentation depth was hapticallyrecognizable during the indentation the participant made in
the real material with their finger. There is a possibility thatthe participants were able to evaluate the similarity between
haptic and visual softness by linking the haptic indentation
depth with the visual one. To explore this possibility, during
the experiment, the participant’s finger position was tracked
and analyzed. Using the data from the tracked finger positions,
we sought to clarify the relationship between visual and haptic
indentation depth.
We averaged the optimized values of the visual indentation© 2023 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission.See https://www.ieee.org/publications/rights/index.html for more information.

This article has been accepted for publication in IEEE Transactions on Haptics. This is the author's version which has not been fully edited andcontent may change prior to final publication. Citation information: DOI 10.1109/TOH.2023.3269016

JOURNAL OF LATEX CLASS FILES, VOL. XX, NO. X, AUGUST 2022

Fig. 5. Relationship between indentation depth in the video clips (visual
indentation depth) and depth of the indentation made by the participant’s
finger (haptic indentation depth).

depth for each combination of participant and material. We
also averaged the indentation depth applied by each participant’s finger (haptic indentation depth) using the data for
tracked finger positions during all trials. The relationshipbetween them is shown in Figure 5. The Pearson correlation
coefficient was 0.77 (p < 0.001). The results of the correlation
analysis indicate that the variation of the optimized values ofthe visual indentation depth was positively related to the variation of the haptic indentation depth (i.e., the indentation madeby the participant’s finger). The strong correlation suggeststhat the participants were able to evaluate the consistency of
haptic and visual softness by linking the haptic indentation
depth with the visual indentation depth. The results shed lighton a new cross-modal matching mechanism that likely operates
in scenarios where participants are not allowed to evaluate
simultaneously the information given to each haptic and visual
modality, but are required to judge the consistency of haptic
and visual softness sequentially.
B. Subjective Rating for the Matching between Haptic andVisual Softness
In the previous section, we found that visual indentation
depth was optimized depending on the material to match its
softness. Still, it was unclear if this optimization actuallycaused the improvement in subjective ratings for softness
matching.
Figure 6(a) shows rating scores for the matching between
haptic and visual softness for each material. To clarify whether
the evolution of optimized visual indentation depth caused thechange in subjective rating for softness matching, we first applied an Aligned Rank Transform (ART) [27], [28] to the data.
In general, the rating scores were not normally distributed, andthus, it was not deemed appropriate to parametrically analyze
the raw scores. ART is a procedure developed to performthe ANOVA with non-normally distributed data, wherein therating score is first “aligned” for each of main effects and7interactions, and then ranked. The obtained rank can thenbe the subject of an ANOVA. Unlike other nonparametricstatistical tests such as the Kruskal-Wallis test, ANOVA withthe use of ART enables analysis using multiple factors withappropriate Type I error rates and suitable powers. We used
trials and material as fixed factors in the ART. The main
effect of trials was significant [F (5, 396) = 18.0, p < 0.001,
ηp2 = 0.19], as was the main effect of material [F (5, 396) =4.5, p < 0.001, ηp2 = 0.05]. The interaction effect was not
significant [F (25, 396) = 1.0, p = 0.44, ηp2 = 0.061].
As post-hoc tests of the significant main effects, we
conducted multiple comparisons with Bonferroni correction
for each of the significant main effects using the Aligned
Rank Transform Contrasts (ART-C) procedure [29]. ART-C
is an additional align-and-rank procedure to facilitate posthoc pairwise comparisons without inflating Type I error rates.
Regarding the main effect of trials, there were significant
differences between the following trial pairs: 1st-5th, 1st-10th,
1st-15th, 1st-20th, 1st-25th, 5th-15th, 5th-20th, and 5th-25th(p < 0.05) (see Figure 6(b)). Regarding the results of themultiple comparisons due to the main effect of materials,
please see Figure 6(c).
In addition, to clarify whether the subjective rating forsoftness matching was significantly improved after optimization, we calculated the 95% confidence interval (CI) of thesubjective rating scores in the last trial for each material
based on 10,000 bootstrap samples [30]. If the Bonferronicorrected 95% CI did not overlap four (i.e., “neutral”), we
could conclude that the subjective rating was significantly
improved over neutral. This analysis showed that the 95% CIs
of subjective rating for all materials were significantly larger
than four.
Consequently, the optimization for the visual indentation
depth improved subjective rating scores for softness matching
up to the 10th trial. Moreover, the rating scores with theultimate optimization value were significantly higher than the“neutral” criterion. The results indicate that the optimization ofvisual indentation depth significantly improves the subjective
consistency of material softness between touch and vision.
IV. D ISCUSSION
The present study for the first time investigated which visualparameters were used for the sequential matching of material
softness. We discovered that visual indentation depth, rather
than visual indentation speed, was used for optimization inhaptic-visual sequential matching (see Figure 4). One of ourprevious studies [19] has shown that both indentation depth
and speed significantly contributed to the “visual” judgmentof material softness. That study also showed that visual
indentation depth had stronger effects on the judgment ofsoftness than visual indentation speed. The present study hasreported that only indentation depth was used by the perceptual
system to match the material softness between haptic andvisual modalities. Thus, it is our conclusion that participants
utilized only visual indentation depth, which has been reported
to have a pronounced influence on visual softness, in thecontext of the sequential matching of haptic-visual softness.

© 2023 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission.See https://www.ieee.org/publications/rights/index.html for more information.

This article has been accepted for publication in IEEE Transactions on Haptics. This is the author's version which has not been fully edited andcontent may change prior to final publication. Citation information: DOI 10.1109/TOH.2023.3269016

JOURNAL OF LATEX CLASS FILES, VOL. XX, NO. X, AUGUST 2022

(b)

(c)

1

Subjective rating for softness matching(a)8

Material ID11cohen’s dcohen’s dFig. 6. (a) Evolution of subjective rating for softness matching for each material. Error bars denote 95 %C.I. (b) Result of multiple comparisons between
trials as a post-hoc test for the main effect of trial on the subjective rating for softness matching. (c) Result of multiple comparisons between materials as a
post-hoc test for the main effect of material on the subjective rating for softness matching.

One thing we should pay attention to is the range of theindentation speed in our experimental settings. The range waslimited to between 2 mm/s and 15 mm/s. If the speed could bemanipulated beyond this range, the effect of indentation speed
might be found to have an effect on perceived softness.
Our results showed that visual indentation depth was highly
correlated with haptic indentation depth. We speculate that theparticipants might imagine ideal values for visual indentation
depth based on the indentation depth of their own finger anduse this to judge the similarity of the softness. This finding
contributes to our understanding of how the perceptual systemjudges the consistency of haptic and visual softness in contexts
where participants are not allowed to evaluate the information
given to each sensory modality at the same time.
As for the degree to which haptic-visual sequential matchingof material softness is achieved as a result of optimization, thesubjective rating scores were significantly higher than 4 (i.e.,“neutral”). Furthermore, the subjective rating scores reached
5 (i.e., “visual and haptic softness somewhat match”) or 6
(i.e., “visual and haptic softness match”) on average for allmaterials. This subjectively high satisfaction regarding thesequential matching indicates that optimization was carriedout successfully. The present study might provide helpful
information for haptic-visual sequential matching design. Asa real-world problem, there are many cases where a designer
wants to match visual softness with haptic softness. For
example, if online sellers want to sell soft cushions online,
it would be helpful if they could match the visual informationin the cushions’ photos/video clips on their website with theactual haptic softness of the cushions. The methods used inthis study are promising for online sellers who want to ensure
the subjective matching of haptic and visual softness of acushion, or other product; the method does not require muchtime investment, with the entire process often being completed
in less than about ten minutes.
The pattern of convergence of subjective rating scoresfor the consistency between haptic and visual softness wasdependent on the type of material (see Supplementary Table 2).
Specifically, rating scores for the stiffer materials (A, B, C, andD) were higher than those for the softer materials (E and F).
The present study configured the initial values in optimizationfor visual indentation depth and visual indentation speeds tobe common across the materials. We suggest that the optimalvalue of the visual indentation depth for the stiffer materials
might be nearer to the initial values than the optimal value forthe softer materials, and that this caused the difference in thepattern of convergence of the consistency rating scores among
the materials. It may be possible to moderate the differencein the rating scores among the materials by appropriately
configuring the initial values in optimization and shortening
the time to convergence.
The potential impact of demand characteristics on participants’ subjective rating scores should be noted. The participant’s task was to select the video clips for a certain material
that gave a similar softness impression to the softness of thehaptic stimuli. Thus, there was a possibility that the participants noticed the purpose of the study and tried to rate theconsistency of haptic-visual softness in accordance with theexperimenter’s expectations. Still, if the demand characteristic
was a unitary cause of the improvement of the rating scores,
one would be required to explain why the rating scores did
not improve after the 10th trial. It is natural to interpret thisto mean that the optimization was completed to produce asufficient level of subjective consistency by the 10th trial. Still,there is also a possibility that the current task is contaminatedwith a demand characteristic, and thus, future studies need to
examine this issue with an improved protocol whereby, forexample, the influence of demand characteristics is reducedby mixing trials in multiple sets or including additional trials
with unrelated stimuli.
The type of material used in our experiment was limited toa specific rubber-like material. Hence, the generalizability of
the conclusion to other categories of elastic materials, such assoft plastics or hard rubbers, is probably promising, but has notbeen confirmed yet. Further research is needed to determine
the extent to which the findings can be generalized to different
categories of elastic materials.
In our experiment, participants were required to indent amaterial with their finger. We decided to ask them to do so
because it is known from exploratory procedure studies thatpeople judge the softness of materials mainly by pushing [31].
We expect that even if the participants were allowed to touch© 2023 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission.See https://www.ieee.org/publications/rights/index.html for more information.

This article has been accepted for publication in IEEE Transactions on Haptics. This is the author's version which has not been fully edited andcontent may change prior to final publication. Citation information: DOI 10.1109/TOH.2023.3269016

JOURNAL OF LATEX CLASS FILES, VOL. XX, NO. X, AUGUST 2022

the material freely, the softness judgment would be performed
by indenting the material with their finger and/or palm. The
previous study just cited has also suggested that different
modes of touch, such as rubbing or lifting, provide information
about the roughness or weight of a material, rather than about
its softness. Thus, if a different mode of touch than indenting
was required, it would not be easy for participants to make anassessment of material softness. As a result, the optimizationof visual indentation depth and speed would not proceed asintended.
Our study successfully identified the visual parameter that
supports haptic-visual softness matching. Due to the easewith which visual indentation depth can be manipulated andpresented in the video clips showing object deformation, ourfindings will be useful in a number of application scenarios.
On the other hand, there is a risk of some ambiguity in thevisual indentation cue due to the camera angle and/or the
distance between the camera and the material. Even in thatcase, we suggest that other visual features may contribute to
the inference of indentation depth, and thus, visual softness.
For example, it is known that texture deformation serves asa cue to softness perception even when the visual indentation
depth is not explicitly displayed [32], [33]. Moreover, otherhaptic and even auditory parameters may also play a part in
the determination of material softness perception. Hence, toestablish the consistency of haptic-visual softness in a realworld setting wherein more parameters are involved in thedetermination of perceptual softness, it may be necessary toconsider optimizations in a higher dimensional space than
the two-dimensional space that the present study assumed.
The exploration of this as yet unknown information providing
effective cues for haptic-visual softness matching is left as anopen issue for future research.

R EFERENCES[1] J. Peck and T. L. Childers, “Individual differences in haptic information
processing: The “need for touch” scale,” Journal of Consumer Research,
vol. 30, no. 3, pp. 430–442, 2003.
[2] Statista, “Retail e-commerce sales worldwide from 2014 to
2026,”
https://www.statista.com/statistics/379046/worldwide-retaile-commerce-sales/#statisticContainer, (Accessed on 30/01/2023).
[3] UNCTAD, “Covid-19 has changed online shopping forever, survey
shows,” https://unctad.org/news/covid-19-has-changed-online-shoppingforever-survey-shows, (Accessed on 30/01/2023).
[4] E. C. Hirschman and M. B. Holbrook, “Hedonic consumption: emerging
concepts, methods and propositions,” Journal of marketing, vol. 46,
no. 3, pp. 92–101, 1982.
[5] M. D. Luca, Multisensory Softness, Perceived Compliance from Multiple
Sources of Information, ser. Springer Series on Touch and Haptic
Systems, 2014.
[6] A. Pasqualotto, M. Ng, Z. Y. Tan, and R. Kitada, “Tactile perception ofpleasantness in relation to perceived softness,” Scientific reports, vol. 10,
no. 1, pp. 1–10, 2020.
[7] R. Kitada, M. Ng, Z. Y. Tan, X. E. Lee, and T. Kochiyama, “Physical correlates of human-like softness elicit high tactile pleasantness,”
Scientific reports, vol. 11, no. 1, pp. 1–10, 2021.
[8] R. L. Klatzky and J. Peck, “Please touch: Object properties that invite
touch,” IEEE Transactions on Haptics, vol. 5, no. 2, pp. 139–147, 2011.
[9] M. Di Luca, B. Knörlein, M. O. Ernst, and M. Harders, “Effectsof visual–haptic asynchronies and loading–unloading movements oncompliance perception,” Brain research bulletin, vol. 85, no. 5, pp. 245–
259, 2011.

9[10] A. Lécuyer, S. Coquillart, A. Kheddar, P. Richard, and P. Coiffet, “Pseudo-haptic feedback: can isometric input devices simulate
force feedback?” in Proceedings IEEE Virtual Reality 2000 (Cat. No.
00CB37048). IEEE, 2000, pp. 83–90.
[11] A. Metzger, A. Lezkan, and K. Drewing, “Integration of serial sensory
information in haptic perception of softness.” Journal of Experimental
Psychology: Human Perception and Performance, vol. 44, no. 4, p. 551,
2018.
[12] C. Cellini, L. Kaim, and K. Drewing, “Visual and haptic integration in
the estimation of softness of deformable objects,” i-Perception, vol. 4,
no. 8, pp. 516–531, 2013.
[13] B. Xiao, W. Bi, X. Jia, H. Wei, and E. H. Adelson, “Can you see
what you feel? color and folding properties affect visual–tactile material
discrimination of fabrics,” Journal of vision, vol. 16, no. 3, pp. 34–34,
2016.
[14] M. Cavdan, K. Drewing, and K. Doerschner, “The look and feel ofsoft are similar across different softness dimensions,” Journal of vision,
vol. 21, no. 10, pp. 20–20, 2021.
[15] M. Cavdan, K. Doerschner, and K. Drewing, “Task and material properties interactively affect softness explorations along different dimensions,”
IEEE Transactions on Haptics, vol. 14, no. 3, pp. 603–614, 2021.
[16] O. Caldiran, H. Z. Tan, and C. Basdogan, “Visuo-haptic discriminationof viscoelastic materials,” IEEE Transactions on Haptics, vol. 12, no. 4,
pp. 438–450, 2019.
[17] E. Fakhoury, P. R. Culmer, and B. Henson, “The effect of indentation
force and displacement on visual perception of compliance,” in 2015
IEEE World Haptics Conference (WHC). IEEE, 2015, pp. 88–93.
[18] V. C. Paulun, F. Schmidt, J. J. R. van Assen, and R. W. Fleming, “Shape,
motion, and optical cues to stiffness of elastic objects,” Journal of vision,
vol. 17, no. 1:20, pp. 1–22, 2017.
[19] Y. Ujitoko and T. Kawabe, “Perceptual judgments for the softness ofmaterials under indentation,” Scientific reports, vol. 12, no. 1, pp. 1–11,
2022.
[20] Y. Ujitoko and T. Kawabe, “Visual estimation of the force applied byanother person,” Scientific reports, vol. 12, no. 1, pp. 1–9, 2022.
[21] M. Piovarči, D. I. Levin, J. Rebello, D. Chen, R. Ďurikovič, H. Pfister,W. Matusik, and P. Didyk, “An interaction-aware, perceptual modelfor non-linear elastic objects,” ACM Transactions on Graphics (TOG),
vol. 35, no. 4, pp. 1–13, 2016.
[22] S.
S.
Center,“Tango(rubberlike),”https://support.stratasys.com/en/materials/polyjet/tango,(Accessed
on 30/01/2023).
[23] W. Chu and Z. Ghahramani, “Preference learning with gaussian processes,” in Proceedings of the 22nd international conference on Machinelearning, 2005, pp. 137–144.
[24] E. Brochu, V. M. Cora, and N. De Freitas, “A tutorial on bayesian
optimization of expensive cost functions, with application to active
user modeling and hierarchical reinforcement learning,” arXiv preprint
arXiv:1012.2599, 2010.
[25] C. Alkhassim, “Gpro,” https://github.com/chariff/GPro, 2021, (Accessedon 14/11/2021).
[26] J. W. Peirce, “Psychopy—psychophysics software in python,” Journalof neuroscience methods, vol. 162, no. 1-2, pp. 8–13, 2007.
[27] J. Higgins, R. Blair, and S. Tashtoush, “The aligned rank transform
procedure,” in Proceedings of the Conference on applied statistics inagriculture (USA), 1990.
[28] J. O. Wobbrock, L. Findlater, D. Gergle, and J. J. Higgins, “The aligned
rank transform for nonparametric factorial analyses using only anova
procedures,” in Proceedings of the SIGCHI conference on human factors
in computing systems, 2011, pp. 143–146.
[29] L. A. Elkin, M. Kay, J. J. Higgins, and J. O. Wobbrock, “An aligned rank
transform procedure for multifactor contrast tests,” in The 34th Annual
ACM Symposium on User Interface Software and Technology, 2021, pp.
754–768.
[30] B. Efron and R. J. Tibshirani, An introduction to the bootstrap. CRC
press, 1994.
[31] S. J. Lederman and R. L. Klatzky, “Hand movements: A window into
haptic object recognition,” Cognitive psychology, vol. 19, no. 3, pp. 342–
368, 1987.
[32] F. Argelaguet, D. A. G. Jáuregui, M. Marchal, and A. Lecuyer, “Elastic
images: Perceiving local elasticity of images through a novel pseudohaptic deformation effect,” ACM Transactions on Applied Perception(TAP), vol. 10, no. 3, pp. 1–14, 2013.
[33] P. Punpongsanon, D. Iwai, and K. Sato, “Softar: Visually manipulating
haptic softness perception in spatial augmented reality,” IEEE transactions on visualization and computer graphics, vol. 21, no. 11, pp.
1279–1288, 2015.

© 2023 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission.See https://www.ieee.org/publications/rights/index.html for more information.

This article has been accepted for publication in IEEE Transactions on Haptics. This is the author's version which has not been fully edited andcontent may change prior to final publication. Citation information: DOI 10.1109/TOH.2023.3269016

JOURNAL OF LATEX CLASS FILES, VOL. XX, NO. X, AUGUST 2022

10

V. B IOGRAPHY S ECTION

Yusuke Ujitoko received the B.E. degree in mechanical engineering and the M.A.E. degree in interdisciplinary information studies from the Universityof Tokyo, Tokyo, Japan, in 2014 and 2016, respectively, and the Ph.D. degree from The University
of Electro-Communications, Chofu, Japan, in 2020.
Since 2020, he has been a Researcher at NTT
Communication Science Laboratory, Atsugi, Japan.
From 2016 to 2020, he was a Member of the R&D
Group, Hitachi, Ltd., Tokyo, Japan. His researchinterests include applied haptic perception and haptic
interfaces.

Takahiro Kawabe is a senior distinguished researcher, the leader of Sensory Representation
Group, Human Information Science Laboratories,
NTT Communication Science Laboratories. He received a Doctor of Psychology from Kyushu University, Fukuoka, in 2005. In 2011, he joined
NTT Communication Science Laboratories, where
he studies basic and applied aspects of humanperception. He received the 2013 JPA Award forInternational Contributions to Psychology: Award
for Distinguished Early and Middle Career Contributions from the Japanese Psychological Association. In 2018, he was also
awarded The Young Scientists’ Prize of The Commendation for Scienceand Technology by the Minister of Education, Culture, Sports, Science andTechnology. He is a member of the Vision Sciences Society and the Vision
Society of Japan.

© 2023 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission.See https://www.ieee.org/publications/rights/index.html for more information.