This article has been accepted for publication in IEEE Transactions on Visualization and Computer Graphics. This is the author's version which has not been fully edited andcontent may change prior to final publication. Citation information: DOI 10.1109/TVCG.2023.3254522

JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2015

1

Softness Perception of Visual Objects Controlled
by Touchless Inputs: The Role of Effective
Distance of Hand Movements
Takahiro Kawabe, Member, IEEE, Yusuke Ujitoko, Member, IEEE,
Abstract—Feedback on the material properties of a visual object is essential in enhancing the users’ perceptual experience of theobject when users control the object with touchless inputs. Focusing on the softness perception of the object, we examined how theeffective distance of hand movements influenced the degree of the object’s softness perceived by users. In the experiments,
participants moved their right hand in front of a camera which tracked their hand position. A textured 2D or 3D object on display
deformed depending on the participant’s hand position. In addition to establishing a ratio of deformation magnitude to the distance ofhand movements, we altered the effective distance of hand movement, within which the hand movement could deform the object.
Participants rated the strength of perceived softness (Experiments 1 and 2) and other perceptual impressions (Experiment 3). A longer
effective distance produced a softer impression of the 2D and 3D objects. The saturation speed of object deformation due to theeffective distance was not a critical determinant. The effective distance also modulated other perceptual impressions than softness.
The role of the effective distance of hand movements on perceptual impressions of objects under touchless control is discussed.
Index Terms—Material perception, Pseudo-haptics, Touchless inputs, Softness✦

1

I NTRODUCTIONPERCEIVING material properties is an essential experience in our daily lives. For example, by stretching acloth fabric, we can perceptually judge whether the fabricis soft or hard, whether the fabric feels good or not, and soon. Similar kinds of interaction between users’ actions andvisual objects likely enhance the perceptual understandingof material properties in virtual and augmented reality
scenes [1], [2], [3]. By controlling visual objects’ motion
direction and speed as visual feedback of user’s actions, it
is possible to give users the impression of various material
properties such as weight/mass [4], [5], [6], [7], [8], [9], stiffness/softness [10], [11], [12], surface roughness [13], [14],and so forth. To understand what kind of visual presentation
method gives the desired impression of material properties
effectively, it is necessary to closely examine the relationships between users’ actions, the type of visual feedback,and the impressions of material properties that are obtained.
Recently, the manipulation of visual objects throughtouchless inputs has attracted attention [15], [16]. Someinterfaces, for example, cars’ dashboard interfaces [17], [18],
KIOSK [19], and digital signage [20] employ touchless inputs. In order to achieve rich interaction between users anda visual object through touchless inputs, it is necessary to
examine in detail what kind of visual feedback of the inputs
is effective in providing the user with an appropriate impression of the material properties of the object. Some previous studies [12], [21] have shown that the physics-based
manipulation of visual feedback can enrich the perception of••E-mail: takkawabe@gmail.com
T. Kawabe and Y. Ujitoko are with NTT Communication Science Laboratories, Nippon Telegraph and Telephone Corporation, Atsugi, Kanagawa,
243-0198, Japan.

Manuscript received April 19, 2005; revised August 26, 2015.

three-dimensional carousels [21], object heaviness [22], andelastic materials [12] controlled by touchless gesture inputs.
One drawback of touchless input systems is that touchless gestures do not serve as practical inputs for controllingthe appearance of visual feedback if not properly captured
by sensing devices such as a camera. In such cases, thevisual feedback is not updated, and thus, only static images
are provided to the user. Let us consider the case whereusers pull a deformable virtual object laterally and generate
its deformation on the basis of touchless gesture inputs.
When the camera no longer captures the user’s pulling
gestures due to occlusion which is something blocking thepath of view, or the limits of the effective angle of view, theobject stops deforming. It was unclear how this unexpectedcessation of deformation would affect the user’s perceptionof the material. While it is possible to employ additional
methods such as electromyography [23], electromagnetic
interfaces [24], air-pressure sensors [25], and Radio Frequency Identification [26] to detect users’ gestures which
are not captured by a camera, this study focuses on the casewhere the only method for detecting gestures is the camera.
Cameras as sensing devices are widespread, often attached
to notebook PCs, and their implementation is low-cost. On
the other hand, specialized sensing devices such as thosementioned above are not yet common and are expensive.
For these reasons, it is worth focusing only on situations
where gestures are detected by a camera.
The purpose of this study was to elucidate whether andhow the cessation of object deformation during touchlessgestures affected the perception of object softness and otherobject properties. The cessation of object deformation likely
causes the change of motion speed. It is known that thechange in motion speed can create the impression of friction,
gravity, or viscosity [11], resistance [27], and collision [28],© 2023 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission.See https://www.ieee.org/publications/rights/index.html for more information.

This article has been accepted for publication in IEEE Transactions on Visualization and Computer Graphics. This is the author's version which has not been fully edited andcontent may change prior to final publication. Citation information: DOI 10.1109/TVCG.2023.3254522

JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2015[29], [30], [31]. On the other hand, it was still unclear
whether the change in motion speed, which was caused by
the cessation of object deformation, was a factor to modulate perceived softness. Thus, in Experiment 1, we explore
whether the perceived softness of the object is altered when
the object stopped deforming at some point with touchlessgestures. Specifically, we manipulate the effective distanceof hand movement within which the virtual object could deform in response to hand movement and examined whether
the effective distance of hand movements could modulate
the perceived softness of the object. In Experiment 2, we
examine whether the saturation speed of object deformation
during touchless inputs, which is varied with the effective
distance of hand movements, can modulate the level of theperceived softness of the object. To further gain insights
into the effect of the cessation of object deformation on
perceptual impression, in Experiment 3, we examine how
the cessation of the object deformation during touchless
inputs can change the perceptual impression of objects
other than softness. Based on the results, we speculate themechanism underlying the change in perceptual softness by
the cessation of object deformation during touchless inputs.
In the second section of this article, we describe theprevious literature relevant to the present study. In the third
section, we specify the issue we treat in this study. In the
fourth and fifth sections, we describe the two experiments
we conducted. In the sixth section, we discuss the significance, another interpretation, and limitations of the present
study.

2

R ELATED S TUDIES

2.1

Softness Perception in Vision

Visual perception of material properties is one of the recent
hot topics in vision science [32], [33]. Most studies on thematerial properties have focused on optical (or surface)properties such as texture and glossiness. In addition, users
can perceive mechanical properties of materials [34], andseveral previous studies have investigated the mechanism
for the perception of the mechanical properties such aselasticity [35], [36], [37], viscosity [38], [39], [40], [41], andfabric stiffness [42], [43], [44].
Above all, the visual perception of softness (or relatedly, stiffness) has attracted the attention of scientists. The
most promising cue to the visual softness perception fora material that is indented by an external body is theindentation depth [45], [46] or the deformation magnitude[47]. Specifically, a greater magnitude of indentation depth
generates a larger magnitude of deformation in the image,producing an impression of greater softness for the indented
material. In addition, the speed of indentation plays aneffective cue in the perception of the softness of material.
A faster indentation generates a faster deformation in theimage, which produces the impression of a greater softnessof the material [35], [47]. However, analyzing image motion
in stimulus video clips, the previous study [47] reported
that deformation magnitudes were stronger cues to theperception of the softness of a material than the deformation
speed.

2

2.2 Softness Perception of Objects Deformed underUser’s Manual Control
The perception of the softness of an elastic object has
been examined in a situation wherein users controlled theappearance of the object displayed on a monitor by manipulating real objects and/or input devices. A previous
study [48] has shown that the judgment of the stiffness ofreal spring devices could be significantly influenced by thevisual feedback of the spring. Another previous study [11]has found that users could discriminate stiffness between areal spring that was controlled in the real world and a visual
spring that was controlled by using an isometric device.
Similarly, a recent study [49] also showed that when users
controlled the deformation of a rectangle on a smartphonemonitor by applying force to the side of the smartphone,
the rectangle’s deformation could generate the impressionof stiffness of the rectangle which was comparable with thestiffness of the real object. In a video see-through system, aprevious study [50] showed that the perceptual judgment ofan object’s softness was strongly altered by the combinationof the appearance of users’ pinching gestures as well as theone of a real elastic object. Although the previous study
was related to the judgment of object softness in augmented
reality, a more recent study [51] has reported that a visualobject under the user’s manual control was judged to besofter in augmented reality than in virtual reality settings.

2.3 Softness/stiffness Perception of Objects in Touchless Controls
It has also been examined how the user’s touchless gesture
to control a visual object could contribute to the generationof the stiffness sensation of the object. There are several
previous studies proposing an AR system that gives users a
pseudo-softness of an object [12], [52] and a pseudo-contact
to an object [53] while the user makes a gesture of controlling the object with a bare hand. Hereinafter, we describe
one of the previous studies in detail. The previous study[12] instructed experimental participants to make a gesture
to pull a visual object displayed on the monitor laterally. A
camera-based hand tracker was used to track the positionof the participant’s hands. The magnitude of deformation
applied to the visual object was changed in accordancewith the tracked position of the hands. The participants
were asked to report the stiffness of the object on a 5-point
scale. The results of that study showed that the reported
stiffness was dependent on the magnitude of horizontal
deformation of the visual object in relation to the distanceof hand movements. Specifically, the stiffness increased inaccordance with the ratio of the deformation to the distanceof the hand movements. The previous study also found that
not only the horizontal deformation but also the verticalone also contributed significantly to the stiffness perception,
consistent with the idea that the sensory system in users
internalizes the physical characteristics called the Poisson
effect [54], [55] and uses the internalized representation of
the Poisson effect to judge the stiffness of the visual object.

© 2023 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission.See https://www.ieee.org/publications/rights/index.html for more information.

This article has been accepted for publication in IEEE Transactions on Visualization and Computer Graphics. This is the author's version which has not been fully edited andcontent may change prior to final publication. Citation information: DOI 10.1109/TVCG.2023.3254522

JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 20153

2.4 Contribution of The Present Study to Softness Perception Research

3

R ESEARCH I SSUE IN THE P RESENT S TUDY

Figure 1 shows a schematic description of the relationships
between the angle of the camera view, a user’s right hand
position, and an occluder, that is something that blocks thecamera view. In the angle of camera view (Figure 1 (a)), a
camera captures images of a user’s right hand (Figure 1 (b)),
and based on the images, a computer calculates the positionof the hand as being at certain coordinates and modifies theappearance of visual objects as feedback. The user’s righthand, when outside the angle of the camera view, will not be
captured (Figure 1 (c)). In addition, as shown in Figure 1 (d),if another object occludes the hand, the images of the user’s
hand will not be captured by the camera. When the imageof the user’s hand is not captured, the position of the user’s
hand is not updated. Consequently, the modification of theappearance of the visual object ceases. It was unclear how
users perceived the material properties of the object whenthe modification of the object’s appearance ceased during
the manipulation by touchless input.

Fig. 1. Graphical explanations of the relationships between the angle ofcamera view, the position of a user’s right hand, and an occluder. Panel(a) shows the angle of the camera view. Panel (b) shows the situation
wherein the user’s right hand falls into the angle of the camera view.
Panel (c) shows the situation wherein the hand is outside of the angle ofthe camera view. Panel (d) shows the situation in which the hand is not
captured by the camera due to the existence of an occluder even though
the hand falls within the angle of camera view.

4

E XPERIMENT 1: T HE R OLE OF E FFECTIVE D IS -TANCE OF H AND M OVEMENT

4.1

PurposeThe purpose of this experiment was to explore whether the
cessation of visual object deformation during the manipulation of the object influenced the participant’s judgment

250 (a)Deformation magnitude (pixels)

Although the previous studies have clarified the visualparameters to determine perceived softness, it is still unclear
how the change in object motion speed, which is likely
caused by the cease of object deformation during touchless
inputs, can influence softness perception. Examining theeffect of the change in object motion speed on the softness
perception will help us to understand the mechanism ofsoftness perception better and to determine the parameterswhen implementing soft objects in touchless inputs.

C/D ratio
0.50
0.75

0.25

(b)

1.00
(c)

200

150

100

50

0

0

5

10 0

5

10 0

5

10

Distance of hand movements (cm)Fig. 2. Variation of deformation magnitudes with the effective distanceof hand movements of (a) 5.4, (b) 7.2, and (c) 9.0 cm. Different lines
indicate the deformation magnitudes for different C/D ratio conditions.

of the object’s softness. The participants moved their righthand in front of a camera which was used to track the
hand’s position, while watching a visual object on themonitor. We manipulated the effective distance of hand
movements within which the hand movement caused theobject deformation (Figure 2). When the effective distanceof hand movements was shorter, the deformation ceased
earlier on the path of the hand movement. In other words,
the longer the effective distance of hand movements, thelarger the magnitude of the object deformation. It was predicted that the rating scores of the softness would increase
with the effective distance of hand movement because thelarger magnitude of the object deformation likely triggereda stronger softness impression of the object. We also tested
the effect of the ratio of the distance of hand movement
to the object deformation magnitude. Hereinafter, we callthe ratio the C(Control)/D(Display) ratio after the previous
studies [56]. Although the definition of the C/D ratio varies
depending on studies [3], in the present study, the larger the
C/D ratio caused the larger deformation with the smaller
hand movements. It was expected that the larger the C/Dratio, the greater the rating scores for the object’s softness.
We also checked the interaction between the effective distance of hand movements and the C/D ratio.
4.2

Method

4.2.1 Participants
Eighty-four (42 female and 42 male) people participated inthis experiment. The participants’ mean age was 40.38 (SD:12.11). The participants’ ages ranged from 20 to 59, and thenumbers of participants in the 20’s, 30’s, 40’s, and 50’s age
generations were 21, 20, 21, and 22, respectively. We did
not ask about the handedness of the participants. All of theparticipants were unaware of the specific purpose of theexperiment. The participants in this study were recruited by© 2023 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission.See https://www.ieee.org/publications/rights/index.html for more information.

This article has been accepted for publication in IEEE Transactions on Visualization and Computer Graphics. This is the author's version which has not been fully edited andcontent may change prior to final publication. Citation information: DOI 10.1109/TVCG.2023.3254522

JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2015

a Japanese online survey company and paid for their participation. Ethical approval for this study was obtained fromthe ethics committee at Nippon Telegraph and Telephone
Corporation (Approval number: R02-009 by NTT Communication Science Laboratories Ethics Committee). The experiments were conducted according to principles originatingin the 2008 Helsinki Declaration. Written informed consent
was obtained from all participants in this study.
4.2.2 Apparatus
This experiment was conducted online. Stimuli were presented on the monitor of the personal computer that theparticipants normally used. The position of the participant’s
hand was tracked by using the web camera which wasavailable in the participant’s environment. The positionof the participant’s hand was calculated using the coordinates of the camera image by using (handsfree.js https://handsfreejs.netlify.app/). The measured mean sampling
rate of the hand tracking was 30.747 Hz (SD: 13.223 Hz).
4.2.3 Stimuli
Stimuli, consisting of a horizontal arrow and a twodimensional rectangle against a black background, were
presented in the upper part of the monitor (see Figure 3).
The arrow indicated the range within which the participants
were required to move their right hand. In front of the arrow,the participants moved their hand from left to right or rightto left. The horizontal length of the arrow was 9 cm (thecorresponding pixel distance varied with the participant’s
environment). The left side of the arrow was located at thehorizontal center of the monitor and at 256 pixels above thevertical center of the monitor. While the system detected
the participant’s hand in front of the arrow, the arrow’scolor was green. Otherwise, its color was gray. By the color
change, we encouraged the participant to set and movetheir hand position in front of the arrow. The color change
unlikely influenced the results of the experiment. The centerof the rectangle was located 384 pixels to the left and 256
pixels above the center of the monitor. The rectangle initiallysubtended 128 (height) × 128 (width) pixels. The rectangle
had gray-scale two-dimensional 16 × 16 random noise as
texture. The noise intensity ranged from 0 to 255 in the RGB
values. The rectangle with the texture was horizontally deformed in accordance with the participant’s hand position.
In an incompressible material, when an elastic material isstretched horizontally, it is naturally compressed vertically,
wherein the magnitude of vertical deformation is smaller
than that of horizontal deformation. The ratio of the verticalto horizontal deformations is called the Poisson’s ratio [57].
Based on the physical characteristics of the deformation ofan elastic material, vertical deformation was added to therectangle so that the deformation had Poisson’s ratio of 0.25.
We chose the value (0.25) of the Poisson’s ratio because thevalue fell in the range of the Poisson’s ratio supporting
the natural impression of the Poisson effect in human observers [54]. Moreover, we used the two-dimensional rather
than the three-dimensional stimuli because the previous
studies [54], [55] have mainly used the two-dimensional
stimuli to assess the perception of object deformation with
the Poisson effect. We experimentally tested the following
two factors. The first factor was a C/D ratio controlled

4in the following four levels (0.25, 0.5, 0.75, and 1). The
second factor was an effective distance of hand movements
defined as the spatial distance between the left terminatorof the arrow and the hand position. The effective distanceof hand movements was controlled in the following threelevels (5.4, 7.2, and 9 cm). When the distance exceeded
9 cm, the deformation ceased. Moreover, the deformation
was also stopped when the camera temporally lost theparticipant’s hand. We chose the maximum value (9 cm)of the effective distance for the following reason. We assumed an ideal situation whereby the participant’s hand
was always separated by 30 cm from the camera with a
60 deg angle of view. In this case, approximately 9 cm was
the maximum hand movement distance the camera could
capture. Considering that the angle of view in the usual
Laptop PC was expected to be more than, or equal to, 60
deg, it was reasonable to use 9 cm as a maximum value ofthe effective distance. 5.4 and 7.2 cm were arbitrarily chosen
as the 60 and 80 % of the maximum value (9 cm). For eachtrial, the maximum deformation magnitude of the rectangle
was determined depending on the C/D ratio and effective
distance of hand movement as shown in Table 1. Figure 2
shows how the magnitude of object deformation changes
with the two factors. As shown in Figure 4, the appearanceof the rectangle drastically changed with the magnitude ofobject deformation applied to it.
TABLE 1
Maximum magnitude of object deformation which varies depending on
C/D ratio and effective distance of hand movements.
C/D ratio
0.25
0.25
0.25
0.50
0.50
0.50
0.75
0.75
0.75
1.00
1.00
1.00

Effective distance ofhand movements (cm)
5.4
7.2
9.0
5.4
7.2
9.0
5.4
7.2
9.0
5.4
7.2
9.0

Maximum magnitude ofobject deformation (pixels)
36
48
60
72
96
120
108
144
180
144
192
240

Fig. 3. A snapshot of a representative experimental setup.

© 2023 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission.See https://www.ieee.org/publications/rights/index.html for more information.

This article has been accepted for publication in IEEE Transactions on Visualization and Computer Graphics. This is the author's version which has not been fully edited andcontent may change prior to final publication. Citation information: DOI 10.1109/TVCG.2023.3254522

JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2015of the camera image. When the participants moved theirhand by more than a critical value (approximately 1/4 of thewidth of the camera image, that is, 9 cm from the horizontal
center of the monitor) from the left terminator of the arrow,the five buttons for the input of rating scores appeared. The
buttons included the following pairs of a digit and a short
description of softness impressions: “1: Very stiff”, “2: Moderately stiff”, “3: Neither stiff nor soft, “4: Moderately soft”,
and “5: Very soft”. The participants reported the softness
judgment of the rectangle by clicking one of the buttons.
Alternatively, they were also allowed to report the softness
judgment by pressing number keys that corresponded tothe digit of the button including the softness description
which represented their softness judgment. The practice
trials consisted of two trials: the first one had a C/D ratioof 1.0 and an effective distance of hand movements of 0.25
and the second one had a C/D ratio of 1.0 and an effective
distance of hand movements of 1. The main trials consistedof twelve trials with the four different C/D ratios (0.25, 0.5,
0.75, and 1.0) and three conditions of the effective distanceof hand movements (0.6, 0.8, and 1). The order of the twelve
trials was randomized within and across the participants.

Original

60 pixels
deformation

120 pixels
deformation

180 pixels
deformation

240 pixels
deformation

Fig. 4. The change in the appearance of object deformation with different
levels of deformation magnitude. The object deforms in both the verticaland horizontal dimensions.

4.2.4

5

Procedure

Based on the previous study [58], before starting an experimental session, the participant was asked to adjust
the size of a rectangle presented on the monitor so thatthe rectangle had an identical size to a credit card (or acard of equal size) which the participant possessed. This
matching was essential to determine the real scale (i.e., incm) which depends on the pixel size of the monitor theparticipant was using for the experiment. After determining
the real scale for the pixel size, we set the length of thearrow at 9 cm. The length setting was important because
we wanted to control the travel distance of the participant’s
hand movement in the main tasks by the length of the arrow.
Then, the participants were asked to move their right handin front of the arrow. After the system successfully detectedthat the participants moved their hand in front of the arrow,the participant moved to the practice and main trials in thisorder. In each trial, the participants were instructed to place
their right hand approximately 30 cm from the camera andmove their hand in front of the arrow while maintaining the
30 cm distance from the camera (see Supplementary Video
1 for experimental scenes). Preliminary examinations withseveral cameras confirmed that moving the hand laterally
for 9 cm at 30 cm from the camera (having approximately
a 60 deg angle of camera view) produced camera images inwhich the hand was moved by roughly 1/4 of the width

4.2.5 Results
Figure 5 shows mean rating scores for the object softness as afunction of the effective distance of hand movement for eachC/D ratio condition. We reviewed the time it took participants to complete the task and found that they completed itwithin a reasonable amount of time (median: 0.858 minutes,
95% confidence intervals: [0.38, 2.12]). Only one participant
took a relatively long time (6.78 minutes) to complete thetask. We checked the data for that participant but foundnothing suspicious. Since there was no reason to exclude
this participant’s data, we used all participants’ data forfurther analysis. First, we checked whether the frame rate ofstimulus presentation influenced the softness rating scores.
Specifically, we calculated Spearman’s correlation between
the rating scores of object softness and frame rates (thatis, sampling rates). We found that there was no significant
correlation between them (rs = 0.037, p = 0.236). Second, weanalyzed how the factors we tested influenced the softness
rating scores. The rating scores are not parametric data.
Hence, we conducted the aligned rank transform [59] of
the rating scores and then a two-way repeated measures ofanalysis of variance (ANOVA) with the C/D ratio and theeffective distance of hand movements as within-participant
factors. The results of the ANOVA are shown in Table 2. The
main effect of the C/D ratio was significant. The main effectof the effective distance was also significant. Interaction of
the two factors was also significant.
TABLE 2ANOVA table for the rating data in Experiment 1.
Factors
C/D ratio
Effective distanceInteraction

df(df.res)3(249)
2(166)
6(498)

F value
147.729
20.3603.952

Pr(>F)<.0001<.0001=.0007

ηp2
0.640
0.197
0.045

As shown in Table 3, multiple comparison tests (with
ART-C proposed by a previous study [60]) showed that eachlevel of the C/D ratio was significantly different from all© 2023 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission.See https://www.ieee.org/publications/rights/index.html for more information.

This article has been accepted for publication in IEEE Transactions on Visualization and Computer Graphics. This is the author's version which has not been fully edited andcontent may change prior to final publication. Citation information: DOI 10.1109/TVCG.2023.3254522

JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2015

Softness rating scores

5.0

0.25

C/D ratio
0.50
0.75

1.00

6

TABLE 4
Results of multiple comparison tests for the effective distance of hand
movement in Experiment 1.
contrast
5.4 - 7.2
5.4 - 9.0
7.2 - 9.0

4.0

estimate-69.8-111.3
-41.4

df
166
166
166

t.ratio-3.962-6.313-2.351

p.value=0.0003<.0001=0.0597

TABLE 5
Simple effect of effective distance in Experiment 1.

3.0

C/D ratio
0.25
0.50
0.75
1.00

2.0

1.0
10.0
6.0
7.0
8.0
9.0
Effective distance of hand movement (cm)Fig. 5. Ratings scores for object softness in Experiment 1 (N = 84). Error
stripes denote 95 % confidence intervals.
TABLE 3Results of multiple comparison tests for function type in Experiment 1.
contrast
0.25 - 0.5
0.25 - 0.75
0.25 - 1
0.5 - 0.75
0.5 - 10.75 - 1

estimate-184.4-335.6-422.7-151.2-238.3
-87.1

df
249
249
249
249
249
249

t.ratio-8.548-15.556-19.594-7.007-11.046-4.038p.value<.0001<.0001<.0001<.0001<.0001=0.0004other levels of the C/D ratio. The results support the resultsin the previous study [12] showing that the C/D ratio was a
critical parameter in determining the stiffness of the visualobject under user’s touchless control.
Table 4 shows the results of the multiple comparison
tests for the effective distance of hand movements. The
rating scores were significantly different between 5.4 and7.2 cm, and 5.4 and 9.0 cm, of the effective distance. The
difference of the rating scores between 7.2 and 9.0 cm of theeffective distance was marginally significant.
We also assessed the simple effect of the significant
interaction. The simple effect of the C/D ratio (Table 7) was
significant for all effective distance conditions. The multiple
comparison tests of the simple effect (Table 8) showed thatthe rating score in a certain C/D condition was significantly
different from other conditions except a pair of the 0.75 and
1.0 C/D ratios for the 7.2 cm effective distance condition.
The simple effect of the effective distance (Table 5) was
also significant when the C/D ratio was more than 0.25.
The multiple comparison tests of the simple effect (Table 6)
showed that when the C/D ratio was 0.5 or more, therating scores in the 5.4 cm effective distance condition weresignificantly lower than those in the 9.0 cm effective distancecondition. Moreover, when the C/D ratio was 0.75 or more,
the rating scores in the 5.4 cm effective distance condition
were significantly lower than those in the 7.2 and 9.0 cm
effective distance condition.

df(df.res)2(166)
2(166)
2(166)
2(166)

F value
0.3531
3.148
9.493
11.876

Pr(>F)=0.7030.046
0.0001<0.001

ηp2
0.004
0.037
0.103
0.125

4.2.6 Discussion
The effective distance of the participant’s hand affected
the softness perception of an object under the participant’s
touchless control. In other words, an earlier cessation of
object deformation gave the impression of a less soft object.
This finding suggests the necessity of taking the cessation of
object deformation into account in giving users the intended
softness impression of the object under the user’s touchless
control.

5 E XPERIMENT 2: T HE R OLE OF S ATURATIONS PEED OF O BJECT D EFORMATION
5.1

PurposeThe purpose of the experiment was to confirm whether
the saturation speed of object deformation during touchless
inputs was the source of the decrease in the softness ratingscores. In Experiment 1, when the effective distance wasapplied to the hand movement, the deformation ceased; thatis, the magnitude of object deformation was discontinuously
saturated at the middle of the hand trajectory (see Figure 2).
Moreover, depending on the level of the effective distance ofhand movements, the saturation speed changed. A smaller
effective distance caused earlier saturation of object deformation. Hence, there was a possibility that the source of theeffect of the effective distance on the softness rating scores
was the saturation of object deformation during touchless
inputs. In Experiment 2, we systematically manipulated thetemporal pattern of object deformation so that the pattern
had a saturation function in several degrees (see Figure 6)
while the overall magnitude of object deformation was kept
constant. Moreover, we also explored how the growth (notthe saturation) of object deformation could infleunce thesoftness perception. We again instructed the participants torate the object’s softness.
5.1.1 Participants
82 (41 female and 41 male) people, who had not participatedin the previous experiment, participated. Their mean age
was 39.63 (SD: 11.40). The participants’ ages ranged from 20
to 59, and the numbers of participants in the 20’s, 30’s, 40’s,
and 50’s age generations were 20, 20, 20, and 22, respectively
All of the participants were unaware of the specific purposeof the experiment.

© 2023 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission.See https://www.ieee.org/publications/rights/index.html for more information.

This article has been accepted for publication in IEEE Transactions on Visualization and Computer Graphics. This is the author's version which has not been fully edited andcontent may change prior to final publication. Citation information: DOI 10.1109/TVCG.2023.3254522

JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2015

7

TABLE 6
Multiple comparisons for the simple effect of effective distance inExperiment 1.
contrast
0.5 C/D ratio
5.4cm - 7.2cm
5.4cm - 9.0cm
7.2cm - 9.0cm
0.75 C/D ratio
5.4cm - 7.2cm
5.4cm - 9.0cm
7.2cm - 9.0cm
1.00 C/D ratio
5.4cm - 7.2cm
5.4cm - 9.0cm
7.2cm - 9.0cm

estimate

df

t.ratiop.value-9.36-22.12-12.76

166
166
166

-1.058-2.500-1.441=0.875=0.040=0.454

-23.1
-37.5
-14.4

166
166
166

-2.662-4.319-1.657=0.026=0.001=0.299

-26
-38
-12

166
166
166

-3.259-4.768-1.508=0.004<0.0001=0.400TABLE 8Multiple comparisons for the simple effect of C/D ratio in Experiment 1.
contrast
5.4cm distance
0.25 - 0.50
0.25 - 0.75
0.25 - 1.00
0.50 - 0.75
0.50 - 1.00
0.75 - 1.00
7.2cm distance
0.25 - 0.50
0.25 - 0.75
0.25 - 1.00
0.50 - 0.75
0.50 - 1.00
0.75 - 1.00
9.0m distance
0.25 - 0.50
0.25 - 0.75
0.25 - 1.00
0.50 - 0.75
0.50 - 1.00
0.75 - 1.00

TABLE 7
Simple effect of the C/D ratio in Experiment 1.
Effective distance
5.4cm
7.2cm
9.0cm

df(df.res)3(249)
3(249)
3(249)

F value
61.68774.16999.486

Pr(>F)<0.0001<0.0001<0.0001

ηp2
0.426
0.472
0.545

estimate

df

t.ratiop.value

-57.0
-99.2
-130.8
-42.2
-73.8
-31.5

249
249
249
249
249
249

-5.593-9.741-12.835-4.147-7.242-3.095<0.0001<.0001<.0001=0.0003<.0001=0.0132-58.6-115.0-141.5
-56.3
-82.8
-26.5

249
249
249
249
249
249

-5.683-11.140-13.706-5.457-8.023-2.566<0.0001<0.0001<0.0001<0.0001<0.0001=0.0652-72.1-127.4-157.9
-55.3
-85.8
-30.5

249
249
249
249
249
249

-7.335-12.958-16.059-5.623-8.723-3.101<0.0001<0.0001<0.0001<0.0001<0.0001=0.0129

5.1.2

Apparatus

The apparatus for this experiment was identical to that usedin the previous experiment. The mean sampling rate of thehand tracking in this experiment was 29.946 Hz (SD: 13.169
Hz).
5.1.3

Stimuli

The stimuli were identical to those used in Experiment 1
except for the following points. The effective distance ofhand movement was kept constant at 9.0 cm. The C/Dratio was controlled in the following three levels: 0.25,
0.5, and 1.0. At each C/D ratio level, we manipulated thetype of functions in the following five levels: Type 1: Early
saturation, Type 2: Late saturation, Type 3: Linear, Type 4:
Early growth, and Type 5: Late growth (Figure 6).
To generate stimuli having the saturation/growth ofobject deformation, we used an exponential function D
defined by using the following formula,

Deformation magnitude (pixels)

250

200

150

100
Type 1
Type 2Type 3
Type 4
Type 5

50

0
0

2
4
6
8
Distance of hand movements (cm)10

Fig. 6. Saturation/growth of object deformation we employed for eachcondition of the five function types in Experiment 2, with the C/D ratio of
1.0 (thus, the maximum magnitude of object deformation of 240 pixels).

D = exp(−s × h).
For the functions of Type 1 and Type 5, s was set at 0.1. For
the functions of Type 2 and Type 4, s was set at 0.05. h was
the distance of hand movements.
Then, we normalized the D into Dnorm so that the valueof the Dnorm distributed in the range between 0 and 1 in thefollowing way,

Dnorm =

D − Dmin,Dmax − Dminwherein Dmin and Dmax are the minimum and maximum values in the given function D.
The deformation magnitudes Df inal in Type 1 and Type
2 functions were defined by the following formula,

Df inal = G × (1 − Dnorm ),and the deformation magnitudes Df inal in Type 4 and Type
5 functions were defined by the following formula,

Df inal = G × Dnorm ,wherein G denotes the maximum deformation magnitudein pixels. When the C/D ratio was 0.25, 0.5, and 1.0, G was60, 120, and 240 pixels, respectively.
5.1.4 Procedure
The procedure was identical to the one used in Experiment
1 except for the following points. The main trials consistedof fifteen trials with the three conditions of C/D ratios(0.25, 0.5, and 1.0) and five conditions of nonlinear functions(Types 1-5). The order of the fifteen trials was randomized
and within and across the participants.

© 2023 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission.See https://www.ieee.org/publications/rights/index.html for more information.

This article has been accepted for publication in IEEE Transactions on Visualization and Computer Graphics. This is the author's version which has not been fully edited andcontent may change prior to final publication. Citation information: DOI 10.1109/TVCG.2023.3254522

JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 20158

5.1.5 Prediction of Results in Terms of the saturation speedof Object Deformation
If the results of Experiment 1 were derived from the variation in the saturation speed of object deformation due to theeffective distance of hand movements, the softness ratingscores should be lower in the Type 1, Type 2, and Type 3
conditions in that order because the object deformation was
saturated earlier in that order. In contrast, since the temporal
non-linearity of object deformation in the Type 4 and Type
5 conditions was not similar to the one in Experiment 1’s
stimuli, it was not appropriate to make predictions for the
results in the Type 4 and Type 5 conditions on the basisof the results of Experiment 1. Nevertheless, we tested thetwo conditions because we wanted to explore the role of thegrowth of object deformation in determining the object’s
softness. If the growth speed of object deformation was
substantial in modulating the perception of object softness,the rating scores would be lower in the Type 5, Type 4 andType 3 conditions in that order because the growth of object
deformation was attenuated in that order.
5.1.6

Results

0.25

Softness rating scores

5.0

C/D ratio
0.50

1.00

4.0

TABLE 9ANOVA table for the rating data in Experiment 2.

3.0

Factors
C/D ratio
Function typeInteraction

2.0

1.0

Figure 7 shows the rating scores for the object softness
for each of the three C/D ratios and five function types. As
in Experiment 1, we transformed the rating scores by using
the ART and conducted a two-way repeated ANOVA with
the C/D ratio and the function types as within-participant
factors.
Table 9 shows the ANOVA results for the rating data
in Experiment 2. Both the main effects of the C/D ratio
and the function type were significant. Interaction between
the two factors was also significant. Multiple comparison
tests (Table 10) for the main effect of the C/D ratio showedthat the rating scores for each C/D ratio condition weresignificantly different from those in all other C/D ratio
conditions. Multiple comparison tests (Table 11) for the main
effect of the function type showed that the rating scores in
the Type 5 condition were significantly different from thosein the Type 1, Type 2, and Type 4 conditions.
We also checked the simple effects of the significant
interaction and the outcome of their multiple comparison
tests. The simple effect of the function type (Table 12) was
significant when the C/D ratios were 0.25 and 1. Multiple
comparisons (Table 13) showed that when the C/D ratio was
0.25, the rating scores were significantly different between
Type 2 and Type 5, and between Type 4 and Type 5. Also,
when the C/D ratio was 1, the rating scores were significantly different between Type 1 and Type 4. The simple
effect of the C/D ratios (Table 14) was significant for all
function type conditions. Multiple comparisons (Table 15)
showed that the rating score in each C/D ratio condition
was significantly different from the one in all other C/D
ratio conditions.

Type 1

Type 2Type 3

Type 4

Type 5

Fig. 7. Softness rating scores for each C/D ratio and function typecondition (N = 82). Error bars denote 95% confidence intervals.

As in Experiment 1, we checked whether the frame
rate of stimulus presentation influenced the softness ratingscores. There was no significant correlation between them(rs = 0.013, p = 0.578).
We again reviewed the time it took participants to complete the task and found that they completed it withina reasonable amount of time (median: 3.22 minutes, 95%
confidence intervals: [1.16, 12.34]). Only one participant took
44 minutes to complete the task. For that participant, only
the first trial took a long time, so it was assumed that she/he
probably performed the task a while after the first screen
was opened. We checked the data for that participant but
found nothing suspicious. Since there was no reason to
exclude this participant’s data, we used all participants’ data
for further analysis.

df(df.res)2(162)
4(324)
8(648)

F value
158.972
4.524
2.175

Pr(>F)<.0001
=.002
=.028

ηp2
0.662
0.052
0.026

TABLE 10
Results of multiple comparison tests for C/D ratio in Experiment 2’s
rating data.
contrast
0.25 - 0.5
0.25 - 1.00
0.50 - 1.00

estimate
-219
-506
-287

df
162
162
162

t.ratio-7.702-17.778-10.076p.value<.0001<.0001<.00015.1.7 Discussion
Although the rating scores significantly varied dependingon the function type defining the saturation/growth speedof object deformation during touchless controls, none ofthe outcomes could account for the results of Experiment
1. Specifically, the rating scores in this experiment did notdecrease in the order of the Type 1, Type 2, and Type 3
conditions, which were not consistent with the results ofExperiment 1. The results indicate that the saturation speedof object deformation could not explain the effect of theeffective distance of hand movements on the object softness
judgments in Experiment 1.

© 2023 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission.See https://www.ieee.org/publications/rights/index.html for more information.

This article has been accepted for publication in IEEE Transactions on Visualization and Computer Graphics. This is the author's version which has not been fully edited andcontent may change prior to final publication. Citation information: DOI 10.1109/TVCG.2023.3254522

JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2015

TABLE 11
Results of multiple comparison tests for C/D ratio in Experiment 2’s
rating data.
contrast
1-2
1-3
1-4
1-5
2-3
2-4
2-5
3-4
3-5
4-5

estimate
12.35
26.72
24.28
93.34
14.37
11.93
80.99
-2.44
66.62
69.06

df
324
324
324
324
324
324
324
324
324
324

t.ratio
0.512
1.108
1.007
3.872
0.596
0.495
3.360
-0.1012.764
2.865

p.value
1.00001.00001.00000.00131.00001.00000.00871.00000.06040.0445TABLE 12
Simple effect of function types in Experiment 2.
C/D ratio
0.25
0.50
1.00

df(df.res)4(324)
4(324)
4(324)

F value
3.340
1.486
2.579

Pr(>F)=0.0106=0.2062=0.0374

ηp2
0.040
0.018
0.031

We observed that the softness rating scores in the Type
5 condition were significantly smaller than the scores in theType 1 and Type 2 conditions. When the participant’s hand
movement distance exceeded 9 cm and the deformationceased, the discontinuity of object deformation was caused.
In the Type 5 condition with late growth deformation patterns, the discontinuity of object deformation was likely
salient in comparison with the Type 1 and Type 2 conditions
with saturation deformation patterns. Such discontinuity ofobject deformation might contribute to the determinationof perceived softness during touchless inputs. The salient
discontinuity of object deformation might serve as a cue tonon-soft materials.

6 E XPERIMENT 3: R ELATIONSHIP B ETWEENS OFTNESS AND OTHER P ERCEPTUAL I MPRESSIONS
6.1

PurposeSo far, we have shown that the effective distance of hand
movements was a strong determinant of softness matchingscores for an object controlled by the participants’ touchless
inputs. On the other hand, it was still unclear how the sudden cease (that is, the discontinuity) of object deformation
affected the softness rating scores. To address the unclear
issue, it would be beneficial to focus on not only softness but
also other perceptual impressions. As described above, thechange in object speed contributes to the change in the impression of friction, gravity, or viscosity [11], resistance [27],and collision [28], [29], [30], [31]. Moreover, in our preliminary observation, the participants informally reported “thesudden change of softness and heaviness” and “something
happened to the object”. Thus, there was a possibility thatthe discontinuity of object deformation caused the change
in perceptual impressions other than softness. Moreover, asa previous study [61] on haptic devices has shown, othertypes of haptic impressions, such as warmness, roughness,size, and weight, could be influenced by the discontinuityof object deformation. Checking the relationship between
the softness impression and other types of impression may

9

TABLE 13Multiple comparisons for the simple effect of function types inExperiment 2.
contrast
0.25 C/D ratio
1-2
1-3
1-4
1-5
2-3
2-4
2-5
3-4
3-5
4-5
1.00 C/D ratio
1-2
1-3
1-4
1-5
2-3
2-4
2-5
3-4
3-5
4-5

estimate

df

t.ratiop.value

-3.74
7.48
-7.01
30.24
11.22
-3.27
33.98
-14.49
22.76
37.25

324
324
324
324
324
324
324
324
324
324

-0.3240.649-0.6082.623
0.973-0.2832.947-1.2561.974
3.230=1.0000=1.0000=1.0000=0.0913=1.0000=1.0000=0.0344=1.0000=0.4923=0.0136

14.9
13.8
39.7
25.6
-1.1
24.8
10.7
25.9
11.8
-14.1

324
324
324
324
324
324
324
324
324
324

1.142
1.058
3.045
1.963
-0.084
1.903
0.821
1.987
0.905
-1.082=1.0000=1.0000=0.0252=0.5051=1.0000=0.5792=1.0000=0.4774=1.0000=1.0000

TABLE 14
Simple effect of C/D ratios in Experiment 2.
Func types
Type 1
Type 2Type 3
Type 4
Type 5df(df.res)2(162)
2(162)
2(162)
2(162)
2(162)

F value
95.69367.40974.34246.41386.600Pr(>F)<0.0001<0.0001<0.0001<0.0001<0.0001

ηp2
0.542
0.454
0.478
0.364
0.517

also give us some insights into the underlying psychologicalmechanism for reducing perceptual softness due to thediscontinuity of object deformation. In this experiment, we
asked the participants to rate eight items regarding various types of perceptual impressions (please refer to the“Procedure” section for detail). Based on the results, we
tried to interpret how the change in softness perception
occurred when the object deformation ceased during touchless inputs. Specifically, we analyzed Spearman’s correlation
coefficients among the rating scores for various perceptual
impressions. Moreover, we conducted a factor analysis to
explore latent variables underlying the determination ofperceptual impressions for object deformation during touchless inputs.
Furthermore, instead of the two-dimensional texture
noise used in the previous experiments, we used the threedimensional object to check whether we could observe thesimilar effect of the C/D ratio and the effective distanceof hand movements on the softness rating scores withthe three-dimensional object. In the previous experiment,
the object to be deformed was a two-dimensional textured
rectangle. In daily life, we often pull and push threedimensional elastic objects. In this respect, it would bemeaningful to check whether the conclusion of the previous
experiments could be extended to the case wherein theparticipants manipulated the three-dimensional object withtouchless inputs.

© 2023 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission.See https://www.ieee.org/publications/rights/index.html for more information.

This article has been accepted for publication in IEEE Transactions on Visualization and Computer Graphics. This is the author's version which has not been fully edited andcontent may change prior to final publication. Citation information: DOI 10.1109/TVCG.2023.3254522

JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2015

TABLE 15
Multiple comparisons for the simple effect of C/D ratio in Experiment 2.
contrast
Type 10.25 - 0.50
0.25 - 1.00
0.50 - 1.00
Type 20.25 - 0.50
0.25 - 1.00
0.50 - 1.00
Type 30.25 - 0.50
0.25 - 1.00
0.50 - 1.00
Type 40.25 - 0.50
0.25 - 1.00
0.50 - 1.00
Type 50.25 - 0.50
0.25 - 1.00
0.50 - 1.00

6.2

estimate

df

t.ratiop.value-45.8-106.3
-60.5

162
162
162

-5.943-13.790-7.847<.0001<.0001<.0001

-37.5
-94.1
-56.6

162
162
162

-4.597-11.532-6.935<.0001<.0001<.0001-42.8-101.5
-58.7

8.36
8.36
8.36

-5.122-12.144-7.022<.0001<.0001<.0001

-42.5
-84.0
-41.5

8.72
8.72
8.72

-4.870-9.634-4.765<.0001<.0001<.0001-41.3-103.6
-62.3

162
162
162

-5.213-13.072-7.859<.0001<.0001<.0001

Participants

189 (93 female and 96 male) people, who had not participated in the previous experiments, participated in thisexperiment. Their mean age was 40.80 (SD: 10.83). The
participants’ ages ranged from 20 to 59, and the numbers ofparticipants in the 20’s, 30’s, 40’s, and 50’s age generations
were 36, 51, 52, and 50, respectively. All of the participants
were unaware of the specific purpose of the experiment.
6.2.1

at the 5.6 cm right of the horizontal center of the display.
The background for the cylinder and the line was black.
The initial width and height of the cylinder were 1.4
and 1.6 cm, respectively. As the participant moved theirright hand toward the right in front of the line, the cylinder
was horizontally elongated toward the right. The maximum
width of the cylinder was 2.0 cm, which occurred in thecondition with a C/D ratio of 1.0 and an effective distance of
9 cm. The amount of elongation was fairly smaller than the
one employed in Experiments 1 and 2. This was because, inthe three-dimensional scene, the larger elongation distorted
the apparent shape due to the camera’s field of view. Specifically, the cylinder distorted in a more periphery visual field.
Although we could use a lower value of the field of viewto correct the distortion, in that case, the appearance of thecylinder was akin to the two-dimensional noises used inExperiments 1 and 2. Hence, we chose this small magnitudeof elongation to present the three-dimensional appearanceof the cylinder reliably. In the conditions with the C/D ratioof 0.25, 0.50, 0.75, and 1.00, the maximum magnitude ofelongation was 1.55, 1.70, 1.85, and 2.0 cm, respectively. In
the conditions with the effective distance of hand distanceof 5.4, 7.2, and 9.0 cm, the elongation ceased when theelongation reached ×0.6, ×0.8, and ×1.0 the maximum
magnitude of elongation which was determined by the C/Dratio.

Before deformation

After deformation

Apparatus

The apparatus for this experiment was identical to that usedin the previous experiments. The mean sampling rate of
the hand tracking in this experiment was 18.859 Hz (SD:12.352 Hz). The lower frame rate in this experiment might
be that rendering the three-dimensional objects in visual
stimuli required more computational time than the twodimensional noises used in Experiments 1 and 2.
6.2.2

10

Fig. 8. The appearance of stimuli as used in Experiment 3. The left
and right panels show the appearance of stimuli before and after object
deformation due to the participant’s touchless inputs.

Stimuli

The stimuli were identical to those employed in the previous experiments except for the following. While the previous experiments used a two-dimensional object with texture noises as a stimulus, this experiment used a threedimensional object (a cylinder) as a stimulus. The cylinder
was rendered by using Three.js (https://threejs.org/). The
surface of the cylinder was textured with the white noise
image as used in Experiments 1 and 2. The cylinder was
displayed on the left side of the display. The left side of thecylinder was located 3cm to the left of the horizontal centerof the display. The cylinder was captured by a camera (in thevirtual scene) with a frustum vertical field of view of 45 deg.
The camera always looked at the center of the coordinate. To
indicate the spatial range for participants to move their righthand, we presented a thick line with 0.3 cm height and 9 cm
width, instead of the arrow as used in Experiments 1 and 2.
The reason for this change was that we felt that the arrow
was not always necessary to indicate the spatial range ofhand movements to the participants. The line was centered

6.2.3

Procedure

In each trial, the participant was asked to move their righthand in front of the line presented on the right side of thedisplay. Despite the effective distance of hand movements,
the list of items to which the participant responded appeared after the participants moved their hand by 9 cm. The
list of the items consisted of impression words for “Soft-Not
soft”, “Cold-Warm”, “Not viscous-Viscous”, “Bumpy-Not
bumpy”, “Small-Large”, “Heavy-Light”, “No friction–With
friction”, and “Collision–No collision.” We chose the itemson the basis of the previous studies [?], [11], [27], [29], [30],[31], [61]. Each item had seven numbers (1-7) with radio
buttons. Participants were instructed to click on the radio
button to the left of the number closest to their impression
written on either side of each question item. We used a
7-point scale, instead of a 5-point scale as in Experiments
1 and 2 because we assumed that a finer scale than theprevious experiments was required for the evaluation of
stimuli of this experiment, which had a narrower stimulus© 2023 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission.See https://www.ieee.org/publications/rights/index.html for more information.

This article has been accepted for publication in IEEE Transactions on Visualization and Computer Graphics. This is the author's version which has not been fully edited andcontent may change prior to final publication. Citation information: DOI 10.1109/TVCG.2023.3254522

JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2015

deformation range than the stimuli of the previous experiments. The participant could proceed to the next trial only
after they responded to all eight items. As in Experiment
1, each participant received 12 trials of 4 conditions of the
C/D ratio and 3 conditions of the effective distance of hand
movements. The trials were conducted in a random orderthat varied among participants.
6.2.4

Results

We again reviewed the time it took participants to complete
the task and found that they completed it within a reasonable amount of time (median: 3.22 minutes). Only one
participant took 44 minutes to complete the task. For thatparticipant, only the first trial took a long time, so it was
assumed that she/he probably performed the task a while
after the first screen was opened. We checked the data forthat participant but found nothing suspicious. Since there
was no reason to exclude this participant’s data, we used allparticipants’ data for further analysis.
For the item “Soft-Not soft”, to make consistency amongexperiments, we subtracted the rating scores from 7 andtreated it as the score of “Not soft-Soft”. The median time theparticipant took to perform the task on each trial was 30.449
seconds (with 95% confidence intervals of [14.203, 119.736]).
As in Experiments 1 and 2, we checked whether the frame
rate of stimulus presentation influenced the softness ratingscores. Different from the previous experiments, there was asignificant correlation between them (rs = 0.100, p < 0.001).
The results indicate that the higher frame rate of stimulus
presentation caused the higher softness rating scores. In
comparison with Experiments 1 and 2, the mean sampling
rate (that is, frame rate) was lower in Experiment 3. As
described above, the lower frame rate in this experimentmight be that rendering the three-dimensional objects invisual stimuli required more computational time than thetwo-dimensional objects used in Experiments 1 and 2. The
lower frame rate in this experiment might cause lower
softness rating scores, and consequently, a significant correlation coefficient might be obtained. On the other hand,the correlation coefficient, while significant, was not veryhigh. Hence, we concluded that the influence of the frame
rate on the softness rating scores was not so strong.
Figure 9 shows the rating scores for each item. Eachgraph shows the scores for each condition of the four levelsof C/D ratios and three levels of the effective distanceof hand movements. As in Experiment 1, we transformed
the rating scores using the ART and conducted a two-way
repeated ANOVA with the C/D ratio and effective distanceof hand movements as within-participant factors.
Table 16 shows the main effect and interaction of thetwo factors. The main effect of the C/D ratio was significant
for the items “Not soft-Soft”, “Cold-Warm”, “Small-Large”,“Heavy-Light”, and “No friction-With friction”. The resultsof the multiple comparison tests for each significant main
effect are shown in Table 17. The main effect of the effective
distance of hand movements was also significant for theitems “Not soft-Soft”, “Cold-Warm”, “Bumpy-Not bumpy”,
and “Heavy-Light”. The results of the multiple comparison
tests for each significant main effect are shown in Table 18.
Interaction between the two factors was significant for

11“Collision-Not collision”. The simple effect of the significant interaction reached significance only for the effective
distance of hand movement when the C/D ratio was 1.0(F2,374 =3.425, p=.0335, ηp2 =0.018). The multiple comparison
test for the significant simple effect showed that the ratingscores for “Collision-No collision” were significantly higher
when the effective distance of hand movements was 5.4 thanwhen it was 9.0 (t374 = 2.466, p = .0423).
TABLE 16
ANOVA table for the rating scores in Experiment 3.
Factors
Not soft - SoftC/D ratio
Effective distanceInteraction
Cold - Warm
C/D ratio
Effective distanceInteraction
Not viscous - ViscousC/D ratio
Effective distanceInteraction
Bumpy - Not bumpy
C/D ratio
Effective distanceInteraction
Small - Large
C/D ratio
Effective distanceInteraction
Heavy - LightC/D ratio
Effective distanceInteraction
No friction - With friction
C/D ratio
Effective distanceInteraction
Collision - No collision
C/D ratio
Effective distanceInteraction

df(df.res)

F value

Pr(>F)

ηp2

3(561)
2(374)
6(1122)81.67511.3591.827<.0001<.0001=.0967

0.304
0.057
0.009

3(561)
2(374)
6(1122)14.6853.523
2.034<.0001=.0305=.0584

0.072
0.018
0.010

3(561)
2(374)
6(1122)0.171
0.02450.5876=.9155=.9757=.74040.001
0.0001
0.003

3(561)
2(374)
6(1122)

1.108
11.8530.8066=.3452<.0001=.5645

0.006
0.060
0.004

3(561)
2(374)
6(1122)3.480
0.78191.718=.0158=.4582
=.113

0.018
0.004
0.009

3(561)
2(374)
6(1122)45.8524.67320.6198<.0001=.009=.7145

0.197
0.024
0.003

3(561)
2(374)
6(1122)24.4372.013
0.527<.0001=.1350=.7889

0.116
0.010
0.003

3(561)
2(374)
6(1122)

1.817
1.63983.143=.1429=.1954=.0046

0.009
0.008
0.017

To explore the relationship among perceptual impressions that we examined, we calculated Spearman’s correlation coefficients among the rating scores of the items. The
obtained correlation coefficients and p-values are shown in
Figure 10. Softness impression was moderately correlatedwith the impression of “Heavy-Light”. That is, a softer
object was judged to be lighter. Interestingly, the softer
impression was also moderately correlated with the scoresof “Cold-Warm” while the scores of “Cold-Warm” wascorrelated more strongly with the scores of “Small-Large”.
The scores of “Not viscous-Viscous” were most correlatedwith the scores of “Bumpy-Not bumpy” while the scores of“Bumpy-Not bumpy” were most correlated with the scoresof “Collision-No collision”. The scores of “Heavy-Light”
were most correlated with the scores of “Not soft-Soft”. The
scores of “No friction-Friction” were most correlated withthe scores of “Collision-No collision”.
The interpretation of the correlation coefficients was not
so easy because the rating scores of a single item werecorrelated with the rating scores of multiple items. To facilitate the interpretation, we conducted a factor analysis
and checked what sort of latent factors was involved with© 2023 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission.See https://www.ieee.org/publications/rights/index.html for more information.

This article has been accepted for publication in IEEE Transactions on Visualization and Computer Graphics. This is the author's version which has not been fully edited andcontent may change prior to final publication. Citation information: DOI 10.1109/TVCG.2023.3254522

JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2015

Not soft - Soft

7

Rating scores7

Not viscous - Viscous

6

6

5

5

5

4

4

4

4

3

3

3

3

2

2

2

2

5

5.4

7

7.2
Small - Large

9.0

1

5.4

7

7.2
Heavy - Light

9.0

1
7

5.4
7.2
9.0
No friction - With friction

1

6

6

6

5

5

5

5

4

4

4

4

3

3

3

3

2

2

2

2

5.4

7.2

9.0

1

5.4

7.2

9.0

1

5.4

7.2

9.0

5.4

1

7.2

9.0

Collision - No collision

7

6

1

Bumpy - Not bumpy

7

6

1

Rating scores

Cold -Warm

7

C/D Ratio
0.25
0.50
0.75
1.00

6

12

5.4

7.2

9.0

Effective distance of hand movement (cm)Fig. 9. Rating scores for each condition of C/D ratio and effective distance of hand movements (N = 189) in Experiment 3. Error bars denote 95%
confidence intervals.

**** p < .0001******* p < .001
Not viscous - Viscous****** p < .01* p < .05****

Bumpy - Not bumpy*****
Small - Large

0.0

****

****

****

****

****

****

****

****

****

****

****

****

****

****

****

Heavy - Light

****

****
No friction - With friction

Heavy - Light****
Small - Large****

Bumpy - Not bumpy

Collision - No collision

Not viscous - Viscous

No friction - With friction

Cold -Warm

6.2.5 Discussion
The results showed that the effective distance of hand
movements still significantly influenced the softness ratingscores even when the participants were asked to judge otherperceptual impressions than softness. The results indicatethat the effect of the effective distance of hand movementson the softness rating scores is not the product of bias due toasking the participants to judge softness impressions only.
On the other hand, not only softness but also otherperceptual impressions such as “Cold-Warm”, “BumpyNot bumpy”, and “Heavy-Light” were also influenced
by the effective distance of hand movements. According
to the results of our factor analysis, “Cold-Warm” and“Heavy-Light” stemmed from the first factor, “No soft-Soft”
stemmed from the second factor, and “Bumpy-Not bumpy”

0.5
Cold -Warm

Not soft - Soft

perceptual impressions for the object deformation during
touchless control. To determine the factor count for thefactor analysis, we first calculated eigenvalues. As shown in
Figure 11, the factor count with the eigenvalues greater than
1 was three. Thus, we conducted the factor analysis witha factor count of three. Table 19 shows the factor score ofeach item after a varimax rotation. The first factor contained
the items of “Cold - Warm”, “Small - Large”, and “Heavy- Light”. The second factor contained “Not soft - Soft”. The
third factor contained “Not viscous - Viscous”, “No friction– With friction”, and “Collision – No collision”.

-0.5

Fig. 10. Correlation matrix among the rating scores in Experiment 3.
Annotations are Spearman’s correlation coefficients and p-values.

stemmed from the third factor. Thus, the discontinuity ofobject deformation, which was caused by the presence of© 2023 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission.See https://www.ieee.org/publications/rights/index.html for more information.

This article has been accepted for publication in IEEE Transactions on Visualization and Computer Graphics. This is the author's version which has not been fully edited andcontent may change prior to final publication. Citation information: DOI 10.1109/TVCG.2023.3254522

JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2015

13

2.5

TABLE 17
Results of multiple comparison tests of the significant main effect of the
C/D ratio in Experiment 3.
estimate

df

t.ratiop.value-211.2-414.1-508.8-202.9-297.5
-94.6

561
561
561
561
561
561

-5.969-11.703-14.377-5.734-8.408-2.674<.0001<.0001<.0001<.0001<.0001=.0462-113.5-144.8-158.8-31.24
-45.2
-13.9

561
561
561
561
561
561

-4.271-5.446-5.971-1.175-1.700-0.525=.0001<.0001<.0001=1.000=.5375=1.000

-66.9
-53.2
-83.5
-13.7
-18.5
-32.3

561
561
561
561
561
561

-2.403-1.908-3.07-0.494-0.666-1.161=.0099=.3408=.0134=1.000=1.000=1.000-143.4-264.6-377.7-121.1-234.2-113.0

561
561
561
561
561
561

-4.235-7.812-11.151-3.577-6.916-3.338=.0002<.0001<.0001=.0022<.0001=.0052

88.1
184.5
253.6
96.4
165.5
69.0

561
561
561
561
561
561

2.776
5.816
7.993
3.039
5.216
2.1770=.0034<.0001<.0001=.0015<.0001=.1793

TABLE 18
Results of multiple comparison tests of the significant main effect of theeffective distance of hand movement in Experiment 3.
contrast
Not soft - Soft
5.4 - 7.2
5.4 - 9.0
7.2 - 9.0
Cold - Warm
5.4 - 7.2
5.4 - 9.0
7.2 - 9.0
Bumpy - Not bumpy
5.4 - 7.2
5.4 - 9.0
7.2 - 9.0
Heavy - Light
5.4 - 7.2
5.4 - 9.0
7.2 - 9.0

estimate

df

t.ratiop.value-79.6-117.8
-38.2

374
374
374

-3.155-4.671-1.516=.0051<.0001=.3910-59.0-117.8
-38.2

374
374
374

-2.642-1.5371.105=.0257=.3751=.8090

-33.5
59.9
93.4

374
374
374

-1.7203.083
4.804=.2582=.0065<.0001

-61.8
74.9
-13.0

374
374
374

-2.363-2.861-0.497=.0558=.0133=1.000the effective distance of hand movements during touchless
inputs, can influence various aspects of perceptual impression.
When the C/D ratio was 1.0, the main effect of theeffective distance of hand movements was significant for
the rating scores of “Collision - No collision”. The results
are consistent with the previous studies [28], [29], [30],[31] showing that motion discontinuity served as a cue to

2.0

Eigenvalue

contrast
Not soft - Soft
0.25 - 0.5
0.25 - 0.75
0.25 - 1
0.5 - 0.75
0.5 - 10.75 - 1
Cold - Warm
0.25 - 0.5
0.25 - 0.75
0.25 - 1
0.5 - 0.75
0.5 - 10.75 - 1
Small - Large
0.25 - 0.5
0.25 - 0.75
0.25 - 1
0.5 - 0.75
0.5 - 10.75 - 1
Heavy - Light
0.25 - 0.5
0.25 - 0.75
0.25 - 1
0.5 - 0.75
0.5 - 10.75 - 1
No friction - With friction
0.25 - 0.5
0.25 - 0.75
0.25 - 1
0.5 - 0.75
0.5 - 10.75 - 1

1.5

1.0

0.5

0.0
1

2

3

4

5

6

7

8

Factor countFig. 11. Eigenvalues as a function of factor counts for the rating scores inExperiment 3. Since the factor count with the eigenvalues greater than
1 was 3, we conducted a factor analysis with the factor count of 3.
TABLE 19
Factor scores for each item and cumulative variance in Experiment 3.
Bold-style values indicate maximum factor factors for each item.
Item
Not soft - Soft
Cold - Warm
Not viscous - Viscous
Bumpy - Not bumpy
Small - Large
Heavy - Light
No friction – With friction
Collision – No collision
Cumulative variancefactor1
0.208337
0.841540
0.006472
0.125152
0.518055
0.482910
0.108375
0.030515
0.160195

factor2
0.971814
0.197321-0.270443-0.020790-0.018932
0.294893-0.190682
0.099745
0.309015

factor3
0.089918-0.150379
0.479372
0.493394
0.060709
0.365981-0.497647
0.489601
0.450130

pseudo-haptic collision.
The factor analysis showed that the three latent variables
might underlie the judgment of perceptual impressions forthe objects controlled by touchless inputs. The first variable, which influenced “Cold-Warm”, “Small-Large”, and“Heavy-Light”, seemed to be involved with the inferenceof the object’s material properties. The second variable,
which influenced “Not soft-Soft”, is likely related to theinference of perceptual softness. The third variable, which
influenced “Not viscous-Viscous”, “No friction–With friction”, and “Collision–No collision”, is possibly related to theinference of interaction between the user (or user’s action)and external objects. Though the list of the latent variable ispossibly not a thorough one due to the limited number of
items we tested, our results indicate that perceptual impressions that the participants conceive during touchless inputs
are multi-facet. To accurately understand the psychological© 2023 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission.See https://www.ieee.org/publications/rights/index.html for more information.

This article has been accepted for publication in IEEE Transactions on Visualization and Computer Graphics. This is the author's version which has not been fully edited andcontent may change prior to final publication. Citation information: DOI 10.1109/TVCG.2023.3254522

JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2015

mechanism, it is necessary to use an expanded set of items
for perceptual impressions.
There were pros and cons to using the 3D object inthis experiment. For pros, we could extend the conclusion
with the 2D objects in Experiment 1 to the case with the
3D objects. For cons, it was not possible to check whether
the results of Experiment 1 with the 2-D objects could be
reproduced by different subjects in a new experiment that
also used 2-D objects. Moreover, the difference in stimuli
caused the difference in the softness rating scores between
Experiments 1 and 3. Compared with Experiment 1, theabsolute level of rating scores for perceptual softness was
lower in this experiment. This might be because the magnitude of object deformation was smaller in this experimentthan in Experiment 1. On the other hand, the effect of
the effective distance of hand movements was significant
in both experiments. Therefore, the discontinuity of object
deformation possibly serves as a cue to perceptual softness
relatively independent of the magnitude of object deformation.

7

C ONCLUSION

7.1

Summary of Results

The present study examined how the effective distance ofhand movements impacted the softness rating scores ofan object under the control of the user’s touchless inputs.
Experiment 1 showed that in addition to the C/D ratio,
the effective distance of hand movements was a criticalfactor in determining the softness rating scores. Experiment
2 showed that the saturation speed of object deformation,
which was inherently triggered by adopting the effective
distance of the hand movements, did not account for thevariation of the rating scores in Experiment 1. In Experiment 3, we observed that the effective distance of hand
movements still significantly influenced the softness ratingscores even when the participants were asked to report otherperceptual impressions than softness. We also reported thatthe effective distance of hand movements affected otherperceptual impressions than softness.
7.2
7.2.1

Significance of the Present Study
Scientific SignificanceThe present study essentially replicated the previous studies
showing that in a passive stimulus viewing, the visual
softness of elastic objects increased with the magnitude of
the object deformation [45], [46], [47]. As described above,
manipulation of the effective distance of hand movements,
which significantly affected the softness rating scores,
caused the variation of the deformation magnitudes. Thus,
the present study showed that the softness rating scoresvaried with the magnitude of object deformation whileusers actively controlled the magnitude of deformation via
their touchless inputs.
7.2.2

Technical SignificanceThe present study showed that the softness judgment ofan object under the control of user’s touchless input wasaltered by the cessation of object deformation during the14

control. Specifically, an earlier cessation of object deformation in the course of deformation manipulation via touchless
control produced a lower softness judgment for the object.
Without considering the effect of the effective distance ofhand movements, it doesn’t seem easy to provide the desired level of object softness to users who controlled thedeformation of the object via their touchless control. In
presenting the softness of an object that is deformed byusers’ touchless inputs, the first important thing to keep inmind is the need to adjust the hand position to fall within
the angle of camera view, so that the deformation of theobject does not stop in the course of the input. In addition,
it must be implemented so that any occluder does not blockthe camera view of the hand. Although the effective distanceof hand movements is likely to cause problems such as theone this study demonstrated, the problems can be avoided
by using a hand position sensing device other than a camera.
Issues arising from sensing failures (in our case, issues
arising from the effective distance of hand movements) area general problem which can occur when we adopt othertouchless sensing devices. For example, a magnetic sensor
such as Polhemus tracker [62] has an effective range ofsensing. When the user’s hand exceeds this range, tracking
cannot be performed. Therefore, there is a possibility thatthe issues examined in our study are common to various
sensing methods for touchless input that have an effective
range of sensing. In addition, the issues might be common
to devices that are not directly related to the sensing oftouchless inputs. For example, a touchpad that is embedded
within a laptop PC cannot sense finger movements outside
the area of the touchpad, and thus there is an effective sensing range. A similar sort of the effect of the effective distanceon softness judgment would be observed in the case of thetouchpad, although it may be necessary to consider theeffect of the actual touch feeling on the softness judgment
as well. In this way, in order to mitigate the effect of sensing
failures on the softness judgment, it is probably necessary
to consider the characteristics of users’ perceptions, which
likely vary according to each sensing method.
So far, we have considered the effect of effective distanceas a negative component for the softness presentation in thetouchless input system and have proposed some methodsof avoiding it. However, it may be also, possible for us
to regard it as a positive component for the presentationof softness impressions. Since the present study showed
the significant interaction between the C/D ratio and theeffective distance, it may be possible to control the level ofobject softness in a finer manner with, rather than without,taking the effect of the effective distance into account.
7.3

Limitations and Future Issues

7.3.1 Another Interpretation of the Present Study
We attributed the effect of effective distance on hand movements to the changes in the magnitude of deformation witheffective distance. However, the results can be interpreted
in a different manner from ours. Namely, the results may
be interpreted in terms of the C/D ratio only. In our stimuli, when the hand was outside the effective distance ofhand movements, any hand movement no longer caused
the deformation. The C/D ratio in this situation can be© 2023 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission.See https://www.ieee.org/publications/rights/index.html for more information.

This article has been accepted for publication in IEEE Transactions on Visualization and Computer Graphics. This is the author's version which has not been fully edited andcontent may change prior to final publication. Citation information: DOI 10.1109/TVCG.2023.3254522

JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2015

considered to be zero. If the sensory system can integrate
and/or average the C/D ratio across time (or along a hand
movement trajectory), the perceived C/D ratio is lower inthe condition with the smaller effective distance of hand
movements. However, there is no previous study to support
the proposal for the temporal integration of the C/D ratio
across time. Future studies are necessary to support thissuggestion.
7.3.2

Spatiotemporal Resolution

The experiment of the present study was conducted online, and thus, the hand-tracking device was limited to thecamera which the participants used. The results suggestthat the touchless input is effective even in a situationin which users controlled the objects via touchless inputs
online. We suggest that the spatiotemporal resolution of thecamera was also enough to clarify the basic property ofobject softness judgments by users. However, using devices
with a higher spatiotemporal resolution may enhance users’
immersive experiences of the object’s softness and/or thesense of agency in controlling the object’s deformation via
touchless inputs. From a scientific perspective, a device withhigh temporal resolution may enable us to check how fine
temporal structures of hand movements could influence theusers’ judgment of the object’s softness.
Similarly, it is an important limitation of the present
study that the frame rate of the stimulus presentation was
not controlled. Although no significant correlations were observed between the softness rating scores and frame rate inExperiments 1 and 2, a significant correlation was observedin Experiment 3. The precise control of the frame rate might
be beneficial to obtain robust experimental data and/or user
experiences in touchless input systems.
7.3.3

Lack of Control of Hand Movement Speed

In the experiments in the present study, we did not control the speed of hand movements. The variation of hand
movement speeds led to the variation in the speed of object
deformation. Thus, the speed of hand movements might
be a potential factor in modulating softness rating scoresbecause it is known that the speed of object deformation is a
modulatory factor of perceived softness [47]. Future studies
need to carefully evaluate the role of the speed of hand
movements in the determination of the perceived softnessof objects controlled by touchless inputs.
7.3.4

Insufficient monitoring of Experimental Task

All experiments in this study were conducted online. Therefore, the experimenter could not monitor how participants
performed the tasks. As far as the time taken for participants
to complete a single trial was monitored, it appeared thatparticipants performed the task within a reasonable amountof time. On the other hand, the other aspects of monitoring,such as how many times the hand tracking failed and inwhat light and sound environment the task was performed,
were unclear. These unmonitored factors could affect theevaluation scores. Experimentation in a controlled environment would reduce this possibility.

15

7.3.5 Implementation with 3D Objects and XR
We believe that the results of the present study will be
useful for implementing interfaces in which users interact
with virtual objects in xR. For example, in online shopping
using xR, the results of this study will be useful for understanding/configuring situations where users pull the cloth
they want to buy and check the softness and appearanceof the cloth. Though we used touchless inputs to deform avirtual object, it would be meaningful to examine whether
there is a difference in results between using a VR controller
to deform an object and using touchless input. Although
we used the three-dimensional object in Experiment 3, therelationship between the participant’s hand movements andcorresponding object deformation could be described in thetwo-dimensional space. In this respect, it is intriguing toextend the results of the present study to the manipulation
in a three-dimensional space and verify whether the effect of
the effective distance of hand movements can be observed in
the extended situation. Such extension is promising with thetouchless inputs because they do not use physical devices,which often restrict users’ hand movements.

R EFERENCES[1]

F. Biocca, J. Kim, and Y. Choi, “Visual touch in virtual environments: An exploratory study of presence, multimodal interfaces,
and cross-modal sensory illusions,” Presence: Teleoperators and Virtual Environments, vol. 10, no. 3, p. 247–265, 2001.
[2] A. Lécuyer, “Simulating haptic feedback using vision: A surveyof research and applications of pseudo-haptic feedback,” Presence:
Teleoperators and Virtual Environments, vol. 18, no. 1, p. 39–53, 2009.
[3] Y. Ujitoko and Y. Ban, “Survey of pseudo-haptics: Haptic feedback
design and application proposals,” IEEE Transactions on Haptics,
vol. 14, no. 4, p. 699–711, 2021.
[4] Y. Ban and Y. Ujitoko, “Enhancing the pseudo-haptic effect on
the touch panel using the virtuual string* *research supported by
the ministry of education, science, sports and culture.” 2018 IEEE
Haptics Symposium (HAPTICS), p. 278–283, 2018.
[5] L. Dominjon, A. Lecuyer, J. Burkhardt, P. Richard, and S. Richir,“Influence of control/display ratio on the perception of massof manipulated objects in virtual environments,” Proceedings. VR
2005, pp. 19–25,, 2005.
[6] Y. Hirao, T. M. Takala, and A. Lécuyer, “Comparing motion-based
versus controller-based pseudo-haptic weight sensations in vr,”
vol. 00, p. 305–310, 2020.
[7] J. Kim, S. Kim, and J. Lee, “The effect of multisensory pseudohaptic feedback on perception of virtual weight,” IEEE Access,vol. 10, p. 5129–5140, 2022.
[8] R. Mandryk, M. Hancock, M. Perry, A. Cox, M. Rietzler, F. Geiselhart, J. Gugenheimer, and E. Rukzio, “Breaking the tracking:Enabling weight perception using perceivable tracking offsets,”
Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems, p. 1–12, 2018.
[9] M. Samad, E. Gatti, A. Hermes, H. Benko, and C. Parise, “Pseudohaptic weight: Changing the perceived weight of virtual objects
by manipulating control-display ratio,” Proceedings of the 2019 CHI
Conference on Human Factors in Computing Systems, p. 1–13, 2019.
[10] F. Argelaguet, D. A. G. Jáuregui, M. Marchal, and A. Lécuyer,“Elastic images: Perceiving local elasticity of images through anovel pseudo-haptic deformation effect,” ACM Transactions on
Applied Perception (TAP), vol. 10, no. 3, p. 17, 2013.
[11] A. Lécuyer, S. Coquillart, A. Kheddar, P. Richard, and P. Coiffet, “Pseudo-haptic feedback?: Can isometric input devices simulate force feedback?” Proceedings IEEE Virtual Reality 2000 (Cat.
No.00CB37048), p. 83–90, 2000.
[12] T. Kawabe, “Mid-air action contributes to pseudo-haptic stiffness
effects,” IEEE Transactions on Haptics, vol. 13, no. 1, p. 18–24, 2020.
[13] Y. Ota, Y. Ujitoko, Y. Ban, S. Sakurai, and K. Hirota, “Surface
roughness judgment during finger exploration is changeable byvisual oscillations,” Lecture Notes in Computer Science, p. 33–41,
2020.

© 2023 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission.See https://www.ieee.org/publications/rights/index.html for more information.

This article has been accepted for publication in IEEE Transactions on Visualization and Computer Graphics. This is the author's version which has not been fully edited andcontent may change prior to final publication. Citation information: DOI 10.1109/TVCG.2023.3254522

JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2015[14] Y. Ujitoko, Y. Ban, and K. Hirota, “Modulating fine roughness
perception of vibrotactile textured surface using pseudo-haptic
effect,” IEEE Transactions on Visualization and Computer Graphics,
vol. 25, no. 5, p. 1981–1990, 2019.
[15] P. Habibi and D. Chattopadhyay, “The impact of handedness
on user performance in touchless input,” International Journal of
Human-Computer Studies, vol. 149, p. 102600, 2021.
[16] L. T. D. Paolis and V. D. Luca, “The effects of touchless interaction
on usability and sense of presence in a virtual environment,”
Virtual Reality, p. 1–21, 2022.
[17] K. Harrington, D. R. Large, G. Burnett, and O. Georgiou, “Exploring the use of mid-air ultrasonic feedback to enhance automotive
user interfaces,” Proceedings of the 10th International Conference on
Automotive User Interfaces and Interactive Vehicular Applications, p.
11–20, 2018.
[18] K. Małecki, A. Nowosielski, and M. Kowalicki, “Gesture-baseduser interface for vehicle on-board system: a questionnaire andresearch approach,” Applied Sciences, vol. 10, no. 18, p. 6620, 2020.
[19] O. Georgiou, H. Limerick, L. Corenthy, M. Perry, M. Maksymenko,
S. Frish, J. Müller, M. Bachynskyi, and J. R. Kim, “Mid-air haptic
interfaces for interactive digital signage and kiosks,” Extended
Abstracts of the 2019 CHI Conference on Human Factors in Computing
Systems, p. 1–9, 2019.
[20] R. Mandryk, M. Hancock, M. Perry, A. Cox, L. Corenthy, M. Giordano, R. Hayden, D. Griffiths, C. Jeffrey, H. Limerick, and et al.,“Touchless tactile displays for digital signage,” Extended Abstractsof the 2018 CHI Conference on Human Factors in Computing Systems,
p. 1–4, 2018.
[21] P. Gaucher, F. Argelaguet, J. Royan, and A. Lécuyer, “A novel 3dcarousel based on pseudo-haptic feedback and gestural interaction
for virtual showcasing,” 2013 IEEE Symposium on 3D User Interfaces(3DUI), p. 55–58, 2013.
[22] T. Kawabe, Y. Ujitoko, and T. Yokosaka, “The relationship between
illusory heaviness sensation and the motion speed of visual feedback in gesture-based touchless inputs,” Frontiers in Psychology,
vol. 13, p. 811881, 2022.
[23] J. McIntosh, C. McNeill, M. Fraser, F. Kerber, M. Löchtefeld, and
A. Krüger, “Empress,” Proceedings of the 2016 CHI Conference on
Human Factors in Computing Systems, p. 2332–2342, 2016.
[24] N. Zhao, G. Dublon, N. Gillian, A. Dementyev, and J. A. Paradiso,“Emi spy: Harnessing electromagnetic interference for low-cost,
rapid prototyping of proxemic interaction,” 2015 IEEE 12th International Conference on Wearable and Implantable Body Sensor Networks(BSN), p. 1–6, 2015.
[25] P.-G. Jung, G. Lim, S. Kim, and K. Kong, “A wearable gesturerecognition device for detecting muscular activities based on
air-pressure sensors,” IEEE Transactions on Industrial Informatics,
vol. 11, no. 2, p. 485–494, 2015.
[26] T. Le, R. A. Bahr, M. M. Tentzeris, B. Song, and C.-p. Wong, “Anovel chipless rfid-based stretchable and wearable hand gesturesensor,” 2015 European Microwave Conference (EuMC), p. 371–374,
2015.
[27] T. Kawabe, Y. Ujitoko, T. Yokosaka, and S. Kuroki, “Sense ofresistance for a cursor moved by user’s keystrokes,” Frontiers in
Psychology, vol. 12, p. 652781, 2021.
[28] T. Hachisu, G. Cirio, M. Marchal, A. Lécuyer, and H. Kajimoto,“Pseudo-haptic feedback augmented with visual and tactile vibrations,” in 2011 IEEE International Symposium on VR Innovation.
IEEE, 2011, pp. 327–328.
[29] ——, “Visual vibrations to simulate taps on different materials,”in Haptic Interaction. Springer, 2015, pp. 297–303.
[30] ——, “Virtual chromatic percussions simulated by pseudo-haptic
and vibrotactile feedback,” in Proceedings of the 8th International
Conference on Advances in Computer Entertainment Technology, 2011,
pp. 1–5.
[31] J. Watanabe, “Pseudo-haptic sensation elicited by background visual motion,” ITE Transactions on Media Technology and Applications,vol. 1, no. 2, pp. 199–202, 2013.
[32] R. W. Fleming, “Visual perception of materials and their properties,” Vision Research, vol. 94, p. 62–75, 2014.
[33] ——, “Material perception,” Annual Review of Vision Science, vol. 3,
no. 1, p. 1–24, 2016.
[34] E. H. Adelson, “On seeing stuff: the perception of materials byhumans and machines,” Human Vision and Electronic Imaging VI, p.
1–12, 2001.

16[35] T. Kawabe and S. Nishida, “Seeing jelly: Judging elasticity of atransparent object,” Proceedings of the ACM Symposium on Applied
Perception, SAP ’16, p. 121–128, 2016.
[36] V. C. Paulun and R. W. Fleming, “Visually inferring elasticity fromthe motion trajectory of bouncing cubes,” Journal of Vision, vol. 20,
no. 6, p. 6–6, 2020.
[37] F. Schmidt, V. C. Paulun, J. J. R. v. Assen, and R. W. Fleming,“Inferring the stiffness of unfamiliar objects from optical, shape,and motion cuesschmidt et al.” Journal of Vision, vol. 17, no. 3, p.
18–18, 2017.
[38] J. J. R. v. Assen, S. Nishida, and R. W. Fleming, “Visual perceptionof liquids: Insights from deep neural networks,” PLOS Computational Biology, vol. 16, no. 8, p. e1008018, 2020.
[39] J. J. R. v. Assen, P. Barla, and R. W. Fleming, “Visual features in
the perception of liquids,” Current Biology, vol. 28, no. 3, pp. 452–
458.e4, 2018.
[40] T. Kawabe, K. Maruya, R. W. Fleming, and S. Nishida, “Seeing
liquids from visual motion,” Vision Research, vol. 109, p. 125–138,
2015.
[41] V. C. Paulun, T. Kawabe, S. Nishida, and R. W. Fleming, “Seeing
liquids from static snapshots,” Vision Research, vol. 115, no. Pt B,
p. 163–174, 2015.
[42] W. Bi, P. Jin, H. Nienborg, and B. Xiao, “Estimating mechanicalproperties of cloth from videos using dense motion trajectories:
Human psychophysics and machine learning,” Journal of Vision,
vol. 18, no. 5, p. 12–12, 2018.
[43] ——, “Manipulating patterns of dynamic deformation elicits theimpression of cloth with varying stiffness,” Journal of Vision,
vol. 19, no. 5, p. 18–18, 2019.
[44] W. Bi and B. Xiao, “Perceptual constancy of mechanical propertiesof cloth under variation of external forces,” p. 19–23, 2016.
[45] E. Fakhoury, P. R. Culmer, and B. Henson, “The effect of indentation force and displacement on visual perception of compliance,”
2015 IEEE World Haptics Conference (WHC), p. 88–93, 2015.
[46] V. C. Paulun, F. Schmidt, J. J. R. v. Assen, and R. W. Fleming,“Shape, motion, and optical cues to stiffness of elastic objects,”
Journal of Vision, vol. 17, no. 1, p. 20–20, 2017.
[47] Y. Ujitoko and T. Kawabe, “Perceptual judgments for the softnessof materials under indentation,” Scientific Reports, vol. 12, no. 1, p.
1761, 2022.
[48] M. Srinivassan, “The impact of visual information on the haptic
perception of stiffness in virtual environments,” Proc. ASME Dynamic Systems and Control Div., vol. 58, pp. 555–559, 1996.
[49] S.-i. Yabe, T. Kimura, H. Kishino, and T. Nojima, “The effectof visually-induced pseudo-haptic on softness.” in ICAT-EGVE(Posters and Demos), 2014.
[50] Y. Ban, T. Narumi, T. Tanikawa, and M. Hirose, “Controlling
perceived stiffness of pinched objects using visual feedback ofhand deformation,” in 2014 IEEE Haptics Symposium (HAPTICS).
IEEE, 2014, pp. 557–562.
[51] Y. Gaffary, B. Le Gouis, M. Marchal, F. Argelaguet, B. Arnaldi,
and A. Lécuyer, “Ar feels “softer” than vr: Haptic perception of
stiffness in augmented versus virtual reality,” IEEE transactions on
visualization and computer graphics, vol. 23, no. 11, pp. 2372–2377,
2017.
[52] M. Sato, S. Suzuki, D. Ebihara, S. Kato, and S. Ishigaki, “Pseudosoftness evaluation in grasping a virtual object with a bare hand,”in ACM SIGGRAPH 2016 Posters, 2016, pp. 1–1.
[53] J. Cauquis, V. R. Mercado, G. Casiez, J.-M. Normand, andA. Lécuyer, ““kapow!”: Studying the design of visual feedback
for representing contacts in extended reality,” in Proceedings of the
28th ACM Symposium on Virtual Reality Software and Technology,
2022, pp. 1–11.
[54] T. Kawabe, “Visual assessment of causality in the poisson effect,”Scientific Reports, vol. 9, no. 1, pp. 1–10, 2019.
[55] ——, “Perceptual properties of the poisson effect,” Frontiers in
Psychology, p. 4044, 2021.
[56] L. Dominjon, A. Lécuyer, J.-M. Burkhardt, P. Richard, and S. Richir,“Influence of control/display ratio on the perception of mass ofmanipulated objects in virtual environments,” in IEEE Proceedings.
VR 2005. Virtual Reality, 2005. IEEE, 2005, pp. 19–25.
[57] G. N. Greaves, A. L. Greer, R. S. Lakes, and T. Rouxel, “Poisson’s
ratio and modern materials,” Nature materials, vol. 10, no. 11, pp.
823–837, 2011.
[58] Q. Li, S. J. Joo, J. D. Yeatman, and K. Reinecke, “Controllingfor participants’ viewing distance in large-scale, psychophysical© 2023 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission.See https://www.ieee.org/publications/rights/index.html for more information.

This article has been accepted for publication in IEEE Transactions on Visualization and Computer Graphics. This is the author's version which has not been fully edited andcontent may change prior to final publication. Citation information: DOI 10.1109/TVCG.2023.3254522

JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 201517

online experiments using a virtual chinrest,” Scientific Reports,
vol. 10, no. 1, p. 904, 2020.
[59] J. O. Wobbrock, L. Findlater, D. Gergle, and J. J. Higgins, “Thealigned rank transform for nonparametric factorial analyses using
only anova procedures,” Proceedings of the 2011 annual conferenceon Human factors in computing systems - CHI ’11, p. 143–146, 2011.
[60] L. A. Elkin, M. Kay, J. J. Higgins, and J. O. Wobbrock, “An aligned
rank transform procedure for multifactor contrast tests,” The 34th
Annual ACM Symposium on User Interface Software and Technology,
p. 754–768, 2021.
[61] D. Wang, K. Ohnishi, and W. Xu, “Multimodal haptic display forvirtual reality: A survey,” IEEE Transactions on Industrial Electronics,
vol. 67, no. 1, pp. 610–623, 2019.
[62] Polhemus, “Human interface guidelines,” https://polhemus.
com/.

Takahiro Kawabe is a Senior Distinguished Researcher and the leader of Sensory Representation Group, Human Information Science Laboratories, NTT Communication Science Laboratories. He received a Doctor of Psychology from
Kyushu University, Fukuoka, in 2005. In 2011, hejoined NTT Communication Science Laboratories, where he studies basic and applied aspectsof human perception. He received the 2013 JPAAward for International Contributions to Psychology: Award for Distinguished Early and Middle
Career Contributions from the Japanese Psychological Association. In
2018, he was also awarded The Young Scientists’ Prize of The Commendation for Science and Technology by the Minister of Education,
Culture, Sports, Science and Technology. He is a member of the Vision
Sciences Society and the Vision Society of Japan.

Yusuke Ujitoko (Member, IEEE) received the
B.E.degree in mechanical engineering and the
M.A.E.degree in interdisciplinary information
studies from the University of Tokyo, Tokyo,
Japan, in 2014 and 2016, respectively, and the
Ph.D. degree from The University of ElectroCommunications, Chofu, Japan, in 2020. Since
2020, he has been a Distinguished Researcher
at NTT Communication Science Laboratory, Atsugi, Japan. From 2016 to 2020, he was a
Member of the R&D Group, Hitachi, Ltd., Tokyo,
Japan. His research interests include applied haptic perception andhaptic interfaces.

© 2023 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission.See https://www.ieee.org/publications/rights/index.html for more information.