This is the author's version of an article that has been published in this journal. Changes were made to this version by the publisher prior to publication.
The final version of record is available at
http://dx.doi.org/10.1109/TOH.2019.2954882
JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2015

1

Influence of Sparse Contact Point and Finger
Penetration In Object on Shape Recognition
Yusuke Ujitoko, Member, IEEE, Sho Sakurai, Member, IEEE, and Koichi Hirota, Member, IEEE,
Abstract—Making a virtual object shape recognizable using a haptic display is one of the major themes of haptic research. In previous
works, multi-point haptic displays have been developed that had a high contact point density between the users’ finger skin and thevirtual object. However, the ideal contact point density that enables intuitive shape recognition has not been determined yet.
Meanwhile, there is also a fundamental problem, that is, real fingers and virtual objects do penetrate, which cannot be solved with such
wearable displays. This study investigated the influence of both contact point density and penetration on the shape recognition
performance. We prepared a real testing environment where the user touched the real object, and where we could simulate both thesparse contact point and the penetration. Specifically, users’ fingers wore thin film coated with glass particles and they touched theurethane foams that deformed flexibly. The result of experiments showed a broad trend where the sparseness of the contact and thesoftness of the object influenced the exploration time required to achieve recognition. In addition, the result suggested that the larger
contact density could make up for the problem of penetration. We confirmed it by conducting two different tasks: (1) exploring the object
surface with the index finger and (2) grasping the object surface with the thumb and the index finger.
Index Terms—Shape Recognition, Wearable Shape Display, Contact Point Density, Finger Penetration

F

1

I NTRODUCTION

H

UMANS are highly skilled in perceiving objects’ shapeeven in the absence of visual information. They cansense the geometry of the real object’s surface based solely
on haptic cues, which comprise kinesthetic and cutaneouscues. Blindfolded people can allegedly perceive the shapeof solid objects with an accuracy of 98% within a few
seconds [1]. People also appear capable of distinguishing
between twelve solid copies of bell peppers just from touch,with the same accuracy as if they used vision alone [2].
Thus, humans can reasonably be expected to recognize
the shape of computer-generated objects in virtual environments if the haptic stimuli are replicated accurately.
Researchers have developed various multiple-contact-point
haptic displays to present the surface geometry of a virtual object to users. As one example of the multi-point
display, pin-array displays raise pins against the humanskin to render force. Traditionally, most devices have beenmechanically grounded [3], [4]. They could present bothkinesthetic and cutaneous cues using robustly grounded
forces with users. However, these devices cannot display
shapes in any position or location around users. In otherwords, the workspace and portability are constrained. Alarge interactive mobile workspace may be useful for the
exploration of virtual spaces.
Recently, more haptic system designs have started ap-•••

Yusuke Ujitoko is with Research & Development Group, Hitachi, Ltd.,
Yokohama, Japan. and is a Graduate Student of the University of ElectroCommunications, Tokyo, Japan. E-mail: yusuke.ujitoko.uz@hitachi.com.
Sho Sakurai is with the Graduate School of Information Systems,
The University of Electro-Communications, Tokyo, Japan. E-mail:
sho@vogue.is.uec.ac.jp
Koichi Hirota is with the Graduate School of Information Systems,
The University of Electro-Communications, Tokyo, Japan. E-mail: hirota@vogue.is.uec.ac.jp

Manuscript received April 19, 2005; revised August 26, 2015.

pearing with wearability in mind, and in this context
wearable, multiple contact point displays have been developed [5], [6]. The mechanical actuators employed usually
include pneumatic arrays and shape memory alloy displays(SMA).
However, currently, the ability to recognize the shape of
virtual objects when using these wearable devices is worse
than when using a bare finger or hand. Tanabe et al. [7]developed vibrotactile whole hand glove with 52 vibrators.
They evaluated the shape recognition performance for four
different 3D virtual objects. The correct answer rate wasapproximately 70% and the response time was close to
20s. Taniguchi et al. [8] developed a pin array display
for the whole hand as dense as the two-point threshold.
The recognition time was close to 40s and the answering
rate accuracy was approximately 70%. These results using
current multi-point haptic displays are far inferior to theones coming from the interaction between an actual finger
and the actual objects, as obtained by [1].
To achieve a higher shape recognition performance, apromising approach is to implement a high contact point
density to provide richer cutaneous cues. Though current
multi-point displays apply contact forces in a sparse distribution (Fig.1 (b)), humans sense the local geometry of
the actual object based on the dense distribution of thecontact force (Fig.1 (a)). Thus, increasing the density of thehuman tactile spatial resolution is widely believed to lead tohigher recognition performance. Thus, actuator integration
technology to increase the contact point density has beendeveloped actively, targeting a higher recognition performance. However, it is not even clear whether a larger contact
point density truly contributes to better shape recognition.
In addition, the exact contact point density enough to realize
intuitive shape recognition has yet to be determined. These
are still unknown in part because the shape recognition

Copyright (c) 2019 IEEE. Personal use is permitted. For any other purposes, permission must be obtained from the IEEE by emailing pubs-permissions@ieee.org.

This is the author's version of an article that has been published in this journal. Changes were made to this version by the publisher prior to publication.
The final version of record is available at
http://dx.doi.org/10.1109/TOH.2019.2954882
JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2015(a) Finger touches real solid surface
dense contact force
real solid
surfacefinger DOES NOT penetrates
into real surfaces(b) Finger touches virtual solid surface

virtual solid
surfacepin matrix
displaysparse contact force
finger penetrates
into virtual surface

2

that controlled the effect of the penetration (Fig.1 (c)).
We conducted shape recognition experiments to observe
the effects of contact density and penetration. To broadly
investigate these effects, we conducted the experiments
assuming two different exploratory procedures [9]. In experiment 1, participants explored the object surface with theindex finger. The exploration in experiment 1 corresponded
to contour following among the exploratory procedure [9].
In experiment 2, participants grasped an object with the
thumb and index fingers. The exploration in experiment 2
corresponded to the enclosure.
The contributions of this study are summarized here:•(c) Finger coated with particles touches real soft surface

real soft
surfaceglass
particlesparse contact force
finger penetrates
into real soft surface•

Fig. 1. (a) The ideal haptic feedback to represent the shape of a solid
object. (b) Wearable pin matrix haptic feedback to represent the shapeof a solid object. The pin’s density defines the sparseness of the contact
force from the virtual solid surface. (c) Simulating the wearable pin
matrix haptic feedback. Users wear a thin finger cot covered with glass
particles. The glass particle size defines the sparseness of the contact
force from the actual soft surface. The softness of the object leads users
to penetrate into the surface.

evaluation in previous studies was performed solely basedon a specific contact point density of the developed device,hence evaluations across various contact point densities
were lacking. Another reason is that the shape recognition
evaluation beyond the maximum contact point density thatcan be developed with current technology is impossible.
Meanwhile, there is an insolvable problem specific towearable display. Indeed, the user’s actual finger moves into
the surface of the virtual object (Fig.1 (b)). Even if the virtual
contact forces are applied, users’ fingers can nudge into theobject. As a result, users can use small kinesthetic cues to
explore surfaces, which would make recognition difficult.
We consider that the problem of finger penetration into theobject could influence the shape recognition.
Based on these considerations, this study investigatedthe effect of the sparse contact point and penetration on
the recognition performance of the object shape. In particular, we had an interest in the impact of the contact point
density when the penetration problem occurred. In order toinvestigate this, we used a real environment where we could
gradually control the contact point density and penetration.
Specifically, users’ fingers wore an elastic film coated withglass particles. We refer to the film coated with particles as
the ”particulate glove” in this paper. The size of the particles
in the particulate glove defined the contact point density.
Because of the particulate glove, we could investigate theeffect of higher contact point density that was impossible
to investigate with current wearable multi-point display.
Users touched the urethane foam with a variable softness

We propose a method for preparing an actual testing
environment where we can gradually control thesparseness of the contact point and the penetration of
the finger in the object independently. To control thecontact point density, we covered the actual user’s
finger with a particulate glove. In order to mimic
penetration, we used urethane foams, which deform
flexibly, as the target of touch.
The result of the experiments demonstrated that boththe sparseness of the contact point and the softnessof the object contributed significantly to the longer
exploration time. We confirmed that the exploration
time decreased as the contact point became denser
even when participants were exposed to the softest
urethane form. It suggested that a higher contact
density of wearable multi-point display could makeup for the problem of the penetration.

2

R ELATED WORK

2.1

Human Shape Recognition

Various attempts were made to disentangle the contributions of the two primary sources of information in haptic
shape recognition: kinesthetic and cutaneous [10]. Kinesthetic mechanoreceptors encode information on the state ofmuscles, tendons, and joints. Cutaneous mechanoreceptors
respond to the deformation of the skin. Both kinesthetic
and cutaneous cues are known to be important for shaperecognition [11], [12], [13], [14]. The work in [11] showedthat the addition of cutaneous cues to kinesthetic cues
significantly improved the recognition of the orientation of asurface. Similarly, recent studies have shown the importanceof cutaneous stimuli in addition to the kinesthetic stimuli
in discerning curvature [15], [16]. As for the compliance
perception, how kinesthetic and cutaneous cues contributeto the perception have been addressed in [17], [18], [19].
Previous work compared their observed data to two
candidate models and investigated how two cues are integrated. The candidate models are Optimal Integration
model [20], which shows how the means and variancesof cues can be pooled, and a Sensory Capture model, in
which the most reliable modality is the only one that isrepresented in the multi-sensory perception [21]. The workin [11], [22] showed that the obtained data in some case fit
to the Optimal Integration model and data in other case fit
to the Sensory Capture model.

Copyright (c) 2019 IEEE. Personal use is permitted. For any other purposes, permission must be obtained from the IEEE by emailing pubs-permissions@ieee.org.

This is the author's version of an article that has been published in this journal. Changes were made to this version by the publisher prior to publication.
The final version of record is available at
http://dx.doi.org/10.1109/TOH.2019.2954882
JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2015

3

2.2 Wearable Multi-point Haptic Display

3

Based on these findings, to improve the shape recognition
performance using a wearable multi-point haptic display,increasing the contact density to provide more cutaneouscues is broadly viewed as important. Kim et al. [6] developed a wearable display composed of a 4×4 pin array on thefingertip. The diameter of the pin was 0.5 mm and pins werearranged in a 1.5 mm interval. The pin moved normallyagainst the skin and normal indentation was achieved.
Sarakoglou et al. [23] also proposed a compact 4 × 4 tactile
array. Caldwell et al. [24] proposed a device that is able tocombine normal indentation and shear stimuli. They used a
4 × 4 pin array with a spatial interval of pins of 1.75 mm.
Aside from pin arrays, another popular set of wearable
systems providing multi-point stimuli is the one exploiting
pneumatic jets. Kim et al. [25] developed a 5 × 5 array of
air jets placed in direct contact with the fingertip and five
additional air nozzles that are in direct contact with eachside of the finger to produce the lateral force. Taniguchi etal. [8] developed a pin array display for the whole hand as
dense as the two-point threshold.
Although in these studies custom displays were builtwith an increasing contact density, the contribution of thiscontact density to the shape recognition performance wasstill unclear. In addition, the shape recognition evaluation
was restricted by the contact density achievable by the display. Recognition could only be evaluated within the contact
density range of the developed displays. A wearable display
with sub-millimeter-scale interval for contact points cannot
be developed with current technology. As a guideline for the
future development of multiple contact point displays, wewould like to know more about the recognition performance
under such sub-millimeter-scale contact density. Thus, it isstill not clear what the ideal contact density is to achieve
intuitive shape recognition.

3.1 Hypotheses

2.3 Effect of Contact Density of Grounded Type Display
Although there are no previous studies on the impact ofcontact density using wearable displays, researchers haveinvestigated these effects using grounded type pin array
displays. The work of [26] studied participants’ shape recognition performance using three passive arrays with distinct
pin spacing values. Results showed that the smaller the pin
spacing the better the shape recognition. Another work evaluated the recognition of shapes [27] and concave/convex
surfaces [28]. The work [29] showed that a tactile arraywith a pin spacing of 1.8 mm and a pin array of size
1mm2 can accurately convey tactile information related tothe inclination of an edge or even to simple tactile shapes.
As in these studies, the effect of the density of the contact
point has been well researched in the field of grounded-type
pin array display.
However, the knowledge accumulated on these effects is
not directly transferable to wearable displays. Indeed, there
are little kinesthetic cues on wearable display although arobust kinesthetic cue exists on the grounded type display.
The evaluation of the impact of the contact point density
under the condition that there are little kinesthetic cues isrequired.

H YPOTHESES AND M ETHOD

We hypothesized that a denser contact point contributes tothe shape recognition performance even when the finger
penetrates into objects.
3.2 Method: Simulation of Sparse Contact Density andFinger Penetration
We prepared a real-world environment in which we mimicked the sparse contact point and the finger penetration andinvestigated the recognition performance of the object shapeby the human. Specifically, by covering the actual finger
with the particulate glove, we simulated a sparse contact
point density with the object (Fig.1(c)).
We used objects made of deformable urethane foam
as a target of exploration. When users pressed a finger
against the deformable urethane foam, the users’ fingers
intruded into the original surface of the urethane foam.
Under this condition, users could use minor kinestheticcues to recognize the shape of the object. The softer theurethane foam users touch is, the more muted kinestheticcues are; eventually they come closer to the condition whereusers’ fingers penetrate the virtual object (Fig.1(c)). Whenthe condition of kinesthetic cues are fixed the cutaneous cuesin terms of the density of contact point can be controlled by
the particulate glove.
3.2.1 Particulate Glove
In order to mimic the issue of a sparse contact point, we
asked users to wear the particulate glove. The illustrationin Fig.2 shows the cross section of the finger covered withtwo distinct particulate gloves. As the diameter of the glass
particles becomes larger, the contact point becomes sparser.
As the diameter of glass particles becomes smaller, thecontact point becomes denser.
Overly large particles may lead to another effect, withthe distance between the finger and the object increasing.
However, we only used large objects in the experiments andthus the change of distance can largely be ignored.

Fig. 2. As the diameter of the glass particles is larger, the contact point
becomes sparser. The diameter of the left glass particles is half of thatof the right ones. The contact density is also half according to the ratio
between diameters.

We used the glass particles (KENIS Ltd.) and thin film
for finger (Misumi ID:RTXY-CT-M). The glass particles wereidentical to the ones used in [30] to create a rough texture.
The particles did not have any sharp edges on the surface ofthe sphere and did not generate unnecessary haptic stimuli.
We followed [30], [31]’s approach which is to systematicallymake the texture plate reproducible. The glass particles werefirst arranged on the thin film using double-sided sticky
tape. After arranging particles on the film, users wore them

Copyright (c) 2019 IEEE. Personal use is permitted. For any other purposes, permission must be obtained from the IEEE by emailing pubs-permissions@ieee.org.

This is the author's version of an article that has been published in this journal. Changes were made to this version by the publisher prior to publication.
The final version of record is available at
http://dx.doi.org/10.1109/TOH.2019.2954882
JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2015

on fingers. The thin film’s thickness was 0.1 mm and thefilm deformed easily by virtue of its elasticity.
We made five types of particulate gloves with distinct
particle sizes and thus five distinct sparse conditions for theexperiments. The diameters of the glass particles were 0.425,
1.194, 2, 3, and 4.3305 mm. The diameters were certified
by the manufacturer (KENIS Ltd.). Fig.3 shows the index
fingers with the specific gloves used in experiment 1. In
experiment 2, participants wore the particulate gloves on
the thumb and the index fingers.

4and on the curved surface of the finger is different or thecurvature of finger slightly varies from person to person,
this study considers that the packing ratio would remain
identical across different particulate gloves.
TABLE 1
The measured packing ratio of glass particles attached to the flat platefor each particle diameter.
#

Particle diameter [mm]Measured packing ratio [%]

d1

0.425

71.5 ± 0.022

d2

1.194

73.4 ± 0.016

d3

2.000

73.2 ± 0.025

d4

3.000

71.7 ± 0.027

d5

4.331

71.8 ± 0.063

3.2.2 Urethane Foam Object
In terms of soft material, we specifically used urethane
foams. We used three types of urethane foam, with distinct levels in hardness. We call them s1 (INOAC CORP,
ECZ), s2 (INOAC CORP, UEM-35G), and s3 (INOAC CORP,
EMO) in ascending order of softness. The specification of
the urethane foams is provided in Table.2. The hardness(25% ILD), density, and compression set described in thetable were certified by the manufacturer. Fig.4 demonstrates
the variability in the deformation of the foam based on
hardness.
TABLE 2The density of the urethane foam in this study.
softness condition

Density[kg/m3 ]Hardness (25% ILD)
[N/314cm3]Compression set
[%]

s1

s2

s3

16±1.535±3.050±5.080

156.8

400.359

4.5

4Fig. 3. Five types of particulate gloves with distinct glass particles
diameters

Though the uniformity of packing ratio, which is the proportion of area covered by glass particles, has been alreadyassured by previous studies [30], [31], the specific valueof the packing ratio has not been clarified yet. Thus, wemeasured the packing ratio of the glass particles attached tothe thin films. This was calculated by counting the numberof particles in certain areas using a microscope when weplaced the glass particles on a flat plate in the same way. We
took three photographs at different positions for each plateand measured the average packing ratio. Table.1 shows thepacking ratio of each plate. These ratios ranged from 71
to 74. Although the attached curvature of the flat surface

Fig. 4. The deformation of the three types of urethane foams when thesame amount of force is applied.

4

E XPERIMENT 1: E XPLORING WITH I NDEX F IN -GER

Experiment 1 was conducted to test the effect of boththe sparseness of the contact point and the penetrationof the finger in the object on the recognition performanceas participants explored surfaces with the index finger.

Copyright (c) 2019 IEEE. Personal use is permitted. For any other purposes, permission must be obtained from the IEEE by emailing pubs-permissions@ieee.org.

This is the author's version of an article that has been published in this journal. Changes were made to this version by the publisher prior to publication.
The final version of record is available at
http://dx.doi.org/10.1109/TOH.2019.2954882
JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2015

5

There were ten participants, seven males and three females,with ages ranging from 21 to 24. The participants were allright-handed. They were screened to ensure that they were
not depressed nor excessively tired as perception would
be altered by physical or emotional states. The University
of Electro-Communications Ethics committee approved thedata acquisition in this paper and written informed consent
was obtained from all participants.
4.1 Experimental System
The participants’ task in this experiment was to discern the
shape of the actual object based solely on haptic informationin the shortest time possible. Participants were seated ona chair (Fig.5). The object was placed approximately 30
cm in front of them on a desk. The base of the object indirect contact with the desk was always the same but theobject was rotated randomly along the z-axis. Participants
placed their dominant hands under a curtain and touched
the object with their index finger.

Fig. 5. Appearance of the experimental system. (Left): Participants touch
the object placed behind the curtain. They were asked to gauge theobject shape based solely on haptic information. (Right): The object wasplaced in a certain position. The participants’ finger was covered with theparticulate gloves.

There were three object shapes. Their cross section waseither square, an equilateral triangle, or trapezoid. The appearance and cross section of the object shapes are shownin Fig.6. Their height and depth were 50 mm. Their crosssections were distinct. We refer to the difference in object
shapes as the shape condition.
(a) square(b) trapezoid(c) equilateraltriangle

30mm
50mm

50
mm

the nine types of objects, which consisted of permutationsin the three shape conditions and three softness conditions.
Participants wore one of the five particulate gloves on
the index finger of their dominant hand as shown in Fig.3.
Participants were only allowed to touch the object with theindex finger area coated with glass particles.
4.2

Task Design

This experiment used a within-participants design. As we
wanted to investigate the general effect of sparseness andpenetration, there were no instructions on specific ways totouch the object or to distinguish between object shapes. The
experiment comprised an initial phase and a test phase.
In the initial phase, participants saw the nine types ofobjects that were used in this experiment. The nine objects
were different in softness and shape. Next, they touched
the objects with a bare index finger for three seconds eachfor adaptation. The participants touched the objects one
by one not only for haptic information but also for visual
information. The participants then wore one of particulate
gloves on their index fingers. Nine objects were placed
behind the curtain sequentially and participants tried to
distinguish their shape individually in one occurrence. Afterthese completed, the test phase started.
During the test phase, participants distinguished theshape of the object by touch with the particle coated index
finger in the shortest time possible. Let us describe the procedure of a trial in the test phase. The experimenter placed
one of the objects in the designated position, randomly
rotating it along the z-axis, and then told participants to start
touching. Participants tried to distinguish the shape andanswered a name in Fig.6 in a forced choice. The exploration
time taken and the name of the shape answered by theparticipants were recorded by the experimenter. After thetest phase, the participants were asked to clarify how to
distinguish the three shapes and wrote free comments forthe experiment.
There were five contact density conditions. For each contact density condition, the participants touched 27 objects(3 shape conditions × 3 softness conditions × 3 times). In
aggregate, one participant touched 135 objects (27 objects ×
5 contact density conditions). The presentation order of thecontact density conditions was pseudo-randomly assigned
across participants. The presentation order of the shapeand softness condition for each contact density condition
was also randomly assigned to be counter-balanced acrossparticipants.
4.3 Data Analysis

70mm

50mm

Fig. 6. The cross section and appearance of three different shapes.

As stated in section 3.2.2, there are three different softness conditions for the objects. Thus, participants touched

We performed a single round of 3 σ clipping to remove
outliers. All the following analysis was performed on thedata that did not include outliers. In order to identify there
was a main effect of softness condition, contact density
condition, or the interaction effect on the exploration time,
we performed a two-way repeated ANOVA of factors ofthese two conditions. We conducted a Shapiro-Wilk test to
check normality and a Mauchly’s test to check the sphericity
criteria in advance of the ANOVA test. If the sphericity
assumptions were violated, Greenhouse-Geisser corrections
were applied to the ANOVA test. As a result of the ANOVA

Copyright (c) 2019 IEEE. Personal use is permitted. For any other purposes, permission must be obtained from the IEEE by emailing pubs-permissions@ieee.org.

This is the author's version of an article that has been published in this journal. Changes were made to this version by the publisher prior to publication.
The final version of record is available at
http://dx.doi.org/10.1109/TOH.2019.2954882
JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2015

test, when there was a significant interaction effect between
the softness condition and contact density condition, TukeyKramer posthoc test for pairwise comparison of data foreach combination of both conditions. Otherwise, and when
there was a main effect of softness condition or contact
density condition, the Tukey-Kramer post-hoc test was conducted for the pairs of condition that was found to besignificant. If the sphericity assumptions were violated, thepost-hoc test was conducted with bonferroni adjustment.

6

and a significant main effect of the contact density condition (F (4, 1335) = 4.26, p = 1.9 × 10−3 ). There wasno significant interaction effect between the two factors(F (8, 1335) = 0.11, p = 0.99). Since there was no interaction
effect, the effect of the softness condition and contact density
condition were analyzed separately.

4.4 Results
Fig.7 shows the average exploration time for each participant across all conditions without removed outlier samples.
We removed 25 outlier samples that were out of 3 σ range.
All of the removed samples were the ones obtained from acertain participant (participant ID:J).
[s]

*
**

* :p<0.05** :p<0.01
Fig. 9. The average exploration time across softness conditions withrespect to contact density condition.

Fig. 7. The average exploration time for each participant across allconditions.

The black line in Fig.9 shows change in the average
exploration time with standard error across softness conditions. We conducted post hoc test with bonferroni adjustment for pair-wise comparison of 5 contact density conditions. As a result, there were significant differences between
d1 and d5 (p < 0.01),and d2 and d5 (p < 0.05).

* :p<0.05**:p<0.01

Fig.8 shows that from a matrix viewpoint. It shows thegradation of the exploration time along both the contact
density and softness axes.

**

exploration
time [s]Fig. 8. The average exploration time for each softness and contact
density condition.

The sphericity test s results indicated that the variance homogeneity assumption was violated for both conditions and interaction effect between them (p < 0.01).
We performed a two-way repeated ANOVA with factors ofsoftness condition (s1, s2, s3) and contact density condition(d1, d2, d3, d4, d5) on the exploration time. According to the
ANOVA results, there was a significant main effect of thesoftness condition (F (2, 1335) = 4.77, p = 8.5 × 10−3 )*Fig. 10. The average exploration time across contact point density
conditions with respect to softness condition.

The black line in Fig.10 shows change in the average
exploration time across contact density conditions. We conducted post hoc test with bonferroni adjustment for pairwise comparison of 3 softness conditions There were significant differences between s1 and s3 (p < 0.01),and s2 ands3 (p < 0.05).
The probability of correct responses for the softness
and contact density condition are shown in Fig.11. We
performed a two-way repeated ANOVA with factors ofsoftness condition (s1, s2, s3) and contact density condition(d1, d2, d3, d4, d5) with the probability of correct responses

Copyright (c) 2019 IEEE. Personal use is permitted. For any other purposes, permission must be obtained from the IEEE by emailing pubs-permissions@ieee.org.

This is the author's version of an article that has been published in this journal. Changes were made to this version by the publisher prior to publication.
The final version of record is available at
http://dx.doi.org/10.1109/TOH.2019.2954882
JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2015

and there was no significance on the main effect of bothconditions and interaction effect.

Fig. 11. The probability of correct answer for softness and contact
density condition.

4.5

Discussion

The result of the ANOVA tests on the two factors of thesoftness and contact density conditions shows that these
two factors had a significant effect on the exploration time. It
suggested that if the contact point was sparse, it took longer
for the participants to distinguish shapes. It also suggested
that when the object was soft, it took longer to distinguishshapes. In addition, the gradation of the exploration time in
Fig.8 and the significantly different pairs identified by the
post-hoc test in Fig.9 and Fig.10 show that there seems to
be a broad trend with a significant divide between the soft,
small contact density condition and the hard, large contact
density condition.
We observe the tendency that the average exploration
time significantly went up with regard to the contact point
density (shown in Fig.9). According to the post-hoc multiple
comparison test, we confirm the significant difference in
exploration time between d1 and d5. The average difference
in exploration time between them was approximately 1.8
second. Also, we confirm the significant difference in exploration time between d2 and d5. The average difference in
exploration time was approximately 1.5 second. Even if theobject was softest (s1), the time required for discrimination
was shortened when the contact point density is higher.
Though there was no significant effect between the contact
density conditions for s1, which was the softest condition,
the average exploration time went up with regard to thecontact point density (shown in Fig.9). Identical trends
materialized in the lines of s2 and s3 in Fig.9). In addition,
a comment from participants supports this analysis, as theysuggested that it was easy to distinguish the square andtrapezoid when the particles were small. We paid attention
to the performance of softest condition s1 with denser density condition d1 or d2 were better than hardest condition s3
with sparser condition d5. This means that denser contact
point density has a possibility of making up for the problemof the penetration.
On the other hand, we found the average exploration
time plateau between s1 and s2 in Fig.10. As a quantitative

7

evidence, there was not a significant difference between s1s2 though there were not between s1-s3 and s2-s3. Oneof the possible explanations is that participants became toresort to the cutaneous cues to recognize the object shapewhen s1 and s2; thus, there was little influence from thedifference of kinesthetic positional cue.
In this experiment, we did not restrict participants in theway they touched the object surface or distinguished theobject shapes. Indeed, we wanted to confirm the impact of
the contact density and the penetration while users freely
explored the invisible objects. Fig.7 reports the variations
in exploration time among participants. One of the probable reasons for the variance is that there were multiple
methods for distinguishing objects and shapes per participant. Regarding how to distinguish among three shapes,
participants commonly indicated in the comments that theyeasily distinguished the equilateral triangle from the othertwo shapes based on the presence or absence of an acutecorner. In contrast, there were various methods to identify
the square and trapezoid that were chosen freely by theparticipants. Four out of ten participants focused on theabsolute inclination angle of the side area. They said thatthey ran their fingers on the side areas and looked forthe side areas whose inclination were not vertical to theground. Other four participants focused on the area of the
top surface. They said that they sensed the area size ofthe top surface by pressing their fingertip onto the surface.
These differences in identification methods could lead tothe variance of exploration time. In addition, another factor
could be the variability in the movement speed of the index
finger as there were no instructions in that regard.

5 E XPERIMENT 2: G RASPING WITH T HUMB ANDI NDEX F INGERS
Experiment 2 was conducted to test the effect of both thesparseness of the contact point and the penetration while
participants grasped objects with their thumb and index
fingers. Although the exploring procedure, the position onwhich to attach the particulate gloves, and the object shapes
were different from experiment 1, the task design and data
analysis were identical.
There were six participants, five males and one female,and their ages ranged from 21 to 24. All participants wereright-handed. They were screened to ensure that they were
not depressed, overly tired as perception would be affected by their physical or emotional states. The University
of Electro-Communications Ethics committee approved thedata acquisition in this paper and written informed consent
was obtained from all participants.
5.1

Experimental System

The experimental system was identical to experiment 1.
Participants placed their dominant hands under a curtain
and grasped the object with their thumb and index fingers(shown in Fig.12).
There were three object shapes. Their cross section waseither a square, a rhombus, or an equilateral triangle. The
appearance and cross section of the object shapes are shownin Fig.13. The cross sections from the top view were different

Copyright (c) 2019 IEEE. Personal use is permitted. For any other purposes, permission must be obtained from the IEEE by emailing pubs-permissions@ieee.org.

This is the author's version of an article that has been published in this journal. Changes were made to this version by the publisher prior to publication.
The final version of record is available at
http://dx.doi.org/10.1109/TOH.2019.2954882
JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 20158

Fig. 12. Appearance of the experimental system. (Left): Participants
touched the object placed behind the curtain. They were asked to gauge
the object shape solely with haptic information. (Right): The object wasplaced in a certain position. Participants’ fingers were covered with theparticulate gloves.

from each other. On the cross section, the average lengthfrom the center to the corner was the same among all shapes,
namely 17.5 mm. The height of the objects was 50 mm.
We refer to the difference in object shapes as the shapecondition. Participants grasped the side areas of the shapewith their thumb and index fingers.
(a) square(c) equilateraltriangle(b) rhombus
30mm

35mm
40mm

17.5mmFig. 14. Index and thumb fingers covered with particulate gloves. Threetypes of particle sizes were used in this experiment.

50mm

5.2
Fig. 13. The cross section and the appearance of three different shapes.

As stated in section 3.2.2, there were three different
softness conditions on objects. Thus, participants grasped
the nine types of objects, consisting of a permutation of thethree shape conditions and three softness conditions.
We conducted informal experiments in advance andfound that it took much longer to complete than the previous experiment. To avoid the participants’ fatigue, we used
three types of particulate gloves on the thumb and index
fingers of their dominant hand as shown in Fig.14 rather
than five types. The participants were only allowed to touch
the object with their thumb and index fingers and the areasin the middle coated with glass particles. In order to prevent
participants from taking a strategy other than grasping, weallowed participants to touch only the side areas and side
corners of the objects.
The object was glued to the top surface on a wood basewhich was tightly attached to the table. The wood base wasfor height adjustment so that the object was aligned withthe height of the subject’s index finger and thumb. Becauseof the glue, the object’s bottom surface did not rotate.

Task Design

The task design was almost identical, but the instructions
on touching the object were different from experiment 1.
There were three contact density conditions. For eachcontact density condition, participants touched 27 objects(3 shape conditions × 3 softness conditions × 3 times). In
aggregate, each participant touched 81 objects (27 objects
× 3 contact density conditions). The presentation order ofthe contact density condition was randomly assigned tobe counter-balanced across participants. The presentation
order of the shape and softness condition for each contact
density condition was also randomly assigned to be counterbalanced across participants.
Data analysis was performed as the same way as experiment 1.
5.3

Results

Fig.15 shows the average exploration time for each participant across all conditions without removed outlier samples.
We performed a single round of 3 σ clipping to remove
outliers as same as experiment 1. 12 outlier samples were
removed and those were the ones obtained from a certain
participant (ID:P).

Copyright (c) 2019 IEEE. Personal use is permitted. For any other purposes, permission must be obtained from the IEEE by emailing pubs-permissions@ieee.org.

This is the author's version of an article that has been published in this journal. Changes were made to this version by the publisher prior to publication.
The final version of record is available at
http://dx.doi.org/10.1109/TOH.2019.2954882
9[s]

JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2015***
* :p<0.05** :p<0.01

Fig. 15. The average exploration time for each participant across allconditions.

Fig.16 shows average exploration time for each softness
and contact density condition from a matrix viewpoint.

Fig. 17. The average exploration time across softness conditions withrespect to contact density condition.

**

exploration
time [s]Fig. 16. The average exploration time for each softness and contact
density condition.

The sphericity test s results indicated that the variancehomogeneity assumption was violated for both conditions
and interaction effect between them (p < 0.01). We performed a two-way repeated ANOVA with softness condition (s1, s2, s3) and contact density condition (d1, d3, d5)
factors on the exploration time. According to the ANOVA
results, there was a significant main effect of the softness condition (F (2, 465) = 4.88, p = 7.0 × 10−3 ) anda significant main effect of the contact density condition(F (2, 465) = 4.70, p = 9.5 × 10−3 ). There was no significant
interaction effect between the two factors (F (4, 465) =0.55, p = 0.69). Since there was no interaction effect, theeffect of the softness condition and contact density condition
were analyzed separately.
The black line in Fig.17 shows change in the average exploration time with standard error across softness
conditions. We conducted post-hoc test with bonferroni
adjustment for pair-wise comparison of 3 contact density
conditions. As a result, there were significant differences
between d1 and d3 (p < 0.05),and d1 and d5 (p < 0.01).
The black line in Fig.18 shows change in the average
exploration time across contact density conditions. We conducted post hoc test with bonferroni adjustment for pair-**:p<0.01

Fig. 18. The average exploration time across contact point density
conditions with respect to softness condition.

wise comparison of 3 softness conditions. There were significant difference between s1 and s3 (p < 0.01).
The probability of finding correct responses for the softness and contact density condition are shown in Fig.19. We
performed a two-way repeated ANOVA with softness condition (s1, s2, s3) and contact density condition (d1, d3, d5)
factors on the probability of correct responses, and there
was no significance on the main effect of both conditions
and interaction effect.
5.4

Discussion

The result of the ANOVA tests on the softness and contact
density conditions factors shows that these two factors hada significant impact on the exploration time. The gradationof the exploration time from Fig.16 and the post-hoc test’s
result in Fig.17 and Fig.18 supports that. These results
are identical to experiment 1. According to the post-hoc
multiple comparisons, differences in average exploration
time between d1 and d5 ,and d3 and d5 were significant.
There were comments from participants indicating that itwas easy to recognize the corner when the particle size wassmall. Looking at Fig.17, the average performance softness
condition s1 with denser density condition d1 was better
than hardest condition s3 with sparser condition d5. This

Copyright (c) 2019 IEEE. Personal use is permitted. For any other purposes, permission must be obtained from the IEEE by emailing pubs-permissions@ieee.org.

This is the author's version of an article that has been published in this journal. Changes were made to this version by the publisher prior to publication.
The final version of record is available at
http://dx.doi.org/10.1109/TOH.2019.2954882
JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 201510

6.2 Design Guidelines for Future Multi-contact-point
Haptic Display

Fig. 19. The probability of a correct answer for softness and contact
density condition.

means that denser contact point density has a possibility ofmaking up for the problem of the penetration.
Regarding discrimination difficulty, it can be said thatthe task in experiment 2 had higher difficulty in discrimination than that in experiment 1. The lower discrimination
accuracy and longer exploration time in experiment 2 support the difficulty of the task. According to Fig.19, it wasmore difficult for participants to discriminate. According to
Fig.15, it took longer to discriminate. In experiment 2, thedifference in exploration time between softness condition
s1 and s2 are noticeable, though that was not in experiment
1. Because the task in experiment 2 was more difficult
than experiment 1, the slight richness of kinesthetic cues
influences the exploration time between condition s1 ands2.

6

G ENERAL D ISCUSSION

6.1 Kinesthetic and Cutaneous Cues Integration
According to the results of the two series of experiments,
we could confirm that the contact density and the softnessof the object influenced the exploration time required forrecognition. Though the exploration procedure was different between experiment 1 and 2, a similar influence wasconfirmed. The results also suggested that increasing thecontact density facilitates shape recognition even when thefinger penetrates into the object.
The previous research [11], [22] investigated the mechanism of sensory integration by fitting the results obtained in
the experiment to sensory integration models such as Optimal Integration model and Sensory Capture model. However, from the viewpoint of the contact point density, it has
not been discussed on the sensory integration of kinesthetic
and cutaneous cues. If sensory capture model was adopted
in our experimental environment and the kinesthetic cue
dominated, it was speculated that exploration time would
not decrease even if contact point density increased. However, in fact, whatever the condition of softness, the increasein contact point density reduced the exploration time. Thus,
the kinesthetic cue did not dominate and the cutaneous cue
was also integrated and used.

Toward the future development of multi-point haptic display, our results suggested the importance of higher contact
density. Let us pay attention to the transition with regard
to the contact point density in Fig.9 and Fig.17. Lines of s3
condition in Fig.9 and Fig.17 show the effect of contact density when the more robust kinesthetic feedback is present.
Lines of s1 and s2 conditions show the effect of contact
density when there is little kinesthetic feedback. Fig.9 showsthat the s1 with d1 or d2 performed better than s3 with d5.
Similarly, Fig.17 shows that the s1 with d1 performed better
than s3 with d5. This indicates that the lacking of kinestheticcues could be compensated by the higher contact density.
Because of the cost and workspace issues with kinesthetic
devices, it would be nice to be able to make up for thelacking of kinesthetic cues with high contact density.
However, the exact density required for the multi-point
display is still unknown, and additional experiments will beneeded in the future. We plan to use particulate glove withhigher contact density than that used in this study to know
the minimum contact density sufficient to realize intuitive
shape recognition. In such an experiment, we are able toevaluate shape recognition beyond the maximum contact
density attainable using current multi-point displays, using
the same method as presented in this study to simulatecontact density and penetration. In addition, we also plan
to make participants to touch soft urethan object with barefinger and use the exploration time as a benchmark.

7

C ONCLUSION

This study proposed a method of preparing a real testing
environment where we can control the density of the contact point and the penetration of the finger in the object
independently. In order to control the density of the contact
point, we covered the actual user’s finger with particulate
gloves. In order to mimic the penetration, we used urethane
foam as a target of touch that deforms flexibly.
We conducted two series of experiments in which participants explored the object solely with the index finger
or grasped with the index finger and thumb. The result ofboth experiments showed that both the sparseness of thecontact point and the softness of the object had a significant
impact on the longer exploration time. We confirmed thatthe exploration time decreased as the contact point became
denser even when participants were exposed to the softest
urethane foam.
As future work, we will study the discriminative threshold on more specific tasks while simulating the sparsecontact density and the penetration.

R EFERENCES
[1]
[2]
[3]

R. L. Klatzky, S. J. Lederman, and V. A. Metzger, “Identifying
objects by touch: An “expert system”,” Perception & Psychophysics,
vol. 37, no. 4, pp. 299–302, 1985.
J. F. Norman, H. F. Norman, A. M. Clayton, J. Lianekhammy,and G. Zielke, “The visual and haptic perception of natural object
shape,” Perception & Psychophysics, vol. 66, no. 2, pp. 342–351, 2004.
Y. Shimizu, S. Saida, and H. Shimura, “Tactile pattern recognition
by graphic display: Importance of 3-d information for haptic
perception of familiar objects,” Perception & Psychophysics, vol. 53,
no. 1, pp. 43–48, 1993.

Copyright (c) 2019 IEEE. Personal use is permitted. For any other purposes, permission must be obtained from the IEEE by emailing pubs-permissions@ieee.org.

This is the author's version of an article that has been published in this journal. Changes were made to this version by the publisher prior to publication.
The final version of record is available at
http://dx.doi.org/10.1109/TOH.2019.2954882
JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2015

[4]

R. D. Howe, W. J. Peine, D. A. Kantarinis, and J. S. Son, “Remotepalpation technology,” IEEE Engineering in Medicine and Biology
Magazine, vol. 14, no. 3, pp. 318–323, 1995.
[5] I. M. Koo, K. Jung, J. C. Koo, J. Nam, Y. K. Lee, and H. R. Choi,“Development of soft-actuator-based wearable tactile display,”
IEEE Transactions on Robotics, vol. 24, no. 3, pp. 549–558, 2008.
[6] S. Kim, C. Kim, G. Yang, T. Yang, B. Han, S. Kang, and D. Kwon,“Small and lightweight tactile display(salt) and its application,” In
Proceedings of IEEE World Haptics Conference 2009, pp. 69–74, 2009.
[7] K. Tanabe, S. Takei, and H. Kajimoto, “The whole hand haptic
glove using numerous linear resonant actuators,” In Proceedings ofIEEE World Haptics Conference, 2015.
[8] T. Taniguchi, S. Sakurai, T. Nojima, and K. Hirota, “Multi-point
pressure sensation display using pneumatic actuators,” In Proceedings of EuroHaptics Conference 2018, pp. 58–67, 2018.
[9] S. J. Lederman and R. L. Klatzky, “Hand movements: A window
into haptic object recognition,” Cognitive Psychology, vol. 19, no. 3,
pp. 342 – 368, 1987.
[10] S. J. Lederman and R. L. Klatzky, “Haptic perception: A tutorial,”
Attention, Perception, and Psychophysics, vol. 71, no. 7, pp. 1439–
1459, 2009.
[11] A. Frisoli, M. Solazzi, M. Reiner, and M. Bergamasco, “The contribution of cutaneous and kinesthetic sensory modalities in haptic
perception of orientation,” Brain Research Bulletin, vol. 85, no. 5,
pp. 260 – 266, 2011.
[12] J. Voisin, Y. Lamarre, and C. E. Chapman, “Haptic discriminationof object shape in humans: contribution of cutaneous and proprioceptive inputs,” Experimental Brain Research, vol. 145, no. 2, pp.
251–260, 2002.
[13] Z. F. Quek, S. B. Schorr, I. Nisky, A. M. Okamura, and W. R.
Provancher, “Augmentation of stiffness perception with a 1degree-of-freedom skin stretch device,” IEEE Transactions on
Human-Machine Systems, vol. 44, no. 6, pp. 731–742, 2014.
[14] Z. F. Quek, S. B. Schorr, I. Nisky, W. R. Provancher, and A. M. Okamura, “Sensory substitution and augmentation using 3-degree-offreedom skin deformation feedback,” IEEE Transactions on Haptics,
vol. 8, no. 2, pp. 209–221, 2015.
[15] A. Frisoli, M. Solazzi, F. Salsedo, and M. Bergamasco, “A fingertip
haptic display for improving curvature discrimination,” Presence,
vol. 17, no. 6, pp. 550–561, 2008.
[16] F. Chinello, M. Malvezzi, C. Pacchierotti, and D. Prattichizzo, “A
three dofs wearable tactile display for exploration and manipulation of virtual objects,” In Proceedings of IEEE Haptics Symposium,
pp. 71–76, 2012.
[17] M. A. Srinivasan and R. H. LaMotte, “Tactual discrimination ofsoftness: abilities and mechanisms,” Somesthesis and the Neurobiology of the Somatosensory Cortex, pp. 123–135, 1996.
[18] W. M. Bergmann Tiest and A. M. L. Kappers, “Cues for haptic
perception of compliance,” IEEE Transactions on Haptics, vol. 2,
no. 4, pp. 189–199, 2009.
[19] A. Metzger and K. Drewing, “Haptically perceived softness ofdeformable stimuli can be manipulated by applying external
forces during the exploration,” In Proceedings of IEEE World Haptics
Conference, pp. 75–81, 2015.
[20] M. O. Ernst and M. S. Banks, “Humans integrate visual and haptic
information in a statistically optimal fashion,” Nature, vol. 415, no.
6870, p. 429, 2002.
[21] M. Kuschel, M. Di Luca, M. Buss, and R. L. Klatzky, “Combinationand integration in the perception of visual-haptic complianceinformation,” IEEE Transactions on Haptics, vol. 3, no. 4, pp. 234–
244, 2010.
[22] F. E. van Beek, R. J. King, C. Brown, and M. D. Luca, “Thecontributions of skin stretch and kinesthetic information to static
weight perception,” In Proceedings of IEEE World Haptics Conference,
WHC 2019, pp. 235–240, 2019.
[23] I. Sarakoglou, N. Tsagarakis, and D. G. Caldwell, “A portablefingertip tactile feedback array - transmission system reliability
and modelling,” In Proceedings of First Joint Eurohaptics Conference
and Symposium on Haptic Interfaces for Virtual Environment andTeleoperator Systems. World Haptics Conference, pp. 547–548, 2005.
[24] D. G. Caldwell, N. Tsagarakis, and C. Giesler, “An integrated
tactile/shear feedback array for stimulation of finger mechanoreceptor,” In Proceedings of IEEE International Conference on Robotics
and Automation, vol. 1, pp. 287–292 vol.1, 1999.
[25] Y. Kim, S. Kim, T. Ha, I. Oakley, W. Woo, and J. Ryu, “Air-jet
button effects in ar,” In Proceedings of Advances in Artificial Reality
and Tele-Existence, pp. 384–391, 2006.

11[26] M. Shimojo, M. Shinohara, and Y. Fukui, “Human shape recognition performance for 3d tactile display,” IEEE Transactions on
Systems, Man, and Cybernetics - Part A: Systems and Humans, vol. 29,
no. 6, pp. 637–644, 1999.
[27] M. Nakatani, H. Kajimoto, N. Kawakami, and S. Tachi, “Tactile
sensation with high-density pin-matrix,” In Proceedings of the 2nd
Symposium on Applied Perception in Graphics and Visualization, pp.
169–169, 2005.
[28] M. Nakatani, N. Kawakami, and S. Tachi, “How human can
discriminate between convex and concave shape from the tactile
stimulus,” In Proceedings of the Annual Meeting of the Cognitive
Science Society, vol. 29, no. 29, 2007.
[29] N. Garcia-Hernandez, N. G. Tsagarakis, and D. G. Caldwell,“Feeling through tactile displays: A study on the effect of the arraydensity and size on the discrimination of tactile patterns,” IEEE
Transactions on Haptics, vol. 4, no. 2, pp. 100–110, 2011.
[30] M. Natsume, Y. Tanaka, and A. M. L. Kappers, “Individual differences in cognitive processing for roughness rating of fine andcoarse textures,” PLOS ONE, vol. 14, no. 1, pp. 1–16, 2019.
[31] H. Tsuboi, M. Inoue, S. Kuroki, H. Mochiyama, and J. Watanabe,“Roughness perception of micro-particulate plate: A study on twosize-mixed stimuli,” In Proceedings of EuroHaptics Conference, pp.
446–452, 2014.

Yusuke Ujitoko is a researcher at Hitachi, Ltd.,
Japan and is also a PhD student at the University
of Electro-Communications from 2019. He received a B.E. degree in mechanical engineeringand M.A.E. degree in inter-disciplinary information studies from the University of Tokyo, Japan,
in 2014 and 2016. His research interests include
applied haptic perception and haptic interfaces.
He is a member of the IEEE.

Sho Sakurai received a B.E. degree in social
and information studies from the Gunma University in 2007, a M.A.E. degree in inter-disciplinary
information studies in 2010 and a PhD degree
in Engineering in 2014 from the University ofTokyo. Currently, she is a project assistant professor in Graduate School of Information Systems, the University of Electro-Communications.
Her research interests are multi-modal/crossmodal interfaces and human-computer interaction. She is also active as a manga artist for introduction of the latest research of virtual reality and artificial intelligence.

Koichi Hirota received a BS degree from the
University of Tokyo, Japan, in 1988. He received
a PhD degree from the University of Tokyo, in
1994. He was then an assistant professor with
the Toyohashi University of Technology in 1995.
In 2000, he was an associate professor with the
University of Tokyo. Currently, he is a professor
in the Department of Human Media Systems,
Graduate School of Information Systems, University of Electro-Communications. His researchinterests include haptic rendering and humaninterfaces. He received several academic awards.

Copyright (c) 2019 IEEE. Personal use is permitted. For any other purposes, permission must be obtained from the IEEE by emailing pubs-permissions@ieee.org.