Atten Percept Psychophys
DOI 10.3758/s13414-013-0588-9

This is the author's version. The final publication
is available at link.springer.com.

VR-MDS: Multidimensional scaling for classification tasksof virtual and real stimuli
S. Okamoto# Psychonomic Society, Inc. 2014

Abstract Evaluating the perceptual similarity between virtualand real sensory stimuli has been a serious problem for virtual
reality interface researchers for a long time. One of the mostcommonly used evaluation methods is a classification task
where assessors classify randomly presented stimuli into multiple candidate types. The results of this method are summarized using two types of confusion matrices, which havedifferent stimulus sets. The present study developed a methodthat computes the locations of simulated and real stimuli in aperceptual space on the basis of the two confusion matrices.
The spatial distribution of the stimuli allows us to visuallyinterpret the perceptual relationships between stimuli and theirperceptual dimensionality. This method is recommended
when the guidance index based on the answer ratios of theconfusion matrices is fairly high.
Keywords Math modeling . Similarity . Statistics
The universal aim of virtual reality technology is to design
simulated stimuli that resemble real stimuli. The perceptual
quality of the stimuli is a primary concern for researchers andusers of simulators. For example, considerable effort has been
devoted to representing the qualities of real tumors in medical
simulators used to train physicians during palpations (Jeon,Choi, & Harders, 2012; Karadogan & Williams, 2013;
Ottermo, 2006). Neuroscientists and psychologists have usedgraphic simulators to obtain pictures of objects under various
lighting and material conditions (Motoyoshi & Matoba, 2012;
Nishio, Goda, & Komatsu, 2012). Several studies have investigated the perceptual qualities of stimuli in other types
S. Okamoto (*)
Department of Mechanical Science & Engineering,
Nagoya University, Nagoya, Japane-mail: okamoto-shogo@mech.nagoya-u.ac.jp

of simulators—for example, sounds (Giordano et al., 2012),
3-D pictures (Mai, Doutre, Nasiopoulos, & Ward, 2012), andhaptic stimuli (Okamura, Cutkosky, & Dennerlein, 2001).
The methodologies used to assess the perceptual propertiesof these stimuli are of crucial importance. Various methods
have been used for this purpose, such as comparing thesimilarities of physical quantities in virtual and real
stimuli—for example, force or acceleration (Kuchenbecker& Niemeyer, 2006; Okamura, Webster, Nolin, Johnson, &Jafry, 2003). However, physical similarities are not sufficient,and researchers often make use of evaluations based on
perceptual similarity.
In some cases, assessors have evaluated the degree of thesubjective quality of sensory feedback using numerical or
graded scales (Dev, Harris, Gutierrez, Shah, & Senger, 2002;
Hikichi, Yasuda, Fukuda, & Sezaki, 2006). Trained assessors
rate the integrated similarity between the percepts of real andvirtual stimuli. Alternatively, they may experience only thevirtual stimuli and rate them. The latter approach is used widelyfor audiovisual content (International Telecommunication
Union, 2009).
In other cases, the similarities between virtual and real
stimuli are evaluated using specific perceptual criteria (Fujita& Ohmori, 2001; Konyo, Tadokoro, & Takamori, 2000;
Watanabe & Fukui, 1995). For example, the assessors may
compare real and virtual tumors only in terms of their compliances or locations beneath tissues.
For virtual reality simulators used for manufacturing or
medical tasks, as well as discussions of the quality of specificstimuli such as tumors or reaction forces delivered via a tool
grip, we need to consider the comprehensive qualities of thetasks experienced by trainees during trials. Questionnaires or
behavioral indices are preferred in these cases. Questionnairebased reality evaluations (e.g., Hendrix & Barfield, 1996; vanBaren & IJsselstejin, 2004; Witmer & Singer, 1998) typicallyAtten Percept Psychophys

involve questions related to the naturalness of the environment, the difficulties of tasks, and so forth.
Behavioral approaches use task performance indices suchas the time required to finish a given task or the accuracy oftask performance (Fukumoto & Sugimura, 2001; Gomoll,
Pappas, Forsythe, & Warner, 2008; Howe, Peine, Kontarinis,& Son, 1995; Kontarinis & Howe, 1995; Pedowitz, Esch, &Snyder, 2002). These indices can be compared during real andinterface-mediated conditions. In general, the values of thesebehavioral indices should be similar in the real and simulated
environments.
All of these methods, including the classification tasks addressed in the present article, are used to assess different aspects
of virtual reality systems. The roles of these assessment techniques for virtual stimuli and simulators are expected to grow inboth academic and industrial settings. This is because perceptual psychologists have been adopting simulators throughwhich highly controlled stimuli are easily produced for theirstudies. In addition, medical virtual reality simulators, such as
surgical and dental simulators, have been emerging, and detailed assessment of their perceptual quality is required.

Classification tasks
When virtual reality simulator researchers want to assess how
well each virtual stimulus is designed and how effectively it
perceptually resembles the corresponding real stimulus, theyoften select classification tasks because of their simplicity andobjectivity. During these tasks, an assessor selects a real
stimulus that is perceptually most similar to a randomly presented virtual or real stimulus from n candidates (n-alternative
forced choice). The quality of the virtual stimulus is thenevaluated by the correct response ratio in a test where thecorresponding virtual and real stimuli are coupled as answers.
Such classification tasks have been used for artificial or simulated stimuli, including sounds caused by physical impacts(Giordano et al., 2012), diagnostic images (Yasnoff, Mui, &Bacus, 1977), speeches (Holyoak & Glass, 1978), and tactile
textures (Yamamoto, Nagasawa, Yamamoto, & Higuchi,2006).
Two types of experiments are used widely to evaluate
stimuli simulators during classification tasks (e.g., Germani,
Mengoni, & Peruzzini, 2013; Okamoto, Ishikawa, Nagano, &Yamada, 2013; Wiertlewski, Lozada, & Hayward, 2011). The
first type comprises virtual-to-real classification tasks whereassessors select the real stimulus that feels most similar to thevirtual stimulus presented. The second type is a real-to-realclassification task, where assessors select a real stimulus thatfeels most similar to a given real stimulus.
Greater similarity in the results of the two tasks indicatesthat the simulators are better at representing real stimuli. For
example, Wiertlewski et al. (2011) designed virtual surfaced

materials using a tactile display. They also prepared real ones
and performed a comparison of the two types of classification
tasks to evaluate how well the virtual materials replicated thereal ones. In this comparison, the correct answer ratios orinformation criteria were calculated from the results of two
tasks and compared (Chen, Park, Dai, & Tan, 2011).

Confusion matrices and multidimensional scaling
methods (MDS)
Confusion matrices are used to display the results of classification tasks. Each cell in the matrix indicates a ratio or
frequency, which shows how often a specific stimulus isselected when another stimulus is presented to participants.
The interpretation of confusion matrices is not always simple,
which means that only the correct answer ratios might bediscussed. However, the incorrect answer ratios also include
important information. Thus, the correct and incorrect answer
ratios need to be considered collectively. To address this issue,
many studies have used a multidimensional scaling (MDS)
approach to visually interpret matrices (e.g., Giordano et al.,2012; Luk et al., 2006; Pols, 1983). This approach is a powerful tool for analyzing the capacities of human perception or
sensory displays by representing the perceptual dissimilarities
between stimuli based on the geometric distances in a multidimensional perceptual space. MDS can be applied to a variety of tasks, although it is computed on the basis of the resultsof classification tasks in the present study. For example,
perceptual closeness between two stimuli is specified using
numbers or analog scales in similarity rating tasks (Cooke,
Jäkel, Wallraven, & Bülthoff, 2007). In clustering or sortingtasks, assessors classify a multitude of stimuli into several
groups on the basis of their perceptual similarity (Bergmann
Tiest & Kappers, 2006; Hollins, Faldowski, Rao, & Young,1993). Dissimilarity indices are computed from these tasks,
before MDS is applied to them.
Unfortunately, existing MDS methods cannot be applied
directly to virtual reality stimuli. As was described above, twotypes of classification tasks are conducted during assessmentsof virtual reality systems, which yield two types of confusion
matrices. The stimulus sets of both matrices are not the same.
The matrix obtained from a virtual-to-real classification taskcomprises both real and virtual stimuli, whereas the matrix
from a real-to-real classification task comprises real stimuli
only. Existing MDS methods cannot cope with this problem.
MDS methods have been developed continually so they candeal with a wider class of problems, although, the uniquenessof the problem of virtual reality interfaces has yet to beconsidered. Some MDS algorithms combine multiple confusion matrices (e.g., Carroll & Chang, 1970; Takane, Young, &De Leeuw, 1977; Tucker & Messick, 1963; Young & Null,1978); however, these algorithms are not considered to beAtten Percept Psychophys

capable of dealing with two confusion matrices that containdifferent stimulus sets. For example, these methods are used to
conduct MDS of multiple matrices obtained from different
assessors using the same stimulus sets. However, our problem
belongs to a different class, as compared with these algorithms.

Objective
The objective of this study is to integrate the two types ofconfusion matrices that are often used to evaluate virtual
reality simulators and to compute the locations of real andvirtual stimuli in a multidimensional space. This method could
become a general evaluation method for virtual reality interfaces, which may facilitate a visual interpretation of the perceptual dissimilarities between simulated and real stimuli, andthe specification of the problems involved with such stimuli.

Technical issues
The classical MDS method requires the distances between all
stimulus pairs. Some later methods are applicable to matrices
with missing values (e.g., Kruskal, 1964a, 1964b), but in thepresent problem where the two matrices comprise different
stimulus sets, the computational accuracy is reduced by a large
number of missing data. Thus, it is desirable that all of the
distances are available, as shown in Fig. 1. Let D sisj be theperceptual distance between the stimuli s i and s j. The s is r orv, which represents real and virtual stimuli, respectively. Thev i corresponds to the virtual stimulus that mimics the real
stimulus r i . The computation methods that have been mentioned thus far are applicable only to D rirj , which is theperceptual distance between real stimuli. D rirj can be acquired
using a similarity choice model (Luce, 1963, 2005), an overlap model (Townsend, 1971; Townsend & Landon, 1982), or
the constant ratio rule (Clarke, 1957; Clarke & Anderson,
1957), which is used in the present study. These methods
successfully compute the interstimulus distances using complete matrices. However, they do not consider the computation

Fig. 1 Distance matrix of real and virtual stimuli (three virtual and real
stimuli)of D rivj and D vivj . Thus, two technical issues are related to thecomputation of the matrix shown in Fig. 1.
The first issue is the computation of D vivj , which is theperceptual distance between two virtual stimuli. Assessors do
not compare pairs of virtual stimuli directly, whereas they
compare pairs of real stimuli and pairs of virtual and real
stimuli during real-to-real and virtual-to-real classification
tasks, respectively. The two confusion matrices obtained fromthese two comparisons do not provide information that is
related directly to the distances between virtual stimuli. This
study shows that D vivj can be computed by letting the real
stimuli serve as intermediates for the virtual stimuli.
The second issue is how to retain the consistency between
two classification tasks that are performed independently. In
order to achieve this consistency, this study computes D rivj ,which is the perceptual distance between the real and virtual
stimuli, by combining the results of the two tasks.
This article presents solutions for the two issues above, forwhich the basic computational methods were presented in(Okamoto & Yamada, 2012), some examples of the method,
and the effectiveness of the computational methods based on
validations.

Multidimensional scaling of real and virtual stimuli
To construct the multidimensional space, we compute theperceptual distances between all stimuli. The computation
methods vary depending on the pairs of stimuli. A metric
MDS method is then applied to the computed distances. The
stimuli are located in a multidimensional space, where thenumber of dimensions is determined using fit measures sotheir distances are maintained well. However, the higher dimensional spaces tend to lose visual clarity.
Model of internal responses toward multiple stimuli basedon signal detection theory
According to multidimensional detection theory (e.g.,
Ashby & Townsend, 1986; Nakatani, 1972), the classification of a stimulus among n candidates is modeled as theclassification of stimuli that are probabilistically distributedin multiple perceptual dimensions. The human internal
response to a physical stimulus is expressed as a normal
distribution, as shown in Fig. 2. The internal response to aphysical stimulus s i follows a joint probability density
function that contains multiple variables (two variables in
the figure), where the center of the distribution S i is acoordinate in perceptual space. These variables are mutually independent, and they construct the perceptual space ofstimuli. We assume an equal variance of distributions,which is commonly accepted. The unit of each dimensionalvariable is set as equal to this variance.

Atten Percept Psychophys

Fig. 2 Internal responses to multiple stimuli in multiple perceptual dimensions and the application of the constant ratio rule

In this model, the stimulus is classified or recognized as s iwhen the response toward s i appears closer to S i than to S j . If
the response is somewhat closer to S j , the stimulus is recognized as s j . The border of this judgment is referred to as
criterion λ ij .
To simplify the handling of stimuli in multidimensions, we
use the constant ratio rule (Clarke, 1957; Clarke & Anderson,
1957), which is a strict form of similarity choice model(Townsend & Landon, 1982). When we apply this rule totwo arbitrary stimuli, s i and s j , their probabilistic and perceptual relationships are found on the line S i -S j as shown in
Fig. 2 (upper left). S i and S j are the projections of S i and S j
on the line, respectively. λij is the intersection of the line
S i -S j with λ ij . We estimate the perceptual distance between
two stimuli on this line as Dsisj ¼ S j −S i .

 2
Z z
1
−x
pﬃﬃﬃﬃﬃﬃ exp
dx
p ¼ g ðzÞ ¼
2
2π
−∞

ð2Þ

Z ðpÞ ¼ g−1 ðpÞ:ð3ÞSimilarly, R j −λ ij is given byPrjrj
R j −λij ¼ Z
Prjrj þ Prjri

On the basis of these equations, the perceptual distance
between two real stimuli is determined byDrirj ¼ R j − R i 
 



ð5Þ


Prjrj
Priri
:
¼ Z
þZ

Priri þ Prirj
Prjrj þ Prjri

Distance between real stimuli
The perceptual distances between real stimuli can be computed simply using the constant ratio rule. Figure 3 shows theinternal responses to a real stimulus r i on the line R i -R j . On
this line,


Priri
λij − Ri ¼ Z
ð1ÞPriri þ Prirj

holds, where Z (p) is the z -score of probability p. Z (p ) is
described byð4Þ

Fig. 3 Internal response to ri on the line R i −R j

Atten Percept Psychophys

Distance between real and virtual stimuli
In a virtual-to-real classification task where the assessorsclassify virtual stimuli as real stimuli, the internal responses
to stimuli are expressed as shown in Fig. 4. When virtual
stimulus v i is presented and its internal response is closer to
R i , the stimulus is classified as r i . However, it is classified as
r j if the response is closer to R j . The border of this judgment
is described as λ ij . The internal response to v i on the line thatpasses V i and the center of R i and R j is expressed as shown in
Fig. 5. λij −V i is described by 
λij − V i ¼ Z



Pviri
:
Pviri þ Pvirj Distance between virtual stimuli
ð6Þ

On the line,


1
−1

1
1

"

Ri − V i
Rj −Vi#" #
2 λ ij − V i
¼
Drirj

ð7Þ

holds, given that λij is placed at the center of Ri and R j ,which implies unbiased assessors. It is also assumed that V i is
closer to R i than to R j , for which a strong violation should be
avoided, as described later in the Example of Poorly Identified
Stimulus Set section. The distances between virtual and real
stimuli are given by
3
2 



−
V
R


i
i
Dviri
 5:
ð8Þ
¼ 4 
Dvirj
R − V jFig. 5 Internal response to v i on the line V i −R i −R jiThese distances are obtained by solving Equation 7.
D rirj is determined as described above using the resultsof the real-to-real classification task. Thus, D viri andD virj are computed using the results of both classification tasks, which combines the two classification tasks
and maintains their consistency.

Finally, we compute the distances between virtual stimuli. The
assessors do not compare virtual stimuli directly in classification tasks. However, the distances between virtual stimuli canbe computed using the judgment criterion for distinguishing
two real stimuli. Figure 6 shows the internal responses towardv i and v j , R i and R j , and their judgment criterion. We consider the space on a line that passes through V i and V j .
Figure 7 shows the responses toward v i and v j and the
projections of R i and R j on this line. Here,


Pviri
λij − V i ¼ Z
ð9ÞPviri þ Pvirj
holds. Similarly,Pvjrj
V j − λij ¼ Z
Pvjrj þ Pvjri

ð10Þ

holds. The distances between two virtual stimuli are computed
using these equations as follows:Dvivj ¼ V j − V i 
 

Pviri
¼ Z
P þP
viri

Fig. 4 Internal response to vi , ri , and rj . The probability densities are notshown to improve the visual clarity




virj



Pvjrj
:
þZ
Pvjrj þ Pvjri Fig. 6 Internal responses to v i and v jð11ÞAtten Percept Psychophys
Table 2 Example 1 (shrunk): Confusion matrix for virtual-to-realclassification
AnsweredPresent

Fig. 7 Internal responses to vi and vj on the line V i −V j

Multidimensional scaling method (MDS)
We use a MDS method to compute the distributions of stimuli
in perceptual space using their perceptual distances. This method determines the coordinates of events (stimuli) in a space
while maintaining the given distances between them. The number of dimensions can be set arbitrarily, although larger dimensions generally represent the distances better. Given that visualinterpretation is the point of MDS methods, two or three
dimensions are preferable. In the present study, we use themetric MDS method developed by Torgerson (1952). In general, metric MDSs with complete data provide precise results.

ExamplesSeveral examples of the confusion matrix and the computed
spatial distributions of stimuli are introduced. First, examples
of a case involving three stimuli are presented. Table 1 shows
the confusion matrix for the real-to-real classification task.
This table was used to compute the stimulus distributions forvarious matrices during the virtual-to-real classification task.
In the following examples, the stimuli are correctly classified
in 84% of cases (2⋅Z(0.84) = 2.0) when the distance between
the two stimuli is 2.

r1

r2

r3

v1

0.7

0.15

0.15

v2
v3

0.15
0.15

0.7
0.15

0.15
0.7

task. The incorrect answer ratios are balanced. Figure 8 shows
the distributions of stimuli computed from Tables 1 and 2. The
distributions of virtual and real stimuli are geometrically similar
because of the balanced incorrect answer ratios. However, thedistribution of virtual stimuli is smaller than that of real stimuli
because of the lower correct answer ratios in the virtual-to-realclassification task.
Example 2 (distorted space)
As is shown in Table 3, we decreased P v2r2 and P v3r3 , while
keeping P v1r1 at the same level as P r1r1 . Thus, we increased
P v2r3 and P v3r2 . In this case, V 1 should be located at the same
coordinates as R 1 because P r1ri = P v1ri (i = 2, 3), whichmeans that v 1 is the same as r 1 in terms of their relationships
with other real stimuli. Furthermore, V 2 and V 3 should approach R 3 and R 2, respectively, because v 2 and v 3 are more
likely to be confused than r 2 and r 3. Figure 9 was computedfrom Tables 1 and 3 and shows the distributions of these stimuli.
Example 3 (expanded space)
As is shown in Table 4, we increased the correct response
ratios of the virtual-to-real classification task, while the incorrect answer ratios were balanced. Figure 10 shows the

Example 1 (shrunk space)
We computed the stimuli distribution where the confusion
matrix of the virtual-to-real task is that shown in Table 2 andthe correct response ratios are lower than those for the real-to-realTable 1 Real-to-real classification task: Confusion matrix (Examples 1–5)
AnsweredPresent

r1
r2
r3

r1

r2

r3

0.8
0.1
0.1

0.1
0.8
0.1

0.1
0.1
0.8

Fig. 8 Example 1 (shrunk): Stimuli distributions

Atten Percept Psychophys
Table 3 Example 2 (distorted): Confusion matrix for virtual-to-realclassification

Table 4 Example 3 (expanded): Confusion matrix for virtual-to-realclassification

AnsweredPresent

Answered

r1

r2

r3

v1

0.8

0.1

0.1

v2
v3

0.1
0.1

0.7
0.2

0.2
0.7

distributions of the stimuli computed from Tables 1 and 4. In
this case, the distributions of virtual stimuli appear to be
expanded, as compared with the real stimuli, whereas thepolygons of the virtual and real stimuli are similar. This may
occur when the features of the real stimuli are well represented
or even emphasized by the virtual simulators.
Example 4 (same matrix)
When the confusion matrices of both classification tasks are
identical, the locations of virtual and real stimuli should also
be the same in a perceptual space. Figure 11 shows theperceptual space computed in this condition using Tables 1
and 5. The coordinates of V i and R i match.

Present

r1

r2

r3

v1

0.9

0.05

0.05

v2
v3

0.05
0.05

0.9
0.05

0.05
0.9

totally confused, as shown in Fig. 12, when they are computedfrom Tables 1 and 6.
Example 6 (shrunk, four stimuli)
The four-stimuli case is addressed here. We use the matrix
shown in Table 7 as the confusion matrix of the real-to-realclassification task. As with Example 1, we decreased thecorrect response ratios while balancing the incorrect answer
ratios, as shown in Table 8. Figure 13 shows the perceptual
space computed from Tables 7 and 8. In this case, a threedimensional space describes the perceptual distances among
stimuli better than does a two-dimensional space. The virtual
stimuli are distributed inside the real stimuli, with profiles
similar to those of the real stimuli.

Example 5 (confused case)Example 7 (distorted, four stimuli)
In the examples above, the diagonal elements of the confusion
matrix used for the virtual-to-real classification task are larger
than the others. In other words, v i is classified most frequently
as r i . However, such an ideal case does not always occur inactual experiments. For example, as is shown in Table 6, v 2
and v 3 can be classified wrongly as r 3 and r 2 , respectively. In
this case, V 2 should be closer to R 3 rather than to R 2. In
addition, V 3 should be located near R 2. Stimuli 2 and 3 are

Finally, we computed the case where the confusion matrix ofthe virtual-to-real classification task (Table 9) was very different from that of the real-to-real classification task (Table 7).
Figure 14 shows the perceptual space computed from Tables 7
and 9. Because P v1r2 and P v2r1 are high in Table 9, V 1 and V 2
tend to be confused, and they should be placed nearby. For thesame reason, the locations of V 3 and V 4 should be close when

Fig. 9 Example 2 (distorted): Stimuli distributions

Fig. 10 Example 3 (expanded): Stimuli distributions

Atten Percept Psychophys
Table 6 Example 5 (confused): Confusion matrix for virtual-to-realclassification
AnsweredPresent

Fig. 11 Example 4 (same): Stimuli distributions

compared with the distance between R 3 and R 4. However,P v1r3 and P v1r4 in Table 9 have a value of 0.05, which is lessthan P r1r3 and P r1r4 in Table 7 (value of 0.07). This meansthat v 1 is less confused with r 3 and r 4. Thus, V 1 should belocated slightly farther from R 3 and R 4. For the same reason,
V 2 should be distant from R 3 and R 4. V 3 and V 4 should also
be distant from R 1 and R 2.
These examples show that our method facilitates the visualinterpretation of the perceptual relationships between real andvirtual stimuli. For greater clarity, we used cases involving asmall number of stimuli, but the geometrical description ismore beneficial as the number of stimuli increases, and theconfusion matrices are more complex. Identifying the perceptual relationships among such stimulus sets solely on the basisof the large matrices is not a simple task.

Example: Application on a footstep simulator

r1

r2

r3

v1

0.8

0.1

0.1

v2
v3

0.1
0.1

0.4
0.5

0.5
0.4

delivered acceleration stimuli to their soles. The accelerations
were synthesized on the basis of those observed when fragile
structures were crushed. The structures were cylinders
consisting of copy paper, aluminum foil, polypropylene(PP), or typing paper.
In a classification task, visually and aurally blocked assessors experienced the real or simulated crushing of cylinders.
Fifteen assessors performed both real-to-real and virtual-toreal classification tasks. Tables 10 and 11 show the averageanswer ratios of the tasks. If we look at only the correct answer
ratios—that is, the diagonal elements of Table 11—we could
conclude that the crushing of copy paper and PP were wellsimulated because their ratios were high, whereas aluminumand typing paper were poorly simulated owing to their low
ratios.
Figure 16 shows the perceptual space obtained by applying
the developed method to the two confusion matrices—that is,Tables 10 and 11. Circles and diamonds express real andvirtual materials, respectively. The configured space indicatesthat the virtual PP closely presented real PP because they wereclosely located in the perceptual space. On the other hand, thevirtual copy paper should be improved because it was distantfrom real copy paper. Nonetheless, virtual copy paper was
easily classified because it was located far from the othermaterials.

An example of virtual reality system analysis is presented
here, using data originally published in Okamoto et al.
(2013). The system is a footstep display through which users
experience the sense of crushing fragile structures with theirfootsteps. As is shown in Fig. 15, the users stepped on an
acrylic plate that was vibrated by voice coil actuators and that

Table 5 Example 4 (same): Confusion matrix for virtual-to-realclassification
AnsweredPresent

v1
v2
v3

r1

r2

r3

0.8
0.1
0.1

0.1
0.8
0.1

0.1
0.1
0.8

Fig. 12 Example 5 (confused): Stimuli distributions

Atten Percept Psychophys
Table 7 Real-to-real classification task: Confusion matrix (Examples 6–7)
AnsweredPresent

r1
r2
r3
r4

r1

r2

r3

r4

0.8
0.07
0.07
0.07

0.07
0.8
0.07
0.07

0.07
0.07
0.8
0.07

0.07
0.07
0.07
0.8

Real and virtual aluminum and typing paper were positioned nearby. Hence, the two materials were frequently confused, as is shown in Tables 10 and 11. In particular, virtual
typing paper tended to be classified as aluminum rather thantyping paper. In the perceptual space, the assessors’ answers
were graphically expressed as virtual typing paper being closer to real aluminum than to real typing paper.
As was described above, the perceptual space graphicallyaids our understanding of confusion matrices. Moreover, aninterpretation of the perceptual dimensions fosters this analysis. For example, from the subjective reports of the assessors,
it was speculated that the horizontal dimension corresponded
to the perceived strength of acceleration or crush stimuli. The
rightward stimuli presented stronger accelerations. This suggested that the magnitude of virtual copy paper stimuli should
be increased such that the position of the virtual copy paper is
moved to the right, becoming more perceptually similar to its
real crushing action. On the other hand, from the assessors’
reports, the vertical dimension likely expresses the quality ofthe perceived crushing action. The stimuli positioned upwardin the space were perceived as the crushing of wet materials,
whereas the downward stimuli represented crushing driermaterial. The profiles of frequency spectra for acceleration
stimuli are linked with such wet and dry sensations (Giordano
et al., 2012). These analyses that use multidimensional perceptual space illustrate how the virtual crushing actions should
be tuned to develop a better understanding of the system and,hence, a better simulator. The combination of the developed
MDS method with systematic approaches for interpreting
perceptual dimensions (e.g., as described in Hollins,

The computational methods and the internal response models
contain various assumptions. These assumptions include
equal variances of distributions, unbiased assessors, and the
independence of the variables that form the perceptual space.
These are naturally dependent on specific virtual reality interfaces and stimulus sets and should be considered carefully foreach experimental task on a case-by-case basis.
Thus, we focus on the validation of the computational
schemes related specifically to our method—that is, the application of the constant ratio rule to the computation of D rivj andD vivj . In this section, we test these computations collectively
using a number of randomly generated samples. Generalvalidations of the constant ratio rule for the computation of
D rirj have been described previously (Clarke, 1957; Clarke &Table 8 Example 6 (4 stimuli, shrank): Confusion matrix for virtual-toreal classification

Table 9 Example 7 (4 stimuli, distorted): Confusion matrix for virtualto-real classification

Fig. 13 Example 6 (4 stimuli, shrunk): Stimuli distributions

Bensmaïa, Karlof, & Young, 2000; Picard, Dacremont,
Valentin, & Giboreau, 2003; Yoshioka & Zhou, 2009) leads
to more consolidated analyses.

Validation of the method

AnsweredPresent

v1
v2
v3
v4

Answered

r1

r2

r3

r4

0.7
0.1
0.1
0.1

0.1
0.7
0.1
0.1

0.1
0.1
0.7
0.1

0.1
0.1
0.1
0.7

Present

v1
v2
v3
v4

r1

r2

r3

r4

0.5
0.4
0.05
0.05

0.4
0.5
0.05
0.05

0.05
0.05
0.6
0.3

0.05
0.05
0.3
0.6

Atten Percept Psychophys

regenerated stimuli spaces were compared using a distancebased index.
Generation of the random reference stimuli space
In a two-dimensional space, three R i (i =1, 2, 3) are randomly
located that are the centers of the internal responses to r i . The
coordinates of each stimulus were constrained so theinterstimuli distances did not exceed 2.5. Thus, the distributions of the stimuli did not expand infinitely. If two stimuli are
2.5 apart, they can be correctly identified at approximately
90% (2⋅Z (0.9) = 2.5). A uniform probability density function
was used to produce the random coordinates. V i was randomly placed near R i within a distance of 0.5.
Production of the confusion matrices
Fig. 14 Example 7 (4 stimuli, distorted): Stimuli distributions

Anderson, 1957; Hodge, 1967; Hodge & Pollack, 1962;
Townsend & Landon, 1982).
Validation methods

The two types of confusion matrices were computed from thereference stimuli distributions.
The probability P sisj of s i being classified as s j is determined byZPsisj ¼ P s j si ¼
f ðSi ÞdS
ð12Þ
Aj

Overview of evaluation procedures
Three, four, and five stimuli cases are validated. Here, theprocedures are described using the three-stimuli cases, but thesame discussion is applicable to the four- and five-stimuli
cases. First, as references, stimuli spaces are randomly generated, before computing their confusion matrices. The developed computational methods then regenerated the stimuli
spaces from these matrices. Finally, the reference andFig. 15 Virtual footstep display!
1−ðx1 − S ix1 Þ 2 −ðx2 − S ix2 Þ 2expþ
f ðSi Þ ¼
;
2π
2
2

ð13Þ

where A j , S i, S ix1, and S ix2 are areas separated by the perpendicular bisectors of R , the center of the internal responses toAtten Percept Psychophys
Table 10 Footstep classification: Answer ratios of real-to-real classification task
Assessors’ Answer

Realmaterial

Copy

AluminumPP

TypingCopy paper

0.97

0.01

0.01

0.01

AluminumPP
Typing paper

0.02
0
0.01

0.51
0.15
0.36

0.22
0.63
0.15

0.25
0.22
0.48

Note. PP, polypropylenes i , and the coordinates of S i in the perceptual space, as shown in
Fig. 17. The criterion located at the center of the two stimuli istheir perpendicular bisector, which means that the internal response of a stimulus is classified as the nearest R i . The probability is given by the surface integral of the probability density
function, which is the joint of two normal distribution functions.
Comparison using an evaluation tester
An evaluation tester based on the interstimuli distances was used,which allowed us to compare the reference stimuli space and theone constructed using the method developed in this article. The
tester is the norm of the vector of distances as follows:
I ¼  Dsisj si ; s j ∈fr1 ; …; rn ; v1 ; …vn g :

ð14Þ

Let I r and I c be the norms of the distance vectors for thereference and constructed stimuli spaces, respectively. These
norms were used as testers of the similarity between both spaces.

Figures 19, 20, and 21 show several examples of random
reference and reconstructed spaces. Cases A–C cover relatively small to large stimuli distributions, which correspond to A–
C in Fig. 18. R i and V i are the locations of the references,whereas R′i and V′i are those of the estimates. All of the
figures demonstrate that the distributions of both symbols arewell matched, while the perceptual spaces are sufficiently wellreconstructed to facilitate visual interpretation.
In contrast to the above well-reconstructed perceptual
spaces, I r and I c tended to lose their linearity when I r was
relatively small. Moreover, in most cases, I c was greater thanI r. These two points were based on the use of the constantratio rule, which is discussed in the following sections.
The constant ratio rule expands the perceptual space
The interstimuli distances of the reconstructed spaces weregreater than those of the reference spaces because I c tended tobe larger than I r.
It is known that the constant ratio rule works correctlyin multidimensional cases better than in unidimensional
cases (Hodge, 1967; Hodge & Pollack, 1962). If morethan three stimuli are placed on or approximately along aline, these stimuli are regarded as unidimensional, so the
distances between these stimuli are exaggerated (see the
Appendix). Even if they are unidimensional, the estimation of the distances is not problematic, provided that they
are placed far apart. Thus, their interstimuli distances tend
to be overestimated when more than three stimuli aregrouped within a small area.
Worst case

Results: High linearity between the reference and constructed
stimuli distributions
Figure 18 shows the plots of I r versus I c for 200 random
samples. Both indices clearly indicate high linearity, where
Pearson’s r = .957. The interstimuli distances of randomly
generated spaces were preserved well in the constructed spaces.
Table 11 Footstep classification: Answer ratios of virtual-to-real classification task
Assessors’ Answer

Virtualmaterial

Copy paper
AluminumPP
Typing paper

Copy

AluminumPP

Typing

0.79
0.033
0.013
0.007

0.047
0.39
0.19
0.48

0.053
0.21
0.65
0.23

0.11
0.36
0.12
0.29

Note. PP, polypropylene
Estimated distances that are significantly larger than the original distances
are in bold

The three worst cases are shown, which correspond to D–F in
Fig. 18. Real and virtual stimuli are grouped closely in suchcases, so I r is naturally low around 1–3.
Figure 22 shows the reference and reconstructed stimuli
spaces for the worst case D. Both spaces appear to be significantly different. This is mainly because the application of theconstant ratio rule overestimates the inter-stimuli distances.
Tables 12 and 13 show the distance matrices for a reference
space and that computed using our method, respectively.
Estimated distances that are significantly larger than the original distances are in bold. First, D r′1r′3 and D r′2r′3 are higher
than D r1r3 and D r2r3. This is because R 1, R 2, and R 3 arelocated close together on a line in Fig. 22. The interstimuli
distances are overestimated in such conditions. D r′1r′3 and D r′
2r′3 were used to compute D v′1r′3, D v′2r′3, and D v′3r′3, so thesethree distances are overestimated. Furthermore, D v′1v′3 andD v′2v′3 are also exaggerated because V 1, V 2, and V 3 areapproximately on a line. Thus, the developed method estimated the distributions of stimuli as being significantly greaterthan the original distributions.

Atten Percept Psychophys

Fig. 16 Perceptual space of real and virtual crinkles

The worst cases E and F are shown in Figs. 23 and 24. Aswith case D, the distributions of the reference stimuli were notpreserved well.

covers 0.02–0.04, according to Fig. 25. This method is suitable
when the variance is ≥ 0.04 for three-stimuli cases. In practice,
we rarely observed such low variances. For example, all of theexamples shown in the Examples section have a variance ≥ 0.08.

Introduction of variance as a guide
Four- and five-stimuli cases
As was described above, this computational method works
effectively for a large proportion of the randomly generated
cases, but the method is not suitable when I r is small. Thus,
we propose checking the variances of all the answer ratios
included in both types of confusion matrices as a quick guideto assessing the suitability of this method’s usage. Figure 25
shows the relationships between the variance and I r. The variance increases with I r. When I r is large, real stimuli are placed at
a distance from each other in perceptual space. Thus, the correct
answer ratios are high, whereas the incorrect answer ratios arelow. The variance of answer ratios is high because the ratios are
imbalanced. By contrast, some of the real stimuli are perceptually close when I r is small. Thus, the incorrect answer ratios are
relatively high and close to the correct answer ratios. This
produces a small variance. Figure 18 shows that the method
functions well when I r ≥ 3. A variance corresponding to I r = 3

Fig. 17 Stimuli distributions and bisectors for the computation of confusion matrices

We also validated the computational methods for four- andfive-stimuli cases in two- and three-dimensional spaces. For
four-stimuli cases, the resulting I r and I c had high linearity(2D, r = .97; 3D, r = .94), which was partially lost with asmall I r as in the three-stimuli case.
On the basis of an analysis similar to the three-stimuli case, theupper limits of the variances of the answer ratios in the confusion
matrices of these I r values were 0.03 and 0.025 for the two- andthree-dimensional cases, respectively. Thus, the application of thedeveloped method to four real and virtual stimuli is suitable if thevariance is greater than 0.03 (2D) or 0.025 (3D).
We also determined a guide for five-stimuli cases after
confirming the high linearity between I r and I c (2D, r = .93;
3D, r = .95). In the five-stimuli cases, the computational
method is suitable when the variance is greater than 0.03
(2D) or 0.02 (3D).

Fig. 18 Indices of interstimuli distances for the reference and reconstructed stimuli spaces (three-stimuli case)Atten Percept Psychophys

Fig. 19 Case A: Example of random spaces (relatively small distribution). The filled and unfilled symbols are references and estimates,
respectively

Fig. 21 Case C: Example of random spaces (relatively large distribution). The filled and unfilled symbols are references and estimates,
respectively

Example of a poorly identified stimulus set
The method we developed should not be applied to a poorly
designed stimulus set where the majority of stimuli are
misclassified completely. This is because the calculation of
P virj and P viri is based on the assumption that V i is closer to

R i than to R j . As is shown in Example 5, a slight violation ofthis assumption is not problematic, but the application to a
stimulus set with a strong violation should be avoided. For
example, as is shown in Fig. 26, we considered a three-stimuli
case where all of the virtual stimuli are classified wrongly. V 1
is located nearer R 3 than R 1. V 2 and V 3 are close to R 1 andFig. 20 Case B: Example of random spaces (middle-sized distribution).
The filled and unfilled symbols are references and estimates, respectively

Fig. 22 Worst case D: Reference and reconstructed stimuli spaces. The
filled and unfilled symbols are references and estimates, respectively

Atten Percept Psychophys
Table 12 Worst case: Distance matrix for the reference space

R2
R3
V1
V2
V3

R1

R2

R3

V1

V2

0.60
0.23
0.23
0.27
0.62

–
0.38
0.50
0.44
0.35

–
0.23
0.21
0.47

–
0.07
0.41

–
0.35

R 2, respectively. Thus, as is shown in Tables 14 and 15, which
are the confusion matrices of R 1–R 3 and V 1–V 3, the virtual
stimuli are associated with the wrong real stimuli. Figure 26
also shows the space reconstructed by applying this method tothese tables. The estimates of the virtual stimuli (V ′i ) shifted
greatly from those in the original locations (V i ). In general, it
is unlikely that it will be necessary to conduct any in-depth
analyses of the catastrophic confusion matrices like those in

Fig. 24 Worst case F: Reference and reconstructed stimuli spaces. The
filled and unfilled symbols are references and estimates, respectively

Table 13 Worst case: Distance matrix for the reconstructed space

R′2
R′3
V′1
V′2
V′3

R′1

R′2

0.68
1.59
0.07
0.28
0.04

–
1.63
0.47
0.04
0.19

R′3

–
1.66
1.59
1.82

V′1

V′2

Table 15, but we should not use the method for poorly classified stimulus sets.
–
0.20
1.62

–
1.78

Fig. 23 Worst case E: Reference and reconstructed stimuli spaces. The
filled and unfilled symbols are references and estimates, respectively

Summary of the validation
This section validated the method used to compute the distances between stimuli. As references, random stimuli spaces
were prepared to generate confusion matrices for real-to-realand virtual-to-real classification tasks. The developed method
was then applied to these matrices to reconstruct the stimuli
space. The reference and reconstructed spaces were comparedusing a tester based on the inter-stimuli distances. The test
indices for both spaces had high linearity. The reconstructed
stimuli spaces preserved the references well. However, theestimated space potentially included significant errors when
the index was small owing to the constant ratio rule. In
practice, the variances of all the answer ratios in the confusion

Fig. 25 I r versus the variance of the answer ratios (three-stimuli case)Atten Percept Psychophys

Fig. 27 Internal response of ra when R a, R b, and R c are on a line inperceptual space

Fig. 26 Example of a poorly designed virtual reality system or stimulus
set. R i and V i indicate the estimates of R ′i and V i′ , respectively

matrices are available as a substitutional index. The developed
method is not recommended if the variance is small.

Conclusions
Classification tasks are widely used for assessing virtual reality interfaces; however, it is difficult to comprehensively

Table 14 Real-to-real classification task: Confusion matrix of a poorly
designed stimulus set
Answered
r1

r2

r3

0.62
0.16
0.17

0.18
0.66
0.18

0.20
0.18
0.65

interpret the confusion matrices resulting from these tasks,
especially when the number of stimuli involved is large. To
avoid this problem, MDS solutions graphically facilitate theunderstanding of how well the simulated stimuli are represented in terms of perceptual similarities between simulated
and real stimuli. However, because of the uniqueness of tasks
when real and virtual stimuli are compared in different tasks,
the earlier MDS solutions are not suitable for the assessmentof virtual reality interfaces.
In this study, we developed computational methods foranalyzing the perceptual distances between real and virtual
stimuli to produce MDS solutions. The solutions to two technical issues were presented. The first challenge was to determine the distances between virtual stimuli, which assessors do
not typically record during classification tasks. Another challenge was to retain the consistency of the two classification
tasks that involved different stimuli. Finally, we validated our
method by applying it to random confusion matrix examples.
This confirmed the effectiveness of the method for the majority of the examples.
Author Note This study was in part supported by MEXT KAKENHI#23135514.

AppendixPresent

r1
r2
r3

Table 15 Virtual-to-real classification task: Confusion matrix of a poorly
designed stimulus set
AnsweredPresent

v1
v2
v3

r1

r2

r3

0.21
0.66
0.15

0.26
0.15
0.64

0.53
0.19
0.21

The constant ratio rule tends to overestimate the interstimuli
distances when the distributions of stimuli are approximately
unidimensional and closely allocated in a perceptual space.
This overestimation is explained as follows. Suppose thecenters of the internal responses to three stimuli r a , r b , andr c are on a line in perceptual space, where the internal response of r a is as shown in Fig. 27. Given that the criterion of
judgment is located at the center of the two stimuli, D ab is
calculated by
Dab ¼ 2 λab − Rað15Þ
¼ 2 ⋅ Z ðPaa Þ:
We apply the rule to r a and r b in Fig. 27, then estimate
D a′b′. Using this rule, Da'b' isAtten Percept Psychophys



D

a0 b0

Paa
¼ 2⋅Z
P
Pab
 aa þ 
Paa
¼ 2⋅Z
:
1−Pac

Thus, because
Paa
;
Paa <
1 − Pac


ð16Þ

ð17Þ

D ab <D a ′b ′. When P ac is sufficiently small—that is, R c isdistant from R a and R b — D ab is almost identical to D a ′b ′.
For the same reason, D b ′ c ′ is larger than D bc . By
contrast, D a ′ c ′ can be greater or less than D ac , depending on the specific case. Overall, the application of theconstant ratio rule tends to overestimate the distances
between stimuli.

References
Ashby, F. G., & Townsend, J. T. (1986). Varieties of perceptual independence. Psychological Review, 93(2), 154–179.
Bergmann Tiest, W. M., & Kappers, A. M. L. (2006). Analysis of haptic
perception of materials by multidimensional scaling and physicalmeasurements of roughness and compressibility. Acta Psychologica,
121(1), 1–20.
Carroll, J. D., & Chang, J. J. (1970). Analysis of individual
differences in multidimensional scaling. Psychometrika, 35,
283–319.
Chen, H., Park, J., Dai, S., & Tan, H. Z. (2011). Design and evaluation ofidentifiable key-click signals for mobile devices. IEEE Transactions
on Haptics, 4(4), 229–241.
Clarke, F. R. (1957). Constant ratio rule for confusion matrices in speechcommunication. Journal of the Acoustical Society of America,
29(6), 751–720.
Clarke, F. R., & Anderson, C. D. (1957). Further test of the constant ratio
rule in speech communication. Journal of the Acoustical Society ofAmerica, 29(12), 1318–1230.
Cooke, T., Jäkel, F., Wallraven, C., & Bülthoff, H. H. (2007). Multimodal
similarity and categorization of novel, three-dimensional objects.
Neuropsychologia, 45, 484–495.
Dev, P., Harris, D., Gutierrez, D., Shah, A., and Senger, S.
(2002). End-to-end performance measurement of internet
based medical applications. Proceedings of the 2002
Ame ri can M ed ic al I nf or ma ti cs As s oc ia tio n Ann u al
Symposium , pages 205–209.
Fujita, K. and Ohmori, H. (2001). A new softness display interface bydynamic fingertip contact area control. Proceedings of the 5th World
Multiconference on Systemics Cybernetics and Informatics , pages
78–82.
Fukumoto, M. and Sugimura, T. (2001). Active click: Tactile
feedback for touch panels. Proceedings of ACM SIGCHI2001 the Conference on Human Factors in Computing
Systems , pages 121–122.
Germani, M., Mengoni, M., and Peruzzini, M. (2013). Electro-tactile
device for material texture simulation. International Journal ofAdvanced Manufacturing Technology .
Giordano, B. L., Visell, Y., Yao, H.-Y., Hayward, V., Cooperstock,
J. R., & McAdams, S. (2012). Identification of walked-uponmaterials in auditory, kinesthetic, haptic, and audio-haptic
conditions. Jounal of the Acoustical Society of America, 131(5),4002–4012.

Gomoll, A., Pappas, G., Forsythe, B., & Warner, J. J. P. (2008). Individual
skill progression on a virtual reality simulator for shoulder arthroscopy: A 3-year follow-up study. The American Journal of Sports
Medicine, 36(6), 1139–1142.
Hendrix, C., & Barfield, W. (1996). Presence within virtual environments
as a function of visual display parameters. Presence: Teleoperatorsand virtual environments, 5, 274–289.
Hikichi, K., Yasuda, Y., Fukuda, A., and Sezaki, K. (2006). The effect of
network delay on remote calligraphic teaching with haptic interfaces. Proceedings of the 5th ACM SIGCOMM Workshop on
Network and System Support for Games , (12).
Hodge, M. H. (1967). Some further tests of the constant ratio rule.
Perception & Psychophysics, 2, 429–437.
Hodge, M. H., & Pollack, I. (1962). Confusion matrix analysis of single
and multidimensional auditory displays. Journal of Experimental
Psychology, 63, 129–142.
Hollins, M., Bensmaïa, S. J., Karlof, K., & Young, F. (2000). Individual
differences in perceptual space for tactile textures: Evidence frommultidimensional scaling. Attention, Perception & Psychophysics,
62(8), 1534–1544.
Hollins, M., Faldowski, R., Rao, S., & Young, F. (1993). Perceptual
dimensions of tactile surface texture: A multidimensional scaling analysis. Attention, Perception & Psychophysics, 54 (6),697–705.
Holyoak, K. J., & Glass, A. L. (1978). Recognition confusions among
quantifiers. Journal of verbal learning and verbal behavior, 17,249–264.
Howe, R. D., Peine, W. J., Kontarinis, D. A., & Son, J. S. (1995). Remotepalpation technology. IEEE Engineering in Medicine and Biology,
14(3), 318–323.
International telecommunication union (2009). Methodology for the subjective assessment of the quality of television pictures.
Recommendation ITU-R BT.500-12 .
Jeon, S., Choi, S., & Harders, M. (2012). Rendering virtual tumors in realtissue mock-ups using haptic augmented. IEEE Transactions on
Haptics, 5(1), 77–84.
Karadogan, E., & Williams, R. L., II. (2013). Haptic modules forpalpatory diagnosis training of medical students. Virtual Reality,
17(1), 45–58.
Kontarinis, D. A., & Howe, R. D. (1995). Tactile display of vibratory
information in teleoperation and virtual environments. Presence,
4(4), 387–402.
Konyo, M., Tadokoro, S., and Takamori, T. (2000). Artificial tactile feel
display using soft gel actuators. Proceeding of the 2000 I.E.
International Conference on Robotics and Automation , pages
3416–3421.
Kruskal, J. B. (1964a). Multidimensional scaling by optimizaing goodness of fit to a nonmetric hypothesis. Psychometrika, 29, 1–27.
Kruskal, J. B. (1964b). Nonmetric multidimensional scaling: A numericalmethod. Psychometrika, 29, 115–129.
Kuchenbecker, K. J. and Niemeyer, G. (2006). Improving telerobotic
touch via high-frequency acceleration matching. Proceedings ofthe 2006 I.E. International Conference on Robotics andAutomation , pages 3893–3898.
Luce, R. D. (1963). Detection and Recognition . Dover Publications.
Luce, R. D. (2005). Individual Choice Behavior: A Theoretical Analysis .
Dover Publications.
Luk, J., Pasquero, J., Little, S., MacLean, K., , Levesque, V., andHayward, V. (2006). A role for haptics in mobile interaction: Initialdesign using a handheld tactile display prototype. Proceedings of the2006 ACM SIGCHI Conference on Human Factors in Computing
Systems , pages 171–180.
Mai, Z., Doutre, C., Nasiopoulos, P., & Ward, R. K. (2012). Rendering 3d high dynamic range images: Subjective evaluation of tonemapping methods and preferred 3-d image attributes. IEEE
Journal of Selected Topics in Signal Processing, 5, 597–610.

Atten Percept Psychophys
Motoyoshi, I., & Matoba, H. (2012). Variability in constancy of theperceived surface reflectance across different illumination statistics.
Vision Research, 53, 30–39.
Nakatani, L. H. (1972). Confusion-choice model for multidimensional
psychophysics. Journal of Mathematical Psychology, 9(1), 104–127.
Nishio, A., Goda, N., & Komatsu, H. (2012). Neural selectivity andrepresentation of gloss in the monkey inferior temporal cortex.
Journal of Neuroscience, 32(31), 10780–10793.
Okamoto, S., Ishikawa, S., Nagano, H., & Yamada, Y. (2013). Spectrumbased synthesis of vibrotactile stimuli: Active footstep display forcrinkle of fragile structures. Virtual Reality, 16(3), 181–191.
Okamoto, S. and Yamada, Y. (2012). Illustrative evaluation index for
haptic interfaces using confusion matrices. in press, Proceedings ofthe 21st IEEE International Symposium on Robot and Human
Interactive Communication .
Okamura, A. M., Cutkosky, M. R., & Dennerlein, J. T. (2001). Realitybased models for vibration feedback in virtual environments. IEEE/AME Transactions on Mechatronics, 6(3), 245–252.
Okamura, A. M., Webster III, R. J., Nolin, J. T., Johnson, K. W., and Jafry,
H. (2003). The haptic scissors: Cutting in virtual environments.
Proc. the 2003 I.E. International Conference on Robotics andAutomation , pages 828–833.
Ottermo, M. V. (2006). Virtual Palpation Gripper . Ph.D. thesis,
Norwegian University of Science & Technology.
Pedowitz, R. A., Esch, J., & Snyder, S. (2002). Evaluation of a virtual
reality simulator for anthroscopy skills development. The Journal of
Arthroscopic and Related Surgery, 18(6), E29.
Picard, D., Dacremont, C., Valentin, D., & Giboreau, A. (2003).
Perceptual dimensions of tactile textures. Acta Psychologica,
114(2), 165–184.
Pols, L. C. W. (1983). Three-mode principal component analysis ofconfusion matrices, based on the identification of dutch consonants,
under various conditions of noise and reverberation. SpeechCommunication, 2(4), 275–293.
Takane, Y., Young, F. W., & De Leeuw, J. (1977). Non-metric individual
differences multidimensional scaling: An alternating least squares
method with optimal scaling features. Psychometrika, 42, 7–67.

Torgerson, W. S. (1952). Multidimensional scaling: I. theory and method.
Psychometrika, 17, 401–419.
Townsend, J. T. (1971). Theoretical analysis of an alphabetic confusion
matrix. Perception & Psychophysics, 9, 40–50.
Townsend, J. T., & Landon, D. E. (1982). An experimental and theoretical investigation of the constant-ratio rule and other models ofvisual letter confusion. Journal of Mathematical Psychology, 25,
119–162.
Tucker, L. R., & Messick, S. (1963). An individual differencesmodel for multidimensional scaling. Psychometrika, 28, 333–
367.
van Baren, J. and IJsselstejin, W. (2004). Measuring Presence: A
Guide to Current Measurement Approaches . IST OmniPres
Project.
Watanabe, T. and Fukui, S. (1995). A method for controlling tactile
sensation of surface roughness using ultrasonic vibration.
Proceeding of the 1995 I.E. International Conference on Robotics
and Automation , pages 1134–1139.
Wiertlewski, M., Lozada, J., & Hayward, V. (2011). The spatial spectrumof tangential skin displacement can encode tactual texture. IEEE
Transactions on Robotics, 27(3), 461–472.
Witmer, B. G., & Singer, M. J. (1998). Measuring presence in virtualenvironment: A presence questionnaire. Presence: Teleoperators andvirtual environments, 7(3), 225–240.
Yamamoto, A., Nagasawa, S., Yamamoto, H., & Higuchi, T.
(2006). Electrostatic tactile display with thin film slider andits application to tactile telepresentation systems. IEEE
Transactions on Visualization and Computer Graphics, 12 (2),
168–177.
Yasnoff, W. A., Mui, J. K., & Bacus, J. W. (1977). Error measures forscene segmentation. Pattern recognition, 9, 217–231.
Yoshioka, T., & Zhou, J. (2009). Factors involved in tactile
texture perception through probes. Advanced Robotics, 23,
747–766.
Young, F. W., & Null, C. H. (1978). Multidimensional scaling of nominal
data: The recovery of metric information with alscal. Psychometrika,
43, 367–379.