(c) 2012 IEEE. Personal use of this material is permitted. However, permission to reprint/republish this material for advertising or promotional purposes or for creating new collective works for resale or for redistribution to serves or lists, or to reuse any copyrighted component of this work in other works must be obtained from the IEEE.
Shogo Okamoto and Yoji Yamada Abstract—For researchers of haptic interfaces, evaluation of the perceptual similarity between virtual and real haptic stimuli has long been a serious problem. One of the most commonly employed evaluation methods is an identification task where assessors identify the type of randomly presented stimuli among multiple candidates. The results of this method are summarized as confusion matrices. We developed a method that allocates all virtual and real stimuli in a perceptual space. The spatial distribution of the stimuli allows us to visually understand the perceptual relationships between the stimuli. A brief validation confirmed that the proposed method is effective in evaluating the perceptual similarity between virtual and real stimuli.
I. I NTRODUCTION
Assessment of the stimuli displayed to users of haptic interfaces is vital. Various methods have been employed for this purpose, for example, comparing the similarity of phys- ical quantities, such as force or skin deformation, between virtual and real stimuli (e.g. [1], [2]). However, studying the physical similarity is not sufficient, and researchers often exploit the evaluation of perceptual similarity. In some cases, assessors evaluate the degree of perceptual reality using numerical scales. In other cases, they conduct sensory evaluations with certain criteria such as the perceived rough- ness of textures or hardness of objects. For each criterion, the similarity between virtual and real stimuli is discussed. Questionnaire-based reality evaluation for virtual reality systems (e.g., [3], [4]) is also applicable to haptic interfaces. Such methods have their advantages and disadvantages; however, a general method does not exist. Researchers tend to select methods appropriate for their own purposes. Identification tasks: Many haptic researchers have chosen identification tasks owing to their simplicity and objectivity (e.g., [5], [6], [7], [8]). In these tasks, a blind- folded assessor chooses a stimulus that feels most similar to the randomly presented stimulus among n candidates ( n- alternative forced choice). Then, the goodness of haptic interfaces is discussed on the basis of correct response ratios. Two types of experiments are commonly performed in identification tasks. In one, the virtual-to-real identification task, assessors choose the real stimulus that feels closest to the virtual stimulus presented to them. In the other, the real-to-real identification task, assessors respond to a real This work was in part supported by MEXT KAKENHI 24700192 and 23135514. The authors are with the Department of Mechanical Science and Engineering, Nagoya University, Japan. stimulus. The results of these two experiments are compared to assess haptic interfaces and stimuli. Some researchers focus on only the correct responses; however, wrong answer ratios also provide important information. Confusion matrices: Confusion matrices are often used to show the results of identification tasks. We acquire two matrices from the two types of experiments mentioned above. One problem is that it is not easy to understand and compare the meanings of wrong answer ratios between these two matrices. Tables I and II show the examples of matrices established by four stimuli. In confusion matrices, diagonal elements indicate correct response ratios and the others are confusion ratios. r i and v i ( i = 1, ...,4) are types of real and virtual stimuli, respectively. v i is a virtual stimulus that mimics r i . P sisj is a probability at which assessors classify s i as s j . First, the real-to-real identification task table (Table I) shows that because P r 1 r 1 and P r 2 r 2 are high, r 1 and r 2 are somewhat easily identified. On the other hand, Pr3r3 and P r4r4 are relatively small and P r3r4 and P r4r3 are considerably high. Hence, r 3 and r 4 are likely to be confused. Second, the virtual-to-real identification task table (Table II) shows that the overall correct response ratios are lower than those of the real-to-real identification task. Thus, virtual stimuli incompletely capture the characteristics of real stimuli, and the confusion ratios elevate. Specifically, v 3 and v 4 are problematic because P v3r3 and P v4r4 are significantly smaller than P r 3r 3 and P r 4r 4 , and P v 3r 4 and Pv4r3 are larger than Pv3r4 and Pv4r3. Also, since Pv2r3 and P v 3r2 are larger than P r2r3 and P r3r2 , v 2 and v 3 feel more similar than the combination of r 2 and r 3 . Finally, P r 1 r 1 and P v 1 r 1 are the same value, which indicates that v 1  well represents r 1 . It is informative to collectively consider both the correct and wrong answer ratios. However, for those who are not familiar with identification tasks, it is not easy to capture the meanings of the wrong answer ratios of the two matrices. Thus, wrong answer ratios, which actually include thoughtful information, tend to be ignored. Correct and wrong answer ratios collectively tell us what improvements are necessary for better haptic interfaces. Objective: The objective of this study is to integrate two types of confusion matrices and construct an illustrative index for intuitive understanding, in which all real and virtual stimuli are placed in a perceptual space. Geometric distances in this space correspond to the perceptual dis- Presented
Real and virtual stimuli placed in perceptual space similarities of the stimuli. The illustrative index allows us to easily obtain the results of identification tasks. There are two technical challenges to establishing such an index. First, two confusion matrices are based on two separately conducted experiments. These matrices must therefore be combined to maintain consistency. Second, assessors do not directly compare pairs of virtual stimuli, whereas they do compare pairs of real stimuli and pairs of virtual and real stimuli in real-to-real and virtual-to-real identification tasks, respectively. Thus, the perceptual distances between virtual stimuli must be computed from unsatisfactory information. Novelty of this study:  To the best of our knowledge, this work is the first attempt to construct a perceptual space of real and virtual stimuli from confusion matrices whereas some measures were established for such matrices. Powerful methods that compute inter-stimuli psychological distances from confusion matrices are the similarity choice model [9], [10] and overlap model [11], [12]. However, these distances are not computed by two confusion matrices that include different stimulus elements. In our problem, two matrices including different elements must be integrated. We use a multidimensional scaling approach to construct a spatial distribution of stimuli. This approach has been used to show the perceptual distances or equivalences between haptic stimuli [13], [14], [15], [16]. For the computa- tion of this approach, perceptual distances between stimuli are required. They were typically acquired by clustering tasks [13], [14] or scaling tasks [15], [16]. In the former tasks, participants clustered the stimuli into several groups based on their similarities. In the latter tasks, the dissimi- larity between two stimuli are specified using a numerical scale. On the other hand, our proposed method allows us to compute a multidimensional scaling method from the results of identification tasks.
II.  ILLUSTRATIVE INDEX FOR IDENTIFICATION TASKS
The illustrative index allocates real and virtual stimuli in a perceptual space, in which geometric distances corre- spond to perceptual dissimilarities. Fig. 1 shows the index constructed by Tables I and II. R i and V i are locations of r i and v i in the perceptual space. Since R 1 and R 2 are located far from the other real stimuli, r 1 and r 2 are easily identified. R 3 and R 4 are closely located and, r 3 and r 4 are frequently confused. The virtual stimuli are distributed inside of the real stimuli, indicating that the correct response ratios of the virtual-to-real identification task are smaller than those of the real-to-real identification task. V 2 , V 3 , and V 4 are perceptually close to each other and their confusion ratios are high. On the other hand, V 1 and R 1 are very close, which suggests that the confusion ratios between v 1 and r i ( i = 2,3,4) are almost the same as those between r 1 and r i (i = 2,3,4). V 2 and R 2 are far from each other and dissimilar; thus, v 2 needs to be improved such that it feels more similar to r 2 . As mentioned above, the introduction of the illustrative index enhances the understanding of perceptual relationships between stimuli. To construct the index, we first compute the perceptual distances between all stimuli. Different methods are used to compute the distances between pairs of real stimuli, real and virtual stimuli, and virtual stimuli. We then apply a multidi- mensional scaling method on the computed distances. The stimuli are located in a two- or three-dimensional space so that their distances are well maintained.
A. Computation of inter-stimuli distance
Let D sis j be the perceptual distance between s i and s j . Given that D sisj = D sjsi , to compute the multidimensional scaling method, the upper triangular part of the distance matrix in Fig. 2 is necessary. D rirj and D vivj are computed from the results of real-to-real and virtual-to-real identi- fication tasks, respectively. In order to integrate the two tasks while maintaining consistency, D rivj is computed by combining the results of both tasks. 1) Model of internal response by signal detection theory: In detection theory, identification of a stimulus among n candidates is modeled as an identification of stimuli on perceptual multi-dimensions. The human internal response to a physical stimulus is expressed as a normal distribution, as shown in Fig. 3. The internal response toward a physical stimulus s i follows a joint probability density function
of multiple variables (two variables in the figure) with the center of distribution S i . These variables are mutually independent and construct the perceptual space of stimuli. In this model, when the response toward s i appears closer to S i than S j , the stimulus is classified or recognized as s i . The border of this judgment is called criterion λ ij . We assume the equality of variance of distributions, which is commonly accepted. The unit of each dimensional variable is set to be equal to this variance. We deal with stimuli on multi-dimensions, following the constant ratio rule [17], [18]. Applying this rule to two arbitrary stimuli s i and s i , their probabilistic and perceptual relationships are discussed on the S i − S j line. As shown in Fig. 3 (upper left), the internal responses of the two stimuli are simplified. We estimate their perceptual distance as D sisj = Sj — Sj on this line. 2) Distance between real stimuli: Fig. 4 shows the internal responses toward a real stimulus r i on the R i − R j line. On this line,
holds, where Z(p) is the z -score of probability p. Z(p) is described by
Similarly, R ,
virtual stimuli into real stimuli, the internal responses of stimuli are expressed as shown in Fig. 5. When virtual stimulus v i is presented and its internal response is closer to R i , the stimulus is classified as r i . On the other hand, if the response is closer to R j , then it is classified as r j . The border of this judgment is described as λ ij . Looking at this internal response on the line that passes V i and the center of R i and R j , the response to v is expressed as shown in Fig. 6. λ i i – V i is described by
locates at the center of R i and R j . The distances between virtual and real stimuli are given by
These distances are given by solving (7). D rirj is de- termined in the above section from the results of the real-to-real identification task. Thus, D viri and D virj are computed by the results of both identification tasks; this combines the two identification tasks and maintains their consistency. In this computation, R i and R j are projected on the V i − R i − R j line. Hence, D viri and D virj tend to be underestimated. 4) Distance between virtual stimuli: Finally, we com- pute the distances between virtual stimuli. In identification tasks, assessors do not directly compare virtual stimuli. However, from the results of the virtual-to-real identification task, the distances between virtual stimuli can be computed. Fig. 7 shows the internal responses toward v i and v j and R i and R j . We look at this space on a line that goes though V i and V j . Fig. 8 shows the responses toward v i and v j and the projections of R i and R j on this line. Here, holds. Similarly,
holds. The distances between two virtual stimuli are com- puted by these equations, as follows: Internal response toward V f and V f
B. Multidimensional scaling method
We use a multidimensional scaling method to compute the distribution of stimuli in the perceptual space from their perceptual distances. This method determines the coordi- nates of events (stimuli) in a space while maintaining the given distances between them. The number of dimensions can be set arbitrarily; however, in general, larger dimensions represent the distances better. Considering that our index is illustrative, two or three dimensions are preferable. In the present study, we use the metric multidimensional scaling developed by Torgerson [19].
III.  EXAMPLE AND BRIEF VALIDATION
We validate the method by checking the consistency be- tween the confusion matrix examples and computed spatial distribution of stimuli. We present examples of the three- stimulus cases. Table III shows the confusion matrix of the real-to-real identification task. We use this table to compute the stimulus distribution for various matrices of the virtual- to-real identification task. In the following examples, if the distance between two stimuli is 2, they are correctly classified at 84% (2 · Z(0.84) = 2.0). As is the nature of multidimensional scaling methods, the orthogonal rotation of the stimuli's coordinates is indeterminate. Example 1 (Shrunk space): First, we compute the stimuli distribution when the confusion matrix of the virtual-to-real task is given as in Table IV, where the correct response ratios are smaller than those for the real- to-real task. The wrong answer ratios are balanced. In this TABLE III REAL-TO-REAL IDENTIFICATION TASK: Confusion matrix (Examples 1–5) Answered
0.1 0.8 TABLE VI E XAMPLE 3 (EXPANDED): C ONFUSION MATRIX OF VIRTUAL-TO-REAL IDENTIFICATION case, the distribution of virtual and real stimuli should be geometrically similar and only their areas should be different. Naturally, the distribution of virtual stimuli should be smaller than that of real stimuli. Fig. 9 shows the distribution of stimuli computed using Tables III and IV. As expected, the distributions of real and virtual stimuli are similar, and the area for virtual stimuli is smaller than that for real stimuli. Example 2 (Distorted space):  As shown in Table V, we decrease Pv2r2 and Pv3r3 , while keeping Pv1r1 as high as P r 1 r 1 . Accordingly, we increase P v2r 3 and P v3r 2 . In this case, V 1 should be located on the same coordinate as TABLE IV E XAMPLE 1 (SHRUNK): C ONFUSION MATRIX OF VIRTUAL-TO-REAL IDENTIFICATION
TABLE VII E XAMPLE 4 ( SAME ): C ONFUSION MATRIX OF VIRTUAL-TO-REAL IDENTIFICATION TABLE V E XAMPLE 2 (DISTORTED): C ONFUSION MATRIX OF VIRTUAL-TO-REAL IDENTIFICATION Answered
TABLE VIII E XAMPLE 5 (CONFUSED): C ONFUSION MATRIX OF VIRTUAL-TO-REAL IDENTIFICATION R 1 because Pr1ri = Pv1ri (i = 2,3), which means that V 1 is the same as R 1 in terms of the relationships with other real stimuli. Also, V 2 and V 3 should approach R 3 and R 2 , respectively, because V 2 and V 3 are more likely to be confused than R 2 and R 3 . Fig. 10 shows the distribution of stimuli computed using Tables III and V. R 1 and V 1 are placed together while V 2 and V 3 are placed closer than R 2 and R 3 . These tendencies are consistent with the above expectations. As a result, the distribution of virtual stimuli is distorted compared with that of real stimuli. Example 3 (Expanded space): As shown in Ta- ble VI, we increase the correct response ratios of the virtual- to-real identification task, and the wrong answer ratios are balanced. In this case, the distribution of virtual stimuli should be larger than that of real stimuli. The polygons com- prising virtual and real stimuli should be similar because the proportions of wrong answer ratios are equal between both identification tasks. Fig. 11 shows the distribution of stimuli computed using Tables III and VI. The polygon comprising virtual stimuli is located outside that of real stimuli. Both polygons are geometrically similar. As a result, the distribution of virtual stimuli appears to be expanded compared with that of real stimuli. This could occur when the features of real stimuli are well represented and even emphasized by virtual stimuli. Example 4 (Same matrix): When the confusion matrices of both identification tasks are identical, the al- locations of virtual and real stimuli in a perceptual space should also be the same. Fig. 12 shows the perceptual space computed under this condition, using Tables III and VII. The coordinates of V i and R i match. Example 5 (Confused case):  In the above examples, the diagonal elements of the confusion matrix of the virtual- to-real identification task are larger than the others. In other words, v i is most frequently classified as r i . However, such an ideal case does not always exist in actual experiments. For example, as shown in Table VIII, v 2 and v 3 can be wrongly classified as r 3 and r 2 , respectively. In this case, V 2 should be closer to R 3 than to R 2 . Also, V 3 should be located near R 2 . Fig. 13 shows the perceptual space computed using Tables III and VIII. Consistent with the above expectations, V 2 is closer to R 3 than to R 2 and V 3 is closer to R 2 . This indicates that stimuli 2 and 3 are totally confused. Summary: The above examples confirm the consis- tency between the confusion matrices and acquired percep- tual space.
IV.  CONCLUSIONS
In this study, we developed an illustrative index for evaluating haptic interfaces, which helped us understand the perceptual similarities between virtual and real haptic stimuli. The index was constructed from the results of iden- tification tasks, which are commonly used for evaluating haptic interfaces. We computed the perceptual distances of stimuli using detection theory. The first challenge was integration of the two tasks: real-to-real and virtual-to- real identification tasks. We computed the distances by combining the results of the two tasks while maintaining their consistency. Another challenge was the acquisition of distances between virtual stimuli, which assessors do not typically compare in identification tasks. Finally, we used a multidimensional scaling method and allocated all virtual and real stimuli in the perceptual space. This allowed us to visually capture how well the virtual stimuli were repre- sented in terms of perceptual similarities. We validated our method by applying it to some confusion matrix examples and confirmed the consistency between the computed spatial distributions of stimuli and the confusion matrices. Through further statistical validations, we hope to construct an even more reliable and useful index for haptic interfaces.