# Visuo-Haptic Mixed Reality Simulation Using Unbound Handheld Tools
## 4 Validation Experiments and Results
### User Evaluation Experiments  
We conducted a face and content validation study in which 15 participants were asked to test the VHMR system with the tennis ball scenario and provide feedback in a questionnaire.  
In this test, after being provided with general information about the system, the user wears the HMD and sees the virtual tennis ball. Then, the user is asked to hold any of the two available tools with the dominant hand and indent the tennis ball radially at different locations for two minutes. One tool change is instructed during each experiment.  
15 healthy participants took part in the study, with ages between 23 and 50 years (M = 32.9; SD = 8.9). These participants are all engineers with different levels of knowledge and experience in force-feedback, virtual reality/mixed reality (VR/MR), and visuo-haptics. Six (40%) participants had used a force-feedback device before, nine (60%) had not. Seven (47%) of the participants had used VR/MR devices and eight (53%) of them had not. Five (33%) participants had experienced visuo-haptic interaction before, while ten (67%) had not.  
An item pool was formed for the questionnaire based on a literature survey. Then the pool was refined, according to the opinions of seven experts in the field, to reflect face and content validity. The final questionnaire in Figure 14 consists of eleven items. Items 1 to 4 are about the overall simulation approach. Items 5 and 6 focus on haptic perception, while 7 and 8 are on vision. Visuo-haptic coordination is evaluated with items 9 to 11. A seven-step Likert scale was used: (1) Very strongly disagree, (2) Strongly disagree, (3) Disagree, (4) Neutral, (5) Agree, (6) Strongly agree, (7) Very strongly agree.  
Figure 14: User evaluation questionnaire and results.  
Figure 13: Contact point deflection versus time plots of the haptic-tip and the visual tip. \(i\) and _ii_ are zoomed-in regions.  
In general, the questionnaire results (Figure 14) reflected a positive view on the part of the participants, with a global mean of 5.3 points. Regarding the overall simulation, the users were very positive about the user-friendliness of the VHMR system and they strongly agreed that using unbound tools and freely changing them enhances realism. Item 2 had the lowest mean and widest range of points in the questionnaire. This was expected, since this item questions the simulation in a broader view. One reason for the synthetic opinion may arise from the physical discomfort of wearing an HMD. Besides, mixed-reality does not transfer the user to someplace else like virtual-reality, so the user may end up performing a peculiar task in a familiar environment. Another reason may be the simulation scenario itself, such that indenting a fixed tennis ball with pointy tools is unlikely to happen in real-life.  
Overall evaluations of the realism of haptic (item 5) and visual (item 7) perceptions had similar scores, concentrated between agree and strongly agree. However, more specific items received lower points. Item 6 feedback suggests that the first encounter with the virtual object could be better, and the Item 8 results indicate alignment issues with the tooltip overlay. The last three items evaluate the spatial alignment, synchronization, and coherence between the visual and haptic stimuli. As a group, these items scored better than the rest of the questionnaire. There was a consensus on the success of visuo-haptic synchronization (item 10), 11 out of 15 participants strongly agreed, while two agreed, and two very strongly agreed.  
In general, participant groups with and without technology experience in force-feedback, VR/MR, and visuo-haptics have provided similar evaluations. However, the ones with experience in VR/MR evaluated the realism of haptic perception (item 5) the highest (Mean: 6.0; Range: 6-6), while the ones experienced in haptics provided the highest score (Mean: 5.5; Range: 5-6) for the realism of visual perception (item 7).