# Visuo-Haptic Mixed Reality Simulation Using Unbound Handheld Tools
## 1 Introduction  
Visuo-haptic mixed reality (VHMR) combines real and virtual objects in the same simulation environment, where a user can see and also touch the objects in the scene. The two sensory modalities not only complement the overall realism of simulation but also cover-up fidelity deficits of each other. For example, visual perception of surface roughness is improved by touch [1], and haptic perception of stiffness is enhanced by vision [2] in virtual environments. Medicine is deemed to be one of the most effective application areas of VHMR [3; 4; 5; 6; 7]. Maintenance training [8; 9; 10] and product design [11] are notable others. The colocation of visual and touch stimuli in VHMR brings new technological challenges that do not exist in conventional delocated virtual reality haptics. If the user interacts with bare hands, then accurate hand tracking is necessary [12; 13]. Barehand interaction is mostly limited to surface haptics with tactile stimuli [14; 15]. Although being at an experimental level, wearable devices such as haptic gloves extend the simulation to have kinesthetic feedback [16; 17; 18]. However, there is then anadditional challenge of removing the device from the visual scene or replacing the hand's view with a virtual one. Most studies with kinesthetic feedback resort to using commercial haptic devices, where the user continuously holds a stylus connected to the device. There is no need for hand tracking since the device can provide an accurate position and orientation. Nevertheless, visual obstruction caused by these bulky devices is a substantial issue. Chroma-key masking [4; 19] and visual removal by image processing [2] of the haptic device are shown to be useful solutions. Once the device is removed from the scene, adding a virtual tool and aligning it with the stylus of the device poses another challenge [20; 21]. Although the tip (or edge) of the tool interacts with the objects in the scene, its handle is harder to render visually because of the occlusion caused by the user's hand. Also, the handle you see and the handle you hold do not usually feel the same. Haptic devices operate in a limited workspace, and they can stably render a limited range of mechanical impedances. Typically, impedance-type devices are better at rendering low inertia, but they are not strong enough for rendering high stiffness. The opposite of this is valid for admittance-type devices. In a VHMR simulation, the user only actually needs haptic feedback when interacting with the objects. Holding a stylus attached to the haptic device at all times constrains the user to the workspace and impedance limits of the device even when it is not needed. This brings us to the motivation of this research. One of the pioneer teams working on colocated visuo-haptic feedback, [PERSON] et al. [19], probably had a similar motivation, such that they built an encountered-type haptic display (ETHD) for their work. ETHDs are typically industrial robots that are configured to provide admittance-type haptic feedback. They position their end-effector at the location of the virtual object and wait for the user to \"encounter\". Although the original idea in [19] falls under the augmented reality category, this approach is especially suited for use with virtual reality, where the user only sees the graphically rendered environment [15]. Studies on ETHDs are limited in number compared to other types of haptic devices, but they focus on barehand interaction in large workspaces. Applications can be categorized into two groups. In the first group, users grasp ETHD's end-effector, which is a physical prop that represents the virtual object or a portion of it. A pneumatic balloon is used as a prop in [22] for rendering non-spherical objects, like a human arm, that is approximated by a chain of spheres with different sizes. An industrial plant training scenario with virtual reality is presented in [23] where the ETHD changes props, such as buttons and valve wheels, as needed.