Real-Time Augmented Reality for Image-Guided Interventions (2009)

The aim of this thesis is to design, develop, and study a new approach to interventional image guidance based on augmented reality visualization – the merging of real and virtual images. This work broadly reviews and categorizes all preliminary augmented reality prototype systems for interventional image guidance. A novel augmented reality setup is introduced and engineered for interventional guidance tasks, integrating pre-operative imaging, namely CT and MRI, as well as real-time ultrasound imaging. Built on a single PC platform with four independent frame-grabber boards and genlocked cameras, this system uniquely combines head-mounted stereoscopic video capture and visualization with optical head and instrument tracking in a very efficient, but yet modular, closed-loop approach. Because of its unique architecture it provides unmatched real-time performance and accuracy of high resolution augmented reality, enabling most complicated guidance tasks. This work advances methods for multi-camera calibration and real-time tracking. A novel head-mounted camera-triplet is described, where two cameras capture the stereoscopic view of the scene while a third, near-infrared camera with a wide-angle lens and a synchronously pulsing infrared flash is used for real-time pose estimation tasks, tracking retroreflective markers in the scene. A specialized calibration technique for the calibration of all intrinsic and relative extrinsic parameters of the camera-triplet with a dedicated calibration object is developed, taking into account the very different optical properties of the three cameras. In respect to real-time optical tracking, a unified approach to pose estimation is developed that includes an optimized cluster design of retroreflective markers, a strategy for marker identification in the tracker camera images, and a method for extrinsic camera parameter estimation based on an initial 3-Point algorithm and non-linear parameter optimizations utilizing all available 2D–3D point correspondences. This unified approach provides precise pose estimates for the camera-triplet in respect to stationary marker frames and marker clusters that are attached to hand-held tools in less than 10ms. The robustness of the tracking results is evaluated in Monte-Carlo simulations and in actual experiments. The introduced augmented reality system provides a novel way to access three-dimensional datasets intuitively in direct relation to real objects and interactions. In comparison to typical screen-based visualization methods, the stereoscopic and kinetic depth cues of the proposed head-mounted approach provide a much higher degree of immersion and, thus, understanding of the data. Furthermore, the system provides the stereoscopic view of the real scene (such as the patient’s body) in direct relation to the virtual data, effectively rendering real objects transparent if the virtual dataset represents otherwise invisible internal details. Interactions with the datasets, which in a screen-based solution need to be resolved through mouse movements and clicks, is given in a way that resembles interactions with real objects by means of tracked instruments. This work makes strong contributions to potential applications of augmented reality in the interventional arena. Studies are performed that show that three-dimensional graphics presented within the actual patient with stereoscopic and kinetic depth cues provide a very compelling and direct access to anatomical information for the planning and guidance of an efficient intervention. Surgeons benefit from this direct overlay of medical images onto the actual patient especially during minimally invasive surgeries. Additional navigational support is given by means of tracked instruments, in particular biopsy and ablation needles, which are visualized as virtual objects within the patient’s actual body. Results of pre-clinical phantom and animal studies are presented, utilizing the introduced augmented reality system for needle placement tasks that evaluate the benefits of the availability of pre-operatively acquired imaging information (e.g., CT or MRI) to interventional radiologists during an intervention. Increased biopsy precision and the absence of continuous x-ray exposure through fluoroscopy are major contributions here.