title: Interactive Rendering of Medical Volumetric Data in Extended Reality
Creators: name: Hüttl, Jana
Publisher: name: Friedrich-Alexander-Universität Erlangen-Nürnberg (FAU)
Year: 2024
Subjects: subject: Martschinke

description: Tomographic imaging techniques like computed tomography (CT) or magnetic resonance imaging (MRI) play an increasingly important role not only for medical diagnosis, but also for surgery planning, patient-doctor communication, medical teaching and forensics. Physically-based volumetric path tracing turns the resulting slice image stack into photo-realistic images of the insides of the human body, enabling non-radiologists like surgeons and even medically inexperienced users like patients, students or legal experts to make use of those imaging modalities. The benefit of three-dimensional renderings of medical data can be further increased by presenting them to the viewer in extended reality (XR), thus either fully immersing the user into data exploration (for virtual reality, VR) or enhancing objects from the physical world, e.g. the skin of a patient, with digital content (for augmented reality, AR). However, for an XR application to be useful for the viewer, it has to be interactive in two ways: The user needs to be provided both with intuitive interaction methods to explore the medical data and with interactive frame rates to experience the effects of those interactions in real-time. As a step towards the first goal (intuitive interaction with medical volumetric data), we present a novel projection-based AR approach to surgery planning by the example of a specific surgical intervention, DIEP flap breast reconstruction. We propose a view-dependent volumetric rendering overlay for the abdominal skin of the patient and a controller-driven, two-step point selection method for finding and labeling internal points of interest directly in situ on the patient. As the labeling of patient-specific landmark points on the skin serving as a preoperative planning step therefore does not depend on classical viewing software for CT/MRI image stacks and the user's spatial imagination anymore, the resulting labels exhibit greatly improved accuracy and the method can easily be utilized by medical laypersons, as we show in a pilot user study. For rendering photo-realistic images of medical volumetric data from tomography using physically-based Monte Carlo path tracing, a large number of light paths needs to be traced in order to reach a smooth, converged image without Monte Carlo noise. This renders interactivity infeasible as only a single path per pixel and frame can be traced in real-time. As a step towards the second goal (interactivity in terms of high frame rates), we therefore propose Adaptive Temporal Sampling to reduce noise in single-sample path tracing, thus bringing physically-based volume rendering to the interactive domain. To this end, we transfer the idea of temporal antialiasing to volume rendering and enable temporal reprojection for volumetric instead of surface data and Monte Carlo instead of shader noise by introducing a representative depth for each pixel, a combined depth and direction check, an improved weighting scheme for arithmetic averaging and an error accumulation method to downweight less appropriate older samples as ghosting countermeasures from temporal antialiasing cannot be applied. Furthermore, we adaptively determine the number of newly generated samples for each pixel using reprojection information, thus dedicating more computational effort to noisier regions. Building upon this original method, we propose an additional learning-based post-processing pass to eradicate remaining noise. We reach interactive frame rates similar to the ones observed in noisy single-sample rendering and nearly noise-free images in an exemplary VR application. Using the proposed methods both for intuitive interaction (projection-based surgery planning prototype) and for interactivity through drastically reduced frame times (Adaptive Temporal Sampling with learning-based extension), we open the door for a more extensive use of interactive XR-based volume rendering applications for non-radiologists and all kinds of medically inexperienced users.