# Hands-On Deformation of Volumetric Anatomical Images on a Touchscreen
###### Abstract  
In this work, we propose a novel metaphor to interact with volumetric anatomical images, e.g., magnetic resonance imaging or computed tomography scans. Beyond simple visual inspection, we empower users to reach the visible anatomical elements directly with their hands, and then move and deform them through natural gestures, while respecting the mechanical behavior of the underlying anatomy. This interaction metaphor relies on novel technical methods that address three major challenges: selection of anatomical elements in volumetric images, mapping of 2D manipulation gestures to 3D transformations, and real-time deformation of the volumetric images. All components of the interaction metaphor have been designed to capture the user's intent in an intuitive manner, solving the mapping from the 2D touchscreen to the visible elements of the 3D volume. As a result, users have the ability to interact with medical volume images much like they would do with physical anatomy, directly with their hands. t +
Footnote †: journal: Applied Sciences  
a successful solution for the definition of visible surfaces within the dense 3D dataset. For deformation, on the other hand, there is no effective method for conveying 3D selection and manipulation of volume data in a natural and intuitive manner using a 2D interface. In this work, we propose a metaphor of natural interaction with medical volume images through tangible direct interaction on a touchscreen. We let users deform the anatomy by touching it directly and performing manipulation operations with their fingers, much like they would do with real anatomy. A key ingredient of our method is to map the 2D touchscreen interaction to 3D volumetric deformation through a rigid handle metaphor. Then, we decompose the interaction process into three operations: selection of handles, mapping of handle transformations, and computation of handle-driven deformations. First, in Section 4, we describe an intuitive, user-guided method for the selection of anatomy elements as volumetric handles. We use an intuitive mapping to define selection parameters directly on the touchscreen, based on visible anatomy elements. Furthermore, we apply a region growing method based on visual settings to segment an anatomical element and conform a handle. Second, in Section 5, we describe an optimization-based method for the computation of rigid handle transformations. We define intuitive cues for direct-touch manipulation of the handles on a touchscreen, and we compute corresponding handle transformations by solving efficiently a closed-form optimization problem. Third, in Section 6, we summarize how we drive the deformation of the full anatomy from the transformation of the handle. To this end, we adapt a corotational coarsening deformation method [1] that solves an elastostatic deformation problem on a coarse finite-element mesh, while respecting the material distribution and boundary conditions imposed on a fine mesh, and then applies the deformation to the embedded medical image through volume resampling [2]. We extend the original corotational coarsening deformation method by supporting additional types of high-resolution boundary conditions, to accommodate handle-based manipulation. In our results, such as Figure 1, we show multiple examples where the user deforms and inspects the anatomy in a medical volume image. We demonstrate that our method is applicable to a wide range of anatomical elements, from large organs to fine elements such as blood vessels. The user performs selection and manipulation actions in a natural way, with smooth transitions. Immediate tangible and visual feedback makes interaction and inspection intuitive and effective. We believe that our methods can serve several clinical applications. The most evident one is pre-operative planning, but our conversations with clinical experts also point to potential use for exploration and diagnosis.