# Breakthrough and Challenging Application: Mixed Reality-Assisted Intracardiac Surgery
## 2 Materials and Methods
### Implementation of the Application  
_Materialise Mimic_ allows to export the 3D model in an STL format. However, the conversion of the file from an STL to FBX format was necessary in order to integrate the model into the 3D environment engine and to enhance the object manageability. For this purpose, Blender 4.3, an open-source 3D creation software, was selected. Blender was also used to make other adjustments to the model. In particular, ensuring the optimal sizing, orientation, and alignment of the components with the desired model configuration is important. In fact, the 3D model of the heart obtained from the segmentation phase consists of nine 3D objects corresponding to the different functional parts of the heart. All the 3D objects have their reference systems in the same position, the center of the global heart mesh. Blender was indispensable for centering the local reference system of each 3D object in the geometric center of its mesh. This step is fundamental for the realization of the interactions (including translation, rotation, and scaling) in the 3D environment engine. In fact, once imported into Unity, the interactions with each object would take place with respect to the local system of the model. In order to be able to structurally reconstruct the global heart according to anatomy, the single 3D object was repositioned with the original coordinates in relation to the global heart system obtained from Blender. In this way, the conformation of the heart was preserved. Subsequently, the finalized object in an FBX format, enriched with distinct color assignments for individual anatomical segments, was generated. The 3D model is then imported into the Unity platform (Unity version 2020.3.30f1, Visual Studio 2019), in which the MR experience is implemented for the HoloLens 2 headset (HL2). To implement interactions specific of MR, the Mixed Reality toolkit (MRTK) from Microsoft was integrated into the Unity project by configuring it to use the HL2 visor. In addition, the UnityVolumeRendering asset was integrated into the project to enable DICOM image display and 3D rendering. Meanwhile, the object with the \"clipping plane\" material provided by Unity was used to make a plane that could dissect the 3D object to allow the internal observation of the mesh. When developing an app for HL2, the size limit of 500 MB for a 3D object that HL2 can process should be taken into consideration. _Materialise_ can segment and then realize the mesh according to the level of details set by the operator. With a hundredth of a millimeter of detail of the order of magnitude, the limit set by the HL2 had been reached by the heart firstly segmented. In this context, the application cannot be built and the device should be used in the Holographic Remoting mode (i.e., connected to the computer via cable so as to utilize the computer's calculation capacity). To match the limitation in a model file dimension supported by the HL2 and to allow the MR experience to be more fluent and stand-alone, the 3D model was smoothed. The details for the segmentation were set to a lower order of magnitude. Once the Unity project configuration is completed, all necessary parameters must be set to compile and deploy the application to the HL2. A comprehensive workflow of the methodology is reported in Figure 1, which summarizes the main steps from the DICOM images to the MR experience realization. The primary function of the MR application is to enable the exploration of a holographic 3D model of the heart reconstructed from CT images. This capability meets the surgeon's requirement for a 360\({}^{\circ}\) observation of each anatomical site and provides a sense of depth that is otherwise unattainable. The user can navigate within the scene and interact with the elements in the scene effortlessly.