# Development of an Augmented Reality Surgical Trainer for Minimally Invasive Pancreatic Surgery
## 2 Materials and Methods  
### Simulator's Architecture  
Targeting minimally invasive pancreatic surgery, the simulator is designed to be included in Step 1 of the medical protocol from Table 1, developed in collaboration with hepato-pancreato-bilary surgeons. Using the simulator, the doctors can visualize and analyze the adequate position of both the patient and the robot in the operating room.  
The simulator is built on a modular architecture that integrates several key components to provide a comprehensive training environment, as can be seen in Figure 1.  
\begin{table}
\begin{tabular}{c c} \hline
**Procedure Step** & **Medical Tasks** \\ \hline \multirow{4}{*}{Step 1: Preplanning} & \multicolumn{2}{c}{Review medical history.} \\  & Plan approach. \\  & 3D organ reconstruction with AR visualization. \\  & Define the patient and robot position. \\ \hline \multirow{4}{*}{Step 2: Preparation} & Induce anesthesia and position the robot. \\  & Calibrate instruments and prepare tools. \\  & CO\({}_{2}\) insufflation and trocar incisions. \\ \hline \multirow{4}{*}{Step 3: Surgical task} & Perform dissection, mobilization, \\  & and tissue resection. \\  & Access the pancreas and adjacent organ retraction. \\  & Seal pancreatic stump, restore digestive tract \\  & through anastomosis if needed, hemostasis, \\  & and insert drains. \\ \hline \multirow{4}{*}{Step 4: Instrument retraction} & Remove instruments, release CO\({}_{2}\), \\  & and undock the robot. \\  & Suture incisions. \\ \hline \multirow{4}{*}{Step 5: Procedure finalizing} & Transfer the patient to intensive care. \\  & Document the procedure \\  & and sterilize all equipment. \\ \hline \end{tabular}
\end{table}
Table 1: Medical protocol for minimally invasive pancreatic surgery.  
Figure 1: The architecture of the simulator.  
The surgical simulator integrates a surgical parallel robot developed for minimally invasive pancreatic surgery, detailed in Section 2.2, with an Omega 7 haptic device [10] used in the simulator's force-feedback system. The Omega 7 haptic device (provided by Force Dimension, Allee de la Petite Prairie 2, CH-1260 Nyon, Switzerland) is a high-precision, force-feedback interface designed to provide realistic tactile sensations and precise control for virtual and robotic environments, featuring six degrees of freedom for advanced simulation applications [11].  
To predict the force used during surgical procedures, the system uses a convolutional neural network (CNN) model. The trained model estimates the force by analyzing visual data from the simulation without the use of physical sensors. During surgical procedures, the simulator can give precise and dynamic force feedback thanks to this model.  
The simulator is developed using the Unity Engine (Editor version 2021.3.16f1), utilizing its physics engine for simulating interactivity and physical behavior [12]. It is targeted for AR environments and particularly for the Microsoft HoloLens 2 device (provided by Microsoft Corporation One Microsoft Way, Redmond, WA 98052-6399, USA), [13]. With the aid of HoloLens 2, the user can visualize 3D organs as well as the robot's structure, enabling the users to interact with them in real time. By tracking spatial mapping, HoloLens precisely determines the direction and movement of surgical tools relative to the patient's body and its internal structures. This integration enhances the surgical process in the preplanning stage, enabling the visualization of the anatomy and the robot's movements during the procedure. By simulating the elasticity of pancreatic tissue, the simulator provides users with feedback on the force used during interactions.