# Development of an Augmented Reality Surgical Trainer for Minimally Invasive Pancreatic Surgery
## 2 Materials and Methods
### Force Prediction and Feedback  
For the identification of the most suitable Artificial Neural Network, a literature survey was performed, analyzing multiple ANNs with respect to their performance in processing real-time images, with the aim of minimizing the network training time while preserving an output accuracy of over 90%. Studies included networks combining RGB images and robot state data [15], modified Inception ResNet V2 for force-feedback estimation [16], and RCNNs integrating CNNs and LSTMs to capture spatial and temporal dynamics for tensile force estimation in robotic surgery [17]. The image-based model for force prediction is integrated into a force-feedback system to estimate forces applied to biological tissues. This model predicts the interaction forces based on visual input, which are then translated into haptic feedback for robotic control in minimally invasive procedures. By replacing traditional sensor-based methods, this approach eliminates the need for physical sensors. The model is based on EfficientNetV2B0, the smallest model from the EfficientNetV2 family, with these models being recognized for their minimal resource usage [18]. The network uses depth-wise separable convolutions to improve computational efficiency and global average pooling (GAP) operations to reduce each feature map to a single scalar value by computing the average of all spatial locations in the feature map. This replaces the need for fully connected layers. Additionally, a dense layer with 1024 neurons is included to help the model recognize more complex patterns. To improve accuracy, the model also includes Squeeze-and-Excitation (SE) blocks for enhancing the network's ability to focus on the most informative feature; residual connections for improving gradient flow, faster training, deeper network formation, and performance; and dropout layers for preventing the overfitting of the model. For this study, the model was initially trained using images of animal pancreas tissue. The data were collected by recording videos of tissue responses under applied forces ranging from 0 to 1 N using a Kuka iiwa LBR 7 R800 robot (provided by KUKA AG, KUKA Aktiengesellschaft, Zugspitzstrasse 140 86165 Augsburg, Germany), equipped with a Robotiq FT 300 force-torque sensor (provided by Robotiq, Robotiq Inc., 966 Chemin Olivier, Suite 500 Levis, QC, G7A 2N1, Canada) and an endoscopic camera (provided by Olympus Corporation, 3500 Corporate Parkway, Center Valley, PA 18034, USA) (Figure 5). The force sensor provided precise force measurements, while the camera captured HD images from multiple angles (frontal, lateral, and oblique) to ensure a diverse dataset. Each video lasted about 30 s, with force levels consistently maintained. Frames were extracted from the videos using the OpenLabeling tool, resulting in 2064 images. These images were labeled with their corresponding force values and stored in a CSV file for model training. The resulting dataset was split into 60% for training, 20% for validation, and 20% for testing. The model predicted forces in the range of 0 to 1 N, in increments of 0.1 N, with no visible deformation in the tissue beyond this threshold. The training, which utilized the PyTorch library (Python 3.13, torchvision 0.11.0), was performed on a 12 th Gen Intel(r) Core(tm) i9-12900K CPU. A batch size of 32, a learning rate of 0.00001, and 50 epochs were used to train the CNN on the CPU. When tested on the test dataset, the model's performance resulted in a mean absolute error (MAE) of 0.0016 and a loss of 0.0011. This shows that there was an average deviation of around 0.0016 N between the actual measured forces and the expected values. Following the initial training, the model was adapted to process Unity-generated synthetic images as well. Specifically, the deformable pancreatic tissue was tested to known forces ranging from 0 to 1 N in Unity, and the resulting images were used for additional training.