# Instrument Detection and Descriptive Gesture Segmentation on a Robotic Surgical Maneuvers Dataset
## 3 Results
### Results for Gesture Segmentation  
This section presents the results of the gesture segmentation network described in Section 2.5.1. We present the results for the two categories of annotations of ROSGMAG40 dataset and the four experimental setups: leave-one-user-out (LOUO), leave-one-supertrial-out (LOSO), leave-one-psm-out (LOPO) and leave-one-trial-out (LOTO). Matlab 2023b software was used to implement the RNN model. Training of the network was performed with the Adam optimization algorithm with an initial learning rate of 0.001. We used a batch size of 8 for 60 epochs (240 iterations). A higher number of training epochs resulted in worse metrics due to overfitting of the network. The total number of observations of the network is 72,843. The size of the test data varies depending on the cross-validation method evaluated, ranging from 15,347 to 36,215. The detection time of the network ranges from 0.0021 to 0.0035 s.  
Table 9 shows the results for LOUO cross-validation scheme. Observing these results we can deduce that the model suffers from a lack of generalization when a user is left out of the training. We can observe that the mean average precision varies from 39% for user _X01_ to 64.9% for _X02_. Though the experimental protocol is the same for all users, these results suggest that the performance is highly dependent on the skill of each user. Moreover, we left freedom to complete the tasks in a random order, i.e., each user had to place six peas on top of the pegs and transfer the six colored sleeves from one side to the other of the pegboard, but the order in which they had to complete the task was not predefined. Table 10 shows the results for the LOSO cross-validation scheme. As in the previous case, the model has difficulties extrapolating the features learned by the network to a task that has not been seen before. Table 11 shows the results for the LOPO cross-validation scheme. In this case, we can observe how the network has poor results for the low-level FGD annotations, but good results for high-level MD annotations, reaching a maximum mAP of 67.5%. Table 12 shows the results for the LOTO cross-validation scheme. This is the experimental setup with more variety of the training data, thus it has the best results of the four cross-validation schemes. Using the test folds of test data 1, which offers the most variety of observations to the network, we reach a 64.65% and 71.39% mAP for the segmentation of PSM1 and PSM2 gestures, respectively, using MD annotations. However, for the segmentation of PSM1 gesture, we reach a maximum precision of 71.39% using test data 5 folds \begin{table}
\begin{tabular}{c c c c c} \hline \hline
**Supertrial Left** & **PSM1 mAP** & **PSM2 mAP** & **PSM1 mAP** & **PSM2 mAP** \\
**Out** & **(MD)** & **(MD)** & **(FGD)** & **(FGD)** \\ \hline Pea on a peg & 56.15\% & 56\% & 46.36\% & 46.67\% \\ Post and sleeve & 52.2\% & 51.9\% & 39.06\% & 43.38\% \\ \hline \hline \end{tabular}
\end{table} 
Table 10: Results for Leave-One-Psm-Out (LOPO) cross-validation scheme. \begin{table}
\begin{tabular}{c c c c c} \hline \hline
**Dominant PSM** & **PSM1 mAP** & **PSM2 mAP** & **PSM1 mAP** & **PSM2 mAP** \\
**(MD)** & **(MD)** & **(FGD)** & **(FGD)** \\ \hline PSM1 & 56.53\% & 67\% & 24.11\% & 37.33\% \\ PSM2 & **65.7\%** & **67.5\%** & 52.47\% & 54.68\% \\ \hline \hline \end{tabular}
\end{table} 
Table 11: Results for Leave-One-Psm-Out (LOPO) cross-validation scheme. \begin{table}
\begin{tabular}{c c c c c} \hline \hline
**User Left Out Id** & **PSM1 mAP** & **PSM2 mAP** & **PSM1 mAP** & **PSM2 mAP** \\
**(MD)** & **(MD)** & **(FGD)** & **(FGD)** \\ \hline X1 & 48.9\% & 39\% & 46.26\% & 23.36\% \\ X2 & 58.8\% & **64.9\%** & 48.16\% & 51.39\% \\ X3 & 50.4\% & 64.2\% & 39.71\% & 51.24\% \\ X4 & **63.0\%** & 61.2\% & 54.05\% & 49.08\% \\ X5 & 54.6\% & 53.6\% & 52.34\% & 52.7\% \\ Mean & 55.14\% & 56.58 & 48.01\% & 45.55\% \\ \hline \hline \end{tabular}
\end{table} 
Table 9: Results for Leave-One-User-Out (LOUO) cross-validation scheme. and FGD annotations.