# Surgical Tool Detection in Open Surgery Videos
## 5 Results  
The performance of the baseline models for the validation and test sets is represented in Table 3. Overall, Faster-RCNN, which is a two-stage detector, clearly outperforms RetinaNet, which is a single-stage detector. In general, two-stage models are considered to be able to provide better detection accuracy. We can find this tendency in our results. The average precision (AP) of the test set was found to be much lower than that of the validation set for all models. This is caused by the way of performing a split. Because we split training, validation, and test sets with _video-based split_ considering the practical scenario, the distribution of categories in the three-set is different. Therefore, a gap in AP between validation and test sets occurs. We conducted experiments with different backbones to explore the dependency of performance from the backbone. Faster-RCNN with a ResNet-X101 backbone presents the highest AP, and RetinaNet with a ResNet-50 backbone presents the lowest for both validation and test sets. The increase in depth and parameters slightly improves the AP in Faster-RCNN. On the other hand, for RetinaNet, the increase in depth and parameters have little effect on the AP. The AP per category given in Tables 4 and 5 shows that each surgical tool was detected with varying degrees of accuracy. This is mainly because of heavily imbalanced data, which naturally incur bias in the model. As a whole, the greater the number of annotated instances we can get, the better AP. For example, the baseline models successfully detected _Mouth Gag_, _Needle Holders_, _Suction cannula_, and _Tweezers_ categories, which have a relatively large number of instances. On the other hand, they fail to detect _Chisel_, _Hammer_, _Pen_, and _Suction Tube_ categories, which have a small number of instances. The class imbalance is the main reason why there is a large gap between the AP of validation data and test data. As shown in Table 1, compared to the validation data, the test data includes the category in which the number of instances is relatively small. This may result in better AP of validation sets rather than that of the test set. \begin{table}
\begin{tabular}{c c c c c c c c c} \hline \hline \multirow{2}{*}{**Model**} & \multirow{2}{*}{**Backbone**} & \multirow{2}{*}{**Iters**} & \multicolumn{3}{c}{**Validation Set**} & \multicolumn{3}{c}{**Test Set**} \\ \cline{3-8}  & & & \(AP\) & \(AP_{50}\) & \(AP_{75}\) & \(AP\) & \(AP_{50}\) & \(AP_{75}\) \\ \hline \multirow{3}{*}{Faster-RCNN} & ResNet-50 & 40K & 48.9 & 63.6 & 55.0 & 27.2 & 41.7 & 29.7 \\  & ResNet-101 & 37K & **51.3** & **66.0** & 56.6 & 28.6 & 42.9 & 32.0 \\  & ResNet-X101 & 40K & **51.3** & 64.5 & **57.0** & **29.7** & **44.2** & **32.8** \\ \hline \multirow{3}{*}{RetinaNet} & ResNet-50 & 41K & **48.1** & **62.4** & **52.2** & 26.7 & 41.0 & 29.1 \\  & ResNet-101 & 42K & 48.0 & 61.8 & 52.1 & **27.8** & **41.5** & **29.9** \\ \hline \hline \end{tabular}
\end{table} 
Table 3: \(AP\), \(AP_{50}\) and \(AP_{75}\) per model for validation and test sets. The best results are highlighted in bold. There are cases where the models fail to detect the categories that have profound instances because of their challenging characteristics. First, the deformable surgical tool category is difficult to be detected. _Gauze_ has the second largest number of instances in the dataset, but the AP of the best baseline model, Faster-RCNN ResNet-X101, is 15.7. This is much worse than the _Mouth Gag_ category, which has almost the same number of instances in the dataset. The reason for this may be that the _Gauze_ category is deformable and its appearance drastically changes every scene. _Suture, Suture Needle_ also has this characteristic. Second, the surgical tool category that has a variety of appearances through instances is difficult to detect. _Retractor_, which is used to separate the edges of a surgical incision, has a different shape and size depending on the situation.