# Surgical Phase Recognition: From Public Datasets to Real-World Data
## 3 Results
### Surgical Phase Recognition on Our Dataset  
We tested how existing surgical phase recognition models, developed on Cholec80, perform when trained on our dataset. Our aim was not an exhaustive comparison of all models but instead to understand how the specificity of our dataset affects the performance of existing models. We trained state-of-the-art surgical phase recognition models on our dataset. Specifically, we considered two different architectures: MSTCN-based (TeCNO) and transformer-based (Trans-SVNet). In accordance with existing studies on Cholec80, stacking temporal models (MSTCN or Trans-SVNet [15]) on top of Resnet-50 backbone boosted the results on all aggregated metrics (Table 3). In particular, the segment-based metric, which takes into account the temporal fragmentation of the predictions, was almost 50 times greater (\(F1@50\): from 0.83 to 40.68 in the second row). The offline MSTCN-based model (TeCNO-Off) reached a top average accuracy of 84.75%, as well as top segment-based scores (\(F1@50\) and edit score). Yet, the best performance in terms of frame-wise precision, recall, and f1-score was achieved by the online TeCNO model. Looking at the phase-wise performance of the models (Figure 4), we observed the largest variability across models on the minority phases (e.g., _gallbladder drainage_, _liver biopsy_). The performance gap between the models was the greatest in the extremely under-represented phase _liver biopsy_. Interestingly, the online TeCNO model performed better than its offline version (TeCNO-Off) in the minority phases. This was not reflected in the aggregated accuracy metric (Table 3). Similarly, the use of weighted cross-entropy loss to train the Trans-SVNet model was crucial to increase the model recall on the minority phases (e.g., _gallbladder drainage_, _liver biopsy_). The Trans-SVNet model without weighted cross-entropy loss had a higher aggregated metric. In the following, we performed a detailed performance analysis of the online TeCNO model. We focused on this model given its higher performance on the minority phases, the characteristic of our dataset. The high standard deviation for precision, recall, and f1-score (Table 3) hinted at a strong variation in the model performance across phases, as observed during the model comparison (Figure 4). This variation was better reflected in the phase-wise precision/recall plots (Figure 5a). In particular, the model struggled to accurately predict the _gallbladder drainage_, _gallbladder retraction_, _liver biopsy_ and _intraoperative cholangiography_. These four phases were less prevalent in our dataset (see Supplementary Figures S2 and S3 online). Note that there were only five and four test videos for _gallbladder drainage_ and _liver biopsy_, respectively. The remaining phases had much higher precision and recall. Notably, _preparation_, _Calot triangle dissection_, _gallbladder dissection_, _cleaning and coagulation_ and _other procedure_ phases had above 76 percent recall. \begin{table}
\begin{tabular}{l l l l l l l} \hline \hline
**Method** & **Accuracy** & **Precision** & **Recall** & **F1-Score** & **F1050** & **Edit** \\ \hline Resnet-50 & 68.94 \(\pm\) 13.89 & 61.54 \(\pm\) 18.13 & 57.46 \(\pm\) 18.18 & 52.89 \(\pm\) 16.54 & 0.83 \(\pm\) 1.17 & 1.79 \(\pm\) 1.31 \\ TeCNO-Off & **84.75 \(\pm\) 13.49** & 67.89 \(\pm\) 23.23 & 69.27 \(\pm\) 24.09 & 64.02 \(\pm\) 22.53 & **40.68 \(\pm\) 23.75** & **39.01 \(\pm\) 25.3** \\ TeCNO [9] & 82.76 \(\pm\) 14.20 & **69.54 \(\pm\) 18.45** & **70.82 \(\pm\) 20.0** & **64.65 \(\pm\) 19.97** & 21.45 \(\pm\) 18.28 & 18.31 \(\pm\) 16.18 \\ Trans-SVNet [15] & 84.05 \(\pm\) 14.41 & 69.02 \(\pm\) 21.66 & 63.73 \(\pm\) 30.05 & 61.70 \(\pm\) 26.88 & 30.04 \(\pm\) 20.13 & 26.18 \(\pm\) 16.75 \\ Trans-SVNet-WCE & 82.97 \(\pm\) 14.48 & 68.33 \(\pm\) 22.08 & 70.67 \(\pm\) 19.72 & 64.10 \(\pm\) 21.95 & 26.46 \(\pm\) 18.76 & 22.41 \(\pm\) 15.66 \\ \hline \hline \end{tabular}
\end{table} 
Table 3: Model performance (mean \(\pm\) std) using aggregated metrics on our dataset.