# Surgical Phase Recognition: From Public Datasets to Real-World Data
## 3 Results
### Surgical Phase Recognition on Our Dataset
The performance of only the Resnet50 backbone is shown in the first row. Stacking the MSTCN on top of Resnet50 boosted the phase segmentation performance (the last two columns in the second row). The std is computed across the videos for _accuracy_, _F1@50_ and _edit score_; across the phases for _precision_, _recall_, and _f1-score_. Figure 4: Comparative phase-specific performance analysis of the models on our dataset. For the minority phases, _gallbladder drainage_, and _liver biopsy_, the **TeCNO**[9] model showed the best performance among others. The performance gap was highly significant for _liver biopsy_, which is an extremely under-represented phase in our dataset, between TeCNO and Trans-SVNet [15] (28 times and 1.57 times greater for recall and precision, respectively). Figure 5: Phase-specific performance analysis of the TeCNO model on our dataset. **(a)** Illustration of phase level precision and recall averaged across videos. One can notice the strong variation in performance between phases. **(b)** The confusion matrix illustrates the confusion between phases. Every sample (frame-wise prediction and ground truth) contributed equally during the computation. Confusion between temporally closer phases was more prevalent than in temporally distant phases. The confusion matrix emphasized the phase-specific errors of the model (Figure 5b). Importantly, confusion with temporally closer phases was more prevalent than with temporally far phases. For example, _clipping_ was mostly misclassified as _cutting_ and _cutting_ as _gallbladder dissection_; in both cases these are their natural successive phase (Table 2). Similarly, _gallbladder packaging_ was mostly misclassified as its most likely successor: _cleaning and coagulation_. Phase-wise performance varied strongly across videos (Figure 6). While the precision/recall averaged across all videos was rather high for most of the phases, some videos displayed poor performance for particular phases. The _gallbladder drainage_, _gallbladder retraction_, _liver biopsy_, and _intraoperative changigography_ phases had the highest variability. Beyond frame-wise metrics, we assessed the model performance using segment-based metrics (Figure 7). The segmental f1-score averaged over phases, _F1@k_, depended on the overlapping _IoU_ threshold \(k\). We measured F1@25 = 25.48 \(\pm\) 18.82, F1@50 = 21.45 \(\pm\) 18.28, F1@75 = 15.42 \(\pm\) 13.96 and the edit score S\({}_{\text{edit}}\) = 18.31 \(\pm\) 16.18. Similar to frame-wise metrics, the high standard deviation underlined a strong disparity in the model performance across videos. The lower segment-based scores compared to the frame-wise f1-score emphasized an over-segmentation of the predictions. Frame-wise f1-score, segment-based edit score, and F1-score were correlated (Figure 7a). Nevertheless, segment-based scores covered a larger range than the frame-wise f1-scores. As an example, _video12_ in Figure 7a had high frame-wise f1-score (84.69%), but low _F1@50_ (7.36%) and edit score (6.74). Looking closer at its phase labels, this video presented two long phases, e.g., _Calot triangle dissection_ and _gallbladder dissection_. These two phases were recognized with low frame-wise error rates which increased the model accuracy. Yet, predictions had lots of over-segmentation errors (fluctuating thin bars); this decreased the performance in terms of segment-based metrics. Figure 6: Phase-specific performance of the TeCNO model showed high variation across videos on our dataset.