ProstaTD: Bridging Surgical Triplet from Classification to Fully Supervised Detection
Authors: Yiliang Chen
Zhixi Li
Cheng Xu
Alex Qinyang Liu
Ruize Cui
Xuemiao Xu
Jeremy Yuen-Chun Teoh
Shengfeng He
Jing Qin
Venue: ICLR 2026 Poster (2026)
Keywords: Surgical Triplet
Endoscopy
Detection
Evaluation
Primary area: applications to computer vision, audio, language, and other modalities
TL;DR: We introduce a new supervised surgical triplet detection task, a novel dataset, two innovative annotation tools, a comprehensive evaluation toolkit, a new benchmark, and a tailored method.

Abstract: Surgical triplet detection is a critical task in surgical video analysis, with significant implications for performance assessment and training novice surgeons. However, existing datasets like CholecT50 lack precise spatial bounding box annotations, rendering triplet classification at the image level insufficient for practical applications. The inclusion of bounding box annotations is essential to make this task meaningful, as they provide the spatial context necessary for accurate analysis and improved model generalizability. To address these shortcomings, we introduce ProstaTD, a large-scale, multi-institutional dataset for surgical triplet detection, developed from the technically demanding domain of robot-assisted prostatectomy. ProstaTD offers clinically defined temporal boundaries and high-precision bounding box annotations for each structured triplet activity. The dataset comprises 71,775 video frames, collected from 21 surgeries performed across multiple institutions, reflecting a broad range of surgical practices and intraoperative conditions. The annotation process was conducted under rigorous medical supervision and involved more than 60 contributors, including practicing surgeons and medically trained annotators, through multiple iterative phases of labeling and verification. To further facilitate future general-purpose surgical annotation, we developed two tailored labeling tools to improve efficiency and scalability in our annotation workflows. In addition, we created a surgical triplet detection evaluation toolkit that enables standardized and reproducible performance assessment across studies. ProstaTD is the largest and most diverse surgical triplet dataset to date, moving the field from simple classification to full detection with precise spatial and temporal boundaries and thereby providing a robust foundation for fair benchmarking.

Published as a conference paper at ICLR 2026
PROSTATD: BRIDGING SURGICAL TRIPLET FROM CLASSIFICATION TO FULLY SUPERVISED DETECTION
Yiliang Chen', Zhixi Li Cheng Xu' Alex Qinyang Liu?, Ruize Cui
Xuemiao Xu", Jeremy Yuen-Chun Teoh", Shengfeng H Jing Qin
Schooo of Nursing, The Hong Kong Polytechnic University
2Nanfang Hospital, Southern Medical University
DDepartment of Surgery, The Chinese University of Hong Kong
Sooth China University of Technology Singapore Management University
ABSTRACT
Surgical triplet detection is critical task in surgical video analysis, with significant implications for performance assessment and training novice surgeons. However, existing datasets like CholecT50 lack precise spatial bounding box annotations, ren- dering triplet classification at the image level insufficient for practical applications. The inclusion of bounding box annotations is essential to make this task meaningful, they provide the spatial context necessary for accurate analysis and improved model generalizability. To address these shortcomings, we introduce ProstaTD, a dataset for surgical triplet detection, developed from the technically demanding domain of aabot-assisted prostatectomy. ProstaTD offers clinically defined temporal boundaries and high- srcc-precision bounding box annota- tions for each structured triplet activity. The dataset comprises 71,775 video frames, collected from 21 surgeries performed across multiple institutions, reflecting a broad range of surgical practices and intraoperative conditions. The annotation process was conducted under rigorous medical supervision and involved more than 60 contributors, including practicing surgeons and medically trained annotators, through multiple iterative phases of labeling and verification. To further facilitate future general-purposs. surgical annotation, we developed two tailored labeling tools to improve efficiency and scalability our annotation workflows. In addition, we created surgical triplet detection evaluation toolkit that enables standardized and reproducible performance assessment across studies. ProstaTD the largest and most diverse surgical triplet dataset to date, moving the field from simple cl classification to full detection with precise spatial and temporal boundaries and therehy providing robust foundation for fair henchmarking. Our implementation can be found at
large-scale,
INTRODUCTION
Surgical triplet detection is fundamental task in surgical data science. It involves identifying <instrument, verb, target>> triplets, which represent the instrument in use, the action performed, and the anatomical target being acted upon, from each frame of surgical videos. This task supports the development of context-aware decision support systems and contributes to improved surgical safety, procedural standardization, and operational efficiency (Murali et al., 2023; Padoy et al., 2012).
The field was initiated with the release of CholecT40 (Nwoye et al., 2020), the first benchmark for surgical triplet classification in laparoscopic cholecystectomy. provided frame-level triplet class labels for training models and was later extended through Cholec (Nwoye et al., 2023a) and CholecT50 (Nwoye et al.. 2022), which included 5 and 10 additional surgical videos. respectively. Currently, CholecT50 is the most widely adopted dataset for the surgical triplet analysis (Xi et al., 2022; Chen et al., 2023; Gui Wang, 2024). While the CholecTriplet 2022 Challenge (Nwoye et al., 2023b) advanced the field by framing surgical triplet detection task involving both recognition
"Corresponding authors: shengfenghe@@ sww..du.ss........ and harry polyu.edu.hk

---

Published as a conference paper at ICLR 2026
(8)
(b)
(c)
(d)
Figure 1: Examples labeled surgical triplets in our ProstaTD dataset. ProstaTD offers clinically defined temporal boundaries and high-precision bounding box annotations for each structured triplet activity, curated from diverse, multi-instittiinall surgical sources, establishing a strong foundation for precise and robust triplet detection. Here, (a) and (b) are examples from our in-house collected videos called Prostate Wellness Hope (PWH) dataset, (c) from the PSI-AVA dataset, and (d) from the ESAD dataset. All videos were comprehensively annotated by our team.
and spatial localization, it relied primarily on weak supervision with frame-level triplet class labels from the C5olecT50 dataset. This constraint limited the ability to achieve precise spatial localization, often resulting in ambiguous triplet predictions.
Despite these contributions, CholecT50 (Nwoye et al., 2022) has three critical limitations:
No bounding box annotations: CholecT50 provides only class labels without spatial localization, which restricts the task to weakly supervised settings and limits localization accuracy.
Unelear and inconsistent temporal boundaries: Although the Cholec paper briefly states that its annotators marked the start and end of cach triplet, the procedure lacks detailed specification. remains ambiguous whether a new triplet activity begins with instrument entry or target contact, and whether it ends with instrument disengagement, change in the target being operated on, instrument exit. Such uncertainty leads to inconsistent labeling and limits the model's ability to capture temporal dynamics and instrument-target interactions.
3. Lack of diversity in data sources: C50lecT50 is collected at single institution, limiting variation workflow and instrument appearance. Differences in manufacturers, as well as surgeons' habits and educational backgrounds, produce rare triplets that often missing, causing models trained on this dataset to overfit to local style and generalize poorly to other clinical settings.
To address these limitations, we develop the first open-source annotation tools specifically designed for surgical triplet annotation. Building upon this, we introduce ProstaTDD, the first scale surgical triplet detection dataset encompassing full-length surgical procedures with precise multi- instrument localization. ProstaTD enables simultaneous and reliable recognition and localization of surgical triplets, a capability essential for modeling instrument- tissue interactions and developing intelligent systems for intraoperative guidance. The dataset includes 21 robot-assisted prostatectomy videos, totaling 71,775 annotated frames. These were sourced from three heterogeneous domains: 4 videos from the publicly available ESAD (Bawa al., 2021), from the PSI-AVA (Valderrama et al., 2022) dataset, and 9 newly acquired from our in-house dataset termed Prostate Wellness Hope (PWH) dataset. Compared to existing cholecystectomy datasets. ProstaTD offers higher instrument concurrency, greater anatomical complexity, and increased domain diversity. These characteristics better reflect the variability and challenges of real-world high-complexity surgical environments. Each frame in ProstaTD is annotated with structured triplet information, including precise bounding boxes for all visible instruments, detailed action (verb) labels, and anatomical target identifiers. All annotations are constrained within expert-defined temporal boundaries curated by experienced urologists, ensuring both clinical accuracy and temporal consistency. Fig. showcases representative examples of the annotated triplets in ProstaTD.
To enable comprehensive and efficient comparisons, we develop an evaluation toolkit and conduct extensive benchmarks using sate-off-the-rr models. We also propose a tailored baseline method that applies a distillation mechanism at the instance level to mitigate triplet imbalance and facilitate fair comparison in future work. Results show that the fine- men-iiaieed spatial annotations in ProstaTD signifi- cantly improve the performance and robustness of models for surgical triplet detection. ProstaTD thus provides comprehensive foundation for advancing surgical video analysis.
2

---

Published as a conference paper at ICLR 2026
Table 1: Comparison of existing surgical datasets with ProstaTID, highlighting the unique attributes required for surgical triplet detection. LC and RARP denote Laparoscopic Cholecystectomy and Robot-Assisted Radical Prostatectomy. Full BBox indicates that every frame is fully annotated with bounding boxes. Complete Surgery denotes that the dataset as whole provides coverage for the entire surgical procedure. Triplet Boundary refers the temporal start and end range of a complete surgical triplet. Multiple Sources indicates that the data is collected from various hospital.
Attributes
Statisties
Dataset
Task
Supervised Full Triplet-like
Triplet
Multiple Complete No.
No.
Detection BBox Structure Boundary Sources Surgery Instances
Triplets
Cholec80-locations
LC
6,471
CholecTrack2)
LC
65,200
ESAD
RARP
/
46,753
PSI-AVA
RARP
5,804
CholecQ
LC
/
14,480
17
C45
LC
146,394
100
CholecT50
LC
161,988
100
ProstaTD (Ours)
RARP
/
196,490
89
In summary, contributions are fourfold:
We introduce a new task with a new dataset for fully supervised surgical triplet detection at the procedure level. To the best of our knowledge, our ProstaTD is the largest surgical dataset with instance-level annotations.
We construct a multi-insticutional surgical triplet dataset featuring detailed labels (precise bounding boxes and standardized triplet boundaries) in more complex surgical scenarios.
We release two open-source annotation tools, which are the first specifically tailored for surgical triplet annotation, along with an open-source evaluation toolkit for benchmarking surgical triplet detection, providing foundation for surgical triplet analysis across diverse surgical procedures.
We introduce the first benchmark for fully supervised surgical triplet detection, providing our tailored method baseline for comparison.
RELATED WORK
While many surgical video datasets have emerged, most are not designed to support triplet-based reasoning. Datasets such as Cholec80-locations (Shi et al., 2020) and CholecTrack (Nwoye al., 2025) provide bounding box annotations but focus primarily on instrument tracking, without structured action- target associations required for triplet detection. A few datasets offer partial triplet-style annotations but are limited in scope, accessibility, or domain relevance. For instance, MISAW (Huaulmé al., 2021) is restricted simulated anastomosis tasks. RLLS-12M (Zhao et al., 2022), Cataract (Lin et al., 2022), and PhacoQ (Lin et al., 2024) are not publicly available, making comparative analysis infeasible. CholecQ (Lin al., 2024) is a notable exception, offering bounding box annotations for triplet-like structures. However, it is constrained to three-second clips from Cholec80 (Twinanda et al., 2016), resulting in high frame redundancy and limited temporal diversity. Consequently, their triplet boundaries can't It cover complete action, so the dataset does not qualify as resource for proper triplet action modeling. Moreover, this limitation also results the dataset lacking full procedure coverage and containing only 17 unique triplet types, far fewer than the 100 in C50 or the 89 our proposed ProstaTD.. These limitations render CholecQ a toy-scale dataset that is inadequate for robust real-world triplet modeling (see Table 1).
To our knowledge, CholecT45 and CholecT50 were the only datasets used for surgical triplet research at the procedure level prior to Pro They originate from subset of cholecystectomy videos in Cholec80, and comprise 146,944 and 161,988 annotated triplet instances across 100 classes, respectively. However, the publicly available labels provided only at the frame level and do not include spatial bounding boxes or explicitly defined temporal boundaries for triplets. Although bounding boxes exist for the test in the TholecTrip1et222 challenge (Nwoye al., 2023b),

---

Published as a conference paper at ICLR 2026
21 Videos Data Sources)
Our PWH (9 vids)
PSI-AVA vids)
ESAD (4 vids)
7 Instruments
Scissors
Forceps
Aspirator
Needle Driver
Grasper
Clip Applier
Endobag
Actions
Coogullate
Dissect
Grusp
Bag
Suck
Clipp
10 Targets
Bladder
Catheter
Seriinal Vesicle
Prostate
Fasclas
Gauze
Endobag
Thread
Fluid
Null
71,775 Frames
196,490 Detection Instances
Triplet 1
grasper, retract, prostate
(cxy. cy W.
Escissors, cut, bladder>
ECTAAA,, Maddor
Triplet 2
(cXp. Cy2, W. h)
89 Triplet Categories
<forceps, retract, bladder> (cx, yy h)
Triplet 3
Figure 2: Overview of the ProstaTD dataset structure. The dataset consists of 21 videos curated from three sources: our in-house PWH, PSI-AVA (Valderrama et al., 2022), and ESAD (Bawa et al., 2021). All videos were comprehensively annotated by our team, covering 7 instruments, 10 actions, and 10 targets. total, the dataset contains 71,775 frames, 196,490 annotated instances, and 89 surgical triplet categories, with each frame labeled using precise bounding boxes.
the test annotations are not publicly accessible. Participants were required to submit predictions for server-side evaluation to obtain results, without direct access to the ground truth. Therefore, CholecT45 and CholecT50 support only surgical triplet classification rather than triplct detection.
The two prostatectomy resources, ESAD (Bawa al., 2021) and PSI-AVA (Valderrama et al., 2022), provide RARP videos, but their original annotations
not designed for surgical triplet detection. ESAD includes full bounding boxes, yet many are coarse or group multiple instruments within single bounding box, while PSI-AVA contains only sparsely annotated frames rather than dense frame-by-frame labels. As for other public prostatectomy datasets such as GraSP (Ayobi et al., 2025), TAPIS (Ayobi al., 2023), and SAR-RARP50 (Psychogyios al., 2023), they share substantial overlap with PSI-AVA videos and were therefore not incorporated into our ProstaTD dataset.
The proposed ProstaTD addresses these limitations and is specifically designed for triplet detection in high-complexity procedures. It is substantially larger than existing datasets, featuring 196,490 annotated surgical triplet instances (see Fig. 2). addition, it offers dense spatial annotations. clinically standardized temporal triplet boundaries. and data collected from multiple institutions. These attributes enable precise localization, consistent temporal labeling, and broad generalizability. which together support more robust and structured analysis of surgery.
3 PROSTATD DATASET
3.1 PROBLEM DEFINITION OF SURGICAL TRIPLET DETECTION
Given video dataset V = (V1, V2. ... Vn) of laparoscopic surgeries, each video V; contains a sequence frames
(Fi, F2...., F.., F. Each frame F, may include multiple instrument instances, annotated with bounding box B {(cz, Cys h)], where (cz, (cy,Cy) is the center, and w. h denote width and height. Each instance has a triplet class label C E (C1, C2.... C.v }. representing specific instrument-actin aottt-actio--aattt target combination, with N the total number of triplets defined by the task. The goal of surgical triplet detection is to identify all valid triplets T, = {C,, Ca...., ... C₁ in each frame F. providing both the spatial location and the semantic label for each interaction.
4

---

Published as a conference paper at ICLR 2026
3.2 PROSTATD DATA FORMAT
For benchmarking, we release annotations in both COCO and YOLO formats. In the YOLO version (see Fig. 2). each annotated instance is represented as (tri, i, a. t, Cz. Cys W. h), where tri, a, denote the triplet, instrument, action, and target identifiers, and (Cz, Cy: w, h) denote the bounding box center, width, and height. This representation follows the standard convention commonly adopted in YOLO-series models. In addition, we provide the COCO format, where (ar, y, h) specifies the top-left corner, width, and height, which is widely used by other detection frameworks. By offering both formats, we ensure compatibility with broad range of models and evaluation pipelines.
3.3 DATA COLLECTION
Our dataset, illustrated in Fig. integrates three sources: in-house PWH Dataset, the publicly available PSI-AVA dataset and the ESAD dataset. Here, ESAD is released under the CC-BY-NC-SA 4.0 license and PSI-AVA is hosted by MIT. The PWH videos were recorded with institutional ethical approval, and the final release of our dataset will also comply with the CC-BY-NC-SA-4, license. Notably, we did not utilize any existing annotations from ESAD or PSI-AVA, even when they were partially relevant to our task. For further details on the original annotations of these datasets, please refer to Appendix A. This decision was driven by differences in data sources and the need for a unified annotation protocol to ensure consistent labeling of surgical actions, targets, and bounding boxes. Accordingly, guided by the expert insights of our team's 's urologist, we developed and strictly followed our own annotation criteria throughout the process.
3.4 ANNOTATION DESIGN
Categories Design. As shown in Fig. 2, our annotation schema comprises 7 surgical instruments, 10 actions, and 10 targets. These categories were determined through consensus among ten professional surgeons, with some labels merged to ensure consistency and reduce redundancy. For details on how labels were merged, please refer to Appendix A.4, For surgical instrument bounding boxes, we adopted consistent strategy: for instruments with well-defined contours, only the head and joint regions were enclosed, whereas instruments with indistinct boundaries, such as "aspirator", were annotated using full-enclosure bounding boxes. These annotation rules were consistently applied across all instrument categories, ensuring uniform quality and reducing ambiguity in model training.
Temporal Boundaries of Triplets. Unlike CholecT50 (Nwoye et al., 2022), we established unified guidelines to define the temporal boundaries of surgical triplets. Importantly, triplet starting points were not annotated based solely on the presence of surgical instrument in the frame, as such an approach resembles instance-level step recognition and lacks alignment with clinical assessment of surgical skill. Following collaborative discussions, we categorized triplets into three types: continuous actions, momentary actions, and null actions. For continuous actions, such lymph node dissection along the iliac vessels, the action was defined to start only when the surgical instrument made contact with was extremely close to the target, and to end when the instrument departed from the same target for more than seconds. For momentary actions, such as cutting sutures with scissors, the temporal window defined more flexibly: annotations included up to 2 seconds before and after the moment of instrument-targer contact. The remaining cases were designated as null actions, where the triplet corresponded to instrument being stationary or moving without meaningful interaction. This clinically informed strategy ensures temporally precise and semantically meaningful annotation of triplet actions. better aligning with real-world surgical practices. Owing these detailed rules, we were able to consolidate annotations from multiple annotators with consistent quality, which is further supported by the inter-rater agreement results Section 5.
3.5 ANNOTATION PROCESS
Labeling Tools. We developed two dedicated annotation applications for ProstaTD: Triplet-labelme, designed for single-frame triplet editing (instrument, action, target) with bounding box assignment, and SurgLabel, tailored for high-throughhut batch labeling of actions and targets across user-defined temporal ranges. Both tools will be released open source to support large-scale triplet annotation across diverse surgical procedures, and they are also broadly applicable to general surgical video annotation, contributing foundation research in this domain. Please see Appendix C for details.
5

---

Published as a conference paper at ICLR 2026
Table 2: Percentage distribution of triplet instances per frame in surgical triplet datasets between CholecT50 (Nwoye et al., 2022) and our ProstaTD.
Dataset
0
1
2
4
5
6
CholecT50 (Nwoye et al., 2022) ProstaTD (Ours)
10.97%
35.18%
47.10%
6.75%
0.00%
0.00%
0.00%
0.91%
6.19%
34.13%
39.75%
16.62%
2.21% %
0.19%
Semi-Automatic Annotation for Surgical Instruments. In the initial phase of annotation, we focused on labeling the types and bounding boxes of surgical instruments used in prostatectomy procedures. To support this task, we recruited 43 additional part-time student assistants, resulting in a total of 48 annotators. Among them, 20 students with strong medical backgrounds were assigned to conduct multiple rounds of review and quality control. Initial annotations were generated based on pre-labeled results from previously developed eystoscopic surgical instrument detection model. Although bladder and prostate surgeries share visual similarities, domain-specific differences require extensive manual refinement. Consequently, corrections on selected videos before using them to fine-tune the instrument detection model for subsequent annotation generation. After five iterative cycles of this semi-automatic training and correction process, we successfully completed both instrument type classification and bounding box annotations for all surgical instruments across the entire video dataset.
team performed least three rounds of manual
Surgical Action and Target Labeling, In the second phase, we enriched the instrument annotations from the first phase by assigning corresponding actions and targets. Unlike the first phase, which could be carried out by student annotators, this stage required substantial medical expertise. Although we employed the batch annotation tool, the annotation and refinement process were considerably more time-consuming. This phase was conducted by ten surgeons and fourteen senior students with strong medical backgrounds. To ensure label accuracy, each frame was reviewed by at least three team members. This process resulted in high-quality ProstaTD dataset comprising 89 triplet categories and 196,490 annotated instances, as illustrated in Fig. 2.
4 ANALYSIS OF THE PROSTATD DATASET
4.1 COMPLEXITY ANALYSIS OF PROSTATI
According to publicly available surgical classification systems (e.g., those used by regional health authorities (Hospital Authority, 2025)), cholecystectomy is typically categorized as major proce- dure, whereas prostatectomy is considered ultra-major, reflecting higher surgical complexity. This distinction highlights the increased technical demands and environmental intricacy associated with prostatectomy procedures. The complexity gap is quantitatively reflected in Table 2, which compares the distribution of concurrent triplet instances in CholecT50 (cholecystectomy) and our proposed ProstaTD (prostatectomy)) datasets. As summarized in Table 2, CholecT50 has 93.25% frames with < 2 triplet instances and none with 2 4. whereas Pro has 58. 38.77% of frames with 2 3 instances. This denser, more crowded scene composition underscores the higher complexity of prostatectomy and poses greater challenges for triplet detection models.
4.2 DISTRIBUTION OF PROSTATD DATASET
Distribution of Individual Components. As shown in Fig. 3, component distributions in Prosta align with the procedure's 's two main phases: dissection and anastomosis. Instrument frequencies reflect this structure: dissection instruments like monopolar scissors and bipolar forceps are most common, while needle drivers peak during anastomosis. Aspirators and graspers are used consistently across both phases for suction and retraction. Endobags and clip appliers are infrequent, as they are restricted to highly specific steps. The action distribution is dominated by retract, reflecting the need to maintain field clarity. The target distribution extends beyond anatomy to include catheters, gauze, and endobags, enhancing clinical realism. Samples may also contain nall actions targets, indicating an instrument is stationary or moving without a meaningful surgical action. Additionally, a detailed analysis of instrument variations discussed in Appendix A.3.

---

Published as a conference paper at ICLR 2026
Insinumants
Actions
Targets
5..02
8000
10MC
4000
36,005
3000
0820
12 130
30.00
3000
13.57
1000
4184
3277
PGT
1770
namdle dlowe
posppe
dep apple
rered
dereed
dksdder
cathuler
Bestae
Ehassind
Broad
Figure3 3: Component statistics in ProstaTD showing instruments (left), actions (center), and targets (right). Counts reflect label frequency and illustrate dataset diversity and task complexity.
Table 3: Cohen's Kappa between independent surgeon reviews and consolidated ground truth five randomly selected 200-frame segments. The bold value highlights the average result.
Surgeon 1
Surgeon 2
Surgeon
Surgeon 4
Surgeon 5
Average
Cohen's Kappa
0.82
0.81
0.84
0.82
0.83
0.82
Distribution of Triplet Components. As shown in Table 6 in the Appendix, triplet frequencies differ noticeably across ESAD, PSI, and PWH. Integrating these distinct sources is essential for capturing clinical variability, which arises from factors like differing surgical preferences, institutional training protocols, and patient-specific conditions. For instance, ESAD features a higher frequency of fascia- related triplets, while PSI-AVA shows prominent vesicle and aspirator retraction patterns. In PWH, endobag interactions are common, reflecting a technique where surgeons position the bag centrally for easier access. Capturing this diversity across sources is crucial for robust model generalization. Further qualitative insights into these distributional differences are discussed in Appendix B.
5 INTER-RATER AGREEMENT
To assess annotation quality, we conducted an inter-rater agreement study. We randomly sampled five non-overlapping video segments of 200 frames each. Five experienced surgeons who were not involved in the original annotation independently reviewed the triplet labels. Each surgeon assessed one segment while viewing the bounding box annotations but blinded to the triplet categories. Agreement with the consolidated ground truth quantified using Cohen's Kappa, with results summarized in Table 3. The average Kappa was 0.82, indicating strong consistency with our annotations. Most discrepancies occurred near anatomical boundaries where target assignment is inherently ambiguous; these cases are mitigated by our detailed written guidelines and expert-driven review protocol. Furthermore, all disputed cases were resolved through a secondary consensus round among senior surgeons, and the final ground truth was refined accordingly to maximize reliability.
6 EXPERIMENTS
6.1 EVALUATION PROTOCOL
To rigorously assess our dataset and models, we adopt a five-fold cross-validation over all 21 surgical videos, rotating the test fold and using the remaining videos for training and validation following our setting. In addition. we introduce an evaluation toolkit ("ivtdmetrics" in Appendix D) tailored for surgical triplet detection benchmarking, supporting mAP at IoU 0.5 and 0.5-0.95, Precision, Recall, and Fi-score, thereby facilitating fair comparison today and providing a basis for future foundation model research. For detailed five-fold composition, please refer to Appendix B.2.
well as
7

---

Published as a conference paper at ICLR 2026
Table 4: Detection performance the ProstaTDD dataset for TT and IVT components. We report mAP at IoU thresholds of 50% ("50") and 50:95 ("95"), together with inference speed (FPS). All results are reported +td (%) over 5-fold cross-validation. Experiments
conducted with input size 640 640 on single NVIDIA RTX 4090 GPU. Bold values with light green background indicate the best results, and underlined values with light purple background indicate the second-bes.
mAP((%)
mAPv(%) 1
mAPr(%) f
mAPivT
Method
FPS 1
50
95
50
95
50
95
50
95
Tripnet-Det' RDV-Det
1.60.4
0.60.3
0.40.1
0.1o.0
331.8
1.80.5
0.60.4
0.3o.1
0.1o.0
146.6
Faster R- R-CNN
73.3
63.2s.4
48.4s.6
42.1s.3
43.58.3
37.65.4
25.94.4
22.63.9
23.4
Cascade R-CNN
69.5s.1
59.65.8
44.66.1
38.55.6
39.58.6
33.65.8
21.64.6
18.74.1
20.6
SSD
74.61.8
64.55.3
50.2s.7
43.75.2
45.4a.1
39.4s.2
27.14.3
23.83.9
82.4
Vit-Det
86.52.8
73.62.8
52.24.6
45.43.8
48.
41.23.8
30.23.9
26.83.6
16.8
Deformable-DETR
75.44.7
65.0s.2
51.16.7
44.55.1
46.3g.2
40.1s.2
27.54.6
24.04.1
RT-DETR
91.6p.a
81.01.6
58.94.7
52.83.3
56.82.4
50.62.2
33.0s.8
29.63.3
66.3
YOLOv10
88.41.3
80.72.2
59.4a.4
54.92.
54.63.2
50.2a.1
34.34.1
31.83.5
200.3
YOLOv11
88.21.4
80.02.5
59.13.2
54.42.1
55.63.6
51.0g.3
34.13.7
31.5a.3
185.2
YOLOv12
88.81.1
80.41.9
59.93.1
54.82.2
54.52.1
49.91.9
34.3a.
31.53.2
204.1
TAPIR
76.14.5
65.84.8
52.35.1
45.64.6
47.15.4
40.54.7
28.44.7
24.64.2
10.6
MCIT-IG
77.41.4
67.21.6
53.64.9
46.94.3
48.45.1
41.8
29.64.5
26.04.0
16.0
TDnet (Ours)
89.91.3
81.02.0
61.72.9
56.32.
55.72.4
50.82.7
36.1s.4
33.13.1
126.6
Weakly supervisd methods.
6.2 COMPARISON METHODS
We establish a benchmark on the proposed ProstaTD dataset by training representative methods, summarized in Table 4, with implementation details provided in Appendix F.I. Note that prior triplet classification approaches are not directly comparable, as they do not output the hounding boxes required for triplet detection. To examine the gap between weakly-supervised and fully-supervised settings, we include two Cholectriplet0222 challenge methods, Tripnet- and RDV-Det (Nwoye al., 2023b). Both rely solely on class labels for weak supervision, employing Class Activation Maps and Non-Maximum Suppression for triplet localization. Other challenge entries are excluded due to incomplete methodological descriptions, unavailable code, and inferior reported performance. supervised baselines are primarily included to highlight the performance gap
These two relative to fully-supervised detectors with bounding box supervision.
We further compare ProstaTD performance using advanced detection architectures. We first in- clude several classical baselines such as Faster R-CNN (Ren et al., 2016). Cascade R-CNN (Cai & Vasconcelos, 2018), SSD (Liu et al., 2016), and ViT-Det (Li et al., 2022). We then evaluate the state-of-the-ard detectors, including Deformable-DETi 2024), well the latest YOLO architectures YOLOv10 (Wang al., 2024), YOLOv11 (Khanam Hussain, 2024), and YOLOv12 (Tian et al., 2025). addition, we include TAPIR (Valderrama al., 2022), framework that leverages temporal information for multi-task surgical scene detection, where we extend it with an additional triplet branch and remove unrelated tasks. We also evaluate MCIT-IG (Sharma et al., 2023), originally designed for semi-supervised surgical triplet detection, which we train in fully-supervised manner on ProstaTD in Table 4. Finally, we present a tailored method named TDnet provide insights for future research. TDnet adopts multi-task learning strategy and integrates self dittlistillation in triplet detection. Further details are provided in Appendix E.
(Zhu al., 2021) and RT-DETR (Zhao et al.,
6.3 RESULTS AND DISCUSSION
Detection Performance Analysis. Among all metrics, mAPIvT is the most critical as evaluates complete triplets. As summarized in Table 4, the two superv-s pipelines, Tripnet-Det and RDV-Det, yield extremely low mAP scores, consistent with the results and findings in the CholecTriplet2222 challenge (Nwoye al.. 2023b). This poor performance is mainly due to their reliance on class labels that provide only weak supervision. which cannot disentangle co- occurring instruments reliably associate instruments with their corresponding actions and targets.
8

---

Published as a conference paper at ICLR 2026
Table 5: Performance of the IVT component on the ProstaTD test set, reporting Precision, Recall, and Fr. Results are add (%) over 5-fold cross-validation. Bold values with light green background indicate the best results, and underlined values with light purple background indicate the second-best.
Method
Precision
Recall 1
F1-score 1
Derformable DETR (Zhu et al., 2021)
36. 14.8
19.72.8
22.73.8
RT-DETR (Zhao et al., 2024)
36.42.9
31.54.2
30.93.7
YOLOv10 (Wang et al., 2024)
34.63.8
34.83.6
31.93.2
YOLOv1 1 (Khanam & Hussain, 2024)
33.53.4
36. 36.13.8
31.53.3
YQLOv12 (Tian et al., 2025)
33.54.6
36.24.0
31.94.0
TAPIR (Valderrama et al., 2022)
35.25.0
20.33.8
23.44.3
MCIT-IG (Sharma al., 2023)
35.54.8
21.03.6
24.14.2
TDnet (Ours)
34.74.3
39.74.3
32.83.6
Conventional detectors such as SSD, Faster R-CNN, Cascade R-CNN, ViT-Det, and Deformable- DETR demonstrate basic recognition ability, but still show large gap compared with modern SOTA models. Among advanced detectors, including YOLOv10, YOLŌv1 1, YOLOv12, and RT-DETR, results are generally comparable. Notably, RT-DETR shows strong potential by achieving the highest instrument score (mAP,@0.5 = 91.6%), indicating that its transformer-basee design can effectively capture instrument-level features. However, this advantage comes at the cost of slower inference speed (66.3 FPS), which limits its practical deployment in real-time surgical applications. Regarding models tailored surgical analysis, TAPIR underperforms despite its use of temporal information. This is largely due to the reliance on sparse annotations for temporal features and the outdated detector backbone. Similarly, MCIT-IG, originally designed semi-supervised two-stage framework for action triplet detection, does not perform effectively under fully-sueer training on our dataset, likely because its architecture is optimized for semi-supervised settings and specialized modules hinder efficiency when labels are complete.
Our proposed TDnet achieves the best results across all major components. It reaches 36.19 % mAPivr @0.5 and 33.1% mAPivi (0.1vv0000000.90 improving upon YOLOv12 from 34.3% to 36.1% 1% and from 31.8% to 33.1 %, respectively. terms of individual components, TDnet attains the highest mAPy and delivers competitive gains on both instrument and target. It also maintains a strong balance between accuracy and efficiency, achieving 126.6 FPS. Notably, as discussed in Appendix F.2, its advantage becomes even more pronounced under the vidco-wiss wise mAP evaluation protocol, further highlighting the robustness our approach across both Trame-wise and vidco-wise assessments.
Rrecision-Recall Analysis. As shown in Table 5, class imbalance in the dataset makes surgical triplet detection highly challenging, leading to generally low Fi-scores across all methods. Deformable DETR lags behind more advanced approaches, with an F, of only 22.7%. highlighting its difficulty in handling long-tail distributions. YOLOvv10, YOLOv11, and YOLOv achieve balanced precision and recall in the mid-30% range, yielding F, around 32%. RT-DETR shows the highest precision (36.4%) but relatively low recall (31.5%), which limits its F1 to 30.9%. TAPIR and MCIT-IG also obtain competitive precision (35.2% and 35.5%) but their recall remains below 22%, resulting in very low F1 scores (23.4% and 24.1%). In contrast, our proposed TDnet achieves the best balance, boosting recall to 39.7% while maintaining a precision of 34.7%. This trade-off results in the highest F, (32.8%), surpassing all baselines. The gain demonstrates that TDnet mitigates the imbalance problem by capturing more true positives without an excessive rise in false positives. Despite these improvements, the absolute scores highlight that surgical triplet detection under heavy class imbalance remains far from solved, leaving considerable space for future progress.
Additional Analysis. Beyond the main comparison, conducted further evaluations, including video-wise metric analysis, ablation studies, confusion matrix analysis, and per-class AP analysis, to demonstrate the effectiveness of our method in mitigating the challenges of triplet prediction. The detailed results of these additional experiments are provided in Appendix F.
7 CONCLUSION
Inspired by the pioneering and influential CholecT50 work from the CAMMA team, We introduced ProstaiT). the first fully-supervised dataset for surgical triplet detection at the procedure level. It

---

Published as a conference paper at ICLR 2026
contains 71.775 frames from 21 multi institiinntttulaaalll surgeries, each with bounding box supervision and clinically verified temporal boundaries. ProstaTD is accompanied by open-source annotation tools, a standardized evaluation toolkit, and baseline benchmarks, forming comprchensive resource for fair and reproducible research in surgical video analysis. To provide a strong reference for future work, we proposed TDnet, baseline method that applies instance-level self-distillation with auxiliary supervision on instruments, actions, and targets. This design alleviates class imbalance and improves the robustness of triplet detection, offering a practical starting point for model development on ProstaTD. Overall, ProstaTD moves the field from coarse triplet classification toward full detection with spatial precision.
REPRODUCIBILITY STATEMENT
Our implementation can be found at http://ggthhbb.oommmooyyyy/yiigt
ETHICAL STATEMENT
All procedures performed in this study with human participants were in accordance with ethical standards and approved by the Institutional Review Board (IRB). The CREC ethics numebr is CREC Ref. No.:2021. No.:0021.600TT. All collected data was thoroughly anonymized and stored in secure facilities with restricted access limited to authorized research team members. The study adhered all relevant data protection regulations and medical research guidelines throughout data collection, processing, and analysis phases. In the following, we discuss the potential positive and negative societal impacts of our work.
Potential Benefits. As a comprehensive dataset for precise bounding box annotations and multi- institutional data, ProstaTD has the potential to significantly advance surgical AI development. The fine-grained annotations enable Al systems to monitor instrument-tissue interactions in real-time, generating objective skill assessments for residents and providing data-driven feedback for con- tinuous surgical quality improvement. Our open-source, multi-center dataset significantly reduces single-source bias and promotes model generalization across different hospitals and surgical set- tings, establishing a new standard for surgical Al validation. This dataset empowers surgical robot manufacturers to pre-train their vision modules directly, potentially accelerating clinical translation and reducing development cycles by months or even years. Moreover, the comprehensive nature of ProstaTD could catalyze breakthrough innovations in surgical automation and safety systems, ultimately contributing to more standardized and safer surgical procedures worldwide.
The dataset might be exploited by unregulated third parties for commercial products, where model failures could directly compromise patient safety. Mitigation Measures: (1) We implement CC-BY-NC-SA-4.0 license requiring all derivatives to maintain non-commercial terms; (2) We recommend incorporating uncertainty estimation and human oversight prompts when deploying models. (3) We explicitly prohibit unauthorized use of the dataset for training LLMs or commercial AI systems, and reserve the right to take legal action against any violations of these terms.
Potential Risks.
ACKNOWLEDGMENTS
We sincerely thank all the annotators from The Hong Kong Polytechnic University (PolyU), Nanfang Hospital Southern Medical University, Prince of Wales Hospital (PWH). South China University of Technology (SCUT) and The University of Hong Kong (HKU) for their dedicated efforts and valuable contributions. This work was supported by a Shenzhen-Hone Hong Kong-Macao Science and Technology Plan Project (Category C Project) under Shenzhen Municipal Science and Technology Innovation Commission (project no SGDX2002082209235000)) a grant under Collaborative Research Fund of Hong Kong Research Grants Council (project no C5055-24G), the Strategic Topics Grant 2025/26, Research Grants Council, HKSAR (Reference no.: STGI/M-445/25-N) the Guangdong Natural Science Funds for Distinguished Young Scholars (No. 2023B1515020097 and the Lee Kong Chian Fellowships. This work was also supported by the Key-Area Research and Development Program of Guangzhou City (No. 2023B01 (OBB2) and the NSFC Key Project (No. U23A20391).
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Nicolás Ayobi, Alejandra
Santiago Rodríguez, and Pablo Arbeláez. Matis: Masked- attention transformers for surgical instrument segmentation. In 2023 IEEE 20th International Symposium on Biomedical Imaging (ISBI), pp. 1-5, 2023. doi: 10.1 109/1SB153787..22... 102.11038811
Nicolás Ayobi, Santiago Rodríguez, Alejandra Pérez, Isabela Hernández, Nicolás Aparicio, Eugénie Dessevres, Sebastián Peña, Jessica Santander, Juan Ignacio Caicedo, Nicolás Fernández, et al. Pixel-wise recognition for holistic surgical scene understanding. Medical Image Analysis, pp- 103726, 2025.
Vivek Singh Bawa, Gurkirt Singh, Francis KapingA, Inna Skarga-Bandurova, Elettra Oleari, Alice Leporini, Carmela Landolfo, Pengfei Zhao, Xi Xiang, Gongning Luo, et al. The saras endoscopic surgeon action detection (esad) dataset: Challenges and methods. arXiv preprint arXiv:2104.03/78 2021.
Zhaowei Cai and Nuno Vasconcelos. Cascade r-cnn: Delving into high quality object detection. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 6154-6162, 2018.
Yiliang Chen, Shengfeng He, Yueming Jin, and Jing Qin. Surgical activity triplet recognition via triplet disentanglement. In International Conference on Medical Image Computing and Computer- Assisted Intervention, pp. 451-461. Springer, 2023.
Shuangchun Gui and Zhenkun Wang. Tail-Enhanced Representation Learning for Surgical Triplet
Recognition In proceedings of Medical Image Computing and Computer Assisted Intervention MICCAI 2024, volume LNCS 15011. Springer Nature Switzerland, October 2024.
Hospital Authority. Operations classification of hong kong. https: /ttp://wwwww/ wwww//wwww..om...oom...wl... ha. org. hk/ fnc/ operations aspx 2aspx1ang-ENNV 2025. Online resource.
Arnaud Huaulmé, Duygu Sarikaya, Kévin Le Mut, Fabien Despinoy, Yonghao Long, Qi Dou, Chin- Boon Chng, Wenjun Lin, Satoshi Kondo, Laura Sravo-Sánchez, al. dicro-surgical anastomose workflow recognition challenge report. Computer Methods and Programs in Biomedicine, 212: 106452, 2021.
Rahima Khanam and Muhammad Hussain. Yolov11: 11: An overview of the key architectural enhance- ments. arXiv preprint arXiv:2410.17722 2024.
Yanghao Li, Hanzi Mao, Ross Girshick, and Kaiming He. Exploring plain vision transformer backbones for object detection. In European conference on computer vision, pp. 280-296. Springer, 2022.
Wenjun Lin, Yan Hu, Luoying Hao, Dan Zhou, Mingming Yang, Huazhu Fu, Cheekong Chui, and Jiang Liu. enstrmmnn--iisuss interaction quintuple detection in surgery videos. In International Conference on Medical Image Computing and Computer-Assiste Intervention, pp. 399-409. Springer, 2022.
Wenjun Lin, Yan Hu, Huazhu Fu, Mingming Yang, Chin-Boop Chng, Ryo Kawasaki, Cheekong Chui, and Jiang Liu. Instrument-tissae interaction detection framework for surgical video understanding. IEEE Transactions on Medical Imaging, 2024.
Wei Liu, Dragomir Anguelov, Dumitru Erhan, Christian Szegedy, Scott Reed, Cheng- Yang Fu, and Alexander C Berg. Ssd: Single shot multibox detector. In European conference on computer vision, pp. 21-37. Springer, 2016.
Aditya Murali, Deepak Alapatt, Pietro Mascagni, Armine Vardazaryan, Alain Garcia, Nariaki Okamoto, Didier Mutter, and Nicolas Padoy. Latent graph representations for critical view of safety assessment. IEEE Transactions on Medical Imaging, pp. 1-1, 2023. doi: 109/90.. 2023.3333034.
Chinedu Innocent Nwoye, Cristians Gonzalez, Tong Yu, Pietro Mascagni, Didier Mutter, Jacques Marescaux, and Nicolas Padoy. Recognition of instrument- tissue interactions in endoscopic videos via action triplets. In International Conference on Medical Image Computing and Computer- Assisted Intervention (MICCAI), pp. 364-374, Springer, 2020.
11

---

Published as a conference paper at ICLR 2026
Chinedu Innocent Nwoye, Tong Yu, Cristians Gonzalez, Barbara Seeliger, Pietro Mascagni, Didier Mutter, Jacques Marescaux, and Nicolas Padoy. Rendezvous: Attention mechanisms for the recognition of surgical action triplets in endoscopic videos. Medical Image Analysis, 78:102433, 2022.
Chinedu Innocent Nwoye, Deepak Alapatt, Tong Yu, Armine Vardazaryan, Fangfang Xia. Zixuan Zhao, Tong Xia, Fucang Jia, Yuxuan Yang. Hao Wang, et al. Cholectriplet2021: A benchmark challenge for surgical action triplet recognition. Medical Image Analysis, 86: 102803, 2023a.
Chinedu Innocent Nwoye, Tong Yu, Saurav Sharma, Aditya Murali, Deepak Alapatt, Armine Var- dazaryan, Kun Yuan, Jonas Hajek, Wolfgang Reiter, Amine Yamlahi. et al. Cholectriple10222. Show me a tool and tell me the triplet- an endoscopic vision challenge for surgical action triplet detection. Medical Image Analysis, 89:102888. 2023b.
Chinedu Innocent Nwoye, Kareem Elgohary, Anvita Srinivas, Fauzan Zaid, Joël Lavanchy, and Nicolas Padoy. Cholectrack20: A multi-perspective tracking dataset for surgical tools. In Proceed- ings of the Computer Vision and Puttern Recognition Conference, pp.
2025.
Nicolas Padoy, Tobias Blum, Seyed-Ahmad Ahmadi, Hubertus Feussner, Marie-Odile Berger, and Nassir Navab. Statistical modeling and recognition of surgical workflow. Medical image analysis, (41))6632-641,0 2012.
Dimitrios Psychogyios, Emanuele Colleoni, Beatrice Van Amsterdam, Chin-Yang Li, Shu- Yu Huang, Yuchong Li, Fucang Jia, Baosheng Zou, Guotai Wang. Yang Liu, et al. Sar-rarp50: Segmentation of surgical instrumentation and action recognition on robot-assisted radical prostatectomy challenge. arXiv preprint arXhv:2401.00496 2023.
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. IEEE transactions pattern analysis and machine intelligence, 39(6): 137-1119, 2016.
Saurav Sharma, Chinedu Innocent Nwoye, Didier Mutter, and Nicolas Padoy. Surgical action triplet detection by mixed supervised learning of instrument-tissue interactions. In International Conference Medical Imuge Computing and Computer-Asistted Intervention, pp. 505-514. Springer, 2023.
Pan Shi, Zijian Zhao, Sanyuan Hu, and Faliang Chang. Real-time surgical tool detection in minimally invasive surgery based on attention-guice guided convolutional neural network. IEEE Access, 8:228853- 228862, 2020.
Yunjie Tian, Qixiang Ye, and David Doermann. Yolov12: Attention-centrie real-time object detectors. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025.
Andru P Twinanda, Sherif Shehata, Didier Mutter, Jacques Marescaux, Michel De Mathelin, and Nicolas Padoy. Endonet: deep architecture for recognition tasks on laparoscopic videos. IEEE transactions on medical imaging, 36(1):86 97, 2016.
Natalia Valderrama, Paola Ruiz Puentes, Isabela Hernández, Nicolás Ayobi, Mathilde Verlyck, Jessica Santander, Juan Caicedo, Nicolás Fernández, and Pablo Arbeláez. Towards holistic surgical scene understanding. In International conference on medical image computing and computer-assiste intervention, pp. 442-452. Springer, 2022.
Ao Wang, Hui Chen, Lihao Liu, Kai Chen, Zijia Lin, Jungong Han, et al. Yolov 10: Real-time end-to- end object detection. Advances in Neural Information Processing Systems, 17:1007844-100011 2024.
Nan Xi, Jingjing Meng, and Junsong Yuan. Forest graph convolutional network for surgical action triplet recognition in endoscopic videos. IEEE Transactions on Circuits and Systems for Video Technology, (112))8550-85.. 2022.
Shang Zhao, Yanzhe Liu. Qiyuan Wang, Dai Sun, Rong Liu, and S Kevin Zhou. Murphy: Relations matter in surgical workflow analysis. arXiv preprint arXiv:22 /2719, 2022.
12

---

Published as a conference paper at ICLR 2026
Yian Zhao, Wenyu Lv. Shangliang Xu, Jinman Wei, Guanzhong Wang, Qingqing Dang, Yi Liu, and Jie Chen. Detrs beat yolos on real-time object detection. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 16965-16974, 2024.
Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai. Deformable detr: Deformable transformers for end-to-end object detection. In International Conference on Learning Representations, 2021.
13

---

Published as a conference paper at ICLR 2026
CONTENTS OF APPENDIX
ProstaTD Dataset Preparation
15
A.I Additional Introduction ESAD and PSI-AVA Datasets
15
A.2 ProstaTD Video Preprocessing
15
A.3 ProstaTD Instrument Taxonomy and Variations
15
A.4 Expert-guided Label Refinement Strategy
17
B PDostaTD Dataset Analysis and Partitioning
17
B.1 Triplet Component Distribution in ProstaTD Dataset
17
B.2 Five-fold Cross-Validanie Dataset Partition Protocol
18
Annotation Software
19
C.I Triplct-labelme Annotation Tool
19
C.2 SurgLabel Annotation Tool
19
Evaluation Toolkit
22
Our Proposed TDnet Network
23
E.1 Network Architecture
23
E.2 Loss Functions
24
F Experiment Details
25
E.I
Implementation Details,
25
F2
Video-wise Comparison Results
................................
25
F3
Ablation ................................
25
Confusion Matrix Analysis
..
27
E5 Per-class Average Precision of IVT component
29
R6 Cost-Aware Ordinal IVT Evaluation
31
F.7
Leave-one-suuree-uut Evaluation
32
G Limitations and Future Works
33
G.1 Long-Tab Tail Distribution and Rare Cases
................................
33
G.2 Temporal Modeling
................................
33
G.3 Continual and Cross-Domain Learning
33
G.4 Bridging to Robotic Assistance and Simulation
34
H Supplementary Information on Annotation Software
34
H.1 Imported ISON File Example
34
H.2 Customizable User Interface Features
36
I The Use of Large Language Models
36
14

---

Published as a conference paper at ICLR 2026
A PROSTATD DATASET PREPARATION
In this section, we first provide an additional introduction to the ESAD and PSI-AVA datasets, then we describe the video preprocessing pipeline to ensure data quality, analyze surgical instrument variations across sources, and conclude with the expert-guided strategy for label consolidation.
A.1 ADDITIONAL INTRODUCTION TO ESAD AND PSI-AVA DATASETS
Both the ESAD and PSI-AVA datasets but their original annotations are not designed for the task of surgical triplet detec- tion. Despite the information presented in Table which indicates that the ESAD dataset (Bawa et al., 2021) includes full bounding box annotations with 46,325 anno- tated instances, these annotations are largely unusable for precise surgical triplet detection. The bounding boxes in ESAD are often coarsely defined, sometimes encompassing an entire in- strument, and in other cases including multiple instruments and anatomical targets within a sin- gle box, illustrated in Fig. 4. Consequently, we did not utilize ESAD's annotations and in- stead performed
collected from prostatectomy procedures,
BladderAnastammeer
re-annotation to ensure precision and consistency for ProstaTDD, and we applied the same unified annotation rules to our internally collected PWH dataset.
Similarly, the PSI-AVA dataset (Valderrama et al., 2022) provides 5,804 annotated instances with sparse bounding box annotations, which do not cover all frames. Although PSI-AVA in- cludes over 5,000 bounding boxes, these annota- tions typically encompass the entire instrument, unlike our ProstaTD dataset, where the annota- tions focus specifically on the instrument's s head and mid-joint positions for greater precision. Therefore, for our 196,490 annotated instances in ProstaTD, also opted to re-annotate rather than adopt PSI-AVA's bounding boxes maintain consistency and alignment with our fine-grained annotation strategy.
Figure Coarse bounding box annotations in ESAD, showing examples where boxes include multiple instruments.
A.2 PROSTATD VIDEO PREPROCESSING
Removal of Non-surgical Content. The preprocessing stage entailed rigorous refinement of surgical videos sourced from our in-house PWH dataset, as well the ESAD (Bawa et al., 2021) 1) and PSI- AVA (Valderrama al., 2022) datasets. The objective to curate a high-quality dataset comprising only relevant surgical content. Initially, non-surgical frames, such as those captured during pre- operative preparation and post-operative recovery, were systematically removed. Additionally, intraoperative segments containing non-surgical visual content were excluded. These included frames with lens contamination necessitating endoscopic cleaning, instrmment-switching. -witchig-iiindudee induced occlusions, and camera deviations from the surgical field for operational purposes.
Quality-oriente Frame Filtering. We also discarded frames with severely degraded visual quality, where even experienced surgeons could not reliably interpret surgical activity. Such instances included extreme lighting artifacts or dense surgical smoke that obscured the operative field. Retaining these frames would hinder effective annotation and risk undermining the dataset's overall reliability.
A.3 PROSTATD INSTRUMENT TAXONOMY AND VARIATIONS
As discussed in the main text, surgical instruments exhibit significant visual variability across sources due to differences in manufacturer design and institutional preferences. Even within a single source, instruments of the same category may appear differently depending their specific surgical applications (c.g., non-da Vinci systems). To illustrate the dataset's 's diversity, we present a comprehensive visualization of all instrument types in Fig. 5, capturing both inter-source variation
15

---

Published as a conference paper at ICLR 2026
PWH Instruments Variations
(a) Scissors
(b) Scisurs? 2
(c) Sciesors
(d) F Forceps
(e) Forceess
B A'Aspirtoor
(g) Clip Appleer
(b) Needle Driver
( Needle Driver 2
i Grasper
Grasper
(1 Endobag
ESAD Instruments Varlations
Scissors
(b) Forceps
(c) Asirrator
Clipp Applier
(e) Clip Applier
(1 Needle Driver
09 Grasper 3
①) Endobag
( Grasper
(h) Grasper 2
PSI-AVA Instruments
Sclewors
(d) Aspicator
(e) Cllp Applier
(1 Needle Driver
Variations
(g) Needle Driver
(h) Grasser
(i Grasper 2
①) Grasper 3
Endobag
Figure 5: Instrument appearance variation across datasets. Representative examples with ground- truth bounding boxes from PWH (red), ESAD (green), and PSI-AVA (blue). Beyond appearance inconsistencies across sources, some instrument variants unique to specific datasets.
and intra-category differences. These examples also serve to intuitively demonstrate our bounding box annotation strategy across diverse instrument classes.
Instrument Variations Across Datasets. Our in-house PWH dataset comprises 12 distinct surgical instrument types, while the ESAD dataset (Bawa et al., 2021) includes 10 instrument types, and PSI-AVA (Valderrama et al., 2022) comprises 11. As depicted in Fig. 5, instruments from PWH (red bounding boxes) include frequently utilized da Vinci robotic instruments such as (a). (d), (f), (g), (h). (i) and (I) in prostatectomy procedures. A physically worn variant is shown in (b), reflecting real-world degradation, and conventional laparoscopic device appears in (c), highlighting the inclusion of non-robotic instruments in minimally invasive procedures.
Instruments from ESAD (green bounding boxes) and PSI-AVA (blue bounding boxes) show partial overlaps, such as graspers (a in both), scissors (b in ESAD, in PSI-AVA), and other shared types like (c), (d), (f), and (i) in ESAD, as well as (a), (c), (d), (e), (f), and (h) in PSI-AVA. These overlaps reflect a high degree of visual similarity, yet subtle design variations evident, for instance, in the color details the grasper (a) between ESAD and PSI-AVA. Additionally, unique instrument types appear in each dataset, such as (g) in ESAD and (b) in PSI-AVA. Compared across all sources, the instruments exhibit notable visual differences, emphasizing inter-dataset variability and intra-category diversity.
The integration of ESAD, PSI-AVA, and our PWH videos into the ProstaTD dataset introduces greater complexity compared existing benchmarks such CholecT50. The resulting diversity in
16

---

Published as a conference paper at ICLR 2026
instrument appearances and variations offers more realistic foundation för developing robust surgical instrument detection systems capable of generalizing across heterogeneous surgical environments and accommodating increasingly complex procedures.
A.4 EXPERT-GUIDED LABEL REFINEMENT STRATEGY
Similar to prior work (Nwoye et al., 2022), we applied systematic label consolidation strategy guided by two core principles: clinical relevance and semantic consistency. This step was essential, as raw surgical action labels often contain redundant or clinically marginal variations that may undermine the dataset's practical applicability.
The consolidation process was performed under clinical expert supervision and comprised two main stages. First, semantically equivalent action triplets were identified and merged into unified super- classes. This grouping operation, denoted as U, was carefully designed to preserve clinical meaning while reducing unnecessary label granularity. Representative examples of this merging strategy are provided in the following:
(grasper, retract, prostate) U (grasper, retract, prostate-apex) U (grasper, retruct, DVC)
(grasper, retruct, prostate)
(Endobag, bag, prostate) U (Endobag, store, prostate)
(Endobag, bag, prostate)
3. (scissors, cut, vas-deferens) U (scissors, cut, seminal-vesicle) - (scissors, cut, seminal-vesicle)
4. (forcep, retract, bladder) U (forcep, retract, bladder-neck) - (forcep. retract, bladder)
(aspirator, suck, fluid U (aspirator, suck, smoke) (aspirator, suck, blood) - (aspirator, suck, fluid)
(scissors, dissect, fascia) U (scissors, dissect, lymph node) U (scissors, dissect, fat) (scissors, dissect, fascia)
B PROSTATD DATASET ANALYSIS AND PARTITIONING
B.1 TRIPLET COMPONENT DISTRIBUTION IN PROSTATD DATASET
Table reports triplet frequencies for ESAD. PSI, and PWH, together with overall totals, using standardized abbreviations for instruments, verbs, and targets. Each source dataset introduces subtle yet meaningful variations in instrument appearance, procedural conventions, and surgical technique, which collectively increase the diversity and difficulty of the learning task. This diversity is deliber- ate design choice to promote generalization across real-world surgical scenarios. The distributional differences further justify integrating these datasets: ESAD exhibits markedly higher frequencies of duscias-relater triplets (c.g., bissssorsddisset.fsssssst
PSI shows distinctive patterns such frequent vesicle- and aspirator-base based retraction events (c.g., aspirator, retract. vesicle): PWH routinely includes Endobag-related contexts, leading frequent bag-centric triplets (e.g.. bag, null, as well cases that are absent in the other datasets. Several triplet categories appear predominantly or even exclu- sively in single dataset, demonstrating that each contributes unique, non-redundant information. Rather than a simple aggregation, their integration ensures broader coverage of clinical variability across institutions and techniques, which is essential for developing robust and generalizable models.
Beyond these distributional differences, ProstaTD also exhibits a pronounced long tail similar to CholecT50 (Nwoye et al., 2022), with many triplet categories appearing only few times. This pattern is amplified by the multi-source composition and by variability in surgical styles, as different surgeons follow their own operational habits. For example, using forceps perform suture instead of a needle driver to save the time of switching instruments. Consequently, numerous infrequent triplets naturally arise. In real surgical settings, is neither feasible to enumerate all possible instrument-action cetpeeentt-atioo-ttrreee combinations nor to exclude human error improvisation, so capturing this variability is necessary for building robust recognition systems. For approaches that decompose triplet prediction into separate instrument, action, and target branches, rare triplets provide valuable stress tests for generalization to uncommon scenarios, a capability that is critical for reliable deployment in the wild.
17

---

Published as a conference paper at ICLR 2026
Table 6: Triplet frequencies by dataset (ESAD, PSI, PWH) with overall totals. Here, "driver" refers to "needle driver", "applier" to "clip applier", and "vesicle" abbreviates "seminal vesicle".
Triplet
ESAD
PSI
PWH
Total
Triplet
ESAD
PSI
PWH
Total
scissors, retract retractbladderr aissors.retrac vesscce
26
359
481
866
21
84
8
113
233
541
403
1177
reissors.ettrac..rosssts
266
617
1326
2209
scissoos, retract, fascias
646
233
278
1157
reisacrs,
92
92
scissons,retrace retract, Findohug
0
99
102
scissors, scissors,coagllll coagulate, coaddertebMaddee blader
30
64
105
scissers.coggulatc vesicle
48
15
scissors,conuul prosttae
36
43
coaguuas.oaaullte.. fascias fascias
105
13
11
129
scissors, cut.,,cuubuue bladder
529
2531
1356
4216
schssoes,cul cul.cateeeeeeeeerr
scissors,cul cuslors,cul.vssice vesicle
192
scissors, prrpoostaee schaorss culeereed
1608
3504
2290
7402
scissors,cut,fan cut.aassatt
280
14
163
457
18
43
31
92
scissors,iissee bladder
375
176
74
625
scissoes, scissoes, vesicle
1177
1673
2767
5617
scissors,iissees dww.sett. prostate
1348
1773
2100
5221
scissors, dissect, fascias scissors,disseer. fascias
4734
750
1725
7209
scissors, mull.
4740
8221
6690
19651
forcopsrrctrac bladder
1448
2696
forcops retrestttrtctctabttcc
35
conceps.retrace vesicle
1888
3696
3137
8721
lorceps.retract www.rrettrrpuuu.c. pprsstate
1704
4163
4444
10311
fonceps.retract fascias
4894
807
2595
petrect. Eindohug
pep cougulatee, bladder
0
413
267
680
lorceps, wtwpppssooullaaeeee...o vesicle
115
4033
518
forceps,coaage coogulate, prostate
18
255
1201
1474
fonceps.oougulate fuscias
341
88
158
587
forceps, dissect, vasicle
16
forceps,dissectti prrosate
24
45
forcops,disse dlisset..fascie
90
0
90
180
forcops.gra grasp.cathece...
77
21
104
forceps. grasp. vesicle
garcopp.grasp..o
wttppsgrrssppsstaaae prostate
fonceps-rray cruupplaccias fascias
443
10
36
489
pareep..rass.gaee gauze
0
60
116
176
forceps.grasy grasp, Endoohg
245
perceps-grnp..hiee grasp.hrrad
764
782
forceps,suture blukker
388
85
123
lorceps prostate
33
104
forceps, suture, fascias
1.5
15
forceps,nul mull
3511
5331
4903
13745
aspirator, retract, bladder
395
4586
3438
8419
aspirator retract, vesiele
14
475
ppprratorretratt.pooo. prostaae
52
248
964
1264
302
134
2324
2760
aspirator retract,Ennobbu uspirator,miill muil, null
13
aspirator, suck,fluid
1167
1189
1477
3833
driver,retract. bladder
203
159
487
849
driver prostate
retver, fascins
288
368
driver. driver.grasp. peeerrraspbbllldeer bladder
0
driver wwwerrrrs..cccceeeeeee catheler
30
30
driver.grasp. pepottttee
0
18
18
driver.grasp. preeggggss.fiiiii.i fascias
7
3
17
triver.grapp.guug gauze
79
griver.gaasp.Eune Endobag
57
driver. driver.grasp.uu thredd
1044
4233
3678
8955
driver,suture bladder
315
382
1331
2028
driver, prostate
567
driver suture, fascias
23
driver, mull.nn Loull
784
2017
2271
5072
grasper, retract, retrett.dee hladder
261
1484
228
1973
rasperrretratt.oofe
0
25
grasperretract, vesicle
461
3074
8786
grasper wetrarc.prossatt
270
2466
3081
5817
ceasper.etract.fee fascias
647
494
1079
2220
grasper,grasp gra.spccathhte catheter
16
88
21
125
graspergraspp..o grasp. vesiele
21
29
56
graspor. grasp. prostate
127
384
pusspr.grasp.fisee fascias
193
138
grasper. gra.tggaa..gaa.. gauze
0
9
32
41
graspen.grapp Einoobug
4
70
362
436
grasper. grasp thread
grasper, mull.
2225
hladdde ppplier.elip bladder
19
87
37
143
applier,elip, vesicle
56
211
24
291
applicr.elip. prostate
100
478
296
874
appliczelip fascias
135
286
429
applier,elip ett.ierrbbb.eddobuuggee Endobag
0
39
39
applier.nuil.nul
660
749
883
2292
Endobag, bag prostate
54
265
451
770
Endobag, bag, fascias
0
0
127
127
Endobag, null
2230
Total:
44061
81179
196490
B.2 FIVE-FOLD CROSS- VALIDATION DATASET PARTITION PROTOCOL.
Table 7: Five-fold cross-validation split for our experimental setup.
Fold
Test Videos
Validation Videos
Training Videos
1
esady1
psiv1
psiv4
pwhv8
psiv?
pwhv5
Remaining videos
2
esadv2
psiv7
pwhy4
pwhv9
psivl
pwhv6
Remaining videos
3
esadv3
psiv14
pwhy1
psiv2
esadvl
pwhy2
Remaining videos
4
esadv4
1siv15
pwhv2
pwhy7
esadv3
pwhy3
Remaining videos
5
psiv3
psiv21
pwhv3
pwhv5
pwhv6
psiv14
pwhv1
Remaining videos
In our experiments, we adopt a five-fold cross-validation scheme over all 21 surgical videos, rotating the test set so that every video (and the rare triplets it contains) appears in testing at least once. For each fold, the remaining videos form the pool for training and validation. In our default protocol, two videos are randomly selected from this pool as the validation set, and the rest are used for training. The validation selections given in Table can be used verbatim to replicate our results. Alternatively, researchers may adopt different protocol that omits the validation set. This is because validation sets often inadvertently contain rare triplets, which can lead to significant performance bias. This issue was also observed in the CholecT50 dataset. While we adhere to the standard validation split to
18

---

Published as a conference paper at ICLR 2026
remain consistent with prior work, skipping the validation set is recommended approach to address data scarcity.
C ANNOTATION SOFTWARE
To our knowledge, before this work there was no open-source annotation tool purpose-built for surgical instance, and none that supports structured triplet labels. In existing general annotation tools, annotators must select each instance and then manually assign one class out of 89 categories, which makes the process slow and error-prone, Such workflow is impractical for large-scale triplet annotation in surgical videos because scales poorly across frames, instruments, actions and targets. Even with a large team, using general tools would take years to produce only small portion of the required labels.
To construct ProstaTD at scale developed dedicated software for triplet labeling (instrument, action, target, bbox). Although the initial motivation came from our prostatectomy dataset, the design is procedure agnostic and transfers to other surgeries with different organ targets and workflow conventions. This provides a practical step toward a general purppos-purpose foundation model for surgical annotation. We introduce two complementary tools. Triplet-labelme supports single-frame frame editing, allowing annotators to assign triplet attributes (instrument, action, target) to every instance in an image and to draw refine the corresponding bounding box for each instance. SurgLabel supports batch annotation by using existing bounding boxes and instance identities together with an adaptive track identity assignment, which propagates and verifies action and target labels across the timeline. Together these tools address diverse needs in surgical video analysis.
C.I TRIPLET-LABELME ANNOTATION TOOL
This tool is adapted from LabelMe' and optimized for surgical use (see Fig. 6). It streamlines bounding box creation and instrument category assignment, and supports single-frame editing of triplet annotations (instrument, action, target) for every instance in image together with the corresponding bounding box. Users can freely modify the action and the target of any selected instance.
While primarily designed for triplet annotation in ProstaTD, the tool is procedure agnostic and can be applied to other surgical tasks such as per-image segmentation, surgical phase and step annotation, or higher-level grouping labels. As shown in Fig. 6, users can import custom JSON file to give each instance the attributes action, target, and track identity. The detailed JSON file and three custom examples are described in Section H.1. Specifically, by modifying the imported JSON, users can restrict the permitted action and target values for a given instrument category to a specified choice set, and can also extend the imported JSON schema to add custom attributes beyond action and target.
Productivity and stability are enhanced through default suto-save, ergonomic shortcut mappings for mode switching, for example changing key "Ctrl+J" to simply key "J", and flexible visualization controls for colors, label transparency, font size, and line thickness. Visualization label boxes are adaptively positioned to avoid overlapping, which keeps the display clear. In addition, we fixed several LabelMe issues that previously caused crashes or made some operations ineffective during large-scale annotation.
C.2 SURGLABEL ANNOTATION TOOL.
After completing instrument instance annotation with Triplet- the next step to assign actions and targets to these instances. Unlike instrument annotation, which can be partially assisted by our trained models, action and target labeling is temporally extended and context-dependent and becomes inefficient if performed frame by frame. To avoid slow per-frame editing and to improve temporal consistency, we designed SurgLabel (see Fig. 7), a span span-based batch annotation interface purpose-built for surgical videos. It treats temporal spans as first-class objects, supports on-the-fly labeling while scrubbing the timeline or scrolling through the dataset, and integrates seamlessly with the instrument instances defined in Triplet-Labelme. Its main capabilities are follows:
19

---

Published as a conference paper at ICLR 2026
File Edit View Help Show-label Fost Colors import-too Import-AT
Spen Oben
p2 (P
Pray
@
huct
Sooe
Delete File
Create
Edit Polygene
Dupliccte Pelypens
Delete
Undo
Brightness Contract
Ftt Window
Mask Model
Dir imag Image
Folygans
Polygone
(eenunery)
scasor's
Forceps
grosper
aspriator
Polygen Lebels
espirattrr
sossors
forcess
Endobag
File Filenerme
hasferestattaaataaa clttaset Full fus/prostaae wostat///wuustt Jowbes
Caata full pwl/
corasata fu prostatee
Attributes
To garget
Trackin:
Figure Our customized Triplet-Labelme tool for bounding box and instrument category annotation with fer-frame triplet editing (instrument, action, target).
Span-based batch labeling, Select one or multiple instrument instances and specify start and end frame indices apply action and target labels contiguous temporal range in single opera- tion, which enables on-the-fly annotation during scrubbing or playback and keeps long maneuvers consistent with minimal interactions.
Open action/target schema. SurgLabel supports arbitrary action and target sets by importing user-defined JSON label schema that specifies, for each instrument class, its admissible actions and targets (see Section H.1 for details and examples). The same schema can be reused across procedures, so teams can standardize switch configurations without code changes.
Support for Diverse Surgical Tasks. The same span-based mechanism applies to image-level annotations such as surgical phase and step, and instance-level annotations such as orientation and free-form descriptions (see Section H.1 for details). As result, SurgLabel is not limited to surgical triplets and can serve diverse labeling tasks, supporting broader data curation for future foundation models.
Support segmentation instances. Although SurgLabel was initially designed for bounding box workflows, also supports segmentation. Action and target labels, well as any schema attributes, can be attached in batch to segmentation instances, and the interface mirrors the box mode (see Fig. 7). This keeps geometric masks and semantic triplets synchronized when instruments persist across spans.
Adaptive track identity assignment. SurgLabel provides automatic per-class instance numbering that remains stable when multiple instruments of the same type are present. By default, relies on spatial and motion heuristics guided by surgical priors: instruments operate from a relatively fixed direction in the camera view, and stay in that sector until fully removed. We then compare instruments of the same class hierarchically using image region, orientation, and keypoint coordinates to separate instances over time. This rule is user-configurable. and switching strategies possible when situations change. The resulting assignment smoothed temporally, so brief crossings or short occlusions rarely cause identity swaps. This resolves most trocar ingress and egress cases, as shown in Fig. 7. Users may also switch to the ByteTrack (Zhang et al., 2022) method via the Sort Method button, but recommend the default because it is more stable, especially under low-frame frame-rate annotation (e.g.. fps). Assigned IDs are temporary and the sort method can be modified any time, and came-class instruments displayed as #1, #2. #3 for unambiguous labeling. Overall, track ID anmotation consuminn,,, particularly in non-robonn non-robotic procedures. Leveraging this prior knowledge-hasee based automatic assignment greatly reduces the workload. Alternatively, if track
inserted through trocars,
20

---

Published as a conference paper at ICLR 2026
bechccute
Chh
CWY obn
(b)
Figure 7: SurgLabel, a purpose-buill interface for span-based based batch labeling of actions and targets with predefined instrument instances. It supports automatic disambiguation of concurrent instruments of the same category by assigning persistent instance tags across time (e.g., #1, #2), ensuring temporal consistency. (a) Detection mode, (b) Segmentation mode.
IDs are already available upstream, our SurgLabel can also directly support batch annotation from them without re-assignment.
User-friendly ergonomics. The interface provides multilingual support, ergonomic shortcuts, se- lection highlighting, and autosave to reduce friction at scale. Overlays adaptive transparency labels do not occlude important content. Different visualization label boxes are adaptively positioned to avoid overlapping, ensuring clear visibility. Annotators navigate frames with the A and D keys, set the start and end frames with the and X keys, and confirm edits with the key, which streamlines repetitive operations.
Rich customization. Per-class colors, label and box opacity, highlight intensity, line thickness, and font weight configurable via the Adjust Display Parameters button and the Adjust Annota-
21

---

Published as a conference paper at ICLR 2026
tion Colors button, allowing teams enforce consistent visual conventions while accommodating individual preferences. For more details, please see Appendix.H.2.
Open sourcing. We will release both tools as open source together with documentation, configuration templates, and example projects, which will facilitate annotation across diverse surgical tasks and accelerate community progress toward surgical foundation models.
D EVALUATION TOOLKIT
To better evaluate surgical triplet detection performance, building upon the prior "ivtmetrics" work in CholecT50 (Nwoye & Padoy, 2022), we have developed enhanced evaluation toolkit named "ivtdmetrics." Compared to the original "ivtmetrics," our toolkit introduces the following key enhance- ments:
Key Evaluation Enhancements
(1) Global Confidence Ranking: implemented global confidence score ranking for mAP calculation instead of image-level ranking, ensuring consistent evaluation across all frames. This follows the COCO evaluation protocol, where predictions across the entire dataset are ranked globally.
(2)
101-Point Interpolation: adopted 101 101-point interpolation for mAP calculation align with COCo-style evaluation standards.
(3) Pseudo Detection Handling:
fixed calculation errors when handling pseudo detections for scenarios where ground truth lacks certain classes but predictions include them.
(4) Precision, Recall, and F1-score Evaluation: added metrics based on single optimal confidence threshold determined by maximizing the F1 score.
(5) mAP@550:5 Evaluation: added mAP@ 50:95 result calculation, averaging over IoU thresh- olds from 0.5 to 0.95.
(6) Support for Verb and Target: added support for verb (V) and target (T) component evaluations alongside instrument (I) and full instrument-vert verbutarrrr (IVT).
(7)
Video-wise Calculation: added support for video- wise metric calculation across all compo- nents, ensuring each surgical procedure contributes equally regardless its duration.
(8) Bug Fixes:
corrected several issues, including errors in the "list2stack" function from the prior work, thereby improving robustness.
More specifically, the "ivtdmetrics" toolkit integrates common detection metrics into unified package, enabling simultancous computation across instruments (I), verbs (V), targets (T), and full
instrument-verb-targe triplets (IVT). It supports mean Average Precision at a fixed IoU threshold of 0.5 (mAP@ 50) and averaged over IoU thresholds from 0.5 to 0.95 in steps of 0.05 (mAP@ 50:95).
Formally, for each component X E VTV, and class c, the Average Precision is defined as
AP. J/" (r) dr (rapz(pMeer (x), x). linspace 1, 101),
where Pixeem (r) denotes the interpolated precision envelope as function of recall, and trapz indicates trapezoidal numerical integration over 101 uniformly spaced recall points.
The mean Average Precision (mAP) for each component is then
Cx mAPx = - AP Xe (L,
Cx
where C'x the number of valid classes for component X.
Moreover, Precision, Recall, and Fl score computed at single optimal confidence threshold de- termined by maximizing the global Fl score across all classes, and results are reported in percentages with higher values indicating better performance (f).
22

---

Published as a conference paper at ICLR 2026
Video-wise mAP. Besides the global evaluation, we also report video-wise metries. Formally, let AP (1) denote the Average Precision of component X E (T,V,T, TVT) and class computed on video v. Given V videos, the video-wise mAP is defined as
Cx AP (t),
mAPvideo E
where Cx is the number of valid classes for component X. this setting, confidence scores are ranked within each video rather than globally, ensuring that the evaluation is self-contained per procedure. This protocol ensures that each video contributes equally, mitigating bias from variable procedure lengths and the absence of certain rare triplets in different five-fold splits.
Our "ivtdmetrics" toolkit publicly available for reproducibility, with detailed implementation and usage examples provided in the accompanying code repository.
E OUR PROPOSED TDNET NETWORK
Following the progress of surgical triplet approaches (Yamlahi al., 2023; Gui et al., 2023) in classification tasks, we propose a tailored baseline method named TDnet to support future research on surgical triplet detection. This method is designed to tackle the severe class imbalance issue in triplet annotations and to provide a robust baseline for subsequent studies. In the following, we introduce TDnet from two complementary aspects: the network architecture and the overall loss function.
E.1 NETWORK ARCHITECTURE
As shown in Table 6, surgical triplet dataset exhibits severe distribution imbalance across categories. We therefore propose a distillation-base baseline for surgical triplet detection on ProstaTD to facilitate future comparisons. Self-distillation supplies softened targets that smooth the label distribution and encode similarity between classes, which reduces overconfidence on frequent categories and improves calibration for rare ones. We are the first to apply this mechanism at the instance level, tying supervision to each positive detection rather than global image labels. This alignment with the detection objective provides more localized gradients and proves more effective in our setting, especially for rare triplets. In addition, the network jointly integrates auxiliary classification heads for instrument, action, and target at the instance level under the supervision of both hard and soft labels, where the hard labels provide exact supervision to the ground truth while the softened teacher logits act as a regularizer that captures ambiguity and proximity between classes. Mixing the two improves generalization on underrepresented triplets without sacrificing fidelity to the annotations.
The overall architecture of our proposed TDnet is illustrated in Fig. 8. It consists of a teacher network and student network, both built upon the same YOLOv12 backbone and detection head. The teacher is first trained to predict triplet detections together with three auxiliary heads for instrument, action, and target. Instead of running separate detection sample assignment for each auxiliary head, we reuse the positive bounding box predictions selected by the triplet head. Specifically, the main head selects top-k bounding box predictions per ground truth triplet using a score- and loU-based criterion to form foreground S. The same foreground corresponding instrument, action, and target labels from the matched triplet are propagated to these positives. Auxiliary losses are computed with binary cross entropy S while the remaining grid points are treated as background. This shared assignment avoids inconsistent supervision across heads and stabilizes training.
is then applied to the auxiliary heads, and the
Formally, let g E G denote ground truth triplet with box by and class cg. and let a E A denote box prediction at grid point, with predicted box b, and triplet head logit z za(g). We define the alignment score:
sa, g) = (()(()))) loU(ba.b))
(4)
where (-) is the sigmoid function and a, 3 2 0 are hyperparameters. For each we select the top-k bounding box predictions by this score:
Sg Topp { (()a)) a E A},
23

---

Published as a conference paper at ICLR 2026
Detection Head
Foreground
Teacher Extractor
Tripbet
Auxiliary Heads (VVV))
Teacher Soft Labels
D.V.F
Audilaay Heads (IN/T)
LAC MIX
Student Extractor
Iriplet Head (, V,77
Ingut Sequence
Student Predictions
Tunable
Siymaid
Forward
Foneground
Detection Head
Prozen
Label Flaston
Supervise
Figure 8: Overview of TDnet. A teacher and a student share a YOLOv12-styl feature extractor. First, the teacher network is trained for triplet detection. During this stage, positive samples selected by the main triplet head define shared foreground mask, which is used to supervise the auxiliary instrument, action, and target heads. In the subsequent self-distillation phase, the frozen teacher provides softened logits. The student then optimizes all four of its classification heads on this same shared foreground using fused BCE objective. This approach mitigates class imbalance and enhances multi-task performance.
and form the shared foreground set:
S = U- Sg.
ga(a) arg max s(a,g) for predictions selected by multiple Sy- gEg
(6)
The labels for the auxiliary heads inherited from Cy and losses are computed only E S. In all our experiments, we set a = 0.5, B 6.0, and k 10.
In the subsequent self- distillation stage, the teacher network is frozen and produces softened logits for the four classification branches: triplet, instrument, action, and target. The student copies the teacher backbone and detection head and keeps them frozen that bounding box prediction geometry and responses remain identical to the teacher. This preserves the positive set selected by the triplet head, and reuse this shared foreground for all branches during self distillation. Since box regression is not updated in this stage, supervision is purely classification on the shared positives. Each student classification head is trained with a fused binary cross entropy objective that blends the teacher probabilities with the original labels. Using softened targets reduces the dominance of frequent classes and improves calibration, and focusing the loss on the shared positives avoids the large Segative-to-positiv imbalance. Mixing teacher predictions with ground truth stabilizes supervision for rare triplets and yields consistent gains across instrument, action, target, and triplet heads.
E.2 LOss FUNCTIONS
Teacher Network. We reuse the shared positive set S defined in Section E. and let A denote all box predictions at grid points across the three feature levels. Here IVT denotes the joint triplet head that predicts the structured instrument, verb, and target combination, and I, V, T denote the auxiliary heads for instrument, verb, and target, respectively. The teacher is trained with a unified objective that applies box and DFL losses on positives, uses BCE all grid-point predictions for the triplet head, and uses BCE on positives only for the auxiliary heads. Formally, the teacher loss is:
(boox Lrox (a) CDFL +
OIVT G
BCE
Σ AE(LV.T]
OA
(B))(a).
BES
BES
hox and DFL. on positives
triplet BCF all predictions
auxiliary BCE positives
24

---

Published as a conference paper at ICLR 2026
Student Network. In the self distillation stage the teacher is frozen and produces softened logits for the four heads. The student backbone and detection head are copied from the teacher and kept frozen so that the positive set S is identical. Distillation is classification only and is computed on 8 for all heads using a fused BCE objective:
student
E
Σ
(IC)
(8)
RE(IVTINV.T) aES
where the mixed soft target
t (a) ad()((a) + (1 a))a))(a))
E [0,1]
(9)
and
CiCE.Mx(( BCE(2,(a), t ta))
(10)
Here 2 (a) and (a) are teacher and student logits for head h, a(.) is the sigmoid function, and y (a) is the original target. Softened teacher logits reduce the dominance of frequent classes and
improve calibration, and focusing supervision S avoids the extreme negative-to-positiee imbalance
while preserving consistent alignment between teacher and student.
F EXPERIMENT DETAILS
F.1 IMPLEMENTATION DETAILS.
All experiments in Section 6 and Appendix F use an input size of 640 x 640 and are trained on an NVIDIA RTX 4090 GPU. For lightweight detectors such as the YOLO series, we use a batch size of 16, whereas larger models like DETR -based architectures are trained with smaller batch sizes (3-4) to fit memory constraints. The learning rate and warmup schedule are adapted according to the corresponding extractor.
For our TDnet, we use an input size of 640x640 and batch size of 16. We set Abox = 7.5, ADFL = 1.5, and oy = 0.5 for all classification heads. The classification loss weights are 0.5 for the IVT head and 0.1 for each of the I, V, and heads. During self distillation, the mixed target assigns 0.8 to the teacher probabilities and 0.2 to the original labels for triplet recognition. Data augmentation includes horizontal flipping with probability 0.5, random scaling by up to 50%, random translation up to 10% of the image size, mosaic augmentation with probability 1.0, and HSV color jitter consisting of hue variation of 0.015, saturation scaling up to 0.7, and value scaling up to 0.4. Optimization performed using SGD with an initial learning rate of 0.0001, momentum of 0.937, and weight decay of 5 x 10-2. The early-stopping patience is set to 100 epochs. Detailed hyperparameter configurations are available in the released configuration files and training scripts in our repository.
F.2 VIDEO-WISE COMPARISON RESULTS
To provide a more comprehensive assessment, we conducted an additional video-wise mAP evaluation, whose definition is given in Appendix D. This protocol computes metrics per video and then averages across videos, which reduces bias from variation in procedure length and ensures that each case receives equal weight.
As shown in Table 8, the overall ranking is consistent with Table 4. Classical detectors such as Faster RCNN, Cascade RCNN, SSD, ViT-Det, and Deformable-DFr remain clearly behind. Task-specific TAPIR and MCIT-IG also underperform. Advanced detectors including RT-DETR (Zhao et al., 2024) and the YOLO family maintain strong results.
Our TDnet retains the top position, and its advantage is more pronounced under the video-wise metric on the critical triplet score. Concretely, TDnet leads on the other three components (mAP,, mAPv, and mAPT), and on the primary triplet metric it surpasses the runner-up YOLOv12 by 2.4% at mAPivr @0.5 and 2.1 % at which are 1.8 0 and 1.3 1.
exceeding the corresponding margins in Table
F.3 ABLATION STUDY
Effect of Multi-task Learning and Self Distillation. TDnet without self distillation, that is TDnet with only multi-task learning, shows the largest degradation relative to TDnet, most clearly mAPy
25

---

Published as a conference paper at ICLR 2026
Table 8: Video-wise detection performance ProstaTDD. Metrics are computed per video and then averaged across videos. Results are reported aad % over 5-fold cross-validation We report video-wise mAP at IoU thresholds (50 and 50:95) for T, and IVT components. All metrics are in % with higher values indicating better performance (1). Bold text with light green background indicates the best result, and underlined text with light purple background indicates the second best.
mAPt (%) t
mAPv (%) f
mAPr (%)
mAPivT (%) f
Method
50
95
50
95
50
95
50
95
Faster RCNN
72.2a.1
62.76.7
47.5a.s
41.65.3
40.64.5
35.04.1
25.14.0
21.73.8
Cascade RCNN
68,65.3
59.0s.9
43.76.7
38.0s.4
38.64.7
33.14.3
21.14.1
18.23.9
SSD
73.64.9
63.96.5
49.4s.4
43.2s.2
42.4a.4
36.94.
26.43.9
23.23.6
Vit-Det
85.92.3
73.02.3
51.63.8
44.7a.3
46.74.3
40.13.3
29.43.5
25.93.1
Deformablc-DETR
74.3s.0
64.1s.5
49.95.4
43.65.2
42.5a.4
37.14.0
25.8a.9
22.53.6
RT-DETR
91.41.6
80.91.e
60.43.8
54.43.3
52.14.
46.53.8
32.83.8
29.33.3
YOLOv10
88.61.5
80.82.4
61.52.5
56.82.6
50.62.6
46.82.3
34.03.5
31.S2.6
YOLOv11
88.21.3
80.71.6
61.73.0
56.82.9
51.82.3
47.72.7
34.52.6
31.72.6
YOLOv12
88.81.2
80.52.2
61.82.6
56.62.9
51.42.
46.92.2
34.7a.6
31,83.4
TAPIR
75.64.4
65.24.6
51.84.9
45.04.4
46.5g.1
39.94.5
27.84.4
24.13.9
MCIT-IG
76.94.1
66.64.6
53.04.s
46.34.1
47.8s.a
41.24.4
29.14.2
25,4a.8
TDnet (Ours)
90.11.8
81.22.2
63.72.3
58.32.7
53.12.3
48.52.1
37.12.9
33.92.9
Table 9: Ablation results for TDnet. MTL denotes multi-task learning across instrument, action and target. The variant without self distillation uses only MTL. The variant without MTL uses only self distillation. TDnet uses both components. We report global mAP for instrument verb V, target and triplet IVT at IoU 50 and 95. Bold text with light green background indicates the best in each column. The mAP values are computed exactly the same way as in Table in Section 6.3.
mAP.
MAPy
mAPr
mA PIVV
Method
50
95
50
95
50
95
50
95
TDnet w/o self-distill
88.51.a
80.5
59.7s.a
S5.02.4
54.6a.4
50.72.a
34.4a.2
31.8s.a
TDnet w/o MTL
89.5
80.92.0
60.9..0
55.72.1
55.52.1
50.72.a
35.4a.7
32.63.2
TDnet
89.9
81.02.0
61.72.9
56.32.1
55.72.4
50.82.7
36.1a.5
33.13.1
and mAP1vT- Concretely, mAPy@O. drops from 61.7% to 59.7% and mAP @0.95 drops from 56.3% to 55.0%. mAPıvr @0.5 drops from 36.1 % to 34.4% and mAPivr @0.95 drops from 33.1 % to 31.8%, with smaller declines on mAP; and mAPr This suggests that directly optimizing three coupled heads complex, increases interference among instrument, action, and target, and weakens the discriminative capacity of the learned representation.
shared feature extractor without auxiliary guidance makes the objective overly
TDnet without multi-task learning, that is TDnet with only self distillation, underperforms the full TDnet by smaller margin. mAPy @0.5 decreases to 60.9% and mAP, @0.95 55.7%. mAP @.5 decreases to 35.4% and mAPivT @0.95 to 32.6%, again with minor drops on mAP, and mAPr. During distillation we align teacher and student detection boxes, which requires freezing the student feature extractor and the detection head so that the geometry remains stable. This constraint limits feature adaptation and therefore constrains the benefit of self distillation alone.
The full TDnet that combines multi-task learning with self distillation achieves the best values in every column Table Distillation stabilizes features and reduces cross-task conflict, while multi-task losses provide complementary supervision for instrument, action, and target. Although the feature extractor and the detection head are frozen during the box alignment stage, the remaining modules continue to learn jointly, which yields consistent gains, especially mAPy and mAPivr where cross-task dependencies are strongest.
Effect of TDnet MTL head different extractors. We replace the YOLOv12 backbone in TDnet with YOLOv10 and YOLOv11 extractors and measure IVT performance under five-fold cross-validation, Results summarized in Table 10.
26

---

mAPivr,95 Published as a conference paper at ICLR 2026
Table 10: Impact of inserting the TDnet MTL. head different extractors (five-fold cross-validation).
We report IVT-level mAP at IoU 50 and 95.
Method
mAPıvr,50
MAPivI,95
YOLOv10
34.34.1
31.8s.s
TDnet with YOLOv10 extractor
35.13.1
32.63.3
YOLOv11
34.13.7
31.5g.a
TDnet with YOLOv1 extractor
34.9s.5
32.3s.1
Table 11: Sensitivity of TDnet to MTT--seff-distilaation fusion weights on a single fold. We report IVT-level mAP at IoU 50 and 95.
Fusion weights (Awr Asu)
mAPivr,50
0.0 1.0
40.9
37.8
0.2: 0.8
40.8
37.7
0.5 0.5
41.3
38.0
0,8 0,2 (TDnet)
41.6
38.2
1.0 : 0.0
41.4
38.1
For both YOLOv10 and YOLOv11, inserting the TDnet MTL head consistently improves mAPivr IoU 50 and 95, indicating that TDnet provides effective and architecture-agnoste gains.
Sensitivity to self- oif--iitillation fusion weights. We further study how TDnet performance changes under different fusion weights between the multi-task and self-distillation losses on a single validation fold. Let AMIL
AsD denote the relative weights; results given in Table 11.
The proposed TDnet setting (0.8: 0.2) achieves the best mAPIvI. and neighboring configurations perform similarly well, suggesting that the self-distillation fusion is effective and not overly sensitive to moderate weight changes.
Long-tail Generalization and Rare-triplet Analysis. We explicitly analyze long-tail generalization on ProstaTD. Appendix F.5 (Per-class Average Precision of IVT Components) reports per-class AP for instruments, verbs, and targets, showing that TDnet improves both frequent and rare classes. To further assess generalization to rare and unseen IVT combinations, we evaluate mAP on triplets with fewer than 300 instances the training set. Results under five-fold cross- coos--vliiation are reported in Table 12.
TDnet outperforms Y0LOv12 on both global and video-wise metrics for rare triplets, indicating improved generalization to long-tail IVT combinations. Complementing this quantitative analysis, Appendix F.4 provides qualitative case studies. For instance, thread is thin and often low-contrast, so the triplet (grasper, grasp, thread) is frequently predicted (grasper, null, null), illustrating the practical difficulties of rare and visually subtle targets and motivating future research on long-tail modeling in surgical IVT detection.
F.4 CONFUSION MATRIX ANALYSIS
Based on five-fold cross-validation, Fig. 9 reports the TDnet confusion matrix with columns nor- malized by ground truth counts, so each column sums and values denote recall for each true class. Several common and visually distinctive triplets are comparatively easy to recognize and therefore form a strong diagonal, including (scissors.null. null,, (forceps.retract.blaat bladder), and cutssorrscutbbladdee
The off-diagonal structure reveals consistent confusions driven by visual similarity and contextual coupling. Instruments from the same family share geometry and appearance, and single-frame evidence can make actions such as grasp and retract hard to separate. Overall, the majority of confusions occur between triplets involving the same instrument, indicating that most errors arise from misrecognizing the action or target rather than the instrument itself. For example, thread is frequently predicted
is thin and often low-contrast, which explains why prrusper.grasp.three. as (grasper.null.null) with 54.5%. Context can also couple action and target. Suction near the Endobag resembles aspirator near fluid, which leads taspirator.etraat,.nndodo being predicted
27

---

Published as a conference paper at ICLR 2026
Table 12: Five-fold cross-validation performance triplets (fewer than 300 instances in the dataset). We report IVT-level mAP at IoU 0.5, both globally and video-wise.
Method
mAPivr (Global)
mAPıvr (Video-wise)
YOLOv12
19.8
20.7
TDnet
20.6
22.1
TDnet Normalized Confusion Matrix Five-Fold Cross Validation
0.8
0.6
Predicted
0.4
Ground
Figure TDnet confusion matrix under five-fold cross-validation ProstaTD. Columns are normalized by ground truth counts and values show recall for each true class. The x-axis shows ground truth and the y-axis shows predictions. Darker cells indicate higher recall. The diagonal shows correct triplet predictions and the off-diagonal structure reveals within-triplet confusions. Abbreviations in labels are asp for aspirator, driver for driver, clipper for clipper applier, coag for coagulate, and vesicle for vesicle.
as (aspirator,suck,fuuk with 53.8%. In addition, rare triplets with limited examples show frequent mutual confusions such as (driver.grasp.gauzz)
addition, we provide separate confusion matrices for the instrument, action, and target components (Fig. 10). They show that instrument confusions are rare, while actions such as Coagulate/Retract (42%) and Suck/Grasp (38%) are frequently confused, and non-organ targets (catheter, gauze, Endobag) are often missed misclassified other organs. Consistent with the triplet-level confusion matrix, these component-wise results highlight that the key challenges in recognizing actions and targets under long-tailed, visually ambiguous conditions.
28

---

Published as a conference paper at ICLR 2026
Component wise Narnalre Contucion Matricee Cross valcaation carkales Hamb
Cart
3dhuera
cotnda"
farden
99A21
3530
Cressag
bug
9.41
barlgruss
50000
Figure 10: Single component normalized confusion matrices of TDnet under five-fold cross- validation on ProstaTD, for instruments (left), actions (middle), and targets (right). Instrument confusions are generally rare, whereas actions and non-organ targets exhibit strong confusion pat- terns, indicating that action and target recognition under long-tailed distributions remain the main challenges.
F.5 PER-CLASS AVERAGE PRECISION OF IVT COMPONENT
To highlight TDnet's contribution to balancing the triplet distribution, we report per-class APIvr an IoU threshold of 0.50 under both global and video-wise protocols, computed as five-fold means. The full results are listed in Table 13, and the protocol definitions follow Section D.
Overall, TDnet outperforms the YOLOv12 baseline on the majority of IVT categories for both global and video-wiss wise evaluation. The gains appear not only on frequent and visually distinctive triplets but also many rare categories, indicating that the combination of multi-task supervision and self distillation improves class balance practice. While a few classes still favor YOLOv12, 12, the prevailing trend is in favor of TDnet.
These per-class results complement the confusion matrix in Fig. The hardest categories remain those with very low support or subtle action and target cues, which show larger variance. Even so, the per-class improvements align with the aggregate advantages reported in Tables 4 and 8.
Table 13: Comparison of YOLOv12 and TDnet Average Precision (AP) for IVT Instrument-AActonn -arron--reet Classes. Values five-fold cross- validation means. Higher values are highlighted in bold with light green background. Here, "driver" refers to "needle driver", and "vesicle" abbre- viates "seminal vesicle".
Y0LOv12
TDnet
Triplet Class (I-V-T)
Global AP
Video-wise AP
Global AP
Video-wise AP
(scissors, retract, bladder)
23.6
21.0
21.5
18.7
(scissors, retract, catheter) (scissors, retract, vesicle) (scissors, retract, prostate) (scissors, retract, fascias) (scissors, retract, gauze) (scissors, retract, Endobag)
56,8
61.2
71.5
70.2
24.7
25.7
23.5
27.9
20.2
22.4
23.7
24.1
17.9
16.8
17.6
18.7
61.4
61.4
58.3
58.3
36.4
45.0
63.5
54.8
(scissors, coagulate, prostate)
33.0
33.0
0.0
0.0
(scissors, coagulate, fascias)
0.0
0.0
45.8
45.8
(scissors, cut, bladder) (scissors, cut, vesicle) (scissors, cut, prostate) (scissors, cut, fascias) (scissors, cut, thread) (scissors, dissect, vesicle) (scissors, dissect. prostate)
64.4
69.0
65.1
74.4
33.2
29.7
42.2
35.8
60.1
64.1
61.2
65.5
4.8
7.0
7.3
6.8
54.0
45,9
57.5
50.0
48.4
52.2
49.6
55.3
42.7
38.4
42,4
40.7
29

---

Published as a conference paper at ICLR 2026
Table 13 continued from previous page
YOLOv12
TDnet
Triplet Class (I-V-T)
Global AP
Video-wis AP
Global AP
Video-wise AP
(scissors, dissect, fascias)
61.8
57.2
65.0
61.0
(scissors, null, null)
77.2
78.2
78.3
79.2
(forceps, retract, bladder) (forceps, retract, vesicle) (forceps, retract, prostate) (forceps, retract, fascias) (forceps, coagulate, bladder) (forceps, coagulate, vesicle) (forceps, coagulate, prostate) (forceps, coagulate, fascias) (forceps, dissect, fascias)
65.8
54.6
68.7
57.6
53,8
56.1
56.2
60.2
60.5
60.0
62.2
60.8
54.9
41.4
57.8
45.0
34.0
30.4
29.4
30.0
14.1
15.0
19.6
22.0
26.1
30.7
35.3
37.8
9.7
10.1
15.8
14.3
7.9
4.1
8.9
8.4
(forceps, grasp, catheter) (forceps, grasp, prostate) (forceps, grasp, fascias) (forceps, grasp, gauze) (forceps, grasp, Endobag) (forceps, grasp, thread) (forceps, suture, bladder) (forceps, suture, prostate) (forceps, null, null) (aspirator, retract, bladder) (aspirator, retract, vesicle) (aspirator, retract, prostate) (aspirator, retract, fascias) (aspirator, suck, fluid) (aspirator, null, null) (driver, retract, bladder) (driver, retract, prostate) (driver, grasp, Endobag) (driver, grasp, thread) (driver, suture, bladder) (driver, suture, prostate) (driver, null, null) (grasper, retract, bladder) (grasper, retract, catheter) (grasper, retract, vesicle) (grasper, retract, prostate) (grasper, retract, fascias) (grasper, grasp, catheter) (grasper, grasp, prostate) (grasper, grasp. fascias) (grasper, grasp, Endobag) (grasper, null, null)
37.4
42.9
32.7
37.2
51.7
25.9
35.6
26.9
40.9
37.9
37.0
37.0
26.4
30.3
23.4
32.5
28.1
52.8
32.1
62.7
69.6
60.3
70.2
60.6
26.8
22.9
8.8
12.7
54.8
54.8
46.2
46.9
70.8
71.4
73.6
74.3
44.3
35.6
46.6
38.2
3.4
2.3
3.4
2.0
46.6
25.5
46.5
25.9
23.1
25.5
19.4
21.0
63.1
62.8
64.0
64.5
27.7
31.9
30.9
35.9
23.4
25.9
14.2
19.4
54.6
55.3
66.8
64.2
12.8
12.8
27.5
28.4
80.0
76.9
80.6
77.3
40.6
44.8
42.5
45.9
46.7
53.3
40.9
51.7
60.8
63.8
61.4
64.5
22.1
18.9
34.9
24.1
45.6
69.2
36.5
73.8
78.4
73.9
74.1
72.8
59.6
54.1
60.5
58.7
26.1
24.0
17.5
17.9
57.1
71.7
59.5
67.7
45.9
52.6
37.2
46.2
51.6
54.7
51.5
47.3
43.3
42.0
55.8
65.2
67.4
62.1
66.4
62.7
(clip applier, clip, vesicle) (clip applier, clip, prostate) (clip applier, clip, fascias) (clip applier, clip, Endobag) (clip applier, null, null) (Endobag, bag, prostate) (Endobag, bag, fascias) (Endobag, null, null)
63.0
55.1
66.3
61.8
59.9
72.8
65.2
75.7
56.4
51.8
66.5
58.9
54.6
53.9
50.6
49.7
72.8
70.6
76.7
75.3
47.9
65.5
49.9
68.2
28.3
28.3
30.8
30.8
21.8
22.9
24.7
35.5
30

---

Published as a conference paper at ICLR 2026
F.6 COST-AWARE ORDINAL IVT EVALUATION
To move beyond a binary evaluation that counts an entire IVT prediction as wrong as soon as any single component is incorrect, we design an ordinal, cost-aware metric that reflects the different importance of each component (bbox, instrument, action, and target), rather than treating all errors equally. particular, we encode the ordinal preference "instrument action target" directly in the scoring rule, and we follow the standard detection protocol to associate predictions with ground-truth boxes: predictions are sorted by confidence and greedily matched to most one ground-truth box each (per image), with each ground truth used at most once, similar to mAP evaluation. Note that this represents only a preliminary proposal for a cost-aware ordinal IVT evaluation metric, and some implementation details may still require further refinement and justification.
Per-prediction cost. All formulas below are first defined for single IVT class. We perform greedy -usdd matching with threshold 0.5. For each prediction if it cannot be matched to any ground-truth box with ToU 2 0.5, its cost-aware score is O; otherwise, we assign non-zero score:
0, Sive (p),
if no matched GT with IoU 2 0.5,
scorevt(())
(11)
otherwise.
For matched pairs, we assign partial credit to each component:
Sive 0.1 + 0.5 H jinstrument correct] + 0.2 H |action correct] + 0.2 123--(aann (ttarget correct), (12)
where [-] is indicator for correct prediction of the corresponding component. The constant 0.1 reflects that the bbox is already correct under the IoU 2 0.5 condition, and thus indicates that at least "something" is happening in that region, even if the instrument, action, or target labels are fully correct. The weight 0.5 encodes the higher ordinal importance of the instrument, while actions and targets share a smaller but equal weight (0.2 each), and the weights sum to 1.0.
Cost-aware precision, recall, and F1. Let P and G be the sets of predictions and ground truths for this single IVT class. Each prediction p E P is matched to at most one ground truth, and each E g is matched to at most one prediction. Unmatched predictions and unmatched ground truths receive zero score.
We define the cost-aware precision and recall as simple averages of the per-prediction scores:
Paust
Σ
scorevr((P),
(13)
PP]
PEP
Reost
scor
(14)
|g]
DEG
where scorervr(g) is the score of the prediction matched to y (or if g is unmatched). We then combine them in a cost-aware Fl-style score:
2. P.Pust Reost
Fl cost
(15)
Pena + Reost +
with small avoid division by zero.
Single-component scores. Using the same matching for this class, we define separate scores for each component. Unmatched predictions (and unmatched GTs)) always contribute 0 for all components. For matched prediction
scorépbox (p) = 0.1,
(16)
scorer(p) = H finstrument correct),
(17)
scorey (p) = JK |lcctinn correct],
(18)
(corer(p) = IK [taarget correct),
(19)
and score:(p) 0 for all {Bbox, V, T} if is unmatched. Plugging score, into Eqs. equa- tion 13-equation 15 (instead score(yr) yields per-class pe... B and FI for bbox, instrument, action, and target.
31

---

Published as a conference paper at ICLR 2026
Table 14:
Cost-aware FI scores (5-fold class-wise macro average over IVT classes) for bbox, instru- ment, action, target, and overall IVT. The overall IVT score is bounded by 1.0, with maximum of 0.1 from bbox IoU matching and 0.5/0.2/0.. /0. from correct instrumenn/acttion/aagg labels, respectively.
Method
F1 Bbox
F1 cuss
FIY cost
FI
FIM
RT-DETR
0,0491
0.2435
0.0859
0.0876
0,4635
YOLOv12
0.0503
0.2508
0.0856
0.0836
0.4676
TDnet
0.0527
0.2627
0.0895
0.0880
0.4871
Table 15:
Leave-one-source--0 IVT performance across hospitals ProstaTD. For each row, models are trained on the remaining two sources and evaluated on the held-out source. Metrics are IVT-level mAP50, mAP and F1 (in %).
RT-DETR
YOLOv12
TDnet
Leave-out source
mAPwT,.0
mAPivT.0s
F1
mAPIvT.50
mAPIvT.0s
mAPrvT.50
mAPivr.05
ESAD
19.3
16.4
16.8
18.4
16.3
17.5
18.5
PSI-AVA
22.1
20.1
19.9
20.6
18.7
20.7
23.1
20.6
23.4
PWH
15.4
13.2
13.9
16.0
14.3
15.5
16.2
14.4
16.0
Average
18.9
16.6
16.9
18.3
16.4
17.9
20.1
17.8
19.9
Aggregation over IVT classes. The above definitions are all per-class. Let (I...., K} index
IVT classes, and let F F and F1 (k.c) denote the overall and single-componett Fl scores for class computed by Eqs. equation 15 and equation 16-equation 19, Our final reported metrics are the
macro-averages over classes:
K K
K
Floost
F1
FI cost
1
cost
(Js.c)
(20)
K
which we report as the cost-aware Fl for overall IVT and for each single component (bbox, instrument, action, target).
This cost-aware metric is used as complementary analysis tool to standard mAP/mAR, providing an ordinal and decomposable view of how much eтror comes from instruments, actions, and targets.
From Table 14, we observe qualitatively different behavior compared to the single-component mAP metrics. While single-component mAP is computed instrument-wing action-wise, or target-wise, our new cost-aware metric is defined IVT-class-wee wise and thus penalizes all components jointly at the triplet level. As a result, even bounding boxes, instruments, actions, and targets all show noticeable room for improvement, especially for triplets where every component tends perform poorly. Nevertheless, our proposed TDnet consistently achieves the best, indicating that its advantage remains under this stricter, triplet-aware evaluation.
As for more complex ordinal schemes, for example prioritizing certain actions or targets more highly in specifie clinical scenarios, this would require detailed input from expert surgeons and task-specific cost design. We view such class- or scenario-specific ordinal cost modeling as promising future work beyond the scope of the current paper.
F.7 LEAVE-ONR--OURRCF-OLT EVALUATION
To assess cross-hospital domain generalization, we conduct leave-one-source-ourr experiments on ProstaTD, where models are trained two hospitals and evaluated on the held-out one. We report IVT-level mAPs0. mAP3005. and Fl score for RT-DETR, Y0LOv12, and TDnet across all permutations in Table 15.
Compared with the intra-hospital setting, all methods suffer a substantial drop in IVT detection perfor- mance in the mone--nne-ourcee-ee configuration, indicating that cross-hospine hospital IVT generalization is challenging on ProstaTD and highlighting its value as a benchmark for future domain generalization and continual learning methods.
32

---

Published as a conference paper at ICLR 2026
G LIMITATIONS AND FUTURE WORKS
G.1 LONG-TAIL DISTRIBUTION AND RARE TRIPLET CASES
Similar to CholecT50 (Nwoye et al., 2022), our Prosta dataset also exhibits uneven data distribution, with significant proportion of rare triplets. These uncommon instances often arise from situational constraints, such as the time cost of instrument switching, where surgeons temporarily repurpose available instruments to perform alternative actions. In addition, some triplets may be missing from the dataset due to anatomical differences between patients, which can lead to additional dissection steps unconventional surgical techniques. In real-world procedures, the potential number of triplet combinations is far greater, influenced by variations in surgical habits, patient conditions, and even occasional errors. However, these rare and abnormal cases are difficult capture due to their infrequent occurrence. Despite their low frequency, they are highly relevant to clinical safety and robustness, making their inclusion vital in medical datasets. Moreover, most current approaches constrain triplet recognition to fixed, predefined class set, which inherently prevents the identification of unseen or abnormal triplets. This limitation underscores the need for models that independently predict instruments, actions, and targets, and then flexibly compose triplet candidates, rather than relying on rigid class enumeration. Although our TDnet is specifically designed for triplet class imbalance, it is clear that there is still significant room for improvement this problem. Therefore, our ProstaTD dataset provides a venue to address this challenge, facilitating future research in surgical action recognition.
G.2 TEMPORAL MODELING
Using temporal information for the surgical triplet tasks is nontrivial. For the surgical triplet classi- fication task, RiT (Sharma et al., 2023) reports a clear gain on the verb metric mAPy from 62.0% to 64.0%, but only small gain on the full triplet metric mAPIvT from 29.4% to 29.7%, indicating that in their method temporal cues help verbs more than the complete triplet. In our surgical triplet detection setting, we also evaluated a temporal multi-task model in the style of TAPIR (Valderrama et al., 2022) and observed that, under our setup, the results in Table 4 are limited and in some cases worse than the single-frame baseline. Since such temporal modules add computation and latency, this cost becomes nontrivial for real-time or assource-constraine deployment and does not offset the limited accuracy gains observed. Modern pipelines commonly perform detection per frame, while temporal consistency is handled in a downstream tracking stage through identity association and tracking supervision. For instance, the recent SurgiTrack (Nwoye & Padoy, 2025) adopts two-stage approach to achieve fine- ine-grained multi-class multi-instrummen tracking. The first stage focuses on using state-of-the-are art per-frame detection models for initial localization. Subsequently, the tracking stage leverages temporal consistency across frames to maintain object association and provide more precise dynamic trajectories for minimally invasive surgeries. Guided by these observations, we do not add temporal modules to TDnet at the detection stage and instead plan to exploit temporal information in a future extension that integrates tracking, where identity supervision can utilize temporal continuity more effectively.
G.3 CONTINUAL AND CROSS- -DOMAIN LEARNING
Our benchmark currently consists of three datasets from different sources, which naturally differ in visual appearance, recording settings, and annotation distributions. The experiments in Appendix F.7 suggest that models trained on one source do not trivially generalize to the others, indicating substantial room for improvement in continual learning and cross-domain generalization within this surgical domain.
At present, TDnet is designed as a single-domain detector and does not explicitly incorporate mechanisms for domain adaptation, domain generalization, continual learning across datasets (e.g., avoiding catastrophic forgetting when adapting from one hospital to another). Designing architectures and training strategies that can leverage all three sources jointly, or adapt sequentially without performance degradation, is therefore an important direction for future work that ProstaTD and the associated datasets can help systematically evaluate.
33

---

Published as a conference paper at ICLR 2026
G.4 BRIDGING TO ROBOTIC ASSISTANCE AND SIMULATION
Our current work focuses perception, and the role of robotic assistance is only partially explored. Spatially grounded triplet detection particularly relevant for robotic assistance tasks such as aspiration control, where systems must reason about where interactions occur to ensure safe movement, avoid collisions, or trigger alerts. Labels alone are insufficient, and bounding boxes provide the necessary spatial context for action execution, navigation, and enforcing no-go zones.
Recent work such as SRT-H (Kim et al., 2025) has already demonstrated that robots can autonomously perform multi-step surgical procedures with minimal human intervention. In such systems, the robot must operate on structured representations of "what tool is doing what to which anatomy" each moment. Our Prosta dataset and triplet detection task provide exactly this kind of structured, spatially grounded information in the form instrument-actiorn aarront triplets.
Although do not yet integrate promising future direction is to use ProstaTD to supply these triplet-based states for training and evaluating autonomous surgical policies. For example, robot could adapt its aspiration behavior enforce safety constraints directly based on detected instrumentt-aciio--aree triplets around critical structures. Realizing this perception- control integration lies beyond the scope of the present work but is a key long-term goal that our dataset is designed to support.
models into closed-loop robotic systems simulators,
H SUPPLEMENTARY INFORMATION ON ANNOTATION SOFTWARE
H.1 IMPORTED JSON FILE EXAMPLE
In this part, we provide additional details about the imported JSON file format, complementing the description in Appendix C. As mentioned earlier, both annotation tools, Triplet-labelme and SurgLabel. rely this ISON file as input. Once imported, the software automatically parses the file and determines whether an attribute corresponds to image-level or instance-leve annotation based the presence of the keyword "image". . In our surgical triplet detection task, only instance-level annotations are used.
The example below shows the schema used for surgical triplet detection task. In addition, provide three customizable examples to illustrate how users can extend the schema to meet different needs, such as adding spatial position information, switching to image-level annotation, or combining image-level and instance- level attributes. After import, the relevant attributes specified in the JSON file are displayed the right-hand panel of the annotation interface, as shown in Fig. 6 and Fig. This adaptive design allows our tools to support not only surgical annotation but also diverse annotation tasks across other domains.
Imported ISON Label Structures for Our Annotation Software
"tools":
"scissors":
"actions": ["retract", "coagulate", "cut", "dissect", "null"],
"cargets": ["bladder", "catheter", "seminal vesicle", "prostate "fascias", "gauze", "Endobag", "thread", "null"] 1,
"forceps":
"actions": ["coagulate", "retract", "grasp", "disscct", "bag", "suture", "null"],
"targets"; ["blacder", "catheter", "seminal vesiele", "prostate
"fascias", "gauze", "thread", "Endobag", "null"] b,
"aspirator":
"actions": ["retract", "suck", "null"|,
"targets": ["blaader", "catheter", "seminal vesicie", "prostate "fascias", "fluid", "Endobag", "mull"]
b,
"needle driver";
34

---

Published as a conference paper at ICLR 2026
"actions": "retract", "grasp", "dissect", "suture", "nuli"),
"targets": "blaader", "catheter", "seminal vesicle", "prostate "fascias", "gauze", "Endobag", "thread", "null"]
"grasper";
"actions": ("retract", "grasp", "dissect", "null"),
"cargets": ("bladder", "catheter", "seminal vesicle", "prostate "fascias", "gauze", "Endobag", "thread", "null"]
"clip applier":
"actions": ["clip", "null"],
"targets": ["bladder", "catheter", "seminal vesicle", "prostate "fascias", "kndobag", "null"]
"Endobag":
"actions": ("bag", "null"],
"targets": ("bladder", "catheter", "seminal vesicle", "prostate "fascias", "null"]
11 Custom Example add spatial position annotation
"tools":
11
"scissors"; !
17
"actions": I...!,
11
"targets": ....],
"position"; ["ieft", "right", "anterior", "posterior", "null
"]
11
1/
1/ Custom Example 2 switch to image level annotation
11 "image";
11. "phase": ["sutoring", "clipping", "dissection",
11.
11 -/--
1/ Custom Example 3 combine image-level and -estrrment--eeel annotation
1/1
11 "image": {
11 "phase"; ["suturing", "clipping"), "lighting"; ["none", "minor", "severe", ...), "smoke": ["yes", "no"),
17
11
"tools":
"forceps";
"actions"; (...],
"targets": [...],
"position": [...],
"targets";
"DVC": {
"bleeding"; ["yes", "no"]
35

---

Published as a conference paper at ICLR 2026
// Note is not required to restrict the schema to "image" or tools"
// Users can design their own JSON structure depending on the annotation task
1/e eveenn the case natural images or other domains
//Our annotation tool can adaptively handle such structures
// enabling modification on a single frame or across the temporal
dimension
H.2 CUSTOMIZABLE USER INTERFACE FEATURES
Stert
1100
Fie Start (Xt)
PsAd f.4i
(b)
Tel.
Adjust Label Colors
Label
G
sdissors
forceps
128
8/18
0.0
espirator
0
2.0
128
clp spoller
128
5.0
128
Endebag
Cence 00c
(c)
Aijust Font Size
Attributes
Action:
Application Fant Siwe b3
Sdart Jue Tel.
Label Font Size: 10
null
Targez:
Shape Per wiithh:
(a)
(d)
Figure 11: Customizable user interface features our annotation tools. (a) SurgLabel auto- matically generates instrument selection panels based the imported JSON schema. (b) Temporal annotation interface in SurgLabel after tool selection. (c, d) Examples of customizable options in Triplet-labelme and SurgLabel after JSON import, including adjustable colors, font weights, and other visual settings, (e) Instrument selection panel automatically generated in Triplet-labelme according the ISON schema.
After importing the JSON file, both Triplet-labelme (Fig. and SurgLabel (Fig. 7) automatically parse the schema and generate instrument selection panels, as illustrated in Fig. (1 and (e). In Triplet-labelme, users can select a specific instrument instance and then perform instance-level modifications of the corresponding action and target in single frame. In contrast, SurgLabel supports batch modifications through the temporal annotation interface shown in Fig. 1b Furthermore, both tools provide customizable options, such as adjusting bounding box thickness and color, label font size, or transparency, as demonstrated Fig. II(c) and (d). These flexible user interface features facilitate efficient annotation and allow adaptation diverse user preferences and annotation requirements.
I THE USE OF LARGE LANGUAGE MODELS
In the preparation of this manuscript, Large Language Model (LLM) was used solely writing aid for grammar checking and refining unclear expressions, ensuring clarity and coherence in the presentation of work.
36

---

Published as a conference paper at ICLR 2026
APPENDIX REFERENCES
Vivek Singh Bawa, Gurkirt Singh, Francis KapingA, Inna Skarga-Bandurove Elettra Oleari, Alice Leporini, Carmela Landolfo, Pengfei Zhao, Xi Xiang, Gongning Luo, et al. The saras endoscopic surgeon action detection (esad) dataset: Challenges and methods. arXiv preprint arXiv:2104.03/72 2021.
Shuangchun Gui, Zhenkun Wang, Jixiang Chen, Xun Zhou, Chen Zhang, and Yi Cao. Mt4mtl-kd: A multi-teacher knowledge distillation framework for triplet recognition. IEEE Transactions Medical Imaging, 13(4):1628-163) 2023.
Woong Kim, Juo-Tung Chen, Pascal Hansen, Lucy Xiaoyang Shi, Antony Goldenberg, Samuel Schmidgall, Paul Maria Scheikl, Anton Deguet, Brandon M White, De Ru Tsai, et al. Srt-h: A hierarchical framework for autonomous surgery via language-conditioned imitation learning. Science robotics, =0(104)ead)5554, 2025.
Chinedu Innocent Nwoye and Nicolas Padoy. Data splits and metrics for benchmarking methods on surgical action triplet datasets. arXiv preprint arXiv:2204.05253: 2022.
Chinedu Innocent Nwoye and Nicolas Padoy. Surgitrack: Fine- fineinr multi-class multi-tool tracking in surgical videos. Medical Image Analysis, 101:103438, 2025.
Chinedu Innocent Nwoye, Tong Yu, Cristians Gonzalez, Barbara Seeliger, Pietro Mascagni, Didier Mutter, Jacques Marescaux, and Nicolas Padoy. Rendezvous: Attention mechanisms for the recognition of surgical action triplets in endoscopic videos. Medical Image Analysis, 78:102433, 2022.
Saurav Sharma, Chinedu Innocent Nwoye, Didier Mutter, and Nicolas Padoy, Rendezvous in time: an attention-based temporal fusion approach for surgical triplet recognition. International Journal of Computer Assisted Radiology and Surgery, 18(6):1053-1108 2023.
Natalia Valderrama, Paola Ruiz Puentes, Isabela Hernández, Nicolás Ayobi, Mathilde Verlyck, Jessica Santander, Juan Caicedo, Nicolás Fernández, and Pablo Arbeláez. Towards holistic surgical scene understanding. In International conference on medical image computing and compuler-assisted intervention, pp. 442-452 Springer, 2022.
Amine Yamlahi, Thuy Nuong Tran, Patrick Godau, Melanie Schellenberg, Dominik Michael, Finn- Henri Smidt, Jan-Hinriet Nölke, Tim J Adler, Minu Dietlinde Tizabi, Chinedu Innocent Nwoye, et Self-distillation for surgical action recognition. In International Conference on Medical Image Computing and Computer-Assister Intervention, pp. 637-646. Springer, 2023.
Yifu Zhang, Peize Sun, Yi Jiang, Dongdong Yu, Fucheng Weng, Zehuan Yuan, Ping Luo, Wenyu Liu, and Xinggang Wang. Bytetrack: Multi-object tracking by associating every detection box. In European conference on computer vision, pp. 1-21. Springer, 2022.
Yian Zhao, Wenyu Lv. Shangliang Xu, Jinman Wei, Guanzhong Wang, Qingqing Dang, Yi Liu, and Jie Chen. Detrs beat yolos on real-time object detection. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 16965-16974, 2024.
37


Reviews:
[{"review_id": "UVGvrQloYF", "reviewer": "Reviewer_TsFH", "summary": "This paper presents ProstaTD, a surgical triplet dataset composed of 21 prostatectomy videos, annotated with instrument-action-target triplets. The dataset integrates multiple source datasets to increase diversity in instruments, actions, and surgical contexts. To support future research, the authors also propose TDnet, a baseline network with self-distillation and multi-task learning for triplet detection. Additionally, they provide two annotation tools and an evaluation toolkit. The dataset aims to support generalizable surgical action recognition and provide a benchmark for surgical triplet detection.", "questions": "The paper is relatively long, with overly detailed and occasionally repetitive descriptions of dataset statistics, annotation tools, and evaluation protocols. It is recommended to consolidate the key points and emphasize the core contributions, namely the dataset, the TDnet baseline, and the evaluation experiments, to improve readability.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "(1)ProstaTD integrates multiple sources (ESAD, PSI, PWH), covering different surgical styles, instrument usage, and rare triplets, increasing clinical coverage and the complexity of real-world surgical scenarios.\n(2)The authors developed dedicated software (Triplet-LabelMe and SurgLabel) supporting structured triplets, temporal propagation, and batch operations. Both tools are open-source, significantly accelerating surgical video annotation and improving reproducibility.\n(3)The ivtdmetrics package unifies I-V-T and IVT metric computation, supports video-wise evaluation, mAP@50:95, and multi-component statistics, facilitating fair comparisons across models.\n(4)ProstaTD and its accompanying tools are planned for open-source release, making high-quality surgical triplet data and annotation tools widely accessible for algorithm development and benchmarking.", "weaknesses": "(1)A large portion of the manuscript is devoted to describing the dataset composition, annotation tools, and triplet statistics, while the experimental contribution is relatively limited. \n(2)Although the TDnet baseline is proposed, the comparisons are primarily against standard detectors. The paper lacks exploration of different architectures, hyperparameter sensitivity, or deeper ablation studies. Additionally, it does not demonstrate the dataset’s utility on other downstream tasks such as surgical phase recognition or multi-instrument tracking.\n(3)While the long-tail distribution is discussed, the paper lacks systematic strategies or metrics to evaluate model generalization on rare or unseen triplets. Additional ablation studies or case analyses would strengthen the argument regarding the dataset’s ability to support learning from rare events.\n(4)Although the dataset is claimed to be procedure-agnostic, there are no experiments demonstrating its transferability to other types of surgeries. The paper lacks quantitative evidence or experiments to support the dataset’s claimed generalizability.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6H4Gavr8gK", "reviewer": "Reviewer_WshK", "summary": "The paper introduces ProstaTD, a multi‑institutional dataset for fully supervised surgical triplet detection (⟨instrument, verb, target⟩) in robot‑assisted prostatectomy, with 71,775 frames, 196,490 instances, 89 triplet classes, 7 instruments / 10 actions / 10 targets, dense instrument bounding boxes, and clinically defined temporal boundaries. It also releases two tailored annotation tools and an evaluation toolkit, establishes a 5‑fold video‑level protocol, and provides a baseline (TDnet) with instance‑level self‑distillation. On the benchmark, TDnet achieves mAP_{IVT}@0.5 = 36.1% (vs. YOLOv12 34.3%) and F1 = 32.8% (Table 4–5). The work moves the community from frame‑level triplet classification (e.g., CholecT45/50) to full detection with explicit spatial/temporal supervision.", "questions": "Can you report leave‑one‑source‑out IVT mAP/F1 to assess cross‑hospital generalization?\n\nAny plan to add target boxes (or weak labels/segmentation) to better learn instrument–target spatial relations?\n\nHow robust are results under video‑wise metrics in the main text (now in Appendix F.2)?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Data contribution: First full‑procedure, multi‑institutional, box‑supervised triplet dataset with standardized temporal boundaries; strong annotation process and κ=0.82.\n\nBenchmark breadth: Comprehensive baselines (from SSD/Faster R‑CNN to RT‑DETR/YOLOv10‑12) with accuracy‑speed trade‑offs.\n\nTools & reproducibility: Open annotation apps + evaluation toolkit; 5‑fold protocol.\n\nEmpirical insight: Higher concurrency/scene density than CholecT50, stressing realistic IVT detection.", "weaknesses": "Domain generalization not tested: No leave‑one‑source‑out (e.g., train on PWH+ESAD, test on PSI‑AVA). Report would bolster the “multi‑institutional” claim.\n\nMissing target boxes: Only instrument boxes are annotated; lack of target localization limits explicit interaction modeling; consider target boxes/segmentation or relation heads.\n\nMethod novelty modest: TDnet is effective but incremental; limited exploration of explicit relation modeling or stronger temporal modules.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "WV5nRf1hO4", "reviewer": "Reviewer_wQHi", "summary": "The paper presents a novel dataset comprising multi-rater triple annotations and instrument bounding boxes for 21 robot-assisted prostatectomies, totaling 71,000 frames, sourced from multiple centers. Moreover, the paper presents the performance of a novel neural network “TDnet” that has been trained on the dataset. The paper also makes the two annotation tools that were used during the dataset's creation available to the public. The appendix, which is extensive in nature, provides insights into a number of areas. These include the annotation software, the model architecture, the experimental setup and the ablation studies.", "questions": "1. The paper claims no bounding boxes are available in CholecT50. However, the datasets overlaps partially with the CholecTrack20 datasets, which provides bounding boxes [1].\nIs the annotation available from both datasets comparable to the presented annotations?\n2. The paper claims inconsistent annotation boundaries in CholecT50, but at the same time relaxes the temporal definition on “momentary action” to “include up to 2 seconds before and after [] the contact“. How is a consistent temporal annotation between annotators ensured?\nHow is the start and end defined when it is not the exact time of contact?\n3. The dataset merged the triplets of the aspirator sucking fluid, smoke and blood, resulting in the second most common triplet. Especially concerning the intraoperative importance of bleeding, why were these triplets merged?\n4. The paper claims a higher complexity compared to CholecT50 as a single frame on average contains more triplets. This is due to the fact that CholecT50 contains laparoscopic surgeries and ProstaTD robot-assisted surgeries. In the later one of the three attached tools is not being used. This is further reflected in the fact that over one quarter of all triplets are made up by an instrument with null action and null target. Is the complexity of the dataset still higher when these null triplets are factored out?\n5. The benchmark is evaluated using mAP, Precision, Recall and F1-Score. All these metrics only distinguish correct and incorrect triplets. Does the impact of confusion differ between detecting the wrong tool compared only confusing the action? Could an ordinal metric like estimated cost provide further insights into the actual confusion of instruments, actions, or targets?\n\n[1] https://github.com/CAMMA-public/cholectrack20", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. important topic that proposes an extensive multi-center multi-rater dataset driving forward the field of triplet detection, encompassing 89 triplets of 7 instruments, 10 actions and 10 targets on 71k frames\n2. The dataset is compared to other triplet datasets, clearly pointing out its benefits.\n3. Further the paper implements a benchmark using state-of-the-art network architectures and the novel TDnet.", "weaknesses": "1. The paper devotes substantial space to criticizing CholecT50 rather than focusing on the independent contributions of the presented dataset.\n2. The description of the TDnet architecture, the experimental design and ablation study are not contained in the paper but only in the appendix.\n\nMinor Comments:\nM1. The highlighting in Appendix Table 10 is inconsistent, as the higher value is not always highlighted in bold.\nM2. The paper cites 7 arXiv sources. Are peer-reviewed publications for these papers available?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6jF0njlkzG", "reviewer": "Reviewer_PPXn", "summary": "ProstaTD is a large-scale dataset for surgical triplet detection in robot-assisted prostatectomy videos. It contains 71,775 annotated frames with 196,490 triplet instances from 21 surgeries across three institutional sources. The dataset addresses limitations in existing surgical video datasets by providing precise spatial bounding box annotations rather than relying solely on image-level labels, and by implementing clinically grounded temporal boundaries for surgical actions. The annotation taxonomy consists of 7 instrument types, 10 actions, and 10 anatomical targets. All annotations were created under medical supervision to ensure clinical validity. The temporal labelling scheme distinguishes between continuous actions (such as grasping) and momentary actions (such as cutting), which improves annotation consistency compared to prior datasets. The dataset exhibits higher surgical complexity than comparable laparoscopic datasets, with more concurrent instruments per frame reflecting actual surgical conditions. The authors benchmarked multiple state-of-the-art detection architectures on ProstaTD. Results show that fully supervised models using bounding box annotations substantially outperform weakly supervised methods that use only class labels. The authors propose TDnet, a new architecture that combines spatial detection with action-target recognition.", "questions": "How do you plan to address the class imbalance of rare triplets ?\n\nIn which clinical context does object detection offer information that the triplet (with action) classification does not account for?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "The paper uses a cross institutional dataset to create a dataset with detailed surgical annotations supervised by experts. Usually the available datasets in this area come from a single institution and one or few surgeons, and the limited variability is known to affect the results. The additional variability will allow testing the generalisation of these methods more thoroughly\n\nThe dataset will be open sourced and baselines of appropriate methods are provided for the predictive Task. The authors also proposed their own variation of a Neural Network that outperforms commonly used methods in the field.\n\nThe dataset includes bounding boxes of the triplets, which is not offered by previous datasets, and it contains more labeled instances of triplets (tool, action, anatomical object) than any previous dataset. The earlier datasets in the field that offer bounding boxes of triplet instances have less than half the annotated instances than this dataset.", "weaknesses": "The details of the proposed Neural Network, TDnet, are not in the main paper but in the appendix. Although the main contribution is an open source dataset, ICLR is mostly a Machine Learning conference and putting forward an algorithmic proposition for the proposed task would make. \n\nWhile TDnet is sensible, including new dependencies in the labels to the optimisation process, it would also be a lot more interesting to propose substantially more innovative algorithmic propositions. Also, TDnet as the best performing model is significant enough to be presented in the main paper and not the appendix.\n\nI am not sure what is the role bounding box detection in this problem. Usually you need a bounding box to identify interactions between objects in an image or something of that sort. However, if you have triplets that consider action you already cover that with a classification problem. I assume the authors have some aspiration in robotics that was not clarified in the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)