Procedure-Aware Surgical Video-language Pretraining with Hierarchical Knowledge Augmentation
Authors: Kun yuan
Vinkle Srivastav
Nassir Navab
Nicolas Padoy
Venue: NeurIPS 2024 spotlight (2024)
Keywords: Surgical Data Science
Video-language Pretraining
Multi-modal
Surgical Foundation Model
Primary area: machine_learning_for_healthcare
TL;DR: Surgical Vision-language Foundation Model (CLIP-like)

Abstract: Surgical video-language pretraining (VLP) faces unique challenges due to the knowledge domain gap and the scarcity of multi-modal data. This study aims to bridge the gap by addressing issues regarding textual information loss in surgical lecture videos and the spatial-temporal challenges of surgical VLP. To tackle these issues, we propose a hierarchical knowledge augmentation approach and a novel Procedure-Encoded Surgical Knowledge-Augmented Video-Language Pretraining (PeskaVLP) framework. The proposed knowledge augmentation approach uses large language models (LLM) to refine and enrich surgical concepts, thus providing comprehensive language supervision and reducing the risk of overfitting. The PeskaVLP framework combines language supervision with visual self-supervision, constructing hard negative samples and employing a Dynamic Time Warping (DTW) based loss function to effectively comprehend the cross-modal procedural alignment. Extensive experiments on multiple public surgical scene understanding and cross-modal retrieval datasets show that our proposed method significantly improves zero-shot transferring performance and offers a generalist visual repre- sentation for further advancements in surgical scene understanding. The source code will be available at https://github.com/CAMMA-public/PeskaVLP.

83 The learned Procedure-Aware Surgical Video-language Pretraining with Hierarchical Knowledge Augmentation
Vinkle Srivastav Nassir Navab Nicolas Padoy 1.2 University of Strasbourg, CNRS, INSERM, ICube, UMR7357, Strasbourg, France
Kun Yuan 1,2,3
2IHU Strasbourg, Strasbourg, France
CAMP, Technische Universität München, Munich, Germany srivaa,,rivastav, a nassir navab@tum.de
srivastav,
Abstract
Surgical video-language pretraining (VLP) faces unique challenges due to the knowledge domain gap and the scarcity of multi-modal data. This study aims bridge the gap by addressing issues regarding textual information loss surgical lecture videos and the spatial-temporal challenges of surgical VLP. To tackle these issues, we propose a hierarchical knowledge augmentation approach and a novel Procedure- Encoded Surgical BoowledgeAuugeenne (PeskaVLP) framework. The proposed knowledge augmentation approach uses large language models (LLM) to refine and enrich surgical concepts, thus providing comprehensive language supervision and reducing the risk of overfitting. The PeskaVLP framework combines language supervision with visual self-supervision, constructing hard negative samples and employing a Dynamic Time Warping (DTW) based loss function to effectively comprehend the cross-modal procedural alignment. Extensive experiments on multiple public surgical scene understanding and cross-modal retrieval datasets show that our proposed method significantly improves zero-shot transferring performance and offers a generalist visual repre- sentation for further advancements in surgical scene understanding. The source code will be available at https: ptttps://githun /tggttttt.......//g/ wtt.chh..oooomeeeeeeemmeeeeeeeee CAMMA MalAAWA-publi/PPssaaa....
Video-Language Pretraining
Introduction
The recent advancements in multi-modal representation learning, particularly with the introduction of CLIP [52]. have led to the development of models capable of understanding wide range of visual concepts using natural language supervision [34] [1]. The expressive natural language has allowed these models to shift from task-specific to more generalist applications
49
these models are robust, facilitating effective performance across diverse visual tasks without the need for task-specific fine-tuning 68, 81]. However, despite the impressive progress made by these models in the general computer vision domain, the effectiveness of these methods in domain-specific settings remains uncertain.
representations
This concern is particularly relevant to the field of Surgical Data Science (SDS), an emerging interdisciplinary domain that utilizes deep learning and computer vision techniques to analyze surgical data [44 [3 74). key component of SDS is the analysis of intraoperative surgical videos captured through endoscopes or laparoscopes. Analyzing these videos presents several unique challenges compared to the general computer vision datasets. Unlike general computer vision last several hours and capture complex and fine-grained activities within a narrow field of view. This requires development of computational approaches to decompose and model the surgical procedures multiple hierarchical levels, including the entire
datasets [47
52 7 surgical videos
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

(a) Clip-level Pairing
Time
Contrastive Learning
open the vein
Video Embedding
Narration Embedding
bend my needle
Time
Abstract Embedding
g
This is the peritoneum
Keystep Embedding
See the artery here
Average Pooling
(b) Phase-level Pairing
Time
open the vein
bend my needle
g
This peritoneum
Vaginal closure
See the artery here
Child-level
Parent-level
(c) Video-level Pairing
Time
case 56-year- woman with abdominal pain due to a uterine myoma. A Scm subserous myoma is located on the posterior
Exploration
Vaginal closure
Child-level
Parent-level
Figure 1: Illustratsion dallo-language pretraining with hierarchical video-text pairs. At phase- and vidco- level, one parent-level text is paired to multiple child-level texts.
procedure 229 phases 67 16
steps
54
atomic actions 66 8 and action triplets 50 62
Moreover, surgical language involves specialized vocabulary, and annotating videos requires clinical expertise, limiting dataset scalability. Consequently, current deep learning applications are restricted to single-centric, sully-supervised, and task-specific approaches
3 31 50, 55 57
74].
To bridge the gap, recent efforts have focused on creating surgical video-text pretraining datasets by curating surgical lecture videos from online e-learning platforms and pairing them with transcribed narrations using audio speech recognition (ASR) methods. Subsequently, a CLIP-style model (76) is trained contrastively match the video clips to their corresponding textual descriptions. Building on this, the HecVL approach introduces hierarchical texts, including phase-level keystep descriptions and video-level summaries that provide hierarchical goals of the surgical procedure (75). However, challenges persist due to the smaller size of the surgical video-language pretraining dataset, noisy transcribed narrations, limited variability in phase-level descriptions, and strong temporal dependen- cies in surgical procedures, where actions and keysteps occur in specifie routine order. These issues hinder the accurate learning multi-modas modal surgical representations.
To address these challenges, we propose Erocedure-Encodedr Surgical Knowledge-Augmet Language Pretraining (PeskaVLP), which boosts data efficacy and tackles the spatial-tempone challenges inherent in surgical procedures from two perspectives. First, we introduce hierarchical knowledge augmentation to mitigate the problem of textual information loss surgical video- language pretraining datasets. We argue that the internal knowledge of LLMs serves as valuable surgical knowledge base, enriching and correcting text descriptions while preserving the original key concepts and meanings. Therefore, we utilize the large language model (LLM) prompted with different behaviors as an external knowledge base to correct, explain, or summarize the hicrarchical texts in the surgical video-languee pretraining dataset, thus providing diverse and better language supervision for multi-modal pretraining. Additionally, it reduces the risk of overfitting by preventing the text encoder from repeatedly encountering the same keystep texts in each epoch.
Video-
From the pretraining objective perspective, we perform the hierarchical video-language pretraining. as shown in Fig. [ with a novel hicrarchy-specific loss, LecNCE. Specifically, we combine language supervision with visual self-supervisson the clip-level pretraining to introduce additional supervision signals within vision modality, making the pretraining efficient with small surgical dataset [76). At phase- and video-level pretraining, we construct hard negative samples by reversing the order of texts, followed by Dynamic Time Warping (DTW) based loss function to learn the

---

temporal alignment between video frames and texts, thus facilitating the understanding of cross-modal procedural alignment during pretraining.
We summarize our contributions as follows: First, we propose an LL.M-based knowledge augmen- tation to handle surgery-specific textual information loss in the dataset, providing more densely interconnected natural language supervision from surgical lecture videos. Second, our proposed hierarchical video-language pretraining method enforces the understanding of the spatial-temporal characteristics of surgical lecture videos at different hierarchical levels, The pretrained PeskaVLP demonstrates state-of of-t-------aann transferability and visual representation to different surgical scene understanding downstream datasets [67 69] 31. across types of surgical procedures and clinical centers. It also shows strong multi-modal alignment ability through the cross cross-mode modal retrieval task at multiple hierarchical levels.
2
Related Works
Surgical Video-Languaee Pretraining: many works have demonstrated the effectiveness of learning
visual representations from the natural language supervision of corresponding text [7] 70, 77 40 46
42
34
These methods conduct contrastive learning (51) match the video clips (or images) with
their corresponding narrations (or captions). Similarly in the medical field, recent works have started to curate large-scale multi-modal data through hospital-sourced chest radiological reports (28 12
and online platforms [76 27 26), c.g., YouTube and Twitter, to perform vision-language pretraining.
However, these works encounter the sample efficiency issue when handling the smaller surgical video-language pretraining dataset (SVL) 176). Recent works improve the data efficacy and zero-shot
performance of CsIl-style models 48 37 25]. However, they do not capture procedural dependency
from the long-form surgical videos beyond the video clip and text matching. Hierarchical pretraining methods 79 75 propose to pair video clips of different durations different hierarchical levels of texts, covering both short- and long-term understanding. Paprika [8) builds a procedural knowledge graph and elicits the knowledge node during the video-language pretraining process.
Textual Augmentation with Knowledge Base: the success of vision-language pretraining is highly dependent on the quality and quantity of available multi-modal data. Recent research 38 shows that a smaller high-quality dataset can outperform a larger low-quality dataset. Common practices improve the quality by textual augmentation, including EDA (37], masked token modeling [65, and captioning loss (72). Recent studies have used synthesized captions from captioming models to achieve notable improvements B3 32 (88. However, they show scalability deficiency and world knowledge loss in models trained with synthetic captions [73]. which their initial benchmark success has largely obscured. To inject the knowledge, K-Lite 63 enriches the texts with WordNet [T5]
and Wiktionary [45] knowledge base. Merlot [78 learns script knowledge representations from millions of YouTube videos, however, knowledge domain gap exists when applying this to the surgical field. The recent advent of large language models like GPT4 (2) and Llama series [66] have been a game-changer, as they encode rich domain-specific knowledge, e.g., clinical knowledge (64). motivating LaCLIP [1] to augment textual inputs through the LLM rewrites.
3 Approach
3.1 Dataset and Contrastive Learning
Learning joint video and language embedding space requires large-scale video-language dataset, however, such datasets are expensive and time-consuming to create in the surgical field. Therefore, the first surgical video-language pretraining dataset, i.e., SVL (76). is proposed by obtaining around a thousand surgical lecture videos from surgical education platforms. SVL collects ~-300 hours of lecture videos accompanied by narration texts obtained using Audio Speech Recognition (ASR) methods, providing 266 video clip-narration pairs for contrastive video-language pretraining. Specifically, short video clips F. and their corresponding narration texts ya are treated as positive pairs P", and the unpaired ones are treated negative pairs A", Then, the contrastive training loss
3

---

[a] Build Knowledge Base
Multi recurrect right inguinal hernia: advantages of the laparoscopic TAPP approach
right inguinal hernia: advantages
laparosoopi: approach
surgeon starts making an incision the right inguinal ingunaarrggon patient
Deep infitrraiing endometriosis (DiE]: Iaparoscopic
treatment with intestinal resection and
GPT4
b. Through the incision, the laparoscopa and other special surgical tooks are
sparing
Deep infiltrating endometriosis: loparoscopic treatment with intestinal resection
Laparoscopic uterine artery clipping origin
N. Laporoscopic uterine artery clipping at origin
10445 917
(b) Hierarchical knowledge Augmentation
patient was admitted with dysphagla, nausea and postoperative vomiting. These
This video belorgs to the General and Digestive, specifically focusing and duodenum surgerfes. The showcases Gastric divertic.lum.
Abstracts
symptoom associated
GERD. Exploration demonstrated
presence disease: gastric diverticulum.
Time
Manipulating
crus, often using dissection tools retractors forceps.
Keysteps
Exploration
Mobilization of right crus
Explain
contrastt swallow follow through confirm...
the intervention begin opening....
small left not presecve. ...
the true identily pertinint
SKB
Proceed with the crus dissection at the level of the diverticulum taking
Narrations
not injure adjacent structures
Figure 2: Hierarchical Knowledge augmentation for hierarchical texts. (a) the process of building surgical step knowledge base. (b) the process of improving hierarchical textual quality based on L.M.M
InfoNCE 51 can be formulated as follows:
Σ
exxo) ((zo))g(va)) g(un)
(Re:8n)EPT
LinfoNCE max f.9
log
(I)
Σ
( g(un) + E el(x)) olrl
(ZorVa JEP
where B represents the batch size. The f and g are visual and textual encoders that generate embedding vectors for videos and texts, respectively. This loss function aligns two modalities by increasing the cosine similarity between paired videos and texts and decreasing the unpaired ones, as shown in Fig. [ (a). Despite reaching an impressive data scale, the imprecision of the ASR system and the scarcity of surgical lecture videos limit the natural language supervision from SVL. Therefore, He 775 proposes to incorporate hierarchical language supervision by extracting additional phase-level keystep and video-level abstract texts from lecture videos' metadata, as shown Fig. [ (b) and (c). this work, we use this hierarchical video-language pretraining dataset and perform hierarchical knowledge augmentation to improve the textual quality.
3.2 Hierarchical Knowledge Augmentation
Quality of language supervision in the multi-modal representation learning matters [ 37
36],
especially when the surgical video-language dataset not "big" enough, e.g., millions of multi-modal samples used in [52, 47 to sufficiently cover the visual-linguistic concepts. In this work, we find that the texts suffer from different types degradation at different hierarchies, failing to provide accurate and broad concepts for pretraining. Specifically, as shown in Fig. 2 narration texts are mostly sentence fragments and casily affected by misspelling errors, therefore altering the original key concepts. The keystep texts mostly short and abstract, resulting in narrow set of linguistic concepts that could show poor transfcrability to the downstream datasets, which usually come with different set of concepts [63] 18 The abstract texts sometimes include redundant and useless information, such as author and citation information.
To address the above hierarchy-speciff textual degradation, we propose a hierarchical knowledge augmentation to correct, explain, and summarize narration, the keystep, and the abstract texts, respectively, by eliciting LLM's encoded surgical knowledge [64]. For each hierarchy, we manually design the system prompt and several input-output examples for L.LM. Thus, we obtain hierarchical LLM assistants with different behaviors of using internal surgical knowledge augment the texts:

---

[a] Cli Clip-level
Contrestive Less
open the vain
my
Augment
This is the peritoneum.
See the artery here
Contractive Loss www.ee
Phase/Video- level
patient was admitted with dysphagia, nausea and postoperative vomiling These symptoms with typical GERD. Exploration demonstrated the presence rare disease: gastric diverticulum.
Parent-level
Aug
by
By
Child- lovel
Exploration
Dissection of greater curvatture
fundoplication
DTW(C)
Video
Child-level
Reversed
Fundopiication
Dissection greater curvature
Exploration
8
by
b2
bg
DTW(CI
Figure 3: The pretraining pipeline different hierarchies. We combine language supervision and visual self- supervision at clip-level pretraining. We conduct the procedure-awei aware contrastive learning at phase/video-les pretraining.
Narration. We ask the LLM to behave "recipe" to come up with list of sequential steps that complete the given surgery. For each lecture video, we feed its title as input and obtain the list of pseudo steps, as shown Fig. (a), building surgical step knowledge base. Then, assign these pseudo steps to narration texts based on textual similarity. This implicitly corrects the typos in transcribed narrations and augments the textual input based on the LLM's surgical knowledge.
Keystep. As shown in Fig. (b), we ask the LLM behave like a "dictionary" explain the meaning of the keystep. Specifically, the LLM assistant expands the given keystep into a description of the main surgical events, anatomies, and instruments involved. This enlarges the textual semantic information of each keystep and provides more expressive language supervision for pertaining.
Abstract. As shown in Fig. 2 (b), we ask the LLM to behave like "summarizer" that captures the key concepts of the given abstract texts, e.g., surgical type, anatomies, and the length of the textual inputs while maintaining the main concepts of the abstract paragraph. In the following experiment, randomly input the original augmented texts for video-language pretraining, Check Appendix H for examples of pre- and post-augmented texts.
This reduces
3.3 Procedure-aware Surgical Video-language Pretraining
We introduce PeskaVLP procedure-aware pretraining framework for the above surgical knowledge- augmented video-language dataset. We emphasize devising pretraining objective LeeNCE for the hierarchical video-text pairs. For clip-level pretraining. LecNC Felip combines language supervision with visual self-supervision to improve data efficiency and boost the scene understanding on visually similar laparoscopic images. LecN C ('Eaaase/vvd considers the procedure awareness during the coarser-level pretraining, through a DTW-based contrastive regularization objective with temporally reversed text sequences negative samples. We apply the dual-encoder model architecture.
3.3.1 Clip-level Pretraining
Language Supervision. The common pretraining objective for dual-encoder model is InfoNCE [51], as denoted in Eq. where matched video text pairs are treated as positive while
5

---

all other pairwise combinations in the batch are regarded as negative, In this work, we also ap- ply InfoN C'E maximize the similarity between short-term video clips and their corresponding narration texts at the clip level, denoted as dlpppp However, this simple objective is data hungry and sensitive to the weakly aligned noisy video-text pairs from small-scale surgical vidco-language datasets, such as SVL (76].
The proposed PeskaVLP approach introduces an additional supervision signal from visual self-supervision to complement noisy language supervision. Specifically, explore the widespread supervision within visual modality to learn generic visual representation. We adopt the simple yet effective SimSiam 11 strategy that aims maximize the similarity between two augmented views. As shown in Fig. 3 (a), during the pretraining, we apply random distortion on the frames of video clips and generate
Visual Self-supervision.
augmented embedding vectors for one video clip. We then apply InfoN C'E to maximize the similarity of these two augmented embeddings by treating them as positive pairs, denoted Liip. This additional supervisory can learn visual features more efficiently and is robust to the distortion of surgical scene images. Finally, the LecNCE loss for clip-level pretraining is the sum these two losses, denoted as LecNCEdip = disp + Lee
clip.
3.3.2 Phase-/Video-level Pretraining
The surgical video-language pretraining presents a unique procedural challenge compared to the existing video-language methods [19] 47 52,
71, 61). The surgical actions and events oceur in a certain order to follow the routine to complete the surgical phase and surgery, e.g., "hook dissecting cystic duct" should happen before "clipper cutting cystic duct" in the "clipping cutting" phase of cholecystectomy surgery. However, prior contrastive learning objectives [46 52 19] omit this temporal dependency and limit the understanding of procedural knowledge in surgical lecture videos.
Our proposed LecNC'E' training objective enables procedural understanding in phase- and video- level pretraining by considering the cross-modal temporal alignment between video frames and text sequence. Specifically, hierarchical texts can form the parent-child correspondence, i.e., abstract (parent-level) and keystep (child-level) texts, keystep (parent-level) and narration (child-level) texts. As shown in Fig. 3 (b). each parent-level text A is paired with a video segment V =fe1....77 }. where the T is the number of frames of the video segment. A is also paired with a child-level text sequence B {bb,...b, where N is the length of this sequence. Then, we build the cost matrix C' E x between video frames and child-level text sequence based on their embeddings. with each element Cs,j computed by distance function D. We adopt the same distance function from (1]
Ci2 D(e,, by) log exp(v.Tb,// b2/3) bR/3)'
v. (od/////f()) b. = s(bbb))/l9)
Ex k= exp exp(V,.b./3)
Using this cost matrix C, we apply Dynamic Time Warping (DTW) to find the minimum cross-modal cost path that aligns the video frames to the text sequence, denoted as DTW (C). We then make reasonable assumption that the global semantics of the text sequence and its reversed version are distinct. Therefore, aligning the video frames to the text sequence should be easier, i.e., incur a lower alignment cost compared to aligning the same video frames when the text sequence is played in reverse. Following this assumption, we temporally reverse the child-level texts into B = (bn,...b)) and build the cost matrix Ć' between V and B, computing the minimum alignment cost DTW We then devise a DTW-based contrastive regularization using hinge loss as follows:
Law max -DC))
where @ the margin between positive and negative samples. This imposed regularization can support fine-grained multi-modal representation learning from weakly paired video frames and texts via temporal alignment. Unlike Paprika [80] which relies on pretrained model [46), our phase-/video-level pretraining provides direct, lightweight, and more adaptable methodology to unseen surgical domains. We do not require the adaption from any existing models, improving the generalization capability. Also, our pretraining process is procedure-avee aware in itself rather than modifying the representation in a second step, streamlining the process and increasing efficiency. We also apply the InfoNCE loss to maximize the similarity between the paired parent-level text, video segment, and child-level texts, denoted as Lineonce- Note that the fonce follows the same pipeline as in Fig. [ (b) and (c). Finally, we achieve the loss LecN CE for phase- video-level

---

Lin fonce + ALdtu, where À is the hyper-paramett parameter to scale two losses. Please refer Appendix for more details about dynamic time warping. Finally, we train the model in an alternating way, using the proposed hierarchical levels of learning objectives. We only train one set of visual and textual encoders for all three levels, ensuring the encoders are optimized for capturing both short-term and long-term semantics. We alternatively train with 25 batches of clip-level samples, followed by 15 and 115 batches of phase- and video-level samples.
pretraining as CecaNCEppes///iieiie
Experiments
Datasets. Our pretraining conducted on the videos SVL [76] dataset. The pertaining dataset includes hierarchical textual annotations from the metadata of the videos 75 We evaluate our model on publicly available surgical phase recognition downstream datasets, i.e., Cholec80 [67 (cholecystectomy from Strasbourg center, AutoLaparo [69] (hystercctomy) from HongKong hospital, MultiBypass 140 [31] (gastric bypass) from both Strasbourg (StrasBypass70) and Bern (BernBy- pass70) centers. These datasets contain untrimmed surgical workflows with frame-wiss phase labels. We also evaluate pretrained model on the cross-modal retrieval task in multiple hierarchical levels with holdout videos in SVL-Retrieval 76 Check Appendix A for more details about pretraining dataset.
Training Parameters. We utilize the dual-encoder architecture with ResNet50 23 as visual encoder and ClinicalBert (24] as textual encoder, respectively. We train the model with a batch size of 120/80/25 for clip-/phase-/video-- iphase-/video-leverr We use Adam optimizer [30] with a learning rate of 5e - 5. We train the model with 4 NVIDIA A100 GPUs each having DRAM of S0 GB for 200 epochs. Temperature parameter 3 for distance function and ó for DaW-base contrastive loss function are fixed as 0.1. Scale factor is set as 0.01.
respectively. We sample 4/16/64 frames for videos of clip-
Evaluation Setup. We evaluate pretrained models using two setups: Zero-Shot evaluation and Few/Full-shot Linear Probing evaluation. For Zero-Shot, we utilize class text prompts, the same as He 75 to compute cosine similarities between image embedding and class text embeddings, classifying images based on the shortest distance. In Linear Probing, the pretrained visual encoder remains frozen when we extract features for each image, subsequently training a linear layer using the SGD optimizer. For few-shot linear probing, train the linear layer with a few numbers of videos, referred to as k-% training, where indicates the percentage of the vidcos used in training. Check Appendix B for more details.
Table Zero-shot phase recognition results. We report Accuracy F1-Score. PeskaVLP outperforms the other methods across different tasks. We report the state-of-the-ar methods that are fine-tuned on the downstream dataset sully-sup manner. However, models fine-tuned on specific downstream datasets show limited generalizability across procedures and institutions.
Model
Dataset
Cholec80
Autolaparo
StrasBypass70
BernBypass70
Average
Cholec80
90.3/-
-/-
-/-
Trans
Autolaparo
82.0/-
-1
BernBypass
-/-
57.3 32.7
85.3/ 62.4
ResNet50
31
StrasBypass
-1-
-1-
90.2 / 79.9
56.7 29.5
-1-
MIL-NCE [46
Howto 100M 100M
7.8/73
9.977.9
5.6/3.1
2.4/2.1
6.475.1
CLIP400M
30.8/13.1
17.479.1
16,9/5.5
14.8/4.1
19,978.0
CLIP 52
Scratch
29.4/ 10.4
15.3
10.9 10.9
6.3 3..5
4.9/
14.0/6.8
SVL.
3
18.2/
15.8
17.8/7.1
21.6/ 12.9
Surg VL.P 76
SVL.
34.7/24.4
21.37 16.6
1..876.9
11.477.2
19.6 13.8
Vl. [75
SVL.
46.7/26.3
23.37 18.9
26.97 TT.3
22.8/13.6
28.77 19.3
PeskaVI.P
SVL.
45.1/34.2
26.5723.6
46.7 / 28.6
45.7722.6
41.07 27.1
4.1 Zero-shot Surgical Phase Recognition
High-qualis Surgical Video-language Dataset. As shown in Table 1 our approach achieves
significant performance improvement over the baselines MIL-NCE [46 and CLIPP (52) pretrained on the natural computer vision datascts, even though pretraining dataset is 10, 000 times smaller

---

than those. Note that when the CLIP model is randomly initialized and then trained with SVL, its performance declines compared to initializing from OpenAI. This shows that our surgical video- language pretraining dataset lacks the scale necessary to adequately pretrain robust video-language model from scratch. ViT 13 9 architectures are sensitive to initialization and excluded from this work. Further insights into the impact of initialization can be found in Appendix C.
Transferability across Surgical Procedures and Centers. Compared the HecVL, our method achieves over 12.3% and 7.8% improvement in absolute accuracy and fl, thanks to our spatial- temporal LecNCC learning objective across multiple hierarchies. Also, the consistent boost on cholecystectomy [67]. hysterectomy [69], and gastric bypass [? procedures show the generalizable and transferable features of Peska VLP. Comparing the results of StrasBypass and BemBypass, we find that Peska VLP can recognize the phases of the same kind of surgery (gastric bypass), even if these surgeries are performed in different centers and follow different procedural routines. More qualitative results can be found in Appendix F.
4.2 Zero-shot Cross-modal Retrieval
Table 2:
We present cross-modal retrieval results the holdout videos, highlighting the best performance in each setting in bold. We additionally include coarser-grained phase-keystep and abstract-video text pairs assess long-term term video and high-level textual understanding.
Clip-Narration
Phase-Keystep
Video-Abstract
method
R@I
R@5
R@ 10
R@1
R@5
R@
10
R@D
R@5
R@ 10
Text-to-Image
CLIP 152
2.9
5.2
6.7
1.7
3.2
6.3
1.2
11.7
25,8
SurgVL. 76
2.8
11.8
16.1
1.6
6.8
11.6
1.3
8.2
15.5
He 77
2.7
11.3
17.2
3.9
13.7
21.3
28.2
74.1
82.3
Peska VLP
3.2
13.2
23.3
6.1
21.0
35.4
38.8
75.3
85.9
Mmage-to-Texs (%)
CLIP |52
1.8
3.9
6.0
0.3
1.2
2.7
0
7.0
16.4
Surg VLP 76
1.3
8.6
13.5
1.0
4.1
7.3
1.3
8.6
14.6
HecVL 75
2.1
9.0
16.2
1.9
8.3
14.8
21.2
65.9
71.8
PeskaVLP
2.4
13.1
21.3
3.4
14.9
24.8
38.8
75.3
81.1
our study, we evaluate pretrained models' cross-modal alignment efficacy by conducting both zero-shot text-to-image and image-to-text retrieval tasks in multiple hierarchical levels. We report the Recall@N metric by identifying the retrieved nearest neighbors for each query and then determining whether the corresponding ground truth element is within the top N nearest neighbors, where N (1,5, 10}. Table 2
shows that our Peska VLP achieves superior performance due to the procedure- aware learning objective in hierarchical pretraining. Particularly, the hierarchical pretraining scheme significantly boosts the cross-modal retrieval at the coarse-graine saarse-grained video-text pairs, comprehending the relationship between long video segments and high-level sentences with surgical terms.
4.3 Few-/Full-shot Linear Probing
General Visual Representation for Surgical Scene Understanding. We present the few- and full-shot linear-probing evaluation in Table It shows that the learned visual representation from PeskaVLP provides general visual representation for surgical scene understanding across surgical procedures. We also find that the MoCo v2 [55] 22 pretrained on the frames of the SVL dataset (second row of Table 3) in a visual self-supervised manner achieves better visual representation than pretraining on public dataset that only contains one type of surgery, e.g., Cholec80 (third row in Table 3). This shows that the cross-procedure surgical pretraining dataset enables better generalizationability.
Knowledge Augmentation and Hierarchical Pretraining. Interestingly, the model pretrained contrastively with short video clips and narrations (SurgVLP) performs worse than MoCo v2 (second row in T in linear probing evaluation. This may because the noisy narrations do not provide accurate natural language supervision for visual representation learning, thus highlighting the
55 22

---

Table 3: Linear- proaang evaluation results. V: supervision is from visual frames. L: supervision is from natural languages. VL: supervision from both visual and language entities.
Model
Dataset
k-%
Cholec80
Autolaparo
StrasBypass70
BemBypass70
100
66.4 / 54.9
57.5/ 44.9
66.2 53,6
64.7/310
ImageNet
ImageNet (V)
10
57.4/ 07.4/42.1 42.3
44.9 30.4
53.3 42.1
53.3 25.6
100
68.2
48.4
71.6/ 58.1
69.6/ 36.5
MoCo v2 55
SVL (V)
10
57.6/43.
49.9/34.6
43.1
59.1 29.1/
100
62.8
51.3/37.4
67.8 / 55.4
MoCo 55
Cholec80 (V)
10
69.6 56.9
45.4/31.7
58.1/45.2
52.7 2
100
64.8/ 50.7
58.5/46.1
63.4 50.6
63.1
CLIP 52
NA (L)
10
57.5/40.0
46.2/31.4
54.3/422.
100
65.97
53.1/42.1
55.17
68.2/35.2
CLIP 92
SVL (L)
58.9/42.3 10
35.3/35.3
18.2/45.2
/
100
50.3
54.3/418
65.87
66.5/34.
VLP 76
SVL (L)
10
55.0
48.5/32.0
57.0/ 44.0
28.5
100
53.2
56.9/44.2
69.8 54.9
70.0/34.4
HecVL 75
SVL (L)
10
56.1
/
60.2 46.8
59.3/
100
69.97
63.1/49.7
71.47 59.5
71,57 37.4
VLP
SVL (VL)
10
50.9
53.1 36.8
63.8 / 50.4
62.9 / 32.7
Table Ablation study on different modifications. Knowledge: knowledge augmentation applied to the pretraining dataset at phase-level (P) and video-level texts (V). P/V: procedure-award pretraining learning objective at phase and video-level. the integration language and visual superviserii at clip-level pretraining. We report 10%-shot linear probing in this table.
LecNCE
Knowledge
Zero-shot
Linear-probing
P/V
C
V
Cholec80
Autolaparo
Cholec80
Autolaparo
X
X
X
X
41.7 26.3
23.3 / 18.9
56.1 40.3
/
V
X
45.5 / 31.0
28.3 25.3 / 20.0
-1 4 34.7
X
X
V
V
42.4 28.1
24.9/ 20.4
58.1 43.2
V
V
43.4 30.3
60.4 48.6 -/-
53.8 39.2
V
V
X
44.0 / 31.8
-/-
-1
V
V
43.7 30.6
-1-
-1-
-1-
V
V
45.1 / 34.2
26.5
61.9/ 50.6
53.1/36.8
StrasBypass70
BernBypass70
StrasBypass70
BernBypass70
/
26.9/ 18.3
22.8 13.6
60.2 46.8
59.3 31.2
x
/
32.3 21.2
23.8 17.5
62.6 / /4.77
60.3 32.3
/.
39.8 / 23.7
25.7 21.33
63.5 48.6
62.2/
45.1 / 34.2
26.5 23.6
63.8 50.4
62.9 / 32.7
importance of visual self-supervision and textual quality. Our model surpasses the prior methods by large margin, showing the efficacy of our hierarchical knowledge augmentation, which denoises the text and improves textual quality. Also, our proposed Lec. promotes the visual encoder through additional visual self-supervision and procedural understanding. We present t-SNE visualizations of learned features in Appendix E, which shows that our multi-modal representations exhibit smaller modality gap, enhancing transferability to vision-and aoson-and-lamnn languag downstream tasks 20
4.4 Ablation Studies
Effect of Knowledge Augmentation. Table presents the effect of our proposed LLM-based hierar- chical snowledge-aware augmentation strategy, applied to the texts of SVL dataset. The first row of the table corresponds to HecVL (75] pretrained on SVL with only conventional visual augmentations, e.g., blurring and so on, without any knowledge augmentation. The results clearly demonstrate that simple visual augmentation strategies exhibit poor robustness as the texts of SVL. are noisy and not diverse enough. Conversely, our knowledge-awar text augmentation consistently improves performance across multiple surgical datasets, highlighting the importance of the textual quality of

---

the surgical video-language pretraining dataset. We found that integrating visual self- supervison with language supervision significantly enhances performance in surgical scene understanding tasks across downstream datasets. Additionally, using procedure-aware learning objective improves surgical phase recognition for routine procedures, such as cholecystectomy (Cholec80), more effectively than complex procedures, like hysterectomy (Autolaparo).
Efffet of Pretraining Objective. Table shows the impact of our learning objective for hierarchical surgical video-language pretraining. When we append visual self-supervision to language supervision the clip-level pretraining. the zero-shot performance is clearly improved. This improvement can be attributed to the added diverse and high-quality supervision. Also, the boost at linear-probing evaluation shows that the combination of language supervision and visual self-supervision leads to a robust visual representation especially with a moderate size of surgical video-language dataset, e.g., SVL.. Table 4 also highlights that the inclusion of LeeNC E with procedure understanding consistently improves performance across most downstream datasets, leading to enhanced accuracy in both zero-shot and linear- aneear-poobing However, performance on the Auto Laparo degrades with this modification. This may be due to challenging or less routined surgical procedures in the pretraining dataset.
5 Conclusion, Limitations and Broader Impact
Conclusion. We have introduced surgical video-language pretraining method for long-term surgical lecture videos and their hierarchical paired texts. Our proposed knowledge augmentation addresses the hierarchical textual information loss by integrating the large language model's internal surgical knowledge, Also, we propose a novel spatial-temporal pretraining objective for video-text pairs of different hierarchies, which addresses the lack of supervision signals problem in small surgical dision-languaae dataset. The proposed LecNCE also addresses the procedural awareness problem, benefiting the long-term cross-modal understanding. The experiments show that our proposed Peska VLP achieves the state-of-the-an art generalized zero-shot ability and visual representation learning that can serve as a general initialization for many surgical scene understanding tasks.
Limitations. While our LLM Lugganeeeetedd strategy enhances textual information, it may overly standardize the text, raising concerns about overfitting during pretraining. Therefore, it is crucial to strike a balance between leveraging LLM capabilities and maintaining the variability present in real-world surgical narratives. To address this, future work will explore incorporating diverse audio inputs and spontaneous narratives into the pretraining process, ensuring that the model retains robustness and adaptability in real-world applications. Additionally, even though the SVL pretraining dataset covers diverse laparoscopic surgeries, it lacks surgeries in different organs, such as the brain and heart. To address this, we plan to expand the pretraining dataset using diverse media such as textbooks, instructional videos, and intraoperative video recordings from diverse sources. We also aim to diversify the pretraining dataset by considering laparoscopic, endoscopic, and microscopic surgeries on different organs, to further mitigate the risk of overfitting and enhance the model's 's generalizability.
Broader Impact. The primary goal of surgical data science is to develop novel context-aware support systems for the operating room by collecting large-scale surgical data and analyzing with modern AI techniques, eventually improving the safcty and efficacy of surgical outcomes. The recent advancements in lision-language-basee multi-modal AI offer significant potential in achieving this goal by enabling the development of more robust and generalizable models. These multi-modal systems have the potential to support clinical decision making, streamline surgical workflows, provide real-time intra-operative guidance to improve surgical precision, reduce errors, and optimize outcomes the operating room. During the development, patient data privacy should be considered as a fundamental ethical requirement. These systems developed on real-world surgical data also hold transformative potential in medical education, enhancing training and skill development in both novice and experienced surgeons.
preision-making,.
10

---

Acknowledgements
We would like extend our deep appreciation to the education platforms, such as Websurg (IRCAD), EAES, and YouTube, for their dedication to providing high-quality educational content freely accessible to learners worldwide. We are especially grateful to the clinicians who have generously contributed their time and expertise to create and share content on these platforms, making this research possible.
This work has received funding from the European Union (ERC, CompSURG, 101088553). Views and opinions expressed are however those of the authors only and do not necessarily reflect those of the European Union or the European Research Council. Neither the European Union nor the granting authority can be held responsible for them. This work was also partially supported by French state funds managed by the ANR under Grants ANR-20- WER-20-CHIL 22-CHHA--OOO CWHAA-002--0 and ANR- 10-IAHU-O2- This work was granted access to the HPC resources of IDRIS under the allocations AD01 1013744R,,,, ADO1 101 13,, and ADO1 101 163 144 made by GENCI. The authors would like to acknowledge the High-Performen
Computing Center of the University of Strasbourg for supporting this work by providing scientific support and access to computing resources. Part of the computing resources were funded by the Equipex Equip Meso project (Programme Investissements d' Avenir) and the CPER Alsacalcul/Big Data.
References
[1] Amro Abbas, Kushal Tirumala, Dániel Simig, Surya Ganguli, and Ari Morcos. Semdedup: Data-cfficient learning at web-scale through semantic deduplication. arXiv preprint arXiv:2303.09540 2023.
[2] Josh Achiam, Steven Adler. Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.00000 08774, 2023.
[3] Deepak Alapatt, Aditya Murali, Vinkle Srivastav, Al4SafeChole Consortium, Pietro Mascagni, and Nicolas Padoy. Jumpstarting surgical computer vision. In International Conference Medical Image Computing and Computer-Assisted Intervention, pages 328-338. Springer, 2024.
[4] Kumar Ashutosh, Rohit Girdhar, Lorenzo Torresani, and Kristen Grauman. Hiervl: Learning hierarchical video-language embeddings. Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 23066-23078, 2023.
[5] AWS. Amazon transcribe medical, 2023.
[6] Nicolás Ayobi, Santiago Rodríguez, Alejandra Pérez, Isabela Hernández, Nicolás Aparicio, Eugénic Dessevres, Sebastián Peña, Jessica Santander, Juan Ignacio Caicedo, Nicolás Feránndzz, et al. Pixel-wise recognition for holistic surgical scene understanding. arXiv preprint arXiv:2401.11174. 2024.
[7] Max Bain, Arsha Nagrani, Gül Varol. and Andrew Zisserman. Frozen in time: A joint video and image encoder for end-to-eud retrieval. In Proceedings the IEEE/CVF International Conference Computer Vision, pages 1728-1738, 2021.
[8] Vivek Singh Bawa, Gurkirt Singh, Francis KapingA, Inna "kkaga-Banndrre Elettra Oleari, Alice Leporini, Carmela Landolfo, Pengfei Zhao, Xi Xiang, Gongning Luo, et al. The saras endoscopic surgeon action detection (esad) dataset: Challenges and methods. arXiv preprint arXiv:2104.03// 2021.
[9] Gedas Bertasius, Heng Wang, and Lorenzo Torresani. Is space-time attention all you need for video understanding? In JCML, volume 2, page 4, 2021.
[10] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9650 9660, 2021.
[11] Xinlei Chen and Kaiming He. Exploring simple siamese representation lcarning. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 15750-15758, 2021.
[12] Zhihong Chen, Maya Varma, Ben-te Delbrouck, Magdalini Paschali, Louis Blankemeier, Dave Van Veen, Jeya Maria Jose Valanarasu, Alaa Youssef, Joseph Paul Cohen, Eduardo Pontes Reis, et al. Chexagent: Towards a foundation model for chest x-ray interpretation. arXiv preprint arXiv:2401.12008, 2024.
11

---

[13] Alexey Dosovitskiy. Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai. Thomas Unterthiner. Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth Transformers for image recognition scale. arXiv preprint arXiv:2010.1192 2020.
16x16 words:
[14] Lijie Fan, Dilip Krishnan, Phillip Isola, Dina Katabi, and Yonglong Tian. Improving clip training with language rewrites. Advances in Neural Information Processing Systems, 36, 2024.
[15] Christiane Fellbaum. WordNet: An electronic lexical database. MIT press. 1998.
[16] Isabel Funke, Dominik Rivoir, Stefanie Krell, and Stefanie Speidel. Tunes: A temporal u-net with self-attention for vidco-based surgical phase recognition. arXiv preprint arXiv:2307.0990 2023.
[17] Xiaojic Gao, Yueming Jin, Yonghao Long, Qi Dou, and Pheng- Ann Heng. Trans-svnet: Accurate phase recognition from surgical videos via hybrid embedding aggregation transformer. In Medical Image Computing and Computer Assisted Interventon-MMICAA 202 24th International Conference, Strashourg. France, September 27-Octoher 1, 2021, Proceedings, Part IV 24, pages 593-603. Springer, 2021.
[18] Robert Geirhos, Patricia Rubisch, Claudio Michaelis, Matthias Bethge, Felix A Wichmann, and Wieland Brendel. Imagenet-trained cnns are biased towards texture; increasing shape bias improves accuracy and robustness. arXiv preprint arXiv:181.12231/ 2018
[19] Kristen Grauman, Andrew Westbury, Eugene Byme, Zachary Chavis, Antonino Furnari, Rohit Girdhar, Jackson Hamburger, Hao Jiang, Miao Liu, Xingyu Liu, et al. Ego4d: Around the world in 3,000 hours of egocentric video. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18995-19012, 2022.
[20] Sophia Gu, Christopher Clark, and Aniruddha Kembhavi. can't believe there's s no images! learning visual tasks using only language supervision. Proceedings of the IEFF/CVF International Conference on Computer Vision, pages 2672-2683, 2023.
[21] Isma Hadji, Konstantinos G Derpanis, and Allan D Jepson. Representation learning via global temporal aligmment and cycle-consistency. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11068-11077 1077, 2021.
[22] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 9729-97338 9738, 2020.
[23] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and puttern recognition, pages 770-778, 2016.
[24] Kexin Huang, Jaan Altosaar, and Rajesh Ranganath. Clinicalbert: Modeling clinical notes and predicting hospital readmission. arXiv preprint arXiv: arXiv:1904.08 1904.0532, 2019.
[25] Shih Cheng Huang. Liyue Shen, Matthew P Lungren, and Serena Yeung. Gloria: multimodal global- local representation learning framework for label-efficient medical image recognition. In Proceedings of the JEEF/CVF International Conference on Computer Vision, pages 3942-3951, 2021.
[26] Zhi Huang, Federico Bianchi, Mert Yuksekgonul, Thomas J Montine, and James Zou. A vanual-language foundation model for pathology image analysis using medical twitter. Nature medicine, 29(9):230--2316 2023.
[27] Wisdom Ikezogwo, Saygin Seyfioglu, Fatemeh Ghezloo, Dylan Geva, Fatwir Sheikh Mohammed, Pa- van Kumar Anand, Ranjay Krishna, and Linda Shapiro. Quilt-Im: One million image-text pairs for histopathology. Advances in Neural Information Processing Systems, 36, 2024.
[28] Alistair FW Johnson, Tom Pollard, Nathaniel R Greenhaum, Matthew P Lungren, Chih-ying Deng, Yifan Peng, Zhiyong Lu, Roger G Mark, Seth J Berkowitz, and Steven Horn.. Minic-cxr-jpe. a large publicly available database of labeled chest radiographs. arXiv preprini 1901:19901 2019.
[29] Siddharth Kannan, Gaurav Yengera, Didier Mutter, Jacques Marescaux, and Nicolas Padoy. Future-state predicting Istm for carly surgery type recognition. IEEE Transactions on Medical Imaging, 39(3):556-56) 566. 2019.
[30] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint mmXiv:1412.00 6980, 2014.
12

---

[31] Joël L LLvannhy, Sanat Ramesh. Diego Dall' Alba. Cristians Gonzalez, Paolo Fiorini, Beat P Müller- Stich, Philipp C Nett, Jacques Marescaux, Didier Mutter, and Nicolas Padoy. Challenges in multi-enntric generalization: phase and step recognition roux-en-y gastric bypass surgery. International journal of computer assisted radiology and surgery, pages 1-9, 2024.
[32] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-imaan pre-training with frozen image encoders and large language models. In International conference on machine learning. pages 19730-19745 PMLR, 2023.
[33] Junnan Li. Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-inan -image pre- pre-training unified vision- languae understanding and generation. In International conference on machine learning. pages 12888-12000 12900. PMLR, 2022.
[34] Kunchang Li. Yali Wang. Yizhuo Li, Yi Wang. Yinan He, Limin Wang, and Yu Qiao. Unmasked teacher: Towards -effcient video foundation models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 19948-19960, 2023.
[35] Wei Li, Linchao Zhu, Longyin Wen, and Yi Yang. Decap: Decoding clip latents for zero-shot captioning via text-only training. arXiv preprint arXiv:2303. 03032, 2023.
[36] Xianhang Li. Zeyu Wang. and Cihang Xie. An inverse scaling law for clip training. Advances in Neurul Information Processing Systems, 36, 2024.
[37] Yangguang Li. Feng Liang, Lichen Zhao, Yufeng Cui, Wanli Ouyang, Jing Shao, Fengwei Yu, and Junjie Yan. Supervision exists everywhere: A data efficient contrastive language-image pre-training paradigm. arXiv preprint arXiv:21 2021.
[38] Zichao Li. Cihang Xie, and Ekin Dogus Cubuk. Scaling (down) clip: A comprehensive analysis of data, architecture, and training strategies. arXiv preprint arXiv:2404.08197 2024.
[39] Victor Weixin Liang. Yuhui Zhang, Yongchan Kwon, Serena Yeung, and James Zou. Mind the gap: Understanding the modality gap in multi-modal contrastive representation leamning. Advances in Neural Information Processing Systems, 35:17612-1708 2022.
[40] Kevin Qinghong Lin, Jinpeng Wang, Mattia Soldan, Michael Wray, Rui Yan, Eric Z Xu, Difei Gao, Rong-Cheng Tu, Wenzhe Zhao, Weijie Kong, et al. Egocentric lidco-laggaaee pretraining. Advances in Newral Information Processing Systems, 35:7575-7555 2022.
[41] Timo Lüddecke and Alexander Ecker. Image segmentation using text and image prompts. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 7086-7096, 2022.
[42] Huaishao Luo. Lei Ji, Botian Shi, Haoyang Huang. Nan Duan, Tianrui Li, Jason Li, Taroon Bharti, and Ming Zhou. Univl: A unified video and language pre-training model for multimodal understanding and generation. arXiv preprint ar arxiv:2002.0635.. 2020.
[43] Lena Maier-Hein. Matthias Eisenmann, Duygu Sarikaya, Keno März, Toby Collins, Anand Malpani, Johannes Fallert, Hubertus Feussner, Stamatia Giannarou, Pietro Mascagni. et al. Surgical data science- from concepts toward clinical translation. Medical image analysis, 76: 10:300 2022.
[44] Lena Maier- Swaroop S Vedula, Stefanie Speidel, Nassir Navab, Ron Kikinis, Adrian Park, Matthias Eisenmann, Hubertus Feussner, Germain Forestier, Stamatia Giannarou, et al. Surgical data science for next-generation interventions. Nature Biomedical Engineering, 1(9):69-669) 2017.
[45] Christian M Meyer and Iryna Gurevych. Wiktionary: A new rival for expert-built lexicons? Exploring the possibilities of collaborative lexicography. na, 2012.
[46] Antoine Miech, Jean-Baptiste Alayrac, Lucas Smaira, Ivan Laptev, Josef Sivic, and Andrew Zisserman. End-to-end learning of visual representations from uncurated instructional vidcos. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1979-9889, 2020.
[47] Antoine Miech, Dimitri Zhukov, Jean- -aattstpt Alayrac. Makarand Tapaswi, Ivan Laptev, and Josef Sivic. Howto100m: Learning a text-vidco embedding by watching hundred million narrated video clips. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2630-2640, 2019.
[48] Norman Mu, Alexander Kirillov, David Wagner, and Saining Xie. Slip: Self-supervision meets language- image pre-training. In Eumpean conference on computer vision, pages 529-544. Springer, 2022.
[49] Bolin Ni, Houwen Peng, Minghao Chen, Songyang Zhang, Gaofeng Meng. Jianlong Fu, Shiming Xiang. and Haibin Ling. Expanding language- image pretrained models for general video recognition. In European Conference on Computer Vision, pages 1-18. Springer, 2022.
13

---

[50] Chinedu Innocent Nwoye, Tong Yu, Cristians Gonzalez, Barbara Seeliger, Pietro Mascagni, Didier Mutter, Jacques Marescaux, and Nicolas Padoy. Rendezvous: Attention mechanisms for the recognition of surgical action triplets endoscopic videos, Medical Image Analysis, 78: 102433, 2022.
[51] Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.000 037480 2018
[52] Alec Radford, Jong Wook Kim, Chris Hallacy. Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748-8788 8763, PMLR, 2021.
[53] Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. Robust speech recognition via large-scale weak supervision. In International Conference on Machine Learning, pages 28492-28518. PMLR, 2023.
[54] Sanat Ramesh, Diego Dall' Alba, Cristians Gonzalez, Tong Yu, Pietro Mascagni, Didier Mutter, Jacques Marescaux, Paolo Fiorini, and Nicolas Padoy. Multi-task temporal convolutional networks for joint recognition of surgical phases and steps in gastric bypass procedures. International journal of computer assisted radiology and surgery, 111-1119,, 2021.
[55] Sanat Ramesh, Vinkle Srivastav, Deepak Alapatt, Tong Yu, Aditya Murali, Luca Sestini, Chinedu Innocent Nwoye, Idris Hamoud, Saurav Sharma, Antoine Fleurentin, et al. Dissecting self-supervised learning methods for surgical computer vision. Medical Image Analysis, 88:102844, 2023.
[56] Tal Ridnik, Emanuel Ben-Baruch, Asaf Noy, and Lihi Zelnik-Manor Imagenet-21K pretraining for the masses. arXiv preprint arXiv:2 109.7 2021.
[57] Dominik Rivoir, Sehastian Bodenstedt, Isabel Funke, Felix von Rechtolsheim, Marius Distler, Jürgen Weitz, and Stefanie Speidel. Rethinking anticipation tasks: Uncertainty-awan anticipation of sparse surgical instrument usage for context-aware assistance. In International Conference on Medical Image Computing and Cosputer-Assisted Intervention, pages 752-762. Springer. 2020.
[58] Noam Rotstein, David Bensaïd, Shaked Brody, Roy Ganz, and Ron Kimmel. Fusecap: Leveraging large language models for enriched fused image captions. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 5689-5700, 2024.
[59] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International journal of computer vision, 115:21 1-252, 2015.
[60] Hiroaki Sakoe and Seibi Chiba. Dynamic programming algorithm optimization for spoken word recognition. IEEE transactions on acoustics, speech, and signal processing, 26(1):43-49, 1978.
[61] Pierre Sermanet, Corey Lynch, Yevgen Chebotar, Jasmine Hsu, Eric Jang, Stefan Schaal, Sergey Levine, and Google Brain. Time-contrastive networks: Self-supervised learning from video. In 2018 IEEE international conference on robotics and automation (ICRA), pages 114- IEEE, 2018
[62] Saurav Sharma, Chinedu Innocent Nwoye, Didier Mutter, and Nicolas Padoy. Surgical action triplet detection by mixed supervised learning of instrument-tissue interactions. In International Conference on Medical Image Computing and Computer-Assiste Intervention, pages 505 514. Springer, 2023.
[63] Sheng Shen, Chunyuan Li, Xiaowei Hu, Yujia Xie, Jianwei Yang. Pengchuan Zhang, Zhe Can, Lijuan Wang. Lu Yuan, Ce Liu, et al. K-lite: Learning transferable visual models with external knowledge. Advances in Neural Information Processing Systems, 35:15558-1555 15573, 2022.
[64] Karan Singhal, Shekoofch Azizi, Tao Tu, S Sara Mahdavi, Jason Wei, Hyung Won Chung, Nathan Scales, Ajay Tanwani, Heather Cole- Cole-Lewis, Stephen Pfohl, et al. Large language models encode clinical knowledge. Nature, 620(7972):172- 180, 2023.
[65] Chen Sun, Austin Myers. Carl Vondrick, Kevin Murphy, and Cordelia Schmid. Videobert: A joint model for video and language representation learning. In Proceedings of the IEEE/CVF international conference on computer vision, pages 7464-7473, 2019.
learning
[66] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anee Lachaux. Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint 1377 2023.
14

---

[67] Andru P Twinanda, Sherif Shehata. Didier Mutter, Jacques Marescaux, Michel De Mathelin, and Nicolas Padoy. Endonet: deep architecture for recognition tasks on laparoscopic videos. IEEE transactions on medical imaging, 36(1):86-97. 2016.
[68] Bairui Wang, Lin Ma. Wei Zhang, and Wei Liu. Reconstruction network for video captioning. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 7622-7631, 2018.
[69] Ziyi Wang. Bo Lu, Yonghao Long, Fangxun Zhong, Tak-Hong Cheung, Qi Dou, and Yunhui Liu. Au- tolaparo: A new dataset of integrated multi-tasks for image-guiden guided surgical automation laparoscopic hysterectomy, In International Conference on Medical Image Computing and Computer- Interven- tion, pages 486-496. Springer, 2022.
[70] Hu Xu, Gargi Ghosh, Po- Po-Yao Huang, Dmytro Okhonko, Armen Aghajanyan, Florian Metze, Luke Zettlemoyer, and Christoph Feichtenhofer. Vidcoclip: Contrastive pre-training for zero-shot video-text understanding. In Proceedings of the 2021 Conference Empirical Methods in Natural Language Processing, pages
2021.
[71] Zihui Sherry Xue and Kristen Grauman. Learning fine-grained view-invariant representations from unpaired ego-exo videos via temporal alignment. Advances in Neural Information Processing Systems, 36, 2024.
[72] Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mojtaba Seyedhosseini, and Yonghui Wu. Coca: Contrastive captioners are image-text foundation models. arXiv preprint arXiv:2205.019 2022.
[73] Qiying Yu, Quan Sun, Xiaosong Zhang, Yufeng Cui, Fan Zhang, Xinlong Wang, and Jingjing Liu. Capsfusion: Rethinking image-text data at scale, arXiv preprint 10..::2310220550 2023.
[74] Kun Yuan, Matthew Holden, Shijian Gao, and Won-Sook Lee. Surgical workflow anticipation using instrument interaction. In Medical Image Computing and Computer Assisted
MICCII 202 24th International Conference, Strasbourg, France, September 27-October 1, 2021, Proceedings, Part IV 24, pages 615-625. Springer, 2021.
[75] Kun Yuan, Vinkle Srivastav, Nassir Navab, and Nicolas Padoy. Heevl: Hierarchical video-language pretraining for zero-shot surgical phase recognition. arXiv preprint arXiv:2405. 10075 2024.
(76] Kun Yuan, Vinkle Srivastav, Tong Yu, Joel Lavanchy, Pictro Mascagni, Nassir Navab, and Nicolas Padoy. Learning moult--moddl representations by watching hundreds of surgical vidco lectures. arXiv preprint 152200 2023.
[77] Xin Yuan, Zhe Lin, Jason Kuen, Jianming Zhang, Yilin Wang, Michael Maire, Ajinkya Kale, and Baldo Faicta, Multimodal contrastive training for visual representation learning. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pages 6995-7004, 2021.
[78] Rowan Zellers, Ximing La, Jack Hessel, Youngjae Yu, Jae Sung Park, Jize Cao, Ali Farhadi, and Yejin Choi. Merlot: Multimodal neural script knowledge models. Advances in Neural Information Processing Systems, 34:23634-23611, 2021.
[79] Bowen Zhang. Hexiang Hu, and Fei Sha. Cross-modal and hierarchical modeling of video and text. In Proceedings of the european conference on computer vision (ECCV), pages 374-390, 2018.
[80] Honglu Zhou, Roberto Martín-Mart Mubbasir Kapadia, Silvio Savarese, and Juan Carlos Niebles. Procedure-aware pretraining for instructional video understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10727-10738. 2023.
[81] Luowei Zhou, Yingbo Zhou, Jason J Corso, Richard Socher, and Caiming Xiong. End-to-end dense video captioning with masked transformer. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 17399-8488 2018
[82] Xueyan Zou, Zi-Yi Dou, Jianwei Yang, Zhe Gan, Linjie Li, Chunyuan Li, Xiyang Dai, Harkirat Behl, Jianfeng Wang, Lu Yuan, et al. Generalized decoding for pixel, image, and language. In Proceedings of the IEEE/CVP Conference on Computer Vision and Pattern Recognition, pages 151 15116-15127, 2023.
[83] Xueyan Zou, Jianwei Yang. Hao Zhang, Feng Li, Linjie Li. Jianfeng Wang. Lijuan Wang, Jianfeng Gao. and Yong Jae Lee. Segment everything everywhere all at once, Advances Neural Information Processing Systems, 36, 2024.
15

---

A Pretraining Dataset
A.1 Videos
We start with the videos that used for surgical vision-language pretraining in [76]. In total, there are 1, 1,326 surgical lecture videos. These videos are transcribed by AWS [5] and Whisper [53] audio speech recognition (ASR) to obtain the corresponding narration texts. Furthermore, we curate the videos' metadata from the online platforms to obtain the extra keystep and abstract texts. In the phase- and video-level pretraining, we need parent- and child-level text correspondences, e.g., keystep and its corresponding narration texts, to perform procedure understanding. Therefore, we filter out the videos that do not have parent-child correspondences. In total, have 1, 007 and 920 videos for phase- and video-level pretraining, respectively.
A.2 Misspelling Error
As the narration texts are generated from the audio using the ASR system, they usually contain many misspelling errors and fragment sentences. Therefore, we apply multiple preprocessing steps to clean the narration texts.
We first built the vocabulary based on the textbook, surgical category labels, and definition words. Specifically, we refer to the academic papers, which define the surgical phases, to curate a list of definition words and build vocabulary that contains the words of interest. We also parse and merge the words from the textbook. total, we obtain a vocabulary of the size of 51, 640 words. Then, we use the built vocabulary along with the spell-checking algorithm to correct the misspelling errors in narration texts. The algorithm utilizes Levenshtein Distance to identify words within edit distances from the original. It then cross-references these permutations (insertions, deletions, replacements, and transpositions) with a word frequency list, prioritizing words with higher occurrence frequencies as potential correct results.
B Evaluation Setup
We provide a detailed description of the downstream tasks and their settings that we apply in the experiment.
Surgical Phase Recognition. Surgical phase recognition is a proxy task test the model's s surgical scene understanding ability. It aims to classify the frame of surgical video into predefined classes (phases), requiring the model to understand the instrument and anatomy's presence and their interactions by extracting visual patterns from the surgical scene image. In this work, we ignore temporal modeling in surgical phase recognition as we focus on multi-modal representation learning. We consider phase recognition as a frame-wise image classification problem. In the surgical phase recognition task, we evaluate the model's performance based on the publicly available datasets, including Cholec80 [67], AutoLaparo 669 and MultiBypass [? J.
Zero-shot Evaluation. As the surgical phase labels are high-level definitions that can be decomposed into a few basic concepts, we manually construct the contextual prompts for phase labels, as shown in Tab. Tab. Gland Tab. 7 Our constructed prompts for the class names are built with the help of clinician's comments, considering the involved surgical instruments, anatomies, and events involved in given surgical phase.
Linear-probing Evaluation. For finear-probing evaluation on the surgical phase recogni- tion downstream datasets, we keep the visual encoder frozen and train a linear classifier on the extracted features. We do not apply any image augmentation during the training. The learning rate is scaled linearly based on the actual batch size. The model is optimized using SGD optimizer with the learning rate 0.001 and weight decay parameter as 0.0005. We train the model for 40 epochs. We fit the model on the training and validation sets and report the performance on the separate test set. For the few-shot linear-probing evaluation, adopt k --percentage shot approach with a slight modification to accommodate the nature of surgical videos, which contain frames from different classes. Specifically, we select 10%
16

---

Table 5: Manually designed prompts for the class names recognize the surgical phase in Cholec80 dataset. We decompose high-level phase definitions into a few basic concepts to form the text prompts.
Phase Labels
Prompts
Preparation
In preparation phase insert trocars to patient abdomen cavity In calot triangle dissection phase use grasper to hold gallbladder and use hook to expose the hepatic triangle area and cystic duct and cystic artery In clip and cut phase use clipper to clip the eystic duct and artery then use scissor to cut them
CalofTrianteeisseccion
ClippingCutiig
In dissection phase use the hook to dissect the connective tissue between gallbladder and liver
Gallblalderrissetttion
Palkkladderuuckngg
In packaging phase I put the gallbladder into the specimen bag In clean and coagulation phase use suction and imiigation to clear the surgical field and coagulate bleeding vessels In retraction phase I grasp the specimen bag and remove from trocar
CleaningCoagulation
GallblaaderRetractionn
Table Manually designed prompts for the class names to recognize the surgical phase in AutoLaparo dataset.
Phase Labels
Prompts
Preparation
I use grasper to grasp and explore the field
Dividing Ligament and Peritoneum
divide ligament and peritoneum
Dividing Uterine Vessels and Ligament
I divide uterine vessels and ligament
Transecting the Vagina
use the dissecting hook to transect the vagina
Specimen Removal
remove the specimen bag and uterus
Suturing
suture the tissue
Washing
Washing
of the video from the training set. This ensures that data leakage is prevented and that the number of samples per class remains similar.
Cross-modal Retrieval. Cross-modal retrieval includes text-based video retrieval and sideo-based
text retrieval. Here, we conduct the cross-modal retrieval three hierarchical levels. We collect 537 clip-narration (clip-level) video-text pairs, 746 phase-keystep (phase-level) video-text pairs, and 86 video-abstract (video-level) video-text pairs from hold-out testing videos of SVI. (76). There more phase-keystep than clip-naraation video-text pairs because some testing videos do not have cleaned narrations and we filter them out. For video embedding generation, we sample multiple frames fro m the video and average pool their image embeddings. We temporally sample 10 frames for dlip-/phase-/video. -video--video-levell videos. We conduct the zero-shot evaluation for the cross-mode moosl retrieval task.
C Architecture & Initialization
As mentioned before, the current surgical vision-language pretrainin dataset lacks the scale necessary to pretrain robust lasion-language model from scratch, therefore good choice of architecture and initialization is important. this section, we conduct the experiment and study the effect different model architectures and initializations, justifying our choice of using ResNet50 architecture with ImageNet initialization as our starting point before the video-language pretraining.
ResNet50. For ImageNet initialization, we use public IMAGENETIK IK_ V1 weights from torchvision. Random initialization means that we random initialize the visual encoder before the hierarchical vasion-language pretraining. These models' textual encoders are initialized from BioClinicalBert [24]. For CLIP initialization, we initialize the visual and textual encoder from OpenAl's weight [52].
17

---

Table 7: Manually designed prompts for the class names recognize the surgical phase in gastric bypass dataset. We use the same prompts for both StrasBypass70 and BernBypas70.. We exclude the "other" class as its definition is ambiguous.
Phase Labels
Prompts
In preparation phase insert trocars to the abdominal cavity and expose of the operating field
Preparation
Gastric pouch creation
I cut the fat tissue and open retrogastric window at stomach grasp and lift the omentum and divide it
Omentum division
see the proximal jejunum and determine the length of the biliary limb. I open the distal jejunum and create the gastrojejunostomy using a stapler. reinforcement of the gastrojejunostomy with an additional suture.
Gastrojejunal anastomosis
I place the retractor and move the gastric tube and detect any leakage of the gastrojejunostony
Anastomosis test
open the mesentery to facilitate the introduction of the stapler and transect the jejunum proximal
Jejunal separation
expose between the alimentary limb and the transverse colon and close it with sutures
Petersen space closure
I expose between the alimentary limb and the transverse colon and close it with sutures
Jejunojejunal anastomosis
Mesenteric defect closure
I expose the mesenteric defect and then close it by stitches
In clean and coagulation phase use suction and irrigation to clear the surgical field and coagulate bleeding vessels remove the instruments, retractor, ports, and camera
Cleaning and coagulation
Disassembling
7ero-shot
Linear-probing (10-shot)
Linear-prohing (full-shot)
Backbone
Tnit.
Cholec80
Autolaparo
Cholec80
Autolaparo
Cholec80
Autolaparo
Random
29.4/ 10.4
15.3/ 10.9
3 /
44.6 25.3
30.7 19.3
ImageNet
34.7 24.4
21.3 / 16.6
55.0/39.9
/
63.5/ 50.3
54.3/ 41.8
ResNet50
CLIP
33.8 / 19.6
18.9/16.2
58.9/ 42.3
/5.3/35.3
64.9/ 55.0
53.1 / 42.1
Random
20.2/
9.1
38.4/
32.1 19.7
4
38.4/ 25.5
ImageNet
42.8/ 25.11
20.5 15.5
57.4 40.5
47.8 /319
60.6/ 48.9
56.3 44.5
VIT-B/16
Dino
35.1
13.9/92
54.7 /39.2
33.1
64.9 51.2
54.0 / 42.4
Table 8: The experiments show that the initialization largely influences the performance of surgical vadeo-language pretraining.
ViT-B/16. For ImageNet initialization, we use weights from the official Google JAX imple- mentation, which is pretrained on ImageNet21k (56) and then finetune on ImageNet k (59] We use the public pretrained weights from [T0) for Dino initialization.
In our work, we choose ResNet50 over Vision Transformer (ViT-B/16) due to its superior perfor- mance and lower parameter amounts in the context of video-language pretraining for surgical data. Our experiments demonstrated that ResNet50, particularly when initialized with CLIP weights, outperformed V across various tasks, including zero-shot and linear-probing evaluations on Cholec80 and Autolaparo datasets. Despite the advanced capabilities of vision transformers, their performance heavily depends on large-scale pretraining datasets, which might not always be available or optimal for specialized domains like surgical scenes. Conversely, convolutional neural networks like ResNet50 have shown robust generalization abilities, even when pretrained on natural images, making them more suitable for our specific application. Additionally, the initialization sensitivity observed ViT-B/16 further justified our preference for ResNet50, ensuring a more reliable and effective starting point for our hierarchical dision-language pretraining.
18

---

D Dynamic Time Warping
After achieving the cost matrix C' and Ć, we perform dynamic time warping (DTW) 660 to find the minimum path to align the frames of video segment V ( the text sequence B = {b.....bx' } and reversed text sequence (b.i) }. respectively, as shown in Algorithm, | We follow 77] to process the DTW function into differentiable, enabling the gradient back-propagation. The differentiable loss function is the same as 21]
A significant advantage of using DTW is that it does not require additional temporal modules, such as recurrent neural networks or attention mechanisms, to model temporal relationships. This simplification allows to focus on learning better representations by directly aligning video frames and text sequences based on their semantics.
Algorithm 1 DTW to align sequences using cost matrix
1: procedure ALIGNSEQUENCESSC, V, B)
2: Let T be the length of sequence V and N be the length of sequence B.
3: Set to 7 and j to N.
4: Initialize distance to 0.
5: while > 0 and j > 0 do
6: distance distance + C[i
7: ir and j > 1 and Cii 1 1] s Cji - []]] and Cli 1]D 1] s cDil -
then
8:
i it
9:
} else if i > and Cii - 1][1 s C[i]l 1] then
10:
6- i
12:
else
j+
14:
end if end while
15:
16:
return distance.
17:
end procedure
E
Modality Gap
Modality gap is a geometric phenomenon observed in the embedding space of multi-modal mod- els 39
This gap illustrates that pretrained multi-modal (vision-language models create joint embedding space where different modalities, such as images and text, are kept at significant distance from each other. During contrastive optimization, this separation created at initialization maintained to the extent that irrelevant image embeddings can be closer to each other than to their corresponding relevant text embeddings. This spatial disparity in the embedding space hinders the model's ability to effectively align and understand the relationships between visual and textual data, leading to suboptimal performance in tasks requiring integrated multi- comprehension. The existence of the modality gap is particularly detrimental when adapting pretrained vision- language models to cross-modal generation tasks, such as image captioning. As highlighted by several studies [35] 20) narrowing modality gap correlates with improved performance in mooss tasks.
As shown in Fig. we visualize the embeddings of videos and their corresponding text descriptions at three hierarchical levels: clip-narration, phase-keystep. and video-abstract. Our proposed model demonstrates significant reduction in the modality gap compared to the Surg model. This align- ment across different hierarchical levels ensures more comprehensive and cohesive understanding of the multi-modk data, leading to superior performance in tasks like image captioning and other vision- language applications.
19

---

Clip-Narration
Phase-Keyster
Video-Abstract
Initialization
SurgVLP
HecVL
PeskaVLP
Image Embedding
Text Embedding
Figure 4: Modality gap visualization in different hierarchical levels. It shows that our model closes the modality gap incurred from the initialization after the hierarchical pretraining.
F Surgical Phase Recognition Results
We demonstrate the zero-shot surgical phase recognition reflect the surgical scene understanding ability of our pretrained model. Our model can identify surgical phases different types of surgical procedures without any finetuning. Both success and failure examples shown.
Surgical Term Understanding, In Fig. 5 we show that the pretrained model excels at identifying the "washing" phase in surgical procedures, demonstrating its capability to accurately recognize high-level surgical activities. This proficiency enhances surgical assistance systems, improving real-time analysis and decision-making in operating rooms.
Instrument Identification. Fig. 6 we demonstrate how the visual embedding is significantly influenced by the presence of surgical instruments. Specifically, in the first row, the semantic meaning of the image changes from "calot triangle dissection" "clip and cut" due to the appearance of hook, even though the other anatomical features remain similar.
G Limitations
As the pretraining process at clip-level requires additional supervision signals, i.e., visual self- supervision, memory and computation overhead increase compared to the vanilla HecVL pretrain- ing. Also, during the phase- and video-level pretraining, the process of dynamic time warping can be time-consuming because is based on dynamic programming, slowing down the pretraining iteration when handling longer-term surgical videos. Additionally, the knowledge augmentation on keystep
20

---

Dividing figmmnnt
Diiidig
Washing
Saturing
Specimen removal
Transecting vagin
Diviiigg utcrine vesse
Dinding leament and
Preparasion
GT. Suturing
GT: Wenking
Sucuring
Spocimion romovol
Dvviing ubertne Ipament
Dividng huument and
Preparacion
Figure 5: Qualitative surgical phase recognition results on hysterectomy. The y-axis is the class names. The x-axis is the probability of each class. The bottom right image shows that the pretrained model understands the blood fluid.
GT: Calot triangle dissection
Calot triangle dissection
Galliladder retraciinn
and cogulation
Gallbleddarr packaging
Galibböder dissection
Calot triangle dissection
Preparacion
GT: Gallbladdrr
GT: coqulation
Gallbledder
Clean and sogulanten
Callbladder paclaging
Galbladder dissection
Calot trlongl dissection
Preparasion
Figure 6: Qualitative surgical phase recognition results on cholecystectomy. The y-axis is the class names. The x-axis is the probability of each class. We find that the pretrained model is triggered by the instrument occurrence, such as hook in the second row.
21

---

and abstract texts need to be modified to fit the other video-language pretraining datasets 4 79 as their hierarchical paired texts are annotated manually. Instead, our knowledge augmentation is more suitable for videos in the wild from online platforms. To address these limitations, future work could focus on developing general textual augmentation strategy using the L.LM's internal knowledge, adapting to the instructional videos that miss keystep and abstract text descriptions. Furthermore, techniques for decentralizing the ladeo-language pretraining could explored, aiming to pretrain with multi-centric rision-language samples while preserving privacy using the federated learning strategy. This could address the scaling problem in surgical vision-language pretraining and improve the generalizationabiiit across the centers.
H
Knowledge Augmentation
Build Surgical Knowledge Base. In Fig. 7 we show that the internal surgical knowledge of large language models can be elicited build the external knowledge base.
Build Surgical Knowledge Base. In Fig. Fig. and Fig. I0 we show that the knowledge of large language model can be used to enrich the semantics of the hierarchical texts, i... narrations, keysteps, and abstracts. Notably, it can explain high-level keystep words into descriptive sentences, enhancing textual diversity and preventing overfitting.
22

---

Laparoscopic right colectomy for cecal cancer 1. Position the patient appropriately laparoscopic right colectomy. This usually involves placing the patient the supine
position.
2. Using trocar, create an access point to the abdomen to allow for the insertion of the laparoscope.
3. Through the laparoscope, verify the presence of the cecal cancer and its location on the right colon.
4. Identify the medial approach and begin the mobilization of the mesentery. Carefully navigate the laparescopic tool to disconnect the right side colon from the rest of the organ.
5. Roam around delicately identify the key anatomic landmarks. This could include the superior mesenteric vein, lleocolic vessels, the duadenum.
6. Complete the divisian of the mesentery intracorporeally. Separate the right colon from the rest of the bowel and carefully
preserve the oncalogic clearance.
7. anastomosis stapler release healthy section of the right colon.
8. Extract the resected right colon extracorporeally through small suprapubic incision. Take caution make small incision possible to ensure minimal harm to the patient.
9. Complete the stapled anastomosis extracorporeally. Connect the healthy section of the colon back to the rest of the organ.
10. After ensuring the anastomosis secure and not leaking, remove the laparoscope, Please note: This is broad outline of the steps undertaken during laparescopic right collctomyy cecal cancer. specific steps may vary based surgeon's expertise, patient's anatomy, and clinical situation.
Redo Nissen fundoplication with stapled-wedge Collis gastroplasty
1. Start the procedure by taking down the previous fundoplication.
2. Follow this by identifying the mechanism underlying the failure the initial repair. Perform an extensive mobilization of the esophagus through the hiatus to achieve adequate length of intra abdominal esophagus.
4. Despite the mobilization, If the esophagus remains too short, perform Collis gastroplasty using the wedge gastrectomy technique over 50 French bougle.
Following the gastroplasty, a 2.5cm of tension-free intre-abdominal esophagus should be achieved.
Repair hiatus with interrupted non absorbable sutures.
7. Finally, perform standard Nissen fundoplication.
Stepwise approach for laparoscopic reversal of Hartmann's 's procedure 1. Position the patient on the operating table after administering general anesthesia to ensure patient comfort and positioning.
Establish pneumoperitoneum
Veress needle inflate the abdomen, creating a space in which work.
3. Insert three trocars (ports) into the patient's abdomen allow for the passage laparoscopic instruments.
4. Inspect the abdomen with laparoscope locate the previous colonic stump and assess adhesions and general abdominal conditions.
5. Begin the process of adhesiolysis, involving the careful separation of adhesions between the abdominal wall and the colon. Proceed with the mobilization colon by carefully performing medial-to-lateral dissection.
7. Divide the colon intra aaraaadddminlllyy using laparoscopic stapler, which seals off the colon and prevents leakage of bowel contents.
8. Identify the rectal stump and mobilize within the pelvis in readiness for the reconnection of the bowel.
9. An anastomosis (connection) is created between the divided colon and the rectal stump, restoring intestinal continuity.
Secure the anastomosis by placing sutures applying surgical staples ensure secure connection leakage.
11. Inspect the whole abdominal cavity visually with the laparoscope checking any signs bleeding, injury any overlooked
issue before ending the procedure.
12. The trocars are then removed, and the incisions sutured. The pneumoperitoneum is deflated.
13. Clean the surgical area thoroughly.
Dress the post-operative wounds correctly.
Laparoscopic extraction of CBD stone after fallure ERCP (duodenal perforation)
1. The surgical area is prepared and patient is positioned for laparoscopic common bile duct (CBD) exploration.
Trocars are inserted at suitable locations in the abdominal region to carry out the procedure.
The gall bladder reached and exposed utilizing laparoscopic
4. The cystic duct identified through careful maneuvering with laparoscopic instruments.
5. A trans-cystic systic approach is taken to explore the Common Bile Duct.
6. In case of large bile duct stones which cannot be extracted through the cystic duct, choledochotomy is performed.
7. The CBD stone visually located using the laparoscopic camera. used The stone is securely extracted from the body through the previously created trocar incisions.
Laparoscopic instruments
the from Common
10. Once the stone is completely removed, the common bile duct and cystic duct are checked for any potential remaining stones blockages.
11. Procedure concludes with the removal all laparoscopic tools and the closure of incisions.
Figure 7: Example of surgical step knowledge base based on the large language models.
23

---

1. Source: and this be for the so be cut the mesh just in the middle about seven centimeter link
Target: Select a mesh of appropriate dimensions that completely covers the hernia defect and extends least centimetres beyond the defect in directions
inferior epigastric
come from here
Target: Utilize dissection instruments make opening between the preperitoneal space the transversalis fascia for access the inguinal region
5. Source: the plain zero be often very thickened in this inflammatory condition and capsule dissection must be perform in
order to help we find the plain and continued dissection
6. Target: Utilize meticulous dissection techniques to divide the blood vessels close to the bowel, ensuring minimal damage to
the surrounding area
7. Source: the sigmoid colon now or most completely release from the lateral side wall
Target: Identify diagnose the patient with diverticulosis and chronic colo-vesical fistula
Source:
morgue correctly define way Target: Proceed with combination of lateral and medial approach for the mobilization of the mesocolan
with dissection
11. Source: percutaneous suture straight needle be insert in the epigastric region and pass towards the of the
right carotid
12. Target: Begin by positioning three ports (5mm, 12mm, 5mm) in the abdomen for laparoscopy
Source: middle colic vessel be clip and divide just above the body the pancreas
demarcation the resection, ensuring to capture all polyps and the other lesion sites observed during preoperative investigations
Target: Locate
Figure Knowledge augmentation on the narration texts.
Source: Opening lesser omentum
Target: The lesser omentum, fatty apron-like
stomach first part the duodenum, opened
allow access the stomach
3. Source: Start of gastric tubulization
4. Target: At this step, the surgeon begins creating tube-like like shape from the remaining portion of the stomach, also known
gastric tubulization
5. Source: End of tubulization
6. Target: This is when the surgeon completes the tubulization process, finalizing the smaller, sleeve-like shape of the stomach
7. Source: Division of greater omentum
Target: this step, surgeon divides the greater omentum, large apron-like fold peritoneum hangs
down from stomach
Source: Jejunojejunostomy
Target: The surgeon creates opening the two loops with cautery hook for passage of the linear stapler and closes
the opening using absorbable sutures
11. Source: Gastrojejunostomyy
12. Target: The surgeon executes the gastrojejunostomy using circular stapler, creating a connection between the stomach
and Jejunum
Source: Closure Petersen's defect
Target: Towards the
surgeon Petersen's space, potential space after Roux-en-Y gastric bypass, prevent
internal herniation
15. Source: Anvil placement
16. Target: The end of nasogastric tube, attached to the anvil, is passed down from the mouth into the stomach
17. Source: Divisian of the ileacolic vessels
18. Target: The surgeon separates the blood vessels connected to the ileum and colon to prevent bleeding during the
procedure
Source: Preparing the anastomosis
Target: surgeon prepares for anastomosis, or surgical connection two parts of
Figure Knowledge augmentation on the keystep texts.
24

---

1. Source:
This edit of live operation demonstrates the performance laparoscopic gastric bypass. demonstrates nicely manoeuvres such as retrocolic placement of the Roux limb and hand-sewn gastrojejunal anastomosis
2. Target:
This video shows laparoscopic gastric bypass surgery, focusing stomach and duodenum procedures and barlatric surgery techniques for marbid obesity treatment. Main activities involve the retrocolic placement of the Roux limb and hand-sewn gastrojejunal anastomosis. They demonstrate the techniques and maneuvers used during this surgery
3. Source: This video shows the case of a female patient presenting with low rectal cancer for which neoadjuvant therapy is used. The author performs totally laparoscopic TME using medial approach. colorectal anastomosis without bowel protection performed
4. Target: This surgical lecture video laparoscopic anterior resection with Total Mesorectal Excision (TME) and medial mobilization of the splenic flexure in a female patient. This procedure is utilized to treat low rectal cancer and involves the use of a medial approach. The video details how to perform colorectal anastomosis without bowel protection. The procedure Is entirely laparoscopic
5. Source: this live educational video, Professor Himpens presents the case of a 34-year-dd old female patient (BMI of 41) with a history morbid obesity since adolescence. She will undergo a laparoscopic sleeve gastrectomy (LSG). The preoperative work-up was normal. of the most commonly performed bariatric procedures. Surgical pitfalls emphasized during the video make sure that LSG achieved adequately and to prevent any potential complications. addition, trocars placement, location of the first firing of the linear stapler, the reasons why oversewing of the staple line is not performed, and thrombosis prophylaxis are also discussed during the procedure
lost 2Kg months before the procedure. Nowadays, laparoscopic sleeve gastrectomy (LSG)
6. Target: This educational video demonstrates laparoscopic sleeve gastrectomy for morbidly obese patient. The surgical procedure involves techniques such as the placement of trocars and the first firing of the linear stapler. also addresses potential surgical pitfalls to ensure the adequate execution of the procedure and prevention complications. The video highlights oversewing of the staple line isn't performed during the procedure and discusses the methods for thrombosis prophylaxis
7. Source: Intrathoracic migration of the fundoplication is one of the most common causes failure after antireflux surgery. When the patient develops symptoms related to the volume of Intramediastinal hemna, the only option is reoperate. Such redos are complex and necessitate thorough and painstaking approach to the potential underlying mechanisms causing intrathoracic migration, namely the length of the esophagus and cruroplasty
8. Target: This surgical video falls under the categories of stomach and duodenum, hiatal hernia, reflux, Nissen fundoplication, and hernia surgery. the video demonstrates reoperation for symptomatic intrathoracic migration of a fundoplication, involving valve repositioning
reinforced crural repair. The principal activities consist of examining underlying mechanisms causing intrathoracic migration such as the length of the esophagus and cruroplasty
9. Source: This video demonstrates transumbilical three-trocar technique for single incision total colectomy and partial proctectomy with intracorporeal side-to-end lleorectal anastomosis using standard laparoscopic instrumentation. The patient is thin 19-year-old boy with BMI of 19 presenting with familial adenomatous polyposis (FAP). The previous colonoscopy has shown 300 polyps in the colon and very few in the distal rectum. Conventional trocars (5mm, 10mm, and 12mm) are used through a 3.5cm transumbilical incision. The ligation of the vessels mostly carried out by the Ligasure-V vessel-sealing device using medial-to-lateral approach. The specimen extracted through umbilical incision after removal of the 10mm and stapling technique
cannulas. The lleorectal anastomosis is carried out intracorporeally using double
10. Target: The video shows transumbilical single incision laparoscopic total colectomy and partial proclectomy with ileorectal anastomosis performed on 19-year-old patient with familial adenomatous polyposis. The surgery primarily uses a three-trocar technique and standard laparoscopic instruments including Ligasure-V vessel-sealing device for ligating vessels. The surgery involves making a 3.5cm transumbilical incision using 5mm, 10mm, and 12mm trocars. The colectomy specimen is extracted through the same umbilical incision. The final ileorectal anastomosis is achieved intracorporeally employing a double stapling method
Figure 10: Knowledge augmentation on the abstract texts.
25

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Our experimental results on multiple datasets consistent with the claims in the abstract and introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitation in the Appendix G.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
26

---

Justification:
We do not include the theoretical assumption and experiments.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proof' can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
We provide the detailed experiment setup in the Experiments section and appendix. Our model evaluated on the public dataset. We will also provide the model weights and config file to reproduce the results.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
27

---

The authors should provide scripts to reproduce all experimental results for the new reproducible, they Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Justification: We are working on cleaning the code for now as a systematic codebase that is related to multiple research works. We will release the code after the acceptance.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.co cc/
for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ https/cccppbbcccguuudddeeoo Coaee//dddeSSbmmmcma ssi onPolicy) for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
proposed method and baselines. only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: See Experiments section.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: The lideu-language pretraining too computationally expensive to provide the error bar. We fix the random seed for the reproducibility.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
28

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (e.g. negative rates).
show in tables or
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: See Experiment section.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that httprr/p...cpublc/cchcssesseeeie
didn't I make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification: Our code follows the code ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- cration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: The social impact is minor in our work as we create the dataset from the open educational platforms, which are open to any learner.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
29

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied partieular applications, let alone deployments. However, if there is direct path any negative applications, the authors should point it out. For example, it is legitimate point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (c.g., gated release models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of MI).
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification: Unlike generative models, this work provides generalist scene understanding as the foundation module for surgical data science. The data are anonymized because the model encoding.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
We will cite the original owners' assets when we release the codebase.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
30

---

If assets are released, the license, copyright information, and terms of use in the thcode has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
package should be provided. For popular datasets, paperswi
latasets.paperssit
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
We will provide the documentation along with the code and dataset.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
We do not involve crowd-sourcing and human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification:
We do involve crowd-sourcing and human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
31

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
32


Reviews:
[{"review_id": "y1b7xOz8Eh", "reviewer": "Reviewer_HH5E", "summary": "The paper presents a new framework called PeskaVLP for surgical video-language pretraining. A hierarchical knowledge augmentation approach is used for enriching text information. The pretraining is implemented with the proposed language supervision and visual self-supervision. A new training objective is proposed for surgical procedural understanding. Extensive experiments are conducted to demonstrate the effectiveness on the surgical phase recognition task and cross-modal retrieval task on multiple downstream dataset.", "questions": "1. What types of surgeries are included in the SVL dataset used in the paper? Is it suitable for the pretraining task? Could it affect the results on the downstream dataset?\n\n2. In Section 3.2, where hierarchical knowledge is augmented by GPT, the authors need to discuss the ability of LLMs to generate accurate textual information to describe the surgical steps in the domain-specific surgical context, especially considering the fine-grained image-text alignment in the clip-level (only 4 frames).\n\n3. In Section 3.2, the authors calculate textual similarity between the pseudo step generated by the LLM and the narration. How is this similarity calculated? Is there an ablation study on the effectiveness of the three behavior in knowledge augmentation?\n\n4. In Section 3.3.1, the authors implement visual self-supervision based on augmentation. Which specific augmentations were used? Do the augmentations affect the corresponding text's semantic information? For example, using flipping could impact descriptions related to left/right information in surgical operation.\n\n5. In Section 3.3.2, procedural information based on surgical phases is used. However, in surgical datasets, such as the cholec80 and AutoLaparo mentioned in the paper, the surgical process does not always follow a linear order defined by Phase 1-N and may include repeated phases. The authors should discuss the applicability of the method design in such situations.\n\n6. In Table 3, for the experimental results on cholec80, Moco (third row) provides the best results, but this is not highlighted in bold in the table. This needs to be corrected and the corresponding discussion should be provided. The same issue appears with the results using Moco (second row) on the StrasBypass70 dataset.", "limitations": "Authors discussed it briefly in the appendix.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. This paper addresses the problem of VLP in the surgical scene. A hierarchical knowledge augmentation is proposed to tackle the problem of lack of textual information in the surgical field.\n2. The paper is generally well-written and easy to follow.", "weaknesses": "1. The explanation of method details is not clear enough, and there is a lack of discussion on some experimental results\n2. The proposed method is based on certain assumptions but lacks a comprehensive consideration of applicability.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**[Q1 SVL Dataset]**\n\n**[Q1.1. Types of surgeries in SVL dataset]** In the rebuttal letter PDF, we have added a table summarizing the top 42 types of surgical videos in the pretraining dataset. As shown in Table 1 in the rebuttal letter PDF, the SVL dataset predominantly contains laparoscopic surgeries, focusing on the stomach, duodenum, hernia, colon, gallbladder, and tumors. Also, the diverse content within each surgical type ensures that the SVL dataset provides generalizable language supervision for representation learning.\n\n**[Q1.2. Suitable for Pretraining]** This paper uses Cholec80, Autolaparo, and MultiBypass datasets for downstream tasks, with Table 1 showing that the SVL pretraining datasets sufficiently cover the required visual concepts. The SVL videos focus on laparoscopic surgeries, covering common concepts like instruments and bleeding, while their descriptive texts offer valuable language supervision for tasks like instrument recognition and adverse event detection.\n\n**[Q1.3. Affect Downstream Dataset]** We hypothesize that the composition of different types of surgical videos in the pretraining dataset will affect performance. While research [1] suggests that altering the pretraining dataset to the downstream dataset improves zero-shot adaptation, we aim to build a generalizable pretrained model and thus do not filter videos based on the downstream datasets.\n\n_[1] Datacomp: In search of the next generation of multimodal datasets_\n\n**[Q2. LLM-Generated Text for Clip-level Alignment]** In this paper, we control the LLM-generated textual quality based on the in-context learning ability of LLMs, which enables faithful predictions with a limited context. By thoughtfully designing contextual input-output examples, LLM can generate diverse and rich texts that enrich the clip-level narration texts, as shown below: \n\n< Table S1>. Original noisy transcription and assigned step text based on LLM.\n| Raw Transcript | Assigned Step Text  |\n|-|-|\n| but you think about two richard deliver from with amoxicillin truck and to work with your left hand with the right four unit   | Placement of five trocars: typically one for the camera, two for the surgeon's instruments, one for the liver retractor, and one for the assistant.   |\n| so I be try to prepare my posterior dissection on the joel | Dissection of the Esophageal Hiatus: Dissect the phrenoesophageal ligament to expose the esophageal hiatus.|\n\nThe generated surgical steps from LLM are not always perfectly aligned with the clip-level frames. To address this, we randomly pick either the original or the augmented texts during video-language pretraining.  \n\nNoisy image-text alignment is a challenge for all CLIP-based methods. Scaling up the dataset could help mitigate this issue by providing a broader context and improving alignment, and we leave this exploration to future research.\n\n**[Q3. Textual Similarity]** Textual similarity is calculated using cosine similarity between feature vectors from pseudo steps and narrations. We use BioClinicalBert to extract 768-dimensional embeddings for each sentence, normalize them, and compute the dot product to get similarity. This approach is consistent with methods used in previous literature.\n\n**[Q4. Effectiveness of Three Levels Augmentation]** Knowledge augmentation from different levels improves performance in different ways, as summarized in the table below:\n\n< Table S2>. Performance on Cholec80.\n| Model| Accuracy / F1-Score |\n|-|-|\n| SurgVLP| 34.7 / 24.4|\n| SurgVLP+Clip-level Augmentation| 36.1 / 26.8 |\n| HecVL | 41.7 / 26.3 |\n| HecVL+Phase-level Augmentation| 44.0 / 31.8 |\n| HecVL+Video-level Augmentation| 43.7 / 30.6|\n\nWe found that phase-level knowledge augmentation significantly improves surgical vision-language pretraining due to its concise and less noisy keystep texts, whereas LLM-generated surgical knowledge base is less effective due to noisy image-text alignment in pseudo-step assignment.\n\n**[Q5. Augmentations Affect Corresponding Text's Semantics]** For visual self-supervision, we use spatial augmentations like random cropping and horizontal flipping, which might introduce vision-language misalignment. We find that around 9% of sentences contain spatial words. Since one video maps to multiple sentences, the actual impact from augmentation is less than 9%, which is a less significant source of noise than misspellings and incomplete transcripts. Also, our experiments show that these augmentations consistently boost performance, with their benefits outweighing potential misalignments.\n\n**[Q6. Procedural Information]** We clarify that the surgical procedural information learned from pretraining is not just a linear sequence of phases but includes complex dependencies among surgical key steps. PeskaVLP learns from various surgical lecture videos, capturing possible paths to complete a surgery. For instance, if one pretraining video includes _[.., gallbladder dissection, cleaning and coagulation, ..]_ and another _[.., gallbladder dissection, gallbladder packing, ..]_, our procedure-aware pretraining can identify multiple possible steps following gallbladder dissection. This procedural information is useful for temporal modeling in surgical phase recognition when applying methods like temporal convolutional networks or task graphs to perform online phase prediction. \n\n**[Q7. Table 3]** We have updated the manuscript and added discussion. The revised manuscript explains that Moco's strong performance on Cholec80 is due to its specialization in cholecystectomy, which limits its generalizability to other surgeries.\n \n< Table S4>. Correct version of table 3 in manuscript.\n| Pretraining | Dataset| Cholec80 | Autolaparo | StrasBypass70 | BernBypass70 |\n|-|-|-|-|-|-|\n| Moco | Cholec80| **73.4 / 62.8** | 51.3 / 37.4 | 67.8 / 55.4| 66.0 / 33.1|\n| Moco| SVL | 68.2 / 55.8 | 59.5 / 48.4 | **71.6** / 58.1| 69.6 / 36.5|\n| PeskaVLP| SVL| 69.9 / 59.8 | **63.1 / 49.7** | 71.4 / **59.5**| **71.5 / 37.4**|", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7Q5nQkdlIh", "reviewer": "Reviewer_smvC", "summary": "This paper proposes a Procedure-Encoded Surgical Knowledge-Augmented Video-Language Pretraining (PeskaVLP) method that enriches language supervision with LLM-refined surgical concepts. It further constructs hard negative samples by reversing the text orders at the phase and video levels and employs a Dynamic Time Warping (DTW) based loss to align multimodal procedures. Extensive experiments on multiple surgical procedures and comprehensive evaluations demonstrate the effectiveness of this framework.", "questions": "- Beyond the current ablation study on PeskaVLP components, would applying the hierarchical knowledge-augmented text data in HecVL improve its performance and if this could yield results competitive with PeskaVLP. This would provide powerful support to verify the extent to which the other components in PeskaVLP contribute to performance, apart from the augmented texts.\n- Although LLM can enhance surgical text quality, is there a concern that the text may become overly standardized? Given that surgeons' narratives in the operating room tend to be more oral, concise, and sometimes include jargon, will there be a performance degradation in real-world, real-time applications where LLM augmentation is impractical?\n- In Appendix E, Figure 4, it would also be interesting if the authors could visualize the embeddings of HecVL, since it performs better than SurgVLP.\n- In Table 3, on Cholec80, Moco pre-trained on Cholec80 (V) has better performance but wasn't in bold, do I misinterpret something?", "limitations": "The authors adequately addressed the limitations. Since the proposed method is tailored for surgical data and applications, it is strongly suggested that the authors include a discussion on the potential negative societal impact of their work.", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The paper is overall well-written, with the background and motivation well-stated.\n- Using LLM to augment surgical video text descriptions is a good idea to enhance the quality of surgical text narration. It establishes a good baseline and guideline for future works that aim to apply LLM in surgical narratives.\n- A more comprehensive parent-child level cross-modal correspondence was designed using DTW than existing works.\n- Demonstration of the proposed method can close the representation gap for different modality, and analysed both successful and complicated examples.", "weaknesses": "- By reading the enriched dataset by LLM in Appendix H, I am concerning that the variation and diversity of narration will be removed by the augmentation. Will that cause any problems?\n- In my opinion, using LLM to refine the text description of surgical videos is the most important contribution of this paper. It would be interesting to see if other components are also effective enough without the knowledge augmentation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**[Q1. Augmentation Removes Variation]** Thank you for pointing out one of the key insights of this work, i.e., using LLM to build a large, versatile, and accurate surgical knowledge base to enrich and correct narrations of different types of videos during the pretraining. Since we enrich the narration based on the built knowledge base, the question becomes if the LLM-generated surgical knowledge base is diverse enough.  \n\nIn this work, we aim to bring the diversity and versatility of the built knowledge base by curating 917 lecture titles, covering diverse surgical procedures such as colorectal, transanal and proctological, cholecystectomy, hernia, and sigmoidectomy surgery. We also manually design input-output examples to instruct the LLM to generate diverse steps. Additionally, during the pretraining, we randomly select either the pseudo step or the original narration text to maintain textual semantics. \n\nOur approach might face risks when LLM generates an over-standardized surgical step knowledge base. However, our experiments show that the downstream zero-shot performance clearly improves when the LLM-generated knowledge base is applied. This implies that the advantages of correcting noisy narration texts outweigh the potential variation and diversity risks. \n\n< Table S1>. Zero-shot phase recognition performance on Cholec80 dataset.\n|            | SurgVLP | SurgVLP + LLM Augmentation |\n|------------|---------|----------------------------|\n| Accuracy   | 34.7    | 36.1                       |\n| F1-Score   | 24.4    | 26.8                       |\n\n**[Q2. Other Components]** In the following table, we show that the combination of visual self-supervision and language supervision at the clip-level vision-language pretraining are effective even without the knowledge augmentation. \n\n< Table S2>. Zero-shot phase recognition on Cholec80 and Autolaparo datasets.\n| Model| Dataset   | Accuracy / F1-Score  |\n|---------|------|--------|\n| HecVL| Cholec80  | 41.7 / 26.3|\n| HecVL+LecNCE{clip}    | Cholec80  | 45.5 / 31.0|\n| HecVL| Autolaparo| 23.3 / 18.9|\n| HecVL+LecNCE{clip}    | Autolaparo| 25.3 / 20.0|\n\nThe results indicate incorporating knowledge augmentation and visual self-supervision can individually benefit the pretraining performance. \n\n**[Q3. Different Levels of Knowledge Augmentation]** Knowledge augmentation from different hierarchies improves the final performance in different ways. We have summarized results in the table below: \n< Table S3>. Performance on Cholec80 dataset with different augmentations.\n\n| Model                              | Accuracy / F1-Score |\n|-----------|---------------------|\n| SurgVLP | 34.7 / 24.4 |\n| SurgVLP+Clip-level Augmentation  | 36.1 / 26.8|\n| PeskaVLP  | 45.1 / 34.2 |\n| HecVL | 41.7 / 26.3  |\n| HecVL+Phase-level Augmentation    | 44.0 / 31.8|\n| HecVL+Video-level Augmentation    | 43.7 / 30.6|\n| PeskaVLP  | 45.1 / 34.2         |\n\nWe observe that applying knowledge-augmented text data in SurgVLP and HecVL achieves a clear improvement, though they still underperform the PeskaVLP. These results demonstrate the effectiveness of the other components of PeskaVLP, i.e., combination of visual self-supervision and language supervision and procedure-aware pretraining objective. \n\n**[Q4. Overly Standardized Texts]** We thank the reviewer for this interesting question. We apply the LLM augmentation strategy for pretraining by considering the goal of representation learning, downstream surgical workflow analysis applications, and the nature of pretraining surgical lecture videos. The goal of PeskaVLP is to learn the correspondence between vision and language modality with the aim of improving the performance on the surgical downstream tasks. Therefore, the jargons and interjection can introduce the noisy alignment, which is filtered out in our pretraining dataset. \n\nOur pretraining videos are primarily lecture videos where narrations are scripted and more formal. The GPT-augmented strategy is well-suited for this type of data, as it enhances the clarity and completeness of the text. \n\nWe acknowledge the potential limitations in real-world applications. For future work, incorporating more diverse and real-time surgical audio [1] into the pretraining process could help improve the performance. \n\n_[1] Jia, Shuyue, et al. \"MedPodGPT: A multilingual audio-augmented large language model for medical research and education.\" medRxiv (2024): 2024-07._\n\n**[Q5. Modality Gap HecVL]** In the rebuttal letter PDF, Figure 1, shows HecVL's embeddings, showing a smaller modality gap for video-abstract pairs compared to SurgVLP. This shows the benefit of hierarchical vision-language pretraining for aligning long-form videos with high-level summaries. PeskaVLP consistently outperforms prior baselines, demonstrating its effectiveness in closing the modality gap and enhancing pretraining for vision-and-language tasks.\n\n**[Q6. Table 3]** Thank you for pointing this out. We have corrected the table to highlight Moco (third row) as providing the best results on the Cholec80 dataset and Moco (second row) on the StrasBypass70 dataset. In the revised manuscript, we have discussed that Moco pretrained on Cholec80 outperforms the others because it is specifically trained on cholecystectomy procedures, thereby losing versatility. This specialization allows Moco to achieve superior performance on Cholec80 but limits its generalizability to other datasets, as summarized below:\n \n< Table S4>. Correct version of table 3 in manuscript.\n| Pretraining | Dataset      | Cholec80 | Autolaparo | StrasBypass70 | BernBypass70 |\n|-------------|--------------|----------|------------|---------------|--------------|\n| Moco        |  Cholec80  | **73.4 / 62.8** | 51.3 / 37.4 | 67.8 / 55.4   | 66.0 / 33.1  |\n| Moco        | SVL | 68.2 / 55.8 | 59.5 / 48.4 | **71.6** / 58.1   | 69.6 / 36.5  |\n| PeskaVLP    | SVL   | 69.9 / 59.8 | **63.1 / 49.7** | 71.4 / **59.5**   | **71.5 / 37.4**  |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fLzJ6lMID0", "reviewer": "Reviewer_EZYT", "summary": "The paper presents a novel approach for enhancing surgical video analysis by incorporating procedural awareness. The authors propose a system that integrates knowledge of surgical procedures to improve the identification, segmentation, and annotation of surgical activities in video footage. This approach aims to address challenges such as the variability of surgical techniques and the complexity of visual data in operating rooms. The contributions of the paper include the development of a procedural model that can be aligned with video data, the creation of annotated datasets for training and evaluation, and the demonstration of improved performance over traditional video analysis methods.", "questions": "Generalizability: How does the system perform across different types of surgeries (like ophthalmic surgery)? Have you tested its effectiveness in various surgical domains beyond the initial scope?", "limitations": "The paper does not adequately address potential limitations and negative societal impacts.", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "1.The integration of procedural knowledge into surgical video analysis is a highly original concept. This approach not only enhances the accuracy of video analysis but also opens new avenues for improving surgical training and documentation.\n\n2.Introduces a novel hierarchical knowledge augmentation technique using large language models to refine surgical concepts. Employs a Dynamic Time Warping-based loss function for effective cross-modal procedural alignment. Demonstrates significant improvements in zero-shot transfer performance across multiple surgical datasets. Provides a robust general visual representation beneficial for various surgical scene understanding tasks.\nWeaknesses:\n\n3.The potential applications of this research in surgical training, intraoperative assistance, and postoperative review are significant. The approach addresses a critical need in medical video analysis, making it highly relevant and impactful.", "weaknesses": "Dataset Limitations: The annotated datasets used for training and evaluation are crucial for the model's success. Expanding the diversity and volume of these datasets would enhance the generalizability of the findings.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**[Q1. Dataset Limitations]** Thank you for the insightful suggestion. In the rebuttal letter pdf, we have added a table to summarize the top 42 types of surgical videos and their amounts in the pretraining dataset. As shown in Table 1 of the rebuttal letter PDF, the SVL dataset predominantly consists of laparoscopic surgeries and covers a diverse range of surgical types, including stomach, duodenum, hernia, colon, gallbladder, and tumor surgeries. This diversity ensures that the SVL dataset provides broad and generalizable language supervision for surgical multi-modal representation learning.            \nIn this paper, the downstream datasets are Cholec80 (Laparoscopic cholecystectomy), AutoLaparo (Laparoscopic hysterectomy), and MultiBypass (Laparoscopic gastric bypass). Our SVL pretraining dataset contains sufficient videos that cover the visual concepts required for these downstream tasks. We agree with the reviewer that expanding the pretraining dataset to cover more types of surgeries can improve diversity and improve generalizability. We leave this exploration to future research endeavors.   \n       \n**[Q2. Generalizability]** Thank you for the insightful question. Evaluating the generalizability of our system across different types of surgeries is crucial for understanding its broader applicability. Our system has been tested extensively on laparoscopic surgeries, including Cholec80 (Laparoscopic cholecystectomy), AutoLaparo (Laparoscopic hysterectomy), and MultiBypass (Laparoscopic gastric bypass). These evaluations demonstrate the model's ability to effectively recognize phases and understand the workflow in these contexts.                   \nSince we do not have ophthalmic surgical videos in our pretraining dataset, we do not test the PeskaVLP’s performance in this domain. Future work would involve incorporating and testing diverse surgical datasets, including ophthalmic surgery, to ensure the model's effectiveness across various surgical domains. This will help us assess the system's capability to generalize and perform effectively in a broader range of surgical contexts.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "x9lmNImh2H", "reviewer": "Reviewer_usyA", "summary": "The paper addresses challenges in surgical video-language pretraining (VLP) due to the knowledge domain gap and scarcity of multi-modal data. It proposes a hierarchical knowledge augmentation approach and the Procedure-Encoded Surgical Knowledge-Augmented Video-Language Pretraining (PeskaVLP) framework. This approach enhances data efficacy and tackles spatial-temporal challenges by combining language supervision with visual self-supervision. Extensive experiments demonstrate significant improvements in zero-shot transferring performance and the generalist visual representation for surgical scene understanding.", "questions": "1. How do you plan to address the limited sample size and diversity in future studies to improve the generalizability of your findings? Consider expanding the dataset to include a more extensive and more diverse sample of surgical procedures to enhance robustness and applicability.\n\n2. What specific methods did you use to handle ASR errors in transcriptions? How did these errors impact your analysis?\n\n3. How do you manage the computational overhead associated with the hierarchical pretraining and dynamic time-warping processes?", "limitations": "The authors have acknowledged the limitations related to dataset size and ASR errors but could elaborate on strategies to mitigate these issues. Specifically, they should discuss plans for expanding the dataset, incorporating more diverse samples, and improving transcription accuracy.\n\nThe positive societal impacts, such as enhancing surgical training and assistance, are well-discussed. However, the authors should address potential negative impacts, such as data privacy and ethical concerns. A detailed discussion on data security measures, user consent protocols, and ethical safeguards is needed.", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "The paper presents a unique approach to surgical video-language pretraining by employing hierarchical knowledge augmentation using LLMs, significantly improving textual data quality and diversity. The PeskaVLP framework innovatively integrates visual and language supervision, addressing the spatial-temporal challenges in surgical scene understanding. The methodology is meticulously validated through extensive zero-shot and linear-probing evaluations on datasets such as Cholec80 and AutoLaparo, demonstrating substantial performance improvements. The clarity of the presentation, with well-organized sections and effective visual aids, facilitates comprehension. The significant contribution lies in enhancing surgical scene understanding and cross-modal retrieval, making it highly valuable for the NeurIPS community. The paper's originality in using hierarchical pretraining and the detailed discussion on model architectures and initialization underscore its quality and significance in advancing surgical data science.", "weaknesses": "Firstly, the dataset size is relatively small, with 1,007 videos for phase-level pretraining and 920 for video-level pretraining, which may limit the generalizability of the findings (as mentioned in the supplementary material). I know the difficulty in collecting medical data, but we must be sure that the presented approach can be generalized to different domains and hospitals. Furthermore, I doubt the methodology's potential to process \"noisy\" videos.    \nExpanding the dataset and including more diverse surgical procedures would improve robustness. \n\nSecondly, while the paper mentions ASR errors in transcriptions, it does not provide a detailed methodology for handling them. Providing specific techniques for improving transcription accuracy would strengthen the study. \n\nAdditionally, the practical implementation of the PeskaVLP framework in real-world surgical contexts is not thoroughly discussed. Detailing strategies for integration into clinical workflows and addressing potential technological barriers would be beneficial.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**[Q1. Plan to Expand Dataset]** Scaling and diversifying the surgical vision-language pretraining dataset is challenging due to privacy concerns and the cost of expert annotations. Even though the SVL pretraining dataset covers diverse laparoscopic surgeries, it lacks surgeries in different organs, such as the brain and heart. We fully agree with the reviewer that expanding the dataset would be crucial for developing more generalizable models. To address this, we plan to expand the pretraining dataset using diverse media such as textbooks, instructional videos, and intra-operative video recordings from diverse sources. We also aim to diversify the pretraining dataset by considering laparoscopic, endoscopic, and microscopic surgeries on different organs. \n\n**[Q2. Transcription Errors and Solution]** ASR errors can be divided into two categories: misspelling errors and incorrect punctuation. In this work, we address these errors using specific methods to mitigate their negative impacts:\n- **Misspelling Errors**: We first follow [1] and use multiple text preprocessing to correct the ASR errors first, including USMLE-based spelling checking. Then, we use the LLM to generate large-scale clean descriptive texts that detail the steps for diverse surgeries. Noisy transcriptions are then assigned to semantically similar step texts based on their semantic similarity, as shown below: \n\n     < Table S2>. Original noisy transcription and augmented text based on LLM.\n     | Raw Transcript | Assigned Step Text  |\n     |----------|----------|\n     | but you think about two richard deliver from with amoxicillin truck and to work with your left hand with the right four unit   | Placement of five trocars: typically one for the camera, two for the surgeon's instruments, one for the liver retractor, and one for the assistant.   |\n     | so I be try to prepare my posterior dissection on the joel     | Dissection of the Esophageal Hiatus: Dissect the phrenoesophageal ligament to expose the esophageal hiatus.   |\n\n    This ensures that the noisy texts are aligned with accurate descriptions. The use of the above LLM-based augmentation significantly improves the model's performance by providing cleaner and more accurate training data. As shown in the table below, SurgVLP shows improvement in accuracy when pretrained with augmented transcriptions. \n\n    < Table S2>. Zero-shot phase recognition performance on Cholec80 dataset.\n\n    |            | SurgVLP | SurgVLP + LLM Augmentation |\n    |------------|---------|----------------------------|\n    | Accuracy   | 34.7    | 36.1                       |\n    | F1-Score   | 24.4    | 26.8                       |\n\n- **Incorrect Punctuation Errors**: We follow SurgVLP and use Whisper, an ASR system based on a language decoder, which generates complete sentences with fewer punctuation errors. Whisper-transcribed sentences help define video correspondence boundaries more accurately. \n\nWe anticipate that the modeling-based approach can also be applied in the future. For example, developing ASR systems specifically trained on surgical data can further reduce errors and improve transcription accuracy. Also, pretraining text encoders on a surgical corpus [2] can make the model more robust to ASR errors.       \n  \n_[1] Ikezogwo, Wisdom, et al. \"Quilt-1m: One million image-text pairs for histopathology.\" Advances in neural information processing systems 36 (2024)._\n\n_[2] Bombieri, Marco, et al. \"Surgicberta: a pre-trained language model for procedural surgical language.\" International Journal of Data Science and Analytics 18.1 (2024): 69-81._\n\n**[Q3. Computational Overhead]** We manage the computational overhead associated with hierarchical pretraining and dynamic time-warping (DTW) processes by using efficient resource allocation and high-speed computational infrastructure. Below is a summary of the time and computational cost when applying hierarchical pretraining and DTW processes:            \n< Table S3>. Training configurations and time for SurgVLP.\n| Configuration             | Training Time | GPU            |\n|---------------------------|---------------|----------------|\n| SurgVLP                   | ~40 Hours     | 1xA100-80G     |\n| + Hierarchical Pretraining| ~90 Hours     | 4xA100-80G     |\n| + Hierarchical + DTW      | ~120 Hours    | 4xA100-80G     |               \n\nIn this work, the speed bottleneck is at the online video loading and preprocessing. In future, we aim to apply the asynchronous processing pipelines to further parallelize the workload and reduce bottlenecks.\n\n**[Q4. Acknowledge the Plans to Mitigate Data Size and ASR Errors ]** Thank you for the insightful suggestion. In the revised manuscript, we have added a discussion section to elaborate the strategies about expanding dataset and improving transcription accuracy, as we discussed in prior responses. \n\n**[Q5. Potential Negative Impacts]** Thank you for the insightful suggestion. As discussed in the paper, our dataset is compiled from open educational platforms accessible to all learners, minimizing any potential societal impact. However, in the rapidly advancing field of surgical data science, there are still risks of privacy concerns and data security associated with the collection and use of surgical video data. Ensuring robust data protection measures and maintaining patient confidentiality is crucial to mitigate these risks.\n\nMoreover, the development of automated intra-operative surgical procedural understanding systems should aim to provide intuitive cognitive aids for surgeons, enhancing their decision-making during complex procedures. These systems should support surgeons by delivering relevant information at critical moments, rather than overwhelming them with extraneous details. We have added a discussion on the potential negative societal impact in the revised manuscript.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank all the reviewers for the insightful comments to improve our work. We are encouraged that the reviewer finds our work an interesting contribution to the community. We have carefully considered each comment from the reviewers and tried to provide detailed answers, clarifying all the issues raised. We hope our responses effectively resolve the concerns raised. We are grateful for the reviewers' time and insightful feedback. If further clarifications or additional experiments are required, please let us know.                                                 
We are pleased to note that the reviewers recognized: 
- This paper addresses a novel surgical vision-language pretraining task that can potentially open new venues for surgical data science [usyA, EZYT] 
- The proposed hierarchical knowledge augmentation improves surgical textual data quality and diversity [all reviewers] 
- The manuscript is well-organized, clearly written, and presented in a reader-friendly way [HH5E, smvC, usyA] 
- This work novelly integrates visual-language supervision and dynamic time warping to learn the cross-modal correspondence [smvC] 
         
We have included a rebuttal letter PDF, including SVL dataset details [All reviewers] and modality gap visualization of HecVL [smvC].

Meta-review: The submission introduces a novel framework for surgical video-language pretraining, which enhances language supervision by integrating surgical concepts refined through large language models (LLMs). The paper presents extensive experiments across multiple surgical procedures and provides a thorough evaluation that underscores the framework's effectiveness. A key novelty is the use of LLMs to augment surgical video text descriptions, addressing the common issue of limited text narration, particularly for key frames. This approach significantly improves the quality and diversity of the generated surgical text.

The reviewers unanimously recommend accepting the paper, particularly after the authors' rebuttal addressed initial concerns. Therefore, I recommend acceptance of this paper. 

The authors are encouraged to incorporate the discussion points and clarifications raised during the review process into the final version. Additionally, it is essential that the authors make the training dataset available by sharing links to the videos used in the study, along with the necessary preprocessing code to generate video-text pairs for training. The source of the dataset and the associated license should be clearly stated to ensure transparency and reproducibility.
Decision: Accept (spotlight)