Procedure-Aware Surgical Video-language Pretraining with Hierarchical Knowledge Augmentation

Authors: kun  yuan
vinkle  srivastav
nassir  navab
nicolas  padoy
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-13 01:52:14

Abstract: Surgical video-language pretraining (VLP) faces unique challenges due to the knowledge domain gap and the scarcity of multi-modal data. This study aims to bridge the gap by addressing issues regarding textual information loss in surgical lecture videos and the spatial-temporal challenges of surgical VLP. To tackle these issues, we propose a hierarchical knowledge augmentation approach and a novel Procedure-Encoded Surgical Knowledge-Augmented Video-Language Pretraining (PeskaVLP) framework. The proposed knowledge augmentation approach uses large language models (LLM) to refine and enrich surgical concepts, thus providing comprehensive language supervision and reducing the risk of overfitting. The PeskaVLP framework combines language supervision with visual self-supervision, constructing hard negative samples and employing a Dynamic Time Warping (DTW) based loss function to effectively comprehend the cross-modal procedural alignment. Extensive experiments on multiple public surgical scene understanding and cross-modal retrieval datasets show that our proposed method significantly improves zero-shot transferring performance and offers a generalist visual repre- sentation for further advancements in surgical scene understanding. The source code will be available at https://github.com/CAMMA-public/PeskaVLP.

Main paper: 
 introduction: 
 The recent advancements in multi-modal representation learning, particularly with the introduction of CLIP [52] , have led to the development of models capable of understanding a wide range of visual concepts using natural language supervision [34, 41] . The expressive natural language has allowed these models to shift from task-specific to more generalist applications [49, 82, 83] . The learned representations of these models are robust, facilitating effective performance across diverse visual tasks without the need for task-specific fine-tuning [68, 81] . However, despite the impressive progress made by these models in the general computer vision domain, the effectiveness of these methods in domain-specific settings remains uncertain. This concern is particularly relevant to the field of Surgical Data Science (SDS), an emerging interdisciplinary domain that utilizes deep learning and computer vision techniques to analyze surgical data [44, 43, 74] . A key component of SDS is the analysis of intraoperative surgical videos captured through endoscopes or laparoscopes. Analyzing these videos presents several unique challenges compared to the general computer vision datasets. Unlike general computer vision datasets [47, 52, 7] , surgical videos can last several hours and capture complex and fine-grained activities within a narrow field of view. This requires development of computational approaches to decompose and model the surgical procedures at multiple hierarchical levels, including the entire procedure [29] , phases [67, 16] , steps [54, 31] , atomic actions [6, 8] , and action triplets [50, 62] . Moreover, surgical language involves specialized vocabulary, and annotating videos requires clinical expertise, limiting dataset scalability. Consequently, current deep learning applications are restricted to single-centric, fully-supervised, and task-specific approaches [3, 6, 31, 50, 55, 57, 67, 69, 74] .To bridge the gap, recent efforts have focused on creating surgical video-text pretraining datasets by curating surgical lecture videos from online e-learning platforms and pairing them with transcribed narrations using audio speech recognition (ASR) methods. Subsequently, a CLIP-style model [76] is trained contrastively to match the video clips to their corresponding textual descriptions. Building on this, the HecVL approach introduces hierarchical texts, including phase-level keystep descriptions and video-level summaries that provide hierarchical goals of the surgical procedure [75] . However, challenges persist due to the smaller size of the surgical video-language pretraining dataset, noisy transcribed narrations, limited variability in phase-level descriptions, and strong temporal dependencies in surgical procedures, where actions and keysteps occur in a specific routine order. These issues hinder the accurate learning of multi-modal surgical representations.To address these challenges, we propose Procedure-Encoded Surgical Knowledge-Augmented Video-Language Pretraining (PeskaVLP), which boosts data efficacy and tackles the spatial-temporal challenges inherent in surgical procedures from two perspectives. First, we introduce hierarchical knowledge augmentation to mitigate the problem of textual information loss in surgical videolanguage pretraining datasets. We argue that the internal knowledge of LLMs serves as a valuable surgical knowledge base, enriching and correcting text descriptions while preserving the original key concepts and meanings. Therefore, we utilize the large language model (LLM) prompted with different behaviors as an external knowledge base to correct, explain, or summarize the hierarchical texts in the surgical video-language pretraining dataset, thus providing diverse and better language supervision for multi-modal pretraining. Additionally, it reduces the risk of overfitting by preventing the text encoder from repeatedly encountering the same keystep texts in each epoch.From the pretraining objective perspective, we perform the hierarchical video-language pretraining, as shown in Fig. 1 , with a novel hierarchy-specific loss, LecN CE. Specifically, we combine language supervision with visual self-supervision at the clip-level pretraining to introduce additional supervision signals within vision modality, making the pretraining efficient with a small surgical dataset [76] . At phase-and video-level pretraining, we construct hard negative samples by reversing the order of texts, followed by a Dynamic Time Warping (DTW) based loss function to learn the temporal alignment between video frames and texts, thus facilitating the understanding of cross-modal procedural alignment during pretraining.We summarize our contributions as follows: First, we propose an LLM-based knowledge augmentation to handle surgery-specific textual information loss in the dataset, providing more densely interconnected natural language supervision from surgical lecture videos. Second, our proposed hierarchical video-language pretraining method enforces the understanding of the spatial-temporal characteristics of surgical lecture videos at different hierarchical levels. The pretrained PeskaVLP demonstrates state-of-the-art transferability and visual representation to different surgical scene understanding downstream datasets [67, 69, 31] , across types of surgical procedures and clinical centers. It also shows strong multi-modal alignment ability through the cross-modal retrieval task at multiple hierarchical levels. 
related works: 
 Surgical Video-Language Pretraining: many works have demonstrated the effectiveness of learning visual representations from the natural language supervision of corresponding text [7, 70, 77, 40, 46, 42, 34] . These methods conduct contrastive learning [51] to match the video clips (or images) with their corresponding narrations (or captions). Similarly in the medical field, recent works have started to curate large-scale multi-modal data through hospital-sourced chest radiological reports [28, 12] and online platforms [76, 27, 26] , e.g., YouTube and Twitter, to perform vision-language pretraining. However, these works encounter the sample efficiency issue when handling the smaller surgical video-language pretraining dataset (SVL) [76] . Recent works improve the data efficacy and zero-shot performance of CLIP-style models [48, 37, 25] . However, they do not capture procedural dependency from the long-form surgical videos beyond the video clip and text matching. Hierarchical pretraining methods [4, 79, 75] propose to pair video clips of different durations to different hierarchical levels of texts, covering both short-and long-term understanding. Paprika [80] builds a procedural knowledge graph and elicits the knowledge node during the video-language pretraining process.Textual Augmentation with Knowledge Base: the success of vision-language pretraining is highly dependent on the quality and quantity of available multi-modal data. Recent research [38] shows that a smaller high-quality dataset can outperform a larger low-quality dataset. Common practices improve the quality by textual augmentation, including EDA [37] , masked token modeling [65] , and captioning loss [72] . Recent studies have used synthesized captions from captioning models to achieve notable improvements [33, 32, 58] . However, they show scalability deficiency and world knowledge loss in models trained with synthetic captions [73] , which their initial benchmark success has largely obscured. To inject the knowledge, K-Lite [63] enriches the texts with WordNet [15] and Wiktionary [45] knowledge base. Merlot [78] learns script knowledge representations from millions of YouTube videos, however, a knowledge domain gap exists when applying this to the surgical field. The recent advent of large language models like GPT4 [2] and Llama series [66] have been a game-changer, as they encode rich domain-specific knowledge, e.g., clinical knowledge [64] ,motivating LaCLIP [14] to augment textual inputs through the LLM rewrites.3 ApproachQuestion: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes] Justification: We discuss the limitation in the Appendix G. Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
dataset and contrastive learning: 
 Learning joint video and language embedding space requires a large-scale video-language dataset, however, such datasets are expensive and time-consuming to create in the surgical field. Therefore, the first surgical video-language pretraining dataset, i.e., SVL [76] , is proposed by obtaining around a thousand surgical lecture videos from surgical education platforms. SVL collects ∼300 hours of lecture videos accompanied by narration texts obtained using Audio Speech Recognition (ASR) methods, providing ∼26k video clip-narration pairs for contrastive video-language pretraining. Specifically, short video clips x c and their corresponding narration texts y n are treated as positive pairs P n , and the unpaired ones are treated as negative pairs N n . Then, the contrastive training loss Inf oN CE [51] can be formulated as follows:EQUATIONwhere B represents the batch size. The f and g are visual and textual encoders that generate embedding vectors for videos and texts, respectively. This loss function aligns two modalities by increasing the cosine similarity between paired videos and texts and decreasing the unpaired ones, as shown in Fig. 1 (a) . Despite reaching an impressive data scale, the imprecision of the ASR system and the scarcity of surgical lecture videos limit the natural language supervision from SVL. Therefore, HecVL [75] proposes to incorporate hierarchical language supervision by extracting additional phase-level keystep and video-level abstract texts from lecture videos' metadata, as shown in Fig. 1 (b) and (c). In this work, we use this hierarchical video-language pretraining dataset and perform hierarchical knowledge augmentation to improve the textual quality.Quality of language supervision in the multi-modal representation learning matters [1, 37, 36] , especially when the surgical video-language dataset is not "big" enough, e.g., millions of multi-modal samples used in [52, 47] , to sufficiently cover the visual-linguistic concepts. In this work, we find that the texts suffer from different types of degradation at different hierarchies, failing to provide accurate and broad concepts for pretraining. Specifically, as shown in Fig. 2 , narration texts are mostly sentence fragments and easily affected by misspelling errors, therefore altering the original key concepts. The keystep texts are mostly short and abstract, resulting in a narrow set of linguistic concepts that could show poor transferability to the downstream datasets, which usually come with a different set of concepts [63, 18] . The abstract texts sometimes include redundant and useless information, such as author and citation information.To address the above hierarchy-specific textual degradation, we propose a hierarchical knowledge augmentation to correct, explain, and summarize the narration, the keystep, and the abstract texts, respectively, by eliciting LLM's encoded surgical knowledge [64] . For each hierarchy, we manually design the system prompt and several input-output examples for LLM. Thus, we obtain hierarchical LLM assistants with different behaviors of using internal surgical knowledge to augment the texts: Figure 3 : The pretraining pipeline of different hierarchies. We combine language supervision and visual selfsupervision at clip-level pretraining. We conduct the procedure-aware contrastive learning at phase/video-level pretraining.Narration. We ask the LLM to behave as a "recipe" to come up with a list of sequential steps that complete the given surgery. For each lecture video, we feed its title as input and obtain the list of pseudo steps, as shown in Fig. 2 (a), building a surgical step knowledge base. Then, we assign these pseudo steps to narration texts based on textual similarity. This implicitly corrects the typos in transcribed narrations and augments the textual input based on the LLM's surgical knowledge.Keystep. As shown in Fig. 2 (b), we ask the LLM to behave like a "dictionary" to explain the meaning of the keystep. Specifically, the LLM assistant expands the given keystep into a description of the main surgical events, anatomies, and instruments involved. This enlarges the textual semantic information of each keystep and provides more expressive language supervision for pertaining.Abstract. As shown in Fig. 2 (b), we ask the LLM to behave like a "summarizer" that captures the key concepts of the given abstract texts, e.g., surgical type, anatomies, and so on. This reduces the length of the textual inputs while maintaining the main concepts of the abstract paragraph. In the following experiment, we randomly input the original or augmented texts for video-language pretraining. Check Appendix H for examples of pre-and post-augmented texts.We introduce PeskaVLP, a procedure-aware pretraining framework for the above surgical knowledgeaugmented video-language dataset. We emphasize devising a pretraining objective LecN CE for the hierarchical video-text pairs. For clip-level pretraining, LecN CE clip combines language supervision with visual self-supervision to improve data efficiency and boost the scene understanding on visually similar laparoscopic images. LecN CE phase/video considers the procedure awareness during the coarser-level pretraining, through a DTW-based contrastive regularization objective with temporally reversed text sequences as negative samples. We apply the dual-encoder as our model architecture.Language Supervision.The common pretraining objective for dual-encoder model is Inf oN CE [51] , as denoted in Eq. 1, where matched video text pairs are treated as positive while all other pairwise combinations in the batch are regarded as negative. In this work, we also apply Inf oN CE to maximize the similarity between short-term video clips and their corresponding narration texts at the clip level, denoted as L vl clip . However, this simple objective is data hungry and sensitive to the weakly aligned noisy video-text pairs from small-scale surgical video-language datasets, such as SVL [76] .Visual Self-supervision. The proposed PeskaVLP approach introduces an additional supervision signal from visual self-supervision to complement noisy language supervision. Specifically, we explore the widespread supervision within visual modality to learn generic visual representation. We adopt the simple yet effective SimSiam [11] strategy that aims to maximize the similarity between two augmented views. As shown in Fig. 3 (a) , during the pretraining, we apply random distortion on the frames of video clips and generate two augmented embedding vectors for one video clip. We then apply Inf oN CE to maximize the similarity of these two augmented embeddings by treating them as positive pairs, denoted as L vv clip . This additional supervisory can learn visual features more efficiently and is robust to the distortion of surgical scene images. Finally, the LecN CE loss for clip-level pretraining is the sum of these two losses, denoted as LecN CE clip = L vl clip + L vv clip .The surgical video-language pretraining presents a unique procedural challenge compared to the existing video-language methods [19, 47, 52, 71, 61] . The surgical actions and events occur in a certain order to follow the routine to complete the surgical phase and surgery, e.g., "hook dissecting cystic duct" should happen before "clipper cutting cystic duct" in the "clipping cutting" phase of cholecystectomy surgery. However, prior contrastive learning objectives [46, 52, 19 ] omit this temporal dependency and limit the understanding of procedural knowledge in surgical lecture videos.Our proposed LecN CE training objective enables procedural understanding in phase-and videolevel pretraining by considering the cross-modal temporal alignment between video frames and text sequence. Specifically, hierarchical texts can form the parent-child correspondence, i.e., abstract (parent-level) and keystep (child-level) texts, keystep (parent-level) and narration (child-level) texts.As shown in Fig. 3 (b), each parent-level text A is paired with a video segment V = {v 1 , ...v T },where the T is the number of frames of the video segment. A is also paired with a child-level text sequence B = {b 1 , ...b N }, where N is the length of this sequence. Then, we build the cost matrix C ∈ R T ×N between video frames and child-level text sequence based on their embeddings, with each element c i,j computed by a distance function D. We adopt the same distance function from [21] :c i,j = D(v i , b j ) = -log exp(ṽ ⊤ i bj /β) N k=1 exp(ṽ ⊤ i bk /β) , ṽi = f (v i )/∥f (v i )∥ 2 bi = g(b i )/∥g(b i )∥ 2(2) Using this cost matrix C, we apply Dynamic Time Warping (DTW) to find the minimum cross-modal cost path that aligns the video frames to the text sequence, denoted as DT W (C). We then make a reasonable assumption that the global semantics of the text sequence and its reversed version are distinct. Therefore, aligning the video frames to the text sequence should be easier, i.e., incur a lower alignment cost compared to aligning the same video frames when the text sequence is played in reverse. Following this assumption, we temporally reverse the child-level texts into B = {b n , ...b 1 } and build the cost matrix Ĉ between V and B, computing the minimum alignment cost DT W ( Ĉ). We then devise a DTW-based contrastive regularization using hinge loss as follows:EQUATIONwhere ϕ is the margin between positive and negative samples. This imposed regularization can support fine-grained multi-modal representation learning from weakly paired video frames and texts via temporal alignment. Unlike Paprika [80] , which relies on a pretrained model [46] , our phase-/video-level pretraining provides a direct, lightweight, and more adaptable methodology to unseen surgical domains. We do not require the adaption from any existing models, improving the generalization capability. Also, our pretraining process is procedure-aware in itself rather than modifying the representation in a second step, streamlining the process and increasing efficiency. We also apply the Inf oN CE loss to maximize the similarity between the paired parent-level text, video segment, and child-level texts, denoted as L inf once . Note that the L inf once follows the same pipeline as in Fig. 1 (b) and (c). Finally, we achieve the loss LecN CE for phase-or video-level pretraining as LecN CE phase/video = L inf once + λL dtw , where λ is the hyper-parameter to scale two losses. Please refer to Appendix D for more details about dynamic time warping. Finally, we train the model in an alternating way, using the proposed hierarchical levels of learning objectives. We only train one set of visual and textual encoders for all three levels, ensuring the encoders are optimized for capturing both short-term and long-term semantics. We alternatively train with 25 batches of clip-level samples, followed by 15 and 115 batches of phase-and video-level samples.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof? Answer: [NA] Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material? Answer: [No] Justification: We are working on cleaning the code for now as it is a systematic codebase that is related to multiple research works. We will release the code after the acceptance. Guidelines:• The answer NA means that paper does not include experiments requiring code. Justification: Our code follows the code of ethics.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.• Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
experiments: 
 Datasets. Our pretraining is conducted on the videos of SVL [76] dataset. The pertaining dataset includes hierarchical textual annotations from the metadata of the videos [75] . We evaluate our model on 3 publicly available surgical phase recognition downstream datasets, i.e., Cholec80 [67] (cholecystectomy) from Strasbourg center, AutoLaparo [69] (hysterectomy) from HongKong hospital, MultiBypass140 [31] (gastric bypass) from both Strasbourg (StrasBypass70) and Bern (BernBy-pass70) centers. These datasets contain untrimmed surgical workflows with frame-wise phase labels.We also evaluate pretrained model on the cross-modal retrieval task in multiple hierarchical levels with holdout videos in SVL-Retrieval [76] . Check Appendix A for more details about pretraining dataset.Training Parameters. We utilize the dual-encoder architecture with ResNet50 [23] as visual encoder and ClinicalBert [24] as textual encoder, respectively. We train the model with a batch size of 120/80/25 for clip-/phase-/video-level, respectively. We sample 4/16/64 frames for videos of clip-/phase-/video-level. We use AdamW optimizer [30] with a learning rate of 5e -5. We train the model with 4 NVIDIA A100 GPUs each having a DRAM of 80 GB for 200 epochs. Temperature parameter β for distance function and ϕ for DTW-base contrastive loss function D are fixed as 0.1. Scale factor λ is set as 0.01.Evaluation Setup. We evaluate pretrained models using two setups: Zero-Shot evaluation and Few/Full-shot Linear Probing evaluation. For Zero-Shot, we utilize class text prompts, the same as HecVL [75] , to compute cosine similarities between image embedding and class text embeddings, classifying images based on the shortest distance. In Linear Probing, the pretrained visual encoder remains frozen when we extract features for each image, subsequently training a linear layer using the SGD optimizer. For few-shot linear probing, we train the linear layer with a few numbers of videos, referred to as k-% training, where k indicates the percentage of all the videos used in training. Check Appendix B for more details.Table 1 : Zero-shot phase recognition results. We report Accuracy / F1-Score. PeskaVLP outperforms the other methods across different tasks. We report the state-of-the-art methods that are fine-tuned on the downstream dataset in a fully-supervised manner. However, models fine-tuned on specific downstream datasets show limited generalizability across procedures and institutions. 1 , our approach achieves a significant performance improvement over the baselines MIL-NCE [46] and CLIP [52] pretrained on the natural computer vision datasets, even though our pretraining dataset is 10, 000 times smaller than those. Note that when the CLIP model is randomly initialized and then trained with SVL, its performance declines compared to initializing from OpenAI. This shows that our surgical videolanguage pretraining dataset lacks the scale necessary to adequately pretrain a robust video-language model from scratch. ViT [13, 9] architectures are sensitive to initialization and excluded from this work. Further insights into the impact of initialization can be found in Appendix C.Transferability across Surgical Procedures and Centers. Compared to the HecVL, our method achieves over 12.3% and 7.8% improvement in absolute accuracy and f1, thanks to our spatialtemporal LecN CE learning objective across multiple hierarchies. Also, the consistent boost on cholecystectomy [67] , hysterectomy [69] , and gastric bypass [? ] procedures show the generalizable and transferable features of PeskaVLP. Comparing the results of StrasBypass and BernBypass, we find that PeskaVLP can recognize the phases of the same kind of surgery (gastric bypass), even if these surgeries are performed in different centers and follow different procedural routines. More qualitative results can be found in Appendix F.Table 2 : We present cross-modal retrieval results on the holdout videos, highlighting the best performance in each setting in bold. We additionally include coarser-grained phase-keystep and abstract-video text pairs to assess long-term video and high-level textual understanding. Clip-Narration Phase-Keystep Video-Abstract method R@1 R@5 R@10 R@1 R@5 R@10 R@1 R@5 R@10Text-to-Image (%) CLIP [52] 2.9 In our study, we evaluate pretrained models' cross-modal alignment efficacy by conducting both zero-shot text-to-image and image-to-text retrieval tasks in multiple hierarchical levels. We report the Recall@N metric by identifying the retrieved nearest neighbors for each query and then determining whether the corresponding ground truth element is within the top N nearest neighbors, where N ∈ {1, 5, 10}. Table 2 shows that our PeskaVLP achieves superior performance due to the procedureaware learning objective in hierarchical pretraining. Particularly, the hierarchical pretraining scheme significantly boosts the cross-modal retrieval at the coarse-grained video-text pairs, comprehending the relationship between long video segments and high-level sentences with surgical terms.General Visual Representation for Surgical Scene Understanding. We present the few-and full-shot linear-probing evaluation in Table 3 . It shows that the learned visual representation from PeskaVLP provides a general visual representation for surgical scene understanding across surgical procedures. We also find that the MoCo v2 [55, 22] pretrained on the frames of the SVL dataset (second row of Table 3 ) in a visual self-supervised manner achieves better visual representation than pretraining on a public dataset that only contains one type of surgery, e.g., Cholec80 (third row in Table 3 ). This shows that the cross-procedure surgical pretraining dataset enables better generalizationability.Knowledge Augmentation and Hierarchical Pretraining. Interestingly, the model pretrained contrastively with short video clips and narrations (SurgVLP) performs worse than MoCo v2 [55, 22] (second row in Table 3 ) in linear probing evaluation. This may be because the noisy narrations do not provide accurate natural language supervision for visual representation learning, thus highlighting the importance of visual self-supervision and textual quality. Our model surpasses the prior methods by a large margin, showing the efficacy of our hierarchical knowledge augmentation, which denoises the text and improves textual quality. Also, our proposed LecN CE promotes the visual encoder through additional visual self-supervision and procedural understanding. We present t-SNE visualizations of learned features in Appendix E, which shows that our multi-modal representations exhibit a smaller modality gap, enhancing transferability to vision-and-language downstream tasks [20, 39] .Effect of Knowledge Augmentation. Table 4 presents the effect of our proposed LLM-based hierarchical knowledge-aware augmentation strategy, applied to the texts of SVL dataset. The first row of the table corresponds to HecVL [75] pretrained on SVL with only conventional visual augmentations, e.g., blurring and so on, without any knowledge augmentation. The results clearly demonstrate that simple visual augmentation strategies exhibit poor robustness as the texts of SVL are noisy and not diverse enough. Conversely, our knowledge-aware text augmentation consistently improves performance across multiple surgical datasets, highlighting the importance of the textual quality of the surgical video-language pretraining dataset. We found that integrating visual self-supervision with language supervision significantly enhances performance in surgical scene understanding tasks across downstream datasets. Additionally, using a procedure-aware learning objective improves surgical phase recognition for routine procedures, such as cholecystectomy (Cholec80), more effectively than complex procedures, like hysterectomy (Autolaparo).Effect of Pretraining Objective. Table 4 shows the impact of our learning objective for hierarchical surgical video-language pretraining. When we append visual self-supervision to language supervision at the clip-level pretraining, the zero-shot performance is clearly improved. This improvement can be attributed to the added diverse and high-quality supervision. Also, the boost at linear-probing evaluation shows that the combination of language supervision and visual self-supervision leads to a robust visual representation especially with a moderate size of surgical video-language dataset, e.g., SVL. Table 4 also highlights that the inclusion of LecN CE with procedure understanding consistently improves performance across most downstream datasets, leading to enhanced accuracy in both zero-shot and linear-probing. However, performance on the AutoLaparo degrades with this modification. This may be due to challenging or less routined surgical procedures in the pretraining dataset. 
conclusion, limitations and broader impact: 
 Conclusion. We have introduced a surgical video-language pretraining method for long-term surgical lecture videos and their hierarchical paired texts. Our proposed knowledge augmentation addresses the hierarchical textual information loss by integrating the large language model's internal surgical knowledge. Also, we propose a novel spatial-temporal pretraining objective for video-text pairs of different hierarchies, which addresses the lack of supervision signals problem in a small surgical vision-language dataset. The proposed LecN CE also addresses the procedural awareness problem, benefiting the long-term cross-modal understanding. The experiments show that our proposed PeskaVLP achieves the state-of-the-art generalized zero-shot ability and visual representation learning that can serve as a general initialization for many surgical scene understanding tasks.Limitations. While our LLM-augmented strategy enhances textual information, it may overly standardize the text, raising concerns about overfitting during pretraining. Therefore, it is crucial to strike a balance between leveraging LLM capabilities and maintaining the variability present in real-world surgical narratives. To address this, future work will explore incorporating diverse audio inputs and spontaneous narratives into the pretraining process, ensuring that the model retains robustness and adaptability in real-world applications. Additionally, even though the SVL pretraining dataset covers diverse laparoscopic surgeries, it lacks surgeries in different organs, such as the brain and heart. To address this, we plan to expand the pretraining dataset using diverse media such as textbooks, instructional videos, and intraoperative video recordings from diverse sources. We also aim to diversify the pretraining dataset by considering laparoscopic, endoscopic, and microscopic surgeries on different organs, to further mitigate the risk of overfitting and enhance the model's generalizability.Broader Impact. The primary goal of surgical data science is to develop novel context-aware support systems for the operating room by collecting large-scale surgical data and analyzing it with modern AI techniques, eventually improving the safety and efficacy of surgical outcomes. The recent advancements in vision-language-based multi-modal AI offer significant potential in achieving this goal by enabling the development of more robust and generalizable models. These multi-modal systems have the potential to support clinical decision-making, streamline surgical workflows, provide real-time intra-operative guidance to improve surgical precision, reduce errors, and optimize outcomes in the operating room. During the development, patient data privacy should be considered as a fundamental ethical requirement. These systems developed on real-world surgical data also hold transformative potential in medical education, enhancing training and skill development in both novice and experienced surgeons. 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: Unlike generative models, this work provides generalist scene understanding as the foundation module for surgical data science. The data are anonymized because of the model encoding.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes]Justification: We will cite the original owners' assets when we release the codebase.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.• If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?Answer: [Yes]Justification: We will provide the documentation along with the code and dataset.Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. 
crowdsourcing and research with human subjects: 
 Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?Answer: [NA] Justification: We do not involve crowd-sourcing and human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector. 
institutional review board (irb) approvals or equivalent for research with human subjects: 
 Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?Answer: [NA] Justification: We do not involve crowd-sourcing and human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.• We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: We start with the videos that are used for surgical vision-language pretraining in [76] . In total, there are 1, 326 surgical lecture videos. These videos are transcribed by AWS [5] and Whisper [53] audio speech recognition (ASR) to obtain the corresponding narration texts. Furthermore, we curate the videos' metadata from the online platforms to obtain the extra keystep and abstract texts. In the phaseand video-level pretraining, we need parent-and child-level text correspondences, e.g., keystep and its corresponding narration texts, to perform procedure understanding. Therefore, we filter out the videos that do not have parent-child correspondences. In total, we have 1, 007 and 920 videos for phase-and video-level pretraining, respectively.As the narration texts are generated from the audio using the ASR system, they usually contain many misspelling errors and fragment sentences. Therefore, we apply multiple preprocessing steps to clean the narration texts.We first built the vocabulary based on the textbook, surgical category labels, and definition words. Specifically, we refer to the academic papers, which define the surgical phases, to curate a list of definition words and build a vocabulary that contains the words of interest. We also parse and merge the words from the textbook. In total, we obtain a vocabulary of the size of 51, 640 words. Then, we use the built vocabulary along with the spell-checking algorithm1 to correct the misspelling errors in narration texts. The algorithm utilizes Levenshtein Distance to identify words within 2 edit distances from the original. It then cross-references these permutations (insertions, deletions, replacements, and transpositions) with a word frequency list, prioritizing words with higher occurrence frequencies as potential correct results.We provide a detailed description of the downstream tasks and their settings that we apply in the experiment.Surgical Phase Recognition. Surgical phase recognition is a proxy task to test the model's surgical scene understanding ability. It aims to classify the frame of surgical video into predefined classes (phases), requiring the model to understand the instrument and anatomy's presence and their interactions by extracting visual patterns from the surgical scene image. In this work, we ignore temporal modeling in surgical phase recognition as we focus on multi-modal representation learning. We consider phase recognition as a frame-wise image classification problem. In the surgical phase recognition task, we evaluate the model's performance based on the publicly available datasets, including Cholec80 [67] , AutoLaparo [69] and MultiBypass [? ].• Zero-shot Evaluation. As the surgical phase labels are high-level definitions that can be decomposed into a few basic concepts, we manually construct the contextual prompts for phase labels, as shown in Tab. 5, Tab. 6 and Tab. 7. Our constructed prompts for the class names are built with the help of clinician's comments, considering the involved surgical instruments, anatomies, and events involved in a given surgical phase.• Linear-probing Evaluation. For linear-probing evaluation on the surgical phase recognition downstream datasets, we keep the visual encoder frozen and train a linear classifier on the extracted features. We do not apply any image augmentation during the training. The learning rate is scaled linearly based on the actual batch size. The model is optimized using SGD optimizer with the learning rate as 0.001 and weight decay parameter as 0.0005. We train the model for 40 epochs. We fit the model on the training and validation sets and report the performance on the separate test set. For the few-shot linear-probing evaluation, we adopt a k-percentage shot approach with a slight modification to accommodate the nature of surgical videos, which contain frames from different classes. Specifically, we select 10% We decompose high-level phase definitions into a few basic concepts to form the text prompts. Cross-modal Retrieval. Cross-modal retrieval includes text-based video retrieval and video-based text retrieval. Here, we conduct the cross-modal retrieval at three hierarchical levels. We collect 537 clip-narration (clip-level) video-text pairs, 746 phase-keystep (phase-level) video-text pairs, and 86 video-abstract (video-level) video-text pairs from hold-out testing videos of SVL [76] . There are more phase-keystep than clip-narration video-text pairs because some testing videos do not have cleaned narrations and we filter them out. For video embedding generation, we sample multiple frames fro m the video and average pool their image embeddings. We temporally sample 10 frames for clip-/phase-/video-level videos. We conduct the zero-shot evaluation for the cross-modal retrieval task.As mentioned before, the current surgical vision-language pretraining dataset lacks the scale necessary to pretrain a robust vision-language model from scratch, therefore a good choice of architecture and initialization is important. In this section, we conduct the experiment and study the effect of different model architectures and initializations, justifying our choice of using ResNet50 architecture with ImageNet initialization as our starting point before the video-language pretraining.• ResNet50. For ImageNet initialization, we use public IMAGENET1K_V1 weights from torchvision. Random initialization means that we random initialize the visual encoder before the hierarchical vision-language pretraining. These models' textual encoders are initialized from BioClinicalBert [24] . For CLIP initialization, we initialize the visual and textual encoder from OpenAI's weight [52] . 8 : The experiments show that the initialization largely influences the performance of surgical video-language pretraining.• ViT-B/16. For ImageNet initialization, we use weights from the official Google JAX implementation, which is pretrained on ImageNet21k [56] and then finetune on ImageNet1k [59] . We use the public pretrained weights from [10] for Dino initialization.In our work, we choose ResNet50 over Vision Transformer (ViT-B/16) due to its superior performance and lower parameter amounts in the context of video-language pretraining for surgical data.Our experiments demonstrated that ResNet50, particularly when initialized with CLIP weights, outperformed ViT-B/16 across various tasks, including zero-shot and linear-probing evaluations on Cholec80 and Autolaparo datasets. Despite the advanced capabilities of vision transformers, their performance heavily depends on large-scale pretraining datasets, which might not always be available or optimal for specialized domains like surgical scenes. Conversely, convolutional neural networks like ResNet50 have shown robust generalization abilities, even when pretrained on natural images, making them more suitable for our specific application. Additionally, the initialization sensitivity observed in ViT-B/16 further justified our preference for ResNet50, ensuring a more reliable and effective starting point for our hierarchical vision-language pretraining.After achieving the cost matrix C and Ĉ, we perform dynamic time warping (DTW) [60] to find the minimum cost path to align the frames of video segment V = {v 1 , ...v T } to the text sequence B = {b 1 , ...b N } and reversed text sequence {b N , ...b 1 }, respectively, as shown in Algorithm. 1. We follow [71] to process the DTW function into differentiable, enabling the gradient back-propagation.The differentiable loss function is the same as [21] .A significant advantage of using DTW is that it does not require additional temporal modules, such as recurrent neural networks or attention mechanisms, to model temporal relationships. This simplification allows us to focus on learning better representations by directly aligning video frames and text sequences based on their semantics.Algorithm 1 DTW to align sequences using cost matrix 1: procedure ALIGNSEQUENCES(C, V, B)2:Let T be the length of sequence V and N be the length of sequence B.Set i to T and j to N .Initialize distance to 0.while i > 0 and j > 0 do return distance. 17: end procedure6: distance = distance + C[i][j] 7: if i > 1 and j > 1 and C[i -1][j -1] ≤ C[i -1][j] and C[i -1][j -1] ≤ C[i][j -1] then 8: i ← i -1 9: j ← j -1 10: else if i > 1 and C[i -1][j] ≤ C[i][j -1] then 11: i ← i -1 12:Modality gap is a geometric phenomenon observed in the embedding space of multi-modal models [39] . This gap illustrates that pretrained multi-modal (vision-language) models create a joint embedding space where different modalities, such as images and text, are kept at a significant distance from each other. During contrastive optimization, this separation created at initialization is maintained to the extent that irrelevant image embeddings can be closer to each other than to their corresponding relevant text embeddings. This spatial disparity in the embedding space hinders the model's ability to effectively align and understand the relationships between visual and textual data, leading to suboptimal performance in tasks requiring integrated multi-modal comprehension. The existence of the modality gap is particularly detrimental when adapting pretrained vision-language models to cross-modal generation tasks, such as image captioning. As highlighted by several studies [35, 20] , narrowing modality gap correlates with improved performance in cross-modal tasks.As shown in Fig. 4 , we visualize the embeddings of videos and their corresponding text descriptions at three hierarchical levels: clip-narration, phase-keystep, and video-abstract. Our proposed model demonstrates a significant reduction in the modality gap compared to the SurgVLP model. This alignment across different hierarchical levels ensures a more comprehensive and cohesive understanding of the multi-modal data, leading to superior performance in tasks like image captioning and other vision-language applications. We demonstrate the zero-shot surgical phase recognition to reflect the surgical scene understanding ability of our pretrained model. Our model can identify surgical phases of different types of surgical procedures without any finetuning. Both success and failure examples are shown. Surgical Term Understanding. In Fig. 5 , we show that the pretrained model excels at identifying the "washing" phase in surgical procedures, demonstrating its capability to accurately recognize high-level surgical activities. This proficiency enhances surgical assistance systems, improving real-time analysis and decision-making in operating rooms.Instrument Identification. In Fig. 6 , we demonstrate how the visual embedding is significantly influenced by the presence of surgical instruments. Specifically, in the first row, the semantic meaning of the image changes from "calot triangle dissection" to "clip and cut" due to the appearance of a hook, even though the other anatomical features remain similar.As the pretraining process at clip-level requires additional supervision signals, i.e., visual selfsupervision, the memory and computation overhead increase compared to the vanilla HecVL pretraining. Also, during the phase-and video-level pretraining, the process of dynamic time warping can be time-consuming because it is based on dynamic programming, slowing down the pretraining iteration when handling longer-term surgical videos. Additionally, the knowledge augmentation on keystep and abstract texts need to be modified to fit the other video-language pretraining datasets [4, 79] as their hierarchical paired texts are annotated manually. Instead, our knowledge augmentation is more suitable for videos in the wild from online platforms. To address these limitations, future work could focus on developing a general textual augmentation strategy using the LLM's internal knowledge, adapting to the instructional videos that miss keystep and abstract text descriptions. Furthermore, techniques for decentralizing the video-language pretraining could be explored, aiming to pretrain with multi-centric vision-language samples while preserving privacy using the federated learning strategy. This could address the scaling problem in surgical vision-language pretraining and improve the generalizationability across the centers.Build Surgical Knowledge Base. In Fig. 7 , we show that the internal surgical knowledge of large language models can be elicited to build the external knowledge base.Build Surgical Knowledge Base. In Fig. 8 , Fig. 9 and Fig. 10 , we show that the knowledge of large language model can be used to enrich the semantics of the hierarchical texts, i.e., narrations, keysteps, and abstracts. Notably, it can explain high-level keystep words into descriptive sentences, enhancing textual diversity and preventing overfitting. NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.https://github.com/barrust/pyspellchecker/ 


Decision: The submission introduces a novel framework for surgical video-language pretraining, which enhances language supervision by integrating surgical concepts refined through large language models (LLMs). The paper presents extensive experiments across multiple surgical procedures and provides a thorough evaluation that underscores the framework's effectiveness. A key novelty is the use of LLMs to augment surgical video text descriptions, addressing the common issue of limited text narration, particularly for key frames. This approach significantly improves the quality and diversity of the generated surgical text.

The reviewers unanimously recommend accepting the paper, particularly after the authors' rebuttal addressed initial concerns. Therefore, I recommend acceptance of this paper. 

The authors are encouraged to incorporate the discussion points and clarifications raised during the review process into the final version. Additionally, it is essential that the authors make the training dataset available by sharing links to the videos used in the study, along with the necessary preprocessing code to generate video-text pairs for training. The source of the dataset and the associated license should be clearly stated to ensure transparency and reproducibility.

Reviews: review: paper_summary: paper_summary: The paper presents a new framework called PeskaVLP for surgical video-language pretraining. A hierarchical knowledge augmentation approach is used for enriching text information. The pretraining is implemented with the proposed language supervision and visual self-supervision. A new training objective is proposed for surgical procedural understanding. Extensive experiments are conducted to demonstrate the effectiveness on the surgical phase recognition task and cross-modal retrieval task on multiple downstream dataset.
strength_weakness: strength_weakness: 1. The explanation of method details is not clear enough, and there is a lack of discussion on some experimental results
2. The proposed method is based on certain assumptions but lacks a comprehensive consideration of applicability.
questions: questions: 1. What types of surgeries are included in the SVL dataset used in the paper? Is it suitable for the pretraining task? Could it affect the results on the downstream dataset?

2. In Section 3.2, where hierarchical knowledge is augmented by GPT, the authors need to discuss the ability of LLMs to generate accurate textual information to describe the surgical steps in the domain-specific surgical context, especially considering the fine-grained image-text alignment in the clip-level (only 4 frames).

3. In Section 3.2, the authors calculate textual similarity between the pseudo step generated by the LLM and the narration. How is this similarity calculated? Is there an ablation study on the effectiveness of the three behavior in knowledge augmentation?

4. In Section 3.3.1, the authors implement visual self-supervision based on augmentation. Which specific augmentations were used? Do the augmentations affect the corresponding text's semantic information? For example, using flipping could impact descriptions related to left/right information in surgical operation.

5. In Section 3.3.2, procedural information based on surgical phases is used. However, in surgical datasets, such as the cholec80 and AutoLaparo mentioned in the paper, the surgical process does not always follow a linear order defined by Phase 1-N and may include repeated phases. The authors should discuss the applicability of the method design in such situations.

6. In Table 3, for the experimental results on cholec80, Moco (third row) provides the best results, but this is not highlighted in bold in the table. This needs to be corrected and the corresponding discussion should be provided. The same issue appears with the results using Moco (second row) on the StrasBypass70 dataset.
limitations: limitations: Authors discussed it briefly in the appendix.
paper_summary: paper_summary: This paper proposes a Procedure-Encoded Surgical Knowledge-Augmented Video-Language Pretraining (PeskaVLP) method that enriches language supervision with LLM-refined surgical concepts. It further constructs hard negative samples by reversing the text orders at the phase and video levels and employs a Dynamic Time Warping (DTW) based loss to align multimodal procedures. Extensive experiments on multiple surgical procedures and comprehensive evaluations demonstrate the effectiveness of this framework.
strength_weakness: strength_weakness: - By reading the enriched dataset by LLM in Appendix H, I am concerning that the variation and diversity of narration will be removed by the augmentation. Will that cause any problems?
- In my opinion, using LLM to refine the text description of surgical videos is the most important contribution of this paper. It would be interesting to see if other components are also effective enough without the knowledge augmentation.
questions: questions: - Beyond the current ablation study on PeskaVLP components, would applying the hierarchical knowledge-augmented text data in HecVL improve its performance and if this could yield results competitive with PeskaVLP. This would provide powerful support to verify the extent to which the other components in PeskaVLP contribute to performance, apart from the augmented texts.
- Although LLM can enhance surgical text quality, is there a concern that the text may become overly standardized? Given that surgeons' narratives in the operating room tend to be more oral, concise, and sometimes include jargon, will there be a performance degradation in real-world, real-time applications where LLM augmentation is impractical?
- In Appendix E, Figure 4, it would also be interesting if the authors could visualize the embeddings of HecVL, since it performs better than SurgVLP.
- In Table 3, on Cholec80, Moco pre-trained on Cholec80 (V) has better performance but wasn't in bold, do I misinterpret something?
limitations: limitations: The authors adequately addressed the limitations. Since the proposed method is tailored for surgical data and applications, it is strongly suggested that the authors include a discussion on the potential negative societal impact of their work.
paper_summary: paper_summary: The paper presents a novel approach for enhancing surgical video analysis by incorporating procedural awareness. The authors propose a system that integrates knowledge of surgical procedures to improve the identification, segmentation, and annotation of surgical activities in video footage. This approach aims to address challenges such as the variability of surgical techniques and the complexity of visual data in operating rooms. The contributions of the paper include the development of a procedural model that can be aligned with video data, the creation of annotated datasets for training and evaluation, and the demonstration of improved performance over traditional video analysis methods.
strength_weakness: strength_weakness: Dataset Limitations: The annotated datasets used for training and evaluation are crucial for the model's success. Expanding the diversity and volume of these datasets would enhance the generalizability of the findings.
questions: questions: Generalizability: How does the system perform across different types of surgeries (like ophthalmic surgery)? Have you tested its effectiveness in various surgical domains beyond the initial scope?
limitations: limitations: The paper does not adequately address potential limitations and negative societal impacts.
paper_summary: paper_summary: The paper addresses challenges in surgical video-language pretraining (VLP) due to the knowledge domain gap and scarcity of multi-modal data. It proposes a hierarchical knowledge augmentation approach and the Procedure-Encoded Surgical Knowledge-Augmented Video-Language Pretraining (PeskaVLP) framework. This approach enhances data efficacy and tackles spatial-temporal challenges by combining language supervision with visual self-supervision. Extensive experiments demonstrate significant improvements in zero-shot transferring performance and the generalist visual representation for surgical scene understanding.
strength_weakness: strength_weakness: Firstly, the dataset size is relatively small, with 1,007 videos for phase-level pretraining and 920 for video-level pretraining, which may limit the generalizability of the findings (as mentioned in the supplementary material). I know the difficulty in collecting medical data, but we must be sure that the presented approach can be generalized to different domains and hospitals. Furthermore, I doubt the methodology's potential to process "noisy" videos.    
Expanding the dataset and including more diverse surgical procedures would improve robustness. 

Secondly, while the paper mentions ASR errors in transcriptions, it does not provide a detailed methodology for handling them. Providing specific techniques for improving transcription accuracy would strengthen the study. 

Additionally, the practical implementation of the PeskaVLP framework in real-world surgical contexts is not thoroughly discussed. Detailing strategies for integration into clinical workflows and addressing potential technological barriers would be beneficial.
questions: questions: 1. How do you plan to address the limited sample size and diversity in future studies to improve the generalizability of your findings? Consider expanding the dataset to include a more extensive and more diverse sample of surgical procedures to enhance robustness and applicability.

2. What specific methods did you use to handle ASR errors in transcriptions? How did these errors impact your analysis?

3. How do you manage the computational overhead associated with the hierarchical pretraining and dynamic time-warping processes?
limitations: limitations: The authors have acknowledged the limitations related to dataset size and ASR errors but could elaborate on strategies to mitigate these issues. Specifically, they should discuss plans for expanding the dataset, incorporating more diverse samples, and improving transcription accuracy.

The positive societal impacts, such as enhancing surgical training and assistance, are well-discussed. However, the authors should address potential negative impacts, such as data privacy and ethical concerns. A detailed discussion on data security measures, user consent protocols, and ethical safeguards is needed.

Comments: title: Rebuttal by Authors for Ethics Review
Rebuttal Follow Up
comment: Thank you for the detailed responses. The rebuttal has addressed most of my concerns, and I would like to raise my score accordingly.
Dear Reviewer, 

Thank you for the recommendation. In the revised manuscript, we will include the GitHub link to the code and dataset repository for this paper. We will make our training dataset available by sharing links to the videos, along with the necessary preprocessing code to generate video-text pairs for training.
I appreciate the authors' response to my review and their efforts to address my concerns. After carefully reviewing the feedback from the other reviewers, I am inclined to maintain my original score, thanks.
**[Q1. Transcription Errors and Solution. ]** In the following Table 1, we quantitatively demonstrate the impact of LLM-based augmentation and transcription processing strategies on transcription accuracy for the zero-shot phase recognition on Cholec80 dataset. We train the SurgVLP model using Clip-level text Augmentation which uses ASR transcripts enhanced by LLM-based augmentation and before mentioned transcription processing strategies. The results are shown in the first two rows on the table. The SurgVLP model when trained with the Clip-level text Augmentation, including transcription processing strategy and LLM-based augmentation, brings – 2.2% improvement in the F1 score.   

<Table 1: Quantitative assessment for the improvements brought by LLM-based augmentation and transcription processing strategies. We report the zero-shot performance of Cholec80 testing set. >

| Setup                                         | Accuracy | F1-Score |
|-----------------------------------------------|----------|----------|
| SurgVLP                                       | 34.7     | 24.4     |
| SurgVLP + Strategy + Clip-level Augmentation  | 36.1     | 26.8     |
| HecVL                                         | 41.7     | 26.3     |
| HecVL + Strategy                              | 43.0     | 29.2     |
| HecVL + Strategy + Phase-level Augmentation   | 44.0     | 31.8     |
| HecVL + Strategy + Video-level Augmentation   | 43.7     | 30.6     |
| PeskaVLP                                      | 45.1     | 34.2     |


Similarly, we also train the HecVL model using processed transcripts without LLM-based augmentation, resulting in improved performance compared to the baseline HecVL. Notably, the improvements from transcription processing can be further enhanced when combined with LLM-based augmentation. 

**[Q2. Computational Overhead]** Thank you for your follow-up comment. In this work, we train directly on the surgical videos without converting them into frames. As a result, a key bottleneck lies in the online video decoding process, where videos must be decoded and processed in real-time to provide training batches. Currently, we are using ffmpeg for the online video decoding process followed by Pytorch to process the decoded frames. In future, we plan to implement a more efficient multi-threaded video decoding pipeline where multiple videos can be decoded in parallel on GPUs using advanced libraries such as NVIDIA Data Loading Library (DALI) [1] or Decord [2]. We expect to minimize the idle GPU time.  

**[Potential Negative Impacts]** Thank you for a follow-up comment. To address patient consent and data privacy in the collection of intra-operative surgical videos, we will ensure that all recordings are fully anonymized by removing any identifiable patient information and out-of-body frames using open-source tools [3]. Specifically, since our focus is on laparoscopic videos, anonymization will involve removing any captions or metadata that could identify the patient, as well as ensuring that any external out-of-body views or patient-identifiable features captured in the video are thoroughly anonymized.  

_[1] https://developer.nvidia.com/dali_

_[2] https://github.com/dmlc/decord_ 

_[3] Lavanchy, Joël L., et al. "Preserving privacy in surgical video analysis using a deep learning classifier to identify out-of-body scenes in endoscopic videos." Scientific reports 13.1 (2023): 9235._
Dear Authors,
Thank you for your detailed rebuttal and for addressing the concerns raised during the review process. Your efforts to expand the dataset, address ASR errors, manage computational overhead, and consider potential negative impacts are appreciated. Here are some follow-up questions and concerns:

Transcription Errors and Solution.
How do you plan to quantitatively evaluate the improvements in transcription accuracy using the proposed LLM-based augmentation and Whisper ASR system? Can you provide detailed comparative results before and after implementing these methods?

Computational Overhead.
Could you elaborate on the specific asynchronous processing pipelines you plan to implement to reduce computational bottlenecks? What are the expected improvements in terms of training time and resource utilization?

Potential Negative Impacts.
While you mentioned minimizing societal impacts by using open educational platforms, how will you handle patient consent and data privacy for intra-operative video recordings? 


Your commitment to improving the manuscript is commendable, and I look forward to seeing these enhancements reflected in your revised submission.