Recognize Any Surgical Object: Unleashing the Power of Weakly-Supervised Data

Authors: Jiajie Li, Brian R Quaranto, Chenhui Xu, Ishan Mishra, Ruiyang Qin, Dancheng Liu, Peter C W Kim, Jinjun Xiong
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: Image Recognition, Vision-Language Pretraining, Image Tagging, Medical Imaging, Surgery

Abstract:
We present RASO, a foundation model designed to Recognize Any Surgical Object, offering robust open-set recognition capabilities across a broad range of surgical procedures and object classes, in both surgical images and videos. RASO leverages a novel weakly-supervised learning framework that generates tag-image-text pairs automatically from large-scale unannotated surgical lecture videos, significantly reducing the need for manual annotations. Our scalable data generation pipeline gathers 2,200 surgical procedures and produces 3.6 million tag annotations across 2,066 unique surgical tags. Our experiments show that RASO achieves improvements of 2.9 mAP, 4.5 mAP, 10.6 mAP, and 7.2 mAP on four standard surgical benchmarks respectively in zero-shot settings, and surpasses state-of-the-art models in supervised surgical action recognition tasks. We will open-source our code, model, and dataset to facilitate further research.

Decision: accept (spotlight)
Reviewer scores: 8, 8, 8, 6

Reviews and discussion:
<thread>
creation date:
1730344755484

thread_type:
official_review

<thread_object>
creator: reviewer_9rar

<content>
summary:
The paper’s contributions include:

(1)RASO Model: A novel open-set recognition model tailored for recognizing diverse surgical objects in images and videos, incorporating temporal-attention fusion for video context. 

(2)Weakly-supervised Learning Framework: An efficient framework leveraging unannotated data, significantly reducing annotation reliance and supporting domains with limited annotated data.

(3)Data Generation Pipeline: A scalable, weakly-supervised pipeline that generates extensive tag-image-text data from over 2,200 surgical procedures, producing comprehensive surgical annotations.

(4)Experimental Results: RASO demonstrates substantial mAP improvements in zero-shot and supervised settings, outperforming current state-of-the-art models in surgical action recognition.

strengths:
(1)The development of RASO introduces an attention-based temporal-fusion mechanism specifically for surgical videos, where temporal information is essential but often neglected in frame-based models. Additionally, the weakly-supervised learning framework and data generation pipeline are highly creative solutions to the prevalent problem of data scarcity, enabling the model to learn from unannotated surgical videos at scale.

(2)Extensive testing on four benchmark datasets, both in zero-shot and supervised settings, provides strong evidence of the model's effectiveness. 

(3)The article is well-structured, the motivation is clear, and the technical explanations are easy to understand

weaknesses:
(1)The model heavily depends on a weakly-supervised dataset generated from surgical lecture videos, which, despite its scale, may introduce considerable noise due to the lack of domain-specific annotation quality control. If we could analyze the approximate distribution of entity keywords(tools or actions, etc.) in the captions, it would allow for a preliminary observation.

(2)While the temporal-attention fusion layer is a key innovation intended to enhance video-based recognition, the paper lacks a rigorous justification for why this layer outperforms simpler alternatives, such as average pooling or recurrent networks. 

(3This architecture includes a label decoder and a text decoder, where the text decoder aligns the visual content with the transcript during training. However, is there potential redundancy between the label decoder and text decoder modules? Would an ablation study be necessary to verify this?)

questions:
I'd like to ask how the frames were extracted. Were they keyframes, sampled at regular intervals, or something else?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730454623363

thread_type:
official_review

<thread_object>
creator: reviewer_a4t2

<content>
summary:
The Paper presents a data generation pipeline for weakly supervised training framework and a foundation model architecture RASO. RASO predicts an open set of tags for single frames while taking video context into account. The data generation pipeline extracts relevant tags from voiceovers of surgical lecture videos. RASO is evaluated in a zero-shot and a supervised setting.

strengths:
- interesting approach which proposes novel architecture, pipeline and data
- Paper is well structured, easy to read and easy to follow the argumentation
- profound evaluation, RASO is compared against 4 SOTA architectures for the zero-shot setting on 4 datasets and 3 SOTA models for the supervised setting on the CholecT50
- Paper presents an ablation study for the pretraining and fine-tuning stages, as well as two different aggregation methods for the video context understanding 
- code, models and dataset will be open-sourced

weaknesses:
- missing details regarding the evaluation, e.g. why was the video aggregation not used in the first two experiments? which data was used for which configuration? ... (see questions)
- no link to code, data, and models in the paper

questions:
- Line 33: Two Datasets are called EndoVis17 and EndoVis18. In both years EndoVis consisted of 4 challenges releasing datasets. I propose to use the challenge names robotic instrument segmentation (RIS) & robotic scene segmentation (RSS)
- Figure 1 is not referenced in the text.
- Every paragraph cites methods or models when firstly used, even if previously already used
- Line 184ff: N and T both represent a number of features, the final embedding size is NxTxD, its unclear what the difference between N&T is and why there are two numbers of frames.
- Line 197 describes the text decoder to take visual features and predicted tags as input while figure 2 shows the encoder to take visual features and encoded tags but no prediction. Fig2 should be improved such that the prediction is taken into account by the text decoder.
- Line 235: The Paragraph heading “Discussion” doesn't really fit, as no results are discussed but rather advantages of the pipeline are listed
- Line 260: How are the generated tags split into the pretraining and fin-tuning set?
- Table1: EndoVis18 (RSS) is listed with 997 images. The dataset consists of 19 snippets with 300 frames each, why/how was it down sampled?
- Table2: why aren't the values of P & R highlighted? Esp. for P the text references the models superiority, this should be visualized in the table.
- Why is the Fbeta with beta=0.5 instead of the more common F1 score used?
- Table3: all values in the table should be given with the same amount of decimals
- Line 417ff: Figure2 introduces the RASO architecture to include a temporal-attention fusion, but the video recognition paragraph compares the video aggregation against a frame-wise approach? How does the frame-wise method work? Which features are aggregated?
- From the video recognition paragraph an Table3c it seems that the temporal-attention fusion was not used in the experiments of Table 1 and 2, as both configuration in Table3c differ from the earlier presented results (Table 1,2,3a&b). Why was the fusion not used as presented in Fig2?
- Table 3a and 3b are similar to the bottom two rows of Table 1 and 2 respectively. No new information is presented. I recommend to reference table 1 & 2 and use the space of Tables 3a&b to further explain the image-wise aggregation or the experimental setup to clarify why  temporal-attention fusion was not used in all experiments.
- Both configurations in Table 3c outperform the results of Table 3a&b. How does the ablation pretraining and fine-tuning change when applying any of the video aggregation methods?
- The paper cites many non-peer-reviewed publication on arXiv. Check if peer-reviewed versions are available (esp lines 539 & 545)
- Some references (lines 608, 623 & 630) are missing details on the journal / publisher
</content>
</thread_object>

</thread>
<thread>
creation date:
1730676424111

thread_type:
official_review

<thread_object>
creator: reviewer_b5e4

<content>
summary:
The paper introduces a recognize-any-object (RAM) foundation model for surgery. The work employs a weakly supervised approach by generating a weakly annotated dataset from web-surgical-like data sources. Data from surgical lectures is processed to generate captions that are further used to generate different levels of annotations, including actions and entities. From the methodological perspective, the model follows RAM-based architectures with an additional temporal attention fusion layer integrated to work on video information.

strengths:
The paper incorporates weak annotations from multiple video sources and investigates the zero-shot generalizability of adapted recognize-anything models in the surgical environment. Generating annotations from surgical videos is a common limitation in the medical domain as it requires expert knowledge. Hence, an approach that allows leveraging this data is interesting from the surgical ML perspective.

weaknesses:
The work presents a modification to a previous RAM model and a pipeline for data generation. Even though leveraging web-based data from unstructured medical sources can be relevant for medical machine-learning purposes, I am unsure about the significance of the contribution from the general machine-learning perspective.

questions:
The work introduces a methodology to generate weak annotations, which can also make it possible to fine-tune existing foundation models with extensive surgical data. In this regard, what are the additional benefits of the introduced model compared with standard foundation models fine-tuned on the weakly annotated data?

It is interesting to see that in Table 4, average pooling performs better than temporal attention in the instrument recognition tasks. What would be the main insights regarding this result?

Figure 3 is written WshiperX, probably instead of WhisperX.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730715418546

thread_type:
official_review

<thread_object>
creator: reviewer_znlc

<content>
summary:
The authors introduced a foundation model aimed at recognizing surgical objects in closed and open set environments. It also adapts to learn from images and videos. They introduce a weakly supervised learning framework to generate tag-image-text pairs from surgical lecture videos. This allowed them to create an interesting dataset that could be helpful for researchers. Experimental results demonstrate that the proposed method outperforms other SOTA methods in zero-shot scenarios and in supervised surgical action recognition tasks.

strengths:
- The paper is well presented and easy to follow.
- Although the technical contribution is limited, the efforts and contributions in the paper are great and are likely to help the research community in the field of surgical tool detection, segmentation and analysis. 
- The evaluation is good and ablation studies reflect how the method could work in different settings.

weaknesses:
1- The main weakness of the paper from a technical perspective is the limited technical contribution. The methodology is heavily based on RAM (Zhang 2024) with an additional temporal attention layer.

2- It is not clear in Figure 2 how the embeddings from I and Vs are processed in the Tag Decoder. How does the network balance the importance of information coming from images or videos?

3- It was not clear why pre-training and fine-tuning were performed for a small number of epochs. Does the loss saturate when fine-tuning for 4 epochs only? 

4- VLM automated annotation: it is not clear the effect of the ability of GPT-4o in generating accurate annotation. An assessment of this step is needed.

questions:
To relate to the 4 points in the weakness section

1- Explain or confirm the main technical differences with RAM

2- Provide more details on how the network balances the importance of image and video embeddings

3- Answer point 3 in the weaknesses

4- Elaborate on 4 in the weaknesses
</content>
</thread_object>

</thread>
<thread>
creation date:
1732042995676

thread_type:
official_comment

<thread_object>
creator: authors

<content>
comment:
We sincerely thank all reviewers for their thoughtful feedback and constructive comments. Your insights have been invaluable in improving both the clarity and quality of our work. **We have submitted a revised version of the paper**, where all updates and modifications are highlighted in the **blue font** for ease of review. Below, we provide a summary of the key improvements and responses made in the revised paper. Detailed individual responses to each reviewer’s comments can be found below.

 **Major Improvements**

1. Figure Updates:
  - Updated Figure 2 to clarify how image embeddings are aggregated for video input, and provide a better illustration of the text decoder for transcript reconstruction joint training.
  - Added a word cloud in Figure 3 to provide an overview of tag frequency, offering a preliminary observation of the dataset’s entity keyword distribution.
2. Revised Text and Tables:
 - Renamed EndoVis18 to RSS in the experiment section and updated related descriptions.
 - Improved Table 4 by cleaning the redundant content and adding an ablation study for supervised video recognition.
 - Clarified the use of temporal-attention fusion for video recognition versus frame-wise aggregation for image-based video recognition baseline.

Thank you again for your valuable time and effort. We hope the revisions and clarifications address your concerns.
</content>
</thread_object>

</thread>
<thread>
creation date:
1766996270294

thread_type:
public_comment

<thread_object>
creator: ~ka_young_kim1

<content>
title:
Clarification on CholecT50

comment:
Thank you very much for your interesting work.

I especially enjoyed reading the experiments on CholecT50, as many existing surgical VLM studies have mainly focused on phase recognition tasks.

As far as I understand, CholecT50 provides several different dataset splits (e.g., as described in https://arxiv.org/pdf/2204.05235). I was wondering if you could kindly clarify which split was used in your experiments.

I was also curious about how frames without any annotated triplets were handled in your experimental setup.

Finally, I would be grateful if you could share a bit more insight into how the threshold was chosen for the zero-shot evaluation on CholecT50. For example, was it tuned on a validation set, fixed empirically, or selected based on some principled criterion?
</content>
</thread_object>

</thread>


Metareview:
metareview:
The submission presents a foundation model aimed at segmenting surgical objects in closed and open set environments.  The reviewers were unanimous that the submission should be accepted.  The significance is primarily in providing a seemingly effective solution to automated surgical video annotation with minimal labeling, leveraging an unannotated dataset.  Effectiveness is shown in zero-shot settings, as well as in supervised surgical action recognition tasks.  On the balance, the submission provides an interesting solution to a medical application, which should find its audience in the ICLR and surgical video analysis communities.