Title: Text Promptable Surgical Instrument Segmentation with Vision-Language Models
Venue: NeurIPS 2023
Accepted: yes
Keywords: Surgical Instrument Segmentation

Vision Language Models

Text Promptable Segmentation
Abstract: In this paper, we propose a novel text promptable surgical instrument segmentation approach to overcome challenges associated with diversity and differentiation of surgical instruments in minimally invasive surgeries. We redefine the task as text promptable, thereby enabling a more nuanced comprehension of surgical instruments and adaptability to new instrument types. Inspired by recent advancements in vision-language models, we leverage pretrained image and text encoders as our model backbone and design a text promptable mask decoder consisting of attention- and convolution-based prompting schemes for surgical instrument segmentation prediction. Our model leverages multiple text prompts for each surgical instrument through a new mixture of prompts mechanism, resulting in enhanced segmentation performance. Additionally, we introduce a hard instrument area reinforcement module to improve image feature comprehension and segmentation precision. Extensive experiments on several surgical instrument segmentation datasets demonstrate our model's superior performance and promising generalization capability. To our knowledge, this is the first implementation of a promptable approach to surgical instrument segmentation, offering significant potential for practical application in the field of robotic-assisted surgery. Code is available at https://github.com/franciszzj/TP-SIS.

Text Promptable Surgical Instrument Segmentation with Vision-Language Models

Abstract

In this paper, we propose a novel text promptable surgical instrument segmentation approach to overcome challenges associated with diversity and differentiation of surgical instruments in minimally invasive surgeries. We redefine the task as text promptable, thereby enabling a more nuanced comprehension of surgical instruments and adaptability to new instrument types. Inspired by recent advancements in vision-language models, we leverage pretrained image and text encoders as our model backbone and design a text promptable mask decoder consisting of attention- and convolution-based prompting schemes for surgical instrument segmentation prediction. Our model leverages multiple text prompts for each surgical instrument through a new mixture of prompts mechanism, resulting in enhanced segmentation performance. Additionally, we introduce a hard instrument area reinforcement module to improve image feature comprehension and segmentation precision. Extensive experiments on several surgical instrument segmentation datasets demonstrate our model's superior performance and promising generalization capability. To our knowledge, this is the first implementation of a promptable approach to surgical instrument segmentation, offering significant potential for practical application in the field of robotic-assisted surgery. Code is available at https://github.com/franciszzj/TP-SIS .

1 Introduction

Minimally invasive surgeries (MIS) have gained widespread attention in various surgical disciplines [16, 48] due to their benefits over traditional open surgery, such as reduced patient discomfort and faster recovery times. Nonetheless, the restricted field of view and indirect vision through endoscopic cameras for existing MIS procedures presents considerable obstacles, making the development of robot-assisted MIS increasingly crucial. Current robot-assisted surgery has to operate under direct control of the surgeon. Enhancing the automatic comprehension of the surgical process through precise instrument segmentation is seen as an essential building block to introduce automation and facilitate robot-surgeon interaction. Despite this recognised importance, existing automatic surgical instrument segmentation faces significant challenges.

First, with fast-paced advances in MIS, there is a surge in the variety of surgical instruments from different vendors. This is however compounded with the lack of a comprehensive and large-scale dataset dedicated to the learning of surgical instrument segmentation. Current methods [43, 20, 12, 55, 5] do not adequately adapt to the ever-changing set of surgical instruments, necessitating re-labelling and re-training of models with the introduction of each new instrument. This has notably hampered the practical application of surgical instrument segmentation within the MIS field.

Second, the segmentation methods face difficulty in distinguishing between different categories of instruments [5], which often have similar appearances. The subtle visual difference across instruments and the often difficult imaging conditions in the confines of the surgical cavity make it harder to differentiate between instrument categories, causing poor segmentation performance.

Overcoming these challenges requires learning-based approaches that are more flexible and robust than the current state of the art. Recent progress in pre-trained vision-language models [33, 36, 47] offer novel opportunities for our research questions as demonstrated by results obtained in diverse downstream computer vision tasks [10, 35, 44, 30, 29, 46, 9]. On the one hand, these vision language models, trained on abundant data, offer robust features which can help compensate for the scarcity of surgical instrument segmentation data. On the other hand, the capability of these models to output aligned image and text features allows for text to supplement information for surgical instrument segmentation, thus simplifying the differentiation between various instrument types.

To address the first challenge, we observe that existing supervised instrument segmentation models [12, 55, 5, 3] are trained on predefined categories, thereby limiting their semantic understanding of instruments outside of the narrow training set. Drawing inspiration from recent advancements in text promptable image segmentation [29, 46] with vision-language models [36], we redefine the task as a text promptable surgical instrument segmentation, thereby enhancing generalization and adaptability to an ever-growing array of new surgical instruments. Figure 1 contrasts the predefined category method with our text-promptable approach. Our model adapts the image and text encoders from a pretrained vision-launguage model, CLIP [36], to extract features from both the surgical image and textural prompt; text promptable mask decoder consisting of attention-based and convolution-based prompting schemes is specifically designed to prompt the surgical instrument in the image from coarse to fine.

Varying text descriptions for each surgical instrument can yield distinct segmentation results. To tackle it, we propose a mixture of prompts mechanism , inspired by mixture of experts (MoE) [31, 41, 37], to leverage diverse prompts. This mechanism inputs multiple source prompts to the model and fuses the resulting predictions via weights generated from a visual-textural gating network. The final segmentation result hence benefits from various prompt information.

To address the second challenge, we aim to enhance the model's capacity to obtain improved image features, thereby reinforcing its ability to segment various surgical instrument categories and delineate precise edge details. We propose a hard instrument area reinforcement module intertwined with the popular image reconstruction approach, masked autoencoder (MAE) [15]. This module performs hard area mining at the site of segmentation error; by masking out the hard area, the module injects an auxiliary image reconstruction to the segmentation model to enahce the understanding of difficult areas in the image features.

To the best of our knowledge, we introduce the first text promptable surgical instrument segmentation approach. Our extensive experiments demonstrate state-of-the-art performance on several surgical instrument segmentation datasets [1, 2, 39, 17]. Additionally, we evaluate the generalization capability of our model through cross-validation between the two datasets, showcasing promising results and emphasizing the significant application potential of our method.

2 Related Work

Surgical instrument segmentation. Most of the current works on surgical instrument segmentation are traditional vision-based models. For instance, TernausNet [43] trains an optimized U-Net [38] model for the segmentation of a restricted variety of surgical instruments. ISINet [12] proposes an instance-based segmentation method that utilizes a temporal consistency module to identify instrument candidates. MF-TapNet [20] utilizes motion flow information through an attention pyramid network and is trained independently for binary, part, and instrument type segmentation tasks. While most approaches rely on convolutional networks, a few recent works [55, 3] have investigated the use of transformer-based methods. For example, MATIS [3] utilizes pixel-wise attention, masked attention modules targeting for instrument areas, and video transformers for temporal information handling. Although the segmentation accuracy has been significantly improved by incorporating various architectures and modules, existing works are limited by their reliance on categorical understanding from the annotated training data; when new categories are introduced, they will have to be re-trained, which requires demanding labelling efforts and computational resources.

Conven&onal vision-based surgical instrument segmenta&on method

Monopolar Curved Scissors

Vision based

Model

Our text promptable surgical instrument segmenta&on method

Bipolar forceps have a slim, elongated tweezer-

like design with opposing 8ps, are silver-color...

Prograsp forceps possess curved scissor-like

handles, specialized grasping tips with jaws ...

Monopolar curved scissors showcase elongated

sha>s, curved cu?ng edges for precise ...

Vision

Language

Model

Image

Ground Truth

Vision-language models. Large pretrained models like CLIP [36] has emerged as the most promising approaches for achieving state of the art results on downstream computer vision tasks. CLIP [36] is a large pretrained vision-language model [45] that aligns text with image modalities in the embedding space to solve downstream tasks like zero-shot image classification [36, 27]. It has been successfully applied to various vision tasks, including video captioning [44], video-text retrieval [30, 10], image generation [35], relation learning [57] and text promptable segmentation [46, 29].

Text promptable segmentation. Text promptable segmentation, also conceptualized as referring expression segmentation in the literature, involves the utilization of natural language expressions as prompts for image segmentation, deviating from the traditional reliance solely on class label annotations for training images [18]. Early works use convolution neural networks (CNNs) and recurrent neural networks (RNNs) to extract visual and textual features which are fused via concatenation and used to perform segmentation [18, 25]. Attention mechanisms are also introduced for exploiting relations between visual and textual features [18, 25, 42, 52]. More recent approaches utilize transformers to perform visual and textual feature fusion either in the encoder [11, 49, 51, 22, 54] or decoder [46, 29, 7].

Large pretrained vision-language models have also been utilized for this task [46, 29, 53, 28, 24]. For instance, CRIS [46] uses the CLIP image and text encoders for multimodal knowledge transfer and a transformer decoder for multimodal fusion and segmentation. CLIPSeg [29] extends prompt-based segmentation beyond text prompts to include image prompts processed by the CLIP image encoder. A recent concurrent work, SAM [24], introduces a transformer-based model capable of generating object masks using either image or text prompts. However, existing promptable segmentation methods are not designed to solve highly specific tasks like surgical instrument segmentation.

3 Method

In this section, we begin by defining the task (Section 3.1), then introduce the image and text encoders inherited from a pretrained vision-language model (CLIP [36]), serving as the feature extractors of our method (Section 3.2). Next, we develop a text promptable mask decoder responsible for decoding the visual features into instrument segments with the help of textual features (Section 3.3). Finally, we elaborate on two key modules: the mixture of prompts module (Section 3.4), designed to enhance segmentation performance through multiple prompts; and the hard instrument area reinforcement module (Section 3.5), focused on improving instrument segmentation on difficult parts.

3.1 Task Definition

Given an image I ∈ R H × W × 3 consisting of surgical instruments for an endoscopic operation, as well as a text description T describing the name, appearance and function of a given instrument, the objective is to obtain the surgical instrument mask M ∈ { 0 , 1 } H × W from I , prompted by the instrument description T . In cases when multiple instruments need to be segmented, we input the model with

CLIP Text

Encoder

CLIP Image

Encoder

FPN

❄

Text Encoder

Image Encoder

Hard Area

Reinforcement

𝑇

!

:

monopolar curved scissors

Visual-Textual

Ga-ng

Network

Prompt Set {

𝑇

!

, 𝑇

"

, 𝑇

#

}

𝑇

"

:

the area represented by ...

𝑇

#

:

monopolar curved scissors

showcase elongated handles ...

×

×

×

+

+

Fusion

ℒ

!"#

ℒ

$"%

(1) Image and Text Encoders

(2) Text Promptable Mask Decoder (3) Mixture of Prompts

𝐹

$

𝐹

%

𝑆

𝐹

%

&

𝐹

%

'

𝐹

%

!"

𝑤

Hard Area Mining

Hard Area

A6en-based

Promp-ng

Conv-based

Promp-ng

(4) Hard Instrument Area Reinforcement

multiple text descriptions corresponding to different categories, and the goal becomes producing the segmentation result M ∈ { 0 , ..., C } H × W , where C represents the number of instrument categories.

3.2 Image and Text Encoders

To accomplish text promptable surgical instrument segmentation, we first encode the input endoscopic image I and instrument description T , obtaining the visual feature F I and textual feature F T , respectively. Importantly, to enforce the semantic congruity between F I and F T , we leverage the strength of the image and text encoders from the popular vision-language model, CLIP [36], pretrained on natural images.

Image encoder. Given the input image I , we utilize and fine tune the image encoder from CLIP [36], which is a vision transformer (ViT) [8], for feature extraction. The ViT-based image encoder comprises 12 layers. We use F l I ∈ R N × D to denote its l -th layer output, where l ranges from 1 to 12, N is the number of visual tokens, and D is the dimension of visual features in ViT. Supposing the patch size of the input image is p , then N = H p × W p .

Multi-scale feature augmentation. Extracting multi-scale features is important for semantic segmentation in order to segment objects in different scales. Whilst the image encoder in CLIP is not originally designed for multiple scales, we equip it with this ability. Specifically, simulating the multi-scale extraction from CNN-based architecture, we take the output features from the 4 -th, 8 -th, and 12 -th layers of the ViT, i.e. F 4 I , F 8 I , F 12 I , representing a wide coverage of level-of-detail for the image; and fuse these features by a feature pyramid network (FPN) [26]. Notice in order to feed these features into FPN, we need to first reshape them into the size of H p × W p × D , then we up-sample F 4 I and down-sample F 12 I both by a factor of 2. Finally, the output of the FPN is reshaped to size of N × D to obtain the multi-scale image feature F I ∈ R N × D .

Text encoder. Given the input description T , we utilize and keep frozen the text encoder from CLIP, which is a transformer encoder [45], for feature extraction. Within the text encoder, T is tokenized with the inclusion of a [ CLS ] token at its beginning, representing the global information of the description. Following the common practice in [46, 29], we use the corresponding feature from the [ CLS ] token as the global textual feature for the description. We denote it by F T ∈ R 1 × D , where D is dimension of the textual feature, equivalent to that of the visual feature.

3.3 Text Promptable Mask Decoder

After obtaining F I and F T , our goal is to decode from F I a score map S ∈ R H × W with the help of F T . Each pixel of S signifies the probability of this pixel belonging to the instrument class described in F T . By thresholding S , it can be transformed into the desired mask M . To achieve this goal, we present two prompting schemes: attention-based prompting and convolution-based prompting, specified below. For a detailed structure figure of the text promptable mask decoder, please refer to the supplementary material.

Attention-based prompting. Inspired by [45], we employ the attention mechanisms to attend the instrument area in F I using F T . First, we compute the self-attention (SA) [45] within F I to promote the foreground area in the endoscopic image. Layer normalization (LN) [4] is applied to F I beforehand while skip connection [14] is applied afterwards. We write out this process: F I S = SA ( LN ( F I )) + F I . Next, we compute the cross-attention (CA) [45] between F I S and F T to localize the instrument area in F I S according to F T . Similarly, this process is written as: F I C = CA ( LN ( F I S ) , F T ) + F I S . Subsequently, we apply the feed-forward network, which is basically a fully connected layer to further polishing F I C , F I F = FFN ( LN ( F I C )) + F I C . SA, CA and FFN are organized in one decoding block, we devise three such blocks sequentially to obtain the attention-based prompted feature F I A ∈ R N × D .

Convolution-based prompting. We design a novel convolution-based prompting scheme to further refine F I A by using F T to convolve it locally. To do this, F I A ∈ R N × D needs to be firstly reshaped to restore its spatial dimensions, i.e. ˜ F I A ∈ R H p × W p × D . Our idea is to transform F T into the convolution parameters, i.e. kernel weights and bias, and use these parameters to convolve ˜ F I A , such that the instrument-related positions will be promoted. The transformation is done via a fully connected (FC) layer: ˜ F T = FC ( F T ) . F T is a vector of dimension D , the FC layer reshapes it to the vector ˜ F T of dimension D × k × k +1 , with ' k × k ' representing the convolution kernel size and ' +1 ' is the extra dimension accounting for bias. This allows decomposition of ˜ F T into convolution weights w ∈ R 1 × D × k × k and bias b ∈ R 1 , which are subsequently used to convolve ˜ F I A ,

Sigmoid is applied to transform the final output into probabilities, hence producing the score map S .

The attention-based prompting computes the attention at each pixel by interacting it with pixels from the whole feature map, while in the convolution-based prompting, each pixel only interacts with its neighboring pixels in the k × k convlution kernel. We can see the former prompting as a global decoding process while the latter as a local decoding process to refine the former.

3.4 Mixture of Prompts

The provision of the prompt is crucial for the surgical instrument segmentation accuracy. Recognizing the impact of different prompts on a given model [56, 50], we propose to effectively leverage the strength of various prompts to facilitate the recognition of surgical instruments. Inspired by mixture of experts (MoE) [21], which divides a complex task into multiple sub-tasks and distributes them amongst multiple experts, we introduce a mixture of prompts (MoP) scheme to combine the segmentation outputs from multiple prompts via a visual-textual gating network.

Acquisition of prompts. There exist multiple strategies for generating prompts from simple to more advanced ones. We developed three types of prompt with increasing complexity. First, we use the name of each instrument class as the prompt. Second, inspired by CLIP's prompt template, we design a similar prompt template in the surgical context, i.e. 'the surgical instrument area represented by the [class name]'. Third, we leverage the recent released large language model GPT-4 [34] to generate prompts. We pose a question template: 'Please describe the appearance of [class name] in endoscopic surgery, change the description to a phrase with a subject, and do not use colons.' Please see supplementary material for the full list of our prompts.

Fusion of prompt predictions. Given the prompt set T = { T 1 , T 2 , T 3 } for an instrument to be segmented, we denote the corresponding textual feature for i -th prompt as F i T . Each F i T is paired the visual feature F I to predict a score map S i as outlined in Section 3.3, resulting in a set of score maps S = { S 1 , S 2 , S 3 } for all prompts. We have parallelled the prompt processing by reshaping different

prompts to the batch dimension, exploiting the transformer structure to concurrently obtain S . To combine the score maps in S , we devise a visual-textual gating network to predict the pixel-wise weight maps.

Visual-textual gating network G . G consists of a 3-layer residual block [14]. It ingests both the visual feature F I and textual feature F p T : we duplicate F T ∈ R 1 × D to F p T ∈ R N × D to match F I 's dimension and concatenate them before inputting to G . The outputs of G are three weight maps corresponding to the three score maps in S . These weights are normalized using softmax operations along the prompt dimension. We calculate the weighted sum of scores maps in S to derive the final score map S et ∈ R H × W . The final mask prediction M is obtained from S et by thresholding it.

3.5 Hard Instrument Area Reinforcement

Current model falls short on distinguishing the accurate category and boundary of surgical instruments due to the complex surgical conditions ( e.g. lighting variance) in endoscopy scenarios. We seek to reinforce its performance on the hard-predicted area in an image by utilizing a MAE-like structure [15] to reconstruct the image especially on hard-predicted area for representation enhancement. We propose a hard instrument area reinforcement module to bolster the model's segmentation accuracy for various instrument classes and their intricate details. Unlike MAE, which pays equal attention to all areas, our objective is to focus the visual encoder on discriminating the challenging instrument area. Therefore, we first introduce a hard area mining scheme to construct a masked image, then feed the visible patches into the decoder to reconstruct the whole image. This module shares the same image encoder with segmentation model.

Hard area mining. Given the estimated segmentation mask M et , we compare it with the ground truth mask M gt to obtain the falsely predicted area, which is taken as the hard instrument area for segmentation. For the input image I , we divide the area of hard regions over that of the whole image to obtain a masking ratio r ∈ (0 , 1) : 1) if r exceeds a pre-defined threshold r t , we randomly unmask some masked pixels till the ratio falls to r t to create the masked image; 2) if r is less than r t , we randomly mask some more unmasked pixels till the ratio meets r t to create the masked image.

Hart area reinforcement. We feed the obtained masked image into the image encoder and employ the same decoder structure as MAE to reconstruct the full image area based on the unmasked area. This helps the model to focus on recognizing the subtle details of the hard-predicted area.

It is worth mentioning that this module is no longer needed during testing.

3.6 Model Training

Our method is based on the pre-trained vision-language model, CLIP. During training, given limited surgical instrument descriptions, we freeze the text encoder and only finetune the image encoder. Two distinct loss functions, L seg for image segmentation and L rec for hard instrument area reinforcement, are introduced.

For L seg , binary cross entropy loss is applied:

where M gt j and M et j denote the labels of ground truth mask M gt and prediction mask M et at the j -th position, respectively.

For L rec , denoting by the I rec and I the reconstructed image and original image, we employ the L2 loss to minimize their pixel-wise distance:

where I rec j and I j represent the reconstructed and original pixel values at j -th location, respectively.

The overall loss function is given by L = L seg + λ L rec , λ serves as the loss weight.

4 Experiments

4.1 Datasets and Metrics

Datasets We evaluate our method on two endoscopic surgical instrument segmentation datasets: EndoVis2017 [1], EndoVis2018 [2]. EndoVis2017 includes 10 videos from the da Vinci robotic system, containing 6 distinct surgical instrument types (bipolar forceps, prograsp forceps, large needle driver, vessel sealer, grasping retractor, monopolar curved scissors) and an ultrasonic probe (classified as other instruments). Following [43], we employ 4-fold cross-validation to assess the model performance. For EndoVis2018, we adopt the widely-used labeling and dataset partitioning method proposed in [12]. This dataset consists of 15 video sequences, with 11 training and 4 testing sequences, and 7 predefined instrument categories (bipolar forceps, prograsp forceps, large needle driver, monopolar curved scissors, ultrasound probe, suction instrument, clip applier). Notably, two categories (suction instrument and clip applier) differ from those in EndoVis2017 (vessel sealer and grasping retractor), enabling cross-category experiments to evaluate our model's generalizability. Additionally, both datasets provide binary and parts segmentation labels. Binary segmentation comprises background tissue and instruments, while parts segmentation distinguishes instrument components as shaft, wrist, and claspers. Besides the two datasets, we have also evaluated our method on the EndoVis2019 [39] and Cholecseg8k [17] datasets in the supplementary material.

Metrics Adopting the evaluation method from [12], we utilize three prevalent IoU-based evaluation metrics: Ch_IoU, ISI_IoU, and mc_IoU. 1) Ch_IoU computes the mean IoU for each category present in the ground truth of an image, then averages them across all image. 2) ISI_IoU extends Ch_IoU by computing mean IoUs for all predicted categorizes regardless of their presence in the ground truth of the image. Ch_IoU is normally greater than or equal to ISI_IoU. 3) mc_IoU is a measurement addressing category imbalance by changing the averaging order in ISI_IoU.

4.2 Implement Details

Vision-Language model. We employ the image and text encoders from the widely-used CLIP model [19], specifically the ViT-B-16 variant pre-trained on the Laion2B [40] dataset, to construct

our promptable surgical instrument segmentation model. The image encoder has a patch size of 16 ( p = 16 ) and the text encoder has a token length limit of 77.

Hyper-parameters. We offer two training/inference default image sizes, 896 × 896 and 448 × 448 , which are compatible with the size requirements for image patching in the ViT-based image encoder [8]. During evaluation, we restore the segmentation prediction to the original image size.Feature dimension D is 1024 . Following [46], we employ a threshold θ = 0 . 35 to transform the score map S into mask M , and select the highest-scoring category per pixel in multi-category cases. For hard instrument area reinforcement, we set mask ratio r to 0.25. Finally, the loss weight λ is 0.5. All hyperparameters are determined empirically by segregating 20% of the training data as a validation set following [46].

Training. In our experiments, we adopt the Adam [23] optimizer with a learning rate of 1 e -4 . We train for 50 epochs, reducing the learning rate to 1 e -5 at the 35-th epoch. To enhance the model's generalization, we apply data augmentation techniques to the image, including random crop, horizontal flip, random rotation, and brightness perturbation. The model is trained on 4 V100 GPUs, the batch size is 16.

4.3 Results

Comparison to state of the art. In Table 1 and Table 2, we compare our method with a series of state of the art methods on the EndoVis2017 and EndoVis2018 datasets, respectively. The methods can be divided into two group depending on whether they have utilized textural prompts for surgical instruments. Ours belongs to the second group. As demonstrated by the results, our method significantly surpasses the state of the art in the first group: e.g. on the EndoVis2017 dataset, +7.36% on Ch_IoU, +5.84% in ISI_IoU, and +9.67% in mc_IoU. Particularly, a smaller difference between Ch_IoU and ISI_IoU for our method indicates it has fewer misclassified categories compared to other methods. Next, comparing to methods in second group, which are originally designed for natural image segmentation, our method yields clearly superior results, validating the effectiveness of our specifically tailored modules for surgical instrument segmentation. Notably, we implement both CRIS and CLIPSeg using their open-source implementations with batch size 64 for training.

Comparison to SAM . Whilst we were working on our work, the segment anything model (SAM) [24] was released concurrently. SAM possesses the capability to output corresponding object masks through visual or text prompts. Its officially released implementation however only contains visual prompts (point, box and mask), lacks text prompting. We thus leverage a community-based solution, lang-segment-anything [32], which enables this function for SAM. We found that this SAM variant struggles with medical prompts ( e.g. the Ch_IoU is only 17.77% and 22.08% on EndoVis2018 and EndoVis2017, respectively), performing significantly inferior than ours. For more details, we refer readers to the supplementary material.

Cross-dataset experiment. We also evaluate our method in the cross-dataset setting by training it on EndoVis2018 and testing it on EndoVis2017, and vice versa. As depicted in Tables 1 and Table 2 ( i.e. Cross-Ours), our method still gets competitive results compared with the state of the art on EndoVis2017, and is on par with state of the art on EndoVis2018, attesting its good generalizability. Note that categories between EndoVis2017 and EndoVis2018 do not fully overlap, the exceptions are the VS and SI categories. Despite this, we achieve very competitive results on them, showcasing the potential capability of our method for open-set text promptable surgical instrument segmentation.

4.4 Ablation Study

We conduct the ablation study on the EndoVis2017 dataset using images of sizes 448 × 448 and Ch_IoU and ISI_IoU as the evaluative metrics.

Image encoder: multi-scale feature augmentation. We ablate the proposed multi-scale feature augmentation (MSFA) in our method and offer the result in Table 3. A clear performance drop can be observed for ours w/o MSFA. MSFA enhances the contextual information for feature representation, thus improving the segmentation performance.

Text encoder: global feature from [CLS] token. We offer an variant by averaging the features from individual word tokens to produce the textural feature F T . The result is in Table 3: one can see that [CLS] → Words decreases Ch_IoU and ISI_IoU scores by 0.46% and 0.48% respectively. This suggests that the [CLS] token can better encapsulate the global information of the text description.

mask ratio

0.25

0.75

77.79

76.92

ISI_IoU

76.45

75.65

Text promptable mask decoder: prompting schemes. We study the importance of the two prompting schemes in Table 4. Results indicate that using only attention-based prompting (Ours w/ AP) outperforms using only convolutional-based prompting (Ours w/ CP) in terms of Ch_IoU and ISI_IoU scores. Moreover, employing both prompting schemes (Ours) sequentially improves the segmentation performance, suggesting that convolution-based prompting can further refine the output of the attention-based approach.

Mixture of prompts: different prompts. We offer the ablation study of MoP in Table 5. Results reveal that 1) using a single and simple prompt ( e.g. Ours w/ P-cls) already performs very well, outperforming the state of the art in Table 1; 2) the proposed mixture of prompts can further improve the segmentation performance. Over the three prompt types, the one from GPT-4 performs the best, owing to its more comprehensive coverage of the knowledge for certain instrument. We have also tried to replace the simple class prompt with prompt generated from another large language model, Bard [13]. Yet, we do not observe significant performance difference, cf. Ours (P-cls → P-Bard). We suspect the reason is that the knowledge contained in the GPT-4 prompt and the Bard prompt highly overlap, given the input token length limit. Note that prompts generated by GPT-4 might slightly vary given different time stamps. We have tried to study this factor, yet find very trivial impact of it.

Mixture of prompts: pixel-wise weight maps. To confirm the need for pixel-wise weight maps in the proposed visual-textual gating network, we develop a simpler version by using the image-wise scalar weight following the design in MoE [6]. These scalar weights are normalized via softmax. Experiments in Table 7 show our pixel-wise weighting scheme is better than the image-wise variant.

Hard instrument area reinforcement: modules and parameters. If we remove this module from our method (Ours w/o HIAR), as shown in Table 6, the performance clearly drops ( e.g. 2.57% and 2.47%). Next, we highlight the efficacy of the hard area mining (HAM) by removing it from HIAR, i.e. Ours w/o HAM; the HIAR module is then rather similar to a typical MAE module. We also observe a clear performance drop, underlining the crucial role of HAM in HIAR. Finally, we vary the making ratio threshold r t over 0.1, 0.25, and 0.5 in Table 6: we can see r t = 0 . 25 works the best.

Hard instrument area reinforcement: mask ratio r t . MAE[15] originally employs a high masking ratio (0.75), but in our study, we argue that a lower threshold (0.25) is more suitable: surgical instruments occupy a relatively small portion of the image, a low masking ratio will make the model concentrating on the hard area. Table 8 reveals a performance decrease with a masking ratio threshold of 0.75 when compared to our default setting of 0.25.

r

t

Ground Truth

Predic.ons

4.5 Computational Analysis

We perform experiments to assess computational complexity and inference speed in Tab. 9. This is achieved by evaluating floating point operations per second (FLOPs) and frames per second (FPS) using a single A100 GPU. We run the experiment on EndoVis2017 [1] by resizing the input image to the default input sizes corresponding to different methods for testing (e.g. 800 × 800 for ISINet, 224 × 224 for MATIS, 416 × 416 for CRIS, 448 × 448 for CLIPSeg and Ours). In Tab. 9, it is evident that our model's computational complexity (FLOPs) and inference speed (FPS) aligns with other adapted text-promptable approaches ( i.e. , CRIS [46] and CLIPSeg [29]), achieving real-time performance suitable for clinical applications. Compared to conventional segmentation methods ( i.e. , ISINet [12] and MATIS [3]), ours appears to be clearly more efficient than ISINet; while it is marginally slower than MATIS, likely due to MATIS having lower complexity (FLOPs).

5 Conclusion

In this paper, we introduce a novel approach for text promptable surgical instrument segmentation leveraging pretrained vision-language model. Our method addresses the challenge of instrument variety by redefining the task as a text promptable segmentation problem, which enhances the model's adaptability to new surgical instruments. Our promptable mask decoder carefully segments the surgical instruments in images via an attention-based and convolution-based prompting schemes. Our mixture of prompts mechanism effectively employs diverse prompts for better segmentation performance. While our hard instrument area reinforcement module intertwined with hard area mining significantly improves the segmentation performance on challenging area. Experimental results demonstrate our model's superior performance over state of the art. Future work will aim to further exploit the potential of this method for real-world surgical scenario.

6 Social Impact

Automated surgical instrument segmentation offers benefits to patients, surgeons, manufacturers, and society. Precise instrument tracking boosts surgical safety and precision, mitigating unintended tissue damage. Such automation eases surgeons' tasks, letting them concentrate on intricate procedures, and augments training for novices. Manufacturers integrating this technology could produce smarter surgical tools, aligning with AI-driven healthcare innovations to elevate patient care.

Acknowledgment

This work was supported by King's Cambridge 1 Access Fund, Tongji Fundamental Research Funds for the Central Universities, and core funding from the Wellcome/EPSRC [WT203148/Z/16/Z; NS/A000049/1]. Computing resources provided by King's Computational Research, Engineering and Technology Environment (CREATE). For the purpose of open access, the authors have applied a CC BY public copyright licence to any Author Accepted Manuscript version arising from this submission. TV is supported by a Medtronic / RAEng Research Chair [RCSRF1819\7\34], and is co-founder and shareholder of Hypervision Surgical. MW and OA are supported by the EPSRC CDT [EP/S022104/1].

References

Max Allan, Alex Shvets, Thomas Kurmann, Zichen Zhang, Rahul Duggal, Yun-Hsuan Su, Nicola Rieke, Iro Laina, Niveditha Kalavakonda, Sebastian Bodenstedt, et al. 2017 robotic instrument segmentation challenge. arXiv preprint , 2019.

Max Allan, Satoshi Kondo, Sebastian Bodenstedt, Stefan Leger, Rahim Kadkhodamohammadi, Imanol Luengo, Felix Fuentes, Evangello Flouty, Ahmed Mohammed, Marius Pedersen, et al. 2018 robotic scene segmentation challenge. arXiv preprint , 2020.

Nicolás Ayobi, Alejandra Pérez-Rondón, Santiago Rodríguez, and Pablo Arbeláez. MATIS: Maskedattention transformers for surgical instrument segmentation. arXiv preprint , 2023.

Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint , 2016.

Britty Baby, Daksh Thapar, Mustafa Chasmai, Tamajit Banerjee, Kunal Dargan, Ashish Suri, Subhashis Banerjee, and Chetan Arora. From forks to forceps: A new framework for instance segmentation of surgical instruments. In WACV , 2023.

Zixiang Chen, Yihe Deng, Yue Wu, Quanquan Gu, and Yuanzhi Li. Towards understanding mixture of experts in deep learning. arXiv preprint , 2022.

Henghui Ding, Chang Liu, Suchen Wang, and Xudong Jiang. Vision-language transformer and query generation for referring segmentation. In ICCV , 2021.

Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In ICLR , 2021.

Yu Du, Fangyun Wei, Zihe Zhang, Miaojing Shi, Yue Gao, and Guoqi Li. Learning to prompt for open-vocabulary object detection with vision-language model. In CVPR , 2022.

Han Fang, Pengfei Xiong, Luhui Xu, and Yu Chen. Clip2video: Mastering video-text retrieval via image clip. arXiv preprint , 2021.

Guang Feng, Zhiwei Hu, Lihe Zhang, and Huchuan Lu. Encoder fusion network with co-attention embedding for referring image segmentation. In CVPR , 2021.

Cristina González, Laura Bravo-Sánchez, and Pablo Arbelaez. Isinet: an instance-based approach for surgical instrument segmentation. In MICCAI , 2020.

Google. Google bard, 2023.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR , 2016.

Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross B. Girshick. Masked autoencoders are scalable vision learners. In CVPR , 2022.

HS Himal. Minimally invasive (laparoscopic) surgery. Surgical Endoscopy And Other Interventional Techniques , pages 1647-1652, 2002.

W-Y Hong, C-L Kao, Y-H Kuo, J-R Wang, W-L Chang, and C-S Shih. Cholecseg8k: a semantic segmentation dataset for laparoscopic cholecystectomy based on cholec80. arXiv preprint , 2020.

Ronghang Hu, Marcus Rohrbach, and Trevor Darrell. Segmentation from natural language expressions. In ECCV , 2016.

Gabriel Ilharco, Mitchell Wortsman, Ross Wightman, Cade Gordon, Nicholas Carlini, Rohan Taori, Achal Dave, Vaishaal Shankar, Hongseok Namkoong, John Miller, Hannaneh Hajishirzi, Ali Farhadi, and Ludwig Schmidt. OpenCLIP, 2021.

Yueming Jin, Keyun Cheng, Qi Dou, and Pheng-Ann Heng. Incorporating temporal prior from motion flow for instrument segmentation in minimally invasive surgery video. In MICCAI , 2019.

Michael I Jordan and Robert A Jacobs. Hierarchical mixtures of experts and the em algorithm. Neural computation , pages 181-214, 1994.

Namyup Kim, Dongwon Kim, Suha Kwak, Cuiling Lan, and Wenjun Zeng. Restr: Convolution-free referring image segmentation using transformers. In CVPR , 2022.

Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR , 2015.

Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al. Segment anything. arXiv preprint , 2023.

Ruiyu Li, Kai-Can Li, Yi-Chun Kuo, Michelle Shu, Xiaojuan Qi, Xiaoyong Shen, and Jiaya Jia. Referring image segmentation via recurrent refinement networks. In CVPR , 2018.

Tsung-Yi Lin, Piotr Dollár, Ross B. Girshick, Kaiming He, Bharath Hariharan, and Serge J. Belongie. Feature pyramid networks for object detection. In CVPR , 2017.

Che Liu, Sibo Cheng, Miaojing Shi, Anand Shah, Wenjia Bai, and Rossella Arcucci. Imitate: Clinical prior guided hierarchical vision-language pre-training. arXiv preprint , 2023.

Jie Liu, Yixiao Zhang, Jie-Neng Chen, Junfei Xiao, Yongyi Lu, Bennett A Landman, Yixuan Yuan, Alan Yuille, Yucheng Tang, and Zongwei Zhou. CLIP-driven universal model for organ segmentation and tumor detection. arXiv preprint , 2023.

Timo Lüddecke and Alexander Ecker. Image segmentation using text and image prompts. In CVPR , 2022.

Huaishao Luo, Lei Ji, Ming Zhong, Yang Chen, Wen Lei, Nan Duan, and Tianrui Li. Clip4clip: An empirical study of clip for end to end video clip retrieval and captioning. Neurocomputing , pages 293-304, 2022.

Saeed Masoudnia and Reza Ebrahimpour. Mixture of experts: a literature survey. Artificial Intelligence Review , pages 275-293, 2014.

Luca Medeiros. Language segment-anything, 2023. URL https://github.com/luca-medeiros/ lang-segment-anything .

Antoine Miech, Jean-Baptiste Alayrac, Lucas Smaira, Ivan Laptev, Josef Sivic, and Andrew Zisserman. End-to-end learning of visual representations from uncurated instructional videos. In CVPR , 2020.

OpenAI. GPT-4 technical report. arXiv preprint , 2023.

Or Patashnik, Zongze Wu, Eli Shechtman, Daniel Cohen-Or, and Dani Lischinski. Styleclip: Text-driven manipulation of stylegan imagery. In ICCV , 2021.

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In ICML , 2021.

Carlos Riquelme, Joan Puigcerver, Basil Mustafa, Maxim Neumann, Rodolphe Jenatton, André Susano Pinto, Daniel Keysers, and Neil Houlsby. Scaling vision with sparse mixture of experts. In NeurIPS , 2021.

Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In MICCAI , 2015.

Tobias Ross, Annika Reinke, Peter M Full, Martin Wagner, Hannes Kenngott, Martin Apitz, Hellena Hempe, Diana Mindroc Filimon, Patrick Scholz, Thuy Nuong Tran, et al. Robust medical instrument segmentation challenge 2019. arXiv preprint , 2020.

Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade W Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, Patrick Schramowski, Srivatsa R Kundurthy, Katherine Crowson, Ludwig Schmidt, Robert Kaczmarczyk, and Jenia Jitsev. LAION-5b: An open large-scale dataset for training next generation image-text models. In NeurIPS Datasets and Benchmarks Track , 2022.

Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc V. Le, Geoffrey E. Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. In ICLR , 2017.

Hengcan Shi, Hongliang Li, Fanman Meng, and Qingbo Wu. Key-word-aware network for referring expression image segmentation. In ECCV , 2018.

Alexey A Shvets, Alexander Rakhlin, Alexandr A Kalinin, and Vladimir I Iglovikov. Automatic instrument segmentation in robot-assisted surgery using deep learning. In ICMLA , 2018.

Mingkang Tang, Zhanyu Wang, Zhenhua Liu, Fengyun Rao, Dian Li, and Xiu Li. Clip4caption: Clip for video caption. In ACM MM , 2021.

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NIPS , 2017.

Zhaoqing Wang, Yu Lu, Qiang Li, Xunqiang Tao, Yandong Guo, Mingming Gong, and Tongliang Liu. Cris: Clip-driven referring image segmentation. In CVPR , 2022.

Zirui Wang, Jiahui Yu, Adams Wei Yu, Zihang Dai, Yulia Tsvetkov, and Yuan Cao. Simvlm: Simple visual language model pretraining with weak supervision. In ICLR , 2022.

Eleanora P Westebring-van der Putten, Richard HM Goossens, Jack J Jakimowicz, and Jenny Dankelman. Haptics in minimally invasive surgery-a review. Minimally Invasive Therapy & Allied Technologies , pages 3-16, 2008.

Jianzong Wu, Xiangtai Li, Xia Li, Henghui Ding, Yunhai Tong, and Dacheng Tao. Towards robust referring image segmentation. arXiv preprint , 2022.

Yinghui Xing, Qirui Wu, De Cheng, Shizhou Zhang, Guoqiang Liang, and Yanning Zhang. Class-aware visual prompt tuning for vision-language pre-trained model. arXiv preprint , 2022.

Zhao Yang, Jiaqi Wang, Yansong Tang, Kai Chen, Hengshuang Zhao, and Philip H. S. Torr. LAVT: language-aware vision transformer for referring image segmentation. In CVPR , 2022.

Linwei Ye, Mrigank Rochan, Zhi Liu, and Yang Wang. Cross-modal self-attention network for referring image segmentation. In CVPR , 2019.

Seonghoon Yu, Paul Hongsuck Seo, and Jeany Son. Zero-shot referring image segmentation with globallocal context features. In CVPR , 2023.

Linfeng Yuan, Miaojing Shi, and Zijie Yue. Losh: Long-short text joint prediction network for referring video object segmentation. arXiv preprint , 2023.

Zixu Zhao, Yueming Jin, and Pheng-Ann Heng. Trasetr: track-to-segment transformer with contrastive query for instance-level instrument segmentation in robotic surgery. In ICRA , 2022.

Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning to prompt for vision-language models. International Journal of Computer Vision , pages 2337-2348, 2022.

Yi Zhu, Zhaoqing Zhu, Bingqian Lin, Xiaodan Liang, Feng Zhao, and Jianzhuang Liu. RelCLIP: Adapting language-image pretraining for visual relationship detection via relational contrastive learning. In EMNLP , 2022.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis manuscript presents a clip-assisted semantic image segmentation method for surgical instruments. In terms of methodology, the proposed work can be viewed as an adaption of CLIPSeg to a domain-specific problem. Compared with CLIPSeg, a mixture of prompt strategy is used for augmenting text prompt information. Hard sample mining is also employed to further improve segmentation performance. The image segmentation network architecture has also been optimized. The proposed method is evaluated on two public surgical instrument segmentation datasets and is shown to outperform some fully-supervised and clip-assisted methods.\n\n**Commented after rebuttal**: I would like to thank the authors, other reviewers, and the ACs. I have carefully read through the rebuttals, and the comments from other reviewers. The discussion with the authors is constructive. Most of my raised concerns have been properly addressed.\n\nSTRENGTHS:\nThe proposed method yields improved performance on two surgical instrument segmentation datasets, compared with some existing fully-supervised/clip-assisted methods.\n\nEmploying text prompt brings additional flexibility in terms of the categories of segmentation targets. This is a major advantage of clip-assisted segmentation over conventional fully-supervised segmentation.\n\nThe proposed method is introduced in sufficient detail and can be easy to follow.\n\nWEAKNESSES:\nThe take-home information for readers may be limited/unclear. Despite that domain-specific adaptions on top of vanilla CLIPSeg leads to improved performance on surgical instrument segmentation, these contributions themselves may not be of sufficient interest to readers: hard sample mining, feature pyramid, generating multiple text prompts are already the common practice and should have been well-known for the community. The authors are encouraged to highlight their key take-home information to readers, and argue how the proposed work brings new knowledge.\n\nIn the abstract and introduction, the authors highlight improved flexibility when dealing with new segmentation targets of clip-assisted segmentation models. However, this point is not sufficiently evaluated and discussed in the experiments: how does CLIPSeg and CRIS that are also clip-driven work in this context?\n\nThe authors may also want to discuss/comment on a closely-related existing work [1].\n\n1. CLIP-Driven Universal Model for Organ Segmentation and Tumor Detection\n\nQUESTIONS:\nJudging by the ablation study (table 3 - 6), the contributions of individual components, except for the text encoder, seem to be quite marginal (the ablated models seem to be already quite strong). Therefore, from the authors' point of view, which factor accounts for most of the improved performance over CLIPSeg and CRIS? Or if there are other components that make a difference? Does the proposed method employ a stronger segmentation backbone or better training technique? The authors are encouraged to provide more details about the compared methods to avoid confusion.\n\nFrom the authors' perspective, what is the core take-home information to the readers?\n\nLIMITATIONS:\nAutomated instrument image segmentation may affect the outcome of a surgery, which may lead to real-world impact to the clinicians and the patient. The authors are encouraged to discuss the potential impact of their work for patients, clinicians, device manufacturers, and the society. \n\nThe authors are also encouraged to discuss if the proposed method exacerbate/mitigate potential risks and biases in image segmentation.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-01"}

comments: SUMMARY:
This paper proposes a novel text promptable surgical instrument segmentation approach to overcome challenges associated with the diversity and differentiation of surgical instruments by using the large CLIP model and a text promptable mask decoder. The experiments show the effectiveness of the proposed method on surgical instrument segmentation. However, I am very concerned with the novelty of the proposed modules due to the limited situations.

STRENGTHS:
1, The paper is well-written and easy to understand.

2, The experiments show the effectiveness of the proposed modules.

WEAKNESSES:
1, There are too few novelties due to most of the proposed modules being explored in the traditional segmentation tasks.

2, The dataset is too limited to prove the effectiveness of the proposed modules. Furthermore, there is no specific module designed for surgical instrument segmentation.

3, Lacking the comparison of inference time.

QUESTIONS:
1, How about comparing the proposed method and the Segment Anything model?

2, What’s the difficulty of surgical instrument segmentation? I understand that there might be a lack of some segmentation datasets leading to overfitting.

LIMITATIONS:
N/A
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2023-07-03

comments: SUMMARY:
This paper presents a text prompt-based surgical instrument segmentation method, which is more scalable to handle the diverse targets in endoscopy videos.

STRENGTHS:
- originality: this is the first work to use text prompt for surgical instrument segmentation
- quality: the performance of the proposed methods surpassed previous methods
- clarity: the paper is well organized
- significance: the method can handle unseen targets, which is desired in clinical scenarios.

WEAKNESSES:
- EndoVis challenge is organized every year. However, this paper validated the method on old datasets (18-19). Why not use the latest dataset and compare it to the challenge winning solutions? e.g., 21-22
http://opencas.dkfz.de/endovis/challenges/2022/

- Reference formats are not consistent.

QUESTIONS:
- Does the hard instrument area identified by the model align with the surgeons' perspective?

- Since the best performance in Table 1-2 still has a large room for further improvement. What are the typical failure modes? What's the potential reason for the model to generate such segmentation errors?

LIMITATIONS:
A more comment task is to segment both instruments and tissues.  It would be great to validate the method in this setting. Here is a public dataset
https://www.kaggle.com/datasets/newslab/cholecseg8k
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2023-07-06

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces a novel approach for surgical instrument segmentation in minimally invasive surgeries. By leveraging text prompts and vision-language models, the proposed method achieves improved segmentation performance. The approach shows promise for practical use in robotic-assisted surgery.\n\nSTRENGTHS:\nThe present work contributes with an innovative and effective approach for text promptable surgical instrument segmentation in minimally invasive surgeries.\n\nThis paper presents a meticulous study of previous work, which is important in the development of the present work. Also, the technical aspects are clearly explained and have also been evaluated using the correct metrics.\n\nAnother strength of this paper is the introduction of a mixture of prompts mechanism. By leveraging multiple text prompts for each surgical instrument, the authors enhance the segmentation performance of their model.\n\nThe experimental evaluation of the proposed model on EndoVis2017 and EndoVis2018 datasets demonstrates its superior performance compared with other works and promising generalization capability.\n\nIn summary, the work is an interesting application of deep learning in the medical area, and it also has a remarkable novelty.\n\nWEAKNESSES:\nRegarding the ablation study, it would be good if the authors could explain why they chose 448x448 as the image size. Aren't some details lost using this size?\n\nIt would be good if the authors could give more details about the dataset, i.e. the average duration of each video for example. \n\nIt would be very positive to also include more datasets, such as EndoVis2019, EndoVis2020 or EndoVis2021.\n\nQUESTIONS:\nDo you plan to include more datasets such as EndoVis2019, EndoVis2020 or EndoVis2021?\n\nLIMITATIONS:\nlimitations are not mentioned, authors should include the limitations in the paper.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-08"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a novel idea of utilizing text prompts and vision-language models to make surgical instrument segmentation more flexible and robust to diversity. The proposed method and custom modules achieve strong results on two endoscopic datasets.\n\nSTRENGTHS:\n1. The paper tackles an important problem in surgical instrument segmentation, which aims to enhance robot-assisted surgery systems. The key idea of using text prompts to improve generalization and adaptability to new instruments is novel.\n2. The method is technically sound, leveraging recent advances in vision-language models like CLIP. The image and text encoder setup makes sense. The text promptable mask decoder uses attention and convolution schemes nicely for decoding.\n3. Several custom modules are proposed to boost segmentation performance: 1) Mixture of prompts leverages multiple prompts effectively. 2) Hard instrument area reinforcement focuses on challenging regions.\n4. Comprehensive experiments on two datasets demonstrate superior performance over state-of-the-art methods. The cross-dataset generalization results are promising. The ablation studies validate the efficacy of individual model components like multi-scale feature extraction, mixture of prompts, and hard area reinforcement.\n\nWEAKNESSES:\n1. The problem definition and goal can be further sharpened. How does text-based prompting specifically help with increasing instrument variety and subtle inter-class differences? This needs more elaboration upfront.\n2. Some architectural details are unclear - like how exactly text features are integrated into the convolutional prompting scheme. More implementation specifics will help reproducibility.\n3. The computational complexity and inference speed are not analyzed. This could be important for practical usage.\n4. More in-depth experimentation on real-world surgical videos and systems would be preferred to further demonstrate applicability.\n\nQUESTIONS:\nHere are some suggestions for authors to consider:\n1. Explain how text prompting helps with adaptation to new instruments and resolving inter-class confusions, with examples.\n2. The method sections explain each component logically but can be more crisp and coherent in places. Authors could provide more implementation specifics for text feature integration into convolutional prompting and gating network.\n3. Consider reporting computational complexity and inference speeds. How does it compare with prior arts?\n4. Consider detecting and segmenting novel instruments not seen during training by using only text prompts.\n\nTypos:\n1. line 58: \"launguage\" should be \"language\".\n2. Mixed use of \"visual-textual\" and \"visual-textural\" at multiple places.\n\nAuthors should also consider discussing the limitations, such as those listed below, in the paper.\n\nLIMITATIONS:\n1. The method has only been evaluated on two datasets with limited surgical scenarios. Performance on more diverse real-world data is unclear.\n2. It relies on high-quality textual prompts, which may not always be available or easy to construct in practice.\n3. The requirement of retraining with new text prompts for new instruments reduces adaptability.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-27"}