Text Promptable Surgical Instrument Segmentation with Vision-Language Models
Authors: Zijian Zhou
Oluwatosin Alabi
Meng Wei
Tom Vercauteren
Miaojing Shi
Venue: NeurIPS 2023 poster (2023)
Keywords: Surgical Instrument Segmentation
Vision Language Models
Text Promptable Segmentation

Abstract: In this paper, we propose a novel text promptable surgical instrument segmentation approach to overcome challenges associated with diversity and differentiation of surgical instruments in minimally invasive surgeries. We redefine the task as text promptable, thereby enabling a more nuanced comprehension of surgical instruments and adaptability to new instrument types. Inspired by recent advancements in vision-language models, we leverage pretrained image and text encoders as our model backbone and design a text promptable mask decoder consisting of attention- and convolution-based prompting schemes for surgical instrument segmentation prediction. Our model leverages multiple text prompts for each surgical instrument through a new mixture of prompts mechanism, resulting in enhanced segmentation performance. Additionally, we introduce a hard instrument area reinforcement module to improve image feature comprehension and segmentation precision. Extensive experiments on several surgical instrument segmentation datasets demonstrate our model's superior performance and promising generalization capability. To our knowledge, this is the first implementation of a promptable approach to surgical instrument segmentation, offering significant potential for practical application in the field of robotic-assisted surgery. Code is available at https://github.com/franciszzj/TP-SIS.

Text Promptable Surgical Instrument Segmentation with Vision-Language Models
Zijian Zhou'
Oluwatosin Alabi²
Meng Wei Tom Vercauteren Miaojing Shi
DDpartment of Informatics, King's College London
2 School of Biomedical Engineering & Imaging Sciences, King's College London
College of Electronic and Information Engineering. Tongji University
@first_namel
@kcll ac uk; mshi rish@ttonji i .ddu. cn
Abstract
In this paper, we propose a novel text promptable surgical instrument segmentation approach to overcome challenges associated with diversity and differentiation of surgical instruments in minimally invasive surgeries. redefine the task as text promptable, thereby enabling more nuanced comprehension of surgical instru- ments and adaptability to new instrument types. Inspired by recent advancements in vision-language models, we leverage pretrained image and text encoders our model backbone and design a text promptable mask decoder consisting of attention- and convolution-basee prompting schemes for surgical instrument seg- mentation prediction. Our model leverages multiple text prompts for each surgical mixture of prompts mechanism, resulting in enhanced segmentation performance. Additionally, we introduce hard instrument area reinforcement module improve image feature comprehension and segmentation precision. Extensive experiments on several surgical instrument segmentation datasets demonstrate our model's 's superior performance and promising generaliza- tion capabilitty To our knowledge, this is the first implementation of promptable approach surgical instrument segmentation, offering significant potential for practical application in the field of robotic-assisted surgery. Code is available at attps://gi thub. com htt///fraaiizziiaoooo/// TTP-SIS.
instrument through
Introduction
Minimally invasive surgeries (MIS) have gained widespread attention in various surgical disci- plines [16, 48] due to their benefits over traditional open surgery, such as reduced patient discomfort and faster recovery times. Nonetheless, the restricted field of view and indirect vision through endo- scopic cameras for existing MIS procedures presents considerable obstacles, making the development of robot-assisted MIS increasingly crucial. Current robot-assisted surgery has to operate under direct control of the surgeon. Enhancing the automatic comprehension of the surgical process through precise instrument segmentation is seen facilitate robot-surgeon interaction. Despite this recognised importance, existing automatic surgical instrument segmentation faces significant challenges.
essential building block to introduce automation and
First, with fast-paced advances in MIS, there is a surge the variety of surgical instruments from different vendors. This is however compounded with the lack of comprehensive and large-scale dataset dedicated to the learning of surgical instrument segmentation. Current methods [43, 20, 12, 55, 5] do not adequately adapt to the ever-changing set of surgical instruments, necessitating re-labelling and re-training of models with the introduction each new instrument. This has notably hampered the practical application of surgical instrument segmentation within the MIS field.
Corresponding author.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Second, the segmentation methods face difficulty in distinguishing between different categories of instruments [5]. which often have similar appearances. The subtle visual difference across instruments and the often difficult imaging conditions in the confines of the surgical cavity make it harder to differentiate between instrument categories, causing poor segmentation performance.
Overcoming these challenges requires bearning-based approaches that are more flexible and robust than the current state of the art. Recent progress in pre-trained vision-language models [33, 36, 47] offer novel opportunities for our research questions as demonstrated by results obtained in diverse downstream computer vision tasks [10, 35, 44, 30, 29, 46, 9]. On the one hand, these vision language models, trained on abundant data, offer robust features which can help compensate for the scarcity of surgical instrument segmentation data. On the other hand, the capability of these models to output aligned image and text features allows for text to supplement information for surgical instrument segmentation, thus simplifying the differentiation between various instrument types.
To address the first challenge, we observe that existing supervised instrument segmentation models [12, 55, 5, 3] are trained on predefined categories, thereby limiting their semantic understanding of instruments outside of the narrow training set. Drawing inspiration from recent advancements in text promptable image segmentation [29, 46] with vision-language models [36], we redefine the task as a text promptable surgical instrument segmentation, thereby enhancing generalization and adaptability to an ever-growing array of new surgical instruments. Figure 1 contrasts the predefined category method with our text- promptable approach. Our model adapts the image and text encoders from pretrained rision-launguee textural prompt; text promptable mask decoder consisting of attention-based and convolution prompting schemes specifically designed to prompt the surgical instrument in the image from coarse to fine.
model. CLIP [36], to extract features from both the surgical image and
Varying text descriptions for each surgical instrument can yield distinct segmentation results. To tackle it, we propose a mixture of prompts mechanism, inspired by mixture of experts (MoE) [31, 37], to leverage diverse prompts. This mechanism inputs multiple source prompts to the model and fuses the resulting predictions via weights generated from a segmentation result hence benefits from various prompt information.
gating network. The final
To address the second challenge, we aim to enhance the model's 's capacity to obtain improved image features, thereby reinforcing its ability segment various surgical instrument categories and delineate precise edge details. We propose a hard instrument area reinforcement module intertwined with the popular image reconstruction approach, masked autoencoder (MAE) [15]. This module performs hard area mining at the site of segmentation error; by masking out the hard area, the module injects an auxiliary image reconstruction to the segmentation model to enahce the understanding of difficult areas in the image features.
To the best of our knowledge, introduce the first text promptable surgical instrument segmentation approach. Our extensive experiments demonstrate state-of-the-ar performance on several surgical instrument segmentation datasets [1,2,39, 17]. Additionally, of our model through cross-validation between the two datasets, showcasing promising results and emphasizing the significant application potential of our method.
evaluate the generalization capability
2 Related Work
Surgical instrument segmentation. Most of the current works on surgical instrument segmentation traditional vision-based models. For instance, TernausNet [43] trains an optimized U-Net [38] model for the segmentation of a restricted variety of surgical instruments. ISINet [12] proposes an instance-based segmentation method that utilizes temporal consistency module to identify instrument candidates. MF-TapNet [20] utilizes motion flow information through an attention pyramid network and is trained independently for binary, part, and instrument type segmentation tasks. While most approaches rely on convolutional networks, few recent works [55, 3] have investigated the use of mansformer-baseed methods. For example, MATIS [3] utilizes wixel-wise attention. masked attention modules targeting for instrument areas, and video transformers for temporal information handling. A Although the segmentation accuracy has been significantly improved by incorporating various architectures and modules, existing works are limited by their reliance on categorical understanding from the annotated training data; when new categories are introduced, they will have to be re-trained, which requires demanding labelling efforts and computational resources.

---

Bipolar Forceps
Vision based Model
Prograsp Forceps
Monopolar Curved Scissors
Image
Conventional vision- based surgical instrument segmentation method
pull leuape love sik sio quiii henner
Vision Language
handles cneciirnt grrsing
Model
Menapcia ar cured ic uan divaaus qpatc
Ground Truth
Our promptable instrument segmentation method
Figure Left: input image and its ground truth. Top Right: conventional vision- surgical instrument segmentation model with predefined categories. Bottom Right: ours leveraging vision- language model for text promptable surgical instrument segmentation.
Dision-language models. Large pretrained models like CLIP [36] has emerged as the most promising approaches for achieving state of the art results on downstream computer vision tasks. CLIP [36] is large pretrained vision-language model [45] that aligns text with image modalities in the embedding space to solve downstream tasks like zero-shot image classification [36, 27]. has been successfully applied to various vision tasks, including video captioning [44], video-text retrieval [30, 10], image generation [35], relation learning [57] and text promptable segmentation [46, 29].
Text promptable segmentation. Text promptable segmentation, also conceptualized referring expression segmentation in the literature, involves the utilization of natural language expressions as prompts for image segmentation, deviating from the traditional reliance solely class label annota- tions for training images [18]. Early works use convolution neural networks (CNNs) and recurrent neural networks (RNNs) extract visual and textual features which are fused via concatenation and used to perform segmentation [18, 25]. Attention mechanisms are also introduced for exploit- ing relations between visual and textual features [18, 25, 42, 52]. More recent approaches utilize transformers to perform visual and textual feature fusion either in the encoder [11, 49, 51, 22, 54] or
decoder [46, 29,77 7].
Large pretrained vision-language models have also been utilized for this task [46, 29, 53, 28, 24J. For instance, CRIS [46] uses the CLIP image and text encoders for multimodal knowledge transfer and transformer decoder for multimodal fusion and segmentation. CLIPSeg [29] extends prompt- based segmentation beyond text prompts to include image prompts processed by the CLIP image encoder. recent concurrent work, SAM [24], introduces a transformer-baat bbaed model capable generating object masks using either image text prompts. However, existing promptable segmentation methods are not designed to solve highly specific tasks like surgical instrument segmentation.
3 Method
In this section, we begin by defining the task (Section 3.1), then introduce the image and text encoders inherited from pretrained vision-language model (CLIP [36]), serving the feature extractors of our method (Section 3.2). Next, we develop a text promptable mask decoder responsible for decoding the visual features into instrument segments with the help of textual features (Section 3.3). Finally, we elaborate on two key modules: the mixture of prompts module (Section 3.4), designed to enhance segmentation performance through multiple prompts; and the hard instrument reinforcement module (Section 3.5), focused on improving instrument segmentation on difficult parts.
3.1 Task Definition
Given an image / Hx33 consisting of surgical instruments for an endoscopic operation, as well as text description 7' describing the name, appearance and function of a given instrument, the objec- tive is to obtain the surgical instrument mask M (0, 1 HW from J, prompted by the instrument description T. In cases when multiple instruments need to be segmented, input the model with

---

Prompt Set F3)
Text Encoder
CLIP Text
Tytte
Fusion
Image Encoder
CLIP Image Encoder
Asian bused Prumptinng
Corns buood Prompting
FPN
gong
(1) Image and Text Encoders
(2) Text Promptable Mosk Decoder (3) Mixture of Prompts
Hard Area Mining
Crec
Reinforcement
(4) Hard instrument Area Reinforcement
Figure 2: An overview of our method. Our method comprises four key modules: 1) image and text encoders derived from the pretrained vision-language model to obtain visual and textual features; 2) a text promptable mask decoder consisting of attention- and convolution based prompting schemes for predicting the score map from image features through text prompts; 3) a mixture of prompts mechanism that utilizes a visual-textual gating network to produce pixel-wise weights for merging different score maps; 4) hard instrument area reinforcement module reinforce image representation learning specifically on hard-predicted area.
multiple text descriptions corresponding to different categories, and the goal becomes producing the segmentation result M E (O,.... C) where represents the number of instrument categories.
3.2 Image and Text Encoders
To accomplish text promptable surgical instrument segmentation, first encode the input endoscopic image and instrument description T, obtaining the visual feature Fí and textual feature Fr, respectively. Importantly, enforce the semantic congruity between F' and FT, leverage the strength of the image and text encoders from the popular vision-language model, CLIP [36], pretrained on natural images.
Image encoder. Given the input image I, we utilize and fine tune the image encoder from CLIP [36], which is vision transformer (ViT) [8], for feature extraction. The ViT-baseh image encoder comprises 12 layers. We use F) E TR X to denote its I-th layer output, where / ranges from 1 to 12, N is the number of visual tokens, and D is the dimension of visual features in ViT. Supposing the patch size of the input image then
W
Multi-scale feature uugmentation, Extracting multi-scale features important semantic segmenta- tion in order to segment objects in different scales. Whilst the image encoder in CLIP is not originally designed for multiple scales, we equip it with this ability. Specifically, simulating the multi-scale extraction from CNN-based architecture, layers of the VIT, Le. F/. FS, F12, representing a wide coverage of level-of- for the image; and fuse these features by a feature pyramid network (FPN) [26]. Notice in order to feed these features into FPN, we need to first reshape them into the size of x x D, then we up-sample F/ and down-sample F22 both by factor of 2. Finally, the output of the FPN is reshaped size of N D to obtain the multi-scale image feature F7 NDD.
take the output features from the 4-th, 8-th, and 12-th
Text encoder. Given the input description T, we utilize and keep frozen the text encoder from CLIP, which is a transformer encoder [45], for feature extraction. Within the text encoder, T is tokenized with the inclusion a [CLS) token at beginning, representing the global information of the description. Following the common practice in [46, 29], we use the corresponding feature from the [CLS) token as the global textual feature for the description.
denote by Fr E RRxD where D is dimension of the textual feature, equivalent to that of the visual feature.

---

3.3 Text Promptable Mask Decoder
After obtaining Fy and Fr, our goal is to decode from F1 a score map S E Rx with the help of Fr. Each pixel of S signifies the probability of this pixel belonging to the instrument class described in Fr. By thresholding S. it can be transformed into the desired mask M. To achieve this goal, we present two prompting schemes: attention-basel prompting and convolution- based prompting, specified below. For a detailed structure figure of the text promptable mask decoder, please refer to the supplementary material.
Attention-based prompting. Inspired by [45], we employ the attention mechanisms to attend the instrument area in F2 using Fr. First, we compute the self-attention (SA) [45] within F, to promote the foreground area in the endoscopic image. Layer normalization (LN) [4] applied to F, beforehand while skip connection [14] is applied afterwards. We write out this process: Fig (F1) + Fr. Next, we compute the cross-attention (CA) [45] between Fig and Fr to localize the instrument area in Fig according to Fr. Similarly, this process is written as: Fic CA(LN( Fi Fr) + Fig- Subsequently, we apply the feed-forward network, which is basically fully connected layer to further polishing Fie. Fir FFN((N(F1ee) + Fie. SA, CA and FFN are organized in one decoding block, we devise three such blocks sequentially to obtain the attention-based prompted feature FIA E RNXD.
Convolution- prompting. We design a novel convolution- based prompting scheme to further refine FI, by using Fr to convolve it locally. To do this, FI, E RNX' needs to be firstly reshaped to restore its spatial dimensions, i.e. FI, E R
x Our idea to transform Fr into the convolution parameters, i.e. kernel weights and bias, and use these parameters to convolve Pr,, such that the instrument--elateed positions will be promoted. The transformation is done via fully connected (FC) layer: FT FC(FT). Fr is a vector of dimension D, the FC layer reshapes it to the vector FT of dimension D x + 1, with "k x k" representing the convolution kernel size and "+1" is the extra dimension accounting for bias. This allows decomposition of Er into convolution weights w E I x and bias E R.,, which are subsequently used convolve FiA.
S = Sigmoid (Conv( (FIa |u,, b)).
Sigmoid is applied to transform the final output into probabilities, hence producing the score map S.
The attention-based prompting computes the attention at each pixel by interacting it with pixels from the whole feature map, while in the convolution-based prompting, each pixel only interacts with its neighboring pixels in the i convlution kernel. We can see the former prompting as a global decoding process while the latter as a local decoding process to refine the former.
3.4 Mixture of Prompts
The provision of the prompt is crucial for the surgical instrument segmentation accuracy. Recognizing the impact of different prompts on given model [56, 50], we propose to effectively leverage the strength of various prompts to facilitate the recognition of surgical instruments. Inspired by mixture of experts (MoE) [21], which divides complex task into multiple sub-tasks and distributes them amongst multiple experts, we introduce a mixture of prompts (MoP) scheme to combine the segmentation outputs from multiple prompts via a visual-textual gating network.
Acquisition of prompts. There exist multiple strategies for generating prompts from simple to more advanced ones. We developed three types of prompt with increasing complexity. First, we use the name of each instrument class as the prompt. Second, inspired by CLIP's prompt template, we design similar prompt template in the surgical context, i.e. "the surgical instrument area represented by the [class name]". Third, we leverage the recent released large language model GPT-4 [34] to generate prompts. We pose a question template: "Please describe the appearance of [class name] in endoscopic surgery, change the description to a phrase with a subject, and do not use colons." Please supplementary material for the full list of our prompts.
Fusion of prompt predictions. Given the prompt set = (Ti, T3} for an instrument to be segmented, we denote the corresponding textual feature for i-th prompt as Fz. Each F is paired the visual feature F, to predict score map S, as outlined in Section 3.3, resulting in a set of score maps S (S1, S2, Sa} for all prompts. We have parallelled the prompt processing by reshaping different
5

---

prompts to the batch dimension, exploiting the transformer structure to concurrently obtain S. To combine the score maps in S, we devise visual-textual gating network predict the pixel-wise weight maps.
Visual-textull gating network G. 9 consists of 3-layer residual block [14]. It ingests both the visual feature Fj and textual feature F:; we duplicate Fr E R to F. E RNxD to match Py's dimension and concatenate them before inputting G. The outputs of & are three weight maps corresponding to the three score maps in S. These weights are normalized using softmax operations along the prompt dimension. We calculate the weighted sum of scores maps in S to derive the final score map Set E ER "R"W The final mask prediction M is obtained from S by thresholding it.
3.5 Hard Instrument Area Reinforcement
Current model falls short on distinguishing the accurate category and boundary of surgical instruments due to the complex surgical conditions (e.g. lighting variance) in endoscopy scenarios. We seek to reinforce its performance the hard-predicted area in image by utilizing MAE-like structure [15] to reconstruct the image especially on hard-predicted area for representation enhancement. We propose hard instrument for various instrument classes and their intricate details. Unlike MAE, which pays equal attention to all areas, our objective is to focus the visual encoder on discriminating the challenging instrument area. Therefore, we first introduce a hard area mining scheme to construct masked image, then feed the visible patches into the decoder to reconstruct the whole image. This module shares the same image encoder with segmentation model.
reinforcement module to bolster the model's segmentation accuracy
Hard area mining. Given the estimated segmentation mask Ms compare it with the ground truth mask M to obtain the falsely predicted area, which taken as the hard instrument area for segmentation. For the input image I, we divide the of hard regions over that of the whole image to obtain a masking ratio E (0, 1): 1) if exceeds pre-defined threshold rt, we randomly unmask some masked pixels till the ratio falls
to create the masked image: 2) if r is less than rz, randomly mask some more unmasked pixels till the ratio meets T; to create the masked image.
Hart area reinforcement. We feed the obtained masked image into the image encoder and employ the same decoder structure as MAE to reconstruct the full image area based on the unmasked area. This helps the model to focus recognizing the subtle details of the hard- predicted area.
is worth mentioning that this module no longer needed during testing.
3.6
Model Training
Our method is based on the pre-trained vision- model, CLIP. During training, given limited surgical instrument descriptions, we freeze the text encoder and only finetune the image encoder. Two distinct loss functions, L for image segmentation and ree for hard instrument area reinforcement, introduced.
For L binary cross entropy loss is applied:
Le = - (M (MMf log MSt + +(1 MM)) log() -M3))
(2)
where Mt and M denote the labels of ground truth mask M and prediction mask M at the j-th position, respectively.
For Ceee denoting by the F and I the reconstructed image and original image, we employ the L2 loss to minimize their pixel-wise distance:
C
Σ Eu 111
(3)
where Iise and I, represent the reconstructed and original pixel values j-th location, respectively. The overall loss function is given by L _ eec À serves as the loss weight.

---

Table 1: Comparison between our method and other fatte-of-the-arr methods on the Vis207 dataset. Methods the first (top) group are conventional supervised methods with predefined categories; methods in the second group (bottom) are text promptable using vision-language models. Pross-Ours in the last row represents the cross-datased experiment.
Method
Ch loU
ISI loU
BF
PF
LND
Vs
GR
MCS
UP
mc Lo__
TernausNet-I1 [43]
35.27
12.67
13.45
12.39
20.51
5.97
1.08
1.00
16.76
10.17
MF-TAPNet [20]
37.35
13.49
16.39
14.11
19.01
8.11
0.31
4.09
13.40
10.77
ISINet [12]
55.62
52.20
38.70
38.50
50.09
27.43
2.01
28.72
12.56
28.96
TraScTR [55]
60.40
65.20
45.20
56.70
55.80
38.90
11.40
31.3
18.20
36.79
S3Net [5]
72.54
71.99
75.08
54.32
61.84
35.5
27.47
43.23
28.38
46.55
MATIS [3]
71.36
66.28
68.37
53.26
53.55
31.89
27.34
21.34
26.53
41,09
CRIS [46]
69.94
67.83
54.87
50.21
68.33
50.12
0.00
43.97
0.00
38.21
CLIPSeg [29]
70.15
65.02
51.29
42.27
49.56
30.12
9.96
30.69
20.05
33.42
Ours (448)
77.79
76.45
69.57
68.91
89.88
82.60
0.00
72.53
0.00
54.78
Ours (896)
79.90
77.83
68.58
73,52
92.74
83.90
0.13
74.70
0.00
56.22
Cross-Ours (896)
72.18
70.44
65.54
58. 19
84.01
67.30
0.06
68.47
0.06
49.09
Table 2: Comparison between our method and other state-of-the-ae methods on the Endo EndoVis2011
dataset.
Method
CoU
IST IoU
BF
PF
LND
SI
CA
MCS
UP
IoU
TernausNet-I [43]
46.22
39.87
44.20
4.67
0.00
0.00
0.00
50.44
0.00
14.19
ME-TAPNet [20]
67.87
39.14
69.23
6.10
11.68
14.00
0.91
70.24
0.57
24.68
ISINet [12]
73.03
70.97
73.83
48.61
30.98
37.68
0.00
88.16
2.16
40.21
TraSeTR [55]
76.20
76.30
53.30
46.50
40.60
13.90
86.30
17.50
47.77
S3Net [5]
75.81
74.02
77.22
50.87
19.83
50.59
0.00
92.12
7.44
42.58
MATIS [3]
84.26
79.12
83.52
41.90
66.18
70.57
0.00
92,96
23.
54.04
CRTS [46]
74.10
72.29
73.58
58.20
47.64
72.14
4.56
45.99
20.18
46.04
CLIPSeg [29]
74.95
69.86
67.25
39.59
36.72
47.27
2.92
79.96
4.22
39.7
Ours (448)
82.67
81.54
81.53
70.18
71.54
90.58
21.46
65.57
57.51
65.48
Ours (896)
84.92
83.61
84,28
73.18
78,88
92.20
23.73
66.67
39.12
65.44
Cross-Ours (896)
66.25
64.92
65.81
56.12
44.72
79.77
1.22
8.97
4.77
37.34
4 Experiments
4.1 Datasets and Metrics
Datasets We evaluate our method on two endoscopic surgical instrument segmentation datasets: Endo Vis2017 [1]. EndoVis2011 [2]. EndoVis2017 includes 10 videos from the da Vinci robotic system, containing 6 distinct surgical instrument types (bipolar forceps, prograsp forceps, large needle driver, vessel sealer, grasping retractor, monopolar curved scissors) and an ultrasonic probe (classified as other instruments). Following [43], we employ 4-fold validatiio to assess the model performance. For Endo Vis2018, we adopt the widely-used labeling and dataset partitioning method proposed in [12]. This dataset consists of 15 video sequences, with 11 training and 4 testing sequences, and predefined instrument categories (bipolar forceps. prograsp forceps. large needle driver, monopolar curved scissors, ultrasound probe, suction instrument, clip applier). Notably, two categories (suction instrument and clip applier) differ from those Endo Vis2017 (vessel sealer and grasping retractor), enabling cross-category experiments to evaluate our model's generalizability. Additionally, both datasets provide binary and parts segmentation labels. Binary segmentation comprises background tissue and instruments, while parts segmentation distinguishes instrument components as shaft, wrist, and claspers. Besides the two datasets, we have also evaluated our method on the Endo Vis2019 [39] and Cholecseg8k [17] datasets in the supplementary material.
Metries Adopting the evaluation method from [12], we utilize three prevalent IoU-based evaluation metrics: Ch IoU, ISI IoU, and mc IoU. 1) Ch IoU computes the mean IoU for each category present the ground truth an image, then averages them across all image. 2) ISI_IoU extends Ch Ch_IoU by computing mean loUs for all predicted categorizes regardless of their presence in the ground truth of the image. Ch IoU Ch_IoU normally greater than or equal to ISI [SI_IoU. addressing category imbalance by changing the averaging order in IOU.
mc_IoU measurement
4.2 Implement Details
Vision-I Languag model. We employ the image and text encoders from the widely-used CLIP model [19], specifically the ViT-B-16 variant pre-trained on the Laion2B [40] dataset, to construct
7

---

our promptable surgical instrument segmentation model. The image encoder has patch size of 16 (p 16) and the text encoder has a token length limit of 77.
prper-parameters. We offer two training/inference default image sizes, 896 x 896 and 448 x 448, which are compatible with the size requirements for image patching in the ViT-based image encoder [8]. During evaluation, we restore the segmentation prediction to the original image Feature dimension D 1024. Following [46], we employ threshold 0 0.35 to transform the score map S into mask M, and select the highest-scoring category per pixel in multi-category cases. For hard instrument area reinforcement, we set mask ratio r to 0.25. Finally, the loss weight À is 0.5. All hyperparameters are determined empirically by segregating 20% of the training data as a validation set following [46].
Training. In our experiments, we adopt the Adam [23] optimizer with a learning rate of 1e 1e-t. We train for 50 epochs, reducing the learning rate 1e-5 at the 35-th epoch. To enhance the model's generalization, we apply data augmentation techniques to the image, including random crop, horizontal flip, random rotation, and brightness perturbation. The model trained on 4 V100 GPUs, the batch size is 16.
4.3 Results
Comparison to state of the art. In Table 1 and Table 2, we compare our method with a series of state of the art methods on the Endo and Endo Vis2018 datasets, respectively. The whether they have utilized textural prompts for
methods can be divided into two group depending surgical instruments. Ours belongs to the second group. As demonstrated by the results, our method significantly surpasses the state of the art in the first group: e.g. on the EndoVis2017 dataset, +7.36% on Ch IoU, +5.84% in ISI IoU, and +9.67% in mc IoU. Particularly. a smaller difference between Ch IoU and ISI IoU for method indicates it has fewer misclassified categories compared to other methods. Next, comparing to methods in second group, which are originally designed for natural image segmentation,
method yields clearly superior results, validating the effectiveness of our specifically tailored modules for surgical instrument segmentation. Notably, we implement both CRIS and CLIPSeg using their open-source implementations with batch size 64 for training.
Comparison to SAM. Whilst we were working on our work, the segment anything model (SAM) [24] was released concurrently. SAM possesses the capability to output corresponding object masks through visual or text prompts. Its officially released implementation however only contains visual prompts (point, box and mask), lacks text prompting. We thus leverage community- based solution, segg--ggmet--nnttieee
[32], which enables this function for SAM. We found that this SAM variant struggles with medical prompts (e.g. the Ch IoU is only 17.77% and 22.08% on Endo Vis2018 Endo 8 and Endo Vi5217,, respectively), performing significantly inferior than ours. For more details, we refer readers to the supplementary material.
Cross-dataset experiment. We also evaluate our method the cross-dataset setting by training on Vis2018 and testing it on Vis207,, and vice versa. As depicted in Tables 1 and Table (i.e. Cross-Ours), our method still gets competitive results compared with the state of the art on Vis201,, and is par with state of the art on Vis201,,018, attesting its good generalizability Note that categories between Endo and Vis201 do not fully overlap, the exceptions are the VS and SI categories. Despite this, achieve very competitive results them, showcasing the potential capability of our method for open-set text promptable surgical instrument segmentation.
4.4 Ablation Study
We conduct the ablation study on the Endo Vis2017 dataset using images of sizes 448 x 448 and Ch Ch_ JoU and ISI_Iol IoU as the evaluative metrics.
Image encoder: multi-scale feature augmentation. We ablate the proposed multi-scale feature augmentation (MSFA) in our method and offer the result in Table 3. A clear performance drop can be observed for ours w/o MSFA. MSFA enhances the contextual information for feature representation, thus improving the segmentation performance.
Text encoder: global feature from [CLS] token. We offer an variant by averaging the features from individual word tokens to produce the textural feature Fr. The result is in Table 3: one can see that [CLS] -> Words decreases Ch Ch_ JoU and ISI_IoU scores by 0.46% and 0.48% respectively. This suggests that the [CLS] token can better encapsulate the global information of the text description.
8

---

Table 3: Ablation study for image and text coder.
Table 4: Ablation study for text promptable mask decoder.
Encoder
IoU
ISL
Decoder
ISL Ch_IoU
Ours
77.79
76.45
Ours
77.79
76.45
Ours w/o MSFA
75.31
74.03
Ours w/ AP
76.46
75.28
Ours ([CIS)) + Words)
77.33
75.97
Ours w/ CP
42.38
37.61
Table 5: Ablation study for mixture of prompts; cls class, tem - template.
Table 6: Ablation study for hard instrument area reinforcement.
MoP
Ch LoU
ISI IoU
HIAR
Ch Jo_
ISI IoU
Ours w/ P-cls
75.67
74.34
Ours
77.79
76.45
Ours w/ P-tem
75.86
74.39
Ours w/o HIAR
75.22
73.98
Ours w/ P-GPT
77.19
76.09
Ours w/o HAM
75.98
74.57
Ours w/ P-cls & tem
76.01
74,53
F: = 0.1
77.03
76.01
Ours w/ P-els & tem & GPT
77.79
76.45
0.25
77.79
76.45
Ours (P-cls - P-Bard)
77.76
76.44
= 0.50
77.57
76.32
Table 7:
Ablation study for pixel-wise weight maps in mixture of prompts.
Table 8: Ablation study for mask ratio thresh-
old ra in hard instrument area reinforcement.
Gating network
Ch IoU
IST IoU
mask ratio rz
Ch IoU
ISI IoU
Pixel-wise
77.79
76.45
0.25
77.79
76.45
Image-wise 76.80
75.77
0.75
76.92
75.65
Text promptable mask decoder: prompting schemes. We study the importance of the two prompting schemes in Table 4. Results indicate that using only attention-based prompting (Ours w/ prompting (Ours w/ CP) in terms of Ch Ch_IoU and ISU scores. Moreover, employing both prompting schemes (Ours) sequentially improves the segmentation performance, suggesting that convolution-bras bassed prompting can further refine the output of the autention-based approach.
AP) outperforms using only convolutional-basee
Mixture of prompts: different prompts. We offer the ablation study of MoP in Table Results reveal that 1) using a single and simple prompt (e.g. Ours w/ P-cls) already performs very well, outperforming the state of the art in Table 1; 2) the proposed mixture of prompts can further improve the segmentation performance. Over the three prompt types, the one from GPT-4 performs the best, owing to its more comprehensive coverage of the knowledge for certain instrument. We have also tried replace the simple class prompt with prompt generated from another large language model, Bard [13]. Yet, we do not observe significant performance difference, of. Ours (P-cls + P-Bard). We suspect the reason is that the knowledge contained in the GPT-4 prompt and the Bard prompt highly overlap, given the input token length limit. Note that prompts generated by GPT-4 might slightly vary given different time stamps. We have tried study this factor, yet find very trivial impact of it.
Mixture of prompts: pixel-wise weight maps. To confirm the need for pixel-wise weight maps in the proposed visual-textue gating network, we develop simpler version by using the image- scalar weight following the design in MoE [6]. These scalar weights are normalized via softmax. Experiments in Table 7 show our pixel-wise weighting scheme is better than the image-wise variant.
Hard instrument area reinforcement: modules and parameters. we remove this module from our method (Ours w/o HIAR), as shown in Table 6, the performance clearly drops (e.g. 2.57% and 2.47%). Next, we highlight the efficacy the hard area mining (HAM) by removing from HIAR, i.e. Ours w/o HAM; the HIAR module is then rather similar to a typical MAE module. We also observe a clear performance drop, underlining the crucial role of HAM in HIAR. Finally, we vary the making ratio threshold r; over 0.1, 0.25, and 0.5 in Table 6: we can see r; 0.25 works the best.
Hard instrument area reinforcement: mask ratio r:- MAE [15] originally employs a high masking ratio (0.75), but in our study, we argue that a lower threshold (0.25) is more suitable: surgical instruments occupy a relatively small portion of the image, a low masking ratio will make the model concentrating on the hard area. Table 8 reveals performance decrease with a masking ratio threshold of 0.75 when compared to our default setting of 0.25.

---

Ground Truth
Predictions
Figure 3: Our segmentation visualization uses distinct colors to represent surgical instrument cate- gories, with ground truth and predictions displayed in the first and second rows, respectively.
4.5 Computational Analysis
We perform experiments to assess computational complexity and inference speed in Tab. 9. This is achieved by evaluating floating point operations per second (FLOPs) and frames per second (FPS) using a single A100 GPU. We run the experiment on Vis2011 [1] by resizing the input image the default input sizes corresponding
different methods for testing (e.g. 800 800 for ISINet,
224 x 224 for MATIS, 416 x 416 for CRIS, 448 x 448 for CLIPSeg and Ours). In Tab. 9, it is evident that our model's computational complexity (FLOPs) and inference speed (FPS) aligns with other adapted pext-promptable approaches (i.e. CRIS [46] and CLIPSeg [29]). achieving real-time performance suitable for clinical applications. Compared to conventional segmentation methods (i.e. ISINet [12] and MATIS [3]), ours appears to be clearly more efficient than ISINet; while it is marginally slower than MATIS, likely due to MATIS having lower complexity (FI.Os)).
Table 9: Computational analysis between ours and representative works on the EndoVis2011
Method
FLOPs (G)
FPS
ISINet [22
264
19
MATIS [3]
66
27
CRIS [46]
196
19
CLIPSeg [29]
127
23
Ours (448)
125
22
Conclusion
In this paper, we introduce a novel approach for text promptable surgical instrument segmentation leveraging pretrained vision-language model. Our method addresses the challenge of instrument variety by redefining the task as text promptable segmentation problem, which enhances the model's s adaptability to new surgical instruments. Our promptable mask decoder carefully segments the surgical instruments in images via an attention-based and convolution based prompting schemes. Our mixture of prompts mechanism effectively employs diverse prompts for better segmentation performance. While our hard instrument area reinforcement module intertwined with hard area mining significantly improves the segmentation performance on challenging area. Experimental results demonstrate model's 's superior performance over state of the art. Future work will aim to further exploit the potential of this method for real-world surgical scenario.
Social Impact
Automated surgical instrument segmentation offers benefits to patients, surgeons, manufacturers, and society. Precise instrument tracking boosts surgical safety and precision, mitigating unintended tissue damage. Such automation eases surgeons' tasks, letting them concentrate intricate procedures, and augments training for novices. Manufacturers integrating this technology could produce smarter surgical tools, aligning with AI-driven healthcare innovations elevate patient care.
10

---

Acknowledgment
This work was supported by King's Cambridge Access Fund, Tongji Fundamental Research Funds for the Central Universities, and core funding from the Wellcome/EPSRC [WT2031 41////8///1/// NS/A000049/11.. Computing resources provided by King's Computational Research, Engineering and Technology Environment (CREATE). For the purpose of open access, the authors have applied a CC BY public copyright licence to any Author Accepted Manuscript version arising from this submission. TVV is supported by Medtronic/ RAEng Research Chair [RCSRFI 8 1917134), and is co-founder and sharcholder of Hypervision Surgical. MW and OA are supported by the EPSRC CDT [EP/S022104/1].
References
[1] Max Allan, Alex Shvets, Thomas Kurmann, Zichen Zhang, Rahul Duggal, Yun-Hsuan Su, Nicola Rieke, Iro Laina, Niveditha Kalavakonda, Sebastian Bodenstedt, et al. 2017 robotic instrument segmentation challenge. arXiv preprint, 2019.
[2] Max Allan, Satoshi Kondo, Sehastian Bodenstedt, Stefan Leger, Rahim Kadkhodamohammaddi Imanol Luengo, Felix Fuentes, Evangello Flouty, Ahmed Mohammed, Marius Pedersen, et al. 2018 robotic scene segmentation challenge. arXiv preprint, 2020.
[3] Nicolás Ayobi, Alejandra Monez-Rondón, Santiago Rodríguez, and Pablo Arbeláez. MATIS: Masked- attention transformers for surgical instrument segmentation. arXiv preprint, 2023.
[4] Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint, 2016.
Britty Baby, Daksh Thapar, Mustafa Chasmai, Tamajit Banerjee, Kunal Dargan, Ashish Suri, Subhashis Banerjee, and Chetan Arora. From forks forceps: new framework for instance segmentation of surgical instruments. In WACV, 2023.
[6] Zixiang Chen, Yihe Deng. Yue Wu, Quanquan Gu, and Yuanzhi Li. Towards understanding mixture of experts in deep learning. arXiv preprint, 2022.
[7] Henghui Ding, Chang Liu, Suchen Wang. and Xudong Jiang. Vision-language transformer and query generation for referring segmentation. In ICCV, 2021.
[8] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition scale. ICLR, 2021.
[9] Yu Du Fangyun Wei, Zihe Zhang, Miaojing Shi, Yue Gao, and Guoqi Li. Learning to prompt for nen-vocabulaary object detection with vision-language model. In CVPR, 2022.
[10] Han Fang, Pengfei Xiong, Luhui Xu, and Yu Chen. Clip2video: Mastering video-text retrieval via image clip, arXiv preprint, 2021.
[11] Guang Feng, Zhiwei Hu, Lihe Zhang, and Huchuan Lu. Encoder fusion network with co-attention embedding for referring image segmentation. In CVPR, 2021.
[12] Cristina González, Laura Bravo-Sánchez, and Pablo Arbelaez. Isinet: instance-based approach for surgical instrument segmentation. In MICCAI, 2020.
[13] Google. Google bard. 2023.
[14] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016.
[15] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross B. Girshick. Masked autoencoders are scalable vision learners. In CVPR, 2022.
[16] HS Himal. Minimally invasive (laparoscopic) surgery. Surgicul Endoxcopy And Other Interventional Techniques, pages 1647-1652, 2002.
[17] W-Y Hong. C-L Kao, Y-H Kuo, J-R Wang. W-L Chang, and C-S Shih. Cholecseg8k: a semantic segmentation dataset for laparoscopic cholecystectomy based cholec80. arXiv preprint, 2020.

---

[18] Ronghang Hu, Marcus Rohrbach, and Trevor Darrell. Segmentation from natural language expressions. In ECCV, 2016.
[19] Gabriel Ilharco, Mitchell Wortsman, Ross Wightman, Cade Gordon, Nicholas Carlini, Rohan Taori, Achal Dave, Vaishaal Shankar, Hongseok Namkoong, John Miller, Hannanch Hajishirzi, Ali Farhadi, and Ludwig Schmidt. OpenCLIP, 2021.
[20] Yueming Jin, Keyun Cheng, Qi Dou, and Pheng- Heng. Incorporating temporal prior from motion flow for instrument segmentation in minimally invasive surgery video. In MICCAI, 2019.
[21] Michael I Jordan and Robert A Jacobs. Hierarchical mixtures of experts and the algorithm. Neural computation, pages 181-214, 1994.
[22] Namyup Kim, Dongwon Kim, Suha Kwak, Cuiling Lan, and Wenjun Zeng. Restr: Convolution freee referring image segmentation using transformers. In CVPR, 2022.
[23] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR, 2015.
[24] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander Berg, Wan Yen Lo, al. Segment anything. arXiv preprint, 2023.
[25] Ruiyu Li, Kai- Li, Yi-Chun Kuo. Michelle Shu. Xiaojuan Qi. Xiaoyong Shen, and Jiaya Jia. Referring image segmentation via recurrent refinement networks. In CVPR, 2018.
[26] Tsung- Yi Lin, Piotr Dollár, Ross B. Girshick, Kaiming He, Bharath Hariharan, and Serge J. Belongie. Feature pyramid networks for object detection. In CVPR, 2017.
[27] Che Liu, Sibo Cheng, Miaojing Shi, Anand Shah, Wenjia Bai, and Rossella Arcucci. Imitate: Clinical prior guided hierarchical vision language pre-training. arXiv preprint, 2023.
[28] Liu, Yixiao Zhang, Jic- Jie-Neng Chen, Junfei Xiao, Yongyi Lu, Bennett A Landman, Yixuan Yuan, Alan Yuille, Yucheng Tang, and Zongwei Zhou. CLIP-driven universal model for organ segmentation and tumor detection. arXiv preprint, 2023.
[29] Timo Lüddecke and Alexander Ecker. Image segmentation using text and image prompts. In CVPR, 2022.
[30] Huaishao Luo, Lei Ji, Ming Zhong, Yang Chen, Wen Lei, Nan Duan, and Tianrui Li. Clip4clip: An empirical study clip for end to end video clip retrieval and captioning. Neurocompating, pages 293-304, 2022.
[31] Saeed Masoudnia and Reza Ebrahimpour. Mixture of experts: literature survey Artificial Intelligence Review, pages 275-293, 2014.
[32] Luca Medeiros. Language segment-anything, 2023. URL https: http://tthtb..omm/ccccoooodooooo seggg--eeeet--teee
[33] Antoine Miech, Jean-Baptiste Alayrac, Lueas Smaira, Ivan Laptev, Josef Sivic, and Andrew Zisserman. End-to-end learning of visual representations from uncurated instructional videos. In CVPR, 2020.
[34] OpenAL. GPT-4 technical report. arXiv preprint, 2023.
[35] Or Patashnik, Zongze Wu, Fli Shechtman, Daniel Cohen-Or, and Dani Lischinski. Styleclip: Text-driven manipulation of stylegan imagery. In ICCV, 2021.
[36] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretehen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In ICML, 2021.
[37] Carlos Riquelme, Joan Puigcerver, Basil Mustafa, Maxim Neumann, Rodolphe Jenatton, André Susano Pinto, Daniel Keysers, and Neil Houlsby. Scaling vision with sparse mixture of experts. In NeurlPS, 2021.
[38] Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In MICCAI, 2015.
[39] Tobias Ross, Annika Reinke, Peter M Full, Martin Wagner, Hannes Kenngott, Martin Apitz, Hellena Hempe, Diana Mindroc Filimon, Patrick Scholz, Thuy Nuong Tran, et al. Robust medical instrument segmentation challenge 2019. arXiv preprint, 2020.
12

---

[40] Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade W Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, Patrick Schramowski. Srivatsa R Kundurthy, Katherine Crowson, Ludwig Schmidt, Robert Kaczmarezyk, and Jenia Jitsev. 55: An open large-scale dataset for training next generation image-text models. In NeurlPS Datasets and Benchmarks Track, 2022.
[41] Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoe V. Le. Geoffrey E. Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gate mixture-of-cxperts layer. In ICLR, 2017.
[42] Hengean Shi, Hongliang Li, Fanman Meng, and Qingbo Wu. Key-word-awand network for referring expression image segmentation. In ECCV, 2018.
[43] Alexey A Shvets, Alexander Rakhlin, Alexandr A Kalinin, and Vladimir I Iglovikov. Automatic instrument segmentation in robot-assisted surgery using deep learning In ICMLA, 2018.
[44] Mingkang Tang. Zhanyu Wang. Zhenhua Liu, Fengyun Rao, Dian Li, and Xiu Li. Clip4caption: Clip for video caption. In ACM MM, 2021.
[45] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. In NIPS, 2017.
[46] Zhaoqing Wang, Yu Lu, Qiang Li, Xungiang Tao, Yandong Guo, Mingming Gong, and Tongliang Liu. Cris: Clip-driven referring image segmentation. In CVPR, 2022.
[47] Zirui Wang, Jiahui Yu, Adams Wei Yu, Zihang Dai, Yulia Tsvetkov, and Yuan Cao. Simvlm: Simple visual language model pretraining with weak supervision. In ICLR, 2022.
[48] Eleanora P Westebring- van der Putten, Richard HM Goossens, Jack J Jakimowicz, and Jenny Dankelman. Haptics in minimally invasive surgery a review, Minimally Invasive Therapy & Allied Technologies, pages 161 2008
[49] Jianzong Wu, Xiangtai Li, Xia Li, Henghui Ding, Yunhai Tong, and Dacheng Tao. Towards robust referring image segmentation. arXiv preprint, 2022.
[50] Yinghui Xing, Qirui Wu, De Cheng. Shizhou Zhang, Guoqiang Liang, and Yanning Zhang. Class- aware visual prompt tuning for vision- laiiunla pre prettrined model. arXiv preprint, 2022.
[51] Zhao Yang. Jiaqi Wang, Yansong Tang. Kai Chen, Hengshuang Zhao, and Philip H. S. Tomr LAVT: language-aware vision transformer for referring image segmentation. In CVPR, 2022.
[52] Linwei Ye, Mrigank Rochan, Zhi Liu, and Yang Wang. Cross-modal self-attention network for referring image segmentation. In CVPR, 2019.
[53] Seonghoon Yu, Paul Hongsuck Seo, and Jeany Son. Zero-shot referring image segmentation with global- local context features. CVPR, 2023.
[54] Linfeng Yuan, Miaojing Shi, and Zijie Yue. Losh: Long-short text joint prediction network for referring video object segmentation. arXiv preprint. 2023.
[55] Zixu Zhao, Yueming Jin, and Pheng-Ann Heng. Trasetr: track-to-segment transformer with contrastive query for instance-level instrument segmentation robotic surgery. ICRA, 2022.
[56] Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning prompt for vision-language models. International Journal of Computer Vision, pages 2337-2348, 2022.
[57] Yi Zhu, Zhaoqing Zhu, Bingqian Lin, Xiaodan Liang, Feng Zhao, and Jianzhuang Liu. ReICLIP: Adapting language-image pretraining for visual relationship detection via relational contrastive learning. In EMNLP, 2022.
13


Reviews:
[{"review_id": "m2nrbDTpkF", "reviewer": "Reviewer_wnjR", "summary": "This paper introduces a novel idea of utilizing text prompts and vision-language models to make surgical instrument segmentation more flexible and robust to diversity. The proposed method and custom modules achieve strong results on two endoscopic datasets.", "questions": "Here are some suggestions for authors to consider:\n1. Explain how text prompting helps with adaptation to new instruments and resolving inter-class confusions, with examples.\n2. The method sections explain each component logically but can be more crisp and coherent in places. Authors could provide more implementation specifics for text feature integration into convolutional prompting and gating network.\n3. Consider reporting computational complexity and inference speeds. How does it compare with prior arts?\n4. Consider detecting and segmenting novel instruments not seen during training by using only text prompts.\n\nTypos:\n1. line 58: \"launguage\" should be \"language\".\n2. Mixed use of \"visual-textual\" and \"visual-textural\" at multiple places.\n\nAuthors should also consider discussing the limitations, such as those listed below, in the paper.", "limitations": "1. The method has only been evaluated on two datasets with limited surgical scenarios. Performance on more diverse real-world data is unclear.\n2. It relies on high-quality textual prompts, which may not always be available or easy to construct in practice.\n3. The requirement of retraining with new text prompts for new instruments reduces adaptability.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The paper tackles an important problem in surgical instrument segmentation, which aims to enhance robot-assisted surgery systems. The key idea of using text prompts to improve generalization and adaptability to new instruments is novel.\n2. The method is technically sound, leveraging recent advances in vision-language models like CLIP. The image and text encoder setup makes sense. The text promptable mask decoder uses attention and convolution schemes nicely for decoding.\n3. Several custom modules are proposed to boost segmentation performance: 1) Mixture of prompts leverages multiple prompts effectively. 2) Hard instrument area reinforcement focuses on challenging regions.\n4. Comprehensive experiments on two datasets demonstrate superior performance over state-of-the-art methods. The cross-dataset generalization results are promising. The ablation studies validate the efficacy of individual model components like multi-scale feature extraction, mixture of prompts, and hard area reinforcement.", "weaknesses": "1. The problem definition and goal can be further sharpened. How does text-based prompting specifically help with increasing instrument variety and subtle inter-class differences? This needs more elaboration upfront.\n2. Some architectural details are unclear - like how exactly text features are integrated into the convolutional prompting scheme. More implementation specifics will help reproducibility.\n3. The computational complexity and inference speed are not analyzed. This could be important for practical usage.\n4. More in-depth experimentation on real-world surgical videos and systems would be preferred to further demonstrate applicability.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### [wnjR-W1, Q1] Explain how text prompting helps with adaptation to new instruments and resolving inter-class confusions, with examples.\n\nFor the adaptation to new instruments,\nunlike previous instrument segmentation methods [36, 18, 11, 47, 5, 3] that require model retraining with new data, our text-promptable method instead offers the open-set potential that can directly segment/infer new instruments in images based on their textual descriptions. \nFor example, in Tab. 2 of the paper we show that, despite \"suction instruments\" appearing only in EndoVis2018, our model trained from EndoVis2017 attains a high IoU of 79.77\\% (SI in Tab. 2 of the paper). This underscores our approach's adaptability to new instruments.\n\nOn the other hand, for resolving inter-class confusion, our text prompting emphasizes unique traits of instrument in descriptions, aiding finer instrument class differentiation.\nFor example, while Bipolar forceps and Prograsp forceps are visually similar (see Fig. 1 in pdf file in global response section), they can be distinguished by \"elongated tweezer-like design\" and \"curved scissor-like handles\" from their textual descriptions, respectively.\nThese textual distinctions can help enhance the visual classification.\n\n### [wnjR-W2, Q2] More details for text feature integration into convolutional prompting and gating network.\n\nGiven the visual and textual feature, $F_I \\in \\mathbb{R}^{N \\times D}$ and $F_T \\in \\mathbb{R}^{1 \\times D}$, respectively, we transform $F_T$ via a fully connected (FC) layer: $\\tilde F_{T} = FC(F_{T})$.\n$F_T$ is a vector of dimension $D$, the FC layer reshapes it to the vector $\\tilde F_{T}$ of dimension $D \\times k \\times k + 1$, with \"$k \\times k$\" representing the convolution kernel size and \"$+1$\" the extra dimension accounting for bias.\nThis allows the decomposition of $\\tilde F_{T}$ into convolution weights $w \\in \\mathbb{R}^{1 \\times D \\times k \\times k}$ and bias $b \\in \\mathbb{R}^{1}$, which are used in subsequent convolutions.\n\nFor the gating network, it consists of a 3-layer residual block [13].\nWe duplicate $F_T \\in \\mathbb{R}^{1 \\times D}$ to $F_T^p \\in \\mathbb{R}^{N \\times D}$ to match $F_I$'s dimension and concatenate them before inputting to $\\mathcal G$. The output of $\\mathcal G$ are three weight maps corresponding to the three score maps in $\\mathcal S$.\nThese weights are normalized using softmax operations along the prompt dimension (they are normalized pixel-wisely across weight maps).\nWe calculate the weighted sum of scores maps in $\\mathcal S$ to derive the final score map.\n\nIn the revised version, we will refine our paper and release the code.\n\n### [wnjR-W3, Q3] Analysis of computational complexity and inference speed.\n\nWe assess the computational complexity and inference speed by evaluating floating point operations per second (FLOPs) and frames per second (FPS) respectively, using a single A100 GPU. We run the experiment on EndoVis2017 by resizing the input image to the default input sizes corresponding to different methods (e.g. $800 \\times 800$ for ISINet, $224 \\times 224$ for MATIS, $416 \\times 416$ for CRIS, $448 \\times 448$ for CLIPSeg and Ours).\nFrom the table below, it's evident that our model's computational complexity (FLOPs) and inference speed (FPS) align with other adapted text-promptable approaches (i.e., CRIS and CLIPSeg), achieving real-time performance suitable for clinical applications. \nOn the other hand, compared to conventional segmentation methods (i.e., ISINet and MATIS), ours appears to be clearly more efficient than ISINet; while it is marginally slower than MATIS [3], likely due to MATIS's small input size.\n\n Method       | FLOPs (G) | FPS \n--------------|-----------|-----\n ISINet [11]  | 264       | 19  \n MATIS [3]    | 66        | 27  \n CRIS [39]    | 196       | 19  \n CLIPSeg [24] | 127       | 23  \n Ours         | 125       | 22  \n\n### [wnjR-W4, L1] Experiments on more diverse real-world data.\n\nThank you for your suggestion.\nThe data in EndoVis2017 and 2018 datasets are indeed from real-world surgical videos.\nTo further validate our approach, we have added experimental results on EndoVis2019 and CholecSeg8k, which both consist of data from real-world surgical videos.\nFor detailed information, please refer to our global response section.\n\n### [wnjR-Q4] Use textural prompts to segment unseen instruments.\n\nIn our study, the cross-dataset experiments between EndoVis2017 and EndoVis2018 (Tabs. 1 & 2 of the paper) indeed underscore the potency of our method for segmenting unseen instruments.\nFor instance, when our model is trained on EndoVis2017, it can actually adeptly handle previously unseen classes, such as suction instrument (SI) in EndoVis2018 by utilizing on only their textural prompts without retraining. \n\n### [wnjR-L2] High-quality textual prompts potentially limit practical use.\n\nFirstly, we have designed well-crafted question templates to guide LLMs like GPT to automatically derive high-quality textural prompts (see Section 6.1 in our supplementary material).\nGiven these predefined question templates, adapting to new instruments becomes straightforward.\nSecondly, our method with simple prompts (not using LLMs) still performs very well (see Tab. 5 in the paper), outperforming SOTA with large margins.\nThirdly, once our model is trained, it is equipped with the open-set potential that can segment new instruments without retraining/finetuning (see cross dataset experiment in Sec 4.3).      \n\n### [wnjR-L3] Retraining with new prompts for different instruments limits adaptability.\n\nAs highlighted in [wnjR-W1, Q1] and [wnjR-L2], our text-promptable approach can adapt to new instruments without retraining/finetuning the model.\n\n### [wnjR-Misc] Typos.\nThanks! We will rectify them in the final version of the paper."}, {"review_id": "dUXLkFjUpV", "reviewer": "Reviewer_NUuZ", "summary": "The paper introduces a novel approach for surgical instrument segmentation in minimally invasive surgeries. By leveraging text prompts and vision-language models, the proposed method achieves improved segmentation performance. The approach shows promise for practical use in robotic-assisted surgery.\n", "questions": "Do you plan to include more datasets such as EndoVis2019, EndoVis2020 or EndoVis2021?", "limitations": "limitations are not mentioned, authors should include the limitations in the paper.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The present work contributes with an innovative and effective approach for text promptable surgical instrument segmentation in minimally invasive surgeries.\n\nThis paper presents a meticulous study of previous work, which is important in the development of the present work. Also, the technical aspects are clearly explained and have also been evaluated using the correct metrics.\n\nAnother strength of this paper is the introduction of a mixture of prompts mechanism. By leveraging multiple text prompts for each surgical instrument, the authors enhance the segmentation performance of their model.\n\nThe experimental evaluation of the proposed model on EndoVis2017 and EndoVis2018 datasets demonstrates its superior performance compared with other works and promising generalization capability.\n\nIn summary, the work is an interesting application of deep learning in the medical area, and it also has a remarkable novelty.", "weaknesses": "Regarding the ablation study, it would be good if the authors could explain why they chose 448x448 as the image size. Aren't some details lost using this size?\n\nIt would be good if the authors could give more details about the dataset, i.e. the average duration of each video for example. \n\nIt would be very positive to also include more datasets, such as EndoVis2019, EndoVis2020 or EndoVis2021.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### [NUuZ-W1] Why chose $448 \\times 448$ as the input image size?\n\nThe previous methods use different input image sizes ranging from $224 \\times 224$ to original image size (i.e.,$1024 \\times 1280$).\nGiven our adoption of a ViT-based image encoder, the input size must conform to ViT's patching requirements, i.e. being divisible by 14, 16, and 32 to fit various ViT models.\nAs such, we choose $448 \\times 448$ as a default setting considering both efficiency and accuracy. Notice that, as with all other methods, our predictions are always reshaped to the original image size for fair evaluation.\n\n### [NUuZ-W2] More details about the datasets.\n\nBelow are detailed descriptions of various datasets.\n\n- EndoVis2017 consists of ten sequences of abdominal porcine procedures, each containing 300 frames sampled at 1 Hz. For training data, the first 225 frames from eight sequences are used, and the remaining 75 frames are kept for testing. Two more sequences with full 300 frames are also reserved for testing.\n\n- EndoVis2018 includes 19 sequences, 15 of them are considered as the training set while the rest 4 as test set. Each sequence is originated from a single porcine training procedure. Redundant frames are manually removed to precisely ensure 300 frames in each sequence.\n\n- EndoVis2019 (Robust MIS) is derived from 30 minimally invasive surgical procedures, including 10 rectal resection, 10 proctocolectomy, and 10 sigmoid resection procedures. A total of 10,040 images are extracted from these procedures. The dataset consists of both training and test cases. Each case contains a 10-second video snippet with 250 endoscopic image frames and a reference annotation for the last frame. \n\n- CholecSeg8K is derived from Cholec80, containing 80 videos of cholecystectomy surgeries performed by 13 surgeons. Each video in CholecSeg8K is recorded at 25 FPS and has annotations for instruments and operation phases. Each video clip contributes 80 image frames, and for each of these frames, the dataset includes raw image data, annotations, and colour masks. In total, the dataset comprises 101 directories with a collection of 8,080 frames.\n\n### [NUuZ-W3, Q1] Experiments on more datasets.\n\nThank you for your suggestion. To further validate our approach, we have added experimental results on EndoVis2019 and CholecSeg8k. For detailed information, please refer to our global response section."}, {"review_id": "yfwyDDb5Xw", "reviewer": "Reviewer_Rt9K", "summary": "This paper presents a text prompt-based surgical instrument segmentation method, which is more scalable to handle the diverse targets in endoscopy videos.", "questions": "- Does the hard instrument area identified by the model align with the surgeons' perspective?\n\n- Since the best performance in Table 1-2 still has a large room for further improvement. What are the typical failure modes? What's the potential reason for the model to generate such segmentation errors?", "limitations": "A more comment task is to segment both instruments and tissues.  It would be great to validate the method in this setting. Here is a public dataset\nhttps://www.kaggle.com/datasets/newslab/cholecseg8k", "rating": 7, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- originality: this is the first work to use text prompt for surgical instrument segmentation\n- quality: the performance of the proposed methods surpassed previous methods\n- clarity: the paper is well organized\n- significance: the method can handle unseen targets, which is desired in clinical scenarios.", "weaknesses": "- EndoVis challenge is organized every year. However, this paper validated the method on old datasets (18-19). Why not use the latest dataset and compare it to the challenge winning solutions? e.g., 21-22\nhttp://opencas.dkfz.de/endovis/challenges/2022/\n\n- Reference formats are not consistent.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### [Rt9K-W1] Why not validate the method on the latest EndoVis datasets and compare with the challenge winners?\n\nPlease refer to our global response for the justification of choosing Endovis2017 and 2018 datasets. \n\nOur method actually compares with winning solutions, like TernausNet-11 [36] in Tab. 1 of the paper, which was that year's winner. Additionally, we research contemporary papers and compare with SOTA methods like S3Net [5] and MATIS [3], underscoring our approach's superiority.\n\n### [Rt9K-W2] Reference formats are not consistent.\n\nWe will correct it in our revised submission.\n\n### [Rt9K-Q1] Does the hard instrument area identified by the model align with the surgeons' perspective?\n\nReferring to the Fig. 2 in pdf file in global response section, we visualize the hard instrument areas and observe that these areas predominantly reside at the instrument's clasper and shaft positions (red rectangles).\nFor the clasper, due to its deep interaction with the tissue, making the image complex and the segmentation challenging.\nThe shaft, on the other hand, presents issues because different instruments often have similar shaft appearances, leading to model misinterpretations.\nAfter consulting with surgeons, they agreed that the clasper is the hard area to identify, aligning with our findings, but they didn't find the shaft as challenging.\nThe difference arises because the instrument clasper, influenced by factors like lighting, can more easily be mistaken for tissue.\nWhile for surgeons, the classification of the instrument shaft is inferred from the clasper, making the shaft a non-challenging area for them.\nHowever, for models, grasping the relationship between the clasper and shaft might not be as intuitive, leading to misclassification.\nThis observation suggests that future work should focus on modeling the relationship between the clasper and shaft to enhance segmentation performance across different parts.\n\n### [Rt9K-Q2] Typical failure modes and reasons.\n\nAs described in the introduction of the paper, surgical instrument segmentation faces two typical difficulties.\nThe first is the continual emergence of new instruments, necessitating frequent model retraining to accommodate these novelties.\nThe second pertains to misclassification when segmenting similar instruments and their boundaries.\nOur paper proposes solutions tailored to these challenges: To address the emergence of new surgical instruments, we pioneer the definition of surgical instrument segmentation in a text-promptable format, enabling the open-set segmentation. We also introduce the Mixture of Prompts (MoP) strategy to enhance the segmentation robustness. MoP addresses instrument similarity by integrating detailed text descriptions, hence enhancing the classification among instruments. Moreover, we introduce the Hard Instrument Area Reinforcement (HIAR) module that further improves the segmentation in areas where instruments and tissues often overlap. HIAR deepens the model's understanding of challenging regions while reducing confusion between similar instruments.\n\nDespite these proposed advancements, challenges persist to certain extent (see Fig. 9 & 10 in supplementary material).\nThe reason is manifold: the alignment between text and image features is not impeccable, and our current method treats different phrases within prompts uniformly, leading to potential identification challenges for new instruments.\nAdditionally, while HIAR alleviates mis-classification issues, it doesn't eradicate them entirely.\nSome ambiguous (hard) areas can be indistinguishable even to human eyes, requiring further exploration for a holistic solution. In the future, we will focus on refining text-image alignment methodologies, leveraging weighting mechanisms to accentuate pivotal distinctions in prompts across instruments, ultimately diminishing segmentation mis-classification, and bolstering instrument edge segmentation through advanced techniques.\n\n### [Rt9K-L1] It would be great to validate the method on CholecSeg8k.\n\nThanks for this suggestion. We have done it accordingly, please refer to the CholecSeg8k experiment section in the global response.\n"}, {"review_id": "Vu1EthPYeR", "reviewer": "Reviewer_9oei", "summary": "This paper proposes a novel text promptable surgical instrument segmentation approach to overcome challenges associated with the diversity and differentiation of surgical instruments by using the large CLIP model and a text promptable mask decoder. The experiments show the effectiveness of the proposed method on surgical instrument segmentation. However, I am very concerned with the novelty of the proposed modules due to the limited situations.", "questions": "1, How about comparing the proposed method and the Segment Anything model?\n\n2, What’s the difficulty of surgical instrument segmentation? I understand that there might be a lack of some segmentation datasets leading to overfitting.", "limitations": "N/A", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1, The paper is well-written and easy to understand.\n\n2, The experiments show the effectiveness of the proposed modules.", "weaknesses": "1, There are too few novelties due to most of the proposed modules being explored in the traditional segmentation tasks.\n\n2, The dataset is too limited to prove the effectiveness of the proposed modules. Furthermore, there is no specific module designed for surgical instrument segmentation.\n\n3, Lacking the comparison of inference time.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### [9oei-W1, W2] Few novelties and modules already explored in traditional segmentation.\n\nWe respectfully disagree.\nFirst, to our knowledge, we're the first to introduce text promptable method in surgical instrument segmentation.\nSecond, through problem-driven thinking, we proposed the Mixture of Prompts (MoP) and Hard Instrument Area Reinforcement (HIAR) modules tailored for specific challenges in this field.\nMoP addresses instrument similarity by integrating detailed text descriptions, hence enhancing the classification among instruments.\nHIAR improves segmentation in areas where instruments and tissues often overlap.\nBoth MoP and HIAR are inspired by advances in computer vision segmentation, but the concrete ideas of mixing the strength of multiple prompts and reinforcing hard regions in a masked auto-encoder are actually novel in the computer vision literature.\nFinally, while designed for this specific task, we believe our method's applicability also extends to other segmentation tasks, underscoring its efficacy.\n\n### [9oei-W2] The dataset is too limited.\n\nIt is a common practice to evaluate methods on the two established datasets, EndoVis 2017 and 2018 (see [36, 18, 11, 47, 5, 3]). To further validate our approach, we have added experimental results on EndoVis2019 and CholecSeg8k. Please refer to our global response section.\n\n### [9oei-W3] Lacking the comparison of inference time.\n\nWe assess the computational complexity and inference speed by evaluating the floating point operations per second (FLOPs) and frames per second (FPS) respectively, using a single A100 GPU.\nWe run the experiment on EndoVis2017 by resizing the input image to the default input sizes corresponding to different methods for testing (e.g. $800 \\times 800$ for ISINet, $224 \\times 224$ for MATIS, $416 \\times 416$ for CRIS, $448 \\times 448$ for CLIPSeg and Ours).\nFrom the table below, it's evident that our model's computational complexity (FLOPs) and inference speed (FPS) align with other adapted text-promptable approaches (i.e., CRIS and CLIPSeg), achieving real-time performance suitable for clinical applications. \nOn the other hand, compared to conventional segmentation methods (i.e., ISINet and MATIS), ours appears to be clearly more efficient than ISINet; while it is marginally slower than MATIS [3], likely due to MATIS's small input size.\n\n Method       | FLOPs (G) | FPS \n--------------|-----------|-----\n ISINet [11]  | 264       | 19  \n MATIS [3]    | 66        | 27  \n CRIS [39]    | 196       | 19  \n CLIPSeg [24] | 127       | 23  \n Ours         | 125       | 22  \n\n### [9oei-Q1] Compare with Segment Anything.\n\nWe value the recommendation to compare our method with the Segment Anything model (SAM), though it is only recently appeared.\nWhile SAM is proficient with various input prompts, its official released implementation only contains visual prompts (point, box and mask), lacks text prompting.\nHowever, visual prompts differ from our task setting, we aim to evaluate the performance of text-promptable SAM.\nWe leverage a community-based solution (lang-segment-anything from luca-medeiros), which enables this function for SAM.\nWe let SAM use the same set of prompts as we do and present the results in following tables.\nWe found that SAM struggles with medical prompts, performing significantly inferior than ours.\nThis suggests challenges with medical concepts without fine-tuning.\nAdditionally, we notice that the text encoder's output in this unofficial implementation of SAM might not align well with its visual encoder's output, potentially leading to decreased performance.\n\nSAM results on EndoVis2017:\n Method     | Ch_IoU | ISI_IoU | mc_IoU \n------------|--------|---------|--------\n SAM       | 17.77   | 14.32    | 10.28   \n Ours (448) | 77.79  | 76.45   | 54.78  \n\nSAM results on EndoVis2018:\n Method     | Ch_IoU | ISI_IoU | mc_IoU \n------------|--------|---------|--------\n SAM        | 22.08  | 17.88    | 12.33   \n Ours (448) | 82.67  | 81.54   | 65.48  \n\n### [9oei-Q2] What’s the difficulty of surgical instrument segmentation? Will the lack of segmentation datasets lead to overfitting?\n\nAs described in the introduction of the paper, surgical instrument segmentation faces two typical difficulties.\nThe first is the continual emergence of new instruments, necessitating frequent model retraining to accommodate these novelties.\nThe second pertains to misclassification during segmentation of similar instruments and their boundaries.\nOur paper proposes solutions tailored to these challenges: To address the emergence of new surgical instruments, we pioneer the definition of surgical instrument segmentation in a text-promptable format, enabling the open-set segmentation. We also introduce the Mixture of Prompts (MoP) strategy to enhance the segmentation robustness. MoP addresses instrument similarity by integrating detailed text descriptions, hence enhancing the classification among instruments. Moreover, we introduce the Hard Instrument Area Reinforcement (HIAR) module that further improves the segmentation in areas where instruments and tissues often overlap. HIAR deepens the model's understanding of challenging regions while reducing confusion between similar instruments.\n\n\nThe lack of surgical instrument segmentation datasets indeed poses an overfitting risk.\nA larger dataset could both enhance the model performance and its robustness.\nThis however is not a free lunch but comes with much more expensive annotation cost. \nThe cross-dataset experiments presented in Tabs. 1 & 2 of the paper demonstrate that our model can bypass this situation with its potential for open-set instrument segmentation.\nFor instance, when our model is trained on EndoVis2017, it can actually adeptly handle previously unseen classes, such as suction instrument (SI) in EndoVis2018 by utilizing only their text prompts without retraining.\n"}, {"review_id": "LQx7B2O3u4", "reviewer": "Reviewer_fFdZ", "summary": "This manuscript presents a clip-assisted semantic image segmentation method for surgical instruments. In terms of methodology, the proposed work can be viewed as an adaption of CLIPSeg to a domain-specific problem. Compared with CLIPSeg, a mixture of prompt strategy is used for augmenting text prompt information. Hard sample mining is also employed to further improve segmentation performance. The image segmentation network architecture has also been optimized. The proposed method is evaluated on two public surgical instrument segmentation datasets and is shown to outperform some fully-supervised and clip-assisted methods.\n\n**Commented after rebuttal**: I would like to thank the authors, other reviewers, and the ACs. I have carefully read through the rebuttals, and the comments from other reviewers. The discussion with the authors is constructive. Most of my raised concerns have been properly addressed.", "questions": "Judging by the ablation study (table 3 - 6), the contributions of individual components, except for the text encoder, seem to be quite marginal (the ablated models seem to be already quite strong). Therefore, from the authors' point of view, which factor accounts for most of the improved performance over CLIPSeg and CRIS? Or if there are other components that make a difference? Does the proposed method employ a stronger segmentation backbone or better training technique? The authors are encouraged to provide more details about the compared methods to avoid confusion.\n\nFrom the authors' perspective, what is the core take-home information to the readers?", "limitations": "Automated instrument image segmentation may affect the outcome of a surgery, which may lead to real-world impact to the clinicians and the patient. The authors are encouraged to discuss the potential impact of their work for patients, clinicians, device manufacturers, and the society. \n\nThe authors are also encouraged to discuss if the proposed method exacerbate/mitigate potential risks and biases in image segmentation.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The proposed method yields improved performance on two surgical instrument segmentation datasets, compared with some existing fully-supervised/clip-assisted methods.\n\nEmploying text prompt brings additional flexibility in terms of the categories of segmentation targets. This is a major advantage of clip-assisted segmentation over conventional fully-supervised segmentation.\n\nThe proposed method is introduced in sufficient detail and can be easy to follow.", "weaknesses": "The take-home information for readers may be limited/unclear. Despite that domain-specific adaptions on top of vanilla CLIPSeg leads to improved performance on surgical instrument segmentation, these contributions themselves may not be of sufficient interest to readers: hard sample mining, feature pyramid, generating multiple text prompts are already the common practice and should have been well-known for the community. The authors are encouraged to highlight their key take-home information to readers, and argue how the proposed work brings new knowledge.\n\nIn the abstract and introduction, the authors highlight improved flexibility when dealing with new segmentation targets of clip-assisted segmentation models. However, this point is not sufficiently evaluated and discussed in the experiments: how does CLIPSeg and CRIS that are also clip-driven work in this context?\n\nThe authors may also want to discuss/comment on a closely-related existing work [1].\n\n1. CLIP-Driven Universal Model for Organ Segmentation and Tumor Detection", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### [fFdZ-W1, Q2] Take-home information for readers.\n\nTo our knowledge, we're the first to introduce text promptable method in surgical instrument segmentation.\nThrough problem-driven thinking, we proposed Mixture of Prompts (MoP) and Hard Instrument Area Reinforcement (HIAR) modules tailored for specific challenges in this field.\nMoP addresses instrument similarity by integrating detailed text descriptions, hence enhancing the classification  among of instruments. \nHIAR improves segmentation in areas where instruments and tissues often overlap.\nBoth MoP and HIAR are inspired by advances in computer vision segmentation, but the concrete ideas of mixing the strength of multiple prompts and reinforcing hard regions in a masked auto-encoder are actually novel in the computer vision literature.\nFinally, while designed for this specific task, we believe our method's applicability also extends to other segmentation tasks, underscoring its efficacy.\n\n### [fFdZ-W2] The performance of CRIS and CLIPSeg for the cross dataset validation.\n\nFor comprehensive analysis, we've added comparisons against CRIS and CLIPSeg by cross dataset validation in following tables.\nIn the following tables, it can be seen that CRIS and CLIPSeg exhibit a significant drop in performance in cross-dataset validation compared to their corresponding counterparts and ours, indicating weaker generalization capabilities for both methods.\n\nCross dataset validation for CRIS and CLIPSeg on EndoVis2017:\n Method           | Ch_IoU | ISI_IoU | mc_IoU \n------------------|--------|---------|--------\n CRIS             | 69.94  | 67.83   | 38.21  \n Cross-CRIS       | 61.33  | 59.87   | 31.52  \n CLIPSeg          | 70.15  | 65.02   | 33.42  \n Cross-CLIPSeg    | 60.73  | 57.26   | 28.71  \n Ours       | 79.90  | 77.83   | 56.22  \n Cross-Ours | 72.18  | 70.44   | 49.09  \n\nCross dataset validation for CRIS and CLIPSeg on EndoVis2018:\n Method           | Ch_IoU | ISI_IoU | mc_IoU \n------------------|--------|---------|--------\n CRIS             | 74.10  | 72.29   | 46.04  \n Cross-CRIS       | 53.91  | 52.18   | 29.46  \n CLIPSeg          | 74.95  | 69.86   | 39.70  \n Cross-CLIPSeg    | 51.35  | 50.77   | 27.62  \n Ours       | 84.92  | 83.61   | 65.44  \n Cross-Ours | 66.25  | 64.92   | 37.34  \n\n### [fFdZ-W3] Discuss the Universal Model.\nThank you for pointing out the \"Universal Model\" paper.\nOur work primarily focuses on surgical instrument segmentation, while the Universal Model targets organ segmentation and tumor detection, thus differing in their research domains.\nThis is why it was not initially cited or discussed in our manuscript.\nHowever, consistent with our approach, the paper also leverages the CLIP vision-language model to enhance the performance of medical image segmentation models.\nWe will cite this work in our paper and discuss it in the appropriate sections.\n\n### [fFdZ-Q1] More explanation for ablation study.\n\nIn Tabs. 3-6 in the paper, we present ablation study results.\nFrom our final model, we remove each single module to observe performance changes.\nNotably, without multi-scale feature augmentation (MSFA), there's a 2.4% performance drop; without Mixture of Prompts (MoP), the drop is 2.1%; and without Hard Instrument Area Reinforcement (HIAR), it's 2.5%.\nThese aren't marginal but significant if we compare the performance increase between the previous two SOTA (S3Net [5] and MATIS [3] in Tab. 1 of the paper). \n\nMoreover, our modules are complementary, omitting both MoP and HIAR leads to a 3.9% drop.\nRemoving MSFA, MoP, and HIAR all together results in a 5.2% decrease; this, as our baseline, is still 2.6% higher than CRIS (Ch_IoU result in Tab. 1 in our paper).\nIt is because our baseline benefits from data augmentation strategies (random crop, horizon flip, random rotate) and employs a ViT-based CLIP model trained on Laion2B.\n\nWe will elaborate on these details in the revised paper to avoid ambiguity.\n\n### [fFdZ-L1] The potential impact for patients, clinicians, device manufacturers, and the society.\n\nEnhanced automated surgical instrument segmentation offers manifold benefits to patients, surgeons, manufacturers, and society.\nPrecise instrument tracking boosts surgical safety and precision, mitigating unintended tissue damage.\nSuch automation eases surgeons' tasks, letting them concentrate on intricate procedures, and augments training for novices.\nManufacturers integrating this technology could produce smarter surgical tools, aligning with AI-driven healthcare innovations to elevate patient care.\n\n### [fFdZ-L2] If the proposed method exacerbates/mitigates potential risks and biases in image segmentation.\n\nModel bias due to non-representative distributions of gender or race in the training data does not apply to instrument segmentation datasets.\nNevertheless, there could still be a human-driven bias in surgical instrument segmentation training data due to surgeons' preference for specific surgical instruments in certain scenarios over other tools.\nThis bias can be mitigated by our text-promptable approach, which allows for using text prompts that are aimed at mitigating these scenarios (e.g. providing specific surgical scenario/function descriptions for certain surgical instrument).   \n\nThe study into the risk and biases of our method differs from this research's current focus; hence we leave further investigations for future works. \n"}]

Author rebuttal:
We thank the constructive comments from all reviewers.
As reviewers say, our key idea of using textural prompts to perform surgical instrument segmentation is novel (wnjR, NUuZ) and interesting (NUuZ). Our paper is overall well organized (Rt9K) and easy to understand (9oei).
Below we first answer the common question raised by several reviewers on our experimental datasets; next, we address the concerns from each reviewer separately.  

## Experiments on more datasets

Several reviewers point out the need to validate our method on more datasets.
We deeply value your comments.
First, we would like to emphasize that it is a common practice [36, 18, 11, 47, 5, 3] to evaluate the surgical instrument segmentation performance on the two established datasets, EndoVis 2017 [1] and 2018 [2].
We followed this practice and should be not disadvantaged.
Next, we delineate the consideration for selecting additional datasets, and then report our results on these datasets.

### Dataset selection
Although the EndoVis Challenge is an annual event, datasets specific to surgical instrument segmentation aren't released every year.
Specifically, the EndoVis2019's Robust-MIS dataset merely differentiates between tissues and instruments, not aligning with our study's focus on segmenting different instrument types. 
Moreover, the instance segmentation in EndoVis2019 also does not align with our paper's problem.
The challenges in EndoVis2020 and 2021 do not address instrument segmentation either.
Regarding the datasets for EndoVis2022 and 2023 challenges, namely SAR-RARP50 and SIMS, their usage in publications is restricted until the release of their respective competition reports.
Given the SAR-RARP50 report's pending release and the ongoing SIMS competition, these datasets are not incorporated into our research.

To further substantiate our approach, we choose to conduct experiments on the instrument binary segmentation task on the EndoVis2019 dataset [R1], as well as instrument and tissue segmentation task on the CholecSeg8k dataset [R2].

### Comparison to state of the art

1. **EndoVis2019**: Consistent with the competition's evaluation protocol [R1], we use the Dice Similarity Coefficient (DSC) and Normalized Surface Dice (NSD) to assess the segmentation performance.
As the following table shows, our method (input size 448) notably surpasses the competition's top performers, with +3% increase in DSC and +2% enhancement in NSD, which demonstrates the superiority of our method. It's worth noting that our approach is designed for multi-class segmentation while is tested for binary class segmentation. Despite this, the performance improvement by ours over SOTA underscores its efficacy.

 Method         | DSC  | NSD  
----------------|------|------
 haoyun [R1]    | 0.89 | 0.89 
 CASIA-SRL [R1] | 0.78 | 0.89 
 Ours (448)           | 0.92 | 0.91 

2. **CholecSeg8k**: Following the protocols from SP-TCN [R4], we split the dataset into training and testing sets (videos 12, 20, 48 and 55 for testing and others for training) and utilize the mean Intersection over Union (mean IoU) as the evaluation metric.
In the following table, the result of our method is 71.03% mean IoU.
It's evident that our method surpasses the current SOTA (SP-TCN) by 1.65% in mean IoU, even though SP-TCN leverages temporal information from videos to boost the performance, while our method solely relies on individual image data.
It's worth noting that for the CholecSeg8k, we use the same prompt generation method described in our paper to obtain prompts for both tissues and instruments. The result demonstrates that prompts for tissues are appropriately generated following our method, further attesting to the generalizability of our method.

 Method                  | mean IoU 
-------------------------|----------
 Swin base [R3]          | 0.6842   
 Swin base + SP-TCN [R4] | 0.6938   
 Ours (448)                    | 0.7103   

## References
[R1] Ross, Tobias, et al. "Robust medical instrument segmentation challenge 2019." arXiv preprint arXiv:2003.10299 (2020).

[R2] Hong, W-Y., et al. "Cholecseg8k: a semantic segmentation dataset for laparoscopic cholecystectomy based on cholec80." arXiv preprint arXiv:2012.12453 (2020).

[R3] Liu, Ze, et al. "Swin transformer: Hierarchical vision transformer using shifted windows." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2021.

[R4] Grammatikopoulou, Maria, et al. "A spatio-temporal network for video semantic segmentation in surgical videos." International Journal of Computer Assisted Radiology and Surgery (2023): 1-8.


Meta-review: After rebuttal, all reviewers land on the positive side. The authors well addressed the main concerns raised by reviewers, e.g., the lack of experiments on more challenging datasets. Thus, the AC recommends acceptance. The authors should include the discussion and new results in the rebuttal into the final version. Besides, the authors should release the code as they promised in the rebuttal.
Decision: Accept (poster)