SlowGuess/NeurIPS_2023_OCR huggingface dataset:
For  $\mathcal{L}^{rec}$ , denoting by the  $I^{\mathrm{rec}}$  and  $I$  the reconstructed image and original image, we employ the L2 loss to minimize their pixel-wise distance:
$$
\mathcal {L} ^ {\text {r e c}} = \sum_ {j} \| I _ {j} ^ {\text {r e c}} - I _ {j} \| ^ {2} \tag {3}
$$
where  $I_{j}^{\mathrm{rec}}$  and  $I_{j}$  represent the reconstructed and original pixel values at  $j$ -th location, respectively.
The overall loss function is given by  $\mathcal{L} = \mathcal{L}^{seg} + \lambda \mathcal{L}^{rec}$ ,  $\lambda$  serves as the loss weight.
Table 1: Comparison between our method and other state-of-the-art methods on the EndoVis2017 dataset. Methods in the first (top) group are conventional supervised methods with predefined categories; methods in the second group (bottom) are text promptable using vision-language models. Cross-Ours in the last row represents the cross-dataset experiment.
Method	Ch_IoU	ISI_IoU	BF	PF	LND	VS	GR	MCS	UP	mc_IoU
TernausNet-11 [43]	35.27	12.67	13.45	12.39	20.51	5.97	1.08	1.00	16.76	10.17
MF-TAPNet [20]	37.35	13.49	16.39	14.11	19.01	8.11	0.31	4.09	13.40	10.77
ISINet [12]	55.62	52.20	38.70	38.50	50.09	27.43	2.01	28.72	12.56	28.96
TraSeTR [55]	60.40	65.20	45.20	56.70	55.80	38.90	11.40	31.3	18.20	36.79
S3Net [5]	72.54	71.99	75.08	54.32	61.84	35.5	27.47	43.23	28.38	46.55
MATIS [3]	71.36	66.28	68.37	53.26	53.55	31.89	27.34	21.34	26.53	41.09
CRIS [46]	69.94	67.83	54.87	50.21	68.33	50.12	0.00	43.97	0.00	38.21
CLIPSeg [29]	70.15	65.02	51.29	42.27	49.56	30.12	9.96	30.69	20.05	33.42
Ours (448)	77.79	76.45	69.57	68.91	89.88	82.60	0.00	72.53	0.00	54.78
Ours (896)	79.90	77.83	68.58	73.52	92.74	83.90	0.13	74.70	0.00	56.22
Cross-Ours (896)	72.18	70.44	65.54	58.19	84.01	67.30	0.06	68.47	0.06	49.09
Table 2: Comparison between our method and other state-of-the-art methods on the EndoVis2018 dataset.
Method	Ch_IoU	ISI_IoU	BF	PF	LND	SI	CA	MCS	UP	mc_IoU
TernausNet-11 [43]	46.22	39.87	44.20	4.67	0.00	0.00	0.00	50.44	0.00	14.19
MF-TAPNet [20]	67.87	39.14	69.23	6.10	11.68	14.00	0.91	70.24	0.57	24.68
ISINet [12]	73.03	70.97	73.83	48.61	30.98	37.68	0.00	88.16	2.16	40.21
TraSeTR [55]	76.20	-	76.30	53.30	46.50	40.60	13.90	86.30	17.50	47.77
S3Net [5]	75.81	74.02	77.22	50.87	19.83	50.59	0.00	92.12	7.44	42.58
MATIS [3]	84.26	79.12	83.52	41.90	66.18	70.57	0.00	92.96	23.13	54.04
CRIS [46]	74.10	72.29	73.58	58.20	47.64	72.14	4.56	45.99	20.18	46.04
CLIPSeg [29]	74.95	69.86	67.25	39.59	36.72	47.27	2.92	79.96	4.22	39.7
Ours (448)	82.67	81.54	81.53	70.18	71.54	90.58	21.46	65.57	57.51	65.48
Ours (896)	84.92	83.61	84.28	73.18	78.88	92.20	23.73	66.67	39.12	65.44
Cross-Ours (896)	66.25	64.92	65.81	56.12	44.72	79.77	1.22	8.97	4.77	37.34
Datasets We evaluate our method on two endoscopic surgical instrument segmentation datasets: EndoVis2017 [1], EndoVis2018 [2]. EndoVis2017 includes 10 videos from the da Vinci robotic system, containing 6 distinct surgical instrument types (bipolar forceps, prograsp forceps, large needle driver, vessel sealer, grasping retractor, monopolar curved scissors) and an ultrasonic probe (classified as other instruments). Following [43], we employ 4-fold cross-validation to assess the model performance. For EndoVis2018, we adopt the widely-used labeling and dataset partitioning method proposed in [12]. This dataset consists of 15 video sequences, with 11 training and 4 testing sequences, and 7 predefined instrument categories (bipolar forceps, prograsp forceps, large needle driver, monopolar curved scissors, ultrasound probe, suction instrument, clip applicer). Notably, two categories (suction instrument and clip applicer) differ from those in EndoVis2017 (vessel sealer and grasping retractor), enabling cross-category experiments to evaluate our model's generalizability. Additionally, both datasets provide binary and parts segmentation labels. Binary segmentation comprises background tissue and instruments, while parts segmentation distinguishes instrument components as shaft, wrist, and claspers. Besides the two datasets, we have also evaluated our method on the EndoVis2019 [39] and Cholecseg8k [17] datasets in the supplementary material.
Metrics Adopting the evaluation method from [12], we utilize three prevalent IoU-based evaluation metrics: Ch_IoU, ISI_IoU, and mc_IoU. 1) Ch_IoU computes the mean IoU for each category present in the ground truth of an image, then averages them across all image. 2) ISI_IoU extends Ch_IoU by computing mean IoUs for all predicted categorizes regardless of their presence in the ground truth of the image. Ch_IoU is normally greater than or equal to ISI_IoU. 3) mc_IoU is a measurement addressing category imbalance by changing the averaging order in ISI_IoU.
4.2 Implement Details
Vision-Language model. We employ the image and text encoders from the widely-used CLIP model [19], specifically the ViT-B-16 variant pre-trained on the Laion2B [40] dataset, to construct
our promptable surgical instrument segmentation model. The image encoder has a patch size of 16  $(p = 16)$  and the text encoder has a token length limit of 77.
Hyper-parameters. We offer two training/inference default image sizes,  $896 \times 896$  and  $448 \times 448$ , which are compatible with the size requirements for image patching in the ViT-based image encoder [8]. During evaluation, we restore the segmentation prediction to the original image size. Feature dimension  $D$  is 1024. Following [46], we employ a threshold  $\theta = 0.35$  to transform the score map  $S$  into mask  $M$ , and select the highest-scoring category per pixel in multi-category cases. For hard instrument area reinforcement, we set mask ratio  $r$  to 0.25. Finally, the loss weight  $\lambda$  is 0.5. All hyperparameters are determined empirically by segregating  $20\%$  of the training data as a validation set following [46].
Training. In our experiments, we adopt the Adam [23] optimizer with a learning rate of  $1e^{-4}$ . We train for 50 epochs, reducing the learning rate to  $1e^{-\frac{1}{5}}$  at the 35-th epoch. To enhance the model's generalization, we apply data augmentation techniques to the image, including random crop, horizontal flip, random rotation, and brightness perturbation. The model is trained on 4 V100 GPUs, the batch size is 16.
Comparison to state of the art. In Table 1 and Table 2, we compare our method with a series of state of the art methods on the EndoVis2017 and EndoVis2018 datasets, respectively. The methods can be divided into two groups depending on whether they have utilized textural prompts for surgical instruments. Ours belongs to the second group. As demonstrated by the results, our method significantly surpasses the state of the art in the first group: e.g. on the EndoVis2017 dataset,  $+7.36\%$  on Ch_IoU,  $+5.84\%$  in ISI_IoU, and  $+9.67\%$  in mc_IoU. Particularly, a smaller difference between Ch_IoU and ISI_IoU for our method indicates it has fewer misclassified categories compared to other methods. Next, comparing to methods in second group, which are originally designed for natural image segmentation, our method yields clearly superior results, validating the effectiveness of our specifically tailored modules for surgical instrument segmentation. Notably, we implement both CRIS and CLIPSeg using their open-source implementations with batch size 64 for training.
Comparison to SAM. Whilst we were working on our work, the segment anything model (SAM) [24] was released concurrently. SAM possesses the capability to output corresponding object masks through visual or text prompts. Its officially released implementation however only contains visual prompts (point, box and mask), lacks text prompting. We thus leverage a community-based solution, lang-segment-anything [32], which enables this function for SAM. We found that this SAM variant struggles with medical prompts (e.g. the Ch_IoU is only  $17.77\%$  and  $22.08\%$  on EndoVis2018 and EndoVis2017, respectively), performing significantly inferior than ours. For more details, we refer readers to the supplementary material.
Cross-dataset experiment. We also evaluate our method in the cross-dataset setting by training it on EndoVis2018 and testing it on EndoVis2017, and vice versa. As depicted in Tables 1 and Table 2 (i.e. Cross-Ours), our method still gets competitive results compared with the state of the art on EndoVis2017, and is on par with state of the art on EndoVis2018, attesting its good generalizability. Note that categories between EndoVis2017 and EndoVis2018 do not fully overlap, the exceptions are the VS and SI categories. Despite this, we achieve very competitive results on them, showcasing the potential capability of our method for open-set text promptable surgical instrument segmentation.
We conduct the ablation study on the EndoVis2017 dataset using images of sizes  $448 \times 448$  and Ch_IoU and ISI_IoU as the evaluative metrics.
Image encoder: multi-scale feature augmentation. We ablate the proposed multi-scale feature augmentation (MSFA) in our method and offer the result in Table 3. A clear performance drop can be observed for ours w/o MSFA. MSFA enhances the contextual information for feature representation, thus improving the segmentation performance.
Text encoder: global feature from [CLS] token. We offer an variant by averaging the features from individual word tokens to produce the textural feature  $F_{T}$ . The result is in Table 3: one can see that [CLS] → Words decreases Ch_IoU and ISI_IoU scores by 0.46% and 0.48% respectively. This suggests that the [CLS] token can better encapsulate the global information of the text description.
Table 3: Ablation study for image and text encoder.
Encoder	Ch_IoU	ISI_IoU
Ours	77.79	76.45
Ours w/o MSFA	75.31	74.03
Ours ([CLS]→Words)	77.33	75.97
Table 5: Ablation study for mixture of prompts;cls-class,tem - template.
MoP	Ch_IoU	ISI_IoU
Ours w/ P-cls	75.67	74.34
Ours w/ P-tem	75.86	74.39
Ours w/ P-GPT	77.19	76.09
Ours w/ P-cls & tem	76.01	74.53
Ours w/ P-cls & tem & GPT	77.79	76.45
Ours (P-cls → P-Bard)	77.76	76.44
Table 7: Ablation study for pixel-wise weight maps in mixture of prompts.
Gating network	Ch_IoU	ISI_IoU
Pixel-wise	77.79	76.45
Image-wise	76.80	75.77
Table 4: Ablation study for text promptable mask decoder.
Decoder	Ch_IoU	ISI_IoU
Ours	77.79	76.45
Ours w/ AP	76.46	75.28
Ours w/ CP	42.38	37.61
Table 6: Ablation study for hard instrument area reinforcement.
HIAR	Ch_IoU	ISI_IoU
Ours	77.79	76.45
Ours w/o HIAR	75.22	73.98
Ours w/o HAM	75.98	74.57
rt=0.1	77.03	76.01
rt=0.25	77.79	76.45
rt=0.50	77.57	76.32
Table 8: Ablation study for mask ratio threshold  $r_t$  in hard instrument area reinforcement.
mask ratio rt	Ch_IoU	ISI_IoU
0.25	77.79	76.45
0.75	76.92	75.65
Text promptable mask decoder: prompting schemes. We study the importance of the two prompting schemes in Table 4. Results indicate that using only attention-based prompting (Ours w/ AP) outperforms using only convolutional-based prompting (Ours w/ CP) in terms of Ch_IoU and ISI_IoU scores. Moreover, employing both prompting schemes (Ours) sequentially improves the segmentation performance, suggesting that convolution-based prompting can further refine the output of the attention-based approach.
Mixture of prompts: different prompts. We offer the ablation study of MoP in Table 5. Results reveal that 1) using a single and simple prompt (e.g. Ours w/ P-cls) already performs very well, outperforming the state of the art in Table 1; 2) the proposed mixture of prompts can further improve the segmentation performance. Over the three prompt types, the one from GPT-4 performs the best, owing to its more comprehensive coverage of the knowledge for certain instrument. We have also tried to replace the simple class prompt with prompt generated from another large language model, Bard [13]. Yet, we do not observe significant performance difference, cf. Ours  $(\mathrm{P -cls}\rightarrow \mathrm{P - Bard})$ . We suspect the reason is that the knowledge contained in the GPT-4 prompt and the Bard prompt highly overlap, given the input token length limit. Note that prompts generated by GPT-4 might slightly vary given different time stamps. We have tried to study this factor, yet find very trivial impact of it.
Mixture of prompts: pixel-wise weight maps. To confirm the need for pixel-wise weight maps in the proposed visual-textual gating network, we develop a simpler version by using the image-wise scalar weight following the design in MoE [6]. These scalar weights are normalized via softmax. Experiments in Table 7 show our pixel-wise weighting scheme is better than the image-wise variant.
Hard instrument area reinforcement: modules and parameters. If we remove this module from our method (Ours w/o HIAR), as shown in Table 6, the performance clearly drops (e.g.  $2.57\%$  and  $2.47\%$ ). Next, we highlight the efficacy of the hard area mining (HAM) by removing it from HIAR, i.e. Ours w/o HAM; the HIAR module is then rather similar to a typical MAE module. We also observe a clear performance drop, underlining the crucial role of HAM in HIAR. Finally, we vary the making ratio threshold  $r_t$  over 0.1, 0.25, and 0.5 in Table 6: we can see  $r_t = 0.25$  works the best.
Hard instrument area reinforcement: mask ratio  $r_t$ . MAE [15] originally employs a high masking ratio (0.75), but in our study, we argue that a lower threshold (0.25) is more suitable: surgical instruments occupy a relatively small portion of the image, a low masking ratio will make the model concentrating on the hard area. Table 8 reveals a performance decrease with a masking ratio threshold of 0.75 when compared to our default setting of 0.25.
Figure 3: Our segmentation visualization uses distinct colors to represent surgical instrument categories, with ground truth and predictions displayed in the first and second rows, respectively.
4.5 Computational Analysis
We perform experiments to assess computational complexity and inference speed in Tab. 9. This is achieved by evaluating floating point operations per second (FLOPs) and frames per second (FPS) using a single A100 GPU. We run the experiment on EndoVis2017 [1] by resizing the input image to the default input sizes corresponding to different methods for testing (e.g.  $800 \times 800$  for ISINet,  $224 \times 224$  for MATIS,  $416 \times 416$  for CRIS,  $448 \times 448$  for CLIPSeg and Ours). In Tab. 9, it is evident that our model's computational complexity (FLOPs) and inference speed (FPS) aligns with other adapted text-promptable approaches (i.e., CRIS [46] and CLIPSeg [29]), achieving real-time performance suitable for clinical applications. Compared to conventional segmentation methods (i.e., ISINet [12] and MATIS [3]), ours appears to be clearly more efficient than ISINet; while it is marginally slower than MATIS, likely due to MATIS having lower complexity (FLOPs).
Table 9: Computational analysis between ours and representative works on the EndoVis2017.
Method	FLOPs (G)	FPS
ISINet [12]	264	19
MATIS [3]	66	27
CRIS [46]	196	19
CLIPSeg [29]	127	23
Ours (448)	125	22
In this paper, we introduce a novel approach for text promptable surgical instrument segmentation leveraging pretrained vision-language model. Our method addresses the challenge of instrument variety by redefining the task as a text promptable segmentation problem, which enhances the model's adaptability to new surgical instruments. Our promptable mask decoder carefully segments the surgical instruments in images via an attention-based and convolution-based prompting schemes. Our mixture of prompts mechanism effectively employs diverse prompts for better segmentation performance. While our hard instrument area reinforcement module intertwined with hard area mining significantly improves the segmentation performance on challenging area. Experimental results demonstrate our model's superior performance over state of the art. Future work will aim to further exploit the potential of this method for real-world surgical scenario.
6 Social Impact
Automated surgical instrument segmentation offers benefits to patients, surgeons, manufacturers, and society. Precise instrument tracking boosts surgical safety and precision, mitigating unintended tissue damage. Such automation eases surgeons' tasks, letting them concentrate on intricate procedures, and augments training for novices. Manufacturers integrating this technology could produce smarter surgical tools, aligning with AI-driven healthcare innovations to elevate patient care.
This work was supported by King's Cambridge 1 Access Fund, Tongji Fundamental Research Funds for the Central Universities, and core funding from the Wellcome/EPSRC [WT203148/Z/16/Z; NS/A000049/1]. Computing resources provided by King's Computational Research, Engineering and Technology Environment (CREATE). For the purpose of open access, the authors have applied a CC BY public copyright licence to any Author Accepted Manuscript version arising from this submission. TV is supported by a Medtronic / RAEng Research Chair [RCSRF1819\7\34], and is co-founder and shareholder of Hypervision Surgical. MW and OA are supported by the EPSRC CDT [EP/S022104/1].
[1] Max Allan, Alex Shvets, Thomas Kurmann, Zichen Zhang, Rahul Duggal, Yun-Hsuan Su, Nicola Rieke, Iro Laina, Niveditha Kalavakonda, Sebastian Bodenstedt, et al. 2017 robotic instrument segmentation challenge. arXiv preprint, 2019.
[2] Max Allan, Satoshi Kondo, Sebastian Bodenstedt, Stefan Leger, Rahim Kadkhodamohammadi, Imanol Luengo, Felix Fuentes, Evangello Flouty, Ahmed Mohammed, Marius Pedersen, et al. 2018 robotic scene segmentation challenge. arXiv preprint, 2020.
[3] Nicolas Ayobi, Alejandra Pérez-Rondón, Santiago Rodríguez, and Pablo Arbeláez. MATIS: Masked-attention transformers for surgical instrument segmentation. arXiv preprint, 2023.
[4] Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint, 2016.
[5] Britty Baby, Daksh Thapar, Mustafa Chasmai, Tamajit Banerjee, Kunal Dargan, Ashish Suri, Subhashis Banerjee, and Chetan Arora. From forks to forceps: A new framework for instance segmentation of surgical instruments. In WACV, 2023.
[6] Zixiang Chen, Yihe Deng, Yue Wu, Quanquan Gu, and Yuanzhi Li. Towards understanding mixture of experts in deep learning. arXiv preprint, 2022.
[7] Henghui Ding, Chang Liu, Suchen Wang, and Xudong Jiang. Vision-language transformer and query generation for referring segmentation. In ICCV, 2021.
[8] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In ICLR, 2021.
[9] Yu Du, Fangyun Wei, Zihe Zhang, Miaojing Shi, Yue Gao, and Guoqi Li. Learning to prompt for open-vocabulary object detection with vision-language model. In CVPR, 2022.
[10] Han Fang, Pengfei Xiong, Luhui Xu, and Yu Chen. Clip2video: Mastering video-text retrieval via image clip. arXiv preprint, 2021.
[11] Guang Feng, Zhiwei Hu, Lihe Zhang, and Hutchuan Lu. Encoder fusion network with co-attention embedding for referring image segmentation. In CVPR, 2021.
[12] Cristina González, Laura Bravo-Sánchez, and Pablo Arbelaez. Isinet: an instance-based approach for surgical instrument segmentation. In MICCAI, 2020.
[13] Google. Google bard, 2023.
[14] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016.
[15] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dólár, and Ross B. Girshick. Masked autoencoders are scalable vision learners. In CVPR, 2022.
[16] HS Himal. Minimally invasive (laparoscopic) surgery. Surgical Endoscopy And Other Interventional Techniques, pages 1647-1652, 2002.
[17] W-Y Hong, C-L Kao, Y-H Kuo, J-R Wang, W-L Chang, and C-S Shih. Cholecseg8k: a semantic segmentation dataset for laparoscopic cholecystectomy based on cholec80. arXiv preprint, 2020.