SlowGuess/NeurIPS_2023_OCR huggingface dataset:
To approximate the gradients required for backpropagation, we adopt IMLE [50, 58]. Let  $\pmb{\theta} = (\pmb{c}^v, \pmb{c}^e)$  be the input to the combinatorial graph matching problem in Eq. (4). The core idea of IMLE is to define a probability distribution  $\rho(\boldsymbol{v}; \pmb{\theta})$  over solutions of the combinatorial optimization problem, where the probability of a solution is proportional to its negative cost, and to estimate  $\partial L / \partial \pmb{\theta}$  through the gradients of the expectation  $\nabla_{\pmb{\theta}} \mathbb{E}_{\hat{\boldsymbol{v}} \sim \rho(\boldsymbol{v}; \pmb{\theta})} [L(\hat{\boldsymbol{v}}, \boldsymbol{v}^*)]$ . Since exact sampling from  $\rho(\boldsymbol{v}; \pmb{\theta})$  is typically intractable, IMLE instead chooses a noise distribution  $\rho(\epsilon)$  and approximates the gradient of the expectation over  $\rho(\boldsymbol{v}; \pmb{\theta})$  with the gradient of the expectation over  $\rho(\epsilon)$
$$
\nabla_ {\boldsymbol {\theta}} \mathbb {E} _ {\hat {\boldsymbol {v}} \sim \rho (\boldsymbol {v}; \boldsymbol {\theta})} [ L (\hat {\boldsymbol {v}}, \boldsymbol {v} ^ {*}) ] \approx \nabla_ {\boldsymbol {\theta}} \mathbb {E} _ {\epsilon \sim \rho (\epsilon)} [ L (\mathrm {G M} (\boldsymbol {\theta} + \epsilon), \boldsymbol {v} ^ {*}) ].
$$
The above approximation invokes the reparameterization trick for a complex discrete distribution. A typical choice for  $\rho (\epsilon)$  is the Gumbel distribution, that is,  $\rho (\epsilon)\sim \mathrm{Gumbel}(0,1)$  [59]. Now, by using a finite-difference approximation of the derivative in the direction of the gradient of the loss  $\nabla_{\tilde{\boldsymbol{v}}}L(\tilde{\boldsymbol{v}},\boldsymbol{v}^{*})$ , we obtain the following estimation rule:
$$
\nabla_ {\boldsymbol {\theta}} \mathbb {E} _ {\tilde {\boldsymbol {v}} \sim p (\boldsymbol {v}; \boldsymbol {\theta})} \left[ L \left(\hat {\boldsymbol {v}}, \boldsymbol {v} ^ {*}\right) \right] \approx \mathbb {E} _ {\epsilon \sim \rho (\epsilon)} \left[ \frac {1}{\lambda} \left\{\tilde {\boldsymbol {v}} - \mathrm {G M} \left(\boldsymbol {\theta} + \epsilon - \lambda \nabla_ {\tilde {\boldsymbol {v}}} L \left(\tilde {\boldsymbol {v}}, \boldsymbol {v} ^ {*}\right)\right) \right\} \right], \tag {6}
$$
Algorithm 1 Forward and Backward Pass for  $c^v$ ,  $c^e$
function FORWARDPASS  $(c^v, c^e)$
// Gumbel noise distribution sampling
$\epsilon ,\epsilon^{\prime}\sim \mathrm{Gumbel}(0,1)$
// Graph-matching with perturbed  $(c^v, c^e)$
$\tilde{\pmb{v}} = \mathbf{GM}\left(\pmb {c}^{\upsilon} + \epsilon ,\pmb{c}^{e} + \epsilon^{\prime}\right)$
// Save values for the backward pass
save  $(\pmb {c}^v,\pmb {c}^e)$ $(\epsilon ,\epsilon^{\prime})$  and  $\tilde{v}$  return  $\tilde{\pmb{v}}$
function BACKWARDPASS  $(\nabla_{\tilde{v}}L(\tilde{v},v^{*}),\lambda)$
load  $(\pmb{c}^v,\pmb{c}^e),(\epsilon ,\epsilon ')$  and  $\tilde{\pmb{v}}$
// Add gradient-based perturbations
$(\pmb{c}_{\lambda}^{v}, \pmb{c}_{\lambda}^{e}) = (\pmb{c}^{v} + \epsilon, \pmb{c}^{e} + \epsilon') - \lambda \nabla_{\tilde{\pmb{v}}} L(\tilde{\pmb{v}}, \pmb{v}^{*})$
// Single sample gradient estimate
$\left(\frac{\partial L}{\partial \boldsymbol{c}^v}, \frac{\partial L}{\partial \boldsymbol{c}^e}\right) = \tilde{\boldsymbol{v}} - \mathsf{GM}(\boldsymbol{c}_\lambda^v, \boldsymbol{c}_\lambda^e)$
return  $\frac{1}{\lambda}\left(\frac{\partial L}{\partial c^v},\frac{\partial L}{\partial c^e}\right)$
where  $\tilde{\pmb{v}} = \mathsf{GM}(\pmb {\theta} + \pmb {\epsilon})$ ,  $\lambda$  is a step size of finite difference approximation. Using a Monte Carlo approximation of the above expectation, the gradient for  $\pmb{\theta}$  is computed as a difference of two or more pairs of perturbed graph-matching outputs. We summarize in Algorithm 1 the forward and backward steps for  $c^v, c^e$ .
Pre-training We utilize Resnet50 [60] and Vision Transformer (ViT-B/16) [61] to train our LVM-Med. For Resnet50, we load pre-trained from ImageNet-1K [62], and SAM Encoder backbone weight [10] for ViT. The raw image is augmented to two different views by using multi-crop techniques as [14] and followed by flip (probability  $50\%$ ), color jitter, random Gaussian blur, and normalization. We trained the LVM-Med with 100 epochs on the collected dataset. The batch size of 3200 is used for ResNet50 and we reduced it to 2800 for ViT due to memory limitation. The model is optimized with Adam [63] with an initial learning rate  $2 \times 10^{-3}$  and reduced halved four times. We use 16 A100-GPUs per with 80GB and complete the training process for LVM-Med with ResNet-50 in five days and LVM-Med with ViT encoder in seven days. Other competitor SSL methods as VicRegl, Twin-Barlon, Dino, etc, are initialized from ResNet-50 pre-trained ImageNet-1K and trained with 100 epochs with default settings as LVM-Med.
To balance samples among different modalities, we combine over-sampling and data augmentation to increase the total samples. Specifically, new samples from minority classes are generated by duplicating images and applying random crop operations covering  $85 - 95\%$  of image regions and then rescaling them to the original resolutions. Note that these augmentations are not used in the self-supervised algorithm (operations  $s, t \sim T$ ) to avoid generating identical distorted versions in this sampling procedure.
Table 1: Summary of datasets and downstream tasks
Evaluation	Downstream Task Data	Modality	Nums	Task
Fine-Tuning	BraTS2018 [64]	3D MRI	285	Tumor Segmentation
Fine-Tuning	MMWHS-CT [65]	3D CT	20	Heart Structures Segmentation
Fine-Tuning	MMWHS-MRI [65]	3D MRI	30	Heart Structures Segmentation
Fine-Tuning	ISIC-2018 [66]	2D Dermoscopy	2596	Skin Leision Segmentation
Fine-Tuning	JSRT [67]	2D X-ray	247	Multi-Organ Segmentation
Fine-Tuning	KvaSir [68]	2D Endoscope	1000	Polyp Segmentation & Detection
Fine-Tuning	Drive [69]	Fundus	40	Vessel Segmentation
Fine-Tuning	BUID [70]	2D Ultrasound	647	Breast Cancer Segmentation
Linear Evaluation & Fine-Tuning	FGADR [71]	Fundus	1841	DR Grading
Linear Evaluation & Fine-Tuning	Brain Tumor Classification	2D MRI	3264	Brain Tumor Classification
Fine-Tuning	Multi-site Prostate MRI Segmentation [72]	3D MRI	116	Prostate Segmentation
Fine-Tuning	VinDr [73]	2D X-ray	18000	Lung Diseases Detection
Figure 3: FGADR performance with top architectures.
**Downstream Tasks** Table 1 lists the datasets and downstream tasks used in our experiments. We cover segmentation, object detection, and image classification problems. It is important to note that in most settings, we utilize simple configurations for all datasets, skipping extra pre-processing for data augmentation. For instance, overlapping image patches with stride operations in the original samples [74] to increase training data in the Drive dataset or combining different 3D MRI modalities to fuse information [75] in the BRATS-2018 are excluded in our downstream setups.
To validate LVM-Med algorithms, we compare with 2D-SSL methods trained in our dataset and foundation models like Clip [3], Align [4], Flava [5], and SAM [6] with pre-trained ViT (Bert for
Align) taken from each method, respectively. During the downstream task, trained SSL weights are then extracted and attached in U-Net for ResNet50 backbone, TransUnet [76] for ViT, and then fine-tuned with training splits of each dataset. Depending on the downstream task's properties, we apply different image resolutions and other parameters like the number of epochs and learning rate for different data domains. Details for these configurations are presented in Appendix.
4.2 2D- and 3D-based segmentation
We evaluate LVM-Med on eight medical segmentation tasks, including five 2D-based and three 3D-based segmentation. In 2D settings, we also compare with 2D supervised architectures, such as U-Net, U-Net++, Attention U-Net, etc. These networks are initialized with ResNet-50 pre-trained ImageNet. Additionally, we investigate the prompt-based segmentation settings inspired by the current success of SAM's zero-shot learning. We utilized the ground truths and added random noise to simulate box-based user prompts as [9]. We next compare three variations of SAM: (i) freezing image and prompt encoders, only fine-tuning mask decoder; (ii) without any training and inference using box prompts; (iii) similar to (i) but replacing the original image encoder by LVM-Med's ViT architecture taken from SAM trained in our dataset.
Table 2: Performance comparison on five 2D segmentation tasks with fully fine-tuning. Results are reported with an average 2D Dice score on three trial times. The best results in each group are in bold, the overall best value, excluding prompt-based segmentation, is underlined.
Method	ISIC-2018 (Skin Lesion)	JSRT (Lung X-ray)	KvaSir (Polyp)	Drive (Vessel)	BUID (Breast Cancer)
2D Supervised Method	Randomly (R50)	86.16 ± 0.14	93.10 ± 0.12	62.85 ± 1.32	59.82 ± 2.00	65.54 ± 0.21
Pre-trained ImageNet [60]	86.87 ± 0.47	94.52 ± 2.66	83.85 ± 1.32	65.12 ± 1.55	72.64 ± 1.14
Attention-Unet [77]	86.81 ± 0.51	94.47 ± 2.71	82.23 ± 1.41	65.02 ± 1.44	72.19 ± 1.16
U-Net ++ [78]	86.71 ± 0.49	94.32 ± 2.81	82.23 ± 1.41	65.38 ± 0.78	73.76 ± 2.83
Trans U-Net [76]	86.60 ± 0.82	89.80 ± 0.35	67.11 ± 0.24	62.63 ± 0.24	67.90 ± 0.40
2D-SSL on medical	Twin-Barlon [13]	86.01 ± 0.07	94.56 ± 3.09	83.00 ± 0.23	65.73 ± 1.46	74.46 ± 1.19
Dino [79]	86.79 ± 0.09	94.84 ± 2.79	79.84 ± 1.62	65.39 ± 0.81	76.21 ± 0.57
SimCLR [15]	87.28 ± 0.21	94.79 ± 2.93	82.20 ± 0.51	65.22 ± 2.18	76.52 ± 0.22
Moco-v2 [17]	87.24 ± 0.14	94.05 ± 3.52	78.24 ± 1.35	64.92 ± 2.21	75.93 ± 1.96
Deepcluster-v2 [20]	86.73 ± 0.42	94.79 ± 2.89	82.69 ± 0.75	64.14 ± 0.92	76.33 ± 0.99
VicRegl [14]	86.27 ± 0.33	94.39 ± 3.25	81.93 ± 0.48	66.17 ± 0.27	75.29 ± 0.64
LVM-Med (R50)	87.76 ± 0.30	95.13 ± 2.64	86.76 ± 0.94	66.97 ± 0.27	78.65 ± 0.72
Foundation Model	Clip [3]	85.98 ± 0.19	89.00 ± 1.08	72.63 ± 0.37	63.01 ± 0.36	70.43 ± 0.24
Flava [5]	86.42 ± 0.10	90.08 ± 0.20	69.47 ± 0.05	61.09 ± 0.45	67.54 ± 1.17
SAM [6]	88.17 ± 0.30	90.68 ± 0.40	70.75 ± 0.60	64.04 ± 0.41	73.07 ± 0.66
LVM-Med (SAM's VIT)	88.41 ± 0.28	90.74 ± 0.47	73.10 ± 0.08	65.49 ± 0.12	77.20 ± 0.42
Prompt-based Seg.	SAM (fixed encoder) [9]	92.42 ± 0.12	92.89 ± 5.24	89.37 ± 0.57	59.74 ± 0.63	87.63 ± 0.67
SAM with Prompt (no-train) [6]	55.78 ± 0.66	61.97 ± 4.48	80.77 ± 0.19	15.12 ± 0.24	78.44 ± 1.01
LVM-Med (SAM's VIT)	92.48 ± 0.07	93.74 ± 4.06	90.09 ± 0.14	63.01 ± 0.02	89.69 ± 0.61
Table 3: 3D segmentation task performance with fine-tuning on three datasets. Results are reported with an average 3D IoU on five trial times. The best results in each group and overall are in bold and underlined.
Method	BraTS	MMWHS-CT	MMWHS-MRI
3D-Transformer [80]	66.54 ± 0.40	67.30 ± 2.29	67.64 ± 2.21
I3D [81]	67.83 ± 0.75	76.63 ± 2.32	66.71 ± 1.27
NiftyNet [82]	60.78 ± 1.60	74.91 ± 2.78	64.60 ± 1.96
Med3D [83]	66.09 ± 1.35	75.01 ± 0.74	63.43 ± 0.61
Model Genesis [32]	67.96 ± 1.29	76.48 ± 2.89	74.53 ± 1.69
Universal Model [84]	72.10 ± 0.67	78.14 ± 0.77	77.52 ± 0.50
TransVW [33]	68.82 ± 0.38	79.74 ± 2.78	75.08 ± 2.04
SwinViT3D [85]	70.58 ± 1.27	70.19 ± 1.23	78.25 ± 1.66
Joint-2D-3D (Depec) [86]	72.81 ± 0.15	83.58 ± 1.54	78.14 ± 1.32
Twin-Barlon [13]	73.30 ± 0.18	84.74 ± 1.01	76.39 ± 2.23
Dino [79]	71.72 ± 0.55	81.08 ± 1.62	70.42 ± 78.74
SimCLR [15]	73.15 ± 0.27	84.60 ± 1.11	76.54 ± 2.22
Moco-v2 [17]	71.97 ± 0.63	75.82 ± 4.20	68.29 ± 0.15
Deepcluster [20]	72.96 ± 0.51	84.03 ± 0.50	79.05 ± 1.63
VicRegl [14]	73.23 ± 0.33	84.72 ± 0.86	76.32 ± 0.78
LVM-Med (R50)	73.58 ± 0.14	84.91 ± 0.77	78.59 ± 0.84
Clip [3]	70.24 ± 1.23	78.5 ± 2.70	65.9 ± 3.98
Flava [5]	71.19 ± 0.48	78.91 ± 2.24	67.14 ± 1.20
SAM (Encoder) [6]	70.11 ± 1.45	77.8 ± 1.60	68.09 ± 5.49
LVM-Med (SAM's VIT)	71.42 ± 0.70	80.78 ± 1.77	69.36 ± 0.18
Table 4: In-out-distribution evaluation for the segmentation task on the Prostate dataset. Results are reported with an average 2D Dice score on three training times.
Method	Multi-site Prostate Segmentation
BMC (Based)	RUNMC	BIDMC	HK	Average
2D Supervised
Random	65.04 ± 2.07	51.44 ± 4.13	9.95 ± 13.56	12.38 ± 7.68	34.7
Pretrained ImageNet [60]	76.47 ± 1.26	62.11 ± 0.85	43.74 ± 4.38	53.90 ± 2.01	59.1
2D SSL on medical data
Twin-Barlon [13]	76.28 ± 1.76	60.09 ± 1.98	32.63 ± 12.32	34.82 ± 15.09	51.0
Dino [79]	77.90 ± 1.15	56.90 ± 1.97	21.53 ± 5.54	30.92 ± 5.41	46.8
SimCLR [15]	76.51 ± 2.07	64.10 ± 4.53	32.88 ± 5.43	42.29 ± 5.98	53.9
Moco-v2 [17]	74.40 ± 0.89	55.49 ± 5.45	27.53 ± 10.18	13.65 ± 14.33	42.8
Deepcluster [20]	77.45 ± 0.35	64.35 ± 3.15	37.73 ± 8.08	44.95 ± 8.57	56.1
Sway [21]	77.59 ± 0.61	57.61 ± 2.16	38.43 ± 12.55	44.90 ± 4.78	54.6
VicRegl [14]	74.85 ± 1.13	54.09 ± 4.35	25.56 ± 5.44	35.45 ± 13.03	47.5
LVM-Med (R50)	80.17 ± 0.55	62.48 ± 2.03	56.76 ± 6.50	52.78 ± 3.04	63.0
Prompt-based Seg.
SAM (Fixed encoder) [9]	95.50 ± 0.29	90.39 ± 0.39	91.41 ± 0.14	91.82 ± 0.26	92.28
SAM with Prompt (no-train) [6]	59.11 ± 1.55	66.95 ± 2.49	59.68 ± 0.49	57.41 ± 2.83	60.79
LVM-Med (SAM's ViT)	95.75 + 0.06	90.40 + 0.36	92.03 + 0.20	92.75 + 0.48	92.73
In 3D settings, we segment 2D slices and merge results for a 3D volume. We also benchmarked with 3D self-supervised methods from [86]. Tables (2) and (3) show that our two versions with ResNet-50 and Sam's ViT hold the best records in each category. For instance, we outperform 2D SSL methods trained on the same dataset, surpassing foundation models such as SAM, Flava, and Clip. In the prompt-based settings, LVM-Med also delivers better performance compared with SAM. Second, LVM-Med achieves the best overall results on seven of eight segmentation tasks, mostly held by
LVM-Med with ResNet-50. The improvement gaps vary on each dataset, for e.g., from  $3 - 5\%$  on Kvasir and BUID compared with 2D supervised methods.
4.3 Linear and finetuning image classification
We analyze LVM-Med on image classification tasks using linear probing (frozen encoders) and fully fine-tuning settings, two popular evaluations used in self-supervised learning. The experiments are conducted on the FGADR Grading and Brain tumor classification tasks. Table (5) presents the average accuracy metric on three training times. LVM-Med (ResNet-50) consistently outperforms other approaches on two datasets. For example, it is better than Clip by  $10.46\%$  and  $8.46\%$  on FGADR and Brain Tumor datasets with linear evaluation. In the foundation model setting, LVM-Med (ViT) also improves SAM's results by  $7.32\%$  and  $4.69\%$  on FGADR with linear and fully-finetuning. Another point we observe is that the overall 2D-SSL methods based on ResNet-50 and trained on the collected medical dataset achieve higher accuracy than foundation models using ViT. We also compare LVM-Med with the top methods on the FGADR dataset, including AFN-Net [87], JCS [88], CoLL [89], and DRG-Net [90]. We choose the DRG-Net as the backbone and replace the employed encoder with our weights (R50). Figure (3) shows that LVM-Med hold the first rank overall.
Table 5: Performance comparison on linear evaluation and fine-tuning classification. The results are reported with average Accuracy on three training times.
Method	Linear Evaluation (Frozen)	Fine-tuning
FGADR	Brain Tumor Cls.	FGADR	Brain Tumor Cls.
Twin-Barlon [13]	66.86 ± 0.41	63.03 ± 0.32	66.37 ± 0.77	74.20 ± 1.38
Dino [79]	65.98 ± 1.91	62.27 ± 0.32	67.35 ± 1.36	71.91 ± 1.55
SimCLR [15]	65.30 ± 1.70	62.52 ± 1.67	67.55 ± 0.28	73.52 ± 3.56
Moco-v2 [17]	65.98 ± 1.04	62.35 ± 1.92	67.55 ± 1.79	74.53 ± 0.43
Deepcluster [20]	65.34 ± 1.93	64.47 ± 0.55	67.94 ± 1.78	73.10 ± 0.55
VicRegl [14]	64.71 ± 0.60	59.64 ± 1.36	65.69 ± 1.46	73.18 ± 2.03
LVM-Med (R50)	68.33 ± 0.48	66.33 ± 0.31	68.32 ± 0.48	76.82 ± 2.23
Clip [3]	57.87 ± 0.50	57.87 ± 0.71	57.48 ± 0.86	34.86 ± 2.27
Flava [5]	31.87 ± 0.69	35.19 ± 0.43	57.18 ± 0.96	34.01 ± 5.97
Algin [4]	36.95 ± 1.04	30.71 ± 2.35	57.28 ± 0.97	63.96 ± 0.04
SAM [6]	55.13 ± 0.41	31.81 ± 4.26	58.75 ± 1.32	60.66 ± 1.36
LVM-Med (SAM's ViT)	62.46 ± 0.86	59.31 ± 0.48	63.44 ± 0.73	67.34 ± 2.08
Table 6: LVM-Med ablation study. Results are reported on an average of five 2D segmentation and two linear classification tasks. The two most important factors are highlighted.
Method	Cls.(Acc)	Seg. (Dice)
LVM-Med (Full)	67.47	83.05
LVM-Med w/o second-order	62.17	80.21
LVM-Med w/o message passing	65.08	81.19
LVM-Med w/o Gumbel noise	64.32	81.37
LVM-Med w/o local similarity	65.67	81.54
4.4 Object detection & In-out-distribution evaluation
Figure 4 indicates our performance on the object detection task using VinDr and Kvasir datasets. We use Faster R-CNN and load ResNet-50 from 2D SSL pretrained weights. Results are presented by Average Precision with IoU=0.5 over three training times. Compared to pre-trainedImagenet, LVM-Med still outperforms by  $1 - 2\%$  though overall, our improvements are smaller than image classification and segmentation tasks.
We also validate LVM-Med performance on the in-out
distribution setting in Table (4) using the segmentation task on the Multi-Prostate dataset. We train LVM-Med and other competitors in BMC data and use the trained models to predict the remaining datasets. Both two versions of LVM-Med with ResNet-50 and ViT, on average, surpass all baselines, which validates the potential abilities of LVM-Med for the in-out-distribution problem.
Figure 4: LVM-Med on object detection.
We do the following settings to evaluate the performance of components used in LVM-Med: (i) LVM-Med without using second-order graph matching conditions, i.e., only solving vertex-to-vertex correspondence problem; (ii) LVM-Med without using message passing network  $g_{\epsilon}$  in Eq. (1) to aggregate information from local connections; (iii) LVM-Med w/o using approximate gradients from Gumbel noise in Eq. (6). For this, we add a constant value to  $c^v$ ,  $c^e$  as prior works [57, 49], and finally (iv) LMV-Med without using local similarity  $c_{ia}^{lo}$  in Eq. (2). Other ablation studies are presented in Appendix. Table (6) indicates that all factors contribute to the final performance, wherein the second-order and Gumbel noise are the two most two important parts.
We have demonstrated that a self-supervised learning technique based on second-order graph-matching, trained on a large-scale medical imaging dataset, significantly enhances performance in various downstream medical imaging tasks compared to other supervised learning methods and foundation models trained on hundreds of millions of image-text instances. Our findings are supported by the benefits shown in two different architectures: ResNet-50 and ViT backbones, which can be used for either end-to-end or prompt-based segmentation.
Limitations and Future Work. We propose to investigate the following points to improve LVM-Med performance. Firstly, extending LVM-Med to a hybrid 2D-3D architecture to allow direct application for 3D medical tasks instead of 2D slices. Secondly, although LVM-Med with ViT backbone utilizes more total parameters, in some cases, it is less effective than LVM-Med ResNet-50. This raises the question of whether a novel approach could improve the performance of ViT architectures. Finally, integrating multi-modal information such as knowledge graphs, bio-text, or electronic health records for LVM-Med is also important to make the model more useful in real-world applications.
This research has been supported by the pAItient project (BMG, 2520DAT0P2), Ophthalmo-AI project (BMBF, 16SV8639) and the Endowed Chair of Applied Artificial Intelligence, Oldenburg University. Binh T. Nguyen wants to thank the University of Science, Vietnam National University in Ho Chi Minh City for their support. Tan Ngoc Pham would like to thank the Vingroup Innovation Foundation (VINIF) for the Master's training scholarship program. The authors thank the International Max Planck Research School for Intelligent Systems (IMPRS-IS) for supporting Duy M. H. Nguyen. Mathias Niepert acknowledges funding by Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) under Germany's Excellence Strategy - EXC and support by the Stuttgart Center for Simulation Science (SimTech).
[1] Veronika Cheptygina, Marleen de Bruijne, and Josien PW Pluim. Not-so-supervised: a survey of semi-supervised, multi-instance, and transfer learning in medical image analysis. Medical image analysis, 54:280-296, 2019.
[2] Georgios A Kaissis, Marcus R Makowski, Daniel Rückert, and Rickmer F Braren. Secure, privacy-preserving and federated machine learning in medical imaging. Nature Machine Intelligence, 2(6):305-311, 2020.
[3] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748-8763. PMLR, 2021.
[4] Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. In International Conference on Machine Learning, pages 4904-4916. PMLR, 2021.
[5] Amanpreet Singh, Ronghang Hu, Vedanuj Goswami, Guillaume Couairon, Wojciech Galuba, Marcus Rohrbach, and Douwe Kiela. Flava: A foundational language and vision alignment model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15638-15650, 2022.
[6] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al. Segment anything. arXiv preprint arXiv:2304.02643, 2023.
[7] Maciej A Mazurowski, Haoyu Dong, Hanxue Gu, Jichen Yang, Nicholas Konz, and Yixin Zhang. Segment anything model for medical image analysis: an experimental study. arXiv preprint arXiv:2304.10517, 2023.
[145] Lhncbc malaria. https://lhncbc.nlm.nih.gov/LHC-downloads/downloads.html#malaria-datasets.
[146] D. Wei, Z. Lin, D. Barranco, N. Wendt, X. Liu, W. Yin, X. Huang, A. Gupta, W. Jang, X. Wang, I. Arganda-Carreras, J. Lichtman, and H. Pfister. Mitoem dataset: Large-scale 3d mitochondria instance segmentation from em images. In International Conference on Medical Image Computing and Computer Assisted Intervention, 2020.
[147] Christian Matek, Sebastian Krappe, Christian Munzenmayer, Torsten Haferlach, and Carsten Marr. Highly accurate differentiation of bone marrow cell morphologies using deep neural networks on a large image data set. Blood, The Journal of the American Society of Hematology, 138(20):1917-1927, 2021.
[148] Safwan S Halabi, Luciano M Prevedello, Jayashree Kalpathy-Cramer, Artem B Mamonov, Alexander Bilbily, Mark Cicero, Ian Pan, Lucas Araujo Pereira, Rafael Teixeira Sousa, Nitamar Abdala, et al. The rsna pediatric bone age machine learning challenge. Radiology, 290(2): 498-503, 2019.
[149] Kaggle dr dataset (eyepacs). Kaggle dataset. URL https://www.kaggle.com/datasets/ mariaherrerot/eyepacspreprocess.
We present below LVM-Med pseudo-code (Section A), implementations used in downstream tasks (Section B), additional ablation studies of LVM-Med (Section C), further prompt-based segmentation results on 3D datasets, image classification benchmark (Section D), predicted masks using the user-based prompt (Section E), and finally the dataset overview (Section F).
A LVM-Med Pseudo-code
First, we provide a pseudo-code for training LVM-Med in Pytorch style:
$\# \mathrm{f}_{\theta}$  : encoder network,  $\mathbf{h}_{\phi}$  : projector network,  $\mathbf{g}_{\epsilon}$  : message passing network, # k_nodes: number of nearest neighbors, Avg: average pooling, # pos: position of image after transform, cos: cosine similarity, #  $\alpha$  : coefficient trades off between global and local costs, L2:L2-distance, #  $\gamma$  : maximum pairs are kept, select_top: select to keep the  $\gamma$  best matches.