Title: LVM-Med: Learning Large-Scale Self-Supervised Vision Models for Medical Imaging via Second-order Graph Matching
Venue: NeurIPS 2023
Accepted: yes
Keywords: medical imaging; self-supervised learning; graph matching; large-vision model
Abstract: Obtaining large pre-trained models that can be fine-tuned to new tasks with limited annotated samples has remained an open challenge for medical imaging data. While pre-trained networks on ImageNet and vision-language foundation models trained on web-scale data are the prevailing approaches, their effectiveness on medical tasks is limited due to the significant domain shift between natural and medical images. To bridge this gap, we introduce LVM-Med, the first family of deep networks trained on large-scale medical datasets. We have collected approximately 1.3 million medical images from 55 publicly available datasets, covering a large number of organs and modalities such as CT, MRI, X-ray, and Ultrasound. We benchmark several state-of-the-art self-supervised algorithms on this dataset and propose a novel self-supervised contrastive learning algorithm using a graph-matching formulation. The proposed approach makes three contributions: (i) it integrates prior pair-wise image similarity metrics based on local and global information; (ii) it captures the structural constraints of feature embeddings through a loss function constructed through a combinatorial graph-matching objective, and (iii) it can be trained efficiently end-to-end using modern gradient-estimation techniques for black-box solvers. We thoroughly evaluate the proposed LVM-Med on 15 downstream medical tasks ranging from segmentation and classification to object detection, and both for the in and out-of-distribution settings. LVM-Med empirically outperforms a number of state-of-the-art supervised, self-supervised, and foundation models. For challenging tasks such as Brain Tumor Classification or Diabetic Retinopathy Grading, LVM-Med improves previous vision-language models trained on 1 billion masks by 6-7%  while using only a ResNet-50.

LVM-Med: Learning Large-Scale Self-Supervised Vision Models for Medical Imaging via Second-order Graph Matching

Abstract

Obtaining large pre-trained models that can be fine-tuned to new tasks with limited annotated samples has remained an open challenge for medical imaging data. While pre-trained deep networks on ImageNet and vision-language foundation models trained on web-scale data are prevailing approaches, their effectiveness on medical tasks is limited due to the significant domain shift between natural and medical images. To bridge this gap, we introduce LVM-Med, the first family of deep networks trained on large-scale medical datasets. We have collected approximately 1 . 3 million in medical images from 55 publicly available datasets, covering a large number of organs and modalities such as CT, MRI, X-ray, and Ultrasound. We benchmark several state-of-the-art self-supervised algorithms on this dataset and propose a novel self-supervised contrastive learning algorithm using a graph matching formulation . The proposed approach makes three contributions: (i) it integrates prior pair-wise image similarity metrics based on local and global information; (ii) it captures the structural constraints of feature embeddings through a loss function constructed via a combinatorial graph-matching objective; and (iii) it can be trained efficiently end-to-end using modern gradient-estimation techniques for black-box solvers. We thoroughly evaluate the proposed LVM-Med on 15 downstream medical tasks ranging from segmentation and classification to object detection, and both for the in and out-of-distribution settings. LVM-Med empirically outperforms a number of state-of-the-art supervised, self-supervised, and foundation models. For challenging tasks such as Brain Tumor Classification or Diabetic Retinopathy Grading, LVM-Med improves previous vision-language models trained on 1 billion masks by 6-7 % while using only a ResNet-50. We release pre-trained models at this link https://github.com/duyhominhnguyen/LVM-Med .

1 Introduction

Constructing large-scale annotated medical image datasets for training deep networks is challenging due to data acquisition complexities, high annotation costs, and privacy concerns [1, 2]. Visionlanguage pretraining has emerged as a promising approach for developing foundational models that support various AI tasks. Methods such as CLIP [3], Align [4], and Flava [5] propose a unified model trained on large-scale image-text data, showing exceptional capabilities and performance across

various tasks. However, their effectiveness in the medical domain still remains unclear. A recent work SAM [6] trains large vision models on over one billion annotated masks from 11M natural images, enabling interactive segmentation. Nevertheless, SAM's zero-shot learning performance is moderate on other datasets [7, 8], highlighting the need for fine-tuning to achieve satisfactory results [9].

Organ

Modality

Brain

Eye

G= (V,Es)

LVM-Med

G' = (v,E)

MRI

x

x

Breast

Tooth

CT

Shoulder

Heart

X-ray

x²

x

Chest

Lung

V={x}→V={x}

Gastrointestinal

Skin

Dermoscopy

V={x} →v

V={x²}→V={x}

Liver

Endoscope

x

x

Kidney

Retinal Image

x2

Blood

x

X

Ultrasound

Hand

Prostate

Fetal

x

CellImage

ContrastiveSSL:

Instance-based SSL:

SimClr, Moco-v2

Twin-Barlon, Dino, VICRegL

To facilitate the development of foundation models in the medical domain, we make two major contributions. First, we have curated a vast collection of 55 publicly available datasets, resulting in approximately 1.3 million medical images covering various body organs and modalities such as CT, MRI, X-ray, ultrasound, and dermoscopy, to name a few. Second, we propose LVM-Med, a novel class of contrastive learning methods, utilizes pre-trained ResNet-50 and a ViT network SAM[10]. We evaluate various instances of LVM-Med relative to popular supervised architectures and vision-language models across 15 medical tasks. To our best knowledge, this is the first time such a large-scale medical dataset has been constructed and used to investigate the capabilities of SSL algorithms.

LVM-Med incorporates a second-order graph-matching formulation, which subsumes and extends a large class of contrastive SSL methods. Given a batch of images, two random transformations are applied to each image, and the resulting transformed images are then fed to an image encoder. The embedding vectors obtained from images in a batch are used to construct two graphs where vertices represent pairs of transformed images generated from the same original one. Through solving a graph-matching problem [11, 12], we learn feature representation such that their encoding serves as suitable priors for a global solution of the graph-matching objective. This approach is distinct from prior contrastive learning methods that focus on merely optimizing pair-wise distances [13, 14] between transformed images or learning contrastive distances with positive and negative samples [15-19]. It is worthwhile noting that previous contrastive learning methods are special instances of our general framework (Figure (1), right).

LVM-Med has several advantages over existing approaches. First, it integrates advanced pair-wise image similarity taken from prior SSL methods into vertex affinities, resulting in both global and local information that can be efficiently fused. Second, it uncovers underlying structures of feature embeddings by utilizing edge constraints, enhancing robustness in the presence of similar entities in medical datasets. Third, though combinatorial problems are typically non-differentiable, LVM-Med can efficiently calculate gradients through the discrete combinatorial loss function using modern implicit maximum likelihood estimation techniques. Consequently, LVM-Med can scale successfully on large-scale data. In a wide range of 15 medical experiments, LVM-Med sets a new state-of-the-art in fully fine-tuning or prompt-based segmentation, linear and fully fine-tuning image classification, and domain generalization, outperforming several vision-language models trained on a hundred million image-text instances.

We summarize major contributions in this work, including:

(i) We present a collection of large-scale medical datasets, serving as a resource for exploring and evaluating self-supervised algorithms.

(ii) We propose LVM-Med, a novel SSL approach based on second-order graph matching. The proposed method is flexible in terms of integrating advanced pair-wise image distance and being able to capture structural feature embedding through the effective utilization of second-order constraints within a global optimization framework.

(iii) On both ResNet-50 and ViT architectures, LVM-Med consistently outperforms multiple existing self-supervised learning techniques and foundation models across a wide range of downstream tasks.

2 Related Work

2.1 Self-supervised learning in medical image analysis

The latest approaches of global feature SSL rely on shared embedding architecture representations that remain invariant to different viewpoints. The variation lies in how these methods prevent collapsing solutions. Clustering methods [20-22] constrain a balanced partition of the samples within a set of cluster assignments. Contrastive methods [15-18] uses negative samples to push far away dissimilar samples from each other through contrastive loss, which can be constructed through memory bank [23], momentum encoder [24], or graph neural network [19]. Unlike contrastive learning, instant-based learning depends on maintaining the informational context of the feature representations by either explicit regularization [13, 25] or architectural design [26, 27]. Our work relates to contrastive and instance-based learning, where a simplified graph-matching version of 1-N or 1-1 reverts to these approaches.

In contrast to global methods, local methods specifically concentrate on acquiring a collection of local features that depict small portions of an image. A contrastive loss function can be used on those feature patches at different criteria such as image region levels [28], or feature maps [29, 14]. These strategies are also widely applied in the medical context, thereby pre-text tasks based on 3D volume's properties, such as reconstructing the spatial context [30], random permutation prediction [31] and self-restoration [32, 33], are proposed. Our LVM-Med model on this aspect can flexible unifying both global and local information by adding them to the affinities matrixes representing the proximity of two graphs, enhancing expressive feature representations.

2.2 Vision-language foundation models

In order to comprehend the multi-modal world using machines, it is necessary to create foundational models that can operate across diverse modalities and domains [34]. CLIP [3] and ALIGN [4] are recognized as groundbreaking explorations in foundation model development. These models demonstrate exceptional proficiency in tasks such as cross-modal alignment and zero-shot classification by learning contrastive pretraining on extensive image-text pairs from the web, despite the presence of noise. To further support multi-modal generation tasks such as visual question answering or video captioning, recent works such as FLAVA [5] and OmniVL [35] are designed to learn cross-modal alignment as well as image-video language models. Conversely, the SAM model [6] utilized a supervised learning strategy with over 1 billion masks on 11 million user-prompt interactions and achieved impressive zero-shot segmentation performance on unseen images. While many efforts have been proposed for natural image domains, limited research has been conducted on large-scale vision models for medical imaging. This motivated us to develop the LVM-Med model.

2.3 Graph matching in visual computing

Graph matching is a fundamental problem in computer vision, which aims to find correspondences between elements of two discrete sets, such as key points in images or vertices of 3D meshes, and used in numerous vision tasks, including 3D vision [36, 37], tracking [38, 39], shape model learning [40, 41], and many others [42-45]. In this framework, the vertices of the matched graphs correspond to the elements of the discrete sets to be matched. Graph edges define the cost structure of the problem, namely, second order, where pairs of matched vertices are penalized in addition to the vertex-to-vertex matchings. This allows us to integrate the underlying geometrical relationship between vertices into account but also makes the optimization problem NP-hard. Therefore, many approximate approaches have been proposed to seek acceptable suboptimal solutions by relaxing discrete constraints [46, 47]. In other directions, gradient estimation techniques for black-box solvers are employed to make the hybrid discrete-continuous matching framework be differentially end-to-end [48-50]. Our LVM-Med follows the latter direction and, for the first time, presents the formulation of contrastive learning as a second-order graph-matching problem.

B = {(x1,x2,..,xN}

x1

x

Avg

s,t~T

田

Encoder

Projector

hΦ

fe

Gs = (V,Es)

Gt = (v,Er)

x

c" e RIv|xIv" ce e RIE"xIEl

MPN ge

x

MPN ge

x

7e

ze

dcv

x

z4

x4

= GM(c",c%)

x4

Forward pass

Lhamm(b, v*)

Backward pass

3 Methodology

3.1 Dataset construction

We provide detailed information about the collected datasets in the Appendix. The data was collected from publicly available resources, which include a diverse set of modalities and body organs as illustrated in Figure 1 (left). The data format is a combination of 2D images and 3D volumes as well as X-ray, MRI, CT, Ultrasounds, etc. For datasets whose data dimensions are 3D volumes, we slice them into 2D images. To avoid potential test data leaking for downstream tasks, we use the default training partition in each dataset; otherwise, we randomly sample with 20% total images. In total, we obtain approximately 1 . 3 million images. More statistics on the dataset are presented in the Appendix.

3.2 Contrastive learning as graph matching

Figure 2 provides an illustration of our LVM-Med method, which learns the feature representation f θ by matching two distorted views derived from the same input image through a graph-matching formulation. Below we describe in detail each component.

3.2.1 Graph construction on feature embedding

Given a batch of N images B = { x 1 , x 2 , .., x N } sampled from a dataset, we generate for each image x i ∈ B two transformed images x s i and x t i by using two transformations s, t ∼ T sampled from T , a set of pre-defined image transformations. After the transformations, each image is of shape ( C × H × W ) , where C is the number of channels and ( H,W ) the original spatial dimensions. These distorted images are fed into an encoder f θ : R C × H × W → R D × R × S to produce two representations y s i = f θ ( x s i ) and y t i = f θ ( x t i ) where D is the number of feature channels and ( R,S ) are the spatial dimensions of the feature map. On each such representation, we perform an average pooling

operation Avg : R D × R × S → R D followed by another projection h ϕ : R D → R F to form two feature embeddings z s i = h ϕ (Avg( y s i )) , and z t i = h ϕ (Avg( y t i )) ∈ R F with F < D .

Given a set of embeddings for a batch B , we construct two graphs G s and G t where, for each pair ( x s i , x t i ) of corresponding distorted images, we add a node representing x s i to G s and a node representing x t i to G t . Hence, for each ℓ ∈ { s, t } , we construct a graph G ℓ = ( V ℓ , E ℓ ) with V ℓ = { x ℓ 1 , ..., x ℓ N } the set of vertices and E ℓ the set of edges e ℓ ij = ( x ℓ i , x ℓ j ) . The node-level feature matrix is given by X ℓ = [ z ℓ 1 ; ... ; z ℓ N ] ∈ R N × F which associates each vertex x ℓ i with its feature embedding z ℓ i . We create edges for each graph G ℓ through a k -nearest neighbors algorithm using the feature matrix X ℓ . The adjacency matrix A ℓ ∈ R N × N is defined as A ℓ ij = 1 if e ℓ ij ∈ E ℓ and A ij = 0 otherwise. With the two graph structures given, we obtain a node-attributed graph G ℓ = ( V ℓ , A ℓ , X ℓ ) on which a graph neural network g ε is used to aggregate the nodes' features. In particular, g ε computes an embedding ˆ Z ℓ = g ε ( X ℓ , A ℓ ) by performing message passing operations. We set g ε to be a graph convolutional network [51, 52] consisting of l +1 layers g ε = { g l , g l -1 , .., g 0 } where the output of layer l is computed as

where I N is the identity matrix modeling self-connections; ˜ D is a diagonal matrix with ˜ D ii = ∑ j A ℓ ij ; g l -1 are the trainable parameters for each layer; σ ( · ) is an activation function; and H ℓ 0 = X ℓ . We use the outputs of the last layer as embeddings for the nodes, that is, ˆ Z ℓ = H ℓ l ∈ R N × F given the shared graph network g ε .

We now have two graphs G s , G t with node attribute matrices ˆ Z s , ˆ Z t , the outputs of the graph neural networks. Next, a graph-matching problem is constructed and solved where the gold matching is given by the pairs ( x s i , x t i ) ∀ i ∈ { 1 , .., N } .

3.2.2 Learning affinities with global and local context

To represent potential connections for a pair of node ( x s i , x t a ) where x s i ∈ G s , x t a ∈ G t , we design a vertex affinity matrix c v ∈ R | V s || V t | where c v ia is the prior (feature-based) similarity between x s i and x t a . An advantage of our formulation is its ability to integrate advanced pair-wise distance can be smoothly integrated to c v ia , resulting in more expressive proximity representation. In particular, we leverage both global and local consistency derived from feature embeddings of distorted images. The global distance used in several prior works can be computed as c glo ia ( x s i , x t a ) = cos(ˆ z s i , ˆ z t a ) where cos( · ) denotes cosine similarity; ˆ z ℓ m is the embedding of x ℓ m ( ℓ ∈ { s, t } , m ∈ { i, a } ) obtained after message passing in Eq. (1).

Compared to global methods that implicitly learn features for the entire image, local methods concentrate on explicitly learning a specific group of features that characterize small regions of the image. As a result, they are more effective for dense prediction tasks such as segmentation [29, 14, 53]. While recent works applied these tactics as a part of pair-wise minimization conditions [54, 28] Instead, we integrate them as a part of vertex costs c v ia and use it to solve the graph matching problem. Indeed, we adapt both location- and feature-based local affinity computed as:

where P = { ( r, s ) | ( r, s ) ∈ [1 , ..., R ] × [1 , .., S ] } be the set of coordinates in the feature map y s i ∈ R D × R × S of x s i ; q ℓ p ( ℓ ∈ { s, t } ) be the feature vector at position p ; m( p ) denote the spatial closest coordinate to p in coordinates of feature map y t a estimated through transformations on original image x i ; finally m ′ ( p ) represents the closest feature vector to p in y t a using l 2 distance. Intuitively, the local cost in Eq. (2) enforces invariance on both spatial location and between embedding space at a local scale. Our final affinity cost is computed as:

3.2.3 Self-supervision through second-order graph matching

While the standard graph matching problem for vertex-to-vertex correspondences can be used in our setting (LAP), it fails to capture the similarity between edges. If there are duplicated entities

represented by distinct nodes in the same graph, the LAP will consider them identical and skip their neighboring relations. For instance, during the image sampling, two consecutive image slides were sampled from a 3D volume, resulting in their appearances have s a small difference. In such cases, it is complicated to correctly identify those augmented images generated from the same one without using information from the relations among connected nodes in the constructed graph. To address this problem, we introduce additional edge costs c e ∈ R | E s || E t | where c e ia,jb represents the similarity between an edge v s ij = ( x s i , x s j ) ∈ E s and v t ab = ( x t a , x t b ) ∈ E t . These edge costs (second-order) are computed as c e ia,jb = cos(( ˆ z s i -ˆ z s j ) , ( ˆ z t a -ˆ z t b )) .

We now establish the second-order graph-matching problem. Denoting v = { 0 , 1 } | V s || V t | be indicator vector of matched vertices, i.e., v ia = 1 if the vertex x s i ∈ V s is matched with x t a ∈ V t and v ia = 0 otherwise. The node correspondence between two graphs G s and G t that minimizes the global condition stated as:

and 1 N be a n -dimension one-value vector. The constraint U ( 1 , 1 ) restricts v satisfying the one-toone matching. Essentially, the Eq. (4) solves the vertex-to-vertex correspondence problem using both node and edges affinities, which can be seen as a form of structural matching (Figure (1),right) and generally can be integrated with higher-order graph constraints as triangle connections or circles. In the experiment, we found out that Eq. (4) significantly improved downstream task performance compared to the pure linear matching approach (Table (6)). Since the Eq. 4 in general is an NP-Hard problem [55] due to its combinatorial nature, we thus use efficient heuristic solvers based on Lagrange decomposition techniques [56].

3.2.4 Backpropagating through a graph matching formulation

With ˆ v = GM ( c v , c e ) a solution obtained from the solver, we use the Hamming distance and an optimal solution v ∗ to define the following loss function

The proposed approach aims to learn the feature representation function f θ such that its output minimizes Eq. (5). However, this is a difficult problem because the partial derivatives of the loss function w.r.t vector costs c v , c e , i.e., ∂L/∂ c v and ∂L/ c e , are zero almost everywhere [48, 57] due to the objective function in Eq. (4) being piece-wise constant, preventing direct gradient-based optimization.

To approximate the gradients required for backpropagation, we adopt IMLE [50, 58]. Let θ = ( c v , c e ) be the input to the combinatorial graph matching problem in Eq. (4). The core idea of IMLE is to define a probability distribution ρ ( v ; θ ) over solutions of the combinatorial optimization problem, where the probability of a solution is proportional to its negative cost, and to estimate ∂L/∂ θ through the gradients of the expectation ∇ θ E ˆ v ∼ ρ ( v ; θ ) [ L ( ˆ v , v ∗ )] . Since exact sampling from ρ ( v ; θ ) is typically intractable, IMLE instead chooses a noise distribution ρ ( ϵ ) and approximates the gradient of the expectation over ρ ( v ; θ ) with the gradient of the expectation over ρ ( ϵ )

The above approximation invokes the reparameterization trick for a complex discrete distribution. A typical choice for ρ ( ϵ ) is the Gumbel distribution, that is, ρ ( ϵ ) ∼ Gumbel(0 , 1) [59]. Now, by using a finite-difference approximation of the derivative in the direction of the gradient of the loss ∇ ˜ v L ( ˜ v , v ∗ ), we obtain the following estimation rule:

Algorithm 1 Forward and Backward Pass for c v , c e

where ˜ v = GM ( θ + ϵ ) , λ is a step size of finite difference approximation. Using a Monte Carlo approximation of the above expectation, the gradient for θ is computed as a difference of two or more pairs of perturbed graph-matching outputs. We summarize in Algorithm 1 the forward and backward steps for c v , c e .

4 Experiments

4.1 Implementation details

Pre-training We utilize Resnet50 [60] and Vision Transformer (ViT-B/16) [61] to train our LVMMed. For Resnet50, we load pre-trained from ImageNet-1K [62], and SAM Encoder backbone weight [10] for ViT. The raw image is augmented to two different views by using multi-crop techniques as [14] and followed by flip (probability 50 % ), color jitter, random Gaussian blur, and normalization. We trained the LVM-Med with 100 epochs on the collected dataset. The batch size of 3200 is used for ResNet50 and we reduced it to 2800 for ViT due to memory limitation. The model is optimized with Adam [63] with an initial learning rate 2 × 10 -3 and reduced halved four times. We use 16 A100-GPUs per with 80 GB and complete the training process for LVM-Med with ResNet-50 in five days and LVM-Med with ViT encoder in seven days. Other competitor SSL methods as VicRegl, Twin-Barlon, Dino, etc, are initialized from ResNet-50 pre-trained ImageNet-1K and trained with 100 epochs with default settings as LVM-Med.

To balance samples among different modalities, we combine over-sampling and data augmentation to increase the total samples. Specifically, new samples from minority classes are generated by duplicating images and applying random crop operations covering 85 -95% of image regions and then rescaling them to the original resolutions. Note that these augmentations are not used in the self-supervised algorithm (operations s, t ∼ T ) to avoid generating identical distorted versions in this sampling procedure.

80

85

90

Kappa

AFN-Net

CoLL

DRG-Net

LVM-Med (DRG-Net)

Figure 3: FGADR performance with top architectures.

Downstream Tasks Table 1 lists the datasets and downstream tasks used in our experiments. We cover segmentation, object detection, and image classification problems. It is important to note that in most settings, we utilize simple configurations for all datasets, skipping extra pre-processing for data augmentation. For instance, overlapping image patches with stride operations in the original samples [74] to increase training data in the Drive dataset or combining different 3D MRI modalities to fuse information [75] in the BRATS-2018 are excluded in our downstream setups.

To validate LVM-Med algorithms, we compare with 2D-SSL methods trained in our dataset and foundation models like Clip [3], Align [4], Flava [5], and SAM [6] with pre-trained ViT (Bert for

Align) taken from each method, respectively. During the downstream task, trained SSL weights are then extracted and attached in U-Net for ResNet50 backbone, TransUnet [76] for ViT, and then fine-tuned with training splits of each dataset. Depending on the downstream task's properties, we apply different image resolutions and other parameters like the number of epochs and learning rate for different data domains. Details for these configurations are presented in Appendix.

4.2 2D- and 3D-based segmentation

We evaluate LVM-Med on eight medical segmentation tasks, including five 2D-based and three 3D-based segmentation. In 2D settings, we also compare with 2D supervised architectures, such as U-Net, U-Net++, Attention U-Net, etc. These networks are initialized with ResNet-50 pre-trained ImageNet. Additionally, we investigate the prompt-based segmentation settings inspired by the current success of SAM's zero-shot learning. We utilized the ground truths and added random noise to simulate box-based user prompts as [9]. We next compare three variations of SAM: (i) freezing image and prompt encoders, only fine-tuning mask decoder; (ii) without any training and inference using box prompts; (iii) similar to (i) but replacing the original image encoder by LVM-Med's ViT architecture taken from SAM trained in our dataset.

In 3D settings, we segment 2D slices and merge results for a 3D volume. We also benchmarked with 3D self-supervised methods from [86]. Tables (2) and (3) show that our two versions with ResNet-50 and Sam's ViT hold the best records in each category. For instance, we outperform 2D SSL methods trained on the same dataset, surpassing foundation models such as SAM, Flava, and Clip. In the prompt-based settings, LVM-Med also delivers better performance compared with SAM. Second, LVM-Med achieves the best overall results on seven of eight segmentation tasks , mostly held by

LVM-Med with ResNet-50. The improvement gaps vary on each dataset, for e.g., from 3 -5% on Kvasir and BUID compared with 2D supervised methods.

4.3 Linear and finetuning image classification

We analyze LVM-Med on image classification tasks using linear probing (frozen encoders) and fully fine-tuning settings, two popular evaluations used in self-supervised learning. The experiments are conducted on the FGADR Grading and Brain tumor classification tasks. Table (5) presents the average accuracy metric on three training times. LVM-Med (ResNet-50) consistently outperforms other approaches on two datasets. For example, it is better than Clip by 10 . 46% and 8 . 46% on FGADR and Brain Tumor datasets with linear evaluation. In the foundation model setting, LVM-Med (ViT) also improves SAM's results by 7 . 32% and 4 . 69% on FGADR with linear and fully-finetuning. Another point we observe is that the overall 2D-SSL methods based on ResNet-50 and trained on the collected medical dataset achieve higher accuracy than foundation models using ViT. We also compare LVM-Med with the top methods on the FGADR dataset, including AFN-Net [87], JCS [88], CoLL [89], and DRG-Net [90]. We choose the DRG-Net as the backbone and replace the employed encoder with our weights (R50). Figure (3) shows that LVM-Med hold the first rank overall.

4.4 Object detection & In-out-distribution evaluation

Figure 4 indicates our performance on the object detection task using VinDr and Kvasir datasets. We use Faster R-CNN and load ResNet-50 from 2D SSL pretrained weights. Results are presented by Average Precision with IoU=0.5 over three training times. Compared to pre-trained Imagenet, LVM-Med still outperforms by 1 -2% though overall, our improvements are smaller than image classification and segmentation tasks.

Figure 4: LVM-Med on object detection. We also validate LVM-Med performance on the in-outdistribution setting in Table (4) using the segmentation task on the Multi-Prostate dataset. We train LVM-Med and other competitors in BMC data and use the trained models to predict the remaining datasets. Both two versions of LVM-Med with ResNet-50 and ViT, on average, surpass all baselines, which validates the potential abilities of LVM-Med for the in-out-distribution problem.

4.5 Ablation study

We do the following settings to evaluate the performance of components used in LVM-Med: (i) LVM-Med without using second-order graph matching conditions, i.e., only solving vertex-to-vertex correspondence problem; (ii) LVM-Med without using message passing network g ϵ in Eq. (1) to aggregate information from local connections; (iii) LVM-Med w/o using approximate gradients from Gumbel noise in Eq. (6). For this, we add a constant value to c v , c e as prior works [57, 49], and finally (iv) LMV-Med without using local similarity c lo ia in Eq. (2). Other ablation studies are presented in Appendix. Table (6) indicates that all factors contribute to the final performance, wherein the second-order and Gumbel noise are the two most two important parts.

0

10

20

30

40

50

60

70

80

90

ImageNet

Twin-Barlon

Dino

SimCLR

Moco-v2

Deepcluster

Swav

VicRegl

LVM-Med

VinDr

Kvasir

Object Detections on VinDr and Kvasir. Results on @mAP50

5 Conclusion

We have demonstrated that a self-supervised learning technique based on second-order graphmatching, trained on a large-scale medical imaging dataset, significantly enhances performance in various downstream medical imaging tasks compared to other supervised learning methods and foundation models trained on hundreds of millions of image-text instances. Our findings are supported by the benefits shown in two different architectures: ResNet-50 and ViT backbones, which can be used for either end-to-end or prompt-based segmentation.

Limitations and Future Work. We propose to investigate the following points to improve LVM-Med performance. Firstly, extending LVM-Med to a hybrid 2D-3D architecture to allow direct application for 3D medical tasks instead of 2D slices. Secondly, although LVM-Med with ViT backbone utilizes more total parameters, in some cases, it is less effective than LVM-Med ResNet-50. This raises the question of whether a novel approach could improve the performance of ViT architectures. Finally, integrating multi-modal information such as knowledge graphs, bio-text, or electronic health records for LVM-Med is also important to make the model more useful in real-world applications.

References

Veronika Cheplygina, Marleen de Bruijne, and Josien PW Pluim. Not-so-supervised: a survey of semi-supervised, multi-instance, and transfer learning in medical image analysis. Medical image analysis , 54:280-296, 2019.

Georgios A Kaissis, Marcus R Makowski, Daniel Rückert, and Rickmer F Braren. Secure, privacy-preserving and federated machine learning in medical imaging. Nature Machine Intelligence , 2(6):305-311, 2020.

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning , pages 8748-8763. PMLR, 2021.

Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, YunHsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. In International Conference on Machine Learning , pages 4904-4916. PMLR, 2021.

Amanpreet Singh, Ronghang Hu, Vedanuj Goswami, Guillaume Couairon, Wojciech Galuba, Marcus Rohrbach, and Douwe Kiela. Flava: A foundational language and vision alignment model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 15638-15650, 2022.

Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al. Segment anything. arXiv preprint arXiv:2304.02643 , 2023.

Maciej A Mazurowski, Haoyu Dong, Hanxue Gu, Jichen Yang, Nicholas Konz, and Yixin Zhang. Segment anything model for medical image analysis: an experimental study. arXiv preprint arXiv:2304.10517 , 2023.

Sheng He, Rina Bao, Jingpeng Li, P Ellen Grant, and Yangming Ou. Accuracy of segmentanything model (sam) in medical image segmentation tasks. arXiv preprint arXiv:2304.09324 , 2023.

Jun Ma and Bo Wang. Segment anything in medical images. arXiv preprint arXiv:2304.12306 , 2023.

Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg, Wan-Yen Lo, Piotr Dollár, and Ross Girshick. Segment anything. arXiv:2304.02643 , 2023.

Hui Sun, Wenju Zhou, and Minrui Fei. A survey on graph matching in computer vision. In 2020 13th International Congress on Image and Signal Processing, BioMedical Engineering and Informatics (CISP-BMEI) , pages 225-230. IEEE, 2020.

Stefan Haller, Lorenz Feineis, Lisa Hutschenreiter, Florian Bernard, Carsten Rother, Dagmar Kainmüller, Paul Swoboda, and Bogdan Savchynskyy. A comparative study of graph matching algorithms in computer vision. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXIII , pages 636-653. Springer, 2022.

Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, and Stéphane Deny. Barlow twins: Selfsupervised learning via redundancy reduction. In International Conference on Machine Learning , pages 12310-12320. PMLR, 2021.

Adrien Bardes, Jean Ponce, and Yann LeCun. Vicregl: Self-supervised learning of local visual features. In NeurIPS , 2022.

Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning , pages 1597-1607. PMLR, 2020.

Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 9729-9738, 2020.

Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved baselines with momentum contrastive learning. arXiv preprint arXiv:2003.04297 , 2020.

Nenad Tomasev, Ioana Bica, Brian McWilliams, Lars Buesing, Razvan Pascanu, Charles Blundell, and Jovana Mitrovic. Pushing the limits of self-supervised resnets: Can we outperform supervised learning without labels on imagenet? arXiv preprint arXiv:2201.05119 , 2022.

Shixiang Tang, Feng Zhu, Lei Bai, Rui Zhao, Chenyu Wang, and Wanli Ouyang. Unifying visual contrastive learning for object recognition from a graph perspective. In Computer VisionECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXVI , pages 649-667. Springer, 2022.

Mathilde Caron, Piotr Bojanowski, Armand Joulin, and Matthijs Douze. Deep clustering for unsupervised learning of visual features. In Proceedings of the European conference on computer vision (ECCV) , pages 132-149, 2018.

Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. Advances in neural information processing systems , 33:9912-9924, 2020.

Xiaohang Zhan, Jiahao Xie, Ziwei Liu, Yew-Soon Ong, and Chen Change Loy. Online deep clustering for unsupervised representation learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 6688-6697, 2020.

Zhirong Wu, Yuanjun Xiong, Stella X Yu, and Dahua Lin. Unsupervised feature learning via non-parametric instance discrimination. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 3733-3742, 2018.

Xinlei Chen, Saining Xie, and Kaiming He. An empirical study of training self-supervised vision transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 9640-9649, 2021.

Quentin Garrido, Yubei Chen, Adrien Bardes, Laurent Najman, and Yann Lecun. On the duality between contrastive and non-contrastive self-supervised learning. arXiv preprint arXiv:2206.02574 , 2022.

Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a new approach to self-supervised learning. Advances in neural information processing systems , 33:21271-21284, 2020.

Kuang-Huei Lee, Anurag Arnab, Sergio Guadarrama, John Canny, and Ian Fischer. Compressive visual representations. Advances in Neural Information Processing Systems , 34: 19538-19552, 2021.

Tete Xiao, Colorado J Reed, Xiaolong Wang, Kurt Keutzer, and Trevor Darrell. Region similarity representation learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 10539-10548, 2021.

Xinlong Wang, Rufeng Zhang, Chunhua Shen, Tao Kong, and Lei Li. Dense contrastive learning for self-supervised visual pre-training. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 3024-3033, 2021.

Xinrui Zhuang, Yuexiang Li, Yifan Hu, Kai Ma, Yujiu Yang, and Yefeng Zheng. Selfsupervised feature learning for 3d medical images by playing a rubik's cube. In International Conference on Medical Image Computing and Computer-Assisted Intervention , pages 420-428. Springer, 2019.

Liang Chen, Paul Bentley, Kensaku Mori, Kazunari Misawa, Michitaka Fujiwara, and Daniel Rueckert. Self-supervised learning for medical image analysis using image context restoration. Medical image analysis , 58:101539, 2019.

Zongwei Zhou, Vatsal Sodha, Jiaxuan Pang, Michael B Gotway, and Jianming Liang. Models genesis. Medical image analysis , 67:101840, 2021.

Fatemeh Haghighi, Mohammad Reza Hosseinzadeh Taher, Zongwei Zhou, Michael B Gotway, and Jianming Liang. Transferable visual words: Exploiting the semantics of anatomical patterns for self-supervised learning. IEEE transactions on medical imaging , 40(10):28572868, 2021.

Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. On the opportunities and risks of foundation models. arXiv preprint arXiv:2108.07258 , 2021.

Junke Wang, Dongdong Chen, Zuxuan Wu, Chong Luo, Luowei Zhou, Yucheng Zhao, Yujia Xie, Ce Liu, Yu-Gang Jiang, and Lu Yuan. Omnivl: One foundation model for image-language and video-language tasks. arXiv preprint arXiv:2209.07526 , 2022.

Jiayi Ma, Xingyu Jiang, Aoxiang Fan, Junjun Jiang, and Junchi Yan. Image matching from handcrafted to deep features: A survey. International Journal of Computer Vision , 129:23-79, 2021.

Dongliang Cao and Florian Bernard. Self-supervised learning for multimodal non-rigid 3d shape matching. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 17735-17744, 2023.

Alper Yilmaz, Omar Javed, and Mubarak Shah. Object tracking: A survey. Acm computing surveys (CSUR) , 38(4):13-es, 2006.

Jeongseok Hyun, Myunggu Kang, Dongyoon Wee, and Dit-Yan Yeung. Detection recovery in online multi-object tracking with sparse graph tracker. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision , pages 4850-4859, 2023.

Tobias Heimann and Hans-Peter Meinzer. Statistical shape models for 3d medical image segmentation: a review. Medical image analysis , 13(4):543-563, 2009.

Paul Roetzer, Zorah Lähner, and Florian Bernard. Conjugate product graphs for globally optimal 2d-3d shape matching. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 21866-21875, 2023.

Yikai Bian, Le Hui, Jianjun Qian, and Jin Xie. Unsupervised domain adaptation for point cloud semantic segmentation via graph matching. In 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) , pages 9899-9904. IEEE, 2022.

Zesen Wu and Mang Ye. Unsupervised visible-infrared person re-identification via progressive graph matching and alternate learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 9548-9558, 2023.

Liang Peng, Nan Wang, Jie Xu, Xiaofeng Zhu, and Xiaoxiao Li. Gate: graph cca for temporal self-supervised learning for label-efficient fmri analysis. IEEE Transactions on Medical Imaging , 42(2):391-402, 2022.

Chang Liu, Shaofeng Zhang, Xiaokang Yang, and Junchi Yan. Self-supervised learning of visual graph matching. In European Conference on Computer Vision , pages 370-388. Springer, 2022.

Ron Zass and Amnon Shashua. Probabilistic graph and hypergraph matching. In 2008 IEEE Conference on Computer Vision and Pattern Recognition , pages 1-8. IEEE, 2008.

Feng Zhou and Fernando De la Torre. Factorized graph matching. IEEE transactions on pattern analysis and machine intelligence , 38(9):1774-1789, 2015.

Marin Vlastelica Poganˇ ci´ c, Anselm Paulus, Vit Musil, Georg Martius, and Michal Rolinek. Differentiation of blackbox combinatorial solvers. In International Conference on Learning Representations , 2020.

Michal Rolínek, Paul Swoboda, Dominik Zietlow, Anselm Paulus, Vít Musil, and Georg Martius. Deep graph matching via blackbox differentiation of combinatorial solvers. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XXVIII 16 , pages 407-424. Springer, 2020.

Mathias Niepert, Pasquale Minervini, and Luca Franceschi. Implicit MLE: backpropagating through discrete exponential family distributions. In NeurIPS , Proceedings of Machine Learning Research. PMLR, 2021.

Thomas N. Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings . OpenReview.net, 2017. URL https://openreview.net/forum?id=SJU4ayYgl .

Felix Wu, Amauri Souza, Tianyi Zhang, Christopher Fifty, Tao Yu, and Kilian Weinberger. Simplifying graph convolutional networks. In International conference on machine learning , pages 6861-6871. PMLR, 2019.

Junwei Yang, Ke Zhang, Zhaolin Cui, Jinming Su, Junfeng Luo, and Xiaolin Wei. Inscon: instance consistency feature representation via self-supervised learning. arXiv preprint arXiv:2203.07688 , 2022.

Zhenda Xie, Yutong Lin, Zheng Zhang, Yue Cao, Stephen Lin, and Han Hu. Propagate yourself: Exploring pixel-level consistency for unsupervised visual representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 16684-16693, 2021.

Rainer E Burkard, Eranda Cela, Panos M Pardalos, and Leonidas S Pitsoulis. The quadratic assignment problem . Springer, 1998.

Paul Swoboda, Carsten Rother, Hassan Abu Alhaija, Dagmar Kainmuller, and Bogdan Savchynskyy. A study of lagrangean decompositions and dual ascent solvers for graph matching. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 16071616, 2017.

Michal Rolínek, Vít Musil, Anselm Paulus, Marin Vlastelica, Claudio Michaelis, and Georg Martius. Optimizing rank-based metrics with blackbox differentiation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 7620-7630, 2020.

Pasquale Minervini, Luca Franceschi, and Mathias Niepert. Adaptive perturbation-based gradient estimation for discrete latent variable models. arXiv preprint arXiv:2209.04862 , 2022.

George Papandreou and Alan L Yuille. Perturb-and-map random fields: Using discrete optimization to learn and sample from energy models. In 2011 International Conference on Computer Vision , pages 193-200. IEEE, 2011.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 770-778, 2016.

Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929 , 2020.

Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A largescale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition , pages 248-255. Ieee, 2009.

Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 , 2014.

Spyridon Bakas, Mauricio Reyes, Andras Jakab, Stefan Bauer, Markus Rempfler, Alessandro Crimi, Russell Takeshi Shinohara, Christoph Berger, Sung Min Ha, Martin Rozycki, et al. Identifying the best machine learning algorithms for brain tumor segmentation, progression assessment, and overall survival prediction in the brats challenge. arXiv preprint arXiv:1811.02629 , 2018.

Xiahai Zhuang and Juan Shen. Multi-scale patch and multi-modality atlases for whole heart segmentation of mri. Medical image analysis , 31:77-87, 2016.

Noel Codella, Veronica Rotemberg, Philipp Tschandl, M Emre Celebi, Stephen Dusza, David Gutman, Brian Helba, Aadi Kalloo, Konstantinos Liopyris, Michael Marchetti, et al. Skin lesion analysis toward melanoma detection 2018: A challenge hosted by the international skin imaging collaboration (isic). arXiv preprint arXiv:1902.03368 , 2019.

Junji Shiraishi, Shigehiko Katsuragawa, Junpei Ikezoe, Tsuneo Matsumoto, Takeshi Kobayashi, Ken-ichi Komatsu, Mitate Matsui, Hiroshi Fujita, Yoshie Kodera, and Kunio Doi. Development of a digital image database for chest radiographs with and without a lung nodule: receiver operating characteristic analysis of radiologists' detection of pulmonary nodules. American Journal of Roentgenology , 174(1):71-74, 2000.

Konstantin Pogorelov, Kristin Ranheim Randel, Carsten Griwodz, Sigrun Losada Eskeland, Thomas de Lange, Dag Johansen, Concetto Spampinato, Duc-Tien Dang-Nguyen, Mathias Lux, Peter Thelin Schmidt, et al. Kvasir: A multi-class image dataset for computer aided gastrointestinal disease detection. In Proceedings of the 8th ACM on Multimedia Systems Conference , pages 164-169, 2017.

Joes Staal, Michael D Abràmoff, Meindert Niemeijer, Max A Viergever, and Bram Van Ginneken. Ridge-based vessel segmentation in color images of the retina. IEEE transactions on medical imaging , 23(4):501-509, 2004.

Walid Al-Dhabyani, Mohammed Gomaa, Hussien Khaled, and Aly Fahmy. Dataset of breast ultrasound images. Data in brief , 28:104863, 2020.

Yi Zhou, Boyang Wang, Lei Huang, Shanshan Cui, and Ling Shao. A benchmark for studying diabetic retinopathy: segmentation, grading, and transferability. IEEE Transactions on Medical Imaging , 40(3):818-828, 2020.

Quande Liu, Qi Dou, Lequan Yu, and Pheng Ann Heng. Ms-net: Multi-site network for improving prostate segmentation with heterogeneous mri data. IEEE Transactions on Medical Imaging , 2020.

Ha Q Nguyen, Khanh Lam, Linh T Le, Hieu H Pham, Dat Q Tran, Dung B Nguyen, Dung D Le, Chi M Pham, Hang TT Tong, Diep H Dinh, et al. Vindr-cxr: An open dataset of chest x-rays with radiologist's annotations. Scientific Data , 9(1):429, 2022.

Sharif Amit Kamran, Khondker Fariha Hossain, Alireza Tavakkoli, Stewart Lee Zuckerbrod, Kenton M Sanders, and Salah A Baker. Rv-gan: Segmenting retinal vascular structure in fundus photographs using a novel multi-scale generative adversarial network. In Medical Image Computing and Computer Assisted Intervention-MICCAI 2021: 24th International Conference, Strasbourg, France, September 27-October 1, 2021, Proceedings, Part VIII 24 , pages 34-44. Springer, 2021.

Andriy Myronenko. 3d mri brain tumor segmentation using autoencoder regularization. In Brainlesion: Glioma, Multiple Sclerosis, Stroke and Traumatic Brain Injuries: 4th International Workshop, BrainLes 2018, Held in Conjunction with MICCAI 2018, Granada, Spain, September 16, 2018, Revised Selected Papers, Part II 4 , pages 311-320. Springer, 2019.

Jieneng Chen, Yongyi Lu, Qihang Yu, Xiangde Luo, Ehsan Adeli, Yan Wang, Le Lu, Alan L Yuille, and Yuyin Zhou. Transunet: Transformers make strong encoders for medical image segmentation. arXiv preprint arXiv:2102.04306 , 2021.

Ozan Oktay, Jo Schlemper, Loic Le Folgoc, Matthew Lee, Mattias Heinrich, Kazunari Misawa, Kensaku Mori, Steven McDonagh, Nils Y Hammerla, Bernhard Kainz, et al. Attention u-net: Learning where to look for the pancreas. arXiv preprint arXiv:1804.03999 , 2018.

Z Zhou, MMR Siddiquee, N Tajbakhsh, and J UNet+ Liang. A nested u-net architecture for medical image segmentation (2018). arXiv preprint arXiv:1807.10165 , 2018.

Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the IEEE/CVF international conference on computer vision , pages 9650-9660, 2021.

Ali Hatamizadeh, Yucheng Tang, Vishwesh Nath, Dong Yang, Andriy Myronenko, Bennett Landman, Holger R Roth, and Daguang Xu. Unetr: Transformers for 3d medical image segmentation. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision , pages 574-584, 2022.

Joao Carreira and Andrew Zisserman. Quo vadis, action recognition? a new model and the kinetics dataset. In proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 6299-6308, 2017.

Eli Gibson, Wenqi Li, Carole Sudre, Lucas Fidon, Dzhoshkun I Shakir, Guotai Wang, Zach Eaton-Rosen, Robert Gray, Tom Doel, Yipeng Hu, et al. NiftyNet: a deep-learning platform for medical imaging. Computer methods and programs in biomedicine , 158:113-122, 2018.

Sihong Chen, Kai Ma, and Yefeng Zheng. Med3d: Transfer learning for 3d medical image analysis. arXiv:1904.00625 , 2019.

Xiaoman Zhang, Ya Zhang, Xiaoyun Zhang, and Yanfeng Wang. Universal model for 3d medical image analysis. ArXiv , abs/2010.06107, 2020.

Yucheng Tang, Dong Yang, Wenqi Li, Holger R Roth, Bennett Landman, Daguang Xu, Vishwesh Nath, and Ali Hatamizadeh. Self-supervised pre-training of swin transformers for 3d medical image analysis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 20730-20740, 2022.

Duy MH Nguyen, Hoang Nguyen, Mai TN Truong, Tri Cao, Binh T Nguyen, Nhat Ho, Paul Swoboda, Shadi Albarqouni, Pengtao Xie, and Daniel Sonntag. Joint self-supervised image-volume representation learning with intra-inter contrastive clustering. arXiv preprint arXiv:2212.01893 , 2022.

Zhiwen Lin, Ruoqian Guo, Yanjie Wang, Bian Wu, Tingting Chen, Wenzhe Wang, Danny Z Chen, and Jian Wu. A framework for identifying diabetic retinopathy based on anti-noise detection and attention-based fusion. In Medical Image Computing and Computer Assisted Intervention-MICCAI 2018: 21st International Conference, Granada, Spain, September 16-20, 2018, Proceedings, Part II 11 , pages 74-82. Springer, 2018.

Yu-Huan Wu, Shang-Hua Gao, Jie Mei, Jun Xu, Deng-Ping Fan, Rong-Guo Zhang, and Ming-Ming Cheng. Jcs: An explainable covid-19 diagnosis system by joint classification and segmentation. IEEE Transactions on Image Processing , 30:3113-3126, 2021.

Yi Zhou, Xiaodong He, Lei Huang, Li Liu, Fan Zhu, Shanshan Cui, and Ling Shao. Collaborative learning of semi-supervised segmentation and classification for medical images. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 2079-2088, 2019.

Hasan Md Tusfiqur, Duy MH Nguyen, Mai TN Truong, Triet A Nguyen, Binh T Nguyen, Michael Barz, Hans-Juergen Profitlich, Ngoc TT Than, Ngan Le, Pengtao Xie, et al. Drgnet: Interactive joint learning of multi-lesion segmentation and classification for diabetic retinopathy grading. arXiv preprint arXiv:2212.14615 , 2022.

Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In Medical Image Computing and Computer-Assisted Intervention-MICCAI 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part III 18 , pages 234-241. Springer, 2015.

Shruti Jadon. A survey of loss functions for semantic segmentation. In 2020 IEEE conference on computational intelligence in bioinformatics and computational biology (CIBCB) , pages 1-7. IEEE, 2020.

Ross Girshick. Fast r-cnn. In Proceedings of the IEEE international conference on computer vision , pages 1440-1448, 2015.

Arnaud AA Setio, Colin Jacobs, Jaap Gelderblom, and Bram van Ginneken. Automatic detection of large pulmonary solid nodules in thoracic ct images. Medical physics , 42(10): 5642-5653, 2015.

Patrick Bilic, Patrick Ferdinand Christ, Eugene Vorontsov, Grzegorz Chlebus, Hao Chen, Qi Dou, Chi-Wing Fu, Xiao Han, Pheng-Ann Heng, Jürgen Hesser, et al. The liver tumor segmentation benchmark (LiTS). arXiv:1901.04056 , 2019.

Amber L Simpson, Michela Antonelli, Spyridon Bakas, Michel Bilello, Keyvan Farahani, Bram Van Ginneken, Annette Kopp-Schneider, Bennett A Landman, Geert Litjens, Bjoern Menze, et al. A large annotated medical image dataset for the development and evaluation of segmentation algorithms. arXiv preprint arXiv:1902.09063 , 2019.

Hanna Borgli, Vajira Thambawita, Pia H Smedsrud, Steven Hicks, Debesh Jha, Sigrun L Eskeland, Kristin Ranheim Randel, Konstantin Pogorelov, Mathias Lux, Duc Tien Dang Nguyen, et al. Hyperkvasir, a comprehensive multi-class image and video dataset for gastrointestinal endoscopy. Scientific data , 7(1):283, 2020.

Bastiaan S Veeling, Jasper Linmans, Jim Winkens, Taco Cohen, and Max Welling. Rotation equivariant CNNs for digital pathology. June 2018.

Babak Ehteshami Bejnordi, Mitko Veta, Paul Johannes Van Diest, Bram Van Ginneken, Nico Karssemeijer, Geert Litjens, Jeroen AWM Van Der Laak, Meyke Hermsen, Quirine F Manson, Maschenka Balkenhol, et al. Diagnostic assessment of deep learning algorithms for detection of lymph node metastases in women with breast cancer. Jama , 318(22):2199-2210, 2017.

Bjoern H. Menze, Andras Jakab, Stefan Bauer, Jayashree Kalpathy-Cramer, Keyvan Farahani, Justin Kirby, Yuliya Burren, Nicole Porz, Johannes Slotboom, Roland Wiest, Levente Lanczi, Elizabeth Gerstner, Marc-André Weber, Tal Arbel, Brian B. Avants, Nicholas Ayache, Patricia Buendia, D. Louis Collins, Nicolas Cordier, Jason J. Corso, Antonio Criminisi, Tilak Das, Hervé Delingette, Ça˘ gatay Demiralp, Christopher R. Durst, Michel Dojat, Senan Doyle, Joana Festa, Florence Forbes, Ezequiel Geremia, Ben Glocker, Polina Golland, Xiaotao Guo, Andac Hamamci, Khan M. Iftekharuddin, Raj Jena, Nigel M. John, Ender Konukoglu, Danial Lashkari, José António Mariz, Raphael Meier, Sérgio Pereira, Doina Precup, Stephen J. Price, Tammy Riklin Raviv, Syed M. S. Reza, Michael Ryan, Duygu Sarikaya, Lawrence Schwartz, Hoo-Chang Shin, Jamie Shotton, Carlos A. Silva, Nuno Sousa, Nagesh K. Subbanna, Gabor Szekely, Thomas J. Taylor, Owen M. Thomas, Nicholas J. Tustison, Gozde Unal, Flor Vasseur, Max Wintermark, Dong Hye Ye, Liang Zhao, Binsheng Zhao, Darko Zikic, Marcel Prastawa, Mauricio Reyes, and Koen Van Leemput. The multimodal brain tumor image segmentation benchmark (brats). IEEE Transactions on Medical Imaging , 34(10):1993-2024, 2015. doi: 10.1109/TMI.2014.2377694.

Christopher T Lloyd, Alessandro Sorichetta, and Andrew J Tatem. High resolution global gridded data for use in population studies. Scientific data , 4(1):1-17, 2017.

Aaron J Grossberg, Abdallah SR Mohamed, Hesham Elhalawani, William C Bennett, Kirk E Smith, Tracy S Nolan, Bowman Williams, Sasikarn Chamchod, Jolien Heukelom, Michael E Kantor, et al. Imaging and clinical data archive for head and neck squamous cell carcinoma patients treated with radiotherapy. Scientific data , 5(1):1-10, 2018.

Patrick Bilic, Patrick Christ, Hongwei Bran Li, Eugene Vorontsov, Avi Ben-Cohen, Georgios Kaissis, Adi Szeskin, Colin Jacobs, Gabriel Efrain Humpire Mamani, Gabriel Chartrand, et al. The liver tumor segmentation benchmark (lits). Medical Image Analysis , 84:102680, 2023.

Jennifer Yin Yee Kwan, Jie Su, Shao Hui Huang, Laleh S Ghoraie, Wei Xu, Biu Chan, Kenneth W Yip, Meredith Giuliani, Andrew Bayley, John Kim, et al. Radiomic biomarkers to refine risk models for distant metastasis in hpv-related oropharyngeal carcinoma. International Journal of Radiation Oncology* Biology* Physics , 102(4):1107-1116, 2018.

Kenneth Clark, Bruce Vendt, Kirk Smith, John Freymann, Justin Kirby, Paul Koppel, Stephen Moore, Stanley Phillips, David Maffitt, Michael Pringle, et al. The cancer imaging archive (tcia): maintaining and operating a public information repository. Journal of digital imaging , 26:1045-1057, 2013.

JYY Kwan et al. Data from radiomic biomarkers to refine risk models for distant metastasis in oropharyngeal carcinoma, the cancer imaging archive, 2019.

P Leavey, A Sengupta, D Rakheja, O Daescu, HB Arunachalam, and R Mishra. Osteosarcoma data from ut southwestern/ut dallas for viable and necrotic tumor assessment [data set]. The Cancer Imaging Archive , 14, 2019.

AA Yorke, GC McDonald, D Solis, and T Guerrero. Pelvic reference data. Cancer Imaging Arch , 2019.

Holger R Roth, Amal Farag, E Turkbey, Le Lu, Jiamin Liu, and Ronald M Summers. Data from pancreas-ct. the cancer imaging archive. IEEE Transactions on Image Processing , 2016.

Holger R Roth, Le Lu, Amal Farag, Hoo-Chang Shin, Jiamin Liu, Evrim B Turkbey, and Ronald M Summers. Deeporgan: Multi-level deep convolutional networks for automated pancreas segmentation. In Medical Image Computing and Computer-Assisted InterventionMICCAI 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part I 18 , pages 556-564. Springer, 2015.

Geert Litjens, Oscar Debats, Jelle Barentsz, Nico Karssemeijer, and Henkjan Huisman. Computer-aided detection of prostate cancer in mri. IEEE transactions on medical imaging , 33(5):1083-1092, 2014.

Geert Litjens, Oscar Debats, Jelle Barentsz, Nico Karssemeijer, and Henkjan Huisman. Prostatex challenge data. The Cancer Imaging Archive, 2017. https://doi.org/10.7937/K9/ TCIA.2017.SMPYMRXQ .

FR Lucchesi and ND Aredes. Radiology data from the cancer genome atlas cervical squamous cell carcinoma and endocervical adenocarcinoma (tcga-cesc) collection. the cancer imaging archive. Cancer Imaging Arch , 2016.

S Kirk, Y Lee, CA Sadow, S Levine, C Roche, E Bonaccio, and J Filiippini. Radiology data from the cancer genome atlas colon adenocarcinoma [tcga-coad] collection. The Cancer Imaging Archive , 2016.

F. R. Lucchesi and N. D. Aredes. The cancer genome atlas esophageal carcinoma collection (tcga-esca) (version 3) [data set]. The Cancer Imaging Archive, 2016.

MWLinehan, R Gautam, CA Sadow, and S Levine. Radiology data from the cancer genome atlas kidney chromophobe [tcga-kich] collection. The Cancer Imaging Archive , 2016.

O Akin, P Elnajjar, M Heller, R Jarosz, B Erickson, S Kirk, and J Filippini. Radiology data from the cancer genome atlas kidney renal clear cell carcinoma [tcga-kirc] collection. The Cancer Imaging Archive , 1310, 2016.

C Roche, E Bonaccio, and J Filippini. Radiology data from the cancer genome atlas sarcoma [tcga-sarc] collection. Cancer Imaging Arch. https://doi. org/10.7937 K , 9, 2016.

Shenggan. Bccd dataset, 2017. URL https://github.com/Shenggan/BCCD_Dataset .

Shiv Gehlot, Anubha Gupta, and Ritu Gupta. Sdct-auxnet θ : Dct augmented stain deconvolutional cnn with auxiliary classifier for cancer diagnosis. Medical image analysis , 61:101661, 2020.

A Gupta and R Gupta. All challenge dataset of isbi 2019 [data set]. URL https://doi.org/10.7937/tcia.2019.dc64i46r , 2019.

Rebecca Sawyer Lee, Francisco Gimenez, Assaf Hoogi, Kanae Kawai Miyake, Mia Gorovoy, and Daniel L Rubin. A curated mammography data set for use in computer-aided detection and diagnosis research. Scientific data , 4(1):1-9, 2017.

R Sawyer Lee, F Gimenez, A Hoogi, and D Rubin. Curated breast imaging subset of ddsm. The Cancer Imaging Archive. DOI: https://doi. org/10.7937 K , 9, 2016.

Linda Wang, Zhong Qiu Lin, and Alexander Wong. Covid-net: a tailored deep convolutional neural network design for detection of covid-19 cases from chest x-ray images. Scientific Reports , 10(1):19549, Nov 2020. ISSN 2045-2322. doi: 10.1038/s41598-020-76550-z. URL https://doi.org/10.1038/s41598-020-76550-z .

Daniel Kermany, Kang Zhang, Michael Goldbaum, et al. Labeled optical coherence tomography (oct) and chest x-ray images for classification. Mendeley data , 2(2):651, 2018.

Salman Maqbool, Aqsa Riaz, Hasan Sajid, and Osman Hasan. m2caiseg: Semantic segmentation of laparoscopic images using convolutional neural networks. arXiv preprint arXiv:2008.10134 , 2020.

Mohamed Amgad, Habiba Elfandy, Hagar Hussein, Lamees A Atteya, Mai A T Elsebaie, Lamia S Abo Elnasr, Rokia A Sakr, Hazem S E Salem, Ahmed F Ismail, Anas M Saad, Joumana Ahmed, Maha A T Elsebaie, Mustafijur Rahman, Inas A Ruhban, Nada M Elgazar, Yahya Alagha, Mohamed H Osman, Ahmed M Alhusseiny, Mariam M Khalaf, Abo-Alela F Younes, Ali Abdulkarim, Duaa M Younes, Ahmed M Gadallah, Ahmad M Elkashash, Salma Y Fala, Basma M Zaki, Jonathan Beezley, Deepak R Chittajallu, David Manthey, David A Gutman, and Lee A D Cooper. Structured crowdsourcing enables convolutional segmentation of histology images. Bioinformatics , 35(18):3461-3467, 02 2019. ISSN 1367-4803. doi: 10.1093/ bioinformatics/btz083. URL https://doi.org/10.1093/bioinformatics/btz083 .

Fabio Cuzzolin, Vivek Singh Bawa, Inna Skarga-Bandurova, Mohamed Mohamed, Jackson Ravindran Charles, Elettra Oleari, Alice Leporini, Carmela Landolfo, Armando Stabile, Francesco Setti, et al. Saras challenge on multi-domain endoscopic surgeon action detection, 2021.

Vivek Singh Bawa, Gurkirt Singh, Francis KapingA, Inna Skarga-Bandurova, Elettra Oleari, Alice Leporini, Carmela Landolfo, Pengfei Zhao, Xi Xiang, Gongning Luo, Kuanquan Wang, Liangzhi Li, Bowen Wang, Shang Zhao, Li Li, Armando Stabile, Francesco Setti, Riccardo Muradore, and Fabio Cuzzolin. The saras endoscopic surgeon action detection (esad) dataset: Challenges and methods, 2021.

Vivek Singh Bawa, Gurkirt Singh, Francis KapingA, Inna Skarga-Bandurova, Alice Leporini, Carmela Landolfo, Armando Stabile, Francesco Setti, Riccardo Muradore, Elettra Oleari, and Fabio Cuzzolin. Esad: Endoscopic surgeon action detection dataset, 2020.

Shoulder x-ray classification. Kaggle dataset . URL https://www.kaggle.com/datasets/ dryari5/shoulder-xray-classification .

Lung masks for shenzhen hospital chest x-ray set. Kaggle dataset . URL https://www. kaggle.com/datasets/yoctoman/shcxr-lung-mask .

Susanne G Mueller, Michael W Weiner, Leon J Thal, Ronald C Petersen, Clifford Jack, William Jagust, John Q Trojanowski, Arthur W Toga, and Laurel Beckett. The alzheimer's disease neuroimaging initiative. Neuroimaging Clinics , 15(4):869-877, 2005.

Ronald Carl Petersen, Paul S Aisen, Laurel A Beckett, Michael C Donohue, Anthony Collins Gamst, Danielle J Harvey, Clifford R Jack, William J Jagust, Leslie M Shaw, Arthur W Toga, et al. Alzheimer's disease neuroimaging initiative (adni): clinical characterization. Neurology , 74(3):201-209, 2010.

Christian Matek, Simone Schwarz, Carsten Marr, and Karsten Spiekermann. A single-cell morphological dataset of leukocytes from aml patients and non-malignant controls (aml-cytomorphology_lmu). The Cancer Imaging Archive (TCIA)[Internet] , 2019.

Christian Matek, Simone Schwarz, Karsten Spiekermann, and Carsten Marr. Human-level recognition of blast cells in acute myeloid leukaemia with convolutional neural networks. Nature Machine Intelligence , 1(11):538-544, 2019.

Aptos 2019 blindness detection. Kaggle dataset . URL https://www.kaggle.com/c/ aptos2019-blindness-detection/data .

Mohamed Amgad, Habiba Elfandy, Hagar Hussein, Lamees A Atteya, Mai AT Elsebaie, Lamia S Abo Elnasr, Rokia A Sakr, Hazem SE Salem, Ahmed F Ismail, Anas M Saad, et al. Structured crowdsourcing enables convolutional segmentation of histology images. Bioinformatics , 35(18):3461-3467, 2019.

Amir Abdi and S Kasaei. Panoramic dental x-rays with segmented mandibles. Mendeley Data, v2 , 2020.

Thomas LA van den Heuvel, Dagmar de Bruijn, Chris L de Korte, and Bram van Ginneken. Automated measurement of fetal head circumference using 2d ultrasound images. PloS one , 13(8):e0200412, 2018.

Hippocampus segmentation in mri images. Kaggle dataset . URL https://www.kaggle. com/datasets/andrewmvd/hippocampus-segmentation-in-mri-images?select= Test .

Skin lesion images for melanoma classification. Kaggle dataset . URL https://www.kaggle. com/datasets/andrewmvd/isic-2019 .

Ahmed Taha, Pechin Lo, Junning Li, and Tao Zhao. Kid-net: convolution networks for kidney vessels segmentation from ct-volumes. In Medical Image Computing and Computer Assisted Intervention-MICCAI 2018: 21st International Conference, Granada, Spain, September 16-20, 2018, Proceedings, Part IV 11 , pages 463-471. Springer, 2018.

Kvasir v2. Kaggle dataset . URL https://www.kaggle.com/datasets/plhalvorsen/ kvasir-v2-a-gastrointestinal-tract-dataset .

Lhncbc malaria. https://lhncbc.nlm.nih.gov/LHC-downloads/downloads.html# malaria-datasets .

D. Wei, Z. Lin, D. Barranco, N. Wendt, X. Liu, W. Yin, X. Huang, A. Gupta, W. Jang, X. Wang, I. Arganda-Carreras, J. Lichtman, and H. Pfister. Mitoem dataset: Large-scale 3d mitochondria instance segmentation from em images. In International Conference on Medical Image Computing and Computer Assisted Intervention , 2020.

Christian Matek, Sebastian Krappe, Christian Münzenmayer, Torsten Haferlach, and Carsten Marr. Highly accurate differentiation of bone marrow cell morphologies using deep neural networks on a large image data set. Blood, The Journal of the American Society of Hematology , 138(20):1917-1927, 2021.

Safwan S Halabi, Luciano M Prevedello, Jayashree Kalpathy-Cramer, Artem B Mamonov, Alexander Bilbily, Mark Cicero, Ian Pan, Lucas Araújo Pereira, Rafael Teixeira Sousa, Nitamar Abdala, et al. The rsna pediatric bone age machine learning challenge. Radiology , 290(2): 498-503, 2019.

Kaggle dr dataset (eyepacs). Kaggle dataset . URL https://www.kaggle.com/datasets/ mariaherrerot/eyepacspreprocess .

Supplementary Material

We present below LVM-Med pseudo-code (Section A), implementations used in downstream tasks (Section B), additional ablation studies of LVM-Med (Section C), further prompt-based segmentation results on 3D datasets, image classification benchmark (Section D), predicted masks using the user-based prompt (Section E), and finally the dataset overview (Section F).

A LVM-Med Pseudo-code

First, we provide a pseudo-code for training LVM-Med in Pytorch style:

We trained LVM-Med with graph size of 16 nodes, each node connected to the top 5 nearest neighbors after using kNN, λ value in Algorithm 1 is 80 , and α = 0 . 8 for associating global- and local-based similarities when computing c v ij . The size of projector h ϕ is 2048 × 128 for ResNet-50, and 768 × 128 for ViT. We configure the message passing network g θ with two convolutional layers of size 128 . For the user-based prompt version, because the SAM model [9] requires an input of shape 256 × 14 × 14 for the mask decoder part, we add two additional convolutional layers with a kernel size of 1 and 3 at the end of ViT backbone to convert from shape 768 × 14 × 14 to the target shape.

B Downstream task setups

B.1 Downstream tasks

Segmentation tasks On 2D-based segmentation tasks, we employ U-Net architecture [91] and load ResNet-50 [60] trained by self-supervised learning algorithms as network backbones. With foundation models, we use TransUnet [76] and take pre-trained ViT models as the backbones. For the prompt-based segmentation, we follow the architecture of SAM [6] consisting of encoder, prompt, and mask decoder layers. We also fine-tune SAM where encoder and prompt networks are frozen, only learning decoder layers [9]. Our LVM-Med for prompt-based setting is similar to [9] except that we substitute SAM's encoders with our weights. We utilize Adam optimizer for all experiments and train architectures with Dice and Cross-Entropy loss [92]. We also normalize the norm-2 of gradient values to stabilize the training step to maximize 1. Table 7 summarizes each dataset's learning rate, number of epochs, and image resolution.

On 3D-based segmentations, we reformulate these tasks as 2D segmentation problems and make predictions on 2D slices taken from 3D volumes. Furthermore, we apply balance sampling to select equally 2D slices covering target regions and other 2D slices, not including the ground truth. Table 8 presents configurations used for 3D datasets; other settings are identical to 2D cases.

Image classification tasks Wetake the feature embedding outputs of each architecture and build one fully connected layer to produce desired classes for image classification tasks. We freeze the encoder layers for the linear evaluation and only train the fully connected layer. For the fully-finetuning, the whole network is trained. The Adam optimizer [63] with cross-entropy loss function and learning rates { 5 × 10 -4 , 10 -3 } are used for Brain Tumor and FGADR, respectively. To benchmark LVMMed with other state-of-the-art methods on FGADR (Figure 3 in paper), we follow the settings of DRG-Net [90] and change their encoder layers by our networks.

Object detection We use Faster-RCNN [93] for object detection tasks. The ResNet-50 of FasterRCNN is replaced by pre-trained weights. In the Vin-Dr dataset, there is a total of 14 objects for, e.g., Aortic enlargement, Atelectasis, Calcification, etc. We use image resolutions of 512 × 512 , Adam solver, and learning rate 10 -4 in 40 epochs. In the Kvasir dataset for polyp detection, we also resize images to a fixed size of 512 × 512 , employ the Adam optimizer with learning rate 2 . 5 × 10 -4 and batch size 8 .

C LVM-Med ablation studies

C.1 Graph sizes and λ in backpropagation

We provide in Figure 5 and Figure 6 LVM-Med performance when changing the number of nodes in graph construction steps G s , G t and λ = 80 used in Algorithm 1 in the backpropagation step. The results are reported on the average Dice score of five 2D segmentation tasks and the average accuracy of two linear classifications on FGADR and Brain Tumor Classification. Figure 5 indicates that 16 is the best value for both classification and segmentation. Increasing the graph's nodes tends to decrease classification performance.

Figure 6 compared different values for λ ∈ { 70 , 80 , 90 , 100 } . We observe that λ = { 80 , 90 } achieve good results for linear classification tasks though λ = { 90 , 100 } decreases segmentation performance.

C.2 Performance on large- and small-scale

We investigate LVM-Med performance when reducing the number of datasets in the pre-training step. Especially, we trained LVM-Med on a small-scale with four datasets: LUNA2016 [94], LiTS2017 [95], BraTS2018 [64], and MSD (Heart) [96]. We compare this version with our default settings trained on 55 datasets (Section F). Two models are evaluated on dice scores of five 2D segmentation tasks, the accuracy metric of two linear image classifications, and mAP50 of two object detection tasks on VinDr and Kvasir detection. Table 9 shows that LMV-Med full leads to better performance overall, especially with the classification settings; the improvement gap is around 3 . 6% . In summary, we conclude that LVM-Med is beneficial when training in large-scale medical settings.

50

60

70

80

90

100

8

16

24

32

Seg. (Dice)

50

60

70

80

90

100

70

80

90

100

Seg. (Dice)

C.3 Performance on weighting global and local similarities

We test with different α = { 0 . 7 , 0 . 8 , 0 . 9 } which used to fuse global- and local-based similarities c v ij . Table 9 demonstrates that α = 0 . 8 is generally the best value in average across segmentation, classification, and object detection tasks.

C.4 Computational complexity

We present a parameter comparison of LVM-Med with other foundation models in Table 10. Our LVM-Med model, based on ResNet-50, has significantly fewer parameters, approximately 3-4 times smaller than models such as Flava or SAM, while still maintaining competitive performance. When utilizing the ViT encoder pre-trained by the SAM method, LVM-Med's parameters are comparable to the Flava model and slightly higher than Clip and Align by 1 . 03 and 1 . 43 times, respectively. However, it is important to note that both LVM-Med and SAM outperform these models by a significant margin in various settings.

D Prompt-based segmentation on 3D datasets and classification tasks

We provide additional results for LVM-Med on 3D-based prompt segmentation and image classification tasks with several fully connected layers.

D.1 Promt-based Segmentation on 3D datasets

We perform experiments on three 3D datasets in Table 11, including BraTS, MMWHS-MRI, and MMWHS-CT. The setup for box prompts follows 2D segmentation cases. We discover that the LMV-Med in 3D cases consistently improves the performance of fine-tuned SAM [9] as in 2D settings and attains a large margin compared with SAM without training [6]. This evidence thus confirms that LVM-Med is also effective under prompt-based scenarios.

D.2 Image classification

We aim to inspect whether foundation models improve their performance given more fully connected layers for image classification tasks with both frozen encoders or fully fine-tuning. For each method in this category and our LVM-Med (ResNet-50 and ViT), we configure two fully connected layers with sizes 512 -256 and 512 -128 for the Brain and FGADR respectively that map from the output dimension of each network to a number of desired classes. Table 12 presents obtained results where new settings are highlighted in color. We notice the following points. (i) Firstly, using more fully connected layers tends to improve the performance of foundation models, especially on linear evaluation. For e.g., the Clip increases from 4 . 79% -9 . 98% on FGADR and Brain Tumor classification tasks, respectively. Similarly, our LVM-Med with SAM's ViT also achieves better results by approximately 1 . 37% and 4 . 82% on those tasks. (ii) Secondly, LVM-Med overall attains the best results in four settings using linear or several fully connected layers with ResNet-50. LVMMed with ViT architecture also delivers the best records on three of four test cases compared with foundation models.

E Visualizing results

We provide qualitative results for prompt-based segmentation in Figure 7. We compare three approaches, including (i) the standard SAM without fine-tuning [6] (second column), (ii) SAM

with encoders and prompt networks are frozen, and only decoder layers are trained as [7] (third column), and (iii) a similar setting as (ii) but encoders taken from LVM-Med version with SAM's ViT architecture (fourth column). For all methods, we simulate box-based prompts using the ground-truth masks and define boxes covering those target regions perturbed by offset values.

Figure 7 demonstrates that the original SAM is prone to generate useless predictions (top and bottom rows) or less precise boundaries. In contrast, updated SAM and LVM-Med produce more accurate results, confirming the importance of fine-tuning to achieve adequate results. Figures in the third and fourth columns also illustrate that SAM tends to over-segment or lacks structures on an object's edges in several cases, while LVM-Med is more stable in those situations (red arrows).

Ground Truth

SAM (no fine-tuning) [6]

SAM (fine-tuning) [9]

LVM-Med (SAM's ViT)

02/201

02/201

922013

02/2013

00:00

00:000

00:020

00:020

CVPS

Q:N

CVPS

CVPS

S:N

!A8

CVPS

Q:N

F Dataset overviews

Table 13 overviews the dataset used in our study. For each dataset, we provide its modality, data dimension, and the total of samples. If the training/testing rate is available (column Train/Test Rate ), we utilize all training data; otherwise, we sample 20% total samples to avoid potential test data leaking for downstream tasks used in the pre-training step. For datasets whose data dimensions are 3D volumes, we sample 2D slices from those formats. Some datasets, such as MSD or ADNI, comprise different sub-datasets inside; we consider these sub-sets as independent ones to avoid confusion during the training steps. In summary, a total of 55 datasets are used with approximately 40% in 3D datasets and 60% in 2D images as presented in Figure 8. Moreover, we also outline ratios between distinct data modalities such as MRI, CT, X-

ray, grayscale types such as Ultrasound, OCT, and finally, color images depicted in Figure 9.

2D datasets

60.0%

3D datasets

40.0%

43.6%

Other Grayscale

3.6%

CT

15.5%

MRI

24.5%

X-ray

12.7%

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis research paper focuses on LVM-Med, a self-supervised learning (SSL) technique designed for medical imaging tasks. Based on a second-order graph matching strategy, LVM-Med is trained on a large-scale medical imaging dataset. The researchers found that the method significantly improves performance on a variety of downstream medical imaging tasks compared to other supervised learning methods and foundation models trained on large quantities of image-text data. These findings were consistent across two different architectures: ResNet-50 and Vision Transformer (ViT).\n\nSTRENGTHS:\n- The LVM-Med method was evaluated on various tasks, including segmentation, object detection, and image classification. The results were compared to foundational models like Clip, Align, Flava, and SAM. It performed particularly well on eight medical segmentation tasks, outperforming both 2D SSL methods trained on the same dataset and foundational models.\n- The training of LVM-Med on a large-scale medical imaging dataset indicates its capacity to handle and learn from large amounts of data, which is often a requirement in the field of medical imaging.\n- The LVM-Med model outperforms both supervised learning methods and foundation models trained on hundreds of millions of image-text instances. This includes a variety of popular models such as Clip, Align, Flava, and SAM.\n- The results show that LVM-Med performs well in in-out-distribution settings, implying a certain level of robustness and generalizability.\n- LVM-Med provides benefits in both end-to-end and prompt-based segmentation tasks. This flexibility can be particularly useful in real-world applications, where various segmentation scenarios may be encountered.\n- The authors also conducted an ablation study, experimenting with variations in LVM-Med's configuration to assess the importance of various components in the overall performance. It was concluded that all factors contribute to the final performance, with the second-order graph matching and Gumbel noise being the most significant.\n\nWEAKNESSES:\n- Only 2D backbone is conducted: The LVM-Med model focuses primarily on 2D backbone. Author should provide more discussion about the challenge for applying this framework in 3D backbone.\n- The LVM-Med model is trained on a large-scale dataset, how can author endure the testing dataset is not leaky in training set. Moreover, the use of a single dataset could potentially limit its generalizability. Ensuring robust performance across diverse datasets from different sources is critical.\n- The model's performance benefits seem to be tied to its training on a large-scale medical imaging dataset. If such a dataset is not available, the performance of the model might be significantly diminished.\n\nQUESTIONS:\nPlease address the comment in weakness part. \nIn addition, though the ViT architecture has more total parameters, in some cases, it is less effective than LVM-Med ResNet-50. More research and discussion would be needed to fully understand this performance discrepancy.\n\nLIMITATIONS:\nNo limitation on limitations and broader societal impacts.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a self-supervised pre-training strategy for medical imaging using a graph matching approach. Each unlabeled image is transformed via a pair of data augmentations and then processed via an encoder network. The augmented pair of images become vertices in a pair of graphs, with vertex features being the encoder outputs and edge connections selected via k-nn. A graph convolutional network is trained for vertex-to-vertex matching of the extracted pair of graphs. The training objective incorporates global and local similarity learning over spatial features along with a second-order edge similarity cost. Due to the combinatorial nature of the objective, gradients for backpropagation are approximated via Implicit MLE. Pre-training is performed over a large scale dataset comprising 55 publicly available datasets and used for multiple downstream fine-tuning tasks including segmentation, detection and classification.\n\nSTRENGTHS:\nThe proposed graph matching technique for self-supervised learning is a novel and significant contribution. Abundant experiments demonstrate the generalizability over downstream tasks, with error bars also included.\n\nWEAKNESSES:\nSome related works on graph matching in computer vision are missing in Section 2.3. \n\nDoi et al. Detecting Object-Level Scene Changes in Images with Viewpoint Differences Using Graph Matching 2022\n\nBian et al. Unsupervised Domain Adaptation for Point Cloud Semantic Segmentation via Graph Matching 2022\n\nWu et al. Unsupervised Visible-Infrared Person Re-Identification via Progressive Graph Matching and Alternate Learning 2023\n\nLiu et al. Self-supervised Learning of Visual Graph Matching 2022\n\nPeng et al. GATE: Graph CCA for Temporal SElf-supervised Learning for Label-efficient fMRI Analysis 2022\n\nThe authors should discuss and contrast with the graph matching objectives and applications in these works to highlight their novelty in self-supervised learning. \n\n\nIntroduction can also be improved to better highlight the contributions. Rather than starting with discussing image-text datasets, the story should highlight the value of self-supervised learning in medical imaging and related works > proposed self-supervised learning method via graph matching > large-scale dataset collected for implementing this method > experiments on downstream tasks.\n\nQUESTIONS:\nCan the authors clarify how they use pre-trained ResNet-50 for downstream segmentation via U-Net? Does this mean that the encoder of U-Net has the same architecture as ResNet-50? This is not clear as the vanilla U-Net architecture (Cicek et al. 2015) is different.\n\nLIMITATIONS:\nLimitations are discussed.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-03"}

comments: SUMMARY:
This paper collects a large medical imaging dataset, and it also shows that a self-supervised learning technique based on second-order graph-matching enhances performance in various downstream medical imaging tasks compared to other supervised learning methods and foundation models trained on image-text instances. The evaluation also considers two different architectures: ResNet-50 and ViT backbones.

STRENGTHS:
- Creating such a large medical imaging dataset is a commendable feat, and is needed by the community.
- The proposed self-supervised task is interesting.
- The evaluation results are comprehensive and thorough.

WEAKNESSES:
None to report.

QUESTIONS:
None.

LIMITATIONS:
These are included in the paper.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2023-07-05

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a set of networks called LVM-Med which are trained on large-scale medical datasets. The authors collected more than a million medical images from more than 50 publicly available datasets of diverse modalities and structures of interest (e.g. CT, MRI, Ultrasound...). In the work, several self-supervised algorithms are benchmarked on the large dataset. Furthermore, this work proposes a self-supervised contrastive learning algorithm based on a second-order graph-matching formulation.\n\nSTRENGTHS:\n- the paper combines an incredibly large number of medical image modalities and images\n\n- I like the formulation of contrastive learning as a graph matching objective \n\n- the method section is comprehensive and the contributions are formalized\n\nWEAKNESSES:\n1) Some aspects of the experimentation are unclear to me. From how I understand the text, the authors aim to compare to a large number of other datasets, baselines (\"In 2D settings, we also compare with 2D supervised architectures, such as U-Net, U-Net++, Attention U-Net, etc.\"), and tasks across 2D and 3D. What I do not understand is how the authors choose their baselines and what they present in the tables. For example, in Table 2. for the Drive segmentation dataset, the authors report 2D supervised Methods (e.g., UNet) with Dice scores ranging from 59 to 65. Clearly, this is not a performance on par with scores reported in other works. From the literature, the state of the art in supervised DRIVE segmentation should be way higher. A fully supervised segmentation baseline on the DRIVE dataset should have 80+ Dice (https://paperswithcode.com/sota/retinal-vessel-segmentation-on-drive). Similarly, the IoU performance of BRATS baselines should be higher https://arxiv.org/pdf/1811.02629.pdf\n\nPotentially I misunderstand what kind of comparisons the authors provide here. Can the authors please explain the choice of their baselines and their experimental settings? \n\n2) The clarity of the writing in some sections should be improved, e.g., in the experimentation.\n\n3) The reproducibility of the results and methods is a concern. I have not seen the code. Furthermore, the sheer amount of computing required  makes reproducibility challenging.\n\nQUESTIONS:\n1) The work has been run on a large set of data. Can the authors provide more detail on the overall computing required to replicate their experiments? \n\n2) The work uses the reparameterization trick to create a complex discrete distribution. Can the authors explain the effect of the backpropagation and what this implies for the learning signal in more detail? \n\n3) Why are there no baselines for alternative ways to define a self-supervised contrastive loss in the presented setting? \n\nI would like to stress that I have an overall positive impression of the work and would be willing to reconsider my rating based on the rebuttal.\n\nLIMITATIONS:\n-", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

comments: SUMMARY:
The paper presents a large scale medical imaging dataset consisting of 1.3M medical images from 55 publicly available datasets along with a new contrastive learning framework based on graph matching. Specifically, the model is firstly pre-trained on the collected dataset, and then finetuned towards different downstream tasks, improvement is observed a cross different datasets and settings.

STRENGTHS:
+ A large scale *medical* dataset for pre-training purposes

+ Well-written and easy to follow

+ Extensive experiments and good results

WEAKNESSES:
- The paper did not provide enough details regarding the proposed datasets, especially in terms of the usage of the dataset, which, in my humble opinion, is pretty important to provide guidance on training medical models on large-scale datasets, e.g., 1. how are 3D data used, are they sliced into 2D data firstly? 2. The data comes from different datasets and maybe in different modalities, any balancing strategy during pre-training? 3. Any augmentations besides multi-crop? (e.g., flip, rotate, color jittering etc). In short, I expect the author provide more details regarding how they utilize the dataset

- I appreciate the author provided dataset statistics details in the supplementary, yet I am curious how are these datasets selected and have the author tried any filtering/curation? As data curation has been considered very important in modern large-scale model training [1], it would be great to have some insights in the medical area.


[1] DataComp: In search of the next generation of multimodal datasets

QUESTIONS:
see weakness

LIMITATIONS:
see weakness
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-06