LVM-Med: Learning Large-Scale Self-Supervised Vision Models for Medical Imaging via Second-order Graph Matching
Authors: Duy Minh Ho Nguyen
Hoang Nguyen
Nghiem Tuong Diep
Tan Ngoc Pham
Tri Cao
Binh T. Nguyen
Paul Swoboda
Nhat Ho
Shadi Albarqouni
Pengtao Xie
Daniel Sonntag
Mathias Niepert
Venue: NeurIPS 2023 poster (2023)
Keywords: medical imaging; self-supervised learning; graph matching; large-vision model
TL;DR: a work on large-scale medical pre-trained models

Abstract: Obtaining large pre-trained models that can be fine-tuned to new tasks with limited annotated samples has remained an open challenge for medical imaging data. While pre-trained networks on ImageNet and vision-language foundation models trained on web-scale data are the prevailing approaches, their effectiveness on medical tasks is limited due to the significant domain shift between natural and medical images. To bridge this gap, we introduce LVM-Med, the first family of deep networks trained on large-scale medical datasets. We have collected approximately 1.3 million medical images from 55 publicly available datasets, covering a large number of organs and modalities such as CT, MRI, X-ray, and Ultrasound. We benchmark several state-of-the-art self-supervised algorithms on this dataset and propose a novel self-supervised contrastive learning algorithm using a graph-matching formulation. The proposed approach makes three contributions: (i) it integrates prior pair-wise image similarity metrics based on local and global information; (ii) it captures the structural constraints of feature embeddings through a loss function constructed through a combinatorial graph-matching objective, and (iii) it can be trained efficiently end-to-end using modern gradient-estimation techniques for black-box solvers. We thoroughly evaluate the proposed LVM-Med on 15 downstream medical tasks ranging from segmentation and classification to object detection, and both for the in and out-of-distribution settings. LVM-Med empirically outperforms a number of state-of-the-art supervised, self-supervised, and foundation models. For challenging tasks such as Brain Tumor Classification or Diabetic Retinopathy Grading, LVM-Med improves previous vision-language models trained on 1 billion masks by 6-7%  while using only a ResNet-50.

LVM-Med: Learning Large-Scale Self-Supervised Vision Models for Medical Imaging via Second-order Graph Matching
Duy M. H. Nguyen* 1,2,3, Hoang Nguyen', Nghiem T. Diep', Tan N. Pham 3,4 Tri Cao Binh Nguyen', Paul Swoboda', Nhat Ho Shadi Albarqount Pengtao Xie
Daniel Sonntag 1 3,11, Mathias Niepert' 112
University of Stuttgart, 2MPPS for Intelligent Systems German Research Center for Artificial Intelligence, *University of Science VNUHCM, Max Planck Institute for Informatics, Universityy of Texas at Austin 'Helmholtz Munich, 8 University Hospital Bonn, 9UC San Diego, 1MMBZUAI, "Oldenburg University.
Abstract
Obtaining large pre-trained models that can be fine-tuned to new tasks with limited annotated samples has remained open challenge for medical imaging data. While pre-trained deep networks on ImageNet and vision-language foundation models trained on web-scale data are prevailing approaches, their effectiveness medical tasks is limited due to the significant domain shift between natural and medical im- ages. To bridge this gap, we introduce LVM-Med, the first family of deep networks trained on large-scale medical datasets. We have collected approximately 1.3 mil- lion in medical images from 55 publicly available datasets, covering a large number of organs and modalities such as CT, MRI, X-ray, and Ultrasound. We benchmark several of-theot-the-att self sulf-supe algorithms on this dataset and propose a novel relf-supervised contrastive learning algorithm using graph matching for- mulation. The proposed approach makes three contributions: (i) it integrates prior pair-wise image similarity metrics based on local and global information; (ii) it captures the structural constraints of feature embeddings through loss function constructed via combinatorial graph-matching objective; and (iii) it can be trained efficiently end-to-end using modern peudient-estimatin techniques for black-box solvers. We thoroughly evaluate the proposed LVM-Med on 15 downstream med- ical tasks ranging from segmentation and classification to object detection, and both for the in and dutto--distributtin settings. LVM-Med empirically outperforms a number of saat--ofthhe-ae art supervised, self-supervised, and foundation models. For challenging tasks such Brain Tumor Classification Diabetic Retinopathy Grading, LVM-Med improves previous vision-languaae models trained on 1 billion masks by 6-7% while using only ResNet-50. We release pre-trained models this link gitttu...b... com/ honm//dyhhominnnguueeLLL.......
Introduction
Constructing large-scale annotated medical image datasets for training deep networks is challenging due to data acquisition complexities, high annotation costs, and privacy concerns [ 2 Vision- language pretraining has emerged as a promising approach for developing foundational models that support various AI tasks. Methods such as CLIP [3), Align [4) and Flava [5] propose unified model trained on large-scale image-text data, showing exceptional capabilities and performance across
"Corresponding authors, CCo-Senior authors.
37th Conference on Neural Information Processing Systems (NcurIPS 2023), New Orleans.

---

various tasks. However, their effectiveness in the medical domain still remains unclear. A recent work SAM [6] trains large vision models on over one billion annotated masks from 11M natural images, enabling interactive segmentation. Nevertheless, SAM's zero-shot learning performance is moderate on other datasets 7 g highlighting the need for fine-tuning to achieve satisfactory results (
Organ
Modality
(V.E)
G7 (K.R)
EVM-Ated
Toot
Rewsee
CT
Heomt
X-Iy
Chedt
Lung
P-00 - M-
Gastrutneetin
S4h
Demoscopy
Y- Lifl 1p
Endoecape
Kicwwyy
Rethaal
MN
Blood
Uttasquund
Prostate
Image
Costrattee5 SoCCk, Mooors
Instanou-aaee ANC: Twie Balke, Dans, VKGllg)
Figure I: (Ieft) Overview of the body organs and modalities in our collected dataset; (right) AVM-Med unifics and extends contrastive and instance-basee self- supervised learning approaches by specifying graph's 's properties.
To facilitate the development of foundation models in the medical domain, we make two major contributions. First, we have curated a vast collection of 55 publicly available datasets, resulting in approximately 1.3 million medical images covering various body organs and modalities such as CT, MRI, X-ray, ultrasound, and dermoscopy. to name a few. Second, we propose LVM-MMed, novel class of contrastive learning methods, utilizes pre-trained ResNet-50 and a ViT network SAM[[O] We evaluate various instances of LVM-Med relative to popular supervised architectures and vision-language models across 15 medical tasks. To our best knowledge, this is the first time such a large-scale medical dataset has been constructed and used to investigate the capabilities of SSL algorithms.
LVM-Med incorporates a second-order graph-matching formulation, which subsumes and extends a large class of contrastive SSL methods. Given batch of images, two random transformations are applied to each image, and the resulting transformed images are then fed to an image encoder. The embedding vectors obtained from images in batch are used to construct two graphs where vertices represent pairs of transformed images generated from the same original one. Through solving a graph-matchin problem [II 12), we learn feature representation such that their encoding serves as suitable priors for a global solution of the graph-matching objective. This approach is distinct from prior contrastive learning methods that focus merely optimizing pair-wise distances [ [4] between transformed images or learning contrastive distances with positive and negative samples 15. It is worthwhile noting that previous contrastive learning methods special instances of our general framework (Figure [T), right).
LVM-Med has several advantages over existing approaches. First, it integrates advanced pair-wise image similarity taken from prior SSL methods into vertex affinities, resulting in both global and local information that can be efficiently fused. Second, it uncovers underlying structures of feature embeddings by utilizing edge constraints, enhancing robustness in the presence of similar entities in medical datasets. Third, though combinatorial problems are typically dindeifferent,iblee LVM-Med can efficiently calculate gradients through the discrete combinatorial loss function using modern implicit maximum likelihood estimation techniques. Consequently, LVM-Med can scale successfully on large-scale data. In a wide range of 15 medical experiments, LVM-Med sets a new state-of-the-an in fully fine-tuning or prompt-base segmentation, linear and fully fine-tuning image classification, and domain generalization, outperforming several vision- language models trained on a hundred million image-text instances.
We summarize major contributions this work, including:
We present a collection of large-scale medical datasets, serving as a resource for exploring and evaluating self-supervised algorithms.

---

(ii) We propose LVM-Med, a novel SSL approach based on second-order graph matching. The proposed method is flexible in terms of integrating advanced pair-wise image distance and being able to capture structural feature embedding through the effective utilization of second-order constraints within a global optimization framework.
(ii) On both ResNet-50 and ViT architectures, LVM-Med consistently outperforms multiple existing self-supervised learning techniques and foundation models across a wide range of downstream tasks.
2 Related Work
2.1 Self-supervised learning in medical image analysis
The latest approaches of global feature SSL rely on shared embedding architecture representations that remain invariant to different viewpoints. The variation lies in how these methods prevent collapsing solutions. Clustering methods (20-22) constrain a balanced partition of the samples within a set of cluster assignments. Contrastive methods (15 uses negative samples to push far away dissimilar samples from each other through contrastive loss, which can be constructed through memory bank [23], momentum encoder [24], or graph neural network [1]]. Unlike contrastive learning, instant-based based learning depends on maintaining the informational context of the feature representations by either explicit regularization [3] 25 or architectural design [26] 27]. Our work relates to contrastive and instance-based learning, where simplified graph-matching version of 1-N or 1-1 reverts to these approaches.
In contrast to global methods, local methods specifically concentrate on acquiring a collection of local features that depict small portions of an image. A contrastive loss function can be used on those
feature patches at different criteria such as image region levels (28). or feature maps [29] 14] These strategies are also widely applied in the medical context, thereby pre-text tasks based on 3D volume's properties, such reconstructing the spatial context [30], random permutation prediction [3] and
self-restoration [32 33), are proposed. Our LVM-Med model on this aspect can flcxible unifying
both global and local information by adding them to the affinities matrixes representing the proximity of two graphs, enhancing expressive feature representations.
2.2 Vision-language foundation models
In order to comprehend the multi-modal world using machines, it is necessary to create foundational models that can operate across diverse modalities and domains 34 CLIP and ALIGN 4 are recognized as groundbreaking explorations in foundation model development. These models demon- strate exceptional proficiency in tasks such as cross-modal alignment and zero-shot classification by learning contrastive pretraining on extensive image-text pairs from the web, despite the presence of noise. To further support captioning, recent works such as FLAVVA 5 and OmniVL 35 are designed to learn cross-modal alignment as well as image-video language models. Conversely, the SAM model 6 utilized a supervised learning strategy with over 1 billion masks on 11 million user-prompt interactions and achieved impressive zero-shot segmentation performance on unseen images. While many efforts have been proposed for natural image domains, limited research has been conducted on large-scale vision models for medical imaging. This motivated us to develop the LVM-Med model.
generation tasks such as visual question answering or video
2.3 Graph matching in visual computing
Graph matching is a fundamental problem in computer vision, which aims to find correspondences between elements of two discrete sets, such key points in images or vertices of 3D meshes, and used in numerous vision tasks, including 3D vision [36] (37), tracking [38] 39, shape model learning 40 and many others 45) In this framework, the vertices of the matched graphs correspond to the elements of the discrete sets to be matched. Graph edges define the cost structure of the problem, namely, second order, where pairs of matched vertices are penalized in addition to the Certex-to-verter matchings. This allows us to integrate the underlying geometrical relationship between vertices into account but also makes the optimization problem harhard. Therefore, many approximate approaches have been proposed to seek acceptable suboptimal solutions by relaxing discrete constraints [46] 47]. In other directions, gradient estimation techniques for black-box solvers are employed to make the hybrid
matching framework be differentially end-to-end [48 Our LVM-Med follows the latter direction and, for the first time, presents the formulation of contrastive learning as a second-orde graph-matching problem.
3

---

(g..
Avg
Projector
fn
hpp
G' (V.E')
G' = (V.E')
xj
e' RIVN
of E QIETIEE
2
MPN
MPN
ar. at
x
9 GM(c",c9
Forward pass
Liama (P, v*)
Backward pass
Figure 2: LVM-Med Overview. Avg is the average pooling layer, MPN denotes for message passing network, GM indicates the combinatorial solver, (c", represents vertex and edge aftinity matrices. For each image æ; in batch size, we generated two distorted versions and fed them into the feature representation fe and another projector ha. The obtained embeddings zf, é (s, t) are used to build two graphs G", G". We further design message passing network 9. that aggregate feature per node by their neighbor information. Then we compute vertex and edge affinities c" c and them solve the graph matching. The output afterward is compared with pairs ground truth (m², i (1,.., N) representing distorted images generated from same sample. the backward pass, we use modern gradient indiintt-stimmation techniques to approximate and
Methodology
3.1 Dataset construction
We provide detailed information about the collected datasets the Appendix. The data was collected from publicly available resources, which include diverse of modalities and body organs as illustrated in Figure T (left). The data format is a combination well as X-ray, MRI, CT, Ultrasounds, etc. For datasets whose data dimensions slice them into 2D images. To avoid potential test data leaking for downstream tasks, we use the default training partition in each dataset; otherwise, randomly sample with 20% total images. In total, we obtain approximately 1.3 million images. More statistics on the dataset are presented in the Appendix.
2D images and 3D volumes as 3D volumes,
3.2 Contrastive learning graph matching
Figure 2 provides illustration of our LVM-Med method, which learns the feature representation fo by matching two distorted views derived from the same input image through a graph-matching formulation. Below we describe in detail each component.
3.2.1 Graph construction on feature embedding
Given a batch of N images B = {æ1, Ø2, ... ..N. sampled from dataset, we generate for each image x; B two transformed images x and x by using two transformations s, T sampled from T, a set of pre-defined image transformations. After the transformations, each image is of shape (C H x W), where the number of channels and (H, W') the original spatial dimensions. These distorted images are fed into encoder so Rxxxxx - DxxS to produce two representations Dd = So (ax) and !! fo(=)) where D the number of feature channels and (R, S) are spatial dimensions of the feature map. On each such representation, perform average pooling

---

operation Avg RD 7 RD followed by another projection hs RR" - R. to form two feature embeddings x = (Avg((())) and क = (A(gg(())) E R with F < D.
Given set of embeddings for batch B, we construct two graphs G and G where, for each pair (2) æ²) of corresponding distorted images, we add a node representing x to G and a node representing if to G Hence, for each & E {8, [8,4), we construct a graph G = (V,, E*) with v²= = = } the set of vertices and E the set of edges efş = (af,, af)). The node-level feature matrix is given by X = [>fi x E RN*E which associates each vertex of with its feature embedding zf. We create edges for each graph G through k-nearest neighbors algorithm using the feature matrix X The adjacency matrix A RNNN is defined as A = 1 if el E E and A, 0 otherwise. With the two graph structures given, we obtain a node-attributed graph G = (V', A xx) on which graph neural network 9a is used aggregate the nodes' features. In particular, Is computes an embedding te = 9s (X A²) by performing message passing operations.
V
We set g5 to be graph convolutional network 52 5 consisting of [++]] layers IE = {9) -1.1: 9o} where the output of layer l is computed as
H = o
H Hf
(1)
where IN is the identity matrix modeling self-connections: D is a diagonal matrix with Dis Σ, Alji g' g011 are the trainable parameters for each layer; a--) an activation function; and H X' We use the outputs of the last layer embeddings for the nodes, that is, 2' Hf NxF given the shared graph network ge-
We now have two graphs G", G with node attribute matrices 2², 2², the outputs of the graph neural networks. Next, graph-matching problem is constructed and solved where the gold matching is given by the pairs (a) x Vi E N}.
3.2.2 Learning affinities with global and local context
To represent potential connections for pair of node (x),,²)) where x; E G", x G', we design vertex affinity matrix e" E RRV"IIV** where di is the prior (feature-based) similarity between an and x An advantage of our formulation is its ability to integrate advanced pair-wise distance can be smoothly integrated cia, resulting in more expressive proximity representation. particular, we leverage both global and local consistency derived from feature embeddings of distorted images. The global distance used in several prior works can be computed as cos(.) denotes cosine similarity; 2 is the embedding of ar (E E {s,t), m {i,a)) obtained after message passing in Eq. I
eg (æf,, xx)) cos( 2) where
Compared to global methods that implicitly learn features for the entire image, local methods concentrate on explicitly learning a specific group of features that characterize small regions of the image. As a result, they are more effective for dense prediction tasks such as segmentation While recent works applied these tactics as a part of pair-wise minimization conditions 28 Instead, we integrate them as part of vertex costs C²2 and use it to solve the graph matching problem. Indeed, we adapt both location- and feature-based local affinity computed as:
29
14
53
54
cia
lo (xf xl)) = EpEP cos Emip)) + EpEP cos cos(g (
(2)
where P f (r) E [1..... R] x []] be the set of coordinates in the feature map of E RDD of a;; of (s, be the feature vector at position y; m(p) denote the spatial closest coordinate to p in coordinates of feature map y estimated through transformations on original image I;; finally m m'(p) represents the closest feature vector to p y¹, using 12 distance. Intuitively, the local cost in Eq. (2) enforces invariance on both spatial location and between embedding space at local scale. Our final affinity cost is computed
cla (a a/)) = a (gglo (at, x²h) + (1 ax)
+ (x0 a.))
(3)
3.2.3 Self-supervision through second-order graph matching
While the standard graph matching problem for vertex-to-verte correspondences can be used in setting (LAP), it fails to capture the similarity between edges. If there are duplicated entities
5

---

represented by distinct nodes in the same graph, the LAP will consider them identical and skip their neighboring relations. For instance, during the image sampling, two consecutive image slides were sampled from 3D volume, resulting in their appearances have small difference. In such cases, it complicated correctly identify those augmented images generated from the same one without using information from the relations among connected nodes in the constructed graph. To address this problem, introduce additional edge costs c² R Fe" | where Saa.. represents the similarity between an edge ofj = ( x3) E E and vib (xt x E E.. These edge costs (second-order) are computed as caa. = cos( 29), (2. 2()).
We now establish the second-order graph-matching problem. Denoting indicator vector of matched vertices, i.e., Uia = if the vertex a; E V is matched with xy E V and Big 0 otherwise. The node correspondence between two graphs G and G that minimizes the global condition stated as:
{0, 13 1 be
GM(C c arg min (l1( where U(1,1) (v E (0, 1) NxN = 11
Ei, Gia -Eijan Via
and Ix be n-dimension one-value vector. The constraint (11 1) restricts satisfying the one-to- one matching. Essentially, the Eq. 4 solves the vertex-to-vertex correspondence problem using both node and edges affinities, which can be seen as form of structural matching (Figure rightt and generally can be integrated with higher-order graph constraints as triangle connections or circles. In the experiment, we found out that Eq. 4 significantly improved downstream task performance compared to the pure linear matching approach (Table (6). Since the Eq. + in general is an NP-Hard problem (55) due to its combinatorial nature, thus efficient heuristic solvers based on Lagrange decomposition techniques [56]
3.2.4 Backpropagating through a graph matching formulation
With è GM(c", c²)) a solution obtained from the solver, we use the Hamming distance and an optimal solution o² to define the following loss function
L(ō, v²) 0. v²) -". 6).
The proposed approach aims to learn the feature representation function fo such that its output minimizes Eq. (5) However, this is a difficult problem because the partial derivatives of the loss function w.r.t vector costs c", c,, i.e., 0L/0c" and 0L/c², are zero almost everywhere [48 57 due to the objective function in Eq. (4) being piece-wise constant, preventing direct gradient-based optimization.
To approximate the gradients required for backpropagation, we adopt IMLE 50 58
Let = (c", c*)
be the input to the combinatorial graph matching problem in Eq. 44. The core idea of IMLE is to define a probability distribution p(u; 0) over solutions of the combinatorial optimization problem, where the probability of a solution proportional to its negative cost, and to estimate aL/00 through the gradients of the expectation VolEa-p(v:0) [L v" v*)]. Since exact sampling from p(v; 0) is typically intractable, IMLE instead chooses a noise distribution p(E) and approximates the gradient of the expectation over p(v; 0) with the gradient of the expectation over p(E)
V Ve [L v²) ~ VelEe-ote + v")]
The above approximation invokes the reparameterization trick for a complex discrete distribution. A typical choice for p(E) is the Gumbel distribution, that is, p(E) Gumbel 1) [59]. Now, by using a finite-difference approximation of the derivative in the direction of the gradient of the loss V (o, v"), we obtain the following estimation rule:
1 1 x
V Ve VeEppi) [L(6,0") 2s EENple)
GM + - AVo((o,0)))
(6)

---

Algorithm Forward and Backward Pass for c², c
function FORWARDPASS(e" c²)
function BACKWARDPASSE V&L(
// Gumbel noise distribution sampling
load (c", )¹), (e, and
e, ~ Gumbel(0, 1)
II Add gradient-bused pertubations
I/ Graphh-mtchhing with perturbed (c",
(c), cg) = (c" + c, + <') - XV, (
i GM (c" c, c. + +')
#1 Single sample gradient estimate
I/ Save values for the backward pass
"Be" de δ GM (5)
BL 8L
save (c", )² e') and T
OL return X De De
return
where ē = GM( + e). À is a step size of finite difference approximation. Using a Monte Carlo approximation of the above expectation, the gradient for 0 is computed as difference of more pairs of perturbed graph-matching outputs. We summarize in Algorithm| the forward and backward steps for c", c..
Experiments
4.1 Implementation details
Pre-training We utilize Resnet50 (60] and Vision Transformer (ViT-B/16) [1] to train our LVM- Med. For Resnet50, we load pre-trained from ImageNet-1K K [62]. and SAM Encoder backhone weight 10] for ViT. The raw image is augmented to two different views by using multi-crop techniques as 14] and followed by flip (probability 50 %), color jitter, random Gaussian blur, and normalization. We trained the LVM-Med with 100 epochs on the collected dataset. The batch size of 3200 used for ResNet50 and we reduced it to 2800 for VIT due to memory limitation. The model is optimized with Adam 63 with an initial learning rate 2 x 10 and reduced halved four times. We use 16 A100-GPUs per with 80GB and complete the training process for LVM-Med with ResNet-50 in five days and LVM-Med with VIT encoder in seven days. Other competitor SSL methods as VicRegl, Twin-Barlon, Dino, etc, are initialized from ResNet-50 pre-trained ImageNet-I IK and trained with 100 epochs with default settings as L.M.-Med
To balance samples among different modalities, we combine over-sampling and data augmentation increase the total samples. Specifically, new samples from minority classes are generated by duplicating images and applying random crop operations covering 85 95% of image regions and then rescaling them the original resolutions. Note that these augmentations are not used in the self-superd self-supervised algorithm (operations t T) to avoid generating identical distorted versions in this sampling procedure.
Table 1: Summary of datasets and downstream tasks
JCS
COLL
DRIS-NN
Evaluation
Downstream Task Data
Modality
Nums
LVM Meed (TRG-AW))
Fine- Tuning
BraTS2018
[4]
MRI
285
Tumor Segmentation
90
Fine- Taning
MMWEES-CT
6
3D
Heart Structures Segmenttaion
$7.6
Fine Timing
MMWHS. MRI
6
3D MRI
Heart Siructures Segmenatton
Fine-Tining
ISIC-2018
20 Demoosoppy
2596
Skin Lision Segmestation
Fine Timing
ISRT
2D X ray
247
Multi Organ Segmentation
Fine-Tuning
KvnSir
6
2D Endessopp
1800
Polyp Segmentation
Detection
12:0
Fine-Tuning
Drive
Randus
40
Vesiel Soymentation
Fine Timing
BUID
20
2D Ulirasound
647
Breust Cancer Segmentation
Fire- Tuning
Evalluttonn
RGADR
Fundus
Godding
76.0
Lincse Evaluntion &
Fine Tuming
Brain Classiticaiion
Tumor Classilication
Fine-Tening
MRI Segmentation
Multi-site Prostate
12
Prostite Segmestation
Fine Tuning
VinDr
73
2D X ray
18000
Long Diseuses Detection
Figure 3: FGADR performance with top architectures.
Downstream Tasks Table]lists the datasets and downstream tasks used in our experiments. We cover segmentation, object detection, and image classification problems. It is important to note that in most settings, we utilize simple configurations for all datasets, skipping extra pre-pr pre-processing for data augmentation. For instance, overlapping image patches with stride operations in the original samples [74] to increase training data in the Drive dataset or combining different 3D MRI modalities to fuse information 75 in the BRATS-2015
excluded in our downstream setups.
To validate L.M-Med algorithms, we compare with 2D-SSL. methods trained in our dataset and foundation models like Clip [3), Align 4. Flava [5), and SAM [6] with pre- pre-trained ViT (Bert for
7

---

Align) taken from each method, respectively. During the downstream task, trained SSL weights are then extracted and attached in U-Net for ResNet50 backbone, TransUnet [76] for ViT, and then fine-tuned with training splits of each dataset. Depending on the downstream task's properties, we apply different image resolutions and other parameters like the number of epochs and learning rate for different data domains. Details for these configurations are presented in Appendix.
4.2 2D- and 3D-based segmentation
We evaluate LVM-Med on eight medical segmentation tasks, including five 2D>-based and three 3D-based segmentation. In 2D settings, we also compare with 2D supervised architectures, such as U-Net, U-Net++ Attention U-Net, etc. These networks are initialized with ResNet-50 pre-trained ImageNet. Additionally, we investigate the prompt-based segmentation settings inspired by the current success of SAM's zero-shot learning. We utilized the ground truths and added random noise to simulate box-based user prompts as [9] We next compare three variations of SAM: (i) freezing image and prompt encoders, only fine-tuning mask decoder; (ii) without any training and inference using box prompts; (iii) similar to (i) but replacing the original image encoder by LVM-Med's s ViT architecture taken from SAM trained in our dataset.
Table 2: Performance comparison on five 2D segmentation tasks with fully fine-tuning. Results are reported with an average 2D Dice score on three trial times. The best results in each group are in bold, the overall best value, excluding prompt-based segmentation, is underlined.
Method
ISIC-2018 (Skin lelon)
JSRT (lamg Xomay
Kvadir (Prlyp)
(Vood))
BUID (Breast Cancer)
Randonlly (150))
Pro ImnaeNa
83,85
1.14
1B Supersiedd Methed
Attertion Irvt
72
M6.11 0.51
94.47 2.71
83.25
1:41
65
72.19- 1:16
U.Net e8 75
56.71 = 0.49
94.32 2 288
82,23 2: 1.41
65.38
9:78
73.76 2.83
Tons (i-Nrt [6
16.60 1 0.12
80:00 0.55
63.11 1 0124
62.61
0.34
67:90-
0.40
Bwis-Baloo
$6.01
3.09
$3.00
SincLA
57.25 0.21
2.03
2.18
0.22
ID-SI on mediral
Missoyy 1
K7.34 0.14
94.05 3.50
75.35
1.55
64.91 3.31
15.91 1.00
Despcluster. 92 A3
56.73 = 0.42
94.79 2 2.89
82,60 2: 0.75
64.14 0.92
76.33 0.59
Vidkegi [4]
16.27 0.33
94.19 3.25
51.05 0.45
66.17 0.27
75.20 0.64
LNM--led (855))
87.76 0.30
V5.13 2.64
M6.76 :: 0.94
0.37 66.9
78.65 0.72
Foundation Model
SAM
68.17 : 0.30
0.10
90:68 0.40
0.20
70:35 0.00
64.04 0.41
0.45
73677 0.66
1.17
LVM-Sled (SAM''s VIT)
19.41 : 0,28
90.74 2: 0.47
73,30 5: 0.05
65.49 9.33
77.20 0.42
(feed recoierr
07:41 0.12
01.39
82.12
0.A5
K7 0.67
Prompt-hasce baseeaaaeedde Seg.
Prosipt trinn
55.76
61.97
98:77
15.12 0.34
1.01
Table 3: 3D segmentation task performance with fine-tuning on three datasets. Results are reported with an average 3D IoU on five trial times. The best results in each group and overall are in bold and underlined.
Table 4: In-out-distribution evaluation for the segmen- tation task on the Prostate dataset. Results are reported with an average 2D Dice score on three training times.
Methad
Bratt
MMWIES-UT
MMWWHS-MRI
Muthod
Mutiiotta Proetade Segmemattion
10. Dazzaomme
500
66. 54
0:40
67.30 1 2.39
67.64 2.21
BMC (Raed)
BUNNIC
BIONI
HK
Averaee
13D [1]
67 + 75
T666 + 2.55
(6:71 : 1.20
Sapersirn
Nety
65.04 76447 1.36
51
9.95
13.56
12.55
SET SU.1
66.001
75.01
Promsited Inugohia (40
62.11 2 9305
41.74 1 4.36
$130 2
Model
67.06 3: 1.29
2:289
74,53 1.09
Llawenal Modil
14
73 10 0:07
78.14 0.177
77:47-115 1.50
Taln Rution [1]
76.36 176
60 09
6: 1.90
13.63 13:33
54.83
15.00
$1.0
TraneW
4 50
79:74 1.7%
2.14
Diss
1.15
3.77
3000
46.5
SwiaViT3D
70.58
1.66
3D
Mi Li
74 00 80
55 69
5:45
37.53
10 16
13:55 à 14.55
17.6
Twin- Burken [k]
73.30 2- 18
54.70 6: 1.01
70.39 2.23
Despctace b0
22:45
64.35 3.15
36.43
9J15
4430 4.74
516
Dis [0)
71.72 3: 0.55
$1.08 2: 1.62
70:42 = 76.74
Villoz [4]
26.12 1.15
54.09 4:35
3,44
35.43 2: 1310
47.5
Suc
15
76.54
LVM--tel
02.45 2305
56.76 6.50
51.78 3/4
60.9
Dopphsrr
VicReei [1]
72.00 4 1551
73.23 2
54.72
44.01 (-050
2: 0.56
76.32 0.78
TuL5 1.63
SAM (Frad eecader
Phongr-fanel
95 50
1M 0:79
0.14
4132 0:30
$1.24
LVM-Mrd (850)
23.58 4 0.14
$4.1 2 0.77
78,59 2: 034
LVM--ltd (SAMV ViT)
05.75 6.96
02.81.11.30
V2.15-1445
$1.11
Clip
71.19
1.20
SAM Faaodda 0
70.11 1:45
77.8 = 1.00
68.09 5.49
LVM-Aled (SAMS VT))
71.42 4 0.70
$0.78 - 1.77
00.36 1.18
In 3D settings, we segment 2D slices and merge results for 3D volume. We also benchmarked with 3D self-supervised methods from [86]. Tables (2) and (3) show that our two versions with ResNet-50 and Sam's ViT hold the best records in each category. For instance, we outperform 2D SSL methods trained on the same dataset, surpassing foundation models such as SAM, Flava, and Clip. In the prompt-based settings, LVM-Med also delivers better performance compared with SAM. Second, LVM-Med achieves the best overall results on seven of eight segmentation tasks, mostly held by
8

---

LVM-Med with ResNet-50. The improvement gaps vary each dataset, for e.g., from 3 - -5% on Kvasir and BUID compared with 2D supervised methods.
4.3 Linear and finetuning image classification
We analyze LVM-Med on image classification tasks using linear probing (frozen encoders) and fully fine-tuning settings, two popular evaluations used in self-supervised learning. The experiments are conducted on the FGADR Grading and Brain tumor classification tasks. Table (5) presents the average accuracy metric on three training times. LVM-Med (ResNet-50) consistently outperforms other approaches on two datasets. For example, it is better than Clip by 10.46% and 8.46% on FGADR and Brain Tumor datasets with linear evaluation. In the foundation model setting, LVM-Med (ViT) also improves SAM's 's results by 7.32% and 4.69% on FGADR with linear and fullu-ninetuning. Another point we observe is that the overall 2D-SSL methods based on ResNet-50 and trained on the collected medical dataset achieve higher accuracy than foundation models using VIT. We also compare WM-Mee with the top methods on the FGADR dataset, including AFN-Net 87, JCS [8]] CoLL 89 and DRG-Net 190] We choose the DRG-Net as the backbone and replace the employed encoder with our weights (R50). Figure 3 shows that LVM-Med hold the first rank overall.
Table 5: Performance comparison linear evaluation and fine-tuning classification. The results are reported with aver- age Accuracy three training times.
Table 6: LVM-Med ablation study.
Results are reported on an average of five 2D segmentation and two lin-
Method
Linear Evaluation (Fruzen)
Fine-tuming
FGADR
Brain Tumor Cls
FGADR
Bruin Tumor
Twins-Brron
66.86 0.41
63.01 + 0.30
96.37 + 0.77
74.20 + 1.38
car classification tasks. The two most important factors are highlighted.
Dise
20
65.98 + 191
62:27 = 0.32
67.35 + 1.36
71.91 = 1.55
Sa R
15
65.30 1.70
62.52 = 1.67
67.55 + 0.28
73.52 = 3.56
M
65.98 104
62:35 1.92
67.55 à: 1.79
74:53 0.43
Derpchoser k0
65.34 199
64.77 + 0.55
67.94 + 1.78
73.10 0.55
ClatAcc)
Seg. (Dice)
Viailey [4]
64:71 + 0.60
59.64 + 1.36
05.60 + 1.46
73.18 200
LVM-Mol (I550)
68.33 0.48
66.33 = 0.31
68.32 + 0.48
76,82 2.23
LVM Med (Full)
67.47
$3.05
SAM
Plans
Clip
Algin
31.87 + 0.69
36.95 = 1.04
55,13 :: 0.41
$1.87 + 0.50
31.51 = 4.26
$7.87 + 0.71
35.19 = 0.43
30.71 = 2.35
58.75 #: 1.32
57.25 + 0.97
$7.45 + 0.85
57.18 0.96
63.96 = 0.04
60.66 =: 1.36
34.01 + 5.97
34.86 + 2:27
LVM Med wio message passing
INM-Mal wio Gurbbel noise
IMM-Med
LVM Med wio second urder
similarity
62.17
65.08
64.32
81.19
81.37
80.21
LYM-Med (SAM's VIT)
62.46 :: 0,86
59.31 : 0.48
63.44 3: 0.75
67.34 2: 2.08
4.4 Object detection & In-out-distribution evaluation
Figure 4
indicates performance on the object de-
tection task using VinDr and Kvasir datasets. We use Faster R-CNN and load ResNet-50 from 2D SSL pre- trained weights. Results are presented by Average Preci- sion with loU=0.5 over three training times. Compared to pre-trained Imagenet, LVM-Med still outperforms by 1-2% though overall, improvements are smaller than image classification and segmentation tasks.
12
$3.77
20
27
We also validate LVM-Med performance on the in-out- Figure 4: LVM-Med object detection. distribution setting in Table (4) using the segmentation task on the Multi-Prostadl dataset. We train LVM-Med and other competitors in BMC data and use the trained models predict the remaining datasets. Both two versions of LVM-Med with ResNet-50 and ViT, on average, surpass all baselines, which validates the potential abilities of LVM-Med for the in-out -out-iistributin problem.
4.5
Ablation study
We do the following settings to evaluate the performance of components used in LVM-Med: (i) LVM-Med without using second-order graph matching conditions, i..., only solving vertex-to-veree correspondence problem; (ii) LVM-Med without using message passing network 9e in Eq. ( to aggregate information from local connections: (iii) LVM-Med w/o using approximate gradients from Gumbel noise in Eq. (6) For this, we add a constant value to e", ce as prior works [57
and finally (iv) LMV-Med without using local similarity de in Eq. (2. Other ablation studies are presented Appendix. Table (6) indicates that all factors contribute to the final performance, wherein
the second-order and Gumbel noise are the two most two important parts.

---

5 Conclusion
We have demonstrated that a self-supervised learning technique based on second-order graph- matching, trained on a large-scale medical imaging dataset, significantly enhances performance in various downstream medical imaging tasks compared to other supervised learning methods and foundation models trained on hundreds of millions of image-text instances. Our findings are supported by the benefits shown in two different architectures: ResNet-50 and ViT backbones, which can be used for either end-to-end or prompt-base based segmentation.
Limitations and Future Work.
We propose investigate the following points to improve LVM-Med performance. Firstly, extending LVM-Med to hybrid 2D-3D architecture to allow direct application for 3D medical tasks instead of 2D slices. Secondly, although LVM-Med with ViT backhone utilizes more total parameters, in some cases, it is less effective than LVM-Med ResNet-50. This raises the question of whether novel approach could improve the performance of ViT architectures. Finally, integrating multi-modal information such as knowledge graphs, bio-text, or electronic health records for LVM-Med is also important to make the model more useful in real-world applications.
Acknowledgements
This research has been supported by the pAltient project (BMG, 2520DATOP2), Ophthalmo-Al project (BMBF, 16SV8639) and the Endowed Chair of Applied Artificial Intelligence, Oldenburg University. Binh T. Nguyen wants to thank the University of Science, Vietnam National University in Ho Chi Minh City for their support. Tan Ngoc Pham would like to thank the Vingroup Inno- vation Foundation (VINIF) for the Master's 's training scholarship program. The authors thank the International Max Planck Research School for Intelligent Systems (IMPRS-IS) for supporting Duy M. H. Nguyen. Mathias Niepert acknowledges funding by Deutsche Forschungsgemeinschaff (DFG, German Research Foundation) under Germany's 's Excellence Strategy EXC and support by the Stuttgart Center for Simulation Science (SimTTec)..
References
[1] Veronika Cheplygina, Marleen de Bruijne, and Josien PW Pluim. Not-so-supervised: a survey of semi-supervised, multi-instance, and transfer learning in medical image analysis. Medical image analysis, 54:280-296, 2019.
[2] Georgios A Kaissis, Marcus R Makowski, Daniel Rückert, and Rickmer F Braren. Secure, prisecy-preserving and federated machine learning in medical imaging. Nature Machine Intelligence, (66) 2020.
[3] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748 8763. PMLR, 2021.
[4] Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoe Le, Yun- Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. In International Conference on Machine Learning, pages 1904-4916. PMLR, 2021.
[5] Amanpreet Singh, Ronghang Hu, Vedanuj Goswami, Guillaume Couairon, Wojeiech Galuba, Marcus Rohrbach, and Douwe Kiela. Flava: A foundational language and vision align- ment model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11638-15650, 2022.
[6] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al. Segment anything. arXiv preprint arXiv:2304.02643 2023.
[7] Maciej A Mazurowski, Haoyu Dong, Hanxue Gu, Jichen Yang, Nicholas Konz, and Yixin Zhang. Segment anything model for medical image analysis: an experimental study, arXiv preprint 2tti:2300.10517, 2023.
10

---

[8] Sheng He, Rina Bao, Jingpeng Li, P Ellen Grant, and Yangming Ou. Accuracy of segment- anything model (sam) in medical image segmentation tasks. arXiv preprint 093444 2023.
[9] Jun Ma and Bo Wang. Segment anything in medical images. arXiv preprint ar 123::2 2023.
[10] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg, Wan- Lo, Piotr Dollár, and Ross Girshick. Segment anything. arXiv:2304. 02643, 2023.
[11] Hui Sun, Wenju Zhou, and Minrui Fei. A survey graph matching in computer vision. In 2020 13th International Congress on Image and Signal Processing, BioMedical Engineering and Informatics (CISP-BMEI) BMEI), pages 225-230. IEEE, 2020.
[12] Stefan Haller, Lorenz Feineis, Lisa Hutschenreiter, Florian Bernard, Carsten Rother, Dagmar Kainmüller, Paul Swoboda, and Bogdan Savchynskyy. A comparative study of graph matching algorithms in computer vision. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXIII, pages 636-653. Springer, 2022.
[13] Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, and Stéphane Deny. Barlow twins: Self- supervised learning via redundancy reduction. In International Conference on Machine Learning, pages 13110-2220. PMLR, 2021.
[14] Adrien Bardes, Jean Ponce, and Yann LeCun. Vicregl: Self-supervised learning of local visual features. NeurIPS, 2022.
[15] Ting Chen, Simon Korbblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pages 1597-1607, PMLR, 2020.
[16] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pages 9729 9738, 2020.
[17] Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved baselines with momentum contrastive learning. arXiv preprint WrXiv22003.04297, 2020
[18] Nenad Tomasev, Ioana Bica, Brian McWilliams, Lars Buesing, Razvan Pascanu, Charles Blun- dell, and Jovana Mitrovic. Pushing the limits of self-supervised resnets: Can we outperform supervised learning without labels imagenet? arXiv preprint 1rxiv:2201.05119 2022.
[19] Shixiang Tang, Feng Zhu, Lei Bai, Rui Zhao, Chenyu Wang, and Wanli Ouyang. Unifying visual contrastive learning for object recognition from graph perspective. In Computer Vision- ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXVI, pages 649 667. Springer, 2022.
[20] Mathilde Caron, Piotr Bojanowski, Armand Joulin, and Matthijs Douze. Deep clustering for unsupervised learning of visual features. In Proceedings of the European conference on computer vision (ECCV), pages 132-149, 2018.
[21] Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. Advances neural information processing systems, 33:9912-99244 2020.
[22] Xiaohang Zhan, Jiahao Xie, Ziwei Liu, Yew-Soon Ong, and Chen Change Loy. Online deep clustering for unsupervised representation learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 6688-6697, 2020.
[23] Zhirong Wu, Yuanjun Xiong, Stella X Yu, and Dahua Lin. Unsupervised feature learning via can-aaramttric instance discrimination. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3733-3743 2018.

---

[24] Xinlei Chen, Saining Xie, and Kaiming He. An empirical study of training self-supervised vision transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9640-9649, 2021.
[25] Quentin Garrido, Yubei Chen, Adrien Bardes, Laurent Najman, and Yann Lecun. On the duality between contrastive and non-contrastive self-supervised learning. arXiv preprint arXiv:2206.0255 2022.
[26] Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a new approach to supersuee learning. Advances in neural information processing systems, 03:21271-21288 2020.
[27] Kuang-Huei Lee, Anurag Arnab, Sergio Guadarrama, John Canny, and Ian Fischer. Com- pressive visual representations. Advances in Neural Information Processing Systems, 34: 19538-19552, 2021.
[28] Tete Xiao, Colorado J Reed, Xiaolong Wang, Kurt Keutzer, and Trevor Darrell. Region similarity representation learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 10539-10548, 2021.
[29] Xinlong Wang, Rufeng Zhang, Chunhua Shen, Tao Kong. and Lei Li. Dense contrastive learning for self-supervised visual pre-training. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3024-3033, 2021.
[30] Xinrui Zhuang, Yuexiang Li, Yifan Hu, Kai Ma, Yujuu Yang, and Yefeng Zheng. Self- supervised feature learning for 3d medical images by playing rubik's cube. In International Conference on Medical Image Computing and Computer-Assion Intervention, pages 420-428. Springer, 2019.
[31] Liang Chen, Paul Bentley, Kensaku Mori, Kazunari Misawa, Michitaka Fujiwara, and Daniel Rueckert. Self-supervised learning for medical image analysis using image context restoration. Medical image analysis, 18:101539, 2019.
[32] Zongwei Zhou, Vatsal Sodha, Jiaxuan Pang, Michael B Gotway, and Jianming Liang. Models genesis. Medical image analysis, 17:10,,4,, 2021.
[33] Fatemeh Haghighi, Mohammad Reza Hosseinzadeh Taher, Zongwei Zhou, Michael Gotway, and Jianming Liang. Transferable visual words: Exploiting the semantics of anatomical patterns for self-supervised learning. IEEE transactions on medical imaging, 40(10))2857- 2868, 2021.
[34] Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx. Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. On the opportunities and risks of foundation models. arXiv preprint arXiv:2108. 100.:52888.. 2021.
[35] Junke Wang, Dongdong Chen, Zuxuan Wu, Chong Luo, Luowei Zhou, Yucheng Zhao, Yujia Xie, Ce Liu, Yu-Gang Jiang, and Lu Yuan. Omnivl: One foundation model for image-language and video-language tasks. arXiv preprint arXiv:2209.00 0756677666 2022.
[36] Jiayi Ma, Xingyu Jiang, Aoxiang Fan, Junjun Jiang, and Junchi Yan. Image matching from handcrafted to deep features: A survey. International Journal of Computer Vision, 129:23-79, 2021.
[37] Dongliang Cao and Florian Bernard. Self-supervised learning for multimodal non-rigid 3d shape matching. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 17735-17744, 2023.
[38] Alper Yilmaz, Omar Javed, and Mubarak Shah. Object tracking: survey. Acm computing surveys (CSUR), 38(4): 2006.
[39] Jeongseok Hyun, Myunggu Kang, Dongyoon Wee, and Dit- Yan Yeung. Detection recovery in online mujti-object tracking with sparse graph tracker. In Proceedings of the IEEF/CVF Winter Conference on Applications of Computer Vision, pages 4850 4859, 2023.
12

---

[40] Tobias Heimann and Hans-Peter Meinzer. Statistical shape models for 3d medical image segmentation: a review. Medical image analysis, 13(4):543-563, 2009.
[41] Paul Roetzer, Zorah Lähner, and Florian Bernard. Conjugate product graphs for globally optimal 2d-3d shape matching. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1866-21875, 2023.
[42] Yikai Bian, Le Hui, Jianjun Qian, and Jin Xie. Unsupervised domain adaptation for point cloud semantic segmentation via graph matching, In 2022 JEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 1899-9908 IEEE, 2022.
[43] Zesen Wu and Mang Ye. Unsupervised visible-infrared person re-identification via progressive graph matching and alternate learning. Proceedings of the IEEF/CVE Conference on Computer Vision and Pattern Recognition, pages 9548-95588 2023.
[44] Liang Peng, Nan Wang. Jie Xu, Xiaofeng Zhu, and Xiaoxiao Li. Gate: graph cca for temporal self-supervised learning for label-efficient fmri analysis. IEEE Transactions on Medical Imaging, 42(2):391 10.39991122100 2022.
[45] Chang Liu, Shaofeng Zhang, Xiaokang Yang, and Junchi Yan. Self-supervised learning of visual graph matching. In European Conference on Computer Vision, pages 370-388. Springer, 2022.
[46] Ron Zass and Amnon Shashua. Probabilistic graph and hypergraph matching. In 2008 IEEE Conference on Computer Vision and Pattern Recognition, pages 1-8. IEFE, 2008.
[47] Feng Zhou and Fernando De la Torre Factorized graph matching. IEEE transactions on pattern analysis and machine intelligence, 38(9):1774-1789 2015.
[48] Marin Vlastelica Pogančié, Anselm Paulus, Vit Musil, Georg Martius, and Michal Rolinek. Differentiation of blackbox combinatorial solvers. In International Conference on Learning Representations, 2020.
[49] Michal Rolínck, Paul Swoboda, Dominik Zietlow, Anselm Paulus, Vit Musil, and Georg Martius. Deep graph matching via blackbox differentiation of combinatorial solvers. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Springer, 2020.
Proceedings, Part XXVIII 16, pages
[50] Mathias Niepert, Pasquale Minervini, and Luca Franceschi.
Implicit MLE: backpropagating through discrete exponential family distributions. In NeurlPS, Proceedings of Machine Learning Research. PMLR, 2021.
[51] Thomas N. Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Truck Proceedings. OpenReview.net, 2017. URL https:///operrviiee net net/forrumid--SJUaay)
[52] Felix Wu, Amauri Souza, Tianyi Zhang, Christopher Fifty, Tao Yu, and Kilian Weinberger. Simplifying graph convolutional networks. In International conference on machine learning, pages 6861-6871. PMLR, 2019.
[53] Junwei Yang, Ke Zhang, Zhaolin Cui, Jinming Su, Junfeng Luo, and Xiaolin Wei. In- scon: instance consistency feature representation via self-supervised learning. arXiv preprint aXXiv:2203.07688 2022.
[54] Zhenda Xie, Yutong Lin, Zheng Zhang, Yue Cao, Stephen Lin, and Han Hu. Propagate yourself: Exploring pixel-level consistency for unsupervised visual representation learning. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 16684-16693, 2021.
[55] Rainer Burkard, Eranda Cela, Panos M Pardalos, and Leonidas S Pitsoulis. The quadratic assignment problem. Springer, 1998.
13

---

[56] Paul Swoboda, Carsten Rother, Hassan Abu Alhaija, Dagmar Kainmuller, and Bogdan Savchyn- skyy. A study of lagrangean decompositions and dual ascent solvers for graph matching. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 1607- 1616, 2017.
[57] Michal Rolinek, Vít Musil, Anselm Paulus, Marin Vlastelica, Claudio Michaelis, and Georg Martius. Optimizing rank-based metries with blackbox differentiation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7620-7630, 2020.
[58] Pasquale Minervini, Luca Franceschi, and Mathias Niepert. Adaptive perturbation-based
gradient estimation for discrete latent variable models. arXiv preprint arXiv:2209.04822 2022.
[59] George Papandreou and Alan L. Yuille. Perturb-and-map random fields: Using discrete optimization learn and sample from energy models. In /0// International Conference on Computer Vision, pages 193-200. IEEE, 2011.
[60] Kaiming He, Xiangyu Zhang. Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[61] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghami, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint wrXiv:2010.11220 2020.
[62] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248-255. Icee, 2009.
[63] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv: /w1.iv::412..o 121.441220..... 2014.
[64] Spyridon Bakas, Mauricio Reyes, Andras Jakab, Stefan Bauer, Markus Rempfler, Alessan- dro Crimi, Russell Takeshi Shinohara, Christoph Berger, Sung Min Ha, Martin Rozycki, et al. Identifying the best machine learning algorithms for brain tumor segmentation, pro- gression assessment, and overall survival prediction in the brats challenge. arXiv preprint arXiv:1811.02629 2018.
[65] Xiahai Zhuang and Juan Shen. Multi-scale patch and multi-modality atlases for whole heart segmentation of mri. Medical image analysis, 31:77-87, 2016.
[66] Noel Codella, Veronica Rotemberg, Philipp Tschandl, M Emre Celebi, Stephen Dusza, David Gutman, Brian Helba, Aadi Kalloo, Konstantinos Liopyris, Michael Marchetti, et al. Skin lesion analysis toward melanoma detection 2018: A challenge hosted by the international skin imaging collaboration (isic). arXiv preprint ar X wetp:9902.3060 2019.
[67] Junji Shiraishi, Shigehiko Katsuragawa, Junpei Ikezoe. Tsuneo Matsumoto, Takeshi Kobayashi, Ken-ichi Komatsu, Mitate Matsui, Hiroshi Fujita, Yoshie Kodera, and Kunio Doi. Development of digital image database for chest radiographs with and without lung nodule: receiver operating characteristic analysis of radiologists' detection of pulmonary nodules. American Journal of Roentgenology. 174(1):71-74 2000.
[68] Konstantin Pogorelov, Kristin Ranheim Randel, Carsten Griwodz, Sigrun Losada Eskeland, Thomas de Lange. Dag Johansen, Concetto Spampinato, Duc-Tien Dang-Nguyen, Mathias Lux, Peter Thelin Schmidt, et al. Kvasir: A mulli-class image dataset for computer aided gastrointestinal disease detection. In Proceedings of the 8th ACM on Multimedia Systems Conference, pages 164-169, 2017.
[69] Joes Staal, Michael D Abråmoff, Meindert Niemeijer, Max A Viergever, and Bram Van Gin- neken. Ridge-based vessel segmentation color images of the retina. IEEE transactions on medical imaging, 23(4):501-509 2004.
14

---

[70] Walid Al-Dhabyani, Mohammed Gomaa, Hussien Khaled, and Aly Fahmy. Dataset of breast ultrasound images. Data in brief, 28: 104863, 2020.
[71] Yi Zhou, Boyang Wang, Lei Huang, Shanshan Cui, and Ling Shao. A benchmark for studying diabetic retinopathy: segmentation, grading, and transferability. IEEE Transactions on Medical Imaging, 40(3):8 18-828, 2020.
[72] Quande Liu, Qi Dou, Lequan Yu, and Pheng Ann Heng. Ms-net: Multi-site network for improving prostate segmentation with heterogeneous mri data. IEEE Transactions on Medical Imaging. 2020.
[73] Ha Q Nguyen, Khanh Lam, Linh Le, Hieu H Pham, Dat Q Tran, Dung B Nguyen, Dung D Le, Chi M Pham, Hang TT Tong, Diep H Dinh, al. Vindr-exr: An open dataset of chest x-rays with radiologist's 's annotations. Scientific Data, 91):4299, 2022.
[74] Sharif Amit Kamran, Khondker Fariha Hossain, Alireza Tavakkoli, Stewart Lee Zuckerbrod, Kenton M Sanders, and Salah A Baker. Rv-gan: Segmenting retinal vascular structure in fundus photographs using a novel multi-scale generative adversarial network. In Medical Image Computing and Computer Assisted Intervention-MIICAN. 2021: 24th International Conference, Strasbourg. France, September 27-October 1, 2021, Proceedings, Part VIII 24, pages 44. Springer, 2021.
[75] Andriy Myronenko. 3d mri brain tumor segmentation using autoencoder regularization. In Brainlesion: Glioma, Multiple Sclerosis, Stroke and Traumatic Brain Injuries: 4th Interna- tional Workshop, BrainLes 2018, Held in Conjunction with MICCAI 2018, Granada, Spain, September 16, 2018, Revised Selected Papers, Part 4, pages 31 1-320. Springer, 2019.
[76] Jieneng Chen, Yongyi Lu, Qihang Yu, Xiangde Luo, Ehsan Adeli, Yan Wang. Le Lu, Alan Yuille, and Yuyin Zhou. Transunet: Transformers make strong encoders for medical image segmentation. arXiv preprint urXiv:2102.043006 2021.
[77] Ozan Oktay, Jo Schlemper, Loie Le Folgoc. Matthew Lee, Mattias Heinrich, Kazunari Misawa, Kensaku Mori, Steven McDonagh, Nils Y Hammerla, Bernhard Kainz, et al. Attention u-net: Learning where to look for the pancreas. arXiv preprint arXiv: 03999, 2018.
[78] Z. Zhou, MMR Siddiquee, N Tajbakhsh, and J UNet+ Liang. A nested u-net architecture for medical image segmentation (2018). arXiv preprint arXiv: /807.10066 2018
[79] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the IEEE/CV international conference on computer vision, pages 9650-9660, 2021.
[80] Ali Hatamizadeh, Yucheng Tang, Vishwesh Nath, Dong Yang, Andriy Myronenko, Bennett Landman, Holger R Roth, and Daguang Xu. Unetr: Transformers for 3d medical image seg- mentation. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 574-584, 2022.
[81] Joao Carreira and Andrew Zisserman. Quo vadis, action recognition? a new model and the kinetics dataset. In proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 6299-629966 6308,, 2017.
[82] Eli Gibson, Wenqi Li. Carole Sudre, Lucas Fidon, Dzhoshkun Shakir, Guotai Wang, Zach Rosen, Robert Gray, Tom Doel, Yipeng Hu, et al. NiftyNet: a deep-lcarning platform for medical imaging. Computer methods and programs in biomedicine, 158: 111-11222222 2018.
[83] Sihong Chen, Kai Ma, and Yefeng Zheng. Med3d: Transfer learning for 3d medical image analysis. arXiv: /904. 00025,. 2019.
[84] Xiaoman Zhang, Ya Zhang, Xiaoyun Zhang, and Yanfeng Wang. Universal model for 3d medical image analysis. ArXiv, abs/2010.06107. 2020.
[85] Yucheng Tang, Dong Yang, Wenqi Li, Holger R Roth, Bennett Landman, Daguang Xu, Vishwesh Nath, and Ali Hatamizadeh. Self-supervised pre-training of swin transformers for 3d medical image analysis. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pages 20730-207440 2022.
15

---

[86] Duy MH Nguyen, Hoang Nguyen, Mai TN Truong, Tri Cao, Binh T Nguyen, Nhat Ho. Paul Swoboda, Shadi Albarqouni, Pengtao Xie, and Daniel Sonntag. Joint self-supervised image-volume representation learning with intra-inter contrastive clustering. arXiv preprint arXiv:2212.01893, 2022.
[87] Zhiwen Lin, Ruoqian Guo, Yanjie Wang, Bian Wu, Tingting Chen, Wenzhe Wang, Danny Z Chen, and Jian Wu. A framework for identifying diabetic retinopathy based on anti-noise detection and attention-bau based fusion. In Medical Image Computing and Computer Assisted iiceveetion-MMCCAA 2018: st International Conference, Granada, Spain, September 16-20, 2018, Proceedings, Part II 11, pages 74- -2. Springer, 2018
[88] Yu-Huan Wu, Shang-Hua Gao. Jie Mei, Jun Xu, Deng-Ping Ping Fan. Rong-Guo Zhang, and Ming-Ming Cheng. Jes: An explainable covid-19 diagnosis system by joint classification and segmentation. IEEE Transactions on Image Processing. 30:3113-3126, 2021.
[89] Yi Zhou, Xiaodong He, Lei Huang, Li Liu, Fan Zhu, Shanshan Cui, and Ling Shao. Collab- orative learning of semi-supervised segmentation and classification for medical images. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 2079-2088, 2019.
[90] Hasan Md Tusfiqur, Duy MH Nguyen, Mai TN Truong, Triet A Nguyen, Binh Nguyen, Michael Barz, Hans-Juergen Profitlich, Ngoc TT Than, Ngan Le, Pengtao Xie, et al. Drg- net: Interactive joint learning of multi-lesion segmentation and classification for diabetic retinopathy grading. arXiv preprint arXiv: 14615, 2022.
[91] Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In Medical Image Computing and Computer-Asiisted DneeventiionMICC.. 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings. Part III 18, pages 234 241. Springer, 2015.
[92] Shruti Jadon. A survey loss functions for semantic segmentation. 2020 IEEE conference computational intelligence in bioinformatics and computational biology (CIBCB), pages 1-7. IEEE, 2020.
[93] Ross Girshick. Fast r-cnn. In Proceedings of the IEEE international conference on computer vision, pages 1440-1448, 2015.
[94] Arnaud AA Setio, Colin Jacobs, Jaap Gelderblom, and Bram van Ginneken. Automatic detection of large pulmonary solid nodules in thoracic images. Medical physies, 42(10): 5642-5651 2015.
[95] Patrick Bilic, Patrick Ferdinand Christ, Eugene Vorontsov, Grzegorz Chlebus, Hao Chen, Qi Dou, Chi-Wing Fu, Xiao Han, Pheng-Ann Heng, Jürgen Hesser, et al. The liver tumor segmentation benchmark (LiTS). 1XXi::901..04050 2019.
[96] Amber L Simpson, Michela Antonelli, Spyridon Bakas, Michel Bilello, Keyvan Farahani, Bram Van Ginneken, Annette Kopp-Schneider, Bennett A Landman, Geert Litjens, Bjoern Menze, et al. A large annotated medical image dataset for the development and evaluation of segmentation algorithms. arXiv preprini arXiv: 0w... 2019.
[97] Hanna Borgli, Vajira Thambawita, Pia H Smedsrud, Steven Hicks, Debesh Jha, Sigrun L Eske- land, Kristin Ranheim Randel, Konstantin Pogorelov, Mathias Lux, Due Tien Dang Nguyen, et al. Hyperkvasir, a comprehensive multi-class image and video dataset for gastrointestinal endoscopy. Scientific data, 7(1):283, 2020.
[98] Bastiaan S Veeling, Jasper Linmans. Jim Winkens, Taco Cohen, and Max Welling. Rotation equivariant CNNs for digital pathology. June 2018.
[99] Babak Ehteshami Bejnordi, Mitko Veta, Paul Johannes Van Diest, Bram Van Ginneken, Nico Karssemeijer, Geert Litjens, Jeroen AWM Van Der Laak, Meyke Hermsen, Quirine F Manson, Maschenka Balkenhol, et al. Diagnostic assessment of deep learning algorithms for detection of lymph node metastases in women with breast cancer. Jama, 3 (18(22)2119922111 2017.
16

---

[100]
Bjoern H. Menze, Andras Jakab, Stefan Bauer, Jayashree Selpathy-Cramee Keyvan Farahani, Justin Kirby, Yuliya Burren, Nicole Porz, Johannes Slotboom, Roland Wiest, Levente Lanezi, Elizabeth Gerstner, Marc-André Weber, Tal Arbel, Brian B. Avants, Nicholas Ayache, Patricia Buendia, D. Louis Collins, Nicolas Cordier, Jason Corso, Antonio Criminisi, Tilak Das, Hervé Delingette, Çağatay Demiralp, Christopher R. Durst, Michel Dojat, Senan Doyle, Joana Festa, Florence Forbes, Ezequiel Geremia, Ben Glocker, Polina Golland, Xiaotao Guo, Andac Hamamci, Khan M. Iftekharuddin, Raj Jena, Nigel M. John, Ender Konukoglu, Danial Lashkari, José António Mariz, Raphael Meier, Sérgio Pereira, Doina Precup, Stephen Price, Tammy Riklin Raviv, Syed M. S. Reza, Michael Ryan, Duygu Sarikaya, Lawrence Schwartz, Hoo-Chang Shin, Jamie Shotton, Carlos A. Silva, Nuno Sousa, Nagesh K. Subbanna, Gabor Szekely, Thomas J. Taylor. Owen M. Thomas, Nicholas J. Tustison, Gozde Unal, Flor Vasseur, Max Wintermark, Dong Hye Ye, Liang Zhao, Binsheng Zhao, Darko Zikic, Marcel Prastawa, Mauricio Reyes, and Koen Van Leemput. The multimodal brain tumor image segmentation benchmark (brats). IEEE Transactions on Medical Imaging, 34(10):1993322)) 2015. doi: 10. (110/T01.2014.337777
[101]
Christopher T Lloyd, Alessandro Sorichetta, and Andrew Tatem. High resolution global gridded data for use in population studies. Scientific data, 4(1):1-17, 2017.
[102] Aaron Grossherg, Abdallah SR Mohamed, Hesham Elhalawani, William C Bennett, Kirk E Smith, Tracy S Nolan, Bowman Williams, Sasikarn Chamchod, Jolien Heukelom, Michael Kantor, et al. Imaging and clinical data archive for head and neck squamous cell carcinoma patients treated with radiotherapy. Scientific data, 5(1):1-10, 2018.
[103] Patrick Bilic, Patrick Christ, Hongwei Bran Li, Eugene Vorontsov, Avi Ben-Cohen, Georgios Kaissis, Adi Szeskin, Colin Jacobs, Gabriel Efrain Humpire Mamani, Gabriel Chartrand, et al. The liver tumor segmentation benchmark (lits). Medical Image Analysis, 84: 102680, 2023.
[104] Jennifer Yin Yee Kwan, Jie Su, Shao Hui Huang, Laleh S Ghoraie, Wei Xu, Biu Chan, Kenneth W Yip, Meredith Giuliani, Andrew Bayley, John Kim, et al. Radiomic biomarkers to refine risk models for distant metastasis in hpy-related oropharyngeal carcinoma. International Journal of Radiation Oncology* Biology* Physics, (02(4):1177-111) 2018.
[105] Kenneth Clark, Bruce Vendt, Kirk Smith, John Freymann, Justin Kirby, Paul Koppel, Stephen Moore, Stanley Phillips, David Maffitt, Michael Pringle, et al. The cancer imaging archive (tcia): maintaining and operating a public information repository. Journal of digital imaging, 26:1045-1057, 2013.
[106] JYY Kwan et al. Data from radiomic biomarkers to refine risk models for distant metastasis in oropharyngeal carcinoma, the cancer imaging archive, 2019.
[107] P Leaveyy A Sengupta, D Rakheja, O Daescu, HB Arunachalam, and R Mishra. Osteosarcoma data from ut southwestern/ut dallas for viable and necrotic tumor assessment [data set]. The Cancer Imaging Archive, 14, 2019.
[108] AA Yorke, GC McDonald, D Solis, and T Guerrero. Pelvic reference data. Cancer Imaging Arch, 2019.
[109] Holger R Roth, Amal Farag, E Turkbey, Le Lu, Jiamin Liu, and Ronald M Summers. Data from pancreas-ct. the cancer imaging archive IEEE Transactions on Image Processing, 2016.
[110] Holger Roth, Le Lu, Amal Farag, Hoo-Chang Shin, Jiamin Liu, Evrim B Turkbey, and Ronald M Summers. Deeporgan: Multi-level deep convolutional networks for automated pancreas segmentation, In Medical Image Computing and Computer-Assiste Intervention- MICCAI 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Pro- ceedings, Purt 1 18, pages 556-564. Springer, 2015.
[111] Geert Litjens, Oscar Debats, Jelle Barentsz, Nico Karssemeijer, and Henkjan Huisman. Computer-aided detection of prostate cancer in mri. IEEE transactions on medical imaging, 33(5):1083-1022 2014.
[112] Geert Litjens, Oscar Debats, Jelle Barentsz, Nico Karssemeijer, and Henkjan Huisman. Prosta- tex challenge data. The Cancer Imaging Archive, 2017. httpps: //doi. org/10 7937/K9) TCIA. 2017. 10117..MPMRRK
17

---

[113] FR Lucchesi and ND Aredes. Radiology data from the cancer genome atlas cervical squamous cell carcinoma and endocervical adenocarcinoma (tega-cese) collection. the cancer imaging archive. Cancer Imaging Arch, 2016.
[114] S Kirk, Y Lee, CA Sadow, S Levine, C Roche, E Bonaccio, and J Filiippini. Radiology data from the cancer genome atlas colon adenocarcinoma [tega-coad] collection. The Cancer Imaging Archive, 2016.
[115] F. Lucchesi and N. Aredes. The cancer genome atlas esophageal carcinoma collection (tega-esca) (version 3) [data set]. The Cancer Imaging Archive, 2016.
[116] MW Linehan, R Gautam, CA Sadow, and S Levine. Radiology data from the cancer genome atlas kidney chromophobe [tega-kich] collection. The Cancer Imaginy Archive, 2016.
[117] O Akin, P Elnajjar, M Heller, R Jarosz, B Erickson, S Kirk, and J Filippini. Radiology data from the cancer genome atlas kidney renal clear cell carcinoma [tega-kire] collection. The Cancer Imaging Archive, 1310, 2016.
[118] Roche, E Bonaccio, and J Filippini. Radiology data from the cancer genome atlas sarcoma [tega-sarc] collection. Cancer Imaging Arch. https://ddoi. org/10.7937 K, 9, 2016.
[119] Shenggan. Beed dataset, 2017. URL htt.p:tt///ttttbbccom com http//ShhggggBBBa /BCCD Dataset
[120] Shiv Gehlot, Anubha Gupta, and Ritu Gupta. Sdet-auxnet0: Det augmented stain deconvolu- tional cnn with auxiliary classifier for cancer diagnosis. Medical image analysis, 61:101661, 2020.
[121] A Gupta and R Gupta. All challenge dataset of isbi 2019 [data set]. https://do..or//1.. http///ii.or//1.773777733 793.79977c7...00000 2019 2019.
URL
[122] Rebecca Sawyer Lee, Francisco Gimenez, Assaf Hoogi, Kanae Kawai Miyake, Mia Gorovoy, and Daniel L Rubin. A curated mammography data set for use in computer-aided detection and diagnosis research. Scientific data, 4(1):1-9, 2017.
[123] R Sawyer Lee, F Gimenez, Hoogi, and D Rubin. Curated breast imaging subset of ddsm. The Cancer Imaging Archive. DOI: https://doi. org/10.7937 K, 9, 2016.
[124] Linda Wang, Zhong Qiu Lin, and Alexander Wong. Covid-net: tailored deep convolutional neural network design for detection of covid-19 cases from chest x-ray images. Scientific Reports, 10(1):19549, Nov 2020. ISSN 2045-2322. doi: 1111038/441598-020--6566000 URL
https://doi. org. 10. 101111///111-0000070707-55
[125] Daniel Kermany, Kang Zhang, Michael Goldbaum, al. Labeled optical coherence tomogra- phy (oct) and chest x-ray images for classification. Mendeley data, 2(2):651, 2018
[126] Salman Maqbool, Aqsa Riaz, Hasan Sajid, and Osman Hasan. m2caiseg: Semantic seg- mentation of laparoscopic images using convolutional neural networks. arXiv preprint arXiv:2008. 101311111220 2020.
[127] Mohamed Amgad, Habiba Elfandy, Hagar Hussein, Lamees A Atteya, Mai A T Elsebaic, Lamia Abo Elnasr, Rokia A Sakr, Hazem S E Salem, Ahmed F Ismail, Anas M Saad, Joumana Ahmed, Maha A T Elsebaie, Mustafijur Rahman, Inas A Ruhban, Nada M Elgazar, Yahya Alagha, Mohamed H Osman, Ahmed M Alhusseiny, Mariam M Khalaf, Abo-Alela F Younes, Ali Abdulkarim, Duaa M Younes, Ahmed M Gadallah, Ahmad M Elkashash, Salma Y Fala, Basma M Zaki, Jonathan Beezley, Deepak R Chittajallu, David Manthey, David A Gutman, and Lee A D Cooper. Structured crowdsourcing enables convolutional segmentation of histology images. Bioinformatics, 35(18):341133467 02 2019. ISSN 1367-4803. doi: 1093/ 1093/ http://00000//iiooooaattttt///aa
bioinformatics/hz003... URL. https:ttss://ooi.cco
[128] Fabio Cuzzolin, Vivek Singh Bawa, Inna son Ravindran Charles, Elettra Oleari, Alice Leporini, Carmela Landolfo, Armando Stabile, Francesco Setti, al. Saras challenge on multi-domain endoscopic surgeon action detection, 2021.
Mohamed Mohamed, Jack-
18

---

[129] Vivek Singh Bawa, Gurkirt Singh, Francis KapingA, Inna Alice Leporini, Carmela Landolfo, Pengfei Zhao, Xi Xiang, Gongning Luo, Kuanquan Wang. Liangzhi Li, Bowen Wang, Shang Zhao, Li Li, Armando Stabile, Francesco Setti, Riccardo Muradore, and Fabio Cuzzolin. The saras endoscopic surgeon action detection (esad) dataset: Challenges and methods, 2021.
Elettra Oleari,
[130] Vivek Singh Bawa, Gurkirt Singh, Francis KapingA, Inna AKarga-Bandurovs Alice Leporini, Carmela Landolfo, Armando Stabile, Francesco Setti, Riccardo Muradore, Elettra Oleari, and Fabio Cuzzolin. Esad: Endoscopic surgeon action detection dataset, 2020.
[131]
Shoulder x-ray classification. Kaggle dataset. URL https: wwggk//lee com/ com/datasets/ http:////shhldeeerraaa-aaaaaaaae
[132] Lung masks for shenzhen hospital chest x-ray set. Kaggle dataset. URL hatpps:/wwt hrggll/.ccoo/aaaaaaatttoooooaaaa
-ask
[133] Susanne G Mueller, Michael W Weiner, Leon J Thal, Ronald C Petersen, Clifford Jack, William Jagust, John Q ''rojanowski, Arthur W Toga, and Laurel Beckett. The alzheimer's disease neuroimaging initiative. Neuroinaging Clinics, 15(4))869 17799-877720 2005.
[134] Ronald Carl Petersen, Paul S Aisen, Laurel Beckett, Michael C Donohue, Anthony Collins Gamst, Danielle J Harvey, Clifford R Jack, William J Jagust, Leslie M Shaw, Arthur W Toga, et al. Alzheimer's disease neuroimaging initiative (adni): clinical characterization. Neurology, (443):201-209,2111
[135] Christian Matek, Simone Schwarz, Carsten Marr, and Karsten Spiekermann. single-cell morphological dataset of leukocytes from aml patients and non-malignant controls (aml- cytomorphology_mme Imm)) The Cancer Imaging Archive (TCIA)[[nteree]) 2019.
[136] Christian Matek, Simone Schwarz, Karsten Spiekermann, and Carsten Marr. Human-level recognition blast cells acute myeloid leukaemia with convolutional neural networks. Nature Machine Intelligence, 1(11):538-544 2019.
[137] Aptos 2019 blindness detection. Kaggle dataset. URL httpps:/www..agglee... c/ aptos2019- htw.::-bbbbinneeeee-eeeettttttee data
[138] Mohamed Amgad, Habiba Elfandy, Hagar Hussein, Lamees A Atteya, Mai AT Elsebaie, Lamia S Abo Elnasr, Rokia A Sakr, Hazem SE Salem, Ahmed F Ismail, Anas M Saad, et al. Structured crowdsourcing enables convolutional segmentation of histology images. Bioinformatics, 35(18);3461-3467 2019.
[139] Amir Abdi and S Kasaei. Panoramic dental x-rays with segmented mandibles. Mendeley Data, v2, 2020.
[140] Thomas LA van den Heuvel, Dagmar de Bruijn, Chris L de Korte, and Bram van Ginneken. Automated measurement of fetal head cireumference using 2d ultrasound images. PloS one, 1388):e0200412, 2018.
[141] Hippocampus segmentation in mri images. Kaggle dataset. URL https///:///w kaggle httm://ttaeeesssmmmmmm////pppppp
wwweeestteeennnnnneeeeeeeeeeeeoe in- in--ri--------------- imagemmseeecss -ag
Test
[142]
Skin lesion images for alasioma Kaggle dataset. URI.ltttps:/www.aagg com/
[143] Ahmed Taha, Pechin Lo, Junning Li, and Tao Zhao. Kid-net: convolution networks for kidney vessels segmentation from ct-volumes. In Medical Image Computing and Computer Assisted Intereetion-MMICAA/ 2018: s International Conference, Granada, Spain, September 16-20, 2018, Proceedings, Part IV 11, pages 463-471. Spriner, 2018
[144] Kvasir v2. Kaggle dataset. URI, https::/p//// kaggle wwt.///aaaatte//laalleeeeee kvasir v2- gastrointestinaal...... -tacttttcttcooo dataset
19

---

[145] Lhnebe malaria.
html#
halaria-datasen
[146] Wei, Z. Lin, D. Barranco, N. Wendt, X. Liu, W. Yin, X. Huang, A. Gupta, W. Jang, X. Wang. I. Arganda-Carreris J. Lichtman, and H. Pfister. Mitoem dataset: Large-scale 3d mitochondria instance segmentation from em images. In International Conference on Medical Image Computing and Computer Assisted Intervention, 2020.
[147] Christian Matek, Sebastian Krappe, Christian Münzenmayer, Torsten Haferlach, and Carsten Marr. Highly accurate differentiation of bone marrow cell morphologies using deep neural networks large image data set. Blood, The Journal of the American Society of Hemalology, (90220):1917-99999 2021.
[148] Safwan S Halabi, Luciano M Prevedello, Jayashree Kalpathy-Crames Artem B Mamonov, Alexander Bilbily, Mark Cicero, Ian Pan, Lucas Araújo Pereira, Rafael Teixeira Sousa, Nitamar Abdala, et al. The rsna pediatric bone age machine learning challenge. Radiology. 290(2): 498-503, 2019.
[149] Kaggle dr dataset (eyepacs). Kaggle dataset. URL http://wwwwaagll....//aattteeeee mari mariaherr aarraalereeeo//eppppppprrrrsssss
20

---

Supplementary Material
We present below LVM-Med pseudo-code (Section A). implementations used in downstream tasks (Section B). additional ablation studies of LVM-Med (Section ( further prompt-based baset segmentation results on 3D datasets, image classification benchmark (Section [). predicted masks using the user-based prompt (Section Bd, and finally the dataset overview (Section
A LVM-Med Pseudo-code
First, we provide a pseudo-code for training LVM-Med in Pytorch style:
# fg: encoder network, h projector network, ge message passing network,
k_nodes: number of nearest neighbors, Avg: average pooling,
# pos: position of image after transform, cos: cosine similarity,
a: coefficient trades off between global and local costs, L L2-distance,
# y: maximum pairs are kept, select_top: select to keep the best matches.
for X in loader: # load batch X = X1, x2, xy] with N samples
# apply two transformations s and
x², Pos* = sX) # x x x
xx) Pos* = ppos pos ... pos k E (s,t}
xt, Pos* = txX)
# compute feature representations y fx(x)); Yt fx*) # feature dimensions NxDxRxS
# applying projection
2 = ha (Avg(Y)))) = (Avg((Y)))) dimensions: NxF
# build graph structures and message passing
G t-nearesttneiighorr (z k_ connectss
G k-nearest-neighhor (Z* _k__connettt)
2. g. (G); 2 - g. (G*, 2z)
# compute vertex and edge affinity matrices
c. Cla.jb cos 23), (2) 2))) affinity between edges vig.
* cos (2; 2 + (1 -a) * local_costlyf, y pos,, post) # affinity xf & x²
c" (c],) E Ruxu, c (cfa.jo) E R| # E be set of edges in Gk k (s,t)
# perturbed costs with Gumbel noise e, ~ Gumbel(O,11 1) c" = + cº c +e'
# solving graph matching and compute loss
9 = CM(c", c²) L(9 0. v') v" 011(1-11 (1-+) +) # compute hamming loss
# update network
L. backward # approximate (ØL/Oc", 0L/Oc*) by Algorithn 1.
Update (hs params), Update ofe params)
define local_coo
def local_cost (y². yi. pos,, pos;)
# location-based local cost yi.. torch.zeros_liikerr? for in R, S:
r', s² argmin ((a post(=', s())) yimu [I. [r.s] s] = =
21

---

Vi_til: Vi,,o select_top (yi, Vi 7)
location_coas cost cos v.
# featured-based local cost
Vi. torch.zeros_likkev]]] for I, a in R, S:
r³, s² argmin(([2(villr.a) si(r', yi r, = y
Vin = select_ton top (yi, vi 7)
feature_coss = cos (ys_yfnm y m11)
return 0. (location_cosst + feature_cost)
We trained LVM-Med with graph size of 16 nodes, each node connected the top 5 near- est neighbors after using KNN, À value in Algorithm is 80, and a 0.8 for associating global- and local-based similarities when computing cjj. The size of projector hgg is 2048 x 128 for ResNet-50, and 768 x 128 for VIT. We configure the message passing network go with two convolutional layers of size 128. For the user-based prompt version, because the SAM model [ requires an input of shape 256 14 x 14 for the mask decoder part, we add two additional convolutional layers with kernel size of and 3 at the end ViT backbone convert from shape 768 x x 14 the target shape.
B Downstream task setups
B.1 Downstream tasks
Segmentation tasks On 2D-based segmentation tasks, we employ U-Net architecture 99 and load ResNet-50 [60] trained by self-supervised learning algorithms as network backbones. With foundation models, use TransUnet (76) and take pre-trained ViT models as the backbones. For the prompt-based segmentation, we follow the architecture of SAM 6 consisting of encoder, prompt, and mask decoder layers. We also fine tune SAM where encoder and prompt networks are frozen, only learning decoder layers [9] Our LVM-Med for prompt-based setting is similar to [9] except that we substitute SAM's encoders with our weights. We utilize Adam optimizer for all experiments and train architectures with Dice and Cross-Entropy loss [92]. We also normalize the norm-2 of gradient values stabilize the training step to maximize 1. Table summarizes each dataset's 's learning rate, number of epochs, and image resolution.
On 3D-based segmentations, we reformulate these tasks as 2D segmentation problems and make predictions on 2D slices taken from 3D volumes. Furthermore, we apply balance sampling to select cqually 2D slices covering target regions and other 2D slices, not including the ground truth. Table presents configurations used for 3D datasets; other settings are identical to 2D cases.
Table 7: Configurations for training 2D segmentation tasks
ISIC-2018 (Skin Lesion)
JSRT (Lang X-eay)
Kvathir (Pulyp)
Drive (Vosel))
BUID (Breast Cancer)
Ir 10-4 opochs 35 batch
Io 10-2. epochs 50 shape 224 224 botch
Ir 10-3, opochs 35 224 224 batch
Ir 10-2, epochs 50 224 batch
Ir 10-4 opachs 50 butch
ResNet-50.
512
256 258
opochs shape 512 x 612 batch size 16
opochs
epochs stape 224 226
opochs
opochs
Foundation Model
shipe 224 224
shape 224 x 224 batch size 16
shape 250 250 butch size
butch size 02
batch size
Ire 30 epochs bil slape 1024 x 1024
Ir = x 10 opachs 501 shope 1024 1024
Ir x 10 opechs 20
Ir 10 opochss 1001
Ir 10 epochs 20 shape 1024 x 1024 batch 16
Prompt-hnsed Seg,
1024 1024
shape 1024 x 1024
batch
botch
batch
botch size
22

---

Table Configurations for 3D-based seaseddsegmentttionn tasks
BraTS
MMWHS-CT
MMWHS-MRI
BMC
Ir= 10 opochs
10-3, opochs 20 shope 224 x 224
Ir= 10-1 10-ch_ppocs shope 224 x 224
Ir= 10-9, spochs 30
ResNet50
shape 224 x 224 batch size 128
shape 224 x 224
batch size 64
butch size 64
batch size 64
Ir= 10 epochs 100 shape batch size 16
Ir 10 epochs 100
Ir= 10 epochs 100
I= 10 epochs 100
Foundation
shape 221 221
shape 224 221
shape 224 221
batch size 16
butch size 16
batch size 16
10 9, epochs shape 1022 x 1024
epochs
epochs
x epochs shape 1021 x 1024 batch size
Prompt-based Seg.
shape 1024 x 1021 batch 16
shape 1024 x 1021 batch 16
batch 16
Image classification tasks We take the feature embedding outputs of each architecture and build one fully connected layer to produce desired classes for image classification tasks. We freeze the encoder layers for the linear evaluation and only train the fully connected layer. For the fully-finetuning, the whole network is trained. The Adam optimizer (33 with cross-entropy loss function and learning rates (5 10-1, 10-3) are used for Brain Tumor and FGADR, respectively. To benchmark L.V-- Med with other state-of-thed thrt methods on FGADR (Figure 3 in paper), we follow the settings of DRG-Net [90] and change their encoder layers by our networks.
Object detection We use Faster-RCNN [93 for object detection tasks. The ResNet-50 of Faster- RCNN is replaced by pre-trained weights. In the Vin-Dr dataset, there is a total 14 objects for, e.g., Aortic enlargement, Atelectasis, Calcification, etc. We use image resolutions of 512 x 512, Adam solver, and learning rate 10 in 40 epochs. In the Kvasir dataset for polyp detection, we also resize images fixed size of 512 512, employ the Adam optimizer with learning rate 2.5 x 10-4 and batch size 8.
C LVM-Med ablation studies
C.1 Graph sizes and in backpropagation
We provide in Figure 5 and Figure L LV-M performance when changing the number of nodes in graph construction steps G" G' and À 80 used in Algorithm I in the backpropagation step. The results reported on the average Dice of two linear classifications on FGADR and Brain Tumor Classification. Figure ( indicates that 16 is the best value for both classification and segmentation. Increasing the graph's nodes tends decrease classification performance.
of five 2D segmentation tasks and the average accuracy
Figure 6 compared different values for À E (70,80) 80, 90, 100). We observe that À (80, 90) achieve good results for linear classification tasks though À = {90, 100} decreases segmentation performance.
C.2 Performance on large- and small-scale
We investigate LVM-Med performance when reducing the number of datasets in the pre-training step. Especially, we trained LVM-Med on small-scale with four datasets: LUNA2016 194]. LITS2017 [95], BraTS2018 64), and MSD (Heart) [96]. We compare this version with our default settings trained on 55 datasets (Section F.. Two models are evaluated on dice scores of five 2D segmentation tasks, the accuracy metric two linear image classifications, and mAP50 of two object detection tasks on VinDr and Kvasir detection. Table shows that LMV-Med full leads to better performance overall, especially with the classification settings; the improvement gap is around 3.6%. In summary, conclude that LVM-Med is beneficial when training in large-scale medical settings.
23

---

Linear Cls. (Acc) Seg. (Dice)
Linear Cls. (Acc)
Seg. (Dice)
100
100
90
90
80
60
60
50
50
70
80
90
100
Figure 5: LVM-Med performance when varying the number of nodes in graph construction.
Figure 6: LVM-Med performance when varying the backpropagation step.
C.3 Performance on weighting global and local similarities
We test with different a {0.7, 0.9} which used fuse global- and local-based similarities d Table demmosttrates that a = 0.8 is generally the best value in average across segmentation, classification, and object detection tasks.
Table 9: LVM-Med ablation studies trained with full data, small-scale, and different hyper- hyper-paranen parameter fusing global- and local-based similarities. Results are reported on an average of five 2D seg- mentation, two linear classifications, and two object detection tasks. The most impacted factors are highlighted.
Method
Cls.(Acc)
Seg. (Dice)
Detect. (mAP50)
LVM-Med (full, 0.8)
67.47
83.05
57.1
LVd-Med (small-scale, a 0.8)
63.83
81.97
56.03
LVM-Med (full, 0.7)
65.89
82.20
56.49
LVM-Med (full, 0.9)
65.03
81.09
57.14
C.4 Computational complexity
We present parameter comparison of LVM-Med with other foundation models in Table 10 Our LVM-Med model, based on ResNet-50, has significantly fewer parameters, approximately 3-4 times smaller than models such as Flava SAM, while still maintaining competitive performance. When utilizing the ViT encoder pre- ire-trained by the SAM method, LVM-Med's parameters are comparable to the Flava model and slightly higher than Clip and Align by 1.03 and 1.43 times, respectively. However, it is important to note that both LVM-Med and SAM outperform these models by significant margin in various settings.
Table 10: Computational complexity of our approaches and other foundation models.
Method LVM-Med (R50)
LVM-Med (ViT)
Clip
Flava 5
Align 4
SAM (Encoder) 6
#Param
25.55 M
88.88 M
85.80 M 86.39 M
62.14M
88.88 M
D Prompt-based segmentation on 3D datasets and classification tasks
provide additional results for L.V--Med on 3D-based prompt segmentation and image classifica- tion tasks with several fully connected layers.
24

---

Table 12: Comparing SSL approaches Foundation models on classification tasks with two evaluation protocols, Linear evaluation and full Fine-tuning, Settings used with several fully connected layers are in eyan. The best results in 2D-SSL and foundation models (two fully connected layers) are in bold; the best results overall are bold and underlined.
Method
Linear Evaluation (Frozen)
Fine-tuning
FGADR (DR Gruding)
Brain Tumor Class,
FGADR (DR Grading)
Brain Tumor Class.
Twin Baston B6]
66.86 = 0.41
63.03 1 0.32
66.37 1: 0.77
74,20 = 1.38
Dino [9]
65,98 1.91
62.27 0.32
67.35
1.36
71.91 1.55
1.67
2D-SLL on medical
Deepeluster 20
Moco-v2
65.34 1.93
65.98 =: 1.04
64.47 + 0.55
62.35 1: 1.92
67.94 + 1.78
67.55 1: 1.79
73,10 = 0.55
74.53 0.43
VicRogl
64,71 0.60
59,64
1,36
65.69
1.46
73.18 2.03
LVMI-Med (R50)
66.67 = 0.84
74.70 =: 0.84
70.58
0.36
78.77 0.78
3
$7.87 0.50
$1.87 + 0.71
57.48 + 0.86
34.86 = 2.27
0.23
1.86
Flava S
32.84 0.12
31.87 0.69
35.19 0.43
24.45 + 4.30
57.18 +: 0.96
56.01 + 0.86
34.01 5.97
33.67 8.11
Foundation Model
Algin
36.95 1.04
30.71 + 2.35
57.28 + 0.97
63.96 + 0.04
0.25
SAM
55.13 1: 0.41
57.48 :: 0.24
31.81 1: 4,26
36.89 :: 1.61
58.75 +: 1.32
58.75 =: 0.99
60.66 1.36
60.07 =: 0.31
LVM-Med (SAM's VIT)
62,46 0.86
59.31 0.48
63.44 1: 0.73
67.34 =: 2.08
D.1 Promt-based Segmentation on 3D datasets
We perform experiments on three 3D datasets in Table including BraTS, MMWHS-MRI, and
MMWHS-CT. The setup for box prompts follows 2D segmentation cases. We discover that the LMV-Med in 3D cases consistently improves the performance of fine-tuned SAM ( as in 2D settings and attains large margin compared with SAM without training [6]. This evidence thus confirms that LVM-Med is also effective under prompt-based scenarios.
Table 11: Prompt-based segmentation on 3D datasets.
Method
BraTS
MMWHS-MRI
MMWHS-CT
SAM (fixed encoder)
85.37
77.64 1.14
76.61 + 1.91
Prompt-based Seg.
SAM with Prompt (no-train) 6
38.97 0.21
59.74 + 0.76
50.25 + 0.33
LVM-Med (SAM's VIT)
85.76 10.07
78.91 + 0.80
78.03 + 0.93
D.2 Image classification
We aim to inspect whether foundation models improve their performance given more fully connected layers for image classification tasks with both frozen encoders fully fine-tuning. For each method in this category and our LVM-Med (ResNer-50 and ViT), we configure two fully connected layers with sizes 512 256 and 512 128 for the Brain and FGADR respectively that map from the output dimension of each network to a number of desired classes. Table 12 presents obtained results where new settings are highlighted in color. We notice the following points. (i) Firstly, using more fully connected layers tends to improve the performance of foundation models, especially on linear evaluation. For e.g., the Clip increases from 4. 49% 9.98% on FGADR and Brain Tumor classification tasks, respectively. Similarly. our LVM-Med with SAM's VIT also achieves better results by approximately 1.37% and 4.82% on those tasks. (ii) Secondly, LVM-Med overall attains the best results in four settings using linear or several fully connected layers with ResNet-50. LVM- Med with ViT architecture also delivers the best records on three four test cases compared with foundation models.
E Visualizing results
provide qualitative results for prompt-based segmentation in Figure We compare three approaches, including (i) the standard SAM without fine-tuning [6] (second column), (ii) SAM
25

---

with encoders and prompt networks are frozen, and only decoder layers are trained as [7] (third column), and (iii) a similar setting as (ii) but encoders taken from LVM-Med version with SAM's ViT architecture (fourth column). For all methods, we simulate box-based prompts using the ground-truth masks and define boxes covering those target regions perturbed by offset values.
Figure 7 demonstrates that the original SAM is prone to generate useless predictions (top and bottom rows) or less precise boundaries. In contrast, updated SAM and LVM-Med produce more accurate results, confirming the importance of fine-tuning to achieve adequate results. Figures in the third and fourth columns also illustrate that SAM tends over- over-segment or lacks structures on an object's edges in several cases, while LVM-Med is more stable in those situations (red arrows).
Ground Truth
SAM (no fine-tuning) 6
SAM (line-tuning) 9
LVM-Med (SAM's VIT)
Visualizing prompt-based predictions three datasets: BUID, Kvasir, and ISIC. Red arrows show differences between SAM (fine-tuning) and L.V--Med using SAM's VIT architecture. Best viewed in color with zoom.
Figure 7:
F
Dataset overviews
Table 13 overviews the dataset used in our study. For each dataset, we provide its modality, data dimension, and the total of samples. If the training/testing rate is available (column Train/Test Rate), we utilize all training data; otherwise, we sample 20% total samples to avoid potential test data leaking for downstream tasks used in the pre-training step. For datasets whose data dimensions are 3D volumes, we sample 2D slices from those formats. Some datasets, such as MSD or ADNI, comprise different sub-datasets inside; we consider these sub-sets as inde- pendent ones to avoid confusion during the training steps. In summary, total of 55 datasets are used with approximately 40% in 3D datasets and 60% in 2D images as presented in Fig- ure Moreover, also outline ratios between distinct data modalities such as MRI, CT, X-
26

---

ray, grayscale types such as Ultrasound, OCT, and finally, color images depicted in Figure
15.0%
datasets 40.0%
Color image 43.5%
datasets
MRI 5%
60.0%
Other Grayscale
Figure chart illustrating the ratio the number of datasets with the 3D and 2D dimension.
Figure 9: Pie chart illustrating the ratio of different
data modalities in collected dataset.
Table 13: Overview our collected medical dataset
No
Data Name
Topic
Disease
Modality Dimension
Train/Test Total Rate?
HyperKvasir
Polyp
Pathological classifi-
Color images
2D
Yes
110079
97
cation
PutchCumelyon
Cells
Histopathologie scans lymph node sections.
Color images
Yes
327680
99
BraTS2018
Brain
Tumor Segmentation
MRI
3D
No
760
[00 [01
64
HNSCC 102
Head Neck
No Label
155
LITS2017
Liver
Segmentation af
CT
3D
No
200
103
Liver and Tumor Lesions
MSD-Heart
Heart
Heart Segmentation
MRI
3D
No
30
96
MSD-Liver
Liver
Liver Segmentation
MRI
3D
No
201
96
MSD- MSD-Lung
Lung
Lung Segmentation
MRI
3D
No
96
96
MSD-
Pancreas
Pancrea
Segmenta-
MRI
3D
No
420
Pancreas
tion
196
10
MSD-
Hepatic
Ves-
Hepatic Vessel Seg- mentation
MRI
3D
No
443
Hepatic Vessel Vessel
sel
(96
MSD-Spleen
Spleen
Spleen Segmentation
MRI
96
12
MSD-Colon
Colon
Colon Segmentation
MRI
3D
No
190
96
13
OPC-
Oropharynx
Label
120
Radiomics
104 106
14
Osteosarcoma-
Osteosarcoma
No Label
Color
2D
No
547
UT 107 108
images
105
15
Panereas- CT
Pancreas
No Label
CT
3D
No
109
10
105
27

---

No
Data Name
Topic
Discase
Modality
Format Default Total Train/Test
Rate
16
Pelvic- Reference-
Pelvic
No Label
CT
3D
No
12
Data
[08 105
17
ProstateX
Prostate
The clinical signifi- of prostate lc- sions prediction
MRI
3D
No
40
111
112
105
18
TCGA-CESC
Cervical
No Label
Color images
2D
No
3977
113 105
19
TCGA-
Colon
No Label
Color images
2D
No
1644
COAD
114 10:5
20
TCGA-ESCA
Cuticle
No Label
Color
No
4427
115]
images
21
HCGA-KICH
Kidney
No Label
Color images
2D
No
2192
[166 [0S
22
TCGA-KIRC
Kidney
No Label
Color images
2D
No
34108
115
TCGA-
Rectum
No Label
Color images
No
248
READ
114 1OS
24
TCGA-SARC
Sarcoma
No Label
Color images
2D
No
624
[18 105
25
TCGA-
Thyroid
No Label
Color images
2D
No
665
THCA
[14 105
26
VinDr
Lung
Abnormal
Disease
X-ray
2D
No
18000
Classification
27
LUNA2016
Lung
Nodule Detection and False Positive Reduc-
CT
No
49386
28
BCCD 19
Cells
Blood cell detection
Color
2D
No
364
images
29
C-
Cells
Leukemia detection
Color
2D
Yes
12529
NMC Leukemia
images
120 121
CHIS-DDSM
Breast
Breast Cancer Classi-
X-ray
No
6774
122
fication
31
COVIDx
Lung
Covid-19 19 Detection
X-ray
2D
Yes
194922
1124
32
Heidelberg
Eye
OCT Imaging Classi-
OCT
2D
Yes
84495
OCT 125
fication
33
m2caiSeg
Laparoscopic
Semantic Segmenta- tion Laparoscopic
Color images
614
116
34
NuCLS 127
Nucleus
Nucleus Segmen- Color images
2D
Yes
1744
tation Detection
Classification
35
SARAS-
Prostatectomy Procedures
Action classification
Color images
2D
Yes
29454
MESAD
in
Prostatectomy
128
29
130
Surgey
36
Shoulder
Shoulder
Shoulder X-ray Clas- sification
X-ray
2D
Yes
1049
X-ray images
from Sun Yat-
Memorial
Hospital [33
28

---

No
Data Name
Topic
Discase
Modality
Format
Default Total Train/Test
Rate
37
Shenzhen
Lung
Lung segmentation
X-ray
2D
No
566
Hospital
X-
ray
132
38
ADNI
1,ST
Brain
Alzheimer's Disease Classification
MRI
3D
No
639
133 134
39
ADNI
3T
Brain
Alzheimer's ss Disease Classification
MRI
3D
No
119
133
134
AML-
Cell
Peripheral smears
blood
Color images
No
18365
Cytomorphology
135
336
105
41
APTOS 2019
Eye
Severity of diabetic
Color
2D
No
3662
137
retinopathy Classifi- cation
images
42
BCSS 138
Cells
Breast cancer seman- tic segmentation
Color
2D
No
151
images
43
Dental Panoramic
Tooth
Mandible segmenta- tion
X-ray
2D
No
116
[39
HC18 140
Fetal head circumfer- ence (HC)
Ultrasound
No
999
45
Hippseg 2011
Brain
Hippocampus mentation
Seg-
MRI
3D
No
3050
[41
46
ISIC lenge [142]
Chal- 2019
Skin
Skin Cancer Classifi- cation
Color images
2D
No
25331
47
KiTS19-21
Kidney
Kidney Segmentation
CT
3D
No
45424
143
48
Kvasir
v2
Gastrointestinal
Gastrointestinal can- Color
No
6000
cer image classifica-
images
49
LHNCBC
Cells
Malaria Classifica- lion
Color images
2D
No
27560
Malaria 145
50
MitoEM
Cells
Mitochondria
In-
MRI/CT
3D
No
1000
[146
stance Segmentation
51
MLL
Bone
Cells
Blood cell classifica- tion
Color
No
171374
Marrow
147
images
52
MMWHS-
Heart
Sub-structure Heart
CT
3D
Yes
40
CT
segmentation
53
MMWHS-
Heart
Sub-structure Heart segmentation
MRI
3D
Yes
40
MRI 65
RSNA
Bone
Bone age prediction
X-ray
No
12611
Age 148
55
EyePACS
Eye
Diabetic Retinopathy Detection
Color Images
2D
Yes
88702
[149
29


Reviews:
[{"review_id": "Jd865DNDGb", "reviewer": "Reviewer_WEtP", "summary": "The paper presents a large scale medical imaging dataset consisting of 1.3M medical images from 55 publicly available datasets along with a new contrastive learning framework based on graph matching. Specifically, the model is firstly pre-trained on the collected dataset, and then finetuned towards different downstream tasks, improvement is observed a cross different datasets and settings.", "questions": "see weakness", "limitations": "see weakness", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "+ A large scale *medical* dataset for pre-training purposes\n\n+ Well-written and easy to follow\n\n+ Extensive experiments and good results", "weaknesses": "- The paper did not provide enough details regarding the proposed datasets, especially in terms of the usage of the dataset, which, in my humble opinion, is pretty important to provide guidance on training medical models on large-scale datasets, e.g., 1. how are 3D data used, are they sliced into 2D data firstly? 2. The data comes from different datasets and maybe in different modalities, any balancing strategy during pre-training? 3. Any augmentations besides multi-crop? (e.g., flip, rotate, color jittering etc). In short, I expect the author provide more details regarding how they utilize the dataset\n\n- I appreciate the author provided dataset statistics details in the supplementary, yet I am curious how are these datasets selected and have the author tried any filtering/curation? As data curation has been considered very important in modern large-scale model training [1], it would be great to have some insights in the medical area.\n\n\n[1] DataComp: In search of the next generation of multimodal datasets\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "\n\nThank you very much for your positive and constructive feedback!\n\n\n**Question 1: Provide more details regarding the usage of the datasets.**\n\n**1.a:  How are 3D data used, are they sliced into 2D data first?**\n\nYes, for 3D volume data, we slice them into 2D images first. We mentioned this in **Section E. Dataset overviews** in the Appendix and will emphasize it more clearly in the revised version.\n\n&nbsp;\n&nbsp;\n\n**1.b: The data comes from different datasets and maybe in different modalities; any balancing strategy during pre-training?**\n\nWe did a statistical summary of the collected datasets (in Figure 5, Appendix), and the results indicate an imbalance between data modalities. To address this imbalance, we used the following strategies. \n\n1. In each mini-batch during pretraining, we randomly select a subset of available modalities, e.g., color image, X-ray, and MRI. \n\n\n2. To balance samples in each modality, we combine over-sampling and data augmentation to increase the total samples. Specifically, new samples from minority classes are generated by duplicating images and applying random crop operations covering 85-95% of image regions and then rescaling them to the original resolutions. Note that these augmentations are not used in the self-supervised algorithm  (operations s, t ~ T) to avoid generating identical distorted versions in this sampling procedure.\n\n     We will further discuss this sampling procedure in Section E. Dataset overviews.\n\n&nbsp;\n&nbsp;\n\n**1.c: Any augmentations besides multi-crop? (e.g., flip, rotate, color jittering, etc.)**\n\nFor the augmentations, we mainly follow prior work [29] that emphasizes the importance of multi-crop augmentation. Though, in the implementation, the method is combined with other widely used operations. In particular, what we use is multi-crop $\\rightarrow$ flip (probability 50%) $\\rightarrow$  color jilter $\\rightarrow$  random Gaussian blur $\\rightarrow$  normalization.\n\nWe will add more details in *Section 4.1 Implementation details*\n\n[29] Vicregl: Self-supervised learning of local visual features \n\n&nbsp;\n&nbsp;\n\n**Question 2: How are these datasets selected? Is any filtering/curation used?**\n\nIn this work, we focus on building a **large-scale model for medical images which only uses raw data without any supervised signals** (e.g., segmentation masks or classification labels). This is essentially different from other multi-modal data where image-text pairs are used as inputs for the algorithm, necessitating careful data curation to prevent skewed or inaccurate information. With this characteristic in mind, we employ specific criteria to pick datasets, outlined as follows:\n\n\n1. Collecting **as many public datasets as possible** whose number of samples is not too small (at least 100 images).\n2. The selected dataset represents **diverse data modalities** (including X-ray, CT, MRI, color images, ultrasound, etc).\n3. The selected data represents **diverse body organs** such as lungs, cells, brains, breasts, etc.\n\nTo define a set of potential datasets, we carefully survey from different sources such as competition challenges (e.g., grand-challenge),  papers,  or benchmarks in medical-related conferences/journals. The collected datasets are then sampled with 20% images if training/testing splittings are unavailable; otherwise, we use all samples in the training set to avoid potential testing data leaking, as discussed in the paper.\n\nWe hope our response resolved most of your concerns, and helped you evaluate our work more positively. If you have other comments, we are happy to address them in the reviewer-author discussion period.\n\n\n\n\n\n\n"}, {"review_id": "yuNxMcwY5x", "reviewer": "Reviewer_jRh7", "summary": "The paper proposes a set of networks called LVM-Med which are trained on large-scale medical datasets. The authors collected more than a million medical images from more than 50 publicly available datasets of diverse modalities and structures of interest (e.g. CT, MRI, Ultrasound...). In the work, several self-supervised algorithms are benchmarked on the large dataset. Furthermore, this work proposes a self-supervised contrastive learning algorithm based on a second-order graph-matching formulation.", "questions": "1) The work has been run on a large set of data. Can the authors provide more detail on the overall computing required to replicate their experiments? \n\n2) The work uses the reparameterization trick to create a complex discrete distribution. Can the authors explain the effect of the backpropagation and what this implies for the learning signal in more detail? \n\n3) Why are there no baselines for alternative ways to define a self-supervised contrastive loss in the presented setting? \n\nI would like to stress that I have an overall positive impression of the work and would be willing to reconsider my rating based on the rebuttal.", "limitations": "-", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- the paper combines an incredibly large number of medical image modalities and images\n\n- I like the formulation of contrastive learning as a graph matching objective \n\n- the method section is comprehensive and the contributions are formalized", "weaknesses": "1) Some aspects of the experimentation are unclear to me. From how I understand the text, the authors aim to compare to a large number of other datasets, baselines (\"In 2D settings, we also compare with 2D supervised architectures, such as U-Net, U-Net++, Attention U-Net, etc.\"), and tasks across 2D and 3D. What I do not understand is how the authors choose their baselines and what they present in the tables. For example, in Table 2. for the Drive segmentation dataset, the authors report 2D supervised Methods (e.g., UNet) with Dice scores ranging from 59 to 65. Clearly, this is not a performance on par with scores reported in other works. From the literature, the state of the art in supervised DRIVE segmentation should be way higher. A fully supervised segmentation baseline on the DRIVE dataset should have 80+ Dice (https://paperswithcode.com/sota/retinal-vessel-segmentation-on-drive). Similarly, the IoU performance of BRATS baselines should be higher https://arxiv.org/pdf/1811.02629.pdf\n\nPotentially I misunderstand what kind of comparisons the authors provide here. Can the authors please explain the choice of their baselines and their experimental settings? \n\n2) The clarity of the writing in some sections should be improved, e.g., in the experimentation.\n\n3) The reproducibility of the results and methods is a concern. I have not seen the code. Furthermore, the sheer amount of computing required  makes reproducibility challenging.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your positive and constructive feedback!\n\n**Question 1: Unclear experiment settings**\n\n**1.a: How to choose baselines?**\n\nWe employ four primary baseline types (e.g., in Table 2):\n\n1.  **2D Supervised Method**: Comparison with standard medical architectures initialized from ImageNet, while our model uses LVM-Med weights.\n2.  **2D-SSL on Medical Data**: Benchmarking our self-supervised algorithm against state-of-the-art SSL methods trained on the same collected dataset.\n3. **Foundation Model**: Evaluating the performance of large vision/vision-language models (e.g., Clip, Flava, SAM) compared to LVM-Med's domain-specific approach.\n4. **Prompt-based Segmentation**:  User-interactive prompt for segmentation masks, highlighting LVM-Med's practicality besides end-to-end training as in (1), (2), and (3).\n\n**1.b: Drive and BRATS segmentation performance lower than literature**\n\nThank you for pointing out these interesting points. After examining the papers mentioned by the Reviewer, we found out that the settings in our work are different, e.g., we use less training data (Drive) or just a single input rather than multiple inputs (BraTS). \n\n- **Drive Segmentation**: Our U-Net baseline with ResNet-50 (LVM-Med) achieved a 65 Dice score, differing from the literature (81 Dice) due to distinct settings. In particular, authors in [2]  use overlapping image patches with a stride of 32 and an image size of 128 × 128 for training. Therefore **they ended up with 4200 images for training (we used 20 images as in the original training set)**. During testing, they applied overlapping image patches again with a stride of 3 and averaged predictions over 20 sub-patches to predict for each image in the test set. \n\n   To provide further  insights, we conduct additional experiments on the Drive dataset using U-net load LVM-Med ResNet-50. Our performance given the same settings as [2] is **84.2 Dice score on average, which works better than the baseline in [2]**.\n\n- **BRATS-2018 Segmentation**:  Our 3D-IoU score is 73 for the whole tumor, compared to 88-90 Dice score by the best method [3]. There are two main different settings: (i) First, **we only use a Flair format for each patient, while [3] combined four available 3D MRI modalities of each patient into the 4-channel image as an input**. (ii) Second, we measure performance on a test set with 95 samples randomly selected from a training set of 285 patients while [3] reported performance on a test set of the BraTS-2018 competition, which is unavailable now because the competition has been closed.\n\n\nIn conclusion, we utilize simple configurations for all datasets, skipping extra pre-processing for data augmentation (Drive) or input fusion (BraTS). We believe these default settings better showcase the benefits of using pre-trained models, especially with limited labeled data. Nevertheless, further experiment details will be included in the Appendix to avoid confusing readers. We appreciate the Reviewer's insightful feedback.\n\n\n[2]  RV-GAN: Segmenting Retinal Vascular Structure in Fundus Photographs using a Novel Multi-scale Generative Adversarial Network, MICCAI-2021\n\n[3] 3D MRI brain tumor segmentation using autoencoder regularization, 4th International Workshop, BrainLes 2018, MICAI.\n\n**Question 2: Reproducibility and code availability**\n\nWe are committed to open-sourcing our code and pre-trained models. **We have sent the code to the Area Chair. We refer the reviewer to the Area Chair for obtaining the code**. In the code, we present detailed instructions in README.md and release ResNet-50 and ViT models trained by LVM-Med, along with the configurations of segmentation, classification, and object detection tasks.\n\n\n**Question 3: Computing details**\n\nAs mentioned in **Section 4.1 Implementation details**, we trained ResNet-50 and ViT-B/16 on our dataset using high-powered GPU systems with 16 A100-GPUs, each with 80GB memory. The ResNet-50 took five days (batch size: 3200 images), and ViT-B took seven days (batch size: 2800 images) for 100 epochs. \n\nWhile the training procedures require a high-powered GPU system, our open-source weights will significantly reduce the time and financial investments necessary for other studies seeking to apply our results in medical applications.\n\n\n**Question 4: Reparameterization trick and backpropagation**\n\nWe utilize IMLE for reparameterizing discrete distributions, outlined in Algorithm 1. To assess the efficacy of this backpropagation approach, we conducted a comparative analysis. Specifically, we compared our method (referred to as LVM-Med (Full) in Table 6) to an alternative technique employing constant value-based perturbation [50] (referred to as LVM-Med w/o Gumbel noise, Table 6). Through comprehensive evaluation across classification and segmentation tasks, we have consistently achieved superior performance than the alternative approaches [50]. For example, segmentation results with IMLE are 83.05, declining to 81.37 using [50].\n\n[50]  Optimizing rank-based metrics with blackbox differentiation, CVPR 2020.\n\n**Question 5: Baselines for contrastive loss in the presented setting**\n\nWe thoroughly compare our approach with alternative contrastive losses (Twin-Barlon, Dino, SimCLR, Moco-v2, VicRegl) in Tables 2, 3, 4, and more (2D-SSL on medical data), which were trained using the same dataset as LVM-Med. Moreover, we demonstrated in Figure 1 (on the right) how the algorithms of LVM-Med can serve as a unified and extending framework for other contrastive SSL algorithms. \n\nWe hope our response resolved most of your concerns, and helped you evaluate our work more positively. If you have other comments, we are happy to address them in the reviewer-author discussion period.\n\n\n\n\n"}, {"review_id": "nEEazmxZvh", "reviewer": "Reviewer_XBHj", "summary": "This paper collects a large medical imaging dataset, and it also shows that a self-supervised learning technique based on second-order graph-matching enhances performance in various downstream medical imaging tasks compared to other supervised learning methods and foundation models trained on image-text instances. The evaluation also considers two different architectures: ResNet-50 and ViT backbones.", "questions": "None.", "limitations": "These are included in the paper.", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "- Creating such a large medical imaging dataset is a commendable feat, and is needed by the community.\n- The proposed self-supervised task is interesting.\n- The evaluation results are comprehensive and thorough.", "weaknesses": "None to report.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "\nThank you very much for your strongly positive feedback! \n\nThis encourages us to continue to improve and extend our research.\n"}, {"review_id": "oU7UeI3qVL", "reviewer": "Reviewer_xKHb", "summary": "This paper proposes a self-supervised pre-training strategy for medical imaging using a graph matching approach. Each unlabeled image is transformed via a pair of data augmentations and then processed via an encoder network. The augmented pair of images become vertices in a pair of graphs, with vertex features being the encoder outputs and edge connections selected via k-nn. A graph convolutional network is trained for vertex-to-vertex matching of the extracted pair of graphs. The training objective incorporates global and local similarity learning over spatial features along with a second-order edge similarity cost. Due to the combinatorial nature of the objective, gradients for backpropagation are approximated via Implicit MLE. Pre-training is performed over a large scale dataset comprising 55 publicly available datasets and used for multiple downstream fine-tuning tasks including segmentation, detection and classification.", "questions": "Can the authors clarify how they use pre-trained ResNet-50 for downstream segmentation via U-Net? Does this mean that the encoder of U-Net has the same architecture as ResNet-50? This is not clear as the vanilla U-Net architecture (Cicek et al. 2015) is different. ", "limitations": "Limitations are discussed. \n", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "The proposed graph matching technique for self-supervised learning is a novel and significant contribution. Abundant experiments demonstrate the generalizability over downstream tasks, with error bars also included. ", "weaknesses": "Some related works on graph matching in computer vision are missing in Section 2.3. \n\nDoi et al. Detecting Object-Level Scene Changes in Images with Viewpoint Differences Using Graph Matching 2022\n\nBian et al. Unsupervised Domain Adaptation for Point Cloud Semantic Segmentation via Graph Matching 2022\n\nWu et al. Unsupervised Visible-Infrared Person Re-Identification via Progressive Graph Matching and Alternate Learning 2023\n\nLiu et al. Self-supervised Learning of Visual Graph Matching 2022\n\nPeng et al. GATE: Graph CCA for Temporal SElf-supervised Learning for Label-efficient fMRI Analysis 2022\n\nThe authors should discuss and contrast with the graph matching objectives and applications in these works to highlight their novelty in self-supervised learning. \n\n\nIntroduction can also be improved to better highlight the contributions. Rather than starting with discussing image-text datasets, the story should highlight the value of self-supervised learning in medical imaging and related works > proposed self-supervised learning method via graph matching > large-scale dataset collected for implementing this method > experiments on downstream tasks. \n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your strongly positive feedback!\n\n\n\n**Question 1: Missing related works on graph matching in computer vision Section 2 & Improving further introduction part to highlight contributions.**\n\n​​We sincerely acknowledge your constructive feedback. Your suggestions are valuable to us, and we will integrate these points properly to enhance both the introduction and the section on related works. These improvements certainly will make the paper more appealing, thereby highlighting our novelties. \n\n\n&nbsp;\n&nbsp;\n&nbsp;\n\n**Question 2: Clarifying how to use pre-trained ResNet-50 for downstream segmentation via U-Net.**\n\nIt means that we replace the encoder network of U-Net architecture (Cicek et al. 2015) with the ResNet-50 architecture (feature outputs after five blocks of ResNet layers). The decoder layer of U-Net is then constructed so that the output at each up-sampling layer has the same feature maps as the original decoders of U-Net. \n\nIn practice, using ResNet-50 or other architectures like VGG or Efficient-Net in UNet is a preferred way and can be applied in several applications such as skin lesions [1], lung segmentation [2], etc. Therefore, we want to show the benefits of LVM-Med for those study cases.\n\n[1] Nguyen, Duy MH, et al. \"TATL: Task agnostic transfer learning for skin attributes detection.\" Medical Image Analysis, 2022 \\\n[2] Cheng, Dorothy, et al. “Transfer Learning U-Net Deep Learning for Lung Ultrasound Segmentation,” Arxiv 2021\n\n\n\n\n"}, {"review_id": "hQ0xTCvLd1", "reviewer": "Reviewer_yfed", "summary": "This research paper focuses on LVM-Med, a self-supervised learning (SSL) technique designed for medical imaging tasks. Based on a second-order graph matching strategy, LVM-Med is trained on a large-scale medical imaging dataset. The researchers found that the method significantly improves performance on a variety of downstream medical imaging tasks compared to other supervised learning methods and foundation models trained on large quantities of image-text data. These findings were consistent across two different architectures: ResNet-50 and Vision Transformer (ViT).", "questions": "Please address the comment in weakness part. \nIn addition, though the ViT architecture has more total parameters, in some cases, it is less effective than LVM-Med ResNet-50. More research and discussion would be needed to fully understand this performance discrepancy.", "limitations": "No limitation on limitations and broader societal impacts.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The LVM-Med method was evaluated on various tasks, including segmentation, object detection, and image classification. The results were compared to foundational models like Clip, Align, Flava, and SAM. It performed particularly well on eight medical segmentation tasks, outperforming both 2D SSL methods trained on the same dataset and foundational models.\n- The training of LVM-Med on a large-scale medical imaging dataset indicates its capacity to handle and learn from large amounts of data, which is often a requirement in the field of medical imaging.\n- The LVM-Med model outperforms both supervised learning methods and foundation models trained on hundreds of millions of image-text instances. This includes a variety of popular models such as Clip, Align, Flava, and SAM.\n- The results show that LVM-Med performs well in in-out-distribution settings, implying a certain level of robustness and generalizability.\n- LVM-Med provides benefits in both end-to-end and prompt-based segmentation tasks. This flexibility can be particularly useful in real-world applications, where various segmentation scenarios may be encountered.\n- The authors also conducted an ablation study, experimenting with variations in LVM-Med's configuration to assess the importance of various components in the overall performance. It was concluded that all factors contribute to the final performance, with the second-order graph matching and Gumbel noise being the most significant.", "weaknesses": "- Only 2D backbone is conducted: The LVM-Med model focuses primarily on 2D backbone. Author should provide more discussion about the challenge for applying this framework in 3D backbone.\n- The LVM-Med model is trained on a large-scale dataset, how can author endure the testing dataset is not leaky in training set. Moreover, the use of a single dataset could potentially limit its generalizability. Ensuring robust performance across diverse datasets from different sources is critical.\n- The model's performance benefits seem to be tied to its training on a large-scale medical imaging dataset. If such a dataset is not available, the performance of the model might be significantly diminished.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your positive and constructive feedback!\n\n\n**Question 1: Add more discussion about the challenge of applying this framework in 3D backbone to the paper.**\n\nThank you for the suggestions. Due to the large workload in dataset collection and experiments, we restricted this work to solving 2D backbone-related downstream tasks. For the 3D backbone for video or 3D volume classification, it requires a dedicated study to investigate further how to extend our proposed algorithm and design optimal architectures. We will add more discussion toward directions in the Limitations and Future Work Section, which includes the following points:\n\n- Extend the architecture in LVM-Med to dynamically receive temporal inputs like frames in videos or consecutive slices of 3D volumes rather than treating them independently (1).\n\n- (1) leads to how to modify the LVM-Med’s graph-matching to encompass combinatorial constraints among internal sections inside 3D  volumes and between slices across different inputs. One possible solution is to leverage the deformable attention mechanisms, which only focus on a flexible small set of major slices conditioned on input data [1], resulting in saving computational complexity and permitting handling the multi-scale feature maps.\n\n[1] Zhuofan Xia et al., “Vision Transformer with Deformable Attention,” CVPR 2022.\n\n&nbsp;\n&nbsp;\n&nbsp;\n\n**Question 2: How can avoid the testing dataset not leaky in the training set?**\n\nWe thank the reviewer for the great questions. In particular, to use LVM-Med and avoid potential testing data leaking during the training steps, there are some typical cases:\n- If the dataset used in downstream tasks does not belong to our 55 collected ones, users can freely download and apply our models.\n- If the dataset belongs to the collection, the user needs to check whether it has default training or testing. If some splitting is available, we follow them and only use the training samples; otherwise, the user needs to exclude the index of images (we sampled 20% of total images for training) trained by LVM-Med. We will release exact information for the later cases in our code repository.\n\n&nbsp;\n&nbsp;\n&nbsp;\n\n**Question 3: The use of a single dataset could potentially limit its generalization. Ensuring robust performance across diverse datasets from different sources is critical**\n\nIn our large-scale medical datasets, collected images cover diverse body organs and data molarities, as demonstrated in Figure 1 (main paper) and Table 7 (Appendix). It is not from a single large dataset. While some datasets have a large number of samples, we address it by applying some balancing strategies in each mini-batch  during the training procedure to avoid a potential bias toward a specific domain.\n\nIn downstream experiments, we selected various types of data modalities, encompassing MRI, CT scans, X-rays, ultrasound images, and color images. Our settings also span eleven distinct organ structures (including tumors, heart, skin, brain, etc). The majority of these experiments have revealed favorable outcomes with LVM-Med compared to alternative reference models. These consistent records, therefore, highlight the strength and adaptability of LVM-Med across different areas of expertise.\n\n&nbsp;\n&nbsp;\n&nbsp;\n\n**Question 4: The model's performance benefits seem to be tied to its training on a large-scale medical imaging dataset. If such a dataset is not available, the performance of the model might be significantly diminished**\n\nWe agree with the reviewer that the achieved performance has strongly aligned with the scale of the dataset we collected. This aspect indeed stands out as our primary discovery and is one of the most critical LVM-Med contributions alongside the novel self-supervised learning approach based on graph matching. We believe that our research will catalyze future investigations into the creation of expansive medical datasets and a deeper exploration of practical applications in real-world medical scenarios, thereby pushing the boundaries of utilizing machine learning within medicine.\n\n\n&nbsp;\n&nbsp;\n&nbsp;\n\n**Question 5: ViT architecture, in some cases, is less effective than LVM-Med ResNet-50. More research and discussion would be needed to understand this performance discrepancy fully.**\n\nIn the **Limitations and Future Work Section**, we discussed this concern, i.e., ViT architecture performance with end-to-end learning. To gain a more comprehensive understanding of this phenomenon and effectively tackle it, we propose the need for additional experimentation. For instance, optimizing hyperparameters like projector heads and token feature dimensions through grid searches during pre-training is important. Another potential solution is integrating the trained architecture and fine-tuning it efficiently for downstream tasks. While our study explores the latter approach, we intend to delve into this aspect and update our findings in publicly available code repositories. For the pre-training hypothesis, we suggest it as a future work for investigation, given the huge amounts of required computations.\n\n\n\n\n\n\n\n\n"}]

Author rebuttal:


We would like to thank all reviewers for the positive and constructive feedback, which we will leverage to improve this work.

We are very encouraged that most reviewers agree that our efforts in **creating such a large medical imaging dataset is important and needed by the community**. Reviewers jRh7, xKHb, and XBHj also appreciate the **LVM-Med algorithms and believe they are novel and interesting**. In addition, Reviewers WEtP, XBHj, xKHb, and yfed acknowledge the **comprehensive and thorough experiments**, which show that LVM-Med consistently **outperforms several state-of-the-art self-supervised algorithms and foundation models**. 

There are a few shared concerns among Reviewers WEtP and jRh7, such as missing details for some of the experiments and improving the introduction by discussing additional relevant work (Reviewer yfed). 

We appreciate your feedback and will take into account these points to improve the manuscript. For instance, missing information will be added to **Section 4.1 Implementation Details - Downstream Tasks** (main paper) and the corresponding sections in the **Appendix**. A part of the Introduction and Related Work sections will be revised based on the references provided by Reviewer yfed.

Below, we address specific concerns raised by individual reviewers in detail. 






Meta-review: This paper collects a large data set of publicly available medical images and conducts a comprehensive study based on graph matching techniques. Additional fine-tuning for specific tasks demonstrated significant improvements in performance for multiple scenarios. The first main contribution of the paper is the novel application of graph-matching methods. The second main contribution is an extensive set of experiments that demonstrate potentially wide applicability of the method. 

The reviewers praised the abundance of experiments, the thorough approach and the novelty of the graph-matching methods for contrastive learning. The reviewer agreed that the results of this paper will benefit the community. During the discussion many remaining concerns of the reviewers  were addressed. This paper will advance our understanding of machine learning methods for medical imaging. Thus, we recommend acceptance.
Decision: Accept (poster)