Curriculum Fine-tuning of Vision Foundation Model for Medical Image Classification Under Label Noise
Authors: Yeonguk Yu
Minhwan Ko
Sungho Shin
Kangmin Kim
Kyoobin Lee
Venue: NeurIPS 2024 poster (2024)
Keywords: Learning with noisy label
medical image classification
Primary area: machine_learning_for_healthcare
TL;DR: We propose CUFIT, a robust fine-tuning method for vision foundation models under noisy label conditions, based on the advantages of linear probing and adapters.

Abstract: Deep neural networks have demonstrated remarkable performance in various vision tasks, but their success heavily depends on the quality of the training data. Noisy labels are a critical issue in medical datasets and can significantly degrade model performance. Previous clean sample selection methods have not utilized the well pre-trained features of vision foundation models (VFMs) and assumed that training begins from scratch. In this paper, we propose CUFIT, a curriculum fine-tuning paradigm of VFMs for medical image classification under label noise. Our method is motivated by the fact that linear probing of VFMs is relatively unaffected by noisy samples, as it does not update the feature extractor of the VFM, thus robustly classifying the training samples. Subsequently, curriculum fine-tuning of two adapters is conducted, starting with clean sample selection from the linear probing phase. Our experimental results demonstrate that CUFIT outperforms previous methods across various medical image benchmarks. Specifically, our method surpasses previous baselines by 5.0\%, 2.1\%, 4.6\%, and 5.8\% at a 40\% noise rate on the HAM10000, APTOS-2019, BloodMnist, and OrgancMnist datasets, respectively. Furthermore, we provide extensive analyses to demonstrate the impact of our method on noisy label detection. For instance, our method shows higher label precision and recall compared to previous approaches. Our work highlights the potential of leveraging VFMs in medical image classification under challenging conditions of noisy labels.

Curriculum Fine-tuning of Vision Foundation Model for Medical Image Classification Under Label Noise
Yeonguk Yu Minhwan Ko
Sungho Shin Kangmin Kim Kyoobin Lee"
Gwangju Institute of Science and Technology (yeon_guk, mhko1998, hogil189, kgmin156)0gm wwt.iiit........................ ac.kr
kyoobinleedgist.ae
Abstract
Deep neural networks have demonstrated remarkable performance in various vision tasks, but their success heavily depends on the quality of the training data. Noisy labels critical issue in medical datasets and significantly degrade model performance. Previous clean sample selection methods have not utilized the well pre-trained features of vision foundation models (VFMs) and assumed that training begins from scratch. In this paper, we propose CUFIT, curriculum fine-tuning paradigm of VFMs for medical image classification under label noise. Our method is motivated by the fact that linear probing of VFMs is relatively unaffected by noisy samples, does not update the feature extractor the VFM. thus robustly classifying the training samples. Subsequently, curriculum fine-tuning of two adapters conducted, starting with clean sample selection from the linear probing phase. Our experimental results demonstrate that CUFIT outperforms previous methods across various medical image benchmarks. Specifically, our method surpasses previous baselines by 5.0%, 2.1%, 4.6%, and 5.8% at 40% noise rate on the HA.M10000, APTOS-2019 BloodMnist, and OrgancMnist datasets, respectively. Furthermore, we provide extensive analyses to demonstrate the impact our method on noisy label detection. For instance, our method shows higher label precision and recall compared to previous approaches. Our work highlights the potential of leveraging VFMs in medical image classification under challenging conditions of noisy labels.
Introduction
Deep neural networks have demonstrated remarkable performance across various tasks, including classification, detection, and segmentation 20
19 57]. medical imaging, these neural networks leverage large amounts of labeled data to train models capable of accurately detecting classifying medical conditions from images such as dermatoscopes, X-rays, MRIs, and CT seans. However, in practical settings, data often contain noisy labels and it is well established that neural networks perform well only when the quality of training data is sufficiently high 42 55. Noisy labels occur when the data annotations-thh labels assigned to training images are incorrect inconsistent. This issue is particularly problematic medical imaging, where annotating images is more complex compared natural images [53 25. Consequently, improving the robustness of neural networks against noisy labels is a crucial area of research, directly affecting the effectiveness and reliability of medical imaging technologies.
large number of algorithms have been developed to address the issue of performance degradation caused by noisy samples [42). In particular, clean sample selection methods, such as MentorNet (24. Co-teaching (17), Co-teaching+ [56). JoCor 47). and CoDis [50), have demonstrated superior performance without requiring modifications to the model architecture or training loss. The core
"Corresponding author: Kyoobin Lec. Our code is available at github.cco/ggiitaalaaCCCCC
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

train params: 2.7K
# train params: 1.1M
Class Token
Class Token
90.0
VFM Layer
VFM Layer
70.0
60.0
Linar probing
VFM Layer
VFM Layer
40.0
Adapter (Rein)
CUFIT (Ours)
0.0
0.1
0.2
0.4 0.6
Noise rale of training data
Linear probing
(b) Using adapters
(c) Performance comparison
Figure 1: Illustration linear probing (a) and adapter usage (b). Specifically, the weights of the foundation model are frozen, while the fully connected layer adapter weights (shown orange) are updated during the training phase. In (c), a performance comparison using a simulated noisy dataset (HAM10000) is presented. It demonstrates that linear probing is more robust to noisy labels compared to the adapter, whereas the adapter outperforms linear probing when there are no noisy labels.
principle behind these methods is that small-loss samples are likely to clean, as they are easier to classify and the model memorizes them faster. Additionally, using two different but homogeneous networks to select small-loss samples for each other more stable than relying on a single model for sample selection. These methods have shown outstanding performance using traditional neural network architectures based on convolutional layers. However, there are practical issues with these methods in aspects: (i) it cannot be guaranteed that these methods will perform equally well with transformer based architectures, which have recently gained significant attention, and (ii) their assumption that training starts from scratch is impractical, as prevents the use of rich features from pre-trained models, which could be beneficial for filtering noisy labels.
Recently, large-scale vision foundation models (VFMs) with transformer-bant based architectures [K] such as CLIP (39. MAE [8], SAM [28), and DINOv2 [7), have gained attention for their perfor- mance and applicability across various tasks. The self- supervisdd training of VFMs large-scale datasets enhances their robustness against various image corruptions and improves their generalization capabilities 10
The inherent robustness and rich features of VFMs can be beneficial for detecting noisy labels. For instance, linear probing of VFMs is relatively unaffected by noisy samples since it does not modify the VFM's feature extractor, preventing the memorization of noisy data, as shown in Figure I However, linear probing does not fully leverage the VFM's capabilities when there is domain gap between the pretraining task of the VFM and the target task (e.g., pretraining on natural images versus medical image classification) 488. To address this issue, some researchers Yet. these adapters might degrade performance by memorizing noisy labels due to their trainable parameters involved in feature extraction. Therefore, we state our research question as follows: How can we use the power of pre-trained vision foundation models for medical image classification in the presence of noisy labels?
48
38
40).
have proposed using trainable fine-tuning adapters for VFMs 22 23 48
In this paper, introduce Curriculum Fine-Tuning paradigm for vision foundation models in medical image classification under noisy labels, called CUFIT. CUFIT is curriculum-learning framework designed fine-tune VFMs with noisy medical datascts. The framework consists of three training modules: the Linear Probing Module (LPM), the Intermediate Adapter Module (IAM), and the Last Adapter Module (LAM). During the training stage, clean samples are selected based on an agreement criterion, where sample is selected if its annotation matches the module's 's prediction. Specifically, the LPM is trained using all available samples, as linear probing robust against noisy labels. Subsequently, the IAM is trained with the samples selected by the LPM, and the LAM trained with the samples selected by the IAM. This inter-module curriculum training (i.e., APM-IAAM+LAMM is beneficial for increasing the number of clean samples available to train the LAM, considering that the I.PM only selects limited number samples due to performance degradation caused by the domain gap. Consequently, CUFIT leverages the LAM for final predictions,

---

offering strong fine-tuning performance for medical image classification in the presence of noisy labels by utilizing the strengths of both linear probing and adapters, as illustrated in Figure Tcc In summary, the main contributions of this paper are as follows:
We introduce CUFIT, a simple yet effective fine-tuning paradigm for medical image classification using VFMs in the presence of noisy labels. This method leverages the robustness of linear probing and the generalization capability of fine-tuning adapters to handle noisy datasets during training stage.
We conduct various experiments demonstrating that CUFIT significantly improves the robustness of VFMs against noisy labels in medical datasets. We show that CUFIT outperforms previous methods across several medical image benchmarks.
We provide extensive analyses to enhance the understanding of our fine-tuning paradigm. Addi- tionally, we validate our framework with various VFMs and adapter configurations.
2
Related Work
Vision foundation models. Vision transformers (ViTs) embed 2D images into ID tokens and model their global correlations using the self-attention mechanism [T3 35 43 ViTs are known to be effective when large datasets are used, and the concept of vision foundation models introduced. Several studies have developed pre-trained vision transformers based on self-supervised learning. For instance, contrastive language-image pre-training (CLIP) provides high-quality visual representations through contrastive learning with a large amount of image-text pairs [39 Additionally, masked auto- encoder (MAE) offers high-capacity models that generalize well through self-supervised learning with masked auto-encoding, where the task is to reconstruct token patches from the given masked tokens [18). Moreover, knowledge distillation with no labels (DINOvI) 77] proposed a teacher- student framework for self- self-training without annotations, resulting in a well cell-generalized ViT. More recently, DINOv2 introduced a self-training framework that combines masked autoencoding and seacher-student training based carefully curated datasets 37..
Since large models and self- in VFMs provide strong generalization capabilities for various tasks, parameter-effiiien fine-tuning has gained attention. ehrameter-efficient fine-tuning (PEFT) aims to adapt foundation models to new tasks by training only few adapter parameters while keeping the model itself frozen. Notably, there is research that proposes low-rank adaptation (LoRA), which introduces trainable rank decomposition matrices into each layer of the transformer architecture in large language models [22]. For the VFMs, visual prompt tuning (23 proposed appending prompts to the input sequence of each transformer block, achieving excellent fine-tuning performance with minimal parameters. Similarly, Adaptformer [9] introduces novel MLP block to replace the original one in transformer blocks, allowing for the use of both original and few trainable parameters. More recently, Wei et al. proposed Rein, which aims adapt VFMs for semantic segmentation with domain generalization capabilities [48]. Also, Dutt er al. investigate PEFT algorithms across both convolutional and large transformer-based networks for medical image classification, demonstrating the effectiveness of PEFT, particularly in the low-data regimes common in medical imaging (14). In this paper, we focus on fine-tuning VFMs for image classification in the presence of noisy labels using adapters. Rather than introducing a new adapter, we utilize an existing adapter within our training paradigm.
Learning with noisy label. Deep neural networks have demonstrated remarkable performance on large-scale datasets. However, it is well-known that neural networks can easily memorize noisy labeled samples, leading to degraded performance. Several studies have been conducted to explore robust supervised learning in the presence of noisy labels. These studies can be categorized into five approaches 42]: (i) robust architectures, (ii) robust regularization, (iii) robust loss functions [15] 33
[9]. (iv) loss adjustment [26 31 32), and (v) sample selection [24 36 17 [6 50). In this paper, we categorize our method as a sample selection method, which selects samples with clean labels from a noisy training dataset. While previous sample selection methods typically consider training from scratch, we focus on training starting from pre-trained model, which is known to be more robust to noisy labels (2T. Additionally, research has explored using CLIP enable robust training by leveraging its text-image matching capability on noisy datasets [49].
3

---

Various methods have been proposed for clean sample selection from noisy datasets. For example, MentorNet introduced the use of a teacher network to guide the student network focus on clean labels (24). Similarly, Decoupling proposed updating two networks by using only the samples with differing predictions between them [6]] Co-teaching also trains two networks simultaneously, updating them based on sample recommendations from each other [17]. Co-teaching+ [56) improved upon Co-teaching by introducing the "update by disagreement" strategy, where only the samples with differing predictions between the two networks are used. More recently, Xia et al. proposed CoDis, an extension of Co-teaching+ that employs an "update by discrepancy" strategy, selecting samples with sigh-iiccrepancy prediction probabilities between the two networks to utilize more samples [50]. These methods are based on the assumption that clean samples can be identified using certain criteria, and that network collaboration is more stable than self-selection which may lead to eroor accumulation. In this paper, we develop our method based on same assumption, but we assume that we start the learning process from the pre-trained VFM.
3 Problem Setup
We consider a k-class classification task using a neural network. Let Rd denote the input space and y E IR* represent the ground-truth -ruth label space. In a typical classification task, the neural network trained to align the input space with the label space. To this end, a training dataset D = {(14 }= is used for supervised learning with cross-entropy loss. In practice, sample (x,, 0:) considered as noisy labeled sample when human-annotated label üs does not match the true label yt. The objective of this paper is to develop a fine-tuning approach for VFMs that is robust to noise and capable of performing accurately on noisy datasets.
Given a pre-trained VFM with parameters Ovrm consisting of a sequence of layers (e.g., attention blocks in ViT (13) L1, L2,... .... Las. where M is the depth of BVrM, the learning objective for a classification problem can be formulated as:
12 Lee (pla, 01).
min 6x
Σ
where O, and foe represent the parameters targeted for updating and the cross-entropy loss, re- spectively. Here, p(- refers to the prediction for a given input using parameters O. We refer to the training process as linear probing when 0g is limited to the parameters of the linear layer O,. Additionally, we refer to the training process as full-tuning when 0, includes 0VEM, and as adapter tuning when it includes adapter parameters Og. which are not part of OVFM.
4 Method
In this section, begin by describing the adapter method for fine-tuning the VFM in Section 4.1 Following this, in Section 4.2 we introduce our method, CUFIT, which utilizes three modules: a linear layer and two adapters, well-pre-trained features of the VFM without updating the feature extractor when handling corrupted samples. Subsequently, the adapters are trained using the samples selected in a curriculum-basee training manner, as shown in Figure[[(i.e (i.e., linear probing - intermediate adapter -> last adapter). This approach helps increase the number of selected samples by reducing the domain gap between the pretraining task and the medical image task. It is important to note that our framework does not train the modules sequentially (i.e., where one module starts training only after another finishes); instead, it trains the modules simultaneously on the current batch, similar to multi-task training.
combat noisy labels. The key idea behind CUFIT is to leverage the
4.1 Learning with Adapter
We consider various adapters for fine-tuning VFMs on medical image datasets.In In particular, adapters like Visual Prompt Tuning (VPT (3)). AdaptFormer (9, Low Rank Adaptation (LoRA (22)) and Rein [48] can be used. These methods have been shown to be efficient for various image and video tasks, even compared to full model training 9 48). Typically, when an adapter is used for fine- fine-tuning, the parameters of the VFM are frozen and not included in the optimization process.
In this section, we briefly introduce how an adapter works. Note that our goal is not to propose a novel adapter but rather to present training paradigm that can be applied to various adapters. For

---

Label actinic kerntosis
Label 2: melanoma
Flow of linear probing module
Flow of adupler module
Label 3: vascular
(0) LP
Block
Block
Block
Linear
M
Label
(b) IAM
MLP
MLP
Linear
Training batch
(9) LAM
MMP
MLP
Linear
CLS
Token
Head
Curriculum Loss Calculation
Figure 2: Illustration of our proposed training framework, CUFIT, which consists of pre-trained VFM and three distinct modules: (a) the linear probing module (LPM), (b) the intermediate adapter module (IAM), and (c) the last adapter module (LAM). During the training stage, the LPM selects clean samples for the IAM based on the agreement criterion, and the IAM selects clean samples for the LPM. During the inference stage, only the LAM used for prediction.
vision transformers (ViTs), the output of the attention block for the given input patches is calculated as follows:
QKT
Attention (Q, K, V) = Softmax
V) I1-1;
(2)
vd
where It-1 is the output token of the previous block.Here, Q. K, and V refer to the query, key, and value vectors, respectively, which are derived from linear projection and LayerNorm (6] applied to
Jt-1- The final output of the block, It, then computed using LayerNorm and an MLP. Without using an adapter, this process is formulated as:
31 MLP(LN((1)) + +3/,
where 2j is the output token of the 1-th block. When an adapter is used, the Eq replaced by the following:
3y = MLP(LN(</)) + + + Adapt(x
(4)
where Adapt(:: of) refers the adapter function for the I-th layer, parameterized by 0f. We consider this process to be an arbitrary function, various adapters can be used.In the last block, the (CLS) token passed to the following linear layer for final image classification.
4.2 Curriculum training of three different modules
We consider pre-trained VFM, @VEM. with a single linear layer parameterized by 6LPM E Rexk an intermediate adapter module parameterized by 9 and last adapter module parameterized by fLAM. where refers to the dimension of the class token (e.g., 384 dimensions for the ViT-small architecture). Then, we propose a curriculum training framework for these three modules, which the LPM is trained with all samples from the given batch, while the adapter modules are trained with filtered samples selected by their corresponding module using the agreement criterion. The agreement criterion refers to method where sample is considered clean if the module's 's prediction matches sample's annotation. The idea behind this criterion is that a robust classifier will correctly predict the sample under the assumption that clean labels are in the majority within noisy class. Therefore, a sample is selected as clean if meets the agreement criterion (c.g., .g. "dog" image with a "dog" annotation). Thus, we build the curriculum training framework based the robustness of the LPM against noisy labels using the agreement criterion.
In particular, the linear probing module (LPM) is trained as follows:
min
Lae (plx @VVMM OLPM)) Uc),
(5)
5

---

which directly represents supervised learning using the given images and corresponding labels. Here, p(a,|6vrM 8 PM) refers the output of the network using θyFи and 8\PM for the given image z;. During the training stage, the intermediate adapter module (IAM) is trained as follows:
min 9IAM
Σ Loa (p(x:://0 0rAM), 9s)1 (arg max p(1::/0rr 0LPM) ū:):
where 1 [ is the indicator function. This simple modification using the indicator function ensures that the adapter module is trained only on selected samples chosen by the linear layer. Finally, the last adapter module (LAM) is trained as follows:
min PLAM
Cec(p(a LAM)) Ds 1 {arg max p(a; (0vMa, 0AAM) = Us }-
The Eq is equivalent to Eq 6 but it uses LAM and IAM instead of IAM and LPM, respectively. This simple yet effective sample selection strategy is well-suited for fine-tuning the VFM on noisy image datasets. Notably, it does not require any hyperparameters like the estimated noise rate, which are commonly needed in previous works 17 47 [50, where they assume the noise rate known in order to select small-loss samples (e.g., selecting 60% of samples in a batch for known noise rate of 40%). After training is completed, only the last adapter module is used to predict the given test image.
5 Experiments
5.1 Settings
Datasets.
We evaluate approach on four simulated noisy label medical multi-class image classi-
fication benchmarks: HAM10000 [44), APTOS-2019 [I]], BloodMnist [3], and OrgancMnist [52). Additionally, conduct evaluation real-world noisy label benchmark, Kaggle-EyePACS 2 In particular, the detail of datasets are as follows:
This dataset contains 10,015 dermatoscopic images for skin lesion classification, with each image classified into one of seven possible disease categories. We use all the images for training, and the evaluation is conducted using the 1,512 test images provided by the ISIC 2018 challenge I.
HAM10000 4.
APTOS-2019 11 This dataset consists of 3,662 retina images taken with fundus photography under various imaging conditions. Each image is rated for the severity of diabetic retinopathy (DR) on a scale from 0 to 4. We use 2,930 images for training and 366 images for evaluation.
BloodMnist (3]: This dataset contains 17,092 images of individual cells, with each image annotated as one of eight possible cell types. We use 11,959 images for training and 3,421 images for evaluation.
OrgancMnist [52] This dataset includes 23,538 images that are center-sliced from the Hounsfield- Unit of 3D images a coronal view. Each image is labeled as one of eleven body organs. We use 12,975 images for training and 8,216 images for evaluation.
This Kaggle competition dataset provides 35, 126 retina images categorized into five DR severity grades for training. which are known to contain noisy labels (25). Specifically, some DR category labels (e.g., mild DR labeled as moderate DR) are noisy, and some images considered normal may actually contain retinal diseases such glaucoma or drusen, which are not included in the classification categories. It is estimated that there is approximately 30%-40% label error this dataset 25 45). We use the original 35,126 training images and their annotations for training, and all images from APTOS-201 for evaluation. Additionally, we use the FGADR [60] dataset for further evaluation.
Kaggle-EyePACS 2
We compare the performance of CUFIT with basic training paradigms: full training, linear probing, and fine-tuning with Rein 48 Additionally, we evaluate our approach against other training-baan based methods, including Co-teaching [17. JoCor [47]. and CoDis [50]. Like ours, these methods not modify the training loss or architecture. Specifically, Co-teaching trains two networks simultancously, with each network selecting small-loss samples from its peer's 's predictions to guide
Baselines.

---

Dataset
Noise rale
Full-iraning
probing
Co-teaching
Method
CoDis
CUFTT
0.T
66.5
75.h
78.6
81.5
NT.T
81.9
82.6
0.2
75.3
72.1
79.1
80.1
HAM10000
0.4
56.1
71.0
54.9
74.3
73.9
74.1
79.1
0.6
59.9
61.9
37.8
67.3
67.1
66.1
70.1
Mean
61.3
71.0
60.8
75.5
75.4
75.5
78.3
0.1
66.8
79.2
82.5
82.8
84,8
83.2
84.2
79.4
APTOS-2019
0.4
69.9
79.5
77.2
79.5
76.0
79.5
81.6
76.3
Mean
62.7
76.3
68.9
79.
79.5
80.1
81.6
0.1
97.2
95.9
98.6
98.5
98.5
99.0
0.2
93.9
96.7
97.6
97.3
97.2
98.8
BloodMnist
0.4
91.8
95.8
69.3
93.7
93.0
93.5
98.3
0.6
90.3
98.2
Mean
92.3
95.0
75.0
94.7
94.0
94.3
98.6
92.1
93.7
0.2
79.9
82.9
82.0
90.9
91.9
90.7
93.6
OrgancMnist
0.4
72.1
79.9
63.8
85.8
85.3
85.8
91.6
0.6
64.5
72.2
43.1
82.8
82.6
81.9
87.4
Mean
75.5
79.6
69.1
87.9
88.0
87.6
91.6
Table I: Average test accuracy (%) on four simulated noisy datasets with different noise levels. The
test accuracy is averaged over the last ten epochs. The best and second-best results in each case highlighted in bold and underline, respectively.
Method
Testset
Full-training
Linear probing
Rein
Co-teaching
JoCor
CoDis
CURIT
APTOS-2015
65.4
70.9
FGADR
14.3
46.4
48.8
44.9
53.1
$3.0
53.7
Total
27.5
59.0
62.3
62.2
63.9
63.8
64.4
Table 2: Average test accuracy (%) real-world noisy datasets (Kaggle-EyePACC for training). After the training is done, we evaluate the model on two datasets: APTOS-2019 and FGADR, The best result and second-best result in each case are highlighted in bold and underline, respectively.
learning. JoCor extends this idea by incorporating co-regularization maximize agreement between the two networks. CoDis further refines this process by selecting samples that not only have small losses but also show high divergence between the two networks. It is important to note that do not compare our proposed framework with state-of- ofte-oofthhe-rr methods that modify the training loss (e.g., semi-supervised learning) model architecture [51] g. In the experiments, we apply these methods to VFMs with adapters, as they do not require specific model architectures, and VFMs with adapters outperform the linear probing of VFMs (i.e., DINOv2 with the Rein adapter is used the default setting for training with Co-teaching, JoCor, and CoDis for a fair comparison).
13 Implementation details. For the experiments, we use DINOv2 37 with the ViT-small backbone basic vision foundation model. Additionally, adapter, originally proposed for domain-gen genuuiningnerrlizeed semantic segmentation of VFMs. In our setup, utilize the class token from the block for classification, rather than the patch tokens from multiple blocks.
Rein [48] as the fine-tuming
We use the PyTorch 4 codebase for our experiments. BloodMnist and OrgancMnist datasets are sourced from MedMnist [54 [55]. We use the ViT-small architecture and the Adam optimizer [27]. All training runs for 100 epochs with batch size of 32. The initial learning rate is 0.001, which decays by factor
at epochs 50, 75, and 90. For full-parameter training, however, we start with an initial learning rate of 0.0001. For the simulated noisy label benchmarks, generate symmetric noise [17 for evaluation, with noise rates set at 10%, 20%, 40%, and 60%.
5.2 Simulated noisy medical image classification benchmark
First, we evaluate our framework on simulated noisy label benchmarks using four medical datasets.
The average classification test accuracy for each dataset is provided in Table T Our framework consistently outperforms previous baselines, demonstrating leveraging the pre- pre-trained features of DINOv2 and the Rein adapter. Notably, framework proves
effectiveness under noisy labels by

---

Linear probing module
Intermediate acupter module
adapter module
CoDis
0.85
0.95
Lahel precision
0.85
0.90
recall
0.85
0.30
Test aoquraayy
1.80
0.75
0,80
0.50
0.75
0.75
0.70
0.70
0.60
0.50
100
100
Epoch
Epoch
Epoch
1.00
1.00
0.85
(d) Lahel precision (%)
0.95
0.95
0.80
0.90
0.90
( Tost sccurayy
0.85
0.85
0.75
(c) Label
0.30
0.75
0.75
0.70
0.70
0.70
0.68
0.65
0.65
0.60
0.60
100 0.60
50
100
50
50
100
Epoch
Epoch
Epoch
Figure 3: Illustration of label precision (a,d), label recall (b.e), and test accuracy (c.f) epoch. The first row is for HAM10000 with 40% noise rate, and the second row is for APTOS-2019 with 40% noise rate.
to be more effective the noise rate increases. For example, CUFIT achieves 0.85% relateively higher accuracy than CoDis on HAM10000 with 10% noise rate, while the improvement rises to 3.7% 60% noise rate. This result indicates that the pre-trained features of the VFM particularly useful for handling noisy labels in the given datasets.
5.3 Real-world noisy medical image classification benchmark
We train DINOv2 with Rein on the real-world benchmark, using the Kaggle-EyePACCS dataset for training and the APTOS-2019 and FGADR datasets for testing. Given the highly imbalanced training (e.g.. approximately 73% of the samples are labeled the normal class), we use weighted cross-entropy loss to train the model. Since previous sample selection methods require noise rate hyperparameter, we employed the noise estimation method from (34), following Co-teaching [17].
In Table 2, we report the classification accuracy on the APTOS-2019 and FGADR datasets, as well the overall accuracy across both datasets. Our method outperforms other baselines the FGADR dataset and the combined dataset, while Co-teaching achieves the highest accuracy on the APTOS-201 2-10 dataset. We believe this discrepancy is due the distribution of normal class 50% in APTOS-2019 and about 5% in FGADR. Co-teaching performs well in classifying the normal class, whereas method excels classifying diseased samples. For example, our method achieves 53.9% macro-average test accuracy, while Co-teaching achieves 48.5% on the combined test set.
camples-approximatee
Discussion
6.1 How does CUFIT works?
So far, we have demonstrated through empirical results that our framework significantly improves the robustness of VFM fine-tuning against noisy labels. However, we have not yet discussed why our framework effective in learning with noisy labels. In Figure 3 we present label precision, label recall, and test accuracy over the number of epochs to illustrate how our framework functions. In principle, higher label precision indicates fewer noisy samples in the selected data, while higher label recall indicates fewer clean samples in the unselected data. We have following three observations:

---

90
Adapter
Lineer pookigg
CUFIT tours)
X35
124
80
764
30
N
11
T14
722
TEL
603
70
%
74
Lanoe jmbling
VPI
AdipE ornore
LofA
Ioin
Lineer prohing
1.
Rein
Linear prihing
VP1
To
Roin
Linver probisg
Adietti
Rain
proting
VPI
LakA
Rain
Linear probiig
VPT
Adagtt
LoRA
Rain
probung
Laki
Rein
Liserr probug
VPT
Adag FareForrr
LoKA
Rein
DINov1-base
MAB-base
DINov2-base
BioMedCLIP
DINOvI-base
MAE-base
DINOv2-base
BioMesCLIIP
HAM10000
rate
(b) APTOS-2011
rate
Figure 4: Test accuracy of our method with various VFMs (DINOvI 7 MAE [8]. DINOv2 K77
and adapters (VPT [23] AdaptFormer Rein 48
We use HAM10000 and APTOS-2019 with
40% noise rate for training.
Dutaset
Noise Rate
Full-training
Lineur probing
CoDis
CUFIT
ResNet
DINOv2
ResNet
DINOv2
ResNet
DINOv2
ResNet
ResNet+rein
DINOv2
0.2
73.1
66.5
71.1
75.6
74.9
80.1
77.7
79.9
82,6
HAM10000
62.6
72.4
0.2
80.4
66.8
80,3
79.2
80.3
82.0
82.4
82.2
84.2
APTOS-201 2019
0.4
64.7
65.9
73.4
79.4
78.1
79.5
80.5
82,2
84,2
Table 3: Average test accuracy on simulated noisy datasets (HAM10000 and APTOS-2019) using the
ResNet50 architecture. Test accuracy is averaged the last ten epochs.
We find that CoDis exhibits lower label precision during training compared to L.M and IAM, suggesting that previous sample selection methods fail to effectively utilize the pre-trained features of VFMs, leading to lower test accuracy. These methods train the network on all training data without sample selection during the early stages training (e.g., epochs to 10 in their default setting). However, this approach may harm the feature extraction capability of VFMs and result degraded performance.
LPM consistently achieves the highest label precision across epochs but has the lowest label recall, indicating that it effectively prevents the memorization of noisy samples. However, because the feature extractor remains unchanged, its overall accuracy is limited, thus selecting only a small number of clean samples.
IAM, the other hand, achieves similar label precision but higher label recall by leveraging the adapter module, which contributes to the improved test accuracy of LAM. This suggests that by adapting the feature extractor through training the adapter few certain clean samples. LAM can be more accurate module. can then provide more clean samples to LAM, resulting in better overall performance.
6.2 Performance comparison across various VFMs and adapters
To validate the performance of CUFIT in various settings, we present experimental results using three VFMs and adapters in Figure 4 We use the same experimental setup (i.e.. 100 epochs with the Adam optimizer) to train the network across all backbones and adapters. Specifically, we utilize four backbones, including DINOv1 77. MAE [8]. DINOv2 (37, and BioMedCLIP (58) and four adapters, including VPT [23]. AdaptFormer 9. LoRA 22). and Rein 48) BioMedClip is CLIP- like model trained with the PMC-15M dataset, which contains 15 million biomedical image-text pairs collected from 4.4 million scientific articles. Our results demonstrate that our framework consistently helps build a robust classifier across different VFMs and adapters. For example, our framework achieves better performance compared to both adapter-based methods and linear probing. Additionally, we observe that linear probing consistently outperforms the adapter method in all cases, indicating that the performance of adapters can be degraded by noisy labels across various adapters.

---

Method
Datasct
Noise rate
Full-trining
Linear probing
Rein
Co-teaching
JoCor
CoDis
CUFIT
CIFARIO
25.9
79.0
24.8
78.2
75.7
76.3
83.9
CI
6.3
25.6
66.7
64.2
73,8
ANIMALION
0.08
74.5
89.1
88.0
92.2
91.9
91.7
92.3
Table 4: Average test accuracy on the natural image dataset with simulated noisy labels (CIFAR, symmetric noise at 80%) and real-world noisy labels (ANIMALION [41,, which has estimated
noise ratio of 8%). The test accuracy is averaged over the last ten epochs. We use DINOv2 with Rein adapter for the experiment. Bold values the best result.
6.3 Performance on CNNs with adapters
We designed CUFIT for VFMs due their strong pre-trained feature extraction capabilities, enabled by self-supervised training on large datasets. However, CNN-based architectures like ResNet [20] also utilize pre-trained weights instead of starting from scratch. Therefore, we validate CUFIT on ImageNet [12] pre-trained ResNet50, with and without the Rein adapter modified for ResNet. The experimental results are shown in Table 3 We observe that our method outperforms other training paradigms when using the ImageNet pre-trained ResNet architecture. Additionally, the Rein adapter for ResNet improves performance, demonstrating that using fewer trainable parameters with an adapter, compared to full training, is beneficial for combating noisy labels. Finally, we show that the more representative pre-trained features of DINOv2 outperform the ImageNet pre-trained features across all training methods.
6.4 Performance on noisy natural image classification benchmark
Since our framework is easily applicable not only to medical image classification but also to natural image classification, we present experimental results on the CIFAR (30] simulated noisy classification benchmarks and ANIMALION [1] real-world noisy classification benchmark Table 4 We validate framework under an extremely high noise rate setting (80%) for CIFAR benchmark, as is intuitive that our framework performs well under low noise rates due to the feature extraction capabilities of VFM. As shown in Table E our framework outperforms other sample selection methods in natural image classification benchmarks as well. This demonstrates the effectiveness of framework, highlighting that using well pre-trained VFMs is beneficial for detecting noisy labels in natural images, as expected.
7 Conclusion
This paper presents curriculum fine-tuning paradigm called CUFIT, designed to robustly fine- tune vision foundation models (VFMs) for medical image classification. Our framework is based on the insight that linear probing of VFMs is robust to noisy labels, as it does not modify the feature extraction process. Building on this, CUFIT consists of three training modules: the linear probing module (LPM), the intermediate adapter module (IAM), and the last adapter module (LAM). These modules are trained simultancously. with each selecting clean samples for the next module. Specifically, while the LPM is trained on all samples, the LPM and LAM select clean samples for the IAM and LAM, respectively (i.e., significantly improves the performance VFMs in the presence of noisy labels for medical image classification. Additionally, we provide extensive analyses to enhance the understanding of CUFIT. We hope our insights inspire future research to further explore the robustness of vision foundation models when learning with noisy labels for various medical imaging tasks.
Experiments demonstrate that CUFIT
Acknowledgments
This work was partly supported by Institute of Information & communications Technology Planning & Evaluation (ITTP) grant funded by the Korea (MST) (No. R1-2022-11055,, Development of Uncertainty-Aware Agents Learning by Asking Questions, 90%) and Institute of Civil Military Technology Cooperation funded by the Defense Acquisition Program Administration and Ministry of Trade, Industry and Energy of Korean government under grant No. 22-CM-GU-08, 10%).
10

---

References
[1] Isic 2018 https://caalllegg...trra-aab
[2] Kaggle diabetic retinopathy detection dibaeticcrreiioopphh-eeettt.t...
competition.
https://www.kggeg www.///gl...c.ee/e/oooo
[3] Andrea Acevedo, Anna Merino González, Edwin Santiago Alférez Baquero, Angel Molina Borrás, Laura Boldú Nebot, and José Rodellar Benedé. A dataset of microscopic peripheral blood cell images for development of automatic recognition systems. Data in brief, 30(article 105474), 2020.
[4] Jason Ansel, Edward Yang, Horace He, Natalia Gimelshein, Animesh Jain, Michael Voznesensky. Bin Bao, Peter Bell, David Berard, Evgeni Burovski, Geeta Chauhan, Anjali Chourdia, Will Constable, Alban Desmaison, Zachary De Vito, Elis Ellison, Will Feng, Jiong Gong, Michael Gschwind, Brian Hirsh, Sherlock Huang, Kshiteej Kalambarkar, Laurent Kirsch, Michael Lazos, Mario Lezcano, Yanbo Liang, Jason Liang. Yinghai Lu, CK Luk, Bert Maher, Yunjic Pan, Christian Puhrsch, Matthias Reso, Mark Saroufim. Marcos Yukio Siraichi, Helen Suk, Michael Suo, Phil Tillet. Eikan Wang. Xiaodong Wang. William Wen. Shunting Zhang. Xu Zhao, Keren Zhou, Richard Zou, Ajit Mathews, Gregory Chanan, Peng Wu, and Soumith Chintala. Py Torch 2: Faster Machine Learning Through Dynamic Python Bytecode Transformation and Graph Compilation. In 29th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume (ASPLOS '24). ACM, April 2024.
[5] Devansh Arpit, Stanislaw Jastrzębski, Nicolas Ballas, David Krueger, Emmanuel Bengio, Maxinder Kan- wal, Tegan Maharaj, Asja Fischer, Aaron Courville, Yoshua Bengio, et al. A closer look at memorization in deep networks. In International conference on machine learning, pages 233-242. PMI.R, 2017.
[6] Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization arXiv preprint arXiv: 160,.0440, 2016.
[7] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the IEEE/CVF international conference computer vision, pages 9650 9660, 2021.
[8] Bingzhi Chen, Zhanhao Ye. Yishu Liu, Zheng Zhang, Jiahui Pan, Biqing Zeng, and Guangming Lu. Combating medical label noise via robust supi-super contrastive learning. In International Conference on Medical Image Computing and Computer-Assister Intervention, pages 562-572. Springer, 2023.
[9] Shoufa Chen, Chongjian Ge, Zhan Tong, Jiangliu Wang, Yibing Song, Jue Wang, and Ping Luo. Adapt- former: Adapting vision transformers for scalable visual recognition. Advances Neural Information Processing Systems, 35: 6666661666668. 2022.
[10] Xi Chen, Lianghua Huang, Yu Liu, Yujun Shen, Deli Zhao, and Hengshuang Zhao. Anydoor: Zero-shot object-level image customization. arXiv preprint arXiv: 09481, 2023.
[11] Omar Dekhil, Ahmed Naglah, Mohamed Shaban, Mohammed Ghazal, Fatma Taher, and Ayman EI- baz. Deep learning based method for computer aided diagnosis diabetic retinopathy. In 20/9 IEFE International Conference on Imaging Systems and Techniques (IST), pages 1-4. IEEE, 2019.
[12] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248-255. Ieee, 2009.
[13] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition scale. In International Conference on Learning Representations. 2021.
[14] Raman Dutt, Linus Ericsson, Pedro Sanchez, Sotirios Tsaftaris, and Timothy Hospedales. Parameter- efficient finc-tuning for medical image amalysis: The missed opportunity. In Medical Imaging with Deep Learning.
[15] Aritra Ghosh, Himanshu Kumar, and Shanti Sastry. Robust loss functions under label noise for deep neural networks. In Proceedings of the AAAI conference on artificial intelligence, volume 31, 2017.
[16] Ross Girshick. Fast r-cnn. In Proceedings of the IEEE international conference on computer vision, pages 1440-144 1448, 2015.
11

---

[17] Bo Han, Quanming Yao, Xingrui Yu, Gang Niu, Miao Xu, Weihua Hu, Ivor Tsang, and Masashi Sugiyama. Co-teaching: Robust training of deep neural networks with extremely noisy labels. Advances in neural information processing systems, 31, 2018.
[18] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the JEEE/CVE conference on computer vision and pattern recognition, pages 16000-16009, 2022.
[19] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In Proceedings of the IEEE international conference computer vision, pages 2961-2969, 2017.
[20] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[21] Dan Hendrycks, Kimin Lee, and Mantas Mazeika. Using pre-training can improve model robustness and uncertainty. In International conference on machine learning, pages 2712-2721. PMLR, 2019.
[22] Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Conference Learning Representations, 2022.
[23] Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardie, Serge Belongie, Bharath Hariharan, and Ser-Nam Lim. Visual prompt tuning. In European Conference on Computer Vision, pages 709-727. Springer, 2022.
[24] Lu Jiang, Zhengyuan Zhou, Thomas Leung, Li-Jia Li, and Li Fei-Fei. Mentomet: Learning data-driven curriculum for very deep neural networks corrupted labels. In International conference on machine learning, pages 2304 2313. PMLR, 2018.
[25] Lie Ju, Xin Wang. Lin Wang. Dwarikanath Mahapatra, Xin Zhao, Quan Zhou, Tongliang Liu, and Zongyuan Ge. Improving medical images classification with label noise using dual-uncertainty estimation. IEEE transactions medical imaging, 1133::1533-1546 2022.
[26] Tachyeon Kim, Jongwoo Ko, JinHwan Choi, Se- Young Yun, et al. Fine samples for learning with noisy labels. Advances in Neural Information Processing Systems, 34:24137-24149, 2021.
[27] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. CoRR, abs/1412.6990. 2014.
[28] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloc Rolland, Laura Gustafson, Tete Xiao, Speneer Whitehead, Alexander C Berg, Wan-Yon Lo, et al. Segment anything. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 2015-4026, 2023.
[29] Jonathan Krause, Benjamin Sapp. Andrew Howard, Howard Zhou, Alexander Toshev, Tom Duerig, James Philbin, and Fei-Fei. The unreasonable effectiveness of noisy data fine-grained recognition. In Computer Vision-ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part III 14, pages 301-320. Springer, 2016.
[30] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
[31] Junnan Li, Richard Socher, and Steven CH Hoi. Dividemix: Learning with noisy labels as semi-supervised learning. arXiv preprint arXiv:2002.0739% 2020.
[32] Junnan Li, Caiming Xiong, and Steven CH Hoi. Learning from noisy data with robust representation learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision pages 9485-9494, 2021.
[33] Sheng Liu, Jonathan Weed, Narges Razavian, and Carlos Fernandez-Granda. Farly-learning reg- ularization prevents memorization of noisy labels. Advances in neural information processing systems, 33:20331-20342 2020.
[34] Tongliang Liu and Dacheng Tao. Classification with noisy labels by importance reweighting. JEFE Transactions on pattern analysis and machine intelligence, 38(3):447-461 2015.
[35] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang. Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF international conference computer vision, pages 10012-10022, 2021.
12

---

[36] Eran Malach and Shai Shalev-Shwarte Decoupling" when to update" from" how to update". Advances in neural information processing systems, 30, 2017.
[37] Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Mare Szaframiec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07193, 2023.
[38] Yu Qiao, Chaoning Zhang, Taegoo Kang, Donghun Kim, Shehbaz Tariq, Chenshuang Zhang, and Choong Seon Hong. Robustness of sam: Segment anything under corruptions and beyond. arXiv preprint X30:: 07713, 2023.
[39] Alec Radford, Jong Wook Kim, Chris Hallacy. Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Míshkin, Jack Clark, al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748-8788 8763. PMLR, 2021.
[40] Poulami Sinhamahapatra, Franziska Schwaiger, Shirsha Bose, Huiyu Wang, Karsten Roscher, and Stephan Guennemann. Finding dino: A plug-and aung-and-pay framework for unsupervised detection of out-of-distributio objects using prototypes. arXiv preprint arXiv:2404.0760 407607644 2024.
[41] Hwanjun Song, Minseok Kim, and Jae- Lee. SELFIE: Refurbishing unclean samples for robust deep learnin.. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pages 5907-5915. PMLR, 09-15 Jun 2019.
[42] Hwanjun Song, Minscok Kim, Dongmin Park, Yooju Shin, and Jac-Gil Gil Lee. Learning from noisy labels with deep neural networks: A survey. IEEE transactions neural networks and learning systems, 2022.
[43] Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. In International conference on machine learning, pages 10347-10357. PMLR, 2021.
[44] Philipp Tschandl, Cliff Rosendahl, and Harald Kittler. The ham10000 dataset, large collection of multi-source dermatoscopic images of common pigmented skin lesions. Scientific data, 5(1):1-9, 2018.
[45] Xin Wang. Lie Ju, Xin Zhao, and Zongyuan Ge. Retinal abnormalities recognition using regional multitask learning- In Medical Image Computing and Computer Assisted Intervention-MICCA 2019: 22nd International Conference, Shenzhen, China, October 13-17, 2019, Proceedings, Part I 22, pages 30-38. Springer, 2019.
[46] Yisen Wang, Xingjun Ma, Zaiyi Chen, Yuan Luo, Jinfeng Yi, and James Bailey. Symmetric cross entropy for robust learning with noisy labels. In Proceedings of the IEEE/CVF international conference on computer vision, pages 322-330, 2019.
[47] Hongxin Wei, Lci Feng, Xiangyu Chen, and Bo An. Combating noisy labels by agreement: A joint training method with co-regularization. In Proceedings of the IEEE/CVF conference on computer vision and puttern recognition, pages 13726-13735, 2020.
[48] Zhixiang Wei, Lin Chen, Yi Jin, Xiaoxiao Ma, Tianle Liu, Pengyang Ling, Ben Wang, Huaian Chen, and Jinjin Zheng. Stronger, fewer, & superior: Harnessing vision foundation models for domain generalized semantic segmentation. arXiv preprint arXiv:2312.04265. 2023.
[49] Cheng- En Wu, Yu Tian, Haichao Yu, Heng Wang. Pedro Morgado, Yu Hen Hu, and Linjie Yang. Why is prompt tuning for vision-language models robust to noisy labels? Proceedings of the IEÉE/CVF International Conference on Computer Vision, pages 15488-15497, 2023.
[50] Xiaobo Xia, Bo Han, Yibing Zhan, Jun Yu, Mingming Gong, Chen Gong, and Tongliang Liu, Combating noisy labels with sample selection by mining -igh-iicrrepaccy examples. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1833- 1843, 2023.
[51] Xiaohan Xing, Zhen Chen, Zhifan Gao, and Yixuan Yuan. Gradient and feature conformity-sterre medical image classification with noisy labels. In International Conference on Medical Image Computing and Computer-Assisted Intervention, pages 75-84, Springer, 2023.
[52] Xuanang Xu, Fugen Zhou, Bo Liu, Dongshan Fu, and Xiangzhi Bai. Efficient multiple organ localization in et image using 3d region proposal network. IEEE transactions on medical imaging. 38(8):1885 1885):885-1894 1898. 2019.
13

---

[53] Cheng Xue, Lequan Yu, Pengfei Chen, Qi Dou, and Pheng- Ann Heng. Robust medical image classification from noisy labeled data with global and local representation guided co-training. IEEE transactions on medical imaging, (1(6):371-13822 2022.
[54] Jiancheng Yang. Rui Shi, and Bingbing Ni. Medmnist classification decathlon: A lightweight automl benchmark for medical image analysis. IEEE 18th International Symposium on Biomedical Imaging (ISBI), pages 191-195, 2021.
[55] Jiancheng Yang, Rui Shi, Donglai Wei, Zequan Liu, Lin Zhao, Bilian Ke, Hanspeter Pfister, and Bingbing Ni. Medmnist v2-a large-seale lightweight benchmark for 2d and 3d biomedical image classification. Scientific Data, 10(1):41, 2023.
[56] Xingrui Yu, Bo Han, Jiangchao Yao, Gang Niu, Ivor Tsang, and Masashi Sugiyama. How does disagreement help gencralization against label corruption? In International conference on machine learning, pages 7164-7173. PMLR, 2019.
[57] Li Yuan, Yunpeng Chen, Tao Wang, Weihao Yu, Yujun Shi. Zi Zi-Hang Jiang, Francis EH Tay, Jiashi Feng. and Shuicheng Yan. Tokens-to-token vit: Training vision transformers from scratch on imagenet. In Proceedings of the IEEE/CVE international conference on computer vision, pages 558-567, 2021.
[58] Sheng Zhang, Yanbo Xu, Naoto Usuyama, Jaspreet Bagga, Robert Tinn, Sam Preston, Rajesh Rao, Mu Wei, Naveen Valluri, Cliff Wong. Matthew Lungren, Tristan Naumann, and Hoifung Poon. Large-scale domain-specific pretraining for biomedical vision-language processing, 2023.
[59] Zhilu Zhang and Mert Sabuneu. Generalized eross entropy loss for training deep neural networks with noisy labels. Advances in neural information processing systems, 31, 2018.
[60] Zhou, Boyang Wang, Lei Huang, Shanshan Cui, and Ling Shao. benchmark studying diabetic retinopathy: segmentation, grading, and transferability. IEFE Transactions on Medical Imaging, 40(3):818- 828, 2020.
14

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We provide the main claims in the introduction and validate it in the experi- ments and discussion sections.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitation in the conclusion.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
15

---

Justification:
We do not provide the theoretical result in the paper.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibiliyy
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We provide the experimental setting of our paper. Also, we provide source code for experiments in supplementary.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
16

---

Answer: [Yes]
Justification: Datasets used in this paper are available publicly. The code will be open- sourced upon publication.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.cm ec/
for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https for more details.
//nips //nips.cc/pil wwwppbblc/guuiicuccccccc
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We give the training and test details in the Experiment section.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer:
[No]
Justification:
We do not report the statistical significance.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
17

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [No]
Justification: Our experiments are relatively free from computational resources since updates adapters of the model using small batch-size.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (c.g., preliminary or failed experiments that didn't 't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics http:::///eeuris..coo http://wwwr...cooooooocccccccaaa
Answer: [Yes]
Justification:
We have reviewed the Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
18

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We have used publicly opened code and data.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
19

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not release new assets,
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
20


Reviews:
[{"review_id": "Vp5KkP2xaZ", "reviewer": "Reviewer_8Ng4", "summary": "The paper presents Cufit, a curriculum fine-tuning paradigm for Vision Foundation Models (VFM) aimed at improving medical image classification under label noise. This method leverages the robust feature extraction capabilities of pre-trained VFMs and employs a linear probing strategy to mitigate the impact of noisy labels. The curriculum fine-tuning process then utilizes clean sample selection to enhance the classification performance. The experimental results demonstrate that Cufit outperforms existing methods on various medical image benchmarks, showing significant improvements in classification accuracy.", "questions": "1. The paper includes experimental comparisons with methods like JoCor and CoDis, but the discussion about these methods' performance is insufficient. The authors should provide a more detailed analysis of why JoCor and CoDis do not perform as well as Cufit. Understanding the strengths and weaknesses of these methods in comparison to Cufit would offer valuable insights. \n\n2. The paper should further discuss the impact of noisy labels on different types of biomedical images. For some image types, noise may be less detrimental, while for others, it could significantly affect diagnostic accuracy. A more detailed exploration of how noise impacts various biomedical image datasets would enhance the comprehensiveness of the study.", "limitations": "N/A", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The presentation is good.", "weaknesses": "1. The paper includes experimental comparisons with methods like JoCor and CoDis, but the discussion about these methods' performance is insufficient. The authors should provide a more detailed analysis of why JoCor and CoDis do not perform as well as Cufit. Understanding the strengths and weaknesses of these methods in comparison to Cufit would offer valuable insights. \n\n2. The paper should further discuss the impact of noisy labels on different types of biomedical images. For some image types, noise may be less detrimental, while for others, it could significantly affect diagnostic accuracy. A more detailed exploration of how noise impacts various biomedical image datasets would enhance the comprehensiveness of the study.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### **Q1: Discussion of previous methods** \nWe greatly appreciate your valuable feedback about including strengths and weaknesses of previous methods in the paper. Our method outperforms previous methods in the medical image classification using VFMs. As shown in Figure 3 in the paper, our modules provide the higher label precision for selecting clean samples, which is based on the linear probing’s robustness for noisy labels. However, previous methods do not leverage the linear probing’s robustness, which leads to lower label precision. The followings are strengths and weaknesses of previous methods we argue.\n- (Strengths) Although previous methods do not leverage the linear probing, they can be applicable to any network. However, our method requires well pre-trained network such as DINOv2 or CLIP to start the training which is our main weakness and their strength against ours.\n- (Weaknesss) Previous methods do not leverage the linear probing’s robustness, which leads to lower label precision. Since they design the method based on the assumption that network is training from the scratch, their methods train the network for all training data without sample selection during early training stage (e.g.,  1 to 10 epoch in their default setting). However, this training for all data may harm the feature extraction ability of VFMs and lead to degraded performance. For instance, the Co-teaching method can have better performance using different sample selection start epoch for APTOS-2019 with 40% noise rate as shown in the following table. This demonstrates that the previous methods have varying performance according to the hyper-parameter and can have the best performance when the sample selection is started at right time (i.e., when a VFM with adapter memorizes clean samples but not yet memorizes noisy samples). \n\n| Sample selection start epoch  | 1 | 3 |  5 | 10 | 25 |\n|:----------------:|:-------------:|:--------------:|:-----:|:-----------:|:-----:|\n|   Test accuracy   |     78.9 |       **80.8** | 80.1 |  79.5 | 76.2 |\n\nWe will include the analysis on the previous methods in the Discussion section and add limitation of our method in the Conclusion. \n\n### **Q2: Discussion of the impact of noisy labels on different types of dataset**\nThank you for your insightful feedback to improve our paper. Our main observation from experiments on the HAM10000 dataset (dermatoscopic images for skin lesion diagnosis) and the APTOS-2019 dataset (fundus photographs for grading diabetic retinopathy from “0-level no DR” to “4-level proliferative DR”) is that noisy labels harm APTOS-2019 less than they do HAM10000. Specifically, the relative performance drops for HAM10000 and APTOS-2019 are 34.3% and 7.1%, respectively (e.g., a 40% noise rate causes a 34.3% performance drop). For example, DINOv2s with Rein achieves 83.6% test accuracy with no noise and 54.9% test accuracy with 40% noise. We believe the relatively non-detrimental effect of noisy labels on APTOS-2019 arises from its continuous disease severity annotations. We will add this discussion to the Experiments section.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5pjxPzXXkF", "reviewer": "Reviewer_1c7f", "summary": "This paper presents a curriculum fine-tuning paradigm called Cufit. This method is designed to fine-tune Vision Foundation Models (VFMs) for medical image classification tasks under the presence of noisy labels. The approach leverages the robustness of linear probing and the generalization capabilities of fine-tuning adapters to improve model performance.", "questions": "- How does the computational cost of Cufit compare to other noise-robust training methods?", "limitations": "None.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Cufit is technically sound and is well-validated through extensive experimental results.\n- The paper is well-structured and effectively conveys the motivation, approach, and outcomes.\n- Demonstrated significant improvements in medical image classification performance under label noise.\n- Applicability to both medical and natural image classification enhances the relevance of the framework.", "weaknesses": "- The training process seems to be complex and computationally intensive.\n- I have concerns about the scalability of the proposed method. It may not scale well for very large datasets or in resource-constrained environments.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### **Q1. Computational cost of Cufit compare to other noise-robust training methods.**\nWe appreciate your feedback. Since the audience may be curious about the computational cost of our method and other training methods, we will provide the resource usage of these methods in the supplementary material. It may seem that our method is complex and computationally intensive. However, it utilizes resources reasonably compared to other methods. For example, we ran experiments on the HAM10000 dataset with a batch size of 32 using an RTX4090 GPU (the number of training images is 10,015) and found that our method performs effectively. The following table provides the memory usage and time cost for various methods with DINOv2-small architecture and Rein adapter.\n\n|                  | Full-training | Linear probing |  Rein | Co-teaching | JoCor | CoDis | Ours |\n|:----------------:|:-------------:|:--------------:|:-----:|:-----------:|:-----:|:-----:|:----:|\n|    Memory (MB)   |     4,350     |       824      | 3,144 |    10,086   | 5,542 | 10,836 | 5,604 |\n| Time (sec/epoch) |      18.6     |       6.6      |  16.5 |     32.9    |  32.7 |  51.4 | 37.7 |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "adl3p4QGD4", "reviewer": "Reviewer_T7UR", "summary": "In this paper, the authors propose a curriculum learning strategy for fine-tuning on noisy medical datasets. The key insight comes from that linear probing with limited training samples can be more robust to label noise. The performance is good compared to the former methods.", "questions": "1. Figure 2 is confusing. The core content in the method may fall in the Curriculum order and the agreement criteria. But for now, it is not clearly reflected in the figure which makes section 4.2. I suggest the author reorganize the figure and detail the purple block.\n\n2. I am not very clear about how the mode is trained. Commonly, when referring to curriculum learning, I guess the model will be trained with multiple steps but in equation 6 authors also modify the loss function which makes it also like a multi-task training pipeline.  This confuses me a lot and the authors have to clarify it more clearly.\n\n3. For the noise simulation. The authors have to state the details more, e.g., how are the correct labels changed just randomly or with some rules? and for multi-label or multi-class medical classification tasks, are there any differences? Similarly, this also confuses me in the method part. If a case with both pneumonia and bone fracture labels but the pneumonia part is wrong, will the whole case be dismissed for the latter training?\n\n4. From lines 203 to 221, in introducing the dataset, it is important to specifically point out what datasets are multi-label tasks and what are multi-class, since in medical these two types all wide exist.\n\n5. In section 6.2, it is also encouraging to explore more  performance differences between general VFM and medical-wise VFM like PMC-CLIP and BiomedCLIP. While considering the rebuttal time limitation, this is just a bonus suggestion.\n\n6. A minor suggestion, I suggest the authors clarify the definition of noise in their introduction part more formally as though for ML domain noise is clear, in medical domain, the noise types are complex which may be hard to understand for the audience in clinical background.", "limitations": "The authors adequately addressed the limitations.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "Generally, I think this is a good paper.  Noisy label is a severe and practical problem for medical scenarios due to diagnosis uncertainty and the intuition behind the proposed method is clearly stated. The performance looks good and the proposed method is clean and efficient.", "weaknesses": "However, I still have to point out some details are not clearly demonstrated. Please check the questions for more details. I will change my score based on the authors' responses.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### **Q1, Q2: Clear description of our method.**\nWe apologize for the unclear term usage. Our method trains the model like a multi-task training approach (i.e., all modules are trained simultaneously for a current given batch). We use the term “curriculum” to represent the order of the agreement criteria (i.e., the sample selection order is LPM -> IAM -> LAM). We will either change the term “curriculum” or include the sentence, “Our method trains modules simultaneously for the current given batch like a multi-task training,” in the method section. Moreover, we will update Figure 2 to clearly show the curriculum order of the training and the agreement criteria in a given training batch. The updated figure is included in the rebuttal PDF. \n\n### **Q3: Description of label noise simulation.**\nWe utilize symmetric noise generation for the experiment. For example, when the noise rate is 0.4 and the number of classes is 6, a sample assigned to class 1 has its annotation changed to one of the other classes with equal probability of 0.08. We will add a detailed description on lines 239-240 (implementation details) and a description of multi-class problem in the preliminary section. Also, we conducted the experiment with multi-class datasets only and will focus on the multi-label classification problem in the future.\n\n### **Q4: Clear dataset description.**\nThank you for your In the current version of paper, we only evaluate the method on multi-class dataset. Thus, All dataset introduced in the paper are multi-class dataset. We will clearly mention it in lines 203 to 221. Also, it will be a future research to apply our method for multi-label classification problems.\n\n### **Q5: Performance using medical-wise VFM.**\nThank you for your insightful suggestion that using medical-domain VFM like PMC-CLIP and BiomedCLIP. We ran the experiments using the image encoder of BiomedCLIP with ViT-base architecture following setting of section 6.2 (i.e., 100 epochs training with Adam optimizer). We find a similar trend in medical-wise VFM, where linear probing has better accuracy than adapter training when there are noisy labels on datasets. The following table demonstrates the experimental results (test accuracy) on BiomedCLIP for HAM10000 and APTOS-2019 with noise rate of 0.4 and we will add the results on Section 6.2.\n\n| dataset | Linear probing |  Rein | Ours |\n|:----------------:|:-------------:|:-----:|:----:|\n|  HAM10000  |     68.9 |     55.6 |  **74.4** |  \n|  APTOS-2019  |     78.6 |      64.2   | **80.0** |  \n\n### **Q6: Definition of the label noise in the Introduction part.**\nWe are thankful for your suggestion. We will add the sentence about definition of noise label in the Introduction section to help understanding the audience in non-ML domain. For example, line 25-26 “Noisy labels occur when the data annotations—the labels assigned to training images—are incorrect or inconsistent” can be replaced as “Noisy labels occur when the data annotations—the labels assigned to training images—are incorrect or inconsistent. In practice, the sample $(x_i, \\hat{y}_i)$ is considered as a noisy labeled sample when human-labeled annotation $\\hat{y}_i$ is not match the true label $y_i$”.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "M37OWgl3Lm", "reviewer": "Reviewer_tyjY", "summary": "The authors propose Cufit, a curriculum fine-tuning method for improving the performance of medical image classification under the noisy labels setting. The method shows strong performance against other baselines on several medical datasets. The authors have also provided results on a non-medical dataset.", "questions": "**Questions**\n\n1. After reading the paper several times, I am yet to understand how Cufit actually works. There is no section dedicated in the text that explains this.\n\n2. Why are methods like LoRA that are very frequently used for doing PEFT in transformer-based models and datasets like CheXpert excluded from the analysis?\n\nPlease address the **Weaknesses** section for more questions.", "limitations": "Please address the **Weaknesses** section.", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "**Strengths**\n\n1. The proposed strategy Cufit outperforms other baselines included in the evaluation\n2. The strategy is agnostic to models (CNNs and ViTs) and images (medical and natural)\n3. Cufit does not require knowledge about certain hyperparameters which are required in other methods.\n4. The authors have presented results on non-medical images as well.", "weaknesses": "**Weaknesses**\n\n1. The paper is very poorly written with spelling and grammatical errors throughout the text. Certain elements in the text are written in a convoluted way that confuses the reader.\n2. The entire paper is about the proposed method “Cufit” but no section in the paper describes the algorithm in detail. Section 6.1 (“How does Cufit work?”) does not talk about the method but only about the results. \n3. It is important to cite previous work on PEFT in medical image analysis such as [1] in the text.\n4. The paper lacks an explanation of the baseline methods used in the evaluation. Methods like CoDis are briefly mentioned in the Related Work section but have not been defined anywhere else. People unfamiliar would not be able to understand Cufit and how it differs from previously proposed methods.\n5. The experiments should include the CheXPert dataset. Apart from being frequently adopted for medical image analysis problems, it would also evaluate the proposed methodology for multi-label classification problems. Furthermore, CheXPert is supposed to contain noisy labels due to automatic labelling from free-text reports. Hence, it would adequately test the proposed method Cufit under a noisy label setting.\n6. To make the experiments more extensive, more datasets and PEFT methods (see [1] for reference) should be included. LoRA is one of the most popular PEFT strategies used for transformer-based models (especially in the case of medical image classification [1]) and should be included in the experiments.\n7. For natural image classification, the authors have adopted the CIFAR dataset. Firstly, in order to provide conclusive results, several natural imaging datasets should be included. Secondly, there are many datasets much more appropriate than CIFAR that should have been used instead.\n\nReferences\n1. Dutt, Raman, et al. \"Parameter-Efficient Fine-Tuning for Medical Image Analysis: The Missed Opportunity.\" *Medical Imaging with Deep Learning*, 2024, https://openreview.net/forum?id=LVRhXa0q5r.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### **Q2, W6:  Experimental results with LoRA**\nWe greatly appreciate your suggestion. We have conducted the experiment primarily using the Rein adapter, as it achieves excellent performance in domain-generalized semantic segmentation, and we believe it has chance to have excellent performance in the medical domain. Given that LoRA is a well-known PEFT method, we agree it should be included. We will add the experimental results using LoRA in Section 6.2, 'Performance comparison across various VFMs and adapters.' The following table provides the comparison results with LoRA on APTOS-2019 and HAM10000 with noise-rate of 40%, showing that our method is also applicable with LoRA. We will include this result in Section 6.2.\n| Dataset | Full-training | Linear probing | LoRA | Co-teaching | JoCor | CoDis | Ours |\n| --- | --- | --- | --- | --- | --- | --- | --- |\n| APTOS-2019 | 69.9 | 79.5 | 63.9 | 79.5 | 80.1 | 80.1 | **81.7** |\n| HAM10000 | 56.1 | 71.0 | 57.0 | 78.1 | 77.2 | 75.0 | **78.8** |\n\n### **W3: Previous work on PEFT in medical image analysis.**\nThank you for pointing out the related work that we missed. We will cite the paper [1] in the related work section to address the importance of PEFT in medical image analysis.\n\n### **Q2, W5: Experiment with CheXPert.**\nThank you for your valuable insight and feedback. In our current version of the paper, we focused on methods for multi-class classification problems in the medical domain, which is why we excluded datasets for multi-label classification problems. However, we agree with your suggestion regarding the CheXPert dataset. It is frequently used in medical image analysis and contains noisy labels due to automatic labeling from free-text reports. Incorporating CheXPert to evaluate Cufit in multi-label classification problems will be a part of our future research.\n\n### **W4: Other dataset for natural image classification.**\nTo validate our method on natural image classification, we further evaluate using the Animal10N dataset, a real-world noisy dataset. We follow the experimental settings outlined in \"SELFIE: Refurbishing Unclean Samples for Robust Deep Learning\" (i.e., batch size of 128 and 100 epochs training). Also, for baseline methods, we set the noise rate of 0.08 which is the estimated noise rate by \"SELFIE\" paper. Given the small performance gap, we conduct the experiment three times and average the results. The following table provides the experimental results using DINOv2-small with the Rein adapter. We will add these results to the paper.\n\n| Dataset | Full-training | Linear probing | Rein | Co-teaching | JoCor | CoDis | Ours |\n| --- | --- | --- | --- | --- | --- | --- | --- |\n| Animal10N | 74.5 | 89.1 | 88.0 | 92.2 | 91.9 | 91.7 | **92.3** |\n\n### **W7: Detailed explanation of the baseline methods.**\nWe agree on your opinion that our paper lacks an explanation of the baselines. We further describe details of the other methods after line 230. For example, the following paragraph will be added in the text.\n- Previous method such as Co-teaching, JoCor, and CoDis involve training two neural networks concurrently on the dataset. In the training stage, each network identifies and discards potentially mislabeled examples, subsequently teaching the other network with the remaining clean examples. Specifically, Co-teaching trains two homogeneous networks simultaneously and cross-updates parameters using R(T)% small-loss instances from the given mini-batch (R(T)% is the estimated noise rate). Additionally, JoCor encourages two different networks to make predictions closer to each other by explicit regularization loss during training to select the small-loss sample with assumption that two networks easily reach an agreement on its predictions for clean samples. CoDis selects high-discrepancy examples with a joint loss (joint loss is equal to cross-entropy loss reduced by Jensen–Shannon (JS) divergence loss), leveraging discrepancy for sample selection, thereby improving selection quality.\n### **Q1 W2: Detailed explanation of our method**\nWe appreciate your feedback. To address lack of detailed explanation of our method, we will update the Section 6.1. and Section 4. to provide description about the method. Our method is designed to train the VFMs with a adapter module robustly for noisy medical image datasets. Our key insight is as follows.\n- Cufit leverages a curriculum learning approach to robustly fine-tune the VFMs for noisy datasets with three key modules: Linear Probing Module (LPM), Intermediate Adapter Module (IAM), and Last Adapter Module (LAM). \n- Linear Probing Module (LPM): The LPM is first trained on all available samples in the given batch. Linear probing is relatively robust against noisy labels as it does not modify the feature extraction capability of VFM, thus preventing memorization of noisy samples. However, since there is no modification in feature extractor, its accuracy is low. \n- Intermediate Adapter Module (IAM): For the given batch, the IAM is trained using samples selected by the LPM based on agreement criteria, which considers a sample as clean if its annotation matches the prediction of the LPM. Compared to LPM, IAM can achieve higher accuracy since it has small PEFT on the feature extraction.\n- Last Adapter Module (LAM): Similarly, the LAM is trained with samples selected by the IAM. This inter-module curriculum training (LPM → IAM → LAM) increases the number of clean samples available for training the LAM, enhancing its performance. Specifically, LPM only selects few samples since its accuracy is lower than IAM. LPM is used for the inference stage.\n### **W1: Spelling and grammatical errors throughout the text.**\nWe apologize for the poor quality of current version of our paper. We have conducted a thorough review of the paper to ensure it has no spelling and grammatical errors throughout the text. Also, we plan to have the professionally English proofreading service to further refine the language and clarity.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We appreciate the reviewers for their valuable comments and constructive feedback on our paper. As summarized by all reviewers, we propose a novel parameter-efficient fine-tuning (PEFT) framework for medical image classification under noisy labels. We believe our framework can outperform previous noise-robust training methods, which focused on CNN networks trained from scratch. 

This rebuttal addresses all reviewers’ concerns. Additionally, we provide a PDF file showing the modified version of Figure 2. Furthermore, we are open to discussions and committed to addressing any additional concerns from the reviewers to ensure the refinement of our paper.

Meta-review: This paper studies vision foundation models for medical image classification from the aspect of parameter-efficient fine-tuning under noisy labels. The topic is interesting and valuable. Reviewers are generally positive for their comments. The strengths of the work include sound method design and rich experiments showing good performance. The weaknesses are mainly concerning lack of detailed analysis on comparison with SOTA, and clarifications on method/experiment implementation details. The rebuttal has satisfactorily answered most of the questions. With the final ratings and replies from reviewers, the AC would like to recommend acceptance for this submission.
Decision: Accept (poster)