Curriculum Fine-tuning of Vision Foundation Model for Medical Image Classification Under Label Noise

Authors: yeonguk  yu
minhwan  ko
sungho  shin
kangmin  kim
kyoobin  lee
Venue: NeurIPS_cc_2024_Conference
Published: 2024-04-23 03:04:22

Abstract: Deep neural networks have demonstrated remarkable performance in various vision tasks, but their success heavily depends on the quality of the training data. Noisy labels are a critical issue in medical datasets and can significantly degrade model performance. Previous clean sample selection methods have not utilized the well pre-trained features of vision foundation models (VFMs) and assumed that training begins from scratch. In this paper, we propose CUFIT, a curriculum fine-tuning paradigm of VFMs for medical image classification under label noise. Our method is motivated by the fact that linear probing of VFMs is relatively unaffected by noisy samples, as it does not update the feature extractor of the VFM, thus robustly classifying the training samples. Subsequently, curriculum fine-tuning of two adapters is conducted, starting with clean sample selection from the linear probing phase. Our experimental results demonstrate that CUFIT outperforms previous methods across various medical image benchmarks. Specifically, our method surpasses previous baselines by 5.0\%, 2.1\%, 4.6\%, and 5.8\% at a 40\% noise rate on the HAM10000, APTOS-2019, BloodMnist, and OrgancMnist datasets, respectively. Furthermore, we provide extensive analyses to demonstrate the impact of our method on noisy label detection. For instance, our method shows higher label precision and recall compared to previous approaches. Our work highlights the potential of leveraging VFMs in medical image classification under challenging conditions of noisy labels.

Main paper: 
 introduction: 
 Deep neural networks have demonstrated remarkable performance across various tasks, including classification, detection, and segmentation [20, 16, 19, 57] . In medical imaging, these neural networks leverage large amounts of labeled data to train models capable of accurately detecting or classifying medical conditions from images such as dermatoscopes, X-rays, MRIs, and CT scans. However, in practical settings, data often contain noisy labels and it is well established that neural networks perform well only when the quality of training data is sufficiently high [42, 29, 5] . Noisy labels occur when the data annotations-the labels assigned to training images-are incorrect or inconsistent. This issue is particularly problematic in medical imaging, where annotating images is more complex compared to natural images [53, 25] . Consequently, improving the robustness of neural networks against noisy labels is a crucial area of research, directly affecting the effectiveness and reliability of medical imaging technologies.A large number of algorithms have been developed to address the issue of performance degradation caused by noisy samples [42] . In particular, clean sample selection methods, such as MentorNet [24] , Co-teaching [17] , Co-teaching+ [56] , JoCor [47] , and CoDis [50] , have demonstrated superior performance without requiring modifications to the model architecture or training loss. The core Figure 1 : Illustration of linear probing (a) and adapter usage (b). Specifically, the weights of the foundation model are frozen, while the fully connected layer or adapter weights (shown in orange) are updated during the training phase. In (c), a performance comparison using a simulated noisy dataset (HAM10000) is presented. It demonstrates that linear probing is more robust to noisy labels compared to the adapter, whereas the adapter outperforms linear probing when there are no noisy labels.principle behind these methods is that small-loss samples are likely to be clean, as they are easier to classify and the model memorizes them faster. Additionally, using two different but homogeneous networks to select small-loss samples for each other is more stable than relying on a single model for sample selection. These methods have shown outstanding performance using traditional neural network architectures based on convolutional layers. However, there are practical issues with these methods in two aspects: (i) it cannot be guaranteed that these methods will perform equally well with transformer-based architectures, which have recently gained significant attention, and (ii) their assumption that training starts from scratch is impractical, as it prevents the use of rich features from pre-trained models, which could be beneficial for filtering noisy labels.Recently, large-scale vision foundation models (VFMs) with transformer-based architectures [13] , such as CLIP [39] , MAE [18] , SAM [28] , and DINOv2 [37] , have gained attention for their performance and applicability across various tasks. The self-supervised training of VFMs on large-scale datasets enhances their robustness against various image corruptions and improves their generalization capabilities [10, 48, 38, 40] . The inherent robustness and rich features of VFMs can be beneficial for detecting noisy labels. For instance, linear probing of VFMs is relatively unaffected by noisy samples since it does not modify the VFM's feature extractor, preventing the memorization of noisy data, as shown in Figure 1 . However, linear probing does not fully leverage the VFM's capabilities when there is a domain gap between the pretraining task of the VFM and the target task (e.g., pretraining on natural images versus medical image classification) [48] . To address this issue, some researchers have proposed using trainable fine-tuning adapters for VFMs [22, 23, 48, 9 ]. Yet, these adapters might degrade performance by memorizing noisy labels due to their trainable parameters involved in feature extraction. Therefore, we state our research question as follows: How can we use the power of pre-trained vision foundation models for medical image classification in the presence of noisy labels?In this paper, we introduce a Curriculum Fine-Tuning paradigm for vision foundation models in medical image classification under noisy labels, called CUFIT. CUFIT is a curriculum-learning framework designed to fine-tune VFMs with noisy medical datasets. The framework consists of three training modules: the Linear Probing Module (LPM), the Intermediate Adapter Module (IAM), and the Last Adapter Module (LAM). During the training stage, clean samples are selected based on an agreement criterion, where a sample is selected if its annotation matches the module's prediction. Specifically, the LPM is trained using all available samples, as linear probing is robust against noisy labels. Subsequently, the IAM is trained with the samples selected by the LPM, and the LAM is trained with the samples selected by the IAM. This inter-module curriculum training (i.e., LPM→IAM→LAM) is beneficial for increasing the number of clean samples available to train the LAM, considering that the LPM only selects a limited number of samples due to performance degradation caused by the domain gap. Consequently, CUFIT leverages the LAM for final predictions, offering strong fine-tuning performance for medical image classification in the presence of noisy labels by utilizing the strengths of both linear probing and adapters, as illustrated in Figure 1-c . In summary, the main contributions of this paper are as follows:• We introduce CUFIT, a simple yet effective fine-tuning paradigm for medical image classification using VFMs in the presence of noisy labels. This method leverages the robustness of linear probing and the generalization capability of fine-tuning adapters to handle noisy datasets during training stage.• We conduct various experiments demonstrating that CUFIT significantly improves the robustness of VFMs against noisy labels in medical datasets. We show that CUFIT outperforms previous methods across several medical image benchmarks.• We provide extensive analyses to enhance the understanding of our fine-tuning paradigm. Additionally, we validate our framework with various VFMs and adapter configurations. 
related work: 
 Vision foundation models. Vision transformers (ViTs) embed 2D images into 1D tokens and model their global correlations using the self-attention mechanism [13, 35, 43] . ViTs are known to be effective when large datasets are used, and the concept of vision foundation models is introduced.Several studies have developed pre-trained vision transformers based on self-supervised learning. For instance, contrastive language-image pre-training (CLIP) provides high-quality visual representations through contrastive learning with a large amount of image-text pairs [39] . Additionally, masked autoencoder (MAE) offers high-capacity models that generalize well through self-supervised learning with masked auto-encoding, where the task is to reconstruct token patches from the given masked tokens [18] . Moreover, knowledge distillation with no labels (DINOv1) [7] proposed a teacherstudent framework for self-training without annotations, resulting in a well-generalized ViT. More recently, DINOv2 introduced a self-training framework that combines masked autoencoding and teacher-student training based on carefully curated datasets [37] .Since large models and self-training in VFMs provide strong generalization capabilities for various tasks, parameter-efficient fine-tuning has gained attention. Parameter-efficient fine-tuning (PEFT) aims to adapt foundation models to new tasks by training only a few adapter parameters while keeping the model itself frozen. Notably, there is research that proposes low-rank adaptation (LoRA), which introduces trainable rank decomposition matrices into each layer of the transformer architecture in large language models [22] . For the VFMs, visual prompt tuning [23] proposed appending prompts to the input sequence of each transformer block, achieving excellent fine-tuning performance with minimal parameters. Similarly, Adaptformer [9] introduces a novel MLP block to replace the original one in transformer blocks, allowing for the use of both original and few trainable parameters. More recently, Wei et al. proposed Rein, which aims to adapt VFMs for semantic segmentation with domain generalization capabilities [48] . Also, Dutt et al. investigate PEFT algorithms across both convolutional and large transformer-based networks for medical image classification, demonstrating the effectiveness of PEFT, particularly in the low-data regimes common in medical imaging [14] . In this paper, we focus on fine-tuning VFMs for image classification in the presence of noisy labels using adapters. Rather than introducing a new adapter, we utilize an existing adapter within our training paradigm.Learning with noisy label. Deep neural networks have demonstrated remarkable performance on large-scale datasets. However, it is well-known that neural networks can easily memorize noisy labeled samples, leading to degraded performance. Several studies have been conducted to explore robust supervised learning in the presence of noisy labels. These studies can be categorized into five approaches [42] : (i) robust architectures, (ii) robust regularization, (iii) robust loss functions [15, 33, 46, 59] , (iv) loss adjustment [26, 31, 32] , and (v) sample selection [24, 36, 17, 56, 50] . In this paper, we categorize our method as a sample selection method, which selects samples with clean labels from a noisy training dataset. While previous sample selection methods typically consider training from scratch, we focus on training starting from a pre-trained model, which is known to be more robust to noisy labels [21] . Additionally, research has explored using CLIP to enable robust training by leveraging its text-image matching capability on noisy datasets [49] .Various methods have been proposed for clean sample selection from noisy datasets. For example, MentorNet introduced the use of a teacher network to guide the student network to focus on clean labels [24] . Similarly, Decoupling proposed updating two networks by using only the samples with differing predictions between them [36] . Co-teaching also trains two networks simultaneously, updating them based on sample recommendations from each other [17] . Co-teaching+ [56] improved upon Co-teaching by introducing the "update by disagreement" strategy, where only the samples with differing predictions between the two networks are used. More recently, Xia et al. proposed CoDis, an extension of Co-teaching+ that employs an "update by discrepancy" strategy, selecting samples with high-discrepancy prediction probabilities between the two networks to utilize more samples [50] . These methods are based on the assumption that clean samples can be identified using certain criteria, and that network collaboration is more stable than self-selection, which may lead to error accumulation. In this paper, we develop our method based on same assumption, but we assume that we start the learning process from the pre-trained VFM.Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes] Justification: We discuss the limitation in the conclusion. Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
problem setup: 
 We consider a k-class classification task using a neural network. Let X ∈ R d denote the input space and Y ∈ R k represent the ground-truth label space. In a typical classification task, the neural network is trained to align the input space with the label space. To this end, a training dataset D = {(x i , ŷi )} n i=1 is used for supervised learning with cross-entropy loss. In practice, a sample (x i , ŷi ) is considered as a noisy labeled sample when human-annotated label ŷi does not match the true label y i . The objective of this paper is to develop a fine-tuning approach for VFMs that is robust to noise and capable of performing accurately on noisy datasets.Given a pre-trained VFM with parameters θ VFM , consisting of a sequence of layers (e.g., attention blocks in ViT [13] ) L 1 , L 2 , ..., L M , where M is the depth of θ VFM , the learning objective for a classification problem can be formulated as:EQUATIONwhere θ t and L ce represent the parameters targeted for updating and the cross-entropy loss, respectively. Here, p(•|θ) refers to the prediction for a given input using parameters θ. We refer to the training process as linear probing when θ t is limited to the parameters of the linear layer θ l . Additionally, we refer to the training process as full-tuning when θ t includes θ VFM , and as adapter tuning when it includes adapter parameters θ a , which are not part of θ VFM .Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof? Answer: [NA]Answer: [Yes] Justification: Datasets used in this paper are available publicly. The code will be opensourced upon publication.Guidelines:• The answer NA means that paper does not include experiments requiring code. • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 
method: 
 In this section, we begin by describing the adapter method for fine-tuning the VFM in Section 4.1. Following this, in Section 4.2, we introduce our method, CUFIT, which utilizes three modules: a linear layer and two adapters, to combat noisy labels. The key idea behind CUFIT is to leverage the well-pre-trained features of the VFM without updating the feature extractor when handling corrupted samples. Subsequently, the adapters are trained using the samples selected in a curriculum-based training manner, as shown in Figure 2 (i.e., linear probing → intermediate adapter → last adapter). This approach helps increase the number of selected samples by reducing the domain gap between the pretraining task and the medical image task. It is important to note that our framework does not train the modules sequentially (i.e., where one module starts training only after another finishes); instead, it trains the modules simultaneously on the current batch, similar to multi-task training.We consider various adapters for fine-tuning VFMs on medical image datasets.In particular, adapters like Visual Prompt Tuning (VPT [23] ), AdaptFormer [9] , Low Rank Adaptation (LoRA [22] ) and Rein [48] can be used. These methods have been shown to be efficient for various image and video tasks, even compared to full model training [9, 48] . Typically, when an adapter is used for fine-tuning, the parameters of the VFM are frozen and not included in the optimization process.In this section, we briefly introduce how an adapter works. Note that our goal is not to propose a novel adapter but rather to present a training paradigm that can be applied to various adapters. For vision transformers (ViTs), the output of the attention block for the given input patches is calculated as follows:EQUATIONwhere x l-1 is the output token of the previous block.Here, Q, K, and V refer to the query, key, and value vectors, respectively, which are derived from linear projection and LayerNorm [6] applied to x l-1 . The final output of the block, x l , is then computed using LayerNorm and an MLP. Without using an adapter, this process is formulated as:EQUATIONwhere x l is the output token of the l-th block. When an adapter is used, the Eq 3 is replaced by the following:EQUATIONwhere Adapt(•; θ a l ) refers to the adapter function for the l-th layer, parameterized by θ a l . We consider this process to be an arbitrary function, as various adapters can be used.In the last block, the [CLS] token is passed to the following linear layer for final image classification.We consider a pre-trained VFM, θ VFM , with a single linear layer parameterized by θ LPM ∈ R c×k , an intermediate adapter module parameterized by θ IAM , and a last adapter module parameterized by θ LAM , where c refers to the dimension of the class token (e.g., 384 dimensions for the ViT-small architecture). Then, we propose a curriculum training framework for these three modules, in which the LPM is trained with all samples from the given batch, while the adapter modules are trained with filtered samples selected by their corresponding module using the agreement criterion. The agreement criterion refers to a method where a sample is considered clean if the module's prediction matches the sample's annotation. The idea behind this criterion is that a robust classifier will correctly predict the sample under the assumption that clean labels are in the majority within a noisy class. Therefore, a sample is selected as clean if it meets the agreement criterion (e.g., a "dog" image with a "dog" annotation). Thus, we build the curriculum training framework based on the robustness of the LPM against noisy labels using the agreement criterion.In particular, the linear probing module (LPM) is trained as follows:EQUATIONwhich directly represents supervised learning using the given images and corresponding labels. Here, p(x i |θ VFM , θ LPM ) refers to the output of the network using θ VFM and θ LPM for the given image x i .During the training stage, the intermediate adapter module (IAM) is trained as follows:EQUATIONwhere 1{•} is the indicator function. This simple modification using the indicator function ensures that the adapter module is trained only on selected samples chosen by the linear layer. Finally, the last adapter module (LAM) is trained as follows:EQUATIONThe Eq 7 is equivalent to Eq 6, but it uses LAM and IAM instead of IAM and LPM, respectively.This simple yet effective sample selection strategy is well-suited for fine-tuning the VFM on noisy image datasets. Notably, it does not require any hyperparameters like the estimated noise rate, which are commonly needed in previous works [17, 47, 50] , where they assume the noise rate is known in order to select small-loss samples (e.g., selecting 60% of samples in a batch for a known noise rate of 40%). After training is completed, only the last adapter module is used to predict the given test image.5 Experiments 
settings: 
 Datasets. We evaluate our approach on four simulated noisy label medical multi-class image classification benchmarks: HAM10000 [44] , APTOS-2019 [11] , BloodMnist [3] , and OrgancMnist [52] . Additionally, we conduct an evaluation on a real-world noisy label benchmark, Kaggle-EyePACS [2].In particular, the detail of datasets are as follows:• HAM10000 [44] : This dataset contains 10,015 dermatoscopic images for skin lesion classification, with each image classified into one of seven possible disease categories. We use all the images for training, and the evaluation is conducted using the 1,512 test images provided by the ISIC 2018 challenge [1] . • APTOS-2019 [11] : This dataset consists of 3,662 retina images taken with fundus photography under various imaging conditions. Each image is rated for the severity of diabetic retinopathy (DR) on a scale from 0 to 4. We use 2,930 images for training and 366 images for evaluation. [25] . Specifically, some DR category labels (e.g., mild DR labeled as moderate DR) are noisy, and some images considered normal may actually contain retinal diseases such as glaucoma or drusen, which are not included in the classification categories. It is estimated that there is approximately a 30%-40% label error in this dataset [25, 45] . We use the original 35,126 training images and their annotations for training, and all images from APTOS-2019 for evaluation. Additionally, we use the FGADR [60] dataset for further evaluation.Baselines. We compare the performance of CUFIT with basic training paradigms: full training, linear probing, and fine-tuning with Rein [48] . Additionally, we evaluate our approach against other training-based methods, including Co-teaching [17] , JoCor [47] , and CoDis [50] . Like ours, these methods do not modify the training loss or architecture. Specifically, Co-teaching trains two networks simultaneously, with each network selecting small-loss samples from its peer's predictions to guide learning. JoCor extends this idea by incorporating co-regularization to maximize agreement between the two networks. CoDis further refines this process by selecting samples that not only have small losses but also show high divergence between the two networks. It is important to note that we do not compare our proposed framework with state-of-the-art methods that modify the training loss (e.g., semi-supervised learning) or model architecture [51, 8] . In the experiments, we apply these methods to VFMs with adapters, as they do not require specific model architectures, and VFMs with adapters outperform the linear probing of VFMs (i.e., DINOv2 with the Rein adapter is used as the default setting for training with Co-teaching, JoCor, and CoDis for a fair comparison).For the experiments, we use DINOv2 [37] with the ViT-small [13] backbone as our basic vision foundation model. Additionally, we use Rein [48] as the fine-tuning adapter, originally proposed for domain-generalized semantic segmentation of VFMs. In our setup, we utilize the class token from the block for classification, rather than the patch tokens from multiple blocks.We use the PyTorch [4] codebase for our experiments. BloodMnist and OrgancMnist datasets are sourced from MedMnist [54, 55] . We use the ViT-small architecture and the Adam optimizer [27] .All training runs for 100 epochs with a batch size of 32. The initial learning rate is 0.001, which decays by a factor of 10 at epochs 50, 75, and 90. For full-parameter training, however, we start with an initial learning rate of 0.0001. For the simulated noisy label benchmarks, we generate symmetric noise [17] for evaluation, with noise rates set at 10%, 20%, 40%, and 60%.First, we evaluate our framework on simulated noisy label benchmarks using four medical datasets. The average classification test accuracy for each dataset is provided in Table 1 . Our framework consistently outperforms previous baselines, demonstrating its effectiveness under noisy labels by leveraging the pre-trained features of DINOv2 and the Rein adapter. Notably, our framework proves to be more effective as the noise rate increases. For example, CUFIT achieves 0.85% relateively higher accuracy than CoDis on HAM10000 with a 10% noise rate, while the improvement rises to 3.7% at a 60% noise rate. This result indicates that the pre-trained features of the VFM are particularly useful for handling noisy labels in the given datasets.We train DINOv2 with Rein on the real-world benchmark, using the Kaggle-EyePACS dataset for training and the APTOS-2019 and FGADR datasets for testing. Given the highly imbalanced training set (e.g., approximately 73% of the samples are labeled as the normal class), we use weighted cross-entropy loss to train the model. Since previous sample selection methods require a noise rate hyperparameter, we employed the noise estimation method from [34] , following Co-teaching [17] .In Table 2 , we report the classification accuracy on the APTOS-2019 and FGADR datasets, as well as the overall accuracy across both datasets. Our method outperforms other baselines on the FGADR dataset and the combined dataset, while Co-teaching achieves the highest accuracy on the APTOS-2019 dataset. We believe this discrepancy is due to the distribution of normal class samples-approximately 50% in APTOS-2019 and about 5% in FGADR. Co-teaching performs well in classifying the normal class, whereas our method excels at classifying diseased samples. For example, our method achieves 53.9% macro-average test accuracy, while Co-teaching achieves 48.5% on the combined test set. 
how does cufit works?: 
 So far, we have demonstrated through empirical results that our framework significantly improves the robustness of VFM fine-tuning against noisy labels. However, we have not yet discussed why our framework is effective in learning with noisy labels. In Figure 3 , we present label precision, label recall, and test accuracy over the number of epochs to illustrate how our framework functions. In principle, higher label precision indicates fewer noisy samples in the selected data, while higher label recall indicates fewer clean samples in the unselected data. We have following three observations:Figure 4 : Test accuracy of our method with various VFMs (DINOv1 [7] , MAE [18] , DINOv2 [37] ) and adapters (VPT [23] , AdaptFormer [9] , Rein [48] ). We use HAM10000 and APTOS-2019 with 40% noise rate for training. • We find that CoDis exhibits lower label precision during training compared to LPM and IAM, suggesting that previous sample selection methods fail to effectively utilize the pre-trained features of VFMs, leading to lower test accuracy. These methods train the network on all training data without sample selection during the early stages of training (e.g., epochs 1 to 10 in their default setting). However, this approach may harm the feature extraction capability of VFMs and result in degraded performance.• LPM consistently achieves the highest label precision across epochs but has the lowest label recall, indicating that it effectively prevents the memorization of noisy samples. However, because the feature extractor remains unchanged, its overall accuracy is limited, thus selecting only a small number of clean samples.• IAM, on the other hand, achieves similar label precision but higher label recall by leveraging the adapter module, which contributes to the improved test accuracy of LAM. This suggests that by adapting the feature extractor through training the adapter on a few certain clean samples, IAM can be a more accurate module. It can then provide more clean samples to LAM, resulting in better overall performance.To validate the performance of CUFIT in various settings, we present experimental results using three VFMs and adapters in Figure 4 . We use the same experimental setup (i.e., 100 epochs with the Adam optimizer) to train the network across all backbones and adapters. Specifically, we utilize four backbones, including DINOv1 [7] , MAE [18] , DINOv2 [37] , and BioMedCLIP [58] and four adapters, including VPT [23] , AdaptFormer [9] , LoRA [22] , and Rein [48] . BioMedClip is a CLIPlike model trained with the PMC-15M dataset, which contains 15 million biomedical image-text pairs collected from 4.4 million scientific articles. Our results demonstrate that our framework consistently helps build a robust classifier across different VFMs and adapters. For example, our framework achieves better performance compared to both adapter-based methods and linear probing. Additionally, we observe that linear probing consistently outperforms the adapter method in all cases, indicating that the performance of adapters can be degraded by noisy labels across various adapters. Table 4 : Average test accuracy on the natural image dataset with simulated noisy labels (CIFAR, symmetric noise at 80%) and real-world noisy labels (ANIMAL10N [41] , which has an estimated noise ratio of 8%). The test accuracy is averaged over the last ten epochs. We use DINOv2 with Rein adapter for the experiment. Bold values the best result.We designed CUFIT for VFMs due to their strong pre-trained feature extraction capabilities, enabled by self-supervised training on large datasets. However, CNN-based architectures like ResNet [20] also utilize pre-trained weights instead of starting from scratch. Therefore, we validate CUFIT on ImageNet [12] pre-trained ResNet50, with and without the Rein adapter modified for ResNet. The experimental results are shown in Table 3 . We observe that our method outperforms other training paradigms when using the ImageNet pre-trained ResNet architecture. Additionally, the Rein adapter for ResNet improves performance, demonstrating that using fewer trainable parameters with an adapter, compared to full training, is beneficial for combating noisy labels. Finally, we show that the more representative pre-trained features of DINOv2 outperform the ImageNet pre-trained features across all training methods.Since our framework is easily applicable not only to medical image classification but also to natural image classification, we present experimental results on the CIFAR [30] simulated noisy classification benchmarks and ANIMAL10N [41] real-world noisy classification benchmark in Table 4 . We validate our framework under an extremely high noise rate setting (80%) for CIFAR benchmark, as it is intuitive that our framework performs well under low noise rates due to the feature extraction capabilities of VFM. As shown in Table 4 , our framework outperforms other sample selection methods in natural image classification benchmarks as well. This demonstrates the effectiveness of our framework, highlighting that using well pre-trained VFMs is beneficial for detecting noisy labels in natural images, as expected.Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes]Justification: We give the training and test details in the Experiment section.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
conclusion: 
 This paper presents a curriculum fine-tuning paradigm called CUFIT, designed to robustly finetune vision foundation models (VFMs) for medical image classification. Our framework is based on the insight that linear probing of VFMs is robust to noisy labels, as it does not modify the feature extraction process. Building on this, CUFIT consists of three training modules: the linear probing module (LPM), the intermediate adapter module (IAM), and the last adapter module (LAM). These modules are trained simultaneously, with each selecting clean samples for the next module. Specifically, while the LPM is trained on all samples, the LPM and IAM select clean samples for the IAM and LAM, respectively (i.e., LPM→IAM→LAM). Experiments demonstrate that CUFIT significantly improves the performance of VFMs in the presence of noisy labels for medical image classification. Additionally, we provide extensive analyses to enhance the understanding of CUFIT. We hope our insights inspire future research to further explore the robustness of vision foundation models when learning with noisy labels for various medical imaging tasks.Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [No] Justification: We do not report the statistical significance.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors).• It should be clear whether the error bar is the standard deviation or the standard error of the mean. • It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified. • For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates). • If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text. 
experiments compute resources: 
 Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?Answer: [No] Justification: Our experiments are relatively free from computational resources since it updates adapters of the model using small batch-size.Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). 
code of ethics: 
 Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines? 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [NA]Justification:Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA]Justification:Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes]Justification: We have used publicly opened code and data.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. 
 
 Appendix: 1. Claims Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? Answer: [Yes] Justification: We provide the main claims in the introduction and validate it in the experiments and discussion sections. Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.Justification: We have reviewed the Code of Ethics.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 


Decision: This paper studies vision foundation models for medical image classification from the aspect of parameter-efficient fine-tuning under noisy labels. The topic is interesting and valuable. Reviewers are generally positive for their comments. The strengths of the work include sound method design and rich experiments showing good performance. The weaknesses are mainly concerning lack of detailed analysis on comparison with SOTA, and clarifications on method/experiment implementation details. The rebuttal has satisfactorily answered most of the questions. With the final ratings and replies from reviewers, the AC would like to recommend acceptance for this submission.

Reviews: review: paper_summary: paper_summary: The paper presents Cufit, a curriculum fine-tuning paradigm for Vision Foundation Models (VFM) aimed at improving medical image classification under label noise. This method leverages the robust feature extraction capabilities of pre-trained VFMs and employs a linear probing strategy to mitigate the impact of noisy labels. The curriculum fine-tuning process then utilizes clean sample selection to enhance the classification performance. The experimental results demonstrate that Cufit outperforms existing methods on various medical image benchmarks, showing significant improvements in classification accuracy.
strength_weakness: strength_weakness: 1. The paper includes experimental comparisons with methods like JoCor and CoDis, but the discussion about these methods' performance is insufficient. The authors should provide a more detailed analysis of why JoCor and CoDis do not perform as well as Cufit. Understanding the strengths and weaknesses of these methods in comparison to Cufit would offer valuable insights. 

2. The paper should further discuss the impact of noisy labels on different types of biomedical images. For some image types, noise may be less detrimental, while for others, it could significantly affect diagnostic accuracy. A more detailed exploration of how noise impacts various biomedical image datasets would enhance the comprehensiveness of the study.
questions: questions: 1. The paper includes experimental comparisons with methods like JoCor and CoDis, but the discussion about these methods' performance is insufficient. The authors should provide a more detailed analysis of why JoCor and CoDis do not perform as well as Cufit. Understanding the strengths and weaknesses of these methods in comparison to Cufit would offer valuable insights. 

2. The paper should further discuss the impact of noisy labels on different types of biomedical images. For some image types, noise may be less detrimental, while for others, it could significantly affect diagnostic accuracy. A more detailed exploration of how noise impacts various biomedical image datasets would enhance the comprehensiveness of the study.
limitations: limitations: N/A
paper_summary: paper_summary: This paper presents a curriculum fine-tuning paradigm called Cufit. This method is designed to fine-tune Vision Foundation Models (VFMs) for medical image classification tasks under the presence of noisy labels. The approach leverages the robustness of linear probing and the generalization capabilities of fine-tuning adapters to improve model performance.
strength_weakness: strength_weakness: - The training process seems to be complex and computationally intensive.
- I have concerns about the scalability of the proposed method. It may not scale well for very large datasets or in resource-constrained environments.
questions: questions: - How does the computational cost of Cufit compare to other noise-robust training methods?
limitations: limitations: None.
paper_summary: paper_summary: In this paper, the authors propose a curriculum learning strategy for fine-tuning on noisy medical datasets. The key insight comes from that linear probing with limited training samples can be more robust to label noise. The performance is good compared to the former methods.
strength_weakness: strength_weakness: However, I still have to point out some details are not clearly demonstrated. Please check the questions for more details. I will change my score based on the authors' responses.
questions: questions: 1. Figure 2 is confusing. The core content in the method may fall in the Curriculum order and the agreement criteria. But for now, it is not clearly reflected in the figure which makes section 4.2. I suggest the author reorganize the figure and detail the purple block.

2. I am not very clear about how the mode is trained. Commonly, when referring to curriculum learning, I guess the model will be trained with multiple steps but in equation 6 authors also modify the loss function which makes it also like a multi-task training pipeline.  This confuses me a lot and the authors have to clarify it more clearly.

3. For the noise simulation. The authors have to state the details more, e.g., how are the correct labels changed just randomly or with some rules? and for multi-label or multi-class medical classification tasks, are there any differences? Similarly, this also confuses me in the method part. If a case with both pneumonia and bone fracture labels but the pneumonia part is wrong, will the whole case be dismissed for the latter training?

4. From lines 203 to 221, in introducing the dataset, it is important to specifically point out what datasets are multi-label tasks and what are multi-class, since in medical these two types all wide exist.

5. In section 6.2, it is also encouraging to explore more  performance differences between general VFM and medical-wise VFM like PMC-CLIP and BiomedCLIP. While considering the rebuttal time limitation, this is just a bonus suggestion.

6. A minor suggestion, I suggest the authors clarify the definition of noise in their introduction part more formally as though for ML domain noise is clear, in medical domain, the noise types are complex which may be hard to understand for the audience in clinical background.
limitations: limitations: The authors adequately addressed the limitations.
paper_summary: paper_summary: The authors propose Cufit, a curriculum fine-tuning method for improving the performance of medical image classification under the noisy labels setting. The method shows strong performance against other baselines on several medical datasets. The authors have also provided results on a non-medical dataset.
strength_weakness: strength_weakness: **Weaknesses**

1. The paper is very poorly written with spelling and grammatical errors throughout the text. Certain elements in the text are written in a convoluted way that confuses the reader.
2. The entire paper is about the proposed method “Cufit” but no section in the paper describes the algorithm in detail. Section 6.1 (“How does Cufit work?”) does not talk about the method but only about the results. 
3. It is important to cite previous work on PEFT in medical image analysis such as [1] in the text.
4. The paper lacks an explanation of the baseline methods used in the evaluation. Methods like CoDis are briefly mentioned in the Related Work section but have not been defined anywhere else. People unfamiliar would not be able to understand Cufit and how it differs from previously proposed methods.
5. The experiments should include the CheXPert dataset. Apart from being frequently adopted for medical image analysis problems, it would also evaluate the proposed methodology for multi-label classification problems. Furthermore, CheXPert is supposed to contain noisy labels due to automatic labelling from free-text reports. Hence, it would adequately test the proposed method Cufit under a noisy label setting.
6. To make the experiments more extensive, more datasets and PEFT methods (see [1] for reference) should be included. LoRA is one of the most popular PEFT strategies used for transformer-based models (especially in the case of medical image classification [1]) and should be included in the experiments.
7. For natural image classification, the authors have adopted the CIFAR dataset. Firstly, in order to provide conclusive results, several natural imaging datasets should be included. Secondly, there are many datasets much more appropriate than CIFAR that should have been used instead.

References
1. Dutt, Raman, et al. "Parameter-Efficient Fine-Tuning for Medical Image Analysis: The Missed Opportunity." *Medical Imaging with Deep Learning*, 2024, https://openreview.net/forum?id=LVRhXa0q5r.
questions: questions: **Questions**

1. After reading the paper several times, I am yet to understand how Cufit actually works. There is no section dedicated in the text that explains this.

2. Why are methods like LoRA that are very frequently used for doing PEFT in transformer-based models and datasets like CheXpert excluded from the analysis?

Please address the **Weaknesses** section for more questions.
limitations: limitations: Please address the **Weaknesses** section.

Comments: title: Thank you for comments!
Thanks for the Rebuttal
Thank you for comments!
Check rebuttal
Check rebuttal
Check rebuttal
Thank you for all comments!
comment: Dear Reviewer tyjY,

Thanks for raising the score rating. We really appreciate your feedback to improve our manuscript. 

Best regards,

The Authors
Thanks for the Rebuttal. My questions have been answered to some extend and I have revised my rating.
Dear Reviewer 1c7f,

We appreciate your efforts in the reviewing process and your positive feedback.

Best regards,

The Authors
I have read the rebuttal and other reviews. I have no more questions. I would like to maintain my initial assessment.
The authors have provided feedback to review comments. Please take time to read the rebuttal and provide response. Thank you.

AC
The authors have provided feedback to review comments. Please take time to read the rebuttal and provide response. Thank you.

AC
The authors have provided feedback to review comments. Please take time to read the rebuttal and provide response. Thank you.

AC
Dear Reviewer T7UR

We would like to thank you again for all your constructive feedback. We will update our final version accordingly. Additionally, we will continue our work on multi-label cases as future work.

Best regards,

The Authors
Thanks a lot for the timely detailed rebuttals. I appreciate the authors' clarifications. Most of my concerns have been solved. My only left concern falls in the multi-class settings, as in healthcare, most cases are multi-label cases since patients may have different diseases, which will somewhat limit the paper's contributions and interests. Thus, sorry that I cannot increase my score more, while I clearly support this paper is worth a weak acceptance.