A Simple and Provable Approach for Learning on Noisy Labeled Multi-modal Medical Images

Authors: nan  wang
zonglin  di
houlin  he
qingchao  jiang
xiaoxiao 2024 li
Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-07 11:04:14

Abstract: Deep learning for medical image classification needs large amounts of carefully labeled data with the aid of domain experts. However, data labeling is vulnerable to noises, which may degrade the ac curacy of classifiers. Given the cost of medical data collection and annotation, it is highly desirable for methods that can effectively utilize noisy labeled data. In addition, efficiency and universality are essential for noisy label training, which requires further research. To address the lack of high-quality labeled medical data and meet algorithm efficiency requirements for clinical application, we propose a simple yet effective approach for multi-field medical images to utilize noisy data, named Pseudo-T correction. Specifically, we design a noisy label filter to divide the training data into clean and noisy samples. Then, we estimate a transition matrix that corrects model predictions based on the partitions of clean and noisy data samples. However, if the model overfits noisy data, noisy samples become more difficult to detect in the filtering step, resulting in inaccurate transition matrix estimation. Therefore, we employ gradient disparity as an effective criterion to decide whether or not to refine the transition matrix in the model’s further training steps. The novel design enables us to build more accurate machine-learning models by leveraging noisy labels. We demonstrate that our method outperforms the state-of-the-art methods on three public medical datasets and achieves superior computational efficiency over the alternatives.

Main paper: 
 introduction: 
 Medical image classification plays a vital role in clinical treatment tasks and diagnosis applications [29, 38, 39] . The success of these classification tasks is contingent upon learning highly discriminative representations using deep neural networks from a huge amount of carefully labeled data with the assistance of domain experts. However, labeling samples by medical experts for medical image classification is expensive, and collecting labels may suffer from the noises [17] . For instance, as demonstrated in Fig. 1 , the malignant histology lymph node images exhibit colors and structures that are strikingly similar to benign ones. This indicates that noisy annotations are an unavoidable part of medical image processing in real-world settings.With the development of deep learning, various methods have been widely used in medical image classification [3, 6, 28] to solve the issue of noisy labeling. There exist two general resolving directions, one is to collect more data [7, 34] , although data collection methods are typically affected by noisy labels. Another way is to fully utilize clean labels while ignoring the noisy labels [11, 12, 18, 36] . Despite making better annotations, these methods are over-complex in the architecture design compared with vanilla classification training, thus becoming sensitive to outliers. Equally, simply and provably processing noisy labeled medical data benign malignant malignant benign malignant benign malignant benignFigure 1 : Examples of typical confusing malignant and benign images from skin lesions and histology lymph nodes, which are prone to inaccurate labeling during the practical annotation process.is significant to assist doctors in diagnosis, few researches have been conducted in medical image classification. Specifically, the existing state-of-the-art method RMIC [38] introduces a complicated teacher-student model that has two carefully designed distillation modules. This is problematic as it costs more resources and time to yield a model. Though the issue can be overcome by optimizing the speed and enhancing the hardware configuration, it is still better to have a lightweight method without sacrificing prediction accuracy. Improving model efficiency is equally important as maintaining model accuracy. Apparently, many clinical applications require realtime decision [23] to allow human-in-the-loop, while a complex model cannot meet such a requirement. Besides, sophisticated deep learning models for healthcare are challenging to install and update on power-constrained devices [21] . Additionally, it is critical to employ simple noisy label training, while the correctness is provable. Such methods have merely been studied so far in medical image classification.Despite their extraordinary effectiveness, it has been repeatedly observed that the performance of these methods is easily influenced by the bias of the training set in complex real-world scenarios [38] . For instance, such bias could come from the variations of disease patterns. A typical issue is that these data biases can cause noise labeling problems. Therefore, an important topic in the realm of medical image analysis is handling multimodal noise. As far as we know, recent research has used cross-modal weak supervision by utilizing convolutional neural networks (CNN). While various previous cross-modal weak supervision methods have shown promise in medical applications, the research conducted to date has been on generating weak labels in an application-specific manner [5] . There is no theoretical analysis that describes the performance expectations of the resulting models, which would support any of these application-specific cross-modal weak supervision approaches. Additionally, these methods can only handle MRI or CT, and the generalizability needs to be improved.By analyzing the existing noisy label learning work for medical image classification, we find that most of the existing methods focus on detecting and utilizing single-mode clean labels only. Nevertheless, these methods are heuristic without provable properties and require much more training parameters than vanilla classification model [18, 21, 23] . Noticing the gaps, we contribute as follows:• We propose a simple and provable noisy label learning strategy that can leverage the labels of clean and noisy data. Specifically, we estimate a transition matrix to correct model outputs to fit the observed labels. • We theoretically prove that our method achieves the same optimal as learning with all clean labels on expectation. To overcome the model overfitting noisy data, we employ gradient disparity as a criterion so that the fixed transition matrix can be accurately modeled by exploiting the noise.• Extensive experiments demonstrate the superior performance of the proposed framework. Our method also has been extensively validated on three public medical datasets with noisy labels, notably outperforming state-of-the-art methods.The goal of our work is to design a simple method for medical image analysis (see Fig. 2 ), while its correctness is provable. The overall training pipeline of our method is still described in Algorithm 1, Motivated by the essential theoretical guarantee in Theorem 3.1, we focus on applying the simple loss forward correction to medical tasks. However, the existing related works [25] typically assume the transition matrix 𝑇 is known, which is impractical in medical applications. Medical image annotation contains complex human decisions, thus the real 𝑇 is difficult to model. To estimate transition matrix 𝑇 for forwarding correction, we first use label prediction by noisy label filter (NLF) (see Section 4.2.1) to distinguish clean samples from noisy samples that serve as the input for training the network. Then, the gradient disparity is applied as a stopping criterion to determine if the matrix is fixed. Finally, the transition matrix is applied to give different confidences to the clean and noisy labels, adapted to the noisy (see Section 4.2). 
related work 2.1 deep learning with noisy labels: 
 To mitigate the impacts of noisy labels on deep learning, researchers proposed approaches to increase the deep neural networks' robustness in recent years.Attempts have been made to evaluate the training labels' correctness. The concept of a supervisor model, MentorNet, is introduced by Jiang et al. [11] . MentorNet is initialized to approximate a predefined sample weighting scheme. During training, it learns with the base deep learning network and re-weights the sample labels based on their reliability. However, the ubiquity of ambiguous hard samples makes the method difficult for medical images, since the images of two classes can be pretty similar. Other than that, other approaches tend to gradually correct the target labels according to the model's prediction. Reed et al. [27] proposed the bootstrapping method, considering the training label confidence by evaluating the model prediction consistency on similar input data. Instead of trusting all training labels to be ground truth, the model's current prediction also plays an important role in deciding the model's label targets. Based on the bootstrapping approach, Iscen et al. [10] penalizes prediction divergence between samples with similar features via an additional consistency loss layer. However, this method introduces additional computational costs to determine the similarity and weighted predictions. It is usually costly for medical image datasets, which are usually large in image feature dimensions. Other than that, the samples with noisy labels are not utilized.Meanwhile, some methods increase the training loss robustness to noisy labels. Sukhbaatar et al. [30] allows the model to capture the label transition matrix by introducing a constrained linear noise layer, handling both label flips and outliers. Patrini et al. [25] proposed two loss correction procedures, which are independent of the neural network's application and architecture. With an assumption of possessing a known label transition matrix T, the loss can be corrected by multiplying with the inverse of T. However, these methods require a certain amount of clean data ahead of time, and the transition matrix is hard to capture due to the complexity of medical image data. Being independent of any prior information, [13] filters noisy labels and trains on the labels with high confidence by selectively applying negative and positive learning. We first input the noisy dataset into our network and divide all samples into noisy and clean groups. To tackle the challenge of noisy label detection, a noisy label filter (NLF) divides all the training data by leveraging the predicted cross-entropy loss of our network. Noisy data are framed by red lines, and their labels need to be corrected. Then, we design a transition matrix that corrects model predictions based on the partitions of clean and noisy data samples. To prevent overfitting of the model to noisy data, we utilize gradient disparity (GD) as a stopping criterion to determine whether to fix the transition matrix 𝑇 in further model training steps.Labeling medical images is expensive and time-consuming as it requires human experts to manually classify them. The data source can also inevitably suffer from noises [17] . Some ambiguous images may confuse exports and some large-scale datasets require automation tools to extract labels, resulting in a certain level of wrong annotation [38] . To address such an issue, many researches were conducted. Some researchers focus on identifying noisy data from clean datasets. Dgani et al. [4] trains a stochastic matrix to determine the likelihood that a label is noisy. Zhu et al. [41] proposes a label quality evaluation criteria to only feed preferable data to optimize the network's parameters. Kim et al. [12] developed the FINE framework to filter out noisy data based on their eigenvectors.Rather than implementing a noisy data filter, some approaches also consider utilizing them to enhance feature detection, and meanwhile, suppressing their influence by re-weighting. Xue et al. [37] designs an iterative learning framework to identify wrongly annotated data and hard samples. A module is also developed to consider the usefulness of those data by assigning appropriate weights. Xue et al. [38] introduces the RMIC method, which involves a co-training framework. The framework includes two student-teacher models, which classify clean and noisy labels independently. Their classifications are fed to one another as new inputs in the next training cycle. Within a student-teacher network, the clean and noisy data are respectively considered by four losses. To increase the teacher network's robustness to noisy labels, its weight is updated by the student network's exponential moving average. The noisy labels are also utilized by a self-supervising learning strategy to improve the network's feature learning. Liu et al. [18] addresses the "noisy supervised domain adaptation" problem for medical imaging segmentation with S-CUDA, which also contains two peer networks.The high-confident noisy labels are identified and corrected by the network's predictions for future reuse.K-mean clustering is one of the most widely used data mining algorithms, grouping up samples with the closest mean. In recent years, many efforts have been made to improve the K-mean algorithm efficiency. Leng et al. [15] chooses cluster initial points by comparing the data point distances with a small threshold. It refines the clusters and decides whether to merge them with an influence factor to measure the similarities between the two clusters. Napoleon and Lakshmi [22] proposes a more efficient K-mean clustering approach of two phases. The clustering centroids are initialized in the first phase, and data points are assigned and clusters are finalized in the second phase. Rauf et al. [26] further improves the clustering efficiency by first classifying data into K arrays of the same size. The clusters are then finalized in the second phase by computing and comparing the distance between data points to the cluster means. 
preliminary 3.1 problem setting: 
 We denote 𝒙 ∈ X as an image sample, ỹ ∈ {𝒆 𝑖 |𝑖 ∈ [𝐾]} is its observed label but possibly noisy (i.e., incorrect), a.k.a corrupted label. Here, 𝐾 is the number of classes, 𝒆 𝑖 is a 𝐾-dimensional onehot vector, whose 𝑖-th entry is 1 and else are 0s. The ground truth label 𝒚 ∈ Y of the sample is unknown. Under these setting, we train a classifier 𝒇 𝜃 to fit the input images X to their noisy labels Ỹ, we aim to optimize the our classifier 𝑓 𝜃 with {X, Ỹ}. The expected risk is defined asEQUATIONwhere L is cross-entropy loss, and we aim to find 𝜃 * := arg min R (𝒇 𝜃 ).Noise. The process of label corruption can be represented by randomly flipping a true label 𝒚 to ỹ with probability 𝑝 ( ỹ|𝒚). Thus, we observe samples from distribution 𝑝 (𝒙, 𝒚), where 𝑝 (𝒙, 𝒚) = 𝒚 𝑝 ( ỹ|𝒚)𝑝 (𝒚|𝒙)𝑝 (𝒙). Such a transition can be expressed as a Markov transition matrix 𝑇 ∈ [0, 1] 𝐾 ×𝐾 , with𝑇 𝑖 𝑗 := 𝑝 ( ỹ = 𝒆 𝑗 |𝒚 = 𝒆 𝑖 ), such that 𝒑 ỹ |𝒙 = 𝑇 ⊤ 𝒑 𝑦 |𝒙 .Risk correction is a simple yet effective approach in learning on label noise [25, 30, 31] , among which forward correction dominates the performance in practice [25] . Without risk correction, training a neural network results in a prediction for the corrupted labels p ( ỹ|𝒙). Forward correction corrects the model prediction using transition matrix 𝑇 . For classification based on minimizing crossentropy, we have the forward corrected loss:EQUATIONEQUATIONObviously, forward correction simply replaces the prediction trained on the noisy label p (𝒚 = 𝒆 𝑗 |𝒙) with 𝑇 𝑖 𝑗 p (𝒚 = 𝒆 𝑗 |𝒙).Optimal in Forward Correction. In our context, the model is optimal in the sense that the minimizer of the corrected loss task coincides with the minimizer of the supervised task on true labels as if the perfect true labels 𝒚 can be observed. Theorem 3.1. (Equal optimal [25] ) Given the forward corrected loss defined in Eq. 4 and 𝑇 is nonsingular, the minimizer of the corrected loss under the noisy label distribution is the same as the minimizer of the original loss under the true label distribution:EQUATIONProof. Consider a link function 𝜓 : Δ 𝐾 -1 ↦ → R 𝑐 , which is invertible, the softmax function is the inverse link function in cross-entropy loss L 𝑐𝑒 = -log 𝑝 (𝒚 = 𝒆 𝑖 ) 𝑖 = -log exp(𝑊 (𝑛) 𝑖 • 𝒙 (𝑛-1) +b (𝑛) 𝑖 ) 𝑐 𝑘=1 exp(𝑊 (𝑛) 𝑘 • 𝒙 (𝑛-1) +b (𝑛) 𝑘 ), where 𝑛 is the index of layers. Let 𝒉(•) represent the features that pass through the softmax function. With proper composite loss properties of cross-entropy, the minimizer satisfies The proof follows [25] . Notice that:EQUATIONEQUATIONEQUATIONwhere 𝜙 -1 = 𝜓 -1 • 𝑇 and 𝜙 = (𝑇 -1 ) • 𝜓 . Eq. 9 is the proper composite with link 𝜙. Finally, from Eq. 9, the loss minimizer is𝒉 * = arg min 𝒉 E 𝒙, ỹ L 𝜙 (𝒚, 𝒉(𝒙)) = 𝜙 (𝑝 ( ỹ|𝒙)) = 𝜓 (𝑇 -1 𝑝 ( ỹ|𝑥)) = 𝜓 (𝑝 (𝒚|𝒙)) □ 4 Equations 4. 
estimating transition matrix 𝑇: 
 The main purpose of this work is to estimate the transition 𝑇 for forward correction described in Section 3.2. To this end, the key step is to estimate the probability of a sample in class 𝑖 is labeled as 𝑗, 𝑇 𝑖 𝑗 = 𝑝 ( ỹ = 𝒆 𝑗 |𝒚 = 𝒆 𝑖 ). We propose to estimate the 𝑇 𝑖 𝑗 from the sample distribution of the training dataset. Given that corrupted labels ỹ ∈ Ỹ are observed for 𝑁 training samples if their corresponding true labels were known, we can count the number of samples 𝑁 𝑖 in class 𝑖. Hereby, the estimated transition matrix is expressed as𝑇 𝑖 𝑗 = 𝑝 ( ỹ = 𝒆 𝑗 |𝒚 = 𝒆 𝑖 ) ≃ | { ỹ𝑛 =𝑗 |𝑛∈ [𝑁 𝑖 ] } | 𝑁 𝑖for 𝑖, 𝑗 ∈ 𝐾, where 𝐾 is the number of classes.Patrini [25] assumes that the transition matrix 𝑇 is known ahead of time. However, it is very hard to estimate such a matrix due to the complexity of medical images. To estimate transition matrix 𝑇 , we first filter out clean data, which are believed to be correctly labeled in the dataset. Inspired by Xue et al. [38] , we use an NLF to separate clean and noisy labels (a.k.a. wrong labels) based on data point-wise classification loss. More specifically, we compute the cross-entropy loss L ( Ŷ, Ỹ) at the end of every epoch, where Ŷ is the prediction result/are the prediction results from 𝑓 𝜃 (X). In our experiments, we use the cross-entropy loss as L. Next, we input the loss into a noisy label filter (NLF) to predict {X, Ỹ} into clear {X 𝑐 , Ỹ𝑐 } and noisy {X 𝑛 , Ỹ𝑛 } samples. In detail, the NLF employs a two-component Gaussian Mixture Model (GMM) to cluster the samples into two groups based on their point-wise cross-entropy losses (i.e., , ℓ 𝑖 = p ( ỹ𝑛 |𝒙 𝑛 )) for data point 𝑛.The NLF method only selects the most confident samples as clean samples at the initializing stage, and the confidence threshold gradually decreases as the model becomes more robust. With a high confidence threshold at the beginning, the consistency of the transition matrix 𝑇 is maintained to the greatest extent. Meanwhile, we added gradient disparity (GD) as a stopping criterion to determine whether or not we should stop updating the transition matrix T.The key challenge exposed is the true label being unknown, thus the number of samples 𝑁 𝑖 within a given true label 𝑖 are unavailable. To tackle this issue, we propose an unsupervised pseudo-labeling strategy to estimate 𝑁 𝑖 . Specifically, we warm up the model without correction for 𝜏 steps and then perform clustering on the data feature representations to form 𝐾 clusters, then assign each cluster a label based on the majority voting on a subset of clean samples within each cluster.We first extract informative low-dimensional representations from the input medical images. Convolution neural networks (CNNs) show their efficiency in extracting the feature. Further, a recent work [16] shows that extracted features are representative and robust even when being trained on label noise. In our method, we use a ResNet-18 [9] as the feature extractor and one linear layer as the classifier training on the observed noisy labels. Then, we perform K-means clustering [26] on the extracted features to split the training data into 𝐾 clusters.Pseudo Labels to Clusters. Towards estimating 𝑁 𝑖 , one step further is to assign a pseudo label to each cluster. For each cluster 𝐶 𝑖 , 0 ≤ 𝑖 < 𝐾, we split the data samples (X 𝐶 𝑖 , Y 𝐶 𝑖 ) in cluster 𝐶 𝑖 into clean (X 𝐶 𝑖 𝑐 , Ỹ𝐶 𝑖 𝑐 ) (correctly labeled) and noisy (X 𝐶 𝑖 𝑐 , Ỹ𝐶 𝑖 𝑛 ) (wrongly labeled) samples using the NLF method given in Section 4.2.1.The label of each cluster depends on the majority voting on the prediction of clean data. To achieve more reliable pseudo labeling, we only choose the top 60% of the clean data given the noise level in medical data annotation is typically small high (i.e., ≤ 40%) for mapping each 𝐶 𝑖 to a non-overlapped pseudo-class. EQUATIONFor Estimate 𝑇 𝑖 𝑗 , the model will learn the clean data in the early epochs based on the memorization effect. Thus, the model tends to find the noisy label and predict the correct transition matrix 𝑇 . Next, the loss from the clean and noisy labels can differ greatly. The NLF can filter the noisy ones. Furthermore, the clustering can distinguish between clean and noisy pairs. These three techniques can help the estimated transition matrix converge to the real one.After we have estimated the transition matrix 𝑇 , we apply the forward correction to the output from the linear layer 𝑓 (𝒙) by multiplying 𝑇 to 𝑓 (𝒙) as 𝑇 𝑓 (𝒙). 𝑇 plays the role of re-weighting 𝑓 (𝒙), making the model give different confidentiality to the clean label and noisy label and also adapt to the noise. The optimization goal is to optimize classifier 𝑓 that minimizes Eq. 4. 
experiment 5.1 experimental setting: 
 We implement our approach with PyTorch [24] 1.8.0 and Torchvision 0.9.0 with 2 NVIDIA 2080Ti GPUs. The models are trained using Adam optimizer [14] by setting coefficients of Adam 𝛽 1 = 0.9, 𝛽 2 = 0.999, and 𝜖 = 10 -8 . The learning rate is 0.0001 and remains constant during the whole training process. The batch size and total epochs are set to 64 and 50, respectively. In addition, we set the warmup step 𝜏 = 10 as the default value. [1, 32] . The dataset is a binary classification for malignant and benign, containing lowresolution images of lymph node sections with about 30, 0000 markers extracted from digital histopathological scans. We randomly select 6200 and 800 images from the training images as our training data and test data, respectively. All the images are resized to the size of 224 × 224.Dataset. This dataset [33] contains 112,120 frontal-view CXR images from 32,717 patients. Each image is labeled with 14 possible pathological findings that are automatically mined from the text reports. As clean testing data, we employed 1,962 manually labeled images [20] . We resized all the skin images to the size of 224 × 224 and normalized each image by subtracting the ImageNet mean and std.The performance of our method under different unsupervised loss weights on the histopathologic datasets. To demonstrate the effectiveness of our algorithm, we compare Pseudo-T correction with five state-of-the-art noisy label learning methods, including:(1) MentorNet (2018) [11] . MentorNet proposes a neural network to supervise the training of the base deep networks. They used a group of clean data to weigh each training data. This method learns a data-driven curriculum dynamically with the network.(2) Co-teaching (2018) [8] . This method trains two deep neural networks to teach each other given every mini-batch by selecting some data of possibly clean labels. Secondly, the two networks communicate with each other about what data should be used for training. Then, each network updates itself and back-propagates the data chosen by its partner networks.(3) ELR (2020) [19] . A semi-supervised learning technique to generate targe probabilities based on the model outputs. Then, ELR develops a regularization term that indirectly prevents memorizing the erroneous labels by guiding the model towards these targets.(4)JoCoR (2020) [35] . A learning paradigm used in a co-training network structure. It considers and trains the two networks as a whole, and aims to reduce their diversity. The two networks first make predictions on the same mini-batch and calculate the joint loss to maximize the agreement between the two classifiers. The small-loss examples are then selected to update the two networks simultaneously; consequently, the two networks become similar.(5) NIB (2021) [40] . A plugin module that separates hard samples from mislabelled samples to further increase the network's performance under a noisy environment. The module consists of a probability transition matrix to generate accumulative soft labels. The divergence between the soft labels and the network's predictions is calculated. With less divergence, the hard samples are identified. Zhang et al. prove that NIB brings improvement to the JoCoR framework with experiments.(6) RMIC (2022) [38] . A collaborative training paradigm with global and local representation learning. They use a self-ensemble model with a noisy label filter to select the clean and noisy samples. Then, the clean samples are trained and applied to eliminate the disturbance from imperfect labeled samples. This method uses a teacher-student co-training strategy for local and global knowledge distillation.Since the data is more imbalanced and higher proportion of hard samples than the lymph node classification task, it should be noted that the skin lesion classification data is more challenging to use. In the malignant vs. benign classification task on the ISIC dataset, Pseudo-T correction significantly outperformed the alternative methods. From Table 1 , it is observed that when there is a 0.05 noise ratio setting, the poor sample utilization rate and incorrect hard sample identification in the mild scenario make co-teaching ineffective, while the biased sample selection and down-weighted hard samples in the self-paced Mentornet model hindered its effectiveness. The RMIC discovered a slight increase in accuracy when the noise ratio was low, but when the noise ratio approached 0.4, the performance drastically declined because their cross-entropy loss on small loss data is not reliable in this situation. Compared with our methods, the RMIC led to a minimal difference in accuracy between decentralized training and the baseline, using a noisy dataset led to a much larger difference between the two. The results show that our method can successfully handle noisy labels since the transition matrix can effectively utilize the hard samples and the noise label filter can filter out the noise labels.To evaluate the generalization ability, Table 1 also presents the testing accuracy (summarized from three independent runs) of Pseudo-T correction and comparison methods on different noise levels. Although the RMIC method achieves the best, 92.75% value at noise equals 0.1 on the HCDD dataset, this method is based on a co-training strategy, so it may be challenging to employ their cross-entropy loss on small loss data. This plausibly implies that Pseudo-T correction would be very helpful and practical in clinical applications by fully using the information of noise data.In addition to successfully handling label noise, our method is lightweight and easy to implement on resource-limited devices. While using ResNet18 as our base model, Pseudo-T correction contains 11.2 million trainable parameters, which is only 1/4 that of RMIC and Co-teaching and around half of the other methods. Compared to RMIC [38] , our method gains comparable results to RMIC while being simple and provable.Results on NIH Chest X-Ray Dataset. The NIH Chest X-Ray dataset is a real-world dataset with real noise. We present the comparison in test accuracy with state-of-the-art methods on the NIH chest X-ray dataset in Table 2 . From this Table 2 , we can observe that Pseudo-T correction still outperformed the baseline (e.g., Cross entropy and Co-teaching) and other methods. For instance, our We report the testing accuracy distinct settings. method achieves 2.4% accuracy gains over ELR. Compared to RMIC, our method achieves better performance with 0.5% accuracy-score gains. We explain this to be that the sample selection is difficult because of the tiny noise proportion and unbalanced sample distribution of the 14 diseases. Overall, this experiment shows the robustness of our method in a real clinical context. Thus, our method is practical with considerably better adaptability.5.4.1 Sanity Check on the Estimated Transition Matrix 𝑇 . The transition matrix 𝑇 is an essential element in improving the classification performance. We conducted multiple experiments to prove the correctness of the transition matrix. For sanity check, we visualize the convergence of transition matrix 𝑇 on the ISIC dataset when the noisy ratio is 0.05 and 0.1. The investigations of 𝑇 are presented in Table 3 . As shown in Table 3 , our method achieves ACC of 0.1 noisy ratio gains over 0.35% compared to the without Pseudo-T correction. In addition, compared to the without transition matrix correction under a 0.4 noisy ratio, Pseudo-T correction added after it achieves significant performance improvement, especially with the increase of 4.7% ACC. Experimental results indicate that the minimization of the corrected loss under the noise distribution corrected by the correction matrix is the same as the minimization of the original loss under the true label distribution. It demonstrates the advantages of using Pseudo-T correction to reduce the risk of training a classification model with incorrect information because the chances of randomly selecting a complementing label that is not a real label are high.To evaluate the stopping criteria (GD) on the ISIC dataset and the HCDD dataset, we report the results in Fig. 3 . It can be seen from Fig. 3 that the proposed Pseudo-T correction produces overall good results under a distinct noise ratio. As shown in Fig. 3 , we report the comparison results with vs. without stopping criteria (GD). From Fig. 3 , the "with GD" achieves the improve-ment of 4.07% compared with "without GD" under 0.4 noise rate on the ISIC dataset. Equally, under the noise rate is 0.05, our method achieves a good result with GD. At a noise rate of 0.4, "with GD" increases the accuracy over "without GD" by 1.3% on the HCDD dataset. Fig. 3 suggests that our method using stopping criteria produces the overall better performance, compared with that without stopping criteria on ISIC and HCDD datasets. This means that our method would be very helpful in noisy data by using gradient disparity as an effective criterion to avoid overfitting noisy data with iterative methods. Similarly, it can be seen from Fig. 4 (b) that our method yields consistently better results within the range of [0.05, 0.1], when 𝜏 is equal to 15. This Fig. 4 (b) also shows that, with 𝜏 = 10, Pseudo-T correction achieves the overall best performances under a noise ratio of 0.2. Also, these results imply that the warmup epoch is an important parameter for maintaining model stability.We further analyze how the warmup epoch 𝜏 on NIH chest X-ray dataset. We ranged 𝜏 ∈ [5, 10, 15, 20] . As shown in Fig. 5 , we can observe that the best selection of 𝜏 is set to 15. From 5 to 15, increasing 𝜏 can improve the classification performance. The performance slightly drops when 𝜏 is set to 20. Thus, we report the performance of the proposed method by setting 𝜏 to 20 on NIH chest X-ray dataset, unless otherwise specified. We find that increasing the 𝜏 may not always improve the performance yet brings extra computational cost. It also demonstrates that increasing 𝜏 is not always the best choice for architectural design. 
conclusion: 
 In this work, we present a simple and provable method by utilizing noisy labels for medical image analysis. The proposed method does not rely on relabeling the noisy labeled data and directly uses noisy labeled data and a lightweight network to promote the learning of robust representation features. Notably, to tackle the realistic but ignored issue of incomplete information in noisy labels, we designed a transition matrix, that corrects model predictions. Furthermore, to avoid underfitting or overfitting deep neural networks trained with iterative methods, resulting in inaccurate transition matrix estimation. Therefore, we employ gradient disparity as a criterion to decide if fixing the transition matrix in the further model training steps. Extensive experiments on three challenging medical datasets, including dermoscopic images, histopathology slide images, X-Ray, about medical image classification tasks with random noise and inter-observer variability, demonstrate that our method obtains state-of-the-art performance. In the future, we will fuse multi-class classification tasks to address the data heterogeneity issue and apply our methods to further improve performance on more datasets. 
 
 Appendix:  


Decision: This paper works on noisy label learning for medical image classification. The rebuttal partially addressed the reviewers’ concerns, and this paper was rated as 1 weak accept, 1 borderline accept and 1 borderline reject. Although some concerns on the technical novelty remains, I tend to agree more on the merits of this paper on simple and provable approach for noisy label learning. Thus, I tend to recommend accept after reconciling the reviewers’ comments. I do suggest the authors revise the manuscript significantly by taking all the reviewers’ comments into consideration.

Reviews: review: paper_summary: paper_summary: The Pseudo-T correction method provides a simple, yet effective, approach to handle noisy labels in medical image classification. It leverages a transition matrix to correct predictions and employs gradient disparity to prevent overfitting. The method demonstrates superior performance on multiple datasets and is computationally efficient, making it a valuable tool for clinical applications.
strength_weakness: strength_weakness: 1. The method presents a novel and theoretically sound approach to a significant problem in medical image classification.
2. Experimental results show strong performance improvements and robustness across multiple datasets.
3. The paper is clear and well-structured, making the methodology accessible.
limitations: limitations: 1. Limited novelty in broader context, as the use of transition matrices for label correction is not entirely new.
2. More comprehensive comparisons with additional recent approaches[1][2] would strengthen the validation.
[1] Hongxin Wei, Lei Feng, Xiangyu Chen, and Bo An. Combating noisy labels by agreement: A joint training method with co-regularization.
[2] Zhang, Z., Li, Y., Wei, H., Ma, K., Xu, T., & Zheng, Y. Alleviating noisy-label effects in image classification via probability transition matrix.

3. Including real-world implementation examples would enhance the practical relevance.
paper_summary: paper_summary: This paper proposes a novel method for medical image analysis, i.e., Pseudo-T correction, to leverage the labels of clean and noisy data. Pseudo-T correction first splits clean and noisy samples via a noisy label filter, and enhances forwarding corrction by taking advantage of transition matrix estimation to fit the observed labels. In the given experimental settings and with the three medical image datasets with noisy labels, the proposed model shows improvements over SOTA methods.
strength_weakness: strength_weakness: 1）The proposed Pseudo-T correction is technically sound by estimating the transition matrix and updating label probabilistic modeling to learn discriminative representation features.

2）Experiments on three real-world datasets verify the effectiveness and practicability of the proposed Pseudo-T correction method.

3）This paper may inspire significant practical applications to medical image analysis tasks, in terms of efficiency and universality of noisy label training.
limitations: limitations: 1) The paper is an incremental method so that the novelty of this proposed method is limited, which estimates a transition matrix to correct model outputs and then employs gradient disparity as a criterion to assign different confidences to the clean and noisy labels in the further model training steps.
2) It is a bit unclear of the motivation behind the technical design of the proposed model. It is claimed that this paper finds that most existing heuristics only detect and utilize single-modal clean label (Line 134). However, I am confused with how this paper solve the “single-modal” dilemma, and provide accurate predictions on multi-modal medical images.  
3) It could be more convincing to vary the noise rate from 0 to 0.9 to demonstrate the robustness to noise
4) Figures can be further improved for better readability. For example, in Figure 2, for noisy data (green) and clean data (yellow), the explanation of the circles framed by red lines is missing.
5) The presentation of this paper needs improvement:

        Line 134: Nevertheless,... -> Moreover,...
        Line 154: ...three medical image noisy labels with public datasets -> ...three public medical image datasets with noisy labels
        Line 267: K-Mean Clustering -> K-Means Clustering
        Line 384-389: The label of the loss minimizer fomulation is missing
        Line 390: The numbering and notation in the formulas are confusing and some marks are missing. Clear and consistent notation is essential for readability and understanding.
        Line 410: The indentation in the flowchart of Algorithm 1 is difficult to follow. What is the output of the training process?
paper_summary: paper_summary: This paper focuses on noisy label learning for medical image classification. The main components are label transition matrix estimation and label correction. The framework is easy to follow. Extensive experiments and ablation studies on three public datasets have been conducted for evaluation.
strength_weakness: strength_weakness: Noisy label learning is of great value in clinical applications. The paper is easy to follow.
limitations: limitations: 1. Nothing related to multi-modal medical images. It is inconsistent with the title.
2. Limited innovation. Both noisy label identification and label correction follow standard operations. The main content namely estimating the transition matrix largely depends on noisy label identification.
3. Simple noise settings. Random flipping is not realistic.
4. Following comment 3, only binary classification is too simple. Noisy multi-label learning is more interesting and more realistic noisy settings are expected. For instance, instance-dependent noise.
5. Insufficient comparison. More advanced noisy label identification methods should be included for comparison.
6. Experiments on multi-label classification are needed.