Label Decoupling and Reconstruction: A Two-Stage Training Framework for Long-tailed Multi-label Medical Image Recognition

Authors: jie  huang
xiaoqin  zhang
guodao  zhang
huiling  chen
zhao-min  chen
yisu  ge
lusi  ye
Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-03-29 03:02:02

Abstract: Deep learning has made significant advancements and breakthroughs in medical image recognition. However, the clinical reality is complex and multifaceted, with patients often suffering from multiple intertwined diseases, not all of which are equally common, leading to medical datasets that are frequently characterized by multi-labels and a long-tailed distribution. In this paper, we propose a method involving label decoupling and reconstruction (LDRNet) to address these two specific challenges. 
  The label decoupling utilizes the fusion of semantic information from both categories and images to capture the class-aware features across different labels. 
  This process not only integrates semantic information from labels and images to improve the model's ability to recognize diseases, but also captures comprehensive features across various labels to facilitate a deeper understanding of disease characteristics within the dataset.
  Following this, our label reconstruction method uses the class-aware features to reconstruct the label distribution. This step generates a diverse array of virtual features for tail categories, promoting unbiased learning for the classifier and significantly enhancing the model’s generalization ability and robustness. Extensive experiments conducted on three multi-label long-tailed medical image datasets, including the Axial Spondyloarthritis Dataset, NIH Chest X-ray 14 Dataset, and ODIR-5K Dataset, have demonstrated that our approach achieves state-of-the-art performance, showcasing its effectiveness in handling the complexities associated with multi-label and long-tailed distributions in medical image recognition.

Main paper: 
 introduction: 
 In recent years, deep learning has achieved significant accomplishments in the field of computer-assisted diagnosis [6, 9, 41, 45] . Medical image recognition, in particular, plays a crucial role in the clinical diagnosis of diseases. The ability to automatically identify specific diseases or conditions in medical images can significantly assist physicians in clinical diagnosis, reduce their workload, and improve diagnostic efficiency. However, in real-world clinical applications, medical images (such as CT scans, X-rays, etc.) often contain multiple diseases, thereby inevitably exhibiting multi-label characteristics. Additionally, due to the varying incidences of diseases, medical image datasets typically show a long-tail distribution, where common cases (head categories) dominate a large number of samples while rare cases (tail categories) are represented in smaller quantities. Therefore, the medical image recognition tasks in real scenarios are confronted with two significant challenges: multilabel and long-tail distribution (refer to Fig. 1 ). Without proper handling, trained models may develop a bias towards head categories, presenting substantial challenges to the task.Considerable efforts have been made towards multi-label medical image recognition. Among the existing works on medical image classification, Zhang et al. [49] introduced a triplet attention mechanism combined with contrastive learning to mine effective information from medical images for learning high-quality label embeddings for all disease labels. He et al. [16] employed a knowledge distillation-based approach to enhance the diagnostic performance of multi-label medical image diseases. Zhou et al. [53] addressed the challenge of extending single-label to multi-label scenarios through a cross-domain transfer method. While these methods have addressed the multi-label issue in medical image recognition, they have not adequately considered the long-tail distribution.Other existing methods are dedicated to tackling the long-tail distribution problem in medical image recognition, which can be broadly categorized into re-sampling and re-weighting strategies. Re-sampling methods [2, 3, 28, 33] aim to create a balanced dataset by under-sampling the head categories and over-sampling the tail categories, thereby assisting the model in better learning the less common categories. Re-weighting approaches [24, 37, 48] address the long-tail issue by adjusting the loss function weights for different categories. Specifically, these methods assign higher weights to the tail categories and lower weights to the head categories, encouraging the model to focus more on the tail categories. However, the aforementioned methods are not applicable to data with multi-label long-tail distribution. Specifically, re-sampling methods struggle to balance sampling across categories due to the influence of label co-occurrence, while re-weighting methods are highly sensitive to weight parameters in a multi-label context, impacting the model's generalization ability and robustness.To address the challenges mentioned above, in this paper, we propose a two-stage model based on label decoupling and reconstruction (LDRNet), using class-aware features to reconstruct the distribution of label feature and training an unbiased classifier, to solve the multi-label medical image recognition problem under long-tail distribution. Specifically, in the first stage, we introduce a label decoupling technique that employs prompts and a Self-Attention-based method to decouple class-aware features from the global image features, and training a biased classifier. In the second stage, we begin by assuming that each category follows a Gaussian distribution, and then reconstruct the Gaussian distribution for each category by the class-aware features obtained in the first stage. Next, we employ a reverse sampling strategy to sample from the Gaussian distributions of each category and generate virtual global image features. Finally, we fine-tune the biased classifier with the generated virtual features to correct its bias. Since our method samples from Gaussian distributions for each category, it generates more diverse features for tail categories, further reducing the classifier's overfitting to tail categories, and thereby enhancing the model's generalization ability and robustness. Additionally, due to the scarcity of publicly available multi-label long-tail distribution medical image datasets, we have collected a long-tail distribution multi-label medical image dataset on Axial Spondyloarthritis to foster development in this field, which we plan to make open source in the future. We evaluated our method on this dataset and two publicly available datasets, and the experimental results demonstrate the superiority of our approach. Our contributions can be summarized as follows:• We propose a LDRNet for long-tail multi-label medical image recognition, which operates through a two-stage training process. In the first stage, we decouple features for each label from the global image, and in the second stage, we reconstruct the feature distribution for each label, then using reverse-sampling to generate more diverse virtual features to obtain the unbiased classifier. 2 Related Work 
multi-label medical images: 
 Multi-label classification tasks have now been extensively studied for predicting multiple category labels [8, 27, 29, 42, 51] . Recently, in the medical image recognition field, to learn the common relations among labels in multi-labels, some methods can be divided into these two major categories: 1) Architecture-Network-based methods, which mainly focus on designing various network variants to learn the commonalities between labels. For example, MLGCN [8] , introduces a GCN-based multi-label classification model to capture the relationships between labels; C-Tran [22] utilizes the Transformer to exploit complex dependencies between visual features and labels; CheXNet [46] , integrates CNN and Transformer modules, where CNN provides rich input through bottom-up feature extraction for the Transformer, and the Transformer guides the feature extraction in CNN with its top-down attention mechanism, to boost performance. 2) Loss-based methods, which mainly involve designing or adjusting the loss function for each label to optimize the model's handling of multi-label images, ensuring it can effectively handle the complexities of label interdependencies and varying error costs. Among these, MLSL [48] designs a softmax-based loss function to reduce intra-label and inter-label ranking errors during training, thereby directly optimizing the ranking loss and AUC; ASL [40] introduces a new asymmetric loss, acting differently on positive and negative samples to balance probabilities across various samples. Kobayashi et al. [21] proposes a dual-approach loss, namely the multi-label classification loss for each sample and the label classification loss for each class, to simultaneously improve sample-wise and class-wise discrimination performance, thereby enhancing classification accuracy.In the medical field, due to the different incidence rates of diseases, medical image datasets often exhibit a long-tail distribution [10, 11] , where common cases account for a large number of samples, and rare cases account for a small number of samples. This long-tail distribution leads to model training bias towards head categories while neglecting tail categories, severely impairing the model's diagnostic performance in real-world scenarios. Therefore, balanced recognition of minority categories is necessary to improve diagnostic performance. To address this issue, most previous methods focus on learning each category in a balance manner, these methods can generally be categorized into three aspects. The re-sampling methods, which include under-sampling for head categories [2] and over-sampling for tail categories [33] , focus on constructing a balanced training set. While the re-weighting methods [24, 37] adjust the long-tail distribution by assigning different weights to the samples. Furthermore, recent studies [36, 52] have proposed using a two-stage decoupling approach to address the long-tail issue, achieving notable results. However, these methods are solely applicable to single-label tasks and are not suitable for multi-label tasks, particularly in the context of medical image recognition. Specifically, for re-sampling methods, when a medical image includes both head and tail categories, it is not feasible to achieve balanced sampling through under-sampling or over-sampling strategies, making it difficult to directly be applied. For re-weighting methods, they are highly sensitive to weight parameters in a multi-label context, leading to unstable model training and decreased model robustness.In real clinical scenarios, where there are both commonality among multiple pathological features and rare pathological features, medical datasets are typically both multi-labeled and long-tail distributed. Therefore, we propose a model for multi-label medical imaging under a long-tail distribution, aiming to simultaneously solve the problems of multi-label and long-tail distribution in real-world medical clinical scenarios. Furthermore, this model also enhances the generalization ability and robustness. 
method 3.1 problem setup: 
 Given a multi-label dataset 𝐷 = {𝑰 , 𝒀 } with 𝐶 categories and 𝑁 samples, where 𝑰 represents the input medical images, and 𝒀 = {𝑦 1 , 𝑦 2 , ..., 𝑦 𝐶 } represents the image-level label annotations, with 𝑦 𝑐 ∈ {0, 1} indicating whether category 𝑐 is present in the image ('1' for presence and '0' for absence). Data in real medical clinical scenarios often exhibit a long-tail distribution, where head categories comprise the majority of the data, while tail categories have very few samples. The presence of a long-tail distribution poses unique challenges for multi-label image classification, primarily manifested in learning algorithms tending to optimize performance for categories with a large number of samples while neglecting those with fewer samples. Our objective is to learn a function 𝑓 (𝑰 ) = 𝒀 ′ , to predict the categories of the input image, and this function 𝑓 is not affected by the long-tail distribution.In this paper, we propose a two-stage method, denoted as LDRNet, to address the multi-label recognition task with long-tail distribution in medical images, the overall framework is shown in Fig. 2 . In the first stage, we utilize a Transformer encoder and prompt strategy to decouple class-aware features. In the second stage, we first reconstruct the Gaussian distribution for each category using class-aware features, then apply a reverse sampling strategy to sample and generate virtual multi-label image global features from the Gaussian distribution to fine-tune the classifier from the first stage. Our LDRNet aims to achieve balanced learning of each category by the classifier.Our aim is to use class-aware features to reconstruct the Gaussian distribution of each category, so we need to obtain the class-aware features. Current mainstream operations for feature decoupling include CAM-based [7] and Self-Attention-based [26] methods. Given the complexity and specialization of medical image data compared to natural images, we adopt the Self-Attention-based method, which has better performance. Specifically, we first use the backbone to obtain global image features, then use learnable query embeddings 𝑸 and a Transformer encoder to decouple the class-aware features from the global features. Formally, given the input multi-label medical image 𝑰 , and using the backbone to obtain global image features 𝑿 ∈ R 𝐷 ×𝐻 ×𝑊 :EQUATIONwhere 𝐷, 𝐻,𝑊 represent the channels, height, and width of the features, respectively. 𝑓 𝑏𝑎𝑐𝑘𝑏𝑜𝑛𝑒 (•) represents the learnable neural network, for which we primarily use densenet [19] and resnet [17] .Then, we propose a prompt strategy to decouple the class features. Specifically, we first define the prompting template: "The Categories of the image are {LN}", where "LN" represents the name of the labels, and use CLIP [39] to obtain the word embedding 𝑾 ∈ R 𝑀 ×𝐷 ′ . Then, we use a learnable MLP to map the dimensions of the word embedding to 𝐷. Next, we define a learnable label embedding 𝑳 ∈ R 𝐶 ×𝐷 as the label query, define 𝑷 𝑤𝑜𝑟𝑑 ∈ R 1×𝐷 and 𝑷 𝑙𝑎𝑏𝑒𝑙 ∈ R 1×𝐷 as the learnable indicator embeddings for the prompting word and label. Finally, we fuse the above embeddings and add the learnable position embedding to obtain the query embedding 𝑸 :𝑸 = [𝑾 + 𝑷 𝑤𝑜𝑟𝑑 : 𝑳 + 𝑷 𝑙𝑎𝑏𝑒𝑙 ] + 𝑷 𝑝𝑜𝑠𝑖𝑡𝑖𝑜𝑛 ∈ R (𝑀+𝐶 ) ×𝐷 , (2)where [:] denotes the concatenation operation, 𝑷 𝑝𝑜𝑠𝑖𝑡𝑖𝑜𝑛 ∈ R (𝑀+𝐶 ) ×𝐷 is the learnable position embedding. Once we have the query embedding 𝑸, we then use the Transformer encoder to implement the decoupling of the category features. Firstly, we reshape the global image features along the spatial dimensions and obtain the flattened feature: where 𝑿 𝑓 ∈ R 𝐻𝑊 ×𝐷 denotes the flattened feature, and 𝑓 𝑟𝑒𝑠ℎ𝑎𝑝𝑒 (•) is the reshape operator. Secondly, we integrate the query embedding 𝑸 and flattened feature 𝑿 𝑓 , and use the Transformer encoder to obtain the class-aware features 𝑿 𝑐𝑙𝑎𝑠𝑠 :EQUATIONEQUATIONEQUATIONwhere 𝑓 𝑒𝑛𝑐𝑜𝑑𝑒𝑟 (•) denotes a standard Transformer encoder, which contains a self-attention module and a feed-forward network. 𝑓 𝑠𝑒𝑙𝑒𝑐𝑡 (•) is the feature selection operation, which selects features from 𝑿 ′ at positions corresponding to the label embedding 𝑳. Then, we can obtain the prediction confidence of class-aware feature ŷ:EQUATIONwhere 𝑓 𝑐𝑙𝑠 (•) represents our classifier, and it tends to exhibit bias. Finally, we employ a Binary Cross-Entropy (BCE) loss to constrain feature learning and achieve the disentangling operation:EQUATIONwhere 𝜎 (•) is the sigmoid function, 𝑦 𝑖,𝑐 is the ground truth label of the 𝑐-th category for the 𝑖-th sample, and ŷ𝑖,𝑐 is the predicted logit for the 𝑐-th category of the 𝑖-th sample.Unlike long-tail distribution single-label image recognition tasks, multi-label image recognition cannot employ re-sampling methods to balance the sampling of each category due to the influence of label co-occurrence relationships. Inspired by [1, 36] , we assume that the feature distribution of each category follows a Gaussian distribution, and propose a Gaussian distribution reconstruction method for multi-label medical imaging under long-tail distribution. This method attempts to reconstruct the Gaussian distribution of each label and uses a reverse sampling strategy to sequentially sample features from the Gaussian distributions to generate a virtual global feature of the image. This generated feature is then used to fine-tune the classifier trained in the first stage, thereby "correcting" the classifier's bias. Specifically, for the class-aware features 𝑿 𝑖 𝑐𝑙𝑎𝑠𝑠 ∈ R 𝐶 ×𝐷 of the 𝑖-th image, we divide along the channel dimension 𝐷 into feature vectors for each category :EQUATIONwhere 𝑿 𝑖,𝑐 𝑐𝑙𝑎𝑠𝑠 ∈ R 1×𝐷 represents the decoupled feature of the 𝑐-th category for the 𝑖-th image.Then, we construct a corresponding Gaussian distribution for each category. Unlike single-label images, since multi-label images contain multiple categories without location annotation information, decoupling processes may still yield class-aware features at corresponding positions for categories not present in the image. Simply reconstructing features for categories present in the image would disrupt the original feature distribution. Therefore, we reconstruct both label-0 and label-1 Gaussian distributions for each category.Formally, given the class-aware feature 𝑿 𝑖,𝑐 𝑐𝑙𝑎𝑠𝑠 for the 𝑐-th category of the 𝑖-th image, we calculate the mean and variance for the 𝑐-th category as follows:EQUATIONEQUATIONwhere 𝑗 ∈ {0, 1} corresponds to label 0 or 1. 𝑁 𝑗 𝑐 represents the number of samples for category 𝑐 with label 𝑗, and 𝜇 𝑗 𝑐 and Σ 𝑗 𝑐 represent the mean and variance of features for category 𝑐 with label 𝑗. Then, based on the calculated mean and variance, we construct the corresponding Gaussian distributions, whose probability density functions can be written as:EQUATIONwhere 𝑗 ∈ {0, 1} corresponds to label 0 or 1, 𝑿 𝑗 𝑐 represents the classaware feature vector with category 𝑐 label 𝑗 , and 𝑓 𝑗 𝑐 (•) represents the probability density functions for category 𝑐 with label 𝑗. Next, we employ a reverse sampling strategy for each category to achieve feature re-sampling. The sampling probability for each category is as follows:EQUATIONwhere 𝑝 𝑐 denotes the probability of category 𝑐, 𝐹 𝑐 represents the frequency of the 𝑐-th category samples, 𝜖 is a constant for numerical stability which is set to 1𝑒-9, and 𝛼 is a hyperparameter that controls the smoothness, which is set to 0.9. Our ablation study will later show the impact of different values of 𝛼 on the sampling probability.Then, for each category, we generate independent random numbers 𝑟 𝑐 to obtain the sampled features, which can be written as:EQUATIONwhere X𝑐 𝑐𝑙𝑎𝑠𝑠 ∈ R 1×𝐷 represents the virtual class-aware feature of the 𝑐-th category sampled from the Gaussian distribution. Finally, we concatenate all sampled features to obtain the final virtual classaware feature X𝑐𝑙𝑎𝑠𝑠 :EQUATIONHowever, since the Gaussian distribution is computed based on the statistical measures of the feature space from the first stage, to further mitigate potential biases, we employ the Expectation-Maximization algorithm [32, 36] to iteratively fine-tune the classifier and encoder. In the maximization step, we freeze the encoder and backbone, and train the classifier on a balanced feature space.In the expectation step, we freeze the classifier, and supervise the encoder and backbone with an additional balance constraint to avoid being contaminated by the long-tail label space again. Therefore, we use BCE to fine-tune the classifier in the maximization step and design a Generalized Cross Entropy (GCE) loss [50] suitable for multi-label in the expectation step, which can be written as:EQUATIONEQUATIONwhere 𝑓 𝑐𝑙𝑠 (•) represents our classifier trained in the first stage, which we fine-tune in the second stage. The hyperparameter 𝑞 is in the range (0, 1] and controls the generalization of the loss function.When 𝑞 is set to 1, the loss function becomes the traditional BCE Loss. In our setup, we set 𝑞 to 0.8, which our subsequent ablation study shows yields optimal performance. 𝑦 𝑖,𝑐 is the ground truth label for the 𝑐-th category of the 𝑖-th sample, and ŷ′ 𝑖,𝑐 is the predicted logit for the 𝑐-th category of the 𝑖-th sample. Through our reconstructed label distribution, the classifier can classify within a Due to a long-tail distribution in the dataset, the head categories include BMER, BMEL, JSR, and JSL, with the remaining categories considered as tail categories.2) NIH Chest X-ray 14 Dataset. The NIH Chest X-ray 14 dataset [44] encompasses 112,120 frontal chest X-ray images from 30,805 patients, each annotated with 14 disease image labels mined through natural language processing from the corresponding radiological reports. The head categories are Pneumothorax, Infiltration, Atelectasis, Emphysema, Edema, Effusion, and Pneumonia, with the remaining categories classified as tail categories.3) ODIR-5K Dataset. The ODIR-5K [23] dataset, comprises a multi-label image dataset with 7,000 images that include patientlevel annotations and image-level diagnostic keywords. The dataset encompasses a range of eye diseases, including Normal (N), Diabetes (D), Glaucoma (G), Cataract (C), Age-related Macular Degeneration (AMD), Hypertension (H), Myopia (M), and Others. Among these, the head categories, consisting of N, D, and O, while the remainings are tail categories.For all datasets, we randomly split the dataset into training, validation, and test sets by a ratio of 7:1:2 at the patient level. Subsequently, we evaluate our methodology on the test set.  
evaluation metrics: 
 In the Axial Spondyloarthritis Dataset, we evaluate our method using three metrics: Area Under the Curve (AUC), mean Accuracy (mACC), and mean Average Precision (mAP), where "mean" denotes the average across all categories. For the NIH Chest X-ray 14 dataset, to fairly compare with current state-of-the-art (SOTA) methods [4, 5, 12, 14, 18, 25, 30, 35, 43, 44, 47] , we adopt AUC as the measurement standard. For the ODIR-5K Dataset, in order to compare with current SOTA methods [13, 15, 16, 20, 23, 24, 34] , we follow the unified official evaluation metrics proposed by [23] , which include Cohen's kappa coefficient, F1 score, and AUC, as well as the Final Score as evaluation metrics.During the training phase, we utilized ImageNet-pretrained models of ResNet-50 or DenseNet-121 as the backbone, depending on the specific dataset and objectives. The learning rate was set at 1𝑒 -4, with beta parameters for the Adam optimizer configured at (0.9, 0.999). A weight decay of 1𝑒 -5 was applied, and the batch size was set to 32 to balance computational efficiency and memory usage. Additionally, we adopted a reverse sampling strategy, with parameters 𝛼 and 𝜖 in Eq.12 set to 0.9 and 1𝑒 -9, respectively. For our loss function in Eq. 16, the parameter 𝑞 was set to 0.8. For input image processing, in the Axial Spondyloarthritis Dataset, images were first resized to dimensions of 256 × 256, followed by a cropping step of fixed size 224 × 224. In contrast, for the NIH Chest X-ray 14 and ODIR-5K Datasets, to maintain consistency with previous SOTA works in terms of input size, the input images were initially resized to 512 × 512 before being cropped to 448 × 448.4.4.1 Comparisons on the Axial Spondyloarthritis Dataset. We first compared our method with some SOTA methods on the Axial Spondyloarthritis Dataset. As shown in Table 1 , this comparison was based on unified data augmentation and data partitioning standards, ensuring the fairness and accuracy of the evaluation. It is evident that our method outperforms previous methods across all evaluation metrics. For instance, our method surpassed the baseline (DenseNet) by 7.8%, 4.8%, and 14.5% in AUC, mACC, and mAP metrics, respectively. Under ResNet, the improvement was 8.8%, 7.5%, and 12.6%. This indicates that our method, through decoupling features and reconstructing label distributions, addresses the bias of classifiers under long-tail distributions, thereby enhancing classifier performance. Compared to other SOTA methods, such as the Two-way Loss [21] with ResNet50 as backbone, our method still excelled by 8.5%, 6.5%, and 8.7% in AUC, mACC, and mAP metrics, respectively. This result further validates the effectiveness of our approach.4.4.2 Comparisons on NIH Chest X-ray 14 Dataset. We also evaluated our proposed model with SOTA methods on the publicly available NIH Chest X-ray 14 dataset. The results are shown in Table 2 .Our method achieved a competitive mean AUC score of 0.8560 across 14 thoracic diseases, comparing favorably with previous methods. Moreover, our method showed significant improvement in performance on tail classes. For example, with Nodule pathology, which is challenging to detect and rare in the dataset, other methods struggle to perform well due to the long-tail distribution. However, our proposed method of label decoupling and reconstruction enables the generation of diverse features for the classifier to learn, thereby better detecting such pathologies to aid diagnosis and treatment.. Similar to the previous datasets, our method outperformed previous approaches on the ODIR-5K dataset, which is shown in the Table 3 . Specifically, our Kappa score exceeded other baselines, demonstrating that our model can provide reliable and accurate predictions across various categories, even in cases of uneven data distribution. The improvement in the F1 score indicates our model's success in balancing precision and recall, which is particularly important for medical image classification. A Table 3 : Comparisons with SOTA on ODIR-5K dataset.Method Kappa F1 AUC Final Score Li et al. [23] 0.3072 0.8495 0.8306 0.6624 Jordi et al. [20] 0.426 0.850 0.805 0.693 Gour and Khanna [13] 0.433 0.853 0.849 0.712 He et al. [15] 0.520 0.886 0.903 0.770 BFENet [34] 0.535 0.892 0.912 0.780 Focal Loss [24] 0.625 0.895 0.930 0.817 SCFKD [16] 0.635 0.911 0.927 0.824 LDRNet 0.663 0.917 0.934 0.838 high F1 score means the model does not miss many cases while also maintaining a low false-positive rate. The high AUC score showcases our model's ability to differentiate between different pathological categories, proving its effectiveness in distinguishing between healthy and pathological samples and reducing the risk of false positives and negatives. This is crucial for improving diagnostic accuracy and treatment outcomes for patients. Furthermore, the leading final score further validates our method's superior overall performance. 4 , where '+' denotes the inclusion of an element, while '-' indicates its absence. From the Table 4 , we can observe that the learning of class-aware features in our first stage shows significant performance improvements over the baseline across three metrics. This indicates that our proposed feature learning strategy effectively captures the global features of complex medical images and decouples them, providing a more robust feature representation for subsequent classification tasks. The enhancement of global features is particularly crucial for medical images, which are subtle and varied. It helps the model to maintain a high recognition rate for common diseases while also improving the ability to recognize rare diseases. Furthermore, the performance significantly improves after using feature decoupling along with label distribution reconstruction. This demonstrates that our method of label distribution reconstruction effectively balances the multi-label medical images under a long-tail distribution, allowing the classifier to learn about each category in a balanced manner. Through this reconstruction, our model can provide a more balanced learning environment when facing imbalanced data distribution, ensuring that each category receives adequate attention. Especially for categories with scarce samples, this approach can significantly enhance the model's ability to learn their features and improve recognition accuracy. Additionally, we observe further improvements in our model's performance across three metrics with the addition of Prompt. This is attributed to the semantic information of labels aiding in the decoupling of features by category. By integrating rich contextual and descriptive information with image data, the Prompt module offers semantic guidance to the model, promoting more accurate recognition of rare and difficult-to-distinguish categories. [38] 0.8183 0.7742 0.7570 CLIP [39] 0.8292 0.7838 0.7596 4.5.3 Effects of Different Resampling Strategies. We present the ablation experiments for different sampling strategies in the second stage as depicted in Eq. 12. As shown in the Table 6 , we observe that sampling from the multi-label reconstructed Gaussian distribution using our reverse sampling strategy outperforms uniform sampling from the same distribution by approximately 2.9%, 1.9%, and 3.2% across three metrics, respectively. We speculate that since the classifier learned in the first stage already exhibits a significant bias, it is necessary to employ a reverse sampling strategy to eliminate this bias, thereby enabling the classifier to distinguish between categories more equitably. Notably, even when our method employs a uniform sampling strategy in its second stage, the results surpass both the baseline and the performance using only the first stage. We attribute this to our proposed reconstruction of the multi-label Gaussian distribution. Even though uniform sampling on a longtailed distribution might result in fewer tail samples compared to head samples, our reconstructed multi-label Gaussian distribution generates a rich variety of features for tail categories.Hyperparameters. In our model's hyperparameter ablation experiments, we first focused on the smoothing parameter 𝛼 within the sampling probability equation in Eq.12.We set the range of 𝛼 values from 0.1 to 1.0, increasing in steps of 0.1, and conducted experiments across three metrics using a uniform ResNet50 as the backbone on our Axial Spondyloarthritis Dataset, with all other settings kept cxwonsistent. The results, as shown in Fig. 3a , indicate that the optimal setting for 𝛼 is 0.9. This finding suggests that a higher 𝛼 value helps the model to more 3b , the result indicates that the optimal setting for 𝑞 is 0.8. This demonstrates that moderately adjusting the generality of the loss function on our dataset can effectively improve the model's generalization capability, thereby optimizing overall performance.In the previous sections, we have conducted some quantitative analyses of our method. In this section, we will carry out a qualitative analysis by performing some visualization experiments on the Axial Spondyloarthritis dataset. As illustrated in Fig. 4 , it is observable that our method predicts all pathological features for these medically challenging images, whereas the baseline only predicts pathological features of medical images containing a few labels, such as shown in row 1 column 2. Notably, for the sample in row 1 column 1, our method predicted two additional pathological features, 'ER' and 'EL', beyond the ground truth. We believe this may be due to the complexity of medical images and noise present in the images.Furthermore, to explore whether our classifier addresses the issue of long-tail distribution, we visualized the L2-Normalization of the parameters in our fine-tuned and baseline classifier. As shown in Fig. 5 , for the baseline classifier, the influence of the long-tail distribution leads to a bias towards head categories, and the parameters of the classifier are fluctuating. In contrast, the fine-tuned classifier is no evident bias towards any category, and the results are quite stable. This indicates that our method, through feature decoupling and label reconstruction, effectively solves the problem of classifier bias in multi-label medical images under a long-tail distribution, enabling the classifier to treat each category more fairly. 
conclusion: 
 In this paper, we introduce a novel two-stage training framework, namely decoupling and reconstruction, to effectively address the inherent long-tail distribution problem in multi-label medical image classification tasks. In the first stage, our method initially employs a technique that combines prompt-based approaches with the Transformer model to decouple specific category features from global image features. In the second stage, we reconstruct Gaussian distributions for each category, and then use reverse-sampling to generate more diverse virtual features to correct the biased classifier. Furthermore, acknowledging the scarcity of publicly available multi-label medical image datasets under long-tail distribution, we have collected a multi-label medical image dataset on ankylosing spondylitis and plan to make it open-source to foster research development in the related field. The experimental results on this dataset, along with two other public multi-label datasets, validate the effectiveness and superiority of our approach. In the future, we will try to combine two stages into one stage. 
 
 Appendix:  


Decision: The paper proposes a two-stage framework for long-tailed multi-label medical image recognition, named Label Decoupling and Reconstruction (LDRNet). The framework aims to address the challenges of multi-label and long-tailed distributions in medical datasets. The first stage involves label decoupling, which extracts class-aware features. In contrast, the second stage involves label reconstruction, using these features to generate diverse virtual features for tail categories, promoting unbiased learning and enhancing model generalization.

Pros:
+ The two-stage label decoupling and reconstruction framework is a novel approach to addressing the long-tail problem in multi-label medical image recognition.
+ The model achieves state-of-the-art performance on multiple datasets, showcasing its effectiveness in handling complex, multi-label, and long-tailed distributions.

Cons:
+ The rebuttal did not significantly solve the problems of the two reviewers.
+ The paper lacks a detailed comparison with existing methods in terms of performance.
+ Further clarification is needed on the detailed process of label decoupling and reconstruction, and the interplay between the two stages.
+ The ablation study is limited to one dataset. A more comprehensive ablation study across all three datasets would validate the method's performance better.

This paper has one weak accept, one borderline accept, and one borderline reject, which is over the bar of acceptance, so I recommend accepting this paper.

Reviews: review: paper_summary: paper_summary: The paper proposes a two-stage framework for long-tailed multi-label medical image recognition. Specifically, it utilizes label decoupling to extract the class-aware features, and then reconstruct the label distribution from class-aware features by reverse sampling strategy and Gaussian distribution reconstruction.
strength_weakness: strength_weakness: 1.The paper proposes a two-stage framework for long-tail multi-label medical image recognition, which can capture class-aware features and reconstruct the label distribution to solve the long-tail problem effectively.
2.The paper collect an Axial Spondyloarthritis Dataset to promote the development of the field of multi-label image recognition under long-tail distribution.
limitations: limitations: 1.Reference [27] used a transformer decoder to capture class-aware features by cross attention on input features and learnable query. In this paper, the class-aware features are obtained through the concatenation of label query and the features as the input of the transformer encoder, and then the features at the corresponding position are selected to obtain the class-aware features. This approach may seem less intuitive than [27] 's. 
2.It is hard to understand the effect of Pword and the Plabel, why not concatenate W and L directly with spatial features directly?
3.Does stage1 and stage2 alternate between each other or does stage2 occur after stage1 has completely ended? If it is alternate, does this mean that Gaussian distribution is built from a batch of data? Does this affect the final result because of the different distribution of the local and the global?
paper_summary: paper_summary: *Problem statement*: medical datasets often contain data with multiple diseases thus samples show multi-label characteristics, with wildly varying distributions, thus models develop biases towards more frequent labels (obviously).

*Why a naïve solution that doesn’t work*: samples have multiple labels with high co-occurrence, so naïve rebalancing methods cannot increase the low sample size class proportionally to the high sample classes.

*Proposed solution*: transformer and natural language based method to decouple class specific features from global image features. Uses class specific features to learn a Gaussian distribution for those features. Sample features from the learnt distributions to generate “virtual” features, which can then be weighted to balance classes with few samples for unbiased classifier training.
I.E., class specific feature resampling to reduce classifier bias. 


Contributions:
1.	New method for rebalancing highly correlated class features
2.	New long tail multiclass medical imaging dataset
3.	Experiments on new dataset and existing datasets, comparison with methods
strength_weakness: strength_weakness: Approach and concept is novel and addresses a real-world problem. The method is mostly described well and appears replicable, with well-defined implementation details. The detailed experiments show the strong performance of the method though some details are missing.
limitations: limitations: *Missing dataset information*: Distribution of class labels not stated, which is the driving force behind the paper. How much overlap between labels is there in each of the datasets? Are the labels independent? Does this matter?

*Method evaluation*: Does the method improve the performance of the tail labels significantly? Or does the accuracy just improve overall? 

Demonstrates an improvement in overall accuracy (AUC, mACC, and mAP), but does not demonstrate that it actually impacted the accuracy of imbalanced classes. Does it lead to better accuracy on the “tail classes”, or does it give a blanket improvement in accuracy?

The above points could be addressed by giving the distribution of the class labels for each of the dataset. For table 2, adding a column with the number of samples per class would allow us to see the impact of the method on different label distributions.

Perhaps a naïve question: is it possible to simply train one model per class (i.e., separately) to address the model’s bias? This might inhibit the learning of the model in some way, but then the correlation between classes does not have to be learnt and overcome by the model.
paper_summary: paper_summary: This paper addresses challenges of multi-label and long-tailed distributions in medical datasets using label decoupling and reconstruction. The method involves utilizing class-aware features to reconstruct label distributions and training an unbiased classifier. By decoupling features in the first stage and reconstructing Gaussian distributions in the second stage, the approach aims to improve model generalization and robustness. The study showcases the effectiveness of the proposed method through experiments on various medical image datasets.
strength_weakness: strength_weakness: The approach enhances the model's ability to recognize diseases, captures comprehensive features across various labels, and improves generalization and robustness. The experiments on different medical image datasets show that the model achieves competitive results and outperforms existing methods, particularly in distinguishing between healthy and pathological samples.
limitations: limitations: 1. The background research is insufficient. In addition to the unsample method mentioned in the article, there are some other strategies that were ignored by the author of this article, such as
[1]Seo, HyeRyeong, et al. "Enhancing Multi-Label Long-Tailed Classification on Chest X-Rays through ML-GCN Augmentation." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.
[2] Zhang, Yilan, et al. "ECL: Class-Enhancement Contrastive Learning for Long-Tailed Skin Lesion Classification." International Conference on Medical Image Computing and Computer-Assisted Intervention. Cham: Springer Nature Switzerland, 2023.
2. At the lines 285-286, transformer-based is not very accurate, because transformer-based is more like a method based on a structure. When introducing the method stream, the main principles should be introduced to avoid misunderstandings.
3. At lines 480-500, why only train the classifier to construct a balanced feature space instead of training the encoder and classifier at the same time? The author should give further explanation
4. The ablation study was not sufficient and was only verified on one data set. In order to fully verify the performance of the method, verification should be done on three data sets.