MONICA: Benchmarking on Long-tailed Medical Image Classification

## Abstract

Abstract Long-tailed learning is considered to be an extremely challenging problem in data imbalance learning. It aims to train well-generalized models from a large number of images that follow a long-tailed class distribution. In the medical field, many diagnostic imaging exams such as dermoscopy and chest radiography yield a long-tailed distribution of complex clinical findings. Recently, long-tailed learning in medical image analysis has garnered significant attention. However, the field currently lacks a unified, strictly formulated, and comprehensive benchmark, which often leads to unfair comparisons and inconclusive results. To help the community improve the evaluation and advance, we build a unified, well-structured codebase called M edical O pe N -source Long-ta I led Classifi CA tion ( MONICA ), which implements over 30 methods developed in relevant fields and evaluated on 12 long-tailed medical datasets covering 6 medical domains. Our work provides valuable practical guidance and insights for the field, offering detailed analysis and discussion on the effectiveness of individual components within the inbuilt state-of-the-art methodologies. We hope this codebase serves as a comprehensive and reproducible benchmark, encouraging further advancements in long-tailed medical image learning. The codebase is publicly available on https://github.com/PyJulie/MONICA .

## 1 Introduction

The deep learning techniques have proven effective for most computer vision tasks benefiting from the grown-up dataset scale (Deng et al., ; Dosovitskiy et al., ; He et al., ). However, training a well-generalized deep-learning-based model is data-driven and requires a balanced data distribution. In the real world, the collected image datasets often exhibit a long-tailed distribution due to
the complex findings and attributes (Ridnik et al., ; Lin et al., ). Models trained from such datasets always result in the overprediction of head classes and underprediction of tail classes (Liu et al., ; Zhang et al., ). In the medical domain, medical image datasets are typically long-tailed because of the natural frequency of diseases in the population and the challenges in collecting sufficient samples of rare conditions (Ju et al., ). However, it is vital to recognize these rare diseases in real-world practice, as they are relatively rare for doctors and may also lack diagnostic capacity.

To address the long-tailed class imbalance, massive deep long-tailed learning studies have been conducted for natural image recognition.
A recent survey (Zhang et al., ) grouped existing methods into three main categories based on their main technical contributions, i.e., re-sampling (Chawla et al., ; Estabrooks et al., ; Liu et al., ; Zhang & Pfister, ), information augmentation (Zhang, ; Zhong et al., ; Li et al., ) and module improvement (Kang et al., ; Tang et al., ; Zhang et al., ). Class Re-sampling aims to balance the distribution by over-sampling the minority-class samples or under-sampling the majority-class samples. Data Augmentation aims to enhance the size and quality of datasets by applying predefined transformations to each data/feature for model training. Module improvement aims to modify the network to better learn from a long-tailed distribution with a specific module design. In Sec. 3.2, we will briefly introduce these methodologies supported in our codebase as groups.

In recent years, research on long-tailed medical image classification (LTMIC) has garnered significant attention. Current research on LTMIC is primarily focused on dermatology (Ju et al., ; Roy et al., ; Mehta et al., ; Zhang et al., ), ophthalmology (Ju et al., ; ; Li et al., ), and radiology (Holste et al., ; ; Jeong et al., ), where more abundant datasets and well-defined diagnostic tasks are available. However, these methods are often evaluated under varying experimental settings, making it difficult to comprehensively compare and select the best approach for practical applications. This challenge is further compounded by the lack of standard benchmarks and pipelines. Overall, these works may share common shortcomings, which arise from the following factors. 1) Datasets. Existing works on LTMIC are evaluated on different datasets. Despite the specialized nature of medical data, we are still curious to explore whether there are highly generalizable methods that can perform well across different datasets, tasks, or medical domains. 2) Partition Schemes. The partition schemes are vita important for long-tailed learning to ensure a fair comparison and metric evaluation. For example, in natural image long-tailed learning, although the training data follows a long-tailed distribution, the test set is often balanced. However, in medical imaging, the test set typically mirrors the distribution of the training set. As a result, relying solely on overall accuracy may not accurately reflect the performance of trained models. 3) Comparison Methodologies The methodologies used for comparing different approaches in LTMIC can vary significantly, which adds another layer of complexity when trying to assess their effectiveness. Due to the lack of standardized comparison practices, such as consistent use of baseline models, evaluation metrics, and reporting standards, it becomes challenging to draw meaningful conclusions across studies. Furthermore, the varying availability of codes and experiment replication further hinder the transparency of fair comparison and the ability to establish a clear understanding of which methods are truly superior. These inconsistencies highlight the need for more unified and comprehensive comparison methodologies in future research.

Our main contributions are summarized as follows: (1) We introduce MONICA, the first comprehensive LTMIC benchmark, where 30+ methodologies from relevant fields are impartially evaluated from scratch across 12 long-tailed medical datasets spanning 6 medical domains. This benchmark covers datasets with varying scales, granularity, and imbalance ratios, offering a robust framework for researchers to objectively assess their models against a wide array of baselines, providing a clear measurement of each method’s effectiveness. (2) We developed a well-structured codebase specifically for customized LTMIC. The framework is modular, featuring decoupled components such as augmentations, well-known backbones, loss functions, optimization strategies, and distributed training. This codebase offers best practices for researchers and engineers to identify applicable methodologies for both pre-defined benchmarks and their own customized datasets. (3) We performed extensive empirical analyses and gave a detailed discussion on valuable insights that suggest promising directions for methodological and evaluation innovations in future LTMIC research.

## 2 Supported Tasks, Benchmarks, and Metrics

### 2.1 Problem Definition and Supported Tasks

LTMIC seeks to learn a deep neural network
model from a training dataset with a long-tailed class distribution. Let $\{x_{i},y_{i}\}^{n}_{i=1}$ be the long-tailed training set, where
each sample $x_{i}$ has a corresponding class label $y_{i}$. The total
number of training set over $K$ classes is $n=\Sigma^{K}_{k=1}n_{k}$, where
$n_{k}$ denotes the data number of class $k$; let $\pi$ denote the vector of
label frequencies, where $\pi=\frac{n_{k}}{n}$ indicates the label frequency
of class $k$ where $\rho$ denoted as imbalance ratio $\rho=\frac{n_{1}}{n_{k}}$. Without loss of generality, a common assumption in
long-tailed learning is when the classes are sorted by
cardinality in decreasing order (i.e., if $i_{1}<i_{2}$, then $n_{i_{1}}>n_{i_{2}}$,
and $n_{1}\gg n_{k}$ ). For a multi-label setting, the class label $y_{i}$ would be a set of Bernoulli distribution $y_{i}\in\{0,1\}^{k}$. Label cardinality $L_{Card}(S)=\frac{1}{n}\Sigma^{n}_{i=1}|y_{i}|$ is commonly used to describe the degree of label co-occurrence in a multi-label dataset.
In this context, MONICA is developed to support the training and evaluation of both single-label / multi-class (MC) and multi-label (ML) long-tailed learning on conducted benchmarks and customized datasets.

### 2.2 Benchmarks and Datasets

**Table 1: Comparison, partition, and statistics of datasets across various medical specialties.**
|  | Dermatology | Ophthalmology | Radiology | Pathology | Hematology | Histology | Gastroenterology |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Dataset | ISIC-2019-LT | DermaMNIST | ODIR | RFMiD | OrganA/C/SMNIST | CheXpert | PathMNIST | BloodMNIST | TissueMNIST | KVASIR |
| Data Modality | Dermatoscope | Dermatoscope | Fundus | Fundus | CT | X-Ray | Pathology | Microscope | Microscope | Endoscope |
| Task | MC | MC | ML | ML | MC | ML | MC | MC | MC | MC |
| Class Number | 8 | 7 | 12 | 29 | 11 | 14 | 9 | 8 | 8 | 14 |
| Imbalance Ratio | 100 / 200 / 500 | 100 | 80 | 310 | 100 | 33 | 100 | 100 | 100 | 20 |
| Train Samples | 10,322 / 9,400 / 8,494 | 6,964 | 7,000 | 1,920 | 16,597 / 7,712 / 9,146 | 178,731 | 29,276 | 4,809 | 109,532 | 4,656 |
| Validation Samples | 400 | 1003 | 1,000 | 640 | 6,491 / 2,392 / 2,452 | 44,683 | 10,004 | 1,712 | 23,640 | 700 |
| Test Samples | 800 | 2005 | 2,000 | 640 | 17,778 / 8,216 / 8,827 | 243 | 7,180 | 3,421 | 47,280 | 1400 |
| Group Split | 2 / 5 / 8 | 1 / 5 / 7 | 3 / 9 / 12 | 6 / 14 / 29 | 3 / 6 / 11 | 4 / 10 / 14 | 2 / 5 / 9 | 3 / 5 / 8 | 3 / 5 / 8 | 4 / 8 / 14 |

To address the challenge of long-tailed medical image
classification, we conducted our benchmark on 12 datasets covering Dermatology (ISIC-2019 (Tschandl et al., ), Dermamnist (Yang et al., )), Ophthalmology (ODIR (), RFMiD ()), Radiology (Organamnist, Organamnist, Organamnist (Yang et al., ), CheXpert (Irvin et al., )), Pathology (Pathmnist), Hematology (Bloodmnist), Histology (Tissuemnist) (Yang et al., ) and Gastroenterology (KVASIR) (Pogorelov et al., ).
To evaluate the ability of existing methodologies under extremely challenging imbalance conditions, some widely-used medical image datasets and tasks with fewer than 8 categories such as Diabetic Retinopathy (Li et al., ) Grading are not considered.

#### 2.2.1 Dermatology Datasets

ISIC-2019-LT (Ju et al., ) is a long-tailed version constructed from ISIC-2019 Challenge (Tschandl et al., ), which aims to classify
8 kinds of diagnostic categories. We follow FlexSampling (Ju et al., ) and sample a subset from a Pareto distribution. With $k$ classes and imbalance ratio $r=\frac{N_{0}}{N_{k-1}}$, the number of samples for class $c\in[0,k)$ can be calculated as $N_{c}={(r^{-(k-1)})}^{c}*N_{0}$. We set $r=\{100,200,500\}$ for three different imbalance levels. We select 50 and 100 images from the remained samples as validation set and test set.

DermaMNIST is created by MedMNIST (Yang et al., ) based on the HAM10000 (Tschandl et al., ), a large collection of multi-source dermatoscopic images of common pigmented skin lesions. The dataset consists of 10, 015 dermatoscopic images categorized as 7 different diseases, formalized as
a multi-class classification task. The original images are split into training, validation and test set with a ratio of 7: 1: 2. We modify the training set with Pareto distribution and imbalance ratio of $r=100$. We keep the use of original validation and test set for evaluation.

#### 2.2.2 Ophthalmology Datasets

Ocular Disease Intelligent Recognition
(ODIR) () is a structured ophthalmic database of 5,000
patients with age, colour fundus photographs of left and right
eyes, and doctors’ diagnostic keywords. Specifically, the
ODIR dataset was originally divided into 8,000 / 1,000 /
2,000 images for training/off-site testing / on-site testing. The classes of annotations can
be divided into two levels: coarse and fine. There are 8 classes
at the coarse level,
where one or more conditions are
given on a patient-level diagnosis, resulting in a multi-label classification challenge.

RFMiD dataset () consists of 3200 fundus images captured using three different fundus cameras with 46 conditions annotated through adjudicated consensus of two senior retinal experts. The RFMiD dataset was originally divided into 1,920 / 640 / 640 images for training/validation/testing. We followed the setting in the RFMiD challenge, the diseases with more than 10 images belong to an independent class and all other disease categories are merged as “OTHER”. This finally constitutes 29 classes (normal + 28 diseases or lesions) for disease classification.

#### 2.2.3 Radiology Datasets

OrganA, C, SMNIST is based on 3D computed tomography (CT) images from Liver Tumor Segmentation Benchmark
(LiTS) (Bilic et al., ). They are renamed from OrganMNIST-Axial/Coronal/Sagittal (in MedMNIST (Yang et al., )), which uses bounding-box annotations of 11 body organs from another study to obtain the organ labels. 2D images are cropped from the center slices of the 3D bounding boxes in
axial/coronal/sagittal views. All three datasets contain the labeling of 11 body organs, resulting in a multi-class classification task.
Originally, 115 and 16 CT scans from the source training set are used as training and validation set, respectively. The 70 CT scans from the source test set are treated as the test set.
We modify the training set with Pareto distribution and imbalance ratio of $r=100$. We keep the use of original validation and test set for evaluation.

CheXpert dataset (Irvin et al., ) is a large dataset that contains 224,316 chest radiographs with 14 kinds of observations. The training labels in the dataset for each observation are either 0 (negative), 1 (positive), or u (uncertain). For convenience, we map all uncertain instances to 0 (negative). The original images are split into training, validation and test set with a ratio of 7: 1: 2.

#### 2.2.4 Others

PathMNIST (Yang et al., ) is constructed for predicting survival from colorectal cancer histology slides, providing a dataset
(NCT-CRC-HE-100K) (Kather et al., ) of 100, 000 non-overlapping image patches from hematoxylin & eosin stained histological images, and
a test dataset (CRC-VAL-HE-7K) of 7, 180 image patches from a different clinical center. The dataset is comprised of 9 types
of tissues as a multi-class classification task. We modify the training set with Pareto distribution and imbalance ratio of $r=100$. We keep the use of original validation and test set for evaluation.

BloodMNIST (Yang et al., ) is based on a dataset of individual normal cells, captured from individuals without infection, hematologic
or oncologic disease and free of any pharmacologic treatment at the moment of blood collection. It contains a total of 17,092
images and is organized into 8 classes. The source dataset was originally split to a ratio of 7 : 1: 2 into training, validation and test
set. We modify the training set with Pareto distribution and imbalance ratio of $r=100$. We keep the use of original validation and test set for evaluation.

TissueMNIST (Yang et al., ) is based on the Broad Bioimage Benchmark Collection (Ljosa et al., ). The dataset contains 236, 386 human
kidney cortex cells, segmented from 3 reference tissue specimens and organized into 8 categories. The source dataset was originally split to a ratio of 7: 1: 2 into training, validation and test
set. We modify the training set with Pareto distribution and imbalance ratio of $r=100$. We keep the use of original validation and test set for evaluation.

Kvasir (Pogorelov et al., ) is a long-tailed dataset of 10,662 gastrointestinal tract images with 23 classes from different anatomical and pathological landmarks. We modify the original dataset with Pareto distribution and imbalance ratio of $r=20$. Those categories with images less than 50 are not included. We select 50 and 100 images from the remained samples as validation set and test set.

## 3 MONICA and Supported Methodologies

### 3.1 Codebase Structure

The whole training process here is fragmented into multiple components, including augmentation (.MONICA.dataset), sampling strategies (.MONICA.sampler), model architectures (.MONICA.models), and loss functions (.MONICA.losses) etc. For instance, vision models are decoupled into several encoders and classification heads according to different methodology designs. This modular architecture allows
researchers to easily craft different counterparts as customized datasets and tasks are needed. With the help of configuration files in .MONICA.configs, users can tailor specialized visual classification models and their associated training strategies with ease.

### 3.2 Supported Methodologies

According to a recent survey (Zhang et al., ), we group existing methods into three main
categories based on their main technical contributions, i.e., class
re-sampling, information augmentation and module improvement. Based on factors such as the availability of open-source code, its impact, and ease of implementation, we have selected and introduced over 30 methodologies supported in MONICA. Note that one methodology may contain more than one of those three taxonomy and we will group them based on their primary motivation and technical contribution when presenting them.

#### 3.2.1 Re-sampling Methodologies

Re-sampling aims to balance the distribution by over-sampling the minority-class samples or under-sampling the majority-class samples following designed schemes. Focal loss (Lin et al., ) is designed to down-weight the loss assigned to well-classified examples, focusing more on hard-to-classify instances. Class-balanced (CB) loss (Cui et al., ) addresses class imbalance by weighting the loss inversely proportional to the effective number of samples per class, thereby reducing the impact of over-represented classes. LADE loss (Hong et al., )
proposed to use them to post-adjust model outputs so that the trained model can be calibrated for arbitrary test class distributions. LDAM loss (Cao et al., ) adjusts the margins for different classes based on their label distribution, promoting larger margins for underrepresented classes to improve their classification accuracy.
EQL (Tan et al., ) directly down-weights the loss values of tail-class samples when they serve as negative labels for head-class samples.
Balanced softmax (Jiawei et al., )
proposed to adjust prediction logits by multiplying by the label
frequencies, so that the bias of class imbalance can be alleviated by
the label prior before computing final losses. VS loss (Kini et al., ) intuitively analyzed the distinct effects of additive and multiplicative logit-adjusted losses, leading to a novel VS loss to combine the advantages of both forms of adjustment.

#### 3.2.2 Information Augmentation Methodologies

Data Augmentation aims to enhance the size and quality of datasets
by applying predefined transformations to each data/feature for
model training. MixUp (Zhang, ) is proposed to improve the model generalization but found to be effective for long-tailed learning by information shared between head and tailed classes. MiSLAS (Zhong et al., )
proposed
to enhance the representation learning with data mixup in the first
stage, while applying a label-aware smoothing strategy for better
classifier generalization in the second stage.
RSG (Wang et al., ) proposed to dynamically estimate a set of feature centers for each class, and use the feature displacement between head-class sample features and their nearest intra-class feature center to augment each tail sample feature. RIDE (Wang et al., ) introduced a knowledge distillation method to reduce the parameters of the multi-expert model by learning a student network with fewer experts.
GCL loss (Li et al., ) perturbs logits with Gaussian noise of varying amplitudes, especially larger for tail classes. Additionally, a class-based effective number sampling strategy with classifier re-training is proposed to mitigate classifier bias.

#### 3.2.3 Module Improvement Methodologies

Decoupling (Kang et al., ; Zhou et al., ) was the pioneering work
to introduce such a two-stage decoupled training scheme. It
empirically evaluated different sampling strategies for representation learning in the first stage and then evaluated different classifier training schemes by fixing
the trained feature extractor in the second stage. In the classifier
learning stage, there are also four methods, including classifier
re-training with class-balanced sampling, the nearest class mean
classifier, the $\tau$-normalized classifier, and the learnable weight-scaling classifier. Range loss (Zhang et al., ) is designed to reduce overall intrapersonal variations while enlarging interpersonal differences simultaneously. Causal classifier (Tang et al., ) resorted to causal inference for keeping the good and removing the bad momentum causal effects in long-tailed learning. DisAlign (Zhang et al., ) innovated the classifier training with a new adaptive logits adjustment strategy. BBN (Zhou et al., ) proposed to use two network branches, i.e., a conventional learning branch and a re-balancing branch,
to handle long-tailed recognition. SADE (Li et al., )
explored a new multi-expert scheme to handle test-agnostic
long-tailed recognition, where the test class distribution can be
either uniform, long-tailed or even inversely long-tailed. SAM (Foret et al., ) minimizes both loss value and loss sharpness by seeking parameters in neighborhoods.

## 4 Experiments

### 4.1 Implementation details

We implement all experiments in PyTorch, ensuring a fair comparison by using unified settings with consistent hyperparameters and architecture choices, unless otherwise specified in the paper. For instance, we use ResNet-50 (He et al., ) as the primary network backbone across all methods, modifying it as needed for certain module improvement methods like BBN. Model training is conducted with the Adam optimizer, using a batch size of 256, a learning rate of 0.0003, and an input size of 224×224, except for CheXpert, where the input size is 512×512. For certain methodologies, such as SAM, we adhere to the specific optimizer and hyperparameters following the original paper. All these designs are for the fairness and the practicality of the comparison on the benchmark. We use top-1 accuracy to evaluate the performance of single-label datasets and mean average precision is adopted for multi-label datasets following DBLoss (Wu et al., ). The main benchmark development and testing are performed using 8 $\times$ NVIDIA RTX4090 GPUs.

**Table 2: The comparison study results on ISIC-2019-LT benchmark.**
| ISIC-2019-LT |  |  |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Imbalance Ratio | $r=100$ | $r=200$ | $r=500$ |  |  |  |  |  |  |  |  |  |
| Methods | Head | Medium | Tail | Avg. | Head | Medium | Tail | Avg. | Head | Medium | Tail | Avg. |
| ERM | 79.00 | 60.67 | 38.33 | 59.33 | 78.50 | 56.67 | 27.00 | 54.06 | 78.00 | 46.67 | 12.67 | 45.78 |
| RS | 69.50 | 61.33 | 49.33 | 60.06 | 76.00 | 62.67 | 36.33 | 58.33 | 78.50 | 44.00 | 19.67 | 47.39 |
| RW | 68.00 | 55.33 | 53.67 | 59.00 | 73.50 | 54.67 | 41.33 | 56.50 | 62.00 | 38.00 | 34.33 | 44.78 |
| Focal | 73.50 | 54.00 | 44.33 | 57.28 | 79.50 | 53.00 | 31.00 | 54.50 | 83.00 | 44.00 | 13.00 | 46.67 |
| CB-Focal | 71.00 | 57.67 | 52.67 | 60.44 | 72.50 | 52.00 | 51.00 | 58.50 | 59.50 | 42.33 | 43.33 | 48.39 |
| LADELoss | 78.50 | 52.33 | 43.67 | 58.17 | 84.00 | 52.33 | 18.67 | 51.67 | 78.50 | 43.00 | 14.00 | 45.17 |
| LDAM | 78.50 | 55.67 | 41.67 | 58.61 | 81.50 | 52.33 | 31.00 | 54.94 | 76.50 | 41.33 | 19.33 | 45.72 |
| BalancedSoftmax | 62.50 | 54.33 | 61.00 | 59.28 | 77.00 | 53.67 | 55.00 | 61.89 | 62.00 | 49.67 | 41.67 | 51.11 |
| VSLoss | 80.00 | 56.33 | 33.67 | 56.67 | 80.50 | 51.00 | 28.67 | 53.39 | 79.00 | 47.67 | 11.00 | 45.89 |
| MixUp | 78.00 | 50.67 | 35.67 | 54.78 | 83.00 | 46.67 | 21.33 | 50.33 | 76.00 | 48.00 | 9.00 | 44.33 |
| MiSLAS | 57.50 | 52.33 | 57.67 | 55.83 | 71.50 | 48.33 | 49.67 | 56.50 | 63.50 | 43.00 | 39.33 | 48.61 |
| GCL | 57.50 | 63.33 | 71.33 | 64.06 | 71.00 | 56.67 | 64.33 | 64.00 | 63.50 | 55.00 | 46.00 | 54.83 |
| cRT | 74.50 | 59.67 | 55.67 | 63.28 | 81.00 | 60.00 | 39.67 | 60.22 | 63.50 | 48.33 | 28.00 | 46.61 |
| LWS | 72.50 | 52.67 | 45.33 | 56.83 | 79.50 | 51.33 | 32.67 | 54.50 | 68.00 | 43.33 | 29.67 | 47.00 |
| KNN | 70.00 | 55.67 | 58.67 | 61.44 | 77.00 | 51.33 | 46.67 | 58.33 | 75.00 | 45.33 | 27.33 | 49.22 |
| LAP | 75.00 | 59.33 | 49.33 | 61.22 | 76.50 | 50.67 | 48.00 | 58.39 | 72.00 | 43.67 | 33.00 | 49.56 |
| De-Confound | 79.00 | 52.33 | 47.67 | 59.67 | 82.50 | 52.67 | 24.33 | 53.17 | 72.50 | 45.33 | 9.67 | 42.50 |
| DisAlign | 81.00 | 60.00 | 52.33 | 64.44 | 78.00 | 59.67 | 52.33 | 63.33 | 68.50 | 49.33 | 37.33 | 51.72 |
| BBN | 82.50 | 58.33 | 46.00 | 62.28 | 76.50 | 62.33 | 31.00 | 54.94 | 75.50 | 50.67 | 19.00 | 48.39 |

**Table 3: The comparison study results on MedMNIST and KVASIR benchmarks.**
|  | BloodMNIST | DermaMNIST | PathMNIST | TissueMNIST |  |  |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Methods | Head | Medium | Tail | Avg | Head | Medium | Tail | Avg. | Head | Medium | Tail | Avg. | Head | Medium | Tail | Avg. |
| ERM | 97.27 | 97.16 | 82.00 | 92.14 | 95.82 | 61.86 | 44.08 | 67.25 | 98.46 | 99.64 | 83.07 | 93.72 | 64.38 | 62.11 | 23.21 | 49.90 |
| RS | 95.75 | 99.20 | 91.17 | 95.37 | 91.95 | 65.75 | 59.75 | 72.48 | 99.26 | 97.47 | 86.78 | 94.50 | 50.26 | 64.71 | 52.88 | 55.95 |
| RW | 97.39 | 99.04 | 88.14 | 94.86 | 87.32 | 65.31 | 69.19 | 73.94 | 96.62 | 99.74 | 88.57 | 94.97 | 59.72 | 66.15 | 43.44 | 56.44 |
| Focal | 97.09 | 99.42 | 80.75 | 92.42 | 95.08 | 59.60 | 60.57 | 71.75 | 96.38 | 98.66 | 87.51 | 94.18 | 61.89 | 63.17 | 22.07 | 49.04 |
| CB-Focal | 96.97 | 98.34 | 85.21 | 93.51 | 84.41 | 63.27 | 73.99 | 73.89 | 98.44 | 99.45 | 84.10 | 94.00 | 58.60 | 67.08 | 45.31 | 56.99 |
| LADELoss | 97.16 | 98.77 | 72.76 | 89.56 | 91.28 | 67.43 | 50.60 | 69.77 | 98.55 | 98.48 | 84.43 | 93.82 | 65.96 | 63.33 | 23.92 | 51.07 |
| LDAM | 96.74 | 98.45 | 87.32 | 94.17 | 92.62 | 63.71 | 53.15 | 69.82 | 97.37 | 98.66 | 86.07 | 94.03 | 63.31 | 64.57 | 14.64 | 47.51 |
| BalancedSoftmax | 96.55 | 98.13 | 90.71 | 95.13 | 89.41 | 67.35 | 76.61 | 77.79 | 96.39 | 99.54 | 88.42 | 94.78 | 53.86 | 68.41 | 52.91 | 58.39 |
| VSLoss | 96.05 | 97.70 | 86.59 | 93.45 | 92.84 | 64.75 | 49.70 | 69.10 | 98.87 | 98.31 | 83.46 | 93.54 | 66.78 | 61.45 | 21.54 | 49.92 |
| MixUp | 98.04 | 98.93 | 79.93 | 92.30 | 95.90 | 59.03 | 54.43 | 69.78 | 99.38 | 99.54 | 84.72 | 94.54 | 65.85 | 62.79 | 13.54 | 47.39 |
| MiSLAS | 93.15 | 99.25 | 81.74 | 91.38 | 65.92 | 67.53 | 75.72 | 69.72 | 96.30 | 98.99 | 88.63 | 94.64 | 42.84 | 64.71 | 49.91 | 52.49 |
| GCL | 96.98 | 99.52 | 90.70 | 95.74 | 68.75 | 73.11 | 90.03 | 77.30 | 96.16 | 99.89 | 87.80 | 94.62 | 64.44 | 69.99 | 33.69 | 56.04 |
| cRT | 97.64 | 99.36 | 87.43 | 94.81 | 88.52 | 73.36 | 70.09 | 77.32 | 93.86 | 99.69 | 90.59 | 94.71 | 59.47 | 67.83 | 38.70 | 55.33 |
| LWS | 90.30 | 75.36 | 1.50 | 55.72 | 44.87 | 7.82 | 1.26 | 17.98 | 59.44 | 32.10 | 23.94 | 38.49 | 59.28 | 33.59 | 0.60 | 31.16 |
| KNN | 93.63 | 95.88 | 83.31 | 90.94 | 85.31 | 66.62 | 59.67 | 70.53 | 93.56 | 99.79 | 89.67 | 94.34 | 54.77 | 55.40 | 51.38 | 53.85 |
| De-Confound | 97.14 | 97.49 | 85.10 | 93.24 | 95.08 | 61.54 | 49.25 | 68.62 | 93.45 | 99.85 | 88.53 | 93.94 | 65.32 | 61.35 | 24.34 | 50.34 |
| DisAlign | 97.11 | 98.98 | 88.82 | 94.97 | 85.68 | 66.04 | 75.72 | 75.81 | 96.70 | 97.95 | 91.85 | 95.50 | 50.20 | 59.73 | 60.21 | 56.71 |
| BBN | 96.24 | 98.23 | 89.39 | 94.62 | 91.87 | 70.98 | 64.02 | 75.62 | 98.14 | 98.95 | 90.33 | 95.80 | 58.62 | 69.68 | 43.29 | 57.19 |
|  | OrganAMNIST | OrganCMNIST | OrganSMNIST | KVASIR |  |  |  |  |  |  |  |  |  |  |  |  |
| Methods | Head | Medium | Tail | Avg. | Head | Medium | Tail | Avg. | Head | Medium | Tail | avg | Head | Medium | Tail | Avg. |
| ERM | 82.67 | 78.20 | 68.56 | 76.48 | 93.50 | 59.20 | 65.18 | 72.63 | 88.33 | 56.89 | 66.90 | 70.71 | 95.50 | 93.25 | 60.33 | 83.03 |
| RS | 76.14 | 82.10 | 63.39 | 73.88 | 92.49 | 57.14 | 64.06 | 71.23 | 89.99 | 52.61 | 66.39 | 69.66 | 95.75 | 94.50 | 62.83 | 84.36 |
| RW | 83.79 | 78.89 | 73.55 | 78.74 | 92.20 | 68.91 | 66.84 | 75.98 | 83.01 | 58.68 | 69.60 | 70.43 | 96.75 | 88.75 | 67.67 | 84.39 |
| Focal | 85.08 | 73.19 | 68.88 | 75.72 | 91.98 | 57.02 | 63.36 | 70.79 | 86.24 | 54.93 | 64.77 | 68.65 | 95.75 | 92.00 | 57.67 | 81.81 |
| CB-Focal | 75.95 | 80.52 | 74.12 | 76.86 | 94.16 | 57.99 | 67.36 | 73.17 | 84.18 | 60.42 | 71.42 | 72.00 | 95.00 | 84.75 | 71.50 | 83.75 |
| LADELoss | 84.54 | 77.00 | 68.73 | 76.76 | 91.22 | 61.58 | 62.86 | 71.88 | 83.32 | 59.04 | 64.92 | 69.09 | 96.25 | 90.00 | 60.83 | 82.36 |
| LDAM | 82.86 | 76.38 | 70.14 | 76.46 | 90.98 | 61.68 | 69.16 | 73.94 | 86.07 | 58.66 | 68.60 | 71.11 | 96.00 | 89.00 | 63.17 | 82.72 |
| BalancedSoftmax | 79.45 | 76.48 | 72.79 | 76.24 | 93.58 | 62.68 | 67.61 | 74.62 | 81.44 | 59.27 | 70.05 | 70.26 | 95.25 | 88.25 | 65.83 | 83.11 |
| VSLoss | 81.59 | 81.73 | 67.85 | 77.06 | 94.79 | 61.42 | 67.08 | 74.43 | 86.46 | 58.66 | 66.70 | 70.61 | 96.75 | 92.75 | 62.83 | 84.11 |
| MixUp | 85.54 | 77.78 | 65.60 | 75.97 | 92.25 | 57.43 | 66.97 | 72.22 | 89.15 | 50.09 | 64.25 | 67.83 | 96.25 | 92.25 | 55.33 | 81.28 |
| MiSLAS | 73.78 | 73.71 | 67.63 | 71.71 | 84.86 | 51.08 | 64.40 | 66.78 | 74.66 | 52.74 | 69.39 | 65.60 | 94.25 | 85.75 | 71.00 | 83.67 |
| GCL | 75.72 | 83.69 | 77.86 | 79.09 | 94.15 | 60.46 | 68.99 | 74.54 | 88.64 | 62.68 | 73.08 | 74.80 | 96.00 | 93.75 | 65.67 | 85.14 |
| cRT | 83.88 | 76.10 | 67.34 | 75.77 | 92.11 | 63.17 | 67.38 | 74.22 | 88.23 | 58.99 | 69.79 | 72.34 | 95.50 | 88.00 | 68.83 | 84.11 |
| LWS | 84.37 | 35.86 | 7.49 | 42.57 | 44.87 | 7.82 | 1.26 | 17.98 | 59.44 | 32.10 | 23.94 | 38.49 | 95.00 | 87.00 | 61.67 | 81.22 |
| KNN | 79.03 | 78.74 | 68.64 | 75.47 | 88.66 | 56.54 | 65.97 | 70.39 | 85.33 | 54.82 | 66.60 | 68.92 | 95.00 | 91.50 | 67.00 | 84.50 |
| De-Confound | 77.50 | 87.72 | 68.87 | 78.03 | 92.56 | 64.81 | 64.58 | 73.98 | 87.03 | 61.35 | 66.07 | 71.48 | 96.25 | 90.50 | 66.50 | 84.42 |
| DisAlign | 80.51 | 78.31 | 69.20 | 76.01 | 93.74 | 62.69 | 72.65 | 76.36 | 88.09 | 58.77 | 66.50 | 71.12 | 95.75 | 89.25 | 68.50 | 84.50 |
| BBN | 74.63 | 77.48 | 67.08 | 73.06 | 91.55 | 63.41 | 63.86 | 72.94 | 87.91 | 59.28 | 71.74 | 72.98 | 95.25 | 89.25 | 70.00 | 84.83 |

### 4.2 Main Results

We conducted extensive experiments on various datasets in the comparison of the state-of-the-art long-tailed learning methods.
We introduced and implemented 30+ methods but only present results for the most relevant ones to avoid redundancy and maintain clarity. Methods with similar or suboptimal performance were excluded to focus on those that best support our key findings within the main motivation’s constraints.

Overall performance evaluation. Table 2 and Table 3 compare the performance of various methods on the ISIC-2019-LT, MedMNIST, and KVASIR benchmarks under different imbalance ratios and datasets. We group the methods as introduced in Sec. 3.2 and filter out these results which do not outperform ERM in terms of any metrics. ERM, as a baseline, generally struggles with tail classes, showing declining performance as imbalance increases. Re-sampling (RS) and re-weighting (RW) methods improve tail class performance but still face challenges under extreme imbalance. Advanced methods like GCL and MiSLAS demonstrate strong results, particularly in handling severe class imbalance, with GCL consistently showing the highest average performance across multiple datasets. Methods like LADELoss, LDAM, and PriorCELoss are effective in improving tail performance, while VSLoss and BalancedSoftmax also perform well, particularly in medium and head classes. In the following sections, we will explore the factors that contribute to the effectiveness of these methods and offer insights into best practices.

Curse of shot-based group evaluation. Improving the performance on tail and medium groups often comes at the cost of reducing accuracy on the head group. For instance, in the ISIC dataset with an imbalance ratio of 100, while the average performance remains similar for GCL and DisAlign, GCL sacrifices a substantial portion of the head group’s performance (from 79.00% to 57.50%) to achieve a superior improvement in the tail group’s performance (from 38.33% to 71.33%). While the trade-off is inevitable, it presents an additional challenge in designing novel metrics that can globally assess performance to meet the demands of real-world practice, particularly in complex disease systems.

Effectiveness of re-sampling across SOTAs. The primary challenge in long-tailed problems is underfitting tail classes due to data imbalance. While resampling is a simple and straightforward approach that shows promising improvements across various datasets and tasks. Also, it is easily incorporated in most state-of-the-art methods along with other module improvements. In this context, further improvements could involve replacing or combining re-sampling modules with alternative approaches, as MONICA offers decoupled training to facilitate these explorations.
Given the diverse implementations and intersections of techniques (e.g., Focal Loss as a component in CBLoss), we summarize their key characteristics—class weighting, modulating factors, and loss formulas—in Table 4 to provide a comprehensive overview of how each method addresses class imbalance through class- or instance-level sampling strategies.

**Table 4: Overview of various loss functions based on re-sampling strategies.**
| Loss Function | Class Weighting | Modulating Factor | Formula |
| --- | --- | --- | --- |
| ERM (Original Sampling) | None (uses original data distribution) | None | $\text{CrossEntropy}(\text{logits},y)$ |
| Re-balanced Sampling | $\frac{1}{n_{c}}$, where $n_{c}$ is the number of samples in class $c$ | None | $\text{CrossEntropy}(\text{logits},y)$ with $P(y=c)\propto\frac{1}{n_{c}}$ |
| Difficulty-based Sampling | Learning difficulty, typically inverse of validation accuracy $a_{c}$ | None | $\text{CrossEntropy}(\text{logits},y)$ with $P(y=c)\propto\frac{1}{a_{c}}$ |
| Focal Loss | Optional per-sample weighting via $\alpha$ | $(1-p_{t})^{\gamma}$, where $p_{t}$ is the probability of correct classification | $-\alpha(1-p_{t})^{\gamma}\log(p_{t})$ |
| CBLoss | The number of samples per class $\frac{1-\beta}{1-\beta^{n_{c}}}$ | None | $\text{weight}\times\text{Loss}(p_{t},y)$ |
| LADELoss | Class weighting based on the number of samples per class | Regularization using ReMINE to prevent overfitting | $-\sum(\text{ReMINE_loss}\times\text{cls_weight})$ |
| LDAM Loss | Margin $m$ adjustment based on the number of samples per class | None | $\text{max}(0,m-\log(\text{prob}[c]))+\text{regularization}$ |
| PriorCELoss | The class prior distribution $\frac{n_{c}}{N}$ | $\text{logits}=x+\log(\text{prior})$ | $\text{loss}=\text{CrossEntropy}(\text{logits},y)$ |
| WeightedSoftmax | The normalized class probabilities | $\text{weight}=-\log(\text{normalized_prob})+1$ | $\text{CE}(\text{logits},y,\text{weight})$ |
| BalancedSoftmax | The number of samples per class, applied to logits | None | $\text{logits}=\log(\text{softmax})-\log(\text{class_distribution})$ |
| VSLoss | The number of samples per class with adjustments | Adjustment factors $\Delta$ and offsets $\iota$ applied to logits | $\text{CrossEntropy}\left(\frac{x_{c}}{\Delta_{c}}+\iota_{c},y\right)$ |

MixUp can improve the feature representation
MixUp (Zhang, ), while often included in data augmentation strategies, does not always improve performance for long-tailed learning when used alone, as shown in the results. However, it shows value in blending head and tail data, facilitating knowledge transfer, reducing head class prediction confidence, and enhancing feature encoder generalization, as evidenced in two-stage decoupling work (Zhong et al., ; Li et al., ). MiSLAS (Zhong et al., )
also empirically observed that data mixup is beneficial to feature learning but has a negative effect on classifier training under the two-stage training scheme. Therefore, assessing MixUp based solely on performance is not fair; integrating it with other methods, as seen in MisLAS and GCL, is essential. The removal of MixUp from GCL in our experiments led to a significant performance decline, e.g, from 64.06% to 62.01% for ISIC-2019-LT with $r=100$, underscoring its crucial role.

Use two-stage training as a general paradigm. Although end-to-end training is often considered to be elegant, addressing long-tailed problems undeniably requires distinct solutions at both the feature and classifier levels and two-stage methods provide significant flexibility in this regard. For feature representation, methods like mixup, as previously discussed, can enhance the representation learning, and self-supervised learning is another potential strategy to improve the generalization of feature encoders. At the classifier level, designing new classifiers can help correct the bias introduced by linear layers. We will delve into these aspects in more detail later.

Dilemmas of self-supervised learning for LTMIC. SSL appears promising for addressing long-tailed problems (Kang et al., ; Cui et al., ), as it works without the need for label supervision, allowing the feature encoder to obtain more generalizable feature representation. We explore the general SSL methodologies such as MoCo (He et al., ), BYOL (Grill et al., ), SimCLR (Chen et al., ), and SSL for long-tailed learning such as SCL (Khosla et al., ), KCL (Kang et al., ) and PaCo (Cui et al., ) but the performance is not satisfactory with catastrophic performance drop. We conclude this for several reasons: (1) The amount of medical imaging data in this study is limited, especially for a LTMIC setting, while SSL often relies on large quantities of unlabeled data. (2) SSL typically depends on specific hyper-parameter designs/tuning such as data augmentations, which are crucial to the second-stage fine-tuning. However, the lack of guidance in these aspects diminishes the possibility of achieving optimal practice. We have provided implementations of some SSL methodologies in our codebase for further modification or improvement by researchers.

Modify classifier to reduce prediction bias. The common practice for image classification is using a linear classifier, and the predictions can be formulated as $p=$Softmax$(WX+b)$. However, long-tailed class imbalance often results in larger classifier weight norms $W$ for head classes than tail classes, which makes the predictions easily biased to dominant classes.
In Fig. 3, We visualize the trade-off between shot-based group performance and weight norms. This indicates that re-sampling strategies such as RW and CBLoss can further calibrate the classifier weight norms.
GCL and DisAlign adopt a simple normalized linear classifier with learnable parameters to scale the predictions, leading to an optimal calibration of the weight norms. There are also some other compacted classifier designs (Kang et al., ; Tang et al., ). Considering that different classifier designs and other strategies, such as resampling methods, may conflict with each other, we still recommend starting with a simple design like a normalized linear classifier in practice.

Figure: Figure 2: The performance and weight norms of the model trained from ISIC-2019-LT ($r$=500).
Refer to caption: x2.png

LTMIC improves out-of-distribution detection. Open Long-tailed Recognition (OLTR) extends long-tailed learning by incorporating out-of-distribution (OOD) detection as an additional task. Models trained on imbalanced datasets typically struggle to generalize well to tail classes, making OLTR especially challenging due to the confusion between tail and OOD samples. To assess whether the LTMIC methods improve OOD detection capabilities, we used the OpenOOD codebase (Yang et al., ) and evaluated six in-built OOD detection methods. We used the model trained on the OrganAMNIST dataset, using its test set as closed-set samples and ImageNet (Deng et al., ) as OOD samples, each with 1,000 randomly selected images. AUROC was used as the evaluation metric for binary classification. As shown in Fig. 3, LTMIC leads to significant performance improvements across various OOD detection methods.

Using imbalanced validation dataset for checkpoint selection. Ideally, a balanced validation set would be preferred for selecting the best model checkpoint, as it ensures a fairer evaluation on the test set. An imbalanced validation set may not fully represent the underlying data distribution, and in extreme cases, some tail categories could contain fewer than 10 samples. As a result, achieving high performance on the validation set may not translate to strong generalization on the test set or in real-world applications. Therefore, the method should demonstrate stability during training, with minimal performance fluctuations as it progresses. Without repeating trials, we conducted following two sets of experiments: (1) For each epoch, we evaluated both the validation and test sets. We then calculated the difference between the best test set performance and the test set performance at the epoch where the validation set achieved its highest performance. (2) We calculated the difference between the test set performance at the epoch with the best validation set performance and the test set performance at the final epoch. The results shown in Fig. 4 indicate that GCL demonstrates both strong overall performance and stable convergence during model training.

Figure: (a)
Refer to caption: x4.png

Multi-label classification is more challenging. Compared to multi-class (MC) classification, multi-label (ML) classification presents additional challenges, particularly due to label co-occurrence, where some labels frequently appear together (e.g., in medical imaging, ”diabetic retinopathy” often co-occurs with ”glaucoma”). This complicates the application of re-sampling strategies that are commonly used in MC problems with new relative imbalance introduced (Wu et al., ; Ju et al., ).
Despite these complexities, many methods designed for MC tasks can be adapted for ML scenarios. Methods such as OLTR (Liu et al., ), RSKD (Ju et al., ), and HKGL (Ju et al., ) show consistent improvements over the standard ERM approach, as demonstrated in results from ODIR and RFMiD (Table 5, based on HKGL (Ju et al., )). Notably, some methods like OLTR can achieve performance gains across head, medium, and tail classes simultaneously in ML tasks. This may be because improving the model’s ability to detect positive samples in tail classes reduces overall misclassification of negative samples, thereby enhancing performance even for head classes. For MC tasks, we resampled datasets using a Pareto distribution to focus on the total number of data samples and the degree of imbalance. However, in ML tasks, the presence of label co-occurrence makes following a Pareto distribution impossible. This highlights the greater complexity of long-tailed multi-label learning compared to multi-class classification, where the interplay of factors such as imbalance ratio, label co-occurrence, and category distribution makes it challenging to draw unified conclusions or insights.

**Table 5: The results of comparison study on multi-label classification datasets. * - denotes the methodology is not supported in MONICA.**
| Dataset | RFMiD | ODIR | CheXpert |  |  |  |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Split | Test Set | Off-Site | On-Site | Test Set |  |  |  |  |  |  |  |  |  |  |  |  |
| Groups | Many | Medium | Few | Average | Many | Medium | few | average | many | medium | Few | Average | Many | Medium | Few | Average |
| ERM | 70.93 | 57.89 | 14.85 | 47.89 | 48.47 | 46.80 | 11.22 | 35.50 | 50.74 | 36.46 | 12.79 | 33.33 | 85.78 | 57.60 | 3.67 | 42.06 |
| RS | 68.67 | 61.48 | 25.94 | 52.03 | 46.34 | 49.27 | 9.07 | 34.89 | 47.91 | 39.10 | 15.35 | 34.12 | 73.28 | 48.76 | 13.65 | 45.23 |
| RW | 70.27 | 60.00 | 18.71 | 49.66 | 50.56 | 48.12 | 11.57 | 36.75 | 51.39 | 37.86 | 17.92 | 35.72 | 74.33 | 46.27 | 14.20 | 44.93 |
| OLTR | 71.25 | 60.22 | 20.77 | 50.75 | 47.37 | 45.02 | 11.86 | 34.75 | 50.11 | 36.01 | 20.78 | 35.63 | - | - | - | - |
| RSKD | 70.55 | 59.63 | 22.15 | 50.78 | 48.09 | 47.78 | 10.82 | 35.56 | 48.89 | 38.61 | 31.21 | 39.57 | - | - | - | - |
| Focal | 70.65 | 55.53 | 16.42 | 47.53 | 46.63 | 46.89 | 13.32 | 35.61 | 47.92 | 35.41 | 10.49 | 31.27 | 72.35 | 55.11 | 9.68 | 45.71 |
| LDAM | 46.67 | 3.19 | 1.18 | 17.01 | 41.14 | 8.22 | 0.48 | 16.61 | 42.97 | 5.10 | 0.55 | 16.21 | 63.63 | 10.14 | 3.29 | 25.58 |
| CBLoss-Focal | 67.73 | 50.89 | 24.65 | 47.77 | 39.30 | 47.44 | 10.00 | 32.25 | 43.40 | 32.31 | 8.60 | 28.10 | 65.83 | 10.45 | 3.00 | 26.30 |
| DBLoss-Focal | 68.16 | 55.27 | 18.94 | 47.46 | 48.39 | 47.11 | 27.83 | 41.11 | 50.06 | 37.60 | 12.96 | 33.54 | 74.44 | 46.12 | 13.95 | 44.83 |
| ASL | 68.25 | 58.25 | 19.59 | 48.70 | 47.93 | 47.89 | 18.57 | 38.13 | 51.69 | 37.36 | 23.70 | 37.58 | 73.86 | 45.08 | 13.50 | 44.15 |
| HKGL | 69.75 | 61.26 | 25.98 | 52.33 | 49.02 | 48.26 | 28.05 | 41.78 | 51.58 | 36.82 | 28.98 | 39.12 | - | - | - | - |

What makes an essential strong baseline? Our analysis shows that the most advanced long-tailed learning methods no longer focus on improving a single strategy. Instead, they integrate re-sampling, information augmentation, and module improvements, as exemplified by GCL.
We would like to emphasize that LTMIC is primarily an engineering-focused effort. These include using more sophisticated augmentation techniques like RandAugment (Cubuk et al., ), employing models with larger parameters, and introducing various other tricks such as learning scheduler, attention mechanism, and knowledge distillation (Hinton et al., ). However, we have reservations about the results of these attempts to maintain focus on the core methods under investigation and avoid diluting the primary contributions of this study. Finally, given the inherent imperfections of long-tailed data, it is unrealistic to assume that a single method can deliver optimal performance across all categories. Therefore, it is necessary to consider trade-offs in performance between different shot-based groups and select methods based on specific needs.

Integration of medical domain prior knowledge. While this paper extensively explores state-of-the-art methods designed for natural image classification and validates their generalization across datasets from various medical domains, we still advocate for the integration of medical domain knowledge to develop specialized techniques, such as hierarchical learning (Ju et al., ), tailored to specific medical challenges. Incorporating prior knowledge helps guide the model to focus on critical features, thereby accelerating convergence and improving training efficiency, finally benefiting the overall performance. More importantly, the integration of clinical insights enhances model interpretability, enabling more transparent and clinically relevant decision-making, from which the importance and value of LTMIC research are exactly highlighted and appreciated.

## 5 Conclusion

In this study, we introduced MONICA, a comprehensive benchmark for long-tailed medical image classification (LTMIC). Our findings emphasize the importance of integrating techniques from multiple aspects including re-sampling, data augmentation, and module improvements, offering valuable practical guidance for future research in LTMIC. The modular design of our codebase further facilitates the application and comparison of these methods across various medical imaging tasks.

Limitations and Future Works This work is largely limited to the implementation of partial existing long-tailed learning works. Due to the unavailability of code, our implementation relies heavily on the details provided in the papers, which may lead to variations in performance. Another limitation is that multi-label learning, as a significant challenge within long-tailed learning, has distinct problem definitions, implementations, and evaluation metrics. However, this paper does not delve deeply into this aspect but instead provides a brief introduction and experimental results on some mainstream datasets. Finally, no novel metrics are proposed for better quantitative analysis. In future work, it is promising to extend our benchmark towards update works and more long-tailed learning tasks such as long-tailed multi-label learning, regression, object detection, and semantic segmentation.

## Appendix A Appendix

**Table 6: The complete comparison study results on ISIC-2019-LT benchmarks.**
| ISIC-2019-LT |  |  |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Imbalance Ratio | $r=100$ | $r=200$ | $r=500$ |  |  |  |  |  |  |  |  |  |
| Methods | Head | Medium | Tail | Average | Head | Medium | Tail | Average | Head | Medium | Tail | Average |
| ERM | 79.00 | 60.67 | 38.33 | 59.33 | 78.50 | 56.67 | 27.00 | 54.06 | 78.00 | 46.67 | 12.67 | 45.78 |
| RS | 69.50 | 61.33 | 49.33 | 60.06 | 76.00 | 62.67 | 36.33 | 58.33 | 78.50 | 44.00 | 19.67 | 47.39 |
| RW | 68.00 | 55.33 | 53.67 | 59.00 | 73.50 | 54.67 | 41.33 | 56.50 | 62.00 | 38.00 | 34.33 | 44.78 |
| MixUp | 78.00 | 50.67 | 35.67 | 54.78 | 83.00 | 46.67 | 21.33 | 50.33 | 76.00 | 48.00 | 9.00 | 44.33 |
| Focal | 73.50 | 54.00 | 44.33 | 57.28 | 79.50 | 53.00 | 31.00 | 54.50 | 83.00 | 44.00 | 13.00 | 46.67 |
| cRT | 74.50 | 59.67 | 55.67 | 63.28 | 81.00 | 60.00 | 39.67 | 60.22 | 63.50 | 48.33 | 28.00 | 46.61 |
| T-Norm | 78.00 | 53.33 | 34.00 | 55.11 | 83.50 | 48.67 | 17.67 | 49.94 | 79.50 | 40.33 | 3.67 | 41.17 |
| LWS | 72.50 | 52.67 | 45.33 | 56.83 | 79.50 | 51.33 | 32.67 | 54.50 | 68.00 | 43.33 | 29.67 | 47.00 |
| KNN | 70.00 | 55.67 | 58.67 | 61.44 | 77.00 | 51.33 | 46.67 | 58.33 | 75.00 | 45.33 | 27.33 | 49.22 |
| CBLoss | 70.00 | 56.33 | 61.33 | 62.56 | 62.00 | 59.33 | 46.67 | 56.00 | 55.00 | 40.33 | 41.67 | 45.67 |
| CBLoss_Focal | 71.00 | 57.67 | 52.67 | 60.44 | 72.50 | 52.00 | 51.00 | 58.50 | 59.50 | 42.33 | 43.33 | 48.39 |
| LADELoss | 78.50 | 52.33 | 43.67 | 58.17 | 84.00 | 52.33 | 18.67 | 51.67 | 78.50 | 43.00 | 14.00 | 45.17 |
| LDAM | 78.50 | 55.67 | 41.67 | 58.61 | 81.50 | 52.33 | 31.00 | 54.94 | 76.50 | 41.33 | 19.33 | 45.72 |
| Logits Adjust Loss | 80.50 | 50.67 | 26.33 | 52.50 | 81.00 | 49.33 | 12.67 | 47.67 | 77.50 | 36.00 | 2.67 | 38.72 |
| Logits Adjust Posthoc | 75.00 | 59.33 | 49.33 | 61.22 | 76.50 | 50.67 | 48.00 | 58.39 | 72.00 | 43.67 | 33.00 | 49.56 |
| PriorCELoss | 65.00 | 57.00 | 56.00 | 59.33 | 75.00 | 50.33 | 48.33 | 57.89 | 62.00 | 52.67 | 43.33 | 52.67 |
| RangeLoss | 29.50 | 12.00 | 0.33 | 13.94 | 44.50 | 9.00 | 4.00 | 19.17 | 46.00 | 5.67 | 0.67 | 17.44 |
| SEQLLoss | 78.00 | 56.33 | 41.00 | 58.44 | 78.00 | 49.67 | 27.00 | 51.56 | 73.50 | 45.33 | 11.33 | 43.39 |
| VSLoss | 80.00 | 56.33 | 33.67 | 56.67 | 80.50 | 51.00 | 28.67 | 53.39 | 79.00 | 47.67 | 11.00 | 45.89 |
| WeightedSoftmax | 70.50 | 58.00 | 48.33 | 58.94 | 74.00 | 59.33 | 36.00 | 56.44 | 73.00 | 48.67 | 22.00 | 47.89 |
| BalancedSoftmax | 62.50 | 54.33 | 61.00 | 59.28 | 77.00 | 53.67 | 55.00 | 61.89 | 62.00 | 49.67 | 41.67 | 51.11 |
| De-Confound | 79.00 | 52.33 | 47.67 | 59.67 | 82.50 | 52.67 | 24.33 | 53.17 | 72.50 | 45.33 | 9.67 | 42.50 |
| DisAlign | 81.00 | 60.00 | 52.33 | 64.44 | 78.00 | 59.67 | 52.33 | 63.33 | 68.50 | 49.33 | 37.33 | 51.72 |
| GCL 1st stage | 72.00 | 56.00 | 50.33 | 59.44 | 78.50 | 55.33 | 32.67 | 55.50 | 78.00 | 41.33 | 31.67 | 50.33 |
| GCL 2nd stage | 57.50 | 63.33 | 71.33 | 64.06 | 71.00 | 56.67 | 64.33 | 64.00 | 63.50 | 55.00 | 46.00 | 54.83 |
| MiSLAS | 57.50 | 52.33 | 57.67 | 55.83 | 71.50 | 48.33 | 49.67 | 56.50 | 63.50 | 43.00 | 39.33 | 48.61 |
| RSG | 72.00 | 35.00 | 0.00 | 35.67 | 78.50 | 23.33 | 0.00 | 33.94 | 76.00 | 23.67 | 0.00 | 33.22 |
| SADE | 34.50 | 19.33 | 48.00 | 33.94 | 31.50 | 11.33 | 52.67 | 31.83 | 23.50 | 13.00 | 48.33 | 28.28 |
| SAM | 77.50 | 51.67 | 26.00 | 51.72 | 82.50 | 50.33 | 16.33 | 49.72 | 76.00 | 36.00 | 9.33 | 40.44 |
| BBN | 82.50 | 58.33 | 46.00 | 62.28 | 76.50 | 62.33 | 31.00 | 54.94 | 75.50 | 50.67 | 19.00 | 48.39 |

**Table 7: The results on ISIC-2019-LT benchmarks with training epochs of 50 and 200.**
| ISIC-2019-LT ($r=100$) |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Training Epochs | 50 | 200 |  |  |  |  |  |  |
| Methods | Head | Medium | Tail | Average | Head | Medium | Tail | Average |
| ERM | 79.00 | 60.67 | 38.33 | 59.33 | 78.50 | 54.00 | 45.00 | 59.17 |
| RS | 69.50 | 61.33 | 49.33 | 60.06 | 74.50 | 59.00 | 53.67 | 62.39 |
| RW | 68.00 | 55.33 | 53.67 | 59.00 | 78.50 | 59.00 | 51.00 | 62.83 |
| MixUp | 78.00 | 50.67 | 35.67 | 54.78 | 82.50 | 53.67 | 42.33 | 59.50 |
| Focal | 73.50 | 54.00 | 44.33 | 57.28 | 77.00 | 60.67 | 43.00 | 60.22 |
| cRT | 74.50 | 59.67 | 55.67 | 63.28 | 75.50 | 53.67 | 55.67 | 61.61 |
| T-Norm | 78.00 | 53.33 | 34.00 | 55.11 | 82.50 | 51.67 | 35.67 | 56.61 |
| LWS | 72.50 | 52.67 | 45.33 | 56.83 | 76.50 | 35.00 | 6.67 | 39.39 |
| KNN | 70.00 | 55.67 | 58.67 | 61.44 | 73.00 | 55.33 | 53.67 | 60.67 |
| CBLoss | 70.00 | 56.33 | 61.33 | 62.56 | 65.50 | 56.00 | 61.33 | 60.94 |
| CBLoss_Focal | 71.00 | 57.67 | 52.67 | 60.44 | 72.50 | 55.67 | 54.33 | 60.83 |
| LADELoss | 78.50 | 52.33 | 43.67 | 58.17 | 81.00 | 58.00 | 44.33 | 61.11 |
| LDAM | 78.50 | 55.67 | 41.67 | 58.61 | 74.50 | 59.00 | 40.00 | 57.83 |
| Logits Adjust Loss | 80.50 | 50.67 | 26.33 | 52.50 | 77.00 | 54.33 | 32.67 | 54.67 |
| Logits Adjust Posthoc | 75.00 | 59.33 | 49.33 | 61.22 | 76.00 | 59.00 | 53.00 | 62.67 |
| PriorCELoss | 65.00 | 57.00 | 56.00 | 59.33 | 73.00 | 58.33 | 56.67 | 62.67 |
| RangeLoss | 29.50 | 12.00 | 0.33 | 13.94 | 50.00 | 0.00 | 0.00 | 16.67 |
| SEQLLoss | 78.00 | 56.33 | 41.00 | 58.44 | 77.00 | 55.67 | 46.00 | 59.56 |
| VSLoss | 80.00 | 56.33 | 33.67 | 56.67 | 80.50 | 59.00 | 41.67 | 60.39 |
| WeightedSoftmax | 70.50 | 58.00 | 48.33 | 58.94 | 76.50 | 52.33 | 48.67 | 59.17 |
| BalancedSoftmax | 62.50 | 54.33 | 61.00 | 59.28 | 75.00 | 59.00 | 49.67 | 61.22 |
| De-Confound | 79.00 | 52.33 | 47.67 | 59.67 | 81.00 | 55.67 | 47.67 | 61.44 |
| DisAlign | 81.00 | 60.00 | 52.33 | 64.44 | 65.50 | 58.67 | 60.67 | 61.61 |
| GCL 1st stage | 72.00 | 56.00 | 50.33 | 59.44 | 78.50 | 59.00 | 56.33 | 64.61 |
| GCL 2nd stage | 57.50 | 63.33 | 71.33 | 64.06 | 72.50 | 63.67 | 63.67 | 66.61 |
| MiSLAS | 57.50 | 52.33 | 57.67 | 55.83 | 62.00 | 57.00 | 58.33 | 59.11 |
| RSG | 72.00 | 35.00 | 0.00 | 35.67 | 72.00 | 60.00 | 51.00 | 61.00 |
| SADE | 34.50 | 19.33 | 48.00 | 33.94 | 65.00 | 49.00 | 47.00 | 53.67 |
| SAM | 77.50 | 51.67 | 26.00 | 51.72 | 80.50 | 50.33 | 33.33 | 54.72 |
| BBN | 82.50 | 58.33 | 46.00 | 62.28 | 74.00 | 61.00 | 53.67 | 62.89 |

How training time impacts method effectiveness. Table 7 presents the results for ISIC-2019-LT ($r=100$) across different methods trained with 50 and 200 epochs. We observed that some methods benefit significantly from extended training; for example, RSG improves from 35.67% AUC at epoch 50 to 61.00% AUC at epoch 200. Conversely, methods like DisAlign exhibit performance degradation with longer training. This suggests potential instability during training and highlights the issue of using an imbalanced validation set to select checkpoints for testing, as discussed in Section 4 of the manuscript. Overall, most methods do not reach optimal performance within 50 epochs, and longer training times may be beneficial for further improvements. However, it is important to emphasize that different methods are designed with varying requirements for learning rates and optimizers depending on different datasets. As such, conclusions based solely on these results may be partial. These findings are intended to offer researchers and engineers guidance on tuning hyperparameters when training with their customized data.

The importance of pre-trained weights initialization. Using ImageNet pre-trained weights as initialization weights is highly beneficial for medical image classification. Despite the huge differences in overall style between domains, ImageNet pretraining offers a common coarse learning representation of texture features and provides a good initialization for optimization convergence. Thus, its benefits will not be further elaborated here. Table 8 presents the performance differences across various methods when using or not using ImageNet for weight initialization. It is worth noting that some methods have modified model structures, making it difficult for these models to leverage ImageNet weights for initialization. However, we would like to emphasize the importance of using well-pretrained models in practice, while the results in the table further corroborate this point.

**Table 8: The results on ISIC-2019-LT benchmarks w/ and w/o ImageNet pre-training ($r=100$).**
| ISIC-2019-LT ($r=100$) |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Initialized Weights | ImageNet-1K | Random |  |  |  |  |  |  |
| Methods | Head | Medium | Tail | Average | Head | Medium | Tail | Average |
| ERM | 79.00 | 60.67 | 38.33 | 59.33 | 73.00 | 21.33 | 0.33 | 31.56 |
| RS | 69.50 | 61.33 | 49.33 | 60.06 | 56.50 | 33.00 | 56.00 | 48.50 |
| RW | 68.00 | 55.33 | 53.67 | 59.00 | 53.00 | 24.33 | 36.67 | 38.00 |
| MixUp | 78.00 | 50.67 | 35.67 | 54.78 | 68.50 | 19.67 | 0.67 | 29.61 |
| Focal | 73.50 | 54.00 | 44.33 | 57.28 | 67.00 | 27.67 | 1.00 | 31.89 |
| cRT | 74.50 | 59.67 | 55.67 | 63.28 | 51.50 | 31.33 | 61.67 | 48.17 |
| T-Norm | 78.00 | 53.33 | 34.00 | 55.11 | 73.50 | 20.33 | 0.00 | 31.28 |
| LWS | 72.50 | 52.67 | 45.33 | 56.83 | 66.00 | 7.00 | 0.00 | 24.33 |
| KNN | 70.00 | 55.67 | 58.67 | 61.44 | 56.50 | 26.00 | 33.00 | 38.50 |
| CBLoss | 70.00 | 56.33 | 61.33 | 62.56 | 46.00 | 26.33 | 31.00 | 34.44 |
| CBLoss_Focal | 71.00 | 57.67 | 52.67 | 60.44 | 48.50 | 34.00 | 26.33 | 36.28 |
| LADELoss | 78.50 | 52.33 | 43.67 | 58.17 | 72.00 | 23.00 | 0.00 | 31.67 |
| LDAM | 78.50 | 55.67 | 41.67 | 58.61 | 71.00 | 19.00 | 0.00 | 30.00 |
| Logits Adjust Loss | 80.50 | 50.67 | 26.33 | 52.50 | 70.50 | 16.67 | 0.00 | 29.06 |
| Logits Adjust Posthoc | 75.00 | 59.33 | 49.33 | 61.22 | 58.50 | 29.33 | 33.33 | 40.39 |
| PriorCELoss | 65.00 | 57.00 | 56.00 | 59.33 | 62.00 | 28.67 | 39.33 | 43.33 |
| RangeLoss | 29.50 | 12.00 | 0.33 | 13.94 | 40.00 | 12.33 | 0.00 | 17.44 |
| SEQLLoss | 78.00 | 56.33 | 41.00 | 58.44 | 73.50 | 26.67 | 0.67 | 33.61 |
| VSLoss | 80.00 | 56.33 | 33.67 | 56.67 | 73.00 | 20.33 | 0.00 | 31.11 |
| WeightedSoftmax | 70.50 | 58.00 | 48.33 | 58.94 | 63.50 | 30.00 | 10.00 | 34.50 |
| BalancedSoftmax | 62.50 | 54.33 | 61.00 | 59.28 | 64.00 | 29.67 | 29.67 | 41.11 |
| De-Confound | 79.00 | 52.33 | 47.67 | 59.67 | 71.50 | 20.33 | 0.33 | 30.72 |
| DisAlign | 81.00 | 60.00 | 52.33 | 64.44 | 50.00 | 27.67 | 49.67 | 42.44 |
| GCL 1st stage | 72.00 | 56.00 | 50.33 | 59.44 | 65.00 | 21.33 | 0.00 | 28.78 |
| GCL 2nd stage | 57.50 | 63.33 | 71.33 | 64.06 | 30.00 | 12.33 | 64.00 | 35.44 |
| MiSLAS | 57.50 | 52.33 | 57.67 | 55.83 | 42.00 | 14.67 | 35.00 | 30.56 |
| RSG | 72.00 | 35.00 | 0.00 | 35.67 | 63.00 | 50.67 | 54.33 | 56.00 |
| SADE | 34.50 | 19.33 | 48.00 | 33.94 | 40.00 | 19.00 | 50.33 | 36.44 |
| SAM | 77.50 | 51.67 | 26.00 | 51.72 | 53.50 | 13.33 | 0.00 | 22.28 |
| BBN | 82.50 | 58.33 | 46.00 | 62.28 | 46.30 | 46.67 | 42.74 | 45.24 |

Additional details for OOD detection. To assess whether the LTMIC methods improve OOD detection capabilities, we used the OpenOOD codebase (Yang et al., ) and evaluated six in-built OOD detection methods. We used the model trained on the Organamnist dataset, using its test set as closed-set samples and ImageNet as OOD samples, each with 1,000 randomly selected images. This task mainly aims to distinguish whether the test sample belongs to known classes from OrganAMNIST or unknown classes from ImageNet. AUROC was used as the evaluation metric for this binary classification. To avoid the impact of retraining models on long-tailed recognition task performance, we selected six post-processing methods, which are as follows: OpenMax (Bendale & Boult, ), MSP (Hendrycks & Gimpel, ), TS (Guo et al., ), ODIN (Liang et al., ), MDS (Lee et al., ), and VIM (Wang et al., ). We used 0.01 for coreset sampling ratio in OpenMax. For ODIN, temperature is set as 1000 and noise is set as 0.0014.

**Table 9: The complete results on MedMNIST and KVASIR benchmarks.**
| Dataset | BloodMNIST | DermaMNIST | PathMNIST | TissueMNIST |  |  |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Methods | Head | Medium | Tail | Average | Head | Medium | Tail | Average | Head | Medium | Tail | Average | Head | Medium | Tail | Average |
| ERM | 97.27 | 97.16 | 82.00 | 92.14 | 95.82 | 61.86 | 44.08 | 67.25 | 98.46 | 99.64 | 83.07 | 93.72 | 64.38 | 62.11 | 23.21 | 49.90 |
| RS | 95.75 | 99.20 | 91.17 | 95.37 | 91.95 | 65.75 | 59.75 | 72.48 | 99.26 | 97.47 | 86.78 | 94.50 | 50.26 | 64.71 | 52.88 | 55.95 |
| RW | 97.39 | 99.04 | 88.14 | 94.86 | 87.32 | 65.31 | 69.19 | 73.94 | 96.62 | 99.74 | 88.57 | 94.97 | 59.72 | 66.15 | 43.44 | 56.44 |
| MixUp | 98.04 | 98.93 | 79.93 | 92.30 | 95.90 | 59.03 | 54.43 | 69.78 | 99.38 | 99.54 | 84.72 | 94.54 | 65.85 | 62.79 | 13.54 | 47.39 |
| Focal | 97.09 | 99.42 | 80.75 | 92.42 | 95.08 | 59.60 | 60.57 | 71.75 | 96.38 | 98.66 | 87.51 | 94.18 | 61.89 | 63.17 | 22.07 | 49.04 |
| cRT | 97.64 | 99.36 | 87.43 | 94.81 | 88.52 | 73.36 | 70.09 | 77.32 | 93.86 | 99.69 | 90.59 | 94.71 | 59.47 | 67.83 | 38.70 | 55.33 |
| T-Norm | 96.45 | 97.97 | 80.79 | 91.74 | 95.97 | 56.76 | 44.16 | 65.63 | 99.47 | 97.12 | 85.51 | 94.03 | 66.57 | 52.56 | 9.82 | 42.98 |
| LWS | 90.30 | 75.36 | 1.50 | 55.72 | 95.90 | 25.97 | 1.73 | 41.20 | 98.41 | 98.30 | 52.25 | 82.98 | 59.28 | 33.59 | 0.60 | 31.16 |
| KNN | 93.61 | 95.88 | 83.31 | 90.94 | 85.31 | 66.62 | 59.67 | 70.53 | 93.56 | 99.79 | 89.67 | 94.34 | 54.77 | 55.40 | 51.38 | 53.85 |
| CBLoss | 96.92 | 98.23 | 85.81 | 93.65 | 83.15 | 63.57 | 76.61 | 74.44 | 97.51 | 99.64 | 87.69 | 94.94 | 60.38 | 67.61 | 43.66 | 57.21 |
| CBLoss_Focal | 96.97 | 98.34 | 85.21 | 93.51 | 84.41 | 63.27 | 73.99 | 73.89 | 98.44 | 99.45 | 84.10 | 94.00 | 58.60 | 67.08 | 45.31 | 56.99 |
| LADELoss | 97.16 | 98.77 | 72.76 | 89.56 | 91.28 | 67.43 | 50.60 | 69.77 | 98.55 | 98.48 | 84.43 | 93.82 | 65.96 | 63.33 | 23.92 | 51.07 |
| LDAM | 96.74 | 98.45 | 87.32 | 94.17 | 92.62 | 63.71 | 53.15 | 69.82 | 97.37 | 98.66 | 86.07 | 94.03 | 63.31 | 64.57 | 14.64 | 47.51 |
| Logits Adjust Loss | 95.41 | 98.18 | 51.39 | 81.66 | 97.39 | 52.12 | 43.63 | 64.38 | 99.58 | 98.58 | 75.68 | 91.28 | 59.79 | 44.62 | 1.23 | 35.21 |
| Logits Adjust Posthoc | 96.65 | 97.91 | 90.27 | 94.94 | 92.62 | 64.53 | 61.47 | 72.87 | 98.07 | 99.17 | 86.19 | 94.48 | 59.03 | 66.35 | 54.78 | 60.06 |
| PriorCELoss | 96.86 | 98.02 | 91.90 | 95.59 | 77.63 | 72.80 | 70.54 | 73.66 | 98.51 | 97.01 | 88.97 | 94.83 | 55.73 | 69.21 | 51.54 | 58.83 |
| RangeLoss | 23.58 | 6.11 | 3.69 | 11.13 | 100.00 | 0.00 | 0.00 | 33.33 | 50.00 | 0.00 | 0.00 | 16.67 | 33.34 | 0.00 | 0.00 | 11.11 |
| SEQLLoss | 97.22 | 97.27 | 85.27 | 93.26 | 89.93 | 69.13 | 55.77 | 71.61 | 97.53 | 99.59 | 88.21 | 95.11 | 65.90 | 62.00 | 24.12 | 50.68 |
| VSLoss | 96.05 | 97.70 | 86.59 | 93.45 | 92.84 | 64.75 | 49.70 | 69.10 | 98.87 | 98.31 | 83.46 | 93.54 | 66.78 | 61.45 | 21.54 | 49.92 |
| WeightedSoftmax | 97.78 | 98.98 | 84.65 | 93.80 | 87.84 | 68.31 | 69.27 | 75.14 | 98.83 | 95.11 | 85.41 | 93.12 | 65.38 | 68.36 | 35.57 | 56.43 |
| BalancedSoftmax | 96.55 | 98.13 | 90.71 | 95.13 | 89.41 | 67.35 | 76.61 | 77.79 | 96.39 | 99.54 | 88.42 | 94.78 | 53.86 | 68.41 | 52.91 | 58.39 |
| De-Confound | 97.14 | 97.49 | 85.10 | 93.24 | 95.08 | 61.54 | 49.25 | 68.62 | 93.45 | 99.85 | 88.53 | 93.94 | 65.32 | 61.35 | 24.34 | 50.34 |
| DisAlign | 97.11 | 98.98 | 88.82 | 94.97 | 85.68 | 66.04 | 75.72 | 75.81 | 96.70 | 97.95 | 91.85 | 95.50 | 50.20 | 59.73 | 60.21 | 56.71 |
| GCL 1st stage | 96.98 | 99.52 | 90.70 | 95.74 | 88.07 | 65.34 | 57.95 | 70.45 | 96.16 | 99.89 | 87.80 | 94.62 | 64.44 | 69.99 | 33.69 | 56.04 |
| GCL 2nd stage | 96.43 | 99.47 | 95.56 | 97.15 | 68.75 | 73.11 | 90.03 | 77.30 | 93.00 | 99.58 | 91.79 | 94.79 | 48.26 | 68.48 | 46.93 | 54.56 |
| MiSLAS | 93.15 | 99.25 | 81.74 | 91.38 | 65.92 | 67.53 | 75.72 | 69.72 | 96.30 | 98.99 | 88.63 | 94.64 | 42.84 | 64.71 | 49.91 | 52.49 |
| RSG | 95.45 | 98.56 | 86.55 | 93.52 | 77.33 | 53.90 | 63.12 | 64.78 | 96.16 | 99.95 | 70.92 | 89.01 | 53.08 | 59.44 | 0.00 | 37.51 |
| SADE | 60.99 | 81.91 | 52.37 | 65.09 | 59.36 | 30.89 | 58.47 | 49.57 | 69.45 | 95.87 | 76.96 | 80.76 | 45.23 | 59.76 | 58.09 | 54.36 |
| SAM | 96.10 | 97.49 | 77.40 | 90.33 | 93.36 | 61.02 | 36.66 | 63.68 | 99.30 | 99.64 | 86.89 | 95.28 | 64.28 | 69.99 | 46.93 | 54.56 |
| BBN | 96.24 | 98.23 | 89.39 | 94.62 | 91.87 | 70.98 | 64.02 | 75.62 | 98.14 | 98.95 | 90.33 | 95.80 | 58.62 | 69.68 | 43.29 | 57.19 |
| Dataset | OrganAMNIST | OrganCMNIST | OrganSMNIST | KVASIR |  |  |  |  |  |  |  |  |  |  |  |  |
| Methods | Head | Medium | Tail | Average | Head | Medium | Tail | Average | Head | Medium | Tail | Average | Head | Medium | Tail | Average |
| ERM | 97.27 | 97.16 | 82.00 | 92.14 | 93.50 | 59.20 | 65.18 | 72.63 | 88.33 | 56.89 | 66.90 | 70.71 | 95.50 | 93.25 | 60.33 | 83.03 |
| RS | 95.75 | 99.20 | 91.17 | 95.37 | 92.49 | 57.14 | 64.06 | 71.23 | 89.99 | 52.61 | 66.39 | 69.66 | 95.75 | 94.50 | 62.83 | 84.36 |
| RW | 97.39 | 99.04 | 88.14 | 94.86 | 92.20 | 68.91 | 66.84 | 75.98 | 83.01 | 58.68 | 69.60 | 70.43 | 96.75 | 88.75 | 67.67 | 84.39 |
| MixUp | 98.04 | 98.93 | 79.93 | 92.30 | 92.25 | 57.43 | 66.97 | 72.22 | 89.15 | 50.09 | 64.25 | 67.83 | 96.25 | 92.25 | 55.33 | 81.28 |
| Focal | 97.09 | 99.42 | 80.75 | 92.42 | 91.98 | 57.02 | 63.36 | 70.79 | 86.24 | 54.93 | 64.77 | 68.65 | 95.75 | 92.00 | 57.67 | 81.81 |
| cRT | 97.64 | 99.36 | 87.43 | 94.81 | 92.11 | 63.17 | 67.38 | 74.22 | 88.23 | 58.99 | 69.79 | 72.34 | 95.50 | 88.00 | 68.83 | 84.11 |
| T-Norm | 96.45 | 97.97 | 80.79 | 91.74 | 92.17 | 55.15 | 63.93 | 70.42 | 86.82 | 54.09 | 62.73 | 67.88 | 98.00 | 89.75 | 59.50 | 82.42 |
| LWS | 90.30 | 75.36 | 1.50 | 55.72 | 44.87 | 7.82 | 1.26 | 17.98 | 59.44 | 32.10 | 23.94 | 38.49 | 95.00 | 87.00 | 61.67 | 81.22 |
| KNN | 93.63 | 95.88 | 83.31 | 90.94 | 88.66 | 56.54 | 65.97 | 70.39 | 85.33 | 54.82 | 66.60 | 68.92 | 95.00 | 91.50 | 67.00 | 84.50 |
| CBLoss | 96.92 | 98.23 | 85.81 | 93.65 | 94.21 | 58.69 | 66.39 | 73.10 | 79.87 | 65.08 | 73.91 | 72.96 | 93.50 | 90.25 | 71.00 | 84.92 |
| CBLoss_Focal | 96.97 | 98.34 | 85.21 | 93.51 | 94.16 | 57.99 | 67.36 | 73.17 | 84.18 | 60.42 | 71.42 | 72.00 | 95.00 | 84.75 | 71.50 | 83.75 |
| LADELoss | 97.16 | 98.77 | 72.76 | 89.56 | 91.22 | 61.58 | 62.86 | 71.88 | 83.32 | 59.04 | 64.92 | 69.09 | 96.25 | 90.00 | 60.83 | 82.36 |
| LDAM | 96.74 | 98.45 | 87.32 | 94.17 | 90.98 | 61.68 | 69.16 | 73.94 | 86.07 | 58.66 | 68.60 | 71.11 | 96.00 | 89.00 | 63.17 | 82.72 |
| Logits Adjust Loss | 95.41 | 98.18 | 51.39 | 81.66 | 91.37 | 55.47 | 56.10 | 67.65 | 89.31 | 53.47 | 56.78 | 66.52 | 96.75 | 93.50 | 48.17 | 79.47 |
| Logits Adjust Posthoc | 96.65 | 97.91 | 90.27 | 94.94 | 91.75 | 62.72 | 71.24 | 75.24 | 82.78 | 62.01 | 69.10 | 71.29 | 93.00 | 89.75 | 70.67 | 84.47 |
| PriorCELoss | 96.86 | 98.02 | 91.90 | 95.59 | 92.83 | 64.86 | 67.17 | 74.95 | 86.08 | 60.96 | 72.36 | 73.13 | 95.50 | 89.25 | 70.33 | 85.03 |
| RangeLoss | 23.58 | 6.11 | 3.69 | 11.13 | 0.00 | 33.33 | 0.00 | 11.11 | 0.00 | 33.33 | 0.00 | 11.11 | 0.00 | 25.50 | 0.00 | 8.50 |
| SEQLLoss | 97.22 | 97.27 | 85.27 | 93.26 | 94.52 | 61.45 | 65.15 | 73.71 | 88.83 | 55.61 | 63.84 | 69.43 | 97.50 | 90.50 | 58.00 | 82.00 |
| VSLoss | 96.05 | 97.70 | 86.59 | 93.45 | 94.79 | 61.42 | 67.08 | 74.43 | 86.46 | 58.66 | 66.70 | 70.61 | 96.75 | 92.75 | 62.83 | 84.11 |
| WeightedSoftmax | 97.78 | 98.98 | 84.65 | 93.80 | 95.06 | 65.70 | 69.36 | 76.70 | 84.02 | 60.64 | 67.67 | 70.77 | 95.25 | 93.25 | 68.17 | 85.56 |
| BalancedSoftmax | 96.55 | 98.13 | 90.71 | 95.13 | 93.58 | 62.68 | 67.61 | 74.62 | 81.44 | 59.27 | 70.05 | 70.26 | 95.25 | 88.25 | 65.83 | 83.11 |
| De-Confound | 97.14 | 97.49 | 85.10 | 93.24 | 92.56 | 64.81 | 64.58 | 73.98 | 87.03 | 61.35 | 66.07 | 71.48 | 96.25 | 90.50 | 66.50 | 84.42 |
| DisAlign | 97.11 | 98.98 | 88.82 | 94.97 | 93.74 | 62.69 | 72.65 | 76.36 | 88.09 | 58.77 | 66.50 | 71.12 | 95.75 | 89.25 | 68.50 | 84.50 |
| GCL 1st stage | 96.98 | 99.52 | 90.70 | 95.74 | 94.15 | 60.46 | 68.99 | 74.54 | 88.64 | 62.68 | 73.08 | 74.80 | 96.00 | 93.75 | 65.67 | 85.14 |
| GCL 2nd stage | 96.43 | 99.47 | 95.56 | 97.15 | 92.99 | 64.40 | 75.42 | 77.60 | 81.80 | 66.23 | 76.79 | 74.94 | 95.25 | 96.25 | 66.17 | 85.89 |
| MiSLAS | 93.15 | 99.25 | 81.74 | 91.38 | 84.86 | 51.08 | 64.40 | 66.78 | 74.66 | 52.74 | 69.39 | 65.60 | 94.25 | 85.75 | 71.00 | 83.67 |
| RSG | 95.45 | 98.56 | 86.55 | 93.52 | 86.48 | 52.89 | 64.18 | 67.85 | 82.77 | 66.06 | 70.06 | 72.96 | 95.50 | 91.50 | 65.33 | 84.11 |
| SADE | 60.99 | 81.91 | 52.37 | 65.09 | 67.88 | 46.28 | 52.30 | 55.49 | 72.74 | 48.62 | 59.27 | 60.21 | 85.75 | 77.00 | 43.50 | 68.75 |
| SAM | 96.10 | 97.49 | 77.40 | 90.33 | 91.34 | 59.89 | 64.26 | 71.83 | 90.77 | 56.11 | 63.36 | 70.08 | 97.00 | 93.25 | 60.50 | 83.58 |
| BBN | 96.24 | 98.23 | 89.39 | 94.62 | 91.55 | 63.41 | 63.86 | 72.94 | 87.91 | 59.28 | 71.74 | 72.98 | 95.25 | 89.25 | 70.00 | 84.83 |

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 3
presentation: 2
questions: - Are their any key trends that you'll observed across the board to narrow down the design space for the future across the general task space? The results are not convincing in any one direction across the board on tasks and methods
- Do you'll think stronger backbones can help learn better features?
- Did you'll consider trying complex augmentation techniques such as AugMix or even learned augmentations?
soundness: 2
strengths: - A good overview of the datasets curated for this work
- important contribution of decoupling the codebase
- A good overview of the method approaches
- practically useful to AI researchers in medical imaging
summary: The authors introduce the problem of long-tailed medical image classification and challenges in the field. Then they develop MONICA which is a package to benchmark various methods ranging from different loss functions, augmentations, etc on the benchmark datasets across medical image classification tasks. They provide an overview of datasets and methods and experiment results on the datasets. The authors additionally share learnings and observations.
weaknesses: - It would help to expand the benchmark datasets and bring in a canonical set for a field such as Camlyon for Pathology, etc. WILDS (medical subset) is a great example of a dataset to bring in to this benchmarking codebase
- Resnet-50 is used as a backbone but the community has generally moved on to more complex backbones such as ConvNext / Swin or foundation model backbones for different datasets. 
- Generally the community uses pretrained backbones rather than training the backbones from the scratch.
- The same backbone is used for every task for fairness but generally a sweep over backbones would help since different modalities and tasks require different approaches
- Top-1 accuracy is an in appropriate metric for model selection in imbalances settings and AUROC, AUPRC, F1 should be used
- error bars are missing in experiments
- More thorough error analysis
- Clearer articulation of novel insights
- Better connection to clinical relevance
- More detailed ablation studies

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 2
questions: See weakness sections. Some more questions below:

Could you elaborate on why the results of self-supervised learning and OOD detection are relevant in this paper? They seem out of place given the main focus on long-tailed classification.

Why did the authors not include a domain-specific analysis (e.g., which methods work better for certain medical fields)? It seems like an important missed opportunity.
soundness: 2
strengths: The paper attempts to provide a comprehensive benchmark for long-tailed medical image classification. The idea of integrating multiple existing methods and datasets into a unified platform could potentially be useful for researchers who want to compare various methodologies under a standardized framework.
summary: This paper presents a unified benchmark for long-tailed learning in the medical domain by integrating several existing datasets and implementing a complete pipeline from data loading to model training and evaluation. The authors claim that the benchmark supports over 30 methods for comparison and provides an analysis of their performances.

- Update after the discussion phase:

Thank you for the detailed responses! I've raised my score. While my concerns are not entirely resolved, I believe with careful revisions, its future version has the potential to be accepted.
weaknesses: 1. Motivation. The paper lacks sufficient justification for evaluating long-tailed problems specifically in medical imaging tasks. While the authors mention some motivations at the beginning, these arguments are not convincing. Is there a fundamental difference between long-tailed problems in medical imaging and those in conventional tasks? Would this difference necessitate different methodologies? Even if the data modalities and evaluation methods are distinct (e.g., balanced vs. imbalanced test sets), would this lead to fundamentally different approaches? The paper analyzes multiple methods based on this premise but fails to provide insightful conclusions, which further deepens my skepticism about the motivation.

2. Dataset Contribution. Although the paper claims to use 12 datasets, 7 of these come from MedMNIST, and several of them are derived from previous work. This reduces the originality of the dataset contribution. Furthermore, the split between multi-class and multi-label datasets is 9/3, respectively. It is worth noting that many existing studies have already utilized MedMNIST for long-tailed learning (https://scholar.google.com/scholar?cites=11226954386823169312&scipsc=1&q=long+tail). Given that 7 out of the 12 datasets in this paper are from MedMNIST, why should users choose MONICA over MedMNIST, which already has extensive use and coverage in the medical imaging field? Additionally, the experimental methods used for multi-class and multi-label datasets are almost entirely different, and the analysis of multi-label results is limited to a single vague statement that multi-label classification is more challenging. This gives the impression that multi-label datasets were included just for the sake of completeness, rather than being a key focus.

3. Code Contribution. The code is not provided in the appendix, nor is there an anonymous GitHub link, which means the authors' claims about the code cannot be verified. By comparison, the NeurIPS D&B track (single-blind review) usually includes dataset or code links, along with information about author affiliations, licenses, and ethics. Although such links may be added after acceptance, this suggests that such work may not be well-suited for ICLR's double-blind review process.

    Additionally, the description of the code structure in Section 3.1 is not particularly informative. The modular design described is basic and lacks novel insights. A more impactful modular design, like in mmdetection, which breaks down components into backbone, neck, and bbox head, would have been more meaningful. As it stands, the description feels unnecessary.

4. Experimental Analysis Lacks of Insights. Comments below:

    - Despite using multiple datasets, the authors only provide a generic / systematic comparison of the methods without analyzing differences across domains. For example, there is no discussion about which methods are better suited for dermatology versus ophthalmology. Almost all discussion is very general, without any specific insights related to medical applications. This diminishes the value of using 12 datasets, as the conclusions drawn are not substantially different from what could be obtained from a single dataset.

    - The analysis in Section 4.2 is poorly organized. There is no clear structure, with the discussion jumping from evaluation metrics (e.g., "Curse of shot-based group evaluation") to re-sampling methods, MixUp, two-stage training, and even self-supervised learning in a seemingly random fashion. Many claims are also not supported by data. The overall takeaway from the experimental section is unclear, and I did not gain any insights on how to design better models.

    - In Section 4.1, there is inconsistency in the training strategies used: some methods use a unified training strategy, while others use the one specified in the original paper (e.g., SAM, Line 306), with no explanation for this discrepancy.

    - There are issues with the tables, such as Table 2, where it is unclear what methods like ERM, cRT, and LWS represent, as they are not referenced properly. Additionally, Section 3.2.3 does not fully align with the table.

    - The categorization of methods is confusing. The authors categorize methods into three types—class re-sampling, information augmentation, and module improvement—but later mention that re-sampling and MixUp are used in many methods, making the classification in Tables 2/3 somewhat meaningless.

    - The discussion on self-supervised learning (Line 398) appears out of place, as it is not introduced earlier.

    - Similarly, the mention of OOD detection (Line 421) is abrupt and lacks context.

    - The section on using an imbalanced validation dataset for checkpoint selection is unclear about its purpose. The conclusion seems to be that GCL exhibits lower fluctuations, but the reasoning and implications are not well explained. Additionally, Figure 4 lacks labels for the x and y axes, making interpretation difficult.

    - Line 475 suddenly states that multi-label classification is more challenging without providing adequate context or analysis.

    - Line 504 claims that "the most advanced long-tailed learning methods no longer focus on improving a single strategy," but this claim is not well-supported by the preceding analysis.

---

confidence: 5
contribution: 2
initial_rating: 5
presentation: 2
questions: Some of the datasets have been changes in terms of distributions, would you share the modified datasets or the code on making the changes?

What are the performance of the SSL models?

There are some typos such as quotation markers etc, please correct.
soundness: 3
strengths: Long-tailed learning is an extremely challenging problem, this work can serves as a comprehensive and reproducible benchmark, encouraging further advancements in long-tailed medical image learning.

It covers most of the strategies that deal with long-tailed problems, and also include 12 datasets from different application domains.
summary: This work introduced Medical OpeN-source Long-taIled ClassifiCAtion (MONICA), a comprehensive benchmark for long-tailed medical image classification (LTMIC). It includes a unified, well-structured codebase integrating over 30 methods developed in relevant
fields and 12 long-tailed medical datasets covering 6 medical domains.
weaknesses: This work doesn't introduce any new datasets or methods. It is a collection of datasets (multi class or multi label) that are already publicly available without justifications as they are many other such kind of long tail datasets available. Also, they have changed some of the original datasets, it would not be useful if they don't share the modified datasets.

They only tried ResNet for the tasks, would be nicer to make comparisons with other models. Also the discussions on SSL models seem not supported by any data.

---

confidence: 4
contribution: 3
initial_rating: 8
presentation: 3
questions: - Is it possible that the chosen hyperparameters used across all methods happen to be more advantageous for certain methods and suboptimal for others? In one sense, using the same set of hyperparameters across methods appears “fair”; however, it may actually be more fair to individually tune each method on each task. I recognize the difficulty of conducting fair comparisons in such a large-scale experimental setting, where it is costly to, e.g., run multiple trials of all experiments. I am not asking the authors to necessarily perform such experiments, but rather to consider this point and perhaps comment on it as a limitation/consideration.
- Can the authors provide a summary of practical suggestions for which methods to use in a few sentences near the Conclusion?
- I might suggest including the **rank** of each method on a given task in all tables. This would also enable you to *quantitatively* assess method performance across tasks (which method has the lowest average/median rank overall?). To make this work logistically (fit all columns in the table), you may need to reduce the precision to one decimal place, e.g.
- The two paragraphs “LTMIC improves out-of-distribution detection” and “Using imbalanced validation dataset for checkpoint selection” are not properly set up. For the former, what does it mean to use “ImageNet as OOD samples, with 1,000 randomly selected images”? What exactly is the task, how is it formulated, and how are experiments conducted? Further, why do we care about this model behavior? For the former, Figure 4 and its findings are confusing – why exactly does this demonstrate “stable convergence”? My general advice: **Use the methods section to describe and prepare the reader to understand everything that appears in the results**. When I come to these results sections, I should already have an idea of what experiments you have performed.

**Minor comments/questions:**
- Avoid editorializing with value judgments: “benchmark is **meticulously** designed”; “we… develop a… **well-structured** codebase”; “our work provides **valuable** practical guidance”. Simply present your work and let the reader make these judgments!
- “data imbalance learning” is not a phrase I have heard. Perhaps “imbalanced learning”?
- “unified, strictly formulated, and comprehensive benchmark”. Unsure what “strictly formulated” means. Could simply say “unified, comprehensive benchmark”
- “we build a… codebase…, which implements over 30 methods… and evaluated on 12… datasets”. It seems that “evaluated on” is the wrong tense; also, what is being evaluated?
- This does not belong in an abstract: “We hope this codebase serves as a comprehensive and reproducible benchmark, encouraging further advancements in long-tailed medical image learning.”
- Often unnecessary inclusion of “the” before concepts: “The deep learning techniques”; “the collected image datasets”; “the long-tailed imbalance”
- “The deep learning techniques have proven effective for most computer vision tasks benefiting from the grown-up dataset scale.” Remove “The”; what does “grown-up dataset scale” mean? “Grown-up” is not the right adjective – be more concrete.
- Refrain from claims like “always result” (line 57) – soften to “usually” or similar
- Confused by this justification: “it is vital to recognize these rare diseases in real-world practice, as they are relatively rare for doctors and may also lack diagnostic capacity.” This reads as “it is vital to recognize rare diseases because they are rare”.
- Line 65: can change “contributions, i.e.,” -> “contributions:”
- Be consistent with capitalization/presentation of terms: “Re-sampling” vs “re-sampling”; “Module improvement” vs. “Module Improvement”; “mnist” vs. “MNIST”; “mixup” vs. “MixUp”; etc.
- Line 82: “we are still curious to explore”. Perhaps just “we aim to explore”?
- “The partition schemes are vita important”. What does “vita” mean?
- The last two paragraphs of the introduction are probably better off being formatted as bulleted or numbered lists. Also, it is unclear why these numbered lists are formatted differently: **1) xxx.** vs. (1) xxx.
- “of class $k$ where $\rho$ denoted as imbalance ratio”. The phrase “denoted as” is awkward + need a comma after $k$
- “a common assumption in long-tailed learning is when the classes are sorted by cardinality in decreasing order” I’m not sure what this means or why this represents an “assumption”. I would just remove this sentence since it does not seem to be used later.
- Line 144: “is a long-tailed version constructed from”. Need to say it is a version “of” something; alternatively, use a word other than “version” like “dataset”
- Inconsistent spacing/use of commas in numbers. “10, 015” -> “10,015”; “3200 fundus images” -> “3,200 fundus images”
- Inconsistent spacing around commas and colons: “training/off-site testing / on-site testin”; “7 : 1: 2”; etc.
- “Liver Tumor Segmentation Benchmark” -> “the Liver Tumor Segmentation Benchmark”
- I realize it is hard to categorize some methods into one bin but GCL loss going in Information Augmentation is interesting, particularly since all other losses fall under re-sampling. It seems to also have module improvement as well.
- “Causal classifier (Tang et al., 2020) resorted to causal inference for keeping the good and removing the bad momentum causal effects in long-tailed learning.” The phrase “resorted to” is strange and has a negative connotation; also, what do “good” and “bad” mean?
- “All these designs are for the fairness and the practicality of the comparison on the benchmark.” Too vague – in what specific way do these support fairness?
- Table 3: Inconsistent “Avg” vs “Avg.” vs “avg”
- Table 4: Consider using a line break occasionally (so one loss function occupies two rows). This would allow you to use a larger font size. Also, be consistent “CrossEntropy” vs “CE”?
- “assessing MixUp based solely on performance is not fair”. Soften to “may not be fair”
- “led to a significant performance decline, e.g,”. Refrain from saying “significant” without statistical significance test + change “e.g,” -> “e.g.,”
- “Use two-stage training as a general paradigm” sounds like a command. Perhaps “Using”?
- Define “SSL” acronym at first use
- “Modify classifier to reduce prediction bias” -> “Classifier modification to reduce prediction bias”
- “In Fig. 2, We visualize” -> “In Fig. 2, we visualize”
- Table 5 indicates the meaning of asterisk, which is never used in the table.
- “models with larger parameters”. The parameters are not “larger” – could say “more parameters” or “a larger parameter count” perhaps.
soundness: 4
strengths: - This work addresses an important problem, namely the variability in dataset/hyperparameters/etc. when evaluating LT learning methods for medical image classification tasks. These variations in setting make head-to-head comparisons difficult, so MONICA serves to provide a “fair playing ground” for these LT learning methods.
- The framework will become publicly available and should serve as an extensible resource going forward for LT medical image classification research.
- The organization and presentation quality of the paper is strong, with helpful use of formatting (typesetting, color, etc.) and high-quality figures.
- Experiments are very thorough, spanning many relevant methods, datasets, and tasks.
- Discussion is thoughtful, going beyond simply displaying all benchmark results. The authors try to synthesize takeaways, provide caveats/limitations, assess out-of-distribution performance, and more.
summary: This paper presents a framework and codebase for structured benchmarking of long-tailed (LT) learning methods on various medical image classification tasks. The benchmark, MONICA, implements over 30 LT learning methods, with comprehensive experiments assessing performance across 12 LT medical image classification datasets spanning 6 different modalities. The experiments analyze which methods, and categories of methods, provide the most benefit to LT medical image classification across tasks in a controlled environment.
weaknesses: - Writing can be improved throughout. See specific comments below for examples of awkward wording, inconsistent naming, grammatical errors, etc.
- It is possible that choosing a fixed set of hyperparameters across methods unintentionally advantages certain methods. Ideally, one could argue that each method should be individually tuned on each task; however, I am aware that this would require a vast amount of resources and time, so I do not consider this a major limitation. More practical solutions to enhance the benchmark would be the following: (i) uncertainty estimates should be provided (e.g., bootstrapped confidence intervals or standard deviations over multiple runs), and (ii) multiple performance metrics should be provided (e.g., AUROC).

Meta-review: The paper introduces MONICA (Medical OpeN-source Long-taIled ClassifiCAtion), a comprehensive benchmark designed for long-tailed medical image classification (LTMIC). MONICA integrates 12 publicly available datasets across six medical domains and implements over 30 long-tailed (LT) learning methods. The framework provides a unified, reproducible platform for evaluating LT learning strategies, aiming to standardize comparisons and advance research in this challenging problem. The authors present a detailed overview of datasets, methods, and experimental results, along with observations and learnings.

Strength: The benchmark addresses the variability in dataset and evaluation settings, offering a fair and standardized framework for LT learning in medical imaging. It covers diverse datasets and strategies, making it practically useful for researchers. The publicly available, extensible codebase encourages further advancements in LTMIC research.

Weakness: The paper does not introduce new datasets or methods, relying on existing publicly available datasets, many of which are derived from prior work (e.g., MedMNIST). Experimental analysis lacks domain-specific insights:the use of a single backbone across tasks, while ensuring fairness, limits exploration of modality-specific optimizations. The results are presented in a way which may be hard to derive clear conclusions from.

Overall, this work could be potentially useful and helpful to standardize comparisons and advance research in long-tailed learning in the medical image research community. Though during the rebuttal, the manuscript has been improved a lot based on the reviewers’ comments including adding more advanced model backbones, more comprehensive evaluation metrics, more compared baselines, I agree with the reviewer’s comment that a more thorough major revision would be appreciated to further refine the work. Thus, I recommend reject based on the current version.

Decision: Reject