Imbalance-aware loss functions improve medical image classification

Authors: daniel  scholz
josef  buchner
jan c peeken
daniel  rueckert
benedikt  wiestler
Venue: MIDL_io_2024_Conference
Published: 2024-01-30 14:22:19

Abstract: Deep learning models offer unprecedented opportunities for diagnosis, prognosis, and treatment planning.
    However, conventional deep learning pipelines often encounter challenges in learning unbiased classifiers within imbalanced data settings, frequently exhibiting bias towards minority classes.
    In this study, we aim to improve medical image classification by effectively addressing class imbalance.
    To this end, we employ differentiable loss functions derived from classification metrics commonly used in imbalanced data settings: Matthews correlation coefficient (MCC) and the F1 score.
    We explore the efficacy of these loss functions both independently and in combination with cross-entropy loss and various batch sampling strategies on diverse medical datasets of 2D fundoscopy and 3D magnetic resonance images.
    Our findings demonstrate that, compared to conventional loss functions, we achieve notable improvements in overall classification performance, with increases of up to +12% in balanced accuracy and up to +51% in class-wise F1 score for minority classes when utilizing cross-entropy coupled with metrics-derived loss. Additionally, we conduct feature visualization to gain insights into the behavior of these features during training with imbalance-aware loss functions. Our visualization reveals a more pronounced clustering of minority classes in the feature space, consistent with our classification results.
    Our results underscore the effectiveness of combining cross-entropy loss with class-imbalance-aware loss functions in training more accurate classifiers, particularly for minority classes.

Main paper: 
 introduction: 
 Deep learning techniques have revolutionized medical image analysis by providing powerful tools for tasks such as diagnosis, prognosis, and treatment planning (Litjens et al., 2017; Oren et al., 2020; Pinto-Coelho, 2023 ). However, one significant challenge that persists in training deep learning models for medical image analysis is the presence of imbalanced data (Mazurowski et al., 2008; Buda et al., 2018; Johnson and Khoshgoftaar, 2019) . In many medical datasets, the distribution of classes is often skewed, with certain classes representing the minority while others dominate. This class imbalance poses a substantial hurdle for conventional deep learning pipelines, as they tend to prioritize learning the majority classes at the expense of the minority ones. Consequently, models trained on imbalanced data may exhibit biased predictions, leading to suboptimal performance, particularly for the underrepresented classes critical for accurate diagnosis or prognosis. (Cluceru et al., 2022; Foltyn-Dumitru et al., 2023) . The resulting bias against minority classes can have potentially grave consequences for patients when utilizing detection and diagnosis systems based on such biased deep-learning classifiers. This problem is further aggravated since commonly used metrics, such as unbalanced accuracy, convey overly optimistic results in imbalanced classification settings (Haixiang et al., 2017) . More appropriate metrics for evaluating classification performance in imbalanced data settings, such as the Matthews correlation coefficient (MCC) (Matthews, 1975; Gorodkin, 2004) and F1 score, have been proposed.Addressing class imbalance is crucial for developing robust and reliable deep learning models that can generalize well across diverse medical imaging datasets and yield clinically relevant insights. In this study, we focus on mitigating the effects of class imbalance in medical image classification tasks using novel approaches derived from imbalance-aware loss functions, aiming to improve the overall performance and equity across all classes. To this end, we comprehensively compare and analyze class-imbalance-aware loss functions in combination with and against established loss functions in two challenging datasets with imbalanced class distributions. In summary, our contributions are as follows:1. We investigate different loss formulations and introduce new combinations of classimbalance-aware losses by integrating the MCC and F1 score with a cross-entropy loss.2. To this end, we comprehensively compare different strategies to address class imbalance, namely, class-imbalance-aware loss functions in combination with oversampling and per-sample weighting, and established loss functions.3. We experimentally show that class-imbalance-aware loss functions increase the performance of challenging classification tasks on diverse medical imaging datasets, as well as the discernability of class representations. 
overcoming class imbalance: 
 Considering how relevant a challenge class imbalance is for training clinically applicable deep learning models, several strategies have been developed to overcome this challenge. These can be broadly grouped into (i) sampling-based, (ii) loss-based, and (iii) synthesis-based approaches.Sampling-based approaches typically aim to oversample the minority class(es) or adjust loss weights. While easy to implement and computationally cheap, this can lead to the model severely overfitting the few samples available in the minority classes (Zheng et al., 2015) .Alternatively, specific loss functions such as focal loss (Lin et al., 2017 ) may be used. Focal loss over-proportionally decreases the loss of an easy sample compared to difficult ones. Since the samples in the minority class are potentially more challenging to classify due to their low prevalence, the model is penalized strongly for misclassifying them. However, the focal formulation does not directly address the class imbalance but rather its "side effect" that minority samples are typically more challenging to classify.Another strategy to tackle class imbalance is creating synthetic examples. Earlier examples of such techniques include the Synthetic Minority Over-sampling Technique (SMOTE) (Chawla et al., 2002) . In SMOTE, the minority class is over-sampled by taking each sample and introducing synthetic examples along the line segments joining any/all of the k minority class nearest neighbors. Linear interpolation in image space, however, rarely gives sensible synthetic samples. Hence, generative models, such as generative adversarial networks (GANs) (Goodfellow et al., 2014; Qasim et al., 2020; Li et al., 2023) or diffusion models (Qin et al., 2023; Dhariwal and Nichol, 2021) have been developed to create realistic examples to supplement the minority classes. While these are powerful approaches, they are also computationally expensive and require considerable effort to develop generative models that produce realistic, helpful minority class examples, which is again challenging. Extending them to different tasks and classes usually requires extensive re-training. 
class-imbalance-aware loss functions: 
 To mitigate the imbalanced data issue in medical imaging datasets, imbalance-aware loss functions emerge to enhance the performance of minority classes. These loss functions generally assign larger loss values to misclassified instances of these less prevalent classes. This adaptation serves to rectify the disparity in their impact on the overall loss calculation. We compare the focal loss with two loss functions derived from the MCC and the F1 score.Focal Loss (Lin et al., 2017) is an often-used loss function for imbalanced deep-learning classification problems. Difficult-to-classify examples often stem from minority classes. These examples are often predicted with low confidence, yielding higher loss values. Hence, the deep learning model is incentivized to optimize for all classes equally. The loss function is given asL focal (p t ) = -(1 -p t ) γ log(p t ) (1)where the exponent γ determines the strength of penalization for samples of class t with predicted probability p t .The F1 score is a valuable metric for assessing classification performance since it summarizes precision and recall into a single number through the harmonic mean. By macro-averaging the F1 score for all classes, we obtain a balanced assessment of the classifier. We leverage this property by deriving a negative differentiable F1 score as a loss function. To this end, we use differentiable true positives (T P ), false positives (F P ), and false negatives (F N ):T P = i∈I y i • ŷi ; F P = i∈I (1 -y i ) • ŷi ; F N = i∈I y i • (1 -ŷi ) (2)where y i is the label and ŷi is the prediction for index i. The precision, recall, and F1 score are defined as:precision = T P T P + F P ; recall = T P T P + F N F 1 = 2 • precision • recall precision + recall (3)We define the corresponding loss function asL F 1 = 1 -F 1 soft ,where F 1 soft is the macro average of the F1 score for each class using differentiable (soft) T P , F P , and F N .Matthew's correlation coefficient (MCC) (Matthews, 1975 ) is a metric that encompasses all four entries of the confusion matrix, namely T P , T N , F P , and F N , into a single value in the binary classification case. It has been argued that the MCC is superior to many other metrics, such as accuracy, F1 score, and the receiver operating characteristic (ROC) area under the curve (AUC) (Chicco and Jurman, 2020; Chicco et al., 2021b,a; Chicco and Jurman, 2023) , because of the normalization term accounting for class imbalance. We define (soft) T P , F P , and F N as above and additionally calculate true negative (T N ):T N = i∈I (1 -y i ) • ŷi (4)From these definitions, M CC sof t is defined as:M CC sof t = (T P • T N ) -(F P • F N ) (T P + F P )(T P + F N )(T N + F P )(T N + F N ) (5)The loss formulation is given as: (Abhishek and Hamarneh, 2021) .L M CC = 1 -M CC sof tThe established cross-entropy loss has desirable theoretical properties. The imbalance-aware losses presented might focus too heavily on the minority class, leading to an unwanted decrease in performance for the majority class. Hence, we also evaluate weighted sums of the F1 and the MCC loss with the cross-entropy loss with equal weights for each loss term.In balanced data scenarios, the samples in a batch are drawn uniformly, i.e., with the same probability, to obtain an equal uniform distribution of each class in a batch. Oversampling One measure to combat class imbalance in deep learning is oversampling the minority class(es) to obtain a more uniform distribution than the original distribution in the dataset. We implement a stratified oversampling technique, i.e., we allocate equal portions of a batch to each class corresponding to equal clinical relevance of each type.Sample weighting Instead of oversampling the minority classes, we can assign higher importance to minority class samples by scaling the influence of a sample according to the prevalence of its class c ∈ C in the loss calculation:EQUATIONWe choose a normalized inverse frequency (Sparck Jones, 1972) scaling for each sample. 
experiment pipeline: 
 An overview of our experimental design is shown in Figure 1 . We conduct our experiments on two diverse and imbalanced datasets. We apply standard preprocessing and augmentation techniques. Two sampling strategies are used to train a ResNet classifier (He et al., 2016) . We compare six loss functions, cross-entropy (CE), focal loss, soft F1, soft MCC, and CE with MCC and F1. For all experiments using the cross-entropy-loss we also compare sample weighting while sampling uniformly. We share our dataset configurations and the code used for our study at https:// github.com/daniel-scholz/address-class-imbalance. For further details of our experiments implementation, refer to Appendix B.To allow the reproduction of our results, our study only uses publicly available datasets.Glioma The first dataset comprises 3D MR images (T1w -/+ contrast, T2w, FLAIR) from large public datasets of adult patients with newly diagnosed gliomas, namely UCSF-PDGM (Calabrese et al., 2022) , EGD (van der Voort et al., 2021), and TCGA (Bakas et al., 2017) . Besides having all four imaging sequences outlined above available, we require biomarker testing for IDH mutation and 1p/19q status in order to classify samples according to the 2021 WHO classification of brain tumors into (a) IDH wildtype glioblastoma, (b) IDH mutant and 1p/19q intact astrocytoma and (c) IDH mutant and 1p/19q codeleted oligodendroglioma (Wen and Packer, 2021) . In total, our dataset contains preoperative MRIs of 1174 patients. The prevalence of glioblastoma (∼80%) in comparison to oligodendroglioma (∼8%) and astrocytoma (∼12%) is striking and consistent throughout all the available datasets, mirroring the real-world distribution. A visualization of the class distributions is shown in the Appendix C, Figure 5 . We hold the TCGA dataset out for testing and use the remaining data for training. For additional robustness analysis, we run each experiment with four different network initializations.The second dataset consists of 1542 individual 2D RGB fundus photographs, of which 786 are healthy controls, 289 photographs show early glaucoma, and 467 are from advanced glaucoma patients (Ahn et al., 2018) . We randomly split the dataset into training ( 3 4 ) and testing ( 1 4 ) data, stratified by class: {no, early, advanced} glaucoma. 
global classification results: 
 Our main results on the test sets are shown in Table 1 (and Appendix A, Table 3, 4, and 5 ).The baseline model (CE loss, uniform sampling) shows reasonable performance, which improves when adding sample weights or oversampling the minority classes for both datasets.The most considerable improvement over the baseline qua balanced accuracy is achieved when using the CE + soft F1 loss combination and oversampling, with a relative improvement of 12.7% on the glioma dataset. We also observe smaller standard deviations for the loss combinations compared to the single class-imbalance-aware losses, F1 and MCC, indicating better training stability for the combination. We explore this further in an ablation study (Appendix A, Table 6 ) for small batch size regimes. The performance on the glaucoma dataset also improves most with the CE + F1 loss combination and oversampling (+10.5%).In addition, we perform a per-class analysis of our methods using the class-wise F1 score, which balances precision and recall (Table 2 ). The baseline training setup yields a classifier biased towards the majority class (glioblastoma / no glaucoma) while performing poorly on the minority classes. The overall improvements in classification performance can be directly traced to improved minority class performance, since majority class performance stays constant across almost all experiments. The CE + F1 loss tremendously improves classification performance on the astrocytoma minority class (+20.3%). The largest improvement when using CE + F1 with oversampling loss is observed on the least prevalent early glaucoma (+51.3%). However, we also observe that using only a class imbalance-aware loss sometimes yields classifiers entirely ignoring one class (e.g., F1 loss in the glaucoma). To investigate the learned representations, we plot the features of the last ResNet layer for the glioma dataset. We use the popular t-distributed stochastic neighbor embedding (tSNE) (Van der Maaten and Hinton, 2008) to project the 256-dimensional feature vectors to 2D for visualization purposes (see Figure 2 ). We observe that representations of the oligodendroglioma are often poorly clustered in this feature space, corresponding to the inferior performance in this class observed in Table 2 . CE + MCC loss with uniform sampling shows a better clustering of the oligodendroglioma features compared to the baseline or the MCC-only loss. 
discussion and conclusion: 
 Class imbalance-aware loss functions relevantly enhance classification performance, primarily by improving recognition of minority classes. Notably, the most substantial enhancements are observed when combining F1 or MCC loss with standard cross-entropy loss. In particular, this combination seems to stabilize training. This points to similarities between classification and segmentation methodologies, exemplified in frameworks like nnUNet (Isensee et al., 2021) , where the combination of Dice (which essentially is the F1 score) and cross-entropy is found to perform best, and underscores the effectiveness of such hybrid approaches. Further, it's worth noting that when employing only class imbalance-aware loss functions, there can be instances where certain classes may be somewhat neglected, a scenario not encountered in the combined approach. In summary, the demonstrated efficacy of class imbalance-aware loss functions, alongside their ease of implementation, computational efficiency, and adaptability across various medical imaging tasks, highlights their potential impact on advancing clinical applications and enhancing the accuracy and reliability of deep learning-based diagnostics in real-world healthcare settings. These properties call for future studies exploring more scenarios in medical image classification with class-imbalance-aware loss functions such as different network architectures or time-series data. Ultimately, we show that integrating loss functions derived from popular metrics such as the F1 score and the MCC with standard cross-entropy loss results in more robust classifiers, with particular benefits for the minority class, thus underscoring its clinical relevance.  
 
 Appendix: For the glaucoma dataset, we also include a range of randomized image intensity and geometry augmentations with a probability of 0.5 each: The set of intensity-changing augmentations consists of randomly adjusting gamma values within the range of 0.5 to 1.5 and contrast adjustment with a gain randomly selected between [5.,10.] . In our geometric augmentations, we randomly flip along the horizontal or vertical axis.Our classifier is a ResNet34 (He et al., 2016) architecture composed of [3; 4; 6; 3] residual blocks, adapted to 3D. We implement the neural network and training using TensorFlow 2.14 (Martín Abadi et al., 2015) , the gamma augmentations with Scikit-image 0.22.0 (van der Walt et al., 2014) , and the Gaussian filtering with Scipy 1.11.3 (Virtanen et al., 2020) . We use Adam optimizer (Kingma and Ba, 2015) , with parameters β 1 = 0.9, β 2 = 0.999, a 


Decision: The paper proposes a novel approach to address class imbalance in medical image classification tasks by integrating class-imbalance-aware loss functions, specifically Matthews correlation coefficient (MCC) and F1 score, with cross-entropy loss. The study demonstrates significant improvements in classification performance, particularly for minority classes, across various medical imaging datasets. While the approach shows promise, reviewers raise concerns about the evaluation methodology and suggest further experiments and analyses to strengthen the findings.

The paper presents a promising approach for addressing class imbalance in medical image classification tasks. To strengthen the findings, reviewers suggest expanding the evaluation to include more target datasets, providing a clearer comparison of different loss function combinations, and conducting additional analyses to enhance the robustness and generalizability of the approach. Addressing these concerns would further solidify the paper's contribution and impact in the field.

Reviews: review: paper_summary: paper_summary: This paper addresses the issue of class imbalance in medical image classification tasks. By integrating class-imbalance-aware loss functions, specifically Matthews correlation coefficient (MCC) and F1 score with cross-entropy loss, the study demonstrates notable improvements in classification performance. Key findings include increases of up to +25% in balanced accuracy and up to +51% in class-wise F1 score for minority classes. The research highlights the effectiveness of combining cross-entropy loss with class-imbalance-aware loss functions for more accurate classifiers, particularly for minority classes.
main_review: main_review: I thank the authors for their comprehensive response to my questions and am satisfied by it. Looking forward to see the additions to the manuscript, I'm convinced it will enhance the paper's impact and relevance to the field.
strength_weakness: strength_weakness: Although the paper is globally well written and easy to understand with a solid evaluation. However, the evaluation seems to be done only on a single test set which might raise some minor concerns:
- First, because it doesn't show how robust the proposed strategy is with regard to the training and test samples.
- Secondly because, if I'm not mistaken, the authors don't disclose the distribution of classes per set (I imagine you performed a stratified split but it would be nice to specify it in the manuscript).
questions: questions: To make the paper more convincing, the authors are advised to add more details about how the experiments are performed (e.g. outlier detection, sets considered for evaluation...)
A good perspective would be also to perform multiple stratified fold evaluation to ascertain the robustness of each strategies to changes in both the training and test sets.
paper_summary: paper_summary: This study introduces differentiable loss functions based on F1 score and MCC targeted for image classification tasks based on unbalanced training data. It addresses an important issue that without proper precautions, model performance of low-sample classes is sub-optimal. The authors introduce the F1 and MCC loss functions that show promising performance when used in combination with cross-entropy loss. The authors test their approach on glioma and glaucoma data.
main_review: main_review: Thank you for the detailed replies to my and other reviewer's comments and for revising the manuscript. I recommend accepting this contribution and believe it is relevant to the MIDL community. The GitHub repository enables others to reproduce and use the proposed method.
strength_weakness: strength_weakness: In my opinion, there is no significant weakness to report; the study is set up nicely and follows good practices in training and evaluating deep neural networks. For a follow-up, testing with different network architectures might be of interest.
questions: questions: None.
paper_summary: paper_summary: A common issue in classification task is that the data can be imbalanced. The authors addresses this data imbalance and explores different combination of loss functions to improve accuracy in medical image classification task. Their findings show that compared to conventional loss functions, utilizing other combination of loss function can improve classification performance significantly.
main_review: main_review: The authors have satisfactorily addressed all of my questions and have made necessary adjustments in their experiments. In line with recommendations, they conducted their experiments multiple times and provided the standard deviation data as well.
strength_weakness: strength_weakness: 1. The authors could utilize more target datasets and show the impact of using different combination of loss functions;
2. The authors could display various percentages of ratios for each class or category, along with the impact of their approach;
3. For evaluation, they can use AUC along side accuracy and F1 score.
4. It is not clear from the manuscript, how many times each experiment were conducted. The authors could show statistical analysis to make their claim stronger.
5. According to their results, the best performance does not come from a single loss combination. It is unclear which one is the best to use. Additionally, it is uncertain if this scenario holds true for different target tasks.
questions: questions: Please see the points mentioned in the section "Weaknesses".

Comments: comment: Thank you for your detailed response addressing the comments raised. I appreciate the thoroughness with which you've handled each point and the efforts made to improve the robustness and reproducibility of your study.

**Evaluation on multiple datasets**
I understand now that while each experiment is evaluated on a single hold-out test dataset, you have tested your methods on two independent datasets, which adds to the generalizability of your findings.

**Robustness analysis** 
It's great to hear that you have performed the experiments multiple times and added the new results to the revised version of the paper. The inclusion of balanced accuracy, MCC, ROC AUC, and F1 score along with standard deviations provides a clearer understanding of the stability and performance of the proposed methods across different training runs.

**Disclosure of class distribution** 
I appreciate the clarification regarding the disclosure of class distribution for both datasets. Adding class distribution diagrams to Appendix C further enhances the transparency of your methodology.

**Additional details on experiment performance** 
The addition of exact splits used for training in the GitHub repository is a commendable step towards improving reproducibility and transparency in your research.

**Outlier Detection**
I understand the difficulty of detecting outliers in the image space and the lack of literature in the field. In response to your comment on the challenge of detecting outliers in oversampling experiments for image data, I recommend exploring the method presented in the paper "Unsupervised Image Outlier Detection with RANSAC" by Chen-Han Tsai and Yu-Shao Peng. It may provide insight on dealing with outliers in the image space, even during oversampling.

**Multiple stratified fold evaluations**
I'm glad to see that you have taken the suggestion to perform multiple stratified fold evaluations into account. The new results with standard deviations across runs provide valuable insights into the stability and robustness of the proposed strategies, particularly for glioma classification.


Overall, I'm satisfied with the responses provided and believe that the revisions made significantly enhance the quality and reliability of your study. Thank you for your diligence and commitment to addressing the concerns raised during the review process.
Thank you for the valuable feedback on our manuscript and for appreciating our “**well-written and easy-to-follow” paper.** We are glad you found our exploration of the impact of various classification loss functions in imbalanced data scenarios to contain a “**comprehensive analysis of these loss functions”** using two public datasets for reproducibility. 

In the following, we would like to address the comments raised:

> more target datasets

We agree that more target datasets can help show the reliability of a method. Yet, we would politely point out that we tested on two completely different datasets of 3D brain MRIs and 2D retina fundus photographs with similar results, showcasing our method's capabilities on different modalities.

> display various percentages of ratios for each class or category, along with the impact of their approach

We agree it would be interesting to investigate different imbalance ratios systematically with respect to the balancing strategies in our paper. To some extent, we already include multiple imbalance ratios in our experiments since our datasets show different degrees of imbalance, but a similar effect of combining class-imbalance aware loss functions with cross-entropy loss: In glioma, the class distribution is {0.08, 0.80, 0.12}, while for the glaucoma dataset, the distribution of the three classes is {0.51, 0.19, 0.30}. We are adding class distribution histograms (Appendix C) to further clarify this point. However, analyzing more different ratios by subsampling the datasets adds another dimension to our experiment grid, making this analysis infeasible to conduct during the short rebuttal period. Nonetheless, we thank the reviewer for raising this important point for future research.

> they can use AUC along side accuracy and F1 score.

We agree that reporting further metrics beyond balanced accuracy and F1 score is interesting, so we added additional tables in the revised paper, reporting AUC and MCC. As expected, we find that the differences in AUC are not as pronounced compared to balanced accuracy, F1, and MCC since the AUC is unduly influenced by the majority class regardless of which averaging (micro, macro) is used through the false positive rate given as $\frac{FP}{FP+TN}$. More individual analysis for each class is given through the class-wise F1 score, which is only dependent on the samples of a class and samples predicted as such. 

> It is not clear from the manuscript, how many times each experiment were conducted

In the revised paper, we conducted the glioma experiment three independent times to assess the stability of our results. These new results confirm our findings that combinations of cross-entropy loss with F1 and MCC loss improve performance in imbalanced data settings. The best method for glioma classification is the CE + F1 combination with oversampling ($0.62_{\pm 0.03}$ balanced accuracy) and the CE + F1 combination with sample weighting ($0.62_{\pm 0.01}$ balanced accuracy). In addition, the small standard deviations across runs show the loss combinations to be stable across runs.

> The authors could show statistical analysis to make their claim stronger.

We agree with the idea of assessing the stability of our proposed loss combination, and, therefore, for the revision, multiple re-runs of our glioma experiment were performed, as discussed in detail above.

> It is unclear which one [loss combination] is the best to use.

While our experiments might not give a conclusive answer to which single combination of sampling and loss function is the best, our experiments clearly show that using the imbalance-aware loss functions derived from F1 or MCC in combination with the cross-entropy loss does, in fact, improve performance reliably across seeds and datasets compared to sample weighting or oversampling, which is usually used to combat class imbalance. Further, we performed an additional ablation experiment for the revised paper to better understand the effect of combining a class-imbalance aware loss with cross-entropy loss. Specifically, we also compared these conditions in low batch size regimes (which are typical for 3D medical imaging data). We found that the combination clearly stabilizes training and leads to a robust performance increase over single losses of +60% ROC AUC at low standard deviations of 0.03. These results clearly add to the picture of combining class-imbalance aware losses with standard cross-entropy loss to improve medical image classification in imbalanced data scenarios. 

> it is uncertain if this scenario holds true for different target tasks.

We are unsure which other target tasks you are referring to here since the loss formulations only apply to the task of classification, where we show our method to work for two entirely different scenarios. We are happy to discuss this further with you.
We sincerely thank the reviewer for the very positive evaluation of our work. In particular, we appreciate you highlighting the “**importance of the issue”** we are addressing and our “**well-motivated” approach** to implementing, comparing, and discussing class-imbalance-aware loss functions. 

> For a follow-up, testing with different network architectures might be of interest.

We agree with your opinion that it would be interesting to see how the different losses influence other network architectures, such as DenseNet or Vision Transformer, for classification. We have added this as a relevant future study to our conclusion.
We thank the reviewer for their insightful comments. We appreciate recognizing our paper's contributions, particularly the “**novel approach for handling class imbalance through** […] **integrating MCC and F1 score with cross-entropy loss”.** This “effectively addresses class imbalance” and “improves classification accuracy and minority class recognition.” We are also glad about the appreciation of our “**evaluation of different loss functions across various medical imaging datasets”** to strengthen the reliability of our findings.

In the following, we would like to address some comments raised:

> [...] the evaluation seems to be done only on a single test set.

Yes and no. While each of our experiments is only evaluated on a single hold-out test dataset, we test our methods on two independent datasets. Please also see our following comment below.

> [...] it doesn't show how robust the proposed strategy is with regard to the training and test samples.

Yes, we agree that a robustness analysis over different training runs is an important add-on to our paper. We reran each experiment three times and added the new results in terms of balanced accuracy (main paper), MCC, ROC AUC, and F1 score (appendix) as averages plus standard deviation to the revised version of the paper. The new results confirm the findings from our submitted paper, showcasing improved classification performance for combinations of CE + {F1, MCC} loss and increased stability when using the combination with cross-entropy loss compared to the imbalance-aware losses (F1, MCC loss) alone (see Appendix A, Table 6). 

> the authors don't disclose the distribution of classes per set

We disclose both datasets' class distribution in Section 4.2 of the paper. We add class distribution diagrams for both datasets to Appendix C for additional clarification. As stated in Section 4.2, we use TCGA for testing for the glioma and the rest for training. We perform a stratified split for glaucoma, maintaining the original class distribution.  

> [...] about the experiments involving sample weighting, has there been a treatment of potential outlier samples in the study.

No, there has not been special treatment of potential outliers in our oversampling experiments, as it is non-trivial to define nor detect outliers in image space. We also could not find any literature regarding explicitly handling outliers during oversampling. We would be glad if you could point us to the body of literature you refer to.

> Add more details about how the experiments are performed (e.g., outlier detection and sets considered for evaluation).

We add the exact splits used for our training to the GitHub repository linked in the paper for better reproducibility.

> A good perspective would also be to perform multiple stratified fold evaluations to ascertain the robustness of each strategy to changes in both the training and test sets.

As outlined above, we fully agree with this suggestion and performed three additional runs for each experiment on the glioma dataset. We added the new results with standard deviations across trials to the revised version of the paper. These new results confirm our findings that combinations of cross-entropy loss with F1 and MCC loss improve performance in imbalanced data settings. The best method for glioma classification is the CE + F1 combination with oversampling ($0.62_{\pm 0.03}$ balanced accuracy) and the CE + F1 combination with sample weighting ($0.62_{\pm 0.01}$ balanced accuracy). In addition, the small standard deviations across runs show the loss combinations to be stable across runs.