Class-Weighted Evaluation Metrics for Imbalanced Data Classification

Abstract: Class distribution skews in imbalanced datasets may lead to models with prediction bias towards majority classes, making fair assessment of classifiers a challenging task. Metrics such as Balanced Accuracy are commonly used to evaluate a classifier’s prediction performance under such scenarios. However, these metrics fall short when classes vary in importance. In this paper, we propose a simple and general-purpose evaluation framework for imbalanced data classification that is sensitive to arbitrary skews in class cardinalities and importances. Experiments with several state-of-the-art classifiers tested on real-world datasets from three different domains show the effectiveness of our framework – not only in evaluating and ranking classifiers, but also training them.

Reviews and author responses:

 While it is true that a linear combination of class-conditioned scores for multi-class evaluation is simple and general, it just isn't a research contribution. In my estimation, the reason you haven't found a reference for this idea is that it is obvious and widely used -- I literally see this basic idea in business and technical reports all the time (e.g., weighted averages of a subset of classes). As stated in my review, the research problems are: (1) how to set the weights and (2) how to determine statistical properties to ensure valid hypothesis testing. Without a principled method, a uniform weighting, a class-balanced weighting, only considering the rarest class, or even a random weighting are all equally valid without a method for eliciting an application-targeted preference. This would be analogous to claiming that a linear combination of objectives is a simple and general formulation for multi-criteria optimization without: (1) establishing a method for setting these weights and (2) demonstrating that the optimization can be computed in a reasonable time (and soundly, identifiably, etc.). 

Evaluations are only as useful as their ability to make real-world decisions and be adopted by a scientific community to make empirical comparisons. In this case, you are 'only' presenting a parameterization and not specifying how to set the parameters. The empirical evaluation section has some interesting points, but the methods contribution is inconsequential from a research perspective.

The authors advocate for class-stratified weighted macro-averaging as an appropriate scalar-valued evaluation for classification under class-imbalance. In the absence of domain expert specified weights, they also propose a weighting function that emphasizes rare classes (referred to as rarity), and a multi-importance criteria based on a normalized product of weightings. Furthermore, they point out that these weightings can be used for training via existing methods in commonly used tools (e.g., class-based instance weighting, class-based loss scaling). Finally, they show performance under different weighting for three log message classification tasks, a sentiment classification task, and a URL classification task (e.g., malware, NSFW, phishing),  — demonstrating that different weighings lead to different orderings of evaluation results and that these weights can be effectively used in training. First of all, I agree that class-stratified weighting can be effective for evaluation and training. However, from a research perspective, this paper has very limited novelty. With respect to class-stratified evaluation (or combining them into a single score), I have seen hundreds of reports that already do this. From a training perspective, this is taught in just about every undergraduate machine learning course (both in training in evaluation). From a training perspective, as the authors point out, this is available in many toolkits (e.g., scikit-learn, SVMlight, TensorFlow, pyTorch), so people are obviously doing this.

Thus, the real question is how to set these weights, which generally falls under the purview of decision theory. There are multiple textbook chapters (and entire textbooks) that address this issues. Given the interest in class imbalance, I would recommend something in the medical decision making community where preference elicitation is widely-studied (e.g., standard gamble). A paper that particularly influenced my thinking is in this space is [A. J. Vickers and E. B. Elkin. Decision curve analysis: A novel method for evaluating prediction models. Medical Decision Making, 26:565–574, 2006.], [but there is ample literature in this space.

Secondly, averaging together a set of point estimates to derive a scalar doesn’t necessarily have well-behaved statisical properties as it depends on what is being averaged together. Specifically, there are cases (e.g., sensitivity and specificity) that are frequently modeled as bivariate normal when aggregating measurements. Without such considerations, it isn’t clear how statistical significance would be evaluated. A paper that influenced my thinking here wrt rigor and scalar values for class imbalance settings is [Jurman, Riccadonna, and Furlanello, A Comparison of MCC and CEN Error Measures in Multi-class Prediction, PLoS One, 7(8): 2012]. Basically, without knowing the distribution properties of the average, it isn’t obvious how to perform hypothesis testing.

Finally, I am not entirely convinced that the motivation regarding a scalar evaluation for multi-class under severe class skew is even that desirable. While subjective, I believe there are potentially good visualization that can be used to elicit class-specific utility — which, as the authors state, are very sensitive for some rare events (e.g., medical diagnosis, (hopefully fictional) doomsday devices). 

This all being said (that this isn’t a suitable paper for ICLR), the writing itself is a bit informal, but easy to understand and some of the observations on the datasets are interesting. However, there isn’t sufficient novelty, potential for impact, or mathematical rigor to accept this paper in a top-tier machine learning conference. The authors advocate for class-stratified weighted macro-averaging as an appropriate scalar-valued evaluation for classification under class-imbalance — justified by a reasonable number of case studies. However, they don’t describe a procedure for setting the weights in a decision theoretic way nor describe the statistical properties of the resulting point estimate for hypothesis testing. Thus, while a reasonable heuristic approach, it lacks rigor for use in critical systems (where high-loss rare events are important). Additionally, there is very limited novelty as this is performed both for training systems and evaluating them in general ML research, application-focused ML research, and industry practice. This paper has some interesting observations, but is not ready for ICLR acceptance. 

 Thanks for your answers - I see the positive aspects of this work, but I keep my recommendation.

 We really appreciate the valuable comments from the reviewer! Please find our response below.

We agree that the importance of each class can be difficult to define. To address this, besides user-defined weights (the most flexible and the only case where the weights should be set by the user or the application), we also propose the option and the calculation of rarity weights, as an objective method to set class weights purely based on the dataset class distribution. This is based on the observation that in many applications, the rarest classes are the most important ones, for example, anomaly detection. For cases when the majority classes are more important, the relatively higher volume naturally serves as higher weights. Thus, we do not explicitly propose an objective weight assignment for such cases.

We propose three different applications in the paper, each with multiple real-world datasets: log parsing, sentiment analysis, and URL classification. Overall, there are actually more than three datasets used in our experiments. For example, for log parsing alone, we have explored four datasets: macOS, BGL, Android, and HDFS. Moreover, the three applications and many datasets in different domains suggest that our framework can be generally applied to many applications and datasets in similar ways.

Based on the rarity importance criteria, if given a large weight on rare classes, the accuracy of these rare classes can be improved, while the accuracy of the majority classes may or may not be affected, which depends on the dataset and the capability of the classifier being used. However, we see this more of a trade-off for users to decide, instead of a defect. Rarity importance criteria can be applied when rare classes are more important, e.g., for applications like anomaly detection, where boosting the accuracy in rare classes is more preferable although this may accompany a slight decrease in majority classes.

 Thanks so much for your kind review! The “too many things” being referred to are simply proposed as different options to the user, which are not necessarily used together. There are two main reasons we propose these many options to apply weights: 1) to show that our framework is general enough and can accept various weights (generality/extensibility); 2) to provide users with different options depending on their needs (flexibility).

 Many thanks to the review and the related works!

In this paper, we address the evaluation of multi-class classifiers when class imbalance is present and class importance varies. We propose a few ways to assign class weights based on user needs to calculate a weighted accuracy measure. We also propose to apply these weights to the loss function used in deep learning in order to improve the accuracy of important classes. The loss function considered in this paper is simply the most commonly used one for multi-class classification with deep learning - cross-entropy loss. This is often sufficient as cross-entropy loss is commonly adopted by default for such tasks.

The first paper mentioned above, “Utility-based Performance Measures”, proposes a similar idea to evaluate model performance on regression tasks, where the model outputs a continuous real value instead of a class label. The proposed approach is compared with MSE and MAD, which are commonly used measures for regression tasks. 

For the second referred paper, fuzzy loss in multi-class classification tasks simply changes the penalty for mis-classification from 1 to the degree of certainty for the class label to predict. Owing to the simplicity and generality metrits of our proposed framework, it can be easily applied to this case. As long as the loss function is an aggregation of all sample losses, our proposed class weights can be easily applied by multiplying the loss aggregation for each class with the corresponding weight, regardless of how each loss value is computed. 


 Correctness: Could you please be more specific about which claims in the paper you found to be incorrect or not well-supported so that we could try to address them?

 We thank all the reviewers for their constructive feedback. Please see below for our responses to individual points raised.

 Thanks a lot for your valuable feedback, and for the additional references!

The evaluation framework we propose follows two key design principles: simplicity and generality. Many other related works to compare models either use a more complicated evaluation process than a simple metric value, or tackle certain domain-specific cases.

In the first reference by Vickers and Elkin, a plot is provided to compare different models with varying thresholds, to deal with medical diagnosis and treatment. This approach can be simplified to use user-defined weights in our proposed framework. 

In the second reference by Jurman and Furlanello, it is true that MCC and CEN take the distribution properties into consideration and consider more on the rarity classes compared with Accuracy measure. However, the Accuracy measure has its advantage in that it’s easier to compute and understand, despite the fact that it unfairly favors the majority classes, which is exactly the defect that our proposed WBA framework aims to address. Our proposed WBA metric can well address the class distribution properties with the proposed rarity weights. Moreover, we also provide an option for user-defined weights, since users may rank class importance differently. Finally, our WBA score is improved from the well-known and commonly-used Accuracy measure, and hence inherits its advantages of easy to compute and understand, which has the potential to be widely applicable.

Lastly, we believe that the two approaches for multi-class evaluation - scalar and visual - are complementary to each other. For use cases with a large number of classes, visualization may be difficult to use, so providing a scalar value could be more practical for evaluating models as well as comparing or ranking multiple of them. Furthermore, a scalar evaluation result is often easier to produce compared with visualization.

 Dear Authors and Reviewers,

Let me first thank you for supporting ICLR 2022: Authors for submitting their contributions, and Reviewers for going through them and sending their comments and remarks!

As the discussion phase has just begun, let me ask Authors to answer all questions appearing in reviews and to defend your paper, and reviewers to check all other reviews to see whether you coincide and to be ready to respond to authors rebuttals.

We are also looking forward for public comments. I hope for a vivid discussion for this paper.

Best regards, AC for Paper 511

The article focuses on a significant problem related to the quality assessment of unbalanced data classifiers, or rather a problem where one deals with a problem where we may observe different misclassification cost for each class. I fully agree that the known and commonly used metrics do not meet expectations, hence their diversity, the analysis presented in the works cited by the author.  A simple and intuitive model has been proposed that modifies the global quality metric by adding class significance weights. However, a problem that, in my opinion, is still unresolved is how to determine the importance of the classes. As a rule, as the authors point out, it should be a user's decision, but as is well known, we have a problem with obtaining this information. Note that the classic probabilistic model of the classifier is based on the minimization of loss function, which leads to the minimization of the overall risk (expected value of the loss function), i.e., one can say that the problem is solved and the proposal contained in the paper does not bring anything new. Unfortunately, we usually do not know the form of the loss function because this is information that an expert should provide. Some methods try to gain the knowledge of the misclassification cost, such as the work on Utility-based Performance Measures (https://www.dcc.fc.up.pt/~ltorgo/Papers/RT08_WEMML.pdf) or methods that try to obtain information about the cost of error using linguistic variables (such as fuzzy loss function - https://link.springer.com/content/pdf/10.1007%2F978-3-540-68168-7_53.pdf). Unfortunately, neither of these approaches fully addresses this problem.
I find interesting comments of the authors contained in sec. 3.2, where it is proposed to calculate the weights according to factors other than those obtained directly from users.

 Recognizing that the problem is interesting, however, taking into account the above, I found that the contribution of this work is insufficient for this conference.



The paper presented a simple yet general-purpose class-sensitive evaluation framework for imbalanced data classification. Their framework is designed to improve the grading of multi-class classifiers in domains where class importance is not evenly distributed. They provided a modular and extensible formulation that can be easily customized to different important criteria and metrics. Experiments with three real-world use cases show the value of a metric based on our framework, Weighted Balanced Accuracy (WBA), over existing metrics – in not only evaluating the classifiers’ test results more sensitively to important criteria but also training them so. The paper addressed a known problem by introducing weights to the rare class and explaining a new theoretical finding to better grade the models applied.

The 2 main goals of simplicity and good generality of models were addressed. The study proposed a new grading metric to an old problem

The research is reproducible and presents simple novel findings

The Weak points are trying to incorporate too many things in the model like user-defined, rarity, multiple, and partially-defined importance.


 Overall, I vote for acceptance as the class imbalance problem is chaotic. It gives classification suboptimal results. Especially when allied with class disjunct


In this paper, the authors have proposed a novel evaluation framework for imbalanced data classification. Specifically, the proposed evaluation metric is designed to improve the grading of multi-class classifiers in domains where class importance is not evenly distributed. Generally speaking, the problem authors paid attention to is really existed and is important in imbalanced classification problem. Moreover, the writing in this paper is very well and is very easy to follow.  In this paper, the authors have proposed a novel evaluation framework for imbalanced data classification. Specifically, the proposed evaluation metric is designed to improve the grading of multi-class classifiers in domains where class importance is not evenly distributed. Generally speaking, the problem authors paid attention to is really existed and is important in imbalanced classification problem. Moreover, the writing in this paper is very well and is very easy to follow. However, the paper still exists the following problems:

The major problem of this paper is that the importance of each class is difficult to define. In this paper, the authors propose four methods in Section 3.2, i.e., importance criteria=user-defined, importance criteria=rarity, multiple importance criteria and partially-defined importance criteria. But in my opinion, none of them is very accurate and objective to define the importance of each class because in real-world applications, even if domain experts are difficult to distinguish the subtle importance differences between some classes. Anyway, it is an interesting and valuable problem for researches to discuss the importance differences between different classes. I think if the authors can provide some other importance criteria that are more objective, this paper will be more valuable.

Besides, since this is a novel evaluation metrics, I think the authors should use a collection of datasets to fully verify the effectiveness of the proposed framework, instead of just three datasets in this paper. 

Finally, if the number of some classes are too rare, according to the rarity importance criteria, these classes will be given a very large weight, if so, whether the accuracy of majority classes will be largely decreased?
 In this paper, the authors have proposed a novel evaluation framework for imbalanced data classification. Specifically, the proposed evaluation metric is designed to improve the grading of multi-class classifiers in domains where class importance is not evenly distributed. Generally speaking, the problem authors paid attention to is really existed and is important in imbalanced classification problem. Moreover, the writing in this paper is very well and is very easy to follow. However, the paper still exists the following problems:

The major problem of this paper is that the importance of each class is difficult to define. In this paper, the authors propose four methods in Section 3.2, i.e., importance criteria=user-defined, importance criteria=rarity, multiple importance criteria and partially-defined importance criteria. But in my opinion, none of them is very accurate and objective to define the importance of each class because in real-world applications, even if domain experts are difficult to distinguish the subtle importance differences between some classes. Anyway, it is an interesting and valuable problem for researches to discuss the importance differences between different classes. I think if the authors can provide some other importance criteria that are more objective, this paper will be more valuable.

Besides, since this is a novel evaluation metrics, I think the authors should use a collection of datasets to fully verify the effectiveness of the proposed framework, instead of just three datasets in this paper. 

Finally, if the number of some classes are too rare, according to the rarity importance criteria, these classes will be given a very large weight, if so, whether the accuracy of majority classes will be largely decreased?

Meta-review: Although two reviewers have given score 6, the other reviews have clearly indicated that the paper is not good enough for ICLR. The paper does not give any new insight into the considered problem, many existing papers are ignored, and the only interesting part about evaluation of label importance is rather very shallow borrowing ideas from other fields without any deep discussion or analysis.

Decision: Reject