Class-Weighted Evaluation Metrics for Imbalanced Data Classification

1 INTRODUCTION . For a broad range of machine learning ( ML ) tasks , predictive modeling in the presence of imbalanced datasets – those with severe distribution skews – has been a long-standing problem ( He & Garcia , 2009 ; Sun et al. , 2009 ; He & Ma , 2013 ; Branco et al. , 2016 ; Hilario et al. , 2018 ; Johnson & Khoshgoftaar , 2019 ) . Imbalanced training datasets lead to models with prediction bias towards majority classes , which in turn results in misclassification of the underrepresented ones . Yet , those minority classes often are the ones that correspond to the most important events of interest ( e.g. , errors in system logs ( Zhu et al. , 2019 ) , infected patients in medical diagnosis ( Cohen et al. , 2006 ) , fraud in financial transactions ( Makki et al. , 2019 ) ) . While there is often an inverse correlation between the class cardinalities and their importance ( i.e. , rare classes are more important than others ) , the core problem here is the mismatch between the way these two distributions are skewed : the ith most common class is not necessarily the ith most important class ( see Figure 1a for an illustration ) . In fact , rarity is one of many potential criteria that can determine the importance of a class , which is usually positively correlated with the costs or risks involved in its misprediction . Ignoring these criteria when dealing with imbalanced data classification may have detrimental consequences . Consider automatic classification of messages in system event logs as an example ( Zhu et al. , 2019 ) . An event log is a temporal sequence of messages that have transpired for a given software system ( e.g. , operating systems , cyber-physical systems ) over a certain time period . Event logs are particularly useful after a system has been deployed , as they can provide the DevOps teams with insights about errors outside of the testing environment , thereby enabling them to debug and improve the system quality . There is typically an inverse correlation between the stability/maturity of a system and the frequency of the errors it produces in its event logs . Furthermore , the message types that appear least frequently in an event log are usually the ones with the greatest importance . A concrete example of this was a rare anomaly in Uber ’ s self-driving car that led to the death of a pedestrian , since the system flagged it as a false positive in its logs ( Efrati , 2018 ) . If this event had not been misclassified and dismissed by the system , the pedestrian death in Arizona may have been avoided . A plethora of approaches have been proposed for building balanced classifiers ( Sun et al. , 2009 ; Branco et al. , 2016 ) . A fundamental issue that still remains an open challenge is the lack of a generally-accepted methodology for measuring classification performance . The traditional metrics , which are designed to evaluate average case performance ( e.g. , Accuracy ) are not capable of correctly assessing the results in presence of arbitrary skew mismatches between class cardinalities and importances . On the other hand , metrics specifically proposed for imbalanced learning are either domain-specific , do not easily generalize beyond two classes , or can not support varying class importance ( e.g. , Balanced Accuracy ) ( Japkowicz , 2013 ) . Let us illustrate the problem with the simple example in Figure 1b . The test dataset consists of 100 data items from 3 classes ( A , B , C ) . The greatest majority of the items belong to class C ( 70 ) , but class B ( 20 ) has the greatest importance ( 0.7 ) . In other words , Cardinality and Importance are both non-uniform and in favor of different classes ( i.e. , representing the top-right quadrant of Figure 1a ) . The confusion matrix on the right shows the results from a classifier run against this test dataset . Unsurprisingly , the classifier performed the best for the majority class C ( 60/70 correct predictions ) . When evaluated using the traditional Accuracy metric , neither Class Cardinality nor Class Importance is taken into account . If Balanced Accuracy is used instead , we observe the degrading impact of the Class Cardinality skew ( 0.38 < 0.65 ) , but Class Importance is still not accounted for . This example demonstrates the need for a new evaluation approach that is sensitive to both Cardinality and Importance skew , as well as any arbitrary correlations between them . This is especially critical for ensuring a fair comparative assessment across multiple classifiers or problem instances . Our goal in this paper is to design an evaluation framework for imbalanced data classification , which can be reliably used to measure , compare , train , and tune classifier performance in a way that is sensitive to non-uniform class importance . We identify two key design principles for such a framework : • Simplicity : It should be intuitive and easy to use and interpret . • Generality : It should be general-purpose , i.e. , ( i ) extensible to an arbitrary number of classes and ( ii ) customizable to any application domain . To meet the first goal , we focus on scalar metrics such as Accuracy ( as opposed to graphical metrics such as ROC curves ) , as they are simpler , more commonly used , and scale well with increasing numbers of classes and models . To meet the second goal , we target the more general n-ary classification problems ( as opposed to binary ) , as well as providing the capability to flexibly adjust class weights to capture non-uniform importance criteria that may vary across application domains . Note that we primarily focus on Accuracy as our base scalar metric in this paper , as it is seen as the de facto metric for classification problems ( Sci ) . However , our framework is general enough to be extended to other scalar metrics , such as Precision and Recall . Similarly , while we deeply examine three use cases ( log parsing , sentiment analysis , URL classification ) , our framework in principle is generally applicable to any domain with imbalanced class and importance distributions . We first provide a brief overview of related work in Section 2 . Section 3 presents our new , classweighted evaluation framework . In Section 4 , we show the practical utility of our framework by applying it over : ( i ) three log parsing systems ( Drain ( He et al. , 2017 ) , MoLFI ( Messaoudi et al. , 2018 ) , Spell ( Du & Li , 2016 ; 2018 ) ) using four real-world benchmarks ( Zhu et al. , 2019 ) ; ( ii ) a variety of deep learning models developed for sentiment analysis on a customer reviews dataset from Amazon ( Ni et al. , 2019 ) ; and ( iii ) an industrial use case for URL classification with real classifiers and datasets from four cyber-security companies . Finally , we conclude in Section 5 . 2 RELATED WORK . Imbalanced Data Classification . Imbalanced data is prevalent in almost every domain ( Cohen et al. , 2006 ; Batuwita & Palade , 2012 ; Makki et al. , 2019 ) . The growing adoption of ML models in diverse application domains has led to a surge in imbalanced data classification research ( He & Garcia , 2009 ; Sun et al. , 2009 ; He & Ma , 2013 ; Branco et al. , 2016 ; Hilario et al. , 2018 ; Johnson & Khoshgoftaar , 2019 ) . While the techniques widely vary , they fall under four basic categories : pre-processing training data to establish balance via sampling techniques ( Estabrooks et al. , 2004 ; Blaszczynski & Stefanowski , 2015 ) , building custom learning techniques for imbalanced training data ( Joshi et al. , 2001 ; Castro & de Pádua Braga , 2013 ) , post-processing predictions from an imbalanced model ( Maloof , 2003 ) , and their hybrids ( Estabrooks & Japkowicz , 2001 ) . In this paper , we do not propose a new imbalanced learning technique , but a general-purpose performance evaluation framework that could be used in the training and/or testing of models for any technique . Section 4 demonstrates the practical utility of our framework for a variety of real ML use cases . Evaluation Metrics . Traditional metrics for evaluating prediction performance such as Accuracy , Sensitivity/Specificity ( and their combination G-mean ) , Precision/Recall ( and their combination FScore ) were not designed with imbalanced data issues in mind ( Japkowicz , 2013 ) . In fact , most of these were originally intended for binary classification problems . To extend them to more than 2 classes , macro-averaging ( i.e. , arithmetic mean over individual class measurements ) is used . Macroaveraging treats classes equally ( Branco et al. , 2016 ) . Balanced Accuracy is a popular averagingbased approach . There are also probabilistic evaluation approaches that extend Balanced Accuracy with Bayesian inference techniques for both binary and multi-class problems ( Brodersen et al. , 2010 ; Carrillo et al. , 2014 ) . Close to our work , Cohen et al . ( 2006 ) introduced the notion of class weights , yet in the specific context of Sensitivity/Specificity for binary classification in the medical domain . Similarly , Batuwita & Palade ( 2012 ) proposed extensions to G-mean for the bio-informatics domain . In addition to these scalar ( a.k.a. , threshold ) metrics , graphical ( a.k.a. , ranking ) evaluation methods such as Receiver Operating Characteristic ( ROC ) curves or Precision-Recall ( PR ) curves ( and the Area Under the Curve ( AUC ) for such curves ) as well as their extensions to imbalanced data / multiclass problems were also investigated ( Weng & Poon , 2008 ; Japkowicz , 2013 ) . While these methods provide more detailed insights into the operational space of classifiers as a whole , they do not easily scale with use in problems with a large number of classes ( Branco et al. , 2016 ) . 3 CLASS-WEIGHTED EVALUATION FRAMEWORK . In this section , we present our new evaluation framework for multi-class learning problems in presence of arbitrary skews among class distributions and/or importances . Our framework builds on and extends commonly used scalar / threshold metrics such as Accuracy . These metrics were originally designed for binary classification problems , where there is typically more emphasis on one class ( the positive class , e.g. , anomalies ) . To adopt them to multi-class problems where there is no such single-class emphasis , each class ’ metric can be computed separately and then an overall aggregation ( i.e. , arithmetic mean ) can be performed . For example , Accuracy has been extended to BalancedAccuracy by following this approach . In our framework , we follow a similar aggregation strategy , however , we do it in a more generalized way that allows custom class weights to capture class importance . Furthermore , these class weights can be based on any importance criteria such as rarity , cost , risk , expected benefits , and possibly a hybrid of multiple such criteria . Therefore , it is critical to provide a flexible formulation that allows users or domain experts to adjust the weights as needed by their problem instance . In what follows , we present our new class-weighted evaluation framework in a top-down fashion . Using the basic notation summarized in Table 1 , we first formulate the general framework , and then we describe how this framework can be customized to different importance criteria scenarios by specializing the weights in a principled manner . For ease of exposition , we first focus on Accuracy as the underlying performance metric , and then we discuss how our approach can be adopted to other similar metrics . Finally , we end this section with a brief discussion of how our framework can be used in model training .

Summary: The authors advocate for class-stratified weighted macro-averaging as an appropriate scalar-valued evaluation for classification under class-imbalance. In the absence of domain expert specified weights, they also propose a weighting function that emphasizes rare classes (referred to as rarity), and a multi-importance criteria based on a normalized product of weightings. Furthermore, they point out that these weightings can be used for training via existing methods in commonly used tools (e.g., class-based instance weighting, class-based loss scaling). Finally, they show performance under different weighting for three log message classification tasks, a sentiment classification task, and a URL classification task (e.g., malware, NSFW, phishing),  — demonstrating that different weighings lead to different orderings of evaluation results and that these weights can be effectively used in training.