Class-Weighted Evaluation Metrics for Imbalanced Data Classification

1 INTRODUCTION . For a broad range of machine learning ( ML ) tasks , predictive modeling in the presence of imbalanced datasets – those with severe distribution skews – has been a long-standing problem ( He & Garcia , 2009 ; Sun et al. , 2009 ; He & Ma , 2013 ; Branco et al. , 2016 ; Hilario et al. , 2018 ; Johnson & Khoshgoftaar , 2019 ) . Imbalanced training datasets lead to models with prediction bias towards majority classes , which in turn results in misclassification of the underrepresented ones . Yet , those minority classes often are the ones that correspond to the most important events of interest ( e.g. , errors in system logs ( Zhu et al. , 2019 ) , infected patients in medical diagnosis ( Cohen et al. , 2006 ) , fraud in financial transactions ( Makki et al. , 2019 ) ) . While there is often an inverse correlation between the class cardinalities and their importance ( i.e. , rare classes are more important than others ) , the core problem here is the mismatch between the way these two distributions are skewed : the ith most common class is not necessarily the ith most important class ( see Figure 1a for an illustration ) . In fact , rarity is one of many potential criteria that can determine the importance of a class , which is usually positively correlated with the costs or risks involved in its misprediction . Ignoring these criteria when dealing with imbalanced data classification may have detrimental consequences . Consider automatic classification of messages in system event logs as an example ( Zhu et al. , 2019 ) . An event log is a temporal sequence of messages that have transpired for a given software system ( e.g. , operating systems , cyber-physical systems ) over a certain time period . Event logs are particularly useful after a system has been deployed , as they can provide the DevOps teams with insights about errors outside of the testing environment , thereby enabling them to debug and improve the system quality . There is typically an inverse correlation between the stability/maturity of a system and the frequency of the errors it produces in its event logs . Furthermore , the message types that appear least frequently in an event log are usually the ones with the greatest importance . A concrete example of this was a rare anomaly in Uber ’ s self-driving car that led to the death of a pedestrian , since the system flagged it as a false positive in its logs ( Efrati , 2018 ) . If this event had not been misclassified and dismissed by the system , the pedestrian death in Arizona may have been avoided . A plethora of approaches have been proposed for building balanced classifiers ( Sun et al. , 2009 ; Branco et al. , 2016 ) . A fundamental issue that still remains an open challenge is the lack of a generally-accepted methodology for measuring classification performance . The traditional metrics , which are designed to evaluate average case performance ( e.g. , Accuracy ) are not capable of correctly assessing the results in presence of arbitrary skew mismatches between class cardinalities and importances . On the other hand , metrics specifically proposed for imbalanced learning are either domain-specific , do not easily generalize beyond two classes , or can not support varying class importance ( e.g. , Balanced Accuracy ) ( Japkowicz , 2013 ) . Let us illustrate the problem with the simple example in Figure 1b . The test dataset consists of 100 data items from 3 classes ( A , B , C ) . The greatest majority of the items belong to class C ( 70 ) , but class B ( 20 ) has the greatest importance ( 0.7 ) . In other words , Cardinality and Importance are both non-uniform and in favor of different classes ( i.e. , representing the top-right quadrant of Figure 1a ) . The confusion matrix on the right shows the results from a classifier run against this test dataset . Unsurprisingly , the classifier performed the best for the majority class C ( 60/70 correct predictions ) . When evaluated using the traditional Accuracy metric , neither Class Cardinality nor Class Importance is taken into account . If Balanced Accuracy is used instead , we observe the degrading impact of the Class Cardinality skew ( 0.38 < 0.65 ) , but Class Importance is still not accounted for . This example demonstrates the need for a new evaluation approach that is sensitive to both Cardinality and Importance skew , as well as any arbitrary correlations between them . This is especially critical for ensuring a fair comparative assessment across multiple classifiers or problem instances . Our goal in this paper is to design an evaluation framework for imbalanced data classification , which can be reliably used to measure , compare , train , and tune classifier performance in a way that is sensitive to non-uniform class importance . We identify two key design principles for such a framework : • Simplicity : It should be intuitive and easy to use and interpret . • Generality : It should be general-purpose , i.e. , ( i ) extensible to an arbitrary number of classes and ( ii ) customizable to any application domain . To meet the first goal , we focus on scalar metrics such as Accuracy ( as opposed to graphical metrics such as ROC curves ) , as they are simpler , more commonly used , and scale well with increasing numbers of classes and models . To meet the second goal , we target the more general n-ary classification problems ( as opposed to binary ) , as well as providing the capability to flexibly adjust class weights to capture non-uniform importance criteria that may vary across application domains . Note that we primarily focus on Accuracy as our base scalar metric in this paper , as it is seen as the de facto metric for classification problems ( Sci ) . However , our framework is general enough to be extended to other scalar metrics , such as Precision and Recall . Similarly , while we deeply examine three use cases ( log parsing , sentiment analysis , URL classification ) , our framework in principle is generally applicable to any domain with imbalanced class and importance distributions . We first provide a brief overview of related work in Section 2 . Section 3 presents our new , classweighted evaluation framework . In Section 4 , we show the practical utility of our framework by applying it over : ( i ) three log parsing systems ( Drain ( He et al. , 2017 ) , MoLFI ( Messaoudi et al. , 2018 ) , Spell ( Du & Li , 2016 ; 2018 ) ) using four real-world benchmarks ( Zhu et al. , 2019 ) ; ( ii ) a variety of deep learning models developed for sentiment analysis on a customer reviews dataset from Amazon ( Ni et al. , 2019 ) ; and ( iii ) an industrial use case for URL classification with real classifiers and datasets from four cyber-security companies . Finally , we conclude in Section 5 . 2 RELATED WORK . Imbalanced Data Classification . Imbalanced data is prevalent in almost every domain ( Cohen et al. , 2006 ; Batuwita & Palade , 2012 ; Makki et al. , 2019 ) . The growing adoption of ML models in diverse application domains has led to a surge in imbalanced data classification research ( He & Garcia , 2009 ; Sun et al. , 2009 ; He & Ma , 2013 ; Branco et al. , 2016 ; Hilario et al. , 2018 ; Johnson & Khoshgoftaar , 2019 ) . While the techniques widely vary , they fall under four basic categories : pre-processing training data to establish balance via sampling techniques ( Estabrooks et al. , 2004 ; Blaszczynski & Stefanowski , 2015 ) , building custom learning techniques for imbalanced training data ( Joshi et al. , 2001 ; Castro & de Pádua Braga , 2013 ) , post-processing predictions from an imbalanced model ( Maloof , 2003 ) , and their hybrids ( Estabrooks & Japkowicz , 2001 ) . In this paper , we do not propose a new imbalanced learning technique , but a general-purpose performance evaluation framework that could be used in the training and/or testing of models for any technique . Section 4 demonstrates the practical utility of our framework for a variety of real ML use cases . Evaluation Metrics . Traditional metrics for evaluating prediction performance such as Accuracy , Sensitivity/Specificity ( and their combination G-mean ) , Precision/Recall ( and their combination FScore ) were not designed with imbalanced data issues in mind ( Japkowicz , 2013 ) . In fact , most of these were originally intended for binary classification problems . To extend them to more than 2 classes , macro-averaging ( i.e. , arithmetic mean over individual class measurements ) is used . Macroaveraging treats classes equally ( Branco et al. , 2016 ) . Balanced Accuracy is a popular averagingbased approach . There are also probabilistic evaluation approaches that extend Balanced Accuracy with Bayesian inference techniques for both binary and multi-class problems ( Brodersen et al. , 2010 ; Carrillo et al. , 2014 ) . Close to our work , Cohen et al . ( 2006 ) introduced the notion of class weights , yet in the specific context of Sensitivity/Specificity for binary classification in the medical domain . Similarly , Batuwita & Palade ( 2012 ) proposed extensions to G-mean for the bio-informatics domain . In addition to these scalar ( a.k.a. , threshold ) metrics , graphical ( a.k.a. , ranking ) evaluation methods such as Receiver Operating Characteristic ( ROC ) curves or Precision-Recall ( PR ) curves ( and the Area Under the Curve ( AUC ) for such curves ) as well as their extensions to imbalanced data / multiclass problems were also investigated ( Weng & Poon , 2008 ; Japkowicz , 2013 ) . While these methods provide more detailed insights into the operational space of classifiers as a whole , they do not easily scale with use in problems with a large number of classes ( Branco et al. , 2016 ) . 3 CLASS-WEIGHTED EVALUATION FRAMEWORK . In this section , we present our new evaluation framework for multi-class learning problems in presence of arbitrary skews among class distributions and/or importances . Our framework builds on and extends commonly used scalar / threshold metrics such as Accuracy . These metrics were originally designed for binary classification problems , where there is typically more emphasis on one class ( the positive class , e.g. , anomalies ) . To adopt them to multi-class problems where there is no such single-class emphasis , each class ’ metric can be computed separately and then an overall aggregation ( i.e. , arithmetic mean ) can be performed . For example , Accuracy has been extended to BalancedAccuracy by following this approach . In our framework , we follow a similar aggregation strategy , however , we do it in a more generalized way that allows custom class weights to capture class importance . Furthermore , these class weights can be based on any importance criteria such as rarity , cost , risk , expected benefits , and possibly a hybrid of multiple such criteria . Therefore , it is critical to provide a flexible formulation that allows users or domain experts to adjust the weights as needed by their problem instance . In what follows , we present our new class-weighted evaluation framework in a top-down fashion . Using the basic notation summarized in Table 1 , we first formulate the general framework , and then we describe how this framework can be customized to different importance criteria scenarios by specializing the weights in a principled manner . For ease of exposition , we first focus on Accuracy as the underlying performance metric , and then we discuss how our approach can be adopted to other similar metrics . Finally , we end this section with a brief discussion of how our framework can be used in model training .

Summary: The paper presents a simple addition to the Balanced Accuracy approach - which the authors refer to as ‘importance’. However, there is nothing in the formulation of this concept which requires that this is an importance and could in fact be any form of weighting. The paper evaluates the new metric - but only agains the Balanced Accuracy metric (which seems quite restrictive).