Class-Weighted Evaluation Metrics for Imbalanced Data Classification
Authors: Min Du
Nesime Tatbul
Brian Rivers
Akhilesh Kumar Gupta
Lucas Hu
Wei Wang
Ryan Marcus
Shengtian Zhou
Insup Lee
Justin Gottschlich
Venue: ICLR 2022 Submitted (2022)
Keywords: Imbalanced data classification
Evaluation metrics
Log parsing
Sentiment analysis
URL classification

Abstract: Class distribution skews in imbalanced datasets may lead to models with prediction bias towards majority classes, making fair assessment of classifiers a challenging task. Metrics such as Balanced Accuracy are commonly used to evaluate a classifier’s prediction performance under such scenarios. However, these metrics fall short when classes vary in importance. In this paper, we propose a simple and general-purpose evaluation framework for imbalanced data classification that is sensitive to arbitrary skews in class cardinalities and importances. Experiments with several state-of-the-art classifiers tested on real-world datasets from three different domains show the effectiveness of our framework – not only in evaluating and ranking classifiers, but also training them.

Under review as a conference paper at ICLR 2022
EVALUATION METRICS FOR IMBALANCED DATA CLASSIFICATION
Anonymous authors Paper under double-blind review
ABSTRACT
Class distribution skews in imbalanced datasets may lead to models with predic- tion bias towards majority classes, making fair assessment of classifiers a chal- lenging task. Metrics such Balanced Accuracy are commonly used to evaluate a classifier's prediction performance under such scenarios. However, these met- ries fall short when classes vary in importance. In this paper. propose simple and general-purpose evaluation framework for imbalanced data classification that is sensitive to arbitrary skews in class cardinalities and importances. Experiments with several state-of-the-ard classifiers tested on real-world datasets from three dif- ferent domains show the effectiveness of our framework not only in evaluating and ranking classifiers, but also training them.
INTRODUCTION
For broad range of machine learning (ML) tasks, predictive modeling in the presence of imbal- anced datasets those with severe distribution skews has been a long-standing problem (He & Garcia, 2009; Sun et al. 2009 He & Ma 2013 Branco et al., 2016) Hilario al. 2018 Johnson & Khoshgoftaar 2019). Imbalanced training datasets lead to models with prediction bias towards majority classes, which in turn results in misclassification of the underrepresented ones. Yet, those minority classes often are the ones that correspond to the most important events of interest (e.g.. errors in system logs (Zhu al., 2019). infected patients in medical diagnosis (Cohen et al. 2006), fraud in financial transactions (Makki et al. 2019)). While there is often an inverse correlation be- tween the class cardinalities and their importance (i.e., rare classes are more important than others), the core problem here is the mismatch between the way these two distributions are skewed: the tth most common class not necessarily the 1h most important class (see Figure la for an illustration). In fact, rarity is one of many potential criteria that can determine the importance of a class, which usually positively correlated with the costs or risks involved in its misprediction. Ignoring these criteria when dealing with imbalanced data classification may have detrimental consequences.
Consider automatic classification of messages in system event logs as an example (Zhu et al., 2019). An event log is a temporal sequence of messages that have transpired for given software system (e.g.. operating systems, cyber-physical systems) over a certain time period. Event logs are partic- ularly useful after a system has been deployed, as they can provide the DevOps teams with insights about errors outside of the testing environment, thereby enabling them to debug and improve the system quality. There is typically and the frequency of the errors it produces in its event logs. Furthermore, the message types that appear least frequently in an event log are usually the ones with the greatest importance. example of this was a rare anomaly in Uber's 's self-driving car that led to the death of a pedestrian, since the system flagged as a false positive in its logs (Efrati) 2018). If this event had not been misclassified and dismissed by the system, the pedestrian death in Arizona may have been avoided.
inverse correlation between the stability/maturity of system
concrete
A plethora of approaches have been proposed for building balanced classifiers (Sun al., 2009 Branco et al. 2016)). A fundamental issue that still remains an open challenge is the lack of generally-accepted methodology for measuring classification performance. The traditional metrics, which are designed to evaluate average case performance (e.g.. Accuracy) not capable of cor- rectly assessing the results in presence of arbitrary skew mismatches between class cardinalitics and importances. On the other hand, metrics specifically proposed for imbalanced learning are ei- ther domain-specific, do not easily generalize beyond two classes, or can not support varying class importance (e.g., Balanced Accuracy) (Japkowizz 2013).
Let us illustrate the problem with the simple example in Figure [b] The test dataset consists of 100 data items from 3 classes (A, B, C). The greatest majority of the items belong class C. (70), but

---

Under review as a conference paper at ICLR 2022
Predicted labels
A
B
Impuriant clasecs aec not that nec
Rare tiassas cue by more important
Class
Cardinality
Importance
4
3
Importance unitern
A
10
0.2
8
20
0.7
Troe
B
C
0.1
C
5
5
60
imhalance or mismmtch issue Accuracy
Rare clasect are not that impoctunt Balanced Accuracy
Accuracy
Correct predictions prodictions
No conzideration of Cardinulity Importance!
- -(1+4160) 100 900-0.65 0.65
uniform
non-unifons
Balanced Accuracy (Accuracy(A) Accuracy(i)) Accuracy())) 13
conrideration
Class Cardinality
-L1/10 + 4:20 60/000003333-1 0.38
Importance!
(a) Skew in distributions
(b) Example
Figure 1: Skew in distributions of Class Cardinality or Class Importance, and the potential mismatch between them render existing accuracy metries unusable in general multi-class prediction problems.
class B (20) has the greatest importance (0.7). In other words, Cardinality and Importance are both non-uniform and in favor of different classes (i.e., representing the top-right quadrant of Figure lak. The confusion matrix on the right shows the results from classifier run against this test dataset. Unsurprisingly, the classifier performed the best for the majority class C (60/70 correct predictions). When evaluated using the traditional Accuracy metric, neither Class Cardinality nor Class Impor- tance is taken into account. If Balanced Accuracy is used instead, we observe the degrading impact of the Class Cardinality skew (0.38 < 0.65), but Class Importance is still not accounted for. This example demonstrates the need for new evaluation approach that is sensitive to both Cardinality and Importance skew, as well as any arbitrary correlations between them. This is especially critical for ensuring fair comparative assessment across multiple classifiers problem instances.
Our goal in this paper is to design an evaluation framework for imbalanced data classification, which can be reliably used to measure, compare, train, and tune classifier performance in a way that is sen- sitive to non-uniform class importance. We identify two key design principles for such a framework:
Simplicity: It should be intuitive and easy and interpret. Generality: It should be general-pun seneral-purpose, i.e., (i) extensible to an arbitrary number of classes and (ii) customizable any application domain.
To meet the first goal, we focus on scalar metrics such as Accuracy (as opposed to graphical metrics such as ROC curves), as they are simpler, more commonly used, and scale well with increasing num- bers of classes and models. To meet the second goal, we target the more general n-ary classification problems (as opposed to binary), as well as providing the capability to flexibly adjust class weights to capture non-uniform importance criteria that may vary across application domains. Note that we primarily focus on Accuracy as our base scalar metric in this paper, as it is seen as the de facto metric for classification problems (Sci). However, our framework is general enough to be extended to other scalar metrics, such as Precision and Recall. Similarly, while we deeply examine three use cases (log parsing, sentiment analysis, URL classification), our framework in principle is generally applicable to any domain with imbalanced class and importance distributions.
We first provide brief overview of related work in Section 2 Section presents our new, class- weighted evaluation framework. In Section 4 we show the practical utility of our framework by applying it over: (i) three log parsing systems (Drain (He et al.) (017). MoLFI (Messaoudi al. 2018), Spell (Du & Li, 2016 (018)) using four real-world benchmarks (Zhu et al. (2019); (ii) variety of deep learning models developed for sentiment analysis on customer reviews dataset from Amazon (Ni et al.) 2019); and (iii) an industrial use case for URL classification with real
classifiers and datasets from four cyber-security companies. Finally, we conclude in Section
RELATED WORK
Imbalanced Data Classification. Imbalanced data is prevalent in almost every domain (Cohen
jct al. 2006 Batuwita & Palade 2012,, Makki et al.) 2019). The growing adoption of ML models
in diverse application domains has led to surge in imbalanced data classification research (He &
Garcia 2009 Sun et al. 2009 He & Ma 2013 Branco et al.) 2016 Hilario et al. 2018 Johnson
& Khoshgoftaar 2019). While the techniques widely vary, they fall under four basic categories:
pre- prceppoccssinn training data to establish balance via sampling techniques (Estabrooks et al., 2004]
Blaszczynski & Stefanowski 2015), building custom learning techniques for imbalanced training
2

---

Under review as a conference paper at ICLR 2022
Table 1: Notation
Notation
Description
N
total number of data items
C
total number of dara item classes
M
number importance eriteria
14
true numbher of data items in class
correcily predicted number of data items in class i
J1
telative trequency of class
Wi
relative weight of 1
relative user-dehuod importance of class
F;
relative rarity of class
relative weight of
impottance criteria j
Accuracy:
Accuracy of class i
data (Joshi et al. 2001; Castro & de Pádua Braga 2013). post-processing predictions from an im- balanced model (Maloof 2003). and their hybrids (Estabrooks & Japkowicz, (001). In this paper, we do not propose new imbalanced learning technique, but a pereral- performance evalua- tion framework that could be used in the training and/or testing of models for any technique. Section 4 demonstrates the practical utility of our framework for variety of real ML use cases.
Evaluation Metries. Traditional metrics for evaluating prediction performance such as Accuracy, Sensitivity/peecffiitty (and their combination G-mean), Precision/Recall (and their combination F- Score) were not designed with imbalanced data issues in mind (Japkowicz. 2013). In fact, most of these were originally intended for binary classification problems. To extend them to more than classes, macro-averaging (i.e., arithmetic mean over individual class measurements) is used. Macro- averaging treats classes equally (Branco et al.] 2016). Balanced Accuracy is popular averaging- based approach. There are also probabilistic evaluation approaches that extend Balanced Accuracy with Bayesian inference techniques for both binary and multi-class problems (Brodersen et al. 2010) Carrillo et al. 2014). Close to our work, Cohen et al. (2006) introduced the notion of class weights, yet the specific context of Sensitivity/Specffiity for binary classification in the medical domain. Similarly, [Batuwita& & Palade (2012) proposed extensions to G-mean for the bio-informatics domain. In addition to these scalar (a.k.a., threshold) metrics, graphical (a.k.a., ranking) evaluation methods such as Receiver Operating Characteristic (ROC) curves or Precision-Recal (PR) curves (and the Area Under the Curve (AUC) for such curves) as well as their extensions to imbalanced data multi- class problems were also investigated (Weng & Poon| 2008 Japkowicz 2013). While these methods provide more detailed insights into the operational space of classifiers as a whole, they do not easily scale with use in problems with large number of classes (Branco al. 2016).
3 CLASS- WEIGHTED EVALUATION FRAMEWORK
In this section, we present our new evaluation framework for multi-class learning problems in pres- ence of arbitrary skews among class distributions and/or importances. Our framework builds on and extends commonly used scalar threshold metrics such as Accuracy. These metrics were origi- nally designed for binary classification problems, where there is typically more emphasis on one class (the positive class, c.g., anomalies). To adopt them to multi-class problems where there is no such single-class emphasis, each class' metric can be computed separately and then an overall aggregation (i.e., arithmetic mean) can be performed. For example, Accuracy has been extended to Balanced/Accuracy by following this approach. In our framework, we follow similar aggre- gation strategy, however, we do it in a more generalized way that allows custom class weights to capture class importance. Furthermore, these class weights can be based on any importance criteria such as rarity, cost, risk, expected benefits, and possibly hybrid of multiple such criteria. There- fore, it is critical to provide flexible formulation that allows users or domain experts to adjust the weights as needed by their problem instance. In what follows, we present our new weightweighted evaluation framework in a top-down fashion. Using the basic notation summarized in Table T we first formulate the general framework, and then we describe how this framework can be customized to different importance criteria scenarios by specializing the weights in a principled manner. For ease of exposition, we first focus on Accuracy as the underlying performance metric, and then we discuss how our approach can be adopted to other similar metrics. Finally, we end this section with brief discussion of how our framework can be used in model training.
3.1 WEIGHTED BALANCED ACCURACY (WBA)
Suppose we are given a test dataset with N data items in it, each of which belongs to one of C distinct classes. Furthermore, each class contains n, of the data items in this dataset. Thus:
3

---

Under review as a conference paper at ICLR 2022
N
The relative frequency of each class i the whole dataset is:
22)
N
Assume a classifier that makes a prediction about the class label of each data item in the test dataset, and correctly predicts p: out of n, labels for given class i, where p: ng. Then, the total number of correct predictions out of all the predictions gives us the overall Accuracy of the classifier:
Σ P
Accuracy
(3)
The classifier's Accuracy, for given class i (a.k.a,, per-class Recall score) can be computed as:
Accuracy,
Ph
(4)
m
BalanceddAccuracy is the macro-average of Accuracy, over all classes in the dataset:
BalancedAccuracy-
Accuracy,
(5)
1m1
The above formulation represents the state of the art in how prediction accuracy is evaluated for multi-class classifiers in presence of imbalanced datasets (i.e., those where fi are not even). While for balanced datasets (i.e., Vi, imbalanced datasets, BalancedAAccurac ensures that the prediction accuracy is not inflated due to high-frequency classes' results dominating over the others'. aadaaneedAecurare works well as long as each class of the same importance, since is the simple arithmetic mean across per-class accuracy measurements (i.e., each class' accuracy contributes evenly to the overall accuracy). As we discussed in earlier sections, in many real-world classification problems, this assumption does not hold. Rather, classifiers must be rewarded higher scores for their prediction performance on more important classes. In order to capture this requirement, we generalize Accuraacy into W "eeightedBalanceddccco Accuracy by extending with per-class importance weights w; as follows:
N/C and fa 1/C) BolancedAAccuaay
Accuracy, for
WeightedBaalonedAAeoree
Σ wi x Accuracy,
66)
This simple yet powerful extension enables to capture both skews and imbalances in class cardi- nalities as well as importances (i.e., the complete design space in Figure Ta}. This general formu-
lation can support any importance criteria for weights as long as 0 < w; S and Σ W; 1. In the following subsections, we present general use of WBA with custom weights, for other scalar metrics, as well as in improving model training.
3.2 WEIGHT CUSTOMIZATION
In multi-class setting, not only may the classes carry different importance weights, but also the cri- teria of importance may vary from one problem or domain another. We now discuss several types of criteria that we think are commonly seen in applications. This is not meant to be an exhaustive list, but provides examples and templates that can be easily tailored to different problems.
Importance criteria User-defined. This is the most general and flexible form of importance cri- teria. The application designer or domain expert specifies the relative weight of each class based on some application-speeffe criteria. As example, the problem might be about classifying images of different types of objects in highway traffic and the user gives higher importance to correct recog- nition of certain objects of interest (e.g., pedestrians, bikes, animals, etc). We express user-defined relative weight of class with u, which is simply used as w; in Equation (i.e. w; = u,).
Importance criteria Rarity, It is often the that the rarer something is, the more noteworthy or valuable it is. multi-class problems, this corresponds to the case when importance i is inversely correlated with its relative frequency of occurrence (fz) in the dataset. For example,
class
4

---

Under review as a conference paper at ICLR 2022
in system log monitoring, log messages for more rarely occurring enrors exceptions (e.g., denial of service attack) are typically of higher importance. Therefore, a classifier that performs well on detecting such messages must be rewarded accordingly. In our framework, capture rarity using weights that are based on normalized inverse class frequencies formulated as follows:
UN
ド 1
Multiple importance criteria. In some problems, importance of a class depends on multiple differ- ent criteria (c.g., both rarity and a uscr-defined criteria). To express class weights in such scenarios, we leverage techniques from multi-criteria decision making and multi-objective optimization (Ti- antophyylou 2000 Helff et al.) 2016). One of the most basic methods using normalized weighted sums based on composite weights (Helff et al. 2016). Composite weights can be computed either in additive or multiplicative form (Tofallis 2014). The multiplicative approach tends to promote weight combinations that are uniformly higher across all criteria, and as such is found to be more preferred approach in application scenarios similar to ours (Helff et al., 2016 Tofallis 2014). While present this approach here, in principle, other approaches from multi-enteria decision making theory could also be used within our framework. Given M different criteria with maj denoting the relative weight of class for criteria j, we can compute the composite weight of a class i as follows:
may
WL EE= ma.s
(8)
For example, if we had two criteria, rarity r and user-defined u with weights r, and uy for each class i, respectively, then the composite weight for class i would be uy
Partially-defined importance criteria. One commonly expected scenario (especially in those clas- sification problems where the number of classes C' can be very large) is that not all of the class importance weights might be supplied by the user. For example, in sentiment analysis use case, the user supplies the weights for all the negative classes, and leaves the others unspecified. Our framework can support such cases by automatically assigning weights the unspecified classes. The default approach is to distribute the remaining portion of weights evenly across all unspecified classes: total weights specified) (number of unspecified classes). If the user prefers an alterna- tive approach (e.g.. distribute the remainder based on rarity of the unspecified classes), this can also be easily supported by our framework.
3.3 METRIC CUSTOMIZAATION
The class-weighted evaluation framework presented above focused on the popular Accuracy metric as the underlying metric of prediction performance. However, our framework follows a general structure based on the idea of weighted macro-averaging with customizable weights, which can essentially be used with any performance metric that can be computed on the basis of a class. For example, the macro-averaging approaches that are already being used for Precision, Recall, and F- Score could easily be extended with our customizable weighing approach by replacing Accuracy in our formulas with one of these metrics.
3.4 MODEL TRAINING IMPROVEMENT USING CLASS WEIGHTS
The customized weights presented in Section 3.2 not only helps to give user- user- user-preferred ranking via the proposed WBA metric, but also helps improve model training. Recall that ML model training aims to minimize the loss between model predictions and ground truth labels. A common practice is to minimize the sum of all per-sample losses. Using Loss; to denote the total loss incurred by all samples within class i, the model loss to minimize in training would be: L Σ Loss;.
By applying class importance weights wi as suggested in Section 3.2 the loss value for important classes takes larger portion the final loss value to minimize. This enables the back- prok-propngttion process to focus more on optimizing the model parameters for the higher weighted classes, and thus improves their accuracy. The model loss incorporating class importance weights becomes:
C =
wi x Loss;
(9)
5

---

Under review as a conference paper at ICLR 2022
The advancement of popular deep learning frameworks have made this process rather straight- forward to implement. For example, the TensorFlow API provides a parameter named which allows user to pass in JSON data structure that specifies a weight value for each class label (TensorFlow). Similarly, the PyTorch API for cross entropy loss also provides parameter to pass in class weights (PyTorch). These provide perfect interfaces for the usage of the importance weights in our WBA framework, both to improve a model's learning ability of the important classes as well as raising the overall WBA score of the classification outcomes.
4 EXPERIMENTAL STUDY
We now present an experimental analysis of our new framework for three application domains. Our goal is to demonstrate the value of WBA compared to existing metrics when evaluating ML models over real-world imbalanced data classification problems. As we will show, often times a traditional metric like Accuracy or Balanced Accuracy will make classifier A seem preferable to classifier B, when reality classifier B is superior. In addition, we also provide an analysis of how WBA can positively impact, not only the testing of models, but also their training. Details of our experiments (inel. code, data, and examples) can be found in the supplementary file and in the appendix.
4.1 USE CASE I: LEARNED LOG PARSING
ML-based log parsers are tools that are designed to automatically learn the structure of event logs generated by hardware and software systems to properly categorize them into event classes (c.g., different error types). In our first study, we used WBA to evaluate 3 ttate-of-the-ar log systems: Drain, Spell, and MoLFI (Du & Li [016) He al. 2017: Messaoudi et al.) 2018). We start by providing an abbreviated description of our experimental setup.
Log Parsing Systems. Drain is rule-based, online log parsing system that encodes the parsing rules in fixed-depth parse tree (He et al.) (017). It performs a pre-processing step for each new log message using regular expressions created by domain experts. Spell. like Drain, is also rule-based; it principally uses the longest common subsequence (LCCS) to find new log message classes (Du & Li 2016). Finally, Mol.Fl casts log parsing as multi-ohjective optimization problem and provides a solution based on genetic programming (Messaoudi et al.] (2018).
Datasets. We test each aforementioned log message classification system with four real-world datasets taken from a public benchmark (Zhu et al. 2019). Each dataset has 2000 log instances randomly sampled from a larger dataset. The macOS dataset contains raw log data generated by the macOS operating system (341 log classes, 237 infrequent classes (i.e., those that have fewer occur- rences in the dataset than the average number of messages per class), and an average class frequency of 5). The BlueGene/l. (BGL.) dataset is a collection of logs from the BlueGene/LL supercomputer system (120 log classes, 101 infrequent classes, and an average class frequency of 16). The An- droid dataset consists of logs from the Android mobile operating system (Zhu al. 2019)] (166 log classes, 127 infrequent classes, and an average class frequency of 16). Finally, the HDFS dataset consists of log data collected from the Hadoop Distributed File System (14 log classes, infrequent classes, and an average class frequency of 142). Overall, the first three datasets are highly skewed in class frequencies, whereas the HDFS dataset is relatively much less skewed (see Appendix[A.. [A.]). Results. For Drain, Spell, and MoLFI, traditional metrics of Precision, Recall, FI-Score, and Accuracy (named Parsing Accuracy in the original papers) were used for training and testing clas- sification performance. None of these metrics are class-sensitive while in log parsing, messages have in fact varying importance across the classes. The importance criteria is rarity: the more rare an error message is, the more important it is to correctly classify this message. To capture this, we configure the WBA to WBArrrity- which automatically assigns weights to WBA based on the dataset classes' inverse frequencies, as described in Section 3.2 Then we evaluate the test results from the 3 parsers over 4 datasets using WBArrrity and compare against traditional metrics in two categories: class-insenan insenssiveee and class-sensitie as shown in Figure
metrics (specifically, F1-Score and Accuracy) agree on how to rank the classification performance of the 3 parsers across all datasets (for macOS and Android, Drain > Spell > MoLFI; for BGL, Drain > MoLFI > Spell; for HDFS, all perform similarly). Since WBArrriit is sensitive to classes' data distribution and importance skews, it makes a completely different judgement. Furthermore, it ranks the techniques differently for each dataset (Drain > MoLFI > Spell in macOS; MoLFI Spell Drain for BGL; Spell Drain > MoLFI for Android; and for HDFS, Spell > Drain and MoLFI). The WBA ranking aligns with our observation on per-class accuracy of the methods. For example, BGL. dataset, although Spell
WBArersty vs. dlass-insensitive Metrics: The

---

Under review as a conference paper at ICLR 2022
Clu www.-aaaatttteiieo Metiica
Cau--arative Matria
8.66
Poiddion
Necml
11.com
Edhaced
Flcil
Scor
Accuriy
(a) macOS (skew 8.454)
(b) BGL (skew 8.900)
Clax- ta Metria
CCasonrtthee Mesic
Drain (15pall
Crann (iSpal BINA
(c) Android (skew 4.822)
(d) HDFS (skew = 0.202)
Figure 2: WBA vs. class-insensitive & agree in all. BA& & WBA agree in (c) & (d) only. WBA disagrees with class-insensitive in all.
metrics for log parsing: F1-Score & Accuracy
has the most mis-classified samples and in most classes (and thus the lowest overall Accuracy and Balanced Accuracy), it has very few mis-classified samples for rare classes, making its WBA higher than Drain. On the other hand, MoLFI performs the best on rare classes and thus has the best WBA yarsty. A discussion on per-class performance can be found Appendix A.. This result validates that WBArrriity provides more sensitive tool for assessing classification performance. WBAversty vs. Balanced Accuracy (BA): As discussed carlier, BA is class-sensitive, but only to dis- tribution imbalance. We can observe the difference between BA and WBA in Figure 2 In macOS and BGL,, where the skew is the highest and rarity is more pronounced, the two metries completely disagree in how they rank the parsers. In contrast, for Android and HDFS, where the skew is lower, there is an overall agreement, although the separation in metric values slightly differ. Of particular importance is the difference seen in Figure 2a We observe that the best performing model is Spell when scored by BA, and Drain when scored by WBAry The reason for this difference due to Spell's and Drain's is differences in their ability to correctly classify infrequent classes, i.e., those that represent failures and errors that require the most immediate response.
4.2 USE CASE 2: SENTIMENT ANALYSIS
In social media and other user-facing domains like e-commerce sites, is often useful to understand the view or feelings ("sentiments") associated with users' behavior or preferences. In the second part of our experimental study, we apply WBA in the context of such a sentiment analysis use case, which involves analyzing text-based product reviews from Amazon's e-commerce websites.
Dataset. The dataset consists of customer reviews and ratings of Amazon products (Ni et al. 2019). The task is to classify the reviews into 5 classes (with 1 being the lowest and 5 being the highest review rating a product can get), where ratings constitute the ground truth class labels. There is high class imbalance in this dataset (54) As shown in the Frequency column of Table 2 Class 5 with the highest customer rating clearly dominates compared to the others. It is known that the distribution of customer review ratings is typically imbalanced and generally follow J-shaped distribution (Mudambi & Schuff 2010, Pavlou & Dimoka, (006).
Sentiment Analysis Models. We compare 4 types of recurrent neural networks (RNN), all con- sisting of an embedding layer with pre-trained word embeddings from (Pennington et al. 2014) followed by a recurrent layer from PyTorch (Subramanian, 2018): RNN, LSTM, GRU, BiLSTM. The hidden state output from the last time step of these are passed to fully-connected layer with input of 256 neurons and output from 5 neurons.
Results. For this use case, we first worked with a user-defined importance criteria borrowed from published studies suggesting that extreme review ratings (classes and 5) carry more importance Mudambi & Schuff, 2010: Pavlou& & Dimoka 2006). Thus, we set the weights as shown in Table 2 (shown as WBA(user) or user in Figure 3).
WBA Other Accuracy Metrics: First, we compare WBA(user) with Accuracy and BalancedAc- curacy (BA) when used as a metric for both training and testing of the 4 DNN models (Figure Ba). We make a few observations: (i) The class-insensitive Accuracy showcases the imbalance problem
7

---

Under review as a conference paper at ICLR 2022
Table 2: Amazon per-class breakdown: Frequencies are highly skewed (skew=2.140); Accuracy; in each model when both trained+tested with user-defined weights wi (same weights as in Figure 3a
Class
Frequency (f+)
Weights (wr)
LSTM
RNN
GRU
BILTMM
0.092
0.7
0.19
0.04
0.16
0.17
2
0.052
0
0
0
0.075
0
4
0.142
0
0
5
0.639
0.3
0.81
0.96
0.84
0.83
8.1A
Weightd slaved
0
6.194
02
Aatuaay
MANSON]
Metri
Inportincs Cotere
WUST DANN CCEU DRLST
WBA vs. Other Metrics (Train+Test)
(b) WBA: Test vs. Train+Test
Figure Amazon results
in classification, as it favors the RNN model which is heavily biased by the majority class (see Accuracy for RNN in Table 2 where class 5 scores 0.96). (ii) The srenuivcy-sensitive BA met- ric finds all models perform similarly. WBA(user), in contrast, identifies LSTM as the best model. Indeed, Table 2 confirms that LSTM performs relatively the best in predicting the most important class, class 1 (0.19 accuracy). Overall, we find that WBA is capable of capturing importance skews, when the frequency skew can be high and biased towards less important classes.
Impact of WBA in Model Training: Next we explore the use of WBA not only in model evaluation, but also in training. We focus on models (LSTTM and RNN), and apply WBA only during testing vs. to both training (by extending loss functions of DNNs with weights as in Section 3.4 and testing. Intuitively, if a model is trained being aware of the importance weights, then it should also perform well when tested against the same criteria. To test this hypothesis, we repeated the experiment for 3 alternative importance criteria: (i) rarity (w) = 0.209, W 0.368, w/ = 0.255, 24 = 0.136, W3- 0.030), (ii) user-defined (i.e., with weights in Table 2 and (iii) composite of the two (wi 0.62, W2 = Wi = W4 = O, Wis- = 0.38). In Figure 3b we observe: (i) Except for rarity, WBA for both LSTM and RNN improves when integrated into model training. This verifies our intuitionn and shows that WBA is a useful metric not only for evaluation, but also for training. (ii) When zoom into rarity, we that although class 2 is the most important, per-class accuracy for class 5 is much higher for both LSTM and RNN in the Test-only case, because both models are still trained heavily biased towards the majority class (5). (iii) Though rarity by itself is not useful in training, when combined with user importance, it visibly improves the WBA scores, This shows that our multi-criteria composition approach is capable of combining importance criteria as intended.
4.3 USE CASE 3: URL CLASSIFICATION
URL classification is a crucial task in the cyber-security industry for managing web traffic. Given a URL, the goal is to categorize the corresponding webpage into several well-defined classes such as benign (i.e., no harmful content), malware, phishing, NSFW (i.e., not safe for work), etc. Given the vast number of URLs available, accurately learning and evaluating URL classifier can greatly help in automatically categorizing webpages without requiring manual labeling (Vallina et al.) 2020). Dataset. Our dataset contains URLs from several different types of categories: benign (e.g., news, sports), NSFW (e.g., drugs, gambling), and malicious (phishing and malware). These URLs were sampled from a variety of third-party sources so as to minimize potential data bias. Many of the phishing and malware URLs were sampled from VirusTotal (VirusTotal). which aggregates various cyber-security vendors' detections into a single site, while benign URLs were sampled from Alexa's Top Sites lists (Amazon), which rank websites according to their popularity. We are unable to reveal the exact URLs used in the analysis due to legal reasons, but we tried to ensure that our dataset is representative of the types of URLs encountered in real-world URL filtering scenarios.
URL Classification Services. We tested the performance of URL filtering products from 4 commer- cial companies, whose names had to be anonymized as A, B, C. D. Since each URL filtering product has its own unique taxonomy of categories, we created mappings from each service's 's category space to single shared category space that we can easily compare the results. Furthermore, since no
8

---

Under review as a conference paper at ICLR 2022
Table 3: Evaluating and ranking the URL classificattio services
Classilication Accuracy
Category
# RRRss
Rarity
Laer wa
Ranking
Service
Service B5
Service C
Service
benign
16762
0.04
0.761
0.815
0.061
0.853
DBAC
NSFW
S276
0.14
0.05
0.965
0.804
0.533
0.767
ABDC
mallware
1913
0.38
0.8
0.890
0.845
0.602
0.872
ADBC
phishing
1675
0.44
0.1
0.968
0.811
0.521
0.771
ABDC
Accuracy
0.826
0.621
0.831
DABC
Balanced Accuracy
0.89%
0.819
0.579
0.816
ABDC
0.929
0.823
0.559
0.812
ABDC
WRAue
0.895
0.856
Table 4: Training and evaluating URLNet model using WBA
Datasel Stitistis
Classification Accuracy
Category
Train #URLs
Test #URLs
Rarity
User
Train with no WBA
Train with rarity
Train with user w.
Wi
benign
10000
6762
0.04
0.05
0.981
0,300
0.458
NSFW
3150
2126
0.14
0.15
0
0.608
0.418
malware
1143
770
0.38
0.45
0.705
0.839
0.895
phishing
1000
675
0.44
0.35
0.782
0.788
0.754
Test with Accuracy
0.745
0,435
0.502
Test with Balanced Accuracy
0.617
0.634
0.631
with WBAmmtt
0.653
NA
Test with WBA.uer
0,640
NA
0.752
company had access to our test dataset sampled from various third-pane parrt-patty sources, no company was able to gain an unfair advantage in this URL classification task.
Results. We submit the sampled URLs from each category to the above services, calculate their per-category accuracy, and compute final accuracy score for competitive analysis.
Evaluating and Ranking the Classifiers: Table 3 shows our results for Companies A-D. For overall Accuracy, D is the best as it has the highest classification accuracy on benign class (i.c., the majority class). With Balanced BaluaccedAccuracyy, we can see that A is the best, since has higher accuracy for non- benign classes compared with other services, and this advantage is more evident with WBArerity shown as a larger gap between their WBA values. B is better than D when using WBAyrrity but if users consider malicious class be the most important and apply very high weight on (e.g.. 0.8), the resulted Improving Model Training with WBA: In this experiment, we adopt our dataset train URLNet model (Le et al. (2018)) by applying various WBA weights as suggested in Section 3.2 to show its effectiveness in terms of improving corresponding WBA metric. As shown Table 60% of the dataset is used for training while the remaining 40% is for evaluation.
would show that D is preferable over B.
ClassifCaasiiicatiin Accu- racy columns show the results under different settings. For vanilla training (without class weights), get a high accuracy for the benign class but low accuracy for the others. The main reason is benign class has the most number of URLs in training data. In order to boost the accuracy for non- benign classes, apply rarity weights. With this, the accuracy for all non-benign and rarer classes are improved (noticeably, from 0 to 0.608 for the NSFW category). Note that malware content is often considered more harmful among all non-benign categories. For the third experiment, ap- ply user-defined weights which assign higher weight on the malware class, successfully boosting its accuracy from 0.839 0.895. It is clear that applying corresponding class weights in training can significantly improve the related WBA accuracy on the test dataset. For instance, when applying rarity weights, WrArrryymmprovee from 0.653 0.761; while WBA improved from 0.640 to 0.752 if the same user-defined weights are used in training. This aligns with our observation that increasing the weights in concerned categories can notably improve the accuracy those cate- gories. As comparison, neither the Accuracy nor the Balanced Accuracy measure does good job ranking the different training methods.
5 CONCLUSION
In this paper, presented simple yet general-purpose class-sensitive evaluation framework for imbalanced data classification. Our framework designed to improve the grading of multi-class classifiers in domains where class importance is not evenly distributed. We provided modular and extensible formulation that can be casily customized to different importance criteria and metrics. Experiments with three real-world use cases show the value of metric based on our framework, Weighted Balanced Accuracy (WBA), over existing metrics in not only evaluating the classifiers' test results more sensitively to importance criteria, but also training them so.

---

Under review as a conference paper at ICLR 2022
ETHICS STATEMENT
This paper introduces a measurement framework for training and evaluating classification models. The primary goal of this framework is enable the creation of more accurate models, especially in presence of imbalances across class distributions and their importances. As with any technological tool, our work could be subject to misuse. In partieular, unfair bias might be introduced learned models by purposefully misconfiguring the class importance weights. To avoid such ethical risks, we urge the community to use our tool in a responsible manner.
REPRODUCIBILITY STATEMENT
This research is highly experimental nature. We provide an appendix section well an addi- tional file with supplementary material in order to document all of our experimental artifacts and how they can be used to reproduce the results we reported in this paper. These artifacts include all datasets used in the experiments, as well as code. Except for the URL classification use case, where the URLs cannot be released either because of our contract with VirusTotal, or due to being pustomerr-snnsttie information. Nevertheless, we have included the class labels and class predictions for each setting, and corresponding scripts to calculate various WBA metrics which can reproduce our results the paper. In particular, the appendix section includes extra experimental results validate our point, as well as the details for each experiment. Our supplementary material contains 3 distinct folders with source code and scripts for the 3 cases in evaluation. Each folder contains README file that instructs how to run the code and reproduce the results.
REFERENCES
Scikit-learn Classification Metrics. ht http://dckkt--llem.oo//taalleled model_estt eva wwa.luaaiio htt.aaa.o..ttttcccccccaaaaaaaooa
Amazon. Alexa Top httpps//ww..lleee.cm///ppttttt
Rukshan Batuwita and Vasile Palade. Adjusted Geometric-mean: A Novel Performance Measure for Imbalanced Bioinformatics Datasets Learning. Journal of Bioinformatics and Computational Biology, 10(4), 2012.
Jerzy Blaszczynski and Jerzy Stefanowski. Neighbourhood Sampling in Bagging for Imbalanced Data. Newrocomputing, 150:529-544 54299255222222 2015.
Paula Branco, Luís Torgo, and Rita P Ribeiro. A Survey of Predictive Modeling on Imbalanced Domains. ACM Computing Surveys (CSUR), 49(2):1-50, 2016.
Kay Henning Brodersen, Cheng Soon Ong, Klaas Enno Stephan, and Joachim M Buhmann. The Balanced Accuracy and Its Posterior Distribution. In 20th International Conference Pattern Recognition (ICPR), pp. 3121-3124, 2010.
Henry Carrillo, Kay H Brodersen, and José A Castellanos. Probabilistic Performance Evaluation for Multi-class Classification using the Posterior Balanced Accuracy. In First Iberian Robotics Conference, pp. 347-361, 2014.
Cristiano Leite Castro and Antônio de Pádua Braga. Novel Cost-Sensitive Approach to Improve the Multilayer Perceptron Performance on Imbalanced Data. JEEE Transactions on Neural Networks and Learning Systems, 24(6):888-899) 2013.
Gilles Cohen, Melanie Hilario ad Hugo Sax, Stephane Hugonnet, and Antoine Geissbuhler. Learn- ing from Imbalanced Data in Surveillance Nosocomial Infection. Artificial Intelligence in Medicine, 37(1):7-18, 2006.
Min Du and Feifei Li. Spell: Streaming Parsing of System Event Logs. In IEEE International Conference on Data Mining (ICDM), pp. 859-864, 2016.
Min Du and Feifei Li. Spell: Online Streaming Parsing of Large Unstructured System Logs. JEEE Transactions Knowledge and Data Engineering (TKDE), (1(11):2213-222 2018.
10

---

Under review as a conference paper at ICLR 2022
Amir Efrati. Uber Finds Deadly Accident Likely Caused By Software Set to Ignore Objects On Road. The Information, 2018. http::/www..ttttttmmmaaoooooo//s seseeerr--ddddddlly soll---uuuuddbb---tttttteeetteee
on-raddd......
Andrew Estabrooks and Nathalie Japkowiez. A Mixture-of-Experts Framework for Learning from Imbalanced Data Sets. In International Conference on Advances in Intelligent Data Analysis
(IDA), pp. 34-43, 2001.
Andrew Estabrooks, Taeho Jo, and Nathalie Japkowicz. Multiple Resampling Method for Learn- ing from Imbalanced Data Sets. Computational Intelligence, (0(1):18-36 36, 2004.
Haibo He and Edwardo A. Garcia. Learning from Imbalanced Data. IEEE Transactions Knowl- edge and Data Engineering (TKDE), 2109):1232222 2009.
Haibo He and Yunqian Ma (eds.). Imbalanced Learning: Foundations, Algorithms, and Applica-
fions. Wiley-IEER Press, July 2013.
Pinjia He, Jieming Zhu, Zibin Zheng, and Michael R Lyu. Drain: An Online Log Parsing Approach with Fixed Depth Tree. In IEEE International Conference on Web Services (ICWS), pp. 33-40, 2017.
Florian Helff, Le Gruenwald, and Laurent d' d'Orazio. Weighted Sum Model for Multi-Objectio Query Optimization for Mobile-Cloud Database Environments. In EDBT/ICTT International Workshop on Mulli-Engine Data Analytics (MEDAL). 2016.
Alberto Fernandez Hilario, Salvador García Lopez, Mikel Galar, Ronaldo C. Prati, Bartosz Krawezyk, and Francisco Herrera. Learning from Imbalanced Data Sets. Springer, 2018.
Natalie Japkowicz. Assessment Metrics for Imbalanced Learning. In Haibo He and Yunqian Ma (eds.), Imbalanced Learning: Foundations, Algorithms, and Applications, pp. 187-206. John Wiley & Sons, 2013.
Justin M. Johnson and Taghi M. Khoshgoftaar. Survey on Deep Learning with Class Imbalance. Journal of Big Data, 6(27), 2019.
Mahesh V. Joshi, Vipin Kumar, and Ramesh C. Agarwal. Evaluating Boosting Algorithms to Clas- sify Rare Classes: Comparison and Improvements. IEEE International Conference on Data Mining (ICDM), pp. 257-264, 2001.
Hung Le, Quang Pham, Doyen Sahoo, and Steven CH Hoi. Urlnet: Learning a url representation with deep learning for malicious url detection. arXiv preprint arXiv: 2018.
Sara Makki, Zainab Assaghir, Yehia Taher, Rafiqul Haque, Mohand-Said Hacid, and Hassan Zeined- dine. An Experimental Study With Imbalanced Classification Approaches for Credit Card Fraud Detection. JEEE Access, 1933010990220 2019.
Marcus A. Maloof. Learning When Data Sets are Imbalanced and When Costs are Unequal and Unknown. ICML Workshop on Learning from Imbalanced Data Sets, 2003.
Salma Messaoudi. Annibale Panichella, Domenico Bianculli, Lionel Briand, and Raimondas Sas- nauskas. A Search-based Approach for Accurate Identification of Log Message Formats. In Proceedings of the 26th Conference on Program Comprehension, pp. 167-177, 2018.
Susan M. Mudambi and David Schuff. Research Note: What Makes Helpful Online Review? A Study of Customer Reviews on Amazon.com. MIS Quarterly, 34(1):185-20,, 2010.
Jianmo Ni. Jiacheng Li, and Julian McAuley, Justifying Recommendations using Distantly- Reviews and Fine-Graid Grained Aspects. In Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP- IJCNLP), pp. 188-197, 2019.
Paul A Paylou and Angelika Dimoka. The Nature and Role of Feedback Text Comments in On- line Marketplaces: Implications for Trust Building. Price Premiums, and Seller Differentiation. Information Systems Research, 1744):392-414, 2006.

---

Under review as a conference paper at ICLR 2022
Jeffrey Pennington, Richard Socher, and Christopher D. Manning. GloVe: Global Vectors for Word
Representation. Empirical Methods Natural Language Processing (EMNLP), pp. 1532- 1543, 2014.
PyTorch. PyTorch cross entropy loss API.
httpss//prrorrd.ooor////sstttt/
generated/ nn www.srrroosoooyooooooooo Losy html
Vishnu Subramanian. Deep Learning with PyTorch: A Practical Approach to Building Neural Net- work Models using PyTorch. Packt Publishing. 2018.
Yanmin Sun, Andrew K. C. Wong, and Mohamed S. Kamel. Classification of Imbalanced Data: A
Review. International Journal of Pattern Recognition and Artificial Intelligence, 23(4) 2009.
TensorFlow. TensorFlow class http:://tww....oooooooo.o//////a
train and evaluaterccase weights
Chris Tofallis. Add or Multiply? Tutorial on Ranking and Choosing with Multiple Criteria.
INFORMS Trunsactions on Education, 14(3):109-119 2014.
Evangelos Triantaphyllou. Multi-criteria Decision Making Methods: A Comparative Study.
Springer, 2000.
Pelayo Vallina, Victor Pochat, Álvaro Feal, Marius Paraschiv, Julien Gamba, Tim Burke, Oliver Hohlfeld, Juan Tapiador, and Narseo Vallina- Rodriue-Z Mis-shapes, mistakes, misfits: An anal- ysis of domain classification services. pp. 598-618, 10 2020. doi: 10.1145/341939944423666
VirusTotal.
VirusTotal Documentation.
https: /suppert www.stotal..cmmmoa
en-us/
Cheng G. Weng and Josiah Poon. A New Evaluation Measure for Imbalanced Datasets. In Proceed- ings of the 7th Australasian Data Mining Conference (AusDM (00), pp. 27-32, 2008
Jieming Zhu, Shilin He, Jinyang Liu, Pinjia He, Qi Xie, Zibin Zheng, and Michael R Lyu. Tools and Benchmarks for Automated Log Parsing. In IEEE/ACM 41st International Conference Software Engineering: Software Engineering in Practice (ICSE-SEIP), pp. 121-130, 2019.
12

---

Under review as a conference paper at ICLR 2022
A APPENDIX
In this appendix, we provide details for the experimental study, including data and code. For further information, please see the supplementary material.
A.1 DETAILS FOR LOG PARSING EXPERIMENTS
For the three log parsing techniques used in Section 4.1 (Drain, Spell, and MoLFI), used the implementations provided by the LogPAI team:
https://github. htwp:/hhuw.ooooooopaa.a.aa.. hotp://oooooaaa///aaaaaee
The four datasets used in these experiments (macOS, BGL, Android, and HDFS) came from the benchmarking data also provided by LogPAI:
ht htcpss//g.tuub..c//oppa////oo
10-
10-3
10-3
10-1
Log message clasess
aa) macOS (skew 8.454)
10
10-3
bolatve
10-3
20-1
message clessss
(b) BGL (skew 8.900)
Android (skew 4, 4.822)
Ielzhe
HDFS (skew 0.202)
Figure 4: Histograms showing the relative frequencies of log parsing classes for the four experimen- tal datasets: All graphs have their y-axes in log scale; green bars show the infrequent classes.
In Figure 4 we show the histograms for the four log datasets together with their skew values. As defined in the Microsoft Excel Documentation, "Skewness characterizes the degree of asymmetry of a distribution around its mean. Positive skewness indicates distribution with an asymmetric tail extending toward more positive values, while negative skewness indicates distribution with
13

---

Under review as a conference paper at ICLR 2022
10
message classes
(a) Drain mis-classification count for macOS dataset
10P
APP
10
Log message clesses
(b) Spell mis miscclassiiication count for macOS dataset
10
Log message
(c) MoLFI mis-classification count for macOS dataset
Figure Misclassification count of three different algorithms on the macOS dataset. x-axis is ranked in descending order of class frequencies, as in Figure Drain performs best on rare classes (right side), which aligns with the WBAraritt ranking shown in Figure 2a
asymmetric tail extending loward more negative values." 0 In our context, skew provides a good indication for the degree of imbalance in class cardinality distributions the larger the skew, the larger the degree of class imbalance.
We also provide data files with class labels (true + predicted) and weights (based on rarity as importance criteria) used in generating the experimental data plotted in Figure 2 as part of our WBA-Evaluator tool implementation included in the supplementary material (can be found under directory).
the TBA-Evaluatos. wxwmples//ooParreeeeeeeeeee
visualization of per-class mis-classification count for each method on each dataset can be found from Figure55to Figure 7 The lower a bar is, the fewer samples being mis-classified in that category. Note that the categories on x axis are ordered ie.. in the same order of Figure 4 As result, the lower the bars the right side of each plot, the better the log parser in classifying infrequent classes for that dataset, and thus the better the WBA rarsttyyysoor should be. As explained by the legends in each figure, the per-class accuracy in each plot aligns with WBArarity metrics in Figure
descending order class frequencies,
14

---

Under review as a conference paper at ICLR 2022
message classes
(a) Drain mis-classification count BGL. dataset
Miscllassiiees 10P
Log message clesses
(b) Spell mis- classssifiicttoon count for BGL datasct
Log message
(c) MoLFI mis-classification count for BGL dataset
Figure 6: Misclassification count of three different algorithms on the BGL dataset. x-axis is ranked in descending order of class frequencies, as in Figure 4b MoLFI performs best on rare classes (right side), which aligns with the WBArrrstt ranking shown Figure 2b
A.2 DETAILS FOR SENTIMENT ANALYSIS EXPERIMENTS
For the sentiment analysis experiments of Section we used sample from the Amazon Customer Reviews dataset provided at:
hittps://nijiie http//oiiii.......mmmmmm.......m github..h.....g.g.. wwt.://aaaaammnnnnaa
html
In Figure 8
we show the histogram for the Amazon dataset. As described in Section 4.2 we im-
plemented RNN-based classifiers to experiment with this dataset. The code for these classifiers can be found in the supplementary material (under the 2ma.ooeeeiiiwaalasssifeeeeee directory) along with a copy of the data (under the Amaaoonlevicevcclasssieeerttttee directory).
We also provide the data files with class labels (true predicted) and weights (user) used in gener- ating the experimental data for LSTM results plotted in Figure 3 and Table can found in our WBA-Evaluator tool implementation included the supplementary material under the -va-Eva uator /ewp//xaalllleeeeeeeee es Amazon/ directory.
2 as example. These
15

---

Under review as a conference paper at ICLR 2022
Log message classes
(a) Drain mis-classification count for Android dataset
cis-ilassiiies
Log message clesses
(b) Spell mis-classification count for Android dataset
Log message
(c) MoLFI miscclassification count for Android dataset
Figure 7: Misclassification count of three different algorithms on the Android dataset. x-axis is ranked in descending order of class frequencies, as in Figure 46 Spell performs best on rare classes (right side), which aligns with the WBArrritt ranking shown in Figure 2c
0.6
Relative Frequency
0.5
0.3
0.2
0.0
Customer Rating
Figure 8: Histogram showing the relative frequencies of the five customer rating classes for the Amazon dataset (skew 2.04)).
A.3 DETAILS FOR URL CLASSIFICATION EXPERIMENTS
For the URL classification experiment in Section 4.3 and specifically for the experiment result in
Table 4 implementation is inherited from the URL work (Le et al. (018)). In particular,
16

---

Under review as a conference paper at ICLR 2022
Table 5: Training and evaluating URLNet model using WBA 2 classes
Dataset Statisties
Clasidication Accuracy
Calegury
Train AURLs
Rarily
Truin no WBA
Truin
#URLs
rarity
user on;
Benign
10N00
6762
0.011
0.2
0.994
0,902
0.950
Phishing
1000
675
0.909
0.8
0.859
0,942
0.861
Test with Accuracy
0.958
0.903
0.942
Test Balanced Accuracy
0.927
0.906
Test with WBAmatty
0.854
0,914
NA
Test with WBA
0.868
NA
0,879
Table 6: Training and evaluating URLNet model using WBA 3 classes
Dotaset Statistics
Classifiction Accuracy
Category
Train
Test #URLs
Karity
User
Train with
Train with
Train with
wi
Wi
rarity wy;
user on;
Benign
10000
6762
0.07
0.1
0.992
0.793
0.177
NSFW
2126
0.5
Phishing
1000
675
0.71
0.4
0.855
0,933
0.901
Test with Accuracy
0.762
0.652
0.374
Test with Balanced Accuracy
0.616
0.611
0.637
Test with WBA
0.673
0.738
NA
Test with WBA
0.441
NA
0.794
changed the URLNet source code support multi-class classification (previously binary), and to apply various weights for model training. The URLNet open source repository is available at:
https://github.com Ant imal www/ htt.:///oomaiiiaaaaeeee/eeNet
Our implementation and WBA-Evaluator hwwpaattorreigggtttttLLee..... directory.
modification can
be
found
at the
In the main context, we show how the proposed WBA weights are able to improve training accuracy for underrepresented classes, Due to page limit, only the evaluation results on 4 classes listed in Figure H Here we include the evaluation results for 2 classes and 3 classes respectively, to further express the effectiveness of WBA weights in improving model training. As shown Table[5] by ap- plying WBAvarity weights and user-defined weights which all focus more on phishing category, the phishing accuracy improved accordingly in both cases, with corresponding WBA metric improved well. In contrast, neither overall accuracy nor Balanced Accuracy is able to show the accuracy improvement the important class. Similarly, Table shows the experiment results for 3 classes. should be noted that vanilla training leads to O accuracy the newly added NSFW category, while another training with rarity weights brings some accuracy to this category, and applying a relatively high user weight for this category boosts its accuracy to over 80%.
A.4 THE
TOOL
In addition to details on our experimental study as described above, we also provide a copy of the WBA-Evallator tool that implements our customizable, blass-weighted evaluation framework de- scribed in Section 3 WBA-Evaluator is written in Python and can be found in the supplementary material along with README that describes how it can be used. In a nutshell, WBA-Evaluator takes as input three files (true class labels, predicted class labels, class weights) and number of con- figuration parameters in the form of commandline arguments, and then generates accuracy scores (BA WBA) specified by these arguments. The WBA-Evaluator implementation comes with two subdirectories: src/ contains the Python source code; exampl m e contains all the input files (labels and weights) and scripts in the scri prript///// subfolder run these. Please see the README. file for more details. Using this tool, results reported in the paper can be reproduced.
17


Reviews:
[{"review_id": "HmKinajGUst", "reviewer": "Reviewer_ircg", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "In this paper, the authors have proposed a novel evaluation framework for imbalanced data classification. Specifically, the proposed evaluation metric is designed to improve the grading of multi-class classifiers in domains where class importance is not evenly distributed. Generally speaking, the problem authors paid attention to is really existed and is important in imbalanced classification problem. Moreover, the writing in this paper is very well and is very easy to follow. However, the paper still exists the following problems:\n\nThe major problem of this paper is that the importance of each class is difficult to define. In this paper, the authors propose four methods in Section 3.2, i.e., importance criteria=user-defined, importance criteria=rarity, multiple importance criteria and partially-defined importance criteria. But in my opinion, none of them is very accurate and objective to define the importance of each class because in real-world applications, even if domain experts are difficult to distinguish the subtle importance differences between some classes. Anyway, it is an interesting and valuable problem for researches to discuss the importance differences between different classes. I think if the authors can provide some other importance criteria that are more objective, this paper will be more valuable.\n\nBesides, since this is a novel evaluation metrics, I think the authors should use a collection of datasets to fully verify the effectiveness of the proposed framework, instead of just three datasets in this paper. \n\nFinally, if the number of some classes are too rare, according to the rarity importance criteria, these classes will be given a very large weight, if so, whether the accuracy of majority classes will be largely decreased?\n", "summary_of_the_paper": "In this paper, the authors have proposed a novel evaluation framework for imbalanced data classification. Specifically, the proposed evaluation metric is designed to improve the grading of multi-class classifiers in domains where class importance is not evenly distributed. Generally speaking, the problem authors paid attention to is really existed and is important in imbalanced classification problem. Moreover, the writing in this paper is very well and is very easy to follow. ", "summary_of_the_review": "In this paper, the authors have proposed a novel evaluation framework for imbalanced data classification. Specifically, the proposed evaluation metric is designed to improve the grading of multi-class classifiers in domains where class importance is not evenly distributed. Generally speaking, the problem authors paid attention to is really existed and is important in imbalanced classification problem. Moreover, the writing in this paper is very well and is very easy to follow. However, the paper still exists the following problems:\n\nThe major problem of this paper is that the importance of each class is difficult to define. In this paper, the authors propose four methods in Section 3.2, i.e., importance criteria=user-defined, importance criteria=rarity, multiple importance criteria and partially-defined importance criteria. But in my opinion, none of them is very accurate and objective to define the importance of each class because in real-world applications, even if domain experts are difficult to distinguish the subtle importance differences between some classes. Anyway, it is an interesting and valuable problem for researches to discuss the importance differences between different classes. I think if the authors can provide some other importance criteria that are more objective, this paper will be more valuable.\n\nBesides, since this is a novel evaluation metrics, I think the authors should use a collection of datasets to fully verify the effectiveness of the proposed framework, instead of just three datasets in this paper. \n\nFinally, if the number of some classes are too rare, according to the rarity importance criteria, these classes will be given a very large weight, if so, whether the accuracy of majority classes will be largely decreased?", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "H-xdzfKameH", "reviewer": "Reviewer_xLmW", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper addressed a known problem by introducing weights to the rare class and explaining a new theoretical finding to better grade the models applied.\n\nThe 2 main goals of simplicity and good generality of models were addressed. The study proposed a new grading metric to an old problem\n\nThe research is reproducible and presents simple novel findings\n\nThe Weak points are trying to incorporate too many things in the model like user-defined, rarity, multiple, and partially-defined importance.\n\n\n", "summary_of_the_paper": "The paper presented a simple yet general-purpose class-sensitive evaluation framework for imbalanced data classification. Their framework is designed to improve the grading of multi-class classifiers in domains where class importance is not evenly distributed. They provided a modular and extensible formulation that can be easily customized to different important criteria and metrics. Experiments with three real-world use cases show the value of a metric based on our framework, Weighted Balanced Accuracy (WBA), over existing metrics – in not only evaluating the classifiers’ test results more sensitively to important criteria but also training them so.", "summary_of_the_review": "Overall, I vote for acceptance as the class imbalance problem is chaotic. It gives classification suboptimal results. Especially when allied with class disjunct\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "CyQ2aMS0S_G", "reviewer": "Reviewer_QJaY", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "First of all, I agree that class-stratified weighting can be effective for evaluation and training. However, from a research perspective, this paper has very limited novelty. With respect to class-stratified evaluation (or combining them into a single score), I have seen hundreds of reports that already do this. From a training perspective, this is taught in just about every undergraduate machine learning course (both in training in evaluation). From a training perspective, as the authors point out, this is available in many toolkits (e.g., scikit-learn, SVMlight, TensorFlow, pyTorch), so people are obviously doing this.\n\nThus, the real question is how to set these weights, which generally falls under the purview of decision theory. There are multiple textbook chapters (and entire textbooks) that address this issues. Given the interest in class imbalance, I would recommend something in the medical decision making community where preference elicitation is widely-studied (e.g., standard gamble). A paper that particularly influenced my thinking is in this space is [A. J. Vickers and E. B. Elkin. Decision curve analysis: A novel method for evaluating prediction models. Medical Decision Making, 26:565–574, 2006.], [but there is ample literature in this space.\n\nSecondly, averaging together a set of point estimates to derive a scalar doesn’t necessarily have well-behaved statisical properties as it depends on what is being averaged together. Specifically, there are cases (e.g., sensitivity and specificity) that are frequently modeled as bivariate normal when aggregating measurements. Without such considerations, it isn’t clear how statistical significance would be evaluated. A paper that influenced my thinking here wrt rigor and scalar values for class imbalance settings is [Jurman, Riccadonna, and Furlanello, A Comparison of MCC and CEN Error Measures in Multi-class Prediction, PLoS One, 7(8): 2012]. Basically, without knowing the distribution properties of the average, it isn’t obvious how to perform hypothesis testing.\n\nFinally, I am not entirely convinced that the motivation regarding a scalar evaluation for multi-class under severe class skew is even that desirable. While subjective, I believe there are potentially good visualization that can be used to elicit class-specific utility — which, as the authors state, are very sensitive for some rare events (e.g., medical diagnosis, (hopefully fictional) doomsday devices). \n\nThis all being said (that this isn’t a suitable paper for ICLR), the writing itself is a bit informal, but easy to understand and some of the observations on the datasets are interesting. However, there isn’t sufficient novelty, potential for impact, or mathematical rigor to accept this paper in a top-tier machine learning conference.", "summary_of_the_paper": "The authors advocate for class-stratified weighted macro-averaging as an appropriate scalar-valued evaluation for classification under class-imbalance. In the absence of domain expert specified weights, they also propose a weighting function that emphasizes rare classes (referred to as rarity), and a multi-importance criteria based on a normalized product of weightings. Furthermore, they point out that these weightings can be used for training via existing methods in commonly used tools (e.g., class-based instance weighting, class-based loss scaling). Finally, they show performance under different weighting for three log message classification tasks, a sentiment classification task, and a URL classification task (e.g., malware, NSFW, phishing),  — demonstrating that different weighings lead to different orderings of evaluation results and that these weights can be effectively used in training.", "summary_of_the_review": "The authors advocate for class-stratified weighted macro-averaging as an appropriate scalar-valued evaluation for classification under class-imbalance — justified by a reasonable number of case studies. However, they don’t describe a procedure for setting the weights in a decision theoretic way nor describe the statistical properties of the resulting point estimate for hypothesis testing. Thus, while a reasonable heuristic approach, it lacks rigor for use in critical systems (where high-loss rare events are important). Additionally, there is very limited novelty as this is performed both for training systems and evaluating them in general ML research, application-focused ML research, and industry practice. This paper has some interesting observations, but is not ready for ICLR acceptance. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "0H-wQ5-BjIo", "reviewer": "Reviewer_WMHg", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "A simple and intuitive model has been proposed that modifies the global quality metric by adding class significance weights. However, a problem that, in my opinion, is still unresolved is how to determine the importance of the classes. As a rule, as the authors point out, it should be a user's decision, but as is well known, we have a problem with obtaining this information. Note that the classic probabilistic model of the classifier is based on the minimization of loss function, which leads to the minimization of the overall risk (expected value of the loss function), i.e., one can say that the problem is solved and the proposal contained in the paper does not bring anything new. Unfortunately, we usually do not know the form of the loss function because this is information that an expert should provide. Some methods try to gain the knowledge of the misclassification cost, such as the work on Utility-based Performance Measures (https://www.dcc.fc.up.pt/~ltorgo/Papers/RT08_WEMML.pdf) or methods that try to obtain information about the cost of error using linguistic variables (such as fuzzy loss function - https://link.springer.com/content/pdf/10.1007%2F978-3-540-68168-7_53.pdf). Unfortunately, neither of these approaches fully addresses this problem.\nI find interesting comments of the authors contained in sec. 3.2, where it is proposed to calculate the weights according to factors other than those obtained directly from users.\n\n", "summary_of_the_paper": "The article focuses on a significant problem related to the quality assessment of unbalanced data classifiers, or rather a problem where one deals with a problem where we may observe different misclassification cost for each class. I fully agree that the known and commonly used metrics do not meet expectations, hence their diversity, the analysis presented in the works cited by the author. ", "summary_of_the_review": "Recognizing that the problem is interesting, however, taking into account the above, I found that the contribution of this work is insufficient for this conference.\n\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: Although two reviewers have given score 6, the other reviews have clearly indicated that the paper is not good enough for ICLR. The paper does not give any new insight into the considered problem, many existing papers are ignored, and the only interesting part about evaluation of label importance is rather very shallow borrowing ideas from other fields without any deep discussion or analysis.
Decision: Reject