Class-Weighted Evaluation Metrics for Imbalanced Data Classification
Authors: Akhilesh Gupta
Nesime Tatbul
Ryan Marcus
Shengtian Zhou
Insup Lee
Justin Gottschlich
Venue:  (2021)
Keywords: Imbalanced data classification
Evaluation metrics
Log parsing
Sentiment analysis

Abstract: Class distribution skews in imbalanced datasets may lead to models with prediction bias towards majority classes, making fair assessment of classifiers a challenging task. Balanced Accuracy is a popular metric used to evaluate a classifier’s prediction performance under such scenarios. However, this metric falls short when classes vary in importance, especially when class importance is skewed differently from class cardinality distributions. In this paper, we propose a simple and general-purpose evaluation framework for imbalanced data classification that is sensitive to arbitrary skews in class cardinalities and importances. Experiments with several state-of-the-art classifiers tested on real-world datasets and benchmarks from two different domains show that our new framework is more effective than Balanced Accuracy –- not only in evaluating and ranking model predictions, but also in training the models themselves.

Under review as a conference paper at ICLR 2021
EVALUATION METRICS FOR IMBALANCED DATA CLASSIFICATION
Anonymous authors Paper under double-blind review
ABSTRACT
Class distribution skews in imbalanced datasets may lead models with predic- tion bias towards majority classes, making fair assessment of classifiers chal- lenging task. Balanced Accuracy is a popular metric used to evaluate a classifier's prediction performance under such scenarios. However, this metric falls short when classes vary in importance, especially when class importance is skewed dif- ferently from class cardinality distributions. In this paper, we propose simple and general-purpose evaluation framework for imbalanced data classification that is sensitive to arbitrary skews in class cardinalities and importances. Experiments with several state-of-the-an classifiers tested on real-world datasets and bench- marks from two different domains show that our new framework is more effective than Balanced Accuracy not only in evaluating and ranking model predictions, but also in training the models themselves.
INTRODUCTION
For broad range of machine learning (ML) tasks, predictive modeling in the presence of imbal- anced datasets those with severe distribution skews has been a long-standing problem (He & Garcia 2009 [Sun al.) 2009 He & Ma 2013 |Baanco et al., [016); Hilario al. 2018 Johnson & Khoshgofttarr 2019). Imbalanced training datasets lead to models with prediction bias towards majority classes, which in turn results in misclassification of the underrepresented ones. Yet, those minority classes often are the ones that correspond to the most important events of interest (c.g., errors in system logs (Zhu al.]|2200 2019). infected patients in medical diagnosis (Cohen et al. 2006), fraud in financial transactions (Makki et al.] 2019)). While there is often an inverse correlation be- tween the class cardinalities and their importance (i.e., rare classes are more important than others), the core problem here is the mismatch between the way these two distributions are skewed: the zth most common class is not necessarily the ith most important class (see Figure Ia for an illustration). In fact, rarity is one of many potential criteria that can determine the importance of class, which is usually positively correlated with the costs or risks involved in its misprediction. Ignoring these criteria when dealing with imbalanced data classification may have detrimental consequences.
Consider automatic classification of messages in system event logs as an example (Zhu et al., 2019)) An event log is temporal sequence of event messages that have transpired for given software system (e.g., operating systems, cyber-physical systems) over a certain period of time. Event logs are particularly useful after a system has been deployed. These logs can provide the DevOps teams with information and insights about errors outside of the testing environment, thereby improving their ability debug and improve the quality of the system. There is typically an inverse correlation between the stability/maturity of a system and the frequency of the errors it produces in its event log. Furthermore, the message types that appear least frequently in an event log are usually the ones with the greatest importance. A concrete example of this was rare anomaly in Uber's self-driving car that led to the death of pedestrian, since the system flagged it as a false positive (Efrati) 2018). If this event had not been misclassified and dismissed by the system, the pedestrian death in Arizona may have been avoided.
A plethora of approaches have been proposed for building balanced classifiers (e.g., resampling to balance datasets, imbalanced learning methods, prediction post-processiee (Sun et al. 2009 Branco et al. 2016)). A fundamental issue that still remains open challenge is the lack of generally- accepted methodology for measuring classification performance. The traditional metrics, which are designed to evaluate average case performance, (e.g., Accuracy) not capable of correctly assess-

---

Under review as a conference paper at ICLR 2021
Predicted labels
A
B
C
Class
Cardinality
Importance
Empgrtaant classes are not that rure
Rare clusaes cue be more inportant
Tree labels
4
Class Insportance uniferm
A
10
0.2
8
4
0.7
C
70
0.1
5
5
60
No imbalance mismatch issue
Raro clasaes are not that important Balanced Accuracy
Accuracy - Cornect predictions All prodictions
No consideration of Cardinality Importance!
Accuracy
-(1+4-60) 100 0.65
uniform
uniforn
Balanced Accuracy (Azcuracy(A) + Accuracy(t) + Accuracy(()) '3
No conrideraiion of Importance!
Class Cardinality
-1/10 + 4/20 60/70) -
(a) Skew in distributions
(b) Example
Figure 1: Skew in distributions of Class Cardinalities or Class Importance, and the potential mis- match between these two distributions render Accuracy and Balanced Accuracy metrics unusable in general multi-class prediction problems.
ing the results in presence of arbitrary skew mismatches between class cardinalities and importances. On the other hand, metrics specifically proposed for imbalanced learning are either domain-speddt do not easily generalize beyond two classes, or can not support varying class importance (e.g., Bal- anced Accuracy) (Japkowicz 2013).
Let us illustrate the problem with the simple example depicted in Figure Tb The dataset consists of 100 data items from 3 classes (A, B, C). The greatest majority of the items belong to class C (70/100), but class B (20/100) has the greatest importance (0.7/1.0). other words, Cardinality and Importance are both non-uniform and in favor of different classes (i.e., falls in the top-right quadrant of Figure Tab. The confusion matrix on the right-hand side shows the results from classifier that was run against this test dataset. Unsurprisingly, the classifier performed the best in labeling the majority class C (60/70 correct predictions). When this result is evaluated using the traditional Accuracy metric, neither Class Cardinality Class Importance is taken into account. If Balanced Accuracy is used instead, we observe the degrading impact of the skew in Class Cardinality (0.38 < 0.65), but Class Importance is still not accounted for. This example demonstrates the need for new evaluation approach that is both sensitive to Cardinality and Importance skew, as well as any arbitrary correlations between them. This is especially critical for ensuring fair assessment of results, when comparing across multiple classifiers or problem instances.
Our goal this paper is to design an evaluation framework for imbalanced data classification, which can be reliably used to measure, compare, train, and tune classifier performance in a way that is sen- sitive to non-uniform class importance. We identify two key design principles for such a framework:
Simplicity: It should be intuitive and easy to use and interpret. Generality: should be ceneral-purpose, i.e., (i) extensible to an arbitrary number of classes and (ii) customizable to any application domain.
To meet the first design goal, we focus on scalar metrics such as Accuracy (as opposed to graph- ical metrics such as ROC curves), they are simpler, more commonly used, and scale well with increasing numbers classes and models. To meet the second design goal, we target the more general π--ay classification problems (as opposed to binary), well as providing the capability to flexibly adjust class weights to capture non-uniform importance criteria that may vary across appli- cation domains. Note that primarily focus on Accuracy our base scalar metric in this paper, as it is seen as the de facto metric for classification problems (Sci). However, our framework is general enough to be extended other scalar metrics, such Precísion and Recall. Similarly, while deeply examine two applications (log parsing and sentiment analysis) in this work, our framework in principle is generally applicable any domain with imbalanced class and importance distributions.
In the rest of this paper, we first provide a brief overview of related work in Section 2 Section 3 presents our new, class-weighted evaluation framework. In Section we show the practical utility of our framework by applying it over three log parsing systems (Drain (He et al. 2017). MoLFI (Mes- saoudi et al. 2018). Spell (Du & Li 2016: 2018)) using four real-world benchmarks (Zuu et al. 2019), as well as over variety of deep learning models developed for sentiment analysis on cus- tomer reviews dataset from Amazon (Ni et al. 2019). Finally, we conclude the paper with brief discussion of future directions.
2

---

Under review as a conference paper at ICLR 2021
2 RELATED WORK
Imbalanced Data Classification, Imbalanced data is prevalent in almost every domain (Cohen
al. 2006 Batuwita & Palade 2012, Makki al., 2019). The growing adoption of ML models in diverse application domains has led to surge in imbalanced data classification research (He &
Gareia 2009 Sun et al.] 2009 He & Ma 2013 [Branco et al.) 2016 Hilario et al. 2018 Johnson & Khhosggfttaar 2019). While the proposed techniques widely vary, they fall under four basic cat- egories: pre- processing training data to establish balance via sampling-bau baasdd techniques (c.g., (Es- labrooks et al, 2004 Blaszczynski & Stefanowski 2015)). building custom learning techniques for imbalanced training data (e.g., (Joshi et al. Castroo & de Pádua Braga 2013)), post-processing predictions from imbalanced model (e.g., (Maloot 2003)), and their hybrids (e.g.. (Estabrooks & Japkowicz| 2001). In this paper, we do not propose new imbalanced learning technique, but performance evaluation framework that could be used in the training and/or testing of models for any technique. Section 4 demonstrates the practical utility of our framework for a variety of ML models from two different application domains.
rather general-purpont
Evaluation Metrics. Traditional metrics for evaluating prediction performance such as Accuracy, Sensitivity/Specifeitt (and their combination G-mean), Precision/Recall (and their combination F- Score) were not designed with imbalanced data issues in mind (Japkowicz) (013). Further, most of these were originally intended for binary classification problems. To extend them to multi-class problems, macro-averaging (i.e., arithmetic mean over individual measurements of each class) is used. Macro-averaging treats each class equally (Branco et al.) 2016). Balanced Accuracy is also one such popular averaging- based approach. There are also probabilistic evaluation approaches that extend Balanced Accuracy with Bayesian inference techniques for both binary and multi-class problems (Brodersen al., 2010 Carrillo et al.) (014)). Closer our work, Cohen al. (2006) in- troduced the notion of class weights, yet in the specific context of Sensitivity/Speccficity for binary classification in the medical domain. Similarly, [Batwwita & Palade (2012) proposed extensions to G-mean for the bio-informatics domain. In addition to these scalar (a.k.a., threshold) metrics, graph- ical (a.k.a,, ranking) evaluation methods such as Receiver Operating Characteristic (ROC) curves or Precision-Recall (PR) curves (and the Area Under the Curve (AUC) for such curves) as well as their extensions to imbalanced data / multi-class problems were also investigated (Weng Poon 2008] Japkowic7 (013). While these methods provide more detailed insights into the operational space of classifiers as a whole, they do not casily scale well with use in problems with a large number of classes (Branco al. 2016).
3 CLASS- WEIGHTEDD EVALUATION FRAMEWORK
In this section, we present our new evaluation framework designed to measure model accuracy for multi-class classification problems in presence of arbitrary skews among class distributions and/or importances. Our framework builds on and extends commonly used scalar / threshold metrics such as Accuracy. These metrics were originally designed for binary classification problems, where there is typically more emphasis on one class (the positive class, e.g., anomalies). To adopt them to multi-class problems where there is no such single-class emphasis, each class' metric can be computed separately and then overall aggregation (i.e., arithmetic mean) be performed. For example, Accuracy has been extended to Balanced by following this approuch. In our framework, we follow similar aggregation strategy, however, do it in more generalized way that allows custom class weights to capture class importance. Furthermore, these class weights can be based on any importance criteria such as rarity, cost, risk, expected benefits, and possibly hybrid of multiple such criteria. Therefore, it is critical to provide flexible formulation that allows users or domain experts to adjust the weights as needed by their problem instance. In what follows, we present our new class- weighte evaluation framework in a top-down fashion. Using the basic notation summarized in Table [ we first formulate the general framework, and then we describe how this framework can be customized to different importance criteria scenarios by specializing the weights in a principled manner. For ease of exposition, we first focus on Accuracy as the underlying performance metric, and then we discuss how our approach can be adopted to other similar metrics.
3.1 WEIGHTED BALANCED ACCURACY (WBA)
Suppose we are given a test dataset with N data items in it, each of which belongs to one of C distinct classes. Furthermore, each class contains n, of the data items in this dataset. Thus:
3

---

Under review as a conference paper at ICLR 2021
Table 1: Notation
Notation
Description
N
total number of data items
C
total number of data item classes
number of importance criteria
true number of data items in class
Pi
correctly predicted number of data items in class
fi
relative frequency of class i
relative weight of class i
relative user-defined importance class
71
relative rarity of class
relative weight of class for importance criteria j
Accuracy;
Accuracy of class
N
n
(I)
1=l
The relative frequency of each class i in the whole dataset is:
Assume a classifier that makes a prediction about the class label of each data item in the test dataset, and manages correctly predict pt out of n, labels for given class i, where pt S n,, Then, the total number of correct predictions out of all the predictions gives us the overall Accuracy of the classifier as follows:
Li=1 P
Accuracy
N
The classifier's Accuracy, for a given class i (a.k.a,, per-class Recall score) can computed as:
Accuracy, Pi Y2
(4)
BaalancedAAccraaaa is the macro-average of these per-class Accuracy measurements over all classes in the dataset:
BalancedAccuracyy C
Accuracy,
The formulation far represents the state of the art in how prediction accuracy is evalu- ated for results of multi-class classifiers in presence of imbalanced datasets (i.e., those where fi are not even). Note that, for balanced datasets (i.e.. Vi,n, n, N/C and fi 1/C), ensures
= Accuracy; whereas for imbalanced datasets, that the prediction accuracy is not inflated due to high-frequency classes' results dominating over the others'. Balanced Accuracy works well as long each class is of the same importance, since it the simple arithmetic mean across per-class accuracy measurements of all classes (i.c., each class' accuracy contributes evenly to the overall accuracy). As we discussed with examples in previous sec- tions, in many real-world classification problems, this assumption does not hold. Rather, classifiers must be rewarded higher scores for their prediction performance more important classes. In order to capture this requirement, we generalize Balanced into W eighled Accuurrrrrr by extending it with per-class importance weights w; follows:
Balanced
WeiihteeDDolaneedAeerrryyy Σ wi x Accuracy,
(6)
The extension above enables us to capture both skews imbalances in class cardinalities as well as importances (i.e., the complete design space in Figure Ta This is general formulation that can
4

---

Under review as a conference paper at ICLR 2021
support any importance criteria for weights as long as 0 S wy S and Σ W; 1. In the next section, we present how weights can be customized for different types of importance criteria.
3.2 WEIGHT CUSTOMIZATION
In a multi-class problem, not only may the classes carry different importance weights, but also the criteria of importance may vary from one problem domain another. In this section, discuss several types of criteria that we think are commonly seen across many application domains. Note that this not meant be an exhaustive list, but it provides examples as well as templates that can be easily tailored to different problems.
Importance criteria User-defined. This is the most general and flexible form of importance cri- teria. The application designer or domain expert specifies the relative weight of each class based on some application-speaeeeepppppppppppp sppcciii criteria. As an example, the problem might be about classifying images of different types of objects in highway traffic and the user gives higher importance to correct recog- nition of certain objects of interest (e-g., pedestrians, bikes, animals, etc). We express user-defined relative weight of class i with us which is simply used as w; in Equation 6
Wi
Importance criteria = Rarity. It is often the case that the rarer something is, the more noteworthy or valuable it is. In multi-class problems, this corresponds to the case when importance of class i is inversely correlated with its relative frequency of occurrence (fi) in the dataset. For example, in system log monitoring, log messages for more rarely occurring errors or exceptions (e.g., denial of service attack) are typically of higher importance. Therefore, a classifier that performs well on detecting such messages must be rewarded accordingly. In our framework, we capture rarity using weights that are based on normalized inverse class frequencies formulated as follows:
UN =
(8)
1= 5
Multiple importance criteria. some classification problems, importance of a class depends on multiple different criteria (e.g., both rarity and user-defined criteria). To express class weights in such scenarios, we can leverage techniques from multi-criteria decision making and multi-objective optimization (Triantaphyllou 2000; Helff al., (016). One of the most basic methods is using normalized weighted sums based on composite weights (Helff et al. (016)). Composite weights can be computed either in additive or multiplicative form (Tofallis) 2014). The multiplicative approach tends to promote weight combinations that are uniformly higher across all criteria, and as such is found to be more preferred approach in application scenarios similar to ours (Helff et al. 2016] Tofallis 2014). While we present this approach here, in principle, other approaches from multi- criteria decision making theory could also be used together with our framework. Given M different criteria with ma,j denoting the relative weight of class for criteria j, we can compute the composite weight of a class as follows:
may
Wi:
ma.j
For example, if we had two criteria, rarity and user-defined u with weights r, and u, for each class i, respectively, then the composite weight for class i would be wi Pixu.
1 Xyx
Partially-defined importance criteria. One commonly expected scenario (especially in those clas- sification problems where the number of classes C' can be very large) that not all of the class importance weights might be supplied by the user. For example, in sentiment analysis use case, the user supplies the weights for all the negative classes, and leaves the others unspecified. Our framework can support such cases by automatically assigning weights to the unspecified classes. The default approach is to distribute the remaining portion of weights evenly across all unspecified classes: (1 total weights specified) (number of unspecified classes). If the user prefers altemna- tive approach (c.g., distribute the remainder based on rarity of the unspecified classes), this can also be easily supported by our framework.
5

---

Under review as a conference paper at ICLR 2021
3.3 METRIC CUSTOMIZATION
The weighte evaluation framework presented above focused on the popular Accuracy metric as the underlying metric of prediction performance. However, our framework follows a general structure based on the idea of weighted macro-averaging with customizable weights, which can essentially used with any performance metric that can be computed on the basis of a class, For example, the macro-averaging approaches that are already being used for Precision, Recall, and their combination F-Score could easily be extended with our customizable weighing approach by replacing Accuracy in our formulas with one of these metries.
4 EXPERIMENTAL ANALYSIS
In this section, we present an experimental analysis of the WBA metric for two application domains. Our primary goal to demonstrate the value of WBA compared to other standard metrics when evaluating ML models over real-world imbalanced data classification problems. As we will show, often times traditional metric like Accuracy or BalancedAccuraac will make classifier A seem preferable to classifier B, when in reality classifier B is superior. In addition, we also provide a brief analysis of how WBA can positively impact, not only the testing of models, but also their training. Further details about this experimental study (including code, data, and examples) can be found in the supplementary material and in Appendix
4.1 USE CASE 1: LEARNED LOG PARSING
ML-based log parsers are tools that are designed to automatically learn the structure of event logs generated by hardware and software systems to properly categorize them into event classes (c.g., different error types). In our first study, we used WBA to evaluate 3 state-of-the -het---rt log systems: Drain, Spell, and MoLFI (Du & Li 2016 He al. 2017: Messaoudi al.) 2018). We start by providing an abbreviated description of our experimental setup.
Log Parsing Systems. Drain is a rule-based, online log parsing system that encodes the pars- ing rules in parse tree with fixed depth (He al., 2017). It performs pre-processing step for each new log message using regular expressions created by experts with domain knowledge. Spell, like Drain, is also baledb it principally uses the longest common subsequence (LCS) to find new classes of log messages (Du & Lij 2016). It parses messages in an online fashion by creating objects for each message type containing information about LCS. Finally, Mol.Fl casts the log parsing multi-objective optimization problem and provides solution based on genetic programming (Mes- saoudi et al.) 2018).
Datasets. We test each aforementioned log message classification system with four real-world datasets taken from a public benchmark (Zhu et al. 2019). Each dataset has 2000 log instances randomly sampled from a larger dataset. The macOS dataset contains raw log data generated by the macOS operating system (341 log classes, 237 infrequent classes (i.e., those that have fewer occur- rences in the dataset than the average number of messages per class), and an average class frequency of 5). The BlueGene/L (BGL) dataset is a collection of logs from the BlueGene/L supercomputer system (120 log classes, 101 infrequent classes, and an average class frequency of 16). The An- droid dataset consists of logs from the Android mobile operating system (Zhu et al. 2019) (166 log classes, 127 infrequent classes, and average class frequency of 16). Finally, the HDFS dataset consists of log data collected from the Hadoop Distributed File System (14 log classes, 8 infrequent classes, and an average class frequency of 142). Overall, the first three datasets are highly skewed in class frequencies, whereas the HDFS dataset is relatively much less skewed (see Appendix Results. For Drain, Spell, and MoLFI, traditional metrics of Precision. Recall, FI-Score, and Accuracy (named Parsing Accuracy in the original papers) were used for training and testing clas- sification performance. None of these metrics are class-sensitive, while in log parsing, messages have in fact varying importance across the classes. The importance criteria is rarity: the more rare an error message is, the more important it is to correctly classify this message. To capture this, we configure the WBA to WBArrrity, which automatically assigns weights to WBA based on the dataset classes' inverse frequencies, as described in Section 3.2 Then we evaluate the test results from the 3 parsers over 4 datasets using WBArrrrtt and compare against traditional metrics in two categories: class- classinssessitive and Plass-sensitive as shown Figure
WBA Warnty vs. Class-insensitive Metrics: The class-insensitive metries (specifically, F1 F1-Score and

---

Under review as a conference paper at ICLR 2021
Cha--maashbe Netris
Chur-ennthe Netiis
0.4
Pecidio
136Ga'
Acautay
Babresic
WEATTMM
Pn..8
V80 (a(0))
Drih CSeell
MaiFi
Drain Lifeel BINAR
(a) macOS (skew 8.454)
(b) BGL (skew 8,900)
Caa iww.ratnaaavvve Metria
Cassoenation Merica
Merks
8.7
0.
0.2
Acsria
Drihn (35pel MAAA
B Drain Lifeel BINAR
(c) Android (skew 4.822)
(d) HDFS (skew 0.202)
Figure 2: WBA vs. class-insensitive & class-sensitive metrics for log parsing: F1-Score & Accuracy agree in all. BA& & WBA agree in (c) & (d) only. WBA disagrees with class-insensitive in all.
Accuracy) agree on how to rank the classification performance of the 3 parsers across all the datasets (for macOS and Android, Drain > Spell > MoLFI; for BGL, Drain MoLFI > Spell; for HDFS, all perform similarly). Since WBArarity is sensitive to classes' data distribution and importance skews, it makes completely different judgement. Furthermore, it ranks the techniques differently for each dataset (Drain >> MoLFI > Spell in macOS; Drain > Spell > MoLFI for BGL; Spell > Drain > MoLFI for Android; and for HDFS, Spell Drain and MoL.FI. This result validates that WBA provides more sensitive tool for assessing classification performance.
WBArerstt vs. Balanced Accuracy (BA): As discussed carlier, BA is class-sensitive but only to dis- tribution imbalance. We can observe the difference between BA and WBA in Figure 2 In macOS and BGL, where the skew is the highest and rarity is more pronounced, the two metries completely disagree in how they rank the parsers. In contrast, for Android and HDFS, where the skew is lower, there is overall agreement, although the separation in metric values slightly differ. Of particular importance is the difference seen in Figure 2a We observe that the best performing model is Spell when scored by BA, and Drain when scored by WBA The reason for this difference due to Spell's and Drain's is differences in their ability to correctly classify infrequent classes, i.e., those that represent failures and errors that require the most immediate response.
4.2 USE CASE 2: SENTIMENT ANALYSIS
In social media and other uscinfacing domains like e-commerce sites, it is often useful understand the view or feelings ("sentiments") associated with users' behavior or preferences. In the second part of our experimental study, we apply WBA in the context of such a sentiment analysis use case, which involves analyzing text-based product reviews from Amazon's e-commerce websites.
Dataset. The dataset consists of customer's reviews and ratings, which we got from Consumer Reviews of Amazon products (Ni et al. 2019). The task is to classify the reviews into 5 classes (with 1 being the lowest and being the highest rating a product can get in a review), where ratings constitute the ground truth class labels. There is high class imbalance in this dataset (skew=2.140) As shown in the Frequency column of Table 2 Class with the highest customer rating clearly dominates compared to the other classes. It is known that the distribution of customer review ratings is typically imbalanced and generally follow a J-shaped distribution (Mudambi & Schuff 2010 Pavlou & Dimoka 2006).
Sentiment Analysis Models. We compare 4 types of recurrent neural networks (RNN), all con- sisting of embedding layer with pre-trained word embeddings from (Pennington al. 2014) followed by a recurrent layer from Py PyTorc (Subramanian, 2018): RNN, LSTM, GRU, BiLSTM. The hidden state output from the last time step of these are passed fully-connected layer with
7

---

Under review as a conference paper at ICLR 2021
Table 2: Amazon per-class breakdown: Frequencies are highly skewed (skew=2.140); Accuracy; in each model when both trained+tested with user-defined weights w; (same weights as Figure 3a
Class
Frequency ((+)
Weights (wr.
LSTM
RNN
GRU
BILSTM
0.092
0.19
0.04
0.17
0.052
0
0
0
3
0.075
0
0
0
4
0.142
0
0
0
0.639
0.3
0.81
0.96
0.84
0.83
VI www.hee--oo Saanore
LIPCO 23
rants
Importance Officia
BIST
B5TT Tran DIAN BRANTRANTOU
(a) WBA vs. Other Metrics (Train+Test)
(b) WBA: Test Train+Test
Figure 3: Amazon results
input of 256 neurons and output from 5 neurons.
Results. For this use case, we first worked with a user-defined importance criteria borrowed from published studies suggesting that extreme review ratings (classes and 5) carry more importance Mudambi & Schuff, 2010: Paylou & Dimoka 2006). Thus, we set the weights as shown in Table 2 (shown as WBA(user) or user in Figure
WBA vs. Other Accuracy Metrics: First, we compare WBA(user) with Accuracy and BalancedAc- curacy (BA) when used as a metric for both training and testing of the DNN models (Figure (aa.. We make a few observations: (i) The class- -ass-insensittve Accuracy showcases the imbalance problem in classification, as it favors the RNN model which is heavily biased by the majority class (see Accuracy, for RNN in Table 2 where class 5 scores 0.96). (ii) The frequency-sensie ric finds all models perform similarly. WBA(user), in contrast, identifies LSTM as the best model. Indeed, Table 2 confirms that LSTM performs relatively the best in predicting the most important class, class 1 (0.19 accuracy). Overall, we find that WBA is capable of capturing importance skews, even when the frequency skew can be high and biased towards less important classes.
BA met-
Impact of WBA in Model Training: Next we explore the use of WBA not only in model evaluation, but also training. We focus on two models (LSTM and RNN), and apply WBA only during test- ing vs. to both training (by modifying loss functions of DNNs to capture class importance weights) and testing. Intuitively, if a model is trained being aware of the importance weights, then it should also perform well when tested against the same criteria. To test this hypothesis, experiment for 3 alternative importance criteria: (i) rarity (w) = 0.209, w2 0.368, w3 = 0.255, Wy 0.136, W3 0.030), (ii) user-defined (i.e., with weights in Table 2 and (iii) composite of the two (wi = 0.62, W2 = Wi W14 = 0, W3 = 0.38). In Figure 3b we observe: (i) Except for rarity. WBA for both LSTM and RNN improves when integrated into model training. This verifies our intuition, and shows that WBA is a useful metric not only for evaluation, but also for training. (ii) When zoom into rarity, we that although class 2 is the most important, per-class accuracy for class 5 is much higher for both LSTM and RNN in the Test-only case, because both models are still trained heavily biased towards the majority class (5). (ii) Though rarity by itself training, when combined with user importance, it visibly improves the WBA scores, This shows that our
repeated the
useful in
composition approach is capable of combining importance criteria as intended.
5 CONCLUSION
In this paper, we presented simple yet general-purpose class-sensitive evaluation framework for imbalanced data classification. Our framework designed to improve the grading of multi-class classifiers in domains where class importance is not evenly distributed. We provided modular and extensible formulation that can be easily customized to different importance criteria and metrics. Experiments with two real-world Weighted Balanced Accuracy (WBA), over existing metrics in not only evaluating the classifiers' test results more sensitively to importance criteria, but also training them so.
cases show the value of metric based
framework,
8

---

Under review as a conference paper at ICLR 2021
REFERENCES
Scikit-learn Classification Metries.
model
Rukshan Batuwita and Vasile Palade. Adjusted Geometric-mean: A Novel Performance Measure for Imbalanced Bioinformatics Datasets Learning. Journal of Bioinformatics and Computational Biology, 10(4), 2012.
Jerzy Blaszczynski and Jerzy Stefanowski. Neighbourhood Sampling in Bagging for Imbalanced Data. Nenrocomputiigg 150:529-542, 2015.
Paula Branco, Luís Torgo, and Rita Ribeiro. A Survey of Predictive Modeling on Imbalanced Domains. ACM Computing Surveys (CSUR), 49(2):1-50, 2016.
Kay Henning Brodersen, Cheng Soon Ong, Klaas Enno Stephan, and Joachim M Buhmann. The Balanced Accuracy and Its Posterior Distribution. In 20th International Conference on Pattern Recognition (ICPR), pp. 3121-3124, 2010.
Henry Carrillo, Kay H Brodersen, and José A Castellanos. Probabilistic Performance Evaluation for Multi-class Classification using the Posterior Balanced Accuracy. In First Iberian Robotics Conference, pp. 347-361, 2014.
Cristiano Leite Castro and António de Pádua Braga. Novel Cost-Sensitive Approach to Improve the Multilayer Perceptron Performance Imbalanced Data. IEEE Transactions Neural Networks and Learning Systems, 24(6))888-899 2013.
Gilles Cohen, Melanie Hilario ad Hugo Sax, Stephane Hugonnet, and Antoine Geissbuhler. Learn- ing from Imbalanced Data in Surveillance of Nosocomial Infection. Artificial Intelligence in Medicine, 37(1):7-18, 2006.
Min Du and Feifei Li. Spell: Streaming Parsing of System Event Logs. In IEEE International Conference on Data Mining (ICDM), pp. 859-864, 2016.
Min Du and Feifei Li. Spell: Online Streaming Parsing of Large Unstructured System Logs. IEEE Transactions on Knowledge and Data Engineering (TKDE), (11):22133-222) 2018.
Amir Efrati. Uber Finds Deadly Accident Likely Caused By Software Set to Ignore Objects htt.rffmmntiooooommmrrrree////bi on-road).
On Road. The Information, 2018.
Andrew Estabrooks and Nathalie Japkowicz. A Mfxture-of-Expprts Framework for Learning from Imbalanced Data Sets. In International Conference on Advances in Intelligent Data Analysis (IDA), pp. 34 34-43, 2001.
Andrew Estabrooks, Taeho Jo, and Nathalie Japkowicz. A Multiple Resampling Method for Learn- ing from Imbalanced Data Sets. Computational Intelligence, 20(1):18-36 2004.
Haibo He and Edwardo A. Garcia. Learning from Imbalanced Data. IEEE Transactions on Knowl- edge and Data Engineering (TKDE), 219)): 2009.
Haibo He and Yunqian Ma (eds.). Imbalanced Learning: Foundations, Algorithms, and Applica-
tions. Wiley-IEEE Press, July 2013.
Pinjia He, Jieming Zhu, Zäbin Zheng, and Michael R Lyu. Drain: An Online Log Parsing Approach with Fixed Depth Tree. In IEEE International Conference on Web Services (ICWS), pp. 33-40, 2017.
Florian Helff, Gruenwald, and Laurent d'Orazio. Weighted Sum Model for Multi-Ohjective Query Optimization for Mobile-Cloud Database Environments. In EDETACDT International Workshop on Multi-Engine Data Analytics (MEDAL.), 2016.
Alberto Fernandez Hilario, Salvador García Lopez, Mikel Galar, Ronaldo C. Prati, Bartosz Krawezyk, and Francisco Herrera. Learning from Imbalanced Data Sets. Springer, 2018.

---

Under review as a conference paper at ICLR 2021
Natalie Japkowicz. Assessment Metrics for Imbalanced Learning. In Haibo He and Yungian Ma (eds.), Imbalanced Learning: Foundations, Algorithmms and Applications, pp. 187-206. John
Wiley & Sons, 2013.
Justin M. Johnson and Taghi M. Khoshgoftaar. Survey on Deep Learning with Class Imbalance. Journal of Big Data, 6(27), 2019.
Mahesh V. Joshi, Vipin Kumar, and Ramesh C. Agarwal. Evaluating Boosting Algorithms to Clas- sify Rare Classes: Comparison and Improvements. IEEE International Conference on Data Mining (ICDM), pp. 257-264, 2001.
Sara Makki, Zainab Assaghir, Yehia Taher, Rafiqul Haque, Mohand-Said Hacid, and Hassan Zeined- dine. An Experimental Study With Imbalanced Classification Approaches for Credit Card Fraud Detection. IEEE Access, 7:93010- 093000-93002, 2019.
Mareus A. Maloof. Learning When Data Sets are Imbalanced and When Costs are Unequal and Unknown. In ICML Workshop on Learning from Imbalanced Data Sets, 2003.
Salma Messaoudi, Annibale Panichella, Domenico Bianculli, Lionel Briand, and Raimondas Sas- nauskas. Search-based Approach for Accurate Identification Log Message Formats. In Proceedings of the 26th Conference on Program Comprehension, pp. 167-177, 2018.
Susan M. Mudambi and David Schuff. Research Note: What Makes Helpful Online Review? A Study of Customer Reviews on Amazon.com. MIS Quarterly, 34(1):185-200, 2010.
Jianmo Ni, Jiacheng Li, and Julian McAuley, Justifying Recommendations using Distantly-Labeled Reviews and Fine-Graiee Grained Aspects. In Conference on Empirical Methods Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP. IJCNLP), pp. 188-197, 2019.
Paul Pavlou and Angelika Dimoka. The Nature and Role Feedback Text Comments in On- line Marketplaces: Implications for Trust Building, Price Premiums, and Seller Differentiation. Information Systems Research, 17(4):392 414, 2006.
Jeffrey Pennington, Richard Socher, and Christopher D. Manning, GloVe: Global Vectors for Word Representation. In Empirical Methods in Natural Language Processing (EMNLP), pp. 1532- 1543, 2014.
Vishnu Subramanian. Deep Learning with PyTorch: A Practical Approach to Building Neural Net- work Models using PyTorch. Packt Publishing, 2018
Yanmin Sun, Andrew K. C. Wong, and Mohamed S. Kamel. Classification of Imbalanced Data: A Review. International Journal of Pattern Recognition and Artificial Intelligence, 23(4):687-719, 2009.
Chris Tofallis. Add or Multiply? A Tutorial on Ranking and Choosing with Multiple Criteria. INFORMS Transactions on Education, 14(3):109-119, 2014.
Evangelos Triantaphyllou. Multi-criteria Decision Making Methods: A Comparative Study.
Springer, 2000.
Cheng G. Weng and Josiah Poon. A New Evaluation Measure for Imbalanced Datasets. In Proceed- ings of the 7th Australasian Data Mining Conference (AusDM '08), pp. 27-32, 2008
Jieming Zhu, Shilin He, Jinyang Liu, Pinjia He, Qi Xie, Zibin Zheng, and Michael R Lyu. Tools and Benchmarks for Automated Log Parsing. In IEEE/ACM 41st International Conference on Software Engineering: Software Engineering in Practice (CSEE-EIP), pp. 121-130, 2019.
10

---

Under review as a conference paper at ICLR 2021
A APPENDIX
In this appendix, we provide details for the experimental study, including data and code. For further information, please see the supplementary material.
A.1 DETAILS FOR LOG PARSING EXPERIMENTS
For the three log parsing techniques used Section (Drain, Spell, and MolFF), we used the implementations provided by the LogPAI team:
https:
com/ htt.h/albooooooooaoooooo
The four datasets used in these experiments (macOS, BGL, Android, and HDFS) came from the benchmarking data also provided by LogPAI:
htt.pp////gttubb.com///oo/
lete we Eeequency
(a) macOS (skew 8.454)
109
Ield te Eequeety
30
(b) BGL (skew = 8.900)
100
Becativ Ineguncy
30"4
message cassess
(c) Android (skew = 4.822)
109
Resative meqqarey
30-4
10-2
30
message
(d) HDFS (skew 0.202)
Figure 4: Histograms showing the relative frequencies of log parsing classes for the four experimen- tal datasets: All graphs have their y-axes in log scale; green bars show the infrequent classes.
In Figure 4 we show the histograms for the four log datasets together with their skew values. As defined in the Microsoft Excel Documentation, "Skewness characterizes the degree of asymmetry of a distribution around its mean. Positive skewness indicates distribution with an asymmetric tail extending toward more positive values, while negative skewness indicates distribution with

---

Under review as a conference paper at ICLR 2021
asymmetric tail extending toward more negative values." " 0 In context, skew provides good indication for the degree of imbalance in class cardinality distributions the larger the skew, the larger the degree of class imbalance.
We also provide data files with class labels (true + predicted) and weights (based on rarity as importance criteria) used in generating the experimental data plotted in Figure 2 as part of our WBA-Evaluator tool implementation included in the supplementary material (can be found under directory).
the WBA-Eva Eval-atoootor/// wewpepeeeeeeeppaeeeeeeee direetrrr
A.2 DETAILS SENTIMENT ANALYSIS EXPERIMENTS
For the sentiment analysis experiments of Section Reviews dataset provided at:
4.2 we used sample from the Amazon Customer
https://nijjaammo. gi hub. 10/ amazam intpx/ndxxxxcmm/ htm
0.5
Relative Frequency
0.5
0.3
0.2
0.0
Rating
Figure 5: Histogram showing the relative frequencies of the five customer rating classes for the Amazon dataset (skew 2.14
In Figure we show the histogram for the Amazon dataset. As described in Section 4.2 we im- plemented RNN-based classifiers to experiment with this dataset. The code for these classifiers can be found in the supplementary material (under the fmazooneeeiieettllsss directory) along with copy of the data (under the amazcoteeiierrcceessfffrrraaaas directory).
We also provide the data files with class labels (true predicted) and weights (user) used in gener- ating the experimental data for LSTM results plotted in Figure B and Table as example. These can found in our WBA-Evaluator tool implementation included the supplementary material under the FBA-Eva wuaatr//eaaaaaaa es Amazon/ directory.
A.3 THE WBA-EVALUATON ToOL
In addition to details on our experimental study as described above, we also provide a copy of the WBA-Evaluator tool that implements our customizable, class-weighted evaluation framework de- scribed in Section WBA-Evaluator is written in Python and can be found in the supplementary material along with README that describes how it can be used. In a nutshell, WBA-Evaluator takes as input three files (true class labels, predicted class labels, class weights) and number of con- figuration parameters in the form of commandline arguments, and then generates accuracy scores (BA WBA) specified by these arguments. The WBA-Evaluator implementation comes with two subdirectories: src/ contains the Python source code; example/ contains all the input files (labels and weights) and scripts in the scripts/ subfolder run these. Please see the README file for more details. Using this tool, results reported in the paper can be reproduced.
12


Reviews:
[{"review_id": "rOfDp6aSwh7", "reviewer": "AnonReviewer6", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents a weighted balanced accuracy to evaulate the performance of multi-class classification. Basically, the performance for a multi-class problem can be evaluated by decomposing the original multi-class problem into a number of binary ones based on one-against-rest manner, and then evaulating the performance scores for each of the binary ones using any well-known metric for binary classification, and then, aggregating the performance scores. The main aim of this paper is to present a weighting scheme when aggregating the scores.\n\nI'm inclined to rejection of this paper. Frankly speaking, I don't think this paper has a significant contribution. the weighting schemes to combine binary metric scores to evaulate the performance of multi-class classification have been well-studied, such as macro-averaging, micro-averaging, as well as importance weighting (manual or data-driven e.g. frequency). The proposed idea is just a simple natural extension of balanced accuracy with a weighting scheme. It is nothing new.", "extra_scores": {}, "extra_text": {}}, {"review_id": "__GtS1g236w", "reviewer": "AnonReviewer5", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a simple and general-purpose evaluation framework for imbalanced data classification that is sensitive to arbitrary skews in class cardinalities and importances.  \n\nI think the problem this paper deals with is very important and of great interest to the wide range of readers. \nIn addition, the paper is generally clearly written and easy to follow.\n\nHowever, I found the novelty and the originality of this paper is not enough for the ICLR standard.  \nAlthough I am not aware of the paper that presents exactly the same concept as this paper, I feel the generalization this work presents is too straightforward that new insights, findings, and benefits brought by this paper to the community are very limited.\nFor example, the micro average (or simply called \"Accuracy\" in this paper) can be regarded as a special form of equation (6), where weights or importance is given by the relative frequency of each class.\nThis importance criteria can be regarded as opposite of one of the presented criteria, \"Rarity\".\nThere may be some cases where the rarer a class is, the more important the class is as explained in the paper, but there may be other cases where more frequent classes are more important. As such, the concept of including importance of each class into an evaluation metric has been implicitly considered. It is true that the paper provides general form that includes the aforementioned case, but I'm afraid the formulation is not novel enough to bring a new value to the community as I mentioned above.\n\nAnother weakness of the paper lies in the experimental analysis.  \nOverall, the analysis is not convincing enough to verify the benefit of the proposed metric.\nFor example, at the end of \"WBA_rarity vs. Class-insensitive Metrics\" in page 7, the authors states \"This result validates that WBA_rarity provides a more sensitive tool for assessing classification performance\". I do not agree with this statement because it is no surprise that different evaluation metrics give different evaluation results, and this alone is not the ground for the validity of the proposed metric. The same argument can be applied to the \"Impact of WBA in Model Training\" in page 8. It is natural that a model trained with a specific criteria results in performing well on the criteria. This is again not the ground for the claim \"our new framework is more effective than Balanced Accuracy – ... also in training the models themselves.\"\n\nThe pros and cons of this paper can be summarized as follows.\n\n\n#### Pros\n1. The paper deals with practically important topic, and presents a simple, easy and intuitive solution.\n1. The paper is clearly written and easy to understand.\n\n#### Cons\n1. The novelty and the originality of this paper is not enough that there is only limited benefits to the community, which does not satisfy the ICLR standard.\n1. The experimental analysis is not convincing enough to support the usefullness of the proposed metric.", "extra_scores": {}, "extra_text": {}}, {"review_id": "GQEzKZb5dGL", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\n\nThis paper proposes a new evaluation framework for imbalanced data. Specifically, they introduce an additional weighted term in the formulation of balanced accuracy. By varying the weights, their framework can be adapted to many application domains. Finally, they use two case studies to illustrate the effectiveness of the proposed measure.\n \nPros:\n\n- The proposed framework is simple and effective. \n- The proposed framework can be used in many application domains.\n\n\nCons:\n\nI have several comments regarding the experimental results:\n\n- How do the authors compute the “skew” score for each dataset? This term is not clearly explained in the paper. \n\n- Since the weighted terms e.g., $w_i$ play an important role in the proposed framework, more examples should be given to explain how to choose these parameters in different application domains.\n\n- In section 4.2, the authors state “ … by modifying loss functions of DNNs to capture class importance weights…”. It is not very clear to see how to implement this.\n\n- When using WBA in model training, what are the performances for the baseline measures, e.g., precision, recall? Also, why choose these values for $w_i$, e.g., w1=0.209, w2=0.368, and w3=0.255? \n\n \nOverall, I think the novelty of this work is limited and the experimental results are not very convincing.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "7d7oK9mzld8", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper presents a simple addition to the Balanced Accuracy approach - which the authors refer to as ‘importance’. However, there is nothing in the formulation of this concept which requires that this is an importance and could in fact be any form of weighting. The paper evaluates the new metric - but only agains the Balanced Accuracy metric (which seems quite restrictive).\n\n## Some major comments on the paper:\n\n1) The proposed evaluation metric appears to be to show whether machine learning approach A is actually better than machine learning approach B. As such one can use the metric to give a value which can be used to compare different approaches. However, in order to judge if a particular approach is better or worse than another you need some way of showing that your metric is correct. The paper lacks a demonstration that your approach actually does give a more appropriate ordering of the machine learning approaches.\n\n2) In the abstract you talk about ‘importance’, however, this concept is not clear at this point. Later in the introduction you explain what importance refers to. Some of this in the abstract would help. The abstract also seems to terse making it difficult to follow.\n\n3) The second paragraph in the introduction seems odd. You are claiming in the previous paragraph that importance need not be the inverse of rarity, however, the example in the second paragraph seems to re-enforce the idea of importance being the inverse of rarity. The text on Uber also doesn’t seem to fit the paper at all and could easily be dropped.\n\n4) Figure 1a needs more discussion in the text.\n\n5) Paragraph starting ‘Let us illustrate the problem’ - this talks about a dataset in the most abstract sense. It would seem this is potentially a concocted example without a real dataset behind it. It would be much better to indicate what dataset this is based on - perhaps detailing the scenario in the appendix. \n\n6) The related work on evaluation metrics seems a little short. How do the other proposed approaches compare to your work?\n\n7) “As we discussed with examples in previous sections, in many real-world classification problems,” - you only give one example and the ‘real-world’ case that this refers to is not provided. Stronger evidence is needed to support this statement.\n\n8) Most sentiment analysis approaches are not based on RNNs - can you justify why you used this approach?\n\n9) The conclusions are rather short and say little. You also claim in the introduction that you will discuss future directions but don't.\n\n## Some more minor comments:\n\n- If a majority is the larges group, what is the ‘greatest majority’? Surely it’s just the majority?\n\n- ‘adopt’ -> ‘adapt’\n\n- There is no punctuation around the equations - For example there should be a full-stop after equations 1 and 2.\n\n- As only equation 6 is referred to in the text why are the others numbered?\n\n- Something going odd in the quotes in “ not inflated due to high-frequency classes’ results dominating over\nthe others’. “\n\n- “must be rewarded higher scores” -> “must be rewarded with higher scores”\n\n- Is equation 7 really needed?\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a weighted balanced accuracy metric to evaluate the performance of imbalanced multiclass classification. The metric is based on a one-versus-all decomposition from multi-class to binary, and then aggregating the metrics on the binary classification sub-problems in a weighted manner. The authors hope to argue that the new metric is more flexible for evaluating classifiers in the imbalanced and importance-varying setting.

The reviewers agree that the proposed framework is simple and applicable to an important problem. Somehow the novelty and significance of the work is pretty limited, as many related metrics (e.g. micro/macro-averaged metrics) exist in the literature. The authors are encouraged to think about stronger reasoning on how useful the "new" metric is. The experiments are also not convincing nor complete enough to verify the benefits of the proposed metric.

Decision: Reject