# Analysis of Cost-Sensitive Algorithms for Degree of Imbalancing

Sai Teja Tangudu, Rajeev Kumar

► To cite this version:

Sai Teja Tangudu, Rajeev Kumar. Analysis of Cost-Sensitive Algorithms for Degree of Imbalancing. 6th International Conference on Computational Intelligence in Data Science (ICCIDS), Feb 2023, Chennai, India. pp.67-75, 10.1007/978-3-031-38296-3_6. hal-05466046

HAL Id: hal-05466046

https://inria.hal.science/hal-05466046v1

Submitted on 19 Jan 2026

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons CC BY 4.0 - Attribution - International License
<figure><img src="image_2.png" /></figure>

ifip

International Federation for Information Processing

This document is the original author manuscript of a paper submitted to an IFIP
conference proceedings or other IFIP publication by Springer Nature. As such, there
may be some differences in the official published version of the paper. Such
differences, if any, are usually due to reformatting during preparation for publication or
minor corrections made by the author(s) during final proofreading of the publication
manuscript.
# Analysis of Cost-Sensitive Algorithms for Degree of Imbalancing

Sai Teja Tanguudu(✉) and Rajeev Kumar

Data to Knowledge (D2K) Lab, School of Computer and Systems Sciences,
Jawaharlal Nehru University, New Delhi 110 067, India

saiteja.makkuva@gmail.com

**Abstract.** Class imbalance, along with other characteristics of the data, such as feature set and class separability, could affect the performance of most machine learning algorithms. This can be attributed to the algorithm's primary assumption about the data being class-balanced and indifferent weightage among different misclassification errors. Class imbalance can be handled using both data and algorithmic-level methods. In this paper, we work on cost-sensitive learning, applied at the algorithmic level. We use the Cost-Sensitive Logistic Regression (CSLR) algorithm as a reference. We propose a methodology to empirically evaluate the performance of cost-sensitive algorithms over varying degrees of imbalanced data. Cost-sensitive learning induces a cost matrix consisting of the weighting scheme of different misclassification errors into the algorithm's training process. This, in turn, forces the model to penalize the misclassification errors according to the skewness of the data to reduce the learning bias of the model towards the majority class. We present empirical evaluations of the reference model over four popular datasets and analyse its behaviour using MAE and Kappa values.

**Keywords:** Machine Learning · Classification · Class Imbalance · Cost-Sensitive Learning · Degree of Imbalance

## 1 Introduction

Class imbalance is a classification problem when the distribution of classes in the training data is primarily skewed toward a single class, making it the majority. Most machine learning algorithms work on the assumption that the classes are balanced. Therefore, with the increase in the degree of imbalance, the difficulty in learning from that data increases for a model [7]. It becomes even more critical in multi-class problems. While dealing with imbalanced data, A model's performance will mostly be biased or slanted toward the dominating class [9]. Such unbalanced class distributions are considerably frequent in real-world classification tasks that include but are not limited to fraud detection, spam detection, medical diagnosis, etc. [12]. Therefore we require specialized techniques to handle the imbalance.
According to Mienye et al., [9], the most popular approaches to handle the class imbalance can be broadly classified into data-level methods, algorithmic-level methods, and a hybrid of both. Data level methods majorly consider either under-sampling or over-sampling, or a combination of both to re-sample the data to reduce the imbalance. As re-sampling varies the distribution of the data samples, these methods can have adverse effects like inducing noise or missing out on some critical samples, which can contribute to the model's learning.

The algorithm-level methods alter the existing learner to eliminate its bias toward the classes that make up the majority. The most common approach applied at the algorithmic level is cost-sensitive learning, which forces the learner to correctly classify minority class samples by imposing a higher penalty on misclassified minority class samples [5]. Cost-sensitive learning is a learning technique that assigns different costs to different types of misclassification, potentially reducing the bias towards the dominant class.

Instead of the typical misclassification error, the conventional motivation for these algorithms would be to reduce the overall cost. Almost all machine learning algorithms consider all types of misclassifications equal, but they may not. So, cost-sensitive learning focuses on the idea that false negatives must be weighed more than false positives during the calculation of error. However, defining an optimized cost matrix can be challenging if not given a prior one [5]. This work aims at understanding the efficiency of cost-sensitive algorithms in dealing with binary classification problems with varying degrees of imbalance. We use different costs for weighing false positives and false negatives, which are provided to the algorithm during the training in class weights. We address the following Research Questions:

**RQ 1:** Does the behaviour of cost-sensitive algorithms vary with varying degrees of imbalanced data?

**RQ 2:** How does the selection of class weights impact the performance of cost-sensitive algorithms?

The research contribution of this work is the proposed methodology that can be used to empirically verify the behavioural patterns of cost-sensitive algorithms and their applicability over imbalanced data. Our methodology empirically verifies whether the behaviour of cost-sensitive algorithms varies with varying degrees of imbalanced data. In this paper, we used class weights as the hyperparameter, with which we verified the behaviour of cost-sensitive algorithms. This hyperparameter can be customized in future works related to cost-sensitive learning algorithms.

The rest of the paper is organized as follows. In Sect. 2, we discuss the previous work in two verticals: designing a cost matrix and modifying learners. We then proposed our methodology and discussed the steps in detail in Sect. 3. Section 4 details the characteristics of the datasets used and their class imbalance. Then we discussed the experimental setup, illustrated and discussed the results obtained from our experiments, and provided the necessary reasoning for the same in Sect. 5. Finally, we conclude our paper with the possible future scope and research directions in Sect. 6.
# 2 Related Work

This section discusses various works in cost-sensitive learning and identifies the associated research gaps. Entire work in cost-sensitive learning can be categorized into two verticals: designing a cost matrix and modifying existing learners to make them cost-sensitive [5].

## 2.1 Designing of Cost Matrix

Defining a cost matrix is usually the core part of any cost-sensitive algorithm. Typically, prior availability of a cost matrix is suitable, except that it requires a domain expert to validate. Moepya et al. [10] used eight different cost matrices to analyse the effectiveness of cost-sensitive algorithms in financial fraud detection using transaction statements. According to them, the true costs of misclassification were hard to find. Therefore, they used pre-determined cost matrices with different levels of cost items. A more popular approach is to use the inverse class distribution ratio for weighing the misclassification errors, i.e., the weight of FP error = the number of majority samples and vice versa. Mienye et al. [9] used this ratio for analysing the performance of cost-sensitive algorithms to handle imbalanced medical data. This method of setting the cost values may be ineffective as an imbalance may not be the sole reason for the inefficiency of a model [6].

Maloof et al. [8] proposed threshold moving, which shifts the output threshold close to majority class instances, making it hard to misclassify samples with higher costs. It employs cost values at the classification phase rather than the training phase. This method is considered unpopular yet effective as a resampling technique for solving imbalance. Domingos et al. [4] proposed a meta-cost approach for solving the class imbalance. Its underlying principle is to create multiple replicates of the original training data and then employ a classifier on each of them. Then estimate each class's probability based on the votes it received and relabels all the examples with that estimated class. The final step reapplies the classifier on the relabeled training data. In case of any modification to the cost matrix, the algorithm needs to repeat only the final stage of the learning process in contrast to the remaining algorithms, where it needs to repeat the entire learning process.

## 2.2 Modification of Learners

Objective functions of the machine learning algorithms weigh all types of errors equally. Cost items are induced into learners' objective functions to make them cost-sensitive. This also forces the learner to reduce the overall cost instead of the training error. Mienye et al. [9] focused on inducing cost values into the objective functions of the classifiers, namely, Logistic Regression, Random Forest, Decision Tree, and XGBoost. They then used the modified objective functions of these algorithms to classify some of the well-known medical imbalanced data sets. This study showed cost-sensitive random forest to be the most effective
among its counterparts in dealing with imbalanced medical data. Sun et al. [12] investigated meta-techniques to classify imbalanced data. They proposed three ways to induce cost values into AdaBoost's weight update formula originating three variants, namely, AdaC1, AdaC2, and AdaC3. They induced cost items outside the exponential function for the first variant, inside the function for the second, and both in and out of the function for the last. Based on the empirical analysis, this paper concluded that the performance of AdaC2 is superior among the variants. Cao et al. [3] presented an approach incorporating a combination of performance metrics, i.e., GMean and AUC, into the Support Vector Machine (SVM) classifier's objective function, making it cost-sensitive. Along with the performance metrics, this work also focused on optimising intrinsic parameters, feature subsets, and cost parameters simultaneously.

Research gaps that can be pointed out from the above literature survey are as follows:

1. Most work focused either on comparing cost-sensitive algorithms to their non-cost-sensitive counterparts or on the effectiveness of a specific cost-sensitive algorithm in handling imbalance. but the general behaviour of the cost-sensitive algorithms is rarely discussed in the works mentioned.

2. response of the cost-sensitive algorithms to various characteristics of the imbalanced data sets, i.e., skewness, data fragmentation, class overlapping, etc. is yet to be explored. This work focuses on the skewness aspect of the imbalanced data.

# 3 The Methodology

This section discusses our proposed methodology to empirically evaluate the behaviour of cost-sensitive algorithms over the varied degree of imbalanced data with various class weights.

## 3.1 Imbalancing Groups

The dataset under study is resampled to produce four imbalanced variations. The first version is characterised by high imbalance, with a minority sample to majority sample ratio close to 5:95; the second and third variants are characterised by moderate class imbalance, with imbalance ratios close to 15:85 and 30:70, respectively. With a ratio close to 50:50, the final variant has very little to no imbalance.

## 3.2 Sampling for Imbalancing

Using Random Under Sampling, we initially create an imbalance in our training data by reducing the number of samples of the minority class and increasing the number of samples of the majority class using the ADASYN oversampling method. As the oversampling methods such as SMOTE and ADASYN cannot generate the exact number of samples required, the imbalance ratios are relatively similar to that of the stated rather than accurate.
## 3.3 Selection of Weights

After setting the imbalance, we created a dictionary of hyper-parameters with different class weights, i.e., 100:1, 10:1, 1:1, 1:10, 1:100, and 1:1000 (refer to Table 1). We can include any class weight for the experiment, but a more reasonable class weight should weigh a minority class instance more than the majority.

## 3.4 Measures for Performance Assessment

Most prominent metrics, such as accuracy, may fail to accurately indicate a classifier's efficiency in dealing with class imbalance. A classifier that predicts the negative class for every training sample can still achieve a high level of accuracy, representing the classifier as efficient. Therefore, we considered a more suitable metric called *Cohen's Kappa* coefficient for our analysis. This can be calculated from the *Confusion Matrix* using the following equation

$$ Kappa = \frac{2 \times (TP \times TN - FP \times FN)}{(TP + FP) \times (FP + TN) + (TP + FN) \times (FN + TN)} $$

# 4 Datasets and Imbalance

This section describes the datasets used for the empirical analysis. This description includes but is not limited to attribute information, pre-processing steps, and class distribution summary. All the attributes present in these datasets have been scaled to values in the range [0, 1] using `MinMaxScaler` available in python's `scikit-learn` library. Following are the details of the datasets used for the empirical analysis.

*Vehicle Dataset:* This four-class dataset [1] has eighteen attributes and four classes, namely, Opel, bus, Saab, and van. The dataset is reclassified into “Van” and “Non Van” classes in order to induce class imbalance. This leaves us with 199 minority samples versus 647 samples of the majority class with an imbalance ratio (IR) of 0.307.

*Pima Indians Diabetes dataset:* This is a dataset<fnref n="1" /> used for binary classification tasks, with class labels diabetic and non-diabetic, with attributes summarizing specific diagnostic metrics of the patient. It contains a total of 268 samples from the positive class and 768 samples from the negative class. There are no null values in the data, but many zeros are recorded for the attributes like “Skin Thickness” and “Insulin” which is absurd. We used the KNNImputer available in the `sklearn` preprocessing module to artificially impute the values in place of zeros (with K = 5).

<footnote n="1">¹ https://www.kaggle.com/datasets/uciml/pima-indians-diabetes-database.</footnote>
*Ionosphere dataset:* This binary class dataset [11] with 351 examples and 34 attributes is used to classify the radar as good or bad. We refer to the “good radar” class with 225 samples as the majority and the bad radar class with 126 samples as the minority. The ‘g’ and ‘b’ labels have been replaced with 1 and 0 as part of pre-processing. For scaling, we have used Standard Scaler instead of MinMaxScaler because of its ability to handle outliers.

*Abalone Dataset:* The age of the abalone is classified by its physical characteristics using the original abalone data set, [2]. There are 29 classes and seven attributes (excluding the discrete feature, i.e., sex). we refer to class “18” as the minority and class “9” as the majority class. They consist of 42 and 689 samples respectively.

# 5 Results

## 5.1 Experimental Setup

Original versions of the datasets are taken from the UCI repository and loaded into Jupyter Note Book (python 3.6) for the experiment. Entire data sets' are used for creating different degrees of imbalanced data. The standard implementation of ADASYN (from *imblearn*) for resampling, Logistic Regression for model generation, RepeatedStratifiedKFold for creating *K* folds, i.e., *K* = 10 and Grid-SearchCV for cross-validation with class weights as hyperparameters are used for the implementation purposes. We instantiated the standard Logistic Regression class present in the sklearn module with the set of class weights mentioned in Table 1. We then empirically evaluated the performance of this model over the four imbalanced variants of the four datasets mentioned in Sect. 4.

## 5.2 Residual Errors

On every imbalanced variant of the dataset, we calculate the mean and standard deviation of mean absolute error for the seven class weights mentioned. Figure 1 illustrate the results using error bars where bars represent the standard deviation of the error; their intercept point represents the mean error over all the weights for a specific variant.

<figure><img src="image_3.png" /><figcaption>Fig. 1. Error distribution for four datasets</figcaption></figure>
## 5.3 Performance Measures

Error bars presented above are based on the mean absolute error, simply the inversion of the accuracy score. As mentioned in Sect. 5.3, we consider the *Kappa* score, i.e., Cohen's *Kappa coefficient*, to be a superior metric in representing the performance of a classifier in cases involving imbalance. In this section, we present the performance trends of our Cost-Sensitive Logistic Regression (CSLR) algorithm in terms of the *Kappa score*.

<table><caption>**Table 1.** Kappa score on the Vehicle dataset</caption><thead><tr><th>Class Weight Degree</th><th>100:1</th><th>10:1</th><th>1:1</th><th>1:10</th><th>1:100</th><th>1:1000</th><th>1:10000</th></tr></thead><tbody><tr><td>5:95</td><td>0.000</td><td>0.000</td><td>0.000</td><td>0.619</td><td>0.242</td><td>0.124</td><td>0.087</td></tr><tr><td>15:85</td><td>0.000</td><td>0.000</td><td>0.426</td><td>0.649</td><td>0.365</td><td>0.275</td><td>0.239</td></tr><tr><td>30:70</td><td>0.012</td><td>0.331</td><td>0.858</td><td>0.677</td><td>0.498</td><td>0.437</td><td>0.394</td></tr><tr><td>50:50</td><td>0.012</td><td>0.331</td><td>0.858</td><td>0.677</td><td>0.498</td><td>0.437</td><td>0.394</td></tr></tbody></table>

Table 1 shows the kappa score over different class weights and degrees of imbalance for the Vehicle data set. Rows represent different degrees of imbalance, whereas columns represent different class weights, i.e., Minority Class Weight to Majority Class Weight Ratio. Figure 2 shows the varying trend of kappa values in the form of error bars, i.e., the error bar represents standard deviation, and the intercept point represents the mean Kappa score over different class weights.

<figure><img src="image_4.png" /><figcaption>**Fig. 2.** Kappa Score Distribution for the datasets.</figcaption></figure>

## 5.4 Discussion

Our experiment's primary objective is to analyze the varying performance of cost-sensitive algorithms, for example, the Cost-Sensitive Logistic Regression (CSLR), dealing with different degrees of class imbalance and the impact of varying cost values on the same. Based on the error bars obtained from our experiment, three of the four data sets support our hypothesis, i.e., varying cost
values influence the performance of CSLR while dealing with a higher degree of class imbalance. But this has little to no impact when it comes to well/almost balanced data. This can be observed in terms of the growing length of error bars, depicting a higher variance of errors, i.e., the unstable performance of the classifier over different class weights. Our experiment on the last data set, i.e., Ionosphere does not show any significant pattern either supporting or contradicting the same. This can be attributed to the data set's unique characteristics, the re-sampling method used to create the variants, and the scaling method used. The classifier's performance on this data set varied when a different re-sampling or scaling method was used during the pre-processing phase.

The means line, i.e., the line joining the mean error over different degrees of imbalance, does not show any significant pattern for data sets. It is neither converging nor diverging over different degrees of imbalance. This can be attributed to the fact that the accuracy score does not represent the cases involving imbalance well, and so is the mean absolute error. This motivated us to provide the results of our experiments in terms of the Kappa score. Table 1 shows the Kappa scores obtained on the vehicle dataset. Figure 2 represent the Kappa score values in the form of error bars over all the datasets. The Kappa mean line is steadily increasing as we move from extreme imbalance to that of a well-balanced variant in three of the four datasets. This shows that the Cost-Sensitive Algorithms perform well on a low degree of imbalance in the data and vice-versa. These error bars do not show any significant pattern supporting our hypothesis. Suppose the kappa score metric is considered superior to the mean absolute error. In that case, these error bars show that the impact of the selection of class weights on the performance of cost-sensitive algorithms over different degrees of imbalance is subjected to the characteristics of the data set and cannot be generalized.

# 6 Conclusion

This paper proposed a methodology for empirically analyzing the behavioural patterns of cost-sensitive algorithms using different degrees of imbalanced data. We consider the Cost-Sensitive Logistic Regression (CSLR) as the reference model to present our methodology. We consider four degrees of imbalanced variants of four popular datasets for our analysis and different levels of class weights as the hyperparameter. The behaviour of the Cost-Sensitive Logistic Regression (CSLR) is then analyzed with the help of two error bars generated based on the mean absolute score and Kappa score, respectively. The discussion section thoroughly discusses their support and resistance to our hypothesis. We suggest the usage of different hyperparameters, different sampling and scaling techniques, and the inclusion of some more degrees of imbalanced variants of the data sets, comparing the performance of different cost-sensitive algorithms over them as directions for future work. This methodology can also be extended to analyze the behavioural patterns of cost-insensitive classifiers and their efficiency in dealing with the imbalance.
References

1. Statlog (Vehicle Silhouettes). UCI Machine Learning Repository

2. Abalone. UCI Machine Learning Repository (1995)

3. Cao, P., Zhao, D., Zaiane, O.: An optimized cost-sensitive SVM for imbalanced data learning. In: Pei, J., Tseng, V.S., Cao, L., Motoda, H., Xu, G. (eds.) PAKDD 2013. LNCS (LNAI), vol. 7819, pp. 280–292. Springer, Heidelberg (2013). https://doi.org/10.1007/978-3-642-37456-2_24

4. Domingos, P.: MetaCost: a general method for making classifiers cost-sensitive. In: Proceedings of 5th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp. 155–164 (1999)

5. Fernández, A., García, S., Galar, M., Prati, R.C., Krawczyk, B., Herrera, F.: Cost-sensitive learning. In: Fernández, A., García, S., Galar, M., Prati, R.C., Krawczyk, B., Herrera, F. (eds.) Learning from Imbalanced Data Sets, pp. 63–78. Springer, Cham (2018). https://doi.org/10.1007/978-3-319-98074-4_4

6. Krawczyk, B.: Learning from imbalanced data: open challenges and future directions. Progr. Artif. Intell. 5(4), 221–232 (2016)

7. Lin, W.C., Tsai, C.F., Hu, Y.H., Jhang, J.S.: Clustering-based undersampling in class-imbalanced data. Inf. Sci. 409, 17–26 (2017)

8. Maloof, M.A.: Learning when data sets are imbalanced and when costs are unequal and unknown. In: ICML Workshop on Learning from Imbalanced Data Sets II, vol. 2, pp. 2–1 (2003)

9. Mienye, I.D., Sun, Y.: Performance analysis of cost-sensitive learning methods with application to imbalanced medical data. Inform. Med. Unlock. 25, 100690 (2021)

10. Moepya, S.O., Akhoury, S.S., Nelwamondo, F.V.: Applying cost-sensitive classification for financial fraud detection under high class-imbalance. In: Proceedings IEEE International Conference on Data Mining Workshop, pp. 183–192. IEEE (2014)

11. Sigillito, V., Wing, S.: Ionosphere. UCI Machine Learning Repository (1989)

12. Sun, Y., Kamel, M.S., Wong, A.K., Wang, Y.: Cost-sensitive boosting for classification of imbalanced data. Pattern Recogn. 40(12), 3358–3378 (2007)