# Editing Training Sets from Imbalanced Data Using Fuzzy-Rough Sets

Do Van Nguyen, Keisuke Ogawa, Kazunori Matsumoto, Masayuki Hashimoto

► To cite this version:

Do Van Nguyen, Keisuke Ogawa, Kazunori Matsumoto, Masayuki Hashimoto. Editing Training Sets from Imbalanced Data Using Fuzzy-Rough Sets. 11th IFIP International Conference on Artificial Intelligence Applications and Innovations (AIAI 2015), Sep 2015, Bayonne, France. pp.115-129, 10.1007/978-3-319-23868-5_9. hal-01385349

HAL Id: hal-01385349

https://inria.hal.science/hal-01385349

Submitted on 21 Oct 2016

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>
# Editing Training Sets from Imbalanced Data
Using Fuzzy-Rough Sets

Do Van Nguyen*, Keisuke Ogawa, Kazunori Matsumoto, and
Masayuki Hashimoto

KDDI Labs, 2-1-15 Ohara, Fujimino, Saitama, 356-8502 Japan

**Abstract.** In this research, we study several instance selection methods based on rough set theory and propose an approach able to deal with inconsistency caused by noise and imbalanced data. Recent attention has focused on the significant results obtained in selecting instances from noisy data using fuzzy-rough sets. For imbalanced data, fuzzy-rough sets approach is also applied before and after using balancing methods in order to improve classification performance. In this study, we propose an approach that uses different criteria for minority and majority classes in fuzzy-rough instance selection. It thus eliminates the step of using balancing techniques employed in controversial approach. We also carry out some experiments, measure classification performance and make comparisons with other methods.

**Keywords:** Rough Set theory · fuzzy-rough sets · classification performance · instance selection · imbalanced data

## 1 Introduction

Rough set model is a machine learning technique for acquiring knowledge from datasets. Rough set theory was first introduced by Pawlak [19,20] in the early 1980s. It is a mathematical approach designed to deal with the ambiguity, vagueness and uncertainty that exist in a database. In knowledge discovery, rough set theory constitutes a sound basis by offering mathematical tools to discover patterns hidden in data. It can, according to [25], be used for feature selection, data reduction, decision rule generation, and pattern extraction.

Recently, rough sets have been implemented for instance selection [3,12,27]. The main ideas of these approaches are extracting fuzzy memberships of positive regions and choosing the instances which have large membership degrees for training phases. However, it is believed that these criteria do not overcome all problems in relation to inconsistency issues.

One issue that may cause an inconsistency problem is imbalanced data [14]. A dataset is imbalanced when the numbers of instances in some classes are much larger than in others. Such classes are called majority classes. The classes with small cardinality are referred to as minority classes.

<footnote>* Corresponding author: va-nguyen@kddilabs.jp; ngdovan@gmail.com</footnote>
In fact, some researchers are trying to use the advantages of rough sets to enhance classification performance when a balancing method is employed [22,23,28]. In these approaches, fuzzy-rough sets are first used to remove low quality instances. Then, a well-known balancing technique called “Synthetic Minority Oversampling Technique (SMOTE)” [4] is employed to form a candidate set. Finally fuzzy-rough sets are used again to select quality instances from the candidate sets. Obviously, such methods need considerable computation time to edit and select instances.

This research will introduce an approach that uses fuzzy-rough sets to balance and select quality instances from imbalanced datasets. In this work, different thresholds for majority and minority classes are used, thus, more items from minority classes can be selected. The experiment results show considerable improvement in classification performance and the advantage with respect to other methods in the literature.

The paper is structured as follows: Section 2 reviews the original rough set theory and an extension called fuzzy-rough set. In section 3, some rough set approaches for dealing with inconsistency are discussed along with their issues. The new algorithms and experiments are introduced in Section 4. Finally, we present our conclusions in Section 5.

# 2 Theoretical Background

## 2.1 Information Systems and Equivalence Relation

An information system is represented as a data table. Each row of this table represents an instance of an object such as people, things, etc. Information about every object is described by object attribute (feature) values.

An information system in the rough sets study is formally defined as a pair $I = (U, A)$, where $U$ is a non-empty finite set of objects called the universe and $A$ is a non-empty finite set of attributes such that $f_a : U \to V_a$ for every $a \in A$ [19,20]. The non-empty discrete value set $V_a$ is called the domain of $a$. The original rough set theory deals with complete information systems in which $\forall x \in U, a \in A, f_a(x)$ is a precise value.

Any information system taking the form $I = (U, A \cup \{d\})$ is called a decision table where $d \notin A$ is called a decision (or label) and elements of $A$ are called conditions. Let $V_d = \{d_1, ..., d_k\}$ denote the value set of the decision attribute, decision $d$ then determines a set of partitions $\{C_1, C_2, ..., C_k\}$ of universe $U$, where $C_i = \{x \in U | f_d(x) = d_i\}, 1 \le i \le k$. Set $C_i$ is called the $i$-th decision class or concept on $U$. We assume that every object in $U$ has a certain decision value in $V_d$.

Formally, in complete information systems, relation $EQU_P(x, y)$, $P \subseteq A$, denotes a binary relation between objects that are equivalent in terms of values of attributes in $P$ [19]. The equivalence relation is reflexive, symmetric, and transitive. Let $E_P(x) = \{y \in U | EQU_P(y, x)\}$ be the set of all objects that are equivalent to $x$ by $P$, which is then called an equivalence class. The family of
all equivalence classes (or partitions) on U based on an equivalence relation is referred to as a category and is denoted by $U/EQU_P$.

From equivalence classes, Pawlak [19,20] defined an approximation space that contains lower and upper approximations denoted by $\underline{appr}_P X$ and $\overline{appr}_P X$, respectively, of set $X \subseteq U$ as follows:

$$ \underline{appr}_P X = \bigcup \{E_P(x) | x \in U, E_P(x) \subseteq X\} = \{x \in U | E_P(x) \subseteq X\}, \quad (1) $$

$$ \overline{appr}_P X = \bigcup \{E_P(x) | x \in U, E_P(x) \cap X \neq \emptyset\} = \{x \in U | E_P(x) \cap X \neq \emptyset\}. \quad (2) $$

In decision table $I = (U, A \cup \{d\})$, the positive region $POS_P(X)$ of set $X$ in terms of attribute set $P \subseteq A$ is defined as:

$$ POS_P(X) = \bigcup \{\underline{appr}_P E_{\{d\}}(x) | x \in X\}. \quad (3) $$

Others region definitions such as negative, boundary and rough set properties can be obtained from the original rough set theory research [19,20].

Apart from using equivalence relations to define a rough set in complete information systems, there are also numerous studies [10,13,16,17,18] that deal with incomplete or imperfect information systems in which data are not described by precise and crisp values.

## 2.2 Fuzzy Rough Set

A classical (crisp) set is normally defined as a collection of elements $x \in X$ that can be finite, countable or over countable. Each single element can either belong to or not belong to a set $X' \subseteq X$. For a fuzzy set, a characteristic function allows various degrees of membership for elements of a given set. Then a fuzzy set $\mathcal{X}$ in $X$ is a set of ordered pairs [31]:

$$ \mathcal{X} = \{(x, \mu_{\mathcal{X}}(x)) | x \in X\}, \quad (4) $$

where $\mu_{\mathcal{X}}(x) \in [0, 1]$ is called the membership function or grade of membership (also the degree of compatibility or degree of truth) of $x$ in $\mathcal{X}$.

In crisp sets, for two special sets $\emptyset$ and $U$ the approximations are simply defined as $\mu_{\underline{appr}_P U}(x) = 1$ and $\mu_{\overline{appr}_P \emptyset}(x) = 0$. Based on the two equivalent definitions, lower and upper approximations may be interpreted as follows: An element $x$ belongs to the lower approximation $\underline{appr}_P X$ if all elements equivalent to $x$ belong to $X$. In other words, $x$ belongs to the lower approximation of $X$ if any element not in $X$ is not equivalent to $x$, namely, $\mu_R(x, y) = 0$. Likewise, $x$ belongs to the upper approximation of $X$ if $\mu_R(x, y) = 1$.

Now the notion of rough sets in crisp sets is extended to included fuzzy sets. Let $\mu_{\mathcal{X}}$ and $\mu_{\mathcal{R}}$ denote the membership functions of set $\mathcal{X}$ and of the set $\{(x, y) \in U \times U | \mathcal{R}(x, y)\}$, respectively. The fuzzy approximation space [21] of fuzzy set $\mathcal{X}$ on $X$ in terms of fuzzy relation $\mathcal{R}(x, y)$ can be defined as follows:

$$ \mu_{\underline{appr}_P} \mathcal{X}(x) = \inf \mathcal{I} \{\mu_{\mathcal{R}}(x, y), \mu_{\mathcal{X}}(y)\}, \quad (5) $$

$$ \mu_{\overline{appr}_P} \mathcal{X}(x) = \sup \mathcal{T} \{\mu_{\mathcal{R}}(x, y), \mu_{\mathcal{X}}(y)\}. \quad (6) $$
where $\mathcal{I}$ and $\mathcal{T}$ are fuzzy implicators <fnref n="1" /> and triangular norms (t-norm) <fnref n="2" />, respectively.

From the above equations, it is noticed that membership of lower and upper approximation for one instance strongly depend on only one other instance since they are defined by minimum and maximum. To soften this definition, Cornelis et. al. [5] suggest a fuzzy-rough set definition using order weight averaging (OWA) aggregation [29]. An OWA operator $F_W$ is a mapping $F: R^n \to R$ using weight vector $W = \langle w_1, ..., w_n \rangle$ such that

$$F_W(a_1, \dots, a_n) = \sum_{i=1}^{n} w_i b_i, \qquad (7)$$

where $b_i$ is the $i^{th}$ largest of the values in $\{a_1, \dots, a_n\}$.

An OWA operator is bounded by the minimum and maximum of vector $W$. It thus can soften minimum and maximum operators. Suppose that there are two vectors $W^{min} = \langle w_1^{min}, \dots, w_n^{min} \rangle$ and $W^{max} = \langle w_1^{max}, \dots, w_n^{max} \rangle$, where $w_1^{min} \le \dots \le w_n^{min}$ and $w_1^{max} \ge \dots \ge w_n^{max}$, we can define OWA fuzzy rough sets as follows:

$$\mu_{\underline{appr}}_{\mathcal{R}}(x) = F_{W^{min}}(\mathcal{I}\{\mu_{\mathcal{R}}(x, y), \mu_{\mathcal{X}}(y)\}), \qquad (8)$$

$$\mu_{\overline{appr}}_{\mathcal{R}}(x) = F_{W^{max}}(\mathcal{T}\{\mu_{\mathcal{R}}(x, y), \mu_{\mathcal{X}}(y)\}). \qquad (9)$$

In fact, the definition of a fuzzy relation on an attribute set depends on individual systems. One example of defining relation methods to deal with incomplete information systems is shown in [15]. Further investigation of the combination between fuzzy and rough sets can be found in [5,8,9,21,30].

## 3 Rough Selection and Problem Stated

Significant research on instance selection using rough sets model was introduced the work of Caballero et al. [3]. For this purpose, the authors calculated approximations and positive regions for each class in training sets. Instances for training phases are then selected in two ways. In the first method, they try to delete all instances in the boundary. The second employs a nearest neighbourhood algorithm to relabel instances in the boundary region. The issues associated with these approaches were raised in Jensen and Cornelis' study [12]. Another possible limitation is that they just deal with crisp sets of attributes in decision tables.

<footnote n="1">¹ A fuzzy implicator is a function $\mathcal{I}: [0, 1] \times [0, 1] \to [0, 1]$ which satisfies the following properties: $\mathcal{I}(0, 0) = 1$; $\mathcal{T}(1, a) = a$; $\mathcal{I}$ is decreasing in the first argument; $\mathcal{I}$ is increasing in the second argument.</footnote>

<footnote n="2">² A t-norm is a function $\mathcal{T}: [0, 1] \times [0, 1] \to [0, 1]$ which satisfies the following properties: Commutativity: $T(a, b) = T(b, a)$; Monotonicity: $T(a, b) \le T(c, d)$ if $a \le c$ and $b \le d$; Associativity: $T(a, T(b, c)) = T(T(a, b), c)$; The number 1 acts as an identity element: $T(a, 1) = a$</footnote>
To deal with the problems identified in the study of Caballero et al., an approach called Fuzzy-Rough Instance Selection (FRIS) has been proposed [12]. The notion of this approach is using memberships of positive regions to determine which instances should be kept and which instances should be discarded for training. First, the method calculates relations among instances and then measures quality of each instance by its positive region membership. An instance will be removed if the membership degree is less than a specified threshold. For the same purpose, another approach of measuring quality of instances was introduced in [27]. Such approaches will undoubtedly refine the positive regions with quality instances.

However, one limitation to be considered is that there is only one threshold used for all classes. In some applications such as medical disease prediction, it is not uncommon to be interested in some specific groups, disease groups for example, rather than the others (healthy groups). When only one threshold is used, once noise is eliminated, valuable instances in interesting classes also disappear.

Some approaches use the advantage of fuzzy-rough to qualify instance sets formed by SMOTE [22,23,28]. Therefore, the issues above may be avoided. The steps of these methodologies can be summarized as follows:

1. Using fuzzy-rough sets to calculate the qualities of every instance and choose instances having high quality.

2. Using SMOTE to create artificial instances from the first step and add to datasets

3. Using fuzzy-rough sets to eliminate low quality instances from datasets after the second step.

There are some differences among these methods. In Ramentol et al.'s study named SMOTE-RSB* [22,23], the first step does not appear. In step 3, the algorithm removes only low quality instances in the artificial instance sets. In Verbiest's research [28], quality measurements are different for step 1 and 3. The first measures performance for imbalanced data while the last deals with balanced information. Thus, they call the algorithm FRIPS-SMOTE-FRBPS (hereinafter referred to as FRPSS in this paper). It is also claimed in [28] that FRPSS produces better training sets compared with SMOTE-RSB*.

It appears that FRPSS is an excellent tool for editing training sets. However, applying three instance selections steps may require considerable computation time compared with other methods.

## 4 Multiple Thresholds Fuzzy Rough Instance Selection

As stated in [12], the Fuzzy-Rough Instance Selection (FRIS) algorithm is quite efficient as it eliminates some instances that might affect the positive region membership of the remaining objects. It thus refines the dataset by choosing quality instances for training. However, we also noted some limitations of conventional methods. An example is shown in Figure 1. If we use fuzzy lower approximation as instance qualities and compare these numbers with the threshold
<figure><img src="image_2.png" /><figcaption>Fig. 1. Histogram to show distribution of approximations and average approximations for a dataset</figcaption></figure>

= 0.7 (FRIS method), numerous positive instances (minority class) will be removed while almost all negative instances will be remained. The same problem occurs with FRPSS in which an average of approximations is used as a quality measurement.

Therefore, in this section, using the advantages of fuzzy rough set instance selection approach, we propose an instance selection method which uses different thresholds for different classes of decision tables. By means of this strategy, SMOTE or other balancing techniques may not be necessary.

## 4.1 MFRIS algorithms

In decision table $I = (U, A \cup \{d\})$, let $R_a(x, y)$ and $\mu_{R_a}(x, y)$ denote a fuzzy relation and membership function, respectively, between objects $x, y \in U$ on attribute $a \in A$. Then the membership function of the relation on an attribute set $P \subseteq A$ is defined as:

$$ \mu_{R_P}(x, y) = \mathcal{T}_{a \in P} \mu_{R_a}(x, y), $$

where $\mathcal{T}$ is a t-norm.

In fact, there are many ways to calculate the membership function of a fuzzy relation between two instances. It is also possible to use distance based similarity such as the normalization of Euclidean distance. It depends on the characteristics of each system.
**Data:**

$X_i, Y_j$, minority and majority classes;

$t_X, t_Y$, the selection thresholds for minority and majority classes.

**Result:** Decision table $(S, A \cup \{d\})$

Calculate membership degree of the relation among objects: $\mu_R(x, y)$;

Calculate membership degree of the approximations for each class;

Calculate the quality measurement of all instances for their classes;

$S \leftarrow \emptyset$;

for $x \in X_i$ do

$$ \left| \begin{array}{ll} \text{if } & \mu_{X_i}^R(x) \geq t_X \text{ then} \\ & S \leftarrow S \cup \{x\}; \\ \text{end} & \end{array} \right. $$

end

for $x \in Y_j$ do

$$ \left| \begin{array}{ll} \text{if } & \mu_{Y_j}^R(x) \geq t_Y \text{ then} \\ & S \leftarrow S \cup \{x\}; \\ \text{end} & \end{array} \right. $$

end

<caption>**Fig. 2. MFRIS1** - The first algorithm: Choosing or eliminating instances for both majority and minority classes</caption>

The membership functions of approximation spaces for an object set $X \subseteq U$ on attribute $P \subseteq A$ can be defined by either equations (5) and (6) or (8) and (9). In this study, as mentioned in Section 2, we must note that in the decision table $I = (U, A \cup \{d\})$ is a single label decision table such that $f_d(x)$ is a precise value. In fuzzy sets, for a single label decision table, $\mu_{C_i}(x) > 0$ if $f_d(x) = d_i$ and $\mu_{C_i}(x) = 0$, otherwise.

In selecting learning instances, we first define a function to measure the quality of an instance for each class $X$ in terms of relation $\mathcal{R}$:

$$ \mu_X^R(x) : \alpha \mu_{\underline{appr}_R} \chi(x) + (1 - \alpha) \mu_{\overline{appr}_R} \chi(x), \quad (10) $$

where $\alpha \in [0, 1]$, $\mathcal{X}$ is the fuzzy set on $X$. In the above equation, note that $x$ may or may not belong to $X$. This is where this approach differs from other approaches. When comparing $\mu_X^R(x)$ for $x \in X$, we can assume that $\alpha = 1$ as in the study of Jensen et al. [12] and $\alpha = 0.5$ as in the approach of Verbiest et al. [28].

Now, let $t_X$ and $t_Y$ be the selection thresholds for minority and majority classes, $\mathcal{X}_i$ and $\mathcal{Y}_j$ be the fuzzy sets on minority class $X_i$ and majority classes $Y_j$, respectively, $\mathcal{R}$ denotes the fuzzy relation on the universal. The set of instances selected for the training phase can be defined as:

$$ S = [\cup_i\{x \in X_i | \mu_{X_i}^R(x) \geq t_X\}] \cup [\cup_j\{x \in Y_j | \mu_{Y_j}^R(x) \geq t_Y\}]. \quad (11) $$

Then, the algorithm to select instances can be described as shown in Figure 2.
In the first algorithm, depending on labels of instances, the quality measurement of every instance on their classes will be compared with the threshold for minority or majority thresholds. This means that an instance will be deleted from a training set if it is low quality even it is in minority classes. The different thresholds may help us to keep more instance in minority classes if necessary.

In addition, we also propose the second algorithm to select instances. The notion of this method is keeping all instances in minority classes while removing or relabelling some instances in majority classes. To describe the algorithm, we first introduce some definitions.

Let $X, Y$ be the family of minority and majority classes respectively. Let $\mathcal{X}_i, \mathcal{Y}_j$ denote fuzzy sets on $X_i, Y_j$, respectively. Thresholds for minority and majority classes are $t_X$ and $t_Y$, respectively. The set of instances for which labels could be changed can be defined as

$$S_{Y \to X} = \bigcup_i \{x \in Y_i | \mu_{\mathcal{Y}_i}^{\mathcal{R}}(x) < t_Y \wedge \sup_j \{\mu_{\mathcal{X}_j}^{\mathcal{R}}(x) - t_X\} \ge 0\}. \quad (12)$$

From the above definition, there are some instances in majority classes whose labels could be changed to a minority class label. The possible minority class set can be determined as follows:

$$Z(x) = \bigcup_i \{X_i | \mu_{\mathcal{X}_i}^{\mathcal{R}}(x) \ge t_X, x \in S_{Y \to X}\}. \quad (13)$$

Then we can re-calculate the class membership functions of an instance $x \in S_{Y \to X} \cap Y_n$ as $\mu_{\mathcal{X}_m}(x) = \mu_{\mathcal{Y}_n}(x); \mu_{\mathcal{Y}_n}(x) = 0$, where $m$ is defined to satisfy: $X_m \in Z(x); \mu_{\mathcal{X}_m}^{\mathcal{R}}(x) \ge \mu_{\mathcal{X}_i}^{\mathcal{R}}(x), \forall i, X_i \in Z(x)$.

Finally, the selected instances for the training phase can be defined as:

$$S = (\bigcup_i X_i) \cup (\bigcup_i \{x \in Y_i | \mu_{Y_i}^{\mathcal{R}}(x) \ge t_Y\}) \cup S_{Y \to X}. \quad (14)$$

The algorithm is then described as shown in Figure 3.

## 4.2 Experiment

Experiments were conducted on datasets from KEEL dataset repository [1] and UCI Machine Learning Repository [2] and our lifestyle-diseases dataset. The dataset lifestyle-diseases is raw data of collected health factors such as age, BMI, and waist circumference,... and the outcomes are related to lifestyle diseases. Datasets and their properties are shown in Table 1. In this table, “IR” shows imbalance rates for the number of instances between majority and minority classes.

Each dataset is divided into three parts for both minority and majority classes. For each train-test, we used two parts to form training data and the rest were used as testing instances.
**Data:**

$X, Y$, the family set of minority and majority classes

$t_X, t_Y$, the selection thresholds for minority and majority classes

**Result:** Decision table $(S, A \cup \{d\})$

Calculate membership degree of the relation among objects: $\mu_R(x, y)$;

Calculate membership degree of the approximations for each class;

Calculate quality measurement of all instances in the majority classes for all classes;

$S_Y \leftarrow \emptyset$;

$S_{Y \to X} \leftarrow \emptyset$;

for $x \in Y_i$ do

if $\mu_{Y_i}^R(x) \ge t_Y$ then
$S_Y \leftarrow S \cup \{x\}$;

else

$Z(x) \leftarrow \bigcup_j \{X_j | \mu_{X_j}^R(x) \ge t_X\}$;
if $Z(x) \ne \emptyset$ then

Find $X_m \in X$ satisfy $\mu_{X_m}^R(x) = \max_j \{\mu_{X_j}^R(x) | X_j \in Z(x)\}$;

if a pair occurs then choose the first $X_m$;

$\mu_{X_m}(x) \leftarrow \mu_{Y_i}(x)$;

$\mu_{Y_i}(x) \leftarrow 0$;

$S_{Y \to X} \leftarrow S_{Y \to X} \cup \{x\}$;

end

end

end

$S \leftarrow \bigcup_i \{X_i \in X\} \cup S_Y \cup S_{Y \to X}$;

**Fig. 3. MFRIS2** - The second algorithm: Choosing, removing or relabelling instances for majority classes

Experiments were conducted using R programming<fnref n="3" /> running on RStudio<fnref n="4" />.

Training data are first cleaned by instance selection algorithms. For rough set based instance selection methods, we measure fuzzy tolerance relations on an attribute by membership function $\mu_{R_a}(x, y) = 1 - \frac{|f_a(x) - f_a(y)|}{l(a)}$, where $l(a)$ is the range of value domain on $a$. In this study, we do not deal with missing values. However, it is possible to define $\mu_{R_a}(x, y)$ for incomplete information by following the study in [16].

The fuzzy tolerance relation on a set of attributes can be calculated by a combination of the relation on each attribute by Lukasiewicz's t-norm $\mathcal{T} = \max(x_2 + x_1 - 1, 0)$. For those which use OWA fuzzy rough sets, the OWA operators are set as follows:

$W^{min} = \langle w_i^{min} \rangle = w_{n+1-i}^{min} = \frac{2^{3-i}}{2^3-1}$ for $i = 1, 2, 3$ and $0$ for $i = 4, ..., n$,

$W^{max} = \langle w_i^{max} \rangle = w_i^{max} = \frac{2^{3-i}}{2^3-1}$ for $i = 1, 2, 3$ and $0$ for $i = 4, ..., n$,
where $n$ is number of instances in datasets.

<footnote n="3">³ http://www.r-project.org/</footnote>

<footnote n="4">⁴ http://www.rstudio.com/</footnote>
<table><caption>**Table 1.** Datasets for experiments</caption><thead><tr><th>Datasets</th><th>Inst...</th><th>Feat...</th><th>IR</th><th>Datasets</th><th>Inst...</th><th>Feat...</th><th>IR</th></tr></thead><tbody><tr><td>pima</td><td>768</td><td>8</td><td>1.87</td><td>shuttle-c0-vs-c4</td><td>1829</td><td>9</td><td>13.87</td></tr><tr><td>haberman</td><td>305</td><td>3</td><td>2.77</td><td>page-blocks-1-3_vs_4</td><td>472</td><td>10</td><td>15.86</td></tr><tr><td>liver_man</td><td>439</td><td>9</td><td>2.78</td><td>yeast4</td><td>1484</td><td>8</td><td>28.10</td></tr><tr><td>vehicle1</td><td>846</td><td>18</td><td>2.90</td><td>winequality-red-4</td><td>1599</td><td>11</td><td>29.17</td></tr><tr><td>blood_transfusion</td><td>748</td><td>4</td><td>3.20</td><td>ozone_one_hr</td><td>1848</td><td>72</td><td>31.42</td></tr><tr><td>life-style-diseases</td><td>4819</td><td>13</td><td>4.04</td><td>yeast5</td><td>1484</td><td>8</td><td>32.73</td></tr><tr><td>segment0</td><td>2308</td><td>19</td><td>6.02</td><td>ecoli-0-1-3-7_vs_2-6</td><td>280</td><td>7</td><td>39.00</td></tr><tr><td>page-blocks0</td><td>5472</td><td>10</td><td>8.79</td><td>yeast6</td><td>1484</td><td>8</td><td>41.40</td></tr><tr><td>yeast-2_vs_4</td><td>514</td><td>8</td><td>9.08</td><td>mammography</td><td>11183</td><td>6</td><td>42.01</td></tr><tr><td>vowel0</td><td>988</td><td>13</td><td>9.98</td><td>winequality-red-8_vs_6-7</td><td>855</td><td>11</td><td>46.50</td></tr><tr><td>glass2</td><td>214</td><td>9</td><td>11.59</td><td>winequality-white-3-9_vs_5</td><td>1482</td><td>11</td><td>58.28</td></tr><tr><td>ozone_eight_hr</td><td>1847</td><td>72</td><td>13.43</td><td>winequality-red-3_vs_5</td><td>691</td><td>11</td><td>68.10</td></tr></tbody></table>

<table><caption>**Table 2.** Confusion matrix for a two-class problem</caption><thead><tr><th></th><th>Positive prediction</th><th>Negative prediction</th></tr></thead><tbody><tr><th>Positive class</th><td>True Positive (TP)</td><td>False Negative (FN)</td></tr><tr><th>Negative class</th><td>False Positive (FP)</td><td>True Negative (TN)</td></tr></tbody></table>

For our approach, we also use OWA fuzzy rough set with above operators and set $\alpha = 0.5$ to calculate $\mu_X^R$. That is the same as instances quality measurement in FRIPS-SMOTE-FRBPS in terms of distinguishing instances quality.

Next, selected data are used for training using several classification techniques including k-Nearest Neighbours [7], Support Vector Machine [6], naive Bayes [24] and Decision Trees [26]. It should be noticed that these data classification toolboxes are available within R data mining packages.

To evaluate classification performance in different cases where classifier techniques are used as well as instance selection methods, we use the confusion matrix (Table 2) to calculate the Area Under the ROC Curver(AUC) [11]. AUC provides a single measure for comparing classification on average.

$$AUC = \frac{1}{2} \left( \frac{TP}{TP + FN} + \frac{TN}{TN + FP} \right). \qquad (15)$$

For comparison, we try to obtain the best performance for each instance selection algorithm. FRPSS suggests its own method of threshold optimization. For FRIS and MFRISs, instance measurements are quantized into a set of a limited number of elements. We then use this set as threshold candidates in order to find out if there is a significant increase in performance.

In general, from the average AUC shown in Table 3, we can see that after using the modification of rough set instances selection (both MFRIS1 and MRIS2), the classification performance increases noticeably in almost datasets. It seems that the use of multiple thresholds is more effective with the increase of IR value. Although the original fuzzy rough instance selection approach does
<table><caption>**Table 3.** Average AUC archived by the different classifiers for each instance selection method (bold numbers show the highest)</caption><thead><tr><th>Datasets</th><th>NONE</th><th>FRIS</th><th>FRPSS</th><th>MFRIS1</th><th>MFRIS2</th></tr></thead><tbody><tr><td>pima</td><td>0.7031</td><td>0.7099</td><td>0.6967</td><td><strong>0.7462</strong></td><td>0.7303</td></tr><tr><td>haberman</td><td>0.5559</td><td>0.5795</td><td>0.6292</td><td><strong>0.6565</strong></td><td>0.6195</td></tr><tr><td>liver_man</td><td>0.5673</td><td>0.5903</td><td>0.6665</td><td><strong>0.7078</strong></td><td>0.6991</td></tr><tr><td>vehicle1</td><td>0.6478</td><td>0.6545</td><td>0.6931</td><td>0.7357</td><td><strong>0.7387</strong></td></tr><tr><td>blood_transfusion</td><td>0.5802</td><td>0.6084</td><td>0.6290</td><td><strong>0.6845</strong></td><td>0.6765</td></tr><tr><td>life-style-diseases</td><td>0.5537</td><td>0.5794</td><td>0.5729</td><td><strong>0.6055</strong></td><td>0.5629</td></tr><tr><td>segment0</td><td>0.8645</td><td>0.8769</td><td>0.8638</td><td><strong>0.9466</strong></td><td>0.9058</td></tr><tr><td>page-blocks0</td><td>0.8177</td><td>0.8433</td><td>0.8650</td><td><strong>0.8963</strong></td><td>0.8870</td></tr><tr><td>yeast-2_vs_4</td><td>0.7892</td><td>0.7958</td><td>0.7678</td><td><strong>0.9102</strong></td><td>0.8856</td></tr><tr><td>vowel0</td><td>0.8996</td><td>0.8443</td><td>0.9482</td><td><strong>0.9556</strong></td><td>0.9498</td></tr><tr><td>glass2</td><td>0.5305</td><td>0.5489</td><td>0.6082</td><td><strong>0.7291</strong></td><td>0.6918</td></tr><tr><td>ozone_eight_hr</td><td>0.5738</td><td>0.5738</td><td><strong>0.7325</strong></td><td>0.5738</td><td>0.5738</td></tr><tr><td>shuttle-c0-vs-c4</td><td>0.9947</td><td>0.6166</td><td>0.9947</td><td><strong>0.9987</strong></td><td>0.9983</td></tr><tr><td>page-blocks-1-3_vs_4</td><td>0.7996</td><td>0.8048</td><td>0.8059</td><td><strong>0.9187</strong></td><td>0.9046</td></tr><tr><td>yeast4</td><td>0.5710</td><td>0.6020</td><td>0.8069</td><td><strong>0.8416</strong></td><td>0.7924</td></tr><tr><td>winequality-red-4</td><td>0.5090</td><td>0.5094</td><td>0.6232</td><td>0.6621</td><td><strong>0.6785</strong></td></tr><tr><td>ozone_one_hr</td><td>0.5659</td><td>0.5659</td><td><strong>0.7485</strong></td><td>0.5659</td><td>0.5659</td></tr><tr><td>yeast5</td><td>0.8081</td><td>0.8081</td><td>0.8736</td><td><strong>0.9759</strong></td><td>0.9421</td></tr><tr><td>ecoli-0-1-3-7_vs_2-6</td><td>0.6487</td><td>0.6519</td><td>0.5625</td><td>0.7766</td><td><strong>0.8593</strong></td></tr><tr><td>yeast6</td><td>0.6634</td><td>0.6879</td><td>0.8019</td><td><strong>0.8996</strong></td><td>0.8587</td></tr><tr><td>mammography</td><td>0.7559</td><td>0.7735</td><td>0.7790</td><td><strong>0.8771</strong></td><td>0.7587</td></tr><tr><td>winequality-red-8_vs_6-7</td><td>0.5085</td><td>0.5102</td><td>0.5896</td><td>0.6661</td><td><strong>0.7278</strong></td></tr><tr><td>winequality-white-3-9_vs_5</td><td>0.5310</td><td>0.5318</td><td>0.5715</td><td>0.7111</td><td><strong>0.7464</strong></td></tr><tr><td>winequality-red-3_vs_5</td><td>0.5252</td><td>0.5128</td><td>0.5385</td><td>0.6481</td><td><strong>0.8376</strong></td></tr></tbody></table>

not deal with imbalanced data, the increase in performance of MFRISs proves that it is possible to use fuzzy rough sets based instance selection with some criteria to overcome the imbalance issue.

Although less consuming time may be an advantage, besides high performance archived, we do not show the time measurements in this version. This is due to the different criteria in choosing thresholds. FRISS, as stated before, has its own method to optimize thresholds. It considers the unique set of all instance measurements as the threshold candidates. This means that the algorithm has to train and test numerous times (nearly as many as the cardinality of training sets). In FRIS and MFRISs, on the other hand, we tried in limited times.

To a comparison of the two algorithms using our approach, MFRIS1 archives better performance for most datasets even though MFRIS2 still improves performance. In the second algorithm, all instances in minority classes are chosen. If such instances do not have sufficient quality, an edited training set from MFRIS2 cannot be better than that from MFRIS1.

When comparing the classifiers, in general, we can see the differences among them in terms of performance evaluation when there is no instance selection method employed (Figure 4, transparency color bars). SVM seems to be sensitive
<figure><img src="image_3.png" /><figcaption>**Fig. 4.** Comparing average of AUC (on all datasets) and increase on average of AUC using MFRISs for different classifiers</figcaption></figure>

<table><caption>**Table 4.** Comparing the thresholds for MFRIS1 that give the highest improvement of each classifier on mammography dataset (the first number is for minority)</caption><thead><tr><th>Training sets</th><th>kNN</th><th>SVM</th><th>nBayes</th><th>DecisionTree</th></tr></thead><tbody><tr><td>fold2_3</td><td>0.5090 0.5966</td><td>0.5300 0.6483</td><td>0.5000 0.5966</td><td>0.5090 0.5966</td></tr><tr><td>fold1_3</td><td>0.5020 0.5841</td><td>0.5220 0.6322</td><td>0.4540 0.5841</td><td>0.5090 0.6322</td></tr><tr><td>fold1_2</td><td>0.5100 0.5911</td><td>0.5280 0.6517</td><td>0.4040 0.5911</td><td>0.5210 0.6517</td></tr></tbody></table>

to imbalance issues, while Bayes provides better performance. This means that for a different type of dataset with respect to imbalance issues, a different choice of classifier may be required.

Figure 4 also illustrates the difference in performance increase (dark color bars). SVM does not show a good result for imbalanced data, nevertheless, its performance improves to a greater extent than the rest. In other words, naive Bayes receives less support from instance selection methods. However, the performance of all classifiers generally improves.

Obviously, optimized thresholds are various among classifiers and among datasets. Table 4 shows that SVM needs to remove more instances by using smaller thresholds in comparing with other classifiers. In addition, Table 5 suggests that for some datasets, it is necessary to edit majority classes by discarding many instances (high majority thresholds) while moving some instances to minority classes.
<table><caption>**Table 5.** Comparing the thresholds for MFRIS2 that the highest improvement of each classifiers on winequality-red-8 vs.6-7 dataset (the first number is for minority)</caption><thead><tr><th>Training sets</th><th>kNN</th><th>SVM</th><th>nBayes</th><th>DecisionTree</th></tr></thead><tbody><tr><td>fold2_3</td><td>0.2560 0.9075</td><td>0.2560 0.9602</td><td>0.1250 0.9360</td><td>0.2140 0.9602</td></tr><tr><td>fold1_3</td><td>0.2130 0.9085</td><td>0.2130 0.8758</td><td>0.1600 0.9085</td><td>0.1840 0.8758</td></tr><tr><td>fold1_2</td><td>0.0000 0.7619</td><td>0.1690 0.9575</td><td>0.0000 0.9082</td><td>0.1440 0.9082</td></tr></tbody></table>

## 5 Conclusion and future work

In this paper, we presented a modification of fuzzy-rough instance selection using multiple thresholds. Retaining the advantages of fuzzy-rough sets when selecting instances, it can measure the quality of instances for training phase in order to select valuable instances. It also deals with imbalanced data by using different thresholds between minority and majority classes.

Some experiments show that these approaches can improve performance considerably. They can also be competitive with state-of-the-art methods. Furthermore, this notion can potentially applied to modify current rough set based learning methods in order to eliminate the instance selection step.

This paper mainly proves the robustness of using multiple thresholds in fuzzy-rough sets based instance selection approach. No algorithm for selecting valuable thresholds is suggested. Therefore, in further research we intend to study a method of optimizing thresholds and will endeavour to discover a better instance quality measurement function.

## References

1. ALCALA-FDEZ, J., FERNANDEZ, A., LUENGO, J., DERRAC, J., AND GARCIA, S. Keel data-mining software tool: Data set repository, integration of algorithms and experimental analysis framework. *Multiple-Valued Logic and Soft Computing* 17, 2-3 (2011), 255-287.

2. BACHE, K., AND LICHMAN, M. UCI Machine Learning Repository, 2013.

3. CABALLERO, Y., BELLO, R., ALVAREZ, D., GAREIA, M. M., AND PIZANO, Y. Improving the k-nn method: Rough set in edit training set. In *Professional Practice in Artificial Intelligence*, J. Debenham, Ed., vol. 218 of IFIP International Federation for Information Processing. Springer US, 2006, pp. 21-30.

4. CHAWLA, N. V., BOWYER, K. W., HALL, L. O., AND KEGELMEYER, W. P. Smote: Synthetic minority over-sampling technique. *Journal of Artificial Intelligence Research* 16 (2002), 321-357.

5. CORNELIS, C., VERBIEST, N., AND JENSEN, R. Ordered weighted average based fuzzy rough sets. In *Rough Set and Knowledge Technology - 5th International Conference, RSKT 2010, Beijing, China, October 15-17, 2010. Proceedings* (2010), pp. 78-85.

6. CORTES, C., AND VAPNIK, V. Support-vector networks. *Machine Learning* 20, 3 (1995), 273-297.

7. COVER, T., AND HART, P. Nearest neighbor pattern classification. *IEEE Trans. Inf. Theor.* 13, 1 (Sept. 1967), 21-27.
8. DUBOIS, D., AND PRADE, H. Rough fuzzy sets and fuzzy rough sets. In *International Journal of General Systems*, vol. 17. 1990, pp. 191–209.

9. DUBOIS, D., AND PRADE, H. Putting rough sets and fuzzy sets together. In *Intelligent Decision Support*, R. Slowinski, Ed., vol. 11 of *Theory and Decision Library*. Springer Netherlands, 1992, pp. 203–232.

10. GRZYMALA-BUSSE, J. W., CLARK, P. G., AND KUEHNHAUSEN, M. Generalized probabilistic approximations of incomplete data. *International Journal of Approximate Reasoning* 55, 1, Part 2 (2014), 180–196. Special issue on Decision-Theoretic Rough Sets.

11. HUANG, J., AND LING, C. Using auc and accuracy in evaluating learning algorithms. *Knowledge and Data Engineering, IEEE Transactions on* 17, 3 (March 2005), 299–310.

12. JENSEN, R., AND CORNELIS, C. Fuzzy-rough instance selection. In *Fuzzy Systems (FUZZ), 2010 IEEE International Conference on* (July 2010), pp. 1–7.

13. KRYSZKIEWICZ, M. Rough set approach to incomplete information systems. *Inf. Sci.* 112, 1-4 (Dec. 1998), 39–49.

14. LOPEZ, V., FERNANDEZ, A., GARCIA, S., PALADE, V., AND HERRERA, F. An insight into classification with imbalanced data: Empirical results and current trends on using data intrinsic characteristics. *Information Sciences* 250, 0 (2013), 113–141.

15. NGUYEN, D. V., YAMADA, K., AND UNEHARA, M. Extended tolerance relation to define a new rough set model in incomplete information systems. *Advances in Fuzzy Systems* (2013). Article ID 372091.

16. NGUYEN, D. V., YAMADA, K., AND UNEHARA, M. On probability of matching in probabilistiy based rough set definitions. In *IEEE-SMC2013* (Manchester, The UK, 2013), pp. 449–454.

17. NGUYEN, D. V., YAMADA, K., AND UNEHARA, M. Rough set approach with imperfect data based on dempster-shafer theory. *Journal of Advanced Computational Intelligence and Intelligent Informatics* 18, 3 (2014), 280–288.

18. NGUYEN, H. S. Discretization problem for rough sets methods. In *Proceedings of the First International Conference on Rough Sets and Current Trends in Computing* (London, UK, UK, 1998), RSCTC '98, Springer-Verlag, pp. 545–552.

19. PAWLAK, Z. Rough sets. *International Journal of Computer and Information Sciences* 11 (1982), 341–356.

20. PAWLAK, Z. *Rough Sets. Theoretical Aspects of Reasoning about Data*. Kluwer Acad., 1991.

21. RADZIKOWSKA, A. M., AND KERRE, E. E. A comparative study of fuzzy rough sets. *Fuzzy Sets Syst.* 126, 2 (Mar. 2002), 137–155.

22. RAMENTOL, E., CABALLERO, Y., BELLO, R., AND HERRERA, F. SMOTE-RSB*: a hybrid preprocessing approach based on oversampling and undersampling for high imbalanced data-sets using SMOTE and rough sets theory. *Knowl. Inf. Syst.* 33, 2 (2011), 245–265.

23. RAMENTOL, E., VERBIEST, N., BELLO, R., CABALLERO, Y., CORNELIS, C., AND HERRERA, F. Smote-frst: a new resampling method using fuzzy rough set theory. In *World Scientific Proceedings Series on Computer Engineering and Decision Making* (2012), C. Kahraman, E. Kerre, and F. T. Bozbura, Eds., vol. 7, World Scientific, pp. 800–805.

24. RISH, I. An empirical study of the naive bayes classifier. Tech. rep., 2001.

25. SKOWRON, A., AND RAUSZER, C. The discernibility matrices and functions in information systems. In *Intelligent Decision Support*, R. Slowinski, Ed., vol. 11 of *Theory and Decision Library*. Springer Netherlands, 1992, pp. 331–362.
26. STROBL, C., MALLEY, J., AND TUTZ, G. An introduction to recursive partitioning: Rationale, application and characteristics of classification and regression trees, bagging and random forests, 2009.

27. VERBIEST, N., CORNELIS, C., AND HERRERA, F. Frps: A fuzzy rough prototype selection method. *Pattern Recognition* 46, 10 (2013), 2770–2782.

28. VERBIEST, N., RAMENTOL, E., CORNELIS, C., AND HERRERA, F. Preprocessing noisy imbalanced datasets using SMOTE enhanced with fuzzy rough prototype selection. *Appl. Soft Comput.* 22 (2014), 511–517.

29. YAGER, R. R. On ordered weighted averaging aggregation operators in multicriteria decisionmaking. *IEEE Trans. Syst. Man Cybern.* 18, 1 (Jan. 1988), 183–190.

30. YAO, Y. Y. Combination of rough and fuzzy sets based on -level sets. In *Rough sets and data mining: Analysis for imprecise data*. Kluwer Academic, 1997, pp. 301–321.

31. ZIMMERMANN, H.-J. *Fuzzy Set Theory and its Applications*. Springer, 2001.