# An Efficient Fuzzy Clustering-Based Approach for Intrusion Detection

Huu Hoa Nguyen, Nouria Harbi, Jérôme Darmont

► To cite this version:

Huu Hoa Nguyen, Nouria Harbi, Jérôme Darmont. An Efficient Fuzzy Clustering-Based Approach for Intrusion Detection. 15th East-European Conference on Advances and Databases and Information Systems (ADBIS 2011), Sep 2011, Vienna, Austria. pp.117-127. hal-00631499

HAL Id: hal-00631499

https://hal.science/hal-00631499v1

Submitted on 12 Oct 2011

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons CC BY 4.0 - Attribution - International License
# An Efficient Fuzzy Clustering-Based Approach for Intrusion Detection

Huu Hoa Nguyen, Nouria Harbi and Jérôme Darmont

Université de Lyon (ERIC Lyon 2) - France
nhhoa@eric.univ-lyon2.fr, {nouria.harbi, jerome.darmont}@univ-lyon2.fr

**Abstract.** The need to increase accuracy in detecting sophisticated cyber attacks poses a great challenge not only to the research community but also to corporations. So far, many approaches have been proposed to cope with this threat. Among them, data mining has brought on remarkable contributions to the intrusion detection problem. However, the generalization ability of data mining-based methods remains limited, and hence detecting sophisticated attacks remains a tough task. In this thread, we present a novel method based on both clustering and classification for developing an efficient intrusion detection system (IDS). The key idea is to take useful information exploited from fuzzy clustering into account for the process of building an IDS. To this aim, we first present cornerstones to construct additional cluster features for a training set. Then, we come up with an algorithm to generate an IDS based on such cluster features and the original input features. Finally, we experimentally prove that our method outperforms several well-known methods.

**Keywords:** classification, fuzzy clustering, intrusion detection, cyber attack.

## 1 Introduction

In recent years, with the dramatically increasing use of network-based services and the vast spectrum of information technology security breaches, more and more organizational information systems are subject to attack by intruders. Among many approaches proposed in the literature to deal with this threat, data mining brings on a noticeable success to the development of high performance intrusion detection systems (IDSs). The preeminence of such an approach lies in its good generalization abilities to correctly classify (or detect) both known and unknown attacks. However, as an inherent essence, the effectiveness of data mining-based IDSs depends heavily upon the quality of IDS datasets. In practice, IDS datasets are often extracted from raw traces in a chaotic system environment, and hence could hold implicit deficiencies, e.g., the existence of noise in class labels due to mistakes in measurement, and the lack of base features. Moreover, due to the sophisticated characteristics of attacks and the diversification of normal events, different data regions could behave differently, i.e., true class labels could seriously be interlaced.

Such factors pose a great difficulty for inducers to identify appropriate decision boundaries from the input space of IDS datasets. In other words, when the input space
is not robust enough to discriminate class labels, making further treatments from alternative knowledge sources as new supplemental features is highly desirable. To this aim, one common approach is to transform the input space into a higher dimensional space from which data are more separable. New additional features can be found by either manual ways based on prior knowledge or automatic analysis methods (e.g., principle component analysis). However, in a high dimensional input space, finding new relevant features is a tough task that often requires human analyses, but derived features are sometimes not as good as expected. As a result, in practice, one often applies standard dimensional-transformation methods (e.g., polynomial, radial basic function) to application domains where class discrimination is ambiguous and additional features are hard to be identified. Yet, such methods are greatly affected by input parameters and data distribution, thus not always outputting a high performance classifier. In this vision, it is desirable to find additional features in a less complex way so that general-purpose algorithms such as Decision Trees (DT) or Support Vector Machines (SVM) can learn the data more efficiently.

Such a context motivates us to propose a novel approach that treats fuzzy cluster information as additional features. These features are selectively incorporated into the input space for building an efficient IDS. we experimentally show that our solution approach is considerably superior to several well-known methods.

The remainder of this paper is organized as follows. Section 2 presents the problem formulation of our approach, whereas section 3 describes our solution for generating an IDS. Section 4 shows the experimental results we achieved. Section 5 finally gives a conclusion of the method we propose.

## 2 Problem formulation

Clustering aims to organize data into groups (clusters) according to their similarities measured by some concepts. Unlike crisp clustering that crisply assigns each data point to a separate cluster, fuzzy clustering allows each data point to belong to various clusters with different membership degrees (or weights). Fuzzy clusters are expressed by their centers (or centroids) that are simultaneously found in the partitioning process of a fuzzy clustering algorithm. The number of clusters (k) is often inputted as a parameter to a fuzzy clustering algorithm. The $n \times k$ membership matrix $W=\{w_{ij} \in [0,1]\}$ of n data points is found in the fuzzy clustering process. For example, Figure 1 describes the instance space of a training set partitioned into four fuzzy clusters, where membership weights that data point $x_1$ belongs to clusters '1', '2', '3', and '4' are 0.3, 0.14, 0.16, and 0.4, respectively.

Let us first denote $S=\{X,Y\}$ the original training set of n data points $X=\{x_1,...,x_n\}$, where each point $x_i$ is an m-dimensional vector ($x_{i1},...,x_{im}$) and assigned to a label $y_i \in Y$ belonging one of the c classes $\Omega=\{\omega_1, ..., \omega_c\}$. Let $B=\{b_i| b_i=\max(w_{ij}), j=1...k\}$ hold the maximum membership weight of each point $x_i$, and $Z=\{z_i| z_i=\argmax_j(w_{ij}), j=1...k\}$ contains the cluster (symbolic) number assigned to each point $x_i$.

For conciseness in describing the approach, we term two column matrices $Z$ and $B$ as two "basic cluster features". In addition, we name the $j$-th column of the membership matrix ($W$) as $P_j$, and term the columns $P_1, ..., P_k$ as "extended cluster features". We
also term the training set added with cluster features {$X, Z, B, P_1, ..., P_k, Y$} as a "manipulated training set". These notations and terminologies are depicted in Figure 1.

<figure><img src="image_2.png" /><figcaption>Fig. 1. A manipulated training set, resulting from adding cluster features into the input space.</figcaption></figure>

The problem formulation follows: "Given a training set $S=\{X,Y\}$ and an inducer $I$, the goal is to find a high performance classifier induced by $I$ over the $m$ initial features of $S$ and the supplemental cluster features $\{Z, B, P_1, P_2, ..., P_k\}$ resulting from a parameterized-by-$k$ fuzzy clustering based on $X$".

Undoubtedly, fuzzy clustering has a great potential in expressing the latently natural relationships between data points. Here, a question is whether information about fuzzy clusters benefits certain inducing types. Basically, there exist some types of inducers to which fuzzy cluster features are helpful. For example, in the SVM context, the decision boundary often falls into a low density region, but the true boundary might not pass through this region, thus resulting in a poor classifier. However, when supplemented with relevant cluster features, data points in high dimensional spaces can become more uniform and discriminatory, hence avoiding an improper separation across this region. In fact, the crucial factor to the success of SVM lies in a kernel trick that maps the initial input space to a much higher dimensional feature space, where the transformed data are expected to be more separable from a linear hyper-plane function. In order words, while other inducers somewhat find dimensionality a curse, blessing of dimensionality can enable SVM to be more effective. Under such a sense, incorporating relevant cluster features into the input space discernibly benefits SVM inducers.

Another consideration relates to the univariate Decision Tree (DT) setting. Due to its greedy characteristic, the DT inducer examines only one ahead partitioning step for growing child trees, rather than considering deeper partitioning steps that can achieve a better tree. This characteristic can lead to an improper tree-growing termination (e.g., the XOR problem), and thus generate a poor classifier. In this vision, cluster features help the DT inducer to determine splits more properly for tree growing.

# 3 Fuzzy Cluster Feature-based Classification

## 3.1 Cluster Feature Generation and Selection

Basically, cluster features can be generated by any fuzzy clustering algorithm. However, for concreteness, we express cluster features with the fuzzy c-means clustering [6], which typically solves the minimization problem to the objective
function of Formula 1. In a common form, the objective function (Formula 1) reaches to a minimum over W (membership matrix) and V (centroids), by Formulas 2 and 3.

$$f_{obj}(X, W, V) = \sum_{j=1}^{k} \sum_{i=1}^{n} w_{ij}^{\alpha} d(x_i, v_j)^2, \text{ subject to the constrain } \sum_{j=1}^{k} w_{ij} = 1 \quad (1)$$

$$v_j = \left( \sum_{i=1}^{n} (w_{ij})^{\alpha} x_i \right) / \left( \sum_{i=1}^{n} (w_{ij})^{\alpha} \right) \quad (2)$$

$$w_{ij} = \left( \frac{1}{d(x_i, v_j)^2} \right)^{\frac{1}{\alpha-1}} / \left( \sum_{q=1}^{k} \left( \frac{1}{d(x_i, v_q)^2} \right)^{\frac{1}{\alpha-1}} \right) \quad (3)$$

where α is a fuzzy constant and d(xᵢ, vⱼ) is the distance from xᵢ (∈ X) to vⱼ (∈ V)

Fuzzy c-means clustering tries to find the best fit for a fixed value of k, the number of clusters. However, as an essential problem of clustering, determining an appropriate parameter k is a tough task. The most common way to find the reasonable number of clusters is to run the clustering with various values of k ∈ {2,..., kmax} and then use a validity measure (e.g., partition coefficient ) to evaluate cluster fitness.

In our approach, however, we need data to be grouped in a way that reveals helpful information for inducers, not for clustering itself, even though the number of clusters might be wrong. In other words, using validity measures to determine the best number of clusters is not reliable enough to derive good cluster features for classifiers. In such a vision, instead of endeavoring to find the best k with validity measures, we use the over-production method to generate several candidate classifiers for different values of k and then evaluate their performance to determine the best one. Evaluating the performance of candidate classifiers can be based either on a validation set or Cross Validation (CV) method [9]. Thus, a proper value of k is simultaneously found in the process of finding a maximum performance classifier from candidate classifiers.

In addition, the use of cluster features should be examined individually for a concrete inducing type. Intuitively, two basic cluster features (Z, B) are benefic enough for DT inducer, instead of including k extended cluster features (P₁,...,Pₖ). By contrast, in the SVM context, it is applicable to employ either only the basic cluster features (Z, B) or all the cluster features (Z, B, P₁,...,Pₖ) for building a classifier. Another solution that can be applied for any inducing type is to employ feature selection techniques (e.g., filter, wrapper) to pick out high merit features from both m initial input features and all (k+2) cluster features. The objective is to apply feature selection techniques on (m+k+2) features to bring about a smaller but more qualitative feature subset than those only on m initial features. Here, note is that feature selection is simultaneously carried out in the process of building candidate classifiers. In a nutshell, formally, there are three possibilities to incorporate cluster features into the initial features (A₁, ..., Aₘ), i.e., (A₁, ..., Aₘ, Z, B), (A₁, ..., Aₘ, Z, B, P₁, ..., Pₖ), or Feature Selection(A₁, ..., Aₘ, Z, B, P₁, ..., Pₖ).

## 3.2 Algorithm for generating a fuzzy cluster feature-based classifier

Our algorithm for generating a classifier from both initial and cluster features, called CFC, is depicted from Figure 2. Related notations are indicated in Table 1.
<table><caption>**Table 1.** Notations used in Figure 2.</caption><thead><tr><th>Notation</th><th>Description</th></tr></thead><tbody><tr><td>C<sub>k</sub></td><td>A candidate classifier resulting from a clustering with k fuzzy clusters.</td></tr><tr><td>C<sub>k*</sub></td><td>The best classifier among |K| candidate classifiers.</td></tr><tr><td>V<sup>k</sup></td><td>A k × m matrix of k centroids obtained from clustering X into k clusters.</td></tr><tr><td>V<sup>k*</sup></td><td>A k* × m matrix of k* centroids, corresponding to C<sub>k*</sub>.</td></tr><tr><td>W<sup>k</sup></td><td>An n × k membership matrix of n data points x<sub>i</sub> ∈ X, corresponding to V<sup>k</sup>.</td></tr><tr><td>B<sup>k</sup></td><td>A column matrix containing the maximum membership weight of each x<sub>i</sub> ∈ X.</td></tr><tr><td>Z<sup>k</sup></td><td>A column matrix representing the cluster (symbolic) number of each x<sub>i</sub> ∈ X.</td></tr><tr><td>Θ</td><td>A horizontal concatenation operator between two matrices.</td></tr></tbody></table>

**Training phase**

**Input:** S={X, Y}: The original training set

I: a base inducer

K: a predefined integer set representing possible number of clusters

ξ: a feature selection technique that returns a specific feature subset

T: a type to employ features for building classifiers

**Output:** C<sub>k*</sub>, V<sup>k*</sup>

1: X' ← Normalize(X) //Normalize continuous features

2: For each k ∈ K do

3: {W<sup>k</sup>, V<sup>k</sup>} ← FuzzyClustering(X', k)

4: B<sup>k</sup> ← {b<sub>i</sub> | b<sub>i</sub> = max(w<sub>ij</sub>), i = 1...n, j = 1...k}

5: Z<sup>k</sup> ← {z<sub>i</sub> | z<sub>i</sub> = arg max<sub>j</sub> (w<sub>ij</sub>), i = 1...n, j = 1...k}

6: Case //D is a manipulated training set

7: T = 1: D ← (X ⊗ Z<sup>k</sup> ⊗ B<sup>k</sup>) //Initial features & basic cluster features

8: T = 2: D ← (X ⊗ Z<sup>k</sup> ⊗ B<sup>k</sup> ⊗ W<sup>k</sup>) //Initial features & all cluster features

9: T = 3:
F ← ξ(X ⊗ Z<sup>k</sup> ⊗ B<sup>k</sup> ⊗ W<sup>k</sup>, Y) //Apply a feature selection
D ← (X ⊗ Z<sup>k</sup> ⊗ B<sup>k</sup> ⊗ W<sup>k</sup>) [F] //Project data by the derived subset

10: End Case

11: C<sub>k</sub> ← I(D, Y) //Build a classifier, using the manipulated training set D & inducer I

12: Performance(C<sub>k</sub>) ← {Average performance of q-fold CV based on (D, Y) and I}

13: End For

14: C<sub>k*</sub> ← arg max<sub>c<sub>i</sub></sub> Performance(C<sub>k</sub>), k ∈ K //Determine one best classifier

15: Return C<sub>k*</sub>, V<sup>k*</sup>

**Operation phase**

16: For an unlabeled testing instance x:

17: x' ← Normalize(x) //Normalize continuous features

18: Compute membership weights (w<sub>j</sub> | j = 1...k*) that x' belongs to v<sub>j</sub> ∈ V<sup>k*</sup> (Formula 3)

19: b ← max(w<sub>j</sub> | j = 1...k*)

20: z ← arg max<sub>j</sub> (w<sub>j</sub> | j = 1...k*)

21: Label x, by taking cluster features {z, b, w<sub>j</sub>} into account, using C<sub>k*</sub>

<caption>**Fig. 2.** Algorithm CFC.</caption>
The key idea is that, for each clustering with different number of clusters ($k \in K$), the algorithm builds and evaluates a candidate classifier from the training set manipulated with a given feature selection type, by q-Fold Cross Validation [9]. The resulting classifier is the one exhibiting maximum performance.

In the training phase, the algorithm first normalizes continuous features (e.g., by a variance-based spread measure) to avoid the dispersion in different ranges (Line 1). Here, it is noticed that the normalized data ($X'$) is merely for clustering purpose, whereas classifiers are built by using the original data ($X$). In addition, instead of executing clustering with parameter $k$ ranging from 2 to a given $kmax$ value, the algorithm uses a predefined set $K=\{k\}$ to mainly focus on important values of $k$, which can be recognized by experiment or prior knowledge (Line 2). As mentioned in Section 3.1, there are three cases to incorporate cluster features into the initial features. Hence, for general purpose, the algorithm introduces an input parameter $T$ for specifying the way to employ features for building classifiers (Lines 6-10). Subsequently, the algorithm builds and evaluates one candidate classifier for each clustering (Lines 11, 12). Here, note is that evaluating candidate classifiers is based on the averaged performance of $q$-fold stratified cross validation from the manipulated training set. Finally, the algorithm determines one best classifier from $|K|$ candidate classifiers, together with a corresponding centroid set (Lines 14, 15).

In the operation phase, for an unlabeled testing instance $x$, the algorithm first normalizes $x$ in the same way as those applied to the training set. Then, cluster features of $x$ are calculated based on the centroid set $V^{k*}$ (Lines 18-20). Finally, the corresponding features are input to classifier $C_{k*}$ for final prediction (Line 21).

# 4 Experiments

## 4.1 Dataset

Our experiments are conducted on the intrusion detection dataset KDD99 [3]. This dataset was derived from the DARPA dataset, a format of TCPdump files captured from the simulation of normal and attack activities in the network environment of an air-force base, created by MIT's Lincoln Laboratory. The KDD99 dataset comprises 494,021 training instances and 311,029 testing instances. Due to data volume, the research community mostly uses small subsets of the dataset for evaluating IDS methods. Each instance in the dataset represents a network connection, i.e., a sequence of network packets starting and ending at some well defined times, between which data flows to and from a source IP address to a target IP address under some well defined protocol. Such a connection instance is described by a 41-dimensional feature vector and labeled with respect to five classes: Normal, Probe, DoS (denial of service), R2L (remote to local), and U2R (user to root).

To facilitate experiments without losing generality, we only use a smaller set of the KDD99 dataset for the purpose of evaluating and comparing our method to others. In particular, the training and testing sets used in our experiments are made up of 33,016 instances and 169,687 instances that are selectively extracted from the KDD99
training and testing sets, respectively. The principle for forming such reduced sets is to get all instances in each small group (attack type), but only a limited amount of instances in each large group, from both the KDD99 training and testing sets. More explicitly, for forming the reduced training, we randomly select five percent of each large group *Neptune*, *smurf*, and *normal*, while gathering all instances in the remaining groups from the KDD99 training set. For sampling the reduced testing set, we randomly select 50 percent of each large group *Neptune*, *smurf*, and *normal*, whereas collecting all instances in the remaining groups from the KDD99 testing set. Class distribution of these two reduced sets is shown in Table 2.

<table><caption>**Table 2.** Class distribution of the reduced training and testing sets used in experiments.</caption><thead><tr><th>Class</th><th>Training set</th><th>Testing set</th><th>Class</th><th>Training set</th><th>Testing set</th></tr></thead><tbody><tr><td>DoS</td><td>22,867</td><td>118,807</td><td>U2R</td><td>52</td><td>70</td></tr><tr><td>Probe</td><td>4,107</td><td>4,166</td><td>Normal</td><td>4,864</td><td>30,297</td></tr><tr><td>R2L</td><td>1,126</td><td>16,347</td><td><strong>Total</strong></td><td><strong>33,016</strong></td><td><strong>169,687</strong></td></tr></tbody></table>

## 4.2 Experiment Setup

In our experiments, the predefined set K is set to {2, 3, ..., 50}. The convergence criterion (termination tolerance) of fuzzy c-means clustering is set to 10⁻⁶, whereas the fuzzy degree (exponent α in Formulas 1-3) is set to 3. On the other hand, continuous futures are normalized by max_min value ranges [6]. To handle different feature types as well as express different merit contributions of features in the Euclidian space, we calculate distances between data points by the metric proposed in Formula 4.

$$d(x_i, v_j)^2 = \sum_q^m G_q \times d_q(x_{iq}, v_{jq})^2 \quad (4)$$

where $G_q$ is information gain of feature q [5], and

$$d_q(x_{iq}, v_{jq}) = \begin{cases} 1, & \text{if } (x_{iq}, v_{jq} \in \{\text{symbolic}\}) \land (x_{iq} \neq v_{jq}) \text{ or } (x_{iq}, v_{jq} \in \{\text{unknown}\}) \\ |x_{iq} - v_{jq}|, & \text{if } x_{iq}, v_{jq} \in \{\text{continuous}\} \\ \frac{|x_{iq} - v_{jq}|}{t-1}, & \text{if } x_{iq}, v_{jq} \in \{\text{ordinals}\}; t = |\{\text{ordinals}\}| \\ 0, & \text{otherwise,} \end{cases}$$

The base inducers (I) tested in our method are the C4.5 decision tree [5] and the SVM [2] with polynomial and radial basic function kernels. The feature selection technique (ξ) used in this experiment is Correlation-based Feature Subset Evaluation (CfsSubsetEval) with genetic search [7]. CfsSubsetEval evaluates the merit of a feature subset by considering the individual predictive ability of each feature along with the degree of redundancy between them. Those subsets that are highly correlated with the class while having low intercorrelation are preferred.

Candidate classifiers are evaluated by an attack type-based stratified cross validation (q=10 folds). The maximum performance classifier is determined based on overall accuracy (i.e., the ratio of the number of correctly classified instances to the total number of instances in the training set).
## 4.3 Experiment Results

The experimental comparison of our method to other well-known methods is featured in Table 3. All the compared classifiers are built from the same training set and tested on the same testing set as described in Section 4.1. Moreover, Figure 4 depicts True Positive Rates (TPRs) and False Positive Rates (FPRs) of classifiers with respect to each class label, whereas Figure 3 portrays average TPRs and FPRs of classifiers. TPR of a class $ω_c$ is the ratio of “the number of correctly classified instances in the class $ω_c$” to “the total number of instances in the class $ω_c$”. FPR of a class $ω_c$ is the ratio of “the number of instances that do not belong to the class $ω_c$ but are classified as $ω_c$” to “the total number of instances that do not belong to the class $ω_c$”.

To have a wider comparative view, we run our algorithm (CFC) with different settings of two parameters (i.e., *I*: base inducer; *T*: the way to employ cluster features for building classifiers). The results of such runs are listed in Rows 10-18 of Table 3.

As shown in Figures 3 and 4, our method, in general, considerably outperforms the others with respect to TPRs in all five classes and on average. Particularly, CFC classifiers are significantly better than all the others in detecting hard classes (i.e., R2L and U2R). On the other hand, FPRs of CFC classifiers are generally lower than those of the others. Our method also considerably improves the classification ability of base inducers (SVM and DT) in both viewpoints, i.e., applying or not applying feature selection. More concretely, by using the same feature selection technique, the SVM classifier built from the manipulated training set (i.e., CFC(*I*=SVM, *T*=3)) is considerably superior to the SVM classifier built from the original training set (i.e., SVM_FS). Similarly, the performance of CFC(*I*=DT, *T*=3) is considerably better than that DT_FS. This tells that applying a feature selection technique on the manipulated training set produces a higher qualitative feature subset (including base features and cluster features) than that on the original training set.

Regarding the SVM context, although we further test PSVM (Polynomial SVM) with exponent degrees ranging from 2 to 6, its performance remains worse than CFC(PSVM(degree=2), *T*={1,2,3}). On average, CFC(PSVM(degree=2), *T*={1,2,3}) gives a 91.96% TPR (with a 2.2% FPR), whereas PSVM(degree={2,...,6}) produces an 86.84% TPR (with a 3.44% FPR). We also test RSVM (Radial Basic Function SVM) with widths Gamma ranging from 0.1 to 1.0, but its performance still underperforms CFC(RSVM(Gamma=0.1), *T*={1,2,3}). More precisely, on average, RSVM(Gamma={0.1,0.2,...,1}) produces an 86.72% TPR (with a 3.62% FPR), whereas CFC(RSVM(Gamma=0.1), *T*={1,2,3}) gives a 91.15% TPR (with a 2.3% FPR). This tells that cluster features benefit SVM in high dimensionality.

<figure><img src="image_3.png" /><figcaption>**Fig. 3.** Average True Positive and False Positive Rates (%) of classifiers</figcaption></figure>
<table><caption>**Table 3.** True Positive and False Positive rates (%) of classifiers.</caption><thead><tr><th>Classifier</th><th></th><th>DoS</th><th>Probe</th><th>R2L</th><th>U2R</th><th>Normal</th><th>Average</th></tr></thead><tbody><tr><td rowspan="2">1. Boosting</td><td>TP</td><td>95.36</td><td>82.48</td><td>5.51</td><td>35.71</td><td>99.22</td><td>87.05</td></tr><tr><td>FP</td><td>0.44</td><td>0.46</td><td>0.03</td><td>0.01</td><td>15.01</td><td>3.00</td></tr><tr><td rowspan="2">2. Bagging</td><td>TP</td><td>94.72</td><td>80.03</td><td>3.46</td><td>42.86</td><td>98.74</td><td>86.27</td></tr><tr><td>FP</td><td>4.64</td><td>0.41</td><td>0.30</td><td>0.02</td><td>14.18</td><td>5.82</td></tr><tr><td rowspan="2">3. NBTree</td><td>TP</td><td>94.53</td><td>83.32</td><td>9.54</td><td>51.43</td><td>98.08</td><td>86.69</td></tr><tr><td>FP</td><td>0.84</td><td>0.62</td><td>0.60</td><td>0.24</td><td>14.22</td><td>3.20</td></tr><tr><td rowspan="2">4. DT</td><td>TP</td><td>94.72</td><td>78.68</td><td>2.84</td><td>51.43</td><td>98.77</td><td>86.18</td></tr><tr><td>FP</td><td>2.85</td><td>0.57</td><td>0.03</td><td>0.10</td><td>14.96</td><td>4.68</td></tr><tr><td rowspan="2">5. DT_FS</td><td>TP</td><td>94.26</td><td>85.60</td><td>7.11</td><td>38.57</td><td>99.06</td><td>86.49</td></tr><tr><td>FP</td><td>0.82</td><td>0.93</td><td>0.26</td><td>0.05</td><td>14.70</td><td>3.25</td></tr><tr><td rowspan="2">6. PSVM</td><td>TP</td><td>95.14</td><td>84.09</td><td>9.43</td><td>38.57</td><td>97.61</td><td>87.03</td></tr><tr><td>FP</td><td>0.91</td><td>0.52</td><td>0.24</td><td>0.01</td><td>14.56</td><td>3.27</td></tr><tr><td rowspan="2">7. PSVM_FS</td><td>TP</td><td>95.44</td><td>73.84</td><td>8.86</td><td>44.29</td><td>97.65</td><td>86.94</td></tr><tr><td>FP</td><td>3.57</td><td>0.22</td><td>0.29</td><td>0.01</td><td>14.00</td><td>5.03</td></tr><tr><td rowspan="2">8. RSVM</td><td>TP</td><td>95.11</td><td>83.99</td><td>9.51</td><td>38.57</td><td>97.62</td><td>87.02</td></tr><tr><td>FP</td><td>0.98</td><td>0.53</td><td>0.23</td><td>0.01</td><td>14.55</td><td>3.32</td></tr><tr><td rowspan="2">9. RSVM_FS</td><td>TP</td><td>94.98</td><td>81.73</td><td>9.92</td><td>40.00</td><td>98.64</td><td>87.09</td></tr><tr><td>FP</td><td>3.04</td><td>0.55</td><td>0.17</td><td>0.01</td><td>13.75</td><td>4.61</td></tr><tr><td rowspan="2">10. CFC(I=DT, T=1)</td><td>TP</td><td>97.69</td><td>88.65</td><td>25.93</td><td>58.57</td><td>99.62</td><td>90.89</td></tr><tr><td>FP</td><td>0.76</td><td>0.53</td><td>0.02</td><td>0.03</td><td>10.12</td><td>2.35</td></tr><tr><td rowspan="2">11. CFC(I=DT, T=2)</td><td>TP</td><td>97.46</td><td>88.24</td><td>21.47</td><td>60.00</td><td>99.03</td><td>90.18</td></tr><tr><td>FP</td><td>1.45</td><td>0.75</td><td>0.03</td><td>0.04</td><td>10.46</td><td>2.90</td></tr><tr><td rowspan="2">12. CFC(I=DT, T=3)</td><td>TP</td><td>98.30</td><td>90.13</td><td>28.01</td><td>62.86</td><td>99.27</td><td>91.49</td></tr><tr><td>FP</td><td>0.70</td><td>0.65</td><td>0.03</td><td>0.04</td><td>9.24</td><td>2.16</td></tr><tr><td rowspan="2">13. CFC(I=PSVM, T=1)</td><td>TP</td><td>98.42</td><td>92.49</td><td>28.19</td><td>68.57</td><td>99.57</td><td>91.70</td></tr><tr><td>FP</td><td>0.81</td><td>0.68</td><td>0.03</td><td>0.03</td><td>8.93</td><td>2.18</td></tr><tr><td rowspan="2">14. CFC(I=PSVM, T=2)</td><td>TP</td><td>98.12</td><td>92.20</td><td>26.27</td><td>75.71</td><td>99.20</td><td>91.24</td></tr><tr><td>FP</td><td>0.87</td><td>0.48</td><td>0.02</td><td>0.04</td><td>9.70</td><td>2.35</td></tr><tr><td rowspan="2">15. CFC(I=PSVM, T=3)</td><td>TP</td><td>98.83</td><td>94.89</td><td>37.62</td><td>74.29</td><td>99.36</td><td>92.92</td></tr><tr><td>FP</td><td>1.08</td><td>0.71</td><td>0.03</td><td>0.03</td><td>7.31</td><td>2.08</td></tr><tr><td rowspan="2">16. CFC(I=RSVM, T=1)</td><td>TP</td><td>97.42</td><td>95.06</td><td>21.61</td><td>68.57</td><td>99.23</td><td>90.37</td></tr><tr><td>FP</td><td>0.76</td><td>0.61</td><td>0.04</td><td>0.02</td><td>10.65</td><td>2.45</td></tr><tr><td rowspan="2">17. CFC(I=RSVM, T=2)</td><td>TP</td><td>98.15</td><td>91.72</td><td>22.51</td><td>72.86</td><td>99.62</td><td>90.96</td></tr><tr><td>FP</td><td>0.83</td><td>0.58</td><td>0.03</td><td>0.03</td><td>9.96</td><td>2.38</td></tr><tr><td rowspan="2">18. CFC(I=RSVM, T=3)</td><td>TP</td><td>98.22</td><td>94.36</td><td>33.81</td><td>68.57</td><td>99.45</td><td>92.12</td></tr><tr><td>FP</td><td>0.79</td><td>0.70</td><td>0.02</td><td>0.03</td><td>8.39</td><td>2.07</td></tr></tbody></table>

- DT refers to the C4.5 decision tree inducer [5] with established input parameters: pruning method = *pessimistic pruning*, confidence=0.2, and Min(#instances per leaf)=6.

- Boosting uses the AdaBoost [8] with parameters: base inducer=DT, # classifiers=10.

- Bagging uses the Bagging [4] with parameters: base inducer=DT, # classifiers=10.

- PSVM refers to SVM inducer with Polynomial Kernel (exponent degree = 2).

- RSVM refers to SVM inducer with Radial Basic Function Kernel (width gamma = 0.1).

- Classifiers 1-9 are trained on the original training set (without cluster features), where classifiers 5, 7, and 9 employ the feature selection technique (ξ) as described in Section 5.2, whereas classifiers 1-4, 6, and 8 do not apply the feature selection technique (ξ).

- Classifiers 10-18 are built from the CFC algorithm whose base inducers have the same parameter settings as *stand-alone* classifiers 4, 6, and 8.

- The column Average is the average weighted by the number of instances on each class.
<figure><img src="image_4.png" /><figcaption>Fig. 4. True Positive Rates (%) of classifiers on each class.</figcaption></figure>

## 5 Conclusion and Future Work

We propose in this paper a novel method in applying data mining to the intrusion detection problem. The incorporation of cluster features resulting from a fuzzy clustering into the training process is proven to be efficient for enhancing the strength of a base classifier. The tactic to achieve a high performance classifier from a training set supplemented with cluster features is addressed. We experimentally show that, as a whole, our method clearly outperforms all the tested methods. Although the experiments are conducted on the KDD99 IDS dataset, the approach we propose can be generally used to improve classification in other application domains. However, to be more objective in evaluating any data mining solution, our future work will be to test the proposed method on other real datasets. In particular, our current effort is fulfilling a honeypot system for gathering both real intrusion and normal traffic activities. Such a real dataset will then be used to evaluate the method we proposed.

## References

1. Amiria, F., Yousefia, M.R., Lucasa, C., Shakeryb, A., Yazdanib, N.: Mutual Information Based Feature Selection for Intrusion Detection Systems. JNCA, V.34, pp.1184-1199 (2011)

2. Platt, J.: Fast Training of Support Vector Machines using Sequential Minimal Optimization. Advances in Kernel Methods - Support Vector Learning, pp. 185-208, MIT Press (1999)

3. UCI KDD Archive, http://kdd.ics.uci.edu/databases/kddcup99/kddcup99.html

4. Breiman, L.: Bagging Predictors. Machine Learning, Vol. 24(2), pp. 123-140 (1996)

5. Quinlan J.R.: C4.5: Programs for Machine Learning. Morgan Kaufmann, San Mateo (1993)

6. Hoppner, F.: Fuzzy Cluster Analysis. John Wiley & Sons, pp. 37-43 (2000)

7. Hall, M.A.: Correlation-based Feature Subset Selection for Machine Learning. Hamilton, New Zealand (1998)

8. Freund, Y., Schapire R.E.: Experiments with a New Boosting Algorithm. In: Thirteenth International Conference on Machine Learning, San Francisco, pp. 148-156 (1996)

9. Andrew, Y.N.: Preventing Overfitting of Cross-Validation Data. ICML, pp. 245-253 (1997)

10. Gupta K.K., Nath, B., Ramamohanarao, K.: Layered Approach Using Conditional Random Fields for Intrusion Detection. IEEE Trans. Dependable Sec. Comput, 7(1), pp. 35-49 (2010)