Comprehensive Detection of Known Attacks Using Integrated Datasets

Authors: chaima  aouiche
bolin  chen
bairong  shen
abdelaziz  aouiche
rajeev k singla
sahraoui  dhelim
Venue: IEEE_org_ICIST_2024_Conference
Published: 2024-08-16 12:01:28

Abstract: Cyberattacks, which are malicious attempts, are continuously increasing, leading to unauthorized data access, services disruptions, and network degradation. Efficient and proactive detection of these attacks is crucial to maintaining the  confidentiality, integrity and availability of the digital environment. In this paper, we present an enhanced and comprehensive approach that cannot only  detects known attacks  but also identifies unknown ones through the integration of three up-to-date datasets and the implementation of  sampling and feature selection techniques. To achieve this, we conducted experiments using two categories of methods: Machine Learning(ML), such as Naive Bayes (NB), Decision Trees (DT), Logistic Regression (LR), K-Nearest Neighbors (KNN), Random Forest (RF), XGBoost, and AdaBoost, and Deep Learning (DL) architectures, including Artificial Neural Networks (ANN), Deep Neural Networks (DNN), Convolutional Neural Networks (CNN), Long Short-Term Memory (LSTM), Gated Recurrent Units (GRU), and Recurrent Neural Networks (RNN). ML models offers easy interpretability, while DL models excel at handling complex patterns. The results from the majority of  models show promising accuracy rates, with 99\% for known attacks, significantly outperforming  previous studies validating the effectiveness of our strategy.

Main paper: 
 i. introduction: 
 The rapid evolution of internet technologies, particularly with the advent of Industry 4.0 and the Internet of Things (IoT), has significantly transformed the digital landscape. These innovations have enabled high levels of connectivity, automation, and data exchange, thereby greatly enhancing daily life. However, the widespread adoption of these technologies has also created new vulnerabilities, which cybercriminals increasingly exploit [1] . This has led to more sophisticated and complex cyberattacks, escalating risks and posing significant threats to individuals, organizations, or governments alike. Consequently, the need for robust cybersecurity measures has never been more critical [2] .Cyberattacks are generally defined as intentional and malicious attempts to damage, disrupt, or gain unauthorized access to digital systems and sensitive information [3] , [4] . While these attacks can be categorized in various ways, we propose that the most practical and applicable approach is to classify them into two main groups: Known Attacks and Unknown Attacks. Known attacks are those that have been publicly disclosed and analyzed by cybersecurity experts. These attacks typically exploit well-documented vulnerabilities and use techniques that can be addressed through established security measures [5] . In contrast, Unknown attacks, also referred to as Zero-Day Attacks, target vulnerabilities that are yet to be discovered by the public and cybersecurity experts, making them particularly dangerous. These attacks are characterized by their lack of detectable signatures in intrusion detection systems, allowing them to bypass traditional security mechanisms [6] .The diversity, dynamic nature and expansion of cyber threats make it a daunting challenge to identify, detect and defend against them. Traditional security solution such as encryption, access control and firewalls, while still necessary, have proven inadequate in identifying these threats [7] , intrusion detection system (IDS) instead demonstrated to be critical to network security, particularly in detecting both known and unknown attacks [8] . In this context, Machine learning(ML) and Deep learning(DL)-based IDS methods have shown great promise in increasing detection accuracy and efficiency, thus overcoming the limitations of conventional methods that are no longer sufficient [5] , [6] , [9] , [10] . Furthermore, the development of benchmarked network traffic based datasets has provided optimum options to accurately identify such attacks [8] [10] . For instance, Shahid et al. [11] used multiple ML algorithms, along with feature selection/dimensionality reduction techniques, to detect different attack classes, using only the CICIDS2017 dataset. Their study found that XGBoost was the best classifier with an accuracy of 99.91%. Ziadoon Kamil et al. [12] also applied both supervised and unsupervised ML algorithms to detect web attacks, showing that The KNN, DT and NB performed very well. Using the CSE-CIC-IDS2018 dataset, Gozde et al. [13] implemented six ML-based IDS coupled with the SMOTE oversampling technique, achieving a high accuracy rate. Similarly, Yung-Chung et al. [14] considered six DL models, with each model recorded accuracy levels above 98%. Recent research have leveraged the most renowned CIC-DDoS2019 datasets to rigorously evaluate their proposed methods. For example, Monika et al. [15] proposed three unsupervised learning algorithms with a GRP feature selection to detect zero-day DDoS attacks, obtaining an accuracy of 94.50%. In a similar effort, Mahrukh et al. [16] analyzed three DL models for both binary and multiclass DDoS attack classification reporting strong accuracy results.Despite the impressive performance achieved by these studies, they may not fully reveal the evolving nature of cyber threats. Fortunately, the usage of combined datasets has been proved to be a powerful approach. In one study, Ali et al. [17] combined the CIC-IDS2017 and CIC-DDoS2019 to enhance the identification of GAN-adversarial DDoS attacks, resulting in a detection ratio between 91.75% and 100%. Another study by Mohamed Selim et al. [18] evaluated four ML algorithms on the CIC-IDS2017, CSE-CICIDS2018, and CIC-DDoS2019, both individually and in combination. Their approach, which utilized a broader and more varied training dataset, resulted in high accuracy for detecting IoV cyberattacks, along with reduced execution times.Compared to the aforementioned studies, this paper aims to present an enhanced and comprehensive approach that can detects known attacks. Firstly, we integrate three up-to-date intrusion datasets together, including:CIC-IDS2017, CSE-CICIDS2018, and CIC-DDoS2019. Secondly, after further preprocessing, we apply the Synthetic Over-sampling Technique (SMOTE) to address class imbalance in the combined dataset. For feature selection, we apply random forest (RF) algorithm, and we optimize hyperparameters using the Tree of Parzen Estimators (TPE). We then evaluate the performance of seven ML models Naive Bayes (NB), Decision Trees (DT), Logistic Regression (LR), K-Nearest Neighbors (KNN), Random Forest (RF), XGBoost, and AdaBoost and five DL models Artificial Neural Networks (ANN), Deep Neural Networks (DNN), Convolutional Neural Networks (CNN), Long Short-Term Memory (LSTM), Gated Recurrent Units (GRU), and Recurrent Neural Networks (RNN)).The rest of this paper is organized as follows: Section 2 details the methods and materials used in this study. Section 3 presents and discusses the experimental results. Section 4 provides the conclusion.The development of this research followed a sequence of four main steps: datasets collection and preprocessing, features selection, models training, validation and evaluation. Additionally, class imbalance handling and parameters optimization, were performed as intermediate steps, as they are particularly important in improving model performance. Each of these steps is further explained in the following sections.The datasets used in this study are: CIC-IDS2017, CSE-CICIDS2018, and CIC-DDoS2019, which collectively provide a comprehensive view of network flow information. These datasets are labelled and open-access developed by the Canadian Institute for Cybersecurity (CIC). They provide realistic network traffic scenarios for testing and evaluating intrusion detection and DDoS detection systems." In this study, all three datasets were subjected to the same preprocessing operations. The following is a summary of the key preprocessing steps:• Initial Data cleaning: First, the individual files within each dataset were merged separately: the eight files from the CIC-IDS2017 dataset, the ten files from the CSE-CIC-IDS2018 dataset, and the eighteen files from the CIC-DDoS2019 dataset. Following this, each merged dataset was then analyzed to detect and remove any 'NaN' values, 'INF' values, and duplicate entries, as well as repeated and unnecessary columns('Unnamed: 0', 'Flow ID', 'Source IP', 'Source Port','Destination IP', 'Destination Port', 'Timestamp' ). Once cleaned, the three datasets were combined, retaining only the common features, which were aligned in the same order across all datasets.• Labels Adjustment and encoding: In the 'Label' column, Labels were standardized by combining similar entries, such as: 'benign' and 'BENIGN' into one 'BE-NIGN' category, 'UDP-lag' and 'UDPLag' into UDP-lag category. Labels representing very small data fractions were removed (Brute Force-XSS:230, SQL Injection: 87, DoS attacks-SlowHTTPTest:4, FTP-BruteForce:1). Then, after grouping the labels into 'benign' and 'malicious'(for all other attack types), they were encoded into numerical values, 0 for 'benign' and 1 for 'malicious' using label encoding.2) Class imbalance using SMOTE: The Synthetic Minority Over-sampling Technique (SMOTE) is an effective method designed to address class imbalance by generating new synthetic samples for the minority class [19] . SMOTE creates new instances within the feature space by following these steps: 1) Nearest Neighbor Selection: For each minority class sample x i , a nearest neighbor x nn is selected. 2) Difference Calculation: The difference between x i andx nn is computed:dif f = x nn -x i (1)3) Scaling the Difference: This difference is scaled by a random factor r between 0 and 1:EQUATION4) Synthetic Sample Creation: The synthetic sample is generated as:EQUATIONMathematically, for each minority class sample x i , where the class label y i =1, the synthetic sample x syn is generated as:EQUATIONIn this study, despite a slight imbalance in the class distribution of newly retained combined-cleaned dataset(Malicious: 11885988, BENIGN:10836635), the SMOTE technique was successfully applied to the training set. This resulted in a balanced class distribution in the training set, with 8320492 instances in both classes (1:8320492, 0:8320492). The training and testing datasets were then normalized using Standard-Scaler for further subsequent analysis.3) Feature selection: Feature selection is the process of identifying the most relevant features, which reduces execution time and improves model performance. There are several techniques available to perform this task including: mutual information, SHAP and principal component analysis with pearson correlation. For this study, feature selection was performed using a random forest classifier, which computed features importance weights [20] . Only the top 20 features with the highest importance score were retained after evaluating model performance with different feature counts. The selected features are presented in "Fig. 1" . To identify known and unknown attacks seven ML and five DL algorithms were chosen. These selection were made due to their proven effectiveness in detecting attacks, as demonstrated in previous studies and discussed below [11] - [18] :• ML algorithms are well-suited for handling complex and large datasets, identifying patterns in nonlinear data, improving model accuracy and processing speed. • DL algorithms are affective at learning and adapting to complex, nonlinear, and long term patterns. 5) Hyperparameter tuning: Hyperparameter optimization is the process of adjusting a model's parameters to enhance its performance. This process aims to find the most effective combination of parameters that maximize the model's accuracy. Traditional methods like grid search and random search systematically explore the hyperparameter space, but these approaches can be computationally expensive. To address these challenges, this study utilizes the Tree of Parzen Estimators (TPE) algorithm, implemented through the Optuna framework. TPE [21] is an advanced Bayesian optimization method that uses probabilistic models to estimate the likelihood of different hyperparameters to select the best parameters. The optimal hyperparameter configurations obtained using 10 trials are listed below: Cross-validation (CV) is a robust technique used to assess the performance and generalizability of machine learning models. It involves dividing the dataset into multiple subsets, or folds, and systematically training the model on some of these folds while validating it on the remaining ones. For k=5, the dataset is split into 5 folds. The model is trained on four folds and validated on the fifth fold. This process is repeated five times, with each fold serving as the validation set once. The results are then averaged to provide a more reliable estimate of the model's overall performance.In this section, we evaluate the effectiveness of our enhanced approach by presenting results using key performance metrics: accuracy, F1 score, precision, and recall. These metrics provide a clear and insightful understanding of how well our method performs.1) Optimal hyperparameter models accuracy: The accuracy of ML/DL models using the optimal hyperparameter values determined by TPE is presented in"TABLE. II". As clearly observed, expect for NB(0.65), AdaBoost(0.96) and GRU(0,99), there was only a slight difference in the models accuracy before and after hyperparameters optimization. This indicates that hyperparameter tuning may not be so important for these models, as they perform with their default parameters. 2) Model's 5-CV accuracy results: As illustrated in "TA-BLE. III" and "Fig. 2 ", all models, except NB, achieved the best performance with minimal standard deviation. Notably, RF, XGBoost, DT and KNN exceeded 99% accuracy, followed closely by LSTM, GRU, DNN and CNN. 3) ML and DL Models Evaluation: ML and DL algorithms were evaluated using the unseen test set. The results obtained are presented in "TABLE. IV" and shown in the confusion matrix depicted in "Fig. 3 ". The matrices indicate that all models were able to correctly classify a significant number of instances. High true positive (TP) and true negative (TN) counts confirmed the ability of the selected models to detect both normal and known attacks. In addition, the highest accuracy, f1-score, precision and recall of RF, XGboost, KNN, GRU and AdaBoost further demonstrated their detection capabilities. NB is the worst classifier suffer the most from overfitting. The mechanism of attacks detection is too complex to be revealed by single dataset. Combining multiple datasets can be leveraged to identify more sophisticated attacks. Recent up-to date datasets represent a powerful source of information about known and new attacks-long term evolution. Our findings underscore the importance of using such datasets in information security tasks including intrusion detections. Our main objective in this paper was not only the detection of known attacks but also unknown ones by excluding four specific attack types (NTP, DNS, SNMP, SSDP and TFTP) from the combined datasets, reserving them for model evaluations. These attack types are existed in CIC-DDoS-2019 day2 but not in day 1. Due to time limitations, we were able to detect only known attacks.CA initialized this study. BC and CA discussed many times to finalize the work plan. BS gave suggestions many times to modify this study. BC and CA conducted the majority of numerical experiments. CA, BC, AO, RK.S and SD drafted the manuscript. Everyone read the manuscript and revised it, and agreed with the final version. 
 
 Appendix:  



Reviews: review: title: title: Manuscript Accept
main_review: main_review: How are the known attacks identified?
What are the concrete contents of three up-to-date datasets?
How do the authors guarantee the universality of the proposed detection mechanism for different attacks?
title: title: This paper presents an enhanced and comprehensive approach. The paper is well written. Here are some comments.
main_review: main_review: Question 1:
Please elaborate on how the integration of the three datasets and the implemented sampling and feature selection techniques contribute to the detection of both known and unknown cyberattacks. What specific challenges were addressed through these techniques?
Question 2:
How do the Machine Learning (ML) models and Deep Learning (DL) architectures compare in terms of their ability to detect known and unknown attacks? Are there specific scenarios or types of attacks where one approach significantly outperforms the other?
Question 3:
The paper reports a 99% accuracy rate for known attacks. Please provide more details on the validation process used to achieve this accuracy. How does this performance compare with state-of-the-art methods in terms of detection rate and false positives?
title: title: This manuscript has a certain degree of innovation and clear simulation figures. It is recommended to accept this paper for publication in IEEE ICIST 2024.
main_review: main_review: This manuscript has a certain degree of innovation and clear simulation figures. Please answer the following review questions.

The paper mentions the integration of three up-to-date datasets and the use of sampling and feature selection techniques. Could you elaborate on the specific reasons why these datasets were chosen and how they contribute to the detection of both known and unknown attacks?

How robust is the proposed approach to detecting unknown attacks, which are often more challenging to identify? What measures were taken to ensure the generalizability of the results to other datasets or attack scenarios?