dataesr/scientific-paragraphs-categorization huggingface dataset:
Gas Pipeline SCADA dataset The gas pipeline system as the water storage tank system is a laboratory-scale SCADA system from the Mississippi State University  [115]. The gas pipeline system (Figures 6. 4 and 6. 5) includes an airtight pipeline connected to a compressor, a pressure meter and a solenoid-controlled relief valve. The pipeline system attempts to maintain the air pressure in the pipeline using a proportional integral derivative (PID) control scheme  [115]. The gas pipeline system dataset contains the same normal and attacks records categories like in Table  6. 1, previously described in chapter 4. But the payload attribute of both systems differ, though having the same network attributes ( 6. 3).
SCADA Datasets Preparation The Water Storage Tank dataset and the Gas Pipeline dataset went through a preparation process i. e. a Min-Max normalization that sets all the datasets values except the labels in the range [0,1], the splitting of the datasets into a training dataset (60 %), a validation dataset (20 %) and test dataset (20 %) using the hold-out cross-validation  [133] , the balancing of training and validation datasets, and finally the one-hot encoding of all the datasets labels. We also considered a version of the datasets containing two categories of records i. e. , one category of normal records (class 0) and one category of anomalous records (class 0). For the anomalous, records, we have binarized the dataset i. e. , all the attacks records are labeled 1. Figures 6. 4 through 6.
13 represent the data distribution of the water storage tank and gas pipeline SCADA training and test dataset (8-class and binary). We notice that the datasets are very imbalanced, e. g. , the water storage tank training dataset normal records times more than 140 the attacks of category 6 (DoS) records, or the gas pipeline normal records is 110 higher than the class 5 (MFCI) attack records. The test sets are imbalanced as well. To avoid undesirable bias in the training process, the training data subsets undergo a balancing process before usage, but all the test data subsets are kept as is. 6. 6. Performance measures 113.
Performance measures Accuracy, recall (or sensitivity), precision, specificity, f1-score  [63] , along with training time and prediction time ,  [151]  are the measures used to evaluate the performances of classifiers. Positive tuples are the tuples of main interest in a dataset. In the present work, the anomaly detection system is detecting anomalies. Hence, the positive tuples are the abnormal ones. Negative tuples are all the other tuples. Let P be the number of positive tuples and N the number of negative tuples. True Positive (TP) represents positive tuples that were correctly labeled by the classifier. True Negative (TN) represents negative tuples that were correctly labeled by the classifier. False Positive (FP) represents negative tuples that were incorrectly labeled as positive. False Negative (FN) represents positives tuples that were incorrectly labeled as negative.
Accuracy (or recognition rate) The design process of neural networks i. e. defining the parameters and the hyper-parameters consists of trial and error, guessing what king of hidden unit, the width, the depth of the network or which learning rate works well  [59] [128]. The network will then be trained, and the parameters and hyper-parameters are evaluated with a validation set. accuracy = T P + T N P + N For imbalanced We have conducted various tests on the datasets by varying the architecture of the stacked autoencoders to 2, 3, 4 and 5 layers, with various parameters and hyperparameters configuration. Specifically, we did the tests for different training epochs i. e. 1, 10, 20, 30, 40, 50. The Table  6. 8 shows two different architectures used for comparison sake, and their training results in Table  6.
Experimental results for the water storage tank dataset We give here the best results obtained for the SCADA Water Storage Tank dataset. The results using the multiclass version of the water storage tank SCADA dataset were obtained for 30 epochs of training, an architecture with 3 hidden layers, a Leaky ReLU activation function, a cross-entropy loss function, a batch size of 128, a learning rate of 0. 01 a sparsity parameter of 0. 05, a noise level of 30%, and 30, 27, 40 number of nodes for layer 1, layer 2 and layer 3 respectively. After 30 epochs, as shown on Figure  6. 15, the validation loss increases even though the training loss continue to decrease. This is why we choose the results at 30 epochs. On the other hand, the results for the binary Water Storage Tank SCADA dataset is a model trained during 30 epochs, which has 3 hidden layers, a batch size of 128, a learning rate of 0.
01 a sparsity parameter of 0. 05, a noise level of 30%, and 38, 24, 20 number of nodes for layer 1, layer 2 and layer 3 respectively. For the multiclass classification with the Water Storage Tank SCADA dataset (Table  6. 19) , we have a detection rate of 0 % for attack classes 1 and 2. Attack class 6 also have a detection rate of only 35 %. The 36 % of False Positive Rate for normal records detection confirms the previous results, as class 1, 2 and 6 as mis-classified as normal data. The binary classification for the same dataset (Table  6. 12) gives a 66 % detection rate, due to a high False Negative rate  (34 %). This clearly shows that some of the attacks are not detected, thus classified as normal records. 120 CHAPTER 6. Implementation and Results For the distributed approach, we create the cluster which has a parameter server and the worker nodes, then we specify the master as the parameter server and the training job is assigned to the workers.
Each worker takes care of only a task which is a portion of the training job. Finally, this time, the SCADA data are read from the Hadoop HDFS database. The command lines used to run the distributed model is : The first command launches the program on the master which is the parameter server which stores the model and distributes it to the 5 workers. The master as a parameter server is also responsible for the model update. The five workers are charged of the bulk of the workload i. e. the computation of the gradient during the Back-propagation optimization algorithm. The computed gradient is sent by each worker to the parameter server for updating the model. The first worker with the task index 0 is the coordinator of the model training job done by the workers. #### With the best model of the water storage tank SCADA dataset, we run the distributed version with that model setting in order to evaluate the training time for different epochs.
Results for Gas Pipeline Dataset We did the test in the same conditions as previously i. e. , using stacked autoencoders of 2, 3, 4 and 5 layers and 1, 10, 20, 30, 40, 50 epochs for each layer. The results are the best ones among all the various configurations for the multiclass and two class classification. The results using the multiclass version of the Gas pipeline SCADA dataset were obtained after training the model with 40 epochs. After 40 epochs, as show on Figure  6. 21, the validation loss increases even though the training loss continue to decrease. This motivates the choice of the results at 40 epochs. The model architecture has 2 hidden layers, a batch size of 128, a learning rate of 0. 01 a sparsity parameter of 0. 05, a noise level of 35%, and 58, 34 number of nodes for layer 1 and layer 2 respectively.
Experimental Results with the Gas Pipeline SCADA dataset With the Gas Pipeline SCADA dataset, the results are better compared to the Water Storage Tank SCADA dataset, as only class 1 with a detection rate of 0 % is totally mis-classified. Here, the detection rate of class 6 improved to 76 %, and for the other class of attacks, the detection rate is above 94 %. The False Positive Rate drops to 10 % for normal records detection. Those good results are confirmed by binary classification with the Gas Pipeline SCADA dataset where we have a detection rate of 89 % and a low False Positive Rate of 2 %. The False Negative rate here has dropped to 11 %, meaning that the mis-classification of attacks as normal records is less important in this dataset.
Conclusion An openstack infrastructure has been setup to conduct the different experiments of this thesis. A standalone instance is used to implement the hybrid deep neural network anomaly detection system, whereas a Hadoop cluster has been deployed for the distributed approach. The water storage tank and gas pipeline SCADA datasets have been pre-processed i. e. minimized, balanced, one-hot encoding of the labels, and splitted into training, test and validation sets. The proposed approach is compared with the Decision Tree, Naive Bayes and Random Forest standard approaches. In all cases of the tests, the proposed approach either outperforms all the baseline approaches or performs almost as well as the best of them which is the Random Forest. The distributed approach of the deep learning based anomaly detection shows a significant reduction of the training time.
• RQ 1. 2 : What are the characteristics of the SCADA dataset used ? The physical process and the resulting SCADA dataset were studied to answer this question. The physical process is a water tank system controlled by a SCADA network. The dataset obtained from the SCADA network has normal records and seven different types of attacks. The study of the distribution of the data showed that the SCADA dataset is highly imbalanced as anomalous records are fewer compared to normal ones. • RQ 1. 3 : How can we design a deep learning based unsupervised feature learning framework that will learn important features from a SCADA dataset? To answer this question, we built a stacked sparse denoising autoencoder for an unsupervised learning of the features of the SCADA network data. Some noise are added to the SCADA input data, and the building block of the unsupervised deep network are denoising autoencoders.
A sparsity constraint is added to the deep network loss function to allow more robust feature learning. The unsupervised deep feature learner is trained one layer at a time by using an unsupervised greedy layer-wise pre-training scheme. The second research question has three sub-questions as well: • RQ 2. 1 : How do we build a deep learning general framework for anomaly detection in SCADA systems? As a response to this question, we designed the general framework of the deep learning based anomaly detection system for SCADA networks. The framework has two main parts i. e a pre-processing engine and an anomaly detection engine. The pre-processing engine is responsible for the splitting of the dataset in training, validation and test data sub-sets. It also is responsible of the data normalization, the data balancing and one-hot encoding of the labels.
Review of contributions Our first contribution is a review of deep neural network-based SCADA anomaly detection systems using feature learning approaches. Then, as a second contribution, we proposed a stacked sparse denoising autoencoder architecture as feature learner for SCADA networks. This framework is able to automatically learn the salient features of the SCADA data that will later be used for classification purpose. A framework for an hybrid deep neural network SCADA anomaly detection system which has a data pre-processing engine and an anomaly detection engine is proposed as the third contribution. To speed-up the training process of the approach, we proposed as a fourth contribution a distributed approach of the deep learning based anomaly detection system, which uses a parameter server and worker nodes.
Limitations The detection rates of the proposed approach for NMRI, CMRI and DoS attack types are lower than other attack types. A possible reason is that those attacks are treated as noise that occurred during the normal operation of the SCADA system. This problem could be mitigated by collecting more training data in order for the system to be able to capture the intrinsic representation of those class of attacks. Features of the SCADA datasets which better describe the measurements of observations could also be developed to better represent the underlined physical process.
Future work As future work, we would collect larger-scale SCADA datasets from the owner to further test the proposed approach. We expect the approach will perform better with larger datasets, especially with more data related to the NMRI, CMRI and DoS attack types. Another future direction is also to combine our unsupervised learning framework with architectures such as RNN or LSTM in order to take into account the time series aspects of the SCADA data. The distributed approach of the anomaly detection system was tested on a cluster which has only classic CPUs. To better evaluate the impact on the training time of the model, we will consider using workers equipped with GPU processors. Detecting anomalies in real-time in SCADA systems is a primary goal to achieve. The approach should be tested with live data streams, by adding stream data acquisition modules to the framework, as well as tested in a SCADA faulty environment to check if the operational anomalies are detected.
It would also be interesting to submit the proposed anomaly detector to realworld SCADA data in order to test its performances in a real-world environment. Finally, the water tank and the gas pipeline datasets contain seven categories of attacks. In the future, a thorough modelization of the SCADA systems attack types will help improve the performance of the approach against Advanced Persistent Threats (APT) or unknown attack types. Contents pour tromper l'opérateur. Les logiciels de réseaux SCADA peuvent être ciblés par des virus, des vers, des chevaux de Troie et des réseaux de zombies. Le smash de pile et la manipulation de pointeurs de fonction pourraient également provoquer un débordement de mémoire tampon et permettre à un attaquant de lancer ses propres programmes contre le système SCADA.
We are using the automatic feature learning capability to unsupervisingly learn SCADA data in order to classify them into normal or anomalous data. We build an hybrid deep neural network anomaly detection system for a Water Tank and Gas Pipeline SCADA systems. Our anomaly detection system is composed of a stacked denoising autoencoder unsupervised feature learner and a softmax classifier. Afterward, we proposed a distributed version of our approach in an attempt to lessen the training time of our deep learning models. The distributed approach is implemented with the TensorFlow Distributed Framework and uses a parameter server to store the model parameters, update them and transmit them to worker nodes which are responsible for calculating the parameters gradient requiring high matrix multiplication during the back-propagation algorithm.
In this thesis, we are using a deep learning approach to propose an efficient hybrid deep neural network for anomaly detection in SCADA systems. The salient features of SCADA data are automatically and unsupervisingly learnt, and then fed to a supervised classifier in order to dertermine if those data are normal or abnormal, i. e if there is a cyber-attack or not. Afterwards, as a response to the challenge caused by high training time of deep learning models, we proposed a distributed approach of our anomaly detection system in order lo lessen the training time of our model.
The stacked auto-encoder has two hidden layers, one with 20 nodes and the second with 10 nodes. The dimension of the learnt features is 10 compared to the 41 original features of the NSL-KDD dataset dataset. The overall process encompasses four steps i. e. a feature learning step with the stacked auto-encoder, a first fine-tuning step where the softmax layer is trained in a supervised manner with labels and training data. The input of this first fine-tuning step is the compressed representation of the data. The following step is a second fine-.
Different kernel size were used for the experimentations. On top of the convolutions layer, a fully connected layer is added for prediction, and dropout is used to prevent overfitting. The authors tested diverse variations of this CNN architecture, by adding a batch normalization layer or by replacing the basic CONV-RELU-POOL block with ( CONV -RELU ) × N-MAXPOOL architecture. They also replaced the convolutionals layers by Inception layers[147]know to provide superior performance while keeping computational cost low. The Inception layers use sparse network connections instead of the fully connections used by convolution layers, hence the reduction of the computational overhead. The experiments were conducted on the Secure Water Treatment testbed (SWaT) dataset, which represents a scaled-down version of a real-world industrial water treatment plant which has36different cyberattacks.
The proposed 1 D CNN model successfully detected 32 out of 36 attacks of the SWaT dataset, representing 89 % of detection rate, which is fairly good, but need to be improved. The comparison of the different architectures shows that LSTMs and inceptionbased convolution converge the fastest and produce the lowest training error 3. 3. Review of unsupervised feature learning in SCADA ADS 59 rate. The anomaly detection algorithm provides high Area Under Curve (AUC), reaching 0. 967 for the eight layers convolutional network. The training and testing times of CNN compared to LSTM network were shorter by a factor 10 to 20 for testing and 15 to 40 for training respectively. Concerning the attack detection performance comparison, the authors show that pure CNN networks demonstrated better anomaly detection results than their LSTM alternatives.
datasets (i. e main class of interest rare), the Accuracy measure is not really relevant. Other metrics such as recall, precision and f1-score are used instead. The False Positive Rate (FPR) or False Alarm Rate (FAR) is the proportion of falsely identifying a normal tuple as anomalous. False Negative Rate (FNR)The False Negative Rate (FNR) is the proportion of falsely identifying an anomalous tuple as a normal one.
23 -Random Forest Gas Pipeline Multiclass ClassificationIn the Water Storage Tank SCADA dataset, the Decision Tree approach detects all the normal records (100 % of recall) with a 88 % of precision, which leads to a False Positive Rate of 38 %. This is why all the attacks classes 1, 2, 3, 5 and 6 recall is 0 %, because those types of attacks are considered as normal records by the classifier. The binary version of the Water Storage Tank SCADA dataset confirms this analysis with the 62 % detection rate of anomalous records and 38% of False Positive Rate. The Gas Pipeline SCADA dataset shows the same results for multiclass classification with the Decision Tree approach. Classes 1, 2, 3, 5 and 6 are mis-classified as normal records and we have a 61 % False Positive rate. For binary classification, we have a poor anomaly detection rate of 39 % and a high False Positive Rate of 61 %.
When doing a multiclass classification on the Water Storage Tank SCADA dataset, with the Random Forest approach, apart attack classes 1 and 2 where the detection rate is 0, this approach gives better detection rate for the other attacks categories (> 96 %) and the normal records category (100 %). We have the 89 % of precision for normal records detection and a False Positive Rate of 34 %. This is because, attacks classes 1 and 2 classified as normal records. Binary classification of the Water Storage Tank dataset confirms the previous results with a 66 % recall for anomalous records and a False Positive Rate of 34 % for normal records. For the multiclass classification, the Gas Pipeline SCADA dataset gives a 96 % of detection rate for normal records and a False Positive Rate of 11 %. Apart from the attack class 1 which has a recall of 20 %, all the other classes of attacks (2 through 7) have a recall above 93 % and a False Positive Rate less or equal to 1 %.
The binary classification of the Gas Pipeline dataset confirm those results with a 96 % recall for normal records with a False Positive Rate of 11 %, and a 89 % recall for anomalous records. For the Water Storage Tank SCADA dataset, based on the F1-score performance, the Decision Tree and Naïve Bayes have overall bad results compared to Random Forest and the proposed approach in multiclass classification. We will focus our comparison on the proposed approach and Random Forest. The proposed approach and Random Forest are totally mis-classifying class 1 and 2 attacks as those classes have a 0 % of recall and 100 % of False Negative Rate. Reconnaissance attacks (class 7) is fully detected by both Random Forest and the proposed approach (recall of 100 %). Random Forest performs slightly better than the proposed approach on class 3 (96 % and 92 % of recall respectively).
Random Forest has higher detection rate than the Hybrid DNN approach on class 4 and class 5 (98 % vs 87 %, and 100 % vs 85 % of recall respectively). The DoS attack is still poorly detected by our approach compared to Random Forest (35 % and 100 % of recall respectively). The binary classification of the same dataset shows that The proposed approach and Random Forest give the best results (66 % of recall and 34 % of False Negative Rate) compared to the Decision Tree and Naïve Bayes approaches. In overall, apart from one attack class, the proposed approach has comparable results with the best of the standard algorithms which is the Random Forest. For multiclass classification with Gas Pipeline dataset, Decision Tree and Naïve Bayes also show poor results compared to Random Forest and the proposed approach.
The proposed approach outperforms Random Forest for class 2, 3 and 4 attacks (98 %, 95 % and 98 % versus 93 %, 93 % and 95 % of recall respectively). The DoS attack recall for the proposed approach, though lower than the one of Random Forest (76 % and 96 % respectively) has increased with this dataset. The class 1 is still not detected by our approach (0 % of recall), but the the detection rate of Random Forest is also still poor for this class (20 % of recall). Figures 6. 20 and 6. 25 show a clear reduction of the training time when using a cluster of computer for deep models training instead of a single computer. One of the big challenge of deep learning models is the high training time required to train those models. By using a distributed approach, implemented with the TensorFlow distributed framework, the training time has been significantly reduced.
Hybrid Deep Neural Network Anomaly Detection System for SCADA Networks Abstract SCADA systems are more and more targeted by cyber-attacks because of vulnerabilities in hardware, software, protocols, communication stack. Those systems nowadays use standard hardware, software, operating systems and protocols. Furthermore, SCADA systems which used to be air-gaped are now interconnected to corporate networks and to the Internet. To thwart those attacks, many solutions have been proposed such as use of firewalls, anti-viruses, encryption, etc. But due to the differences between traditional IT and SCADA systems, the proposed solutions are not always applicable to the latter. Intrusion detection systems (IDS) are complementary solutions to secure SCADA networks. Collecting and labeling huge SCADA data is not always feasible as it requires human expert intervention and it is a time consuming process.
Les conséquences d'une attaque sur les systèmes SCADA peuvent être des pertes de production, des pertes financières, des catastrophes environnementales et même des pertes en vies humaines. CiteValdes2009 [169]. Les réseaux SCADA présentent des vulnérabilités dans différents domaines, tels que le périmètre réseau, les protocoles, les appareils de terrain, les bases de données et les liens de communication. Nous pouvons classer les attaques sur les réseaux SCADA de la manière suivante: attaques sur le matériel, attaques sur les logiciels et attaques sur la pile de communications [169] [150]. Après un accès à distance non autorisé aux périphériques SCADA, l'attaquant pourrait injecter de fausses données, telles que la modification des paramètres des automates, ce qui peut causer une défaillance du périphérique ou couper une alarme.