dataesr/scientific-paragraphs-categorization huggingface dataset:
Poor access policies, traceability and patching are mentioned in the second vulnerabilities category. The third categories of vulnerabilities involves the various operating systems found in SCADA networks such as Linux, SCO Unix, Windows NT, Windows 2000 Server, Windows XP, and 20 CHAPTER 1. SCADA Systems Windows 2003 Server. Those OS as well as the supported applications are not well patched leaving the SCADA servers exposed to attacks. Finally, the last vulnerabilities category i. e. the SCADA communication protocol vulnerabilities are mainly due to TCP/IP version of traditional protocols such as Modbus, DNP3, Profibus and Fieldbus. The shift to modern protocols open new possibilities to attackers because those protocols do not apply integrity checking of commands between Master and slaves, nor do they perform any authentication mechanism between Master and slaves.
[35]  recommend consistent security policy, well-designed network architecture, system hardening by shutting down unnecessary services, ports and software, two-factor password and data encryption for remote connections are also mandatory to secure SCADA systems. Cryptographic methods such as cryptographic algorithms (RC4, DES, AES and RSA) are suggested by  [46] , Message authentication through Integrity protection with hash and digital signature, Key distribution and Entity authentication are used to achieve confidentiality, integrity, authentication and nonrepudiation security objectives. He also recommend security in communication protocols like PPP, PAP, CHAP, WEP in Link Layer, IPSec in Network Layer, SSL and SSH in Transport Layer, Digest Authentication and PGP in Application Layer. Communications are also secured by the means of Firewalls and Intrusion Detection Systems.
The system architecture consists of three parts which are a component analysis module, a detection module, and an original CHAPTER 1. SCADA Systems inference module. The component analysis module that automatically analyzes system variables to extract components and generates a graph describing the valid system states and another graph of relationships between the variables. Here, alternating vectors which record the alternating relations between two continuous states are also used to represent the real-time states of a component in normal operation. The second module is a detection module that uses the state graph to generate an invalid state alert if a new alternation vector is not found in the state graph or an invalid transition alert if the current state could not be reached from a previous state.
Input, hidden and output layers The Input layer represent how we get input data fed into the network. It has the same number of neurons as the number of input features. Each neuron in the input layer represents a unique attribute in your dataset. On the other hand, There is one or more hidden layers in a neural network. The weight values on the connections between the layers are how neural networks encode the learned information extracted from the raw training data. Hidden layers introduce nonlinearity modeling in neural networks. The Output layer is the final layer in a network. It receives input from the previous hidden layer, applies an activation function, and returns an output representing your model's prediction. The answer or prediction from the model is in the output layer. The final output may be a real valued output (regression) or a set of probabilities (classification).
In SGD, we compute the gradient and parameter vector update after every training sample. SGD speed-up learning and also parallelizes well. In between Batch gradient descent and SGD lies mini-batch gradient descent, where a batch of samples more than a single training example and less than the full training dataset is used to compute the gradient. The loss function can be minimized by estimating the impact of small variations of the parameter value on the loss function. Let s W be the set of parameters and E(W ) the loss function. The gradient descent algorithm allow the minimization of E(W ) by iteratively adjusting W as follow  [101]. W k = W k-1 - ∂E(W ) ∂W Where is a scalar constant called the learning rate. This is the Weights update with Gradient Descent.
In real world, most data are complex, and building good predictors on those data means learning complex functions too, which are best represented by multiple levels of non-linear operations, i. e. deep architectures  [155]. Unsupervised feature learning can be done by using clustering on data using algorithms such as K-means  [13] , or by training stacked auto-encoders or convolutional networks  [156]. In order to detected anomalies due to data/command injection, reconnaissance or Denial-of-Service (DoS) attacks on a gas pipeline SCADA system,  [50]  propose an anomaly detection approach consisting of a Bloom filter anomaly detector and a time-series level anomaly detector (Figure  3. 2). The former one is a packet-level anomaly detector which checks a packet signature in its database. The database stores network patterns and communication pattern signature as they are stable in a SCADA system.
If the analyzed package signature is not in the Bloom filter, the packet is considered anomalous. The latter detector receives normal packet that pass the Bloom filter for a time-series level anomaly detection, which uses its power of information memorization for an extended number of time steps to predict the behavior of the next time step. a Bloom filter is a probabilistic data structure that is used to test whether an element is a member of a set. It is commonly used as an in memory data structure which size is limited by the availability of RAM space on the machine  [40]. Because of the limited memory and computing resources of some of SCADA components, using a Bloom filter as a fast and light-weighted packet level anomaly detector is important. It efficiently stores the signature database of normal network packets and detects anomalies thereafter.
On the other hand, the time-series anomaly detector is a Stacked Long Short Term Memory (LSTM) Network-based Anomaly Detector (Figure  3. 1) which takes the input of time-series x(t -1), x(t -2),. , learns their higher dimensional feature representations, and then uses those features to predict the next data point x(t). Furthermore, the predicted data point can be used to classify if x(t) is anomalous by checking the similarity between x(t) and x(t). The LSTM network model is then trained to minimize a softmax loss function suited for multi-class classification  [59] [128]. The evaluation of the combined anomaly detection framework on a gas pipeline SCADA dataset  [115]  gives an accuracy of 92 %, which is higher compared to other approaches. However, the training time of the LSTM model of 35 min 54CHAPTER 3.
Review of SCADA Anomaly Detection Systems using Deep Feature Learning A during 50 epochs is rather high. The poor performance of the latter two categories of attacks is due to the lack of sufficient amount of data related to R2L and U2R (0. 04 % and 0. 79 % respectively). 9 to 10 % training data samples for R2L and U2R categories of attacks as with the probe attacks would have given better detection results. However, the approach proposed by  [132]  gives promising results in feature learning and good detection rate for some classes of attacks detection. It uses the NSL-KDD dataset, an improved version of KDD Cup 99  [149]  which is a general bench-marking dataset for network intrusion 56CHAPTER 3. Review of SCADA Anomaly Detection Systems using Deep Feature Learning A detection research, created twenty years ago.
CNN/LSTM anomaly detection in SCADA The Secure Water Treatment testbed (SWaT) dataset contains up to 36 different cyber-attacks. To evaluate the use of unsupervised feature learning for intrusion detection in such system,  [92]   The proposed CDBN efficiently reveal the high-dimensional temporal behavior features of the unobservable FDI attacks that bypass the SVE mechanism with a high accuracy rate over 94% even in the presence of occasional operation faults, meaning that unknown attacks could be detected. Wind turbines usually operate in harsh and variable environment, making theirs components such as gearbox, main bearing, generator, inverter and controller subjet to failure. This situation can lead to unavailability and even destruction, causing expensive repair costs of wind turbines.
68CHAPTER 4. Building an Unsupervised Deep Neural Network Feature Learning Framewor After the presentation of the water storage tank SCADA dataset used for this work in the first section, we give the building process of the stacked sparse denoising auto-encoder in the following section. Thereafter, a section is devoted to the loss function, activation function, parameters and hyper-parameters. In the fifth section, we define the algorithm needed to train the model before the conclusion.
SCADA Dataset used The dataset used is obtained from a testbed of the Mississippi State University SCADA Security Laboratory and Power and Energy Research laboratory. The physical system is a water storage tank system  [115]. The records of the dataset were captured from the control system of the water storage tank that models oil storage tanks found in industries like chemical or refineries  [55]. The physical process is made of two storage tanks (a primary and a secondary one), a pump that moves the water from the secondary storage tank to the primary one, a relieve valve which allows the water to flow from the primary storage tank to the secondary tank, and a sensor which indicates the water level in the primary tank as a percentage of total capacity. The control system of the water storage tank system have three part : an HMI that allows a human operator to monitor and control the water storage tank, a Master Terminal Unit (MTU), a Remote Terminal Unit (RTU) and communication links.
Dataset attributes The datasets contain two types of attributes i. e. the network traffic attributes and the payload attributes. The topology and services of the SCADA systems are relatively static compared to traditional IT. Therefore, network traffic attributes can be used to describe normal traffic patterns and thus detect fraudulent activities. Network traffic attributes include the device address, function code, packet length, packet error control information, and the time interval between packets. The attributes of the payload content provide information about the state of the SCADA system. They are useful for detecting the cause of abnormal behavior in hardware (eg PLCs). These attributes include sensor measurements, control command values and physical process states (see Table  4. 2).
KL(ρ|| ∧ ρ j ) = ρ log ρ ∧ ρ j + (1 -ρ) log 1 -ρ 1 - ∧ ρ j Where ρ and ∧ ρ j representing the means of two Bernoulli random variables. If we set the value of ρ very small (0. 05 for example), and ∧ ρ j being the mean of the activation of a given node over a the dataset, that will cause the node to rarely activate, thus introducing the sparsity.
Sparse Denoising auto-encoder The sparse denoising auto-encoder is a denoising autoencoder with a sparse parameter. The denoising autoencoder, instead of merely minimizing the loss function L(x, g(f (x))) 4. 4. Sparse Denoising Autoencoder (SpDAE) 75 where f (x) is the coder and g(f (x)) the decoder, minimizes L(x, g(f ( x))) where x is a copy of x that has been corrupted by some form of noise. The sparse autoencoder on the other hand adds a sparsity penalty Ω(h) on the code layer h in addition to the reconstruction error  [59]  : L(x, g(f (x))) + Ω(h) where g(h) is the decoder output and h = f (x), the encoder output. The sparsity term of an autoencoder make it respond to unique statistical features of the dataset it has been trained on, rather than simply acting as an identity function. The sparse denoising autoencoder will then minimize the loss function L(x, g(f ( x))) + Ω(h).
Stacked Sparse Denoising Autoencoder (SSpDAE) 4. 5. 1 Process of building the SSpDAE In the process of stacking the sparse denoising auto-encoders, the input of the first auto-encoder is the SCADA dataset features, and the hidden layer is h 1 with some number of nodes. The decoder part of the first auto-encoder is discarded. For the second auto-encoder, the input layer is the hidden layer h 1 of the first auto-encoder, and the hidden layer of the second auto-encoder is h 2. We do this process until we reach the n th auto-encoder which input is the hidden layer h n-1 4. 5. Stacked Sparse Denoising Autoencoder (SSpDAE) Now that the stacked denoising auto-encoder global structure is defined, in the next section, we will define the loss function, the activation function, parameters and hyper-parameters like the number of layer, the number of nodes per layer and the regularization used to avoid overfitting.
Conclusion In this thesis, we have used a water storage tank testbed SCADA dataset that have normal and anomalous records. The attributes of the dataset are either from the network or measures from the physical process observations. The dataset contains normal as well as attacks records. There is an overall  28   As a basis of the proposed architecture, we choose the stacked denoising autoencoder (SDAE) based on its performance in prior studies, to which we added a sparse parameter to make the feature learning process more robust. Afterwards, we showed the process of building the stacked sparse denoising autoencoder (SSpDAE), where the first sparse denoising autoencoder (SpDAE) input is the SCADA data attributes, and for the subsequent layers, the output layer of a previous SpDAE is discarded, and its hidden layer is the input layer of the current SpDAE.
Data Splitting into Training, Validation and Testing Sets To split the datasets, we use a cross-validation technique. The two most commonly used cross-validation techniques are the hold-out cross-validation and the k-fold cross-validation  [133]. The hold-out cross-validation is a widely-used cross validation technique that is efficient and easy to use. The dataset is separated into three mutually disjoint subsets, i. e. , a training, a validation and a testing sets. There is no restriction on the size of the three data subsets, making this technique suited for big datasets. The model is trained on the training set and the validation one is periodically used to evaluate its performance to avoid overfitting. The training is stopped when the performance on the validation set is good enough or when it stops giving better results.
Dataset Balancing Many approaches exist for unbalanced datasets balancing, e. g. random oversampling, random undersampling, the Synthetic Minority Oversampling Technique (SMOTE)  [27]  or the Adaptive Synthetic (ADASYN)  [65]  sampling methods. Oversampling methods balance training dataset by increasing the number of minority class example, while undersampling methods balance training dataset by decreasing the number majority class examples. In section 6. 5 of the next chapter, the dataset distribution shows that some minority class has very few samples compared to normal instances. We then focus the balancing method on oversampling, SMOTE, and ADASYN. Tests results with the SCADA dataset give better results when applying random oversampling.
Supervised Classification Module The SCADA dataset has records from normal operation as well as those caught from seven different categories of attacks. There is an overall eight categories of records in the dataset. Hence, we are solving a multiclass classification problem which consists in discriminating the different records of the dataset with respect to their correct category. A layer with nodes using a softmax activation function is suited for multiclass classification problems  [45]   [127]. Therefore, in the proposed architecture, we use on top of the unsupervised deep feature learner, a supervised classifier which is a softmax layer (Figure  5. 4). When using a softmax layer for a multiclass modeling problem, we only care about the best score across these classes, and we use an arg-max() function of the softmax output layer to get the highest score of all the classes.
Loss Function of the Supervised Classifier The loss function that will be defined for the supervised classifier training will also be used to train the hybrid deep neural network. The supervised classifier defined in the previous section is a softmax layer which output is a probability distribution over all the classes. The input data of the supervised classifier and the hybrid DNN use labeled data. In the section 5. 3 concerning the SCADA datasets pre-processing, those labels have been one-hot encoded. This is actually a probability distribution where the actual class value is 1 and all the others 0. We then need a loss function capable of comparing two probability distribution (the actual label of data one-hot encoded and the outcome of the classification output by the softmax function).
e. h 1 is used as the input of the second auto-encoder. We train the second auto-encoder and likewise, another abstraction of the input features is the second hidden layer h 2. Finally, in the same way, the n-th auto-encoder is trained, and a final abstraction of the input feature is represented by the layer-wise-th hidden layer h n. for l ← 1 to L do foreach example ∈ SCADA_dataset do H ← f (W X + b) // f is the Leaky ReLU activation X ← f (W H + b ) / * Minimize J(θ) = 1 m m i=1 ( 1 2 (h W ,b (x (i) ) -x (i) ) 2 ) + β k i=1 ρ log ρ ∧ ρ i + (1 -ρ) log 1-ρ 1- ∧ ρ i * / W , b ← Minimize J(θ) end learnt_f eatures ← H L end 5. 5.
Softmax Supervised Layer Training Step The supervised classification module is a softmax layer. The transfer function for each node of is a softmax function, which maps each output to a vector of probabilities. The softmax layer has eight nodes corresponding to the number of the different types of the dataset records. This layer is appended to the stacked sparse denoising autoencoder (Fig 5. 6 ). After the greedy layer-wise unsupervised for i ← 1 to training_epoch do W , b ← DN N _U nsupervised_FL (SCADA_dataset) X, ← SCADA_dataset H 1 ← f (W 1 X + b 1 ) // f is the Leaky ReLU activation for l ← 2 to L do H l ← f (W l H l-1 + b l ) end / * P (Y = i|x, W , b) = sof tmax i (W x + b) = e W i x+b i j e W j for i ← 1 to training_epoch do X ← SCADA_dataset H 1 ← f (W 1 X + b 1 ) // f is the Leaky ReLU activation for l ← 2 to L do H l ← f (W l H l-1 + b l ) end pred_labels ← sof tmax(H L W s + b s ) / *.
Algorithme 5 : DNN_Detection Data : network, SCADA_test_dataset, W , b, W s , b s Result : pred_labels X ← SCADA_test_dataset H 1 ← f (W 1 X + b 1 ) // f is the Leaky ReLU activation for l ← 2 to L do H l ← f (W l H l-1 + b l ) end pred_labels ← sof tmax(H L W s + b s ).
First Generation of Distributed Machine Learning Systems : DistBelief The first generation of Distributed Machine Learning Systems was the DistBelief  [39] , a software framework that can use computer cluster with thousands of machines to train large models. On each node, the user defines the computation and the message that should be passed during the computation. In case of large models, the user can partition it across several machines of the cluster. DisBelief seamlessly manages the parallelization of the computation, the communication, the synchronisation as well as data transfer between machine. To enable the distribution of the training across multiple model instances, they propose two new large-scale distributed optimization procedures i. e. Downpour SGD and Sandblaster L-BFGS which are distributed versions of the Stochastic Gradient Descent (SGD) and the Limited memory Broyden-Fletcher-Goldfarb-Shanno (L-BFGS)  [171]  method respectively.
In the Downpour SGD approach, the training dataset is divided into a number of subsets and a copy of the model is run on each of the subset. A centralized parameter server keeps the current state of all models parameters scattered across the nodes. On the other hand, the Sandblaster L-BFGS uses a different approach where a coordinator issues commands like dot product, scaling, coefficient-wise addition, multiplication, etc. , that is performed by each parameter server shard independently, with the results being stored locally on the same shard. In TensorFlow, the client which is the main component uses Session interface to communicate with the master, and one or more worker processes. The graph nodes are executed on computational devices such as classical CPU or Graphical Processing Units (GPU).
Proposed Distributed Deep Neural Network Anomaly Detection System for SCADA In our approach, multiple autoencoders are stacked, and the feature learning uses a greedy layer-wise pre-training. But it is shown in practice that in deep neural networks, going deeper than 3 layers does not help much more in terms of performance  [74]. Our main focus is the size of data that could be very important in SCADA networks. The distributed approach of the hybrid deep neural network anomaly detection system for SCADA will then use the data parallelism to speed up the training process. The calculation of the gradient in the model training that consists of multiplication of large matrices while iterating over a large dataset is time consuming. To speed up the Stochastic Gradient Descent training algorithm, we parallelize the computation of the gradient for a mini-batch across mini-batch elements  [2].
Several distributed deep learning like Dryad  [77] , Flume  [25] , CIEL  [117] , Naiad  [118]  and Spark  [167]  exist. But the Distributed TensorFlow framework  [2]  uses a hybrid data flow model that borrow elements from the previous frameworks without their shortcomings. Furthermore, TensorFlow is more flexible and allows the expression of a wide variety of machine leaning models and optimization algorithms compared to distributed deep learning approaches like DistBelief  [39] , the Adam Project  [31]  and the Parameter Server Project  [36]. An Hadoop  [160]  cluster with different nodes will be used and the training data will be stored on the Hadoop HDFS database. Thus, the TensorFlow distributed framework will parallelize the computation of the gradient of the mini-batches across the different nodes and a parameter server with update the weights on the master (Figure  5.
Introduction This last chapter is intended for the implementation of the proposed hybrid deep neural network based anomaly detection systems for SCADA networks. After the introduction, the second section shows the development environment setup used for this thesis. The following section, gives details of the second dataset used i. e. the gas pipeline SCADA dataset. The fourth and fifth sections respectively illustrates the preparation of the two datasets before usage and their distributions. In the sixth section, we show the different performance measures used to compare the approach with baseline methods. Sections seven through nine are used to show the different results of the proposed approach with the SCADA water storage tank and gas pipeline datasets. We discuss about the experimentation results in section ten before a conclusion.
Development environment setup For the different experiments of the present work, we setup an infrastructure via OpenStack. The hybrid deep neural network anomaly detection system was implemented on a server with 8vcpu, 8 Go of RAM and 10 Go hard drive. The server is running a Ubuntu 18. 04. 2 LTS operating system. The Deep Learning framework used is TensorFlow 1. 5. 0 running on Python 2. 7.
CHAPTER 6. Implementation and Results For the distributed approach of the anomaly detection system, we setup an Hadoop cluster (Figure  6. 1). The Hadoop version is 2. 7. 3. The cluster has one master and five workers. Each node has 8vcpu, 8 Go of RAM, 10 Go hard drive and runs a Ubuntu 18. 04. 2 LTS operating system. Distributed Deep Learning is powered by Distributed TensorFlow (version 1. 5. 0) running on python 2. 7. The HDFS file system on the master node is used to store the SCADA datasets. In chapter 4, we gave a broad description of the Water Storage Tank System SCADA dataset used (Figures 6. 2 and 6. 3). Table  6. 1 and Table  6. 2 respectively show the water storage tank SCADA dataset records categories and attributes.