# A Case Study on Virtual HPC Container Clusters and Machine Learning Applications
## 4 ML on Docker
### Anomaly Detection Application
#### 4.2.3 Implementation  
This subsection only describes the most important parts of the source code of the anomaly detection application. In the project, training the neural network and detecting anomalies are divided into two separate scripts. For clarity, the code presented here is slightly modified and shortened, without additional helper functions, and all parameters stored in variables are hardcoded. In reality, most of these parameters can be changed from the command line. Listing 2 shows a section of the code responsible for importing the necessary modules and setting parameters for ML and anomaly detection. The value normal_percentile=0.99 was chosen after a grid search (0.90-0.995) that maximized the F1-score on a hand-labeled validation subset; lower percentiles produced excessive false positives, whereas 0.995 missed short spikes. ```
import tensorflow from tensorflow import keras from sklearn.preprocessingimportStandardScaler importnumpy import horovod.keras as horovod import pandas csv_file = \"stock_data.csv\" time_steps = 4 learning_rate = 0.01 optimizer = keras.optimizers.Adam(learning_rate) loss_function = \"MSE\" epochs = 300 callbacks = [] batch_size = 64 normal_percentile = 0.99
```  
The variables in Listing 2 have the following purposes:  
- csv_file--name of the file containing training data,
- time_steps--number of samples in a single training sequence,
- learning_rate--learning rate coefficient,
- optimizer--optimization algorithm used for calculating new weights during neural network training (Adam algorithm uses stochastic gradient descent),
- loss_function--loss function (MSE--Mean Squared Error),
- epochs--number of epochs (complete training cycles using the entire training set),
- callbacks--list of callbacks controlling the learning process,
- batch_size--size of the training batch (amount of data taken at once from the dataset during training),
- normal_percentile--the percentile of the error distribution, above which values will be classified as anomalies. Before training, the data must be read from the source and appropriately transformed. The code segment in Listing 3 accomplishes this task. Data are read from a CSV file (line 6), then the unnecessary column storing the timestamp of the measurements is removed (line 7). The feature values are normalized using the StandardScaler object, which scales them by first subtracting their means and then dividing by their standard deviations (line 11). The table is then transformed into a dictionary mapping container instance IDs to their corresponding CPU and memory usage measurements (lines 13-14). Next, each long sequence of measurements needs to be split into smaller subsequences of size time_steps, and then all subsequences are added to a master list. This is achieved using a helper function from the Keras module (line 19). In the given time series train_series, a window of size time_steps is moved from the beginning to the end, one step at a time, extracting the contained sequences. The mentioned function returns a Dataset object, but an array of numpy.array is needed, so the received object is transformed, and then appended to the master list train_x (lines 23-25). The sequences stored in train_x serve as both input and output data for the neural network. The network aims to replicate the given time series of size time_steps as accurately as possible. Data from train_x are fed into the network, and output values close to the input are expected. A training set train_dataset is created based on train_x (line 26), and its batch size is determined using the batch method (line 27).