## 2 Related Deep Learning Algorithms
### Description of Experiment
In the production environment, this threshold should be lowered due to the target maximization of the parameter sensitivity \(=\)\(\frac{number\;of\;true\;positives}{number\;of\;true\;positives-number\;of\;false\; negatives}\) (True Positive--correct detection of an anomaly; False Negative--wrong classification of an anomaly as a normal sample). Ultimately, values similar to the threshold should be collected for labeling and used to provide additional training for the model. For experimental purposes, the primary indication of the model's efficiency was the classification's precision. The Adam optimizer was used during the training process, and it did not stop until the accuracy of the test data decreased. Each model's training was conducted six times for various initial learning rates (0.01, 0.001, and 0.0001) and batch sizes (32,128). In total, 324 experiments were conducted. The training process of the deep learning models in the paper, focusing on hyperparameters, training duration, and steps taken to prevent overfitting, can be summarized as follows:  
1. Selection of Hyperparameters: * Autoencoder (AE) and Variational Autoencoder (VAE): The selection of hyperparameters was crucial for both AE and VAE models. The hidden and latent dimensions were varied to understand their impact on model performance. Specifically, hidden dimensions were experimented with at 64, 128, or 256, and latent dimensions at 8, 16, 32, 64, 128, or 256. This variation allows for an exploration of how the complexity of the model affects its ability to reconstruct and detect anomalies. * Cost Function Choices: The VAE model used a specialized cost function comprising reconstruction and regularization costs. This choice of cost function is essential as it balances the model's ability to reconstruct input data while accurately maintaining a well-organized hidden space. 2. Training Duration: * Iterations Per Epoch: Each epoch of the training process consisted of approximately 2000 iterations. This indicates a comprehensive training procedure per epoch, allowing the models to learn sufficiently from the training data in each pass. * Total Iterations: The training continued until there was no further improvement in the test set performance. This approach led to training durations that varied significantly, with some models reaching up to 200,000 iterations while others concluded around 20,000 iterations. * Adaptive Training Duration: This adaptive approach to training duration, which continues until the test performance plateaus, effectively ensures that the models are neither under- nor over-trained. It allows each model to learn at its own pace based on the architecture's complexity and the data's nature. 3. Preventing Overfitting: * Dynamic Stopping Criterion: The decision to stop training when the test set performance on accuracy ceased to improve is an example of a dynamic stopping criterion, akin to early stopping. This method is crucial in preventing overfitting, as it ensures that the model does not continue to learn the idiosyncrasies of the training data that do not generalize to unseen data. * Monitoring Test Performance: Regularly monitoring the model's performance on the test set during training allows for a balanced approach, ensuring that the model is effectively learning the underlying patterns in the data without memorizing the training set. Regularization Techniques: Regularization in the cost function of the VAE is a direct method to prevent overfitting. The Kullback-Leibler divergence in the VAE's cost function ensures that the model memorizes the training data and learns a more generalized representation. * Hyperparameter Tuning: This study implicitly addressed overfitting by experimenting with different hyperparameters, particularly the size of the latent space in autoencoders. Smaller latent spaces can help prevent the model from capturing too much noise and overfitting.