# A Comparison of Deep Learning Algorithms for Anomaly Detection in Discrete Mechanical Systems
## 4 Network Architecture and Training
### Autoencoder  
Both the machine learning algorithms, that in the following will be respectively referred as PINN-CAE and DD-CAE, share the same neural network architecture (characterized by a total number of 31,240 trainable parameters), based on a convolutional autoencoder. Autoencoders are unsupervised learning algorithms which, after several transformation and data compression series, aim to reconstruct the input at the output with the least distortion. This technique is widely used to remove noise and for compressing and visualizing high-dimensional data [33]. Convolutional neural networks (CNNs), on which convolutional autoencoders (CAEs) are based, are a class of artificial neural networks (ANNs) which make use of algorithms based on convolution operations and are characterized by many advantages: (i) each neuron is no longer connected to all neurons of the previous layer but to a smaller portion, reducing the parameters and speeding up convergence and (ii) dimension reduction allows the removal of trivial features while retaining useful information [34]. In this paper, the autoencoder was built using the package TensorFlow and assembled with Keras API. As reported in Figure 7, the model has 11 layers of the following types:  
1. **Separable convolutional 1D layer**: This layer applies 1D-convolutional windows separately to every channel. Then, it mixes the channel by point-wise multiplication. Indeed, the application of convolutional layers is proven to be particularly efficient in the analysis of time series [35]. 2. **MaxPooling layer**: A limitation of convolutional layers is that they greatly increase the number of parameters in the output tensors compared to the input ones; when many filters are involved, the magnitude of the tensors grows exponentially. For this reason, a pooling layer usually follows a convolutional one. Its purpose is to sub-sample the feature map by retaining only the most attractive information extracted by the convolutional layer. Thereare many possible pooling functions, but in this work, the MaxPooling function is adopted, which takes only the max value out of a predefined sub-matrix. 3. **Dropout layer**: A common problem in the development of an ANN is overfitting; this occurs when a model learns from a particular random feature in the training data so that it is able to perfectly manage that set, but these learned concepts may not apply to new data, leading to poor performance. Dropout is a form of regularization, i.e., an approach that makes the network more robust in the training phase by forcing the network to learn general and recurrent patterns. During training, if a tensor passes through a dropout layer, some of its values are randomly dropped according to the dropout probability, i.e., the fraction of input's elements whose value is set to zero. During testing, no values become zero, but the output is scaled by a factor equal to the dropout probability. Sometimes, the values are adjusted by the same fraction only in training to leave the test and prediction phases untouched. Some guidelines to manage the dropout layer can be found in [36]. 4. **Dense layer**: This is the simplest and most straight-forward type of layer that can be used. It is used to define the latent space. All the neurons in a dense layer are connected to all the neurons in the previous layer. Every connection is characterized by a weight, which multiplies the input value. The dense layer defines a bias \(b\) and an activation function \(f\). If \(x\) is the input tensor, \(z\) is the output tensor, and \(W\) is the weights tensor, the mathematical equation of a dense layer is: \[z=f(\tilde{z})=f(Wx+b)\] (11) where \(\tilde{z}\) is called the weighted sum of the input. 5. **Transposed convolutional 1D layer**: This is a type of convolutional layer that can be used to increase the spatial resolution of an input tensor while maintaining a connectivity pattern that is compatible with some convolutional layer.