# Toward Explainable AutoEncoder-Based Diagnosis of Dynamical Systems
## 3 Preliminaries
### AutoEncoders  
This section introduces the autoencoder and reviews how it has been used for diagnosis. We first describe the vanilla (or traditional) AE, and the outline the standard variants, namely the denoising autoencoder (DAE), sparse AE, contractive autoencoder (CAE), and variational autoencoder (VAE). An introduction of AEs is presented in [29; 30].  
Autoencoders have widely been used for anomaly detection and fault detection [1]. Given data \(\xi\), a vanilla autoencoder \(\mathcal{A}\) consists of two parts: (1) an encoder \(f_{\theta_{E}}(\xi)=h\) that creates the latent code \(h\) in terms of parameters \(\theta_{E^{\prime}}(2)\) a decoder \(g_{\theta_{D}}(h)=\xi\) that re-creates the input from the latent code \(h\) in terms of decoder parameters \(\theta_{D}\).  
Figure 1 depicts an autoencoder for diagnosing faults in time-series systems. An autoencoder projects the input data \(\xi\) into a smaller subspace, called the code \(z\), such that linear classifiers can be used on the structure of the code. Autoencoders can thus perform nonlinear dimensionality reduction, as well as classification. Mathematically, learning an autoencoder can be expressed as the following optimization task:  
\[\min_{f,\delta}\ \parallel\xi-g\circ(f(\xi))\parallel_{2}^{2} \tag{5}\]  
When we train \(\mathcal{A}\) with nominal data, then an input \(\xi\) that is anomalous will produce a higher than average anomaly score. However, in many AE-based methods, the loss function \(\mathcal{L}\) is typically defined in terms of MSE, or of a regularized balancing of (a) accuracy-based MSE and (b) some sparsity metric over the code \(\mathcal{L}=\mathcal{L}_{MSE}+\lambda\mathcal{L}_{sparsity}\), where \(\lambda\) is the regularization parameter.  
Figure 1: The autoencoder approach we adopt, where the input \(\xi\) is a encoded to a code \(z\) and then decoded to create the reconstructed input \(\xi\). We use the loss function \(L(\xi,\xi)\) to control the quality of the reconstruction.  
We can use \(\mathcal{A}\) as a decision tool to identify faults, where an input \(\xi\) is anomalous if some measure (an anomaly score) over the reconstruction \(\xi\) and input \(\xi\) by more than a threshold \(\epsilon\). Several definitions of the anomaly score have been proposed, such as reconstruction error (\(\epsilon_{R}=|\xi-\xi|\)), or Mahalanobis distance, \(\epsilon_{M}=\sqrt{(\xi-\xi)^{T}\Sigma^{-1}(\xi-\xi)}\), where \(\Sigma\) is the covariance matrix of the training input.  
Other AE approaches add regularization to _reduce and/or minimize the size_ of the latent code. Common approaches include the sparse autoencoder (which minimizes the number of active neurons in the code, i.e., \(\min_{z\in\xi}|z|\)), or the contractive autoencoder (which limits the sensitivity to small changes in the input, using the Frobenius norm \(\|\cdot\|_{F}\) of the Jacobian matrix of the encoder).  
A problem with using \(\mathcal{A}\) is the lack of semantics for the approach, and the challenge of scaling to complex systems, e.g., dynamical systems such as process-control factories. Further, the loss function associated with Equation (5) best addresses single-fault diagnosis (fault/no-fault) cases: it aims to match the input \(x\) with the output \(\hat{x}\). This representation focuses on identifying an anomalous sequence; hence, it does not address fault isolation.