## 2 Materials and Methods
### Hybrid Model LSTM-MA-AE
Figure 7:0  
The encoded representation \(\mathbf{out}_{\text{enc}}\) is first decoded by means of an MA, which uses  
Figure 7:1  
where \(\mathbf{H}_{\text{t-1}}\) is the previous hidden layer of the LSTM unit at the end of the decoder, which at the beginning is a tensor of zeros. Next, the MA output of the decoder is concatenated with the historical data \(\mathbf{R}_{\text{t}}\) to be reconstructed:  
Figure 7:2  
A fully connected layer \(FL\) is then used to learn even more of the data representation, and the output of this is used as input to two LSTM layers, followed by _LayerNorm_:  
Figure 7:3  
where \(\mathbf{H}_{\text{t}}\) feeds as a \(Q\) back to the MA to obtain the reconstructed predicted data. Finally, the decoder output is projected to the original dimension with a \(FL\) to reconstruct the \(\hat{\mathbf{Z}}\) time series, using Equation (27). Figure 7:4  
The pseudocode for LSTM-MA-AE is presented in Algorithm 1. ```
1:Input: SCADA data, where \(\mathbf{X}=[\mathbf{x}_{1},\mathbf{x}_{2},\dots,\mathbf{x}_{T}]\in\mathbb{R}^{T \times p}\), \(T\) is the time series length and \(v\) is the number of variables. 2:Initialize:
3: Load configuration parameters: hidden layers \(hl\), batch size \(bs\), epochs \(e\), patience \(p\)
4: Initialize model, loss criterion \(\mathcal{L}\), optimizer Adam, and learning rate \(Lr\)
5: Initialize variables: best_val_loss = \(\infty\), patience_counter = 0, \(\mathcal{L}_{val}=0\)
6: Load \(X_{training}\), \(X_{test}\), \(X_{val}\in\mathbb{R}^{(e,q,p)}\), where \(e\) is the number of sequences and \(q\) is the sequence length. 7: Create DataLoader iterators for training and testing data with \(bs\)
8:for each epoch \(e\)do
9:Training Phase:
10: Set model to training mode
11: Initialize training loss to zero
12:for each batch in \(X_{training}\)do
13: Zero the parameter gradients
14:\(output=\) LSTM-MA-AE(\(batch\))
15: Compute loss \(\mathcal{L}=criterion(output,batch)\)
16: Concatenate \(params\_without\_bias=Concat\_params(LSTM\)-MA-AE)
17: Compute regularization term \(\mathcal{L}_{reg}=\mathcal{L}+Reg\cdot L1_{norm}(params)\)
18: Perform backpropagation
19: Update model parameters \(\mathcal{L}_{reg}\_backward()\)
20: Accumulate training loss \(\mathcal{L}_{train}+=\mathcal{L}_{reg}\)
21:endfor
22: Compute average training loss \(\mathcal{L}_{train\_avg}=\mathcal{L}_{train}/\)number of batches
23:Validation Phase:
24: Set model to evaluation mode
25:for each batch in \(X_{val}\)do
26: Disable gradient computation
27:\(output_{val}=\) LSTM-MA-AE(\(batch\))
28: Compute loss \(\mathcal{L}_{val}=criterion(output_{val},batch)\)
29: Accumulate validation loss \(\mathcal{L}_{val}+=\mathcal{L}_{val}\)
30:endfor
31: Compute average validation loss \(\mathcal{L}_{val\_avg}=\mathcal{L}_{val}/\)number of batches
32:Early Stopping:
33:if\(\mathcal{L}_{val\_avg}<\)best_val_loss then
34: Update best_val_loss = \(\mathcal{L}_{val\_avg}\)
35: Save model parameters and reset patience_counter = 0
36:else
37: Increment patience_counter \(+=1\)
38:endif
39:if patience_counter \(>\)\(p\)then
40: Break training loop early
41:endif
42:endfor
43:Test the model using \(X_{test}\)
44: Reconstruction data \(output_{test}=\) LSTM-MA-AE(\(X_{test}\))
45: System of prediction faults
46: Loss of each variable \(\mathcal{L}_{test}=criterion(output_{test},X_{test})\)
47: Smoothing \(\mathcal{L}_{test}\)
48: Apply threshold with EWMA and sliding window anomaly detection
49: Fault warning;

### Exponential Weighted Moving Average (EWMA)

Within the failure prediction mechanism, EWMA is used. This technique applies greater weight to more recent data with higher errors. This technique is defined by Equation (28) [44]. Figure 7:5

where \(\alpha\) is the smoothing factor, \(a_{t}\) is the value of the reconstruction loss in time \(t\) and EWMA\({}_{t-1}\) is the value of EWMA in time \(t-1\). The threshold value at the upper bound of EWMA is calculated as shown in Equation (29).