# Neural Network System for Predicting Anomalous Data in Applied Sensor Systems
## 2 Materials and Methods
### Development of the Predictor Model  
The predictor is a recurrent neural network (RNN) with a long short-term memory (LSTM) layer [37; 38] (Figure 3). The RNN training is carried out on the dataset prepared by the preprocessor. During use, the neural network takes a subsequence of actual sensor values preceding the current value as input and outputs a predicted value. The LSTM layer consists of identical LSTM blocks (Figure 4), whose number matches the subsequence length \(n\) defined during preprocessing. These blocks collectively form a hidden state vector h, with its length set by the system operator. A Dropout layer randomly deactivates a specified neuron fraction in vector h to prevent overfitting, typically deactivating 20%, though this parameter can be adjusted by the operator. The Dense layer  
Figure 3: Block diagram of the predictor model. applies the modified (special) rectified linear unit (ReLU) activation function (Appendix A) to transform the data and obtain the predicted value \(i_{i+1}\). Each LSTM block comprises a block state and several filter layers. The block state is a vector carrying information from previous time steps and propagates through the entire chain of LSTM blocks. An LSTM block contains three types of filters (\"gates\"), an input gate, a forget gate, and an output gate, which regulate the data amount to be saved, forgotten, and output, respectively. Figure 4 illustrates the following notations: the \"Control gate\" is a control unit with a sigmoidal activation function; the \"Recording gate\" is a recording unit with a tangential activation function; the \"Output gate\" is an output unit with a sigmoidal activation function; \(c_{t-1}\) is the memory tensor at the previous time step; \(c_{t}\) is the memory tensor at the current time step; \(h_{t-1}\) is the network output tensor at the previous time step; \(h_{t}\) is the network output tensor at the current time step; \(x_{t}\) is the input tensor at the current time step; \(n_{t}\) is the tensor at the control unit output; \(r_{t}\) is the tensor at the recording unit output; and \(W_{\alpha t}\), \(W_{lm}\), \(W_{xn}\), \(W_{hr}\), \(W_{xr}\), \(W_{ch}\), \(W_{hh}\), and \(W_{xh}\) are the weight coefficients connecting inputs to units. The LSTM mechanism addresses the vanishing gradient problem by regulating the flow of new information into the state vector \(c_{t}\), as well as the network's output state \(h_{t}\), and updating its state \(c_{t}\). The vector filters are determined by the following expressions:  
\[i_{t}=\sigma\cdot(U_{i}\cdot x_{t}+W_{i}\cdot h_{t-1}), \tag{9}\]  
\[f_{t}=\sigma\cdot\left(U_{f}\cdot x_{t}+W_{f}\cdot h_{f-1}\right), \tag{10}\]  
\[o_{t}=\sigma\cdot(U_{o}\cdot x_{t}+W_{o}\cdot h_{o-1}), \tag{11}\]  
\[g_{t}=\sigma\cdot\left(U_{g}\cdot x_{t}+W_{g}\cdot h_{g-1}\right), \tag{12}\]  
where \(\sigma\) is the sigmoid function; \(x\) is the input sequence; \(h\) is the network cell hidden state vector; and \(U_{i}\), \(U_{f}\), \(U_{o}\), \(U_{g}\), \(W_{i}\), \(W_{f}\), \(W_{o}\), \(W_{g}\) are the network filter weight matrices for the training sequence element indices \(i,f,o,g\), \(t\). Based on the network filter values, its internal state (internal memory) and hidden state vectors are determined as follows:  
\[c_{t}=\tanh(i_{t}\circ g_{t}+f_{t}\circ c_{t-1}), \tag{13}\]  
\[h_{t}=o_{t}\circ c_{t}, \tag{14}\]  
where \(c_{t}\) represents the network cell state vector, \(h_{t}\) denotes the hidden state vector, and \"\(\circ\)\" indicates element-wise multiplication. Figure 4: Diagram of the variable memory LSTM network [37]. In the Dropout layer, certain elements of the activation vector are randomly set to zero to prevent overfitting.