# Deep Neural Networks for Spatial-Temporal Cyber-Physical Systems: A Survey
## 3 DNNs in CPS-Based Spatial-Temporal Data
### DNN Techniques
#### 3.3.2 RNN  
It refers to the DNNs designed to process sequential data, such as time series or text data, by maintaining an internal memory or state. The basic idea behind RNNs is that the output at a given time step is impacted not only by the current input but also by the previous inputs and the current state of the network. The network comprises a series of interconnected recurrent cells, which process the input at each time step and update the internal state of the network. Each cell takes the current and previous states as input, produces an output, and passes a new state to the next cell in the sequence. The backpropagation through time method trains RNNs, a variant of the standard backpropagation algorithm used to update the network weights based on the error signal at each time step. RNNs are suitable for application areas such as language modeling, speech recognition, etc. They are particularly effective for tasks requiring an understanding of the temporal dependencies of sequential data. Nonetheless, it is a challenging issue to train and prone to vanishing and exploding gradients. Various mechanisms have been developed to address the drawbacks, such as gradient clipping and regularization techniques (e.g., dropout). RNN has two representative variants: one is LSTM and the other is GRU, which are briefly explained below. * LSTM: It is an RNN variant designed to deal with challenges in RNNs in handling sequential data. Since conventional RNNs suffer from the _\"vanishing gradient problem\"_ that limits their capability of capturing long-term dependencies between input and output sequences. LSTMs improve on that by remembering and selectively forgetting information over longer time horizons, making them effective for modeling sequences of variable lengths, such as natural language processing (text or speech). An LSTM consists of four main components: a memory cell, three gating units (an input, a forget, and an output gate), and an activation function. The memory cell stores information over long periods and passes it on to the next time step. The input gate controls data flow into the memory cell according to the current and previous outputs. Based on the input and output from the previous and current cycles, the forget gate determines the data to be erased from the memory cell. The output gate determines the output based on the current input and the current state of the memory cell. The activation function, mostly a hyperbolic tangent or sigmoid function, is used to compute the cell's current state. At each iteration, the LSTM unit receives an input vector and a hidden state vector from the previous time step and produces an output vector and a new hidden state vector. The input vector is passed through the input gate, and the output gate determines the output vector. The forget gate determines what information to keep from the previous hidden state, and the memory cell updates its internal state based on the input and the last hidden state. The updated memory cell is then passed on to the next time step. LSTMs have been effective in various applications, including speech recognition, machine translation, image captioning, and music composition. They are often combined with other DNNs, such as CNNs or attention mechanisms, to realize performance on given tasks. * GRU: It is a gating mechanism for RNN. It was introduced to serve as a simplified version of the complex LSTM. Like other RNNs, GRU processes sequential input data, such as text or time series, by maintaining a hidden state that captures information about the past sequence elements. It also uses a gating mechanism to selectively update and reset the hidden state, enabling it to capture longer-term dependencies in the sequence. GRU has two gating mechanisms: the reset gate and the update gate. The reset gate decides the past hidden state to forget, while the update gate makes the current input to incorporate into the new hidden state.