# Action Recognition Algorithm of Spatio-Temporal Differential LSTM Based on Feature Enhancement
## 3 St-D Lstm
### The Internal Structure of the ST-D LSTM  
[PERSON] et al. [19] proposed the ST-LSTM structure for spatio-temporal sequence prediction; it can realize information transmission between different layers of LSTM units. ST-LSTM is improved based on the ConvLSTM [20] structure. Vertically, spatial information memory states between the LSTM units at different layers are similar to the horizontal memory states of the ConvLSTM unit, and the spatio-temporal memory module is added based on the original horizontal memory state. The ST-LSTM transmits the information of hidden layers, and increases the transmission of spatial information in the vertical direction, to realize the transmission of memory information between different layers in this time step. ST-LSTM is the core part of the PredRNN algorithm. For action recognition, limb position change is a vital feature; that is, the time change and position change should be considered at the same time. The zigzag transfer method enables the stacked ST-LSTM unit to transfer the spatial state longitudinally at each time step. Although the PredRNN algorithm considers both temporal and spatial features through the zigzag cross-layer connection, it ignores changes of temporal and spatial features. For this reason, the SpatioTemporal Differential LSTM (ST-D LSTM) unit is proposed, with the idea of spatio-temporal variation based on the spatial memory state of the ST-LSTM unit. The ST-D LSTM is similar to the LSTM. It also contains the forgetting gate, the input gate, and the output gate. Furthermore, the ST-D LSTM unit also contains two cell states: the temporal memory module \(C_{t-1}^{l}\) and the spatial memory module \(S_{t}^{l-1}\). The temporal memory module stores the temporal characteristic information of the previous \(t-1\) moments in the same layer units, while the spatial memory module stores the spatial characteristic information of different layer units. \(x_{t}\) represents input in the ST-D LSTM unit; \(h_{t-1}^{l}\) is the hidden layer state. \(k_{t}\), \(i_{t}\) and \(f_{t}\) are the conversion mechanism, the input gate and the output door of temporal memory, respectively. \(k_{t}^{\prime}\), \(i_{t}^{\prime}\) and \(f_{t}^{\prime}\) are the conversion mechanism, the input, and the output door of the spatial memory, respectively. The output gate \(o_{t}\) combines temporal memory and spatial memory. Similarly to the differential part in the PID control, the differential module of spatial memory state is added to the original LSTM unit according to the connection mode of the input gate. The \"future\" error, that is the characteristic change information, is introduced into the present state by integral calculation, so that the network can improve the accuracy and stability. In addition, the input differential module is added at the same time to increase the propagation of spatial features in the same layer of the LSTM unit along the horizontal time step, so that the network can take into account the temporal information, the limb moving speed and trajectory. The ST-D LSTM internal structure diagram is shown in Figure 1.  
In the mathematical model, \(\mathbf{t}\) is a small value, so the input differential \(\frac{\mathrm{d}x(t)}{dt}\) is approximated to \(x_{t}-x_{t-1}\), that is \(\frac{\mathrm{d}x(t)}{dt}\approx x_{t}-x_{t-1}\). Similarly, the spatial memory differential can be expressed as \(S_{t}^{l-1}-S_{t-1}^{l-2}\). Approximation can make the calculation easier while realizing the differentiation of the input and spatial state. The differential processing is  
Figure 1: The internal structure diagram of the ST-D LSTM. transmitted. For the unit in the first layer at time t, the differentiation between the spatial memory state of the previous time step \(S^{l}_{t-1}\) and that of the time step \(S^{l}_{t-2}\) is added; that is, \(S^{l}_{t-1}-S^{l}_{t-2}\). The spatial memory state output of the unit is divided into two directions, one direction continues the longitudinal spatial memory transmission, and the other direction performs the differential calculation.