## 2 Related Work
### SA-ConvLSTM  
The difference between SA-ConvLSTM [4] and a standard LSTM lies in the presence or absence of SAM. The internal structure of SA-ConvLSTM is depicted in Figure 2, and the internal structure of SAM is illustrated in Figure 3. ConvLSTM uses the following equations:  
\[\hat{X}_{t}=SA(X_{t}) \tag{2}\] \[\hat{H}_{t}=SA(\hat{H}_{t-1})\]  
where \(\hat{X}_{t}\) denotes the input at time \(t\), and \(\hat{H}_{t-1}\) denotes the input state at time \(t-1\). \(X_{t}\) and \(H_{t}\) denote the outputs at time \(t\). \(SA\) denotes the processing of the SA-ConvLSTM, \(Q_{h}\) is the query, \(K_{h}\) is the key, and \(V_{h}\) is the value.  
The self-attention mechanism ConvLSTM is the SAM, enabling the model to selectively concentrate on significant aspects of the data. This enhancement bolsters the ability of the model to discern and assimilate complex patterns and interconnections. The SAM module was embedded into ConvLSTM for the self-attention mechanism. SAM was expressly crafted to harness long-term spatial dependencies, thereby augmenting the long-term memory capacity of the model.  
The SAM is partitioned into three segments: feature aggregation, memory updating, and output generation. The feature aggregation component accepts the features of the current timestep along with the memory from the previous step and applies self-attention to both inputs. This process refines the ability of model to capture intricate spatiotemporal contexts. Memory updating is pivotal for the model to retain salient patterns and historical trends, thus enabling the capture of long-range dependencies within the spatiotemporal spectrum. The module functions by refining the memory state using the assimilated features.  
Figure 3: Internal structure of SAM.  
Figure 2: SA-ConvLSTM cell.  
Regarding feature aggregation, the similarity score of each point pair is obtained using matrix products as follows:  
\[e_{\Delta}=Q_{h}^{T}K_{\Delta}\in\mathbb{R}^{N\times N},\Delta\in\{h,m\}. \tag{3}\]  
All weights are obtained by applying a softmax function to each column:  
\[\alpha_{i,j}=\frac{\exp e_{\Delta,i,j}}{\sum_{k=1}^{N}\exp e_{\Delta,i,k}}, \Delta\in\{h,m\},i,j\in\{1,2,\dots,N\}. \tag{4}\]  
The \(i\)-th aggregated feature is then computed using the weighted sum over all locations \(N\):  
\[Z_{\Delta,i}=\sum_{j=1}^{N}\alpha_{\Delta,i,j}V_{\Delta,j}=\begin{cases}\sum_{ j=1}^{N}\alpha_{h,i,j}(W_{h\
u}H_{t,j}),&\text{if }\Delta=h\\ \sum_{j=1}^{N}\alpha_{m,i,j}(W_{m\
u}M_{t-1,j}),&\text{if }\Delta=m\end{cases} \tag{5}\]  
and finally, the aggregated feature \(Z\) is obtained:  
\[Z=W_{z}[Z_{h};Z_{m}] \tag{6}\]  
For memory updating, using a gating mechanism, SAM adaptively updates the memory M to capture long-range spatiotemporal dependencies. The aggregated feature \(Z\) and original input \(H_{t}\) are used to generate the fused feature \(g_{t}^{\prime}\) with input gate \(i_{t}^{\prime}\). In addition, \(1-i_{t}^{\prime}\) is used to reduce the number of parameters. The updating process can be formulated as follows:  
\[i_{t}^{\prime} =\sigma(W_{m;zi}\otimes Z+W_{m;hi}\otimes H_{t}+b_{m;i}) \tag{7}\] \[g_{t}^{\prime} =\tanh(W_{m;gg}\otimes Z+W_{m;bg}\otimes H_{t}+b_{m;g})\] \[M_{t} =(1-i_{t}^{\prime})\cdot M_{t-1}+i_{t}^{\prime}\cdot g_{t}^{\prime}\]  
Here, memory \(M\) is updated using not only the convolutional operation but also the aggregated feature \(Z_{t}\) to obtain the global spatial dependence in a timely manner. Finally, the SAM output feature, \(\hat{H}\), is the result of the dot product of the output gate \(o_{t}^{\prime}\) and updated memory \(M_{t}\), which can be formulated as follows:  
\[o_{t}^{\prime} =\sigma(W_{m;zo}\otimes Z+W_{m;yho}\otimes H_{t}+b_{m;o}) \tag{8}\] \[\hat{H}_{t} =o_{t}^{\prime}\cdot M_{t}\]