## 3 Methodology
### Learning Modules
#### 3.2.2 Attention Mechanism  
As one of the most representative identifiers of transformers, the attention mechanism is proposed as a query-key-value (QKV) model to learn long-range dependencies without recurrent structures. Given the matrices \(\mathbf{Q}\in\mathbb{R}^{L_{q}\times D_{k}}\), \(\mathbf{K}\in\mathbb{R}^{L_{k}\times D_{k}}\), and \(\mathbf{V}\in\mathbb{R}^{L_{k}\times D_{v}}\) as the projected queries, keys, and values, the single-head version of standard attention mechanism can be formalized as \(\mathcal{A}(\mathbf{Q},\mathbf{K},\mathbf{V})=\text{Softmax}(\frac{\mathbf{Q} \mathbf{K}}{\sqrt{D_{k}}})\mathbf{V}\), where \(L_{q},L_{k}\) denote the length of queries and keys/values; \(D_{k},D_{v}\) denote the projected dimensions; \(\frac{1}{\sqrt{D_{k}}}\) is the scale factor to avoid \(\text{Softmax}(\cdot)\) yielding extremely small gradients. Furthermore, the multi-head version is as follows:  
\[\begin{split}&\mathcal{A}_{\text{multi-head}}(\mathbf{Q},\mathbf{K}, \mathbf{V})=\text{Concat}(\text{head}_{1},\cdots,\text{head}_{H})\mathbf{W} ^{O},\\ &\text{where head}_{i}=\mathcal{A}(\mathbf{Q}\mathbf{W}^{O}_{i}, \mathbf{K}\mathbf{W}^{K}_{i},\mathbf{V}\mathbf{W}^{V}_{i}).\end{split} \tag{5}\]  
The queries, keys, and values with dimension \(D\) are mapped into \(H\) heads (i.e., subspaces) by \(\mathbf{W}^{O}_{i},\mathbf{W}^{K}_{i}\in\mathbb{R}^{D\times D_{k}}\), and \(\mathbf{V}\mathbf{W}^{V}_{i}\in\mathbb{R}^{D\times D_{v}}\). Then the outputs of these heads are concatenated and mapped back to \(D\) by \(\mathbf{W}^{O}\in\mathbb{R}^{HD_{v}\times D}\). In most cases, \(D_{k}=D_{v}=D/H\). The standard transformer has two types of multi-head attention: self-attention and cross-attention. For self-attention, its projected queries, keys, and values are shared with the same source, while the key-value pairs of cross-attention are typically from the output of encoder.  
In practice, the attention modules used in DualET are modified to improve the performance of capturing dependencies. First, we design an additional cross-domain attention module to discover the correlation between the features of images and sequences. Concretely, the queries are the temporal features of the decoder, and the key-value pairs are the cloud information from the RSI encoder. Furthermore, we perform the fast Fourier transform (FFT) on the input and the inverse FFT on the output:  
\[\mathcal{A}_{\text{FFT}}(\mathbf{Q},\mathbf{K},\mathbf{V})=\mathcal{F}^{-1}( \mathcal{A}(\mathcal{F}(\mathbf{Q}),\mathcal{F}(\mathbf{K}),\mathcal{F}( \mathbf{V}))) \tag{6}\]  
where \(\mathcal{F},\mathcal{F}^{-1}\) denote the FFT and its inverse, which are also used in the self-attention module of the LSI encoder. The FFT plays a key role in signal processing because it can rapidly convert a signal from the time/space domain to the frequency domain (and vice versa) and describe relationships between these domains [37]. It is defined by  
\[\mathbf{\overline{X}}_{k}=\sum_{m=0}^{L-1}e^{-2\pi ik\cdot(m/L)}\mathbf{X}_{m},\quad k=0,\dots,L-1 \tag{7}\]  
Based on FFT, the attention module can discover the frequency correlation between queries and keys. In addition, we employ the ProbSparse attention mechanism [38] as the self-attention and cross-attention modules of the decoder to improve their performances.