# LSTNet: A Reference-Based Learning Spectral Transformer Network for Spectral Super-Resolution
## 3 Proposed Method
### Spectral Transformer Module  
As shown in Figure 1, the STM takes \(\mathbf{Y}\), \(\mathbf{Y}_{ref}\), and \(\mathbf{X}_{ref}\) as inputs and exports soft attention and hard attention into the SRM module. First, the feature extraction sub-networks (as discussed below) are used to generate the corresponding features \(Q\) and \(K\) from \(\mathbf{Y}\) and \(\mathbf{Y}_{ref}\), in which the feature extraction sub-networks involve shared parameters. As the features \(Q\) and \(K\) are extracted from different images using the same feature extraction network, these  
Figure 1: Network architecture of the proposed LSTNet. The input MSI is the image to be processed. The ref-MSI and ref-HSI are a paired MSI and HSI, used as reference images. \(Q\) and \(K\) are the features extracted from the input MSI and ref-MSI, denoting query and key, where the used feature extraction sub-networks are shared parameters. \(V\) denotes the spectral values transformed from the ref-HSI. Relevance denotes the spatial attention relevance between the input MSI and ref-MSI. features contain more useful spatial texture information to further compute the spatial relevance between \(\mathbf{Y}\) and \(\mathbf{Y}_{n\!f\!f}\). The process of feature extraction can be expressed as:  
\[Q=f_{FE}(\mathbf{Y}), \tag{2}\]  
\[K=f_{FE}(\mathbf{Y}_{n\!f\!f}), \tag{3}\]  
where \(f_{FE}()\) denotes the feature extractor. The basic elements of the attention mechanism are \(Q\) (query), \(K\) (key), and \(V\) (value). As we wish to transform the spectral structure from the reference HSI, we use ref-HSI directly as \(V\). Relevance measurement aims to compute the spatial relevance between \(\mathbf{Y}\) and \(\mathbf{Y}_{n\!f\!f}\) by estimating the spatial similarity between \(Q\) and \(K\). Since both are multispectral images, the spatial similarity facilitates the migration of spectral information on the ref-HSI. Following the setting of attention [34; 41], to calculate the relevance, \(Q\) and \(K\) are unfolded into patches to make use of the spatial information of the local neighborhood. The patches are denoted as \(q_{i}\in\mathbb{R}^{(n\times p\times p)\times(W\times H)}\) and \(k_{j}\in\mathbb{R}^{(n\times p\times p)\times(W\times H)}\), where \(i\) and \(j\) denote the \(i\)th and \(j\)th patches, \(n\) denotes the number of features of \(Q\) and \(K\), and \(p\) denotes the patch size. Then, the patches must be normalized, such that the relevance \(r_{ij}\) can be calculated by using the inner product between each \(q_{i}\) and \(k_{j}\):  
\[r_{i,j}=\frac{q_{i}}{\|q_{i}\|}\cdot\frac{k_{j}}{\|k_{j}\|}, \tag{4}\]  
where \(\|\|\) denotes the normalization process. Hard attention in the STM describes the most relevant spectrum in \(V\) for each \(q_{i}\), which is the raw spectral information migrated directly from the ref-HSI. Here, we use an argmax function to calculate the index location of the maximum value for each relevance \(r_{ij}\), along the dimension of \(k_{j}\). The resulting indices are the position of the most relevant spectrum in \(V\). The hard attention mechanism can be presented as:  
\[h_{i}=\text{arg}\underset{j}{\text{max}}r_{i,j}, \tag{5}\]  
where \(h_{i}\) denotes the index location of each relevant maximum. All of the index locations for the most relevant spectra are formed as a location map, \(\mathbf{H}\), in order to select the spectrum directly from the ref-HSI \(V\). The selected relevance spectrum is further transferred into the spectral reconstruction module in order to help recover \(\mathbf{X}\). The transformed spectrum features from the hard attention can be formulated as:  
\[\mathbf{F}_{h}=V_{\mathbf{H}}. \tag{6}\]  
Soft attention in the STM denotes the spectrum synthesized by the relevance weights. In contrast to the hard attention, where we use an argmax function to determine the index location of the maximum value, soft attention uses the maximum relevance to soften the transformed spectrum from the hard attention.