# Inf-OSRGAN: Optimized Blind Super-Resolution GAN for Infrared Images
## 3 Method
### Spatially Correlative Loss  
To maintain the accuracy and perceptual quality of the reconstructed images within the degradation space described in Section 3.1, we introduced a spatially correlative loss function, namely F-LSeSim [23], that constrains the reconstruction outcomes by calculating the correlation between feature maps, paying more attention to maintaining the structural consistency and detail information to improve the accuracy and realism of the generated images. F-LSeSim  
We introduced the constraint concept of F-LSeSim into the super-resolution task. The computation process is detailed as follows:  
We generate contrastive maps at the patch level, following PatchNCE [58]. Let \(v=S_{x_{i}}\in\mathbb{R}^{1\times N_{p}}\) denote the spatially correlative map associated with the \"query\" patch in the ground-truth image. \(v^{+}=S_{x_{i}}\in\mathbb{R}^{1\times N_{p}}\) is a positive sample at the same location as the query region in the generated image \(G(x)\). \(v^{-}\in\mathbb{R}^{K\times N_{p}}\) is negative samples. Figure 3: The complete network architecture of Inf-OSRGAN includes a generator based on RRDBNet and a discriminator utilizing U-Net. The degradation process employs our proposed model, and during the training phase, a combination of L1 loss, perceptual loss, and spatially correlative loss was utilized. The query patches are positively matched with patches located at identical positions in the generated image \(G(x)\) and negatively matched with patches taken from different positions within in \(G(x)\) or from other images. A total of \(K=255\) negative patches are utilized. The LSeSim contrast loss is given Equation (7):  
\[\mathcal{L}_{c}=-\log\frac{e^{sim(\mathbf{v},\mathbf{v}^{+})/\tau}}{e^{sim(\mathbf{v},\mathbf{v }^{+})/\tau}+\sum_{k=1}^{K}e^{sim(\mathbf{v},\mathbf{v}_{k}^{-})/\tau}} \tag{7}\]  
In Equation (7), \(sim(\mathbf{v},\mathbf{v}^{+})=\mathbf{v}^{T}\mathbf{v}^{+}/\|\mathbf{v}\|\|\mathbf{v}^{+}\|\) denotes the cosine similarity of the two spatially correlated patches and \(\tau\) act as the temperature parameter. To minimize the loss, LSeSim promotes the closeness of corresponding patches that share similar structures, even if they have very different visual appearances, maintaining the structural coherence of the images. The loss in Equation (7) is exclusively employed to optimize the structural representation network. FSeSim generates spatially correlative maps by calculating the self-similarity between features extracted from LSeSim. These maps elucidate the spatial relationships among various structures within the images, rather than their visual appearance. Let \(x\) be the generated image and \(\hat{y}\) be the corresponding ground-truth image, respectively. Using LSeSim, we can extract features from the two input images, denoted as \(f_{x}\) and \(f_{\hat{y}}\). For each query point in the image, the self-similarity between this feature and the features of other points in the image is computed, creating what is known as a spatially correlative map, formally represented as Equation (8):  
\[S_{x_{i}}=(f_{x_{i}})^{T}(f_{x_{i}}) \tag{8}\]  
where \(f_{x_{i}}^{T}\in\mathbb{R}^{1}\) represents the feature of a particular query point \(x_{i}\), and the corresponding features contained within a patch composed of \(N_{p}\) points are included in \(f_{x_{i}}\in\mathbb{R}^{N_{p}}\). By combining self-similarity maps of multiple query points \(S_{x}=[S_{x_{1}};S_{x_{2}};\dots;S_{x_{s}}]\in\mathbb{R}^{N_{s}\times N_{p}}\), a spatially correlative map of the entire image is formed, where \(N_{s}\) stands for the number of sampled patches. These maps show the spatial relationships between different regions in the image. Subsequently, a comparison is made between several structural similarity maps, focusing on the ground-truth \(\hat{y}\) and generated image \(x\), as Equation (9) shows below:  
\[\mathcal{L}_{s}=d(S_{x},S_{\hat{y}}) \tag{9}\]  
In this equation, the spatially correlative map corresponding to the ground-truth images is signified by \(S_{\hat{y}}\).