Learning Cross-Spectral Prior for Image Super-Resolution

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-07 04:42:50

Abstract: With the rising interest in multi-camera cross-spectral systems, cross-spectral images have been widely used in computer vision and image processing. Therefore, an effective super-resolution (SR) method is significant in providing high-resolution (HR) cross-spectral images for different research and applications. However, existing SR methods rarely consider utilizing cross-spectral information to assist the SR of visible images and cannot handle the complex degradation (noise, high brightness, low light) and misalignment problem in low-resolution (LR) cross-spectral images. Here, we first explore the potential of using near-infrared (NIR) image guidance for better SR, based on the observation that NIR images can preserve valuable information for recovering adequate image details. To take full advantage of the cross-spectral prior, we propose a novel $\textbf{C}$ross-$\textbf{S}$pectral $\textbf{P}$rior guided image $\textbf{SR}$ approach ($\textbf{CSPSR}$). Concretely, we design a cross-view matching (CVM) module and a dynamic multi-modal fusion (DMF) module to enhance the spatial correlation between cross-spectral images and to bridge the multi-modal feature gap, respectively. The DMF module facilitates adaptive feature adaptation and effective information transmission through a dynamic convolution and a cross-spectral feature transfer (CSFT) unit. Extensive experiments demonstrate the effectiveness of our CSPSR, which can exploit the prominent cross-spectral information to produce state-of-the-art results.

Main paper: 
 related work 2.1 single-modal image super-resolution: 
 Benefiting from the development of deep learning, singleimage super-resolution (SISR) has achieved remarkable advances over previous reconstruction-based methods [2, 20, 34, 54] . As the first convolutional neural network (CNN) based SISR method, SRCNN [9] learns the LR-to-HR mapping and achieves remarkable advances. Following this fashion, a large number of deep learning-based SISR methods [3, 19, 23, 26, 36, 42, 47, 50, 52, 55-57, 59, 62, 64] ) have been developed to improve the objective accuracy or perceptual quality of the SISR results.Most SISR approaches adopt MSE or MAE as a loss function and target high PSNR/SSIM by proposing various deep network architectures. For instance, VDSR [19] constructs a deeper SISR network with 20 convolution layers. EDSR [23] builds a very deep and wide network by cascading modified residual blocks. DRFN [56] adopts a deep recurrence learning strategy to enlarge the receptive field and utilizes transposed convolution for upsampling. ENLCN [47] proposes an efficient non-local contrastive attention module to model long-range visual features and leverage more relevant nonlocal features in an image. HAT [3] combines both channel attention and self-attention schemes to utilize global image statistics. To decrease the computational cost, NLSN [59] proposes a Non-Local Sparse Attention (NLSA) and embeds it into a residual network to enforce sparsity in the Non-Local attention module, as well as largely reduce its computational cost. SMSR [42] learns sparse spatial and channel masks to identify important locations and mark redundant channels in those unimportant regions for efficient SR.To obtain better SR performance, some methods use additional images prior to assisting the super-resolution. Specifically, multi-frame super-resolution [31] , reference-based superresolution (RefSR) [27, 48, 53, 60, 65] , and stereo image super-resolution [7, 10, 17, 24, 43, 51, 68] methods exploit images from adjacent frames, reference images and cross-view images, respectively, to provide beneficial guidance for SR and achieve a huge breakthrough.In addition to single-modal visible images, multi-modal images, including infrared images and near-infrared (NIR) images, have also been regarded as a prior in some RGB image restoration tasks, such as the low-light image enhancement [33, 63, 69] , image dehazing [21, 38] , image restoration [16, 46] , and image denoising [44] .For instance, some image enhancement methods [33, 63, 69] use the contrast and texture information in infrared images to guide the enhancement of low-light VIS images. The NIR image-guided colour image denoising method [44] fuses NIR images and noisy colour images to eliminate image noise and transfer detail structure from guided images by simply concatenating the two input images.In order to utilize the multi-modal images in SR, most current multi-modal image super-resolution (MMSR) methods regard RGB images as the guidance for the SR of images from other modalities, including depth image [8, 30, 35, 37, 40, 41, 45, 49, 58, 70] , multi-spectral image [22, 28, 61] , thermal image [35, 49] , NIR image [35] . Some self-supervised MMSR methods [30, 35, 49] LR space or using the weakly supervised cross-modal transformation manner. CMSR [49] is proposed to super-resolve the thermal, NIR, and depth images under the guidance of RGB images. However, these RGB-guided MMSR methods require high-resolution RGB images as reference, which are unavailable in some real-world scenes.Current MMSR methods ignore the potential of the NIR image for improving the VIS image quality and require pixellevel aligned multi-modal images. NIR images have not been considered to assist the SR of visible (RGB) images. In addition, their practicability is limited when the HR RGB image is unavailable. 
nir-vis image analysis: 
 To effectively introduce the NIR guidance into the VIS image SR, we first analyze the inherent characteristics of the NIR-VIS images. Figure 2 demonstrates the visual and statistical characteristics of the NIR and VIS images.As shown in Figure 2 (a), the NIR images retain better brightness contrast and richer texture details under poor lighting conditions, while the VIS images suffer from detail loss in the overexposed or dark areas. Furthermore, the NIR images provide simpler and clearer boundaries of texts, as they are more sensitive to illumination change. In comparison, the VIS images need to handle complex colours, leading to blurred or jagged edges between different objects. In addition, VIS images are susceptible to illumination, fog, and other bad weather, and NIR images can well resist these disturbances. Furthermore, we also observe that the NIR images preserve accurate high-frequency information, which is helpful in providing reasonable guidance and few low-frequency disturbances for recovering adequate image details.Then, we also show the distribution of average pixel intensity in NIR/VIS images from three benchmark datasets (RGB-NIR stereo [66] , NIRScene [1] , and RANUS [6] ) in Figure 2 (b). Compared to VIS images, the pixel intensity distribution of NIR images is relatively more uniform, demonstrating that NIR images can provide beneficial and complementary guidance for recovering accurate and rich VIS image details. Figure 2 (c ) visualizes the intensity variation of pixels along the red dotted lines in the left NIR/VIS image. The NIR image presents a larger step response between mountains and clouds, therefore, it retains clearer boundaries. In addition, for the rich-textured VIS image areas with a high-frequency intensity variation, the NIR image has weaker intensity changes, which ensures more valuable NIR information is provided and avoids the disturbance to details that already exist in the VIS image.Based on the above analysis, the visual and statistical difference between the NIR-VIS images inspires us to take full advantage of the beneficial information from the NIR image for guiding the SR of the VIS image. 
method: 
 To take full advantage of the cross-spectral guidance for better SR, we propose a cross-spectral prior guided image superresolution (CSPSR) approach. We will dedicate to stating the proposed approach in detail in the following subsections.Figure 3 demonstrates the overall workflow of the CSPSR framework. We adopt a dual-branch structure to super-resolve images of different modalities, as the two SR branches can extract valuable features, that are more relevant to SR, to provide more appropriate guidance. To bridge the multimodal feature gap and enhance the cross-view sub-pixel correspondence of cross-spectral images, we design a crossview matching (CVM) module and a dynamic multi-modal fusion (DMF) module, which can enhance the correspondence between two views and fuse features of different modalities, respectively. We will introduce the two modules in detail in the following subsections.As shown in Figure 3 𝑉 𝐼𝑆 , 𝐹 0 𝑁 𝐼𝑅 ∈ 𝑅 𝐻×𝑊 ×𝐶 ) of the inputs (𝐼 𝑙 𝐿𝑅 , 𝑁 𝐼𝑅 𝑙 ) with two convolution layers and a residual block. The residual block is composed of two convolution layers and a residual connection. 𝐻, 𝑊, 𝐶 denote the height, width, and channel number of the features (𝐹𝑁𝐼𝑅, 𝐹𝑉 𝐼𝑆 ).To adaptively fuse the NIR and VIS image features, the DMF module, consisting of a dynamic convolution operation and a cross-spectral feature transfer (CSFT) unit, is inserted between two SR branches. The DMF module takes the NIR image feature (𝐹 𝑡-1 𝑁 𝐼𝑅,𝑡=1:𝑇 ) and the VIS image feature (𝐹 𝑡-1 𝑉 𝐼𝑆,𝑡=1:𝑇 ) as input and is repeated for 𝑇 times to fully combine the NIR and VIS image features. After 𝑇 DMF modules, the outputs (𝐹 𝑇 𝑉 𝐼𝑆 , 𝐹 𝑇 𝑁 𝐼𝑅 ) of the last DMF module in each SR branch are delivered to a convolution layer and a pixel shuffle layer to generate the final SR results (𝐼 𝑙 𝑆𝑅 , 𝑁 𝐼𝑅 𝑙 𝑆𝑅 ).As shown in Figure 3 , the CVM module enhances the global correspondence between the cross-spectral images based on the cross-spectral disparity prediction model [66] . In order to enhance the robustness of the CSPSR model, we predict rough left-right disparities {𝑑𝑖𝑠𝑝 𝑙 , 𝑑𝑖𝑠𝑝 𝑟 } based on the lowfrequency LR images directly. Thus, we can also allocate more training effort to the subsequent SR modules by decreasing difficulty and releasing the burden of training the CVM module.Then, we offset all pixels (𝑖, 𝑗) in the right NIR image according to the 𝑑𝑖𝑠𝑝 𝑙 , which can accurately align crossspectral images to the same view, and generate the new NIR image in the left view (𝑁 𝐼𝑅 𝑙 ), facilitating stronger sub-pixel correspondence between the NIR/VIS images.𝑁 𝐼𝑅 𝑙 (𝑖, 𝑗) = 𝑁 𝐼𝑅 𝑟 (𝑖, 𝑗 -𝑑𝑖𝑠𝑝 𝑙 (𝑖, 𝑗))(1) Figure 4 gives an example of the output of the CVM module. By enhancing the pixel-level correlation between cross-spectral images, the convolutional calculation with a limited receptive field can take full advantage of the aligned NIR image to restore more accurate VIS image details.As shown in Figure 3 , the 𝑡-th dynamic multi-modal fusion (DMF) module first adopts a dynamic convolution to adapt the 𝐹 𝑡-1 𝑁 𝐼𝑅 , resulting in a new NIR image feature (𝐹 ′𝑡-1 𝑁 𝐼𝑅 ). Then, to transfer knowledge from the NIR image to the VIS image, we feed the 𝐹 𝑡-1 𝑉 𝐼𝑆 and the 𝐹 ′𝑡-1 𝑁 𝐼𝑅 to the CSFT unit and obtain a new VIS image feature (𝐹 ′𝑡-1 𝑉 𝐼𝑆 ), containing the information in the NIR image feature. Finally, the input features (𝐹 𝑡 𝑁 𝐼𝑅 , 𝐹 𝑡 𝑉 𝐼𝑆 ) of the next DMF module (𝐷𝑀 𝐹𝑡+1) are obtained by two convolution layers.Dynamic convolution. Inspired by the dynamic upsampling filter [18] , we introduce a dynamic convolution to conduct content-adaptive feature adaption. First, we concatenate the 𝐹 𝑡-1 𝑉 𝐼𝑆 and the 𝐹 𝑡-1 𝑁 𝐼𝑅 and apply a convolution layer to learn ). Then, we reshape all vectors 𝐾(𝑖, 𝑗)𝑖=1:𝐻,𝑗=1:𝑊 ∈ 𝑅 1×1×𝑘 2 to generate 𝐻 × 𝑊 filters of size 𝑅 𝑘×𝑘 . Finally, the new NIR image feature (𝐹 ′𝑡-1 𝑁 𝐼𝑅 ) can be obtained by position-aware filtering the 𝐹 𝑡-1 𝑁 𝐼𝑅 with the 𝐾. As the kernel is learned based on the NIR-VIS features, it can flexibly adapt the NIR feature and make it more compatible to enhance the VIS feature.Cross-spectral feature transfer (CSFT) unit. The aligned images can provide more relevant information, which makes it easier to guide the SR of the left VIS image. However, due to the issue of occlusions, the CVM module cannot guarantee sub-pixel matching accuracy, which may degrade the SR performance and introduce unwanted artefacts. Therefore, how to effectively extract and utilize helpful information in the aligned cross-spectral images is significant. In addition, fusing features of different modalities adaptively is also a key point that needs to be solved.To address the above problems and make full use of the NIR image for better SR, we propose a confidence-based CSFT unit, where the multi-modal features are weighted for better feature fusion between cross-spectral images. As shown in Figure 3 , the CSFT unit can adaptively fuse the NIR-VIS image features (𝐹 ′𝑡-1 𝑁 𝐼𝑅 , 𝐹 𝑡-1 𝑉 𝐼𝑆 ) to output a new feature (𝐹 ′𝑡-1 𝑉 𝐼𝑆 ) through learnable spatial and channel-wise attention. Thus, the feature 𝐹 𝑡-1 𝑉 𝐼𝑆 , combining abundant and useful information of the NIR image and VIS image, is helpful for restoring more texture details in the VIS image. The specific workflow of the CSFT unit is as follows.As we mentioned before, some VIS image regions with low contrast typically lose many details. From this observation, we first learn a spatial weight (𝑤𝑠1 ∈ 𝑅 𝐻×𝑊 ×1 ) based on the 𝐹 𝑡-1 𝑉 𝐼𝑆 through a convolution layer with kernel size 3 × 3 × 1 to indicate image areas with poor quality. Considering that cross-spectral images have different intensity ranges and visual effects, we apply two batch normalization (BN) layers on 𝐹 ′𝑡-1 𝑁 𝐼𝑅 , 𝐹 𝑡-1 𝑉 𝐼𝑆 to unify the NIR-VIS image features and highlight the relative difference. Then, to rebalance features of different modalities, we also learn a channel weight (𝑤𝑐 ∈ 𝑅 1×1×2𝐶 ) by passing the 𝐹 ′𝑡-1 𝑁 𝐼𝑅 and 𝐹 𝑡-1 𝑉 𝐼𝑆 to a global average pooling layer, a convolution layer with kernel size 1 × 1 × 2𝐶, and a sigmoid activation layer, respectively. Next, 𝑤𝑐 is split into two vectors (𝑤𝑐1, 𝑤𝑐2 ∈ 𝑅 1×1×𝐶 ) to re-weight the 𝐹 ′𝑡-1 𝑁 𝐼𝑅 , 𝐹 𝑡-1 𝑉 𝐼𝑆 based on their channel-wise correlation and significance.Given the concatenation of 𝐹 ′𝑡-1 𝑁 𝐼𝑅 and 𝐹 𝑡-1 𝑉 𝐼𝑆 , another spatial weight map (𝑤𝑠2) is generated through a convolution layer with kernel size 3 × 3 × 1 to identify the accurate and useful NIR image features. Finally, we combine the 𝐹 ′𝑡-1 𝑁 𝐼𝑅 and 𝐹 𝑡-1 𝑉 𝐼𝑆 by weighted summation based on the weights 𝑤𝑠1, 𝑤𝑠2, 𝑤𝑐1, 𝑤𝑐2 to transfer the abundant information from the NIR image to the VIS image.EQUATIONwhere × denotes the element-wise multiplication.Given the VIS-NIR outputs (𝐼 𝑙 𝑆𝑅 , 𝑁 𝐼𝑅 𝑙 𝑆𝑅 ) and the GT crossspectral images (𝐼 𝑙 𝐻𝑅 , 𝑁 𝐼𝑅 𝑟 𝐻𝑅 ), our CSPSR model is trained end-to-end using the final loss (ℒ) in Eq.( 3). In addition to the mean square error (MSE) loss, used to constrain the pixel-level accuracy between the 𝐼 𝑙 𝑆𝑅 and 𝐼 𝑙 𝐻𝑅 , we also propose a cross-spectral spatial consistency loss (ℒ𝐶𝑆𝑆𝐶 ) for optimizing the network to learn better NIR information. Since calculating the MSE between the misaligned NIR images (𝑁 𝐼𝑅 𝑙 𝑆𝑅 , 𝑁 𝐼𝑅 𝑟 𝐻𝑅 ) directly will lead to spatial artefacts in the NIR image, we first obtain the HR left NIR image (𝑁 𝐼𝑅 𝑙 𝐻𝑅 ) and the SR right NIR image (𝑁 𝐼𝑅 𝑟 𝑆𝑅 ) by warping the 𝑁 𝐼𝑅 𝑟 𝐻𝑅 and the 𝑁 𝐼𝑅 𝑙 𝑆𝑅 based on the disparity (𝑑𝑖𝑠𝑝 𝑙 , 𝑑𝑖𝑠𝑝 𝑟 ) (Eq.( 1)). Then, the ℒ𝐶𝑆𝑆𝐶 calculates the MSE between the aligned NIR image pairs, including (𝑁 𝐼𝑅 𝑙 𝑆𝑅 , 𝑁 𝐼𝑅 𝑙 𝐻𝑅 ) and (𝑁 𝐼𝑅 𝑟 𝑆𝑅 , 𝑁 𝐼𝑅 𝑟 𝐻𝑅 ).EQUATIONwhere the weight 𝜆1 is set to 0.1.The final CSPSR model contains 15 DMF modules (𝑇 = 15) in total. The kernel size of the convolution layers in two SR branches is 3 × 3 and the feature number 𝐶 is 64. 
experiments: 
 This section mainly introduces experimental settings and reports the performance of our approach by conducting the Table 1 : Quantitative comparison with state-of-the-art SR approaches. The average PSNR↑/SSIM↑ of VIS images on three NIR-VIS datasets (RGB-NIR stereo [66] , NIRScene [1] , RANUS [6] ). ↑ denotes the higher, the better. The best results are highlighted in bold. The GFLOPs/Param(MB) denote the calculation amounts and the parameter amounts of different SR models. , ENLCN [47] , HAT [3] ). The ×4 VIS SR results on the RGB-NIR stereo [66] and NIRScene [1] datasets. comparison experiment and the ablation experiment. Furthermore, the cross-spectral disparity prediction results are provided to verify the practicability of the CSPSR.We train our models on the RGB-NIR stereo dataset [66] , which contains 12 videos in total. The RGB-NIR stereo dataset contains 8 training videos and 4 testing videos, where each video covers 500 pairs of RGB-NIR cross-view frames with spatial size 582×429. We employ the RGB-NIR stereo, the NIRScene [1] (477 pairs of NIR-VIS images), and the RANUS [6] (40k pairs of NIR-VIS images) datasets for evaluation. Specifically, images in four testing videos of the RGB-NIR stereo dataset, 46 pairs of NIR-VIS images, corresponding to 9 categories, from the NIRScene dataset, and the 50th subset (104 NIR-VIS image pairs) in the RANUS dataset are adopted to test different SR models.During training, each mini-batch contains 32 pairs of crossspectral image patches of size 120×120. The LR NIR/VIS images are generated by bicubic interpolation with scales 2, and 4. We augment the training data by randomly downscaling, flipping, and rotating images. To measure the SR results, we adopt the peak signal-to-noise ratio (PSNR) and structure similarity (SSIM) [67] . The higher PSNR/SSIM indicates better performance.All models are based on the Pytorch implementation and optimized by Adam [32] with 𝛽1 = 0.9, 𝛽2 = 0.999. All experiments are conducted on an Nvidia GTX2080Ti GPU (128G RAM). Our model is optimized for 300 epochs with a learning rate of 1e-4.Table 2 : Quantitative comparison with prior-guided SR methods, including StereoSR (PASSR [43] , NAF-SSR [7] ), reference-based SR (MASA [27] ), and multi-modal SR (CMSR [49] ) models. SISR methods. To our best knowledge, existing SR methods have never explored the NIR guidance for the SR of VIS images. In order to compare with the state-of-the-art, we train and evaluate several representative SISR models, including EDSR [23] , RCAN [64] , SMSR [42] , NLSN [59] , ENLCN [47] , HAT [3] , on the VIS images of the RGB-NIR stereo dataset. Table 1 reports the quantitative SR results (PSNR/SSIM) of the VIS images on three datasets and also compares the calculation amounts (GFLOPs) and parameter amounts of different SR models. Our CSPSR substantially achieves the best PSNR/SSIM with acceptable calculation amount and parameter amount. Figure 5 demonstrates that the proposed CSPSR is able to produce photo-realistic images with accurate structure and clear textures, resulting in satisfactory visual performance.Prior-guided SR methods. To further verify whether the existing prior-guided SR methods work well on our problem, we retrain the StereoSR (PASSR [43] , NAFSSR [7] ), the reference-based SR (MASA [27] ), and the multi-modal SR (CMSR [49] ) models on our training data. Specifically, we replace the stereo VIS images with the NIR-VIS images as the input for the StereoSR model. Since our model only requires LR NIR image guidance, we take the Bicubic upscaled NIR image instead of the HR VIS image as the reference to guide the SR of the VIS image to the MASA and the CMSR models for fair comparison. We also construct the CMSR (w/ HR NIR) by taking the HR NIR image as guidance, which reflects the possible upper bound of the NIR-guided VIS image SR. Note that, the NIR-VIS images are first aligned through the cross-spectral stereo matching model [66] before SR.Table 2 and Figure 6 demonstrate the quantitative and visual comparison on ×4 SR. Though our CSPSR only leverages LR NIR image, its SR results are comparable with that of the HR NIR image-guided CMSR (w/ HR NIR), which indicates the effectiveness of our strategy to exploit cross-spectral and cross-view information for SR.To verify the effectiveness of the proposed module for using NIR image information, we retrain a SISR model [47] (ENL-CN w/ NIR), which takes the concatenation of RGB and NIR images as input. Table 3 reports the ×4 SR results on the RGB-NIR stereo dataset [66] . We can observe a slight We use '𝐹 𝑢𝑙𝑙' to denote the complete model, based on which the '𝑤/𝑜 𝐶𝑉 𝑀 ' and the '𝑤/𝑜 𝐶𝑆𝐹 𝑇 ' remove the CVM module and the CSFT unit, respectively. The '𝑤/𝑜 𝐶𝑉 𝑀 ' fuses NIR-VIS image features by directly concatenating them. The comparison verifies that the CVM module enables the NIR image to provide a more valuable reference for recovering image details and the CSFT module is helpful for the model to better fuse and utilize multi-modal features for the VIS image SR. The '𝑤/𝑜 𝐷𝑦𝐶𝑜𝑛𝑣' replaces the dynamic convolution with the common convolution.The '𝑤/𝑜 ℒ𝐶𝑆𝑆𝐶 ' only calculates the MSE between the SR results and the GT of VIS images and ignores the explicit constraint for the NIR SR branch and the CVM module, obtaining lower accuracy. In conclusion, the '𝐹 𝑢𝑙𝑙' can adaptively exploit valuable NIR image information, leading to the highest PSNR/SSIM. In Figure 7 , the SR results of the '𝐹 𝑢𝑙𝑙' contain finer textures that are sharper and closer to the GT, compared with that of the 'w/o CVM' and 'w/o CSFT'.All the above experiments demonstrate the reasonability and effectiveness of the proposed network architecture and modules, which facilitate better SR by fully using the guidance of the cross-view NIR image.Table 5 reports the disparity RMSE, calculated on different SR results on the video '0224 0742' of the RGB-NIR stereo dataset [66] . Figure 8 visualizes the cross-spectral disparity. This experiment demonstrates that our CSPSR provides high-quality cross-spectral images for the downstream task.To further demonstrate the practical value of the CSPSR, we estimate the cross-spectral disparity between the SR VIS image and the HR NIR image with a cross-spectral stereo matching [66] method. All test images of the RGB-NIR Stereo dataset are labelled with material segments in 8 classes, including common, light, glass, glossy, vegetation, skin, clothing, and bag. By following this work, we evaluate the disparity accuracy through the disparity root mean square error (RMSE) in pixels for each material.The main contribution of this paper is introducing crossspectral guidance in SR of images from the multi-camera cross-spectral system. Though the near-infrared images provide benefits for the super-resolution of visible images, as we mentioned before, it may not help much for some areas with normal brightness, where the visible images contain enough high-frequency information. Therefore, how to better exploit the near-infrared image guidance for more efficient computing returns, needs to be further studied. 
conclusion: 
 In this paper, we have proposed a novel cross-spectral prior guided image super-resolution (CSPSR) approach, which makes the first attempt to introduce the near-infrared (NIR) image to assist the visible image SR. The proposed CSPSR can reasonably exploit the cross-spectral guidance for recovering accurate structures and clear details through a cross-view matching (CVM) module and a dynamic multi-modal fusion (DMF) module. Specifically, the CVM module enhances the cross-view correspondence, which facilitates cross-spectral images providing more valuable and appropriate guidance for SR. The DMF module adopts a dynamic convolution and a cross-spectral feature transfer unit to adaptively enhance the multi-modal features and fully fuse the cross-spectral information for predicting abundant and realistic image details. Extensive experiments have demonstrated that the CSPSR can take full advantage of the NIR information to restore high-quality images with accurate structures and clear details, obtaining superior SR results compared to the state-of-theart.  
: 
 , ENLCN [47] , HAT [3] ) for the ×4 SR. The proposed approach can effectively use the NIR image as guidance to restore a better high-resolution VIS image with clear details and fine structure.popular. Cross-spectral images receive wide attention in the computer vision field and provide strong benefits for numerous practical applications, such as scene parsing [5] , person re-identification [11, 29] , face recognition [4, [13] [14] [15] , object detection [12, 25, 39] . These applications always require high-resolution (HR) images. Therefore, the super-resolution of cross-spectral images (CSSR), producing high-resolution (HR) images from their low-resolution (LR) versions, is significant. However, real-world cross-spectral images always suffer from complex degradation, such as noise, high brightness, and low light, making the CSSR challenging. Existing SR methods (NLSN [59] , ENLCN [47] , HAT [3] ) cannot perform well on cross-spectral images (see Figure 1 ).The near-infrared (NIR) image and the visible (VIS) RG-B image of the cross-spectral images in Figure 2 present different inherent characteristics. Compared with the VIS image, the NIR image retains better brightness contrast and richer texture details in some overexposed or dark areas and provides clearer boundaries of the texts, as it is robust to the change of colour and is sensitive to the change of illumination. In addition, the NIR image can resist the disturbance of bad imaging conditions, such as low illumination, and fog. Therefore, the NIR image is able to preserve some valuable information for recovering adequate VIS image details and to provide many benefits for the VIS image SR.The above analysis inspires us to propose a cross-spectral prior guided super-resolution (CSPSR) approach by introducing the NIR guidance into the SR of the VIS image. Figure 1 (a) demonstrates a pair of LR cross-spectral images and the disparity between them. The multi-modal feature gap and cross-view pixel misalignment make the CSPSR challenging. To our knowledge, most current SR methods mainly work on images of single-modal or single-view and rarely exploit the cross-spectral image information (e.g., NIR) to guide the (VIS) image SR. Therefore, it is suboptimal to directly apply existing SR methods to the CSPSR.How to bridge the multi-modal feature gap and enhance the cross-view sub-pixel correspondence are prominent insights of this paper. The proposed CSPSR approach can restore accurate HR images by taking full advantage of the cross-spectral and cross-view image information. Specifically, we enhance the pixel-level correspondence between different views through a cross-view matching (CVM) module to provide more appropriate NIR guidance for the VIS image. To fully fuse multi-modal features of cross-spectral images, we design a dynamic multi-modal fusion (DMF) module, composed of a dynamic convolution and a confidence-based cross-spectral feature transfer (CSFT) unit. The dynamic convolution adaptively adapts the NIR feature to better match the VIS feature, and the CSFT unit transfers reliable knowledge from the NIR image to the VIS image by learning different confidence maps.As shown in Figure 1 (b), our CSPSR can effectively utilize the NIR information to produce an HR VIS image with clear textures and structures, that are closer to the ground truth (GT). The highlights of our work are as follows.• We analyze the inherent characteristics of the crossspectral images and propose a novel cross-spectral prior guided image super-resolution (CSPSR) approach, which introduces the NIR image to assist the SR of the VIS image for the first time. • We propose a cross-view matching (CVM) module and a dynamic multi-modal fusion (DMF) module to take full advantage of the cross-spectral and cross-view image information for better SR. The CVM module enhances the cross-view spatial correspondence. The DM-F module adaptively bridges the multi-modal feature gap and transfers cross-spectral knowledge through a dynamic convolution and a cross-spectral feature transfer (CSFT) unit. • We design a dual-branch framework for extracting multi-modal features of cross-spectral images to provide an informative reference related to SR. 
 
 Appendix:  


Decision: The paper proposes a novel model network CSPSR to exploit the cross-spectral and cross-view characteristics of images to restore high-resolution image.
The paper is well written. Several minor concerns have been raised about experimentation. The authors should aim to address those in the final draft.

Reviews: review: paper_summary: paper_summary: The paper tries to use near-infrared (NIR) image guidance for better SR, based on the observation that NIR images can preserve valuable information for recovering adequate image details.
strength_weakness: strength_weakness: 1. The analysis of cross-spectral SR in this paper is very detailed, pointing out some challenges, and Motivation is reasonable.
2. Compared with some algorithms, the proposed algorithm achieves better results.
3. The description of this article is relatively clear.
4. The experiments are relatively sufficient, and all parts have been tested.
limitations: limitations: The method of using cross-spectral priors was proposed a long time ago, and the use of priors lacks innovation.
paper_summary: paper_summary: This paper introduces an innovative approach to image super-resolution termed Cross-Spectral Prior Guided Image Super-Resolution (CSPSR), marking the first instance where a Near-Infrared (NIR) image is utilized to enhance the super-resolution of a Visible (VIS) image. The DMF module is designed to dynamically connect the gap between different feature modalities and facilitate the transfer of knowledge across spectral domains through the use of dynamic convolution and a Cross-Spectral Feature Transfer (CSFT) unit in a dual-branch framework .
strength_weakness: strength_weakness: This paper is well-structured, and the English expression is good.
This paper is a work to explore cross-spectral prior guided image super-resolution (CSPSR) approach, which introduces the NIR image to assist the SR of the VIS image for the first time.
The experiments in this paper retrained six SISR methods and five Prior-guided SR methods on the RGB-NIR stereo dataset. Extensive comparative experiments demonstrate the superior performance of the methods proposed in this paper.
limitations: limitations: The experiment regarding Cross-spectral disparity estimation results on Super-Resolution (SR) images only compared the effects of Single Image Super-Resolution (SISR) methods. However, judging from the previous experiments, Prior-guided Super-Resolution methods may have better experimental outcomes. Comparing only SISR is not convincing; it lacks more reasonable comparative methods. For instance, "Learning Spatial-Spectral Prior for Super-Resolution of Hyperspectral Imagery."

Limited novelty: This paper merely designs a dynamic multi-modal fusion module based on dynamic convolution, which employs a dual-branch approach, and this is not a particularly novel practice.
paper_summary: paper_summary: This paper uses the characteristics of cross-spectral images to design a model network CSPSR. It utilizes cross-spectral and cross-view image information to restore HR images. The network enhances pixel correspondence through cross-view matching module. And the author uses dynamic convolution and a confidence-based cross-spectral feature transfer unit to build a dynamic multi-modal fusion module to fuse the features of the cross-spectral image, and then transfer the features of the NIR image to the VIS image.
strength_weakness: strength_weakness: The paper proposes to fuse features of cross-spectral images to guide super-resolution, which is a very interesting idea.

The experiment was very thorough.
limitations: limitations: 1.Although combining cross-spectral image features is a good idea, I have not been able to see its specific effect on VIS image super-resolution from the analysis you provided. I hope you can provide a more detailed explanation.
2.The paper states that CSPSR can use the characteristics of NIR images to guide the super-resolution of VIS images, but the data in the experiment mainly uses cross-spectral images, so I have doubts about whether this network structure can have good general applicability.
paper_summary: paper_summary: This paper proposes an image super-resolution method that utilizes cross-spectral information to assist visible light images. The authors designed a cross-view matching (CVM) module and a dynamic multi-modal fusion (DMF) module to enhance the spatial correlation between cross-spectral images.
strength_weakness: strength_weakness: Using cross-spectral information to assist super-resolution is a relatively innovative idea. To my knowledge, no one has ever done this. The experiments in the paper are also quite sufficient. Comparing the images, we can see that the method is improved compared with the previous methods.
limitations: limitations: The data set used in the article's experiment is not commonly used in super-resolution such as Set5 and Urban100. I can understand the authors' desire to demonstrate the advantages of CSPSR on cross-spectral images. However, as a super-resolution method, it should still provide data sets other than cross-spectral images. The comparison methods in this paper are not aimed at cross-spectral images. I think the authors should provide some existing methods that work on cross-spectral images in the comparison method. This will make the experiment more convincing.