Reference-based Burst Super-resolution

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-03-14 05:25:42

Abstract: Burst super-resolution (BurstSR) utilizes signal information from multiple adjacent frames successively taken to restore rich textures. However, due to hand tremors and other image degradation factors, even recent BurstSR methods struggle to reconstruct finely textured images. On the other hand, reference-based super-resolution (RefSR) leverages the high-fidelity reference (Ref) image to recover detailed contents. Nevertheless, if there is no correspondence between the Ref and the low-resolution (LR) image, the degraded output is derived. To overcome the limitations of existing BurstSR and RefSR methods, we newly introduce a reference-based burst super-resolution (RefBSR) that utilizes burst frames and a high-resolution (HR) external Ref image. The RefBSR can restore the HR image by properly fusing the benefits of burst frames and a Ref image. To this end, we propose the first RefBSR framework that consists of Ref-burst feature matching and burst feature-aware Ref texture transfer (BRTT) modules. In addition, our method adaptively integrates features with better quality between Ref and burst features using Ref-burst adaptive feature fusion (RBAF). To train and evaluate our method, we provide a new dataset of Ref-burst pairs collected by commercial smartphones. The proposed method achieves state-of-the-art performance compared to both existing RefSR and BurstSR methods, and we demonstrate its effectiveness through comprehensive experiments. The source codes and the newly constructed dataset will be made publicly available for further research.

Main paper: 
 introduction: 
 Burst photography has become popular in recent years with the increasing use of smartphones. Since burst super-resolution (BurstSR) utilizes sub-pixels in multi-frames, high-resolution (HR) images can be obtained. One of the key challenges of BurstSR is to align and fuse sub-pixels between frames into a base frame (i.e.the first frame). However, due to different brightnesses or large movements between frames, sub-pixels can be misaligned, leading to incorrect results. On the other hand, reference-based super-resolution (RefSR) Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from permissions@acm.org. ACM MM, 2024, Melbourne, Australia On the other hand, in case of failed matching, the performance of the RefSR method is degraded as shown in the yellow dotted box. If the RefSR method is less helpful due to failed matching, sub-pixels of the burst frame can be utilized to recover the region as shown in the yellow dotted box of the BurstSR result. These examples demonstrate the advantages and disadvantages of each technique. Based on the above observations, the goal of this paper is to combine the strengths of each method and mitigate the weaknesses of them. In this regard, we introduce a new unexplored task: reference-based burst super-resolution (RefBSR) that integrates the two methods. The outcomes of RefBSR showcased in the red dotted box in Fig. 1 preserve details of the window textures present in the building. These results closely match the detailed outcomes obtained from RefSR. On the other hand, the results of RefBSR presented in the yellow dotted box show the textural details in the exterior of the building, similar to the detailed results of BurstSR. Therefore, our proposed RefBSR integrates the high-frequency texture from the Ref image and the multiple clues from the sub-pixels in the burst frames.To achieve a suitable fusion of the two approaches, we propose the burst feature-aware Ref texture transfer (BRTT) module. The BRTT module aligns the Ref feature using each burst feature individually into the grid of the base frame.  
burst super-resolution: 
 In contrast to RefSR, burst super-resolution (BurstSR) [1-3, 6-8, 12, 14, 19-22] exploits multiple shifted LR frames that contain the same contents to recover the high-quality image. Therefore, it is important to align features from multi-frames and aggregate them in the BurstSR task. In recent years, many BurstSR approaches based on deep learning have been proposed. For instance, Deudon et al.proposed HighResNet [6] using multi-frames for remote sensing images. Also, Bhat et al. [1] proposed DBSR based on PWC [26] network for the feature alignment and aggregation using attention weights. For improving the performance, MFIR [2] using the deep reparameterization of MAP and EBSR [20] based on the PCD with multi-scales mechanism were suggested. Furthermore, Luo et al. [19] introduced BSRT based on an optical flow estimator to align more robustly with PCD [28] module. Recently, Mehta et al. [22] proposed GMTNet that utilizes the attention mechanism to generate the offset for better alignment with the deformable convolution. In Section 3.2, we first perform the base and reference frame matching to obtain a flow field 𝑉 𝑏 1 →𝑟 from the base frame to the reference frame. In Section 3.3, we obtain burst feature-aware aligned Ref featuresReference 𝐼 ! Burst Frames ℐ " RBAF 𝐹 $ !→" ! % 𝐹 $ & % RBAF 𝐹 " " # RBAF 𝐹 $ !→" ! ' 𝐹 $ & ' … BRTT BRTT BRTT 𝐹 " $→& ! # Output 𝐼 '$ 𝑉 " ! →$ 𝐹 $ × N Reference Encoder Encoder Decoder 𝐹 # " " →" ! %&' ( 𝐹 $ " ! →" ! 𝐹 $ " ) →" ! 𝐹 $ " * →" !{ F𝑖 𝑟 →𝑏 1 } 𝑁𝑖=1 by transferring a Ref feature to the base frame based on burst frame features using the BRTT module. The Ref feature 𝐹 𝑟 is extracted from 𝐼 𝑟 by VGG16 [25] , while 𝑁 burst features { F𝑏 𝑖 →𝑏 1 } 𝑁 𝑖=1 , which are aligned to the base frame, are extracted from I 𝑏 by FG-DCN [5, 19] . In Section 3.4, we develop the Ref-burst adaptive feature fusion to combine { F𝑖𝑟 →𝑏 1 } 𝑁 𝑖=1and { F𝑏 𝑖 →𝑏 1 } 𝑁 𝑖=1 effectively. Last, the decoder generates the superresolved frame 𝐼 𝑠𝑟 from the fused feature. 
base and reference frame matching: 
 In order to transfer the texture from the Ref image 𝐼 𝑟 to the lowresolution base frame 𝐼 𝑏 1 effectively, it needs to find correspondences between 𝐼 𝑟 and 𝐼 𝑏 1 . We first obtain a base frame matching feature 𝑀 𝑏 1 = 𝜙 (𝐼 𝑏 1 ) and a Ref matching feature 𝑀 𝑟 = 𝜙 (𝐼 𝑟 ) by employing a feature extractor 𝜙 (•) in [23] . Then, we compute the patch-based correlation between features 𝑀 𝑏 1 and 𝑀 𝑟 to obtain the flow filed 𝑉 𝑏 1 →𝑟 for matching from the base frame to the Ref frame. The correlation at pixel x for a flow vector v is defined asEQUATIONwhere 2𝑘 + 1 is a patch size. Thus, 𝐶 (x, v) denotes the correlation between patches centered at x in 𝑀 𝑏 1 and centered at x + v in 𝑀 𝑟 . Note that we compute all correlations between every pixel in the base and Ref frames for global matching. Then, the flow vector 𝑉 𝑏 1 →𝑟 (x) for pixel x is defined asEQUATION3. 𝐹 𝑟 →𝑏 1 (x) = 𝐹 𝑟 (x + 𝑉 𝑏 1 →𝑟 (x)).(3) Here, 𝐹 𝑟 (x + 𝑉 𝑏 1 →𝑟 (x)) is computed by bilinear interpolation. Considering that some pixels in the base frame may not be well matched to the Ref frame, we refine the flow 𝑉 𝑖 𝑏 1 →𝑟 by exploiting the aligned burst feature F𝑏 𝑖 →𝑏 1 . Thus, 𝐹 𝑟 →𝑏 1 and F𝑏 𝑖 →𝑏 1 are concatenated and processed by a 3 × 3 convolution layer to obtain the dynamic flow-based offset (DFO) Ṽ 𝑖 𝑏 1 →𝑟 , which is given by ṼEQUATIONwhere Conv(•) and [•, •] denote the convolution and concatenation operations. We adopt a deformable convolution to generate a burst feature-aware aligned Ref featureEQUATION)where DCN is a function for the deformable convolution and the dynamic flow-based offset Ṽ 𝑖 𝑏 1 →𝑟 is used in the deformable convolution. BRTT is repeated for each burst feature, and thus burst feature-aware aligned Ref features { F𝑖 𝑟 →𝑏 1 } 𝑁 𝑖=1 are obtained.We combine the aligned Ref and burst features {( F𝑖𝑟 →𝑏 1 , F𝑏 𝑖 →𝑏 1 )} 𝑁 𝑖=1to integrate the information of high-frequency texture in the Ref image and multiple clues of the sub-pixels in the burst frames. Even though the two features can be simply concatenated for the feature fusion, we expect the feature fusion to be enhanced by assigning adaptive weights along feature channels for each position. For instance, in Figure 4 Next, we compute a weighted feature F𝑖 𝑤 using the adaptive fusion weight map 𝐴 asEQUATIONwhere ⊗ denotes an element-wise multiplication operation. To this end, the weighted feature F𝑖 𝑤 is processed by a convolution layer and a residual block to yield a fused Ref-burst feature F𝑖 𝑓 . We repeat the adaptive feature fusion process for all pairs {( F𝑖𝑟 →𝑏 1 , F𝑏 𝑖 →𝑏 1 )} 𝑁 𝑖=1 , resulting in 𝑁 fused features { F𝑖 𝑓 } 𝑁 𝑖=1 .Finally, the 𝑁 fused features are concatenated and processed by the decoder to produce the super-resolved frame 𝐼 𝑠𝑟 . A detailed explanation of the decoder is in the supplementary material.To accomplish our goal of reference-based burst super-resolution, we construct a new dataset, called RefBSR, for the training and testing stages.For dataset construction, we first capture 232 pairs of RAW Ref and burst images using the custom camera app in iPhone 14 pro. Specifically, we obtain 14 burst frames for each sample using continuous shooting. To collect Ref images, we take a picture of scenes with different locations, viewpoints, and time from the burst frames. Unlike the BurstSR dataset [1] , we collect the dataset by hand without a tripod for realistic hand tremors. In other words, some burst frames in our dataset may contain motions with each other due to hand tremors.Since the captured images are in RAW format, we transform the first burst frames and Ref images into 3-channel color images through demosaicking, where the transformed first burst frames are used as the ground truth (GT). For LR images, the burst frames are downsampled ×1/4 by bicubic interpolation with the Bayer  
experiments 4.1 training details: 
 Datasets and Evaluation. For both training and testing, we use the representative synthetic burst dataset [1] , as well as our Ref-BSR dataset as introduced in Section 3.5. Given that the existing synthetic burst dataset [1] exclusively provides burst frames, we generate the synthetic Ref images. We apply a random perturbation to the GT images between 0-20 to create Ref images as introduced in [10] . The synthetic dataset provides input burst frames of size 4×96×96 with a Bayer pattern and Ref images of size 3×384×384. The size of GT image is also 3×384×384. In consequence, we utilize 46,839 pairs for the training dataset and 300 pairs for the testing set. We employ PSNR, SSIM [29] , and LPIPS [35] metrics for evaluation. Note that all methods are evaluated on the linear sensor space.Results are evaluated on the linear sensor for the metrics.Implementation details. To ensure smooth training, we first train the proposed network with the synthetic dataset, and then train it with the RefBSR dataset. We set epoch to 80 and batch size to 16 for training with the synthetic dataset. We select ADAM [11] as the optimizer and set the learning rate to 1e-4 and the betas to 𝛽 1 = 0.9 and 𝛽 2 = 0.99. For training with the RefBSR dataset, we opt for the same setting with the optimizer, learning rate and betas as the synthetic dataset, while we set epoch to 130 and batch size to 8. The proposed model is trained using the 𝐿 1 lossEQUATION)where 𝐼 𝑠𝑟 and 𝐼 𝑔𝑡 denote the super-resolved result and the groundtruth, respectively. Our method is trained for about 3 to 4 days with A6000 GPUs.Quantitative Comparison. We compare our method with the SISR [15] , RefSR [10, 34] , BurstSR [1, 2, 8] , and RefVSR [13] methods. Note that RefBSR is a newly unexplored task, and thus we pick the state-of-the-art SISR, RefSR, and BurstSR, RefVSR to validate the effectiveness of the proposed network. For fairness, all existing methods are firstly trained with the synthetic dataset and then trained on the RefBSR dataset as done in our model training. We train SISR methods using the base frame only, Also, RefSR methods are trained with the base and Ref images, while BurstSR methods are trained with all burst frames. Table 1 shows quantitative results on the synthetic and the Ref-BSR datasets. In the RefSR method, 𝐶 2 -Matching [10] and LMR [34] provide a lower performance than SwinIR [15] on the RefBSR dataset, and we assume that the feature extractor for matching in [10] degrades the performance. LMR † indicates another experiment with different feature extractor [23] for matching. When the feature extractor in LMR is replaced with the more effective feature extractor [23] , LMR † outperforms SwinIR on the RefBSR dataset. In BurstSR methods, since Burstormer [8] does not utilize the flow estimator, it yields a lower performance than the previous BurstSR models [1, 2] on the RefBSR dataset, which contains realistic movements. Finally, our proposed method outperforms the existing methods for all metrics on both synthetic and RefBSR datasets. This indicates that our model effectively combines the high-frequency texture from the Ref image and the multiple subpixel cues in the burst frames. Comparing our RefBSR model to the second best method MFIR [2] , we achieve a performance improvement of over 1.45dB in terms of PSNR on the RefBSR dataset. Qualitative Comparison. Figure 6 shows qualitative results of the proposed method, and the state-of-the-art models including SwinIR [15] , LMR [34] , and MFIR [2] on the RefBSR dataset. Our model faithfully recovers detailed textures by fusing multiple LR frames and the Ref image. For instance, in the 3rd row, even though the Ref image and LR base frame contain different brightness, the textures of the structure in our model are visible more clearly than in other models. Also, in the last row, our result contains highfidelity letters, e.g.'2023', as compared with other methods.To demonstrate the effectiveness of our proposed model, we perform ablation studies on the synthetic and the RefBSR dataset.In Table 2 , we design the baseline model Base without the aligned burst feature in (4) in the BRTT module and without the RBAF module. Thus, in Base, only Ref feature is used to update the flow filed 𝑉 𝑏 1 →𝑟 in (4) without the information of burst features. Also, in Base, we concatenate the aligned burst and Ref features and pass it to the RSB blocks without the RBAF module. Next, DFO+Base indicates that the dynamic flow-based offsets (DFO) are included in Base, which is equivalent to the proposed model without the RBAF module. Finally, RBAF+DFO+Base is the final version of our model. In Table 2 , we observe that the performance gradually increases on the synthetic and RefBSR datasets as we add each module one by one. Further analysis of each method is detailed in the following explanations.Analysis on BRTT Module. We assert that various aligned reference features can be obtained by exploiting the burst features, and those burst feature-aware Ref features transfer rich textures in the Ref image to the base frame effectively. In Tab. 2, when we add the aligned burst features to generate the dynamic flow-based offsets (DFO+Base), there is a performance improvement against Analysis on RBAF Module. By applying the adaptive weights to the aligned Ref feature and the aligned burst feature, the RBAF module ensures that the two features are fused properly. In Tab. 2, we verify the effectiveness of the RBAF module in terms of quantitative results on both datasets. As shown in Figure 8 , when we add the RBAF module to DFO+Base, RBAF+DFO+Base exhibits more faithful restoration. Specifically, the structure of the windows of Base+DFO+RBAF on two scenes is clearly reconstructed. These visual results, as well as the quantitative results, demonstrate that the proposed RBAF module focuses more on the important information in each feature. Approach of Ref-Burst Fusion. To combine the aligned burst and the Ref features, it is an issue "how" to integrate burst and Ref features. One of the naive approaches is that all burst features are concatenated and then the Ref feature is fused to the simply aggregated burst feature. We refer to this approach as the group-fusion method. Specifically, in the group-fusion method, the aligned burst feature in (4) for BRTT and (6) for RBAF is replaced with the simply aggregated burst feature. Another option is the individual-fusion method that fuses the Ref feature to each burst feature, which is adopted in our model. Table 3 quantitatively compares the SR performance of the group and individual fusion methods. We see that the individual fusion approach is more effective than the group method. The advantage of individual fusion over group fusion is that we can obtain dynamic flow-based offsets using individual burst features. In contrast, the group fusion provides only one offset for the alignment of Ref image. These dynamic flow-based offsets work well to exploit sub-pixel information in multiple burst frames, and thus remarkable performance is achieved. Furthermore, Figure 9 illustrates that the individual fusion method produces relatively sharper results than the group method. These observations demonstrate that the individual fusion method is more effective than the group fusion method.Effect of Reference and Burst Frame. When we construct the synthetic dataset as described in Section 4.1, we employ a random permutation to the ground-truth image for generating Ref images. In the Ref image generation process, we divide the samples into three similarity levels according to the permutation by following [10] : easy (0-10), medium (10) (11) (12) (13) (14) (15) (16) (17) (18) (19) (20) , and hard (20) (21) (22) (23) (24) (25) (26) (27) (28) (29) (30) . Table 5 shows that the performance of our model gradually increases as the similarity between base and Ref images increases. This trend demonstrates that our model exploits the reference information effectively. Dual Camera Setting. Our model readily extends to the Dual lens setting. We conduct analyses on dual rigs. For this system, we collect a total of 72 sets of datasets by configuring bursts to wide-angle and utilizing Ref as telephoto image. In the dual camera setting, RefSR encounters a challenge due to the absence of correspondences between the wide-angle as the burst frames and the telephoto as the Ref image, leading to a degradation in quality. One plausible approach to address this issue is to consider employing burst frames in regions lacking correspondences. As demonstrated in Table 6 , our model exhibits superior performance compared to previous RefSR, BurstSR, RefVSR method. Furthermore, in the Figure 10 , we observe remarkably sharp results in the areas overlapping with Ref. Moreover, in the non-overlapping regions with Ref, there is sufficient utilization of burst frames to capture detailed information. These results demonstrate that our model operates robustly in dual camera system. 
conclusion: 
 We introduce a new framework for the reference-based superresolution (RefBSR) with a new dataset. The proposed RefBSR framework reconstructs the super-resolved image using the lowresolution burst frames and the reference image as the inputs. For this purpose, the burst feature aware Ref texture transfer (BRTT) module conveys the fine texture in the reference image to each burst feature. To achieve the reference alignment, we utilize the burst feature to obtain the wide range of the aligned reference feature in the BRTT module. The ref-burst adaptive feature Fusion (RBAF) gives the attention weights to each aligned burst-reference feature and fuses them. We provide a new dataset for the RefBSR task consisting of pairs of the RAW burst frames, the reference image and the ground truth image. Extensive experiments verify our framework and modules on the proposed dataset.  
 
 Appendix:  


Decision: The authors provided a response and all the reviewers are generally positive with respect to the contributions from this paper.

The meta-reviewer agrees with the reviewers and invites the authors to further refine their work for the camera ready by benefiting from the received feedback and integrating contents from the authors' response.

Reviews: review: paper_summary: paper_summary: This paper proposes a reference-based burst super-resolution (RefBSR) method that combines the advantages of Burst Super-Resolution (BurstSR) and Reference-based Super-Resolution (RefSR) to enhance the quality of super-resolved images. By harnessing both multiple burst frames and a high-resolution reference image, RefBSR efficiently restores finely textured images that were previously challenging due to factors such as hand tremors. The proposed framework incorporates innovative modules like Ref-burst feature matching and Burst Feature-Aware Reference Texture Transfer (BRTT), alongside Ref-burst Adaptive Feature Fusion (RBAF), which adaptively integrates high-quality features from both reference and burst sources. The proposed dataset consisting of Ref-burst pairs collected from commercial smartphones supports the training and evaluation of this framework, which has demonstrated state-of-the-art performance in comprehensive experiments.
strength_weakness: strength_weakness: 1. Combines BurstSR and RefSR methodologies to leverage the strengths of both, enhancing texture restoration in super-resolved images.
2. Provides a dataset specifically designed for RefBSR, allowing for training and comprehensive evaluation of the method.
3. Achieves superior results compared to existing BurstSR and RefSR methods, as demonstrated in extensive experiments.
limitations: limitations: 1. The motivation and issues are not clear. The authors state that "if there is no correspondence between the Ref and the low-resolution (LR) images". However, RefSR has considered correspondence matching between them. Could you provide more details on this?

2. In Figure 1, the result of RefSR seems to be problematic. The based frame has vertical textures, but it is removed by RefSR. This may not happen in RefSR.

3. The authors state that the proposed method is the first framework for reference-based burst super-resolution (RefBSR). However, there is one paper [a] also using reference-based techniques. Could you discuss the difference between them?

    [a] Burst Super-Resolution with Diffusion Models for Improving Perceptual Quality 

4. In Figure 2, what happens when Reference I_r is replaced with a certain frame in Burst Frames? It is more practical in the real-world setting.

5. In Table 1, the name of the paper [34] sometimes is MRefSR, and sometimes LMR (see Figure 6). The authors should use one name.

6. In Figure 6, there is only one visual result about RefSR. Could you provide more results about RefSR in the main paper?

7. The authors should compare and discuss the efficiency of different methods, including inference time, and model size.
paper_summary: paper_summary: This paper introduces a novel reference-based burst super-resolution (RefBSR) method to combine the strengths of Burst Super-Resolution (BurstSR) and Reference Super-Resolution (RefSR) while overcoming their limitations. RefBSR adaptively fuse the Ref and the burst features to achieve high-quality image reconstruction through the proposed burst feature-aware Ref texture transfer (BRTT) and Ref-burst adaptive feature fusion (RBAF) modules. Additionally, they also provide a new dataset of Ref-burst pairs collected using commercial smartphones for training and evaluation. Experimental results demonstrate that the proposed method outperforms existing RefSR and BurstSR methods.
strength_weakness: strength_weakness: 1. The paper is well-written and straightforward to follow.
2. RefBSR effectively integrates the BurstSR and RefSR, leading to superior high-resolution image reconstruction with enhanced detail and texture fidelity.
3. They collect a new dataset consisting of 2,287 pairs of burst frames and Ref images.
4. Extensive experiments demonstrate superior performance.
limitations: limitations: 1. As the first paper to use both burst and Ref images, the motivation behind this idea is not sufficiently explained.

2. The experiments are all conducted on the proposed dataset. This raises concerns about the generalizability of this method. It remains to be seen whether the method is still effective on other datasets.

3. The methods for comparison are trained on either burst or Ref images alone. Thus, the fairness of the experiments and the effectiveness of the proposed method need further clarification.
paper_summary: paper_summary: The authors combine reference-based and burst super-resolution to utilize burst frames and a high-resolution (HR) external Ref image, which achieves state-of-the-art performance compared to both existing RefSR and BurstSR methods. Further, they provide a new dataset of Ref-burst pairs collected by commercial smartphones for training and testing.
strength_weakness: strength_weakness: Reasonable structure and novel idea. Solid dataset and evaluation. Sufficient reproducibility and practicality.
limitations: limitations: 1. In 3.2, the authors estimate flow between low-resolution base frame I_{b1} and the Ref image I_r. However, I_r contains many irrelevant/noisy information to I_{b1} (they are not adjacent/neighboring frames). How to avoid irrelevant referenced information disturbing feature fusion?
2. Performance is promising. But compared to BurstSR methods, RefBSR has Ref frame, and compared to RefSR methods, RefBSR has several adjacent frames. It would be better if compared to Burst + Ref SR methods.
paper_summary: paper_summary: The paper proposes a novel reference-based burst super-resolution method (RefBSR) that combines the advantages of burst frames and high-resolution reference images to significantly enhance image reconstruction quality, and demonstrates its superior performance through comprehensive experiments. The paper also introduces a dataset based on images captured by smartphones.
strength_weakness: strength_weakness: +The proposed dataset is highly practical, and the method is both practical and effective.

+The experimental results in the paper are robust, and the experiments are thorough.
limitations: limitations: -Why not evaluate on the RealMCVSR dataset?

-Why does RefVSR perform worse than non-ref burst SR methods?

-In reference-based SR tasks, aligning the reference frame with other frames is crucial. The paper should provide more analysis on whether the improvements come from the alignment process itself or from the implicit alignment achieved by the feature fusion module.

-Although I find the method in the paper to be quite engineering-focused, the challenge in reference-based approaches lies in the selection and alignment of the references. The paper does not provide sufficient insight into this point. However, the proposed dual-camera dataset and experimental results may advance the development in this direction.