ReFIR: Grounding Large Restoration Models with Retrieval Augmentation
Authors: Hang Guo
Tao Dai
Zhihao Ouyang
Taolin Zhang
Yaohua Zha
Bin Chen
Shu-Tao Xia
Venue: NeurIPS 2024 poster (2024)
Keywords: Image Restoration
Diffusion Model
Retrieval Augmented Generation
Primary area: machine_vision
TL;DR: This paper propose a training-free framework to alleviate the hallucination of existing diffusion-based restoration models producing unfaithful contents or textures.

Abstract: Recent advances in diffusion-based Large Restoration Models (LRMs) have significantly improved photo-realistic image restoration by leveraging the internal knowledge embedded within model weights. However, existing LRMs often suffer from the hallucination dilemma, i.e., producing incorrect contents or textures when dealing with severe degradations, due to their heavy reliance on limited internal knowledge. In this paper, we propose an orthogonal solution called the Retrieval-augmented Framework for Image Restoration (ReFIR), which incorporates retrieved images as external knowledge to extend the knowledge boundary of existing LRMs in generating details faithful to the original scene.  Specifically, we first introduce the nearest neighbor lookup to retrieve content-relevant high-quality images as reference, after which we propose the cross-image injection to modify existing LRMs to utilize high-quality textures from retrieved images. Thanks to the additional external knowledge, our ReFIR can well handle the hallucination challenge and facilitate faithfully results. Extensive experiments demonstrate that ReFIR can achieve not only high-fidelity but also realistic restoration results. Importantly, our ReFIR requires no training and is adaptable to various LRMs.

ReFIR: Grounding Large Restoration Models with Retrieval Augmentation
Hang Guo!
Tao D
Zhihao Ouyang²
Taolin Zhang'
Yaohua Zha
Bin Chen
Shu-tao Xia
Tsinghua University
Shenzhen University Aitistaai
4 Harbin Institute of Technology *Pngg Cheng Laboratory
https: https://github.com wttp./////cccc/.R......
Abstract
Recent advances in diffusion-basee Large Restoration Models (LRMs) have sig- nificantly improved photo-realistic image restoration by leveraging the internal knowledge embedded within model weights. However, existing LRMs often suf- fer from the hallucination dilemma, i.e., producing incorrect contents or textures when dealing with severe degradations, due to their heavy reliance on limited internal knowledge. In this paper, we propose an orthogonal solution called the Retrieval-augmentte Framework for Image Restoration (ReFIR), which incorpo- rates retrieved images as external knowledge to extend the knowledge boundary existing LRMs in generating details faithful to the original scene. Specifically, we first introduce the nearest neighbor lookup to retrieve content-relevant high-quality images reference, after which we propose the cross- cross-image injection to modify existing LRMs to utilize high-quality textures from retrieved images. Thanks the additional external knowledge, tion challenge and facilitate faithfully results. Extensive experiments demonstrate that ReFIR achieve not only high-fidelity but also realistic restoration results. Importantly, our ReFIR requires no training and is adaptable to various LRMs.
ReFIR well handle the hallueina-
Introduction
Restoring a high-quality image (HQ) from its low-quality counterpart (LQ) a well-known ill posed problem and has been studied over the years [1, 2, 3, 4, 5, 6, 7, 8, 9]. Previous efforts attempt to handle this problem through employing various neural network architectures, including CNNs, GANs and Transformers. Recently, diffusion models [10, 11] have emerged as promising alternative, delivering noteworthy results in real-world image restoration [12, 13, 14]. In particular, some works [15, 16, 17, 18, 19, 20] have successfully leveraged the powerful generative prior of pre-trained text-to-image (T2I) diffusion models for scaling up, to obtain the Large Restoration Model (LRM) with billions of parameters, bringing significant progress in restoring photo-realistic images.
Although scaling up restoration models has achieved remarkable success, existing LRMs may not always produce results that are faithful to the original scene, particularly when faced with heavily degraded images that surpass the LRMs' capabilities (see Fig. 1). This issue is similar to the hallucination problem observed in large language models (LLMs) [21, 22], egg. ChatGPT might generate nonsense responses when highly specialized questions exceed its knowledge boundary. Similarly, if one LRM has never seen specific scene, it will struggle to restore corresponding images faithfully. By analogizing LLM to LRM, we define the phenomenon where LRMs generate textures inconsistent with the original scene when facing hard samples the hallucination of LRMs.
To address the hallucination problem in LRMs, simply expanding the internal knowledge through additional training data and parameters might seem straightforward, but it can significantly increase
CCoresppndingg author: Tao Dai (daitao.edu @ gmmil.comm..
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Reference
Ground Truth
SeeSR
SUPIR
Ours
Figure 1: Existing LRMs encounter hallucination issues. i.e, generating contents or details that
deviate from the original scene, when dealing with challenging degradations. By incorporating the proposed ReFIR to existing LRMs [19] without any training. the additional external knowledge facilitates producing more faithful results. Please zoom in for better visualization.
computational and storage costs. Instead, this work considers another orthogonal strategy that enhances the external knowledge of LRMs without adding parameter counts. Drawing inspiration from the retrieval-augmente retrieved high-quality content-relevant images as external knowledge to alleviate the hallucination of LRMs. However, applying RAG to image restoration poses specific challenges. Specifically, in natural language, simply feeding the retrieved documents along with the original user query to LLMss can allow it to produce grounded responses. However, in the context of image restoration, allowing low-quality images to attend to retrieved images during their restoration process is non-trivial, which motivates us to develop novel techniques to enable LRMs to utilize external knowledge in restoration.
generation (RAG) used in LLMs [23, 24, 25], we aim to use the
To this end, we delve deep into the working mechanisms of LRMs for insightful observations. Details of the experimental setup are described in Sec. 3. Our key findings indicate that the workflow of LRMs can be divided into two distinct stages: the Denoising Structure Reconstruction stage, during which the self-attention in the ControlNet [26] reconstructs a clear overall structure from the noised representation. After that, in the Detail Texture Restoration stage, the self-attention in the UNet [27] decoder fills acene-specific textures based on the denoised structure map. Based on these findings, a natural solution emerged: we can transfer high-quality, scene-specific textures from the retrieved images to the low-quality images during the detail texture restoration stage. In this way, the restored image is allowed a consistent texture with the retrieved image, thus mitigating the hallucination.
Inspired by the above observation, in this work, we propose the for Image Restoration, dubbed ReFIR, to offer a simple but effective way to expand the knowledge boundary of LRMs using the external knowledge from the retrieved images. Specifically, we first construct the retriever which employs the nearest neighbor lookup in the semantic embedding space to retrieve content-relevan reference images in the high-quality image database. After that, we develop the cross image injection which modifies the self-attention layer of original LRMs to enable the queries from the low-quality denoising chain to attend to the keys and values from the denoising chain of retrieved reference. To avoid the domain preference problem during injection, we propose separate attention to perform intra-chain and inter-chain attention, respectively. Given the spatial misalignment between the LQ and the retrieved HQ, we further adopt spatial adaptive gating to mask meaningless pixels during injection. At last, we employ the distribution alignment to narrow the domain gap between LQ and retrieved images. Thanks to the proposed ReFIR, the restoration of the LQ image can make full use of the external knowledge from the reference to generate high-fidelity images. Notably, the proposed pipeline is training-frre free and can be applied to multiple LRMs.
Framework
The contribution of this paper can be summarized as: (i) We introduce retrieval- -ugmented restoration, a novel concept to mitigate the hallucination problems in existing LRMs. (ii) We conduct an in- depth analysis of the working mechanisms of LRMs, based on which we propose training-free framework to utilize the retrieved images. (iii) Extensive experiments validate that our proposed method effectively mitigates hallucination and is applicable to a broad spectrum of existing LRMs.

---

Fourer Apalyls on Features from Controlitet
Fourler Analysa Factures from Decoder
Cardi afh
Bocc Slock
Block fleck
Dooodar
Block_1
Block_
Block_3 Block_4
Figure 2: In-depth visualization about the working mechanism of LRM. Left: we use PCA to visualize the top three principal components latent extracted from the self-attention layer of the ControlNet and UNet decoder. Right: quantitative power spectrum of the corresponding latent using Fourier analysis. More visualization can be found Appendix
Related Works
2.1 Diffusion Model for Image Restoration
Diffusion models have recently achieved significant advancements across various computer vision tasks [28, 29, 30, 31, 32, 33, 34, 35, 36, 37]. In the realm of image restoration, early explorations often involved training diffusion models from scratch to obtain the restoration tailored models [38, 39, 40, 41]. While these models are capable of producing high-fidelity results, they usually fall short of generating perceptually pleasing images. To leverage the powerful generative capabilities of large pre-traines rre-trained text-to-image diffusion models like Stable Diffusion
[38], recent attempts [15, 16, 17, 42, 18, 19] have focused on using the ControlNet [43] with LQ image as the condition to generate HQ images. Benefiting from the scaling law [44], these large restoration models with billions of parameters have shown impressive restoration results with photo-realistic textures and details. However, similar to the large language models, when the user query, i.e., the I.Q image in this setting, exceeds the knowledge boundary of the large models, the models often fail to generate meaningful or correct responses, which is unacceptable for image restoration tasks that pursue high-fidelity.
Reference-based Image Super-resolution
Compared with single image super-resolution [1, 4, 2], Reference- Image Super-Resolun (RefSR) can achieve enhanced performance by employing content-similar reference images as the additional input, and has attracted great research interests in the past few years [45, 46, 47J. For instance, C2-Matching [47] introduces a teacher-student correlation distillation and a dynamic DCN aggregation module for more precise alignment between low-quality and reference images. Following this. DATSR [48] employs reciprocal learning and SwinTransformer to further boost performance. Additionally, MRefSR [49] introduces simple baseline to facilitate RefSR with multiple reference images. It is worth mentioning that despite both using additional images as references, our proposed retrieval augmented restoration pipeline differs from previous RefSR methods in several key aspects. Firstly, current RefSR models are typically small-scale due to limited training data, leading to performance degradation under challenging real-world conditions. Secondly, most RefSR methods can only use one single reference image and even fail to work in the absence of reference images. Thirdly, different from ReISR models that require training, our method can inject image-specific external knowledge into LRMs in a training-free manner. We give a detailed discussion about the difference in Appendix B.
2.3 Retrieval Augmented Generation
In the domain of natural language processing, Retrieval-Augmented Generation (RAG) leverages the strengths pre-trained Large Language Models (LLMs) combined with knowledge retrieved from an external document database to enhance the quality of generated content [21, 22]. Typically, RAG system initially retrieves documents relevant to the user's query from the knowledge base and then integrates the retrieved document along with the original user query into the LLMs without any tuning to generate response. Even when no relevant document available, this system can still operate by using the internal knowledge embedded in the L.MSs parameters. The integration of RAG allows I.IMMs to produce outputs that are not only contextually rich but also factually accurate, effectively mitigating the hallucination problem in knowledge-intensin tasks [23, 24, 25]. In this

---

Reference Image Rctrieval
High-fidelity Image Restoration
Feature Extractor
Srodeiyy
Low qualtt
GEICI
Bebrinedd Imagee
High-quality Image D
Figure 3: Our ReFIR consists of stages: the Reference Image Retrieval stage employs the retriever R. to search content-relevant images from high-quality image database DD and then the High-fidelity Image Restoration stage restores HQ image with reference images IR condition. The proposed framework is highly generic and can be applied to multiple existing LRMs without any training fine-tuning.
work, we extend the concept of RAG to image processing and propose retrieval-augmente -augmented restoration alleviate the hallucination issues LRMs. By utilizing external textures embedded in the retrieved reference images, our tuning-free free framework significantly facilitates faithful restoration results.
3 Probing Large Restoration Models
In order manipulate the LRM that it can utilize the retrieved reference images external knowledge, we first delve into the underlying mechanism existing LRMs to find useful insights. We choose the current popular LRM method SUPIR [19] a representative. Inspired by previous image editing efforts [50, 31, 30, 29), which show that the self-attention layer of diffusion models contains important spatial correlation of an image,
thus follow this clue and employ the PCA to visualize the principal components of the latent from self-attention layers of SUPIR. We further utilize the Fourier analysis [51] to allow for quantitative results. The results shown in Fig. 2.
can be seen that the ControlNet of the LRM can denoise the latent as the layers deepen, facilitating the reconstruction of clear overall structure. However, this process is accompanied by reduction in the feehfrreuencyy meaningful texture of the original image. This qualitative visualization can be also verified by the frequency characteristic plots. with high-frequency components decaying as layer number increases. On the other hand, the role of the UNet decoder is significantly different. Based on the previous clear structural map, the decoder restores the figh-frequency details and textures with the help of skip connections, which is also shown through the strengthening high-frequency component in the decoder's frequency curve.
Considering the above observations, we can divide the image restoration process the LRM into two phases: the Denosing Structure Reconstruction phase in the ControlNet, and the Detail Texture Restoration phase in the UNet decoder. Inspired by these probing experiments, in this work, we employ the detail texture restoration nature in the self-attention layer of the decoder to inject the high-fidelity textures retrieved images into the restoration process the low-quality image.
4 Methodology
This work considers using retrieved reference images as explicit part of the model. In contrast to the existing restoration pipeline, our ReFIR is parameterized by not only the internal knowledge from the network weights but also the external knowledge retrieved from suitable data representations. Fig. 3 gives an overview our ReFIR. the following part, we will first give the technical details of the retriever for reference image retrieval in Sec. 4.1, followed by the cross image injection to inject external data knowledge into the restoration process of LRMs Sec. 4.2.
4.1 Nearest Neighbor Lookup for Reference Image Retrieval
Our reference image retrieval system can be represented as a binary set (D, R.,, where is fixed database containing large number of HQ images, and R. denotes a non-parametric retriever to obtain the retrieved image set IR which consists of elements and subset of D given a query LQ
image l7q xx i.e. R I1Q, D + IR. where IR C D and IR| k. Idcally, R has to be

---

(a) Separate Attention
Source Reference Chain Cs
Houd ontrrtiionn
BeTrENE Fatrrnnc trage
noice
x (discard)
Injectoo
(b) Spatial Adaptive Gating
X High quality Image
notse
0
LoN Qla Iraee
Target Restoration Chain Cr
Figure 4: An illustration of eross image injection. Both Cr and Cs share the same model weights.
designed such that it provides the model with beneficial data representations from D to help restore images containing details faithful the original scenes.
In this work, we implement a conceptually simple solution of R. which uses the query image Lg to retrieve its nearest neighbor in using cosine similarity in the compact feature space derived from any feature extractors, such as VGG [52], ResNet [53] or CLIP [54]. Since the D is fixed, in practice, we can pre-extract and store the compact feature before training. Given sufficiently large database D. this strategy ensures that the set of neighbors IR shares sufficient semantic consistency with LQQ and thus provides useful visual information for the restoration. Although this scheme seems simple, we show that it is efficient and effective, please see Sec. 5.3 for discussion.
4.2 Cross Image Injection for High- Image Restoration
Given the retrieved reference images I R(ILQ D), we further propose the cross image injection to allow the original LRMs to use the external knowledge from I As shown in Fig. 4, we first construct two parallel denoising chains: the target restoration chain Cr which used to restore IzQ, and the source reference chain Cs which unfolds IR into denoising time steps. After that, we introduce separate attention to separately perform attention within and between chains, followed by spatial adaptive gating to filter out irrelevant pixels. At last, we use the distribution alignment to mitigate the domain gap between chains. More details are given below.
Separate attention. To allow the Cr to learn the knowledge from the Cs. an effective interaction between the latents crucial. Inspired by the observation in Sec. 3, aim transfer the knowledge embedded the self-attention layer of Cs 's decoder to the counterpart of Cr- To this end, we modify the original self-attention in Cr to proposed separate attention. The core idea of separate attention is to add "inter-chain cross-attention" to the original "intra-chain self-attention" so that Cr can attend high-quality texture knowledge from Cs while preserving its original features. As shown in Fig. 4(a), formally, denote Qr, Kr, VT as the query, key and value from the Cr, and Ks, Vs as the key and value from the Cs. the intra-chain self-attention preserves the original attention of Cr to obtain the output O Ontrra and the inter-chain cross-attention uses the Qr to query the Kş and Vss to facilitate Cr utilizing the knowledge from Cs to get the result Onterr. In short, the proposed separate attention can be formalized as follows:
Ointra Attention (QriiionnnQTnnn KT, Vr). Oenter Attention (Qr, Ks. Vs).
(I)
It is worth mentioning that directly using Qr to query the concatenate results of Kr and Kş can only yield sub-optimal results due to the domain preference issue, i.e., Qr will prefer latent from the same domain Cy even though Cs is more helpful for reconstruction. By using the proposed separate attention, the Qr is separated to attend Ky and Ks, thus effectively mitigating this problem. We give more discussion in Sec. 5.3.
Spatial adaptive gating. We then consider fusing the separate attention results Ointra and Ointer- The main challenge is the spatial misalignment between IL and IR. For instance, the same objects
5

---

may appear in different locations in ILQ and IR. or some objects in L7 may not present in IR and vice versa. As a result, some pixels in QT may not find the corresponding reference in Ks, resulting in some pixels in Ointer meaningless.
To address this spatial misalignment, we propose the spatial adaptive gating selectively fuse Omtrre and O Onter without introducing additional parameters (Fig. 4(b)). Specifically, given latents at specific denoising blocks from CT and Cs. respectively, we first flatten them along the spatial dimension to obtain hT, hy E CCHW Next, we compute their pixel-wise cosine similarity to obtain the similarity matrix sim E TR- xHW Since the i-th row of sim represents the similarity of the i-th pixel in hr to all the pixels in hs. therefore, large sum of the i-th row indicates a large impact of hy in restoring the i-th pixel of hT. Following this idea, we summation over the i-th row of the sim to approximate the utiliity of hş to the i-th pixel of hr. Finally, we reshape this summation results back to 2DD shape and use min-max normalization to restrict the range to [0, 1]. to get the pixel-wise mask M for adaptive gated fusion:
Osuse= = (1- - sM) @ O + s.M @ Omterr
(2)
where is user-defined scalar to control the degree which the restored image attends the retrieved images, @ denotes the Hardamard product, and 1 is an all one tensor with the same shape as M.
Distribution alignment. Using the Ofuse to replace the original intra-chain self-attention results Ountra scems to be a promising way to integrate useful external knowledge from Cs. However, it should be noticed that there is domain gap between Cr and Cg due to the image quality and content differences, and thus a direct insertion of Ointra into Cr will result in a distribution shift of the original denoising chain in Cr-
To this end, we propose the distribution alignment as a complementary to calibrate the distribution shift. Specifically, considering the latent in the diffusion chain is Gaussian, we propose to use the Adaptive Instance Normalization (AdaIN) [55] to align the mean and variance of Juse to the original statistics of Osntro:
O'uuse AdaIN(O Juse, Ointra);
(3)
where AdaIN(u, (t, v) denotes replacing the mean and variance of with the corresponding part of v. Finally, we replace the original self-attention result in Cr with the well-aligned O'usce to finish the cross image injection process.
Experiments
5.1 Experiments Setup
Datasets and metrics. In this work, we include experiments with two difficulty levels for performance evaluation. The first setup considers restoration with manually provided ideal reference images, which share high content similarity with the LQ image, to evaluate the ability to utilize the reference knowledge. The datasets for this setting employ the widely used RefSR dataset including CUFEDS [56, 57] and WR-SR [47], in which the reference images are already provided. Since these datasets only contain HQ images, we thus use the sccond-order degradation model from Real-ESRGAN [8] with x4 down nown-sampling scale to generate the real-world degraded images. The second setup turns to more challenging practice where the reference images have to be retrieved using the retriever, and we use the RealPhoto60 [19] which contains 60 real-world degraded images without ground truth for evaluation. And we use DIV2K [58] as the high-quality image database for retrieval and employ the image encoder of VGG16 [52] as the feature extractor. As for the evaluation metrics, we use both the fidelity metrics containing PSNR and SSIM, as well as the perceptual metries including LPIPS [59], NIQE [60], FID [61], MUISQ [62], and CLIPIQA [63], to assess the performance of the different methods.
Implementation details. For a fair comparison, we use one reference image if not specified. Experiments with multiple reference images are given in Appendix A. Following the common practice of existing LRMs [19, 17, 18, 16], the IQ is up-sampled to the desired size using Bicubic before going through the LRMs. We use reflective padding to ensure the input size of Cr and Cs are the same. We use fixed random seeds for results reproducibility in all experiments. The hyperparameters of different baselines follow their original settings. We apply the proposed retrieval augmented restoration framework to two popular LRMs, namely SeeSR [18] and SUPIR [19], and denoted the models augmented with our ReFIR as "SeeSR+ReFiR"" and "SUPIR+ReFIR" respectively.

---

Table 1: Quantitative comparison with state ofate-of-the-are RefSR methods, GAN-based methods, and Diffusion-basse methods on real-world image super-resolution. Our ReFIR achieves consistent performance improvements in both fidelity and perceptual quality.
CUFEDS
WR-SR LPIPS.
Method
PSNRT
SSIM
LPIPS|
NIQEL
FID
PSNR
SSIMT
NIQE!
FID!
C2-Matching [47]
20.77
0.5169
0.7282
8.4438
282.43
22.63
0.5627
0.7177
8.3238
157.61
DATSR [48]
20.75
0.5130
0.7301
8.6765
282.19
22.62
0.5620
0.7210
8.4329
157.54
MrefSR [49]
20.84
0.5218
0.7853
9.6524
286.44
22.68
0.5703
0.7748
9.7742
156.57
BSRGAN [9]
20.22
0.5256
0.4135
4.2204
203.17
22.07
0.5735
0.4073
3.8703
133.50
Real-ESRGAN [8]
20.31
0.5543
0.3698
3,8832
175.91
22.14
0.5974
0.3631
3.7001
97.88
StableSR [16]
20.46
0.4480
0.6532
6.3433
292.69
21.22
0.4421
0.5899
5,2040
145.07
DiABIR [15]
19.76
0.4886
0.3820
3.5629
154.75
21.30
0.5284
0.3938
3.8736
76.05
PASD [17]
20.22
0.4959
0.5252
5.4828
208.64
21.12
0.5254
0.4292
4.2505
98.16
SeeSR [18]
19.94
0.5195
0.3660
3.7912
142.92
21.73
0.5658
0.3501
4.0155
65.78
ReeIR
20.32
0.5289
0.3338
3.7831
134.62
21.86
0.5664
0.3460
3 3,9089
61.22
Aimprovement
+0.38
+0.0094
+0.0322
+0.0081
+8.30
+0.13
+0.0006
+0.0041
+0.1066
+4.56
SUPIR
18.97
0,4665
0,4807
4,5624
168.20
20,91
0,5426
0,3791
3.7587
75.85
ReFIR+ReFIR
19.00
0,4729
0,4341
4,2085
148.69
21.02
0,5497
0,3785
3.7478
71.82
Aimprovement
+0.03
+0.0064
+0.0466
+0.3539
+19.57
+0.11
+0.0071
+0.0006
+0.0109
+4.03
0n:
FA.50
0058
2A50
Ureedd losb digh- -pb-zaiit Image
SUPI
Secil A4
Gooaad TrESs HASh QUATT IEage
SWISR
Simik
CESA
Uoaa High
Irage
SUP3
SL GUETI
Figure 5: Quantitative comparison on RefSR dataset. The results using our ReFIR are bolded. Please zoom for better visualization.
5.2 Comparison to State-of-the-Arts
Restoration with ideal reference. We first compare on the RefSR dataset with real-world degradation. The compared methods includes state-of-the-at RefSR methods [47, 48, 49], GAN-based methods [9, 81, and recent Diffusion- based methods [16, 19, 18, 17, 15]. Tab. 1 gives the results. It can be seen that our method brings significant gains in all metrics on both fidelity (PSNR, SSIM) and perceptual quality (LPIPS. NIQE, FID) for the LRMs. Taking SUPIR as an example, our method brings FID improvement of even 19.57 on the CUFED5 dataset. Moreover, similar performance gains can also be observed in SeeSR. For instance, equipping our ReFIR to SeeSR can lead to 0.38dB PSNR improvement, demonstrating the generalization of our ReFIR. is noteworthy that the above superiority is obtained without any training or fine-tuning. Moreover, we also give visual comparisons in Fig. 5, and it can be seen that our method can generate details that are faithful to the original scene with the help of external knowledge from retrieved reference images.
Restoration in the wild. The above experiments on RefSR datasets focus on utilizing the already provided reference images from the dataset, which applies when the user has relevant HQ images. In this section, we turn more challenging scenarios in which the reference image has be obtained by retrieval. Since the ground truth of RealPhoto datasets is unavailable, we use non-reference image quality assessment metrics, i.e. NIQE, MUSIQ, and CLIPIQA for evaluation. As shown in Tab. 2, our approach continues to produce significant gains over its Ren-ReFIR counterparts. For instance, our SeeSR+ RRFIR surpasses the original SeeSR by 0.2866 NIQE and 1.59 MUSIQ. Since the retrieved image can not serve as an ideal reference, the above favorable results demonstrate the robustness of our ReFIR in the face of real-world retrieved images. We also give quantitative results in Fig. 6. Even under severe real-world degradation, our method maintains good perceptual quality.
7

---

Table Quantitative comparison on real-world degradation with RealPhoto datasets.
StableSR [16]
DifiBIR
PASD
CCSR
SceSR
SUPIR
SeeSR+ReFIR SUPIR+ReFIR
Metrics
[15]
[17]
[42]
[18]
[19]
(Ours)
(Ours)
NIQEL
3.7695
2.8458
5.1603
5.5082
4.7432
3,5076 4,4566(+0.2866)
3,4593
MUSIQU
51.95
65.20
49.01
32.26
55.54
59.84
60.49(+0.65) 0.6732(+0.0157)
57.13(+1.59)
CLIPIQAT
0.6852
0.7845
0.5863
0.4568
0.6575
0.5692
0.5722(+0.003)
SERRASH
CUSH
StablesR
DitiBiR
PASO
CUSH
Low quality Image
SuSSR
SUPIR IR SL IR
Low-qualtyy Image
See5R
SUPIR
SWSR SARRRRSSRRRRRRRRRRR
Figure 6: Quantitative comparison on RealPhoto dataset. More results provided in Appendix H.
Table 3: Comparison of model complexity before and after incorporating our ReFIR. We use input image with the resolution of 2048 x 2048 to evaluate the GPU memory and the inference time on one single 80G NVIDIA A100 GPU.
Method
PSNR1
SSIM LPIPS..
FID
#param. GPU Memory, Inference Time.
SeeSR [18]
19.94
0.5195
0.3660
142.92
2.04B
24.4G
76.5s
SUPIR [19]
18.97
0.4665
0.4807
168.26
3.87B
37.3G
146.4s
ReeSR+ReFIR
20.32
0.5289
0.3338
134.62
2.04B
40.9G
170.7s
SUPIR+RefIR
19.00
0.4729
0.4341
148.69
3.87B
51.4G
322.8s
Complexity analysis. Tab. 3 gives the comparison of the computational complexity, including the number of parameters, GPU cost, and the inference latency. We also give the restoration performance for more comprehensive comparison. As for the parameters, our ReFIR can facilitate both fidelity and realistic image restoration using the same #param as the original base LRMs. For the GPU memory, since our ReFIR two images input, i.e., LQ image, and reference image, the GPU cost will become larger than the original one. For instance, it rises 1.38 times the increase of RUPIR+ReFIR than the original SUPIR model. Moreover, the inference time also increases due to more inputs as well as the additional interaction between two chains. In the future, we will delve deep into the effective utilization of retrieved images while maintaining efficiency.
5.3 Ablation Studies
Effectiveness of the reference retriever. In order to obtain content-relevant retrieved images, present a simple but inference-efficien retriever R that uses the high-level semantic vectors from the pre-trained deep models for similarity matching in the high-quality image dataset D. Despite the simple design, we here demonstrate its effectiveness in Fig. 7. Since semantically consistent images usually contain similar textures, egg., the texture in the first elephant image can help in the restoration of the LQ elephant image, and thus the proposed retriever can yield satisfactory retrieval results. Although texture-based retrieval may be better choice for image restoration, usually necessitates additional training of new retrieval models. For simplicity, we adopt semantic-based retrieval and leave the exploration of more advanced reference retrievers for future work.
Ablation on cross image injection. In the proposed cross image injection, we use separate at- tention (SA), spatial adaptive gating (SG), and distribution alignment (DA) for effective external knowledge injection. Here, we ablate to validate the effectiveness of different components. We SUPIR+ReFIR
the CUFEDS dataset and use the scalar weighted when SG is removed. The results are shown in Tab. 4. One can see that using fixed scalar weights instead of spatial adaptive gating results in 0.18 NIQE drop. This is because not all pixels of the reference image are useful, and thus fine-grain gated mask is needed. Moreover, removing the distribution alignment also impairs performance, e.g., 4.36 FID drop, since the distribution of raw fusion results O fuee does not match Cr, and directly inject O yuse to the denoising chain of Cr can cause sub-optimal results.
representative
8

---

Netrooed
Topl Rabloved
Tops Rabiowe
New
PealPhoto 010
DIVKK 0117
PholPhoto 055 Guere Image
DIVIK 0004
Ptooo 017
DIV2 0158
THUT REUroved
Toot Wetroveel leraae
Top! Bastived
AAA
ResiPtooooco 051
DIV2 0629
RealPhoto 060
DIV2K (201
Resi 012
DIV2K 0070
Figure 7: The retrieval results with RealPhoto60 dataset [19] as the query images and DIV2K [58] as the HQ image database.
Table 4: Effectiveness of different components in cross image injection.
Table 5: Ablation experiments on different positions of cross-image injection.
SA SG DA DAJPSNRT SSIM NIQEL
FIDL
Injection Position
|PSNR| SSIM: NIQEJ FID
18.97
0.4665 4.5624
168.2 26
No Injection
18.97 0.4665 4.5624 168.26
19.09
0,4799
4.3893 150.81
Encoder& Decoder
19.09
0.4773 4.5241 153.43
19.12
0.4724
4.2275
153.05
Facoder Only
19.08
0.4689
4.6977 168.26
19.00
0.4729 4.2085 148.69
Decoder Only
19.00
0.4729 4,2085
148.69
Domain preference problem. The motivation behind the proposed separate attention is to address the domain preference problem, i.e., the attention in Cy will prefer to use latent from the same chain even though the latent from Cs is more help- ful for reconstruction. To verify the existence of the domain preference, we use the ground truth H as the input of Cs and compute the normalized attention scores between QT and KT, Qr and Kg. It can be seen in Fig. that even using the spatially strictly aligned TH the reference, Qr still has significantly high attention for the latent from the same chain, indicating that the domain preference problem interferes with the CT's utilization of external knowledge in Cs. By contrast, the proposed separate attention can effectively mitigate this problem by forcing the Qr to separately attend Kr and Kg.
vice Scou
Figure 8: The normalized attention scores obtained by averaging all samples and all time steps.
Other choices on injection position. In Sec. 3, we find the diffusion decoder responsible for restoring textures. Based on this observation we propose to apply cross-image injection the UNet decoder. Here, we ablate to analyze the impact of different cross-image injection positions. The results shown in Tab. 5. It can be seen that performing cross-image injection only on the encoder will cause 19.57 FID drops. This is because the encoder focuses on the structure reconstruction, thus transferring the structure of Cs will destroy the layout of the Cr. Moreover, performing injection only in the decoder achieves the best results since it can transfer the high-quality textures from the Cs. Due the page limit, more ablation experiments can be seen Appendix C.
5.4 Discussions
What is the impact of the control scale? The scale in Eq. (2) can control the extent to which the LRMs use external knowledge from the retrieved reference image for restoration. Here, we conduct an ablation study explore the effect of s. The results shown in Fig. 9. It can be seen that when s takes smaller values, the model mainly uses the internal knowledge embedded its own parameters, which can make the model hallucinate when the degradation is severe. For example, the model produces incorrect textures when s = O. As s increases, the model starts to use external knowledge from the retrieved reference image, from which the model's hallucination problem be alleviated. We also provide quantitative ablation experiments on in Appendix C.
How much do the reference images affect performance? In the proposed framework, the retrieved images IR crucial in alleviating hallucinations. Here, we try to answer the role of In during restoration process, by manually controlling different types of retrieved images. As shown in Tab. 6,

---

HQ Image
NoReference (s=0)
5-0.5
0
5-1.0
Figure 9: Ablation visualization the control scale s. As increases, the LRM utilizes the external
knowledge from retrieved reference images to mitigate hallucination. Zoom in for better effects.
Table 6: The performance impact of reference images. NoRef means no reference image used. HQRef denotes the corresponding IQ is used as the reference. SelfRef represents using x4 bicubic upsampling of LQ for ref- erence. Random means randomly selecting a high-quality image as the reference.
Figure 10: An explanation of how the pro- posed retrieval augmented framework af- fects the restoration process of existing LRMs.
Latent Manifool
Settings
PSNRT SSIM LPIPS| NIQR]
FTD.
Finally Modified X-1
NoRef
18.97
0,4665 0.4807 4.5624
168.26
from LRM
HQRef
19,41
0.5033 0.3928 4.0764
137.52
SelfRef
19.16
0,4795 0.4761 4.5501
163,94
Random
19.53
0,5138 0.5354
5,3796
223.47
from
Baseline
19.00 0.4729 0.4341 4.2085
148.69
Current latent X:
broweedppd
find that using the exact ground truth InQ the IR further improve the performance, which can be seen as an ideal up-bound. Interestingly, using ILQ itself as its own retrieved image instead brings slight improvement compared with no retrieval, which we attribute to the regularization effect from the distribution alignment strategy. Finally, randomly selecting a high-quality reference image resulted in huge performance degradation, suggesting that the content correlation is more important than the image quality for a favorable retrieved reference image.
How does the proposed ReFIR work? Extensive experiments have shown the state-of-the-art performance of our ReFIR. However, it seems not straightforward to understand how the retrieved reference images influence the image restoration process of the original LRMs. Here, we give an intuitive explanation. As shown Fig. 10, for the latent at the t-th time step on the latent manifold, there are two forces in different directions pulling produce the latent at the next t 1-th time step. One force from the internal knowledge of frozen weights in LRMs, and the other is the external knowledge from the retrieved reference image through the proposed cross image injection mechanism. These two forces ultimately determine the latent of the next time step. Therefore, a restored image from our ReFIR utilize both the internal knowledge in the original LRMss well as the external knowledge in the retrieved image, thus alleviating the hallucination the LRMs.
Conclusion
This paper presents ReFIR, a training-free and generic framework that can alleviate the halluci- nation of LRMs to facilitate high-fidelity and photo-realistic restoration results through retrieval augmentation. We introduce the nearest neighbor lookup as a simple retriever obtain relevant high-quality images and further propose the cross-image injection which employs separate attention to transfer knowledge while avoiding the domain preference problem, the spatial adaptive gating to address the spatial misalignment, and the distribution alignment to mitigate the domain gap during injection. Through expanding the knowledge boundary using the additional external knowledge from retrieved images, ReFIR exhibits significant improvements on both fidelity and perceptual quality, as demonstrated through extensive qualitative and quantitative evaluations. Moreover, with its training-foe free and generic nature, our ReFIR can be easily applied to multiple LRMs.
10

---

Acknowledgements
This work is supported in part by the National Natural Science Foundation of China, under Grant (62302309,62171222)) Shenzhen Science and Technology Program (ICY100220818101014083 JCYJ202208 01000020000101012222252520 and the PCNL KEY project (PCI2002ASS-1),
References
[1] Chao Dong. Chen Change Loy. Kaiming He, and Xiaoou Tang. Learning deep convolutional network for image super-resolution. In ECCV, pages 184 199. Springer, 2014. 1,
[2] Tao Dai, Jianrui Cai, Yongbing Zhang, Shu-Tao Xia, and Lei Zhang. Second-order attention network for single image super-resolution. In CVPR, pages 11065-11074, 2019. 1,3
[3] Hang Guo, Jinmin Li. Tao Dai, Zhihao Ouyang. Xudong Ren, and Shu- Tao Xia. MambalR: A simple baseline for image restoration with space model. arXiv preprint arXiv:2402. 2024. 1,
Kai Zhang, Wangmeng Zuo, Shuhang Gu, and Lei Zhang. Learning deep enn denoiser prior for image restoration. In CVPR, pages 3929-3938, 2017. 1,3
[5] Hanting Chen. Yunhe Wang. Tianyu Guo, Chang Xu, Yiping Deng. Zhenhua Liu, Siwei Ma, Chunjing Xu, Chao Xu, and Wen Gao. Pre-trained image processing transformer. In CVPR, pages 12299-12310, 2021.
[6] Wenbo Li, Xin Lu, Shengju Qian, Jiangbo Lu, Xiangyu Zhang, and Jiaya Jia. On efficient transformer-based image pre-training for low-level vision. arXiv preprint 12.:::2112.1117: 2021.
[7] Jingyun Liang. Jiezhang Cao, Guolei Sun. Kai Zhang. Luc Van Gool, and Radu Timofte. SwinIR: Image restoration using swin transformer. In ICCV, pages 1833-1844, 2021.
[8] Xintao Wang, Liangbin Xie, Chao Dong, and Ying Shan. Real-ESRGAN: Training real-world blind super-resolution with pure synthetic data. In ICCV, pages 1905-1914, 2021. 6,
[9] Kai Zhang, Jingyun Liang, Lue Van Gool, and Radu Timofte. Designing practical degradation model for deep blind image super-resolution. In ICCV, pages 4791-4800, 2021. 7,
[10] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. NeurlPS, 33:6840- 6851, 2020.
[11] Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502, 2020.
[12] Ben Fei, Zhaoyang Lyu, Liang Pan, Junzhe Zhang, Weidong Yang. Tianyue Luo, Bo Zhang, and Bo Dai. Generative diffusion prior for unified image restoration and enhancement. CVPR, pages 9935 9946, 2023.
[13] Bahjat Kawar, Michael Elad, Stefano Ermon, and Jiaming Song. Denoising diffusion restoration models. NewrIPS, 35:23593-23606, 2022.
[14] Yinhuai Wang. Jiwen Yu, and Jian Zhang. Zero-shot image restoration using denoising diffusion null-space model. arXiv preprint 2xXiv:2212.00400 2022. 1
[15] Xinqi Lin, Jingwen He, Ziyan Chen, Zhaoyang Lyu, Ben Fei, Bo Dai, Wanli Ouyang, Yu Qiao, and Chao Dong. DifIBIR: Towards blind image restoration with generative diffusion prior. arXiv preprint arXiv:2308. 15070, 2023. 7,8
[16] Jianyi Wang, Zongsheng Yue, Shangchen Zhou, Kelvin CK Chan, and Chen Change Loy. Exploiting diffusion prior for real-world image super-resolution. arXiv preprint arXiv:2305.07015 2023. 1, 6, 7,
[17] Tao Yang, Peiran Ren, Xuansong Xie, and Lei Zhang. Pixel-aware stable diffusion for realistic image super-resolution and personalized stylization. arXiv preprint arXiv:2308. 14469, 2023. 1, 3, 3,6,7 7,
[18] Rongyuan Wu, Tao Yang, Lingchen Sun, Zhengqiang Zhang, Shuai Li, and Lci Zhang. SeeSR: Towards comantics-aware real-world image super-resolution. arXiv preprint arXiv:2311. 2023. 1, 6, 8,8, 15
[19] Fanghua Yu, Jinjin Gu, Zheyuan Li, Jinfan Hu, Xiangtao Kong, Xintao Wang. Jingwen He, Yu Qiao, and Chao Dong Scaling Up to Excellence: Practicing model scaling for choto-realistic image restoration in the wild. arXiv preprint arXiv:2401.13621 2024. 1, 2. 3, 4, 6, 7, 8, 15. 21

---

[20] Jianze Li, Jiezhang Cao, Zichen Zou, Xiongfei Su, Xin Yuan, Yulun Zhang. Yong Guo, and Xiaokang Yang. Distillation-free one-step diffusion for real-world image super-resolution. arXiv preprini arXiv:2410.04224. 2024.
[21] Sewon Min, Kalpesh Krishna, Xinxi I Lyu, Mike Lewis, Wen-tau Yih, Pang Wei Koh, Mohit lyyer, Luke Zettlemoyer, and Hamnanch Hajishirzi. FactScore: Fine-grained atomic evaluation of factual precision in long form text generation. urXiv preprint arXiv:2305. 14251, 2023. 1.3
[22] Alex Mallen, Akari Asai, Victor Zhong, Rujarshi Das, Daniel Khashabi, and Hannaneh Hajishirzi. When not trust language models: Investigating effectiveness of parametric and non-parametric memories. arXiv preprint arXiv:2212.10511. 2022. 1.3
[23] Ori Ram, Yoav Levine, Itay Dalmedigos, Dor Muhlgay, Amnon Shashua, Kevin Meyton-Bronn and Yoav Shoham. In-context retrieval-augmentedd language models. Transactions of the Association for Computational Linguistics, 11:1116-1311, 2023. 2,3
[24] Akari Asai, Zeqiu Wu, Yizhong Wang, Avirup Sil, and Hannanch Hajishirzi. Self-RAG: Leamingg to retrieve, generate, and critique through self-reflection. arXiv preprint arX1v:2310./1511, 2023. 2,
[25] Akari Asai, Sewon Min. Zexuan Zhong, and Danqi Chen. Retrieval-based language models and applications.
In ACL, pages 41-46, 2023. 2,3
[26] Lymin Zhang, Anyi Rao, and Mancesh Agrawala. Adding conditional control to text-to-image diffusion models. In ICCV, pages 3836-3847, 2023. 2
[27] Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-Ner: Convolutional networks for biomedical image segmentation. In MICCAI, pages 234-241. Springer, 2015. 2
[28] William Peebles and Saining Xie. Scalable diffusion models with transformers. ICCV, pages 4195-4205,
2023. 3
[29] Mingdeng Cao, Xintao Wang, Zhongang Qi, Ying Shan, Xiaohu Qie, and Yinqiang Zheng. MasaCtrl: Tuning-free mutual self-attention control for consistent image synthesis and editing. In ICCV, pages 22560-22570, 2023. 3, 4, 16
[30] Yuechen Zhang, Jinho Xing, Eric Lo, and Jiaya Jia. Real-world image variation hy aligning diffusion inversion chain. NeurlPS, 36, 2024. 3, 16
[31] Jing Gu, Yilin Wang, Nanxuan Zhao, Tsu-Jui Fu, Wei Xiong, Qing Liu, Zhifei Zhang, He Zhang. Jianming Zhang, HyunJoon Jung, al. PhotoSwap: Personalized subject swapping in images. NeurlPS, 36, 2024. 3, 4, 16
[32] Andreas Blattannn, Robin Rombach, Kaan Oktay, Jonas Müller, and Bjöm Ommer. Retrieval-augmennte diffusion models. NeurlPS, 35:15309-15322 2022. 3
[33] Yue Ma. Yingging He, Xiaodong Cun, Xintao Wang, Siran Chen. Xiu Li, and Qifeng Chen. Follow your Pose: Pose-guided text-to-vidco generation using pose-free videos. In AAAI, volume 38, pages 4117-4125, 2024. 3
[34] Yue Ma, Yingging He. Hongfa Wang. Andong Wang, Chenyang Qi, Chengfei Cai, Xiu Li, Zhifeng Li, Heung- Heung-Yeung Shum, Wei Liu, et al. prompts. arXiv preprint arXiv:2403.0828 2024.
Open-domain regional image animation via short
[35] Yue Ma. Hongyu Liu, Hongfa Wang. Heng Pan, Yingging He, Junkun Yuan, Ailing Zeng, Chengfei Cai, Heung- Yeung Shum, Wei Liu, et al. Follow- Your Emoji: Fine-controllable and expressive freestyle portrait animation. arXiv preprint arXiv:2406.00 01900, 2024.
[36] Yue Ma, Xiaodong Cun, Yingqing He, Chenyang Qi, Xintao Wang, Ying Shan, Xiu Li, and Qifeng Chen. MagicStick: Controllable video editing via control handle transformations. arXiv preprint arXiv:2312.0 030447, 2023.
[37] Jiangshan Wang, Yue Ma, Jiayi Guo, Yicheng Xiao, Gao Huang, and Xiu Li. COVE: Unlcashing the diffusion feature correspondence for consistent video editing. arXiv preprint arXiv:2406.08835 2024. 3
[38] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High-resolution image synthesis with latent diffusion models. In CVPR, pages 10684-10695, 2022. 3
[39] Shuyao Shang. Zhengyang Shan, Guangxing Liu, LunQian Wang, XingHua Wang, Zekai Zhang, and Jinglin Zhang. ResDiff: Combining enn and diffusion model for image superoresolution. In Proceedings of the AAAI Conference Artificial Intelligence, volume 38, pages 8975 7983, 2024. 3
12

---

[40] Zongsheng Yue, Jianyi Wang, and Chen Change Loy. ResShift: Efficient diffusion model for image super-resolution by residual shifting, NeurIPS, 36, 2024.
[41] Yi Zhang, Xiaoyu Shi, Dasong Li, Xiaogang Wang, Jian Wang, and Hongsheng Li. A unified conditional framework for diffusion- based image restoration. NeurIPS, 36, 2024. 3
[42] Lingchen Sun, Rongyuan Wu, Zhenggiang Zhang, Hongwei Yong, and Lei Zhang. Improving the stability of diffusion models for content consistent super-resolution. arXiv preprint vrXiv:2411.00877 2023. 3,
[43] Lvmin Zhang, Anyi Rao, and Mancesh Agrawala. Adding conditional control to text-to-image diffusion models. In ICCV, pages 3836-3847, 2023. 3
[44] Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models, arXiv preprint arXiv:2001.0833
2020. 3
[45] Fuzhi Yang. Huan Yang. Jianlong Fu, Hongtao Lu, and Baining Guo. Learning texture transformer network for image super-resolution. In CVPR, pages 5791-5800, 2020. 3
[46] Liying Lu, Wenbo Li, Xin Tao, Jiangbo Lu, and Jiaya Jia. Masa-SR: Matching acceleration and spatial adaptation for reference-basee image super-resolution. In CVPR, pages 5368-6377, 2021. 2021 3
[47] Yuming Jiang, Kelvin CK Chan, Xintao Wang. Chen Change Loy, and Ziwei Liu. Robust reference- based super-resolution via C2- 02-matching, In CVPR, pages 2103-211 2112, 2021. 3, 6, 7, 15
[48] Jiczhang Cao, Jingyun Liang. Kai Zhang, Yawei Li, Yulun Zhang, Wenguan Wang, and Lue Van Gool. Reference-based image super-resolution with deformable attention transformer. In ECCV, pages 325-342. Springer, 2022. 3, 7, 15
[49] Lin Zhang, Xin Li, Dongliang Hc, Fu Li, Errui Ding, and Zhaoxiang Zhang. Lmr: A large-scale multi- reference dataset for seerence-based super-resolution. In ICCV, pages 13118-13127, 2023. 3, 7, 15
[50] Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or Prompt-to- prompt image editing with cross attention control. arXiv preprint arXiv:2208.0166 2022. 4, 16
[51] Namuk Park and Songkuk Kim. How do vision transformers work? In ICLR, 2021. 4
[52] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recogni- tion. arXiv preprint arXiv: 111.144995555 1556, 2014. 5, 6
[53] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, pages 770-778, 2016. 5
[54] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. pages 8748 8763. PMLR, 2021. 5
[55] Xun Huang and Serge Belongie. Arbitrary style transfer in real-time with adaptive instance normalization.
In ICCV, 2017.
[56] Zhifei Zhang, Zhaowen Wang, The Lin, and Hairong Qi. Image super-resolution by neural texture transfer. In CVPR, pages 7982-7991, 2019. 6
[57] Yufei Wang, Zhe Lin, Xiaohui Shen, Radomir Mech, Gavin Miller, and Garrison W Cottrell. Event-specific image importance. CVPR, pages 4810-4819, 2016.
[58] Radu Timofte, Eirikur Agustsson, Luc Van Gool, Ming-Hsuan Yang. and Lei Zhang. NTIRE 2017 challenge on single image super-resolution: Methods and results. In CVPRW, pages 1 114-125, 2017. 6,
[59] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as perceptual metrie. In CVPR, pages 586-595, 2018. 6
[60] Anish Mital, Rajiv Soundararajan, and Alan C Bovik. Making a "completely blind" image quality analyzer. IEEE Signal processing letters, 2013)2009-12,, 2012. 6
[61] Martin Heusel. Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to local nash equilibrium. NewrIPS, 30, 2017.
[62] Junjie Ke, Qifei Wang. Yilin Wang, Peyman Milanfar, and Feng Yang. MUSIQ: Multi-scale image quality transformer. In ICCV, pages 202. 6
13

---

[63] Jianyi Wang. Kelvin CK Chan, and Chen Change Loy. Exploring CLIP for assessing the look and feel of images. In AAAI, volume 37, pages 2555-2563, 2023.
[64] Chenggang Yan, Biao Gong. Yuxuan Wei, and Yue Gao. Deep multi-view enhancement hashing for image retrieval. IEEE TPAMI, 43(44:14445-444 1451, 2020. 17
[65] Siteng Huang, Biao Gong, Yulin Pan, Jianwen Jiang. Yiliang Lv, Yuyuan Li, and Donglin Wang. VoP: Text-video co-operative prompt tuning for cross-modal retrieval. In CVPR, pages 6565-6574, 2023. 17
14

---

Appendix
A Adaptive Multi-reference Injection
Technical details. In the main paper, we mainly focus the case of one single retrieved image. However, in practice, there may be multiple available reference images at hand, and using multi- ple reference images for resemblance could intuitively gain better performance. To this end, we extend the original cross- cross-image injection to allow to incorporation of multiple reference images for reconstruction. Our key idea is to modify the scale factor s in Eq. (2) from scalar into vector: 8 = {81, $2, ... 8k), where Σ 8n, = 1. Each 8n can be obtained by computing the cosine similarity between LQ and the corresponding n-th retrieved image in I followed by Softmax normalization. Then we can modify the original single-reference cross-image injection of Eq. (2) to the following multi-reference version:
fuuee = (1 Σ Snn & Ointer n=
Mn) @ Ointra:
(4)
where M, denotes the gated mask of the n-th reference image.
Experiments with multiple reference images. For experiments with multiple reference images, we use SUPIR+ReFIR as a representative. Since the CUFED5 dataset contains multiple reference images, we directly use the provided images as the retrieved reference for reproducibility. Tab. gives the results. It can be seen that using multiple reference images produces better results than one single reference image, e.g. the 2.08 improvement FID. However, is worth noting that the marginal gain from adding reference images is diminishing, accompanied by notable increase in computational cost. Therefore, practice, we use one single reference image to balance the model performance and inference efficiency.
B
More Discussions
Difference from the other methods. Our ReFIR uses retrieved images the reference for high- fidelity restoration. Despite both RefSR methods and ours appears the reference image, we would like to clarify the difference between ReFIR and previous RefSR methods. Firstly, current RefSR models [47, 48, 49] are typically small-scale (#param <50M) and use simple Bicubic degradation, while our ReFIR focuses on the recent diffusion- based large-scale restoration model (#param >1B) for more challenging real-world SR. Secondly, most RefSR methods can only use one reference image and even fail work in the absence of reference images, by contrast, our ReFIR can flexibly use ~ k images. Thirdly, different from RefSR models that require training, our method can be applied in various LRMs training-free manner.
Performance under extreme conditions. Since our Re- FIR relies on the retrieved images, it is interesting to plore extreme situations when highly relevant and high- quality reference images scarce even unavailable. To this end, we introduce the fallback strategies to handle this situation. Specifically, since our method does not modify the parameters of LRMs, we can directly use the original inference pipeline of the LRM without using reference images. We denote this as origin Lingin_rmmm. In addition, we also use the BLIP model to caption the LR image to obtain the text prompt, which will then be fed into the StableDiffusion.00 model to generate semantic-similar high-quality images the reference. We denote this as gen_ref. We use SeeSR [18] as represen- tative, on the real-world degradation dataset RealPhoto60 [19]. We first give the results in which all LR images adopt the fallback strategies in Tab. It can be seen that using the SD2.0 generated images as the fallback image can bring slightly improvement compared with noReference. After that, we further develop task-oriented adaptive strategies to enhance the performance of ReFIR in real-world scenarios. In detial, we respectively use the retrieved images and the gen_ref to generate the results. And then we select the one with larger task score as the final result. We denote as ada gen ref. From Tab. 7, it can be seen that the task-orienter strategy achieves a significant
Table 7: Results of all LR images using the fallback strategies.
setup
[IIEEL
MUSIQU
CLIPIQAT
origin_Irm
4.7432
55.54
0.6575
gen gen_ref ref
4.6923
55.98
0.6602
ada ada_gen_no ref
4.3464
57.68
0.6942
ReFIR
4.4986
57.01
0.6759
15

---

Table Experiments on extending to use multiple retrieved images for restoration. The inference
time is evaluated on A100 GPU.
settings
GPU Memory
Inference Time.
PSNR*
SSIMT
LPIPS|
NIQEJ
FID
NoRef
37.3G
146.4s
18.97
0.4665
0,4807
4.5624
168.26
OneRef
51.4G
322.8s 499.2s
18.86
0.4623
0.4492
4.2317
156.10
TwoRef
65.5G
18.78
0.4676
0.4296
4.2315
154.02
0.4
19.0
0.400
19.0
0.90
50al
60800
0.400
182.0
0.420
91.5
156.0
154.0
60.0
048.
0.
0.4
0.8
0.0
1.5
32
1.4
04
0.
2.5
20
122
144
$0.00
Figure 11: Quantitative ablation results the control scales using SUPIR+ReFIR CUFED5.
performance improvement against previous ReFIR baselines, e.g., 0.0183 CLIPIQA improvements, due the fact that it works in the output end. However, this setup is accompanied by a larger inference time, and further acceleration on this fallback strategies can be an promising future work.
Computational overhead from retrieval and attention modification. Since we employ additional Ref images as input and modify the attention layers, we adiscuss the impact of these trchnuques on the inferenve efficiency. First, in order to reduce the computational overhead of the retrieval process, we pre-calculated the feature vectors of images in the retrieval database before inference. Furthermore, the cosine similarity between the LR image vectors and all retrieval vectors is computed in parallel. These strategy results in an almost negligible (less than 3% inference time) cost computational overhead. Second, the modification of self-attention layers only happens in the last 20 timestep in the decoder layers, i.e., only 12% attention layers are modified while the left is kept intact. These analysis is also supported by practice, in which find these two process only take up <5% inference time, with most computational cost coming from the original LRM. Future LRM acceleration (e.g. pruning, quantization, one-step diffusion) will benefit ReFIR, and we will explore more efficient implementation the future.
Why use the self-attention as the external knowledge? In the proposed cross-image injection, use the features of the self-attention layer of the Cs's 's decoder as external knowledge to guide Cr to produce textures faithful to the original scene. Here, we give the reason behind this. Firstly, previous image-to-imadt tomeagge efforts [29, 31, 30, 50], e.g., image editing. has demonstrated through extensive experiments that the self-attention layer of the diffusion model contains important spatial correlations images, which inspired us to follow this clue to utilize this prior. Secondly, leveraging the attention mechanism allows CT to query features in Cs without any training, whereas using features from other parts of Cs may require introducing additional training.
What about the quality of cross-image attention? In the proposed cross cross-image image injection, the inter-chain attention is used to perform attention between QT and Ks. Considering the domain gap between Cr and Cs due to the input quality difference, one may ask whether the results of the inter-chain attention are meaningful. Here, We visualize the attention map to validate the effectiveness of the inter-chain attention (see Fig. 13). It can be seen that for given query pixel query in Cr.
16

---

the inter-chain attention can effectively overcome the spatial misalignment, and find relevant pixel features in Cg for reference.
C More Ablation Results
Quantitative ablation the control scale. We also provide quantitative ablation results
the control scale 8 in Fig. 11. It can be seen that when is too small, the LRM will mainly use the knowledge contained within its parameters restore high-quality images, which can lead to performance degradation due to the hallucination problem. On the other hand, when s too large, the LRM will overuse the content in the retrieved reference image, thus producing patterns that are not present in the original LQ image. In practice, we adopt a moderate = 0.5 to trade off the hallucination and the overuse of the reference image.
Other choices for cross image injection. The proposed cross image injection mitigates the domain preference problem by using separate attention to promote latent in Cr attend Cs- Here, we conduct ablation to study the impact of different design choices of cross image injection. As shown in Tab. 9, directly replacing the original self- attention results from Ointra in Cr with corresponding latent in Cs causes severe performance degradation, due to the significant loss of original knowledge in Cr. In addition, using Qr query the concatenation results of KT and Kg also causes a performance drop, which further confirms that the domain preference problem, i.e., Qr prefers to use latent from the same chain Cr, even though Cs is more helpful for reconstruction.
Table 9: Ablation experiments on different cross image injection designs.
settings
|PSNR| SSIM
NIQE!
FID
replace concat
18.84
0.4385
4.26
182.82
18.89
0.4691
4.19
156.03
baseline
19.00
0.4729
4.21
148.69
D Statistical Significance on Performance
In Tab. 1 of the main paper, we give the performance gains of incorporating the proposed ReFIR into the existing LRMs. Considering the randomness of the generative models, we give the performance fluctuations of ReFIR under multiple trials with exactly the same experimental setting and random seed. The results are given in Tab. 10. It can be seen that the randomness of the diffusion-bass generative model is very small when using fixed seed, reducing the disturbance from noise errors for evaluation. In addition, we further use hypothesis testing to verify the significance of performance gains, and the test results reject the original hypothesis HO at 95% confidence level all metrics and datasets, indicating that the performance gains from the proposed method are statistically significant.
E
Extension to Specific Restoration Scenarios
An important application of our method is in scenarios with high fidelity demands, such as scene text images with specific stylistic structure, or face images with identity preservation, and here we preliminarily explore the application of the proposed ReFIR to real-world face image restoration. The results are given in Fig. 15. It can be seen that by using a high quality image of specific person's identity as a reference, the resulting restoration results can better preserve the person's 's attributes. However, it should be noted that this experiment is just preliminary attempt, and will leave the further improvement of our ReFIR for specific downstream restoration tasks for future work.
F
Limitation and Future Works
Although the proposed ReFIR can effectively mitigate the hallucination of LRMs by introducing external knowledge from retrieved reference images, the proposed framework can be further improved in the following aspects. First, since the computational complexity of the current LRMs is costly, the computational complexity will be further increased when using the proposed method, which may hinder the use of resource-constrained mobile devices. With the advent of accelerated diffusion- image restoration methods in the future, we believe that the proposed method can further improve its efficiency. In addition, this paper proposes simple retriever based on semantie vector matching for presentation. With the development of image retrieval techniques [64, 65], designing specialized
17

---

Table 10: Performance fluctuations under different experiment trials. We use ten trails to obtain stable fluctuation range.
CUDEDS LPIPS,
WR-SR LPIPSL
settings
PSNR1
SSIM1
NIQE4
FID.
NIQE,
FID
ReIIR
20.32
0.5289
0.3338
3.7831
134.62
21.86
0.5664
0,3460
3.9089
61.22
Numerical flucuatiios
+0.0013
+000001
+000002
+0.0001
+0.03
+0.001
+ 0.0001
+-0.0002
+0.0001
+ 0.02
SUPIR-ReFiR
19.00
0.4729
0.4341
4.2085
3.7478
71.82
Numericul fluctuations
+0,055
+0.0003
+0.001
+0.012
+0.5
+0.002
+0.0003
+0.0006
+0.007
+0.3
retrievers, e.g., using textures as key matching cues, will further improve the performance. Finally, for some slightly degraded images, which can already be handled well by only using the internal knowledge of the LRMSs, designing hyper-networks to adaptively decide whether to use retrieval augmentation or is also promising. We leave the above considerations for future work.
G Broader Impact
The development of our ReFIR offers significant positive societal impacts, including advancements medical imaging, historical preservation, and media restoration by enhancing the fidelity and realism of image restoration. However, it also poses potential negative societal impacts, such the misuse of improved restoration capabilities for generating disinformation, deepfakes, and surveillance, raising ethical concerns about privacy, security, and faimess. To mitigate these risks, implementing safeguards like gated releases of models, monitoring mechanisms, and transparency in model training and deployment is crucial. Continuous ethical evaluation and adherence to strict guidelines essential to prevent potential harms.
Additional Visual Results
In this section, we provide more visual results, which are organized follows:
In Fig. 12, we give more samples of the PCA visualization on the top three principal components of the self-attention layer latent.
In Fig. 13, give visualization of the attention map from the cross-image injection, to help better understand the feasibility of cross-image attention.
Fig. 14 gives more quantitative comparison results against the state-of-the-a method on real-world degradation without ground truth.
Fig. 15 gives the visualization results of the extension experiments of applying the proposed ReFIR blind face image restoration.
18

---

Original Image
Depth
ControlNe:
OF
Decader
ControlNes
Decaler
ControlNet
Decoder
ControlNet
Decoder
Controll
Decader
Figure 12: Additional visualization on the top three principal components of the self-attention layer latent of PCA. The latent is extracted from first self-attention layer within blocks control net and unet decoder.
19

---

Query Point
LQ Image
Reference
Attention
LO image
Queery Point
Attention Map
neter neference
Query Point
Image
Attention Map
Reference
LQ Image
Query Paint
Actention Map
Reference
LQ Image
Query Paint
Reference
Attention
Figure 13: Additional visualization on the attention maps from the cross image injection. It can be seen that the query pixel in the Cr can well attend similarly region from Cs.
20

---

StableSR
DiIIIR
PASD
CCSR
Low-quality Image
SeeSR
SUPIR
SoeSR+ReFIR
SUPIR+ReFIR
StableSR
DiffBIR
PASD
CCSR
Low-quality Image
SeeSR
SUPIR
SeeSR +ReFIR
SUPIR+ReFIR
StableSR
DiffBIR
PASD
CCSR
Low-quality Image
SeeSR
SUPIR
SeeSR+RRe +ReFIR
SUPIR +ReFIR
StableSR
DiffBIR
PASD
CCSR
Low-quality Image
SeeSR
SUPIR
SeeSR+ReFIR
SUPIR+ReFIS
Figure 14: Additional qualitative comparison with state-of-the. methods on RealPhoto60 [19].
Please zoom in for better effects.
21

---

Low-quality Image
SeeSR
SUPIR
SeeSR +ReFIR
SUPIR+RReIR
Low-quality Image
SeeSR
SUPIR
SeeSR +ReFIR
SUPIR +ReFIR
Low-quality Image
SeeSR
SUPIR
+ReFIR
Nww-quality Image
SeeSR
SUPIR
SeeSR+ +ReFIR
RReFIR
Figure 15: Visualization results of applying the proposed ReFIR to the downstream specific domain of blind face image restoration. Please zoom in for better effects.
22

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the
paper's contributions and scope?
Answer: [Yes]
Justification: We have clearly outlined the main contributions of this paper in the abstract and introduction sections.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We have discussed detail the limitation of this work in Appendix F.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
23

---

All assumptions in the paper are either derived from the conclusions of previous research or validated through extensive experiments.
Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We have provided a detailed description of our algorithm's process and implementation specifics in the paper, and we will release our code after review.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
24

---

reproducible, they Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Justification: We will release our code after review, but we have already provided a detailed explanation of how to implement our algorithm and the specific implementation details in the paper.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https: htt///nips..p/ cc/
public/ guides CodeSubmii s siionoooicyy for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" an acceptable answer. Papers cannot be rejected simply for including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: / /nnps cc/ publi icc////// cu des CodeS ubmi ss i onPol ierolicc icy) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permirted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We have provided detailed experimental details in both the paper and the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: We have provided the performance fluctuation under different random seeds in Appendix D.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
25

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative eπτor rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We provided the computational resources we used for experiments in the ??.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https:/ /neurips.....//
Answer: [Yes]
Justification: We have checked in every respect that this work conform with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We have provided detailed discussion on possible both positive and negative societal impact Appendix G.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
26

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
This paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: Yes, the creators and original owners of assets used in this paper are prop- erly credited, and the license and terms of use are explicitly mentioned and respected, as evidenced by thorough citations.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
27

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
assets are released, the license, copyright information, and terms of use the package should be provided. For popular datasets, papers thoode com/ dat a seets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
This paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmodel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
This paper does involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
28

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
29


Reviews:
[{"review_id": "pr15AF37gP", "reviewer": "Reviewer_NDC8", "summary": "The paper titled \"ReFIR: Grounding Large Restoration Models with Retrieval Augmentation\" introduces a novel framework called Retrieval-augmented Framework for Image Restoration (ReFIR). This method addresses a significant issue in diffusion-based Large Restoration Models (LRMs) — the tendency to generate hallucinatory outputs when faced with heavily degraded images, akin to issues faced by large language models. ReFIR mitigates this by incorporating external high-quality, content-relevant images retrieved via a nearest neighbor lookup in the semantic embedding space. These images serve as a source of external knowledge, enabling the restoration model to produce more accurate and faithful details. The framework modifies the self-attention layers of existing LRMs to integrate textures from the retrieved images, significantly enhancing the model's ability to restore images without any additional training required. Extensive experiments demonstrate that ReFIR achieves realistic and high-fidelity restoration results, confirming its effectiveness across various existing LRMs. This training-free, adaptable approach significantly expands the knowledge boundary of LRMs, offering a promising solution to their inherent limitations.", "questions": "* Given that the paper primarily demonstrates ReFIR's performance on specific types of image degradations, can the authors clarify how well the framework performs across a broader spectrum of degradation scenarios, including less common or more complex types? Further elaboration on its performance in varied real-world conditions could significantly clarify the framework’s versatility and robustness.\n\n*  Can the authors provide more details on the computational efficiency of the retrieval process integrated within the ReFIR framework? Specifically, what are the impacts on processing time and computational resources when applying ReFIR to large-scale restoration models, and are there strategies in place to optimize this process in real-time applications?\n\n*  In cases where the retrieval process fails to find sufficiently relevant or high-quality images, what strategies does ReFIR employ to ensure the quality of the restoration output? A detailed explanation of fallback mechanisms or alternative approaches when ideal reference images are not available would be helpful in assessing the framework's reliability and functionality in less-than-ideal conditions.", "limitations": "Since ReFIR relies on external datasets for retrieving reference images, there is a potential for bias if those datasets are not diverse or inclusive of various demographic groups and scenarios. This could inadvertently lead to biases in restored images, particularly in sensitive applications. The authors should discuss the measures taken to ensure the diversity of the datasets and address potential biases in the image retrieval and restoration process.", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* The paper is well-organized and articulates complex concepts with clarity, making it accessible to both experts and those new to the field. The significance of ReFIR lies in its potential to profoundly impact practical applications involving image restoration, such as in digital forensics, media restoration, and medical imaging, where accuracy and fidelity are paramount. This approach offers a scalable and adaptable solution that can be applied across various existing models without the need for retraining, highlighting its utility in improving the practical deployment of deep learning models for image restoration.\n\n* The ReFIR framework introduces a unique method of integrating retrieval-augmented techniques into image restoration models, addressing the hallucination dilemma often faced by large restoration models. By borrowing the concept of retrieval-augmented generation from natural language processing and adapting it for image processing, ReFIR creatively leverages external image databases to enhance the detail and fidelity of restored images, significantly expanding the utility of existing large-scale models.\n\n* The implementation of ReFIR demonstrates robust quality through extensive testing and innovative adaptations. The framework modifies existing large restoration models by integrating high-quality, content-relevant images during the restoration process. This method has shown to significantly enhance the performance of these models, as validated by comprehensive experiments that not only improve quantitative metrics like PSNR and SSIM but also yield visually more accurate restorations.", "weaknesses": "* The effectiveness of ReFIR is significantly dependent on the quality and relevance of the images retrieved from external databases. This reliance could limit the framework's effectiveness in scenarios where highly relevant and high-quality reference images are scarce or unavailable. Addressing this, the authors could explore mechanisms to assess and ensure the relevance and quality of retrieved images dynamically during the restoration process or develop fallback strategies when suitable images are not found.\n\n* While the paper mentions that the framework does not require additional training, it does not thoroughly address the potential increases in computational overhead and latency introduced by the retrieval process and the modification of self-attention layers. This could be particularly challenging when deploying in real-time or resource-constrained environments. Future iterations of the framework could benefit from a detailed analysis of computational costs and potential optimizations to streamline the retrieval and integration processes.\n\n* The experimental setup primarily demonstrates the effectiveness of ReFIR under controlled conditions with specific types of image degradations. To fully validate the robustness and generalizability of the framework, additional testing across a broader spectrum of real-world scenarios and more diverse degradation types would be beneficial. This would help in understanding the limitations and operational range of ReFIR in practical applications, ensuring it can effectively handle unexpected or uncommon degradation patterns.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### [Q1: Reliance on the image quality and relevance] \n\n> The effectiveness of ReFIR is significantly dependent on the quality and relevance of the images retrieved from external databases. This reliance could limit the framework's effectiveness in scenarios where highly relevant and high-quality reference images are scarce or unavailable. Addressing this, the authors could explore mechanisms to assess and ensure the relevance and quality of retrieved images dynamically during the restoration process or develop fallback strategies when suitable images are not found.\n\nWe understand your concerns, and we have thus conducted a thorough experiment presented in the **global author rebuttal part**. \n\nWith these newly developed fallback and filtering strategies, our ReFIR is further improved under the real-world retrieval settings, relieving the dependence on retrieval system.\n\n### [Q2: Computational overhead from retrieval and attention modification]\n> While the paper mentions that the framework does not require additional training, it does not thoroughly address the potential increases in computational overhead and latency introduced by the retrieval process and the modification of self-attention layers. This could be particularly challenging when deploying in real-time or resource-constrained environments. Future iterations of the framework could benefit from a detailed analysis of computational costs and potential optimizations to streamline the retrieval and integration processes.\n\nIn order to reduce the computational overhead of the retrieval process, we **pre-calculated** the feature vectors of all images in the retrieval database before inference. Furthermore, the cosine similarity between the LR image vectors and all retrieval vectors is computed **in parallel**. These strategy results in an **almost negligible** (less than 3% inference time) cost of computational overhead.\n\nThe modification of self-attention layers only happens in the last 20 timestep in the decoder layers, i.e., **only 12%** attention layers are modified while the left is kept intact.\n\nThese analysis is also supported by practice, in which we find these two process only take up <5% inference time, with most computational cost coming from the original LRM. Future LRM acceleration (e.g. pruning, quantization) will benefit our ReFIR, and we will explore more efficient implementation in the future.\n\n### [Q3: Testing on broader degradations]\n> The experimental setup primarily demonstrates the effectiveness of ReFIR under controlled conditions with specific types of image degradations. To fully validate the robustness and generalizability of the framework, additional testing across a broader spectrum of real-world scenarios and more diverse degradation types would be beneficial. This would help in understanding the limitations and operational range of ReFIR in practical applications, ensuring it can effectively handle unexpected or uncommon degradation patterns.\n\nAs suggested, we test the robustness of our ReFIR framework on another two challenging real-world mixed degradations, i.e., low-resolution$\\times$4+noise$\\sigma$=50, as well as low-resolution$\\times4$+JPEG$q=30$. The results are as follows.\n\n\nTableA low-resolution$\\times$4+noise$\\sigma$=30 on the WR-SR dataset\n| Metric | SeeSR | SeeSR+ReFIR |\n|--------|-------|-------------|\n| PSNR   | 23.06 | 23.16       |\n| SSIM   | 0.6273| 0.6298      |\n| LPIPS  | 0.2710| 0.2698      |\n| NIQE   | 3.8580|  3.7266           |\n| FID    | 44.10 | 42.78       |\n\nTableB low-resolution$\\times4$+JPEG$q=30$ on the WR-SR dataset\n| Metric | SeeSR    | SeeSR+ReFIR |\n|--------|----------|-------------|\n| PSNR   | 23.84    | 23.91       |\n| SSIM   | 0.6679   | 0.6686      |\n| LPIPS  | 0.2291   | 0.2287      |\n| NIQE   | 3.8723   | 3.8148      |\n| FID    | 36.74    | 35.56       |\n\nIt can be seen that our ReFIR framework maintains its effectiveness on real-world hybrid degradations, demonstrating its robustness and generalizability.\n\n\n\n### [Q4: Possible bias from retrival database]\n\nWe agree with you. Currently, we only use publicly available, high-quality academic data as the retrieval database. In the future, we will pay attention to content diversity and potential bias in constructing larger scale retrieved data, e.g., we could use VLMs or LLMs for automatic filtering followed by human review.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7IiEh38HuW", "reviewer": "Reviewer_SJc6", "summary": "This paper presents a plug-and-play approach to enhance the quality of Diffusion-based super-resolution models by leveraging reference images. The authors exploit CLIP to effectively filter high-definition images with similar semantics from a pre-trained dataset. These reference images are then employed to replace the original diffusion model's features within the attention operation, leading to improved super-resolution results.", "questions": "1. The paper does not explicitly address the potential issue of similar features being missed due to image chunking in pre-trained SD models like SeeSR and SUPIR. Consider discussing whether the model retrieves features from the corresponding chunk of the reference image or employs a different strategy to handle chunked inputs.\n2. It would be helpful to clarify whether the features used for matching in the reference image retrieval process are pre-computed or computed online.\n3. The paper explores the concept of internal knowledge (learned by the LRM) and external knowledge (introduced through reference images). While directly training the LRM on the reference image dataset seems like an alternative approach, it's important to consider the potential drawbacks, such as overfitting to the specific reference data and reduced generalizability. Discussing these trade-offs would provide a more nuanced understanding of the benefits of using reference images as external knowledge.", "limitations": "They have.", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The paper makes a valuable contribution by demonstrating how introducing external knowledge via CLIP filtering can effectively address the hallucination problem in Diffusion-based super-resolution models.\n2. The paper presents promising quantitative and qualitative results that support the effectiveness of the proposed method.", "weaknesses": "1. Figures 5 and 6 would benefit from including the reference images selected by the model alongside the processed results. This would allow for a clearer evaluation of the effectiveness of the reference image selection process. Consider revising the figures to include the reference images or provide additional visualizations that demonstrate the impact of the reference information on the final output.\n2. Tables 1 and 2 highlight a significant performance boost from ReFIR on the RefSR dataset, which diminishes in real-world scenarios. This raises questions about ReFIR's ability to handle diverse real-world data. Can the authors provide additional evidence or analysis to demonstrate ReFIR's effectiveness in retrieving relevant reference images for real-world scenarios, even when the similarity might not be as high as in the RefSR dataset? Or, could we generate reference images automatically just like CoSeR? The comparison and more detailed discussion is necessary.\n3. The appendix appears to contain crucial information about the reference image retrieval stage, such as the selection criteria for the feature extractor. This information should be moved to the main body of the paper for better clarity and transparency.\n4. The paper should discuss the computational cost and time consumption associated with the \"Source Reference Chain\" process. While the performance improvement is valuable, quantifying the trade-off in terms of computational resources is essential.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### [Q1: Reference image presentation]\n\nThanks for your advice, we will add the retrieved image in Fig.5 and Fig.6 in the revision to improve the presentation quality.\n\n### [Q2: Improving ReFIR's ability in real-world scenarios]\n\n> Tables 1 and 2 highlight a significant performance ...  detailed discussion is necessary.\n\nTo further improve the ability of ReFIR in real-world applications, we further explore several techniques to improve its robustness. The related analysis and experiments are shown in the **global author rebuttal**.\n\nEquipped with the fallback mechanisms (not using reference or generating Ref like CoSeR) as well as the adaptive filtering strategies (relevance/quality/task-based filtering), our ReFIR can alleviate the dilemma when high-correlated and high-quality images are scarce or unavailable, enhancing its ability in real-world scenarios.\n\n\n### [Q3: Moving the retrieval stage details to main body]\n\nDue to the 9-page limit for submission this year (1 page less than usual), we are sorry for placing the image retrival details in the appendix. We will add more details about the image retrieval stage in the main body in the revision.\n\n\n### [Q4: Computaion and time cost from Source Reference Chain]\n\n> The paper should discuss the computational cost and time consumption associated with the \"Source Reference Chain\" process. While the performance improvement is valuable, quantifying the trade-off in terms of computational resources is essential.\n\n\nSince the \"Target Restoration Chain\" only receives the corresponding same time-step and layer features from the \"Source Reference Chain\", this property thus allow the **batch acceleration** for efficient implementation. Specifically, these two chains are implemented sharing one LRM weight, and simply increasing the batch size from 1 (LR only in original NoRef LRM) to 2 (LR+Ref in our ReFIR).\n\nTherefore, the computational cost and time consumption of ReFIR is roughly similar to the original LRM with batchsize set to 2, with a slight overhead increase from the additional cross-image injection. The comparison of model complexity before and after incorporating our ReFIR are given in Table7 of Addpendix, and here we also give this reuslt as follows:\n\nTableA. Comparison on computation and time cost. The input resolution is $2048\\times2048$, and is evaluated on one 80G A100 GPU.\n\n| Method       | input          | #param | GPU memory | Inference time |\n|--------------|----------------|--------|------------|------------------|\n| SeeSR| batchsize=2    | 2.04B  | 40.2G| 160.2s|\n| SUPIR| batchsize=2    | 3.87B  | 51.1G| 312.2s|\n| SeeSR+ReFIR  | 1xLR + 1xRef   | 2.04B  | 40.9G      | 170.7s|\n| SUPIR+ReFIR  | 1xLR + 1xRef   | 3.87B  | 51.4G      | 322.8s|\n\n\n### [Q5: Dicussion on patchfied images ]\n\n> Consider discussing whether the model retrieves features from the corresponding chunk of the reference image or employs a different strategy to handle chunked inputs.\n\nThe patchfy is usually used when the input resolution is large. Although it is intuitive to use the corresponding patch from the large Reference Image as the reference branch's input, it can inevitably incur significant performance degradation when the LR and Ref possess spatial misalignment. \n\nHere, we introduce another solution which simply uses existing techniques to deal with this problem. Specifically, we employ the Multi-reference injection, shown in Appendix-A, in which we use all the patches from the reference image to guide the restoration of a certain LR input patch. In this way, the LR image can maintain the global receptive filed from the Ref. In experiment, we choose the number of patches as 4 for a 2048x2048 input from RealPhoto60 dataset, resulting in 4 small patches with size 512x512. The results are shown as bellow.\n\n| setup          | GPU cost | NIQE    | MUSIQ  | CLIPIQA       |\n|----------------|----------|---------|--------|---------------|\n| MultiRef-for-patchfied images | 29.8G   | 4.6543 | 56.20  | 0.6598        |\n| baseline       | 40.6G    | 4.4566  | 57.13  | 0.6732        |\n\nTo some extent, introducing the multi-reference injection can alleviate the perofrmace drop while maintaining modest memory footprint. However, similar to exsting SD-based restoration works, the inherent tradeoff between whole-image receptive filed and the low GPU cost still exist. Our framework may potentially benefit from future acceleration works.\n\n### [Q6: Computation mode for retrieval]\n\n> It would be helpful to clarify whether the features used for matching in the reference image retrieval process are pre-computed or computed online.\n\nSince the retrival database is agnostic to the specific LR inputs, we thus **pre-compute** the retrival vectors for fast inference. We will make it more clear in the revision.\n\n\n### [Q7: Converting refernece image as internal knowledge]\n\nDue to the backpropagation of LRM needs huge GPU costs, e.g.  64 Nvidia A6000 GPUs for training SUPIR, we are not able to practically convert reference images as internal knowledge through fully fine-tuning.\n\nHowever, we point out that the idea of injecting downstream knowdge through fine-tuning has been explored in the image genration community, such as DreamBooth. During the subject-driven adaptation of dreambooth, the authors of dreambooth also encountered diffusion model's overfitting for specific subjects, and they propose the prior preservation loss to solve this problem. \n\nAs for the image restoration, it is difficult to get calibration datasets for prior preservation since the content of LR images can be various, unlike the similar topic in image generation. Moreover, in real-world applications, it is also difficult to obtain scene-specific data for training in advance, and even if we get it, it requires additional training time. In contrast, our ReFIR framework does not need the scene-specific data in advance and can adapt to specific scenes in a training-free manner.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "VecEa28ek2", "reviewer": "Reviewer_UnRT", "summary": "The paper introduces ReFIR, a novel method designed to enhance the capabilities of Large Restoration Models by incorporating external knowledge through the retrieval of high-quality, content-relevant images. \nThe main contribution is to (1) Gives both quantitative and qualitative results on how existing LRM works. (2) Propose a novel solution which is training-free and generic to alleviate the hallucination problem of LRMs.", "questions": "1. How does the retrieval system scale with increasingly large HQ retrieval datasets, and what are the computational implications of scaling?\n2. How might different retrieval algorithms (beyond nearest neighbor lookups) impact the performance of ReFIR?", "limitations": "Further exploration into the scalability, efficiency, and the robustness of the proposed method as mentioned in the Weakness would be helpful for broader application.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The idea of using external data representation instead of the model parameters seems interesting, which can be used in parallel with other model-oriented approaches.\n2. The experiments are solid and the performance are noteworthy. The authors apply the proposed technique into various LRMs and compare with sota under settings with different difficulty levels. From Table1 and Table2, the ReFIR methods achieves significant gains on both fidelity and perceptual metrics, demonstrating the superiority.\n3. The framework can be potentially used for multiple Diffusion-based restoration models without additional training, making it a cost-effective solution.\n4. The paper is well organized and easy to follow.", "weaknesses": "1. Difference form other methods. The proposed method seems relevant to some of works in image editing tasks, such as MasaCrtl and Prompt2prompt, which also use training-free technique to modify the behavior of the diffusion model. A detailed explanation about the differences between the proposed ReFIR method and these methods would be beneficial.\n2. Dependency on the Quality of Retrieved Images. From Table5, it seems the performance of ReFIR heavily relies on the relevance and quality of the retrieved images. If the retrieval system performs sub-optimally, it could adversely affect the restoration outcomes.\n3. Complexity in Implementation. The proposed cross-image injection and spatial adaptive gating mechanisms introduce additional operation, moreover, the input image also contains extra reference images, which might challenge efficient implementation in practice.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### [Q1: Difference from other works]\n\n> Difference form other methods. The proposed method seems relevant to some of works in image editing tasks, such as MasaCrtl and Prompt2prompt, which also use training-free technique to modify the behavior of the diffusion model. A detailed explanation about the differences between the proposed ReFIR method and these methods would be beneficial.\n\nAlthough both ReFIR and previous image editing work are both training-free, however, we point out that they differ in the following ways. \n\n- The specific techniques are different. Although both modify the attention layer, our ReFIR further introduces the Spatial Adaptive Gating to solve the spatial misalignment between LR and Ref, and proposes strategies to allow the injection of multiple reference images. In addition, we also propose the novel Distribution Alignment to alleviate the domain gap between two diffusion process chains.\n- The goals are different. Previous image editing work modifies the attention layer for controllable generation, while our ReFIR aims to inject external knowledge to alleviate the hallucination problem of existing LRMs.\n- Finally, in addition to incorporating reference images through training-free attention modification, another contribution of this work is the proposal of a feasible retrieval system to facilitate the acquisition of external knowledge from retrieved databases.\n\n### [Q2: Dependency on the Quality of Retrieved Images]\n\n> From Table5, it seems the performance of ReFIR heavily relies on the relevance and quality of the retrieved images. If the retrieval system performs sub-optimally, it could adversely affect the restoration outcomes.\n\nTo make our ReFIR more robust under conditions when the relevance and quality of the retrieved images is poor, we further explore techniques including fallback strategy and the adaptive filtering policy. Please see the global author rebuttal part for more details.\n\nUsing these techniques can further improve the robustness of ReFIR when the retrieval system performs sub-optimally. We will explore more techniques for improvements in the future.\n\n\n### [Q3: Complexity in implementation]\n\n> The proposed cross-image injection and spatial adaptive gating mechanisms introduce additional operation. Moreover, the input image also contains extra reference images, which might challenge efficient implementation in practice.\n\nActually, since the modification of the attention layer only happens in the decoder in the last 20 timesteps, which only accounts for 12% of all the layers, the cross-image injection and spatial adaptive gating rise **almost negligible** increase in computational cost. We provide the specific cost comparison as follows.\n\n| setup                     | input             | #param | GPU memory | Inference time |\n|---------------------------|-------------------|--------|------------|------------------|\n| w/o cross-image injection  | batchsize=2      | 3.87B  | 51.1G      | 312.2s          |\n| and spatial adaptive gating| 1xLR + 1xRef     | 3.87B  | 51.4G      | 322.8s          |\n\n\nSince the LR chain only receives the corresponding same layer features from the Ref chain, this property thus allow the **batch acceleration** for efficient implementation. Specifically, the input batch size is increased from 1 (LR only in original NoRef LRM) to 2 (LR+Ref in our ReFIR).  Therefore, the batch parallel computation of both LR and Ref make it efficient for practical implementation.\n\n\n### [Q4: Scaling up with lager retrieval database]\n\n> How does the retrieval system scale with increasingly large HQ retrieval datasets, and what are the computational implications of scaling?\n\n\nAs suggested, we use ImageNet as the larger database for scaling up the retrieval system. The results are as follows. \n\n\n| setup                 | NIQE    | MUSIQ  | CLIPIQA |\n|----------------------|---------|--------|---------|\n| ImageNet as Retrival | 4.4233  | 57.14  | 0.6770  |\n| baseline             | 4.4566  | 57.13  | 0.6732  |\n\nIt can be seen that increasing dataset size can improve the relevance and thus improves the performance. Furthermore, since the calculation of similarity between LR and retrieved dataset is parallel, we find it brings almost negligible computational cost when scaling up retrieved dataset.\n\n\n### [Q5: Explore differeent retrieval algorithms]\n\n> How might different retrieval algorithms (beyond nearest neighbor lookups) impact the performance of ReFIR?\n\nAs suggested, we use the TopK retrieval algorithm to explore the impact of different retrieval algorithms. Specifically, we select images with top 3 large similarity as reference inputs, and use the multi-reference injection technique to allow multiple reference images. The experimental results are shown below.\n\n| setup     | NIQE    | MUSIQ  | CLIPIQA |\n|-----------|---------|--------|---------|\n| Top3-reference      | 4.4217  | 57.25  | 0.6749  |\n| baseline  | 4.4566  | 57.13  | 0.6732  |\n\n\nIt can be seen that using the TopK retrieval algorithm can bring more relevance to the reference image, thus improving the performance.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "sHaqoyDv2D", "reviewer": "Reviewer_qg3y", "summary": "This work introduces a novel training-free paradigm which uses the retrieval augmentation to expand the knowledge boundaries of existing large restoration models by incorporating reference images as external knowledge to facilitate the restoration of high-fidelity details. The authors propose the nearest neighborhood lookup to retrieve reference images, followed by the cross-image injection to fuse external knowledge into the LRM. Qualitative and quantitative experiments demonstrate the effectiveness of the proposed method.", "questions": "In the spatial adaptive alignment module, considering the size of the similarity matrix $sim$ as $HW \\times HW$, which requires computing per-pixel similarity, does this introduce significant inference costs? Will the authors open-source the related details and codes?", "limitations": "Please see the above weaknesses.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Although diffusion-based restoration models can produce realistic results, their inherent randomness often leads to outputs that are not faithful to the original scene. This work considers an approach by introducing additional reference images to mitigate this challenge.\n2. The idea of this work interesting. Utilizing RAG techniques, which has been widely applied in NLP, for low-level computer vision tasks is promising. The authors provide a detailed quantitative and qualitative analysis of the workings mechanism of existing LRMs, based on which they propose specific designs to inject external knowledge into LRMs.\n3. The authors propose specific solutions to address challenges such as domain preference issues and spatial misalignment when directly using high-quality reference images. The framework appears to be generic and requires no training, making the proposed method easily applicable to various existing LRMs.", "weaknesses": "1. In Table 1, why there is a variation of the performance improvement when applying the proposed method to different LRMs? For example, there is a 0.38 dB PSNR improvement in SeeSR but only a 0.03 dB improvement in SUPIR. Additionally, why do the restored images produced by different methods seems still different when using the proposed ReFIR?\n2. It seems the proposed method requires additional reference images as input, which may increase inference costs and time.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### [Q1: Variation of the outputs from different LRMs]\n\n> In Table 1, why there is a variation of the performance improvement when applying the proposed method to different LRMs? For example, there is a 0.38 dB PSNR improvement in SeeSR but only a 0.03 dB improvement in SUPIR. Additionally, why do the restored images produced by different methods seems still different when using the proposed ReFIR?\n\n\nIn fact, we have also noticed this phenomenon and we think it can be well explained based on Fig.8 in the main paper. As shown in Fig.8, for the latent at the t-th time step, there are two forces in different directions pulling it to produce the latent at the next t−1-th time step. One force is from the internal knowledge of frozen weights in LRMs, and the other is the external knowledge from the retrieved reference image. Therefore, although the external knowledge of different LRMs is the same (the reference image is the same), due to their different internal knowledge (different pre-training parameters), it eventually resulting in different LRMs producing different restoration results.\n\n\n### [Q2: Inference costs from additional reference image]\n\n> It seems the proposed method requires additional reference images as input, which may increase inference costs and time.\n\nTo ensure efficient implementation, we use batch accerlaration to process LR and Ref paralley, whose cost is roughly similar to increasing batchsize from 1 to 2. Moreover, although using the reference image inevitably incurs computational cost compared to direct inputing LR for inference, the additional reference can mitigate the hulucination of LRMs, resulting in significant performance gains. We provide a comprehensive comparison regarding performance and effectiveness as follows.\n\n\n| setup       | NIQE    | MUSIQ  | CLIPIQA | #param | GPU memory | Inference time |\n|-------------|---------|--------|---------|--------|------------|------------------|\n| SeeSR-NoRef | 4.7432  | 55.54  | 0.6575  | 2.04B  | 24.4G      | 76.5s            |\n| SeeSR-ReFIR | 4.4566  | 57.13  | 0.6732  | 2.04B  | 40.9G      | 170.7s           |\n\n\n### [Q3: Efficiency of the adaptive alignment module]\n\n> In the spatial adaptive alignment module, considering the size of the similarity matrix sim as HW×HW, which requires computing per-pixel similarity, does this introduce significant inference costs? \n\nWe are sorry to cause your confusion. In fact, the computation of the similarity matrix is quite efficient. Since we find the pixel-wise similarity between LR and Ref does not change too much among different layers, we thus compute the **all layer shared** similarity maxtrix for one UNet block, resulting in only 12 calculation for one timestep. We will make it more clear in the revision.\n\n\n### [Q4: Code open-source]\n\n> Will the authors open-source the related details and codes?\n\nYes! We will release all the code when the paper is accepted.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
## Global Author Rebuttal
### **[1. Remarks by authors]**
We would like to express our sincere gratitude to all the reviewers for taking their time reviewing our work and providing fruitful reviews that have definitely improved the paper. We are encouraged that they find our method
- "offers a scalable and adaptable solution that can be applied across various existing models without the need for retraining" (Reviewer NDC8, Reviewer UnRT, Reviewer qg3y)
- "addressing the hallucination dilemma often faced by large restoration models" (Reviewer NDC8)
- "creatively leverages external image databases to enhance the detail and fidelity of restored image" (Reviewer NDC8)
- "not only improves quantitative metrics like PSNR and SSIM but also yields visually more accurate restorations" (Reviewer NDC8, Reviewer SJc6, Reviewer UnRT)
- "The experiments are solid" (Reviewer UnRT)

During this rebuttal period, we have tried our best and made a detailed response to address the concerns raised by reviewers. If you have any further questions,  we will actively discuss with you during the author-reviewer discussion period.

---

### **[2. Futher exploration on the retrival in the wild]**

We have noticed Reviewer NDC8, Reviewer SJc6, and Reviewer UnRT rise concerns on the performance of our ReFIR when highly relevant and high-quality reference images are scarce or unavailable. Here, we first discuss the mechanism already existed in ReFIR for mitigating uncorrelation problems. After that, we explore several new techniques to further improve the model ability in this extreme situation.

**2.1 Spatial Adaptive Gating for relevance filtering** 

In Sec4.2, we introduce the Spatial Adaptive Gating (SAG) to resolve the spatial misalignment between LR images and Ref images. Since the mask $M$ in Eq.2 contains pixel-wise similarities between LR and Ref, we point out that this similarity-aware mask can **filter out low correlated pixels** from the reference image, thus imporving robustness in the wild. The visualization of this similarity mask $M$ is shown in Fig.13 of the Appendix.


**2.2 Exploration for Further Improvments**

Furthermore, as suggested by reviewers, we further develop several new techniques for further improvement.

We first develop **fallback strategies** to handle the situation where reference images are not available. This includes two possible choices: 

1.  Since our method does not modify the parameters of LRMs, we can directly use the original inference pipeline of the LRM without using reference images. We denote this as `origin_lrm`.

2.  We use the BLIP model to caption the LR image to obtain the text prompt, which will then be fed into the StableDiffusion2.0 model to generate semamtic-similar high-quanlity images as the reference. We denote this as `gen_ref`.

We then develop the **adaptive filtering policy** to assess and ensure the relevance and quality of retrieved images. This includes three alternatives:


1. We set the cosine similarity threshold, and use the retrieved image as the reference only if the similarity is greater than the threshold, otherwise use the image provided by the above fallback strategy. We denote this as $r_{rele}$, and set $r_{rele}$ as 0.6 through ablation.

2. We use the non-reference image quality assessment metric CLIPIQA score as a threshold, and use retrieved image only when its quality is greater than the threshold, otherwise the fallback strategy is used. We denote it as $r_{qual}$, and set $r_{qual}$=0.6246 obtained from the mean CLIPIQA of the retrieval database.

3. Task-oriented adaptive filtering. We respectively use the retrieved image and the fallback strategy to generate the result. And then we select the one with a larger task score as the final result. We denote it as $r_{task}$

Due to limited rebuttal time, we use SeeSR large restoration model as a representative, using the real-world degradation dataset RealPhoto60.

We first give the results in which all LR images adopt the fallback strategies in TableA.

TableA Results of all images using fallback strategies

| setup     | NIQE    | MUSIQ  | CLIPIQA |
|-----------|---------|--------|---------|
| origin_lrm | 4.7432  | 55.54  | 0.6575  |
| gen_ref   | 4.6923  | 55.98  | 0.6602  |
| ReFIR     | 4.4986  | 57.01  | 0.6759  |

It can be seen that using the SD2.0 generated images as the fallback image can bring slightly improvement compared with noRefernce.  But it is still inferior to the results using the retrieved dataset, which we argue is due to the fact that there is a  knowledge overlap between generated images and the LRMs.


Then we combine the two fallback strateges and three filtering policies to obtain a more robust RAG system in the real-world senarios, as shown in Table B.

TableB Experimental results of combining different fallback and adaptive filtering strategies


| Metrics    | origin_lrm+$r_{rele}$ | origin_lrm+$r_{qual} $| origin_lrm+$r_{task}$ | gen_ref+$r_{rele}$ | gen_ref+$r_{qual}$ | gen_ref+$r_{task}$ | ReFIR-baseline |
|------------|----------------------|------------------|------------------|------------------|----------------------|----------------------|-----------------|
| NIQE       | 4.4982| 4.4943| 4.3891| 4.4978| 4.4923| 4.3464| 4.4986|
| MUSIQ      | 56.78| 57.12| 57.77| 56.95| 57.11| 57.68| 57.01           |
| CLIPIQA    | 0.6730| 0.6745| 0.6898|0.6743| 0.6770| 0.6942|   0.6759       |


It can be seen that using the SD generated image as the reference is better than not using a reference image, supporting the results in Table A. In addition, the task-oriented filtering strategy achieves a significant performance improvement due to the fact that it works in the output end, but is is accompanied by a larger inference time. Using SD-generated image as fallback and adopt the quality-based filtering act as a competitive alternative to the our preivous baseline.

Meta-review: The final ratings of this paper are Weak Accept, Weak Accept, Borderline Accept, and Borderline Accept. The paper introduces a training-free method to enhance Large Restoration Models (LRMs) by integrating high-quality, content-relevant images retrieved from external databases. This approach addresses the hallucination problem in diffusion-based LRMs by incorporating textures from retrieved images into the self-attention layers, which significantly improves restoration accuracy without additional training. Extensive experiments confirm ReFIR’s effectiveness across various LRMs, making it a promising solution for practical applications. However, the method’s reliance on the quality of retrieved images and potential computational overhead are noted as limitations, with further exploration needed into scalability, efficiency, and robustness. The authors’ rebuttal has addressed the majority of concerns from reviewers, resulting in an increase in rating.
Decision: Accept (poster)