InstantIR: Blind Image Restoration with Instant Generative Reference

## Abstract

Abstract Handling test-time unknown degradation is the major challenge in Blind Image Restoration (BIR), necessitating high model generalization. An effective strategy is to incorporate prior knowledge, either from human input or generative model. In this paper, we introduce Instant-reference Image Restoration ( InstantIR ), a novel diffusion-based BIR method which dynamically adjusts generation condition during inference. We first extract a compact representation of the input via a pre-trained vision encoder. At each generation step, this representation is used to decode current diffusion latent and instantiate it in the generative prior. The degraded image is then encoded with this reference, providing robust generation condition. We observe the variance of generative references fluctuate with degradation intensity, which we further leverage as an indicator for developing a sampling algorithm adaptive to input quality. Extensive experiments demonstrate InstantIR achieves state-of-the-art performance and offering outstanding visual quality. Through modulating generative references with textual description, InstantIR can restore extreme degradation and additionally feature creative restoration.

## 1 Introduction

Image restoration seeks to recover High-Quality (HQ) visual details from Low-Quality (LQ) images. This technology has a wide range of important applications. It can enhance social media contents to improve user experience Chao et al. (). It also functions at the heart in industries like autonomous driving Patil et al. () and robotics Porav et al. () by improving adaptability in diverse environments, as well as assists object detector in adverse conditions Sun et al. ().

Image restoration remains a long-standing challenge extending beyond its practical application. The information loss during degradation makes a single LQ image corresponds to multiple plausible restorations. This ill-posed problem is further exacerbated in Blind Image Restoration (BIR), where models are tested under unknown degradation. A common strategy is to leverage prior knowledge. Reference-IR models use other HQ images to modulate LQ features, requiring additional inputs with similar contents but richer visual details Lu et al. (). Generative approaches, on the other hand, directly learn the HQ image distribution. The input is first encoded into a hidden variables $z$, which servers as the generation condition to sample HQ image from the learned distribution $p(y|z)$. Although generative methods achieve single-image restoration, they are prone to hallucinations that produce artifacts in restoration Yang et al. (). This happens when the encoder fails to retrieve accurate hidden variable due to the input distribution shift in degradation. Existing methods improve robustness by training on more diverse synthetic degradation data or introduce discrete feature codebook. We argue that these are only shot-term solutions. Alternative methods are pendding to be explored to better address unknown inputs in BIR.

In this paper, we present InstantIR, a dynamic restoration pipeline that iteratively refines generation condition using a pre-trained Diffusion Probabilistic Model (DPM). InstantIR employs two complementary way for processing input LQ image. First, a pre-trained vision encoder extracts compact representation from degraded content. The encoder’s high compression rate enhances the robustness in the extracted representation, while retaining only high-level semantics and structural information. Next, we introduce the Previewer module, a distilled DPM capable of one-step generation. At each generation step, the previewer decodes current diffusion latent using the compact representation, providing a restoration preview resembles original input in high-level features. This preview serves as an instant generative reference to guide the Aggregator in encoding identity and other fine-grained missing from the compact representation. We observe in experiments that the previewer tends to decode aggressively when the input is clear, resulting in high variance in restoration previews. We take this as a reliable indicator of input image quality, and develop an adaptive sampling algorithm that amplifies the fine-grained encoding with relatively high quality inputs. Additionally, we find the previewer is controllable through text prompts, which produces diverse generative references and enables semantic editing with restoration.
Our contributions are as follows:

- 1.
We explore a novel BIR method that iteratively aligns with the generative prior to address unknown degradation;
- 2.
We introduce a novel architecture based on pre-trained DPM, which dynamically adjusts the generation condition by previewing intermediate outputs;
- 3.
We develop sampling algorithms tailored for our pipeline, enabling both adaptive and controllable restoration to text prompts;
- 4.
We perform extensive evaluations to validate the effectiveness of the proposed methods.

## 2 Related Work

### 2.1 Diffusion Model

DPM is a class of generative model that generate data by iteratively denoising from Gaussian noise Sohl-Dickstein et al. (); Ho et al. (); Song et al. (). Typically, a neural network with a UNet architecture Ronneberger et al. () is trained to predict the noise added at each inference step. DPM offers superior mode coverage compared to Variational Autoencoders (VAE) Kingma & Welling () and outperform GAN-based models Goodfellow et al. () in generation quality without the need of adversarial training Dhariwal & Nichol (). These advantages establish DPM as the leading approach in vision generative models. By incorporating additional inputs, DPMs can learn diverse conditional distributions Nichol & Dhariwal (), with the most widely used application being text-to-image (T2I) generation Rombach et al. (); Saharia et al. (); Ramesh et al. (). Leveraging the flexibility of text inputs and the vast amount of text-image training data Schuhmann et al. (), these models are capable of generating images with exceptional visual quality and remarkable diversity, forming the foundation for many subsequent excellent work in vision generative models Wang et al. (; ).

### 2.2 Blind Image Restoration

The task setting makes BIR particular valuable in real-world applications. The major challenge in BIR is the input distribution gap between training and testing data. Previous work have explored multiple ways to address this issue. Feature quantification is widely used in generative-based methods Esser et al. (); Van Den Oord et al. (); Zhou et al. (). They align the encoded LQ image features to a learnable feature codebook, ensuring the input to generator is unaffected by domain shifts. However, this hard alignment constraints the generation diversity and quality by the capacity of the discrete codebook. Previous work have also explored the application of powerful DPM in BIR. Some approaches design specialized architectures and train DPMs from scratch Saharia et al. (); Sahak et al. (); Li et al. (), while the others apply additional modules on pre-trained T2I model Wang et al. (); Yu et al. (); Sun et al. (), leveraging their large-scale prior. In many practical scenarios, HQ images with similar contents, such as those from photo albums or video frames, are available. This has spurred interest in restoring images using reference-based methods Cao et al. (); Jiang et al. (); Lu et al. (); Xia et al. (); Yang et al. (); Zhang et al. (). They adopt regression models to learn how to transfer high-quality features to LQ images, enhancing details restoration.

## 3 Methodology

Figure: Figure 2: The overall pipeline of InstantIR. InstantIR adopts a novel previewing mechanism to actively align with generative prior. This is achieved by three key modules: 1) DCP for compact LQ image representation encoding; 2) Previewer for decoding it into generative prior; and 3) Aggregator for integrating the generative reference and LQ input into sampling conditions.
Refer to caption: x2.png

The distribution gap between training and testing data exacerbates the ill-posed nature of BIR, causing hallucinations in generation-based IR models and producing artifacts. We attribute this to the error in encoding LQ image, and propose a generative restoration pipeline that refines the LQ encodings with generative references. This is achieved by exploiting the reverse process of DPM. Specifically, we first encode the LQ image into a compact representation via pre-trained vision encoder, capturing global structure and semantics to initiate diffusion generation. Conditioned on this embedding, our Previewer module generates a restoration preview at each diffusion time-step. The preview resembles to the input image with more plausible details, and they are further fused in the Aggregator module to preserve fidelity. Finally, the adjusted LQ encoding is used to control the pre-trained DPM for a fine-grained diffusion step.

### 3.1 Preliminaries

DPM involves two stochastic processes named forward and reverse process Ho et al. (). In the forward process, i.i.d. Gaussian noise is progressively added to the image $\bm{x}$. The marginal distribution of diffusion latent $\bm{x}_{t}$ follows $\mathcal{N}\left(\alpha_{t}\bm{x},\beta_{t}\bm{I}\right)$, where $\alpha_{t}$ and $\beta_{t}$ are hyperparameters defining the forward process. $\bm{x}_{t}$ converges to pure noise as $t$ increases, and the reverse process generates images by inverting the forward process. Generally, we train a neural-network to predict the noise added at each time-step by minimizing the diffusion loss:

$$ $\mathcal{L}_{diff}=\mathbb{E}\left[\lVert\bm{\epsilon}_{\theta}\left(\bm{x}_{t },t\right)-\bm{\epsilon}\rVert^{2}\right],$ (1) $$

where $\bm{\epsilon}_{\theta}$ denotes the noise-prediction network. At each step in the reverse process, we can retrieve a denoising sample with the predicted noise and re-parameterization Karras et al. ():

$$ $\bm{\hat{x}}=\frac{\bm{x}_{t}-\beta_{t}\bm{\epsilon}_{\theta}\left(\bm{x}_{t}, t\right)}{\alpha_{t}}.$ (2) $$

In the open-sourced T2I model Stable Diffusion (SD) Rombach et al. (), the noise-prediction network $\bm{\epsilon}_{\theta}$ is additionally conditioned on a text input that describes the target image. Moreover, SD employs a VAE to move the input $\bm{x}_{t}$ into latent space $\bm{z}_{t}$, compressing inputs by a factor of 48 and significantly reduces the memory usage to enable image generation up to $512^{2}$ resolution.

### 3.2 Architecture

The restoration pipeline of InstantIR consists of three key modules: Degradation Content Perceptor (DCP) for compact LQ image encoding, Instant Restoration Previewer for generating references on-the-fly during the reverse process, and Latent Aggregator for integrating restoration references.

#### Degradation Content Perceptor

We employ the pre-trained DINO Oquab et al. () for providing compact LQ image representation. Compared to CLIP Radford et al. (), a common choice in image editing Ye et al. (), DINO’s self-supervised training with data augmentation improves robustness of the encoded features. The extracted LQ representation is modulated by a learnable Resampler Han et al. () and projected as context to the cross-attention layers of diffusion UNet. For the $l$-th cross-attention block, we introduce an additional cross-attention operation:

$$ $\bm{f}_{out}^{l}=\bm{f}_{in}^{l}+\texttt{CrossAttn}\left(\bm{f}_{in}^{l},\bm{c }_{txt}\right)+w^{l}\cdot\texttt{CrossAttn}\left(\bm{f}_{in}^{l},\Phi\left(\bm {c}_{lq},t\right)\right),$ (3) $$

where $\Phi$ denotes the DCP module and $\bm{c}_{lq}$ is the LQ context matrix. We retain the text cross-attention here as it is a crucial part of the pre-trained T2I model that synthesizes high-level semantics. Jointly training DCP with textual transformation allows it to focus on low-level information absent in the other modality. We introduce a hyper-parameter $w^{l}$ to regulate their behaviors. Note that the DCP also takes time-step $t$ as input to establish temporal dependency in the output. Specifically, we use adaptive layer-normalization to modulate the context matrix from the DCP according to time-step $t$:

$$ $\Phi\left(\bm{x},t\right)=\bm{\mathcal{T}}_{scale}\odot\texttt{LayerNorm}\left (\bm{c}_{lq}\right)+\bm{\mathcal{T}}_{shift},$ (4) $$

where, $\bm{\mathcal{T}}_{scale},\bm{\mathcal{T}}_{shift}$ are calculated from the time-step. We train the DCP module on a frozen diffusion model using the standard diffusion loss in Eq. 1.

#### Instant Restoration Previewer

The compact representation encoded by the DCP, while robust against degradation, losses high-level information. We introduce Previewer, a diffusion model generates from current diffusion latent instead of noise, to decode generative references from the DCP encoding. Decoding at each diffusion time-step requires $\left(T\left(T+1\right)/2\right)$ network forward passes with the vanilla T2I model. To streamline this process, we fine-tune the Previewer using consistency distillation Luo et al. () to make it a one-step generator. For diffusion latent $\bm{z}_{s}$ at time-step $s$, we first obtain the Previewer output conditioned solely on $\bm{c}_{lq}$. Then, we perform a diffusion step using the pre-trained model from $\bm{z}_{s}$, conditioned on both $\bm{c}_{lq}$ and $\bm{c}_{txt}$, to reach $\bm{z}_{t}$. $\bm{z}_{t}$ is regarded as the ground-truth diffusion latent at time-step $t$ in the sampling trajectory. Finally, we get the preview of $\bm{z}_{t}$, again conditioned solely on $\bm{c}_{lq}$. The consistency distillation loss is then calculated by:

$$ $\mathcal{L}_{dist}=\lVert\Psi\left(\bm{z}_{s},s,\Phi\left(\bm{c}_{lq},s\right) \right)-\texttt{StopGrad}\left(\Psi\left(\bm{z}_{t},t,\Phi\left(\bm{c}_{lq},t \right)\right)\right)\rVert^{2},$ (5) $$

where $\Psi$ denotes the previewer model. Additionally, Eq. 5 trains the previewer to follow the sampling trajectory without $\bm{c}_{txt}$, removing its dependency on text conditions which are typically unavailable in BIR tasks. The consistency constraint Song et al. () of enforcing consistent outputs across time-step enabling the Previewer to decode generative references on-the-fly.

#### Latent Aggregator

The primary challenge in the BIR task is the input distribution shift. Previous work address this by aligning LQ features with reference HQ images or a learned feature codebook. The former takes extra inputs, while the latter is limited to a specific domain by the codebook capacity. In contrast, we generate reference features directly from diffusion prior. Since the compact embedding $\bm{c}_{lq}$ retains only high-level information, it is insufficient for the Previewer to reconstruct HQ images at larger time-steps, as shown in Fig. 6. Relying solely on reference preview incurs error accumulation, so the Aggregator anchors preview to the original input to prevent divergence in the reverse process. The input LQ image is encoded into SD’s latent space and spatially concatenated with the preview. This expanded input remains compatible to the diffusion UNet, allowing the Aggregator to be initialized as a trainable copy of UNet compression path following Zhang et al. (). We remove text cross-attention layers to make the Aggregator lightweight and independent of textual conditions like the Previewer. The preview and LQ hidden featrues are fused in the spatial-attention layers, which are further integrated via Spatial Feature Transform (SFT) Wang et al. (). For hidden feature $\bm{H}^{l}$ at the $l$-th layer in the Aggregator, we first split it spatially into $\bm{h}_{p}^{l}$ and $\bm{h}_{o}^{l}$, corresponding to the hidden features of preview and LQ latent, and integrate them with SFT:

$$ $\bm{h}^{l}_{res}=\left(1+\bm{\alpha}^{l}\right)\odot\bm{h}_{p}^{l}+\bm{\beta}^ {l};\bm{h}_{p}^{l},\bm{h}_{o}^{l}=\texttt{Split}\left(\bm{H}^{l}\right),$ (6) $$

where $\bm{\alpha}^{l},\bm{\beta}^{l}=\mathcal{M}_{\theta}^{l}(\bm{h}_{o}^{l})$ are two affine transformation parameters calculated from the feature map of LQ latent at this level. We extract multi-level features $\left\{\bm{h}_{res}^{l}\right\}_{l=1}^{L}$ from Aggregator using Eq. 6, and inject them into the corresponding part of U-Net expansion path through residual connections.

### 3.3 Adaptive Restoration

Figure: Figure 3: The evolution of the Previewer outputs during generation. (a) L2-distances between previews and denoising means; (b) temporal differences of the Previewer trajectory, measured by L2-distances between adjacent points; (c) relative distances between previews and denoising means.
Refer to caption: x3.png

InstantIR processes LQ image through two complementary ways: 1) extracting compact representation using the DCP, which is robust to degradation but loses fine-grained information; 2) encoding via the lossless SD-VAE and integrating with restoration preview, which is prone to errors in the SD-VAE. Under severe degradation, InstantIR may produce samples deviate from the target HQ image. In such cases, restoration previews exhibit small variation, suggesting the DCP struggles to provide guidance according to the input. We further analyze the trajectory of restoration previews during the reverse process, compare it with the denoising predictions from Eq. 2. We assess them on four degradation levels: HQ image, 4x downsampling, 8x downsampling and synthetic multi-degradation, representing decreasing input quality. Fig. 3 (a) illustrates the L2-distance between these two trajectories, which increases monotonically as input quality improves. A pronounced disparity between preview and ordinary denoising prediction represents the Previewer is confident with the guidance, suggesting the input LQ image is informative. Based on this observation, we use the relative difference between two predictions as an indicator of input quality:

$$ $\delta=\frac{\lVert\Psi\left(\bm{z}_{t},t,\Phi\left(\bm{c}_{lq},t\right)\right )-\bm{\hat{z}}_{t}\rVert^{2}}{\lVert\Psi\left(\bm{z}_{t},t,\Phi\left(\bm{c}_{ lq},t\right)\right)-\Psi\left(\bm{z}_{t+1},t+1,\Phi\left(\bm{c}_{lq},t+1\right )\right)\rVert^{2}},$ (7) $$

where $\bm{\hat{z}}_{t}$ is given by Eq. 2. From Fig. 3 (b) we can see the Previewer is unstable at the beginning. The consistency training in Eq. 5 drives it to decode aggressively, causing large prediction variance during early reverse process where the input diffusion latent is too noisy. Normalizing the L2-distance between trajectories with Previewer’s temporal difference effectively mitigates the temporal correlation as illustrated in Fig. 3 (c). A larger $\delta$ indicates higher input quality, and the conditional signals from the Aggregator should be amplified to preserve fine-grained information from the original input. On the other hand, DPM is known to first generate low-frequency features such as global structure, and add high-frequency details in the later stage of the reverse process. A decreasing $\delta$ encourages InstantIR to produce diverse by exploiting the LQ image representation. We provide pseudo-code of the proposed adaptive restoration (AdaRes) algorithm in Alg. 1.

Figure: Algorithm 1 Adaptive Restoration

Surprisingly, although only the DCP module is explicitly trained on text-image data, InstantIR demonstrates notable creativity following textual descriptions. By employing a text-guided Previewer, we can generate diverse restoration variations with compound semantics from both modalities. However, these variation samples can conflict with the original input, making them ineligible as generative references. Inspired by previous work in image editing, we disable the Aggregator at later stage generation and let InstantIR renders semantic details according to LQ representation and text prompt. This ensures the low-frequency features are succeeded from the Aggregator, meanwhile prevents the high-frequency semantics and noise from entering the final results.

## 4 Experiments

### 4.1 Implementation Details

InstantIR is built on SDXL Podell et al. () accompanied by a two-stage training strategy. In Stage-I, we train the DCP module on a frozen SDXL, followed by consistency distillation of the Previewer (see Sec. 3.2). The Previewer is trained by applying Low-Rank Adaptation (LoRA) Hu et al. () on the base SDXL model for efficiency. By toggling the Previewer LoRA, we can seamlessly switch between the Previewer and SDXL, reducing memory footprint. After obtaining the DCP and Previewer LoRA, we proceed to Stage-II Aggregator training. The two-stage training ensures the Aggregator receives high-quality previews since the beginning of its training course.

We adopt SDXL’s data preprocessing and conduct training on $1024^{2}$ resolution. In both two stages we use the AdamW Loshchilov () optimizer with a learning rate of $1\times 10^{-4}$. In Stage-I, we train the DCP module using a batch size of 256 over 200K steps, and distill the Previewer for another 30K steps with the same batch size. We train the Aggregator with a batch size of 96 over 200K steps in Stage-II. The entire training process spans approximately 9 days on 8 Nvidia H800 GPUs.

To enable Classifier-free Guidance (CFG) Ho & Salimans () sampling, we apply LQ image dropout with a probability of 15% in both stages training. In all test experiments, we employ 30 steps DDIM sampling Song et al. () with a CFG scale of $7.0$.

**Table 1: Quantitative comparisons on both synthetic validation data and public real-world dataset. We highlight the best results in bold and the second best with underline.**
| Dataset | Model | PSNR | SSIM | LPIPS | CLIPIQA | MANIQA | MUSIQ |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Synthetic | BSRGAN | 20.21 | 0.5214 | 0.7793 | 0.2072 | 0.2076 | 17.53 |
| Real-ESRGAN | 19.92 | 0.5317 | 0.7554 | 0.2102 | 0.2331 | 17.39 |  |
| StableSR | 20.42 | 0.5388 | 0.3751 | 0.4672 | 0.2602 | 52.33 |  |
| CoSeR | 19.92 | 0.5114 | 0.3353 | 0.6651 | 0.4152 | 67.51 |  |
| SUPIR | 20.46 | 0.4990 | 0.4090 | 0.4875 | 0.3081 | 56.43 |  |
| InstantIR (ours) | 18.54 | 0.5126 | 0.3986 | 0.5497 | 0.4379 | 68.59 |  |
| Real-world | BSRGAN | 26.38 | 0.7651 | 0.4120 | 0.3151 | 0.2147 | 28.58 |
| Real-ESRGAN | 27.29 | 0.7894 | 0.4173 | 0.2532 | 0.2398 | 25.66 |  |
| StableSR | 26.40 | 0.7721 | 0.2597 | 0.4501 | 0.2947 | 48.79 |  |
| CoSeR | 25.59 | 0.7402 | 0.2788 | 0.5809 | 0.3941 | 60.51 |  |
| SUPIR | 26.41 | 0.7358 | 0.3639 | 0.3869 | 0.2721 | 42.72 |  |
| InstantIR (ours) | 21.75 | 0.6766 | 0.3686 | 0.5401 | 0.4819 | 65.32 |  |

### 4.2 Experimental Configuration

#### Training Data

We synthesis LQ-HQ image pairs using Real-ESRGAN Wang et al. () with the default setting. As mentioned in Sec. 3.2, we conduct Stage-I training on the JourneyDB dataset Sun et al. (), a generated dataset with descriptive captions. While JourneyDB images are of extreme quality, they lack the textures in real-world images. Hence for Stage-II training, we incorporate publicly available texture-rich datasets to enhance model’s ability to produce realistic visual details. Specifically, we use DIV2K Agustsson & Timofte (), LSDIR Li et al. (), Flickr2K Timofte et al. () and FFHQ Karras et al. ().

#### Test Setting

For a comprehensive evaluation, we test InstantIR on a synthetic dataset and public benchmarks following previous work. We synthesize $2,000$ multi-degradation samples from DIV2K and LSDIR validation sets using Real-ESRGAN pipeline, filtering out images smaller than $1024^{2}$ to ensure ground-truth quality. We include a small portion of JourneyDB validation data to enhance benchmark diversity. We conduct evaluations on RealSR Cai et al. () and DRealSR Wei et al. () to assess model performance on real-world LQ images. We report full-reference metrics PSNR, SSIM, LPIPS Zhang et al. (), if ground-truth targets are available, and non-reference metrics MANIQA Yang et al. (), CLIPIQA Wang et al. (), MUSIQ Ke et al. () to quantitatively compare InstantIR with other models.

### 4.3 Comparing to Existing Methods

We compare InstantIR with state-of-the-art models, including StableSR Wang et al. (), CoSeR Sun et al. (), SUPIR Yu et al. (), BSRGAN Zhang et al. () and Real-ESRGAN Wang et al. (). Since some of them are limited to $512^{2}$ resolution, we consider two test scenarios for a fair comparison: 1) models are tested on $512^{2}$ images with outputs of 1024-models scaled accordingly; 2) following SUPIR, the models are tested on $1024^{2}$ images by cropping $512^{2}$ patch as inputs to 512-models, metrics are evaluated on the cropped area only.

#### Quantitative Comparison

The results are summarized in Tab. 1. InstantIR continuously achieves the highest MUSIQ and MANIQA scores across all test settings, outperfoming the second best by large margins up to 22% in MANIQA and 8% in MUSIQ. Notably in scenario 1, despite halving the input data, InstantIR still performs comparably to SOTA models. While CoSeR achieves the best CLIPIQA scores closely followed by InstantIR, restorations from 1024-models SUPIR and InstantIR are rich in details as shown in Fig. 4. We also observe the misalignment of PSNR and SSIM scores with visual quality as reported in the literature Yu et al. (); Wang et al. (). We include these metrics here for reference purpose.

Figure: Figure 4: Qualitative comparisons on real-world LQ images. Restorations from InstantIR are rich in details with global semantic consistency. Better viewed zoom in.
Refer to caption: x4.png

#### Qualitative Comparison

We provide some restoration samples on real-world LQ images in Fig. 4. Through leveraging the previewing mechanism, InstantIR actively aligns with generative prior, reducing hallucinations and producing sharp yet realistic details. In the second row of Fig. 4, while SUPIR’s result contains rich textures, the absence of global semantic guidance causes the diver’s body and mask to blend together. In contrast, the cognitive encoder in CoSeR helps it identifies statues in the second example. CoSeR employs a feature codebook to handle unknown degradations, which limits the generation of complex textures on the statues. Notably in the first row of Fig. 4, InstantIR is the only one that successfully recovers all four faces without distortion, suggesting its superior ability in capturing semantic and reproduce realistic details from diverse degradations.

Figure: (a) In-domain previews enhance detail restoration.
Refer to caption: x5.png

Figure: Figure 6: Visual examples of generative references. The first row on the right is generated by DCP trained on image-only data, while the second row is produced by DCP used in InstantIR, which is trained on text-image pairs.
Refer to caption: x7.png

### 4.4 Restoration with Previewing

#### In-domain Reference for Detail Enhancement

Reference-based BIR models improve detail restoration by transferring high-quality textures from HQ references. InstantIR achieves this by querying the T2I model, eliminating additional inputs. In Fig. 5(a), we disable the Previewer to see the effect of generative references. Here InstantIR infers solely with LQ images, which is beneficial to fidelity preservation but bad for visual quality. This is also reflected in Tab. 2(b) where all quantitative metrics deteriorate except PSNR and SSIM. Moreover, InstantIR equipped with Alg. 1 further improves the non-reference metrics, suggesting its flexibility in to different conditions.

#### Out-domain Reference for Creative Restoration

Fig. 5(b) shows more creative restoration samples. Owing to the efficiency of our Aggregator in integrating reference latents, InstantIR is able to perform high-level semantic editing during restoration, altering specific attributes of the subject and leaving other visual details like global structure and layout intact. We empirically find InstantIR offer better prompt-following ability under heavy degradation.

**Table 2: Ablation studies. The best results are highlighted in bold.**
|  | PSNR | SSIM | LPIPS | CLIPIQA | MANIQA | MUSIQ |
| --- | --- | --- | --- | --- | --- | --- |
| Baseline | 21.40 | 0.6775 | 0.3173 | 0.5433 | 0.4024 | 66.35 |
| +Distillation | 24.24 | 0.6963 | 0.4306 | 0.2453 | 0.2145 | 38.33 |
| +Noisy Previews | 23.07 | 0.7312 | 0.3830 | 0.3767 | 0.2924 | 49.23 |

### 4.5 Ablation Study

#### DCP Training on Text Domain

We compare training DCP module with and without textual condition. Due to limited computational resources, we did not proceed to train the subsequent Previewer and Aggregator for the DCP trained on image-only data. For comparison, we provide some visual examples of their generative references across diffusion time-steps in Fig. 6. As shown in the first row of Fig. 6, the generative references from the image-only DCP differ significantly from the input LQ image at early stage, retaining only coarse semantic like “a bird standing on a rocky surface.” In contrast, DCP trained with text descriptions preserves most of the low-level information, including global hue, structure, layout, and even the subject’s category (penguin) and its pose.

#### Previewer Consistency Distillation

We validate the necessity of consistency constraints in Previewer. We experiment with using predictions from Eq. 2 as reference inputs to the Aggregator. The second row in Tab. 2(b) shows a significant drop in the non-reference metrics. In fact, the prediction in Eq. 2 is close to the distribution mean at each time-step Karras et al. (). Previewer with consistency distillation can directly sample from the data distribution, providing more informative generative references.

#### Fresh Noise to Restoration Previews

We additionally train an Aggregator that injects fresh noise to reference latents according to diffusion time-step. The noisy preview latent follows the same distribution as current diffusion latent, making the overall pipeline resemble a ControlNet model Zhang et al. (). As shown in the third row of Tab. 2(b), InstantIR significantly outperforms ControlNet with LQ image as conditional inputs. This highlights the effectiveness of the previewing mechanism in InstantIR for adjusting generation conditions during inference.

## 5 Conclusion

In this paper, we explore a novel method to address unknown degradations in BIR tasks. Through exploiting the generation process of DPM, we propose to actively align with the generative prior to reduce the errors in encoding LQ image. Our pipeline is implemented based on pre-trained SDXL model, referred to as InstantIR. Extensive experiments demonstrate the exceptional restoration capability of InstantIR, delivering SOTA performance in quantitative metrics and visual quality. However, we observe some disparity in reference metrics such as PSNR and SSIM compared to SOTA models, which might because of the excessive generative prior diminishes fidelity. Future work will explore approach to improve the interaction between generative prior and conditions, as well as ways to refine the previewer to produce more reliable references.

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: 1. The details of Instant Restoration Previewer is not well written. The authors only know that the previewer is a diffusion model. The same things happened on aggregator and DCP. In other word, Fig. 2 need to be redrawn in a detailed way.
2. Do the authors train DINOv2 for DCP from scratch? Also how to distill the diffusion model fro Previewer from a pre-trained DINOv2? It is confusing as it is said in Sec. 4.1.
3. As a workflow of blind image restoration, it contains deblur, SR, denoise and dejpeg. What is the differences between all-in-one and it? Can it also be evaluated on the all-in-one setting, such as Perceive-IR, PromptIR or ProRes?
4. The performance of the proposed method is not superior than other compared methods. The authors should press more on the essential keys of this paper. 
5. The ability of editing LQ images seems to be derived from DiNOv2. The authors should provide more examples of editing scenario and discuss it in details.

Scores can be raised upon replies.
soundness: 3
strengths: Good theoretical analysis and well written manuscript. Some experimental results are used to prove the effectiveness of design.
summary: A good baseline for blind image restoration. Although the performance is not good, the editing ability is good in some real-world applications.
weaknesses: The performance on blind image restoration is not so good as other methods. The pipeline is simple and not novel.

---

confidence: 5
contribution: 3
initial_rating: 5
presentation: 2
questions: Please see the above Weaknesses.
soundness: 3
strengths: 1. The design of the proposed method seems to be reasonable. In fact, different images should be enhanced with different generation ability, previous works do not address this issue. Meanwhile, this work introduces a solution for this important issue, which is important.
2.  The flexibility of the proposed method is verified, which may contribute to the usage of the diffusion model for image restoration.
3.  The paper summarizes the advantages of INSTANTIR and points out possible future research directions, including improving the interaction between generated priors and conditions, which is important in this topic.
summary: The paper presents a novel approach to Blind Image Restoration (BIR) called Instant-reference Image Restoration (INSTANTIR), which addresses the challenge of model generalization under unknown degradation conditions. The authors propose a diffusion-based method that dynamically adjusts the generation conditions during inference by utilizing prior knowledge extracted from a pre-trained vision encoder.  Overall, the paper contributes a significant advancement in the field of image restoration by effectively integrating generative models with adaptive sampling strategies, demonstrating robust results across various degradation levels.
weaknesses: 1. The figure seems to be naive, I need to read the method carefully to get the key idea of the proposed method. Can authors provide a more detailed figure for both Sec.3.2 and Sec.3.3 to make it more clear?
2. In Sec.3.3, the authors claim "By employing a text-guided Previewer, we can generate diverse restoration variations with compound semantics from both modalities. However, these variation samples can conflict with the original input, making them ineligible as generative references." However, the authors do not provide visual analysis or references to support this opinion.
3. It seems the quality of the reference produced by the Previewer seems to be important for the final restoration effect, can authors provide deep analysis for the effect of the reference quality beyond simply visual results?
4. Since there are two diffusion models in the architecture, the computation cost and efficacy of the proposed method should be discussed.

---

confidence: 5
contribution: 2
initial_rating: 5
presentation: 2
questions: Please see weakness.
soundness: 2
strengths: It is not easy  to conclude what's the major strengths of this paper since most of the components are pretty straightforward compared with other paper. I will give authors a chance to answer what's the  Strengths from authors point of view.
summary: The paper INSTANT-IR addresses the challenge of handling unknown test-time degradation in Blind Image Restoration (BIR) using diffusion model 

The method extracts a compact representation of the input image using a pre-trained vision encoder. During each diffusion step, this representation decodes the current diffusion latent state, embedding it into the generative prior. The degraded image is encoded with this reference, which creates a robust generation condition. By observing that the variance of generative references changes with degradation intensity, the authors develop a sampling algorithm that adapts to input quality.

Experiments show that INSTANT-IR achieves state-of-the-art performance, providing excellent visual quality. By adjusting generative references with textual descriptions, it can handle extreme degradation and even perform creative restoration.
weaknesses: Here are some of the points I am not clear by reading this paper. 

1. Why DINO as the model to provide compact LQ image representation. From my understanding, DINO is a self-supervised learned representation that is very good for  semantic understanding. In a sense, the dino feature is highly compact and salient so that a lot of low-level cues are missing in the DINO feature. And Image Restoration is a pretty low-level task which need to pay more on the details rather than saliency.   That's why it doesn't make sense to me that DINO is a good choice here. 

2. If the Generative Reference is generated from LQ image representation, it is not clear what extra information that could bring in. In addiiton, in Line 211-212, what high-level information is missing in LQ and it is super un-clear. 

3. For the case with prompt, how did author train it ? And what if the prompt and LR images are totally different like a women LR image but with a man as prompt etc. Please discuss more in this part and it is totally missing in this paper. 

Overall, I didn't see any significant novelty in this paper since authors just use Dino feature and then decode it to be a reference image to guide the generation process.

---

confidence: 5
contribution: 2
initial_rating: 5
presentation: 3
questions: 1. Could you please explain why the proposed method is superior to previous ones, specifically referring to the perception-distortion tradeoff? (see weakness 1). Why is it appropriate to say that InstantIR is SOTA, given the perception-distortion tradeoff? What evidence in the paper convinces that the proposed approach is more effective than previous ones? For example, if previous methods were trained slightly differently (with different hyper-parameters), they may attain the same perceptual quality as InstantIR, but with better distortion (e.g., PSNR, SSIM).

2. How can we apply/train InstantIR in other image modalities besides natural images?
soundness: 2
strengths: 1. The paper is overall clear and well written.
2. The proposed method is, to my knowledge, novel.
3. Connection to related work is good.
4. Experiments are thorough, including a convincing ablation study.
summary: This paper proposes a new methodology for blind image restoration.
The main idea revolves around a new way to provide the degraded image as a condition to a pre-trained diffusion model (which is, to my knowledge, is typically done with a ControlNet).
Specifically, a pre-trained vision encoder first extracts a compact representation of the degraded image. This representation is combined with the diffusion state at each step, to form a preview of the generated image corresponding to that step (using a previewer). The previewed image is then used as the input condition for the diffusion model to produce the result in the next step.
weaknesses: 1. Compared to previous methods, the proposed method (InstantIR) seems to attain better perceptual quality, but this comes at the expense of worse distortion (e.g., PSNR, SSIM). Due to the perception-distortion tradeoff [1], this means that the proposed method cannot be claimed to be "SOTA", but just another point of the perception-distortion pareto frontier. Namely, to really be SOTA, a given algorithm needs to attain **better perceptual quality and better distortion simultaneously**, thus approaching the true perception-distortion tradeoff bound (dominating previous methods). Thus, it is not clear whether the proposed method is effective, because previous methods may also improve their perceptual quality at the expense of distortion (e.g., by tuning some hyper-parameters) and perhaps beat the proposed approach in terms of distortion at a given level of perceptual quality. Unfortunately, the authors do not discuss the perception-distortion tradeoff at all, which I believe is highly important given the fact that the proposed method's distortion is compromised in all data sets.
Note that the authors do briefly mention this limitation in the conclusion section (the compromised distortion compared to previous methods). But again, I believe that this should be thoroughly discussed, and it should be confirmed that the proposed approach is more effective than previous ones.

2. The conclusion/discussion section is quite short and limited. For example, a limitation I would specify is the design choice itself, which necessitates a degradation concept perceptor (e.g., DINO). What about other image modalities besides natural images (e.g., medical images, infra-red, compressed sensing), which DINO cannot handle? How can we then train InstantIR in such circumstances? Do we need to come up with a DINO-equivalent model to operate on images in other domains? My point is that InstantIR cannot be easily generalized/applied to other image modalities. I believe that this should be discussed, since many practitioners are working with other image modalities where blind image restoration is a crucial task.

3. While the proposed approach is novel, it is conceptually not that much different than previous methods (e.g., [2]). Specifically, the proposed method aims to generate samples from the posterior distribution (of natural images given a degraded image), which is an approach done in many previous works. Besides the empirical results, which, in my opinion, do not convince that the proposed approach is superior to previous ones (see point 1.), it is not clear why InstantIR offers any additional advantage over previous methods in terms of approximating the true sampler from the posterior.

4. The authors do not report NIQE scores for perceptual quality, which is standard in such works (e.g., [3]).

[1] Yochai Blau and Tomer Michaeli, The Perception-Distortion Tradeoff, CVPR 2018.

[2] Xinqi Lin et al., DiffBIR: Towards Blind Image Restoration with Generative Diffusion Prior, arXiv 2023

[3] Xintao Wang et al., Real-ESRGAN: Training Real-World Blind Super-Resolution with Pure Synthetic Data, ICCV 2021

Meta-review: This paper proposes InstantIR, a new method for blind image restoration (BIR) that uses a diffusion-based approach. The authors claim their method achieves state-of-the-art performance in BIR, offering high perceptual quality. They also claim their method is flexible and can be used to restore extremely degraded images. 

Strengths:
- The paper is generally well-written and easy to understand.  
- The method is able to restore extremely degraded images.  
- The method is flexible and can be used to generate a variety of outputs.  

Weaknesses:
- The performance of the method is not superior to other compared methods in terms of PSNR and SSIM.  
- The pipeline doesn't offer significant novelty, the contribution is very limited.  
- The authors do not provide enough examples of editing scenarios.  
- Discussion about important topics such as the perception-distortion tradeoff is very limited.

In its current presentation, the weakness overweight the strengths.  Despite revisions, some reviewers remain unconvinced of the method's practical value and impact.

Decision: Reject