Denoising as Adaptation: Noise-Space Domain Adaptation for Image Restoration
Authors: Kang Liao
Zongsheng Yue
Zhouxia Wang
Chen Change Loy
Venue: ICLR 2025 Poster (2025)
Keywords: Image Restoration
Domain Adaptation
Diffusion Loss
Primary area: generative models
TL;DR: We propose a novel domain adaptation method in the noise space for image restoration, guided by a diffusion loss that leverages auxiliary conditional inputs.

Abstract: Although learning-based image restoration methods have made significant progress, they still struggle with limited generalization to real-world scenarios due to the substantial domain gap caused by training on synthetic data. Existing methods address this issue by improving data synthesis pipelines, estimating degradation kernels, employing deep internal learning, and performing domain adaptation and regularization. Previous domain adaptation methods have sought to bridge the domain gap by learning domain-invariant knowledge in either feature or pixel space. However, these techniques often struggle to extend to low-level vision tasks within a stable and compact framework. In this paper, we show that it is possible to perform domain adaptation via the noise space using diffusion models. In particular, by leveraging the unique property of how auxiliary conditional inputs influence the multi-step denoising process, we derive a meaningful *diffusion loss* that guides the restoration model in progressively aligning both restored synthetic and real-world outputs with a target clean distribution. We refer to this method as *denoising as adaptation*. To prevent shortcuts during joint training, we present crucial strategies such as channel-shuffling layer and residual-swapping contrastive learning in the diffusion model. They implicitly blur the boundaries between conditioned synthetic and real data and prevent the reliance of the model on easily distinguishable features. Experimental results on three classical image restoration tasks, namely denoising, deblurring, and deraining, demonstrate the effectiveness of the proposed method.

Published as a conference paper at ICLR 2025
DENOISING AS ADAPTATION: NOISE-SPACE DOMAIN ADAPTATION FOR IMAGE RESTORATION
Kang Liao, Zongsheng Yue, Zhouxia Wang, Chen Change Loy
S-Lab, Nanyang Technological University (kang. 1iao, zongsheng... yue,, zhouxia. wang, ccloy) entu. edu.sg
https://kanglia0922 http:///ggiaiii....iiu..u.....ba htt.:u/b.eo/poojectssooooeeeeeee
ABSTRACT
Although learning- based image restoration methods have made significant progress, they still struggle with limited generalization to real-world scenarios due to the substantial domain gap caused by training on synthetic data. Existing methods address this issue by improving data synthesis pipelines, estimating degradation kernels, employing deep internal learning, and performing domain adaptation and regularization. Previous domain adaptation methods have sought to bridge the domain gap by learning innaaininvaariat knowledge in either feature or pixel space. However, these techniques often struggle extend to low-level vision tasks within a stable and compact framework. In this paper, we show that it is possible perform domain adaptation via the noise space using diffusion models. In particular, by leveraging the unique property of how auxiliary conditional inputs influence the multi-step denoising process, we derive meaningful diffusion loss that guides the restoration model in progressively aligning both restored synthetic and real-world outputs with a target clean distribution. We refer to this method as denoising as adaptation. To prevent shortcuts during joint training, we present crucial strategies such as channel-shuffling layer and residual-swapping contrastive learning in the diffusion model. They implicitly blur the boundaries between conditioned synthetic and real data and prevent the reliance of the model on easily distinguishable features. Experimental results on three classical image restoration tasks, namely denoising, deblurring, and deraining, demonstrate the effectiveness of the proposed method.
INTRODUCTION
Image restoration is a long-standing yet challenging problem in computer vision. It includes variety of sub-tasks, e.g., denoising (Zhang et al. 2017 Yue et al., 2024), deblurring (Pan et al., 2016 Ren et al. 2020), and deraining (Fu et al. 2017: Wang et al.) 2021). each of which has received research attention. Many methods are based on deep learning. typically following a supervised available in real-world contexts, i.e., degrada-
learning pipeline. Since annotated samples tion is unknown, a common technique is to generate synthetic low-quality data from high-quality images based on assumptions on the degradation process to obtain training pairs. This technique has achieved considerable success but is not perfect, as synthetic data cannot cover all unknown or unpredictable degradation factors, which can vary wildly due uncontrollable environmental conditions. Consequently, existing methods often struggle to generalize well to real-world scenarios.
Extensive studies have been conducted to address the lack of real-worke training data. Some restora- tion methods improve the data synthesis pipeline to generate more realistic degraded inputs for training (Zhang al., 2023) Luo et al., 2022). Other blind restoration approaches estimate the degradation kernel from the real degraded input during inference and use it as a conditional input to et al., 2019). Unsupervised methods (Lehtinen
guide the restoration (Gu et al. 2019
al. 2018 Shocher et al. 2018 Chen et al. 2023 Ren et al. 2020 Lee et al. 2022) enhance input quality without relying on predefined pairs of clean and degraded images. These methods often use deep internal learning self-supervised learning, where the model learns to predict clean images directly from the noisy or distorted data itself. In this paper, we investigate the problem assuming the existence of both synthetic data and real-world degraded images. This scenario fits typical

---

Published as a conference paper at ICLR 2025
Dittusion Process: Mix the input with Causstian noise
Prodkcton Etror bx1e S
Symtbviic Lahel
Systhatic Dhaged
Gi Naie
Restoration Networks
Dillusion Model
Lox
Diflaaion Loss
Prodecind Neiss
Coaditinn
Eupu
Rosl Duapee
Comuptton Condition
Gradient Gradanoe: The of conditions matters
(a) Diffusion Performance under Different Conditions
(b) Demoising a Adaptation
Figure 1: (a) The prediction error of diffusion model is highly dependent on the quality of the conditional inputs. In this experiment, we introduce an additional condition alongside the original noisy input. This condition the same target image but corrupted with additive white Gaussian noise at noise level a E [0, 80). More details can be found in the Appendix[AI]. (b) The restoration network is optimized to provide "good" conditions to minimize the diffusion model's Iss noise prediction enror, aiming for a clean target distribution.
domain adaptation setting, where existing methods can be categorized into feature-space ( Tzeng al., 2014 Ganin & Lempitsky, 2015
Long et al. 2015 Tzeng al., 2015 Bousmalis al. 2016)
and pixel-space (Taigman et al. 2016
Shrivastava et al. 2017 (oussalssss et al. (217)) approaches.
Both paradigms have their weaknesses: aligning high-level deep representations in feature space may overlook low-level variations essential for image restoration, while pixel-space approaches often involve computationally intensive adversarial paradigms that can lead to instability during training.
In this work, we present novel domain adaptation method for image restoration, which allows for a meaningful diffusion loss to mitigate the domain gap between synthetic and real-world degraded images. Our main idea stems from the observation shown in Fig. [T). Here, measure the noise prediction error of a diffusion model conditioned on a noisy version of the target image. The trend in Fig. a[) shows that conditions with fewer corruption levels facilitate lower prediction errors of the diffusion model. In other words, "good" conditions give low diffusion loss, and "bad" conditions lead to high diffusion loss. While such behavior may be expected, it reveals an interesting property of how conditional inputs could influence the prediction error of diffusion model. Our method leverages this phenomenon by taming a diffusion model conditioned on both the restored synthetic image and restored real image from the restoration network, as shown in Fig. [b). Both networks are jointly trained, with the restoration network optimized to provide good conditions to minimize the diffusion model's noise prediction error, aiming for a clean target distribution. Such goal drives the restoration network to learn to improve the quality of its outputs. After training, the diffusion model is discarded, leaving only the trained restoration network for inference.
While the multi-step denoising process aids the restoration network, a potential shortcut learning could arise: the diffusion model learns to recognize conditions based on their channel index or pixel similarity to noisy synthetic labels, thereby neglecting real data. To mitigate this issue, we propose crucial strategies to fool the diffusion model, making it hard to discriminate between these two conditions. Specifically, we incorporate a channel-shufling layer into the diffusion model and design residual-swapping contrastive learning strategy ensure the model genuinely learns to restore images accurately, rather than relying on easily distinguishable features. These strategies implicitly blur the boundaries between synthetic and real data, ensuring that both contribute effectively during joint training and facilitating their alignment with the target distribution.
To verify the effectiveness of our method, we conducted extensive experiments on three classi- cal image restoration tasks (denoising, deblurring, and deraining), showing promising restoration performance and scalability different networks. In summary, we make the following contributions:
Our work represents the first attempt addressing domain adaptation in the noise space for image restoration. We show the unique benefits from diffusson loss in eliminating the gap between the synthetic and real-world data, which cannot be achieved using existing losses.
To eliminate the shortcut learning in joint training, design strategies to fool the diffusion model, making it difficult to distinguish between synthetic and real conditions, thereby encouraging both align consistently with the target clean distribution.
2

---

Published as a conference paper at ICLR 2025
Our method offers general and flexible adaptation strategy applicable beyond specific restoration tasks. It requires no prior knowledge of noise distribution or degradation models and is compatible with various restoration networks. The diffusion model is discarded after training, incurring no extra computational cost during restoration inference.
RELATED WORK
Image Restoration aims to recover images degraded by factors like noise, blur, or data loss.
Driven
largely by the capabilities of various networks (Dong et al.,
2014
2015;
Zamir et al., 2022 Liang
ct al.
2021), significant advancements have been made in sub- fields such as image denoising Zhang
et al. 2021 Ren et al.]
2021: Guo et al.]
2019; Kim al.
2020
2024 Fu et al., 2023 Kousha
et al.
2022). image deblurring Kupyn et al.
2018:
Suin et al.
2020 Zhang et al.. 2019), and image
deraining (Jiang et al.
2020 Purohit al.
2021 Ren et al.
2019 Yang al.. 2017). In image
restoration, loss functions are essential for training models. For example, the LI loss minimizes average absolute pixel differences, ensuring pixel-wise accuracy. Perceptual loss uses pre-trained networks to compare high-level features, ensuring perceptual similarity. Adversarial loss involves a discriminator distinguishing between real and synthetic images, pushing the generator create more realistic outputs. However, the models trained on synthetic images with these conventional losses still cannot escape from a significant drop in performance when applied to real-world domains.
To address the mismatch between training and testing degradations, some supervised image restoration techniques (Zhang et al., 2023; Luo et al. 2022) improve the data synthesis pipeline, focusing on creating training degradation distribution that balances accuracy and generalization in real-world scenarios. Some methods (Gu et al., 2019 BeillKligler et al. 2019) estimate and correct the degradation kernels improve the restoration quality. Our work orthogonal to these methods, aiming to bridge the gap between training and testing degradations.
Unsupervised learning methods for image restoration leverage models that do not rely paired training samples (Huang et al.) 2021 Chen et al. 2023; Huo et al.] 2023; Chen et al.) 2024). Techniques like Noise2Noise (Lehtinen et al. 2018), Noise2 (Krull et al. 2019), and Deep Image Prior (Ulyanov et al., 2018) exploit the intrinsic properties of images, where the network learns to restore images by understanding the natural image statistics or by self-supervision. These approaches have proven effective in restoration tasks, achieving impressive results comparable to supervised learning methods. However, they often struggle with handling highly complex or corrupted images due to their reliance on learned distributions and intrinsic image properties, which may not fully capture intricate details and show limited generalization to other tasks.
Domain Adaptation. The concept of domain adaptation is proposed to eliminate the discrepancy between the source domains and target domains (Saenko et al. 2010 Torralba & Efros) 201 to facilitate the generalization ability of learning models. Previous methods can be categorized into feature-space and pixel-space approaches. For example, feature-space adaptation methods adjust the extracted features from networks to align across different domains. Among these methods, some classical techniques are developed like minimizing the distance between feature spaces ( (Tzeng let al. 2014 Long al.. 2015) and introducing domain adversarial objectives (Ganin & Lempitsky 2015 Tzeng et al., 2017 7). Aligning high levels of deep representation may overlook crucial low-
level variances that are essential for target tasks such as image restoration. In contrast, pixel-space
adaptation methods (Liu & Tuzel [2016] Taigman et al. 2016 Shrivastava et al. 2017 Bousmalis
ctt al. 2017) achieve distribution alignment directly the raw pixel level, by translating source data to match the "style" of a target domain. While they are easier to understand and verify for effectiveness
from domain-shifted visualizations, pixel-space adaptation methods require careful tuning and can be unstable during training. Recent methods (Hoffman et al. 2018)] Zheng et al. 2018 Chen et al. 2019) compensate for the limitation of isolated domain adaptation by jointly aligning feature space and pixel space. However, they tend to be computationally demanding due to the need to train multiple networks and the complexity of the cycle consistency loss (Zhu et al. 2017). Different from the above ceature-space and pixel-space methods, we propose new noise-space solution that preserves low-level appearance across different domains within a compact and stable framework.
Diffusion Model. Diffusion models (Sohl-Dickstedl et al. 2015 Ho et al., 2020 Nichol & Dhariwal
202I) have gained significant attention in generative modeling. They work by gradually transforming simple distribution into complex distribution in series of steps, reversing the diffusion process.
3

---

Published as a conference paper at ICLR 2025
This approach shows remarkable success in -ixteto- generation (Saharia et al.
2022b; Ruiz
eet al.] (2023) and image restoration (Saharia et al. 2022a c Yue et al. (003). Often, conditions are fed
to the diftusion model for conditional generation, such text (Rombach et al. 2022), class label (Ho
& Salimans 2022), visual prompt (Bar et al. 2022), and low-resolution image (Wang et al. 2024). to
facilitate the approximation of the target distribution. Some recent works propose to adapt di diffusion
models for image restoration and its related tasks such as blind JPEG restoration (Welker et al. 20224)
open-set image restoration (Gou et al. 2024), and classification of degraded images (Daultani et al. 2024). However, they require the diffusion model in both the training and inference stages. In this work, we show that the diffusion's forward denoising process has the potential to serve as a training proxy task to improve the generalization ability of the image restoration model.
3 METHODOL.OGY
Problem Definition. We start by formulating the problem of noise-space domain adaptation in the context of image restoration. Given a labeled datase (T from a synthetic domain and an unlabeled dataset from a real-world domain, we aim to train model on both the synthetic and real data that can generalize well to the real-world domain. Supposed that D = [(af. } denotes the labeled dataset containing N samples from the source synthetic domain and D = {œx J denotes the unlabeled dataset with N" samples from the target real-world domain, where y" is the clean image, x" is the corresponding synthetic degraded image, and x" is the real-world degraded image.
Image Restoration Baseline. The restoration network can be formulated as deep neural network G(:;0) with learnable parameter 0g. This network is trained to prediet the ground truth y from its degraded observation æ" on the synthetic domain. Our domain adaptation is not limited to a specific type of network architecture. One can choose from existing networks such as DrCNN (Zhang let al. 2017). U-Net (Yue et al., 2019). RCAN (Zhang et al. 2018b), and SwinIR (Liang et al. (021). The approach also orthogonal to existing loss functions used in image restoration, e.g., L1 or L2 loss, Charbonnier loss (Zamir et al. 2021), perceptual loss (Johnson et al.) 2016), and adversarial loss (Wang et al., 2018). To better validate the generality of the proposed approach, we adopt the widely used U-Net architecture f-(-) and the Charbonnier loss LRs. as our baseline. In the joint training. the diffusion model is trained using a diffusion objective, Lpy.. while the restoration network is updated using both the Lis and L The diffusion model is discarded after training.
3.1 NOISE-SPACE DOMAIN ADAPTATION
Ideally, the ground truth images and those restored images by an image restoration model from both synthetic and real-world data should lie in a shared distribution of high-quality clean images. However, attaining such an ideal model that can universally map any degraded images onto the distribution, is exceedingly challenging. Suppose a high-quality image æ as a realization derives from a random vector X, which belongs to the clean distribution Px. We then define the restored synthetic and real-world outputs from the restoration network as X and Xr. In this work, we investigate developing meaningful diffusion loss guide the conditional distributions of both synthetic and real-world outputs aligned to the target clean distribution, i.e., Px = Px- = P
Given the commonly adopted case where the ground truth images from the synthetic dataset available, we first explore adapting the target clean distribution with a perspective of paired data. Without loss of generality, let consider synthetic degraded image x with its ground truth y from the synthetic domain and a real degraded image x" from the real-world domain. Using the restoration network G(-; 0G). we can obtain the restored images ÿ" and ÿ", respectively. Then, based on our observation that the predicted eтror of a diffusion model is highly dependent on the quality of the conditional inputs, we incorporate a multi-step denoising process as a proxy task into the training process. It employs the predicted images ÿ¹ and zj" as conditions to help the diffusion model fit the clean distribution. Following the notations in DDPM (Ho et al. 22000), we denote the diffusion model as 6g and formulate its optimization to the following objective:
Lmi E ||e En (j"IC(j" ()),
(1)
where ÿJ" Vary" + /1 ā,e, N I), ā. is the hyper-paramees of the noise schedule, and C(.,)) denotes the channel-wise concatenation. During the joint training, supervision from the
Following the notations in domain adaptation, we use "lahel" represent the ground truth image.
4

---

Published as a conference paper at ICLR 2025
Iaput
Restered
PSNR
Synthetic Data
Epuch
PSNK
Real Data
Epoch 10
30
100 200
400
Epoch
Figure 2: During the joint training. the restored synthetic images smoothly converge the expected distribution over the epochs. However, the model tends find a shortcut real data by matching similarity between the conditions and the paired clean image or remembering the channel index. Consequently, the restoration network learns to corrupt the high frequeny details in real-world images and the diffusion model tends to ignore them.
diffusion loss will pack-propagate the conditions ÿ and D" if they under-restored, iee,, far away from the expected distribution. This encourages the restoration network to align ý² and ø" as closely possible to the target domain. Partieularly, the knowledge "leaked" by the diffusion's 's input plays an important role, potentially offering degradation-free guidance to help adapt the degraded real-world images into the clean distribution. More discussions presented in Section A5.2
The joint training, however, could lead to trivial solutions or shorteuts, as shown in Fig. For example, easy to distinguish the synthetic and real-world conditions by the pixel similarity between D and " or the channel index. Consequently, the restoration network will cheat the diffusion network by roughly degrading the gegh-frequency information in real-world images. As illustrated in Fig. (bottom) we identify three stages in this training process: (I) Diffusion network struggles to recognize which conditions aid denoising as both are heavily degraded, promoting the restoration network to enhance both; (II) Synthetic image is clearly restored and is easy to discriminate from its appearance; (III) The diffusion model tends to distinguish between the conditions, leading to focus on the synthetic data while ignoring the real- world data.
3.2 ELIMINATING SHORTCUT LEARNING IN DIFFUSION
To avoid the above shortcut in the diffusion model, as shown in Fig. 3, we first propose channel shufiling layer fes to randomly shuffle the channel index of synthetic and real-world conditions each iteration before con- catenating them, i.e., C g²) "²) We show in the experiments that this strategy important bridge the gap between synthetic and real data.
Sample
Condittonss
Pained Nocay GT
Attract
Repal
Notse GT
In addition channel shuffling, devise residual- swapping contrastive learning strategy to ensure the net- work learns restore genuinely instead of overfitting the paired synthetic appearance. Using the ground truth noise as the anchor, we construct positive example no de- rived from Eq. [] goos en (j* C(i", .")), i.e., the expected noise from the diffusion model conditioned on restored synthetic and real-world images. We then swap the residual maps of these two conditions and formulate a negative example e neg as follows:
Negalive Sumple
Connitoooe
Legena
X
Residual Stoppping
Stuffling Layer
Figure 3: The proposed solution to eliminate the shortcut learning in diffusion.
neg Eg (j" CCj ".".""""""" ))t)), 1
x
+ R", "="== x" + R*,
where Rs and R are the estimated residual maps of the synthetic and real- world images from the restoration network, and t is the pixel-wise addition operator. By swapping the residual of two conditions, we constrain the diffusion model to repel the distance between the wrong restored results
We omit the shuffling operator fce for notation clarity in the following presentation.
5

---

Published as a conference paper at ICLR 2025
and the expected clean distribution regardless of their context relation. Based on the positive, negative, and anchor examples, a compact residual-swapping contrastive learning can be formulated as:
Cc max (Ile - || ||E <n=0/1 ||2 + 6,0)
(3)
where & denotes a predefined margin to separate the positive and negative samples. In this way, the loss of diffusion model takes the mean of Eq. T and Eq. 3 forming the final diffusion loss DDj Using the above strategies, we challenge the diffusion model to distinguish between synthetic and real conditions based on trivial solutions, encouraging both to align with the target clean distribution.
In the above formulation, the restored synthetic image of the condition, denoted as ý", and the input to the diffusion model, represented ÿ², form pair of data with evident pixel-wise similarity. This similarity can potentially mislead the diffusion model to ignore the real restored image y" in condition as analyzed in Fig. [ It is important note that the target distribution encapsulates the domain knowledge of high-quality clean images, including but not limited to the ground truth images in the synthetic dataset. Motivated by this observation, the proposed method can be further extended by replacing the noisy input ÿ with y". defined as ÿ" ya,y" + vI â,c, where y is randomly sampled from an unpaired extensive high-quality image dataset. This strategy disrupts the pixel-wise similarity between the synthetic condition and the diffusion input, thus enforcing the diffusion model to guide both the synthetic and real conditions predicted by the restoration network at the domain level. We will provide ablation on this setting in Appendix A4.1
3.3 TRAINING
In the proposed training strategy, the image restoration model is jointly optimized by:
L Lites + ADif
Following previous works (Ganin & Lempitsky 2015), we gradually change ADIF from 0 to 3 to avoid distractions for the main image restoration task during the early stages of the training process:
ADIF = ( 1 cxp(-7.)))) -1)
(5)
where y and 3 are empirically set to 5 and 0.2 in all experiments, respectively. And p min (7+ where n denotes the current epoch index and N represents the total number of training epochs.
1),
3.4 DISCUSSION
The proposed denoising adaption is reminiscent of the domain adversarial objective proposed by (Ganin & Lempitsky 2015). The main difference is that we do not use a domain classifier with gradient reversal layer but diffusion network for the loss. We categorize methods like (Ganin & Lempitsky] 2015) as feature-space domain adaptation approaches. Unlike these approaches, we show that denoising as adaptation is more well-suited for image restoration as can better preserve low-level appearance in the pixel-wise noise space. Compared to pixel-space approaches that usually require multiple generator and discriminator networks, our method adopts compact framework incorporating only single additional denoising U-Net, ensuring stable adaptation training. After training, the diffusion network is discarded, requiring only the restoration network for inference. The framework comparison of the above three types of methods is presented in Appendix A2
4 EXPERIMENTS
Dataset. For image denoising, we follow previous works (Zhang et al. 2018a 8a Zamir et al. 2022) and construct the synthetic training dataset based on DIV2K ( (Timotte et al. (017]) Flickr2K (Nah et al. 2019), WED (Ma et 2016), and BSD (Martin et al. 2001). The noisy images are obtained by adding the additive white Gaussian noise (AWGN) of noise level o E [0, 75] to the source clean images. We use the training dataset of SIDD (Abdelhamed et al. 2018) as the real-world data. For image deraining, the synthetic and real-world training datasets are respectively obtained from Rain13K (Yang et al. (017)) and SPA (Wang et al.) 2019). For image deblurring, GoPro (Nah et al.) 2017) and RealBlur-J (Rim et al., (020)) are selected as the synthetic and real- wol-world training datasets, respectively. Please note that we only use the degraded images from these real- world datasets (without

---

Published as a conference paper at ICLR 2025
hppl
CoCADA
NetNe
G
GR
GR
GR
GR
GRD
GR
GR
GR
GR
GR
DR
DR
DR
PR
PR
DR
PR
DR
DR
GR
GR
GR
GR
GR
GR
GR
GR
GR
GR
Figure 4: Visual comparison of the image denoising task on SIDD test dataset (Ahdelhamed ct al. 2018). PSNR (dB) is marked for each comparison sample.
Table Quantitative evaluation of the image denoising task on SIDD test dataset. syn, real, both denote the model is trained on synthetic, real- world (w/o GT), and both synthetic and real-world (w/o GT) datasets, respectively. CRe.. Laan. Lovs, and LDs denotes the pixel-wise restoration loss (i.e. Charbonnier loss), generative adversarial loss, original loss exploited in the paper, and the proposed diffusion loss, respectively.
Metries
Vanilla
DANN
DSN
PixelDA
CyCADA
Ne2Ne MaskedD
Ours
Space
Feature
Feature
Pixel
Feature&Pixel
Noise
Train Data
syn
both
both
both
both
real
real
both
Train Loss
LREx
CRextGGan Laan
CReatGGan
Loss
Coss
DRect
PSNR
26.58
30.09
28.40
29.24
30.81
25.61
28.51
34.71
SSIM
0.6132
0.7832
0.6984
0.7611
0.8067
0,5647
0.7196
0.9202
LPIPS J
0.3171
0.1348
0.2265
0.1403
0.1256
0.3039
0.2348
0.0903
the ground truth) for training purposes. For large-scale unpaired clean images, all images in the MS-COCO dataset (Lin al. 2014) are used. The test images of the real-world datasets (SIDD, SPA, RealBlur-J) are employed to evaluate the performance of the corresponding image restoration models.
Training Settings. To train the diffusion model, we adopt a conditioning and the linear noise schedule ranging from 1e-6 to 1e-2 following previous works (Saharia al.) 2022a Chen et al. 2020). Moreover, the EMA strategy with a decaying factor of 0.9999 is also used across our experiments. Both the restoration and diffusion networks with random cropping and rotation for data augmentation. Our model is trained with a fixed learning rate 5e-5 using Adam (Kingma & Ba 2014) algorithm and the batch size is set to 40.
trained on 128 x 128 patches, which are processed
Metrics. The performance of various methods is mainly evaluated using the classical metrics: PSNR, SSIM, and LPIPS. For the image deraining, calculate PSNR/SSIM using the Y channel in YCbCr color space following existing methods (Jiang et al. 2020
Purohit et al. 2021 Zamir et al. 2022).
4.1 COMPARISONS WITH STATE-OF-TII-AAR METHODS
We implement the image restoration network using handy and classical U-Net architecture, which trained with the proposed noise-space domain adaptation strategy. To validate its effectiveness, compare the proposed method with previous domain adaptation approaches, including DANN (Ganin Lempitsky 2015), DSN (Bousmalis al., 2016), PixeIDA (Bousmalis et al., 2017). and Cy- CADA (Hoffman al. 2018), covering the feature-space and pixel-space solutions. For the purpose of a fair comparison, we retrained these methods with the same standard settings and datasets. Besides, also consider some unsupervised restoration methods and representative supervised methods such as Ne2Ne (Huang et al. 2021). MaskedD (Chen et al. 2023). NLCL (Ye et al. 2022 SelfDeblur (Ren al. 2020), VDIP (Huo al. 2023), and Restormer (Zamir et 2022).
Comparison Results. The quantitative and qualitative comparison results are shown Tab. and Fig. 4 From the comparison results, the proposed method leads the comparison methods three image restoration tasks. In particular, previous fcature-space domain adaptation methods (Ganin & Bousmalis al. 2016; Hoffman al. 2018) fail to perceive the crucial low-level
Lempitsky 2015:
7

---

Published as a conference paper at ICLR 2025
Table 2: Quantitative evaluation of the image deraining task SPA test dataset Wang al
2019
Metrics
Vanilla
DANN
DSN
PixelDA
CyCADA
NLCLI
Restormer
Ours
Space
Feature
Feature
Pixel
Featuro&PPxel
Noise
Train Data
syn
both
both
both
both
real
syw
both
PSNR f SSIM LPIPS
33.04
32.21
33,56
30,20
32.21
20.68
34.17
34.39
0.9540
0.9443
0,9552
0.9288
0,9442
0.8412
0.9492
0.9571
0.0477
0.0597
0.0512
0.0758
0.0597
0.0967
0.0488
0.0462
Table 3: Quantitative evaluation the image dehlurring task on RealBlur-J test dataset (Rim
2020
Metries
Vanilla
DANN
DSN
PixeIDA
CyCADA
SelfDeblur
VDIP
Ours
Space
Feature
Feature
Pixel
Feature&Pixel
Noise
Train Data
xyи
both
both
both
both
real
real
both
PSNR 1
26.27
26.11
26.28
24.71
26.36
23.23
24.89
26.46
SSIM LPIPS
0.8012
0.7945
0.8003
0.7646
0,7936
0.6699
0.7404
0.8048
0.1389
0.1345
0.1380
0.1583
0.1340
0,1340
0.1589
0.1363
information and pixel-space domain adaptation methods (Bousmalis et al. 2017) |offman et al. 2018)
yield inferior results since the precise style transfer between two domains is hard to control during the adversarial training. Moreover, the self-supervised and unsupervised restoration methods (Huang
al., 2021 Chen et al. 2023 Ye et al., 2022 Huo et al. 2023) show noticeable artifacts and limited generalization performance due to some inevitable information loss and hand-crafted designs on
specific degradations. By contrast, our method ensures a general and fine domain adaptation in the
pixel-wise noise space across various tasks, without introducing unstable training.
Analysis. From the above results, we can observe that the proposed method enables noticeable improvements beyond the baseline on the tasks involved with feghhfreyuency noises, such as image denoising. In particular, +-8. 13/0.3070 gains on PSNR/SSIM are achieved. We argue that the target of image denoising naturally fits that of the forward denoising process in the diffusion model. It is more sensitive to other Gaussian-like noises in the pre-sampled noise space. Thus, an intense diffusion loss would be back-propagated if the conditioned images under-restored, and the restoration network tries to eliminate the noises on both the synthetic and real-world images as much as possible.
4.2 ABLATION STUDIES
We conduct ablation studies regarding the sampled noise levels of the diffusion model, determined by the time-step f, and the training strategies to avoid shortcut learning, as shown in Tab, 4 and Fig. Concretely, with low noise intensity, e.g., [1. 100], it is easy for the diffusion model to discriminate the similarity of paired synthetic data even when the restored conditions are under-restored. As a result, the shorteut learning occurs earlier during the training process and the real-world degraded image is heavily corrupted by the restoration network, of which most details are filtered. On the other hand, when the intensity of the sampled noise high, e.g. t [900, 1000], the diffusion model is hard to converge and the whole framework has fallen into a local optimum. By sampling the noise from a more diverse range with E [1, 1000]. the restored results can be gradually adapted to the clean distribution. Moreover, the generalization ability of the restoration network gains further improvement using the designed channel shuffling layer (CS) and sssidua-swaaping contrastive learning strategy (RS), which effectively eliminates the shortcut learning of the diffusion model. Therefore, higher restoration performance on real-world images and more realistic visual appearance can be observed from (d) to (c) and (f) in Tab. and Fig. 6 We also demonstrate that both synthetic data and real data are indispensable for our domain adaptation, excluding each of them would lead to dramatic degradation in real-world performance (shown in the last two in Tab. 44.. Particularly for excluding the real data, the performance is almost degraded to that of the Vanilla model.
4.3 SCALABILITY
Comparisons. In this work, aim present general domain adaptation strategy for various restoration tasks, which is scalable to any restoration network. In particular, a basic and lightweight U-Net is used to validate the effectiveness of our method. However, such architecture essentially
8

---

Published as a conference paper at ICLR 2025
Input
DANN
DSN
PiseIDA
Input
Vanilla
DANN
KS
PaxeIDA
CyCADA
y blookks
CyCADA
NLCL
Restermer
Ours
KS
KS
KS
KS
SeltD
VIEP
Our
Figure Visual comparison of image deraining and image dehlurring tasks on SPA Wang et al. 2019 and RealBlur-J Rim ct al. 2020) test datasets. PSNR (dB) is marked for each comparison sample.
Tnput
GT
Noise Sampling Range
Strategy CS RS
Metries
Exp.
[1. 100] 1900, 1000] [I, 1000)
PSNR1
SSIM
(a)
26.58
0.6132
(b)
16.77
0.6070
(c)
27.36
0.6590
(d)
32.07
0.8706
(c)
(d)
(f)
(c)
32.91
0.9082
(1) (Ouns)
V
<
34.71
0.9202
(Only syn)
26.83
0.6286
(Only reaf)
32.60
0.8831
Table 4: Ablation studies of variant networks the SIDD test image denoising dataset. CS and RS represent the proposed channel shuffligg layer and residual-swapping contrastive learning strategies, respectively.
Figure Visual comparison results of ablation studies. The complete version of the proposed method achieves the best restoration results with visually pleasant appcarances.
limits the upper bound of the restoration performance compared to some recent self-supervised works (Jang al. 2021 Lee et 2022; Jang al., 2024] Cai al. 202I) tailored specific tasks.
Here, we provide experiments to demonstrate higher performance can be achieved using advanced restoration networks with the proposed adaptation strategy. The comparison results are shown in Table[5] In this experiment, we employ a restoration network based on U-Net architecture with deeper layers (named Ours*, the complexity details of different restoration networks are listed in Table A2 of Appendix). The results demonstrate that denoising performance on the SIDD test set has been improved from 34.71 dB to 35.52 dB. Moreover, we show the proposed method can generalize well to other unseen real-world datasets in Fig. | These datasets not encountered during the network's training and fall outside the distribution the trained datasets.
We believe more powerful restoration networks extraordinary performance for specific tasks is not the goal of this work.
enable further improvements, but pursuing
Discussion. Compared to the self sule--upe methods (Chen al. 2023 Ren et al. 2020 Huo et al. Jang
2023 Jang et al. 2021 Lee et al., 2022), our work shows the following unique strengths: it is not bounded to the specific tasks; it free the prior knowledge of underlying noise distribution and degradation mode; and is friendly to the type of preceding restoration networks. We also argue the difference between domain adaptation and self-supervised learning methods: Domain adaptation transfers knowledge from one domain to another with different distributions, improving performance in new, unseen environments. Self-supervised learning, on the other hand, learns from unlabeled data by generating pseudo-labed or exploring the target distribution from the data itself. Both approaches reduce the reliance on large labeled data but address different challenges: domain adaptation focuses on bridging domain gaps, and self-supervise learning leverages data's 's inherent structure.
Performance vs. Complexity. We validate the scalability of the proposed method using different variants of U-Net-based restoration networks and other types of architectures, such as the Transformer- based network (Wang et al. 2022). In particular, we classify these networks based on their model sizes and obtain: Unet-T, Unet-S (the model applied in Sec. I.I. Unet-B, Uformer-T, Uformer-S, and Uformer-B. B. More details are listed in the Appendix. The quantitative results vs. computational costs are shown in Fig. As
observe, the complexity increases, the vanilla restoration network (orange elements) tends to overfit the training synthetic dataset and perform worse on the test real-world dataset. In contrast, the proposed method improve the generalization ability of

---

Published as a conference paper at ICLR 2025
Metrics
C2N
AP-BSNI
Ours
Ours*
Ulomenn B
Space
Noise
Noise
34
Type
sS
SS
DA
DA
PENN (cl))
U'oemer-S
Train Data
both
real
both
both
Učsemer
30
Uttaeee
PSNR f
35.35
34,90
34.71
35.52
UnetTT
SSIM f
0.9370
0.9000
0.9202
0.9297
Unat-S Urat
26
Vanilla
Table 5: Ours* denotes using more advanced restoration net- work with deeper layers, trained by our domain adaptation strat- egy. and DA represent self-supervised and domain adap- tation methods, respectively. The asymmetric pixel-shuflle downsampling for the blind-spot network is exploited.
Vanillatuur
50
100
GMACs
Figure 7: Scalability of the proposed method different network architectures.
POST
POST
Figure 8: Visual results of the proposed method unseen real-world datasets: the denoising test dataset DND (Plotz & Roth 2017) and deraining test dataset 'Real-Intereet" Yang 2017).
restoration models with various sizes (blue elements). It is also interesting that for each type of architecture, our method can facilitate better performance as the complexity of the restoration network increases, demonstrating its effectiveness addressing the overfitting problem of large models.
4.4 LIMITATION
The natural mission of the diffusion model is to predict the noises mixed the input, which are sampled from figh-frequency distribution. Diffusion models excel at capturing and modeling these small-scale variations due to their ability to learn fine-gres ine-grained details through their denoising process. Thus, more intuitive improvements can be observed in image denoising and deraining tasks, which typically involve sigh-frequency noises in images. By contrast, artifacts in blurred images, which consist of smooth, gradual changes in intensity, can be less sensitive for diffusion models. They affect larger regions of the image and require the model to correct broad, sweeping distortions rather than fine details. Consequently, diffusion models may struggle fully restore images with low-frequency noise compared to those with high-frequency noise. We leave it as one of the future work.
CONCLUSION
In this work, have presented a novel approach that harnesses the diffusion model proxy network to address the domain adaptation issues in image restoration tasks. Different from previous feature- space and pixel-space adaptation approaches, the proposed method adapts the restored results the target clean distribution in the pixel-wise noise space, resulting in significant low-level appearance improvements within a compact and stable training framework. To mitigate the shorteut issue arising from the joint training of the restoration and diffusion models, we randomly shuffle the channel index of two conditions and propose a residual-swapping contrastive learning strategy to prevent the model from discriminating the conditions based on the paired similarity. Furthermore, the proposed method can be extended by relaxing the input constraint the diffusson model, introducing diverse unpaired clean images as denoising input. Experimental results demonstrate the effectiveness of our approach over saauue--ppaee and pixel-space domain adaptation methods, as well as its scalability surpassing that of self-supervised methods across range of image restoration tasks.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENT
This study is supported under the RIE2020 Industry Alignment Fund- - Industryy Collaboration Projects (IAF-ICP) Funding Initiative, well cash and in-kind contribution from the industry partner(s).
REFERENCES
AbdeIrahman Abdelhamed, Stephen Lin, and Michael S Brown. A high-quality denoising dataset for smartphone cameras. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattem Recognition, 2018.
Amir Bar, Yossi Gandelsman, Trevor Darrell, Amir Globerson, and Alexei Efros. Visual prompting via image inpainting. Advances Neural Information Processing Systems, 2022.
Sefi Bell-Kligler, Assaf Shocher, and Michal Irani. Blind super-resolution kernel estimation using an internal-gan. Advances in Neural Information Processing Systems, 2019.
Konstantinos Bousmalis, George Trigeorgis, Nathan Silberman, Dilip Krishnan, and Dumitru Erhan. Domain separation networks. Advances in Neural Information Processing Systems, 2016.
Konstantinos Bousmalis, Nathan Silberman, David Dohan, Dumitru Erhan, and Dilip Krishnan. Unsupervised pixel-level domain adaptation with generative adversarial networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2017.
Yuanhao Cai, Xiaowan Hu, Haoqian Wang, Yulun Zhang, Hanspeter Pfister, and Donglai Wei. Learn- ing to generate realistic noisy images via pixel-level noise-aware adversarial training. Advances in Neural Information Processing Systems, 34:3259-3213 2021.
Haoyu Chen, Jinjin Gu, Yihao Liu, Salma Abdel Magid, Chao Dong, Qiong Wang, Hanspeter Pfister, and Lei Zhu. Masked image training for generalizable deep image denoising. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, 2023.
Lufei Chen, Xiangpeng Tian, Shuhua Xiong, Yinjie Lei, and Chao Ren. Unsupervised blind image deblurring based on self-enhancement, In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024.
Nanxin Chen, Yu Zhang, Heiga Zen, Ron J Weiss, Mohammad Norouzi, and William Chan. Wavegrad: Estimating gradients for waveform generation. arXiv preprint 2=Xiv:2009.007/3 2020.
Yuhua Chen, Wen Li, Xiaoran Chen, and Lue Van Gool. Learning semantic segmentation from synthetic data: A geometrically guided input-output adaptation approach. In Proceedings of the IEFF/CVF Conference on Computer Vision and Puttern Recognition, 2019.
Dinesh Daultani, Masayuki Tanaka, Masatoshi Okutomi, and Kazuki Endo. Diffusion-based adapta- tion for classification of unknown degraded images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5982-5991, 2024.
Chao Dong, Chen Change Loy, Kaiming He, and Xiaoou Tang. Learning deep convolutional network for image super-resolution. In Proceedings of the European Conference on Computer Vision, 2014.
Chao Dong, Chen Change Loy, Kaiming He, and Xiaoou Tang. Image super-resolution using deep convolutional networks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 38(2): 295-307, 2015.
Xueyang Fu, Jiabin Huang, Delu Zeng, Yue Huang, Xinghao Ding, and John Paisley, Removing rain from single images via deep detail network. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017.
Zixuan Fu, Lanqing Guo, and Bihan Wen. srgb real noise synthesizing with neighboring correlation- aware noise model. In Proceedings of the JEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1683-1691, 2023.
11

---

Published as a conference paper at ICLR 2025
Yaroslav Ganin and Victor Lempitsky. Unsupervised domain adaptation by backpropagation. In International Conference on Machine Learning, 2015.
Yuanbiao Gou, Haiyu Zhao, Boyun Li, Xinyan Xiao, and Xi Peng. Test-time degradation adaptation for open-set image restoration. In Forty-first International Conference Machine Learning, 2024.
Jinjin Gu, Hannan Lu, Wangmeng Zuo, and Chao Dong. Blind super-resolution with iterative kernel correction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, 2019..
Shi Guo, Zifei Yan, Kai Zhang, Wangmeng Zuo, and Lei Zhang. Toward convolutional blind denoising of real photographs. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, 2019.
Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. In Advances in Neural Information Processing Systems Workshop, 2022.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 2020.
Judy Hoffman, Eric Tzeng, Taesung Park, Jun-Yan Zhu, Phillip Isola, Kate Saenko, Alexei Efros, and Trevor Darrell. Cycada: Cycle-consistent adversarial domain adaptation. In International Conference on Machine Learning, 2018.
Tao Huang, Songjiang Li, Xu Jia, Huchuan Lu, and Jianzhuang Liu. Neighbor2neighbor: Self- supervised denoising from single noisy images. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, 2021.
Dong Huo, Abbas Masoumzadeh, Rafsanjany Kushol, and Yee-Hong Yang. Blind image deconvo- lution using variational deep image prior. IEEE Transactions Pattern Analysis and Machine Intelligence, 2023.
Geonwoon Jang, Wooscok Lee, Sanghyun Son, and Kyoung Mu Lee. C2n: Practical generative noise modeling for real-world demoising. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 2350-2359, 2021.
Hyemi Jang, Junsung Park, Dahuin Jung, Jaihyun Lew, Ho Bae, and Sungroh Yoon. Puca: patch- unshuffle and channel attention for enhanced self-supervised image denoising. Advances in Neural Information Processing Systems, 36, 2024.
Kui Jiang, Zhongyuan Wang. Peng Yi, Chen Chen, Baojin Huang, Yimin Luo, Jiayi Ma, and Junjun Jiang. Multi-scale progressive fusion network for single image deraining. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020.
Justin Johnson, Alexandre Alahi, and Li Fei-Fei. Perceptual losses for real-time style transfer and super-resolution. In Proceedings of the European Conference on Computer Vision, 2016.
Dongjin Kim, Donggoo Jung, Sungyong Baik, and Tae Hyun Kim. srgb real noise modeling via noise- aware sampling with normalizing flows. In The Twelfth International Conference on Learning Representations, 2024.
Yoonsik Kim, Jae Woong Soh, Gu Yong Park, and nam Ik Cho. Transfer learning from synthetic to real-noise denoising with adaptive instance normalization. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint 142266641,.690, 2014.
Shayan Kousha, Ali Maleky, Michael S Brown, and Marcus A Brubaker. Modeling srgb camera noise with normalizing flows. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 17463-17471, 2022.
Alexander Krull, Tim- Buchholz, and Florian Jug. Noise2void-leaminng denoising from single noisy images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019.
12

---

Published as a conference paper at ICLR 2025
Orest Kupyn, Volodymyr Budzan, Mykola Mykhailych, Dmytto Mishkin, and Jifí Matas. Deblurgan: Blind motion deblurring using conditional adversarial networks. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, 2018.
Wooseok Lee, Sanghyun Son, and Kyoung Mu Lee. Ap-bsn: Self-supervised denoising for real-world images via asymmetric pd and blind-spot network. In Proceedings of the IEEE/CVF Conference Computer Viston and Pattem Recognition, pp. 17725-17734 2022.
Jaakko Lehtinen, Jacob Munkherg, Jon Hasselgren, Samuli Laine, Tero Karras, Miik Aittala, and Timo Aila. Noise2noise: Learning image restoration without clean data. arXiv preprint 1eX3::8803044180 2018.
Tianhong Li, Yonglong Tian, He Li, Mingyang Deng, and Kaiming He. Autoregressive image generation without vector quantization. arXiv preprint arXiv:2406. 1838, 2024.
Jingyun Liang, Jiezhang Cao, Guolei Sun, Kai Zhang, Luc Van Gool, and Radu Timofte. Swinir: Image restoration using swin transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, 2021.
Tsung-Yi Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft COCO: Common objects in context. In Proceedings of the European Conference on Computer Vision, 2014.
Ming-Yi Yu Liu and Oncel Tuzel. Coupled generative adversarial networks. Advances in Neural Information Processing Systems, 2016.
Mingsheng Long, Yue Cao, Jianmin Wang, and Michael Jordan. Learning transferable features with deep adaptation networks. In International Conference on Machine Learning, 2015.
Zhengxiong Luo, Yan Huang, Shang Li, Liang Wang, and Tieniu Tan. Learning the degradation distribution for blind image super-resolution. arXiv preprint ar arXiv:2203.04963 2022.
Kede Ma, Zhengfang Duanmu, Qingbo Wu, Zhou Wang. Hongwei Yong. Hongliang Li, and Lei
Zhang. Waterloo exploration database: New challenges for image quality assessment models.
IEEE Transactions on Image Processing, 26(2):1004-1016 2016.
David Martin, Charless Fowlkes, Doron Tal, and Jitendra Malik. A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics. In Proceedings Eighth IEEE International Conference on Computer Vision, 2001.
Seungjun Nah, Tae Hyun Kim, and Kyoung Mu Lee. Deep multi-scale convolutional neural network for dynamic scene deblurring. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, 2017.
Seungjun Nah, Radu Timofte, Sungyong Baik, Seokil Hong, Gyeongsik Moon, Sanghyun Son, and Kyoung Mu Lee. Ntire 2019 challenge on video deblurring: Methods and results. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, 2019.
Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In International Conference on Machine Learning, 2021.
Jinshan Pan, Deqing Sun, Hanspeter Pfister, and Ming-Hsuan Yang. Blind image deblurring using dark channel prior. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2016.
Tobias Plotz and Stefan Roth. Benchmarking denoising algorithms with real photographs. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2017.
Kuldeep Purohit, Maitreya Suin, AN Rajagopalan, and Vishnu Naresh Boddeti. Spatially image restoration using distortion-guided networks. In Proceedings of the IEEF/CVE International Conference on Computer Vision, 2021.
Chao Ren, Xiaohai He, Chuncheng Wang, and Zhibo Zhao. Adaptive consistency prior based deep network for image denoising. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, 2021.
13

---

Published as a conference paper at ICLR 2025
Dongwei Ren, Wangmeng Zuo, Qinghua Hu, Pengfei Zhu, and Deyu Meng. Progressive image deraining networks: A better and simpler baseline. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, 2019.
Dongwei Ren, Kai Zhang, Qilong Wang. Qinghua Hu, and Wangmeng Zuo. Neural blind deconvo- lution using deep priors. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020.
Jaesung Rim, Haeyun Lee, Jucheol Won, and Sunghyun Cho. Real-worid blur dataset for learning and benchmarking deblurring algorithms. In Proceedings of the European Conference on Computer Vision, 2020.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition, pp.
2022.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In Medical Image Computing and Computer-Assisted Intervention, 2015.
Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman. Dreambooth: Fine tuning text-to-inn text-to-image image diffusion models for subject-driven generation. In Pro- ceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, 2023.
Kate Saenko, Brian Kulis, Mario Fritz, and Trevor Darrell. Adapting visual category models to new domains. In Proceedings of the European Conference on Computer Vision, 2010.
Chitwan Saharia, William Chan, Huiwen Chang, Chris Jonathan Ho, Tim Salimans, David Fleet, and Mohammad Norouzi. Palette: Image-to-image diffusion models. In ACM SIGGRAPHI 2022 Conference Proceedings, 2022a.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Bureu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. Advances in Neural Information Processing Systems, 2022b.
Chitwan Saharia, Jonathan Ho, William Chan, Tim Salimans, David Fleet, and Mohammad Norouzi. Image super-resolution via iterative refinement. IEEE Transactions Pattern Analysis and Machine Intelligence, 45(4):4713-4726, 2022c.
Assaf Shocher, Naday Cohen, and Michal Irani. "zero-shot" super-resolution using deep internal learning. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, 2018.
Ashish Shrivastava, Tomas Pfister, Oncel Tuzel, Joshua Susskind, Wenda Wang, and Russell Webb. Learning from simulated and unsupervised images through adversarial training. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, 2017.
Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. International Conference on Machine Learning. 2015.
Maitreya Suin, Kuldeep Purohit, and AN Rajagopalan. Spatially-attentive aach-hierarchical network for adaptive motion deblurring. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020.
Yaniv Taigman, Adam Polyak, and Lior Wolf. Unsupervised cross-domain image generation. arXiv 2016.
preprint
Radu Timofte, Eirikur Agustsson, Lue Van Gool, Ming-Hsuaa Yang, and Lei Zhang. Ntire 2017 challenge on single image super-resolution: Methods and results. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, 2017.
Antonio Torralba and Alexei A Efros. Unbiased look at dataset bias. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, 201 1.
14

---

Published as a conference paper at ICLR 2025
Eric Tzeng, Judy Hoffman, Ning Zhang, Kate Saenko, and Trevor Darrell. Deep domain confusion: Maximizing for domain invariance. arXiv preprint arXiv.1412.3474, 2014.
Eric Tzeng, Judy Hoffman, Trevor Darrell, and Kate Saenko. Simultaneous deep transfer across domains and tasks. In Proceedings of the IEEE International Conference on Computer Vision, 2015.
Eric Tzeng, Judy Hoffman, Kate Saenko, and Trevor Darrell. Adversarial discriminative domain adap- tation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2017.
Dmitry Ulyanov, Andrea Vedaldi, and Victor Lempitsky. Deep image prior. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, 2018.
Hong Wang. Zongsheng Yue, Qi Xie, Qian Zhao, Yefeng Zheng, and Deyu Meng. From rain generation to rain removal. In Proceedings of the IEEF/CVF Conference on Computer Vision and Puttern Recognition, 2021.
Jianyi Wang, Zongsheng Yue, Shangchen Zhou, Kelvin CK Chan, and Chen Change Loy. Exploiting diffusion prior for real-world image super-resolution. International Journal of Computer Vision, 132(12):5929- 99919999999999922 2024.
Tianyu Wang. Xin Yang, Ke Xu, Shaozhe Chen, Qiang Zhang, and Rynson WH Lau. Spatial attentive single-image deraining with high quality real rain dataset. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019.
Xintao Wang, Ke Yu, Shixiang Wu, Jinjin Gu, Yihao Liu, Chao Dong, Yu Qiao, and Chen Change Loy. Esrgan: Enhanced super-resolution generative adversarial networks. In Proceedings of the Euro- pean Conference on Computer Vision Workshops, 2018.
Zhendong Wang, Xiaodong Cun, Jianmin Bao, Wengang Zhou, Jianzhuang Liu, and Houqiang Li. Uformer: general u-shaped transformer for image restoration. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022.
Simon Welker, Henry N Chapman, and Timo Gerkmann. Driftrec: Adapting diffusion models to blind jpeg restoration. IEEE Transactions on Image Processing, 2024.
Wenhan Yang, Robby T Tan, Jiashi Feng, Jiaying Liu, Zongming Guo, and Shuicheng Yan. Deep joint rain detection and removal from single image. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2017.
Yuntong Ye, Changfeng Yu, Yi Chang, Lin Zhu, Xi-Le Zhao, Luxin Yan, and Yonghong Tian. Unsupervised deraining: Where contrastive learning meets self-similarity. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022.
Zongsheng Yue, Hongwei Yong, Qian Zhao, Deyu Meng, and Lei Zhang. Variational denoising network: Toward blind noise modeling and removal. In Advances Neural Information Processing Systems, 2019.
Zongsheng Yue, Jianyi Wang, and Chen Change Loy. Resshift: Efficient diffusion model for image super-resolution by residual shifting. In Advances in Neural Information Processing Systems, 2023.
Zongsheng Yue, Hongwei Yong, Qian Zhao, Lei Zhang, Deyu Meng, and Kwan Yee K Wong. Deep variational network toward blind image restoration. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024.
Syed Waqas Zamir, Aditya Arora, Salman Khan, Munawar Hayat, Fahad Shahbaz Khan, Ming-Hsuan Yang, and Ling Shao. Multi-stage progressive image restoration. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, 2021.
Syed Waqas Zamir, Aditya Arora, Salman Khan, Munawar Hayat, Fahad Shahbaz Khan, and Ming-Hsuan Hsuan Yang. Restormer: Efficient transformer for high-resolution image restoration. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, 2022.
15

---

Published as a conference paper at ICLR 2025
Hongguang Zhang, Yuchao Dai, Hongdong Li, and Piotr Koniusz. Deep stacked hierarchical multi- patch network for image deblurring. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, 2019.
Kai Zhang, Wangmeng Zuo, Yunjin Chen, Deyu Meng, and Lei Zhang. Beyond Gaussian denoiser: Residual learning of deep CNN for image denoising. IEEE Transactions on Image Processing, 26 (7):3142-3155, 2017.
Kai Zhang, Wangmeng Zuo, and Lei Zhang. FFDNet: Toward fast and flexible solution for cnn-based image denoising. IEEE Transactions Image Processing, 27(9):468-44222 201 8a.
Kai Zhang, Yawei Li, Wangmeng Zuo, Lei Zhang, Luc Van Gool, and Radu Timofte. Plug-and-plas image restoration with deep denoiser prior. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(10):6360-633 6376, 2021.
Ruofan Zhang, Jinjin Gu, Haoyu Chen, Chao Dong. Yulun Zhang, and Wenming Yang. Crafting training degradation distribution for the cccuracy-generaaizztiion trade-off in real-world super- resolution. In International Conference on Machine Learning. 2023.
Yulun Zhang, Kunpeng Li, Kai Li, Lichen Wang, Bineng Zhong, and Yun Fu. Image super-resolution using very deep residual channel attention networks. In Proceedings of the European Conference on Computer Vision, 2018b.
Chuanxia Zheng, Tat-Jen Cham, and Jianfei Cai. T2net: Synthetic-to-realsttie translation for solving single-image depth estimation tasks. In Proceedings of the European Conference Computer Vision, pp. 767-783, 2018.
Jun- Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros. Unpaired image-to-image translation using cycle-consistent adversarial networks. In Proceedings of the IEFE International Conference on Computer Vision, 2017.
16

---

Published as a conference paper at ICLR 2025
APPENDIX
AI IMPLEMENTATION DETAILS
A1.1 CONDITION EVALUATION ON DIFFUSION MODEL
This work is inspired by the beneficial effects that favorable conditions facilitate the denoising process of the diffusion model, as shown in Fig I aa. In this preliminary experiment, we first condition and train the diffusion model with an additional input in addition to its conventional input. Then, we test the noise prediction performance of this model under different conditions. To be specific, the condition by adding the additive white Gaussian noise (AWGN) noise level a E [0, 80) to its original clean images, which performed 1, 1,000 images in the MS-COCO test dataset (Lin et al.) 2014). The noise prediction error of the diffusion model is evaluated using the mean square error (MSE) metric.
corrupt
A1.2 COMPARISON SETTINGS
In comparison experiments, we mainly compare the proposed approach with three types of previous methods: domain adaptation methods, including DANN (Ganin & Lempitsky 2015). DSN [Bous malis al., 2016)) PixeIDA (Bousmalis et al., 2017). and CyCADA (Hoffman al. 2018): unsu-
pervised image restoration methods, including Ne2Ne (Huang et al. (021). MaskedD (Chen et al. 2023), NLCL. (Ye et al. 2022), SelfDeblur (Ren et al. 2020). and VDIP (Huo et al., 2023); some representative supervised methods which serve as strong baselines in image restoration such as Restormer (Zamir et al. 2022), to comprehensively evaluate generalization performance of different methods. MaskedD (Chen et al. 2023) proposes masked training to enhance the generalization performance denoising networks, showing the potential to be directly applicable to real-world scenarios. It shares the same goal with our work.
A1.3 SCALABILITY EVALUATION
To provide comprehensive evaluation of the proposed method, we apply six variants of the image restoration network in our experiments, including three variants of convolution-based network (Ron) nebergee et al. 2015): Unet-T (Tiny), Unet-S (Small), and Unet-B (Base); and three variants of Transformer- based network (Wang et al.] 2022): Uformer-T (Tiny), Uformer-S (Small), and Uformer- (Base). These variants differ in the number of feature channels (C) and the count of layers at each encoder and decoder stage. The specific configurations, computational cost, and the parameter numbers are detailed below:
Unet-T: C=32, depths of Encoder = (2, 2, 22, 2), GMACs: 3.14G, Parameter: 2.14M,
Unet-S: C=64, depths of Encoder (2. 2,, GMACs: 12.48G, Parameter: 8.56M,
Unet-B: C=76, depths of Encoder 2, 2), GMACs: 17.58G, Parameter: 12.07M,
Uformer-T: C=16, depths of Encoder (2, 2), GMACs: 15.49G, Parameter: 9.50M,
Uformer-S: C=32, depths of Encoder 22 2), GMACs: 34.76G, Parameter: 21.38M,
Uformer-B: C=32, depths of Encoder [1, 2, 8, 8), GMACs: 86.97G, Parameter: 53.58M,
and the depths of the Decoder match those of the Encoder.
A2 DISCUSSION ON DIFFERENT DOMAIN ADAPTATION METHODS
As discussed in Sec. 3.4 we described the effectiveness of the proposed method beyond the previous feature-space and pixel-space domain adaptation methods. We further show their specific framework in Fig. Al In contrast to previous adaptation methods, our method is free to a domain classifier or discriminator by introducing meaningful diffusion loss function.
17

---

Published as a conference paper at ICLR 2025
Function N etwookk
Transkation Netwock
Function Network
AC
Tre
Dill anion Netoo bock
Discrtminator
y
Natuo
Domain Classifier
Function Network
(a) Feature- Space DA
(b) Pixel-Space DA
(c) Noise- Noise-Space DA
Systhetic Input
X Real W ord Iaput
Synthetic Output
St: Real W orld Outpet
x Transterred Iuput
Concat
Random Noise Sampling
Target Distribution
Noisy Input
Figure A1: Overview of different domain adaptation (DA) approaches. (a) Fcature-space DA aligns the intermediate features across source and target domains. (b) Pixel-space DA translates source data to the "style" of the target domain through adversarial learning. (c) The proposed noise-space DA is specifically designed for image restoration. It gradually adapts the results from both source and target domains to the target clean image distribution, via multi-step denoising. Particularly, the function network represents restoration network in the context of image restoration.
A3 ADDITIONAL ANALYSIS OF THE ABLATIONS
We provided an ablation study to show the necessity of real data, in which only the synthetic or real data conditions onto the diffusion model. The quantitative results of the SIDD test dataset listed in Table. 4 It is noteworthy that both synthetic and real data are essential for effective domain adaptation in diffusion models. Omitting either type results in significant decline real-world performance. In particular, when real data is excluded, the performance nearly degrades to the level of a Vanilla model. We further analyze the necessity of each condition as follows: (1) Real data typically acts as a "bad" condition that introduces extra noises to the diffusion model, because the restoration network cannot restore it well under the domain gap. Consequently, valid and strong diffusion loss would backpropagate to the restoration network, promoting it learns to provide "good" conditions. As a benefit of the proposed strategies to eliminate the shortcut, the model progressively adapts the real data into the target clean distribution in multi-step denoising manner. (2) Synthetic data in two conditions can provide useful guidance in the early training stage, ensuring the diffusion model continuously focuses on these condition channels.
A4 ADDITIONAL COMPARISON RESULTS
A4.1 EXTENSION
As mentioned in Sec. our method can extend to the unpaired condition case by relaxing the diffusion's 's input with the image from other clean datasets. Thus, the shortcut issue can be poten- tially eliminated since the trivial solutions such as matching the pixel's similarity between input and condition do not exist. Such an extension keeps the channel shuffling layer but is free to the residual swapping contrastive learning. We show the quantitative evaluation in Tab, [A] The results demonstrate that although the condition and diffusion input are unpaired, our method can still learn to adapt the restored results from the synthetic and real-wor world domains to the clean image distribution, which also complements the restoration performance of the paired solution in
Table A1: Quantitative metrics of the proposed method (Ours) and extension on unpaired condition case (Our- Ex). The results are formed with PSNR/SSIM/LPIPS, The best and second best scores are highlighted and underlined.
Task
Ours
Ours-Ex
Denoising 34.71/0.9202/0.0903
33.44/0.8938/0.1 1064
Deraining 34.39/0.9571/0.0466
Deblurring 26.46/7.8044/0.1366 26.44/0.803000.111
some tasks like deraining and deblurring.
18

---

Published as a conference paper at ICLR 2025
Table A2: Complexity comparison of the image restoration methods: Parameter (M), GMACs (G).
Metrics
C2N
AP-BSN
Restormer
Selfdeblur
MaskedD
VDIP
Ours
Ours
Parameter
164.57
3.10
26.09
3.10
12.00
3.00
8.56
65.19
GMACs
280.48
60.62
35.24
23.83
188.03
23.77
12,48
22.46
Input
Input
Ours
Ours
Ours
Ours
Figure A2: Visual results detailed textures and high-fire freeueeeeenyy components. The proposed method serves general learning strategy for the image restoration task, offering scalability across different restoration networks. also enables performance improvements as the complexity the restoration network increases (Ours*).
A4.2 MORE ADVANCED RESTORATION NETWORKS
As discussed in Sec. 4.3 the proposed domain adaptation method offers strong scalability across various image restoration networks. Additionally, by employing more advanced restoration networks with the proposed denoising as adaptation (Ours*), the performance can be further improved, yielding results that are more perceptually aligned with the ground truth as illustrated in Fig. A2 The complexity comparison of different image restoration networks is listed in Table A2
A4.3 ADDITIONAL VISUAL COMPARISON RESULTS
We visualize more comparison results the image denoising task in Fig. A3 image deraining task in Fig. A4, and image deblurring task in Fig. A5 In particular, we name the proposed method and its extension as 'Ours' and 'Ours-Ex's respectively.
A4.4 ADDITIONAL. VISUAL RESULTS ON OTHER REAL--WORLE DATASETS
To show the generalization ability of the proposed method, we also visualize the restored results of the proposed method other real-world datasets (Plotz & Roth 2017 Yang et 2017) in Fig. A6
Fig. A7 Fig. A8 Please note that these datasets were not seen during the network's training and fall outside the distribution of the trained datasets.
A4.5 FAILURE CASE
We show the failure case of our method and comparison methods in Fig A9 Particularly. our method fails to restore images with challenging degraded distortions such as strong noises and dut-of-distribution noises. These real-world degradations induce significant gap compared with the synthetic dataset, burdening the learning model effectively adapt the restored results into the clean domain.
A4.6 ANALYSIS ON TRAINING DYNAMICS AND COMPLEXITY
To demonstrate the impact of the introduced diffusion loss during training, we visualize the related metrics of training dynamics in Fig. A10 (left). It is easy to find that the restoration model trained only with L1 loss on the synthetic dataset tends to overfit quickly and performs poorly on the real-world validation By contrast, the diffusion loss effectively guide the restoration model to adapt to the real-world domain in a multi-step denoising manner, consistently improving the restoration performance the real-world validation set.
19

---

Published as a conference paper at ICLR 2025
OC.
OR DRY ERASE
Vanilla
DANN
PixeIDA
CyCADA
Ne2Ne
MaskedD
Ours
Ours-Ex
tixel
Vanilla
DANN
PixeIDA
CyCADA
ne by Go
Ne2Ne
MaskedD
Ours
Ours-Ex
Vanilla
DANN
PixeIDA
CyCADA
Ne2Ne
MaskedD
Ours
Ours-Ex
Vanilla
DANN
PixeIDA
CyCADA
Ne2Ne
MaskedD
Ours
Ours-Ex
Figure A3: Visual comparison of the image denoising task on SIDD test dataset (Abdelhamed et al. 2018
20

---

Published as a conference paper at ICLR 2025
DANN
DSN
PixeIDA
CyCADA
NLCL
Restormer
Ours
Ours-Ex
Storyblo
DANN
DSN
PixeIDA
CyCADA
NLCL
Restormer
Ours
Ours-Ex
DANN
DSN
PixcIDA
CyCADA
NL.L
Restoreer
Ours
Ours-Ex
DANN
DSN
PixeIDA
CyCADA
NLCL
Restormer
Ours
Ours-Ex
Figure A4: Visual comparison of the image deraining task SPA test dataset
Waan al.
2019
21

---

Published as a conference paper at ICLR 2025
Inpul
DANN
PixeIDA
SellDeblur
Ours
Ours-Ex
Input
DANN
PixeIDA
SelfDeblur
Ours
Ours-Ex
Input
DANN
PixelDA
SelfDeblur
Ours
Qurs-Ex
7HT DGB
DGB4722
EE ISA
Input
DANN
PixeIDA
SeliDeblur
Our
Ours-Ex
Figure A5: Visual comparison the image deblurring task on RealBlur-J (Rim al. 2020 test dataset.
22

---

Published as a conference paper at ICLR 2025
Input
Figure A6: Visual results of proposed method on unseen DND real- world denoising test dataset Plotz
Roth 2017).
23

---

Published as a conference paper at ICLR 2025
MiskedD>
Figure A7: Visual results of the proposed method on unseen DND real-world denoising test dataset Plltt & Roth 2017).
24

---

Published as a conference paper at ICLR 2025
Input
Restormer
Ours
Figure A8: Visual results of the proposed method on unseen 'Real-Intemeet real-world deraining test datasct Yang al. 2017
25

---

Published as a conference paper at ICLR 2025
Input
Ours
Comparison
Figure A9: Failure of the proposed method and comparison methods.
on Validetion
Quantitative Evallation SIDD
25-
20-
-04
22
L1 LOBS
50
100
Eboch
200
Difieeet
Ofti 2n
Modal
Figure A10: Analysis of training dynamics (left) and model complexity (right) of the proposed method.
Moreover, we show the results validating the diffusion model with different complexities in Fig. A10 (right). To be specific, classify the diffusion models into three types based on their complexitics in each layer: Diffusion-T: [32. 32, 64, 64], Diffusion-S: [32, 64, 128, 128]. Diffusion- B: [64, 128, 256, 512] (exploited in this paper). As can observe, the real-world restoration performance gains further improvements as the complexity of the diffusion model increases, i.e., from Diffusion-T Diffusion-B. We also provide deep analysis of diffusion loss in restoration tasks when compared with MAR (Li et al. 2024): MAR models the per-token probability distribution using small MLP the diffusion model. It trained jointly with the AR model to achieve efficient image generation. In particular, the tokens are small in size and represent high-level semantic features. By contrast, our diffusion model serves for the low-level image restoration problem. It directly adapts the restoration results at the dense and wide-scale pixel level, requiring accurate discrimination on the rich texture of images. Therefore, in the context of adapting the restoration model to the real-world domain, the diffusion model cannot be extremely simplified.
DnCNN
Restormer
SwinIR
Vanilla
Ours
Vanilla
Ours
Vanilla
Ours
PSNR
25.98
30.46
24.10
33.98
30.86
34.85
SSIM 0.5911
0.7637
0.5194
0.9183
0.7544
0.9153
Table A3: Performance comparison on SIDD test set of the commonly-used and SOTA restoration models under different training strategics. Vanilla denotes training the model on the paired synthetic datasets and Ours denotes training with the proposed noise-space domain adaptation strategy.
26

---

Published as a conference paper at ICLR 2025
Algorithm 1 Training Process of The Proposed Framework
Input: Degraded images x" and ground truth images y" from synthetic domain D degraded images x" from real-world domain DD"
Output: Restored synthetic image 1j" and restored real-world image "'
1: Initialization on the image restoration model fa(-) and diffusion model Es(-)
2: repeat
3: for all a: and y E D x3 E Dr do
4: Df < fo(())) 93 + fo()))
5: Calculate the restoration loss CRee with ÿf and y using Charbonnier loss
6: Sample the diffusion's imput yf = va.y, + vI- djE, (),,)), [1
7: Predict the noise 4 (ū, C(j 07). t), calculate the diffusion loss D
8: Update ea(.) using LDE
9: Update fo(-) using Eq with Res and D
10: end for
11:
until Convergence
Algorithm 2 One-Pass Inference of The Proposed Framework
Require: Degraded image x", trained image restoration model fa (-) 1: O" +- fa("")
2: return y
A4.7 VALIDATION ON MORE RESTORATION MODELS
Our work contributes to novel and general domain adaptation strategy for image restoration, which cannot be replaced by current self-supervised methods. To this end, we further validated our method on the commonly used and SOTA restoration models, such as DrCNN, Restormer, and SwinIR. The quantitative evaluations of these comparison methods are reported in Table. As we can observe, all restoration models trained on synthetic datasets failed to generalize well to the real-world dataset. By incorporating the proposed domain adaptation training strategy, the woal-world performance of these models gains significant improvements, demonstrating the favorable generalization and scalability of our method.
A5 ADDITIONAL TECHNOLOGICAL DETAILS
A5.1 TRAINING AND INFERENCE
To provide clear distinction and description of the training and inference stages, show their detailed processes in Alg. T and Alg. 2 respectively. For clarity, we omit the strategies to eliminate the shorteut solutions (exploited in the 7th step of Alg. [ such the channel shuffling layer and residual-swapping contrastive learning.
A5.2 INFORMATION GUIDANCE BETWEEN TWO DOMAINS
Our motivation for this work derived from an interesting observation: the prediction error of conditional diffusion model relies on the quality of the conditions (as shown in Fig. (a)). Therefore, guided by the back- propppaoeee diffusion loss, the restoration network optimized to provide "good" conditions to minimize the diffusion model's noise prediction error, aiming for clean image distribution. During this joint training, the synthetic GT serves as the denoising target the diffusion model, which potentially offers realistic textures to help adapt the degraded real-worle images into the clean distribution. In other words, the clean knowledge/ifforaatinn input (in multi-step denoising manner) plays important role in bridging the gap between different domains.
"leaked" by the diffusion's
Generally, the ground truth images and those restored images by restoration model, whether from synthetic or real-world domains, should reside within a common distribution of high-quality, clean images. However, the appearance of synthetic GT and restored real-world data unrelated, leading
27

---

Published as a conference paper at ICLR 2025
the diffusion model exploit a shorteut, overfitting its denoising capability by relying solely on the paired synthetic data. To this end, we further design crucial strategies (e.g., channel-shufling layer and respdig contrastive learning) to implicitly blur the boundaries between conditioned synthetie and real data and prevent the reliance of the model on easily distinguishable features. As result, the proactive "leakage" from clean distribution to degraded images can effectively work during the whole training process, consistently improving the restoration performance on real-world images.
28


Reviews:
[{"review_id": "n0KYigXm4b", "reviewer": "Reviewer_5su7", "summary": "The core of the method is a two-model setup: a restoration network and a diffusion model. The restoration network is trained to clean degraded images, while the diffusion model acts as a noise-space guide, enforcing an alignment between synthetic and real-world restorations and the clean target distribution. The authors validate their approach on tasks such as image denoising, deblurring, and deraining, showing it consistently outperforms both feature-space and pixel-space domain adaptation methods.", "questions": "Can the method be trained with multiple tasks simultaneously since it is technically (low-level) task agnostic?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The strengths of this work lie in the following: \n- A novel approach for iteratively using a diffusion model as a proxy model for denoising in noise space \n- The authors introduce channel-shuffling and residual-swapping strategies, ensuring robust generalization. \n- The diffusion model is used only during training\n- The final restoration model achieves SOTA results across multiple restoration tasks.", "weaknesses": "For de-raining results, the authors should show results where the background is not empty, e.g. where there is more content in the image besides a gray background as in Figure 5.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "L4g8yTX6MX", "reviewer": "Reviewer_s2ny", "summary": "This paper presents an image restoration approach that utilizes diffusion models to address domain adaptation issues. The method adjusts restored results in the pixel-wise noise space, resulting in significant improvements in low-level visual appearance while operating within a compact and stable training framework. To avoid shortcut learning, the method employs channel index shuffling and a residual-swapping contrastive learning strategy. Experimental results demonstrate this approach outperforms existing feature-space and pixel-space domain adaptation methods across various tasks, such as image denoising, deraining, and deblurring.", "questions": "Could the authors give concrete examples of situations where their proposed method fails to deliver the expected results?\n\nI am impressed by the significant improvements reported by the authors. I wonder how the results would compare if the model were trained directly on a paired real dataset and then tested on corresponding real data.\n\nI expect a direct and clear explanation from the authors regarding how the knowledge is \"leaked\" from the synthetic domain to the real domain in the proposed method.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The method enhances image restoration performance by effectively bridging the gap between synthetic and real-world data.\n\n2. The approach can be easily integrated with various restoration networks, demonstrating improved performance even with more complex architectures.", "weaknesses": "1. The writing in this paper is not very clear, and I found the third section particularly confusing regarding how the proposed method leverages the mapping learned from synthetic datasets to be applied to real datasets. There must be some form of \"information leakage\" involved—this is a proactive \"leakage\" that helps bridge the gap between different domains. However, the authors do not provide a clear explanation of this process, and it appears that channel shuffling plays a key role in this function. A more explicit clarification from the authors would be beneficial.\n\n2. The paper lacks runtime for the proposed method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "zzISfLw4oC", "reviewer": "Reviewer_kUVz", "summary": "The introduces an approach to improve the generalization of image restoration models by addressing the domain gap between synthetic and real-world data using noise-space adaptation with diffusion models. The method utilizes diffusion loss, which is optimized during joint training with the restoration network. Key strategies, such as channel-shuffling and residual-swapping contrastive learning, are employed to prevent shortcut learning, ensuring the model effectively restores images rather than relying on trivial cues. The diffusion model is then discarded, leaving a robust restoration network. Experimental results show the method outperforms existing domain adaptation techniques.", "questions": "Please refer to the Weaknesses for the main questions.\n\n1. How does the model perform on synthetic degradations? Are there significant performance trade-offs due to the domain-adaptation process?\n2. Would it be possible to provide quantitative metrics (e.g., PSNR) alongside the visual results (e.g., in Figures 4 and 5) to better illustrate the restoration performance?\n3. The paper seems to mainly demonstrate the performance on denoising task, which shows the highest performance gain. Why does the proposed idea fit substantially better for denoising?", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The idea of incorporating diffusion loss in domain adaptation for image restoration is novel.\n2. The proposed method can be applied to a variety of restoration tasks, including denoising, deblurring, and deraining, with demonstrated performance gains in the experiments.\n3. The use of channel-shuffling and residual-swapping to prevent shortcut learning is well-motivated, and ablation studies show that this technique is crucial for performance improvement.\n4. The paper is well-written and easy to understand.", "weaknesses": "1. Although ground-truth images from the real dataset are not used for training, degraded images from the same dataset are presented during training, which blurs the distinction between the training and inference stages. To better validate the generalization ability of the proposed framework, driven solely by the inference stage, evaluation on fully unseen datasets (e.g., Nam or PolyU datasets for denoising) should be included.\n2. The authors do not provide an analysis of training time, which seems to be a key limitation of the proposed approach. There should be a detailed comparison of training complexity with other models, not just comparisons of inference performance.\n3. Ablation studies on the sensitivity to hyperparameters (e.g., beta, gamma) are lacking, particularly as the authors state that these values were chosen empirically.\n4. The performance gains compared to Vanilla baseline (trained only with synthetic datasets), while present, are relatively small for deraining and deblurring tasks. Especially for deblurring task, gain is less than 0.2dB.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3pgBBitH6v", "reviewer": "Reviewer_vDAP", "summary": "- While traditional domain-adaptation methods have focused on adaptation in the feature or pixel-space, this paper emphasizes adaptation in the noise space by leveraging the diffusion model process. \n- The training strategy using the diffusion process includes channel-shuffling and residual-swapping contrastive learning to prevent shortcuts learning. Additionally, during inference, only the restoration model is used independently from the diffusion model, making this approach memory-efficient as well.", "questions": "- This paper utilizes U-Net and U-Former for simplicity. In my opinion, the key contribution lies in applying the paper’s novel learning strategy to general restoration networks. Therefore, I am curious whether the proposed strategy also performs well on SOTA methods such as **Restormer** or **NAFNet**[1*], and whether these SOTA methods can surpass the performance on real-world data when trained with both real-world and synthetic data.\n- Based on the SIDD test set, **NAFlow**[2*], **NeCA**[3*], and **sRGB-Flow**[4*] show much lower performance compared to the restoration achieved with DnCNN using synthetic datasets. According to the claim, the proposed method should be able to bridge the gap between synthetic and real data, thus delivering better performance. It would also be helpful if the paper provided performance results using DnCNN[5*] or other commonly used denoising networks to further validate the effectiveness of the approach.\n---\n[1*] Chen, Liangyu, et al. \"Simple baselines for image restoration.\" European conference on computer vision. Cham: Springer Nature Switzerland, 2022.\n\n[2*] Kim, Dongjin et al. \"sRGB Real Noise Modeling via Noise-Aware Sampling with Normalizing Flows.\" The Twelfth International Conference on Learning Representations. 2024.\n\n[3*] Fu, Zixuan, Lanqing Guo, and Bihan Wen. \"sRGB Real Noise Synthesizing with Neighboring Correlation-Aware Noise Model.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.\n\n[4*] Kousha, Shayan, et al. \"Modeling srgb camera noise with normalizing flows.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.\n\n[5*] Zhang, Kai, et al. \"Beyond a gaussian denoiser: Residual learning of deep cnn for image denoising.\" IEEE transactions on image processing 26.7 (2017): 3142-3155.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- Domain adaptation in the noise space using the diffusion model process is innovative.\n- Although this paper focuses on general restoration tasks such as denoising, deraining, and deblurring, it also potentially explains why the training method using synthetic data struggles to effectively address real data in the denoising problem, and why joint training with both types of data has been ineffective.\n- The training strategy to prevent shortcuts is a novel contribution to the restoration task. In fact, shortcut phenomena can occur not only in restoration problems but also in tasks that involve predicting high-frequency components. While it raises the question of whether this strategy can be applied to general CNN networks as well as diffusion models, the paper demonstrates its effectiveness.", "weaknesses": "- The performance of the comparison methods used in the experiments is severely degraded. For example, the methods used for comparison in **Table 1** are either outdated or inappropriate. Although relatively recent methods such as **Ne2Ne** and **MaskedD** were used, since these methods do not target denoising using real data, it is questionable whether the results obtained by training these methods on real-world data are reasonable.\n- Additionally, since the method presented in the paper uses extra data along with synthetic data for each task, it would be more appropriate to compare it with methods such as **PNGAN**[1*], which use the same training data, as seen in **Table 1**.\n- The experiments in **Table 4** demonstrate that the noise sampling range of [1, 1000] is appropriate. Although the model does not require the diffusion process during inference, we believe this range imposes a significant burden during training. Considering this, for example, **DnCNN**[2*] can achieve a gain of approximately 35 dB on the SIDD test dataset with simple training using only L1 loss, even when trained directly on real data.\n---\n[1*] Cai, Yuanhao, et al. \"Learning to generate realistic noisy images via pixel-level noise-aware adversarial training.\" Advances in Neural Information Processing Systems 34 (2021): 3259-3270.\n\n[2*] Zhang, Kai, et al. \"Beyond a gaussian denoiser: Residual learning of deep cnn for image denoising.\" *IEEE transactions on image processing* 26.7 (2017): 3142-3155.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "J9DniNIVkP", "reviewer": "Reviewer_qhFa", "summary": "* This work proposes to address domain adaptation in the noise space for image restoration \n* The authors propose channel shuffling and residual-swapping contrastive learning strategy to overcome the shortcut learning issue\n* Experiments on image deraining, denoising and deblurring validate the effectiveness", "questions": "Please see weakness", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* Generalization is a challenging and vital problem in image restoration. The paper proposes to work towards generalization using noise-space adaption.\n* The work is well motivated. Especially in Figure 1, the authors find condition of diffusion models can serve as a proxy to discriminate clean distributions. This is interesting for me.\n* The paper is well-written. The method is clearly claimed.", "weaknesses": "* It seems the paper lacks direct evidence to support diffusion loss is effective. That is, what is the performance if we replace the diffusion loss with perceptual loss, gan loss or nothing. I think this validation is important.\n* I suggest to add some visual results of other compared methods in Figure A6-A8.\n* Can you provide more details about the used diffusion model? What is the impact of the introduced diffusion loss on training dynamics(eg, loss, accuracy on validation set)? Whether the diffusion model can be extremely simplified (such as a single MLP layer like [1])\n[1] Autoregressive Image Generation without Vector Quantization", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)