Adaptive Domain Learning for Cross-domain Image Denoising

Authors: zian  qian
chenyang  qi
ka lung law
hao  fu
chenyang  lei
qifeng  chen
  sensetime
hkisi-cas  cair
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-14 11:17:06

Abstract: Different camera sensors have different noise patterns, and thus an image denoising model trained on one sensor often does not generalize well to a different sensor. One plausible solution is to collect a large dataset for each sensor for training or fine-tuning, which is inevitably time-consuming. To address this cross-domain challenge, we present a novel adaptive domain learning (ADL) scheme for cross-domain RAW image denoising by utilizing existing data from different sensors (source domain) plus a small amount of data from the new sensor (target domain). The ADL training scheme automatically removes the data in the source domain that are harmful to fine-tuning a model for the target domain (some data are harmful as adding them during training lowers the performance due to domain gaps). Also, we introduce a modulation module to adopt sensor-specific information (sensor type and ISO) to understand input data for image denoising. We conduct extensive experiments on public datasets with various smartphone and DSLR cameras, which show our proposed model outperforms prior work on cross-domain image denoising, given a small amount of image data from the target domain sensor.

Main paper: 
 introduction: 
 Noise generated by electronic sensors in a RAW image is inevitable. Over the past few years, learningbased methods have made significant progress in RAW image denoising [5, 22, 25, 42] . However, building a large-scale real-world dataset with noise-clean pairs for training a denoising model is time-consuming and labor-intensive. It is hard to collect ground truth that is noise-free and has no misalignment with the input noisy data. Moreover, due to the different noise distributions of different sensors (such as read noise and shot noise), the collected data from a particular sensor usually cannot be used to train the denoising model of other sensors, which causes a waste of resources. Therefore, it is important to develop a method to solve this problem.Existing solutions to data scarcity in RAW image denoising can be divided into two categories, noise calibration [42, 47, 27] and self-supervise denoising [22, 18, 25, 41, 20] . Noise synthesis and calibration methods first build a noise model, optimize for noise parameters according to a particular camera, and then synthesize training pairs from the noise model to train a network. Self-supervised denoising is designed based on the blindspots schemes. When the input noisy image masks out some pixels and forms a similar but different image from the input, the network learns to denoise instead of identity mapping. Therefore, the network can learn to denoise without pairwise noise-clean data.While the noise synthesis and calibration methods are top-performing ones for RAW data denoising and self-supervised denoising does not need to collect pairwise data, both of them have their practical limitations. First, noise synthesis and calibration methods are not able to obtain the exact noise model of the real noise. For example, fixed pattern noise such as dark signal non-uniformity (DSNU) and Photo-response non-uniformity (PRNU) are not included in the model. As a result, some of the sampled noise training pairs might be harmful to the training of the denoising model (i.e., decrease in performance). Second, building a calibration model still needs to collect data under particular circumstances. Third, these models can only be used to synthesize training data for specific sensors, which leads to a waste of resources. On the other hand, self-supervised denoising is designed under some unverified assumptions of noise distribution. First, the noise distribution has zero means. Second, the noise in different pixels is independent of each other. These assumptions do not match the noise in the real world, especially when the noise distribution is complicated. Therefore, self-supervised denoising does not achieve state-of-the-art denoising performance.Different from prior work, we solve this problem by proposing a cross-domain RAW image denoising method, adaptive domain learning (ADL). Our method can utilize existing RAW image denoising datasets from various sensors (source domains) combined with very little data from a new sensor (target domain) together to train a denoising model for that new sensor. Some data in a source domain may be harmful to fine-tuning a model due to the large domain gap: for instance, synthetic data may be harmful to training a model for real-world applications if the synthetic data imposes unrealistic and unreasonable assumptions. In such cases, our method dynamically evaluates whether a data sample from a source domain is beneficial or harmful by evaluating the performance on a small validation set of the target domain, before and after fine-tuning the model on this data sample. If the performance improves after fine-tuning, we can use this data sample for training; otherwise, we should ignore it. As for the network architecture, we design a modulation network that takes sensor-dependent information as input (sensor types and ISO), which aligns the features from different sensors into the same space and ensembles useful common knowledge for denoising.To evaluate our proposed model with ADL, we compare our model against prior methods on diverse real-world public datasets [2, 42, 5] captured by both smartphone and DLSR cameras. The results demonstrate that our method outperforms the prior work and shows consistent state-of-the-art performance with ADL on RAW data denoising, given a small amount of data in the target domain. We also demonstrate that our ADL can be applied to fine-tuning existing noise calibration models with cross-domain data to further improve its performance.The contributions of this work can be summarized as follows.• We propose a novel adaptive domain learning (ADL) strategy that can train a model with little data from a new sensor (target domain), by automatically leveraging useful information and removing useless data from existing RAW denoising data from other sensors (source domains).• A customized modulation strategy is applied to provide sensor-specific information, which helps our network adapt to different sensors and noise distributions.• Our model outperforms prior methods in cross-domain image denoising in the target domain with little data.2 Related WorkQuestion: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?Answer: [Yes] Justification: We have a contribution summary in the introduction section.Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 
raw data denoising: 
 In recent years, methods based on RAW data denoising draw a lot of attention [29, 5, 42, 47, 4, 44, 25, 1, 48, 27] . SID [5] shows that RAW image denoising can perform well with a naive U-net architecture. Besides, they find difficulties in collecting large-scale datasets. Aware that collecting datasets is the research bottleneck, many approaches attempt to synthesize more realistic data. UIP [4] and CycleISP [44] attempt to inverse the image signal processing pipeline and synthesize noise in RAW space to train a RAW denoising framework. However, the generated pseudo-RAW data still has great differences compared to real RAW data. Jin et al. [16] utilize different noise distribution parameters to form a simulation camera to train a network. However, they still need to build a noise model to synthesize data. Another kind of approach is the noise calibration method [42, 47, 27, 45] . In the source domain adaptive learning stage, in iteration t, data from the source domain will be used to update the network parameter from θ t-1 to θ ′ . Then a dynamic validation set will judge whether the data is useful. If so, set θ t = θ ′ and repeat the process. If not, retrieve the network parameter from θ ′ to θ t-1 . Finally, the target domain data will be used to fine-tune θ ′ to θ T .Wei et al. [42] and Zhang et al. [47] analyze the noise source from the electronic pipeline of the DSLR cameras and build corresponding noise model to synthesize data. However, the domain gap still exists between the synthetic noise and the real noise. Besides, the noise calibration model is usually designed for a specific sensor, and hence no generalization ability and is not reusable. Our ADL can overcome the domain gap between real and synthetic data by removing harmful data. We can also utilize our ADL algorithm to fine-tune the existing noise calibration method to further improve its performance. Lehtinen et al. [22] find that pairs of two low-quality images with very similar content are enough to train a denoise model, forming the self-supervised denoising field.As the modified version of Noise2noise, methods that can learn to denoise with a single image are developed [18, 21, 41, 12, 20, 31, 14, 45] . They design a blind-spot network to force the model to learn the mapping from noisy to noise-free. In such cases, no paired data is needed for training. However, their method has some assumptions about noise distribution, which is usually not the case in the real world. Due to the above constraints, these methods usually cannot reach state-of-theart performance compared to other noise-to-clean supervised methods. Our method can solve the problem of data collection while keeping state-of-the-art performance.The gap between different domains (synthetic and real, daylight and night, etc.) is a great challenge in the field of low-level vision. To solve the problem, a set of approaches based on meta-transfer learning is proposed [32, 35, 17, 6, 43, 39, 40, 34, 15, 8] . Park et al. [32] and Soh et al. [35] utilize MAML algorithm [9] in super-resolution to obtain a model with better initialization implicitly from the source domain, then fine-tune it to fit the target domain better. Kim et al. [17] transferred the useful features from the synthesis noise model to the real-world noise model to overcome the domain gap problem between real-world noise and synthetic noise with an adaptive instance normalization layer [37, 13, 24, 33] to help the synthetic noise better adapt to real-world noise. For the metatransfer learning method, it is very hard to tell whether the implicitly learned information is useful or not. Some of the transferred features might be harmful. In contrast, our ADL can learn common knowledge and remove harmful information explicitly.Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [NA]Justification: No limitations currently.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
method: 
 In this section, we introduce the three steps in our adaptive domain learning pipeline: target domain pretraining, source domain adaptive learning, and target domain fine-tuning. The overall pipeline is illustrated in Figure 1 .Given the small training set of the target domain T adp , we first pre-train a model for the target domain by minimizing pixel-wise L 1 Loss.Target domain pre-training has benefits in two aspects. First, although the domain gap exists between the source domain and the target domain, denoising is a task that shares similar implicit feature representations. Therefore, pre-training can provide better initialization for the adaptive domain learning stage. Second, there is only very little data from the target domain, and the data from the source domain can be 100 times more than the data in the target domain. Pre-training on the target domain can improve the robustness and ensure the dominant position of the target domain data in the whole training process to prevent our model from overfitting to the source domain.However, due to the domain gap between the source domain and the target domain, not all the data from the source domain contribute to the training of the target domain, some data might be harmful and will lead to performance reduction. Therefore, we proposed adaptive domain learning (ADL), to eliminate harmful data and make use of the one that has contributions to our model.In each iteration t, we sample a batch of training data S ′ from some source domain S(i), and adapted our pretrained parameter θ t-1 to θ ′ byEQUATIONwhere α is the learning rate and L(S ′ ) is the L 1 loss defined on S ′ .To tell whether the data batch S ′ has contributions to our model, we evaluate the updated parameter θ ′ on a target domain validation set T val that is sampled from the target domain dataset T adp . The selection of the validation set in each iteration t is crucial to the performance of our method. Fixed validation set selection for each iteration may make the training stuck in the local minima and easily overfit to the validation set. To avoid these problems from happening, in each iteration t, we randomly sampled a dynamic validation set V ′ of size k from the target domain dataset T adp to let our model explore the feature space in a stochastic way. On the other hand, the rest of the dataset from T adp , denoted as T T rain , will combined with S ′ to provide the correct direction for the training process. At the beginning of the training, k is set to 20% of the size of T adp and increases during the training process. At the end of the training, 50% of T adp will be used.Moreover, inspired by [16] , when the size of T adp is extremely small, i.e., smaller than 10, we intentionally select the data that has very diverse system gain from T adp to form V ′ in each iteration to avoid the over-fitting problem.Dynamic average PSNR We evaluate whether the data batch S ′ is useful or not by comparing the PSNR of the result of the updated network parameter θ ′ to the PSNR of the result of previous iteration θ t on the sampled validation set V ′ . However, hard criteria based on PSNR usually make the training procedure unstable under the setting of the dynamic validation set. When the size of the dynamic validation set V ′ is small, the variance of PSNR is large and thus is not that reliable. Some useful data might be removed accidentally. In such cases, we want our model to take a data batch S ′ as useful data if S ′ has a trend to improve the performance of our model. We design soft criteria based on PSNR by maintaining a priority queue Q eval of max size M that stores the value of highest PSNR value in the history during the training process. Q eval is ranked by the value of PSNR in ascending order. We denote the PSNR on our dynamic validation set V ′ of model θ ′ at iteration t in our training process as Eval(V ′ , θ t ). At the beginning of the adaptive domain learning stage, we push Eval(V ′ , θ t=0 ) into Q eval . During the training, if the PSNR of the updated parameter θ ′ on the dynamic validation set V ′ , Eval(V ′ , θ ′ ), is higher than the average PSNR in Q eval , we keep the updated parameter θ ′ and push Eval(V ′ , θ ′ ) into Q eval and pop out the first element in Q eval if it is full. Else, we retrieve the network parameter from θ ′ to θ t-1 . This process can be characterized asEQUATIONAlgorithm 1 Adaptive domain learning (ADL) Require: S 1 , . . . , S n : training sets of n source domains Require: T adp : The small target domain datasetRequire: Q eval : priority queue with max length M that stores the PSNR.1: Initialize a model of θ 0 by pretraining on T adp 2: for t ← 1 to T do 3:Randomly sample images S ′ from some domain S i 4:Randomly sample images V ′ from T adp , the rest partT train = T adp -V ′ 5:Merge S ′ and T train by S ′ = T train + S ′ 6:θ ′ ← θ t-1 -α∇ θt-1 L(S ′ ) 7: if Eval(V ′ , θ ′ ) > 1 m m i=1 Q eval i then 8: θ t = θ ′ 9: if Q.size() == M then 10: Q eval .pop() 11: Q eval .push(Eval(V ′ , θ ′ )) 12:else 13:θ t = θ t-1 14: Fine-tune the model of θ T on T adpIn the final stage, we fine-tune the network parameter θ ′ obtained in the previous stage using the target domain training set and update the network parameter to θ T . The detail of our adaptive domain learning algorithm is illustrated in Algorithm 1To let our network better utilize the information from sensors that have different noise distributions, we need to adjust the feature space of different inputs. For a RAW data D captured by a CMOS sensor, we can model its noise by:EQUATIONwhere K is the system gain, N dep is the signal dependent noise and N indep is the signal independent noise. Based on this modeling, we propose a channel-wise modulation network to adjust the feature space by embedding two easy-to-access parameters, the sensor type and the ISO in our network. ISO is proportional to system gain K, while the sensor type can help the network know how to utilize the ISO to learn the signal-dependent noise N dep and recognize the signal-independent noise N indep .Given the one-hot encoding of the sensor type p ∈ R 1×n and the corresponding ISO s ∈ R 1×n (duplicate n times in the vector), our channel-wise modulation layer transfers the concatenated metadata (p, s) into a channel-wise scale γ and shifts β byEQUATIONEQUATIONwhere γ, β ∈ R 1×C , and MLP β and MLP γ are two four layer Multi-Layer Perceptrons. Let the feature map of the i-th convolution layer be F i ∈ R H×W ×C , we embed the sensor-specific data to F i by a channel-wise linear combination byEQUATIONNote that the type of the input metadata of our channel-wise modulation strategy is not fixed. The input concatenated vector can be extended as long as more meta information is provided along with the data.Blind2Unblind [41] Transfer learning [17] Our error map0 1 Figure 2:The error map of our method compares against state-of-the-art approaches. The first row is the result from the SIDD dataset, and the second row is the result from the SID dataset. We can see that our method is able to generate the image with smaller errors and less noise compared to previous work.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Answer: [Yes]• It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified. • For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates). • If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text. 
experimental setup: 
 Datasets. We evaluate the performance of our ADL and modulation on the dataset captured by smartphones in normal light conditions (SIDD dataset [2] ), and the dataset captured by DLSR cameras in extremely low light conditions (ELD [42] and SID [5] dataset). Compared to RAW data captured by smartphones, RAW data captured by DSLR cameras in extremely low light environments is more difficult to denoise because the noise distribution is more complicated and the noise level is larger. Moreover, the domain gap between the RAW data captured by different DSLR cameras is larger than the domain gap between the RAW data captured by smartphones.SIDD [2] is a popular RAW denoising dataset that contains 160 pairs of noisy and ground-truth RAW data from 5 different smartphone cameras (G4, GP, IP, N6, S6) of different scenes. ELD Dataset [42] contains RAW data captured by 3 different brands of DSLR cameras (Nikon, Canon, Sony) with different ISO and light factors, while the SID dataset captured RAW data using 2 different brands of DSLR cameras (Sony and Fuji). Note that the ELD dataset and SID dataset [5] are using the same DSLR camera (Sony A7S2). We only use the data captured by this camera from SID dataset to keep the domain gap between each set of domains. Besides, the input RAW data of the ELD and SID datasets are captured in extremely low light environments, while the ground truth is captured in normal light conditions. We follow the training strategy in [5] by multiplying a light factor by the input RAW data to keep the input RAW data and ground-truth RAW data in the same space.Baselines and training settings. To evaluate the performance of our framework, we compare our method against several baselines: the fundamental baselines pre-train and then fine-tune. Selfsupervise denoising methods: Blind2unblind [41] , ZS-N2N [26] , and DIP [38] . Meta transfer learning method MZSR [35] , Prabhakar et al. [34] and Kim et al. [17] (denoted as transfer learning). Traditional approach BM3D [7] . Calibration Free Method Led [16] . Note that we try our best to find all possible work that has the same goal as our method for the comprehensive baseline comparisons.Although the approach might be different, we make the experiment as fair as possible with proper settings.We compare the performance of our method against the above baselines by cross-validation on each sensor of all three datasets. In each experiment, we take one sensor as the target domain to represent the sensor with a very small number of data (around 20 pairs of data). The data from all other sensors will form the source domain, which represents the existing dataset. For the baselines, we only use the data from the target domain for the training of all self-supervised denoising methods, no data from the source domain is used since cross-domain data will reduce the overall performance of these methods. For fine-tuning, we first pre-train the model with the data from the source domain using U-net, then use the data from the target domain for fine-tuning. For DIP [38] We train our ADL in 300k iterations, using AdamW as the optimizer with a learning rate of 3 × 10 -3 .We quantitatively evaluate our method by comparing the PSNR against other baselines on the smartphone dataset SIDD and DSLR camera dataset ELD and SID. Note that we only report PSNR [11] because there are no other systematic evaluation metrics designed for RAW data. Other popular evaluation metrics like LPIPS [46] is not suitable for RAW data. We also present the SSIM [11] metrics in the supplementary material. The result is illustrated in Table 1 . Here, "fine-tuning" denotes the experiment training on the source domain and fine-tuning on the target domain. The camera name on the top row means that we take this sensor as the target domain while keeping the other sensors as the source domain. For example, "G4" in Table 1 means that this experiment takes G4 as the target domain and all data from the other four sensors, GP, IP, N6, and S6 will form the source domain dataset. From the table, we can see that our proposed method has the best performance on both the smartphone dataset and the DSLR dataset. To be specific, our method has 0.71dB and 0.39dB performance gains on average compared to MZSR [35] and transfer learning [17] baseline. The PSNR values in the table of the ELD and SID dataset are much lower than those in the table of the SIDD dataset because the ELD and SID datasets are captured in extremely low light environments: the noise level is higher, and the scenes are much more complicated than the SIDD dataset. Note that although the AIN module in Kim et al. [17] has a similar function and architecture to our modulation strategy, they can only estimate the noise level, which is very limited when the source domain contains data from many sensors. Our modulation strategy is more extensible and can somehow provide more hyper information(if provided in the dataset) and can let the network know the difference between the sensors.Qualitative Result Since the PSNR of the RAW denoising is relatively high and difficult to tell the difference from the human visual perspective, We evaluate our method qualitatively by comparing the error map against all three baselines. As illustrated in Figure 2 (a)The PSNR result of applying our ADL to fine-tune the existing noise calibration model. "FT" means naive fine-tuning, "Single" means only using data from the corresponding sensor to fine-tune, while "Multiple" means using data from all sensors in the corresponding dataset to fine-tune.Sensor Led [16] visualize, we transfer the ground truth into the sRGB domain using LibRAW to better demonstrate the color and details. For the restored RAW images, it is hard to use a well-designed ISP (Image Signal Processing) pipeline to obtain visually pleasant sRGB images for different sensors, we only demonstrate the result on RAW space. The error map is calculated in RAW space. We can see that the error between the ground truth and our output noise-free image is much smaller compared to all baseline methods. For more qualitative results, please refer to supplementary material.Noise calibration methods Although the noise calibration method is powerful, the calibrated model still does not include out-of-model noise such as fixed-pattern noise. Thus, fine-tuning using real data can further improve the performance of the model trained by those synthetic data. However, when the real data used for fine-tuning is scarce, the improvement is usually marginal. In such cases, we may want to utilize more data from different domains to further improve the result. In this section, we analyze the performance of applying our ADL to fine-tune the existing noise calibration model with data from multi-domain. We utilize the noise calibration method proposed by Zhang et al. [47] . The result is illustrated in Table 2 (a). Here "Single" means that only the data from that corresponding sensor is used for fine-tuning, while "Multiple" means that the data from all sensors in that dataset is used for fine-tuning. The result shows that when utilizing limited data from a single domain, the improvement of both naive fine-tuning and ADL is marginal. When we use more data from multiple domains, naive fine-tuning cannot learn useful information from various domains and thus leads to a drop in the final performance. However, our ADL can remove the harmful data, and ensemble the useful information from the different domains to help the training.Calibration-free methods Different from the noise calibration method, the calibration-free method Led [16] embed noise distribution from the simulated camera into the pre-train model. Although LED is a calibration-free method, the network also has no prior knowledge of the noise distribution of the target domain data during the pre-train stage. The large intensity distribution gap between the simulation cameras and the test set will lead to low performance. However, our ADL has prior knowledge of the target domain noise intensity distribution throughout the training process, which can gain similar robustness to the calibration method. As illustrated in Table 2 (b), we replace the synthetic data from the well-calibrated simulated camera in Led [16] with the data from the source domain(data from existing sensors), which is the same as our method in the experiments. Our method can outperform Led [16] in this case. This is because the performance of Led [16] highly depends on the prior knowledge of noise distribution learning from the simulated camera. Our method will not use data with a huge gap in intensity distribution between the source domain and the target domain.PMN [8] PMN [8] aims to overcome the bottleneck of the learnability in real RAW denoising by reforming the data. It can be generalized and applied to all real RAW image denoising methods and improve their performance including our ADL. As illustrated in [8] on the SIDD (G4, GP, IP, N6, and S6), ELD (Nikon, and Canon), and SID (Fuji and Sony) datasets. PMN can improve the performance of our ADL. The camera name on the top row means that we keep this camera as the target domain and use data from other cameras as the source domain. The domain gap in the deep learning model of other tasks is usually caused by the scene between the training data and test data. As for RAW image denoising, the difference in noise intensity among different sensors is more crucial. For example, if the noise intensity of two sensors is the same, then the noise model trained on one sensor can be generalized to the other. On the contrary, the model will fail if the noise intensity between these two sensors is very different. Based on this observation, we can say that the source domain data with similar noise intensity to the target domain data is useful, while the data with a large noise intensity difference is harmful. However, in cross-domain training, the harmful data has a more negative impact, because in cross-domain training, the model will tend to compromise different domains to reach the global minimum. As illustrated in Table 5 , we utilize the target domain data from the ELD dataset as the base set, and we build two Harmful datasets. Here "Harmful1" is synthesized by using the ground truth of the SIDD dataset that is captured in bright light conditions and naive Gaussian noise with noise level "harmful" σ = 30, and "Harmful2" set is the data pairs that have mis-alignment. For example, the input and ground truth are from different scenes, or the ground truth is black. In these cases, even though we add more data, the performance still drops. However, our ADL ignores the harmful data and always optimizes the model towards the noise intensity of the target domain.We conduct an ablation study on the effectiveness of our ADL and modulation strategy. The ablation study is conducted on SID and ELD datasets with the same training settings and configuration as in the previous section. We ablate over the strategies of our method by training models without applying the target domain pretraining, source domain adaptive learning, and modulation(including sensor type and ISO modulation). As illustrated in Table 4 , the result demonstrated that the target domain pretraining, source domain adaptive learning, ISO and sensor type modulation, and dynamic validation set all contribute to the final result. The source domain adaptive learning, which automatically evaluates whether data from the source domain is harmful or not, is the most crucial strategy in our framework. 5 : The PSNR and SSIM result of our ADL comparing to naive fine-tuning on base set and synthetic harmful dataset. Here "FT" means naive fine-tuning, "Base" means only using data from the corresponding sensor to fine-tune, while "Harmful1" means using naive Gaussian synthetic data with the different light conditions to fine-tune, and "Harmful2" means using the misaligned input and ground truth data to fine-tune.To evaluate how our method performs on different sizes of the target domain data, we conduct the ablation study on two sensors, G4 in the SIDD dataset and Sony in the SID dataset. Figure 3 demonstrates the PSNR against the size of the target domain dataset T adp compared to the fundamental baseline fine-tuning and our method without using the dynamic validation set strategy and diverse system gain selection strategy. It can be observed that our method can outperform fine-tuning when the size of the target domain data is extremely small. Besides, our dynamic validation set strategy also prevents the training from over-fitting when the target domain data is scarce. 
conclusion: 
 We have proposed a novel adaptive domain learning (ADL) scheme for cross-domain image denoising. We leverage the data from other sensors to help the training of the data from new sensors in a smart fashion: ADL removes harmful data and utilizes useful data from the source domain to improve the performance in the target domain. Our proposed modulation strategy provides extra camera-specific information, which helps differentiate the noise patterns of input data. We evaluate our method on smartphone and DSLR camera datasets, and the results demonstrate that our method outperforms state-of-the-art approaches in cross-domain image denoising. Moreover, we show that ADL can also be easily extended to image deblurring. We believe ADL is general and can be generalized to other cross-domain tasks, which can be further explored in the future.  
experiments compute resources: 
 Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?Answer: [Yes] Justification: We discuss the compute resources in the experiment and supplementary material.Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). 
code of ethics: 
 Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines? 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [NA] Justification: Our paper have no societal impact.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: Our paper has no such risks.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: We mentioned them explicitly.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. 
 
 Appendix: We present the detail of our network architecture. The detail design of our modulation strategy is illustrated in figure 6 . We utilize Dense-Unet as our backbone. We downsample the input feature map 4 times and the number of channels is set to 64, 128, 256, 512, and 1024 respectively. There is a total of 5 layers in each dense block. Leaky ReLU [30] is used as the activation function in each layer.We investigate how our network uses the data from the source domain to help the training of the target domain. We draw a heatmap to demonstrate the training process of our network. As illustrated in Figure 4 , we show the training process of sensor G4 on the SIDD dataset. The x-axis represents the epoch number during the training, and the y-axis represents the type of sensor. Each block in the figure represents the percentage of the data that is judged to have contributed to the training of the target domain in a particular epoch. For example, the value in the top left corner in Figure 4 is 0.75, which means that 75% of the data from sensor GP has contributed to the training of the target domain in the first epoch. It can be seen that at the beginning of the training, most of the data from the source domain contributed to the training of the target domain. As the epoch number and the size of the dynamic validation set increase, more and more data has no contribution to the training of the target domain, which means the useful information hidden in the source domain has been discovered.Another information we can observe from this heatmap is which sensor from the source domain has a much smaller domain gap compared to the target domain. As illustrated in Figure 4 (a), sensor S6 has only 15% of data that has contributed to the training of the target domain, and the percentage quickly reduces to zero in the following epoch. This indicates that there is a huge domain gap between sensor S6 and the target domain sensor G4, and the data from sensor S6 has almost no contribution to the training of G4. In contrast, the percentage of data usage from sensor IP keep at a high range as the epoch number increases, which indicates that sensor IP has the smallest domain gap to the target domain compared to the other sensors. Moreover, comparing 4 (a) and (b), we can tell that the data captured by the DSLR camera has a larger domain gap than the data captured by the smartphone. This is because the sensor of a DSLR camera is more complicated and has more noise sources than the sensor of a smartphone. Based on the observations, we can sample data with different percentages of the sensors from the source domain and shorten the training time.We demonstrate that our framework can also be applied to image deblurring and image dehazing to overcome the domain gap between the synthetic data and real data. For image deblurring, we synthesize blur kernel with three different sizes 21 × 21, 31 × 31, 41 × 41 by following the blur generation method proposed in [19] . We perform our experiment by training a Dense-Unet [10] (a)The SSIM result of applying our ADL to fine-tune the existing noise calibration model. Here "FT" means naive fine-tuning, "Single" means only using data from the corresponding sensor to fine-tune, while "Multiple" means using data from all sensors in the corresponding dataset to fine-tune.Sensor Led [16] a popular deblurring dataset Gopro [28] . We apply synthetic blur kernels on the ground-truth clear images and form 320 training image pairs as the source domain, and randomly select other 8 training pairs in the Gopro dataset as the target domain training set T train and 16 images to form the target domain validation set T val . The other 50 images are used as the test set, with no overlapping on the scene between the validation set, training set, and test set. For image dehazing, we utilize the realworld indoor dehazing dataset, I-haze [3] as the target domain and the synthetic dataset RESIDE [23] as the source domain. We employ the state-of-the-art dehazing method Dehazeformer [36] as the backbone. We evaluate the performance of our result by comparing it with the fine-tuning pipeline using two evaluation metrics, PSNR and SSIM. The quantitative results are illustrated in Table 6 . Here, "direct" means training on synthetic data and directly testing on real data without any fine-tuning. The quantitative result demonstrates that our method has higher PSNR and SSIM than fine-tuning.We demonstrate additional qualitative results of our method against Blind2unblind [41] and transfer [17] on smartphone dataset SIDD [2] and DLSR datasets ELD [42] and SID [5] , as illustrated in Figure 5 . We can see that the error between the ground truth and our output noise-free image is much smaller compared to all baseline methods.We also demonstrate the qualitative result of our method compared to the fundamental baseline fine-tuning on image denoising and image dehazing in figure 7 . Our method is able to recover images that are much cleaner and finer.[41][17] Ours0 1Figure 5 : The error map of our method compares against state-of-the-art approaches. The first row is the result from the SIDD dataset, and the second row is the result from the SID dataset. We can see that our method is able to generate the image with smaller errors and less noise compared to previous work. We demonstrate the SSIM value for section 4.3 and section 4.5. The results are demonstrated in Table 7 and Table 8 .NeurIPS Paper ChecklistJustification: We are with NeurIPS code of ethics in every aspect.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 


Decision: This paper proposes a novel Adaptive Domain Learning (ADL) method for cross-domain RAW image denoising, leveraging both source and limited target domain data to enhance performance. While the method demonstrates superior performance across several public datasets and is technically sound, there are concerns regarding its novelty and comprehensive comparison with existing methods. The paper received mixed reviews regarding its contribution and presentation. The authors provided a thorough response addressing the concerns raised by the reviewers, leading to a final average score of 5.25. The AC carefully reviewed the paper and all the reviewers' comments. Considering the good performance of the proposed framework on benchmark datasets and its potential impact for the community, the AC agrees to accept the paper and strongly recommends incorporating the content from the rebuttal into the final version.

Reviews: review: paper_summary: paper_summary: This paper addresses the challenge of cross-domain RAW image denoising due to varying noise patterns from different camera sensors (bit depths, color). The authors propose an Adaptive Domain Learning (ADL) scheme that leverages existing data from various sensors and a small amount of data from a new sensor to train an effective denoising model. The ADL scheme selectively uses beneficial data from the source domain while discarding harmful data, and introduces a modulation module to incorporate sensor-specific information. The proposed model outperforms prior methods on public datasets, demonstrating state-of-the-art performance with limited target domain data.
strength_weakness: strength_weakness: 1. Compared with RAW2RAW mapping: There are existing methods for raw-to-raw mapping, such as "Semi-Supervised Raw-to-Raw Mapping." It would be interesting to explore whether source domain raw images can be mapped to the target domain and then used for training based on the noise model. This could potentially provide a more direct comparison and integration of data from different domains.

2. Data Requirements: The authors used over 20 captured raw images, which is significantly more than the "Two pairs" approach used in LED. With these raw images, could a noise model be used to simulate noise instead? Is there a comparison available? This would help to understand whether the additional raw images provide a significant advantage over using simulated noise from a noise model.
questions: questions: The current ADL settings require more data compared to the LED method. In Table 2b, it is not clear how the LED method was specifically trained—whether it involved fine-tuning with limited target domain data. Additionally, comparing the performance of ADL with LED under the "Two pairs" condition would provide valuable insights into the effectiveness and efficiency of both approaches. This could help understand the trade-offs between data requirements and performance outcomes.
limitations: limitations: A more explicit discussion on the potential impact of extremely limited target domain data (like two pairs) on the model's performance would provide additional clarity.
paper_summary: paper_summary: This paper address the cross-domain image denoising problem with a small number of target domain training samples. The authors propose an adaptive domain learning (ADL) strategy that dynamically selects useful training samples from both source and target domains to improve performance.  Additionally, the paper leverages channel-wise modulation layer to model different noise distribution caused by sensor type and ISO. Experimental results showcase the advantages of the proposed method.
strength_weakness: strength_weakness: 1. **Lacking Clarity on Definitions and Practical Benefits:**  The paper actually discusses the few-shot domain adaptation problem but fails to clearly define the specific settings and scenarios being addressed. Also, this paper does not clearly specify the conditions under which the proposed ADL strategy provides the most benefit. For example, how does the performance of ADL compare with simpler strategies like mixing the source and target domain data as the number of target samples increases? What is the threshold number of samples where ADL proves advantageous? Is this number large enough so that collecting target domain data larger than the threshold can be seen as impractical.

2. **Lack of comparison to adequate baselines**: The paper would benefit from a more comprehensive comparison with existing  few-shot domain adaptation method in low level vision [1,2]. Comparing the proposed method with approaches in different settings, such as self-supervised learning, may not be fair or relevant. It would be more informative and meaningful to compare against methods with similar objectives and constraints.

3. **Lack of Task-Specific Design and Motivation**: The proposed ADL strategy is not specifically tailored for the denoising task, which reduces its effectiveness and relevance. Additionally, the paper does not clearly explain the motivation behind combining ADL with the modulation module. The paper would benefit from a clearer explanation of how these components interact and complement each other.

4. **Presentation Issues:** There are inconsistencies in the presentation of results, such as the caption in Table 3 mentioning "Mod" (modulation) without corresponding columns in the table.  

[1] K. Prabhakar, Vishal Vinod, et al. "Few-Shot Domain Adaptation for Low Light RAW Image Enhancement." British Machine Vision Conference 2021.

[2] Bo Jiang, Yao Lu, et al. "Few-Shot Learning for Image Denoising." 2023.
questions: questions: 1. As the size of the target domain dataset increases, how does the performance of the ADL strategy compare with simpler strategies such as mixing the source and target domain datasets? Is there a threshold number of target samples beyond which ADL does not offer significant benefits?

2. What are the computational overheads associated with the dynamic validation set used in the ADL strategy? Is the process efficient enough?
limitations: limitations: Limitations are currently not discussed in this paper, which may not be so adequate.
paper_summary: paper_summary: This paper proposes a novel adaptive domain learning (ADL) approach for cross-domain RAW image denoising problem. The ADL scheme allows to train models for target domains with limited data by leveraging data from other source domains. The harmful data from the source domain is automatically removing during training. Authors also propose a new modulation approach to encode the sensor type and ISO to help the network to adapt to different noise distributions. The extensive experimental results demonstrate the effectiveness of the proposed method.
strength_weakness: strength_weakness: 1. The qualitative results only show the error maps. Actual denoised images (or patches) in the sRGB should be provided for better visual comparison of the methods. In addition, quantitative analysis with metrics such as LPIPS should be done on the sRGB methods
2. It seems that the already trained network cannot easily adapted to new data. The method encodes the sensor type as a one-hot vector. Therefore, the new source domain data (new sensor) cannot be incorporated to fine-tune the network and the training should be done from scratch.
questions: questions: Have you tried other metrics than PSNR for dynamic validation?
limitations: limitations: None
paper_summary: paper_summary: This paper introduce adaptive domain learning (ADL) for cross domain raw image denoising problem. The author also introduce a module to embed sensor information into the network. Experiments on public datasets with various smartphone and DSLR cameras demonstrate proposed model outperforms prior work on cross-domain image denoising with a small amount of image data from target domain.
strength_weakness: strength_weakness: 1. Novelty Issue: Although I am reluctant to mention this, the proposed pipeline is quite common and widely available in domain adaptation research. The main credit is that the author applied it to raw image denoising. However, it may require more innovative insights to meet the standards of the NeurIPS community.
2. Embedding sensor information to help the network distinguish the domain is also a common technique used in many research works. I don't believe it can be considered a significant contribution.
3. In line 210, 20 pairs of data in the target domain is actually not a very small number for a raw image dataset. I am curious about the significance of using these target domain data and how it can contribute to real advancements in raw image denoising applications.
4. The author should fine-tune PMN (the recent SOTA method) with the target domain data for a more thorough comparison.
5. Minor Issues: There is a misalignment in Table 2. It would be better to note SSID, ELD, and SID in Table 1 instead of using camera names only.

PMN (Learnability Enhancement for Low-Light Raw Image Denoising: A Data Perspective)
questions: questions: Primarily comparing PSNR with self-supervised denoising methods is somewhat unfair because the primary advantage of these methods is not high PSNR. Instead, why not structure the experimental section to compare with state-of-the-art performance methods?
limitations: limitations: No

Comments: comment: Thank you very much for the score improvement and your constructive feedback. We will further polish the paper in the final revision.
Hello,

Thank you for your response. 

After careful consideration of all the rebuttals and available reviews/discussions, I have decided to maintain my original score. I believe this to be the most appropriate reflection of my overall impression and evaluation of the paper.

Best regards,

Reviewer G2YE
Thank you for your clarification. I am willing to raise my score.
We value your feedback and are eager to address any further questions or concerns you may have. If you have had a chance to review our response and have additional thoughts, we would greatly appreciate your input.
We value your feedback and are eager to address any further questions or concerns you may have. If you have had a chance to review our response and have additional thoughts, we would greatly appreciate your input.
We value your feedback and are eager to address any further questions or concerns you may have. If you have had a chance to review our response and have additional thoughts, we would greatly appreciate your input.
We thank you for affirming and approving the application of ADL to RAW denoising and other clinical aspects and for your valuable suggestions. We will improve our paper further in the revision. Besides, we would really appreciate it if you could point out the specific work that a similar pipeline is proposed in domain adaptation research so that we can learn from it and include it in our literature review. We were eager to address any further questions or concerns you may have. If you have had a chance to review our response and have additional thoughts, we would greatly appreciate your input.
Thank you for your reply. We would like to further discuss the question with you as below:

***- Unfair comparison with the baselines***

We try our best to find all possible work that has the ***same goal*** as our method for the ***comprehensive*** of the baselines. We compare our ADL with the source-free domain adaptation and self-supervised denoising method just for reference, and we will emphasize this point in our revision. The source domain data of our ADL can be easily accessed. The source domain data could be synthetic data or data from any other sensors. Due to the robustness of our ADL, the harmful data from the source domain will be automatically removed. For the self-supervised denoising method, as illustrated in section 4.1, line 211 in the main paper, in our experiment, the self-supervised denoising method did not use any source domain data and used all the data from the target domain data, which is the same as their own setting.
Thanks to the authors for their rebuttal and extra experiment. However, some of my questions have not been properly answered:

**Q1**: Regarding the setting, this approach requires source data in domain adaptation. However, source data is not available in many cases (source-free domain adaptation). If those compared methods are source-free, the comparison may not be fair.

**Q2**: We are happy to see the additional results. However, the authors did not address my concern: "Comparing the proposed method with self-supervised learning may not be fair," because SSL does not require target domain data with ground truth.
Thank you for your reply. We would like to further discuss the question with you as below:

***- Data requirement and PMN***

Collecting paired real data for RAW denoising requires considerable human labor and equipment support. Collecting 160 pairs of real data requires a lot more effort than collecting 20 pairs of real data. The PMN can reach better performance with less data because they overcome the bottleneck of learnability in real RAW denoising. Moreover, PMN is a data reformation method, their method can be applied to any training strategy and network architecture and has similar performance improvement. We applied the DSC strategy proposed in PMN to our ADL. The PSNR on the Sony sensor improved by 0.41dB, and the PSNR on the Nikon sensor improved by 0.38 dB.

***- Comparison with self-supervised denoising is not fair***

We try our best to find all possible work that has the ***same goal*** as our method for the ***comprehensive*** of the baselines.  We compare our ADL with the self-supervised denoising method just for reference. Moreover, as illustrated in section 4.1, line 211 in the main paper, in our experiment,  the self-supervised denoising method did not use any source domain data and used all the data from the target domain data, which is the same as their own setting. Compared to our ADL, they use a lot more data and do not involve any cross-domain learning.

***- The practical applications of ADL***

Here are two examples of practical applications for our ADL in real-world scenarios:
1. As the iteration of the smartphone and DLSR cameras become faster and faster in recent years, collecting a large RAW denoising dataset for each of these sensors to build noise calibration models or single domain supervised learning is very labor-demanding. Moreover, the dataset for specific sensors cannot be used in the training of sensors in the future and therefore causes a waste of resources. With the help of our ADL, we only have to collect a small dataset with around 20 pairs of data. Besides, the dataset we collect for old sensors can also be reused and serve as the source domain to help with the training of the new sensors.
2. Collecting paired real RAW images is difficult and may collect bad data, such as misalignment, without the help of professional equipment, such as robot arms. Moreover, synthetic data may also have outliers, which are very different from the noise distributions of real-world data. These bad data are hard to detect and can lead to significant performance drops in supervised learning. As illustrated in section 4.4, the robustness of our ADL can avoid the performance drop brought by these bad data.
We thank you for you valuable feedback. We would like to further discuss the problem with you.
You can set the one-hot encoding vector size to be some large number, for example, greater than 100. We believe that 100 is enough for real practice. Very large number of source domains barely appear in real scenarios.
Hello,

Thank you for your response. I would like to reply to the contents one by one and discuss with the author as follow:

- ***Novelty issue***

As in my initial review, I expressed concerns that the proposed pipeline is quite common in domain adaptation research. While I acknowledge the main credit to the authors for applying these methods to the raw image denoising field, I believe the paper may require more innovative insights to inspire readers, as is expected from publications at NeurIPS. For me, novelty is not a decisive factor for rejection if other aspects are excellent, in other words, I believe there is still much room for improvement to elevate the paper's quality to the acceptance standard.

 - ***Data requirement and PMN***

As presented in the PMN paper, they obtained significantly better results than SFRN, according to my previous test on PMN's code, they utilized about 160 data pairs. This context led me to comment that "20 pairs of data in the target domain is actually not a very small number for a raw image dataset." However, based on Figure 3 and the author's clarifications, the performance with just 6 paired data is substantially lower than with 26 paired data (around a 3 dB difference), which is significant. I am not yet convinced by the author's rebuttal on this point and would appreciate further explanation.

 - ***Comparison with self-supervised denoising is not fair***

Self-supervised denoising methods typically use no clean data (no supervision), and unsupervised denoising uses unpaired data. In contrast, the method proposed by the authors not only requires paired data from the source domain for supervision but also from the target domain, which should naturally result in better PSNR/SSIM scores. This is why I question the fairness of using these as comparable baselines for performance comparisons. Additionally, the results presented in the tables do not show a significant performance advantage over self-supervised or unsupervised methods, despite with the support of substantial data and supervision signals.

- ***Additionally***

I am curious about the significance of using these target domain data and how it can contribute to real advancements in raw image denoising applications. Does this domain adaptation strategy have any practical applications in real-world scenarios for raw image denoising?

Looking forward to discussion with you.

Best regards,

Reviewer G2YE
I appreciate the authors' efforts in addressing my comments and providing additional experimental results for the SSIM metric. However, I am still not convinced that the method is flexible for fine-tuning new source domain data. Even though the one-hot encoding vector size can be set higher than the number of sensor types, we can still have more new sensor types in the future that exceed the vector size