Image-free Pre-training for Low-Level Vision

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-03-11 05:43:19

Abstract: The constrained data scale in low-level vision often induces the demon overfitting hazard for restoration networks, necessitating the adoption of the pre-training paradigm. Mirroring the success of the high-level pre-training approaches, recent methods in the low-level community aim to derive general visual representation from extensive data with synthesized degradation. In this paper, we propose a new perspective beyond the data-driven image pre-training paradigm for low-level vision, building upon the following examination. First, unlike the semantic extraction prevalent in high-level vision tasks, low-level vision primarily focuses on the continuous and content-agnostic pixel-level regression, indicating that the diversified image contents inherent in large-scale data are potentially unnecessary for low-level vision pre-training. Secondary, considering the low-level degradations are highly relevant to the frequency spectrum, we discern that the low-level pre-training paradigm can be implemented in the Fourier space with fostered degradation sensibility. Therefore, we develop an Image-Free Pre-training (IFP) paradigm, a novel low-level pre-training approach with necessity of single randomly sampled Gaussian noise image, streamlining complicated data collection and synthesis procedure. The principle of the IFP involves reconstructing the original Gaussian noise from the randomly perturbed counterpart with partially masked spectrum band, facilitating the capability for robust spectrum representation extraction in response to the capricious downstream degradations. Extensive experiments demonstrate the significant improvements brought by the IFP paradigm to various downstream tasks, such as 1.31dB performance boost in low-light enhancement for Restormer, and improvements of 1.2dB in deblurring and 2.42dB in deraining for Uformer.

Main paper: 
 introduction: 
 In order to alleviate the overfitting problem and bring further improvement when data is limited, image pre-training has received widespread attention in the field of computer vision. However, unlike the remarkable success in high-level vision, the application of pre-training in low-level vision remains underdeveloped. In mainstream opinion, this is attributed to the lack of large-scale low-level vision datasets comparable to ImageNet [8] in the community. Adhering to the perspective that the scale of data plays a decisive role in the effectiveness of pre-training, pioneer works [4, 7, 23] attempt to synthesize paired degradation data on ImageNet [8] through sundry data corruption pipelines and perform corresponding image restoration tasks. These data-driven pre-training methods meet the challenge of introducing target degradation into the pre-training phase. This requirement hinders their applicability to high-cost tasks where the synthesis of degradation is a challenge. Recently, the emergence of DegAE [28] makes it possible for high-cost tasks to benefit from pre-training. It no longer focuses on data acquisition but designs a new pre-training paradigm that learns a general low-level vision representation by transferring the degradation between images. The new paradigm significantly improves the downstream image restoration tasks after fine-tuning, even if the target degradation does not appear in the pre-training stage. However, DegAE [28] does not deviate from the traditional pre-training paradigm of learning from data. Because it still relies on The pre-training effect of different image types adopted in IFP is similar. Random Init denotes the baseline without IFP. Real images are randomly selected from ImageNet [8] . More details are in Table 5 .manually generating degradation data pairs exceeding 10G. Moreover, it requires an additional degradation information injection module, which necessitates significant computational resources and may potentially cause task bias in downstream applications. Different from previous data-driven works, we propose a dataefficient and time-saving low-level pre-training paradigm called Image-Free Pre-training (IFP). As shown in Fig. 1 , IFP requires only one randomly generated Gaussian noise image. However, it can enhance the performance of Restormer [45] on motion deblurring task by 0.66 dB. Specifically, IFP performs Masked Image Modeling (MIM) [14, 37, 41] in the frequency domain, enabling the model to deduce the information of the masked bands from the known bands. Such design is based on the following observations:First, we find that different degradation types show different disturbance patterns to the spectrum in the frequency domain. As illustrated in Fig. 2 (a), blurring predominantly eliminates highfrequency details, rain primarily introduces low-frequency information, and low-light augments low-frequency while diminishing high-frequency. Therefore, image restoration can be interpreted as the process of reconstructing any destroyed frequency band by using the unchanged ones. Frequency domain MIM enables models to learn occlusion invariance on the spectrum, thereby making it suitable as the pre-training task for low-level vision.Second, unlike semantic extraction in high-level vision, the core intention in low-level pre-training lies in the continuous pixel-level regression ability, which is principally content-agnostic. Inspired by the generative models that takes Gaussian noise to synthesis various content images, we take the Gaussian noise as input and simulate different frequency disturbances by random frequency masking to investigate content-agnostic pre-training. It can be observed from Fig. 2(b ) that albeit the model has only seen a randomly generated Gaussian noise image during pre-training, it still gets a strong image reconstruction ability after frequency MIM pretraining. Moreover, we show the influence of image types on the IFP effect in Fig. 2(c ). With applying our method on real images from ImageNet [8] dataset, the performance increase is consistent compared with random noise. These findings imply that the effectiveness of IFP does not stem from the image and the diversity of image content inherent in large-scale data is not essential.In conclusion, IFP is a new attempt in the field of low-level vision pre-training. To our knowledge, it is the first method that deviates from the traditional pre-training paradigm of learning from data. It means that IFP is independent on downstream tasks, and the pre-trained representations are general and transferable. Moreover, only one Gaussian noise image means the computational cost comes to be extremely low. In summary, our contributions include: During pre-training, 𝐼 𝑜 is converted to the frequency domain by FFT to obtain its amplitude spectrum 𝐴 𝑜 and phase spectrum 𝑃 𝑜 . The Frequency-aware Masking (FAM) strategy simulate random degradation effect on different frequencies. Specifically, the amplitude spectrum is randomly masked and the phase spectrum is kept unchanged. After iFFT, the noise whose amplitude information has been corrupted is sent to the encoder (e.g., ViT, CNN) and the original image is reconstructed using a lightweight decoder. At the downstream task fine-tuning stage, we initialize the model with encoder parameters obtained during the pre-training stage, replace the decoder with a simple convolutional layer, and then fine-tune the whole network with data from a specific low-level task. 
related work 2.1 image restoration: 
 The purpose of image restoration is to reconstruct high quality natural images from the observed degraded images (e.g. noise, blur, rain drops) by removing degradations. Early methods typically focuses on incorporating various natural image priors along with handcrafted features for specific degradation removal tasks [1, 30] . Recently, deep learning based methods have made compelling progress on various image restoration tasks. For instance, SRCNN [9] introduces an end-to-end convolutional neural network for superresolution task. Zhang et al. [50] proposes the first deep learning method DnCNN for denoising task, DehazeNet [3] and MSCNN [34] for image dehazing task, DeblurGAN [20] and Deblurgan-v2 [21] for image motion deblurring task. In addition, with the advent of ViT [10] , the transformer, due to its excellent performance in modeling global dependencies and superior adaptability to the input content, has been introduced into vision tasks. For instance, IPT [4] , Uformer [36] , SwinIR [26] and Restormer [45] are notable examples of such methods that have significantly contributed to advancements in the image restoration area.Pre-training can help the model find a good initialization, which is an effective way to alleviate model overfitting problem in datascarce scenarios, but there is little exploration in the low-level vision field. Most low-level vision pre-training methods [4, 7, 23] concentrate on tasks where downstream training data is easy to obtain. The pre-training phase can be supplanted by adding downstream training data, thereby undermining the true purpose of pre-training. Based on the challenge of procuring training data for downstream tasks, DegAE [28] divides low-level tasks into lowcost and high-cost categories. Focusing on only high-cost tasks, DegAE [28] designs a pretext task centered around image degradation transfer, taking into account the specific characteristics of low-level tasks. The downstream tasks are significantly improved with relatively less pre-training data. However, it still requires over 10G of data that has undergone a series of manual preprocessing, along with additional structural modules. In contrast, our proposed IFP does not require any preprocessing, and the data requirement is only a randomly generated Gaussian noise image, which means the degradation invariance is obtained by design rather than data. 
method: 
 The overall architecture of IFP is shown in Fig. 3 , which uses the target model as the encoder and a simple CNN as the decoder. One randomly generated Gaussian noise image with a random amplitude spectrum mask is used as input to perform image reconstruction, and the lost amplitude spectrum information is attempted to be restored from the generated image. Despite performing masking in the frequency domain, we still take the converted spatial images as input such that the model would not suffer an input domain gap between pre-training and fine-tuning. In the forthcoming sections, we will elaborate on how to generate the noise input in Sec. Previous research [19] has shown that the benefit from MIM is free of content information in the spatial domain. In Sec. 4.5, we confirm that IFP similarly exhibits this characteristic in the frequency domain. Therefore, for the input, we select Gaussian noise I (x) ∽ N (𝜇, 𝛿 2 ) with 𝜇 = 0 and 𝛿 = 1 as standard to simplify natural image representation. The theoretical foundation of this operation is rooted in the fact that its statistical property aligns with the Gray-World Color Constancy Hypothesis [2] , which posits that for an image with large color variations, the averages of the three RGB components converge to the same gray value K. In the pretraining stage, we only initialize a Gaussian noise image randomly at the beginning of the process.Our Frequency-aware Masking (FAM) is performed on the amplitude spectrum for the following reasons: (1) The effect of degradation on the original image appears as the random destruction of low or high frequency information in Fourier space. (2) Phase represents the content information and spatial details while amplitude represents the global statistical properties. We want to avoid the influence of specific image content when designing the pretext task. The FAM operation is simple. Specifically, since the destruction of degradation on the frequency spectrum is center-symmetric and discrete, we don't divide the input image into patches like MAE [14] , nor do we directly use a high-pass or low-pass filter to mask the continuous frequency band like [40] , but regarding the amplitude spectrum of the image as a combination of concentric squares with unit width. Then, following MAE [14] , we randomly sample these concentric squares at a 25% ratio and follow a uniform distribution.Algorithm 1 provides the pseudo-code to show the details of FAM. Such operation can largely eliminate the information redundancy caused by the continuity of frequency distribution.Algorithm 1 Pseudocode of FAM operation in IFP. We use the model that needs to be pre-trained as the encoder of the IFP model during pre-training phase, which does not have a fixed form. Since there is no single architecture that works best on all low-level vision tasks, in this paper we employ two state-of-the-art Transformer architectures in low-level vision -Ufomer [36] and Restormer [45] as our encoder for multiple tasks. For the model that we use as an encoder, nothing else is done except to modify the channel number of the last convolutional layer from 3 to 64 to accommodate subsequent decoder. This operation ensures that the original architecture of the model is largely preserved, allowing it to retain its inherent strengths while benefiting from the additional learning provided by the pre-training process. Since the decoder is only used in the pre-training stage to perform image reconstruction tasks, its architecture can be flexibly designed in a way that is independent of the encoder design. As long as its input is compatible with the output of the encoder and simple enough, it can be in any form. Follow DegAE [28] , we simply take a CNN with four residual blocks [15] as our decoder and it will be replaced by a single convolution layer in the downstream task fine-tuning stage.Our IFP reconstructs the input by predicting the missing values on the amplitude spectrum. Specifically, the loss function consists of amplitude spectrum reconstruction loss and pixel-wise reconstruction loss, which can be formulated as:EQUATIONwhere 𝐼 𝑜 denotes the original Gaussian noise image, 𝐼 𝑟 is the reconstructed image, 𝜆 is the balanced weight. 𝑀 represents the binary masking map, ⊙ is the Hadamard product between matrices, 𝐴 𝑜 and 𝐴 𝑟 are the amplitude spectra corresponding to 𝐼 𝑜 and 𝐼 𝑟 respectively. The first part of the formula is the pixel-wise reconstruction loss, while the second part is the amplitude spectrum reconstruction loss. We follow the settings in MAE [14] , and only calculate the masked region in practical application.IFP is the first attempt to exploit task-agnostic data for low-level vision pre-training. We argue that by learning occlusion invariance in the frequency domain, IFP enables the model to extract robust spectral representation regardless of whether it encounters a clean or degraded image. To substantiate this claim, we take Restormer [45] as the backbone and perform the visual analysis of its output features before and after pre-training. As shown in Fig. 4 , with the help of IFP pretraining, the gap between different degradations is reduced, proving advantageous in response to the unpredictable disturbances caused by downstream degradations. 
experiment: 
 We conduct extensive experiments to show the effectiveness of our proposed IFP method. Specifically, We evaluate the proposed IFP pre-training paradigm on several high-cost low-level tasks, including image deraining, image deblurring, and low-light image enhancement. In the following sections, we explain the implementation details and show the IFP's performance on different tasks. Due to the space limit, we show the experimental results for the low-cost task in the supplementary file.We implement IFP on a single NVIDIA Geforce RTX 3090 GPU. For pre-training, the learning rate is initialized as 2e-4. Adam optimizer [18] with 𝛽 1 = 0.9 and 𝛽 2 = 0.99 is adopted. The Gaussian noise input selected from I (x) ∽ N (0, 1) is in the shape of 224×224.The batch size is one and a total of 30K iterations are executed.After pre-training, we initialize the model with the parameters of the encoder from the first stage and then fine-tune it on specific downstream datasets. We employ the same training policy across different backbones for fairness and convenience. We adopt Rain13K dataset for fine-tuning, which includes 13,712 clean-rain image pairs collected from multiple datasets [11, 24, 25, 29, 44] and is newly-adopted in [6, 36, 45, 46] . For testing, we adopt Rain100L [43] , Rain100H [43] , Test100 [49] , Test1200 [48] and Test2800 [12] datasets. The PSNR and SSIM values is calculated on the Y channel in the YCbCr color space. We compare our IFP with state-of-the-art methods, including five derain methods: De-rainNet [11] , RESCAN [24] ,PreNet [33] , MSPFN [16] ,MPRNet [46] , and one state-of-the-art pre-training method: DegAE [28] . The quantitative results are shown in Table 1 and the visual results are shown in Fig. 5 . It shows that IFP helps improve the performance of Restormer and Uformer on all five datasets. The improvement is the most obvious on Rain100H dataset, with Uformer producing an improvement of 2.42dB and Restormer producing a performance improvement of 0.84dB. From the visualization results in Fig. 6 , IFP pre-training can help the target model to remove rain more thoroughly to produce better visual effects.For image deblurring, we adopt GoPro [31] dataset for training and testing. It consists of 3214 pairs of blurred and clean images extracted from 33 sequences. The blurred images are generated by averaging varying number (7-13) of successive latent frames to produce varied blur. Follow previous methods [31, 47] , we use 2103 image pairs for training and the remaining 1111 pairs for testing. the compared methods that we select are DeblurGAN [20] , DeblurGAN-v2 [21] , SRN [35] , SPAIR [32] , HINet [6] , MPRNet [46] , IPT [4] , NAFNet [5] and DegAE [28] . As can be seen from the quantitative results of motion deblur in Table 2 , IFP has brought improvements of 0.60 dB and 1.02 dB to Restormer and Uformer on GoPro dataset, respectively. Although neither Restormer nor Uformer achieves the best performance on the GoPro dataset, it does not affect the validation of our method. One can achieve better results by using a higher-performance backbone. Moreover, as illustrated in Fig. 6 , with application of IFP pre-training, the model becomes more proficient in eliminating blur from the image and the image restored is closer to the target with more details.IFP can also bring considerable improvement on low-light enhancement task. We use the LOL dataset [38] as the fine-tuning dataset, which contains 500 low/normal-light image pairs. In the training, we merely employ 450 image pairs, and no synthetic images are used. We report the results of IFP along with eight state-of-the-art methods TAPE [27] , AirNet [22] , PairLIE [13] , EnlightenGAN [17] , STAR [42] , URetinexNet [39] , SwinIR [26] and DegAE [28] . As can be seen from the results in Table 3 and Fig. 5 , with the introduction of IFP, the performance of the model has been significantly improved. Specifically, Uformer and Restormer achieve 0.94 dB and 1.31dB improvements on the LOL dataset, respectively. What's 29.55 0.934 SRN [35] 30.26 0.934 SPAIR [32] 32.06 0.953 MPRNet [46] 32.66 0.959 DegAE(Restormer) [28] 32.67 0.928 HINet [6] 32.71 0.959 NAFNet [5] 32 We present the ablation experiments on image deraining task with Restormer [45] as the backbone to verify the effectiveness of our method. Basically, we explore the impact of pre-training iterations, the number of Gaussian noise images, and the masking ratio on the proposed IFP method. Additionally, to verify the data-agnostic characteristic of the IFP pre-training method, we conducted comparative experiments using different real images and Gaussian noise images. We still train on the Rain13K dataset [11, 24, 25, 29, 44] and test on five benchmark datasets: Rain100L [43] , Rain100H [43] , [26] 17.81 0.723 AirNet [22] 18.18 0.735 TAPE [27] 18.97 0.621 STAR [42] 19.30 0.579 PairLIE [13] 19.51 0.736 URetinexNet [39] 19.84 0.826 DegAE(Restormer) [28] 23 Test100 [49] , Test1200 [48] and Test2800 [12] . The average performance across these test datasets is used as the evaluation criterion for the models' performance.Type of pre-training input. In order to verify whether the performance of the IFP pre-training method is affected by the input image content, we conduct a total of 10 repeated experiments on the image deraining task using 5 randomly generated Gaussian noise images and 5 real images randomly selected from ImageNet [8] . The final results are shown in Table 5 . It can be observed that the difference between pure Gaussian noise images and real images in terms of downstream fine-tuning improvement is very small. The performance of Gaussian noise images is even more stable than that of real images. This result indicates that the effectiveness of the IFP method does not stem from the content of the image. Such data-agnostic invariance indicates that IFP pre-training method is We explore the influence of pre-training degree on the downstream deraining task. As shown in Table 4 , we are surprised to find that when the number of pretraining iterations reaches about 10K, there is a certain performance improvement. When the number of iterations reaches about 30K, the pre-training achieves the ideal effect, and the performance improvement reaches 0.47dB. When the number of iterations reaches about 40K, it can still significantly improve the model's performance, but the effect slightly decreases. We hypothesize that this is because overfitting tends to occur when performing MIM training on a very small training set, as the network can easily "remember" the unique training image. Masking ratio. We also explore the influence of the masking ratio on IFP, and the results are shown in Fig. 7 . It can be observed that the effect of IFP is closely related to the masking ratio. A relatively high masking ratio can bring a good initialization to downstream tasks. Similar to the spatial MIM method [14] , the IFP method works best when the masking ratio reaches 75%.Number of noise images. The outstanding advantage of our method is that only one noise image is used during pre-training.Here we explore the effect of increasing the number of training inputs on model performance, and the results are shown in Table 6 . It suggests that an increase in the quantity of noise images can actually hinder the progress of pre-training. We hypothesize that this phenomenon is due to the fact that the efficacy of the IFP is not intrinsically derived from the data and one image is enough for IFP to learn. Consequently, an increase in data complexity merely introduces negative interference to the model, amplifies the training challenge, thus leads to the reduction in the effectiveness of IFP. 
conclusion: 
 Due to the constrained data scale related to many degradation restoration problems and the black-box nature of low-level model learning process, pre-training has not been well developed in the low-level vision field. In this paper, we propose a bold idea when thinking about solving the problem: can pre-training be done without using any task-related data? We analyze the impact of different degradation on images from the frequency domain perspective, and then introduce a new low-level self-supervised pre-training paradigm, called Image-Free Pre-training (IFP). The final results provide many surprises, and experimental results show that even if only one randomly generated Gaussian noise image is used for IFP pre-training, it can still bring significant improvement to a wide range of downstream tasks. This groundbreaking work forces us to rethink the complexity of image features in low-level vision problems, and we hope that our work will inspire research on the interpretability of low-level vision networks.  
 
 Appendix:  


Decision: This paper presented a new image-free pre-training (IFP) scheme for low-level vision/image restoration tasks. The paper received review comments from four reviewers. The strengths of this work include: the new and interesting idea of image-free pre-training, and extensive experiments including ablation study to validate the main idea and claims.
The key idea of using pure noise to train models to perform downstream tasks would be a good contribution to the community and may inspire follow-up works. However, there are some limitations that could be addressed to make the paper stronger: Insufficient clarifications for some claims and statements, lack of sufficient details in some descriptions and illustrations, real-world applications, and the two-stage design. Some of them were addressed in the rebuttal, and the authors are suggested to include them in the final version of the paper. All four reviewers recommended positive scores, with three weak accept and one borderline accept.

Overall, the contribution and strengths of this paper overwhelm the limitations, and this work would be of interest to a large group of audience in ACM MM. As a result, the AC recommends Accept.

Reviews: review: paper_summary: paper_summary: This paper proposes an Image-Free Pre-training (IFP) paradigm, a novel low-level pre-training approach with necessity of single randomly sampled Gaussian noise image, streamlining complicated data collection and synthesis procedure. The principle of the IFP involves reconstructing the original Gaussian noise from the randomly perturbed counterpart with partially masked spectrum band, facilitating the capability for robust spectrum representation extraction in response to the capricious downstream degradations.
strength_weakness: strength_weakness: 1. This paper proposes a pre-training paradigm, called Image-Free pre-training (IFP).
2. This paper proposes the frequency-aware masking strategy to enable the degradation-agnostic general representation learning by reinforcing the capability of the model for perceiving the susceptible spectra variations for downstream tasks.
3. IFP is the first method that deviates from the traditional pre-training paradigm of learning from data.
limitations: limitations: 1. From Table 1-3, it is obviously that the proposed IFP can improve the performance of low-level vision methods. However, how is the performance on real-world data? Could you provide qualitative and quantitative comparisons on real-world datasets? I would like to understand that the effectiveness on real data.
2. The amount of parameters and computational complexity of the proposed IFP should be discussed.
3. Could you provide more comparisons with pre-training methods? 
4. From Table 1-3, the compared methods seem to be out of dated. The superiority of the proposed IFP cannot comprehensively demonstrated.
5. How to determine the optimal hyperparameters?
6. Could you provide the ablation study of the optimization function?
paper_summary: paper_summary: This paper introduces Image-Free Pre-training (IFP), a novel low-level pre-training paradigm for vision restoration tasks, utilizing a single Gaussian noise image and focusing on frequency domain manipulation for spectrum representation extraction in first stage pretraining; In second stage, the pretrained model is fine-tuned in various downstream tasks, which demonstrates significant improvements in various tasks such as deblurring, deraining, and low-light image enhancement.
strength_weakness: strength_weakness: The idea of the paper, i.e. using pure noise for the first stage pre-training is novel and interesting, and the experiments are adequate. The innovation of the paper comes mainly from the use of pure noise to train the model in the first stage, which in my understanding is similar to allowing the model to learn a better initial distribution of parameters, making it easier to train downstream tasks closer to the global optimum. This process has a very small cost in terms of data and time and can bring significant performance improvements, which I think is valuable for the community.
limitations: limitations: 1.	The blurred images shown in the experimental part of the paper are still sharp, which does not reflect the effect of deblurring. Some deblurring results with high-degree blur need to be shown, e.g., those blurred images where the content of the image is no longer readable.
2.	Some of the tasks in the paper use datasets that are purely synthetic, such as GoPro in the deblurring task. Thus, some experiments on real-world data should be conducted, which is important to prove the effectiveness in reality of the proposed method. 
3.	The proposed method is two-stage, thus the experimental results of the one-stage should be ablated, i.e: for each task, the input data could consist of two parts (task data and noise) for one-stage training. This experiment helps us to understand whether it is really necessary to choose a two-stage process, as one-stage tends to be more convenient and performs better.
paper_summary: paper_summary: This paper proposes a novel paradigm for low-level vision pre-training called Image-free Pre-training (IFP). It introduces a new perspective for low-level pre-training beyond data-driven image pre-training. IFP requires only A SINGLE randomly generated Gaussian noise image for pretraining, which is really interesting. IFP applies random masking on the frequency domain on the noise image, simulating random degradations in the Fourier domain for content-agnostic representation learning. Experiments demonstrate effectiveness of IFP on various downstream tasks like deraining, deblurring and low-light enhancement, achieving performance boost compared to existing baselines.
strength_weakness: strength_weakness: -	The paper is well written and the motivation, method and experiments are clearly explained.
-	The core idea of IFP is novel in that it deviates from traditional data-driven pre-training and operates directly in the frequency domain using only noise, which is an interesting new perspective for low-level vision.
-	The authors conduct plenty of ablation studies and obtain lots of interesting observations.
-	The visual and quantitative results show the effectiveness of IFP.
limitations: limitations: -	In Fig. 2, how are the t-SNE results obtained? In what domain and how each point obtained from the image pixel?
-	In Fig. 2(c), what is the hollow circle in the plot?
-	How well is the reconstruction result in the Image-free masking pretraining? (reconstructed noise image)
-	In Fig. 3, the ‘random process’ is too large.
-	Explanation of how IFP enables learning degradation-invariant representations could be clearer. What exactly is learned from reconstructing the noise?
-	Impact of noise type (other than Gaussian) is worth exploring.
paper_summary: paper_summary: This paper presents a novel pre-training method for low-level vision that requires no data. The proposed method utilizes FFT to transform into frequency domain and then realize pre-training by Mask-Image-Modelling. With a single noisy image sampled from Gaussian distribution, this method has achieved great competitive improvement compared to other pre-training methods like DegAE, which is hungry with a huge amount of data and requires a long time to train. 
Overall, this paper implements pre-training in a very novel and efficient way. I think it is of high quality and deserves to be accepted. Open source is also expected.
strength_weakness: strength_weakness: The motivation for this article is very clear and convincing. Different degradations show different distributions in the frequency domain, so it is very reasonable to restore different frequency components in the frequency domain as pre-training. At the same time, the authors point out that low-level vision does not rely on semantics, so only pure noise images are needed to satisfy such pre-training, which ease the hunger for data.
The proposed method is very novel and effective. Experiments are also fully conducted, demonstrating the effectiveness of the proposed method and its advantages over DegAE: it only requires a short period of pre-training and a single noise picture.
limitations: limitations: In the ablation study Table.6, increasing the number of noise pictures affects the performance. I think the author's explanation for this is too brief and not very convincing.