Networks are Slacking Off: Understanding Generalization Problem in Image Deraining
Authors: Jinjin Gu
Xianzheng Ma
Xiangtao Kong
Yu Qiao
Chao Dong
Venue: NeurIPS 2023 poster (2023)
Keywords: Image Deraining
Generalization
Interpretation

Abstract: Deep deraining networks consistently encounter substantial generalization issues when deployed in real-world applications, although they are successful in laboratory benchmarks. A prevailing perspective in deep learning encourages using highly complex data for training, with the expectation that richer image background content will facilitate overcoming the generalization problem. However, through comprehensive and systematic experimentation, we discover that this strategy does not enhance the generalization capability of these networks. On the contrary, it exacerbates the tendency of networks to overfit specific degradations. Our experiments reveal that better generalization in a deraining network can be achieved by simplifying the complexity of the training background images. This is because that the networks are ``slacking off'' during training, that is, learning the least complex elements in the image background and degradation to minimize training loss. When the background images are less complex than the rain streaks, the network will prioritize the background reconstruction, thereby suppressing overfitting the rain patterns and leading to improved generalization performance. Our research offers a valuable perspective and methodology for better understanding the generalization problem in low-level vision tasks and displays promising potential for practical application.

Networks are Slacking Off: Understanding Generalization Problem in Image Deraining
Jinjin Gu 1.
Xianzheng Ma' Xiangtao Kongl.3 Yu Qiao 1.3 Chao Dong 1.3,
Shanghai AI Laboratory The University of Sydney Shenzhen Institutes Advanced Technology, Chinese Academy of Sciences jinj in. gu@sydedyy edu au, xianzhengma@pjlab org. {xt.kong yu. qiao, chao. dong)osiat ac.cn
Abstract
Deep deraining networks consistently encounter substantial generalization issues when deployed in real-world applications, although they are successful in labo- ratory benchmarks. A prevailing perspective in deep learning encourages using highly complex data for training, with the expectation that richer image background content will facilitate overcoming the generalization problem. However, through comprehensive and systematic experimentation, we discover that this strategy does not enhance the generalization capability these networks. On the contrary, it exacerbates the tendency of networks to overfit specific degradations. Our experi- a deraining network can be achieved by
ments reveal that better generalization simplifying the complexity of the training background images. This is because that the networks are "slacking off" during training, that is, learning the least complex elements in the image background and degradation minimize training loss. When the background images are less complex than the rain streaks, the network will prioritize the background reconstruction, thereby suppressing overfitting the rain patterns and leading to improved generalization performance. Our research offers a valuable perspective and methodology for better understanding the generalization problem in low-level vision tasks and displays promising potential for practical application.
Introduction
The burgeoning progress in deep learning has produced a series of promising low-level vision networks that significantly surpass traditional methods benchmark tests [T0] 30 12]. However, the intrinsic overfitting issue has prevented these deep models from real-world applications, especially
when the real-world degradation differs a lot from the training data [20] 31 33 [3]. We call this dilemma the generalization problem. As traditional and important low-level vision task, image deraining also faces a severe generalization problem. Existing deraining models tend to do nothing for the rain streaks that are beyond their training distribution, see Figure for example. The model trained with synthetic rainy images cannot remove rain streak for images with different rain patterns the training data.
Despite its significance, the generalization problem the context deraining tasks not comprehen- sively explored in the existing literature. Understanding the underlying causes of these generalization performance issues is imperative to propose effective solutions. However, analyzing generalization in low-level task is far from straightforward, as it is not simple extension of the generalization research in high-level vision tasks. This paper aims to take the pioneering step towards a more profound understanding of this challenge.
"Conesponding author.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

(a) Input Rainy Image
(b) Baseline Model
(c) Baseliee Model, More Background Training Data
(d) Baseine Model More Rain
According Conclusion
Reducing Traning
Figure I: The existing deraining models suffer from severe generalization problems. After training with synthetic rainy images, when feeding (a) an image with different rain streaks, its output (b) shows limited deraining effect. Two intuitive ways to improve generalization performance, including (c) adding background images and (d) adding rain patterns, cannot effectively relieve the generalization issue. In this paper, we provide a new counter-intuitive insight: (e) we improve the generalization ability of the deraining networks by using much less training background images for training.
As a typical decomposition problem, image deraining utilizes a relatively simple linear superimposi- tion degradation model. When the network fails to generalize, the rain streaks persist, and the image background remains unaffected. This intuitive and quantifiable phenomenon is well-suited for our study. Contrary previous works that rely only on overall image quality evaluation, we propose to decouple the deraining task into rain removal and background reconstruction. These two components are then separately analyzed. A key motivation behind this approach that the unsatisfactory perfor- mance can be attributed to "the unsuccessful removal of rain streaks" or "the poor reconstruction of image background". Without distinction, an image with successful rain streak removal but poor background reconstruction will also have bad quantitative results. Given that the generalization problem the deraining task is mainly related to the removal of rain streaks, we argue that those results that can remove rain will be more enlightening, regardless of the background reconstruction effect. Our research method allows us to minimize the impact of other less relevant factors on the effectiveness of rain removal.
In this paper, we show that the generalization problem arises when the network overfits the degradation, i.e., the rain patterns present in the training set. One significant factor contributing to this outcome the inappropriate training objective. We start our research with the most fundamental element in formulating the training objective- the training data. Numerous studies have attempted to enhance real-world performance by increasing the complexity of training data. This approach originates from a natural but unproven "acknowledgement" that augmenting the quantity of training data can rectify the generalization problem. This "acknowledgement" has also permeated the deraining community, suggesting that network trained with more diverse training set (both in terms of background images and rain streaks) can better generalize to unseen scenarios. However, this approach does not effectively address the generalization problem in deraining. We argue that this issue arises precisely because the network is provided with an excess of background data during training. Consequently, the model fails to learn reconstruct the image content but instead overfits the degradation. We arrive at some counter counter-intuitive conclusions by employing our analysis method, which separately measures background reconstruction and deraining effect.
Our key findings. We find that deep networks are slacking off during training, aiming to reduce the loss in the quickest way. This behaviour leads to poor generalization performance. The improper objective set for training is one of the primary contributors to this issue. Our finding indicates that deep networks exhibit a tendency to learn the less complex element between image content and additive degradation.
Specifically, the network naturally learns to overfit the rain streaks when the background complexity is higher than the rain complexity. However, when rain streaks in real-world scenarios deviate from the training set, the network tends to disregard them. At this point, the network will not remove rain streaks from the image and exhibit poor generalization performance. Conversely, training the model using a less complex background image set demonstrates superior generalization ability, as illustrated in Figure (e). This is because that when the complexity of the training image background is less than that of the rain patterns, the network will again take a shortcut to reduce the loss. In this case, the network learns to reconstruct the background rather than overfitting the rain streaks. When faced

---

with rain streaks outside the training set, the network takes priority to reconstruct the background, thus avoiding failure caused by the inability to recognize new rain streaks. It is also important to note that the model's performance is determined by the removal of rain and the quality of background reconstruction. Reducing the background complexity of the training data might inevitably lead to subpar reconstruction performance. Nonetheless, our results revealed that model trained with just 256 images can already handle most image components effectively. These counter-intuitive phenomena have not been revealed in previous literature.
Implication. Our results highlight the critical role of the training objective in determining a model's generalization ability. An inappropriate or incomplete training objective creates a loophole for deep networks to "slack off". While we anticipate that the network will learn the rich semantics inherent in natural images, it is often overlooked that the model can also achieve learning objectives through shortcuts. This results in poor generalization performance of low-Ievel vision models. Our findings also highlight that model with good generalization ability should learn the distribution of natural images rather than overfitting the degradation.
1.1 Related Works
This research primarily pertains to the field of deraining. However, unlike most of the existing deraining research, we do not propose new network structures, loss functions, or datasets. Our objective is to analyze and understand the generalization problem within the context of the deraining task. Due to space constraints, reviews of deraining works are provided in Appendix A We then review previous works focusing on interpretability and generalization in low-level vision.
Deep learning interpretability research aims to understand the mechanism of deep learning methods and obtain clues about the success/failure of these methods. We are not convinced to move forward in the right direction without a deep understanding of these working mechanisms. The research on deep learning interpretability follows long line of works, most focusing on the classification task
76 36.. Low-level vision tasks have also embraced great success with powerful deep learning techniques. There are also works on interpretability for these deep low-level networks
[7
48
46
49
44] For the generalization problem in low-level vision, these problems often arise when the testing degradation does not match the training degradation, e.g., different downsampling kernels 20 31 27 /4 and noise distributions [2 7 The existing works either develop blind methods to include more degradation possibilities in the training process or make the training data closer to real-world applications. Only little work has been proposed to study the reasons for this lack of generalization performance [33] 34]. More details of these previous works can also be found in supplementary material. No research has attempted to investigate the interpretation of the training process of low-level vision networks, especially from the perspective of the generalization problem.
19
61 37
2
Analysis Method
We aim investigate how different training objectives influence network behaviour and generalization performance. Before detailing our observations, we will outline our experimental designs and quantitative analytical methods.
2.1 Construction of Training Objective
The training data and the loss function jointly determine the training objective of a deep network. We set various training objectives to observe the changes in the generalization performance of different deraining models. As shown Figure (left), rainy image O can be modelled using a linear model O = B + R, where B is the image background, and R is the additive rain image. We change the training objectives with different background images and rain streaks.
Background Images. Typically, image backgrounds are sampled from street view images [17] or natural image datasets [3] II as these images close to the application scenarios of deraining. In literature, previous works [3 M claim that the model can learn the prior knowledge of reconstructing these scenes by training on a large number of such background images. We break this common sense by constructing different training background image sets from the following two aspects.
First, we change the number of training background images. The amount of training data partially determines the complexity of network learning. We argue that as too many background images are provided for training, the model cannot faithfully learn to reconstruct the image content but turns to overfit the degradation patterns. We reduce the complexity of the background images to see how the network behaviour changes in extreme scenarios. In our experiments, we use 8, 16, 32, 64, 128,
3

---

(a) Background image Spacos
(b) Rain Streaks for Training and Tasting
Urban100
Training Rain Small Range
Tst t
m 6
Mangs
Training Rain Medium Range
DIV2K
Training Large Range
Celob
R100L Rain for Testing
Figure 2: (a) Background images from different image categories. can be seen that the structure of the face image (CelebA) is relatively complex. Natural images (DIV2K) contain natural textures and patterns. The patterns in Mangal09 and Urban100 100 are artificially created - Manga images have sharp edges, while Urban images contain lot of repeating patterns and self-similarrties. (b) Different synthetic rain streak distributions that were used in our experiments.
Synthetc images deraining
Eni-grained analysis deraining results.
Evatuating
Reio Ramoval
Compare
Eacegrooudd Image
Aoditive Homm Image
Compare
Denwinnng Result
Straakes Region Mask
Ground Truth
Evaitaating
Rairy Image
Baniground Recanstucion
Figure 3: (Left) The illustration of the rainy image synthesis. (Right) Our fine-grained analysis of the deraining results.
256, 512, and 1024 background images" to build the training datasets, respectively. We also use large number of images (up to 30,000) to simulate the typical situation when the image background sufficiently sampled.
In addition to the data scale, the image content will also affect network learning. For example, is easier for the network to fit the distribution of images with regular patterns. A face image containing both short- and long-term dependent structures is more complex than a skyscraper with only repeated lines and grids 22. We select image distribution as the second aspect of our dataset construction. We sample from four image datasets that are distinct from each other: CelebA (face images) [35]
DIV2K (natural images) [50), Manga109 (comic images)
[38] and Urban 100 (building images)
Some examples of these images shown in Figure (a).
Rain streaks synthesis. Since it is hard to collect large number of real-world rainy/clean image pairs, we follow the previous deraining works [66 3 to synthesize rainy images for research. We use two kinds of rain streaks for training and testing separately. For training, we use the computational model to render the streaks left on the image by raindrops of varying sizes, densities, falling speeds, and directions. This model allows us sample rain streaks from different distributions. We adopt three rain image ranges for training, where different ranges may lead to different generalization effects; detailed settings. For testing, we use the synthetic rain patterns (63 Although the rain streaks are visually similar humans, they still pose huge generalization challenge existing models.
Table Different rain streaks settings.
Range
Quantity
Width
Length
Direction
Small
2200, 300
[5]
30, 31
[-5",5"]
Medium
200, 300
(5.7.9)
20. 40
-300, 300
Large
200, 300
(1.3.5.7.9]
[,600]
-70", 70")
Figure (b) for convenient visualization and Table for the
Loss Function. In low-level vision, the loss function is usually defined by the difference between the output image and the ground truth. We use the linorm loss without losing generality.
In actual training, we crop large images of different sizes into image patches of size 128 128 for training. this article, unless otherwise stated, these quantities refer to the number of patches.
The re-implementation of the PhotoShop rain streaks synthesis method. Please refer to this link

---

Res SwindR
Rain Region Removal
ResNet
60
UNee
Better Generalization
Generalization
50
16
32
B4
128
258
512
1024
10k
30k
16
32
64
128
256
512
1024
10k
30k
Training Background
Training Background Patch (DI22))
Restvet SwinlR UNet
Rain Ragion Renoval
50
ResNer
45
Generalization
30
UNet
Generalization
32
64
128
256
512
1024
3600
8
16
32
64
128
258
512
1024
2600
Training Background Patch Number (Mangu109)
Training Background Patch Number (Urban100)
Figure 4: The relationship between the number of training background images and their rain removal performance. The z-axis represents the image number, and the y-axis represents the rain removal effect En. Higher ER means better rain removal performance. The test rain patterns are not included in the training set. Thus, the effect of rain removal this time reflects the generalization performance. The qualitative results are obtained using ResNet.
2.2 Decoupling Analysis of Rain Removal Results
The standard evaluation of a deraining model involves computing similarity metrics between its output and the ground truth images [8]. However, such an evaluation strategy may lead unfair comparison. For example, image with perfect background reconstruction but inferior rain removal may have higher PSNR value than that with perfect rain removal but a slightly flawed background reconstruction (e.g., color shift). Drawing conclusions from such metrics may lead to biased conclusions about model's generalization capability. Generally speaking, the generalization performance of a deraining model mainly shown in the form of removing unseen rain. The background reconstruction may affect the visual effect, but it does not correlate directly with the effectiveness of rain removal. However, background reconstruction affects the quantitative result of traditional rain removal metrics. Therefore, we discuss the removal of rain streaks separately from the reconstruction of the background. We can achieve this goal with a simple mask-based decoupling method.
can seen from Figure (b) that the pixels in the additive rain image R without rain streaks should be black, while rain streaks will appear brighter. After synthesis, these black areas reflect the background area, while the brighter areas indicate the rainy regions. A perfect rain removal effect should do minimal damage to the background area and remove the additive signal from the rain streaks area. By processing R to a binary mask M using a threshold i, where MP.)) = if R S t. and M [..) = 1 if [(.j
segment the output image Ô into the rain streaks part O Q M and the background part Ô @ (1 (). We then have two metrics:
ER v /E[( @ M O @ M) gives the Rain Removal performance. A network with poor generalization will not remove rain streaks. This term measures the changes made by the network in the rainy regions. A higher value reflects better rain removal performance.
Eg B[(Ō o (1 - M) B o (1 M)) gives the effect of Background Reconstruc- tion by comparing the background regions with the ground truth. A large error in this term means poor reconstruction quality.
2.3 Deep Models
We summarize existing networks into three main categories our experiments. The first category includes networks composed of convolutional layers and deep residual connections, and we use
5

---

128
256
512
1024
30,000
Region Removal
60
Medium Small
60
6.15
145
655
M45
545
141
11
40
Large Range
Range
30
Smalll Range
M45
041
64 128
30k
Training Backgoumd Image Numbers
When trained with different rain ranges, the model exhibits different rain removal effects. The y-axis represents the quantitative rain removal effect. When the rain removal performance is lowered to the blue dashed line, the qualitative effect of removing rain decreases significantly. We use ResNet in this experiment.
Figure 5:
Normalized Rain Removal Effect
1.0
Mult- Stale Structural Complexisy
Figure 6: (a) The relationship between the number of training images and their normalized rain removal perfor- mance. When the y value is lowered to the grey dashed line, the qualitative ef- feet of removing rain starts to decrease significantly. (b) The averaged com- plexities of different image categories given by (2).
0.8
0.34
0.6
0.32
0.4
Urban100
CelebA
0.30
0.2
Manga109
DF2K
0.28
0.0
(a)
Training Backgouud Complexity Level
16
12
64
128
256
512
1024
(b)
CelebA
OP2K
Manga
Urban
the ResNet (88 as a representative. The second category includes networks with encoder-decoder
designs, and layers to extract global and multi-scale features, which have been proven successful in many deraining networks. The last category includes image processing Transformer networks. Transformer [45] 73 751 is a new network structure characterized by self-attention operations. We select SwinIR [30] as representative. For more training settings, please check the supplementary material.
UNet [42 representative. UNet introduces down-sampling and up-sampling
3 Understanding Generalization
Next, we conduct experiments based on the above analysis method to explore the counter-intuitive phenomenon terms of generalization in the deraining task. Our analysis includes two aspects rain removal (Section 3.1 and background reconstruction (Section 3.2)
3.1 Generalization Rain Removal
We analyze the rain removal effect unseen rain streaks. Since use different types of rain streaks for training and testing, the rain removal results can reflect generalization performance. After extensive experiments, arrive at the following observations.
Training with fewer background images produces better deraining effects. In this experiment, we use the middle-level range of rain streaks and different background image sets to generate different training datasets, described in Section 2.1 Experiments are conducted across all four categories of images. We then test the rain removal performance ER of these models. The test images utilize rain streaks proposed in 63] which are different from the rain distribution of the training set. The testing background images are sampled from the corresponding categories and differ from those in the training set. The experimental results are presented in Figure 4 Despite variations in background images and network architectures, consistent trend is evident across all experimental results. Notably, deraining models trained on only eight images demonstrate a noteworthy ability handle unseen rain streaks. In contrast, models trained using large number of images exhibit an inability to address these rain streaks. Such a phenomenon challenges traditional expectations. Between these polarized states, the rain removal efficacy diminishes as the quantity of training images increases. Once the image number ascends to 256, the networks have predominantly lost their deraining capability. As the image number increases from 1024 to 30,000, there's 's minimal variance in models' rain removal performance - they all fail to remove unseen rain streaks. This trend is also reflected the qualitative results.

---

Background Reconatuccion
40
Reshet
50
ResNal SwinlR
45
ReaNet
80
ResNer
-66
SwintR
UNot
UNot
40
UNit
SwhiR
SwniR
UNot
20
35
40
26
30
30
20
25
20
16
15
10
8:
256
16 39 128 5121024
258 5121004
32
128 5121004
Training Patch Number (ColobA)
Training Backgoumd Patch Number (DIV2K)
Training Patch Number
Training Backgguum Patch Number (Unban100)
(Mangall0)) 109)
Figure 7: The relationship between the number of training images and their background reconstruc- tion effect. For each plot, the z-axis represents the image number, and the y-axis represents the reconstruction eπrоr of the background Eg.
Here, we attempt to explain this interesting counter-intuitive phenomenon. Although we set the training objective as removing rain streaks from images by comparing the output to the target, the network has two alternative strategies to minimize this training loss. The first strategy involves recognizing and removing rain streaks, while the second consists of recognizing and reconstructing the image background. If the learning strategy is not specified, the network tends to gravitate towards the simpler strategy. When a large number of background images are utilized in training, learning to reconstruct backgrounds becomes significantly more complex than learning to remove rain streaks. Consequently, the network chooses to recognize and remove the rain. This decision, however, can lead to overfitting problem: when new rain streaks differ from those used in training, the network fails to recognize and remove them. Conversely, when the background image is comprised of only a few image images, learning the background becomes easier than learning rain streaks. In this scenario, the network recognizes image components in the background without overfitting the features of rain streaks. As result, the model demonstrates superior rain removal capability in images with unseen rain streaks.
The relative complexity between the background and rain determines the network's behaviour. To corroborate the abovementioned conjecture, modify the range of the rain streaks used in training. as outlined in Section 2. When employing a medium-level rain range, the rain removal effect diminishes when training with 64 background images. According to our explanation, larger rain range complicates the network's i task of learning the rain pattern. As a result, the rain removal effect does not deteriorate until a larger number of background images are used for training. The experimental results are shown in Figure [ It can be observed that the rain removal effect decreases across all three training rain ranges as the number of background images increases. When sufficient background images are used for training (30,000 images), even with large training range for rain, the resulting model struggles to deliver satisfactory rain removal performance on unseen rain streaks. This suggests that the large rain range does not encompass our testing scenarios and cannot bring generalization ability on test images. If training with the large rain range, the network displays significant drop in rain removal performance only when more than 512 background images are used for training. Conversely, a model trained on a small rain range cannot exhibit satisfactory rain removal even with only 16 background training image images. These results indicate that the relative relationship between the background image and rain streaks influences network behaviours. The complexity learning difficulty of the medium-rang rondium-range rain is approximately less than that of 64 training background images. In comparison, the complexity of the large-range rain is approximately less than that of 512 training background images. Depending on the situation, the network tends to take shortcuts or select the easier learning pathway.
A more complex background dataset makes learning harder for the network. Next, we modify the category of the background images used for training and monitor the resulting behaviours of the models. We normalize the deraining effect to a range between 0 and to facilitate comparison across different image categories. The results are shown in Figure (a). The most intuitive conclusion is that, even when the number of training images remains consistent, different image categories can lead to different rain removal performance. For instance, in the case of CelebA images, increasing the training set from 8 to 16 images leads to notable decrease in deraining performance. This disparity more stark compared to natural image - increasing to 16 images does not result in a significant drop in rain removal performance. Moreover, for images sampled from the Mangal 009 and Urban 100 categories, the rain removal performance does not exhibit a significant drop until the image number exceeds 32. According to our interpretation, the more complex image categories will prompt the
7

---

(a] Input Raiiy Image
Cbi SPONet
IDI RCDNet
(0) RosNet (10k patches, Large Rain)
(6) (Large
patches)
(s) polches, Large Rain)
Truth
Figure 8: Visualization of the deraining results on synthetic image. Zoom in for better comparison.
Training Objective
ResNet
SPDNNet
[68]
RCDNer
[4]
Table 2: Quantitative com-
Back.
Rango
Es
Ep
PSNR
Es 1
Eg
PSNR
Ex 1
Eg
PSNR
parisons between different
30k
Medium
31.24
10.79
25.15
33,63
5.49
30.51
26.55
5.41
28.54
models. 1 means the higher the better while + means the
Medium
53.33
25.02
20.87
45.47
14.78
512
Large
39.88
8.91
28.57
37,53
7.16
29 60
256
Large
45.64
16.51
24.30
38.87
8.03
29.40
40,40
8.52
29.08
lower the better.
Large
23.53
21.45
43.20
25.67
13.72
model experience an earlier performance decline as the number of training images increases. Our results suggest that the complexity of these four image categories can be ranked in ascending order as CelebA, DIV2K, Mangal09, and Urban100.
The observed sequence roughly aligns with human perceptual tendencies. Face images, as represented by the CelebA dataset, exhibit strong global and local structures. While DIV2K images are abundant in local texture, they maintain a relatively simple global structure. Contrastingly, Manga images, although free of complex textures, often contain text elements and detailed edges. Lastly, Urban images are chiefly characterized by recurrent patterns, such as stripes and grids. To corroborate our conclusion, validate it using complexity metric derived from mathematical model. Bagrov el al. [2) proposed a computational method for estimating the structural complexity of natural pat- terns/images. We compute the multi-scale structural complexity for these four image categories, and the results corroborate our observed ordering, as depicted in Figure (b). This provides some mathematical evidence to support our claim.
3.2 Reconstruction on Background
The aforementioned results indicate that the deraining capability can be enhanced by limiting the complexity of the background images used for training. However, utilizing only a restricted set of background images also has drawbacks. While it prevents the network from overfitting rain patterns, it may conversely prompt the network to overfit the limited background images. We also conduct experiments to address this particular concern.
Using the decoupled evaluation metric EB described Section 2.2 we are able to assess the recon- struction of the background separately. The results in Figure show that as the number of training images increases, the quality of background reconstruction also improves. Remarkably, training with just 256 background images can already yield satisfactory background reconstruction performance. Adding more training images beyond this point does not significantly improve performance. These findings are surprising and counter-intuitive. We typically assume that training low-level vision models requires a large number of images. However, our research suggests that training with an excessive number of background images does not necessarily enhance the reconstruction performance but rather exacerbates the model's tendency to overfit rain streaks. Another unexpected result is that a model trained with merely 256 images can already handle most image components. This might imply that the complexity of image components and features for a low-level vision network might not be as high as commonly believed.
Relationships with large models and large data trends. We have also noticed recent trends around large models and large data. Many studies point out that larger models (from billions to hundreds of billions of parameters) and larger data can lead to the emergence of capabilities that previously unimaginable [26]. This seems to contradict the discovery described in this work.
8

---

ROONet
to SPONeY
LRt Syn2Fea 3Sm
(e) SyndRenk- Reel
RCDMee Tralned wy 206
Figure 9: Qualitative results on real-world test images. Zoom in for better comparison.
However, the models that can achieve significant gains from large amount of data and model parameters are some generative-based models, such as GPT and diffusion models [4T.. The large-scale dataset is not the only critical factor to the remarkable ability of these large models. The training method is also very important. Giving a large-scale dataset to simple task will not make the model generalize to other tasks. In low-level vision, we argue that the task of learning an end-to-end input-to-output mapping via simple image comparison loss may be relatively easy. In contrast, important factor for the success of the large language model is its generative pre-training learning method. This learning method enables the model to learn the content rather than the mapping described by the imput and output. It is also the core point of this paper that solving the generalization problem requires learning the content of the data, instead of the degradations.
4 Implication
This paper does not propose any new algorithms directly, but can provide insights on how to improve the generalization performance of low-level models. Our experiments have yielded three significant practical findings: (1) By limiting the number of background images used in training, the network can focus more on learning the image content instead of overfitting the rain streaks; (2) Enlarging the range of rain streaks in the training set can allow for the use of more background images in training; (3) Surprisingly, a small number of background images can yield satisfactory reconstruction performance. These findings can be directly applied to enhance the generalization capability of existing models with minimal modifications. Our strategy is straightforward: find a balance between the number of background images and the range of rain streaks in the training set avoid overfitting.
Some quantitative results are presented in Table 2 We use three deraining models as baselines (ResNet, SPDNet [68], RCDNet (54)) and demonstrate the power of the proposed strategy. We train our baseline models by using 30,000 background images and medium-range rain. The test set the R R100 dataset [63]. We quantify the deraining and background reconstruction effects according to the decouple evaluation metrics FR and EB. We also calculate the PSNR performance as a reference. It can be seen that using the existing training methods cannot generalize well to the unseen rain of R100, which is shown by the poor deraining performance in Table 2 However, due to the learning on large number of images, the reconstruction errors of the baseline models are generally lower. Thus, the PSNR values cannot objectively reflect the effect of rain removal. We reduce the training background images to 64, which is the upper limit of the image number that make the model generalize under medium-range rain. At this time, the rain removal performance has been greatly improved, but at the cost of background reconstruction performance. By enlarging the rain range and training with more background images, we are able to achieve a trade-off between rain removal performance and background reconstruction.
Figure g shows a qualitative comparison of these models under different training objectives. can be seen that even with the advanced network structure design, the rain removal effects of the baseline models of SPDNet and RCDNet are not satisfactory. Using a larger range of rain can bring limited improvements. In the case of medium-range rain, reducing the background image to 64 significantly improves the rain removal effect and results in unstable image reconstruction. When the rain range is enlarged, and the training background is set to 128 images, the model can show excellent performance in rain removal and background reconstruction. Note that we do not use additional data or improve the network structure throughout the process. We only adjust the training data.
We also present the comparison on real images in Figure 9 addition, semi-supervised methods 58 23 have also been used to improve the deraining effect real images, and we also include the

---

representative method Syn2Real [66] 67]. Syn2Real-Syn is trained on synthetic data, and Syn2Real- Real is trained on synthetic labelled data and real unlabeled data. Due to the difference in the distribution of rain streaks, the models trained using synthetic data cannot produce satisfactory rain removal effects. When obtaining some real images, Syn2Real-Real can indeed achieve some improvement. However, these improvements are not brought by improving the generalization ability, Because these methods manage to convert "rain outside the training set" to "rain inside the training set". Since data collection is extremely difficult, this method still faces great challenges in practice. Our method improves generalization performance and achieves better results on test images. We also find that artifacts may appear in the background part of the output images. Given that our training utilizes only 256 images without any specific techniques, the outcome, while expected, showcases our prowess in the removal of rain streaks. Indeed, while competing methods might sidestep these artifacts, they fall short in removing rain streaks effectively. We recognize various solutions existing rectify the artifact issue from introducing post-processing modules to leveraging this outcome to steer another deraining network. These methods deviate from the interpretability topic of this work, and we hope diseuss technical means to improve generalization performance further in future works.
Conclusion and Insights
We investigate the generalization problem in deraining networks. While our work focuses on image deraining, our key conclusions can provide insights for the broader field of low-level vision. We argue that the gencralization problem in low-level vision cannot be attributed solely to insufficient network capacity. Instead, discover that existing training strategies do not promote generalization. Networks tend to overfit the degradation patterns in the training set, leading to poor generalization performance for unseen degradation patterns. Moreover, the current need for practical interpretability tools for low-level vision models presents a significant obstacle. These tools are necessary to understand what the low-level model is learning and why it learns this way. This gap between the ideal and the reality compels us to reconsider the importance of interpretability in low-level vision networks. Our work provides necessary perspective on this issue. These insights also need to be tested and validated in other low-level vision tasks. Our findings highlight the importance of continued research in this area, and we hope that our work will inspire further investigations.
This work was supported in part by the National Natural Science Foundation of China under Grant (62276251, 62272450), the Joint Lab of CAS-HK, the National Key R&D Program of China (NO. 20222D0160100) and in part by the Youth Innovation Promotion Association of Chinese Academy of Sciences (No. 2020356).
Acknowledgement
References
[1] Pablo Arbelaez, Michael Maire, Charless Fowlkes, and Jitendra Malik. Contour detection and hierarchical image segmentation. IEEE transactions on pattern analysis and machine intelligence, 33(5):888999 2010.
[2] Andrey A Bagrov, Ilia A lakovlev, Askar A Iliasov, Mikhail Katsnelson, and Vladimir V Mazurenko. Multiscale structural complexity of natural patterns. Proceedings of the National Academy of Sciences. 17744))30241-3025 2020.
[3] Tom Brown, Benjamin Mann. Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
[4] Yi Chang, Luxin Yan, and Sheng Zhong. Transformed low-rank model for line pattern noise removal. In P Proceedings of the IEEE international conference computer vision, page 1726-1734, 2017.
[5] Chenghao Chen and Hao Li. Robust representation learning with feedback for single image deraining, In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, page 7752-7751, 2021.
[6] Hanting Chen, Yunhe Wang. Tianyu Guo, Chang Xu, Yiping Deng. Zhenhua Liu, Siwei Ma. Chunjing Xu. Chao Xu, and Wen Gao. Pre- traitrained image processing transformer. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, page 12299-12311 2021.
10

---

[7] Haoyu Chen, Jinjin Gu, Yihao Liu, Salma Abdel Magid, Chao Dong, Qiong Wang, Hanspeter Pfister, and Lei Zhu. Masked image training for generalizable deep image denoising. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023.
[8] Xiang Chen, Hao Li, Minggiang Li, and Jinshan Pan. Learning a sparse transformer network for effective image deraining. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pages 5896-599699 5905, 2023.
[9] Xiang Chen, Jinshan Pan, Kui Jiang, Yufeng Li, Yufeng Huang, Caihua Kong, Longgang Dai, and Zhentao Fan. Unpaired deep image deraining using dual contrastive learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, page 2017-2026 2022.
[10] Chao Dong, Chen Change Loy, Kaiming He, and Xiaoou Tang. Image super-resolution using deep convolutional networks. IEEE transactions on pattern analysis and machine intelligence, 38(2):295-307, 2015.
[11] Yingjun Du, Jun Xu, Xiantong Zhen, Ming-Ming Cheng, and Ling Shao. Conditional variational image deraining. IEEE Transactions on Image Processing. 29:6288 6301, 2020.
[12] Xucyang Fu, Jiabin Huang, Xinghao Ding, Yinghao Liao, and John Paislcy. Clcaring the skies: deep network architecture for single-image rain removal. JEEE Transactions on Image Processing, 26(4):294429556 2017.
[13] Xueyang Fu, Jiabin Huang. Delu Zeng. Yue Huang, Xinghao Ding, and John Paisley. Removing rain from single images via deep detail network. In Proceedings of the IEFE conference on computer vision and pattern recognition, page 3855-3863, 2017.
[14] Xueyang Fu, Borong Liang, Yue Huang, Xinghao Ding, and John Paisley. Lightweight pyramid networks for image deraining. IEEE transactions neural networks and learning systems, (1(6))1794 1807, 2019.
[15] Xueyang Fu, Qi Qi, Zheng-Jun Zha, Yurui Zhu, and Xinghao Ding. Rain streak removal via dual graph convolutional network. Proceedings of the AAAI Conference Artificial Intelligence, volume 35, page 1352-1360, 2021.
[16] Kshitiz Garg and Shree Nayar. Photorealistic rendering of rain streaks. ACM Transactions on Graphics (TOG), 1563)999-11022 2006.
[17] Andreas Geiger, Philip Lenz, and Raquel Urtasun, Are we ready for autonomous driving? kitti vision benchmark suite. In 2012 IEEE conference on computer vision and pattern recognition, pages 3354-3361. IEEE, 2012.
[18] Jinjin Gu, Haoming Cai, Haoyu Chen, Xiaoxing Ye, Jimmy Ren, and Chao Dong. Pipal: a large-scale image quality assessment dataset for perceptual image restoration. In European Conference Computer Vision, pages 633-651. Springer, 2020.
[19] Jinjin Gu and Chao Dong. Interpreting super-resolution networks with local attribution maps. In Pro- ceedings of the IEEE/CVP Conference Computer Vision and Pattern Recognition, pages 9199-9208, 2021.
[20] Jinjin Gu, Hannan Lu. Wangmeng Zuo, and Chao Dong. Blind super-resolution with iterative kernel correction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1604-1613, 2019.
[21] Shuhang Gu, Deyu Meng, Wangmeng Zuo, and Lei Zhang. Joint convolutional analysis and synthesis sparse representation for single image layer separation. In Proceedings of the IEEE International Conference on Computer Vision, page 1016- 2017.
[22] Shi Guo, Zifei Yan, Kai Zhang, Wangmeng Zuo, and Lei Zhang. Toward convolutional blind denoising of real photographs. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 1712-1722, 2019.
[23] Huaibo Huang, Aijing Yu, and Ran He. Memory oriented transfer learning for cemi-supervised image deraining. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7732-7741, 2021.
[24] Jia-Bin Huang, Abhishek Singh, and Narendra Ahuja. Single image super-resolution from transformed self exeelaxepps In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5197-5206, 2015.
11

---

[25] Kui Jiang, Zhongyuan Wang, Peng Yi, Chen Chen, Baojin Huang. Yimin Luo, Jiayi Ma, and Junjun Jiang. Multi-scale progressive fusion network for single image deraining. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, page 8346-8355, 2020.
[26] Jared Kaplan, Sam McCandlish, Tom Henighan. Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeftrey Wu, and Dario Amodei. Sealing laws for neural language models, arXiv preprint arXiv:2001.08.0 2020.
[27] Xiangtao Kong, Xina Liu, Jinjin Gu, Yu Qiao, and Chao Dong. Reflash dropout in image super-resolution. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6002- 6012, 2022.
[28] Christian Ledig, Lucas Theis, Ferenc Huszár, Jose Caballero, Andrew Cunningham, Alejandro Acosta, Andrew Aitken, Alykhan Tejani, Johannes Totz. Zehan Wang, et al. Photo-realistic single image super- resolution using generative adversarial network. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4681-4690, 2017.
[29] Ruoteng Li, Loong-Fah Cheong, and Robby T Tan. Heavy rain image restoration: Integrating physics model and conditional adversarial learning. In Proceedings of the IEEE/CVF Conference Computer Vision and Pottern Recognition, page 1633-1642, 2019.
[30] Jingyun Liang, Jiezhang Cao, Guolei Sun, Kai Zhang. Luc Van Gool, and Radu Timofte. Swinir: Image restoration using swin transformer. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 1833-1844, 2021.
[31] Anran Liu, Yihao Liu, Jinjin Gu, Yu Qiao, and Chao Dong. Blind image super-resolution: A survey and beyond. IEEE Transactions on Pattern Analysis and Machine Intelligence. 2022.
[32] Xing Liu, Masanori Suganuma, Zhun Sun, and Takayuki Okatani. Dual residual networks leveraging the potential of paired operations for image restoration. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, page 7007-7016, 2019.
[33] Yihao Liu, Anran Liu, Jinjin Gu, Zhipeng Zhang, Wenhao Wu, Yu Qiao, and Chao Dong. Discowering" semantics" in super-resolution networks. arXiv preprint arXiv: 008,00 2021.
[34] Yihao Liu, Hengyuan Zhao, Jinjin Gu, Yu Qiao, and Chao Dong. Evaluating the generalization ability of super-resolution networks. IEEE Transactions on pattern analysis and machine intelligence, 2023.
[35] Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of the IEEE international conference on computer vision, pages 3730-3738. 2015.
[36] Scott M Lundberg and Su-In Lee. A unified approach to interpreting model predictions. Advances in neural information processing systems, 30, 2017
[37] Salma Abdel Magid, Zudi Lin, Donglai Wei, Yulun Zhang, Jinjin Gu, and Hanspeter Pfister. Texture-based error analysis for image super-resolution In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pages 21 211221 2022.
[38] Yusuke Matsui, Kota Ito, Yuji Aramaki, Azuma Fujimoto, Toru Ogawa, Toshihiko Yamasaki, and Kiyoharu Aizawa, Sketch-based manga retrieval using mangal09 dataset. Multimedia Tools and Applications. 76(20):2181122188 1811-1188811 2017.
[39] Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang. Zachary De Vito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Leter. Automatic differentiation in pytorch. 2017.
[40] Dongwei Ren, Wangmeng Zuo, Qinghua Hu, Pengfei Zhu, and Deyu Meng. Progressive image deraining networks: A better and simpler baseline. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, page 3937-3946, 2019.
[41] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-rcsolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and puttern recognition, pages 10684-10695, 2022.
[42] Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In International Conference on Medical image computing and computer-usiisted intervention, pages 231. Springer, 2015.
[43] Gerald Schaefer and Michal Stich. Ucid: An uncompressed color image database. In Storage and Retrieval Methods and Applications for Multimedia 2004, volume 5307, pages 472-480. SPIE, 2003.
12

---

[44] Shuwei Shi, Jinjin Gu. Liangbin Xie, Xintao Wang, Yujuu Yang, and Chao Dong. Rethinking alignment in video super-resolution transformers. In Advances in Neural Information Processing Systems, 2022.
[45] Shuwei Shi, Jinjin Gu, Liangbin Xie, Xintao Wang. Yuju Yang. and Chao Dong. Rethinking alignment in video super-resolution transformers. Advances in Neural Information Processing Systems, 2022.
[46] Avanti Shrikumar, Peyton Greenside. and Anshul Kundaje. Learning important features through propagat- ing activation differences. In International conference on machine learning, pages 3145-3153, PMLR, 2017.
[47] Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman, Deep inside convolutional networks: Visualising image classification models and saliency maps. arXiv preprint arXiv: 13/2.6034, 2013.
[48] Jost Tobias Springenberg. Alexey Dosovitskiy, Thomas Brox, and Martin Riedmiller. Striving for simplicity: The all convolutional net, arXiv preprint arXiv:1412.680 2014.
[49] Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep networks. In International conference machine learning, pages 3319-333300 19-3228. PMLR, 2017.
[50] Radu Timofte, Eirikur Agustsson, Luc Van Gool, Ming-Hsuan Yang, and Lei Zhang. Ntire 2017 challenge single image super- resoluton: Methods and results. In Proceedings of the IEEE conference computer vision and pattern recognition workshops, pages 114 125, 2017.
[51] Cong Wang, Yutong Wu, Zhixun Su, and Junyang Chen. Joint self-attention and scalo-aggregation for self- calibrated deraining network. Proceedings of the 28th ACM International Conference on Multimedia, page 2517-2525, 2020.
[52] Cong Wang. Xiaoying Xing. Yutong Wu, Zhixun Su, and Junyang Chen. Desfi: Deep cross- scale fusion network for single image rain removal. In Proceedings of the 28th ACM international conference on multimedia, page 1643-1651, 2020.
[53] Guoqing Wang. Changming Sun, and Arcot Sowmya. Erl-net: Entangled representation learning for single image de-raining. In Proceedings of the IEEE/CVF International Conference Computer Vision, page 5644-5652, 2019.
[54] Hong Wang, Qi Xie, Qian Zhao, and Deyu Meng. A model-driven deep neural network for single image rain removal. Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, page 3103-3112, 2020.
[55] Tianyu Wang, Xin Yang. Ke Xu, Shaozhe Chen, Qiang Zhang, and Rynson WH Lau. Spatial attentive single-imaee deraining with high quality real rain dataset. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, page 12270- 12279, 2019.
[56] Yinglong Wang, Yibing Song, Chao Ma, and Bing Zeng. Rethinking image deraining via rain streaks and vapors. In European Conference on Computer Vision, page 367-382. Springer, 2020.
[57] Zheng Wang, Jianwu Li, and Ge Song. Dtdn: Dual-task de rainnngn network. In Proceedings of the 27th ACM international conference on multimedia, page 1833-1841 2019.
[58] Wei Wei, Deyu Meng, Qian Zhao, Zongben Xu, and Ying Wu. Semi-supervised transfer learning for image rain removal. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pages 3877-3886, 2019.
[59] Yanyan Wei, Zhao Zhang. Haijun Zhang, Richang Hong, and Meng Wang. A Searse-to-fine multi-strent hybrid deraining network for single image deraining. In 2019 IEEF international conference on data mining (ICDM), page 028-637. IEEF, 2019.
[60] Jie Xiao, Man Zhou, Xueyang Fu, Aiping Liu, and Zueng-Jun Zha. Improving de-raining generalization via neural reorganization. In Proceedings of the IEEE/CVF International Conference Computer Vision, page 4987 199449996996 2021.
[61] Liangbin Xie, Xintao Wang, Chao Dong, Zhongang Qi, and Ying Shan. Finding discriminative filters for specific degradations blind supor-resolution. Advances in Neural Information Processing Systems, 34:51-61, 2021.
[62] Wenhan Yang, Jiaying Liu, Shuai Yang, and Zongming Guo. Scale- single image deraining via visibility-ehhancee enhanced recurrent wavelet learning. IEEE Transactions on Image Processing, 28(6):2948-22 2961. 2019.
13

---

[63] Wenhan Yang, Robby T Tan. Jiashi Feng, Jiaying Liu, Zongming Guo, and Shuicheng Yan. Deep joint rain detection and removal from single image. In Proceedings of the IEEE conference on computer vision and puttern recognition, page 1357-1366, 2017.
[64] Youzhao Yang and Hong Lu. Single image deraining via recurrent hierarchy enhancement network. In Proceedings of the 27th ACM International Conference on Multimedia, page 1814-1822, 2019.
[65] Rajeev Yasarla and Vishal M Patel. Uncertainty guided multi-scale residual learning-using cycle spinning enn for single image de-raining. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, page 8405 8414, 2019.
[66] Rajeev Yasarla, Vishwanath Sindagi, and Vishal Patel. Syn2real transfer learning image deraining using gaussian processes. In Proceedings of the IEEE/CVF conference on computer vision and puttern recognition, page 2726-273666 2736, 2020.
[67] Rajeev Yasarla, Vishwanath Sindagi, and Vishal M Patel. Semi-superrised image deraining using gaussian processes. IEEE Transactions Image Processing, 30:6570-6558 2021.
[68] Qiaosi Yi, Juncheng Li, Qinyan Dai, Faming Fang, Guixu Zhang, and Tieyong Zeng. Structure-preserving deraining with residue channel prior guidance. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 4238 4247, 2021.
[69] Weijiang Yu, Zhe Huang, Wayne Zhang, Litong Feng, and Nong Xiao. Gradual network for single image de-raining. In Proceedings of the 27th ACM international conference multimedia, page 1795-1804, 2019.
[70] Syed Waqas Zamir, Aditya Arora, Salman Khan, Munawar Hayat, Fahad Shahbaz Khan, Ming-Hsuan Yang. and Ling Shao. Multi Multi-stage computer vision and pattern recognition, page 14821-14831, 2021.
stage progressive image restoration. Proceedings of the IEEE/CVF conference on
[71] He Zhang and Vishal M Patel. Density-award aware single image de-raining using a multi stream dense network. In Proceedings of the IEEE conference on computer vision and pattern recognition, page 695 195-704, 2018.
[72] Zhang, Vishwanath Sindagi, and Vishal M Patel. Image de-raining using conditional generative adversarial network. IEEE transactions on circuits and systems for video technology, 2019.
[73] Jiale Zhang, Yulun Zhang, Jinjin Gu, Yongbing Zhang, Linghe Kong, and Xin Yuan. Accurate image restoration with attention retractable transformer. In International Conference on Learnnng Representations, 2023.
[74] Ruofan Zhang, Jinjin Gu, Haoyu Chen, Chao Dong, Yulun Zhang, and Wenming Yang. Crafting training degradation distribution for the securacy-generallee trade-off. In International conference machine learning. PMLR, 2023.
[75] Chen Zheng, Yulun Zhang, Jinjin Gu, Yongbing Zhang, Linghe Kong, and Xin Yuan. Cross aggregation transformer for image restoration. Advances Neural Information Processing Systems, 2022.
[76] Bolei Zhou, David Bau, Aude Oliva, and Antonio Torralba. Interpreting deep visual representations via network dissection. IEFE transactions on pattern analysis and machine intelligence, 41(9):2131-2144 2018.
[77] Man Zhou, Xiao, Yifan Chang, Xueyang Fu, Aiping Liu, Jinshan Pan, and Zheng-Jun Zha. Image de-raining via continual learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, page 4907-4916, 2021.
[78] Lei Zhu, Chi- Chi-Wing Fu, Dani Lischinski, and Pheng-Ann Heng. Joint bi-layer optimization for single-image rain streak removal. In Proceedings of the IEEE international conference on computer vision, page 2526-2534, 2017.
14

---

Appendix
A
Other Related Work
A.1
Image Deraining
Many methods have been proposed to develop state-of-the-an deraining networks. These works
include deep networks designs [12 [53. residual networks [13
32],
recurrent networks [40] 64 62].
multi-task [1] and multi-scale designs 12] 65|
69
[59
52
70 sparsity-based image modeling
21 78). low-rank prior [4) model-driven solutions 154
56
attention mechanism [51 6 [5].
Transformer-basee network [8), adversarial learning (29), representation learning [5), semi-supervised
666 and unsupervised learning [ Deep learning methods are data-hungry, but collecting rain streaks and background image pairs is challenging. A lot of works have been proposed to synthesize rain streaks with better results. Garg er al. [16] first propose a physically- -based photo-realistic rendering method for synthesizing rain streaks. Zhang et al. [1] and Fu et al. [12] use Photoshop software to manually add rain effects to images to build the synthetic paired data. Due to the poor generalization performance of existing methods, models trained on synthetic images were found to be ineffective in real-world scenarios. Some works [63 72 551 that have contributed
real-world collected deraining datasets. However, acquiring these datasets is still expensive and cannot solve the problem of poor generalization. There are also works that mention the generalization issue of the deraining models. Xiao el al. (60] and Zhou et al. [77] attempt to improve the generalization ability of deraining networks by accumulating knowledge from multiple synthetic rain datasets, as most existing methods can only learn the mapping on single dataset for the deraining task. However, this attempt does not allow the network to generalize beyond the training set.
In addition, semi- supervise methods [58 23 have also been used to improve the deraining effect on real images, and we also include the representative method Syn2Real 666 67]. There are some semi-supers cumi-supervised deraining methods [38] 23 66 67 are proposed to improve the performance of deraining models in real-world scenarios. When obtaining some real images similar to the test images, these works can achieve some improvement. However, these improvements are not brought about by improving the generalization ability. Their solution is to include real test images in the training set, even if we don't 't have corresponding clean images. These methods are effective when we can determine the characteristics of the test image. However, more is needed to solve the generalization problem. Because these methods manage to convert "rain outside the training set" to "rain inside the training set".. Since data collection is extremely difficult, this method still faces great challenges in practice.
A.2 Low-Level Vision Interpretabiliity
Next, we provide a detailed review of existing work on low-level visual interpretability. Gu and Dong [19] bring the first interpretability tool for super-resolution networks. Xie et [1. find the most discriminative filters for each specific degradation in blind SR network, whose weights, positions, and connections are important for the specific function in blind SR. Magid el al. (37] use a texture classifier to assign images with semantic labels in order to identify global and local sources of SR errors. Shi et al. [45) show that Transformers can directly utilize multi-frame information from unaligned frames, and alignment methods sometimes harmful to Transformers in video super-resolution. They use a lot of interpretability analysis methods in their work. The closest work to this paper is the deep degradation representation proposed by Liu et al. [33]. They argue that SR networks tend to overfit degradations and show degradation "semantics" inside the network. The presence of these representations often means decrease in generalization ability. The utilization of this knowledge can guide us to analyze and evaluate the generalization performance of SR methods 34
B Transferability of Limited Training Images
At the end of the main text, we described method to improve the generalization performance of the deraining network by reducing the number of training background image images. However, this method will overfit the image content when the number of training background images is very small. In the main text, we investigate the risk of reducing the number of training images when testing on the same image category. Recall that with the increase of training images, the reconstruction of the background becomes better. Training with 256 background images can already bring a good background reconstruction effect. Continuing to add training images does not further improve the performance of background reconstruction.
15

---

ResNet
SwiniR
UNet
60-
CeiebA Urban Manca DN22
60
Celeba
50
CelebA
Background Recorstruciion
Urbon
Rncorstruction
Urban
50
50-
Manga DIV2K
40
Manga
DIVZK
40
30
Background
30-
Background
20
20
10
10
10
16
32
64
128 256 512 1024 30k
8
16
52
64
128 256 siz 1024 30k
16
32
de
120 256 siz 1024 30k
Training Background Patch Number
Training Background Patch Number
Training fackground Patch Number
100
CelebA
100
CelebA
Urban
Urban
Removal
90
Manga
Rain Rogian Ramoval
Mansa
60
80
DN2
80-
DNV2
70-
70-
Rain Region
55
60
50
Calbb
50
5
Manga
Urban
40
40
DN2K
256 Training Background Patch
16
32
64 128
$12 1024 30%
16
92
64
128
256
$12 1024 30k
16 35
64
128
256 S12 1024 30x
Training Background Patch Number
Training Background Putch Number
Figure 10: The relationship between the number of training images and their background reconstruc- tion performance (upper row) and rain removal performance (lower row). The test image set for these six plots is the DIV2K set. We train the model with all four image categories to validate the performance when the image distribution mismatch. For background reconstruction, lower values on the y-axis mean better background reconstruction. For rain removal, higher values on the y-axis mean better rain removal performance.
Background Reconatructon
Reshet 32 Channel, Blocks
Resivet 16 Channel, Blocks
90
ResNet 32 Channel, Blacks
ResNet 16 Channel, Blocks
10-
ResNeet 64 Channel, 16 Bocks
Rain Region Derovial
ResNet Channel, 16 Blocks
30
60
20
50
16
32
64
128
256
512
1024
8
16 32
64
128
256
512
1024
(a)
Training Backgoumd Complexity
(b)
Training Background Patch Number
Figure 11: The relationship between the number of training images and their background reconstruc- tion (a) and rain removal (b) performance. The test image set for these two plots is the DIV2K set. We train the models with different model complexities. For background reconstruction, a lower value on the y-axis means better. For the rain removal effect, a higher value on the y-axis means better. The test rain patterns are not in the training set. The effect of rain removal at this time reflects the generalization performance.
In this section, we investigate whether the proposed scheme is still robust when the training and testing image distributions are significantly different. We train the models on four image categories and then test them using the DIV2K image category. This simulates the situation when the background image distribution differs from the test set. We observe the behaviour of models trained with different numbers of images. The results are shown in Figure TO We draw the following conclusions. First of all, even if the distribution of training background images is different, the models trained using the images of CclebA and Urban categories can still perform similarly to the model trained by DIV2K images. These models can reconstruct background images well when training images reach 256 or more. At this time, the difference in the distribution of these training sets and DIV2K does not bring significant differences. The rain removal effect of these models is also similar. Second, we found that the model trained with the Manga image differed from others. The models trained with the Manga images are generally worse at background reconstruction than other models. Even with larger image numbers, the models trained with the Manga images still cannot achieve similar performance to other models. For rain removal, the model trained with the Manga images also performs the worst. These results are in line with our expectations because the Manga images significantly differ from other images, especially for the low-level image components. Although the other three categories of images
16

---

Inpad Rainy Image
SPO
RCON
Aestel Trared of 10k paxtes, Larg Rair
RCDNet Trained
SCDRet Trinnd
RCONet Trannd
Grund
HoptRaary Image
RCDNet Tisindd Large Rai
ECDRet Trannd W' pointes
RCD.ett Trained 125 posches, Large Rain
C Gonnd Tuuh
InpuRRaity
SPONet
RCDNet
RosNet Toined
ACDNet Trained ei Large Rail
R.COket Trained G4 potches
RCDWet Trained w/ 125 patches, Large Rain
GourdTuuh
Figure 12: Visualization of the deraining results. Zoom for better comparison.
also different from each other in image structure, texture type, and other characteristics, they all belong to the category of natural images. Whereas the Manga images contain artificial textures and edges, which are quite different from other images.
There is a large image reconstruction error when training with images whose distribution very different from that of the test set images. This is reasonable because, in this case, even using large number of training images cannot bridge the error caused by the distribution mismatch. We are pleased that the method of training using limited background images is robust image content to considerable extent, as long as the training images are natural images. This is consistent with our practice. In practice, also found that the results are stable long more than 256 image images are used for training. There is no significant performance change due to the content of the selected training images.
C Discussion about Model Complexity
The impact of model complexity on generalization performance cannot ignored either. general, models with smaller capacity are less prone to overfitting, although this is in exchange for generally lower performance. We next investigate the validity of our conclusions under models with different complexity. Specifically, we use three different sizes of ResNet architectures and conduct experiments
17

---

on the DIV2K dataset, aligning with the experiments presented in Figure and Figure 7 The results are illustrated in Figure [T] One can see from the results that all three models of varying complexities exhibit similar behaviours in rain removal performance. When the training background complexity on the lower side, the more complex models deliver superior rain removal generalization (as can be seen in the right figure, where higher is better). Conversely, with the increase of the training background complexities, these more complex models encounter larger generalization challenges. More importantly, the inflection point, where this transition occurs, remains consistent irrespective of the model's complexity. This confirms our explanation that within this training framework, the behaviour of the network is primarily determined by the relative complexity between the training background image and the rain degradation. Similar trends are noticeable in the context of background reconstruction. For more complex models, the quality of background reconstruction tends to degrade more when trained with a limited number of images. This can be attributed to the pronounced overfitting tendency of larger models. Interestingly, the rain removal performance is enhanced at this time, indicating the model's focus on image content. As the number of training images increases, the larger models also yield improved background reconstruction, which aligns with our expectations. It is crucial to highlight that the transition in network behaviour remains consistent across models of differing complexities. This reaffirms our conclusion that, within this framework, the network's behaviour is partly steered by the relative complexity interplay between the training background images and the rain degradation.
D More Results
We provide more results of different deraining models in Figure 12 and Figure 13 Note that we did not use additional data nor improve the network structure throughout the process. We only adjust the training objective. Although the effect of the output image can be further improved, it shows the practical value and application potential of our conclusions.
E Limitation.
Our work mainly takes the deraining task as breakthrough point, and attempts make a general summary of the generalization problem in low-level vision. Due to the differences between different low-level tasks, the analysis methods in this paper, especially the fine-grained analysis methods, may not be directly used on some other tasks. However, our work can still bring novel insights to the low-level vision field.
Our work also attempts to improve existing deraining models. But these improvements are based on the simple usage of some key conclusions of our work. Although shown to be effective, believe that these methods are still far from ideal. We only demonstrate the application potential of the knowledge presented in this work and have no intention of proposing of-te-of- algorithms or models. Research efforts are still needed to develop more robust deraining algorithms using our conclusions.
F
Reproducibility Statement
F11 Resources
The models used in our work are taken directly from their respective official sources. Our code is built under the BasicSR framework https://github..o htt.ttuu..moooooooeeeeeeeeeeeeee
The deraining model SPDNet [68] is available at xinntao/BasicSR for better code organization. deraining model RCDNet 5
http://ihhhb..omooooyy///////eee available at https:: gi thub. com//hongvangO/RRODeet The
The
training and testing datasets used in our work are all publicly available.
F.2 Network Training
Due to space constraints, we do not deseribe our training method in detail in the main text. Here, describe the training method to reproduce our results. A total of 150 models were involved in our experiments. We used the same training configuration for all models. We use Adam for training. The initial learning rate is 2 x 10-4 and 31 0.9, 32 = 0.99. For each network, we fixed the number of training iterations to 250,000. The batch size is 16. Input rainy images are of size 128x 228 The cosine annealing learning strategy is applied to adjust the learning rate. The period of cosine is 250,000 iterations. All models are built using the Py PyTorch framework (39 and trained with NVIDIA A100 GPUs
F3 Availability
All the trained models and code will be publicly available.
18

---

G Ethics Statement
This study does not involve any human subjects, practices to data set releases, potentially harmful insights, methodologies and applications, potential conflicts of interest and sponsorship, discrimi- nation/bias/faimees concerns, privacy and security issues, legal compliance, and research integrity issues. We do not anticipate any direct misuse of our contribution due to its theoretical nature.
19

---

HopuRaarty Image
SPONet
RCDNet
RecNet Taired 10K caxces.. La gc Rain
ACDNeL Traned in Large Rain
S.DOkot Trainod 64 porchics
RCONet Trained v/ 125 paschss Large Rain
GoundTuhh
Ioputt Palit Image
SPOK
RCDNNc
SusNot Traired ot pwwchss
RCDNet: Trained Large
SCDRet Trained potchtss
#CDNet: Trainss 128 patthes Raih
GourdTuch
Figure 13: Visualization of the deraining results. Zoom in for better comparison.
20


Reviews:
[{"review_id": "bnfd6rsAYW", "reviewer": "Reviewer_UxXt", "summary": "This paper makes observations on the generalization problem of deraining networks by conducting extensive experiments, on the perspective of training data. This paper claims to find out some underexplored insights that less training data with less complex background can lead to better rain removal ability at the cost of the background reconstruction ability and that these conclusions can benefit the future research of deraining community. Two new quantitative metrics are proposed to isolate the evaluation of ability of rain removal and background restoration and experiments are based on them.", "questions": "See the weaknesses above.\n\nIt is interesting to see such study on low level vision which is underexplored, but still, the observations made in this paper have some problems. I disagree that the background quality can be the cost of removing more rain, which is against the purpose of image restoration, because usually a less complex method tends to remove both rain and background due to less capacity. \n\nThe isolation of evaluation of ability of removing rain and restoring background also fails. Authors should reconsider their definition of reconstructing background. For deraining, the problematic areas are only rain regions rather than those unaffected background areas. The removal of rain and reconstruction of background take place in the same area while keeping the unaffected background unchanged is also important. Previous models years ago cannot handle them at the same time due to their inferior model capability and that's why recent models doing nothing to rain and keeping background tidy can get higher PSNR.\n\nFor a better improvement of this paper, I recommend authors to try newly proposed powerful models and find out what the difference is. The perspective of model complexity is also important when discussing the generalization problem.", "limitations": "NA.", "rating": 2, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. It is interesting and valuable to investigate the reason why existing deraining methods fail to generalize to unseen rain streaks, from the perspective of training data.\n\n2. It is reasonable to separately evaluate the ability of rain removal and background restoration of deraining models, for the two parts both have influence on the final restoration performance.\n\n3. It is good to conduct extensive experiments to validate the assumptions and useful for those new researchers to deraining tasks.\n\n4. The paper is easy to follow.", "weaknesses": "Despite the interesting motivation and efforts on studying the generalization problem of deraining methods, there are some basic problems for this paper.\n\n1. It is negative for the paper to hold the opinion that the background restoration quality can be sacrificed to get better rain removal ability. It is opposite to the practice of most image restoration tasks, i.e., keeping the degradation-free areas unchanged. Deraining, which consists of removing the rain streaks AND restoring those lost background details occluded by these rain streaks, is special that most part of the image is actually unaffected given only rain streaks exist and there is no rain accumulation. Thus, the objective of deraining is to remove as many rain streaks as possible when keeping the background unchanged. It agrees with the loss functions used to train models, that keeping the output more similar to the ground truth. The observation found in this paper, is actually not the deraining models overfit to the pattern of rain streaks, rather that models try to keep the background unchanged and do nothing to the image if rain streaks are not recognized. Training with less data indeed makes models to remove more rain, but also makes them to remove background details. That's why we need more powerful models to train on more diverse data to get better performance on real cases.\n\n2. The higher value of proposed metric 'Rain Removal Performance' doesn't mean the better ability of rain removal. As mentioned above, the goal of deraining is to remove the rain streaks and restore the occluded background textures. Thus, simply calculating the difference between between the rain region of rainy and output images, cannot directly tell the ability of removing rain. For example, in the extreme case, the rain removal ability is 'the best' when the rain region is all set to zeros because the difference between the zeros and white rain is maximized, but it is unwanted for deraining. In addition, the difference between the rainy and ground truth in rain regions is different for different backgrounds. Therefore, it cannot directly tell the ability of rain removal by the relative value of the proposed metric.\n\n3. The lower value of proposed metric 'Background reconstrution' doesn't mean the better ability of background reconstrution. Instead, it can only means that the models do less harm to background details when removing rain streaks. To be more specific, the reconstructed background only lies within the rain region rather than the unaffected background areas. Together with the above point, the metric of 'Rain Removal Performance' is a compoistion of the ability of removing rain and reconstructing background. Thus, the proposed two metrics acturally do not isolate the evaluation of removing rain and reconstructing background. Additionally, it can be found that only a certain amount of background images like 256-1k can help models to learn to keep unaffected areas unchanged, which may be good for future data synthsis.\n\n4. The rain region is obtained by thresholding to calculate the new metrics. It is weird because the experiments are conducted on full synthetic data and the rain is manually superimposed on clean images, which means the accurate rain areas are already known in advance. The threshold value can also have negative effects on the correctness of observations.\n\n5. It is infeasible to make more conclusions than that L1 loss for training is not the optimal choice when the whole training is based on L1 loss. Training with L1 loss aims to tell models to produce more similar outputs to ground truth pixel-wisely and evaluating the performance with PSNR and SSIM is persuing better visual quality. While this paper proposes new evaluation metrics and can only prove that L1 loss cannot help models to remove more rain at the cost of removing background details as expected by the paper. It is more important to make real solutions based on the proposed metrics to guide the training of models to get better results. But it is not considered in this paper.\n\n6. It is simple to make seem-to-be-valuable observations via various experiments and such efforts should be respected. However, such conclusions can also be harmful to researchers who are new to this area if no absolute therories supports such insights. Authors also knows that better background preservation can lead to higher PSNR but remove less rain streaks as described in Sec 2.2. The marching direction of the community should be removing more rain streaks and handle more types of rain while keeping the background textures unchanged rather than removing more rain at the cost of background quality. It is recommended that authors also examine the generalization problem from the perspective of model complexity, and you may find that less complex models tends to remove more rain streaks at the cost of removing more backgrounds rather than doing nothing like more complex models.\n\n7. Let's make a step backward, assuming these observations in this paper are insightful, the strategy proposed by authors that trying to find a balance of complexity of background and rain streaks should be applicable to all deraining models, no matter what architectures and complexities they have. However, the quantitative and qualitative results show that RCDNet may be the only benefited model of the proposed strategy. Thus, it is not a general solution that less training data brings better balance and generalization.\n\n8. The selected SPDNet and RCDNet, etc. are actually not the state-of-the-art models in deraining. There are many more powerful deraining-oriented models or general backbone models in both CNN-based and Transformer-based architectures these years. It is more insightful to examine how these new powerful models behave when the training data of background images is decreasing.\n\n9. For the second implication in Sec 4., is there any chance that larger rain range occludes more background areas and models need more data to learn how to keep unaffected regions unchanged and restore those occluded ones? Different perspectives can lead to various conclusions based on these experimental results.\n\n10. As the LLM and SAM models in NLP and CV areas show up, it is more likely that surprisingly large amounts of training data is the key to generalization problem. However, it is hard to get pair-wise ground truth for so many low-level image restoration tasks, thus it is also valuable to study the possibility of training models to be better with proper data. But the insights claimed in this paper seem to be normal observations when conducting experiments. Most importantly, authors do not give any new solutions based on these observations like new loss functions. Actually, it is promising to design new loss functions in the spirit of isolation of removing rain and restoring background.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank Reviewer UxXt (R#1) for his\\her efforts. We next responded to R#1's comments. To save space, we set some abbreviations: generalization (GNL), background (BG).\n\nR#1's resistance to data reduction likely arises from a misinterpretation of our intentions. Through experiments, we aim to challenge prevailing notions of low-level vision GNL. We maintain that not all future methods necessitate data reduction for superior GNL performance. Moreover, we do not claim there is a trade-off between BG reconstruction and deraining effects. As an interpretability study, we neither propose a universally applicable method nor implies the practical viability of the method used in this paper. Our perspectives are neither \"negative\" nor \"harmful\". In contrast, our view is insightful and worth discussing.\n\nConducting experiments to shed new light on the GNL issue is very valuable for the community. While there is an undeniable need for more robust models and efficient loss functions, our results have show interesting and important phenomena. We argue that exploring the causes of the GNL problem holds more importance than proposing a new method or metric, particularly in an era where studies investigating low-level vision interpretability are limited. We take pride in this contribution.\n\n`Weakness 1`: The R#1's statement is common in the existing literature, and it is also the \"unproven acknowledgment\" mentioned in our paper. But, how does the model recognize the BG to keep and the rain to remove? The prevailing academic assumption posits that the model automatically opts to learn the BG. This paper's insight is to challenge this belief through experimentation. Maintaining an \"unchanged BG\" necessitates identifying either rain or BG, which is where the GNL issue originates. The R#1's statement, \"do nothing...if not recognized\" implies R#1's belief that current models identify rain, and their inability to generalize stems from failed rain recognition, thereby contradicting R#1's own critique of our work.\n\n`Weakness 2 and 3`: These metrics are proposed for separate quantitative studies of rain removal and BG reconstruction, and we do not purport them as universally applicable. (1) For deraining, given rain's usual high brightness, this metric can reflect rain removal efficacy. While susceptible to corner cases, we've provided visual results that corroborate our metrics' effectiveness. The corner case mentioned by R#1 do not undermine this metric. Hypothetically, if a method outputs an entirely black in rain part, it has technically \"removed\" the rain. (2) Regarding BG reconstruction, including rain-affected areas in computations would allow the deraining effect to influence BG reconstruction evaluation. This violates our original intention of studying rain removal and BG reconstruction separately. The deraining effects would be reflected in whole-image performance indicators, and is not what this paper trying to do.\n\n`Weakness 4`: Understandably, we know the precise location of raindrops, and the non-zero part of the raindrop image should represent the background. However, the raindrop synthesis method results in a broad non-zero area surrounding the raindrops, causing the mask's range to exceed actual raindrop dimensions. Thresholding can mitigate this issue. Given our low threshold, this process doesn't impact our conclusions.\n\n`Weakness 5 and 6`: Please see the rebuttal PDF.\n\n`In weakness 7 \"...RCDNet may be the only benefited model...\" and in weakness 8 \"The selected...are not SOTA\"`: We respectfully disagree with your comments. Our results consistently show that both Unet, ResNet and Transformer SwinIR gain from the described training method. Almost all experiments speak to these facts. RCDNet and SPDNet serve as additional models to align with existing deraining research. As a paper centered on elucidating the deraining model's generalization mechanism, surpassing existing state-of-the-art results isn't requisite. Our research encompasses various models from existing literature, including the classic Unet-shaped network, the CNN network, and SwinIR. Thus, our conclusions exhibit considerable widely validity. Although employing newer models or examining all models would be ideal, it is unfeasible for a 9-page paper.\n\n`Weakness 9`: This is a good question. First of all, we are aware that the range of actual occlusion by rain may have a little influence on the experiments. Therefore, in different ranges of rain settings, the number of raindrops is the same to reduce the impact caused by occlusion. The reviewers also mentioned that due to occlusion, the model is forced to learn more complex BG reconstruction. In fact, according to our insight, if the model is forced to learn to reconstruct the BG, this also improves the GNL ability of the model. A recent paper proves our point [3]. Their experiments were conducted on the denoising task. They found that the existing denoising model does not learn the content of the image (this is consistent with our point of view), causing the GNL performance of the denoising model to fail. Through deliberate masked training, the network is forced to learn the reconstruction of image content, This improves the GNL performance.\n\n`Weakness 10`: The large data is not the only key to the amazing ability of large models. The training method is also very important. Giving a large data to a simple task will not make the model generalize to other tasks. In low-level vision, we argue that the task of learning an end-to-end input-to-output mapping via a simple image comparison loss may be too simple. An important factor for the success of the large language model is its generative pre-training learning method. This learning method enables the model to learn the content of the image, rather than the mapping described by the input and output. This is also the core point of this paper, that solving the GNL problem requires learning the content of the image, not degradation."}, {"review_id": "jfIe2xRkBO", "reviewer": "Reviewer_e5xd", "summary": "This paper is working on understanding generalization problem in image deraining. Several interesting findings are: 1) training with fewer background images leads to better deraining effect; 2) the relative complexity between the background and rain determines the network behavior; 3) a more complex background set makes it harder for the network to learn.", "questions": "see the weakness part.", "limitations": "yes, authors provided some limit of this paper, for example, this paper doesn't propose any new algorithms directly.", "rating": 3, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "there are some interesting findings in this paper for image deraining for example\n1) training with fewer background images leads to better deraining effect.\n2) the tradeoff between deraining and background reconstruction.", "weaknesses": "1) although there are interesting finding, the overall contribution is limited, i.e., authors didn't provide a good solution to solve the problem. for example for figure 9, although results from author removed rain, but it also generated much more artifacts in the background which probably not good for image deraining task. I would expect more contribution from authors to make this paper accepted by NeurIPS. \n\n2) My feeling is that there is fundamental limit in the current training for image derain, the simulated rain streaks is not real. I wondered wether the current findings from author because of this fundamental limit. \n\n3) some related papers like [1] is not cited or compared. \n\n[1] Jie Xiao, Man Zhou, Xueyang Fu, Aiping Liu, Zheng-Jun Zha, Improving De-raining Generalization via Neural Reorganization, ICCV 2021.\n\nI have carefully read authors rebuttal and still thought the quality of this paper could be improved. So will keep my rating.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank Reviewer e5xd (R#2) for his\\her efforts. Thank you for finding our conclusions interesting. Our response to your concerns is as follows:\n\n`In weakness 1) \"the overall contribution is limited, i.e., authors didn't provide a good solution to solve the problem.\" and \"I would expect more contribution from authors to make this paper accepted by NeurIPS.\"`:\nWe respectfully disagree with the R#2's comment. It has been a long time that low-level vision research has predominantly focused on results. Many researchers overlook the fundamental causes of problems, espacially since deep leanring is applied in low-level vision. It seems that any research that doesn't directly yield positive results, cannot be directly applied to algorithm development, or doesn't introduce a new method should neither exist nor be reported or discussed. We are disappointed and frustrated by this prevalent understanding within the field. By this rationale, many seminal papers should never have been accepted. We do not believe that introducing a novel algorithm is a prerequisite for a paper to be accepted by NeurIPS. Our work is devoted to elucidating the underlying causes of the generalization issue in low-level vision, complemented by explorative experiments. The conclusions drawn in our paper can inspire a lot of low-level vision models that are generalizable in real-world applications.\n\nFurthermore, our research has demonstrated, through experimentation, that effects unattainable in current research paradigms can be achieved with just a straightforward approach. Such insights can directly inspire numerous studies. For instance, a recent work [3] which focused on denoising tasks. They found that existing denoising models don't learn image content (which aligns with our views), leading to failures in their generalization capabilities. Through deliberate masked training, they forced networks to learn content reconstruction, thereby enhancing generalization performance. Although the specific methods differ, the underlying approach is to shift training objectives towards understanding image content, rather than overfitting to degradation. This underscores the value of our work. Therefore, we strongly contest the reviewer's critique of our contributions.\n\n`In weakness 1) \"for figure 9, although results from author removed rain, but it also generated much more artifacts\"`:\nWe are confused by the R#2's comment. Figure 9 presents a comparison of our method with existing techniques on real images. In fact, our approach successfully removed the white raindrops while preserving the texture in the background. In contrast, most methods we compared against couldn't effectively eliminate the raindrops. This alone validates the efficacy of our method. We did not observe the artifacts mentioned by the reviewer in the images. We surmise that the reviewer might have mistaken the residual background textures for artifacts. These textures are intrinsic to the background and are not the result of any shortcomings in our outcomes.\n\n`In weakness 2) \"My feeling is that there is fundamental limit in the current training for image derain, the simulated rain streaks is not real.\"`:\nThis statement of the R#2 is correct. The generalization problem in low-level vision refers to the performance drop or model failure that occurs when the test degradation is inconsistent with the training degradation. As we explain in the text L316 to L325. There are two main ideas to solve the generalization problem so that the model can be applied in real scenarios. One is to make the training degradation as close as possible to the testing degradation. Existing attempts to propose image synthesis methods, or through real-world data pairs, or unpaired training methods using real data are essentially this idea. Another idea is to improve the generalization performance of the model. Allows a model trained on one type of data to work on a different test data. This is the focus of this paper. In order to study the generalization problem, it is very important to construct the scene of the generalization problem. The rain simulation method we use provides a suitable research scenario for the study of generalization problems. We can use more accurate raindrop simulation methods to improve its effect in real scenes, but this has little to do with the exploration of generalization in this paper.\n\n`In weakness 3) \"some related papers like [1] is not cited or compared\"`:\nThanks to R#2 for the question. However, we have discussed the literature mentioned by the reviewers in the supplementary material. In the supplementary material we provide a more detailed literature review. The [32] citation is the paper mentioned by the R#2.\n"}, {"review_id": "HThNPguuIM", "reviewer": "Reviewer_w1Er", "summary": "This paper proposed to explore the problem of generalization problem in image deraining. The authors find that deep networks tend to overfit the degradation patterns present in the training set, which leads to poor generalization performance for unseen degradation patterns.", "questions": "Is the main findings of this paper applicable to other low-level vision tasks? Has the author conducted any extension experiments for other tasks?", "limitations": "The authors adequately addressed the limitations.", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Writing fairly well.\n\n2. The problem is interesting. It is very interesting to explore such findings that better generalization in a deraining network can be achieved by simplifying the complexity of the training data. None of previous works have discussed these issues.", "weaknesses": "1. Why did the author create a new dataset without conducting experiments on previous existing benchmarks. The author should explain this issue.\n\n2. Is there any connection between the generalization problem proposed by the author and the interpretability of the low-level vision networks? I am confused about the conclusion of this paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank Reviewer w1Er (R#3) for his\\her efforts. We also thank R#3 for finding our conclusions interesting and novel. Our response to your concerns is as follows:\n\n`In Weakness 1): \"Why did the author create a new dataset without conducting experiments on previous existing benchmarks.\"`:\nWe appreciate the reviewer's question. In this paper, our aim is to showcase specific behaviors exhibited by models when using neural networks to tackle the rain removal problem, achieved through meticulously designed experiments. In previous benchmarks and research settings, all variables were intermingled, making it challenging to isolate elements such as the distribution of rain or the distribution of background images. This complication hindered our past analyzes of model performance. Therefore, in this study, we manually curated our training data and training paradigm, striving to se partly discuss and analyze each variable. We do not intend to create a new benchmark, we just study the behavior of deep low-level vision methods in a purer environment.\n\n`In Weakness 2): \"Is there any connection between the generalization problem proposed by the author and the interpretability of the low-level vision networks?\"`:\nThe issue of interpretability in low-level vision encompasses various aspects. Gu and Dong [11] delved into how deep models in low-level vision utilize input information. Xie et al. [33] investigated from the perspective of specific network filters. Magid and colleagues [19] analyzed low-level vision models based on the errors in their results. Shi et al. [22] studied the alignment modules of video super-resolution networks. Meanwhile, Liu and his team [17] analyzed the semantic degradation in low-level vision.\n\nAnalyzing the generalization issues in low-level vision and pinpointing the inherent causes of such issues is one of the important topics in low-level vision interpretability research. However, this question has not yet been explored in existing literature. As the first paper dedicated to explaining generalization problems in low-level vision, this work holds significant value.\n\n`In Question: \"Is the main findings of this paper applicable to other low-level vision tasks? Has the author conducted any extension experiments for other tasks?\"`:\nThank you for the question. Indeed, our findings can be applied to many other low-level vision tasks, such as real-world image restoration, which closely align with the RealESRGAN setting [R1], and image denoising. We have conducted very similar experiments, and the conclusions align. A recent paper proves our point [3]. Their experiments were conducted on the denoising task. They found that the existing denoising model does not learn the content of the image (this is consistent with our point of view), causing the generalization performance of the denoising model to fail. Through deliberate masked training, the network is forced to learn the reconstruction of image content, This improves the generalization performance.\n\nThe reasons for focusing on deraining as the subject of our study are described in lines L34 to L43 of the paper. To summarize, the primary reason is that the degradation in deraining is relatively simple, and our proposed method allows for a clear separation between deraining performance and background reconstruction performance. In contrast, other low-level vision tasks are challenging to study through quantitative means for controlled variables; conclusions can only be drawn qualitatively based on the outcomes.\n\n[R1] Wang, Xintao, et al. \"Real-esrgan: Training real-world blind super-resolution with pure synthetic data.\" Proceedings of the IEEE/CVF international conference on computer vision. 2021.\n"}, {"review_id": "H1aKXJ8K9Z", "reviewer": "Reviewer_FkGF", "summary": "This paper explores the generalization problem of image deraining. The authors claim that deep networks display a tendency to learn the less complex element in the task of separating image content from additive degradation. Overall, this paper contributes some new ideas and new insight to the community, but there are still some weaknesses that can be further improved.", "questions": "See the above Weaknesses part.", "limitations": "The author has clarified the limitations in the supplementary materials.", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "S1. The paper is well-organized and clearly written. \n\nS2. The authors point out the the generalization problem in the context of deraining tasks, which contributes some important ideas for the community. The key findings of this paper left a deep impression on me. \n\n\n", "weaknesses": "W1. The author chose two CNN-based methods, RCDNet and SPDNet, to validate the main finding of this paper. I am curious whether the latest Transformer-based image deraining method still has the same conclusion. For example, Chen, et al. \"Learning A Sparse Transformer Network for Effective Image Deraining.\" CVPR, 2023. In fact, the generalization performance of CNN-based and Transformer-based deraining methods is different. I suggest the author could add the experimental analysis of the Transformer-based deraining method.\n\nW2. The authors claim that the relative complexity between the background and rain determines the network behavior. In previous studies, Rain100L and Rain100H are two classic image deraining benchmark datasets that share the same background image. As far as I know, Rain100H (rain is more complex) has better generalization performance than Rain100L. I am curious about how to define the relative complexity between the background and rain. The expression here needs further improvement, which is a bit difficult to understand.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank reviewer FkGF (R#4) for his review. We also thank R#4 for acknowledging our conclusions and contributions. Our response to your concerns is as follows:\n\n`In Weakness 1): \"I am curious whether the latest Transformer-based image deraining method still has the same conclusion.\"`: Thank you for the reviewer's suggestion. In our paper, we primarily utilized SwinIR as a representative of the Transformer networks. Our results indicate that, when trained under the same conditions, SwinIR does not exhibit superior generalization performance compared to CNN-based networks. Given SwinIR's role as a representative method of Transformers and its inherently simple and effective Design design, we infer that Transformers do not inherently offer enhanced generalization capabilities. However, we cannot conclude that no Transformer design can achieve substantial improvements in generalization. There might be newer methods that address the generalization issues of Transformers: reference [3] serves as a case in point. While they also used SwinIR, they introduce the attention and input mask module, significantly enhancing the model's generalization capacity. Based on our tests, the model proposed in [3] is also effective in improving the generalization capability for the deraining task. The paper mentioned by the reviewer has been accepted for CVPR2023. Since it was not publicly published at the time of our submission, we did not discuss or cite it in our initial version. We have added a discussion about this paper in our revised version. We appreciate the reviewer bringing this to our attention.\n\n`In Weakness 2): \"As far as I know, Rain100H (rain is more complex) has better generalization performance than Rain100L.\"`:\nThank you for the question. Compared to Rain100L, Rain100H features denser rain, larger raindrops, and more obscured background content. The enhanced generalization ability of models trained on the Rain100H dataset might stem from the broader extent of occlusion. This intriguingly aligns with the approach utilized in reference [3]. In that work, they discovered that when training denoising networks, adding a random mask layer to the input image largely transforms the denoising task into an inpainting task. Consequently, in an effort to minimize the loss, the network is compelled to learn background reconstruction. This, in turn, results in improved generalization performance, which is consistent with our paper's findings. Returning to the reviewer's question, due to the more extensive background occlusion by rain in Rain100H, the network focuses more on learning image content rather than raindrop distribution. This might be the reason for its superior generalization capability.\n\n`In Weakness 2): \"I am curious about how to define the relative complexity between the background and rain.\"`:\nThank you for the question. Admittedly, this is a challenging question to address. Defining a quantitative measure for such complexity is daunting. Complexity isn't solely about the data; the design of the model itself also plays a significant role. Consider a network specifically crafted for learning raindrops or human faces; for such architectures, the \"complexity\" of learning raindrops or faces might be relatively lower. At this juncture, we cannot provide a mathematically sound measure that quantifies all these elements. However, our approach manifests this relativity in another manner. As the number of images increases, their learning complexity monotonically rises, while the complexity of the rain remains constant. We observed a shift in network behavior throughout this rising process, affirming the existence of relative complexity.\n\nFrom another perspective, in real-world applications, the complexity of background images always greatly surpasses that of degradation. This accounts for the prevalent generalization problems in low-level vision tasks. Overcoming generalization issues by merely reducing the complexity of background images isn't appropriate. This realization inspires us to develop more sophisticated training strategies and models.\n\nReviewer can also encouraged to check the new experiment provided in the rebuttal pdf for more information."}]

Author rebuttal:
Reply to all reviewers and area chairs for short abstracts


Dear Reviewers and Area Chairs,
We thank all reviewers and regional chairs for their valuable time. We sincerely appreciate all comments. They gave positive, high-quality comments to our paper and provided a lot of constructive feedback.

We are encouraged by:
1. **All reviewers agree that our findings and conclusions are interesting.**
2. Reviewers UxXt, reviewer w1Er, and reviewer FkGF **agree that our work is valuable and contributes to the field.** Reviewer w1Er commented "None of the previous works have addressed these issues"
3. Reviewers UxXt, reviewer w1Er and reviewer FkGF **agreed that our paper is well written and easy to understand.**

Our research aims to uncover the root causes of low-level vision challenges, with a particular focus on the generalization problem that we study experimentally. But for a long time, low-level vision research only emphasized benchmark results. Many researchers are not concerned with the root cause of the problem and thus ignore interpretability research, especially since deep learning is applied to low-level vision. We noticed that some reviewers were confused about whether our paper gave a way to surpass SOTA and the final performance metrics. It seems that any research that does not directly produce positive results, is not directly applicable to algorithm development, or does not introduce new methods should not exist, and should not be reported or discussed.

**We would like to point out that there should not be only one kind of paper appearing in the low-level vision community.** As a low-level vision interpretability research, this paper has a unique structure. The way to contribute to the field is also different from other papers that focus on novel methods. Hope the reviewers will again consider this paper as a contribution to the low-level vision science. We have replied to each reviewer individually to address any comments. We also provide experiments on model complexity.

We look forward to addressing any concerns the reviewers had. I noticed that we can continue discussions on the Openreview system for some time to come. If the reviewers has any questions, we would be happy to continue the discussion.

Once again we thank all reviewers and regional chairs!

Paper 798 author.


Meta-review: The reviews for this paper were extremely polarized, even after rebuttal and discussions. The two reviewers strongly in favor argued that the novel insights should outweigh the non stellar empirical results. The other two reviewers argued that good empirical results are necessary (e.g. in background reconstruction). I side with the favorable authors as good insights are valuable to the community, although I acknowledge that the empirical results are also important. I encourage the authors to take feedback from all reviewers and incorporate them into the final version.
Decision: Accept (poster)