Networks are Slacking Off: Understanding Generalization Problem in Image Deraining
Authors: Jinjin Gu
Xianzheng Ma
Xiangtao Kong
Yu Qiao
Chao Dong
Venue: Submitted to ICLR 2023 (2023)

Abstract: Deep low-level networks are successful in laboratory benchmarks, but still suffer from severe generalization problems in real-world applications, especially for the deraining task. An ``acknowledgement'' of deep learning drives us to use the training data with higher complexity, expecting the network to learn richer knowledge to overcome generalization problems. Through extensive systematic experiments, we show that this approach fails to improve their generalization ability but instead makes the networks overfit to degradations even more. Our experiments establish that it is capable of training a deraining network with better generalization by reducing the training data complexity. Because the networks are slacking off during training, i.e. learn the less complex element in the image content and degradation to reduce the training loss. When the background image is less complex than the rain streak, the network will focus on the reconstruction of the background without overfitting the rain patterns, thus achieving a good generalization effect. Our research demonstrates excellent application potential and provides an indispensable perspective and research methodology for understanding the generalization problem of low-level vision.


Under review as a conference paper at ICLR 2023
NETWORKS ARE SLACKING OFF: UNDERSTANDING GENERALIZATION PROBLEM IN IMAGE DERAINING
Anonymous authors Paper under double-blind review
ABSTRACT
Deep low-level networks are successful in laboratory benchmarks, but still suffer from severe generalization problems in real-world applications, especially for the deraining task. An "acknowledgemen"" of deep learning drives us to use the train- ing data with higher complexity, expecting the network to learn richer knowledge to overcome generalization problems. Through extensive systematic experiments, we show that this approach fails to improve their generalization ability but instead makes the networks overfit to degradations even more. Our experiments establish training a deraining network with better generalization by re-
that it is capable ducing the training data complexity. Because the networks training, i.e. learn the less complex element in the image content and degradation reduce the training loss. When the background image is less complex than the rain streak, the network will focus on the reconstruction of the background with- out overfitting the rain patterns, thus achieving good generalization effect. Our research demonstrates excellent application potential and provides an indispens- able perspective and research methodology for understanding the generalization problem of low-level vision.
slacking off during
INTRODUCTION
The whirlwind of progress in deep learning has produced a steady stream of promising low-level vision networks, which significantly outperform traditional methods in existing benchmark datasets. However, the intrinsic overfitting issue has prevented these deep models from real-world applica- tions, especially when the degradation differs lot from the training data. We call this dilemma the generalization problem. Although important, this problem not well studied in low-level vision literature. We need more in-depth analysis and understanding, before proposing effective solutions.
Understanding generalization in low-level vision is by no means easy. It is not a naive extension of the generalization research in high-level vision. We need dedicated analysis tools to interpret new phenomena. In this paper, we hope to build stepping stone towards a more in-depth understanding of this problem. To achieve this goal, we select representative low-level vision task as the break- through point, and design quantitative analysis methods for several controlling factors. The heart of our methodology stated as follows.
Select deraining as the representative task. Low-level vision includes many tasks, such im- age denoising and super-resolution, which have different characteristics. A general understanding of generalization across all low-level vision tasks cannot be built in day. Thus, we choose the image deraining task as a representative. Image deraining aims to remove the undesired rain streaks in an image. There are two considerations for selecting the deraining task. First, as a typical decomposi- tion problem, image deraining has a relatively simple degradation model (a linear superimposition model). This will facilitate our research and enable the usage of many quantitative mcasurements. Second, the deraining task suffers from severe generalization problem. Existing deraining models tend to do nothing for the rain streaks that are beyond their training distribution. See Figure for an example. This phenomenon is very intuitive and easy to quantify.
Analyze from the perspective of training data. We argue that the generalization problem is due to the network overfitting the degradation (the rain patterns in the deraining task). The main reason for this result is the inappropriate training objective. We start analysis with the most basic and indispensable factor in constructing the training objective- training data. There has been a lot of works trying improve real-world performance by improving the complexity of training data. This

---

Under review as a conference paper at ICLR 2023
(a) Input Rainy Image
(b) Basaline Model
(c) Baseline Model, More Background Training Data
[d) Baseiine Model, More Rain Data
(e) Reducing Traning cata According to our Conclusion
Figure I: The existing deraining models suffer from severe generalization problems. After training with synthetic rainy images, when feeding (a) an image with different rain streaks, its output (b) shows limited effect. Two intuitive ways to improve generalization performance - (c) adding back- ground images, and (d) adding rain patterns, cannot effectively relieve the generalization issue. In this paper, we provide a new counter-intuitiin insight - (e) we improve the generalization ability of the deraining networks by selecting much less training background images for training, not more.
comes from a natural but unproven "acknowledgment" in low-level vision that more training data can solve the generalization problem. This acknowledgment also influences the deraining committee: when the network sees more (both background images and rain streaks), can generalize better to more weal-world scenarios. However, the generalization problem of deraining is NOT solved in this way. These methods still do not work on rain patterns that have not yet been collected. We argue that because too much background data is provided for training, the model cannot learn to reconstruct the image content and can only overfit to the degradation. Therefore, we propose to reduce the number of training background images in our study, rather than increase it further.
Our analysis methods. To systematically study the changes in model behavior brought about by changing training objectives, we construct a number of training sets consisting of different back- ground images. We first investigate the effect on the number of training set images by overfitting the model on very few (16 even 8) images. By switching between different image categories, we study the network behaviour when fitting images of different complexity levels. We study the rela- tionship between the complexity of the background image set and the generalization performance of deraining through extensive quantitative experiments.
Except for constructing training objectives, we also perform fine-grained analysis of the model outputs. Previous works simply use the overall image quality as the performance indicator, such as PSNR. However, the reason for the quality deterioration may be either the unsuccessful removal of rain streaks or the poor reconstruction of the image background. Thus we decouple the deraining task as rain removal and background reconstruction, which are studied separately. Since the gener- alization problem in the deraining task is mainly related to the removal of rain streaks, fine-grained analysis can exclude the influence of other factors.
Our key findings. We find that deep networks are slacking off during training. They take shortcuts in reducing the loss, resulting in poor generalization performance. This is due to the inappropriate objective we set for training. Our key finding can be summarized as:
Between the image content and the additive degradation, deep networks tend to learn the less complex element in the separation lask.
Specifically, in the common training data with high background complexity and low rain complexity, the network will naturally learn to identify and separate rain streaks, because they are less complex and easier to learn. But when the real situation deviates from its depiction of rain, the network tends to ignore them and gets poor generalization performance. On the contrary, when we train the model on less complex background image set, it exhibits better generalization ability, see Figure (e). The reason is that when the complexity of the training image background is smaller than that of the rain patterns, the network will also take shortcut to reduce the loss, i.e., remember the recon- struction of the background instead of overfitting to the rain streaks. Except for the removal of rain, the performance of the model is also determined by the background reconstruction. Reducing the background complexity of the training data could inevitably produce unsatisfactory reconstruction results. However, our results show that the model trained only 256 images can already handle
2

---

Under review as a conference paper at ICLR 2023
most of the image components. These counter-intuitive phenomena have never been studied and valued in the literature.
Implication. Given the current literature, our results are interesting and inspiring. Our results demonstrate the importance of the training objective in determining the generalization ability. An inappropriate and incomplete training objective provides an opportunity for deep networks "slack off"" Although we hope that the network can learn the rich semantics in natural images, it is of- ten overlooked that the low-level vision system can achieve learning goals through some shortcuts. While these shortcuts lead to poor generalization performance of the model. Our results also illus- trate that a model with good generalization ability should learn the distribution of the natural images themselves, rather than overfit the degradation. By simply exploiting our findings, we can make the simplest networks exhibit excellent generalization capabilities. This shows that our findings have great potential for application.
1.1 RELATED WORKS
This work is first related to dereaining research. But do not propose new network structures, loss functions, or datasets like most existing deraining works. Our work is aimed at the analysis and understanding of the generalization problem in the deraining task. Due to the limited space, the de- raining works are reviewed in Appendix A We next review previous works about interpretability and understanding of generalization in low-level vision.
Deep learning interpretability research aims to understand the mechanism of deep learning methods and obtain clues about the success or failure of these methods. Without a deep understanding of these working mechanisms, we are not convinced to move forward in the right direction. The research on deep learning interpretability follows a long line of works, most of them focusing on the classification task Simonyan et al.] (2013): Springenberg et al. (Srrkus
(Shrikumar et al. (2017); Sundararajan et al. (2017); Zhou et al. 2018): Lundberg & Leel (2017). Most low-level vision tasks have also embraced great success with powerful deep learning techniques. There are also works on interpretability for these deep low-level networks Gu & Dong (202)); Xie et al. (202I); Magid let al. (2022). For the generalization problem in low-level vision, these problems often arise when the lesting degradation model does not match the degradation used training, e.g., different down- sampling kernel Gu et al. (2019); Liu et al. (2022a); Kong et al. (2022) and noise distribution Guo jet al. (2019). The existing works either develop blind restoration methods to include more degrada- tion possibilities in the training process or make the training data closer real-world applications.
Only a little work has been proposed to study the reasons for this lack of generalization performance Liu et al. (2021; 2022b). More details of these previous works can also be found Appendix. No research has attempted to investigate the interpretation of the training process of low-level vision networks, especially from the perspective of the generalization problem. Our work fills this gap.
2 ANALYSIS METHOD
Our goal is to explore how different training objectives affect the network behavior and general- ization performance. Before introducing our observations, we need to describe the experimental designs and quantitative analytical methods in this section.
2.1 CONSTRUCTION OF TRAINING OBJECTIVE
The training objective of deep network is jointly determined by the training data and the loss function. We set a variety of training objectives in order to observe the changes in the generalization performance of different deraining models. As shown in Figure 3 (left), a rainy image O can be roughly modeled using linear model O B R. where B is the image background, and R is the additive rain streaks. We will change the training objectives with different background images and rain streaks.
Background Images. Typically, image backgrounds are sampled from street view images Geiger jet al. (2012) or natural image datasets SShhaffer & Stich (2003); AArbelazz et al. (20TO), as these images are close to the application scenarios of deraining. In the literature, the previous works Fu et al. (2017b); Zhang & Patel (0018) claim that the model can learn the prior knowledge of reconstructing these scenes by training on large number of such background images. We break this common sense by constructing different training background image sets from the following two aspects.
For the first aspect, we change the number of background images. We argue that as too much background data are provided for training, the model cannot faithfully lear to reconstruct the image
3

---

Under review as a conference paper at ICLR 2023
(a) Background image Spacas
(b) Rain Streaks for Training Tasting
Urben100
Training Rain Small Kange
T,t te
Manga105
Training Medium Range
Training Rain Large Range
DIV2K
CalebA
R100L Rain for Tosting
Figure 2: (a) Background images from different image datasets. It can be seen that the structure of the face image (CelebA) is relatively complex. Natural image patches (DIV2K) contain natural tex- tures and patterns. The patterns in Mangal09 and Urban 100 are artificially created - Manga images have sharp edges, while Urban images contains lot of repeating patterns and self-similarities. (b) Rain streaks used our experiments.
content but overfit the degradation patterns. To demonstrate this comment, we reduce the complexity of the background images to see how the network behavior changes in extreme scenarios. In our experiments, use 8, 16, 32, 64, 128, 256, 512, and 1024 background image patches of size 128 x 128 to build the training datasets, respectively. We also use a large number of patches (up to 30,000) simulate the common situation when the image background is sufficiently sampled.
In addition to the number of images, the image content will also affect the learning procedure. For images with many self-similar regular patterns, is easier for the network to remember and extract semantic features. While a face image that contains both short- and long-term dependent structures is apparently more complex than a skyscraper that consists of just repeated lines and grids Bagrov et al. (2020). We carefully choose the image distribution as the second aspect of our dataset construction. We sample from four image datasets that are distinct from each other: CelebA (face images) Liu et al. (2015), DIV2K (natural images) Timofte et al. (2017). Mangal09 (comic images) Matsui et al. (2017). and Uerban 100 (building image) Huang et a.. (2015). Some examples of these images are shown in Figure ( (a).
Rain streaks synthesis. Since it is hard to collect large number of real-world rainy/clean image pairs, we follow the previous deraining works Garg & Nayar (2006); Fu et al. (2017b) to synthesize rainy images for research. We use
Table I: Different rain streaks synthesis ranges.
Range
Quantity
Width
Length
Direction
Small
200, 300
[5}
30,31
[-5".5"]
Medium
200, 300
(5.7.9)
200. 40j
-30" 30")
Large
200, 300
(1.3.5.7.))
5.600]
-70", 70")
two different kinds of rain streaks for training and testing, separately. For training, we use the com-
putational model [ to render the streaks left on the image by raindrops of varying sizes, densities, falling speeds, and directions. This model allows us to sample rain streaks from different distribu- tions. We adopt three rain image ranges for training, where different ranges may lead to different generalization effects, see Figure (b) for convenient visualization and Table T for the detailed set-up. For testing, we use the synthetic rain patterns presented by Yang et al. (2017). Although in both cases the rain streaks simulated and visually similar to humans, they still pose huge generalization challenge to existing deep models.
Loss Function. In low-level vision, the loss function is usually defined by the difference between the output image and the ground truth. our study, we use the -nom loss to simplify the problem. In this setting, our conclusions do not lose generality and can be extended to other similarity- training loss functions.
2.2 FINE-GRAINED ANALYSIS.
Generally, the evaluation of deraining model is to compute similarity metrics (e.g., PSNR) be- tween the output and ground truth images Gu et al. (2020). However, such an evaluation on the whole image may lead to unfair comparison. For example, image with perfect background recon-
'The reimplementation of the PhotoShop rain streaks synthesis method. Please refer to https::/wss.. wtt.:oom//po.... httpa//eeaaaaaaao/oooooooeeeeaee wwwp:ehhh--ttttte-eeeeeeeeeee
4

---

Under review as a conference paper at ICLR 2023
Synthelic imeges for deraining
fine-grined analysis deraining reeults.
Evaluating Eain Removal
Compare
Bacaground Image
Additive Kain
Image
Compare
Deraining Result
Rain Streaks Ragian Mask
Ground Truth
Evoluating
Kainy image
Background Recomstuction
Figure 3: (Left) The illustration of the rainy image synthesis. (Right) Our fineggrained analysis of the deraining results.
struction but inferior rain removal may have a higher PSNR value than that with perfect rain removal but inferior background reconstruction (e.g., color shift). Such quantitative results would introduce systematic errors in our study.
We discuss the removal of rain streaks separately from the reconstruction of the background regions. Generalization performance of a deraining model is mainly shown in the form of removing unseen rain. The reconstruction of the background may affect the visual effect but is irrelevant to the removal of rain marks. Pixels in R without rain streaks should be black, while rain streaks will appear brighter, as shown in Figure (b). After synthesis, these black areas reflect the background area, while the brighter areas indicate the rainy regions. A perfect rain removal effect should do minimal damage the background area and remove the additive signal from the rain streaks area.
By processing the R to binary mask M using threshold t. where ( = 0 if R(x,5] I t and
Mp.1 if Ri.5i the background part Ô @ (1 - M). We then have two performance numbers:
can segment the output image Ô into the rain streaks part Ô M and
Rain Removal Performance: En B[( @ M O @ M) gives the effect of rain re- moval. A network with poor generalization will not remove rain streaks and make minimal changes to the image. This term measures the changes made by the network in the rainy regions. A higher value reflects better rain removal performance.
Background Reconstruction: EB [B( o (1 - M) - B o (1 - M] gives the ef- feet of background reconstruction by comparing the background regions to the ground truth. A high error in this term means poor overall reconstruction quality.
2.3 DEEP MODELS
We summarize existing networks into three main categories. The first category is a network com- posed of convolutional layers and deep residual connections, and use the ResNet [edig et al. (017) as a representative. The second category is the network with an encoder-decoder design, and we use [oonneenerrrrr et al. (2055) as a representative. Compared with ResNet, UNet introduces down-sampling and up- samplinn layers to extract global and multi- features, which have been proven successful many deraining networks. The last category image processing Transformer. Transformer (Shi et al. ; Chen et al. 2021 is a new network structure characterized by self- attention operations. We include SwinIR Liang et al. (202T) as representative Transformer in our study. For the training settings of these models, please check Appendix E.2
UNDERSTANDING GENERALIZATION
In this section, we conduct experiments based on the above analysis methods. Our analysis consists of two aspects the rain removal effect and the effect of background reconstruction.
3.1 GENERALIZATION ON RAIN REMOVAL
We analyze the rain removal effect unseen rain streaks. Notably, streaks for training and testing, the results in this section all represent generalization performance. After comprehensive experiments, obtain the following interesting observations.
different kinds of rain
Training with fewer background images leads to better deraining performance. First, fix the range rain streaks to medium level and then replace the background images to build different training objectives. We conduct experiments all four categories of images. For each category,
5

---

Under review as a conference paper at ICLR 2023
Ros SwiniR UNet
Rain Regi on Removal
Res
60
60
UNet
Better
Generalization
50
Generalization
50
32
120
255
612
1024
10k
30k
16
64
128
256
512
1024
10k:
Training Background Patch Number (CelebA)
Training Background Patch Number (DIV21)
Rai Removel
Restier SwintR
Rain Region Ramoval
50
UNet
60
ResNet SwintR
45
Better Generalization
UNet
Generalization
258
1024
3600
32
64
128
258
512
1024
2600
Training Background Patch Number (Manga100)
Training Background Patch Number (Urban100)
The relationship between the number of training patches and their rain removal perfor- mance. For each plot, the z-axis represents the patch number, and the y-axis represents the quanti- tative rain removal effect ER- Higher values on the y-axis mean better rain removal. The test rain patterns are not the training set. The effect of rain removal at this time reflects the generalization performance. The qualitative results obtained using ResNet.
Figure 4:
Region Removel
60
60
Medium
145
565
415
145
525
145
041
M55
Large Range
Medium Range
30
Smalll Range
8 16 32 64 128 256 512 1024 30k Training Backgoumd Image Numbers
Figure 5: When trained with different rain ranges, the model exhibits different rain removal effects. The y-axis represents the quantitative rain removal effect. When the rain removal performance is lowered the blue dashed line, the qualitative effect of removing rain starts decrease significantly. We mark these cases and their corresponding effects with colors. We use ResNet in this experiment.
the training set with different amounts of image patches. We test the rain removal effect of these models. The testing images adopt rain streaks as in Yang et al. (2017). Their background images sampled from each category and are different from the training set. The experimental results are shown in Figure [4 It can be seen that these experimental results speak to the same trend, despite different background images and networks. Specifically, the deraining models trained on eight image patches can surprisingly handle unseen rain streaks. On the contrary, models trained with large number of background images cannot remove these rain streaks. This no longer in line with our common sense. Between these two extreme states, the rain removal effect is getting worse with the increase in the number of training images. When the patch number increases to 256, the networks have already lost most of their rain removal ability. While the number of patches increased from 1024 to 30k, the rain removal effects do not change significantly (they all fail to remove rain). This can also be observed from the qualitative results.
We present our explanation of this interesting phenomenon. Although we describe the training objective removing rain streaks from images, there are two strategies for the network to reduce the training loss. One is to recognize and remove rain streaks, and the other is to recognize and reconstruct the image background. If we do not specify the learning strategy, the network will choose the simpler one of these two strategies. When large number of background patches are used in training, learning reconstruct backgrounds is much more complicated than learning remove

---

Under review as a conference paper at ICLR 2023
Normalized Rain Removal Effect
1.0
Multi Multi-Scal Stale Structural Complexity
60.36
Figure (a) The relationship between the number of training patches and their normalized rain removal performance. When the y value is lowered to the grey dashed line, the qualitative ef- feet of removing rain starts to decrease significantly. (b) The averaged complexity of differ- ent image categories given by Bagrov et al. (2020).
0.8
0.34
0.6
0.32
0.4
Urban100
0.30
CelebA
0.2
Manga109
DF2K
0.28
0.0
8
16
32
64
128
256
512
1024
CelebA
DF2K
Manga
Urban
(a)
(b)
Training Backgoumd Complexity Level
Background Reconetruction
-36
40
Reshist SwintR UNet
50
SwinIR
ResNal
45
60 ReaNet
SwhiR
SwniR
Reshet
UNee
40
UNet
UNel
-30
35
40
26
30
20
25
20
16
15
10
8: 32
128 258 5121024
16 32
128 5121024
32 64 258 5121024
32
128 5121024
Training Reckgrums Patch Numbor (CaledA)
Thening Blecegurdd Patch Number
Trwiiing (Manga1000)
Paech hammber
Trwining Backgromm Phecc Number (Urton100)
(DIV2X)
Figure 7: The relationship between the number of training patches and their background reconstruc- tion effect. For each plot, the z-axis represents the patch number, and the y-axis represents the reconstruction error of the background.
rain. Then the network chooses recognize rain and remove it. This will result in an overfitting problem: when new rain streaks deviate from the training ones, the network fails to recognize and remove them. On the contrary, when the background image consists of only a few image patches, learning the background is easier than learning rain streaks. The network will recognize image components in the background, and ignore the features of rain streaks. Thus, the model shows better rain removal effects in images with unseen rain streaks.
The relative complexity between the background and rain determines the network behavior. To verify the above conjecture, we change the range of the rain streaks used in training according to Section 2.1 When using the medium rain range, the rain removal effect weakens when training using 64 background patches. According to our explanation, larger rain streak range makes it harder for the network to learn the rain pattern. Therefore, the rain removal effect will not be weakened until more background patches are used for training. The results of this experiment are shown in Figure 5 As can be seen, in all three training rain ranges, the rain removal effects decrease as the number of background patches increases. When there enough background images for training (30k patches), even the large rain range of training rain cannot make the final model achieve sufficient rain removal performance, indicating that the large rain range does not cover our testing cases. When training with a large rain range, the network shows a significant drop in rain removal performance until it is trained with more than 512 background patches. The model trained on a small rain range cannot show good rain removal effect even when there are only 16 background training image patches. These results indicate that the network behaviors are affected by the relative relationship between the background image and rain streaks. The complexity or learning difficulty of the medium range rain approximately less than 64 training patches, while the complexity of the large range rain is approximately less than 512 training background patches. The network will slack off, and "select" the easier way to learn in different situations.
A more complex background set makes it harder for the network to learn. We next change the category of the background images used for training and observe the models' behavior. To compare across different image categories, we normalize the deraining effect to [0, 1]. The results are shown in Figure The most intuitive conclusion that even with the same number of training patches, different image categories can lead to different rain removal effects. For CelebA images, when the patch number increases from 8 to 16, its deraining performance begins to drop sharply. For the natural image patches, increasing the patch number 16 does not cause such a rain removal performance drop. In contrast, for image patches from Mangal09 and Urban 100, the rain removal performance does not drop significantly until the patch number is larger than 32. According to our
7

---

better while & means Under review as a conference paper at ICLR 2023
IN lout Rany Iaae
SPONNY
HI1 Rodeer parthes Large
IN) INRERRM (Lane
patthe)
Large Beee
Crmund
Figure 8: Visualization of the deraining results on a synthetic image, Zoom in for better comparison.
Table 2: Quantitative
Training Objective
ResNet
SPDNet Yi et al. 2202 PSNK
RCDNe
Wang et al.
2020c
Back.
Range
Eg 7
Pg
PSNR
Ex.
Fg
Fx 1
Fg
PSNK
comparisons between
Medium
31.24
10.79
33.63
30.51
28.54
different models. t
64
Medium
53.33
25.02
20.87
45.47
14.78
25.32
means the higher the
Large
39.88
28.57
7.16
29.60
256
Large
45.64
16.51
24.30
38.87
8.03
29.40
40.40
8.52
29.08
128
Large
51.75
23.53
21.45
43.20
14.59
25.67
44.67
13.72
26.09
the lower the better.
explanation, the number of training patches increases, the more complex image categories will make the models suffer from performance drop earlier. Our results suggest that the complexity of these four image categories can be in decent order as CelebA, DIV2K, Mangal09 and Urban 100.
This roughly matches our human perception. Face images have strong global and local structures. DIV2K images are rich in texture but have simple global structure. Manga images lack complex textures but often contain text and complex edges. Urban images consist of repetitive stripes and grids. We then verify our conclusion with a system complexity calculated by a mathematical model. Bagrov et al. (2020) propose a computational method to estimate the structural complexity of natural patterns including natural images. We calculate the multi-scale structure complexity for these four image categories, and the results exhibit the same order, see (bg This provides mathematical evidence for our argument.
3.2 RECONSTRUCTION ON BACKGROUND
The above results show that the deraining ability can be improved by reducing the background image used for training. But training with only limited background images is not at cost. Reducing the training images prevents the network from overfitting the rain patterns, but causes the network overfit the limited background images. We also conduct experiments on this issue.
With the decoupled evaluation metrics En described in Section 2.2 we can measure the reconstruc- tion of the background independently. The results are shown in Figure 7 With the increase of train- ing images, the reconstruction of the background becomes better. It can be seen that training with 256 background patches can already bring a good background reconstruction effect. Continuing to add training images does not further improve the performance of background reconstruction. This conclusion surprising, as this goes against our intuition that training the low-level vision model requires lot of images. Our research shows that training with large number of background im- ages does not lead to large gain in reconstruction performance, but instead exacerbates the model's overfitting to rain streaks. Another surprising finding is that the model trained with only 256 images can already handle most the image components. This may indicate that image components and features not complex
usually thought for low-level vision network.
IMPLICATION
Improve the existing deraining models. Although this paper does not directly propose any al- gorithms, our conclusions can shed light on improving the removal of unknown rain streaks by existing models. Our experiments have three important practical findings: (1) Reducing the number of background images can make the network focus on learning image content instead of overfitting rain streaks. (2) Expanding the range of rain streaks allows us use more background images for training. (3) A few background images can already achieve good reconstruction performance. These findings can be directly used to improve the generalization ability of existing models with minimal changes. Our strategy is simple: find a balance between background images and rain range to avoid overfitting rain streaks.
8

---

Under review as a conference paper at ICLR 2023
(e) vour
tol PRZDNee
(c) SroNee
Set Synahas -Sm
(e) Sayie Heal- taad
ID HICENNe Erained of 2cbi
Figure 9: Qualitative results on real-world test images. Zoom in for better comparison.
Some quantitative results are presented in Table 2 We use three deraining models as baselines (ResNet, SPDNet et al. (202T). RCDNet Wang et al. (2020ch) and demonstrate the power of the proposed simple strategy. We use 30K background images and the medium range rain to train our baseline models. The test set is the R100 dataset Yang al. (207). We quantify the deraining effect and the background reconstruction effect according to the decoupie evaluation metrics ER and Eg- We also test PSNR as a reference. It can be seen that using the existing training methods cannot generalize well the unseen rain of R100, which is shown by the poor deraining performance in Table However, due to the learning on a large number of images, the reconstruction errors of the baseline models are generally low. Thus the PSNR values cannot objectively reflect the rain removal effect. We reduce the training background images to 64, which is the upper limit of the image number that can make the model generalize under medium range rain. At this time, the rain removal performance has greatly improved, but at the cost of background reconstruction performance. By enlarging the rain range and training with more background images, we are able to achieve trade- off between rain removal performance and background reconstruction.
Figure shows a qualitative comparison of these models under different training objectives. It can be seen that even with the advanced network structure design, the rain removal effects of the baseline models of SPDNet and RCDNet are not satisfactory. Using a larger range of rain can bring limited improvements. In the case of medium range rain, reducing the background image to 64 significantly improved the rain removal effect and resulted in unstable image reconstruction. When the rain range is enlarged, and the training background is set to 128 patches, the model can show excellent performance in rain removal and background reconstruction. Note that we do not use additional data improve the network structure throughout the process. We only adjust the training data.
We also present the comparison on real images in Figure 9 In addition, semi muni--sperviied methods Wei et al.] (2019a); Huang et al.] (2021) have also been used to improve the deraining effect on real images, and we also include the representative method Syn2Real Yasarla et al. (2020) 2021)
Syn2Real-Syn is trained on synthetic data, and Syn2Real-Real is trained synthetic labeled data and real unlabeled data. Due to the difference in the distribution of rain streaks, the models trained using synthetic data can not generate satisfactory rain removal effects. When obtaining some real images, Syn2Real-Resd can indeed achieve some improvement. However, these improvements are brought about by improving the generalization ability. Because these methods manage to convert "rain outside the training set" to "rain inside the training set". Since data collection is extremely difficult, this method still faces great challenges practice. Our method improves generalization performance and achieves better results test images.
CONCLUSION AND INSIGHTS
In this work, we study the generalization problem of deraining networks. Although we take image deraining as representative, our key conclusions can provide insights for low-level vision. We argue that the generalization problem in low-level vision cannot be attributed to insufficient net- work capacity or training data. Instead, we find that existing training strategies do not encourage generalization. The networks only learn to overfit the degradations, and thus can hardly generalize well to unseen degradations. A feasible solution is to guide the network to learn image distributions instead of degradations. But this also poses challenges when the image distributions Furthermore, due to the lack of effective interpretability tools, we cannot explore what the low-level model learns and answer why it learns in this way. This gap between ideal and reality prompts to rethink the importance of low-level network interpretability. and this work provides an indispensable perspective this. These insights also need to be verified in other low-level vision tasks.
complex.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Pablo Arbelaez, Michael Maire, Charless Fowlkes, and Jitendra Malik. Contour detection and hi- erarchical image segmentation. IEEE transactions on pattern analysis and machine intelligence, 33(5):898-9) 916, 2010.
Andrey A Bagrov, Ilia A lakovlev, Askar A Iliasov, Mikhail Katsnelson, and Vladimir V Mazurenko. Multiscale structural complexity of natural patterns. Proceedings of the National Academy of Sciences, 11745)::0241-30251 2020.
Yi Chang, Luxin Yan, and Sheng Zhong. Transformed low-rank model for line pattern noise re- moval. In Proceedings of the IEEE international conference computer vision, pp. 7726-1734, 2017.
Chenghao Chen and Hao Li. Robust representation learning with feedback for single image derain- ing. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7742-7751, 2021.
Hanting Chen, Yunhe Wang. Tianyu Guo. Chang Xu, Yiping Deng. Zhenhua Liu, Siwei Ma. Chun- jing Xu, Chao Xu, and Wen Gao. Pre-trained image processing transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12299-12310, 2021.
Xiang Chen, Jinshan Pan, Kui Jiang, Yufeng Li, Yufeng Huang, Caihua Kong, Longgang Dai, and Zhentao Fan. Unpaired deep image deraining using dual contrastive learning. In Proceedings of the JEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2017-2026, 2022.
Yingjun Du, Jun Xu, Xiantong Zhen, Ming-Ming Cheng, and Ling Shao. Conditional variational image deraining. IEEE Transactions on Image Processing, 29:6288-6300 2020.
Xueyang Fu, Jiabin Huang, Xinghao Ding. Yinghao Liao, and John Paisley. Clearing the skies: A deep network architecture for single-image rain removal. IEEE Transactions on Image Process- ing. 26(6):2944 2956, 2017a.
Xueyang Fu, Jiabin Huang, Delu Zeng, Yue Huang, Xinghao Ding, and John Paisley. Removing rain from single images via a deep detail network. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3855-3863, 2017b.
Xueyang Fu, Borong Liang, Yue Huang, Xinghao Ding, and John Paisley. Lightweight pyramid networks for image deraining. IEEE transactions neural networks and learning systems, 31 (6):1794-1807, 2019.
Xueyang Fu, Qi Qi, Zheng-Jun Zha, Yurui Zhu, and Xinghao Ding. Rain streak removal via dual graph convolutional network. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp. 1352-1360, 2021.
Kshitiz Garg and Shree Nayar. Photorealistic rendering of rain streaks. ACM Transactions on Graphics (TOG), 25(3):996-1003 1002, 2006.
Andreas Geiger, Philip Lenz, and Raquel Urtasun. Are ready for autonomous driving? the kitti vision benchmark suite. In 2012 IEEE conference on computer vision and pattern recognition, pp. 1354-33611 IEEE, 2012.
Jinjin Gu and Chao Dong. Interpreting super-resolution networks with local attribution maps. Pro- ceedings of the IEEF/CVE Conference Computer Vision and Pattern Recognition, pp. 9199- 9208, 2021.
Jinjin Gu, Hannan Lu, Wangmeng Zuo, and Chao Dong. Blind super-resolution with iterative ker- nel correction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1604-1613, 2019.
Jinjin Gu, Haoming Cai. Haoyu Chen, Xiaoxing Ye, Jimmy Ren, and Chao Dong. Pipal: large- scale image quality assessment dataset for perceptual image restoration. In European Conference Computer Vision pp. 633-651. Springer, 2020.
10

---

Under review as a conference paper at ICLR 2023
Shuhang Gu, Deyu Meng, Wangmeng Zuo, and Lei Zhang. Joint convolutional analysis and synthe- sis sparse representation for single image layer separation. In Proceedings of the IEEE Interna- tional Conference on Computer Vision, pp. 1708-1716 2017.
2017.
Shi Guo, Zifei Yan, Kai Zhang, Wangmeng Zuo, and Lei Zhang. Toward convolutional blind de- noising of real photographs. In Proceedings of the IEEE/CVF conference computer vision and paltern recognition, pp. 1712-1722, 2019.
Huaibo Huang, Aijing Yu, and Ran He. Memory oriented transfer learning for semi-supervised image deraining. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7732-7741, 2021.
Jia-Bin Huang, Abhishek Singh, and Narendra Ahuja. Single image super-resolution from trans- formed exemllxx In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 5197-5206, 2015.
Kui Jiang, Zhongyuan Wang. Peng Yi, Chen Chen, Baojin Huang, Yimin Luo, Jiayi Ma, and Junjun Jiang. Multi-scale progressive fusion network for single image deraining. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognilion, pp. 8346-8355, 2020.
Xiangtao Kong, Xina Liu, Jinjin Gu, Yu Qiao, and Chao Dong. Reflash dropout in image super- resolution. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recog- nilion, pp. 6002-6012, 2022.
Christian Ledig. Lucas Theis, Ferenc Huszár, Jose Caballero, Andrew Cunningham, Alejandro Acosta, Andrew Aitken, Alykhan Tejani, Johannes Totz, Zehan Wang. et al. Photo-realistic sin- gle image super-resolution using generative adversarial network. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4681-4486 16900, 2017.
Ruoteng Li, Loong-Fab Cheong, and Robby T Tan. Heavy rain image restoration: Integrating physics model and conditional adversarial learning. In Proceedings of the IEEE/CVF Conference Computer Vision and Puttern Recognition, pp. 1633-1642, 2019.
Jingyun Liang, Jiezhang Cao, Guolei Sun, Kai Zhang, Lue Van Gool, and Radu Timofte. Swinir: Image restoration using swin transformer. In Proceedings of the IEEE/CVF International Confer- ence on Computer Vision, pp. 1833-1844, 2021.
Anran Liu, Yihao Liu, Jinjin Gu, Yu Qiao, and Chao Dong. Blind image super-resolution: A survey and beyond. IEEE Transactions Pattern Analysis and Machine Intelligence, 2022a.
Xing Liu, Masanori Suganuma, Zhun Sun, and Takayuki Okatani. Dual residual networks lever- aging the potential of paired operations for image restoration. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7007-7016, 2019.
Yihao Liu, Anran Liu, Jinjin Gu, Zhipeng Zhang, Wenhao Wu, Yu Qiao, and Chao Dong. Discov- ering" semantics" in super-resolution networks. arXiv preprint arXiv:2108.00 004000 2021.
Yihao Liu, Hengyuan Zhao, Jinjin Gu, Yu Qiao, and Chao Dong. Evaluating the generalization ability of super-resolution networks. arXiv preprint arXiv:2205.07011 2022b.
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face autributes in the wild. In Proceedings of the IEEE international conference on computer vision, pp. 3730-3738, 2015.
Scott M Lundberg and Su-In Lee. A unified approach to interpreting model predictions. Advances in neural information processing systems, 30, 2017.
Salma Abdel Magid, Zudi Lin, Donglai Wei, Yulun Zhang, Jinjin Gu, and Hanspeter Pfister. Texture- based error analysis for image super-resolution. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2118-2127, 2022.
Yusuke Matsui, Kota Ito, Yuji Aramaki, Azuma Fujimoto, Toru Ogawa, Toshihiko Yamasaki, and Kiyoharu Aizawa. Sketch-based manga retrieval using manga109 dataset. Multimedia Tools and Applications, 76(20):2181) 111-211811-28888 2017.
11

---

Under review as a conference paper at ICLR 2023
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary Vitito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. 2017.
Dongwei Ren, Wangmeng Zuo, Qinghua Hu, Pengfei Zhu, and Deyu Meng. Progressive image deraining networks: A better and simpler baseline. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3937-3946, 2019.
Olaf Ronneherger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedi- cal image segmentation. In International Conference on Medical image computing and computer- assisted intervention, pp. 234-24422 241. Springer, 2015.
Gerald Schaefer and Michal Stich. Ucid: An uncompressed color image database. In Storage and Retrieval Methods and Applications for Multimedia 2004, volume 5307, pp. 472-480. SPIE, 2003.
Shuwei Shi, Jinjin Gu, Liangbin Xie, Xintao Wang, Yujiu Yang, and Chao Dong. Rethinking align- ment in video super-resolution transformers. Advances in Neural Information Processing Systems, 2022.
Avanti Shrikumar, Peyton Greenside, and Anshul Kundaje. Learning important features through propagating activation differences. In International conference on machine learning, pp. 3155- 3153. PMLR, 2017.
Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Deep inside convolutional networks: Vi- sualising image classification models and saliency maps. arXiv preprint arXiv:/312.6034 2013.
network::
Jost Tobias Springenberg, Alexey Dosovitskiy, Thomas Brox, and Martin Riedmiller. Striving for simplicity: The all convolutional net. arXiv preprint arXiv: 14X2.:14/2.6806 2014.
Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep networks. In International conference on machine learning, pp. 3319-3328. PMLR, 2017.
Radu Timofte, Eirikur Agustsson, Luc Van Gool, Ming-Hsuan Hsuan Yang, and Lei Zhang. Ntire 2017 challenge on single image super-resolution: Methods and results. In Proceedings of the IEEE conference on computer vision and pattern recognition workshops. pp. 125, 2017.
Cong Wang, Yutong Wu, Zhixun Su, and Junyang Chen. Joint self-attention and scale scale-aggregation for self-calibrated deraining network. Proceedings of the 28th ACM International Conference Multimedia, pp. 2517-2525, 2020a.
Cong Wang, Xiaoying Xing, Yutong Wu, Zhixun Su, and Junyang Chen. Desfn: Deep cross-scale fusion network for single image rain removal. In Proceedings of the 28th ACM international conference on multimedia, pp. 1643-1651, 2020b.
Guoqing Wang, Changming Sun, and Arcot Sowmya. Erl-net: Entangled representation learning for single image de-raining. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 11444-665,, 2019a
Hong Wang, Qi Xie, Qian Zhao, and Deyu Meng. model-driven deep neural network for single image rain removal. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3103-31 3--31122 12, 2020c.
Tianyu Wang, Xin Yang, Ke Xu, Shaozhe Chen, Qiang Zhang, and Rynson WH Lau. Spatial atten- tive single-image deraining with a high quality real rain dataset. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12270-12275 2019b.
Yinglong Wang, Yibing Song, Chao Ma, and Bing Zeng. Rethinking image deraining via rain streaks and vapors. In European Conference Computer Vision, pp. 367-382. Springer, 2020d.
Zheng Wang. Jianwu Li, and Ge Song. Dtdn: Dual-task de-raining network. In Proceedings of the 27th ACM international conference multimedia, pp. 1833-1841, 2019c.
12

---

Under review as a conference paper at ICLR 2023
Wei Wei, Deyu Meng, Qian Zhao, Zongben Xu, and Ying Wu. Semi-supervised transfer learning for image rain removal. In Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pp. 3877-3886, 2019a.
Yanyan Wei, Zhao Zhang. Haijun Zhang, Richang Hong. and Meng Wang. A coarse-to-fin tone multi- stream hybrid deraining network for single image deraining, In 2019 IEEE international confer- ence on data mining (ICDM), pp.
IEFE, 2019b.
Jie Xiao, Man Zhou, Xucyang Fu, Aiping Liu, and Zheng-Jun Zha. Improving de-raining general- ization via neural reorganization. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. -987-4996 2021.
Liangbin Xie, Xintao Wang, Chao Dong, Zhongang Qi, and Ying Shan. Finding discriminative filters for specific degradations in blind super-resolution. Advances in Neural Information Processing Systems, 34:51-48 61, 2021.
Wenhan Yang. Robby T Tan, Jiashi Feng, Jiaying Liu, Zongming Guo, and Shuicheng Yan. Deep joint rain detection and removal from a single image. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1357-1366, 2017.
Wenhan Yang, Jiaying Liu, Shuai Yang, and Zongming Guo. Scale-free single image deraining via visiaeldty-enhaneedd recurrent wavelet learning. IEEE Transactions on Image Processing, 28(6): 2948-2961, 2019.
Youzhao Yang and Hong Lu. Single image deraining via recurrent hierarchy enhancement network. In Proceedings of the 27th ACM International Conference on Multimedia, pp. 1814-1822, 2019.
Rajeev Yasarla and Vishal M Patel. Uncertainty guided multi-scale residual learning-using ey- cle spinning enn for single image de-raining. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 8405-8414, 2019.
Rajeev Yasarla, Vishwanath A Sindagi, and Vishal Patel. Syn2real transfer learning for image deraining using gaussian processes. In Proceedings the IEEE/CVF conference computer vision and pattern recognition, pp. 2726-2736, 2020.
Rajeev Yasarla, Vishwanath A Sindagi, and Vishal M Patel. Semi-supervised image deraining using gaussian processes. IEEE Transactions on Image Processing, 30:6570-6582, 2021.
Qiaosi Yi, Juncheng Li, Qinyan Dai, Faming Fang, Guixu Zhang, and Tieyong Zeng. Structure- preserving deraining with residue channel prior guidance. In Proceedings of the IEEE/CVF In- lernational Conference on Computer Vision, pp. 4238 2021.
Weijiang Yu, Zhe Huang, Wayne Zhang, Litong Feng, and Nong Xiao. Gradual network for single image de-raining. Proceedings of the 27th ACM international conference on multimedia, pp- 1795-1804, 2019.
Syed Waqas Zamir, Aditya Arora, Salman Khan, Munawar Hayat, Fahad Shahbaz Khan, Ming- Hsuan Yang, and Ling Shao. Multi-stage progressive image restoration. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 14821-14831, 2021.
He Zhang and Vishal Patel. Density-aware single image de-raining using a multi-stream dense network. In Proceedings of the IEEE conference computer vision and pattern recognition, pp- 695-704, 2018.
He Zhang, Vishwanath Sindagi, and Vishal M Patel. Image de-raining using conditional generative adversarial network. IEEE transactions on circuits and systems for video technology, 30(11); 3943-3956, 2019.
Bolei Zhou, David Bau, Aude Oliva, and Antonio Torralba. Interpreting deep visual representations via network dissection. JEEE transactions on pattern analysis and machine intelligence, 41(9): 2131-2145, 2018.
13

---

Under review as a conference paper at ICLR 2023
Man Zhou, Jie Xiao, Yifan Chang, Xueyang Fu, Aiping Liu, Jinshan Pan, and Zheng-Jun Zha. Image de-raining via continual learning. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4907-4916, 2021.
Lei Zhu, Chi-Wing Fu, Dani Lischinski, and Pheng-Ann Heng. Joint bi-layer optimization for single-image rain streak removal. In Proceedings of the IEEE international conference on com- puter vision. pp. 2526 2534, 2017.
14

---

Under review as a conference paper at ICLR 2023
APPENDIX
A OTHER RELATED WORK
A.1 IMAGE DERAINING
Many methods have been proposed to develop state-of-the-art deraining networks. These works include deep networks designs Fu et al. (2017a); Wang et al. (2019a), residual networks Fu et al. (017b): Liu et al. (2019), recurrent networks Ren et al. (2019); Yang & Lu (2019): Yang et al. (2019). multi- task Wang et al. ) Du et al. (2020) and multi-scale designs Jiang et al. (2020);
Fu et al. (2019); Yasarla & Patel (2019) ; Yu et al. (2019); Wei et al. (2019b); Wang et al. (2020b):
Zamir et al. (2021).
pearsity-based image modeling Gu et al. (2017) Zhu et al. (2017), low-rank prior Chang et al. (2017). model-driven solutions Wang al. (2020c d), attention mechanism Wang
(ct al. g Chen et al. (202]) Fu et al. (202T). adversarial learning Li et al. (2019), representation
2020) and unsupervised learning Chen learning Chen & Li (021), semi-supervised Yasarla et al.)
jet al. (2022). Deep learning methods are data-hungry but collecting rain streaks and background image pairs are challenging. A lot of works have been proposed synthesize rain streaks with better results. Garg & Nayar 2006) first propose a physically-base
photo-realistic rendering method for synthesizing rain streaks. Zhang & Patel (0018) and Fu et al. (2017a) use Photoshop software to manually add rain effects to images to build the synthetic paired data. Due to the poor generalization performance of existing methods, models trained on synthetic images were found to be ineffective in real-world scenarios. Some works Yang et al. (2017); /Zhang et al. (2019); Wang et al. (2019b) that have contributed to real collected deraining datasets. However, acquiring these datasets still expensive and cannot solve the problem of poor generalization. There are also works that mentioned the generalization issue of the deraining models. Xiao et al. (202I) and Zhou et al. (202I) attempt to improve the generalization ability of deraining networks by accumulating knowledge from multiple synthetic rain datasets, as most existing methods can only learn the mapping on a single dataset for the deraining task. But this attempt does not allow the network generalize beyond the training set.
In addition, semi-supervised methods Wei et al. (2019)) Huang et al. (2021) have also been used to improve the deraining effect real images, and we also include the representative method Ren2Real Yasarla et al. (2020) 202T). There are some semi-supervised deraining methods Wei et al. 2019a) Huang et al. (2021) Yasarla al. (2020) (02I) are proposed improve the performance of derain- ing models in real-world scenarios. When obtaining some real images similar to the test images, these works can indeed achieve some improvement. However, these improvements are not brought about by improving the generalization ability. Their solution is to include real test images in the training set, even if we don't have corresponding clean images. These methods are effective when we can determine the characteristics of the test image. But this does not solve the generalization problem. Because these methods manage to convert "rain outside the training set" to "rain inside the training set". Since data collection is extremely difficult, this method still faces great challenges in practice.
A.2 LOW-LEVEL VISION INTERPRETABILIE
We provide detailed review of existing work on low-level visual interpretability. Gu & Dong (222)) bring the first interpretability tool for super-resolution networks. Xie et al. (2021) find the most discriminative filters for each specific degradation in blind SR network, whose weights, positions, and connections are important for the specific function blind SR. [Maidd et al. (2022) texture classifier to assign patches with semantic labels, in order to identify global and local sources of SR errors. [Shi et al. (2022) show that Transformers can directly utilize multi-frame information from unaligned frames, and alignment methods are sometimes harmful to Transformers in video super-resolution. They use a lot of interpretability analysis methods in their work. The closest work this paper is the deep degradation representation proposed by Liu et al. (202I). They argue that SR networks tend to overfit to degradations and show degradation "semantics" inside the network. The presence of these representations often means a decrease in generalization ability. The utilization of this knowledge can guide us to analyze and evaluate the generalization performance of SR methods Liu et al. (20226).
B TRANSFERABILITY OF LIMITED TRAINING PATCHES
At the end of the main text, we propose method to improve the generalization performance of the deraining network by reducing the number of training background image patches. However, this method will overfit the image content when the number of training background patches is very
15

---

Under review as a conference paper at ICLR 2023
ResNet
SwiniR
UNet
60
Ce-ebA
60-
CelebA
50
ColebA
Recorststtionn
Urban Manga DVV2K
Background Rucomistuction
Urban
Barkgroondd Recomtructton
Urban
50-
Manga DWVK
400
Manga
DWV2K
40
40-
50
30
20
20
10
10
16
32
64
128 256 512 1024 30k
16
32
64
128 250 siz 1024 30k
16
52
64
128
256 512 1024 30k
Trairing Background Patch
Training Background Putch Number
Training Background Putch Number
100
CelebA
100
CelebA
Urban
Urban
80
90
DIV2
Manga
Rain Rogion Removal
80
Manga
DWV2K
60
70
70
55
Region
60-
60
50
50
Urban
Manga
40
40
40
DWVK
16 125 256 512 3024 30k
256 Twining Barkground
16 32 64
128
512 1024 30k
52 G4 128 256 5iz 1024 30k Training Background Panch Number
Training Background Patch Number
Figure 10: The relationship between the number of training patches and their rain removal or back- ground reconstruction performance. The test image set for these six plots is the DIV2K set. We train the model with all four image categories to validate the performance when the image distribution mismatch. For background reconstruction Eg, lower values on the y-axis mean better background reconstruction. For rain removal effect En, higher values on the y-axis mean better rain removal. The test rain patterns are not in the training set. The effect of rain removal at this time reflects the generalization performance.
small. In Section 3.2 we investigate the risk of reducing the number of training patches when testing on the same image category. Recall that with the increase of training images, the reconstruction of the background becomes better. Training with 256 background patches can already bring a good background reconstruction effect. Continuing to add training images does not further improve the performance of background reconstruction.
In this section, we investigate whether the proposed scheme is still robust when the training and lesting patch distributions are significantly different. We train the model on four image categories and then test using the DIV2K image category. This simulates the situation when the background image distribution differs from the test set. We observe the behavior of models trained with different numbers of patches. The results are shown in Figure T0 We can draw the following conclusions. First of all, even if the distribution of training background images is very different, the model trained using the images of CelebA and Urban categories can still perform similarly to the model trained by DIV2K patches. These models can reconstruct background images well when training patches reach 128 or more. At this time, the difference in the distribution of these training sets and DIV2K does not bring significant differences. The rain removal effect of these models is also similar. Second, we found that the model trained with Manga image patches differed from others. The model trained with manga image patches is generally worse at background reconstruction than other models. Even when the number of patches is large, the model trained on manga cannot achieve similar performance to other models. For rain removal, the model trained with manga also performs the worst. This result is in line with expectations because manga images are significantly different from other images, especially in the underlying image components. Although the other three types of images differ greatly in image structure, texture type and other characteristics, they all belong to the category of natural images. Whereas Manga images contain artificial textures and edges, which are quite different from other images.
There is a large image reconstruction error when training with images whose distribution very different from that of the test set images. This is reasonable to some extent, because in this case, even using a large number of training images cannot bridge the error caused by this distribution mismatch. And are pleased that the method of training using limited background images is robust to image content to a considerable extent, as long as the training images are natural images. This is consistent with our practice. In the process of actually using method, also found that as long as more than 256 image patches are used for training, the results are stable. There is no significant performance change due the content of the selected training patches.
16

---

Under review as a conference paper at ICLR 2023
Inputaart by Imago
SPOR
ACONet
ResNet Trained 10K patchs. La 12 Rain
Larg
Fopel Rairy Image
paxcass Large
ACDNN Traeed Lange
LCDRW. Traned
Dfw. Tranned 125 Fon, Rain
Goonnd Tush
Imsaa
BCDNNe Traked en Large Bair
LCC.bel Traited 64 palt bets
RCD Trained w/ 125 pertes, Large Bain
Caoudd Tuch
Figure 11: Visualization of the deraining results. Zoom for better comparison.
C MORE RESULTS
We provide more results of different deraining models in Figure 1] and Figure 12 Note that we did not use additional data nor improve the network structure throughout the process. We only adjust the training objective. Although the effeet of the output image can be further improved, it shows conclusions' practical value and application potential.
D LIMITATION.
Our work mainly takes the deraining task as breakthrough point, and attempts make general summary the generalization problem in low-level vision. Due the differences between different low-level tasks, the analysis methods in this paper, especially the fine fine-grained analysis methods, may not be directly used on some other tasks. But we believe our work can still bring novel insights to the entire low-level vision field.
Our work also attempts to improve existing deraining models. But these improvements are based on the simple usage of some key conclusions of our work. Although shown effective, believe that these methods are still far from ideal. We only demonstrate the application potential of the knowledge presented in this work and have intention to propose state-of-the-art algorithms
17

---

Under review as a conference paper at ICLR 2023
models. Research efforts are still needed to develop more robust deraining algorithms using conclusions.
E
REPRODUCIBILITY STATEMENT
E.1 RESOURCES
The models used in our work are taken directly from their respective official sources. Our code built under the BasicSR framework ter code organization. The deraining model SPDNet Yi et al. (2021) is available at https: Srrithab.coo//oyiees/////11 DNe The deraining model RCDNet Wang al. (2020c) is avail-
com/ citm/xinnttat anttao//ooooooooooooooooooooo BasiicRsssssss / for bet-
able athhttss: //gg: com hhwpg///gwwggggognettttttt The training and testing datasets used in our work are all publicly available.
E.2 NETWORK TRAINING
Due to space constraints, we do not describe our training method in detail in the main text. Here we describe the training method reproduce our results. total of 150 models were involved in our experiments. We used the same training configuration for all models. We use Adam for training. The initial learning rate x 10-4 and 31 0.9, f2 0.99. For each network, we fixed the number of training iterations to 250,000. The batch size is 16, input rainy images are of size 128x The cosine annealing learning strategy is applied to adjust the learning rate. The period of cosine is 250,000 iterations. All models are built using the PyTorch framework Paszke et al. (2017) and trained with NVIDIA A100 GPUs.
E.3 AVAILABILITY
All the trained models and code will be publicly available.
F ETHICS STATEMENT
This study does not involve any human subjects, practices data set releases, potentially harmful insights, methodologies and applications, potential conflicts of interest and sponsorship, discrimi- nation/bias/farness concerns, privacy and security issues, legal compliance, and research integrity issues. We do not anticipate any direct misuse of our contribution due to its theoretical nature.
18

---

Under review as a conference paper at ICLR 2023
RopeRRaity Image
SPOket
ACONet
ResNet Trared at 10 psiches ge Ram
Large
parches, Large
InpuRRaity Image
SPONet
RCDNet
RosNet Trined pwwces.ccccccooococc
AciDNee Trained wil large Baie
Dit,e,, Truined wy pattites
REDOWe Triased w/ 178 prrtee, Large Rain
Figure 12: Visualization of the deraining results. Zoom in for better comparison.
19


Reviews:
[{"review_id": "jOG8S4OWNBe", "reviewer": "Reviewer_DKEu", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposed a new finding that between the image content and the additive degradation, deep networks tend to learn the less complex element in the separation task. To solve the generalization problem in image deraining, it tried to train the deraining networks with the fewer and less complex background images.\nIn my opinion, the proposed finding is novel and effective to explore the generalization problem in images deraining, which utilizes less training data and less complex background images, and this finding differs from previous acknowledgement that deep learning methods need to use the training data with higher complexity to learn richer knowledge to overcome generalization problems. Furthermore, to prove this finding, the paper conducts extensive experiments, including carefully choose the different images’ background and design different rain streaks. The experimental design and the analysis are clear to understand. The paper writing is also professional with idea organizations and presentations. \n", "summary_of_the_review": "The finding in this paper is novel and subversive, but it doesn’t propose a better way to achieve the balance between restoring the image background and removing rain streaks. It doesn’t show a significant improvement in generalization performance compared to the existing rain removal methods. As to the experiment results shown in the paper, I don’t find this method is superior to other deraining methods. I think this paper needs to do more experiments to prove the effectiveness of this finding. ", "strength_and_weaknesses": "Strength ：\n1.\tThe advantage of this paper is that it tries to improve the generalization performance of images deraining by changing the training objective, and finds that training with fewer data and less complex background images can improve the generalization in deraining. \n2.\tDifferent from the existing acknowledgement, this method doesn’t need to train the deraining networks with amounts of data and extremely complex background images and rain streaks.\n\nWeaknesses ：\n1.\tThe disadvantage is that although this paper finds the impact of the training objective on the performance of rain removal generalization, it doesn’t propose a better way to achieve the balance between restoring the image background and removing rain streaks. \n2.\tIt doesn’t show a significant improvement in generalization performance compared to the existing rain removal methods. \n3.\tI think this paper needs to do more experiments to prove the effectiveness of this finding.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "fCokvL3oEB", "reviewer": "Reviewer_bcPF", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper presents generalization problems of the network in performing the low-level vision task, i.e. authors perform a study by performing the different deraining experiments and convey that when trained with a less complex background, networks learn the less complex element in the image content and degradation to reduce the training.  The paper finally claims that authors experimental study demonstrates an excellent methodology for understanding the generalization of the problem of low-level vision.", "summary_of_the_review": "- This is good work but can be improved before publishing the work", "strength_and_weaknesses": "Strengths:\n- For the experimental authors conducted on three different types of networks, U-Net, ResNet, SWIN-IR. these experiments are beneficial in understanding the behavior of different types of networks in different training conditions.\n- Compares rain regions and backgrounds with different metrics (Rain Removal Performance and Background Reconstruction). But these metrics are just l2-loss in background regions and rain regions which are not excellent metrics since they don't measure the quality of background structures or quality of rain regions, and networks are prone to degrade background structures by over deraining.\n- Paper is well written with a detailed study of each experiment performed.\nWeaknesses:\n- section 4 Implications:\n     - Observations from the experiments are not excellent: \"Our strategy is simple: find a balance between background images and rain\nrange to avoid overfitting to rain streaks.\" This conclusion statement is naive since this statement is very obvious and doesn't require any experiments and study to get to this conclusion.\n    - It would have been better if authors propose a training strategy or a loss that addresses generalization problems in low-level visions\n    - Even though the experimental study is good, the conclusion and implications are not contributing to addressing the generalization problems of low-level vision.\n- Authors claim that networks have generalization problems in addressing deraining, but few existing works are proposed different generalization techniques in overcoming these generalizations differences between synthetic rain and real rain. Can authors perform similar experiments and study these semi-supervised deraining methods? Since it is important to know for the reader whether these semi-supervised deraining methods are overcoming these generalization problems in low-level vision\n- Experiments on real rainy images are not performed.\n- Can authors also discuss how their study can be generalized to other low-level vision tasks like de-hazing, desnowing, low-light enhancement? Since most of the comparisons and experiments are more specific to deraining. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "Ux8fiBLirNY", "reviewer": "Reviewer_LTSX", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper has an interesting finding for improving generalization ability of image deraining. By reducing the background images in training data, the deraining model will tend to learn background context instead of learning rain streaks, preventing deraining model overfitting to rain streaks patterns. The finding is interesting and may help improve generalization ability of restoration models in other low level vision tasks. ", "summary_of_the_review": "The finding is interesting, but it needs more validations to convince that the conclusion is solid. With the current experiments, it is not sufficient to support the finding is correct for deraining task. If the validation cannot be done during rebuttal, I will not suggest to accept it, because it is dangeous to mislead the comminity. ", "strength_and_weaknesses": "Strengths: without modification on network architecture, existing deraing methods can be trained to have better generalization ability by reducing background images in training data. It is interesting and surpersing. \n\nWeakness:\n1. Although it is interesting, this finding is dangous to works for specific data and task. If it is solid for deraining, this paper also contributes to this field. But the training data and testing data distribution should be well discussed and evaluated. If the distribution of background images in training and testing data, the conclusion has nothing to endorse. Also, in Fig. 7, when patch number continues to increase, reconstruction error will increase?\n\n2. Experiments are not sufficient to convince me. Cross-evaluation should be conducted on public datasets to show generalization ability of existing deraining models are indeed improved. \n\n3. typos: Sec 3.2 that deraining the deraining ability", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "R2fsr9Bn3OD", "reviewer": "Reviewer_kgGu", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a new solution to the generalization problem of image de-raining, that is, focusing on learning the image background  can improve the generalization ability. Based on the above conclusions, the author conducts experiments by modifying the objective function and training samples to verify the effectiveness of the proposed training scheme.", "summary_of_the_review": "Overall, the idea so this paper is somewhat novel. However, the generalization problem in the field of image deraining lies in the difference between sythetic data and real-world data,  which has not been verified by experiments in this paper.", "strength_and_weaknesses": "Strength：\n+ The authors' perspective on solving the generalization problem is interesting.\n\nWeaknesses:\n- Lack of experiments on real-world rainy images.\n- There are many typos. The author should polish the manuscript.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject