SlowGuess/NeurIPS_2023_OCR huggingface dataset:
The relative complexity between the background and rain determines the network's behaviour. To corroborate the abovementioned conjecture, we modify the range of the rain streaks used in training, as outlined in Section 2.1. When employing a medium-level rain range, the rain removal effect diminishes when training with 64 background images. According to our explanation, a larger rain range complicates the network's task of learning the rain pattern. As a result, the rain removal effect does not deteriorate until a larger number of background images are used for training. The experimental results are shown in Figure 5. It can be observed that the rain removal effect decreases across all three training rain ranges as the number of background images increases. When sufficient background images are used for training (30,000 images), even with a large training range for rain, the resulting model struggles to deliver satisfactory rain removal performance on unseen rain streaks. This suggests that the large rain range does not encompass our testing scenarios and cannot bring generalization ability on test images. If training with the large rain range, the network displays a significant drop in rain removal performance only when more than 512 background images are used for training. Conversely, a model trained on a small rain range cannot exhibit satisfactory rain removal even with only 16 background training image images. These results indicate that the relative relationship between the background image and rain streaks influences network behaviours. The complexity or learning difficulty of the medium-range rain is approximately less than that of 64 training background images. In comparison, the complexity of the large-range rain is approximately less than that of 512 training background images. Depending on the situation, the network tends to take shortcuts or select the easier learning pathway.
A more complex background dataset makes learning harder for the network. Next, we modify the category of the background images used for training and monitor the resulting behaviours of the models. We normalize the deraining effect to a range between 0 and 1 to facilitate comparison across different image categories. The results are shown in Figure 6 (a). The most intuitive conclusion is that, even when the number of training images remains consistent, different image categories can lead to different rain removal performance. For instance, in the case of CelebA images, increasing the training set from 8 to 16 images leads to a notable decrease in deraining performance. This disparity is more stark compared to natural image – increasing to 16 images does not result in a significant drop in rain removal performance. Moreover, for images sampled from the Manga109 and Urban100 categories, the rain removal performance does not exhibit a significant drop until the image number exceeds 32. According to our interpretation, the more complex image categories will prompt the
(b) SPDNet
(c) RCDNet
(d) ResNet (10k patches, Large Rain)
(e) RCDNet (Large Rain)
Figure 8: Visualization of the deraining results on a synthetic image. Zoom in for better comparison.
(f) RCDNet (64 patches)
(g) RCDNet (128 patches, Large Rain)
(h) Ground Truth
Training Back.	Objective Range	ResNet	SPDNet [68]	RCDNet [54]
ER↑	EB↓	PSNR↑	ER↑	EB↓	PSNR↑	ER↑	EB↓	PSNR↑
30k	Medium	31.24	10.79	25.15	33.63	5.49	30.51	26.55	5.41	28.54
64	Medium	53.33	25.02	20.87	-	-	-	45.47	14.78	25.32
512	Large	-	-	-	39.88	8.91	28.57	37.53	7.16	29.60
256	Large	45.64	16.51	24.30	38.87	8.03	29.40	40.40	8.52	29.08
128	Large	51.75	23.53	21.45	43.20	14.59	25.67	44.67	13.72	26.09
Table 2: Quantitative comparisons between different models.  $\uparrow$  means the higher the better while  $\downarrow$  means the lower the better.
model to experience an earlier performance decline as the number of training images increases. Our results suggest that the complexity of these four image categories can be ranked in ascending order as CelebA, DIV2K, Manga109, and Urban100.
The observed sequence roughly aligns with human perceptual tendencies. Face images, as represented by the CelebA dataset, exhibit strong global and local structures. While DIV2K images are abundant in local texture, they maintain a relatively simple global structure. Contrastingly, Manga images, although free of complex textures, often contain text elements and detailed edges. Lastly, Urban images are chiefly characterized by recurrent patterns, such as stripes and grids. To corroborate our conclusion, we validate it using a complexity metric derived from a mathematical model. Bagrov et al.[2] proposed a computational method for estimating the structural complexity of natural patterns/images. We compute the multi-scale structural complexity for these four image categories, and the results corroborate our observed ordering, as depicted in Figure 6 (b). This provides some mathematical evidence to support our claim.
3.2 Reconstruction on Background
The aforementioned results indicate that the deraining capability can be enhanced by limiting the complexity of the background images used for training. However, utilizing only a restricted set of background images also has drawbacks. While it prevents the network from overfitting rain patterns, it may conversely prompt the network to overfit the limited background images. We also conduct experiments to address this particular concern.
Using the decoupled evaluation metric  $E_B$  described in Section 2.2, we are able to assess the reconstruction of the background separately. The results in Figure 7 show that as the number of training images increases, the quality of background reconstruction also improves. Remarkably, training with just 256 background images can already yield a satisfactory background reconstruction performance. Adding more training images beyond this point does not significantly improve performance. These findings are surprising and counter-intuitive. We typically assume that training low-level vision models requires a large number of images. However, our research suggests that training with an excessive number of background images does not necessarily enhance the reconstruction performance but rather exacerbates the model's tendency to overfit rain streaks. Another unexpected result is that a model trained with merely 256 images can already handle most image components. This might imply that the complexity of image components and features for a low-level vision network might not be as high as commonly believed.
Relationships with large models and large data trends. We have also noticed recent trends around large models and large data. Many studies point out that larger models (from billions to hundreds of billions of parameters) and larger data can lead to the emergence of capabilities that are previously unimaginable [26]. This seems to contradict the discovery described in this work.
Figure 9: Qualitative results on real-world test images. Zoom in for better comparison.
However, the models that can achieve significant gains from a large amount of data and model parameters are some generative-based models, such as GPT [3] and diffusion models [41]. The large-scale dataset is not the only critical factor to the remarkable ability of these large models. The training method is also very important. Giving a large-scale dataset to a simple task will not make the model generalize to other tasks. In low-level vision, we argue that the task of learning an end-to-end input-to-output mapping via a simple image comparison loss may be relatively easy. In contrast, an important factor for the success of the large language model is its generative pre-training learning method. This learning method enables the model to learn the content rather than the mapping described by the input and output. It is also the core point of this paper that solving the generalization problem requires learning the content of the data, instead of the degradations.
4 Implication
This paper does not propose any new algorithms directly, but can provide insights on how to improve the generalization performance of low-level models. Our experiments have yielded three significant practical findings: (1) By limiting the number of background images used in training, the network can focus more on learning the image content instead of overfitting the rain streaks; (2) Enlarging the range of rain streaks in the training set can allow for the use of more background images in training; (3) Surprisingly, a small number of background images can yield satisfactory reconstruction performance. These findings can be directly applied to enhance the generalization capability of existing models with minimal modifications. Our strategy is straightforward: find a balance between the number of background images and the range of rain streaks in the training set to avoid overfitting.
Some quantitative results are presented in Table 2. We use three deraining models as baselines (ResNet, SPDNet [68], RCDNet [54]) and demonstrate the power of the proposed strategy. We train our baseline models by using 30,000 background images and medium-range rain. The test set is the R100 dataset [63]. We quantify the deraining and background reconstruction effects according to the decouple evaluation metrics  $E_{R}$  and  $E_{B}$ . We also calculate the PSNR performance as a reference. It can be seen that using the existing training methods cannot generalize well to the unseen rain of R100, which is shown by the poor deraining performance in Table 2. However, due to the learning on a large number of images, the reconstruction errors of the baseline models are generally lower. Thus, the PSNR values cannot objectively reflect the effect of rain removal. We reduce the training background images to 64, which is the upper limit of the image number that can make the model generalize under medium-range rain. At this time, the rain removal performance has been greatly improved, but at the cost of background reconstruction performance. By enlarging the rain range and training with more background images, we are able to achieve a trade-off between rain removal performance and background reconstruction.
Figure 8 shows a qualitative comparison of these models under different training objectives. It can be seen that even with the advanced network structure design, the rain removal effects of the baseline models of SPDNet and RCDNet are not satisfactory. Using a larger range of rain can bring limited improvements. In the case of medium-range rain, reducing the background image to 64 significantly improves the rain removal effect and results in unstable image reconstruction. When the rain range is enlarged, and the training background is set to 128 images, the model can show excellent performance in rain removal and background reconstruction. Note that we do not use additional data or improve the network structure throughout the process. We only adjust the training data.
We also present the comparison on real images in Figure 9. In addition, semi-supervised methods [58, 23] have also been used to improve the deraining effect on real images, and we also include the
representative method Syn2Real [66, 67]. Syn2Real-Syn is trained on synthetic data, and Syn2Real-Real is trained on synthetic labelled data and real unlabeled data. Due to the difference in the distribution of rain streaks, the models trained using synthetic data cannot produce satisfactory rain removal effects. When obtaining some real images, Syn2Real-Real can indeed achieve some improvement. However, these improvements are not brought by improving the generalization ability. Because these methods manage to convert "rain outside the training set" to "rain inside the training set". Since data collection is extremely difficult, this method still faces great challenges in practice. Our method improves generalization performance and achieves better results on test images. We also find that artifacts may appear in the background part of the output images. Given that our training utilizes only 256 images without any specific techniques, the outcome, while expected, showcases our prowess in the removal of rain streaks. Indeed, while competing methods might sidestep these artifacts, they fall short in removing rain streaks effectively. We recognize various solutions existing to rectify the artifact issue – from introducing post-processing modules to leveraging this outcome to steer another deraining network. These methods deviate from the interpretability topic of this work, and we hope to discuss technical means to improve generalization performance further in future works.
5 Conclusion and Insights
We investigate the generalization problem in deraining networks. While our work focuses on image deraining, our key conclusions can provide insights for the broader field of low-level vision. We argue that the generalization problem in low-level vision cannot be attributed solely to insufficient network capacity. Instead, we discover that existing training strategies do not promote generalization. Networks tend to overfit the degradation patterns in the training set, leading to poor generalization performance for unseen degradation patterns. Moreover, the current need for practical interpretability tools for low-level vision models presents a significant obstacle. These tools are necessary to understand what the low-level model is learning and why it learns this way. This gap between the ideal and the reality compels us to reconsider the importance of interpretability in low-level vision networks. Our work provides a necessary perspective on this issue. These insights also need to be tested and validated in other low-level vision tasks. Our findings highlight the importance of continued research in this area, and we hope that our work will inspire further investigations.
Acknowledgement This work was supported in part by the National Natural Science Foundation of China under Grant (62276251, 62272450), the Joint Lab of CAS-HK, the National Key R&D Program of China (NO. 2022ZD0160100), and in part by the Youth Innovation Promotion Association of Chinese Academy of Sciences (No. 2020356).
[1] Pablo Arbelaez, Michael Maire, Charless Fowlkes, and Jitendra Malik. Contour detection and hierarchical image segmentation. IEEE transactions on pattern analysis and machine intelligence, 33(5):898-916, 2010.
[2] Andrey A Bagrov, Ilia A Iakovlev, Askar A Iliasov, Mikhail I Katsnelson, and Vladimir V Mazurenko. Multiscale structural complexity of natural patterns. Proceedings of the National Academy of Sciences, 117(48):30241-30251, 2020.
[3] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
[4] Yi Chang, Luxin Yan, and Sheng Zhong. Transformed low-rank model for line pattern noise removal. In Proceedings of the IEEE international conference on computer vision, page 1726-1734, 2017.
[5] Chenghao Chen and Hao Li. Robust representation learning with feedback for single image deraining. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, page 7742-7751, 2021.
[6] Hanting Chen, Yunhe Wang, Tianyu Guo, Chang Xu, Yiping Deng, Zhenhua Liu, Siwei Ma, Chunjing Xu, Chao Xu, and Wen Gao. Pre-trained image processing transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, page 12299-12310, 2021.
A Other Related Work
A.1 Image Deraining
Many methods have been proposed to develop state-of-the-art deraining networks. These works include deep networks designs [12, 53], residual networks [13, 32], recurrent networks [40, 64, 62], multi-task [57, 11] and multi-scale designs [25, 14, 65, 69, 59, 52, 70], sparsity-based image modeling [21, 78], low-rank prior [4], model-driven solutions [54, 56], attention mechanism [51, 6, 15], Transformer-based network [8], adversarial learning [29], representation learning [5], semi-supervised [66] and unsupervised learning [9]. Deep learning methods are data-hungry, but collecting rain streaks and background image pairs is challenging. A lot of works have been proposed to synthesize rain streaks with better results. Garg et al.[16] first propose a physically-based photo-realistic rendering method for synthesizing rain streaks. Zhang et al.[71] and Fu et al.[12] use Photoshop software to manually add rain effects to images to build the synthetic paired data. Due to the poor generalization performance of existing methods, models trained on synthetic images were found to be ineffective in real-world scenarios. Some works [63, 72, 55] that have contributed to real-world collected deraining datasets. However, acquiring these datasets is still expensive and cannot solve the problem of poor generalization. There are also works that mention the generalization issue of the deraining models. Xiao et al.[60] and Zhou et al.[77] attempt to improve the generalization ability of deraining networks by accumulating knowledge from multiple synthetic rain datasets, as most existing methods can only learn the mapping on a single dataset for the deraining task. However, this attempt does not allow the network to generalize beyond the training set.
In addition, semi-supervised methods [58, 23] have also been used to improve the deraining effect on real images, and we also include the representative method Syn2Real [66, 67]. There are some semi-supervised deraining methods [58, 23, 66, 67] are proposed to improve the performance of deraining models in real-world scenarios. When obtaining some real images similar to the test images, these works can achieve some improvement. However, these improvements are not brought about by improving the generalization ability. Their solution is to include real test images in the training set, even if we don't have corresponding clean images. These methods are effective when we can determine the characteristics of the test image. However, more is needed to solve the generalization problem. Because these methods manage to convert "rain outside the training set" to "rain inside the training set". Since data collection is extremely difficult, this method still faces great challenges in practice.
A.2 Low-Level Vision Interpretability
Next, we provide a detailed review of existing work on low-level visual interpretability. Gu and Dong [19] bring the first interpretability tool for super-resolution networks. Xie et al.[61] find the most discriminative filters for each specific degradation in a blind SR network, whose weights, positions, and connections are important for the specific function in blind SR. Magid et al.[37] use a texture classifier to assign images with semantic labels in order to identify global and local sources of SR errors. Shi et al.[45] show that Transformers can directly utilize multi-frame information from unaligned frames, and alignment methods are sometimes harmful to Transformers in video super-resolution. They use a lot of interpretability analysis methods in their work. The closest work to this paper is the deep degradation representation proposed by Liu et al.[33]. They argue that SR networks tend to overfit degradations and show degradation "semantics" inside the network. The presence of these representations often means a decrease in generalization ability. The utilization of this knowledge can guide us to analyze and evaluate the generalization performance of SR methods [34].
B Transferability of Limited Training Images
At the end of the main text, we described a method to improve the generalization performance of the deraining network by reducing the number of training background image images. However, this method will overfit the image content when the number of training background images is very small. In the main text, we investigate the risk of reducing the number of training images when testing on the same image category. Recall that with the increase of training images, the reconstruction of the background becomes better. Training with 256 background images can already bring a good background reconstruction effect. Continuing to add training images does not further improve the performance of background reconstruction.
Figure 10: The relationship between the number of training images and their background reconstruction performance (upper row) and rain removal performance (lower row). The test image set for these six plots is the DIV2K set. We train the model with all four image categories to validate the performance when the image distribution mismatch. For background reconstruction, lower values on the  $y$ -axis mean better background reconstruction. For rain removal, higher values on the  $y$ -axis mean better rain removal performance.
Figure 11: The relationship between the number of training images and their background reconstruction (a) and rain removal (b) performance. The test image set for these two plots is the DIV2K set. We train the models with different model complexities. For background reconstruction, a lower value on the  $y$ -axis means better. For the rain removal effect, a higher value on the  $y$ -axis means better. The test rain patterns are not in the training set. The effect of rain removal at this time reflects the generalization performance.
In this section, we investigate whether the proposed scheme is still robust when the training and testing image distributions are significantly different. We train the models on four image categories and then test them using the DIV2K image category. This simulates the situation when the background image distribution differs from the test set. We observe the behaviour of models trained with different numbers of images. The results are shown in Figure 10. We can draw the following conclusions. First of all, even if the distribution of training background images is different, the models trained using the images of CelebA and Urban categories can still perform similarly to the model trained by DIV2K images. These models can reconstruct background images well when training images reach 256 or more. At this time, the difference in the distribution of these training sets and DIV2K does not bring significant differences. The rain removal effect of these models is also similar. Second, we found that the model trained with the Manga image differed from others. The models trained with the Manga images are generally worse at background reconstruction than other models. Even with larger image numbers, the models trained with the Manga images still cannot achieve similar performance to other models. For rain removal, the model trained with the Manga images also performs the worst. These results are in line with our expectations because the Manga images significantly differ from other images, especially for the low-level image components. Although the other three categories of images
Input Rainy Image
SPDNet
RCDNet
ResNet, Trained w/ 10k patches, Large Rain
RCDNet, Trained w/ Large Rain
RCDNet, Trained w/ 64 patches
RCDNet, Trained w/ 128 patches, Large Rain
RCDNet, Trained w/ Large Rain
Figure 12: Visualization of the deraining results. Zoom in for better comparison.
are also different from each other in image structure, texture type, and other characteristics, they all belong to the category of natural images. Whereas the Manga images contain artificial textures and edges, which are quite different from other images.
There is a large image reconstruction error when training with images whose distribution is very different from that of the test set images. This is reasonable because, in this case, even using a large number of training images cannot bridge the error caused by the distribution mismatch. We are pleased that the method of training using limited background images is robust to image content to a considerable extent, as long as the training images are natural images. This is consistent with our practice. In practice, we also found that the results are stable as long as more than 256 image images are used for training. There is no significant performance change due to the content of the selected training images.
C Discussion about Model Complexity