SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Jinjin Gu $^{1,2}$  Xianzheng Ma $^{1}$  Xiangtao Kong $^{1,3}$  Yu Qiao $^{1,3}$  Chao Dong $^{1,3,*}$
$^{1}$  Shanghai AI Laboratory  $^{2}$  The University of Sydney
3 Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences
jinjin.gu@sydney.edu.au, xianzhengma@pjlab.org.cn
{xt.kong, yu.qiao, chao.dong}@siat.ac.cn
Deep deraining networks consistently encounter substantial generalization issues when deployed in real-world applications, although they are successful in laboratory benchmarks. A prevailing perspective in deep learning encourages using highly complex data for training, with the expectation that richer image background content will facilitate overcoming the generalization problem. However, through comprehensive and systematic experimentation, we discover that this strategy does not enhance the generalization capability of these networks. On the contrary, it exacerbates the tendency of networks to overfit specific degradations. Our experiments reveal that better generalization in a deraining network can be achieved by simplifying the complexity of the training background images. This is because that the networks are "slacking off" during training, that is, learning the least complex elements in the image background and degradation to minimize training loss. When the background images are less complex than the rain streaks, the network will prioritize the background reconstruction, thereby suppressing overfitting the rain patterns and leading to improved generalization performance. Our research offers a valuable perspective and methodology for better understanding the generalization problem in low-level vision tasks and displays promising potential for practical application.
The burgeoning progress in deep learning has produced a series of promising low-level vision networks that significantly surpass traditional methods in benchmark tests [10, 30, 12]. However, the intrinsic overfitting issue has prevented these deep models from real-world applications, especially when the real-world degradation differs a lot from the training data [20, 31, 33, 34]. We call this dilemma the generalization problem. As a traditional and important low-level vision task, image deraining also faces a severe generalization problem. Existing deraining models tend to do nothing for the rain streaks that are beyond their training distribution, see Figure 1 for an example. The model trained with synthetic rainy images cannot remove rain streak for images with different rain patterns as the training data.
Despite its significance, the generalization problem in the context of deraining tasks is not comprehensively explored in the existing literature. Understanding the underlying causes of these generalization performance issues is imperative to propose effective solutions. However, analyzing generalization in a low-level task is far from straightforward, as it is not a simple extension of the generalization research in high-level vision tasks. This paper aims to take the pioneering step towards a more profound understanding of this challenge.
(a) Input Rainy Image
(b) Baseline Model
(c) Baseline Model, w/ More Background Training Data
(d) Baseline Model, w/ More Rain Data
(e) Reducing Training data According to our Conclusion
Figure 1: The existing deraining models suffer from severe generalization problems. After training with synthetic rainy images, when feeding (a) an image with different rain streaks, its output (b) shows a limited deraining effect. Two intuitive ways to improve generalization performance, including (c) adding background images and (d) adding rain patterns, cannot effectively relieve the generalization issue. In this paper, we provide a new counter-intuitive insight: (e) we improve the generalization ability of the deraining networks by using much less training background images for training.
As a typical decomposition problem, image deraining utilizes a relatively simple linear superimposition degradation model. When the network fails to generalize, the rain streaks persist, and the image background remains unaffected. This intuitive and quantifiable phenomenon is well-suited for our study. Contrary to previous works that rely only on overall image quality evaluation, we propose to decouple the deraining task into rain removal and background reconstruction. These two components are then separately analyzed. A key motivation behind this approach that the unsatisfactory performance can be attributed to "the unsuccessful removal of rain streaks" or "the poor reconstruction of image background". Without distinction, an image with successful rain streak removal but poor background reconstruction will also have bad quantitative results. Given that the generalization problem in the deraining task is mainly related to the removal of rain streaks, we argue that those results that can remove rain will be more enlightening, regardless of the background reconstruction effect. Our research method allows us to minimize the impact of other less relevant factors on the effectiveness of rain removal.
In this paper, we show that the generalization problem arises when the network overfits the degradation, i.e., the rain patterns present in the training set. One significant factor contributing to this outcome is the inappropriate training objective. We start our research with the most fundamental element in formulating the training objective – the training data. Numerous studies have attempted to enhance real-world performance by increasing the complexity of training data. This approach originates from a natural but unproven “acknowledgement” that augmenting the quantity of training data can rectify the generalization problem. This “acknowledgement” has also permeated the deraining community, suggesting that a network trained with a more diverse training set (both in terms of background images and rain streaks) can better generalize to unseen scenarios. However, this approach does not effectively address the generalization problem in deraining. We argue that this issue arises precisely because the network is provided with an excess of background data during training. Consequently, the model fails to learn to reconstruct the image content but instead overfits the degradation. We arrive at some counter-intuitive conclusions by employing our analysis method, which separately measures background reconstruction and deraining effect.
Our key findings. We find that deep networks are slacking off during training, aiming to reduce the loss in the quickest way. This behaviour leads to poor generalization performance. The improper objective set for training is one of the primary contributors to this issue. Our finding indicates that deep networks exhibit a tendency to learn the less complex element between image content and additive degradation.
Specifically, the network naturally learns to overfit the rain streaks when the background complexity is higher than the rain complexity. However, when rain streaks in real-world scenarios deviate from the training set, the network tends to disregard them. At this point, the network will not remove rain streaks from the image and exhibit poor generalization performance. Conversely, training the model using a less complex background image set demonstrates superior generalization ability, as illustrated in Figure 1 (e). This is because that when the complexity of the training image background is less than that of the rain patterns, the network will again take a shortcut to reduce the loss. In this case, the network learns to reconstruct the background rather than overfitting the rain streaks. When faced
with rain streaks outside the training set, the network takes priority to reconstruct the background, thus avoiding failure caused by the inability to recognize new rain streaks. It is also important to note that the model's performance is determined by the removal of rain and the quality of background reconstruction. Reducing the background complexity of the training data might inevitably lead to subpar reconstruction performance. Nonetheless, our results revealed that a model trained with just 256 images can already handle most image components effectively. These counter-intuitive phenomena have not been revealed in previous literature.
**Implication.** Our results highlight the critical role of the training objective in determining a model's generalization ability. An inappropriate or incomplete training objective creates a loophole for deep networks to "slack off". While we anticipate that the network will learn the rich semantics inherent in natural images, it is often overlooked that the model can also achieve learning objectives through shortcuts. This results in poor generalization performance of low-level vision models. Our findings also highlight that a model with good generalization ability should learn the distribution of natural images rather than overfitting the degradation.
This research primarily pertains to the field of deraining. However, unlike most of the existing deraining research, we do not propose new network structures, loss functions, or datasets. Our objective is to analyze and understand the generalization problem within the context of the deraining task. Due to space constraints, reviews of deraining works are provided in Appendix A.1. We then review previous works focusing on interpretability and generalization in low-level vision.
Deep learning interpretability research aims to understand the mechanism of deep learning methods and obtain clues about the success/failure of these methods. We are not convinced to move forward in the right direction without a deep understanding of these working mechanisms. The research on deep learning interpretability follows a long line of works, most focusing on the classification task [47, 48, 46, 49, 76, 36]. Low-level vision tasks have also embraced great success with powerful deep learning techniques. There are also works on interpretability for these deep low-level networks [19, 61, 37, 44]. For the generalization problem in low-level vision, these problems often arise when the testing degradation does not match the training degradation, e.g., different downsampling kernels [20, 31, 27, 74] and noise distributions [22, 7]. The existing works either develop blind methods to include more degradation possibilities in the training process or make the training data closer to real-world applications. Only a little work has been proposed to study the reasons for this lack of generalization performance [33, 34]. More details of these previous works can also be found in supplementary material. No research has attempted to investigate the interpretation of the training process of low-level vision networks, especially from the perspective of the generalization problem.
2 Analysis Method
We aim to investigate how different training objectives influence network behaviour and generalization performance. Before detailing our observations, we will outline our experimental designs and quantitative analytical methods.
2.1 Construction of Training Objective
The training data and the loss function jointly determine the training objective of a deep network. We set various training objectives to observe the changes in the generalization performance of different deraining models. As shown in Figure 3 (left), a rainy image  $O$  can be modelled using a linear model  $O = B + R$ , where  $B$  is the image background, and  $R$  is the additive rain image. We change the training objectives with different background images and rain streaks.
Background Images. Typically, image backgrounds are sampled from street view images [17] or natural image datasets [43, 1], as these images are close to the application scenarios of deraining. In literature, previous works [13, 71] claim that the model can learn the prior knowledge of reconstructing these scenes by training on a large number of such background images. We break this common sense by constructing different training background image sets from the following two aspects.
First, we change the number of training background images. The amount of training data partially determines the complexity of network learning. We argue that as too many background images are provided for training, the model cannot faithfully learn to reconstruct the image content but turns to overfit the degradation patterns. We reduce the complexity of the background images to see how the network behaviour changes in extreme scenarios. In our experiments, we use 8, 16, 32, 64, 128,
(a) Background Image Spaces
(b) Rain Streaks for Training and Testing
Figure 2: (a) Background images from different image categories. It can be seen that the structure of the face image (CelebA) is relatively complex. Natural images (DIV2K) contain natural textures and patterns. The patterns in Manga109 and Urban100 are artificially created - Manga images have sharp edges, while Urban images contain a lot of repeating patterns and self-similarities. (b) Different synthetic rain streak distributions that were used in our experiments.
Figure 3: (Left) The illustration of the rainy image synthesis. (Right) Our fine-grained analysis of the deraining results.
256, 512, and 1024 background images² to build the training datasets, respectively. We also use a large number of images (up to 30,000) to simulate the typical situation when the image background is sufficiently sampled.
In addition to the data scale, the image content will also affect network learning. For example, it is easier for the network to fit the distribution of images with regular patterns. A face image containing both short- and long-term dependent structures is more complex than a skyscraper with only repeated lines and grids [2]. We select image distribution as the second aspect of our dataset construction. We sample from four image datasets that are distinct from each other: CelebA (face images) [35], DIV2K (natural images) [50], Manga109 (comic images) [38], and Urban100 (building images) [24]. Some examples of these images are shown in Figure 2 (a).
Rain streaks synthesis. Since it is hard to collect a large number of real-world rainy/clean image pairs, we follow the previous deraining works [16, 13] to synthesize rainy images for research. We use two kinds of rain streaks for training and testing separately. For training, we
Table 1: Different rain streaks settings.
Range	Quantity	Width	Length	Direction
Small	[200,300]	{5}	[30,31]	[-5°,5°]
Medium	[200,300]	{5,7,9}	[20,40]	[-30°,30°]
Large	[200,300]	{1,3,5,7,9}	[5,60]	[-70°,70°]
use the computational model  ${}^{3}$  to render the streaks left on the image by raindrops of varying sizes, densities, falling speeds, and directions. This model allows us to sample rain streaks from different distributions. We adopt three rain image ranges for training, where different ranges may lead to different generalization effects; see Figure 2 (b) for a convenient visualization and Table 1 for the detailed settings. For testing, we use the synthetic rain patterns in [63]. Although the rain streaks are visually similar to humans, they still pose a huge generalization challenge to existing models.
Loss Function. In low-level vision, the loss function is usually defined by the difference between the output image and the ground truth. We use the  $l_{1}$ -norm loss without losing generality.
2In actual training, we crop large images of different sizes into image patches of size  $128 \times 128$  for training. In this article, unless otherwise stated, these quantities refer to the number of patches.
3The re-implementation of the PhotoShop rain streaks synthesis method. Please refer to this link.
Figure 4: The relationship between the number of training background images and their rain removal performance. The  $x$ -axis represents the image number, and the  $y$ -axis represents the rain removal effect  $E_{R}$ . Higher  $E_{R}$  means better rain removal performance. The test rain patterns are not included in the training set. Thus, the effect of rain removal at this time reflects the generalization performance. The qualitative results are obtained using ResNet.
2.2 Decoupling Analysis of Rain Removal Results
The standard evaluation of a deraining model involves computing similarity metrics between its output and the ground truth images [18]. However, such an evaluation strategy may lead to unfair comparison. For example, an image with perfect background reconstruction but inferior rain removal may have a higher PSNR value than that with perfect rain removal but a slightly flawed background reconstruction (e.g., color shift). Drawing conclusions from such metrics may lead to biased conclusions about a model's generalization capability. Generally speaking, the generalization performance of a deraining model is mainly shown in the form of removing unseen rain. The background reconstruction may affect the visual effect, but it does not correlate directly with the effectiveness of rain removal. However, background reconstruction affects the quantitative result of traditional rain removal metrics. Therefore, we discuss the removal of rain streaks separately from the reconstruction of the background. We can achieve this goal with a simple mask-based decoupling method.
It can be seen from Figure 2 (b) that the pixels in the additive rain image  $R$  without rain streaks should be black, while rain streaks will appear brighter. After synthesis, these black areas reflect the background area, while the brighter areas indicate the rainy regions. A perfect rain removal effect should do minimal damage to the background area and remove the additive signal from the rain streaks area. By processing  $R$  to a binary mask  $M$  using a threshold  $t$ , where  $M_{[i,j]} = 0$  if  $R_{[i,j]} \leq t$  and  $M_{[i,j]} = 1$  if  $R_{[i,j]} > t$ , we can segment the output image  $\tilde{O}$  into the rain streaks part  $\tilde{O} \odot M$  and the background part  $\tilde{O} \odot (1 - M)$ . We then have two metrics:
-  $E_{R} = \sqrt{\mathbb{E}[(\tilde{O}\odot M - O\odot M)^{2}]}$  gives the Rain Removal performance. A network with poor generalization will not remove rain streaks. This term measures the changes made by the network in the rainy regions. A higher value reflects better rain removal performance.
-  $E_{B} = \sqrt{\mathbb{E}[(\tilde{O}\odot(1 - M) - B\odot(1 - M))^{2}]}$  gives the effect of Background Reconstruction by comparing the background regions with the ground truth. A large error in this term means poor reconstruction quality.
2.3 Deep Models
We summarize existing networks into three main categories in our experiments. The first category includes networks composed of convolutional layers and deep residual connections, and we use
Figure 5: When trained with different rain ranges, the model exhibits different rain removal effects. The  $y$ -axis represents the quantitative rain removal effect. When the rain removal performance is lowered to the blue dashed line, the qualitative effect of removing rain decreases significantly. We use ResNet in this experiment.
Figure 6: (a) The relationship between the number of training images and their normalized rain removal performance. When the  $y$  value is lowered to the grey dashed line, the qualitative effect of removing rain starts to decrease significantly. (b) The averaged complexities of different image categories given by [2].
the ResNet [28] as a representative. The second category includes networks with encoder-decoder designs, and we use UNet [42] as a representative. UNet introduces down-sampling and up-sampling layers to extract global and multi-scale features, which have been proven successful in many deraining networks. The last category includes image processing Transformer networks. Transformer [45, 73, 75] is a new network structure characterized by self-attention operations. We select SwinIR [30] as a representative. For more training settings, please check the supplementary material.
3 Understanding Generalization
Next, we conduct experiments based on the above analysis method to explore the counter-intuitive phenomenon in terms of generalization in the deraining task. Our analysis includes two aspects - rain removal (Section 3.1) and background reconstruction (Section 3.2).
3.1 Generalization on Rain Removal
We analyze the rain removal effect on unseen rain streaks. Since we use different types of rain streaks for training and testing, the rain removal results can reflect generalization performance. After extensive experiments, we arrive at the following observations.
Training with fewer background images produces better deraining effects. In this experiment, we use the middle-level range of rain streaks and different background image sets to generate different training datasets, as described in Section 2.1. Experiments are conducted across all four categories of images. We then test the rain removal performance  $E_{R}$  of these models. The test images utilize rain streaks proposed in [63], which are different from the rain distribution of the training set. The testing background images are sampled from the corresponding categories and differ from those in the training set. The experimental results are presented in Figure 4. Despite variations in background images and network architectures, a consistent trend is evident across all experimental results. Notably, deraining models trained on only eight images demonstrate a noteworthy ability to handle unseen rain streaks. In contrast, models trained using a large number of images exhibit an inability to address these rain streaks. Such a phenomenon challenges traditional expectations. Between these polarized states, the rain removal efficacy diminishes as the quantity of training images increases. Once the image number ascends to 256, the networks have predominantly lost their deraining capability. As the image number increases from 1024 to 30,000, there's minimal variance in models' rain removal performance - they all fail to remove unseen rain streaks. This trend is also reflected in the qualitative results.
Figure 7: The relationship between the number of training images and their background reconstruction effect. For each plot, the  $x$ -axis represents the image number, and the  $y$ -axis represents the reconstruction error of the background  $E_B$ .
Here, we attempt to explain this interesting counter-intuitive phenomenon. Although we set the training objective as removing rain streaks from images by comparing the output to the target, the network has two alternative strategies to minimize this training loss. The first strategy involves recognizing and removing rain streaks, while the second consists of recognizing and reconstructing the image background. If the learning strategy is not specified, the network tends to gravitate towards the simpler strategy. When a large number of background images are utilized in training, learning to reconstruct backgrounds becomes significantly more complex than learning to remove rain streaks. Consequently, the network chooses to recognize and remove the rain. This decision, however, can lead to an overfitting problem: when new rain streaks differ from those used in training, the network fails to recognize and remove them. Conversely, when the background image is comprised of only a few image images, learning the background becomes easier than learning rain streaks. In this scenario, the network recognizes image components in the background without overfitting the features of rain streaks. As a result, the model demonstrates superior rain removal capability in images with unseen rain streaks.