SlowGuess/NeurIPS_2023_OCR huggingface dataset:
The impact of model complexity on generalization performance cannot be ignored either. In general, models with smaller capacity are less prone to overfitting, although this is in exchange for generally lower performance. We next investigate the validity of our conclusions under models with different complexity. Specifically, we use three different sizes of ResNet architectures and conduct experiments
on the DIV2K dataset, aligning with the experiments presented in Figure 4 and Figure 7. The results are illustrated in Figure 11. One can see from the results that all three models of varying complexities exhibit similar behaviours in rain removal performance. When the training background complexity is on the lower side, the more complex models deliver superior rain removal generalization (as can be seen in the right figure, where higher is better). Conversely, with the increase of the training background complexities, these more complex models encounter larger generalization challenges. More importantly, the inflection point, where this transition occurs, remains consistent irrespective of the model's complexity. This confirms our explanation that within this training framework, the behaviour of the network is primarily determined by the relative complexity between the training background image and the rain degradation. Similar trends are noticeable in the context of background reconstruction. For more complex models, the quality of background reconstruction tends to degrade more when trained with a limited number of images. This can be attributed to the pronounced overfitting tendency of larger models. Interestingly, the rain removal performance is enhanced at this time, indicating the model's focus on image content. As the number of training images increases, the larger models also yield improved background reconstruction, which aligns with our expectations. It is crucial to highlight that the transition in network behaviour remains consistent across models of differing complexities. This reaffirms our conclusion that, within this framework, the network's behaviour is partly steered by the relative complexity interplay between the training background images and the rain degradation.
D More Results
We provide more results of different deraining models in Figure 12 and Figure 13. Note that we did not use additional data nor improve the network structure throughout the process. We only adjust the training objective. Although the effect of the output image can be further improved, it shows the practical value and application potential of our conclusions.
E Limitation.
Our work mainly takes the deraining task as a breakthrough point, and attempts to make a general summary of the generalization problem in low-level vision. Due to the differences between different low-level tasks, the analysis methods in this paper, especially the fine-grained analysis methods, may not be directly used on some other tasks. However, our work can still bring novel insights to the low-level vision field.
Our work also attempts to improve existing deraining models. But these improvements are based on the simple usage of some key conclusions of our work. Although shown to be effective, we believe that these methods are still far from ideal. We only demonstrate the application potential of the knowledge presented in this work and have no intention of proposing state-of-the-art algorithms or models. Research efforts are still needed to develop more robust deraining algorithms using our conclusions.
F Reproducibility Statement
F.1 Resources
The models used in our work are taken directly from their respective official sources. Our code is built under the BasicSR framework https://github.com/xinntao/BasicSR for better code organization. The deraining model SPDNet [68] is available at https://github.com/Joyies/SPDNet. The deraining model RCDNet [54] is available at https://github.com/hongwang01/RCDNet. The training and testing datasets used in our work are all publicly available.
F.2 Network Training
Due to space constraints, we do not describe our training method in detail in the main text. Here, we describe the training method to reproduce our results. A total of 150 models were involved in our experiments. We used the same training configuration for all models. We use Adam for training. The initial learning rate is  $2 \times 10^{-4}$  and  $\beta_{1} = 0.9$ ,  $\beta_{2} = 0.99$ . For each network, we fixed the number of training iterations to 250,000. The batch size is 16. Input noisy images are of size  $128 \times 128$ . The cosine annealing learning strategy is applied to adjust the learning rate. The period of cosine is 250,000 iterations. All models are built using the PyTorch framework [39] and trained with NVIDIA A100 GPUs.
F.3 Availability
All the trained models and code will be publicly available.
G Ethics Statement
This study does not involve any human subjects, practices to data set releases, potentially harmful insights, methodologies and applications, potential conflicts of interest and sponsorship, discrimination/bias/fairness concerns, privacy and security issues, legal compliance, and research integrity issues. We do not anticipate any direct misuse of our contribution due to its theoretical nature.
Input Rainy Image
RCDNet, Trained w/ Large Rain
Figure 13: Visualization of the deraining results. Zoom in for better comparison.
ResNet, Trained w/ 10k patches, Large Rain
Ground Truth