Rainmer: Learning Multi-view Representations for Comprehensive Image Deraining and Beyond

Authors: peirong  ma
zhiquan  he
hong  lu
Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-07 15:54:50

Abstract: We address image deraining under complex backgrounds, diverse rain scenarios, and varying illumination conditions, representing a highly practical and challenging problem. Our approach utilizes synthetic, real-world, and nighttime datasets, wherein rich backgrounds, multiple degradation types, and diverse illumination conditions coexist. The primary challenge in training models on these datasets arises from the discrepancies among them, potentially leading to conflicts or competition during the training period. To address this issue, we first align the distribution of synthetic, real-world and nighttime datasets. Then we propose a novel contrastive learning strategy to extract multi-view (multiple) representations that effectively capture image details, degradations, and illuminations, thereby facilitating training across all datasets. Regarding multiple representations as profitable prompts for deraining, we devise a prompting strategy to integrate them into the decoding process. This contributes to a potent deraining model, dubbed Rainmer. Additionally, a spatial-channel interaction module is introduced to fully exploit cues when extracting multi-view representations. Extensive experiments on synthetic, real-world, and nighttime datasets demonstrate that Rainmer outperforms current representative methods. Moreover, Rainmer achieves superior performance on the All-in-One image restoration dataset, underscoring its effectiveness. Furthermore, quantitative results reveal that Rainmer significantly improves object detection performance on both daytime and nighttime rainy datasets. These observations substantiate the potential of Rainmer for practical applications.

Main paper: 
 introduction: 
 Rain occurs approximately eight times more frequently than fog and snow according to the statistics of the latest large autonomous driving dataset [1] . As the most common adverse weather condition, rain significantly impairs outdoor computer vision applications. Recently, the sixth UG 2 + Prize Challenge highlighted the importance and urgency of real-world image deraining tasks.Deep learning-based methods [4, 16, 30, 50-52, 54, 57, 58, 63 ] have made significant progress in image deraining in decades. However, the majority of these algorithms are primarily trained on synthetic datasets [4, 16, 30, 51, 52, 54, 57] , which limits their effectiveness in real-world and nighttime scenarios. Consequently, a fundamental question arises: What factors contribute to the challenges in establishing a comprehensive image deraining model?Constructing paired real-world and nighttime datasets has long been challenging in the image de-raining literature. Because capturing rainy images and clean images simultaneously while maintaining pixel-wise consistency is usually rendered impossible. Therefore, the majority of methods rely on synthetic datasets [8, 20, 51, 63] to develop powerful deraining models, which often struggle in real-world rainy environments and nighttime scenarios. Recently, Wang et al. [45] proposes to generate pseudo clean backgrounds from real rainy videos to create paired datasets. Subsequently, Ba et al. [2] and Zhang et al. [62] introduce a time multiplexing technique to collect paired rainy and clean images from online video streams at different timestamps. Though effective, these methods mainly collect images in static scenes with little motion movement and cannot be adapted to heavy rainy scenes as well as low-light conditions, restricting the richness of backgrounds and illuminations. Most recently, Zhang et al. [60] create a paired nighttime rainy dataset leveraging the weather simulation technique in the GTAV game. Hence, a more feasible approach to developing comprehensive deraining models is to combine the advantages of synthetic, realworld, and nighttime datasets. The benefits are three-fold: 1) Rich background scenes from synthetic datasets. Synthetic datasets [8, 20, 52] usually utilize high-quality images from BSD [29] and UCID [38] to synthesize rainy images; 2) Rich degradations from synthetic and real-world datasets. Synthetic approaches can control the rain by tuning hyper-parameters, while the real-world dataset contains complex rain effects, e.g., rain streaks, rain veiling effect, blur, and color distortions; 3) Diverse ambient lighting conditions from nighttime datasets. Rain in the daytime is often monochromatic [59, 65] while being colored by ambient lights in nighttime scenarios [60] . Current approaches [16, 30, 57, 58] attempt to acquire decent deraining ability by directly training models on mixed synthetic datasets. However, this approach suffers from larger dataset discrepancies arising from various degradations and significant illumination contrasts. There are also All-in-One methods [19, 21, 67] developed to address multiple weather conditions, but they pay little attention to the diversity and complexity of rain itself. Additionally, we observe significant differences in the rain density distribution between daytime and nighttime datasets, which may impede the learning process.To address the aforementioned issues, we first align the rain density distributions among synthetic, real-world, and nighttime datasets. Typically, we find that a large number of black blocks in rainy night images are meaningless for training and should be removed. Then, we aim to extract multi-view (multiple) representations from rainy images that characterize image details, degradations, and illuminations, as illustrated in Fig. 1 . Existing representation learning approaches focus on learning degradationrelated representation [19, 44] , discriminating between rain and backgrounds [5, 56] and extracting joint rain-/detail-aware representation [35] , which are not capable of simultaneously perceiving image details, degradations, and illuminations. Inspired by [35] , we utilize detail-/degradation-aware representation to capture image backgrounds, rain streaks, and blur effects. This joint representation facilitates learning overall rain densities. Moreover, a chromatic representation is explored to capture color distortions, rain veiling effects, and illuminations. These two kinds of representations constitute multi-view representations, which are efficiently learned with the proposed contrastive learning approach in an unsupervised manner. These representations are expected to model discrepancies among datasets well and promote dataset collaboration. Beyond, they serve as valuable prompts to guide the deraining process as shown in Fig. 1 . Hence, we further devise a prompting strategy that contributes to Rainmer's design. Unlike current prompting strategies [33, 39] which employ a fixed number of prompts, the proposed Rainmer directly extracts prompts from input images, functioning effectively at larger dataset scales. Additionally, Rainmer includes a Spatial-channel Interaction Module (SCIM), which facilitates the full exploitation of channel information when extracting multi-view (multiple) representations. In this paper, we make the following contributions:1) We propose to learn a comprehensive image deraining model leveraging the combinations of synthetic, real-world, and nighttime datasets. A rain density distribution alignment strategy is introduced to mitigate gaps among these datasets.2) To effectively learn from synthetic, real-world, and nighttime datasets, we propose a multi-view representation learning method aimed at capturing backgrounds, degradations, and illuminations. On top of these representations, we develop Rainmer, which utilizes multiple representations to prompt image deraining. Additionally, a SCIM module is incorporated into Rainmer.3) We conduct extensive experiments on synthetic, real-world, and nighttime datasets, where the proposed Rainmer outperforms current state-of-the-art. Remarkable performance on the All-in-One image restoration dataset further emphasizes the superiority of the proposed approach. Moreover, experimental results on downstream object detection indicate a significant improvement in mean Average Precision (mAP) on both daytime and nighttime datasets. 
related work: 
 Single Image Deraining focuses on the removal of rain effects to restore clean backgrounds for outdoor computer vision applications. Prior-based deraining methods, utilizing techniques such as Gaussian Mixture Models [23] and dictionary learning [10, 27] , achieve this goal through iterative optimization but often incur significant computational burdens and struggle with generalization. With the advent of deep learning, learning-based approaches [8, 30, 35, 43, 52, 54, 58] have emerged, dramatically enhancing deraining ability over the past decades. In the realm of learning-based deraining, multi-scale design [9, 42, 53] , attention mechanisms [4, 13, 41, 45] , recurrent units [36, 37, 54] , and multi-stage processing [52, 58] have been extensively explored to adress complex and accumulated rain streaks. There are contemporary semi-supervised and unsupervised methods [5, 14, 47, 55, 66] that attempt to incorporate unlabeled real-world data for training, often yielding undesired performance. Current outstanding deraining methods [6, 33, 50] primarily focus on synthetic datasets, with limited attention given to real-world and nighttime environments. Researchers have begun to develop realworld rainy datasets [2, 62] , as well as nighttime dataset [60, 61] . Albeit successful, comprehensive image deraining with complex backgrounds, diverse rain degradations, and varying illumination conditions remains a largely unexplored and urgent problem. Representation Learning-based Image Restoration aims to produce high-quality results by leveraging interactions between intermediate features and abstract representations. Wei et al. [47] represent synthetic and real rain using Gaussian Mixture Models, where Kullback-Leiber divergence [7] is imposed to transfer knowledge from synthetic images to unlabeled real rainy images. The architecture of the proposed Rainmer. Rainmer initially extracts multi-view representations 𝑧 chr and 𝑧 det-g from input image using mixture of experts (MoEs). Then, Rainmer employs a U-Net-based architecture to remove rain effects, where we develop MPM to prompt the decoding process utilizing multiple representations. Additionally, we propose SCIM to maximize spatial-channel interaction when extracting representations.Huang et al. [14] introduce a soft-updating strategy to facilitate interaction among representations of synthetic and real rainy images. Recently, contrastive learning [12] has been widely adopted in image restoration [19, 44, 49, 56] . Typically, Wang et al. [44] extract degradation representations with contrastive learning to guide image super-resolution. Li et al. [19] employ degradation representations to generate parameters of deformable convolution layers, thereby enhancing All-in-One image restoration. In contrast, Transweather [39] incorporates learnable weather-type queries to facilitate feature interaction. Following [39] , Potlapalli et al. [33] introduce a fixed number of learnable prompts to guide the image restoration process. Leveraging a pre-trained model to extract degradation representation, Wang et al. [40] devise a prompting image restorer. Though effective, these methods mainly focus on utilizing degradation-aware representations, neglecting image details and illumination. Most recently, Ran et al. [35] propose to learn joint rain-/detail-aware representations to remove complex rain effects, while overlooking illumination differences. 
comprehensive image deraining: 
 In this paper, we aim to address image deraining leveraging synthetic, real-world, and nighttime datasets. Denote D syn , D real , and D night as the synthetic, real-world and nighttime datasets, respectively. And let D = {x 𝑖 , y 𝑖 } 𝑁 𝑖=1 be their union, where x 𝑖 and y 𝑖 represent 𝑖-th rainy and clean images, and 𝑁 is the total number of samples. To handle discrepancies among these datasets, we propose to learn multiple representations that effectively perceive image details, degradations, and illuminations. Specifically, given rainy image x, we employ a chromatic encoder𝐸 and a detail-/degradation-aware encoder 𝑓 det-g 𝐸 (see Fig. 1 ) to extract corresponding representations following:EQUATIONwhere 𝑧 chr and 𝑧 det-g are 𝑑-dimensional spherical vectors, indicating multi-view (multiple) representations. We introduce a novel contrastive learning strategy to learn 𝑓 chr 𝐸 and 𝑓 det-g 𝐸 , with training objective formulated as:EQUATIONwhere L chr contra and L det-g contra denote contrastive learning losses that supervise the training of 𝑓 chr 𝐸 and 𝑓 det-g 𝐸 . Fig. 2 presents the details of encoders, which share a mixture of experts (MoEs) and possess specific feature projectors denoted as 𝜓 chr and 𝜓 det-g . The representation learning procedure will be elaborated in Section 3.2.Aside from effectively perceiving details, degradations, and illuminations, the multi-view representations could be exploited to assist the deraining procedure. To this end, we further devise Rainmer, which contains Multi-view representations Prompting Modules (MPM) to prompt the decoding process, as depicted in Fig. 2 . In addition to the vector-level prompting in MPM, we introduce a Spatial-channel Interaction Module (SCIM) to exploit rich spatialchannel interaction when extracting representations. Details of Rainmer will be discussed in Section 3.3. Generally, this prompting deraining process can be formulated as: where y p is the restored image, and 𝑓 represents the deraining process of Rainmer as shown in Fig. 2 . The prediction of restored image y p can be supervised with a reconstruction loss:EQUATIONEQUATIONwhere we employ the combination of Charbonnier loss [17] (first term) and SSIM [46] loss (second term), 𝜖 = 10 -3 . In summary, We train Rainmer end-to-end using the combination of L recon and L contra :EQUATIONwhere 𝜆 is a hyper-parameter.The ambient lighting conditions may induce a large gap between daytime and nighttime datasets, which is harmful to training. We investigate the rain density distribution of synthetic, real-world, and nighttime datasets. Typically, we choose Rain13K [16] (synthetic, noted as SynRain), GT-Rain [2] (real-world), and GTAV-NightRain [60] (nighttime) datasets for analysis. Note that rainy images captured at night usually have much lower pixel intensities compared to daytime images, which influences rain density calculation. Hence we introduce a rain density @ illuminance metric:EQUATIONwhere x and y are paired rainy and clean images. ℎ, 𝑤 represents spatial coordinates, and 𝑐 means color channel index. The metric 𝜌 in Eq. ( 6) describes the extent to which the image is corrupted by rain when compared to the overall pixel intensity of the background and rainy image. It can be seen that SynRain and GT-Rain share similar statistics of 𝜌, while presenting a large difference against GTAV-NightRain. Specifically, images with 𝜌 < 0.08 in GTAV-NightRain account for about 90% proportion. In fact, we observe that this is due to plenties of black regions in nighttime images with 𝜌 ≈ 0 (see Fig. 3 (a)), which contains almost no information and is meaningless. To address this problem, we first crop all images in GTAV-NightRain into non-overlap 512 × 512 images, resulting in GTAV-crop. Then we remove a proportion of images with 𝜌 < 0.08 in GTAV-crop by matching the average statistics of 𝜌 in the intervals [0.00, 0.04] and [0.04, 0.08] of SynRain and GT-Rain. Finally, we obtain the GTAV-balance dataset, which presents similar statistics to SynRain and GT-Rain as illustrated in Fig. 3 (b).Contrastive learning [12] has been proven an elegant and successful approach to learning image representation in an unsupervised manner [19, 35, 44] . In this section, we depict the proposed contrastive learning method to learn multi-view representations in detail. To extract representation as shown in Fig. 2 from x, we first utilize a mixture of experts to obtain low-level spatial features:EQUATIONwhere 𝑓 MoE indicates the mixture of experts, and 𝑓 conv is a convolutional layer to expand channels. In practice, we implement 𝑓 MoE with eight experts following [4] . 𝐹 spa is the low-level feature with rich spatial information to perceive image details, degradations, and illuminations. Utilizing chromatic projector 𝜓 chr and detail-/degradation-aware projector 𝜓 det-g containing three convolutional layers, we extract chromatic information and detail/degradation information from 𝐹 spa by:EQUATIONwhere 𝐹 chr and 𝐹 det-g denote corresponding deep features, respectively. Details of 𝜓 chr and 𝜓 det-g are provided in the supplementary materials. Then we can obtain image chromatic representation 𝑧 chr and degradation representation 𝑧 deg following:EQUATIONwhere GAP(•) means global average pooling operation. Note that GAP will destroy the spatial information in feature, hence we obtain detail representation 𝑍 det while preserving spatial information via:EQUATION)where Pool(•) is a pooling operation with 8 × 8 kernel and stride 8.Detail-/Degradation-aware Representation Learning. Ran et al. [35] have developed an efficient approach to extract joint rain-/detail-aware representations. The underlying philosophy is that by pushing rainy image x apart from negatives with blurred backgrounds and negatives with most dissimilar rain effects simultaneously, the model can in turn learn rain-/detail-aware representations. Inspired by this, we construct 𝑁 𝑏 detail-aware samples by employing Gaussian blur on clean image y following [35] , denoted as {y 𝑏 𝑗 } 𝑁 𝑏 𝑗=1 . To construct negatives for learning degradation, we maintain a rain layer archive A 𝑟 similar to [35] . Different from [35] , given rainy image x, we retrieve most dissimilar rain layers from A 𝑟 to construct negatives by:EQUATIONwhere 𝑁 𝑟 means the number of degradation-aware negatives, r and r ′ denote the rain layer of x and rain layer in archive, respectively. The svd(•) calculates the singular values in descending order, which is robust to translation and rotation. Utilizing detail-aware negatives {y 𝑏 𝑗 } 𝑁 𝑏 𝑗=1 and degradation-aware negatives simulated by y + r 𝑗 , we calculate corresponding negative logits for contrastive learning by:EQUATIONwhere 𝑍 det 𝑗 and 𝑧 deg 𝑗 are calculated using Eq. ( 10) and Eq. ( 9), corresponding to y 𝑏 𝑗 and y + r 𝑗 , respectively. By employing data augmentation to x, we can calculate positaive logits 𝑠 det+ and 𝑠 deg+ similar to Eq. ( 12). Hence, the contrastive learning loss L det-g contra in Eq. ( 2) is formulated as:EQUATION)When prompting restoration, 𝑍 det and 𝑧 deg are combined to 𝑧 det-g since they both come from 𝐹 det-g . Chromatic Representation Learning. To perceive color distortions and illumination in rainy images, we compute a chromatic vector u ∈ R 3 for image x via:u 𝑐 = 1 𝐻𝑊 ∑︁ ℎ,𝑤x ℎ,𝑤,𝑐 , 𝑐 ∈ {𝑅, 𝐺, 𝐵},where 𝐻 and 𝑊 denote the spatial resolution, and u characterizes both the color property and overall illuminance. We can re-render the chromatic property of x with another u ′ by:EQUATIONwhere ⊙ means element-wise multiplication, and x is the re-rendering result. Eq. ( 15) provides a way to construct chromatic-aware negatives. To this end, we maintain a chromatic vector archive A 𝑐 and retrieve 𝑁 𝑐 most dissimilar vectors {u 𝑗 } 𝑁 𝑐 𝑗=1 from it given rainy image x. The {u 𝑗 } 𝑁 𝑐 𝑗=1 is then employed to simulate chromatic-aware negatives { x𝑗 } 𝑁 𝑐 𝑗=1 using Eq. ( 15). Similar to Eqs. ( 12) and ( 13), we derive chromatic contrastive loss L chr contra in Eq. ( 2) below:EQUATIONwhere 𝑧 chr , 𝑧 chr+ , and 𝑧 chr 𝑗 are chromatic representations for rainy image x, clean image y, and negatives x𝑗 , respectively. By pulling chromatic representations of x and y together, the proposed method could implicitly address color distortions raised by rain effects.As shown in Fig. 2 , Rainmer first employs MoEs to extract lowlevel features 𝐹 cha , and then processes features through a 4-level encoder-decoder architecture. In the end, the features from the decoder undergo another MoEs to obtain final restored result. At the 𝑙-th level, the encoder comprises 𝑁 𝑙 Multi-head Self-attention Blocks (MSB) as shown in Fig. 2 . Each MSB is a LayerNorm (LN)-Multi-head Self-attention (MSA)-LayerNorm-Feedforward Network (FFN) architecture (see Fig. 2 ). The 𝑙-th decoder contains 𝑁 𝑙 Multihead Self-attention Prompting Blocks (MSPB), with a Multi-view Prompting Module (MPM) inserted behind MSA of the MSB as illustrated in Fig. 2 . The MPM utilizes multi-view representations to prompt the decoding process, thereby producing high-quality results. Additionally, we introduce a Spatial-channel Interaction Module (SCIM) to enhance the low-level features 𝐹 cha . In practice, we implement the MoEs, MSA, and FFN following [4] , where sparse attention and mixed-scale feedforward network are employed.Multi-view Prompting Module (MPM). Given an input feature 𝐹 in , multi-view representations 𝑧 chr and 𝑧 det-g , MPM first computes spatial response maps 𝑅 chr and 𝑅 det-g corresponding to 𝑧 chr and 𝑧 det-g via:EQUATIONwhere G 𝑡 , 𝑡 ∈ {chr, det-g} represents a Conv-GELU-Conv-Sigmoid gate. Utilizing the response map, MPM integrates multi-view representations by:EQUATION)where H 𝑡 , 𝑡 ∈ {chr, det-g} denotes a Conv-SiLU transformation. To merge 𝐹 𝑡 𝑝 , 𝑡 ∈ {chr, det-g}, MPM employs a learnable vector 𝑧 p to calculate the response of chromatic representation by 𝛼 (𝑧 chr , 𝑧 p ) = (1 + 𝑧 chr • 𝑧 p )/2. Finally, MPM produces weighted output:EQUATION)Spatial-channel Interaction Module (SCIM). As shown in Eqs. ( 8) to (10) , all multi-view representations are from 𝐹 spa , which contains rich information. Hence, we propose the SCIM to explore the mutual spatial-channel interaction between 𝐹 spa and low-level features 𝐹 cha from MoEs. SCIM subsequently outputs an enhanced feature for removing various rain effects as well as restoring details. Fig. 2 presents the details of SCIM. 
experiments: 
 In this section, we conduct extensive experiments to assess the efficacy of our approach. Specifically, negative numbers 𝑁 𝑏 , 𝑁 𝑟 , and 𝑁 𝑐 in Section 3.2 are set to 4 following [35, 48] . The capacities of archives A 𝑟 and A 𝑐 are 256, which are dynamically updated with the current data batch in a queue-like manner. The dimensions of all representations are 128 following [12] . In Rainmer, the numbers of MSB (MSPB) in each encoder (decoder) level are [4, 6, 6, 8] . Rainmer comprises about 36M parameters, with an increase of 2M parameters compared to [4] . We implement Rainmer using the PyTorch [32] framework. During training, the hyper-parameter 𝜆 in Eq. ( 5) is set to 0.1. The batch size is 8 with 128 × 128 patch size, and the gradient accumulation technique is employed. All experiments are conducted on an NVIDIA Tesla V100 GPU. We adopt the AdamW optimizer [26] to train Rainmer for 300K iterations. The learning rate starts at 3𝑒 -4 for the initial 92K iterations and then decreases to 1𝑒 -6 for the remaining 208K iterations using the cosine annealing scheme [25] .Datasets. We select Rain13K [16] , GT-Rain [2] , and the GTAVbalance dataset in Section 3.1 for benchmarking image deraining methods. Specifically, the synthetic Rain13K comprises 13,712 image pairs for training, along with 5 testing sets: Rain100L [52] , Rain100H [52] , Test100 [64] , Test1200 [63] , and Test2800 [8] . The challenging real-world GT-Rain comprises 26,124 pairs for training and 2100 for testing. The nighttime GTAV-balance, inherited from GTAV-NightRain [60] , contains 12,321 pairs for training and 3587 for evaluation, all with a resolution of 512 × 512. Additionally, we conduct experiments on an All-in-One image restoration dataset AllWeather [19] , which includes 18,069 training pairs and three testing sets: Outdoor-Rain [20] , RainDrop [34] , and Snow100K-L [24] .For the comparison of object detection in night rainy scenes, we utilize the set3 testing set of GTAV-NightRain, containing 1860 samples with resolutions of 1920 × 1080. Metrics. For quantitative comparison of image deraining and Allin-One image restoration, we adopt widely used PSNR [15] and SSIM [46] metrics following [31, 57] . Additionally, we utilize the mean average precision [3] (mAP) metric to evaluate object detection performance across different intersection over union (IoU) thresholds.We compare the proposed Rainmer with eleven representative methods, including six CNN-based methods (PReNet [37] , BRN [36] , RCDNet [43] , EfDerain [11] , AirNet [19] , and unsupervised NLCL [56] ), four recent Transformer methods (IDT [50] , DRSformer [4] , Promp-tIR [33] , and CoIC [35] with DRSformer backbone), and the hybrid CNN-Transformer method HCT-FFN [6] . Note that AirNet and CoIC are contrastive learning-based methods, while PromptIR is prompting-based. To ensure fair comparisons, we re-train all selected methods on mixed Rain13K, GT-Rain, and GTAV-balance datasets following their official instructions. Quantitative results are tabulated in Table 1 . Rainmer has outperformed all selected methods with the highest average PSNR/SSIM metric across all datasets, substantiating the superiority of the proposed method. Specifically, Rainmer has brought PSNR improvements of 1.12dB, 0.38dB, 1.16dB, 0.19dB, and 0.64dB over the previous best results on four synthetic datasets (Rain100L, Rain100H, Test100, and Test2800) and the GTAV-balance dataset, respectively. Without chromatic representation, both AirNet and CoIC fail to obtain outstanding results on GTAV-balance. Surprisingly, NLCL obtains the worst performance, demonstrating huge challenge of unsupervised deraining with large rain discrepancies. However, Rainmer fails to surpass previous methods, e.g., CoIC on GT-Rain, which may be attributed to the limitations of constructing negatives by linear addition. Fig. 4 provides a visual comparison on both daytime and nighttime datasets. The results on the first two rows in Fig. 4 indicate that Rainmer can better restore image details. For nighttime image deraining, Rainmer is capable of efficiently removing complex and colored rain streaks. Both the quantitative and qualitative results have substantiated the efficacy of the proposed method. Fig. 5 provides six real-world deraining examples. Typically, both the unsupervised NLCL and the recent DRSformer struggle to remove complex rain effects compared to the proposed Rainmer. This indicates that Rainmer exhibits strong real-world deraining ability.We further conduct experiments on the AllWeather dataset to verify the effectiveness of Rainmer. We select NAS [21] , Tran-sWeather [39] , WeatherDiff 64 [31] , WeatherDiff 128 [31] , and WGWS-Net [67] for comparison. Specifically, TransWeather employs learnable weather queries for prompt image restoration. Table 2 reports quantitative results. The proposed Rainmer has dramatically outperformed all methods on Outdoor-Rain and Snow100k-L datasets, offering 1.21dB and 0.26dB PSNR improvements against the previous best method, respectively. However, Rainmer has not achieved the best performance on RainDrop, which may be attributed to the imprecise calculation of raindrop layers when synthesizing degradation-aware negatives. We also provide a visual comparison in Fig. 6 . Compared to the proposed Rainmer, the recent WeatherDiff 128 cannot correct background color under rain veiling effect in Outdoor-Rain. Moreover, WeatherDiff 128 fails to remove raindrop and snow effects in Fig. 6 (b) & (c ). These observations assess the superiority of the proposed approach in image restoration.While removing rain effects efficiently, image deraining methods may not consistently improve object detection performance under rainy scenarios [22] . Therefore, we further investigate object detection on daytime datasets (Test100 and Test1200) and nighttime datasets (GTAV-NightRain (set3)). The real-world GT-Rain is excluded due to incomplete content by cropping. Specifically, we utilize the recent RTMDet [28] for detection. Following [18] , we generate labels from detection results on clean images while excluding low-confidence results. Quantitative results 12 on daytime and nighttime datasets are reported in Tables 3 and 4 , respectively. Notably, Rainmer has obtained the highest mAP .50:.95 metric over all datasets. A visual comparison on nighttime rainy image is presented in Fig. 7 , where HCT-FFN fails to detect the "car" object while Rainmer successfully detects all objects. Summarizing results in Tables 1, 3 and 4, we further investigate the correlation between improvements of image quality metrics and mAP metric. The result is presented in Fig. 8 , where we observe that higher PSNR and SSIM improvements cannot always bring a higher mAP value. Only with a significant gain in PSNR/SSIM can we achieve a higher mAP metric.In this section, we conduct experiments to investigate the effectiveness of the proposed multi-view (multiple) representations, SCIM, and prompt weighting strategy. Specifically, we treat Rainmer without multi-view representations, MPM, and SCIM as our baseline. Effect of Multi-view Representations. We first examine the efficacy of multi-view representations. As illustrated in Table 5 , Rainmer produces degraded average PSNR metrics when removing 1 IDT [50] is excluded due to high inference burden with fixed input size 128 × 128. 2 unsup. NLCL [56] is excluded due to its bad performance. one representation. This demonstrates that multi-view representation is necessary to acquire superior performance. Effect of SCIM. As tabulated in Table 5 , without SCIM, Rainmer has suffered significant performance drops on Rain100L and Rain100H datasets, resulting in average PSNR degradation. This observation verifies the necessity of SCIM.Effect of Prompt Weighting in Eq. (19) . As shown in Table 6 , Rainmer without weighting has undergone severe PSNR metric drops on Rain100L and Rain100H datasets, offering even worse performance over Rainmer only with 𝑧 chr representation. This observation indicates that directly adding prompting results degrades performance, which can be mitigated with a weighting strategy. 
conclusion and future work: 
 We address more challenging and practical image deraining by leveraging synthetic, real-world, and nighttime datasets. In this scenario, complex backgrounds, diverse rain effects, and varying illuminations coexist, potentially inducing competition and conflicts. To tackle this problem, we propose to learn multi-view (multiple) representations that efficiently perceive image details, degradations, and illuminations. Then, we develop Rainmer, a potent deraining model that integrates multiple representations for prompting restoration.Extensive experiments on synthetic, real-world, and nighttime image deraining, as well as All-in-One image restoration, substantiate the efficacy of Rainmer. Furthermore, Rainmer has been proven to significantly improve object detection performance. In the future, we will develop more effective representation learning methods for universal image restoration under various extreme conditions. 
 
 Appendix:  


Decision: This paper was reviewed by four experts in the field. The paper received mixed reviews BA, BA, BA, BR. The reviewers approved the performance of the proposed method. The reviewers also raised several concerns including the significance of the task solved in the paper. Based on the reviews, the AC would like to recommend the acceptance of this paper and suggest the authors include the added experiments in the rebuttal to the final version. However, the authors should add a more detailed discussion about the problem itself and include more all-in-one methods in comparison.

Reviews: review: paper_summary: paper_summary: The paper presents "Rainmer," a U-Net-based model for image deraining that employs multi-view representations to address complex real-world and nighttime scenarios. It introduces a contrastive learning strategy to capture details, degradations, and illuminations, and integrates these representations into the decoding process with a prompting strategy. Rainmer also includes a Spatial-channel Interaction Module (SCIM) for enhanced feature extraction. Experiments show Rainmer outperforms current methods, with improved object detection capabilities in rainy conditions.
strength_weakness: strength_weakness: 1. It captures a comprehensive set of image attributes including details, degradations, and illuminations by learning from diverse datasets, which significantly enhances the model's ability to handle complex deraining tasks.

2. Rainmer demonstrates superior performance over existing methods, with notable improvements in object detection accuracy in rainy conditions, highlighting its practical utility and effectiveness in real-world applications.
limitations: limitations: 1. The model might be optimized for heavy rain scenarios and could potentially perform less effectively on images with minor or light rain effects.
2. Unsupervised methods often demonstrate superior performance in deraining natural scenes due to their reduced risk of overfitting to synthetic datasets. Conducting additional comparative experiments with other unsupervised learning models can further validate the model's advantages in natural settings. For instance, comparisons with models such as "Unsupervised Image Deraining Optimization Model Driven Deep CNN",  "UConNet: Unsupervised Controllable Network for Image and Video Deraining" and "Unsupervised deraining: Where contrastive learning meets self-similarity" on datasets featuring natural scenes could be insightful. Assessments could utilize no-reference metrics or alternative evaluation methods to ascertain the model's strengths in real-world scenarios.
paper_summary: paper_summary: The authors propose a method that leverages synthetic, real-world, and nighttime datasets to train a more robust deraining model. They address the challenges posed by the discrepancies among these datasets by aligning their distributions.
A contrastive learning strategy is introduced to extract multi-view representations that capture various aspects of the image, including details, degradations, and illuminations. 
The paper introduces Rainmer, a U-Net-based deraining model. This model incorporates multi-view representations as prompts during the decoding process to enhance its deraining capability. 
Extensive experiments on three datasets demonstrate that Rainmer outperforms existing state-of-the-art methods. The model also shows improvements in object detection performance in both daytime and nighttime rainy conditions.
strength_weakness: strength_weakness: This paper utilize multi-view representations extracted through a  contrastive learning strategy. 
By aligning distributions of synthetic, real-world, and nighttime datasets and integrating multi-view representations into the decoding process, Rainmer outperforms existing methods on various datasets, including All-in-One image restoration, and significantly improves object detection performance in both daytime and nighttime rainy scenarios.
limitations: limitations: 1. The reviewer notices that Rainmer does not achieve the best results on the GT Rain dataset and there is a significant gap compared to the second method. Can the author discuss the details of the results on this dataset?
2. The Related Work section should be enriched by incorporating relevant literature on contrastive learning in the field, which will enhance the discussion of the innovative aspects of the proposed work. For example：
[1] Ye Y, Yu C, Chang Y, et al. Unsupervised deraining: Where contrastive learning meets self-similarity[C]//Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2022: 5821-5830.
[2] Wu H, Qu Y, Lin S, et al. Contrastive learning for compact single image dehazing[C]//Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2021: 10551-10560.
3. What does the symbol E_(x, y) in Formula 4 represent?
4. Since this is a new setting for image rain removal, can the author provide comparison results on real images to verify the effectiveness of the method.
paper_summary: paper_summary: This paper proposes a method called Rainmer for comprehensive image deraining. The proposed method aims to address the challenges posed by synthetic, real-world, and nighttime datasets by learning multi-view representations that effectively perceive image details, degradations, and illuminations. Experimental results demonstrate that Rainmer outperforms state-of-the-art methods on various datasets, including synthetic, real-world, and nighttime datasets. The proposed method also achieves a significant improvement in mean Average Precision (mAP) on both daytime and nighttime datasets for object detection tasks.
strength_weakness: strength_weakness: 1.	The writing is well and the algorithmic details are clear.
2.	Solving the problem of deraining under various scenes is practical.
3.	The idea of multi-view representation learning is insightful.
4.	The results show the superior performance on each deraining dataset and detection dataset.
limitations: limitations: 1.	Why the authors emphasize the U-Net-based design? Is it one of your contributions? It maybe not proper to call a network with attention blocks as U-Net-based architecture.
2.	I can understand that 8 x 8 kernel better preserve the spatial information than global manner, but with an 8 times down-sampling process, the detail information will still be destroyed. And no relative ablation study appears.
3.	Why the authors group detail and degradation representations? I suppose that the details are more related to the content of the image, which should be consistent when meeting different degradations. The reason of this joint training manner is not clear.
4.	In Fig. 4, why only the bottom two rows have zoom-in box?
5.	The effectiveness of prompting weighting is limited.
paper_summary: paper_summary: This paper proposes an approach for image deraining under various, complex scenarios, including synthetic, real-world and nighttime data. They devise a framework consisting of data distribution alignment module, a contrastive learning for multi-view representation learning, a spatial-channel interaction module. Experiments on synthetic, real-world and nighttime datsets are conducted to validate its superior performance. Evaluation on downstream tasks is also displayed to illustrate its practical application.
strength_weakness: strength_weakness: It is interesting and practical to design a more generalizable method for adverse weather removal tasks.
A large amount of experiments are conducted on multiple datasets to prove the versatility of the proposed method.
Evaluation on downstream tasks like object detection is great, demonstrating the practical applications of the proposed method.
limitations: limitations: 1. This paper proposes an approach for comprehensive image deraining. However, there is no part tailored towards the characteristics of rain streak. Why not directly develop an All-in-One method considering deraining, dehazing and desnowing, similar to NAS, TransWeather? The problem setting is strange.
2. The definition of ``multi-view'' is ambiguous. It is hard to understand why chromatic, detail and degradation share such relation.
3. More All-in-One methods should be compared in Tab. 2, such as TKL, WGWS-Net.
4. Fig. 7 is surprising to me. The performance oscillation happened without regular patterns. More analysis should be given why a better PSNR/SSIM even brings a significant performance drop on downstream tasks.
5. More cases should be given in Fig. 6 to better illustrate practical applications.
6. Visualization results on real-world data are highly desired because this work focuses on generalizable image deraining.
7. Observed from Tab. 5, each proposed module contributes very limited performance gain. It seems the whole framework is designed based on a strong baseline.