Harnessing Joint Rain-/Detail-aware Representations to Eliminate Intricate Rains
Authors: Wu Ran
Peirong Ma
Zhiquan He
Hao Ren
Hong Lu
Venue: ICLR 2024 poster (2024)
Keywords: Joint rain-/detail-aware representation learning
contrastive learning
context-based modulation mechanism
image deraining
Primary area: representation learning for computer vision, audio, language, and other modalities
TL;DR: This study introduces a joint rain- and detail-aware representation learning approach for training powerful image deraining models from multiple dataset sources.

Abstract: Recent advances in image deraining have focused on training powerful models on mixed multiple datasets comprising diverse rain types and backgrounds. However, this approach tends to overlook the inherent differences among rainy images, leading to suboptimal results. To overcome this limitation, we focus on addressing various rainy images by delving into meaningful representations that encapsulate both the rain and background components. Leveraging these representations as instructive guidance, we put forth a Context-based Instance-level Modulation (CoI-M) mechanism adept at efficiently modulating CNN- or Transformer-based models. Furthermore, we devise a rain-/detail-aware contrastive learning strategy to help extract joint rain-/detail-aware representations. By integrating CoI-M with the rain-/detail-aware Contrastive learning, we develop [CoIC](https://github.com/Schizophreni/CoIC), an innovative and potent algorithm tailored for training models on mixed datasets. Moreover, CoIC offers insight into modeling relationships of datasets, quantitatively assessing the impact of rain and details on restoration, and unveiling distinct behaviors of models given diverse inputs. Extensive experiments validate the efficacy of CoIC in boosting the deraining ability of CNN and Transformer models. CoIC also enhances the deraining prowess remarkably when real-world dataset is included.

HARNESSING JOINT RAIN-IDETAL--WARE REPRE- SENTATIONS TO ELIMINATE INTRICATE RAINS
Wu Ran 1.2 Peirong Ma Zhiquan Hel.2, Hao Ren Hong Lu
School of Computer Science, Fudan University Shanghai Key Lab of Intelligent Information Processing (wran21, 2qhe22 @m. em. fudan. edu. en, (prma20, hren1 7, hongl lunglllp u fudan. edu. cn
ABSTRACT
Recent advances in image deraining have focused on training powerful models on mixed multiple datasets comprising diverse rain types and backgrounds. How- ever, this approach tends overlook the inherent differences among rainy images, leading to suboptimal results. To overcome this limitation, we focus on address- ing various rainy images by delving into meaningful representations that encapsu- late both the rain and background components. Leveraging these representations as instructive guidance, we put forth Context-based Instance-level Modulation (CoI-M) mechanism adept efficiently modulating CNN- Transformer-basee models. Furthermore, we devise a rain-/detail-awes -aware contrastive learning strategy to help extract joint rain-/detail-aware representations. By integrating Col-M with the rain-/detail-awerd aware Contrastive learning, we develop CoIC" an innovative and potent algorithm tailored for training models mixed datasets. Morcover, CoIC offers insight into modeling relationships datasets, quantitatively assessing the impact of rain and details on restoration, and unveiling distinct behaviors of mod- els given diverse inputs. Extensive experiments validate the efficacy of CoIC in boosting the deraining ability of CNN and Transformer models. CoIC also en- hances the deraining prowess remarkably when real- weal-world dataset is included.
INTRODUCTION
Images contaminated with rain can severely impair the performance of outdoor computer vision systems, including self-driving and video surveillance (Wang et al. | [002a].. To mitigate the influence of rain, numerous image deraining methods have emerged over the past decades with the objective of
restoring pristine backgrounds from their rain-corrupted counterparts. Recent years have witnessed
the notable success of the learning-based methods (Pu et al. 2017 Li et al, 2018 Yang & Lu 2019
Zamir et al. 2021 Mou et al. 2022 Zamir et al. 2022 Xiao et al.
2022
Ozdenizci & Legenstein
2023). which leverage large labeled datasets train sophisticated image deraining models.
A number of contemporary learning- based methods Yang et al. 2017 Li et al. 2018 Yang et al,
2019 Xiao al., 2022) exclusively train and validate models single dataset. However, such a strategy is infeasible for practical applications, as the requisite training time and physical storage scale linearly with the number of distinct datasets. Furthermore, synthetic dataset tend exhibit restricted diversity in rain characteristics like orientation, thickness, and density, as it is generated utilizing a unitary simulation technique, e.g., photorealistic rendering (Garg Nayar 2006), phys- ical modeling (Li et al. 2019), and Photoshop simulation As a consequence, models trained on single dataset frequently y generalize poor to others. Recent work (Zamir al.] 2021 2022 Mou et al. 2022 Wang et al. 2023) has explored training models amalgamated datasets drawn from multiple sources, yielding enhanced performance under adverse rainy conditions while avoiding overfitting to specific datasets. Nevertheless, these methods directly mix all datasets, which risks
" "Correspondigg author
I "oode is available at: httppp/s//gthhbb.cco///c/hhhahaa
Rain httpgn/ghttppe./....oooooesseeer http::/www..pppppp.ooooooooooooa

---

neglecting the discrepancies among datasets and resulting in suboptimal optimization. As illustrated in Figure I (a), rain density across mixed datasets exhibits a long- taig distribution spanning a wide range. Consequently, models directly trained mixed datasets with suboptimal optimization may exhibit poor real-world deraining ability, as illustrated in Figure (c).
To address the aforementioned limitations, we propose to learn adaptive image deraining through training on mixed datasets. The goal is exploit the commonalities and discrep- ancies among datasets for training. Specifically, the model ar- chitecture and base parameters are shared (commonalities), while image representations are extracted to modulate the model's 's inference process (discrepancies). These representa- tions provide instructive guidance for a novel Context-based Instance-level Modulation (Col-M) mechanism, which can efficiently modulates both CNN and Transformer architec- tures. Col-M is also verified to improve the performances of existing models trained on mixed datasets.
Roall world Seis Sess.iing Cal
MI Roe Beruiiing CORC
Figure (a) rain density distribu- tion. (b) ruin-/detall-wwareeee in-
Further analysis reveals that all these rain-/detail-aware repre- sentations form unified and meaningful embedding space, where images with light rain are primarily characterized by background detail, while heavy rainy images are distin- guished more by the rain itself. A statistical analysis in Fig- ure (b) indicates that the embedding becomes increasingly rain-aware and less detail-aware as rain density increases. This suggests that gathering both background and rain fac- tors becomes crucial when rain density spans a wide range, which is neglected in previous works (Li et al., 2022a Ye et al. 2022). These observations motivate learning meaningful joint embedding space that perceives various rains and background details.
tensities with respect to rain den- sity. (c) & (d) real-world de- raining results of DGUNet (Mou et al. (022) trained on three mixed datasets without and with the pro- posed CoIC, respectively.
Contrastive learning has been widely adopted learn image representations in an unsupervised manner.He He et al. (2020) propose content-related instance-diseriminaate contrastive learning algo- rithm, where the degradation factor is overlooked. More recently, contrastive tearning-based image restoration approaches have sprung up. WWang et al. (2021) assume that degradations of two im- ages are different. However, such an assumption may be infeasible for rainy images when multiple datasets mixed. Li et al. (2022a) focus learning discriminative rain, fog, and snow represen- tations. Ye et al. (0022) attempt to separate rain layer from background. [Chen et al. (2022) pro- pose dual contrastive learning approach to push rainy and clean images apart. Nevertheless, (Li et al. 2022a Ye et al. [2022] Chen et al. (022) all fail to perceive coupled intricate rains and back- ground details. Wu et al. (2023) introduce contrastive learning as perceptual constraint for image super-resolution, but it can only distinguish between degraded and high-quality images. Therefore, learning joint rain-/detail-awee aware representations serves as non-trivial problem, considering diverse coupling modes of rain and background among datasets. To this end, we propose novel joint rain-/detail-aware contrastive learning approach. Further, by integrating it with Col-M, we develop CoIC, a strategy to train pigommerformance, generalizable CNN or Transformer models using mixed datasets. As illustrated in Figure (c) & (d), models trained under the CoIC strategy demonstrate improved deraining ability to world rainy images.
Contributions. In this work, we propose to learn adaptive deraining models on mixed datasets through exploring joint rain-/detail- aware representations. The key contributions are follows: (1). We introduce novel context-based instance-level modulation mechanism for learning adaptive image deraining models mixed datasets. Rain-/detail-award representations provide instructive guidance for modulation procedure. (2). To extract rain-/detail-aware representations effectively, we develop a joint hin-/detail-awarr contrastive learning strategy. This strategy facilitates the learning of high-quality representations for Col-M. (3). By integrating Col-M and the proposed contrastive learning strategy, we introduce the CoIC to enhance deraining performance for models trained on mixed datasets. CoIC provides insight into exploring the relationships between datasets and enables quantitative assessment of the impact of rain and image details. Experimental results demonstrate that CoIC can significantly improve the performance of CNN and Transformer models, as well as enhance their deraining ability on real rainy images by including real-world dataset for training.
2

---

2
RELATED WORK
Image Deraining. Traditional image deraining methods foeus separating rain components by utilizing carefully designed priors, such as Gaussian mixture model (Li et al. 2016), sparse repre- sentation learning (Gu et al. 2017 Fu et al.) 201T), and directional gradient prior (Ran al. 2020). Generally, these methods based on hand-crafted priors tend to lack generalization ability and impose high computation burdens. With the rapid development of deep learning, various deep neural net- works have been explored for image deraining. [uu et al. (2017) propose the pioncering deep residual network for image deraining. Subsequent methods by Li et al. 20188; Yang & Lu (2019); Ren al. (2019) incorporate recurrent units to model accumulated rain. Li et al. (219)) introduce depth infor- mation remove heavy rain effects. Considering the cooperation between synthetic and world
data. Yasarla et al. (2020) propose a semi-supervised approach. More recently, Ye et al. (2022);
Chen et al. 2022) propose contrastive learning-baa unsupervised approaches. Additionally, Xiao
al. (2022) ; Chen et al. (2023) have designed effective transformer models. Note that majonity of
deraining models are trained on individual datasets, restricting them to ad adverse rain types and back-
ground scenes. Thus, some recent work (Zamir et al. 2021 Mou al., 2022 Zamir et al.. [022] Wang et al.] 2023) leverage multiple datasets to train more robust models. However, simply mixing and training amalgamated datasets overlooks potential discrepancies among them, consequently resulting in suboptimal results. To address this issue, we propose to learn adaptive image deraining models by exploring meaningful rain-/detail-aware representations. The proposed approach helps models efficiently capture both commonalities and discrepancies across multiple datasets.
Image Restoration with Contrastive Learning. Contrastive learning has emerged as an efficient approach for unsupervised representation learning (Chen et al. [2020a]b b) He et al., 2020). The under- lying philosophy is to pull similar examples (positives) together while pushing dissimilar examples (negatives) apart in the latent feature space. Some recent work (Wu et al., 2021; Li et al. 20226 Fu et al. 2023) has employed contrastive learning as auxiliary constraints for model training. Ye al. (2022); Chen al. (2022) propose unsupervised deraining methods by pushing rain and clean components apart. Wang et al. (202T) and [i et al. (2022a) focus on extracting degradation repre- sentations for image resouurenl and all-in-one restoration, respectively. Most recently, [Zheng (et al. (2024) employ contrastive learning to generate realistic rainy images by controlling the amount of rain. These off-the-shelf approaches either learn only degradation representations only discrim- inate between rain and clean background, which cannot perceive coupled rain and backgrounds.
Image Restoration with Model Modulation. He al. (2019) inserts adaptive modulation mod- ules to control image restoration on continuous levels. Fan el al. (2019) proposes to generate image operator parameters with hand-crafted guidances. Recently, Li al. (2022a) introduce deformable convolution into image restoration, where its offset and scales are generated with deep feature guid- ance. Inspired by (Hu al. 202T), we propose a computation-friee context- based modulation mechanism using vector-level guidance.
3 METHODOL.OGY
3.1 PROBLEM DEFINITION
Given mixed multiple rainy datasets D UMM, D:, our objective is to learn the optimal parameters d* for model that minimizes the overall empirical cost across datasets, i.e., arg min, ((: D). Notably, M is the number of datasets and denotes any arbitrary loss function. A straightforward approach, as proposed in recent work (Tiang et al.) 2020 Zamir et a 2021: 2022] Wang al., 2023), to directly train the model on mixed datasets. We argue that such an approach risks neglecting the discrepancies among datasets, e.g., the rain types, the background scenes, and the rain simulation techniques, resulting in suboptimal performance. Moreover, directly training mixed datasets impedes the model's 's ability to assimilate knowledge across datasets. Therefore we propose an adaptive image deraining approach, termed CoIC, to facilitate training models on mixed datasets. An overview of the proposed CoIC is presented in Figure [ (a). In particular, a unified and structural latent embedding space & for 'D is constructed capture rain-/detail-awes properties for all images. Formally. we aim to optimize ((0,E, D). For simplicity, we decompose it by:
E(0, E: D) e (0(E):)) + XP(E:
3

---

Rain Context-based Instance-level
Detall Detall-aeas OAL30
modulation (Col-M)
Searching rain layer bank Add most dissimilar rain
Rain-aware negetiive exemplars
RO
Guidonce
Gaussian blur
Getail aware negative exemp
Modulated model
D
Main-/detail--wee
Instance speciicc Result f(x, 2)
= D
contrastive learning
(a) An overview of the proposed CoIC
(b) Rain--detail-wwart negative exemplars generation
Figure 2: (a) The framework of the proposed CoIC. We extract instance- representations with the help of tain-/detail-wwant contrastive learning strategy. Leveraging these representations as in- structive guidance, we then utilize Col-M to modulate the model's parameters, yielding adaptive deraining results. (b) Generation of rain-/detail-avee negative exemplars.
where the first term is data- data-fidelity term, and the second term indicates an embedding space con- straint. À is hyper-paramett parameter which is heuristically tuned in Appendix A.4
3.2 INSTANCE- LEVEL REPRESENTATION EXTRACTION
Let ((xg,15)) y denote the mixed multiple rainy dataset, where æj and 8/ are the paired rainy input and ground truth, respectively, and N is the total number of pairs. To obtain the rain-/detail- aware representation, we employ encoder comprising feature extractor E, Global Average Pooling (GAP) operation, and a subspace projector e to embed the rainy image E R " x3 into d-dimensional spherical vector Id (see Figure 3 for the details). Applying to x produces feature map F = F(x) which captures rich spatial and channel information related to rain and image details. Considering the property F, we introduce a contrastive learning strategy, detailed in Section 3.3 to learn rain-/detail-awe embedding space. We then project the feature map F into a d-dimensional vector as shown Figure] by
& (GAP(F))
(2)
1 (GAP(F)) 112
where á indicates the subspace projector. As a result, the latent vector encapsulates rain-/detail- aware information, which can be leveraged to guide the adaptive image deraining.
3.3 JOINT RAIN-/DETAIL-WWAN CONTRASTIVE LEARNING
As previously discussed, the embedding space & (see in Figure 2 (a)) characterizes both rain and image details. Concretely, the encoder is required focus more on image details for light rain images while concentrating more on rain for heavy rain images (see in Figure ] (b)). To achieve this, we develop a rain-/detail- aware contrastive learning approach by care- fully designing negative exemplars (see Fig- ure (b)).
BN
Negative Exemplars in Rain-aware Case. Given rainy-clean image pair (x, y), the en- coder should discriminate the rain in æ. To this end, we leverage a rain layer bank noted as DR = T2. ... Tw} to generate nega- tive exemplar follows. we first retrieve DR to obtain a rain layer that is most dissimilar to the rain in which is determined by
Feature extractio E
Subsoace projector &b
Figure 3: The encoder for extracting instance-level representations comprises three components: fea- ture extractor, GAP layer, and subspace projector. LReLU indicates the LeakyReLU activation.
arg max (x - y) -1|11-
(3)
rEDR
Utilizing f. generate negative exemplar + F, which contains the most dissimilar rain to x but preserving the same background. In practice, the rain layer bank is constructed from the data batch, where cross-dataset simulation is facilitated.
4

---

Negative Exemplars in Detail-aware Case. Recently, Wu et al. (2023) has developed efficient contrastive learning-bus based perceptual loss for image super-resolution, where the blurred editions of input image considered as negative exemplars. Motivated by this, the detail-aware information can be exploited through distinguishing between the details in rainy image from the blurred clean image. Specifically, given rainy-clean pair (ar,y), we generate N₂ negative exemplars by blurring the clean image y to obtain (u]13-)
With the negative exemplars i and {u9), we formulate the proposed contrastive learning loss as eons FFFe)
foontr - -log ecoo(F.Fk) ecos(F.F)) + No cus(F.FF)
where F E(x). Fi E(r*), F E()). and F? Ely)) are spatial features extracted by feature extractor E and its momentum-updatte version É (following MoCo (He et al. 2020)) con- taining rich rain-/detail-aware information. cos(,.) denotes channel-wise cosine similarity. Here, x denotes an augmentation of æ. As result, foeatra forms the embedding space constraint term in equation | i.e., the P(E; D). The rain-/detail-att aaware information in F will propagate to the vector 2 via equation which can guide the modulation for both the CNN and Transformer models.
How to Assess the Impact of Rain and Details? As stated previously, the feature extractor E effec- tively characterizes both rain- and detail-related properties, enabling assessment of the model's reliance on rain cues versus background details. Specifically, given a rainy-clean pair (I. y). the dependance on background details can be qualified Çn cos (GAA(E(())) GAP(E(y))) where a higher Çn indicates greater dependence on details. To compute (R for rain effect, we build a clean image bank denoted as Dg and use it to retrieve the most dissimilar background by
= arg max be
|1y
bll1-
(5)
Subsequently, we construct the image x' b+ (z-y) which retains the rain in while incorporating maximally dissimilar background. Formally, <n is defined as cos (GAA(E(Z))) GAP(E(22)) A higher Çn indicates greater reliance on rain. Figure (bb) provides a rain-/detail- awareness analysis utilizing ÇB and Çn-
Discussion. The Gaussian blur may not effectively model the degraded background excessively rainy images where occlusions dominate. However, in such case, the rain-aware information typi- cally dominates, thereby mitigating the limitations of Gaussian blur.
3.4 CONTEXT-BASED INSTANCE-LEVED MODULATION
Leveraging the rain/detail-red reelatdd information encapsulated in from equation we propose Context-based Instance-level Modulation (Col-M) mechanism to modulate each convolutional layer in CNN model and self-attention layer in a Transformer model under the guidance of z.
Modulate CNN-based Model. The convolutional layer is the fundamental building block of CNN models, comprised of C' filters each with dimensions C' x k x k. Notably, C' and C' denote the number of input and output channels, while & refers to the spatial size of the kernel, respectively. Let the filters in the 1-th convolutional layer be represented by W E R RRxx0xxxx Given an input feature FL, the standard convolution operation yields
(6) where denotes the convolutional operation and 6' is bias. We aim modulate the output F++1 by modulating W under the guidance of instance-level embedding 2 and the context in Fl. Inspired by the low-rank adaptation in large language models (Hu et al. 2021). we generate an adaptive weight A IR CR xlek to modulate W Spefically, we first generate three vectors utilizing and the context in Fll following
p++ W + F +
QQ' 2 MLP([2, G))(F")))
where Q' E RC, ne E RC, and Z R (by reshaping) are cropped from the output of MLP, and MLP is a two-layer multi-layer perceptron. Then we generate A' by
12 Re
A 0.,
(8)
a..
Eas 2 ael
5

---

where T/g. 1/sigmodd Q² + Ryy). Note that Tee induce channel-wist temperature The = / Eo, ( 22. min Z 2) of standard Softmax operation in equation 8 (Proof and an in-depth analysis can be found in Appendix A.II). which controls the receptive filed of a l: k convolutional kernel. Moreover, the nonlinearity in equation
S can efficiently increase the rank of A Utilizing the adaptive weight A', we can modulate the convolution operation in equation 6 via
p = (A'W") * F' +b' = F++ + AW' * F*,
where the last term indicates an adaptive alternation to Fi+1 in equation 6 with AW 1)W! Furthermore, if all elements in Z' are equal, we can derive from equation and that AW 0 (T' + oo), implying no modulation under this condition. This modulation approach thereby enables efficient modulation of the CNN models under the guidance of instance-level rep- resentation 2 and the feature context. In practice, equation 9 can be efficiently computed in parallel utilizing group convolution. A PyTorch-style implementation is provided in Appendix A.2
Modulate Transformer-based Architecture. In Transformer-basee models, where the self- attention layer is a core building block, we propose incorporating the instance-level representation to develop rross-attention mechanism as follows:
Q = XW MLP(2),
K, V = (X +c)
(X
+ We)
QK
Y Softmax
V.
(10)
where X and Y denote the input feature and output cross-attention result, respectively. In particular, represents the context generated from representation in equation 2 MLP refers to SiLU acti- vation followed by a fully-connected layer. Cross-attention mechanism has been efficiently adopted to image translation tasks (Rombach et al.) 2022). Owing to the cross-altention formulation in equa- tion I0 the model can efficiently generate adaptive deraining results guided by the instance-level representation z. Similar to equation 9 the modulation of K (or V) in equation I0 can be formu- lated as K- = X XW* K + X LAWK, where XAW* can be equivalently transferred into AX XW The alternation AX- MLP(2) thus equivalent to equation 10.
IO
The proposed Col-M facilitates adaptive deraining in layer-wise modulated manner. Let f far 2) denote the restored image under the guidance of 2. The data fidel-iiylityy loss term equation I can be derived below:
{(((E); D) (( (f(x, z), E span((=)),
(11)
where l(-,) represents arbitrary data- cata-fidelity loss function or a combination of multiple loss func- tions, e.g., MSE or L1 loss, and & denotes the joint http/ddetall-waarr embedding space spanned by all instance-level representations. The in-depth analyses of the embedding space can be found in Section 4.2 and Appendix A.6
4 EXPERIMENTAL RESULTS
4.1 SETTINGS
Synthetic and Real-world Datasets. We conduct extensive experiments utilizing five commonly adopted synthetic datasets: Rain200L. & Rain200H (Yang et al. 2017). Rain800 (Zhang et al., 2019). DID-Data (Zhang & 2018)), and DDN-Data (Fu al. 2017). Rain200L and Rain200H contain light and heavy rain respectively, each with 1800 image pairs for training and 200 for evaluation. Rain800 is generated using Photoshop[] with diverse and accumulated rain types varying in orien- tation, intensity, and density. It has 700 pairs for training and 100 for testing. DID-Data generated utilizing Photoshop comprises three rain density levels, each with 4000/400 pairs for training/test- ing. DDN-Data consists of 12,600 training and 1400 testing pairs with 14 rain augmentations. In total, we amalgamate all 5 training sets comprising 28, larger than current mixed dataset Rain13K (Jiang al. 2020) ). To evaluate the real-world deraining ability, we use the real-world dataset from (Wang et al. 2019) comprising 146 challenging rainy images, which we denote as RealInt.
28,900 900 pairs as the mixed training set (much

---

Table 1: Quantitative comparison of five representative models trained mixed multiple datasets.
The last column demonstrates the real-world deraining quality on RealInt.
Mathoch
PSNR
Rain2001
SSIM
PSNR 7
Ran72111
SSIM 7
PSNR 1
Raistt))
SSIM
PSNR
DID-Dum
5S0M
PSNR
DON-D
SSIM 1
Realler
Sys23Rcal Keaata.sial.
30.83
0.288
0.6788
0.879
0.0033
4.0052
21.44
16:04
04732
0.777
21.28
4,7640
BRN Rie al. [10ts]
35.11
09534
27.65
0,5810
2615
08652
11.52
0.015
$2.40
0.441
4,7008
BRN
CCC (0rs)
3788
2845-00
26133
0.33
3481
32.926333
0.9466
4.946311
RCUNE
Wase
12220
36.73
0,737
0.747
0.9656
0.816
32.85
0.9451
4,5781
16
DGUUNa
Mox dial 5022
39.00
0.9866
31.00
0.9146
30:43
0.9088
35,10
0.9624
33,99
0.0555
4,7040
DCUUNet
19.8
0.966
31477
699552
30 10
35.11m
69027/0000
X3.99
Nowrdt ct plon
1957
GRI15
09912
0.9865
DRStomne: Chen et 2023
29.34
0.9858
30.42
0,9057
0.9114
34,96
0,9647
33.92
0.0541
4.7562
DRSteme
0,8862
W N0.5
29, 10
38
B INC
Table 2: Quantitative results of further trained DRSformer with SPAData.
Methots
PSNR 7
SS00 1
PSNR 7
Rana
SS0M
PSNK
Raistto
SS031
PSKR
DiD-Daaa
SSIM 1
PSNS 1
DDN Data
SSIM 1
PSNK
SPADea
SSIM
DRSEaraan
50.32
R0850
20:37
0,9000
35.85
0:3070
34.91
DJ9607
35.71
0U5540
45.46
0.5666
DRSIara ColC (Ours)
39,70
26.30 UN
8 0,908 mous
29,7 28,73
0.14
36,00
88556 38
46 80.5%
Evaluation Metries. Following previous work (Zamir et al. 2021 2022), adopt two commonly used quantitative metrics for evaluations: Peak Signal-to-Noise Ratio (PSNR) (Huynh-Thie & Ghan- bari 2008) and Structural Similarity Index (SSIM) (Wang et al., 2004). For real-world images, we utilize the Natural Image Quality Evaluator (NIQE) metric (Mittal et al. 2012).
Training Settings. The base channel number in the feature extractor is set to 32. After each down- sampling operation, the channel number is doubled. All LeakyReLU layers in the feature extractor have a negative slope of 0.1. The output dimension of the subspace projector 128, corresponding following (He et al., (020)_ For rain-/detail-awaan contrastive
to the dimension of in equation 2 learning, the number of detail-aware negative exemplars is set to No- = 4 as suggested in (Wu et al.) 2023). The blurred negative exemplars generated using Gaussian blur with sigma uniformly sampled from interval [0.3, The hyper-parameter À balancing the contribution of the contrastive loss in equation is empirically set to 0.2. Experiments are implemented in Py (Paszke al., 2019) on NVIDIA GeForce GTX 3090 GPUs.
4.2
MAIN RESULTS
Comparison on Benchmark Datasets. We first verify the effectiveness of CoIC through train- ing models on a mixture of five synthetic datasets. Specifically, three recent CNN models are selected, including BRN (Ren et al., 2020), RCDNet (Wang et al.] 2020), and DGUNet (Mou)
et al. 2022). along with two Transformer models, IDT (Xiao et al., 2022 and DRSformer (Chen et al.
2023 .. A model complexity analysis, along with training details and training histories are
provided in Appendix A.3 & A.5 respectively. Additionally, two representative semi-supervised
method Syn2Real (Yasarla et al. 2020) and contrastive bamming-based unsupervised method DCD- GAN (Chen et al. 2022) are included for comparsion. The evaluation results are tabulated in Table Due to the complexity of rain and backgrounds, both Ren2Real and DCD-GAN fail with unsatis- factory performances. It can be seen that CoIC has brought significant performance improvements across all synthetic datasets, while also exhibiting superior real-world deraining capabilities. This provides evidence that the proposed CoIC approach can substantially enhance deraining perfor- mance when training models on mixed datasets. A visual comparison is included in Appendix(A.. Note that Mixture of Experts (MoE) modules in DRSformer help it tolerate discrepancies across synthetic datasets, resulting in marginal improvement. However, DRSformer faces significant chal- lenges when incorporating a real-world dataset for further training (please refer to Table 2).
To further investigate the efficacy of the proposed CoIC when training models on individual datasets, experiments are conducted with 10 representative methods: DDN (Fu et al., 2017). DID-
MDN (Zhang & Patel 2018), JORDER-E (Yang et al. 2019), PReNet (Ren et al. 2019). RCD- Net (Wang et al.] 2020). DGCN (Fu et al. 2021). SPDNet (Yi et al.
2021 Restormer (Zamir et al. 2022), Uformer (Wang al. 2022b), and DRSformer (Chen al.
2023 The heavy rain dataset, Rain200H is chosen for benchmarking. Quantitative results are presented in Table where we find
that DRSformer with CoIC has achieved the best PSNR metric compared to other methods, offer-
7

---

diaepphoo Av
(a) Rainy
(b) DRSformer w/o ColC. (c) DRSformer w/o ColC' (d) DRSformer w/ CoIC (adding SPAData)
(e) DRSformer w/ CotC (adding SPAData)
Figure 4: Real-world image deraining comparsion challenging rainy images from Reallnt.
ing 0.09dB PSNR gain over DRSformer. This indicates that the proposed CoIC can also improve deraining performance on individual datasets.
Table 3: Quantitative compari- of image deraining methods
Table 4: Comparison the Reallnt dataset. Restormer! denotes the official pre-trained model.
Table 5: Ablation on the proposed
Col-M modulation strategy, the
on Rain200H dataset.
contrastive learning loss, and the type of negative exemplars (rain- and detail-aware). Values marked with t demonstrate the PSNR im-
DEN or al Mi113]
76
1189155
Values marked with
DID NToX Zhink Sin.SPaaa Papal 2014
26.61
(U8242
specify the NIQE gain.
BCDNET
2000
30.34
[19948
provement against the first row.
SMONe
DOCN
Viat 2011
31.15
31.28
I19125
EU0007
6.273 ou
4.5955
Founse warherrrayan Rawhel
Restome Frani
No
48
31.40
DRSterae
1007
Riin
11.1
DRSteema- - Culc. (Oun)
32.36
IU0227
4,005
Bali- Daal--aaalaaaa
4822
314800
31.5
Real-world Deraining Transferred from Synthetic Datasets. Our subsequent analysis examines the real-world deraining capabilities by training models on an individual dataset (e.g., Rain800), directly mixed multiple datasets, and mixed multiple datasets employing the pro- posed CoIC. We select DGUNet and IDT baselines. Additionally, we include the official pre-trained Restormer") denoted Restormer for comparison. Table 4 reports the evaluation comparisons. Both DGUNet and IDT trained solely on the Rain800 dataset demonstrates the worst performance. Training on the mixed mul- tiple datasets has improved generalization for both DGUNet and IDT. Notably, IDT surpasses mixed datasets.
(4) UIMP couuliation IDT (b) Average intra-intaa-baatte similtattiis
Figure 5: (a) UMAP visualization of learned rep- resentations. (b) Average intra-/interddataset sim- ilarities. "L", "H", and "A" indicate light, heavy, and accumulated rain, respectively.
Restormert when trained Equipped with the proposed CoIC, DGUNet and IDT achieves the best real-world derain- ing quality, validating the superiority of our proposed method. Visual results are provided in Ap- pendix A.8
Further Training Incorporating Real-world Dataset. To fully explore the potential of the pro- posed CoIC, we add real-world dataset SPAData (Wang et al. 2019) further train DRStormer
https:///itth/n com/ 30/ /ee//ettttrree
8

---

obtained using mixed synthetic datasets. The SPAData contains 28,500/1000 image pairs for train- ing and evaluation. Surprisingly, we observe that DRSformer with CoIC has obtained remarkable real-world deraining ability shown in Figure (see more high-quality results in Appendix [A.9]. On the other hand, DRSformer with CoIC has significantly outperformed the direct training edition, as can be seen in 2 This suggests that CoIC can help learn comprehensive deraining model adept at handling both synthetic and real rains. We also conduct comparison by training DRSformer on five synthetic datasets and SPAData from scratch Appendix A.10
Joint Rain-/Detail-aware Representation Visualization. We employ UMAP (MeInnes et al., 2018) to project all instance-level representations onto two-dimensional spherical surface. For simplicity, the longitude and latitude are plotted in (a) The IDT trained on mixed five syn- thetic datasets selected for visualization, with 400 examples randomly sampled from each dataset: Rain2001., Rain200H, Rain800, DID_1, DDD,,,DDD DID_3, DDN_4, and DDN_12 (details of datasets are in Ap- pendix AA.6). Additionally, all 146 examples from Realint and 400 clean images in Rain2001, are included. The results in 5 demonstrate that rainy images from different datasets may share similar embeddings. We further plot the intra-finter-datase similarities in Figure (b) using these embeddings. More in-depth analyses are provided in the Appendix A.6
4.3 ABLATION STUDY
Effectiveness of the CoI-M. We first study the efficacy of the proposed Col-M. we train DGUNet on the mixed dataset of Rain200L, Rain200H, and Rain800. Quantitative results are tabulated in Table 5 Typically, DGUNet with Col-M has improved PSNR by 0.06dB on the Rain800 dataset, indicating performance gains from learning adaptive deraining. Furthermore, a real-world com- parison presented in Figure 6 suggest that DGUNet without Col- M tends to overlook the real rain (Figure 6 (a)). In contrast, DGUNet with Col-M readily perceives the real rain and effi- ciently eliminates the rain streaks (Figure (b)).
Effectiveness of the Rain-Dettail-aware Negative Exemplars. We train DGUNet on the mixed dataset Rain2001., Rain200H, and Rain800 to analyze the impact of rain-/detail-aware negative exemplars. Quantitative results are presented in Ta Incorpo- rating detail-aware and rain-aware negative exemplars consecu- tively enhances the deraining performances. With both exemplar types, CoIC provides considerable PSNR gain of 0.23dB the Rain800 dataset comprising complex rain. However, rain-aware negative exemplars alone brings no improvement on the Rain800 dataset. This occurs because the encoder finds a trivial solution of Coomr (see in equation 4 by only discriminating heavy rain from other rain, owing to the arg max in equation 3 Figure 6 (c) & (d) verify this explanation by visualizing the embedding space. In summary, rain-/detail-ave -ware aware negative exemplars mutually reinforce the learning of instance-level representations, hence significantly improving performance.
TO TER-OEET
Figure 6: (a) & (b) A real- world deraining result w/o and w/ Col-M. (c) & (d) UMAP vi- sualization of learned embed- ding space.
CONCLUSION AND FUTURE WORK
Large image deraining models have sprung up recently pursuing superior performance. However, the potential of datasets remains nearly untapped. We find that inherent discrepancies among datasets may sub-optimize models, and shrinks their capabilities. To address this, we develop a novel and effective CoIC method for adaptive deraining. We first propose a novel contrastive learning strategy to explore joint nann/detail-awan representations. Leveraging these representations as instructive guidance, we introduce Col-M to perform layer-wise modulation of CNNs and Transformers. Exper- imental results on synthetic and real-world datasets demonstrate the superiority CoIC. Moreover, CoIC enables exploring relationships across datasets, and model's behaviors. Furthermore, superior real-world deraining performances with CoIC are observed when further incorporating real-world dataset to train model. In the future, we anticipate extending CoIC to learn more practical derain- ing models that could handle diverse rains coupled with fog, dim light, blur, noise, and color shift. Moreover, expanding CoIC to all-in-one image restoration task is also promising.
9

---

ACKNOWLEDGEMENT
This work was supported by Key Area Support Plan of Guangdong Province for Jihua Laboratory (X19005ITB190).
REFERENCES
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In Proceedings of the International Conference on Machine Learning. pp. 1597-1607. PMLR, 2020a.
Xiang Chen, Jinshan Pan, Kui Jiang, Yufeng Li, Yufeng Huang, Caihua Kong, Longgang Dai, and Zhentao Fan. Unpaired deep imgge deraining using dual contrastive learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2017-2026. 2022.
Xiang Chen, Hao Li, Mingqiang Li, and Jinshan Pan. Learning a sparse transformer network for ef- fective image deraining. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5896-5905, 2023.
Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved baselines with momentum contrastive learning. arXiv preprint arXiv:2003.04297 2020b.
Qingnan Fan, Dongdong Chen, Lu Yuan, Gang Hua, Nenghai Yu, and Baoquan Chen. A general decoupled learning framework for parameterized image operators. IEEE Transactions on Pattern Analysis and Machine Intelligence, 43(1):33 47, 2019.
Huiyuan Fu, Wenkai Zheng, Xiangyu Meng, Xin Wang, Chuanming Wang, and Huadong Ma. You do not need additional priors or regularizers in retinex-based low-light image enhancement. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 18125- 18134, 2023.
Xueyang Fu, Jiabin Huang, Delu Zeng, Yue Huang, Xinghao Ding, and John Paisley. Removing rain from single images via deep detail network. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3855-3863, 2017.
Xueyang Fu, Qi Qi, Zheng-Jun Zha, Yurui Zhu, and Xinghao Ding. Rain streak removal via dual graph convolutional network. In Proceedings of the AAAI Conference Artificial Intelligence, volume 35, pp. 1352-1360, 2021.
Yu-Hsiang Fu, Li-Wei Kang, Chia-Wen Lin, and Chiou-Ting Hsu. Single-frame-based rain removal via image decomposition. IEEE International Conference on Acoustics, Speech and SP, pp. 2011.
Kshitiz Garg and Shree Nayar. Photorealistic rendering of rain streaks. ACM Transactions on Graphics, 1000) 2006.
Shuhang Gu, Deyu Meng, Wangmeng Zuo, and Lei Zhang. Joint convolutional analysis and synthe- sis sparse representation for single image layer separation. In Proceedings of the IEEE Interna- tional Conference on Computer Vision, pp. 1708- 1716, 2017.
Jingwen He, Chao Dong, and Yu Qiao. Modulating image restoration with continual levels via adaptive feature modification layers. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 11056-11064, 2019.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings the IEEE Conference Computer Vision and Pattern Recognition, pp. 9729-9738, 2020.
Edward J Hu, Yelong Shen. Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2 106.2/00.096655 2021.
10

---

Quan Huynh-Thu and Mohammed Ghanbari. Scope of validity of psnr in image/video quality 2008.
sessment. Electronics Letters, 44(13):800-80) 2008.
Kui Jiang, Zhongyuan Wang, Peng Yi, Chen Chen, Baojin Huang, Yimin Luo, Jiayi Ma, and Junjun Jiang. Multi-scale progressive fusion network for single image deraining. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 8346-8355, 2020.
Boyun Li, Xiao Liu, Peng Hu, Zhongqin Wu, Jiancheng Lv, and Xi Peng. All-in-one image restora- tion for unknown corruption. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 17452-17462, 2022a.
Fei Li, Lingfeng Shen, Yang Mi, and Zhenbo Li. Drenet: Dynamic image restoration contrastive network. In Proceedings of the European Conference on Computer Vision, pp. 514-532. Springer, 2022b.
Ruoteng Li, Loong-Fah Cheong, and Robby T Tan. Heavy rain image restoration: Integrating physics model and conditional adversarial learning. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 1633-1642, 2019.
Xia Li, Jianlong Wu, Zhouchen Lin, Hong Liu, and Hongbin Zha. Recurrent squeeze-and-eecitation context aggregation net for single image deraining. In Proceedingy of the European Conference on Computer Vision, pp. 254-269, 2018.
Yu Li, Robby T Tan, Xiaojie Guo, Jiangbo Lu, and Michael S Brown. Rain streak removal using layer priors. In Proceedings of the IEFE Conference on Computer Vision and Pattern Recognition, pp. 9736-2744, 2016.
David Martin, Charless Fowlkes, Doron Tal, and Jitendra Malik. A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics. In Proceedings of the IEEE International Conference on Computer Vision, pp. 416-423. IEEE, 2001.
Leland McInnes, John Healy, and James Melville. Umap: Uniform manifold approximation and projection for dimension reduction. arXiv preprint arXiv:1802.03422 2018.
Anish Mittal, Rajiv Soundararajan, and Alan C Bovik. Making "completely blind" image quality analyzer. IEEE Signal Processing Letters, 20(3):20 2012.
Chong Mou, Qian Wang, and Jian Zhang. Deep generalized unfolding networks for image restora- tion. In Proceedings of the IEEE Conference Computer Vision and Puttern Recognition, pp- 17399-17410, 2022.
Ozan Özdenizei and Robert Legenstein. Restoring vision in adverse weather conditions with patch- based denoising diffusion models. IEEE Transactions on Pattern Analysis and Machine Intelli- gence, 2023.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, high-peeformance deep learning library. In Proceedings of the Annual Conference on Neural Information Processing Systems, volume 32, 2019.
Pytorch: imperative style,
Wu Ran, Youzhao Yang, and Hong Lu. Single image rain removal boosting via directional gradient. In Proceedings of the IEEE International Conference on Multimedia and Expo, pp. 1-6, 2020.
Dongwei Ren, Wangmeng Zuo, Qinghua Hu, Pengfei Zhu, and Deyu Meng. Progressive image deraining networks: A better and simpler baseline. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition, pp. 3937-3946, 2019.
Dongwei Ren, Wei Shang, Pengfei Zhu, Qinghua Hu, Deyu Meng, and Wangmeng Zuo. Single image deraining using bilateral recurrent network. IEEE Transactions on Image Processing, 29: 6852-6863, 2020.
11

---

Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE Conference on Computer Viston and Pattern Recognition, pp. 10684-10695, 2022.
Gerald Schaefer and Michal Stich. Ucid: An uncompressed color image database. In Storage and Retrieval Methods and Applications for Multimedia, volume 5307, pp. 472-480. SPIE, 2003.
Cong Wang, Jinshan Pan, and Xiao-Ming Wu. Online-updated high-order collaborative networks for single image deraining. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pp. 2406-2413, 2022a.
Hong Wang, Qi Xie, Qian Zhao, and Deyu Meng. A model-driven deep neural network for single image rain removal. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1003-1122, 2020.
Longguang Wang, Yingqian Wang, Xiaoyu Dong, Qingyu Xu, Jungang Yang, Wei An, and Yulan Guo. Unsupervised degradation representation learning for blind super-resolution. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 10581-10590, 2021.
Tianyu Wang, Xin Yang, Ke Xu, Shaozhe Chen, Qiang Zhang, and Rynson WH Lau. Spatial attentive single-image deraining with a high quality real rain dataset. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 12270-12279, 2019.
Yinglong Wang, Chao Ma, and Jianzhuang Liu, Smartassign: Learning a smart knowledge assign- ment strategy for deraining and desnowing. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition, pp. 3677-3686, 2023.
Zhendong Wang, Xiaodong Cun, Jianmin Bao, Wengang Zhou, Jianzhuang Liu, and Houqiang Li. Uformer: A general u-shaped transformer for image restoration. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 17683-17693, 2022b.
Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli. Image quality assessment: From error visibility to structural similarity. IEFE Transactions on Image Processing, 13(4): 600-612, 2004.
Gang Wu, Junjun Jiang, and Xianming Liu. practical contrastive learning framework for single- image super-resolution. IEEE Transactions on Neural Networks and Learning Systems, 2023.
Haiyan Wu, Yanyun Qu, Shaohui Lin, Jian Zhou, Ruizhi Qiao, Zhizhong Zhang, Yuan Xie, and Lizhuang Ma. Contrastive learning for compact single image dehazing. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 10551-10560, 2021.
Jie Xiao, Xueyang Fu, Aiping Liu, Feng Wu, and Zheng-Jun Zha Image de-raining transformer. IEEE Transactions Pattern Analysis and Machine Intelligence, 2022.
Wenhan Yang. Robby T Tan, Jiashi Feng. Jiaying Liu, Zongming Guo, and Shuicheng Yan. Deep joint rain detection and removal from single image. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1357-1366, 2017.
Wenhan Yang, Robby T Tan, Jiashi Feng, Zongming Guo, Shuicheng Yan, and Jiaying Liu. Joint rain detection and removal from single image with contextualized deep networks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 42(6))1377-1393, 2019.
Youzhao Yang and Hong Lu. Single image deraining via recurrent hierarchy enhancement network. In ACM International Conference on Multimedia, pp. 18 1814-1822, 2019.
Rajeev Yasarla, Vishwanath A Sindagi, and Vishal M Patel. Syn2real transfer learning for image deraining using gaussian processes, In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2726-2736, 2020.
Yuntong Ye, Changfeng Yu, Yi Chang, Lin Zhu, Xi-le Zhao, Luxin Yan, and Yonghong Tian. Unsu- pervised deraining: Where contrastive learning meets self-similarity. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5821-5830, 2022.
12

---

Qiaosi Yi, Juncheng Li, Qinyan Dai, Faming Fang, Guixu Zhang, and Tieyong Zeng. Structure- preserving deraining with residue channel prior guidance. In Proceedings of the IEEE Interna- tional Conference on Computer Vision, pp. 4238 02884424,240 2021.
Syed Waqas Zamir, Aditya Arora, Salman Khan, Munawar Hayat, Fahad Shahbaz Khan, Ming- Hsuan Yang, and Ling Shao. Multi-stage progressive image restoration. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 14821-14831, 2021.
Syed Waqas Zamir, Aditya Arora, Salman Khan, Munawar Hayat, Fahad Shahbaz Khan, and Ming- Hsuan Yang. Restormer: Efficient transformer for high-resolution image restoration. In Pro- ceedings of the IEEE Conference on Computer Vision and Patten Recognition, pp. 5728-5739, 2022.
He Zhang and Vishal M Patel. Density-awe aware single image de- de-raining using a multi-stream dense network. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 695-704, 2018.
He Zhang, Vishwanath Sindagi, and Vishal M Patel. Image de-raining using conditional generative adversarial network. IEEE Transactions on Circuits and Systems for Video Technology, 30(11): 3943-3956, 2019.
Shen Zheng, Changjie Lu, and Srinivasa G Narasimhan. Tpsence: Towards artifact-free realistic rain generation for deraining and object detection in rain. In Proceedings of the IEEE Winter Conference on Applications of Computer Vision, pp. 5394-5403 2024.
13

---

A APPENDIX
A.1 DERIVATION OF THE CHANNEL- WISE TEMPERATURE
Please recall that the proposed Col-M generates adaptive weight A c, following equation 8 which is re-written below:
A ,a,3 Edeg e Bilai
(12)
For Softmax operation on any give score s 181,81:"" sn) and positive scalar temperature t, it outputs
Softmax(s, 1); = 8i/t
Σ, E
(13)
Generally, the score s is obtained by shared classification head for all samples. However, the proposed Col-M equips different convolutional layers with different MLPs that output both , and Tee' hence the amplitude of Z in equation 12 also influences the Softmax result. To compare the generated temperatures for all convolutional layers, proper normalization of Z 2,, is indispensable. Specifically, let Zz, = Zu. - min Za.e, where Z 2 Additionally, denote Za.s = n'r 0.9
where Ea,9 r a.g = 1. We can re-formulate equation 12 following equation 13 to
A .,,8 Eas gy " ar Softmax Ted /1²),
(14)
where 12d is normalization of z conntraiied by Ea, I af 1. Note that yt can be calculated by
1e' En,g Zu.s En.
E(Z,, min Z Zg,9).
(15)
D a,8
a,3
a
Therefore the induced temperature is Tool Ea, ( min (,,0). Equivalently, we choose the temperature to be the form of The = Tee Eo,6 (Z) operation Ea,e a to be averaging operation. In this way, equation 14 becomes
min Zz,g). where we let the summation
A A 3 = 1 2
Ted hx)) = 2 Softmax
(16)
How does CoIC Modulate the Model? Formally, large temperature T! pro- duces an approximately uniform distribution of Softmax operation. According to equa- tion 16 all elements in A nearly become 1, demonstrating no modulation (all spa- tial weights of convolutional kernel are important). the generated A! will collapse to spatial coordinates with large n a..9 demonstrating the convolution layer is modu- lated to focus on local regions with shrinked receptive field (spatial weights of kernel focusing on informative neigh- bors important). A shrinked receptive field of ker- nel (i.e., small temperature T(d:) is observed in the deep en- coder layers where skip connection is employed, as well the deep decoder layers which produce high quality restored images (Please refer to Figure 7.. We conjecture that the non-modulated deep encoder layers exhibit large receptive field due downsampling and stacked convolutional lay- ers. However, the skip connection from encoder to decoder for reconstructing details restricts that the features passed from encoder should capture rich image details without blur, hence the encoder is required to focus on informative local regions, corresponding to low temperatures. Conversely, the features passed to deep decoder layers contain rich details and possess high resolution, therefore small temperatures are required to focus on local regions to avoid blurry result.
Conversely, when T is small,
enzeddrr
dassdder (stags
eocoder (stage
layer
(stgge
(slage
Figure 7: Average layer-wise log T with 95% confidence interval of DGUNet.
14

---

Algorithm Col-M for CNN layer, PyTorch-like
import torch. nn functional as 2 import torch. nn as nn Customize module to modulate CNN layers
class ColM_Conv2d (nn. (nn.duleee))
def _wnit_nn_______ (self, in_e, out_e, k_size, stride, pad, grps, bias) gEps: number of conv groups
self.bias = bias
self.stride stride self.padding pad
10
self.groups grps
11
self.conv Cin cut_c, k_size, stride,
12
pad, groups-gps, bias-bias)
def forward adaptive_w):
x: shape (b, C, w)
15
adaptive_w indicates generated weights using context in and rain-/detall-wware representation 2
16
if adaptive_w is None:
return self.conv(1 else:
19
# adaptive_M, shape (b, cut_e, in_c // grps, k_aize, k_size) b, c, h, w x.shape it self.bias:
20
bias self. colf.ccov.aee wonv. bias. www.waterreppatttia (b) repeat bias
23
medulate conv weight using adaptive_W
24
weight self. conv. wow..ooovii..i.... www.gh...squueee..............ee out_c, in_e // grps, ksize, k_size)
(0) + adaptive_w shape: (b,
x.view (1, -1, h, w) # absorb batch_size dimension to channel dimension, shape (1, bein_c, h, w)
26
weight neight.view absorb batch_size to out_e dimension, shape: (bxout_e, in // grps, sizee, k_size)
in_c // self. k_size, k_size) #
out F. (x weight=weight, bias=bias, stride=self.sttie padding-sel
cropus-bxsel..groupe # shape:
(1, brout_e, h', w() return out.view (b, c, out shape [2], out shape (3])
29
To verify conjecture, here select the U-Net like DGUNet elucidate how the receptive field is layer-wisely modulated given different inputs. Figure 7 plots layer-wise log T across the Rain200L, Rain200H, and Rain800 datasets. Deep encoder layers (with skip connections) and deep decoder layers exhibit small temperatures, concentrating more on informative local regions for detail restoration. Additionally, the encoder's bottleneck layer has large temperature, enabling rich deep feature extraction. Given different inputs, differences manifesl primarily in the decoder, demon- strating that effective modulation occurs when features from encoders and decoders are interacted at different scales. These results have substantiated the above conjecture.
A.2 PYTORCH-LIKE PSEUDO CODE FOR EQUATION
In Section 3.4 we propose context-bane based instance-level modulation mechanism to modulate the weights of all CNN layers in an efficient way. Typically, we follow equation 9 which we re-write below:
pl+1 = (A'w') F + F++1 AW F!.
(17)
However, the generated adaptive weight instance-relate which requires a for-loop to calculate equation 17 for data batch. The for-loop implementation will cause high computation cost and sub- sequently increase the training time. Therefore, we introduce Algorithm T to compute equation 17 in parallel with the help of group convolution operation.
15

---

(0) "sorain-Is2 pag" (Rain2008.)
(b) "sorain-Txx2 pag" (Rain2008)
(c) pung (Rain500)
1d0"i6jpe"" (DID) 1)
Gc) 110752 jps" (DID 31
( 4.P (DDN 4)
(6) 149 12.jPg' (DDN 12)
(b) "rain 051.png" (Reallnt) "ratn=1 peggp"""""""""""" (Reallett () "nomins gon" (Clenn)
Figure 8: Visual examples of images from different datasets for instance-leve representation visu- alization. Images marked with orange box contain similar light rain patterns. Images marked with green box are characterized by similar accumulated rain. Be aware that both (h) (i) are real-world rainy images.
Table 6: Model complexity analysis for BRN, RCDNet, DGUNet, IDT, and DRSformer in terms of #P, FLOPS, and testing time. #P indicates the number of parameters. #AP denotes the increased parameters with CoIC.
Methods
Input size
#P (M)
w/ ColC
FLOPs (G)
FLOPs w/ ColC (G)
Testing time (ms)
Testing time ColC (ms)
BRN
512 512
0.38
0,21
392.9
391.3
333,24 38.1
364.2:t 33,0
RCDNet
389.5
DGUNet
512 512
3.63
1.62
396.8
397.2
161 0+-5.5
198.8-66.8
IDT
128 128
16.42
2,53
7.3
7.6
59.8+22.7
83,2=1103
DRSformer
33.67
833.6+0..5
A.3 MODEL COMPLEXITY ANALYSIS
In this section, we provide comprehensive model complexity comparison between selected BRN, RCDNet, DGUNet, RCDNet, and DRSformer baselines and them equipped with the proposed CoIC. We evaluate the complexities in terms of the number of parameters (#P), FLOPS, and testing time. The results are provided in Table 6 Note that IDT can only accept input of size 128 128 due to spatial window-based self-attention mechanism. The increased parameters brought by CoIC depend on the intrinsic model architectures. By comparing the performance in Table [ for IDT, we can conclude that the performance gain is not from the increased parameters. Additionally, the increased testing time comes mainly from the encoding process (about 30 ms), which demonstrates that the modulation process brings nearly no extra inference burden.
A.4 BALANCE BETWEEN DATA-FIDELITY LOss AND CONTRASTIVE Loss
The hyperparameter À balances the contrastive loss con- tribution in equation D We train DGUNet on the mixed dataset of Rain200L, Rain200H, and Rain800 with X [0.0, 0.05, 0.1, 0.2, 0.8) to examine its impact. Figure displays the results. Notably, the contrastive loss slight im- proves the performance on Rain200H and Rain200L, which characterized by homogenous rain. In contrast, CoIC with contrastive loss stably improves Rain800 performance, demonstrates its superiority for complex and accumulated rain. We heuristically choose the best X = 0.2 for our ex- periments.
40.5
Figure 9: Ablation on the hyperpa- rameter X.
16

---

Table 7: Quantitative results of DRSformer trained mixed synthetic datasets with different patch
sizes,
Methods
Purch size
PSNR
Rain2001
SSIM
PSNR
Rain20UH
SSIM 1
PSNR
Rain800
SSIM
PSNR
DID-Data
SSIM
PSNR
DDN--Daa
SSIM
DRSfomeer
x
0,9858
34.96
33.92
0.9541
DRSfomeer e ColC (Ours)
96 x 96
39.81
0,9862
30.50
0.9076
29.92
0.9115
35.01
0.9614
33.94
0.9545
DRSfomer
128 x 128
39.97
0,9868
30.78
0.9112
30.09
0.9114
35.13
0.9624
54.03
0.9556
DRSfomer ColC (Ours)
0,9869
0,9118
0.9159
35.17
0.9629
34.11
0.9567
A.5 TRAINING DETAILS AND HISTORIES OF ALL MODELS
In this section, we provide training details for BRN, RCDNet, DGUNet, IDT, and DRSformer. Specifically, both BRN w/o and w/ CoIC trained on 100 100 image patches, with a batch size of 12 for about 260k iterations. We train RCDNet w/o and w/ CoIC on image patches of size 64 x 64 with batch size 16 for about 260k iterations. For the large DGUNet, we train it w/o and w/ ColC on image patches of size 128 128 with batch size 16 for about 400k iterations until converge. The Transformer model IDT w/o and w/ CoIC are trained on image patches of size 128 x 128 with batch size 8 for about 300k iterations. We train DRSformer on mixed synthetic datasets on 96 x 96 image patches with batch size 4. A 96 x 96 image patch setting enables DRSformer to fit on a single NVIDIA 3090 24G GPU. However, since DRSformer utilizes spatial sparse attention mechanism, small image patch size may degrade the performances. Therefore, we next train DRSformer w/o and w/ ColC on 128 x 128 image patches, which further improves the performances five synthetic datasets (Please refer to Table 7
For the purpose of further training DRSformer pre-trained on mixed synthetic datasets with real- world SPAData. We continue to train both DRSformer w/o and w/ CoIC on mixed synthetic and SPAData for about another 105k iterations. Figure 10 (a)-(e) display the training data-fidelity loss curves for BRN, RCDNet, DGUNet, IDT, and DRSformer trained on mixed synthetic datasets. Fig- ure 10 (f) shows the further training loss history of DRSformer after incorporating real-worke SPA- Data dataset. The comparison in Figure (f) demonstrates that the proposed ColC can help DRS- former learn better when exposed to both synthetic and real-world data.
4.50
1000
3,30
-0135
Trainina US5
Training
-080
2.0
0690
032
3.0
452
600
BRN
(b) RCDNet
(c) DGUNet
13133
-324
2.3139
Training
Training
Training
2313
3310e
-350
/ 600
R 600
(d) IDT
(c) DRSformer
(f) DRSformer further training
Figure 10: Training loss curves for BRN, RCDNet, DGUNet, IDT, and DRSformer.
A.6 ANALYSTS ON JOINT RAIN-/DETATI-AWARE REPRESENTATIONS
We present the dataset details for the visualization of instance-level representations in Section 4.2 As stated in Section 4.2 400 examples from Rain2001,, Rain200H, Rain800, DID 1, DID 3, DDN 4,
17

---

BRN
RCDNet
(c) DGUNet
(d) IDT
(c) DRSformer
Figure 11: Top: Joint rain-ddetail-wwan representations visualization for different models. Bottom: Average intra- and inter-dataset similarities learned of different models utilizing the proposed CoIC. "L" "H", and "A" represent light, heavy, and accumulate rain, respectively.
DDN. 12 are randomly sampled for visualization. Additionally, all 146 real-world rainy images from RealInt and 400 clean images (noted as Clean) from Rain200L. are included. Recall that DID-Data comprises three density levels (i.e., low, medium, and high), hence we denote the first (smallest) density level as DID_1, while the third (largest) density level as DID.3. In other words, the images DID_1 and DID_3 are characterized with light rain and heavy rain, respectively. As for the DDN- Data which contains 14 kinds of rain augmentation, (DDN_4) and twelfth augmentation (DDN_12) with heavy rain for visualization. We also present visual example for these datasets in Figure g where their rain patterns be well perceived visually. We can easily observe that the real-world rainy image in Figure g (h) comprises light rain pattern similar to synthetic images in Figure g (a) (from Rain200L) and Figure 8 (f) (from DDN 4). Additionally, the real-world rainy image in Figure 8 (i) contains similar accunulated rain synthetic rainy images in Figure (c) (from Rain800) and Figure g (d) (from DID_I). These observations suggest that training models on mixed multiple datasets can improve its generalization ability on diverse real-world rainy images.
select its fourth augmentation with light rain
With images from Rain200L, Rain200H, Rain800, DID I, DID 3, DDN 4, DDN 12, RealInt, and Clean, we can employ the pre-trained feature extractor to obtain their instance-level representa- tions. We project these nigh-dimensional representations onto a two-dimensional spherical surface. For better visualization, we plot the longitude and latitude of these projected representations, Fur- ther, these representations enable us to calculate the intra- and inter-dataset average similarities, which turn reveal the relationships among datasets. Specifically, BRN, RCDNet, DGUNet, IDT, and DRSformer trained mixed five synthetic datasets (Rain200L & H. Rain800, DID-Data and DDN- DDN-Data) utilizing the proposed CoIC are selected. The visualization results are displayed in Fig- ure T for all these models. It can be that the learned embedding space of all five models neither density-level nor dataset-leve discriminative. This is due to the fact that rainy images from two different datasets may contain similar rain pattern (see Figure g (a) vs. Figure g (f), and Fig- ure 8 (c) vs. Figure g (e)). Therefore, the rain-aware property of instance-level representations may not contribute to dataset-level discriminative embedding space. On the other hand, the ground truths of different datasets may come from the same dataset, e.g., BSD MMrrtin et al. 2001) and UCID (Schaefer& Stich, 2003) dataset, hence the detail-awar property of instance-level represen- Iations may not result in well-separated dataset-level or density-level embedding space.
The learned average intra-/inter-dataset similarities are provided in Figure [1 finter-dataset similarities learned by different models varies, demonstrating that pre-defined crite- rion to discriminate rain and details cannot be applied to all models. For instance, both as transform- ers, IDT tends to be more sensitive to the differences between light and heavy rain than DRSformer as shown in Figure II (d) & (e). Furthermore, the negative similarity between two datasets (e.g., Rain200L and Rain200H for IDT in Figure IT (d)) may imply dataset collision or dataset com- petition when directly training models on these datasets. Our proposed CoIC method encourages different models learn decent joint rain-/detail-awan embedding spaces.
The average intra-
18

---

Rain800
Rain200H
Rainy
(b) RCDNet (mix)
(c) RCDNEr ColC
H1YT (mix)
(e) IDT +
(1) Clean
Figure 12: Qualitative comparison for RCDNet and IDT. Please zoom in for more details.
A.7 VISUAL DERAINING COMPARISON ON SYNTHETIC DATASETS
We conduct visual deraining comparison synthetic datasets in this section demonstrate the effectiveness of the proposed CoIC method. Concretely, two examples from Rain800 dataset and four rainy images from the heavy rain Rain200H dataset are selected. We utilize pre-trained RCDNet and IDT on mixed five synthetic datasets for qualitative evaluation. Figure displays the result. be seen from Figure 12 that both RCDNet and IDT employing the proposed CoIC can better eliminate rain effect as well as recover details, owing to the joint rain-/detail-awe -awar guidance.
A.8 MORE REAL-WORLD DERAINING RESULTS BY USING SYNTHETIC DATA
In this section, we provide more real-world deraining examples to examine the generalization abili- ties of models trained on individual dataset (e.g., Rain800), directly on mixed multiple datasets, and on mixed datasets employing the proposed ColC. Typically, we select the representative CNN model DGUNet, along with the Transformer model IDT as baseline models. Additionally, the official pow- erful pre-trained Restormer, noted Restormer', is included for comparison. The qualitative re- sults are presented in Figure 13, where images contaminated with various kinds of rain are included. Models (DGUNet and IDTT) trained on an individual dataset tend to overlook rain streaks with high pixel intensities or result in over-smooth results, demonstrating poor perceptual abilities on rain and image details. In contrast, models trained directly on multiple datasets can well handle complex rain occasions. However, these models may fail to strike good balance between removing rain and preserving details (see the first, third, and fourth row in Figure 13 (g)). On the contrary, models trained utilizing CoIC can well balance rain removal and detail restoration (see the first, third, and fourth row in Figure B (h)), owing to the exploration of joint rain-/detail-aware representations.
These results have verified the superiority of the proposed CoIC method.
19

---

(a) Raisy
(b) Restormet
(c) EXGUNNe twie Mix)
(d) DGUNer (Mix) (e) DGUNet ColC
(D: Ewoo Mao
(e) IDT (mik)
(h) IDT Colc
Figure 13: More real-world deraining comparison results for DGUNet and IDT. Please zoom in for more details.
Table 8: Quantitative results of DRSformer w/o and w/ ColC trained on mixed synthetic and real- world datasets.
Methods
Training
PSNK 1 SSIM
Rain2000
PSNR
Ratz200H
SSEM
PSNR 7
Rain800
SSIM
PSNR
DID Datz
SSIM
PSNR 1
DON Dat
SSIM "
PSNK 1
SPADaO
SSEM
DRStommer
DRStomer 1 ColC. (Ours)
twor-ttage
39.70
29.32
0,9850
0.9860
29:27
36.31
0,9000
0,9058
23.85
29.73
0.9070
0.9143
34.91
35.82
1.9601
0.9618
33.94
33:71
0.8540
0.9556
46.03
45.45
0,5896
0,9903
DRStomee
DRStomer
(Ours)
from vonadok
A.9
MORE REAL-WORLD VISUAL DERAINING RESULTS BY TRAINING INCORPORATING SPADATA
Here provided more visual real-world deraining results by further training DRSformer w/o and w/ the proposed CoIC incorporating real-world SPAData dataset. The results are displayed in Figure 14 where we can observe that DRSformer w/ CoIC can achieve much better deraining performances.
A. TRAINING ON SYNTHETIC AND REAL-WORLD DATASETS FROM SCRATCH
In Section we have explored the potential of the proposed CoIC by incorporating real-world dataset SPAData and further training DRSformer. This two-stage training approach allows the pre- trained DRSformer to first assimilate deraining knowledge from five synthetic datasets, thereby facilitating its learning process upon the addition of real-world SPAData. Therefore, we further con- duct experiments on training DRSformer utilizing both five synthetic datasets and real-world SPA- Data from scratch, where the larger discrepancies among datasets may impede the learning process. In practice, we train DRSformer w/o and w/ the proposed ColC for about 405K iterations, which equals to the total training iterations the two-stage approach. Quantitative results are provided Table 3 where DRSformer with the proposed CoIC has outperformed vanilla DRSformer overall in six datasets, demonstrating the effectiveness of the devised method. Surprisingly, compared to the two-stage training approach, DRSformer trained from scratch not shown superior performance among Rain2001. Rain200H, Rain800, DID-Data, and DDN-Data. Consequently, the improvement brought by the proposed CoIC shrinks. However, DRSformer training from scratch has exhibited remarkable deraining ability on the real-world dataset SPAData, achieving 46.52dB (w/o CoIC) and 46.76dB (w/ CoIC) in terms of PSNR metric. We conjecture that this counter-intune intuiiive result is due the extremely imbalanced data scale and small batch size. The official SPAData contains 28,500 training image pairs with high resolution, resulting in a training dataset containing 638,473 examples. When training DRSformer using five synthetic datasets and SPAData from scratch with
20

---

(n) Rainy
(b) DRSformer w/o CoIC
(c) DRSformer w/o CoIC (d) DRSformmr CotC: further trained adding SPA
(c) DRSformer ColC further trained adding SPAData
Figure 14: More real-world deraining comparison results by further training DRSformer w/o and w/ ColC after adding SPAData. Please zoom in for more details.
batch size of 4, almost all images from SPAData throughout the training process, impeding the proposed contrastive learning process and further diminishing the performance gain on the five synthetic datasets. Conversely, as the training primarily dominated by SPAData, both DRSformer without and with CoIC have achieved remarkable PSNR and SSIM metrics. In summary, extreme data seale imbalance accompanying small batch size harmful and should be circumvented during the training process.
21


Reviews:
[{"review_id": "g69gjAjqye", "reviewer": "Reviewer_nGEY", "summary": "This paper proposes an instance-specific de-raining models by exploring representations that characterize both the rain and background components in rainy images. The authors first propose a rain-/detailaware contrastive learning strategy to explore joint rain-/detail-aware representations. Leveraging these representations as instructive guidance, the authors furhter introduce CoI-M to perform layer-wise modulation of CNNs and Transformers.", "questions": "See Weaknesses", "limitations": "", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1, The paper is well-written. \n\n2, The paper may be meaningful to some extent for the deraining community.\n\n3, The results reveal that the proposed method consistently improves previous baselines.\n\n4, The proposed instance-specific method may be useful to other low-level vision tasks, e.g., dehazing, desnowing, super-resolution, etc.", "weaknesses": "Although this paper has some strengths, I am still confused by following questions:\n\n1, The proposed techniques seem to be a guidance for deraining, i.e., the authors attempt to constrain an instance-specific representation to guide the deraining.\n\nAlthough this, the reviewer would like to know the increased parameters and FLOPs.\n\n2, Whether the training iterations of the proposed method are the same with the applied method? i.e., BRN, RCDNet, DGUNet, IDT, DRSformer. If not, the comparisons are unfair.\n\n3, The reviewers would like to see the training curve comparisons, e.g., BRN, RCDNet, DGUNet, IDT, DRSformer vs. BRN+ CoIC, RCDNet+ CoIC, DGUNet+ CoIC, IDT+ CoIC, DRSformer+ CoIC.\n\n4, The paper title is the  INSTANCE-SPECIFIC IMAGE DE-RAINING, the reviewer thinks that this is not suitable since the authors also train the deep models on Rain13K. The dataset can be regarded as one entirety. It is hard to reflect the 'INSTANCE'. Whether it has a better title? The reviewer thinks that if the paper does a continual learning for deraining and demonstrates the CoIC can consistently improve the  performance on each dataset, the 'INSTANCE' may be suitable.\n\n5, Deraining methods with high PSNR and SSIM usually tend to overfit to synthetic datasets. Hence, deraining performance would be worse on real datasets. Whether the authors can further solve this problem? \n\n6, Can the rain-aware negative exemplars be images with real rain streaks? Whether the authors consider to explore the real images to participate in training to improve the generalization to real scenes? \n\n7, I have observed that the authors mention the generalization many times. However, only training on synthetic dataset is hard to improve the generalization because of overfitting.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "mdSflDdsWQ", "reviewer": "Reviewer_zqPW", "summary": "The paper focuses on tackling the single image deraining problem, wherein the authors introduce instance-specific de-raining models. These models are designed to delve into meaningful representations that capture the distinct characteristics of both the rainy elements and the background components in images affected by rain. Authors propose Context-based Instance-specific Modulation (CoI-M) mechanism which can modulate CNN- or Transformer-based to learn the representations specific to rain details and background. Authors also propose  rain-/detail-aware contrastive learning strategy to help extract joint rain-/detail-aware instance-specific representations. Integrating these modules authors claim that the proposed method can handle multiple different rain datasets.", "questions": "Please refer weaknesses", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- proposed Context-based Instance-specific Modulation (CoI-M) mechanism which can modulate CNN- or Transformer-based to learn the representations specific to rain details and background\n     - employed feature extractor E,  and a Global Average Pooling (GAP) operation to capture rich spatial and channel information related to rain and image details.\n- proposed contrastive learning strategy to help extract joint rain-/detail-aware instance-specific representations\n     - to make the encoder discriminate the rain in (x,y) pair of rainy images they use leverage a rain layer bank noted a D_{R}\n     - proposed contrastive learning based loss to learn the difference in representations of different rains and also image representations.\n- Proposed Context-based Instance-aware Modulation (CoI-M) mechanism to modulate features at different layers in CNN, and attention layers in transformer based networks, to learn the embedding space spanned by all instance-specific representations.", "weaknesses": "- the comparisons are inadequate. There is rich literature in semi-supervised and continual learning that focuses on representation learning for different types of rain and image representations. Authors failed to compare with the existing methods.\n    -  Memory Oriented Transfer Learning for Semi-Supervised Image Deraining, CVPR 2021.\n        proposes a representation learning based approach where they learn basis vectors (which called memory) to represent the rain and adapt them using them for different datasets to minimize the differences the datasets. The proposed Col-C is also doing similar to this, so it will be easy for the reader to understand the benefits of CoI-C if authors can compare with this method.\n    - Syn2Real Transfer Learning for Image Deraining using Gaussian Processes, CVPR 2020.\n        proposes a Gaussian process based pesudo labeling approach where enable the encoder to learn representation rain and image using generated pesudo label. Note the here in Syn2Real they formulate the joint Gaussian distribution to generated a pesudo label for the unknown or unlabeled image, in way they selecting the k-nearest labeled images and maximizing correlation between unlabeled and labeled  similar type of rain images. Also minimizing correlation between k-nearest farthest labeled images and  unlabeled and labeled  similar type of rain images. Thus Syn2Real approach can be compared to proposed contrastive loss and CoI-C approach, to understand benefits of CoI-C.\n     - Unpaired Deep Image Deraining Using Dual Contrastive Learning, CVPR 2022.\n       proposed a contrastive based approach to learn the representation of rain and image and guide the networks to learn removal network and image generator networks in an unsupervised cyce-GAN approach. Thus, it would be great to see comparisons of CoI-C against this method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "INisDd7Nwl", "reviewer": "Reviewer_v7hb", "summary": "This paper proposes a context-based instance-specific modulation (CoI-M) method for learning adaptive image de-raining models with mixed datasets. The goal is to exploit the commonalities and discrepancies among datasets for training. This mechanism can efficiently modulates both CNN and Transformer architectures. CoI-M is also verified to improve the performances of existing models when training on mixed datasets.", "questions": "See the above Weaknesses part.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "+ As to the paper structure, I think it is clear and easy to follow. \n\n+ The authors claims that images with light rain are primarily characterized by background detail, while heavy rainy images are distinguished more by the rain itself. The statistical analysis in Figure 1 is suitable.\n\n+ The experimental results indicate that the proposed method can further improve performance.", "weaknesses": "- It is not clear how to select the positive and negative exemplars. The author should discuss the differences with relevant contrastive learning-based deraining methods [1-2].\n[1] Chen et al, Unpaired deep image deraining using dual contrastive learning, CVPR2022.\n[2] Ye et al, Unsupervised deraining: Where contrastive learning meets self-similarity, CVPR2022\n\n- I suggest the author conduct further validation on the mixed dataset Rain13K.\n\n- The figures included in the manuscript are rendered with small font sizes and low resolution, which makes them challenging for reviewers to examine and comprehend. I recommend revising the figures to meet these requirements to improve the overall quality and accessibility of the visuals in the manuscript.\n\n-----------------------After Rebuttal---------------------------\n\nThank you for your feedback. The rebuttal addressed my concerns well. I have decided to increase my score.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "JypIx7ZrgT", "reviewer": "Reviewer_iZXv", "summary": "Existing de-raining methods tend to overlook the inherent differences between datasets. To address this limitation, this paper develops a rain-/detail-aware contrastive learning strategy to extract a representation, and proposes a Context-based Instance-specific Modulation mechanism, which uses the representation to modulate models. This approach helps existing methods boost the de-raining ability.", "questions": "Please see 'Weaknesses'.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. This paper focuses on training de-raining models on amalgamated datasets.\n2. The experiments are sufficient and the results are better than others.\n3. Visualization results on RealInt demonstrate the generalization capability of this method on real scenes.", "weaknesses": "1. The main concern is the necessity of the proposed method. In fact, when the deraining model is strong enough, it may be able to learn the differences between datasets. For example, in Table 1, the proposed method has limited performance improvement on DRSformer and DGUNet.\n2. The novelty of the method may be limited. The method of representation extraction based on contrastive learning has been widely explored in the blind super-resolution field [1]. The paper needs to further explain the differences with them.\n[1] Unsupervised Degradation Representation Learning for Blind Super-Resolution. CVPR 2021.\n3. It would be better to show the increase in the number of parameters and changes in inference time brought by the proposed method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fC4ZnwgtWd", "reviewer": "Reviewer_rX1q", "summary": "Summary.\n\nThis paper proposes the CoIC algorithm to train CNN and Transformer based models to be instance-specific, using mixed datasets, for single image rain removal. The CoIC algorithm includes a Context-based Instance-specific Modulation (CoI-M) mechanism and a rain-/detail-aware contrastive learning strategy.", "questions": "Please see above.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Strengths.\n\nThe paper is generally easy to read and follow.\nThe proposed approach seems to work well for CNN based models.\nExperiments (with tables and figures) are provided to analyze the proposed method.\nThe proposed method seems to work well on the RealInt (Wang et al., 2019), in terms of NIQE.\nThe idea of generating negative examples for both rain and background and modulating the image-specific information into networks sounds interesting.", "weaknesses": "Weaknesses/Questions.\n\n\nThe Introduction mentions that there is a long-tail distribution across mixed datasets. As most (if not all) datasets are synthetic, would it be easier/better to synthesize more rain images to handle such an unbalanced distribution? Another question is whether training using a mixed dataset improves the individual performance of one model on each dataset. These answers are not easily found in this paper. \n\nFigure 1(b) shows one observation of this paper that the image is gradually dominated by rain when rain density increases, which, however, sounds superficial. The resulting (from this observation) learning of a joint embedding space for both rain and detail information is based on contrastive learning, while it is not explained why using contrastive learning here as it seems that rain density is the most important factor. Meanwhile, the paper simply says it is distinct from previous instance-discriminative contrastive learning (He et al., 2020) without explanation.\n\nThe Introduction mentions that the embedding space of this paper is not instance-level discriminative, which is hard to understand as the paper aims to learn instance-specific deraining features (If not discriminative enough, how can one model learn the specific information?) The second point it mentions is that the background content is not discriminative, which I do not know whether it is correct, does not explain why it is not suitable for a standard contrastive learning strategy. After I read the method section, it seems to me that the main difference is to construct negative examples based on rain types (selected from another dataset) and Gaussian blurred background. The authors may correct me if I misunderstood but it seems to me that the discussion in the introduction does not match what they do in the method.\n\nThe paper uses Gaussian blur to construct negative exemplars of the background image, which needs further justification as it may not represent the degraded backgrounds. Usually when rain streaks are large or dense, there are more occlusions than blur, which the Gaussian blur may not model.\n\n\n\nThe related work section should be expanded. 1. There are certainly several rain removal methods not cited, especially for heavy rain restoration, although they do not combine multiple datasets for training. 2. The modulation is one of the key techniques in the proposed method but there is no literature review on it.\n\nIn Table 1, the performance gain over DRSformer (Chen et al., 2023) is rather limited. The paper needs to show more advantages, otherwise, it seems not worth doing considering the efforts and the performance gains.\n\nThe visual results are not impressive, as we may easily see the remained rain streaks in, e.g., Figure 4, Figure 12. I understand that there are improvements by comparing the results of methods with and without CoIC. However, it seems to me that these visual results are still failure cases (due to obvious rain streaks, in Figure 12, (g) is comparable to or even better than (h)). Combining it with Table 1, really makes others wonder about the necessity of using the proposed approach.\n\n%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%\nI am glad to see that the rebuttal addresses my previous concerns/questions. I would like to raise my rating to accept and I hope that the authors will incorporate all information in this rebuttal into their final revision.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)