Generative Status Estimation and Information Decoupling for Image Rain Removal
Authors: Di Lin
Xin WANG
Jia Shen
Renjie Zhang
Ruonan Liu
Miaohui Wang
Wuyuan Xie
Qing Guo
Ping Li
Venue: NeurIPS 2022 Accept (2022)
Keywords: Rain Removal
Image Restoration
Generative Model

Abstract: Image rain removal requires the accurate separation between the pixels of the rain streaks and object textures. But the confusing appearances of rains and objects lead to the misunderstanding of pixels, thus remaining the rain streaks or missing the object details in the result. In this paper, we propose SEIDNet equipped with the generative Status Estimation and Information Decoupling for rain removal. In the status estimation, we embed the pixel-wise statuses into the status space, where each status indicates a pixel of the rain or object. The status space allows sampling multiple statuses for a pixel, thus capturing the confusing rain or object. In the information decoupling, we respect the pixel-wise statuses, decoupling the appearance information of rain and object from the pixel. Based on the decoupled information, we construct the kernel space, where multiple kernels are sampled for the pixel to remove the rain and recover the object appearance. We evaluate SEIDNet on the public datasets, achieving state-of-the-art performances of image rain removal. The experimental results also demonstrate the generalization of SEIDNet, which can be easily extended to achieve state-of-the-art performances on other image restoration tasks (e.g., snow, haze, and shadow removal).

Generative Status Estimation and Information Decoupling for Image Rain Removal
Di Lin Xin Wang Jia Shen' Renjie Zhang', Ruonan Liu', Miaohui Wang', Wuyuan Xie Qing Guo", and Ping Li
Tianjin University, China
The Hong Kong Polytechnic University, Hong Kong SShenzhen University, China *Center for Frontier AI Research, STSR,,, Singapore p. p.li@poly..euuhkk
Abstract
Image rain removal requires the accurate separation between the pixels of the rain streaks and object textures. But the confusing appearances of rains and objects lead the misunderstanding of pixels, thus remaining the rain streaks missing the object details in the result. In this paper, we propose SEIDNet equipped with the generative Status Estimation and Information Decoupling for rain removal. In the status estimation, we embed the pixel-wise statuses into the status space, where each status indicates a pixel of the rain or object. The status space allows sampling multiple statuses for pixel, thus capturing the confusing rain or object. In the information decoupling, we respect the pixel-wise statuses, decoupling the appearance information of rain and object from the pixel. Based on the decoupled information, we construct the kernel space, where multiple kernels are sampled for the pixel to remove the rain and recover the object appearance. We evaluate SEIDNet on the public datasets, achieving -fate-of-the-art performances of image rain removal. The experimental results also demonstrate the generalization of SEIDNet, which can be easily extended to achieve state-of-the-arr performances on other image restoration tasks (e.g., snow, haze, and shadow removal), We release the implementation of SEIDNet via https://gghhb..cmmxxxxx000551111
Introduction
Image rain removal relies on the understanding of the appearances of the rains and the objects. Most of the current methods employ the discriminative network to learn the visual features of the pixels, for representing the visual appearances of the rain streaks and object textures. Based on the visual features, the network learns the shared 2 3 4 5 6 7] or dynamic convolutional kernels [8] [ for rain removal on the pixels. Intuitively, the kernels for rain removal respect the pixel-wise statuses (i.e., rain or object), reducing the rainy intensities and recovering the object details of the pixels.
The challenge of rain removal mainly stems from the fact that some of the rains are similar the objects (see the input regions in Figure aa). The confusion between the rains and the objects
inevitably encoded into the visual features. yields the inappropriate kernels for the pixels, contributing to the erroneous results like leaving the rain streaks and removing the object textures in the image (see the predictions in the pink rectangles of Figure [b)). The discriminative network provides a deterministic kernel for a pixel. But it loses the chance of computing more appropriate
Di Lin and Xin Wang contributed equally to this work.
Ping Li is the corresponding author.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Imput
Kemel
Prediction Ground Truth
(a) Rainy Image
(b) Result of Rain Removal
Figure 1: the left-most column, the input regions of the rainy image (a) have similar rain streaks and object textures. The top/bottom pair of input regions processed by the similar kernels. In the result (b), these kernels yield good predictions in the blue rectangles, but remaining rains and missing textures in the pink rectangles. Here, employ EiDeRain [8 to estimate the kernels.
kernels for a misunderstood pixel. The popular methods 2
3
10
12
K 14
15
16
17
18
use different kernels to process the rains several stages. Yet, confusing information the early stages still misleads the kernel computation at the later stages.
In this paper, we present a novel approach for capturing the confusing information of the pixels. The key idea is to use the generative network to learn the probability distribution of the pixel-wise status and the dynamic convolutional kernel. The distribution captures the correlation between status and kernel. Given the distribution, we regard the feature of the pixel as the condition, generating multiple statuses and kernels. The statuses facilitate more focused learning of the dynamic kernels, which capture the rain and object characteristics. Compared to single status, multiple statuses can be used for generating more kernels, which provide more opportunities for refining the result of rain removal.
We construct SEIDNet, a generative network equipped with the pixel-wise Status Estimation and Information Decoupling for rain removal. We illustrate the construction of SEIDNet in Figure 2 We use the rainy image and the object laye as the training sample. In the status estimation (see Figure aa)) we subtract the object layer from the rainy image, achieving the statuses of all pixels in the rainy image. We construct conditional variational auto-encoder (CVAB) to embed the pixel-wise statuses of different pixels into the status space. With the status space, we use the feature of the pixel as the condition to generate multiple statuses that capture the confusing appearance of the pixel.
In the information decoupling (see Figure (b))), represent the decoupled information of rain and object. We exploit these feature maps to learn the kernels. We employ another CVAE to embed the learned kernels into the kernel space, where the feature and the status of the pixel as the condition to generate the kernel. The status estimation can generate multiple statuses, helping generate multiple kernels. These kernels effectively reduce the confusing information of the pixel, finally producing better result of rain removal.
the statuses to yield the feature maps that
We evaluate SEIDNet the public datasets for rain removal (i.e., Rain 100H [66,, Rain [T6], Rain 1400 19), Rain 13K [00] and SPA Bl), achieving state-of oftt--ofttheann art performances. Furthermore, extend SEIDNet to various image restoration tasks (c.g., snow, haze, and shadow removal), where SEIDNet also surpasses the recent methods on the public datasets 20 21 22).
2 Related work
2.1 Discriminative Networks
The recent methods use the deep discriminative network to learn the visual features of rain and object. Zhang et al. [4] propose a residual-awen aware classifier to recognize the rain density. Yang et al. [16] use large convolutional kernels to capture rich visual information for recovering the details of the
dense rain streaks and object textures. Zhang et al. [23), Wang et al. [3) and Li et al. [3] use the
We refer the object layer the image without rain.

---

Object Layer
Status Map
Conditin
Eneoder
Trpui Input
Emeader
Condition
Conditoo
Feature Masking
CVAE V_
Status Map R
Kamnl
CVAE V.
Object Laye: Of
Map
Map K'
Kainy lnange
Feature Map F
(6) Status Fstimation
(b) Information Decoupling
P
H, H OZ
My 1 OZ
Status Map R
Encoder
Rain Festure Map F
Kernel Map K
Status Map
Decoder
Featuuee Map
Kernel Map K
Decoder
Map R
Map
comm
Condition
Condition
Feature Map
I-R
Object
Map F
Feature Map F
Festure Masking
CVAE V.
Subtraction
multiplication
Figure 2: The training architecture of SEIDNet. The architecture has the (a) status estimation and (b) information decoupling. We use (c) the CVAE for learning the status space, (d) the feature masking for yielding the rain and object feature maps, and (e) the CVAE for learning the kernel space.
multi-scale information to capture the appearances of rains and objects in local and global ranges. Li al. [22 use the dilated convolution to capture the multi-scale image information for predicting the diverse shapes of the rain streaks. Wang et al. (2) propose the convolutional dictionary model, where rain kernels are used for capturing the appearances of the rain streaks and object textures.
The discriminative networks are also used for learning the correlation between rain and object terms of their appearances. Luo et al. 24 combine the sparse coding and the greedy pursuit algorithms to separate the rain and object layers. the patch-based priors of rain and object. Deng al. [26] a unified network for rain removal and repair of object textures. Fu et al. 19 propose the detail layer of the rain streaks and object contours, for reducing the impact of low-frequency object textures the rain removal.
et al. 25 utilize Gaussian Mixture Models (GMMs) to involve
Based on the pixel-wisc wise features, the discriminative networks compute the shared or the dynamic convolutional kernels [819] for rain removal on the pixels. Yet. the similar appearances of the rains and the objects let the discriminative network predict the problematic kernels. In contrast, use the generative network that learns the probability distribution of the pixel-wise status. With the distribution, we sample multiple statuses for capturing the confusing information the pixel. assists the computation of kernels for processing the pixels of the rain streaks and object textures.
2.2 Generative Networks
The generative networks, such as the variational auto-encoders (VAEs) 27
and the 34 35), have been widely used for image
28 29
30
generative adversarial networks (GANs) 31 32 generation. Some the methods employ the generative networks to recover the object layer of the rainy image. Li et al. 14 input the rainy image to the conditional GAN that preserves the object textures. Rui et al. [17] resort to the visual attention mechanism, attending to the important image regions with rain streaks. Zhang al. (23] propose the delicate loss function to alleviate the artifacts generated by GAN the object layer. Du et al. [36] equip the conditional VAE [37 to the spatial density estimation of the rain streaks for
33
accurate rain removal.
The existing methods use a stand-alone generative network for embedding the features of the rains and objects into the latent space. But the confusing rains and objects may mislead the construction of

---

the latent space. These methods may achieve the latent vector of the rain (or object) from the latent space, where the vector mistakenly represents the object (or rain). Instead, we utilize couple of CVAEs, for learning the probability distribution of the pixel-wise status and dynamic kernel. We use the first CVAF to sample multiple statuses from the latent space, capturing the confusing information of the pixel. With the second CVAE, we use multiple statuses to sample the kernels. These kernels are learned from the decoupled information of the rains and objects. They better reduce the confusing information and refine the pixel intensities in the result.
Method Overview
We introduce the probability distribution of the pixel-wise status and kernel. The distribution is learned by CVAEs in the status estimation and information decoupling of SEIDNet.
compute the visual feature map F TR BxW xC and the status map E TR. Hx F(x, u) E RdC represents the feature vector of the pixel located at (z, y) in the rainy image. R(x, y) E RR is the score for the pixel. A higher (or lower) score indicates the status of the rain (or object). We compute the kernel map K E
Probability Distribution For the rainy image I Hx
HR [ (SXSxx))
where ((x; y) E I& XxCC is the S x S kernel with C' channels for the pixel at (I, v). We construct the generative network to learn the probability distribution P(R, K F), which takes condition as the feature map F for computing the status map R. and the kernel map K. We formulate P(R, K F) as:
F(R,K|F) = P(R, K F. 2) P(Z) dz.
E HRHx is a set of latent variables generated by the normal distribution PZZ). We factorize the distribution P(R. K F, Z) into two distributions, P(K R. F, Z) and P(R. F, Z), as:
P(R, K F. Z, P(K| R. Z) P(R F.
The factorization of probability distribution enables a more focused learning of the dynamic kernel based on the pixel-wise status. We use two CVAEs in the status estimation and information decoupling to learn the conditional distributions (RR F, Z) and P(K) F. Z), respectively.
Status Estimation As illustrated in Figure a), we use the rainy image and the object layer O Hx train the CVAE V.e- V.e learns the distribution P(R. F, Z), for modeling the status space. We subtract O from I, to achieve the status map R. Vse embeds the statuses in R. into the status space. As illustrate in Figure 2c), Vee regards the feature map as the condition, for sampling the status map R' from the status space. We regard R' as the estimation of R.
Information Decoupling We use the status map R and the feature map F to train the CVAE Via in the information decoupling (see Figure b)). Via learns the distribution P(K R. F, Z) that models the kernel space. First, we feed R and F into the feature masking (see Figure d)). We decouple into the rain feature map F, E "xx and the object feature map F. E XxxXC We use F, and F, compute the kernel map K that removes the rains and recovers the objects. Next, Ved embeds K into the kernel space. As illustrated in Figure e), Vid regards F and as the condition, for sampling the kemee map K' from the kernel space. K estimates K for rain removal.
4 Architecture of SEIDNet
Below, we introduce the training and testing architectures of SEIDNet for rain removal.
4.1 Training Architecture
Status Estimation We illustrate the training architecture of SEIDNet in Figure 2 In the status estimation (see Figure 2 a)), achieve the status map R #x as:
R (cconv O)).
(3)
where R(x,, y) E [0, 1]. and comu are the sigmoid function and a set of convolutional layers. For the rainy image I IR x33. we use the convolution to learn the feature map F E 18 xxxx WxC We

---

H
Decoder
Condition
Decodlr
My M
Status Map R
Map
My 02
Condition
Kainy
Feature
Kamel K*
Object Layer
Status Map
Kernel
K
Generation Status Mapa
Generation Kemel
Flament-arss multiglication
Average
D Concataratee
@
Convolution
Figure 3:
The testing architecture of SEIDNet. It contains the condition and decoder branches of CVAEs, for generating (a) status maps and (b) kernel maps. For brevity, only illustrate two pairs and (R", K"). More maps can be generated for testing.
of status and kernel maps, i.e., (R" mt K
input R and F the CVAE Vse (see Figure c)), whose encoder, condition, and decoder branches are formulated as:
My: o,] = encoder( R, F)), [uy. و condition(F))
= decoder([F, μ4, ,,-to,0Z)) (4)
H is the feature concatenation along the channel dimension. C is the element-wise multiplication.
The encoder and condition branches take input as [R, F] and F, yielding the mean value maps Mr, μ, E R H and the standard deviation maps or: of IR Hx XC We use the normal distribution (00, 1) to generate the latent variable map Z IR #XXXC We input F, Z and (H,, σ,) to the decoder, yielding the status map R'
RR # We use the sigmoid function to normalize R
During the network training, use T--moorm (denoted as I2) to penalize the difference between the status map and the estimated counterpart R'. Besides, we compute Kullback-Leibler Divergence (denoted as K L) between the Gaussian distributions G(ur,, σx) and G( ,r)) It allows to use (ny, f) in place of (pr,, σT) that are unavailable during testing. We define a = 4 as the weight of KL divergence. L2-norm and KL divergence compose the status estimation loss Lae
Lee = (R,, R') + aKL(G o,) G(ps,
(5)
This loss guides Vsc learn the distribution P(R F, Z) Eq. 2 which models the status space.
Information Decoupling In the information decoupling (see Figure 2(b)), we pass the status map R and the feature map F into the feature masking (see Figure d)), achieving the feature maps Fr, F. E x xC in Eq. (6). Based on Fr, F, that respectively represent the appearances of rains and objects in the image I, we compute the kernel maps K,, K, E H x(5x5xC)
F, R K, (F,, F. =1 R O., K, = (F.v(F.).
(6)
We use the status map R weight K, and K,, yielding the kernel map K E RRxW as:
K R K,
R) K,
As illustrated Figure e],,, the CVAE Via in the information decoupling also has the encoder, condition and decoder branches, which formulated as:
[M4, =e]=end [K, F. R)), ( Rl), K' K'=decoder([P) R. p. 0
Z
(8)
We input KK F. R] and [F. R] into the encoder and condition branches, respectively, producing the mean value maps pig Mcc E R x xC and the standard deviation maps o G. E xC We feed F, R. μh o 3 Z] into the decoder, yielding the kernel map K' Hx (SXXXX)
We resort to the L2- norm and KL. divergence to construct the information decoupling loss Lid as:
Lia (K, K') + aKL(G σk) 9(pc. a.)).
Via learns the conditional distribution P(K| R. F. Z) in Eq. 2 which models the kernel space.
5

---

Deraining and Overall Losses Given the kernel map K', we perform the convolution on the rainy image I and estimate the object layer O'. As formulated in the left of Eq. (TO). the convolution (denoted as () is pixel-wise. where the pixel [ær, y) is processed by the kernel (x,,v) We use the structural similarity (denoted as SSTM) loss and 7--noomm to penalize the difference between the object layer O and the estimated layer O', yielding the deraining loss L. in the right of Eq. (T).
O' O' (x, y) = (T, v) @ [(x, v), Lie L2 O') + (O, O'),
(10)
where 3 = 0.2. With the losses Lees Lid and Lie. we form the overall loss L as:
L Lise + Lia Lide-
(11)
4.2 Testing Architecture
Kernel Generation During the network testing, we follow the convention to remove all encoder
branches of the CVAEs V.e and Via. It results in the testing architecture (see Figure 3). We formulate
the process of using Vse and Via to generate the status map R. and the kernel map K as:
p o = condition(F)), R decoder (F,, μy + o ② ))).
(12)
11 oe! condition(FF. R"),, K decoder (F, R. 1e" + t. о I).
As illustrated in Figure 3 a), we use the rainy image I to compute the the visual feature map We pass F into the condition branch of Vse. achieving the mean value map by and the standard deviation map o We use the normal distribution to generate the latent variable map z" As formulated in Eq. (12). the decoder of Vee uses F, Zn and (H,, of to generate the status map R.
As illustrated Figure b)), we use the condition branch of Via, which takes input as and tM to compute the mean value map pa,"" and the standard deviation map o² As formulated in Eq. (12). the decoder of Vid uses F, R Z and (1r" o.") to generate the kernel map K"
Kernel Aggregation We use the normal distribution to generate an array of latent variable maps { m 1, 1,.... N). Each latent variable map be used by Eq. 112 produce set of kernel maps {Km m 1, 1,....N. N). As illustrate in Figure (b), we sum these kernel maps as:
K
ΣΣ KT
(13)
N
m=
where K" E (XXWX(SXSXCC is convoluted with the rainy image I for computing the object layer.
Experiments
5.1 Experimental Datasets
We compare SEIDNet with state-of-the-art methods, on the Rain [6], Rain [6]. Rain1400 [19], and SPA 3 datasets. These datasets provide 1, 800/200/12, 600/638, 492 images for training, along with 100/100/1, 400//,, 000 images for testing. We also evaluate different methods on the Rain 1 13K [10) dataset, which provides 13, 712 images for training. The test set of Rain13K con- tains 4, 300 images, which are taken from the test sets of Test100 (3). Test [4, Test2800 [19], Rain 1 00H, and Rain 100L. Rain allows the models be trained the unified data and evaluated on the separate test sets, thus justifying the model generalization. We report the performances of rain removal in terms of peak signal-to-noise ratio (PSNR) and structural similarity (SSIM).
5.2 Ablation Study of SEIDNet
Below, we use the test set of Rain 100H 16 for the major evaluation of SEIDNet.
Sensitivity to the Number of Kernels Given the testing architecture of SEIDNet, we generate and aggregate multiple kernels for removing the rains on the image. We change the number N of the generated kernels for each pixel, evaluating the impact on the performances. We choose the number N from the set (1, 2, 4, 8, 16, 32). We report the computational overheads (i.e., GPU memory and testing time) in Figure 4 a b), along with the performances (i.c., PSNR and SSIM) in Figure d)..

---

74
6,885
3355
33,221
33221
0:3227
119027
0.5527
50
0.445
32.55
0329
3.6
32 381
0,1042
00
07
133
6065
6.105
0 8o0p lefr
0083
0.000
32:55
31
31.755
D421
0,9122
03131
16
32
16
32
(a) GPU Memory
(b) Testing Time
(c) PSNR
(d) SSIM
Figure 4: Sensitivities of GPU memory in GB (a), testing time per image in seconds (b), PSNR (c)
and SSIM (d) to the number of kernels.
With N 1, generate a single kernel for each pixel the rainy image, yielding 31.457 PSNR and 0.9142 SSIM on the test set of Rain 100H. With N 8, we considerably improve the performances (up 33.217 PSNR and 0.9327 SSIM). This is because more kernels better capture the visual patterns of rain streaks and object textures. Too many kernels (e.g.. N = 32) saturate the performances, but needing more computations. Below, we use N as default.
Analysis of Network Components The status estima- tion and information decoupling of SEIDNet are based on the generative CVAEs for computing the statuses and kernels. To evaluate the generative power for rain removal, we experiment with using the discriminative networks for the status estimation and information de- coupling. For the status estimation, we resort to the convolution for predicting single status map, based on the feature map of the rainy image. For the information decoupling, we pass the visual feature map and the sta- tus map to the convolution, predicting a single kernel map for the rainy image. The alternative architectures are provided in the supplementary material.
SE
ID
Memory
Time
PSNR
SSIM
D
D
1.0
0.0815
29.43
0.8809
G
D
1.2
0.0643
29.62
0.8849
D
G
1.9
0,0704
31.42
0.9172
G
G
2.1
0.0832
33,22
0.9327
Table 1: We refer to SE, ID, Memory and
Time as the status estimation, information decoupling, GPU memory (GB) and test- ing time (second). D and G indicate the discriminative and generative networks, re- spectively. The performances reported on the test set of Rain
First, we remove all CVABs, only using the discriminative network for the status estimation and information decoupling. In this case, we yield the unsatisfactory 29.43 PSNR and 0.8809 SSIM (see the first row of Table I The discriminative network only provides a pair of status and kernel maps for the rainy image, missing the critical patterns of rains and objects.
Next, we the discriminative network for either the status estimation or the information decoupling. There is a CVAE for capturing the diverse patterns of the statuses or kernels, leading to better results (see the second and third rows of Table Th than the discriminative networks. Yet, the performances achieved by a CVAE are lower than SEIDNet. This is because SEIDNet employs a couple of CVAEs comprehensively model the correlation between each pair of status and kernel. We analyze the correlation between the status and the kernel in the supplementary material.
Various Combinations of Networks In Table 2 compare different strategies of using the discriminative and generative networks for deraining. Similar to SEI- DNet, we use 132 convolutional layers to construct a discriminative network for predicting the kernel maps. The generative SEIDNet outperforms the discriminative network (see the first and second rows).
Network
Memory
Time
PSNR
SSIM
D
0.9
0.0453
29.91
0.8905
G
2.1
0.0832
33.22
0.9327
D+G
0.0879
33.11
0.9310
Table 2: D or G means the discriminative or
generative network that estimates the ker- nels for rain removal. The performances are reported on the test set 100H.
We combine the discriminative and generative networks by averaging the kernel maps. This combination increases the network parameters but degrades the performances (see the last row). A smarter combination of the discriminative and generative networks needed for reducing the parameters and improving the performances. We illustrate the compared networks in the supplementary material.
Different Ways of Using CVAEs SEIDNet has a pair of CVAES that model the factorized distributions of the status and the kernel. We compare SEIDNet with the alternative methods, which use a CVAE without distribution factorization. We list the results in Table 3 Again, we illustrate the compared networks in the sup- plementary material.
Method
Memory
PSNR
SSIM
One CVAE for K
1.6
0.0703
25.21
0.7929
One CVAE for (K, R)
19
0.0774
29.03
0.8963
Two CVABs for (K, R)
2.1
0.0832
33.22
0.9327
Table 3: K/R is the kernel/status maps
yielded by various probability factoriza- tions. We list the results on the test set of 10On
7

---

First, we set the single CVAE that only takes the visual feature map of the rainy image as the condition. This method directly generates the kernel maps. Because the status maps are unavailable for enabling the focused learning of the kernel maps, this method yields lower performances (see the first row).
Image
Input
RCDNet
MPR
SPDNct
SEIDNet
Figure 5: Visual results on the image deraining task. We zoom in the image regions (see the blue rectangles) compare deraining results of different methods.
the single CVAE gen- erate the status and kernel maps. Again, this CVAE takes the visual feature map of the rainy image as the only condition.
Second,
Subset A
Subset B
Subset C
Method
PSNR
SSIM
PSNR
SSIM
PSNR
SSIM
MPR 10
35.57
0,9519
36.83
0,9579
37.65
0.9630
ErDeRain X
34.37
0,9556
36.18
0,9629
35.44
0.9576
The status and kernel maps are generated by the separate decoder branches. The single CVAE only depends on the train- ing loss of the status map, for implicitly guiding the generation of the kernel map. It is less effective than SEIDNet, where
SPDNer IS
31.84
0.9094
33.57
0.9254
32.93
0.9162
SEIDNet
39.07
0,9813
39.90
0.9799
Table 4: We compare SEIDNet with state-of-the-art meth- ods on three subsets of SPA dataset. The performances are reported terms of PSNR and SSIM. the generation of the kernel map straightforwardly guided by the status map. As result, SEIDNet outperforms this single CVAE (see the last two rows of Table
Rain100H
Rain100L
Rain1400
SPA
Rain13K
Method
PSNR
SSIM
PSNR
SSIM
PSNR
SSIM
PSNR
SSIM
PReNet
11
29.46
0.8988
37.48
0.9792
32.66
0.9347
40.16
0.9816
31.47
0.9129
JORDER 16
30.50
0.8967
38.59
0.9834
32.00
0.9347
40.78
0.9811
31.39
0.9118
SPANet
3
25.11
0.8332
35.33
0.9694
29.85
0.9148
40.23
0.9838
29.83
0.8951
RCDNel 38
31.28
0.9090
39.99
0.9860
33.04
0.9339
41.47
0.9854
32.02
0.9411
CVID 36
27.93
0.8765
37.83
0.9882
28.69
0.8722
34.45
0.9437
28.19
0.8584
10
30.64
0.9040
0.9564
33.28
0.9540
43.59
33.27
0.9510
BiDeRain 8
31.14
0.8990
35.04
0.9634
32.91
0.9323
43.77
0.9894
32.11
0.9416
SPDNet [B
32.68
0.9202
39.59
0.9854
32.89
0.9444
43.555
0.9875
32.01
0.9367
Our SEIDNet
33.22
0.9327
40.66
0.9865
34.84
0.9626
44.96
0.9911
33.62
0.9539
Table 5: We compare SEIDNet with ofate-of-the-art methods on the test sets of Rain 100H. Rain 100L,
Rain1400 1400, SPA and Rain13K. The performances reported in terms of PSNR and SSIM.
Discussion on Size of Training and Testing Dataset Conducting experiments on the unbalanced training and testing splits may reduce the confidence of SEIDNet. Thus, we reduce the training data in the extremely unbalanced dataset (i.e., SPA dataset with 638K/1K images for training and testing). This done by randomly sampling 1K training images from SPA dataset. The random sample is three-fold, thus forming three different subsets, each of which contains 1K images for training different models. The trained models are evaluated on the 1K images in the original test set. With different subsets for training, SEIDNet yields better results than other methods (see the Table 4

---

Rain100H
Rain1001
Test100
Test1200
Test2800
Overall
Method
PSNR
SSIM
PSNR
SSIM
PSNR
SSIM
PSNR
SSIM
PSNR
SSIM
PSNR
SSIM
PReNet (1]
27.02
0.8655
32.61
0,9513
24.89
0.8564
31.54
0.9136
31.79
0.9151
31.47
0.9130
JORDER [6]
27.43
0.8677
32.42
0.9476
24.29
0.8542
31.44
0.9110
31.72
0.9145
31.39
0.9118
SPANet 3
26.88
0.8536
31.26
0.9247
23.17
0.7853
29.93
0.8928
30.07
0.9004
29.83
0.8951
RCDNet 38
30.17
0.8876
35.06
0.9603
23.79
0.8303
31.68
0.9294
32.41
0.9513
32.02
0,9411
CVTD
26.25
0.8444
30.53
0.9025
23.23
0.7824
0.8401
28.50
0.3685
0.8584
MPR 10
30.47
0.8926
36.45
0,9669
30.29
0.9139
32,98
0.9397
33.47
0.9587
33.26
0.9510
EIDeRain [
30.44
0.8954
35.45
0,9645
27.67
0.8874
31,41
0.9260
32.53
0.9511
32.11
0.9416
SPDNet [+]
30.56
0.8956
35.37
0.9621
24.87
0.8349
31.49
0,9152
32.59
0.9501
32.
0.9367
Our SelDNet
31.18
0.8993
36.83
0.9657
30.29
0.9148
33.16
0.9442
33,93
0.9611
33.62
0.9539
Table 6: performances are reported in terms of PSNR and SSIM.
We compare SEIDNet with state-of-the-ar methods on the test sets of Rain 13K. The
Snow I00K-S
Snow 100K-M
Snow 100K-L
Overall
Method
PSNR
SSLM
PSNR
SSIM
PSNK
SSIM
PSNK
SSIM
DuRN-S-P
32.27
0.9497
30.92
0.9398
27.21
0.8891
30.12
0.9261
Composition GAN
31.21
0.9431
DesnowNet
32.33
0,9500
30.87
0.9409
27.17
0.8983
30.11
0.9296
DS GAN
46
33.43
0,9641
31.88
0,9570
28.07
0.9211
31.11
0.9473
HDCWNet
47
33.21
0,9623
32,38
0.9541
28.13
0.9253
31.24
0.9472
DDMSNet
34.34
0.9445
32.89
0,9330
28.85
0.8772
32.03
0.9183
RFMPRaLSGAN 49
33.68
0.9690
30.47
0.9500
29.38
0.9440
31.17
0.9540
RSRNet 50
30.52
0.9444
Our SEIDNet
35.01
0.9765
33.45
0.9711
29.84
0.9454
32.77
0.9643
Table 7: We compare SEIDNet with state-of-the-art methods on the test sets of Snow1 100K. The performances are reported in terms of PSNR and SSIM.
5.3 Comparison with State-of-the-Art Methods
In Table we compare SEIDNet with the recent methods on the Rain100H, Rain Rain 1400, SPA, and Rain13K datasets, where SEIDNet outperforms other methods. We show the deraining results of the competitive methods in Figure
SEIDNet is trained on the unified data, outperforming other methods on test sets of Rain13K. Thus, SEID- Net shows better generalization across different test sets. We average the performances on test sets and report the results in Table 5 We provide the results on the separate test sets (i.c., the test sets of Test100, Test1200 4.. Test2800 (191. Rain 100H, and Rain100L) in the Table The proposed SEIDNet outperforms ohter methods the separate test sets, which justifies the generalization of the deraining model.
IT'S Subset
OTS Subse(
Method
PSNR
SSIM
PSNR
SSIM
Grid-Net
19
32.16 16
0.9836
30.86
0.9820
MSBDN
40
33.67
0.9850
33.48
0.9820
FEA-Net
41
36.39
0.9886
33.57
0,9840
AECR-Nel
37.17
0.9901
33.84
0.9837
D-Former
44
40.05
0.9960
34,95
0.9840
SEIDNet
40.62
35.72
0.9951
Table 8: We compare SEIDNee with other methods on ITS&OTS. The results are listed in terms of PSNR and SSIM.
5.4 Extensive Comparison on Different Tasks
To evaluate the generalization of SEIDNet on differen tasks, we use SEIDNet to resolve the desnow. dehaze, and deshadow tasks. We report the performances on different tasks in Tables 8 We provide the visual results of SEIDNet on different tasks in Figure
g
and 10
we use Snow100K dataset (20 for evalu- ation. Snow 100K has three subsets, i.e., Snow 100K- S. SSwSSSwow---- flakes the images have small, medium, and large sizes. The images of 10oo00 only contains the small snow flakes. The snow flakes Snow 100K- M have small and medium sizes, while those in Snow 100K-L have small, medium, and large sizes. In each subset, there about 17K images for train- ing/testing. We also average the results (see "Overall") on Snow 100K-S, Snow 100KMM, and Snow 100KLL.
In Table 7
Method
Shadow
Non-Shadow
All
ST-CGAN
13.4
7.7
8.7
and mo0w000--I, where the snow
DeshadowNet
51
15.9
6.0
Mask-GAN
4.0
SP+M-Net
53
3.0
PMDNes [4
9.7
3.0
4,0
ARFNet 55
6.5
3.8
4.2
CRFormer [6
5.9
2.9
3,4
Our SEIDNet
6.4
3.4
3.9
Table 9: We compare SEIDNet with other methods on the test set of ISTD+ The per- formances are reported in term of RMSE.
In Table we use the ITS&OTS dataset 21 to justify the generalization of SEIDNet on the haze removal task. In the ITS&OTS dataset, the ITS subset

---

Input Images
Desnow Results
Input Images
Dehaze Results
Input Images Deshadow Results
Figure 6: Visual results on Snow 100K, ITS&OTS and ISTD test sets.
contains 110,000 indoor images with haze. The OTS subset has 313,950 outdoor images with haze.
We train SEIDNet on the ITS and OTS subsets, respectively. The network is evaluated on the indoor and outdoor test sets, respectively, where each test set contains 500 images. Table S we also report the results of other methods for haze removal.
In Table and T0 we use the adjusted ISTD (ISTD+) 57 and ISTD datasets to evaluate the performances of different methods on the shadow removal task. The images in the ISTD dataset are taken from 135 different scenarios. There are 1,330 and 540 images for training and testing, respectively. Le et al. [57] adjust the color inconsistency between the shadow and shadow free images of ISTD and achieve the ISTD+ dataset. We use the image regions with/without shadow test different methods (see the performances in "Shadow" and "Non-Shadow") We also use the full images to test the methods and report the performances "All".
6 Conclusions
The latest progress of rain removal benefits from deep discrininative networks trained on large-scale datasets. In this paper, have proposed generative network, SEIDNet, to generate the pixel-wise status and kernel for rain removal. SEIDNet has two CVAEs, which model the factorized probability distributions. It learns the status and kernel spaces. In contrast the discrim- inative networks, SEIDNet enables the generation of multiple statuses for the pixel. for capturing the confu- sion between the appearances of rains and objects. We employ these statuses for generating multiple kernels, reducing the confusing information and refining the deraining result on the pixel. SEIDNet achieves state-of-the-an art performances on the public datasets. In the future work, plan to explore an effective strategy of combining the discriminative and generative networks for rain removal.
Method
Shadow
Non-Shadow
All
Mask-GAN
52
12.67
6.68
7.41
ARGAN
S
9.21
6.27
6.63
60
6.39
RIS-GAN
6.31
DHAN
61
8.14
6.04
6.37
CANet [+
8.86
6,07
6.15
AEFNet 55
7.77
5.56
5.92
CRFomeer 56
7.32
5.82
6.07
Our SEIDNet
5.08
5.47
Table 10: We compare SEIDNet with other methods on the test set of ISTD. The per- formances are reported in term of RMSE.
Negative Societal Impacts
Our approach can be broadly applied in many scenarios (e.g., autonomous vehicles and video surveil- lance). One should be cautious of the problematic results, which may give rise to the infringement of privacy or economic interest.
Acknowledgements
We thank the anonymous reviewers for their constructive comments. This work was supported by The Hong Kong Polytechnic University under Grant P0030419, Grant P0042740, and Grant P0035358.
10

---

References
[1] Xiaowei Hu, Chiwing Fu, Lei Zhu, and Phengann Heng. Depth-attentional features for single-image rain removal. In IEEE CVPR, pages 8022-8033 1031, 2019.
Hong Wang. Qi Xie, Qian Zhao, and Deyu Meng. A model-driven deep neural network for single image rain removal. IEEE Conference on Computer Vision and Pattern Recognition, pages 3103-31 12, 2020.
[3] Tianyu Wang, Xin Yang. Ke Xu, Shaozhe Chen, Qiang Zhang, and Rynson W.H. Lau. Spatial attentive single-image deraining with a high quality real rain dataset. In IEEE CVPR, pages 12270-12279, 2019.
[4] Zhang and Vishal M Patel. Density- aware single image de-raining using a multi stream dense network. In IEEE CVPR, pages 695-704, 2018.
[5] Yang Liu, Ziyu Yue, Jinshan Pan, and Zhixun Su. Unpaired learning for deep image deraining with rain direction regularizer. In JEEE ICCV, pages 4753-4761, 2021.
[6] Xu, Xin Tian, Xin Yang, Bancai Yin, and Rynson WH Lau. Intensity-aware single-image deraining with semantic and color regularization. IEEE Transactions on Image Processing, 30:8497-8509, 2021.
[7] Yinglong Wang, Chao Ma, and Bing Zeng. Aulti-decoding deraining network and quasi-sparsigy based training. In IEEE CVPR, pages 13375 13384, 2021.
[8] Qing Guo, Jingyang Sun, Felix Juefei-Xu, Lei Ma, Xiaofei Xie, Wei Feng. Yang Liu, and Jianjun Zhao Elficientderain: Learning pixel-wise dilation filtering for high-efficiency single-image deraining. In Proceedings of the AAAI Conference Artificial Intelligence, pages 1487-1495, 2021.
[9] Ye-Tao Wang, Xi-Le Zhao, Tai- Tai-Xiang Jiang, Liang-Jian Deng, Yi Chang, and Ting-Zhu Huang. Rain streaks removal for single image via kernel-guided convolutional neural network. IEEE Transactions on Neural Networks and Learning Systems, 32(8):3664-3666 2020.
[10] Syed Waqas Zamir, Aditya Arora, Salman Khan, Munawar Hayat, Fahad Shahbaz Khan, Ming- -ssan Yang, and Ling Shao. Multi-stage progressive image restoration. In IEEE CVPR, pages 14821-14831, 2021.
[11] Dongwei Ren, Wangmeng Zuo, Qinghua Hu, Pengfei Zhu, and Deyu Meng. Progressive image deraining networks: A better and simpler baseline. In IEEE CVPR, pages 3937-3946, 2019.
[12] Xia Li, Jianlong Wu, Zhouchen Lin, Hong Liu, and Hongbin Zha. Recurrent squeeze-and-excitttion context aggregation net for single image deraining. In ECCV, pages 254 144-269, 2018.
[13] Ruoteng Li, Loong-Fah Chcong, and Robby T Tan. Single image deraining using scalc-aware multi-stage recurrent network. arXiv preprint arXiv: 1712.0680, 2017.
[14] Ruoteng Li. Loong-Far Fah Cheong, and Robby T Tan. Heavy rain image restoration: Integrating physics model and conditional adversarial learning. In IEEE CVPR, pages 1633 1142, 2019.
[15] Wenhan Yang, Robby T Tan, Jiashi Feng, Jiaying Liu, Zongming Guo, and Shuicheng Yan. Deep joint rain detection and removal from single image. In IEEE CVPR, pages 1357-1366, 2017.
[16] Wenhan Yang, Robby T Tan, Jiashi Feng. Jiaying Liu, Shuicheng Yan, and Zongming Guo. Joint rain detection and remowal from single image with contextualized deep networks. IEEF Transactions on Puttern Analysis and Machine Intelligence, 42(6):1377-1393 2019.
[17] Rui Qian, Robby Tan, Wenhan Yang. Jiajun Su, and Jiaying Liu. Attentive generative adversarial network for raindrop removal from a single image. In IEEE CVPR, pages 2482-2491, 2018
[18] Qiaosi Yi, Juncheng Li, Qinyan Dai, Faming Fang, Guixu Zhang, and Tieyong Zeng. Structure -rreeerviipeeeeiingge deraining with residue channel prior guidance. In IEEE ICCV, pages 4238-4247, 2021.
[19] Xueyang Fu, Jiabin Huang, Delu Zeng. Huang Yue, and John Paisley. Removing rain from single images via a deep detail network. In IEEE CVPR, pages 3855-3863, 2017."
[20] Yun-Fu Liu, Da- Jaw, Shih-Chia Huang, and Jenq-Neng Hwang. Desnownet: Context-aware deep network for snow removal. IEEE Transactions on Image Processing, 2766)330 2018.
[21] Boyi Li, Wenqi Ren, Dengpan Fu, Dacheng Tao, Dan Feng. Wenjun Zeng. and Zhangyang Wang. Bench- marking single-iman image dehazing and beyond. IEEE Transactions on Image Processing, 28(1):492-55) 505, 2018.
11

---

[22] Jifeng Wang, Xiang Li, and Jian Yang. Stacked conditional generative adversarial networks for jointly learning shadow detection and shadow removal. In IEEE CVPR. pages 1788-1797, 2018.
[23] He Zhang, Vishwanath Sindagi, and Vishal M Patel. Image de-raining using conditional generative adversarial network. IEFF Transactions on Circuits and Systems for Video Technology, 30(1 10(11):94333350 2019.
[24] Yu Luo, Yong Xu, and Hui Ji. Removing rain from single image via discriminative sparse coding. In IEEE ICCV, pages 3397-3405, 2015.
[25] Yu Li, Robby T Tan, Xiaojic Guo, Jiangbo Lu, and Michael S Brown. Rain streak removal using layer priors. In IEEE CVPR, pages 2736- 773442744, 2016.
[26] Sen Deng, Mingqiang Wei, Jun Wang, Yidan Feng, Luming Liang, Haoran Xie, Fu Lee Wang. and Meng Wang. Detail-recovery image deraining via context aggregation networks. In IEEE CVPR, pages 18660-14569, 2020.
[27] Diederik P Kingma and Max Welling, Auto- aato-encoding variational bayes. arXiv preprint arXiv:1312.6114, 2013.
[28] Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic backpropagation and approx- imate inference in deep generative models. In International Conference Machine Learning, pages 1278-1286, 2014.
[29] Zhijie Wu, Xiang Wang, Di Lin, Dani Lischinski, Daniel Cohen-Or, and Hui Huang. Sagnet: Structure- aware generative network for 3d 3d-shape modeling, ACM Transactions Graphics, 38(4): 14, 2019.
[30] Xin Jin, Zhibo Chen, Jianxin Lin, Zhikai Chen, and Wei Zhou. Unsupervised single image deraining with self-supervised constraints. In IEEE International Conference Image Processing, pages 2761-2765, 2019.
[31] Ian Goodfellow, Jean Pougetabadie, Mehdi Mirza, Bing Xu, David Wardefarley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In NeurIPS, volume 27, 2014.
[32] Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks. arXiv preprint urXiv:1511.06434 2015.
[33] Yanyan Wei, Zhao Zhang, Yang Wang, Mingliang Xu, Yi Yang. Shuicheng Yan, and Meng Wang. Deraineyclegan: Rain attentive cyclegan for single image deraining and rainmaking. IEEE Transactions on Image Processing, 30:4788 4801, 2021.
[34] Yuntong Ye, Yi Chang, Hanyu Zhou, and Luxin Yan. Closing the loop: Joint rain generation and removal via disentangled image translation. In IEEE CVPR, pages 2053-2062, 2021.
[35] Zheng Dong, Ke Xu, Yin Yang, Hujun Bao, Weiwei Xu, and Rynson WH Lau. Location-aware single image reflection removal. In IEEE ICCV, pages 5017-5026, 2021.
[36] Yingjun Du, Jun Xu, Xiantong Zhen, Ming-Ming Cheng, and Ling Shao. Conditional variational image deraining, IEEE Transactions on Image Processing, 29:6288-6301, 2020.
[37] Kihyuk Sohn, Honglak Lee, and Xinchen Yan. Learning structured output representation using deep conditional generative models. In NeurlPS, pages 3483-3491, 2015.
[38] Hong Wang, Qi Xie, Qian Zhao, and Deyu Meng. A model-driven deep neural network for single image rain removal. In IEEE CVPR, pages 3103-311 3122, 2020.
[39] Xiaohong Liu, Yongrui Ma, Zhihao Shi, and Jun Chen. Griddehazenet: Attention-based multi-scale network for image dehazing. In IEEE ICCV, pages 7314-7323, 2019.
[40] Hang Dong. Jinshan Pan, Lei Xiang, Zhe Hu, Xinyi Zhang, Fei Wang, and Ming-Hsuan Yang. Multi-scale boosted dehazing network with dense feature fusion. In IEEE CVPR, pages 2157-2167, 2020.
[41] Xu Qin, Zhilin Wang. Yuanchao Bai, Xiaodong Xie, and Huizhu Jia. Ffa-net: Feature fusion attention network for single image dehazing. In AAAI, pages 1 11908 11915, 2020.
[42] Haiyan Wu, Yanyun Qu. Shaohui Lin, Jian Zhou, Ruizhi Qiao. Zhizhong Zhang, Yuan Xie, and Lizhuang Ma. Contrastive learning for compact single image dehazing. In IEEE CVPR, pages 10551-10560, 2021.
[43] Yuda Song. Zhuqing He, Hui Qian, and Xin Du. Vision transformers for single image dehazing. arXiv preprint arXiv: 03883, 2022.
12

---

[44] Xing Liu, Masanori Suganuma. Zhun Sun, and Takayuki Okatani. Dual residual networks leveraging the potential of paired operations for image restoration. In IEEE CVPR, pages 7007-7016, 2019.
[45] Zhi Li, Juan Zhang, Zhijun Fang, Bo Huang, Xiaoyan Jiang, Yongbin Gao, and Jenq-Neng Hwang. Single image snow removal via composition generative adversarial networks. IEEE Access, 7:25016-25025, 2019.
[46] Da-Wei Jaw, Shih-Chia Huang, and Sy- Yen Kuo. Desnowgan: efficient single image snow removal framework using cross-resolution lateral connection and gans. IEEE Transactions on Circuits and Systems for Video Technology, 31(4):132-1133 1350, 2020.
[47] Wei-Ting Chen, Hao-Ye Yu Fang. Cheng-Lin Hsieh, Cheng-Che Tsai, I Chea, Jian-Jiun Ding, Sy-Yen Yen Kuo, et al. All snow removed: Single image desnowing algorithm using hicrarchical dual-tree complex wavelet representation and contradict channel loss. In IEEE ICCV, pages 4196-4290 2021.
[48] Kaihao Zhang, Rongqing Li. Yanjiang Yu. Wenhan Luo, and Changsheng Li. Deep dense multi-scale network for snow removal using semantic and depth priors. IEEE Transactions Image Processing, 30:7419-7431, 2021.
[49] Thailcang Sung and Hyo Jong Lec. Removing snow from single image using a residual frequency module and perceptual ralsgan. IEEE Access, 9:152047-152056, 2021.
[50] Hamidreza Fazlali, Shahram Shirani, Michael Bradford, and Thia Kirubarajan. Single image rain/snow removal using distortion type information. Multimedia Tools and Applications, pages 1-27, 2022.
[51] Liangqiong Qu, Jiandong Tian, Shengfeng He, Yandong Tang, and Rynson WHI Lau. Deshadownet: multi-context embedding deep network for shadow removal. In IEEE CVPR, pages 4067-4075, 2017.
[52] Xiaowei Hu, Yitong Jiang, Chi- Shi-Wing Fu, and Pheng-Ann Heng. Mask saak-shadowgan: Learning to remove shadows from unpaired data. In JEEE ICCV, pages 2472-2481, 2019.
[53] Hieu Le and Dimitris Samaras. Physics-based shadow image decomposition for shadow removal. IEEE Transactions on Pattern Analysis and Machine Intelligence, (01):1-1, 2021.
[54] Hieu Le and Dimitris Samaras. From shadow segmentation to shadow removal. In ECCV, pages 264-281, 2020.
[55] Lan Fu, Changqing Zhou, Qing Guo, Felix Juefei-Xu, Hongkai Yu, Wei Feng, Yang Liu, and Song Wang. Auto-exposure fusion for single-image shadow removal. In IEEE CVPR, pages 10571-10580, 2021.
[56] Jin Wan, Hui Yin. Zhenyao Wu, Xinyi Wu. Zhihao Liu, and Song Wang. Crformer: A cross-region transformer for shadow removal. arXiv preprint 2277v2220..01600, 2022.
[57] Hieu Le and Dimitris Samaras. Shadow removal via shadow image decomposition. In IEEE ICCV, pages 8578 8587, 2019.
[58] Bin Ding, Chengjiang I Long, I Ling Zhang, and Chunxia Xiao. Argan: Attentive recurrent generative adversarial network for shadow detection and removal. In IEEE ICCV, pages 10213-10222, 2019.
[59] Xiaowei Hu, Chi- Biig Fu, Lei Zhu, Jing Qin, and Pheng-Am Ann Heng. Direction- aware spatial context fea- tures for shadow detection and removal. IEEE Transactions on Puttern Analysis and Machine Intelligence. 42(1 1):2755-2808, 2019.
[60] Ling Zhang, Chengjiang Long, Xiaolong Zhang, and Chunxia Xiao. Ris-gan: Explore residual and illumination with generative adversarial networks for shadow removal. In AAAI, pages 12829-12836. 2020.
[61] Xiaodong Cun, Chi-Man Pun, and Cheng Shi. Towards ghast-free shadow removal via dual hicrarchical aggregation network and shadow matting gan. In AAAI, pages 10680-10687, 2020.
[62] Zipei Chen. Chengjiang Long, Ling Zhang, and Chunxia Xiao, Canet: A context-aware network for shadow removal. In IEEE ICCV, pages 4743-4752, 2021.
13

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] Please see the section Limita-
tion in the supplementary material.
(c) Did you discuss any potential negative societal impacts your work? [Yes] Please see
the section of Negative Societal Impacts.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full of assumptions all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as URL)? [Yes] Please see the section of Implementation Details in the supplementary material.
(b) Did you specify all the training details (c.g., g., data splits, hyperparameters, how they were chosen)? [Yes] Please see the section of Implementation Details in the supplementary material.
(c) Did you report error bars (e.g., with respect the random seed after running experi- ments multiple times)? [N/A]
(d) Did you include the total amount of compute and the type of resources used (e.g.. type of GPUs, internal cluster, or cloud provider)? [Yes] Please see the section of Implementation Details in the supplementary material.
4. If you using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any assets either in the supplemental material as URL? [N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you're 're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information offensive content? [N/A]
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid participants and the total amount spent on participant compensation? [N/A]
14


Reviews:
[{"review_id": "yHV12xyKbBf", "reviewer": "Reviewer_4SNd", "summary": "A CNN structure based on the combination of two Conditionally Variational Auto-Encoders (CVAE) is proposed for image rain removal. Two CVAE are used, one for generating hypothesis on rain presence at a pixel and on for the scale of the rain steak. The proposed structure is tested on several database and compared to several others methods. It is also applied separately to snow, haze and shadow removal.  ", "questions": "In the supplementary material, shown maps of rainy pixels have numerous errors on bright objects. May it be possible de comment on this point ? \nOne may notice that the size of the test database is small or very small compared to the size of the learning databases in several cases in the experiments. This may induce a lack of confidence on the proposed results that should be interesting to discuss ?\n ", "limitations": "No societal impact.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The proposed structure is interestingly introduced and seems original. The evaluation seems substantial and show improvements over 5 others methods. It is interesting that the proposed structure can work on other applications such as snow, haze and shadow removal.  ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "u33v0wrZ5s6", "reviewer": "Reviewer_aYKW", "summary": "The paper presents a CVAE(conditional variational auto-encoder) based method of single image rain removal. The key idea is to use the generative network VAE to learn the probability distribution of the pixel-wise rain status P(R|F,Z) and the derainning kernels P(K|F, R, Z), where R, F is the status map and the kernels, respectively. F are the extracted features of the input image, and Z is the latent variables generated by the normal distribution P(Z). For the inference stage, they use the rainy image I to compute F, and randonly generate m Zs. By using F, m Zs, they compute m Rs and consequently m Ks. The final derain kenerl map K is the mean value map of m Ks. The experimental results demostrate the effictiveness of the proposed method on the deraining benchmarks of Rain100H, Rain100L, Rain1400, SPA, the snow removal task dataset of Snow100k, and the haze removal task dataset of ITS&OTS. \n", "questions": "Please provide the clear defination of the function \"conv\" in EQ3, EQ6. If it is a convolution, it should have two varibles but not one. if not, please give the clear explaination. \nPlease explain the \"encoder\", \"condition\", and \"decoder\" operations in the paper. If they are CNN blocks, please decribe them clearly, especially where the mean and standard deviation are computed from by the \"condition\" block ( condition(F) in EQ4,  condition(F, R) in EQ12). \nPlease explain the equations of the information decoupling. Please explain the conv function in Eq6 and fig. 2. Is the conv a fixed numarical fountion? How do you promise the current formulation get correct K of  the input pair?      \n\nPlease provide more explaination about the discriminative networks they used in the comparison experiments. No information is provided in the current draft. ", "limitations": "The authors have not provide discussion about the limitations or failure cases of their work. The authors should include such discussions. In ablation study, more discussions and analysis are needed about the comparison of the proposed CVAE with GAN or other discriminative methods. ", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The proposed method is an extension of [Du et.al. TIP 2020]. Instead of using CVAE to directly estimate P(Y|F,Z) as Du et. al., they formulate the problem as P(R, K| F, Z) and factorize the distribution P(R, K| F, Z) = P(K|R,F,Z) P(R|F, Z), so that they can decouple the latent factors to learn P(R|F,Z) and P(K|R, F, Z), respectively. They experimental results show the effictiveness of the proposed formulation. \n\nHowever, the paper presentation is lack of clarity. There are many places in the method(both formulation and architecture) which are not clear. Especially about information decoupling, the authors should give a more explaination about why they use the corrent formulation of K (EQ6~9), and how it can get correct K as ground truth.   ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "nFOWanRn1Tf", "reviewer": "Reviewer_YErN", "summary": "Paper proposes an image deraining method where they compute pixel status in status space that indicates whether the pixel is belonging to rain or an object.  this information is used in the decoupling layer to construct kernel space for the pixel to derain and recover a clean background.", "questions": "Please refer weaknesses", "limitations": "Please refer weaknesses", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Weaknesses:\n- Can the proposed method handle multiple degradations like images with rain and fog or images with fog and snow.\n- comparison with a few recent papers is missing\n[a] Tu et al., Maxim: Multi-axis mlp for image processing, cvpy 2022\n[b] Valanarasu et al., Transweather: Transformer-based restoration of images degraded by adverse weather conditions, cvpr 2022.\n\n- can authors elaborate or provide pseudo-code for training and testing\n- Proposed methods seem complex to implement, will the code be made public\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}, {"review_id": "WrZIzoaM5jI", "reviewer": "Reviewer_CNK5", "summary": "The paper discusses a method for image rain removal based on two CVAE models. The first CVAE learns the distribution of rain condition (status), conditioned on the image features. The second CVAE learns the distribution of kernels that are used to remove rain effects, conditioned on the rainy status and the image features. During testing, multiple status maps and kernels are repeatedly sampled from the distribution to remove rainy effects from the input image. ", "questions": "Please see the weaknesses. ", "limitations": "Some limitations have been discussed in the s.m. It's unclear to me when the method doesn't work well in comparison to other approaches and why.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\nThe technical details are clear. The idea of using CVAE to learn the distribution of status estimation and the corresponding restoration kernels is novel. The results are strong and the framework seems to work well for not just rain removal but also other image restoration tasks including snow, haze, and shadow removal. \n\nWeaknesses:\n\nMy main concern is that it is unclear to me how the method works. Conceptually, I think we aim to correctly estimate the status maps and the kernels, given the image feature? With the correct status map and kernels then we can remove rainy effects properly. However, the two CVAEs sample several status maps as well as multiple kernels, conditioned on the image features. I don’t understand how these randomly sampled status maps approximate rainy status in the input image. Should an input rainy image be associated with exactly one status map? What does it mean (conceptually) to randomly sample multiple status maps given a single input image? Moreover, I also don’t understand why taking the average of randomly sampled kernels is better than one kernel. While I am aware that the authors have provided quantitative evidence in the s.m. that using these CVAEs is better than using regression models directly, I don’t think the authors have provided proper explanations for the effectiveness of their method.\n\nThe motivation also doesn’t make much sense to me since I can’t connect the motivation to the proposed method. Why the “confusing information of the pixels” can be resolved by sampling multiple pseudo rainy status maps? \n\nThe shadow removal experiment should be conducted on the Adjusted ISTD dataset (shadow image decomposition for shadow removal - ICCV19) since the original ISTD dataset exhibit a major color shift between input and GT images.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept