The Thirty-Fifth AAAI Conference on Artificial Intelligence (AAAI-21)

EfficientDeRain: Learning Pixel-wise Dilation Filtering for High-Efficiency
Single-Image Deraining
Qing Guo1∗ , Jingyang Sun2 * , Felix Juefei-Xu3 , Lei Ma2 †Xiaofei Xie1† , Wei Feng4 , Yang Liu1 , Jianjun Zhao2
1
Nanyang Technological University, Singapore
Kyushu University, Japan 3 Alibaba Group, USA
4
College of Intelligence and Computing, Tianjin University, China
2

PSNR Comparison on Rain100H

Abstract
Single-image deraining is rather challenging due to the unknown rain model. Existing methods often make specific assumptions of the rain model, which can hardly cover many
diverse circumstances in the real world, compelling themto employ complex optimization or progressive refinement.
This, however, significantly affects these methods’ efficiency
and effectiveness for many efficiency-critical applications.
To fill this gap, in this paper, we regard the single-image
deraining as a general image-enhancing problem and originally propose a model-free deraining method, i.e., EfficientDeRain, which is able to process a rainy image within 10 ms(i.e., around 6 ms on average), over 80 times faster than thestate-of-the-art method (i.e., RCDNet), while achieving similar de-rain effects. We first propose the novel pixel-wise dilation filtering. In particular, a rainy image is filtered withthe pixel-wise kernels estimated from a kernel prediction network, by which suitable multi-scale kernels for each pixel
can be efficiently predicted. Then, to eliminate the gap between synthetic and real data, we further propose an effective data augmentation method (i.e., RainMix) that helps
to train network for handling real rainy images. We perform comprehensive evaluation on both synthetic and realworld rainy datasets to demonstrate the effectiveness and efficiency of our method. We release the model and code in
https://github.com/tsingqguo/efficientderain.git.

1

4.8

74.8 144.8

Time (ms)SSIM Comparison on Rain100H

424.8

4.8

74.8 144.8

Time (ms)Real Rainy Image

EfDeRain w/o RainMix (Ours)

RCDNet (CVPR2020)

EfDeRain (Ours)

424.8

Figure 1: Top: Comparison results (i.e., PSNR vs. Time &SSIM vs. Time) on the Rain100H dataset. Down: an example of using EfDeRain, EfDeRain without RainMix and thestate-of-the-art RCDNet (Wang et al. 2020a) to handle a realrainy image. Note that, the time cost of all compared methods are one-by-one evaluated on the same PC.

Introduction

Rain patterns or streaks captured by outdoor vision systems(e.g., stationary image or dynamic video sequence), often
lead to sharp intensity fluctuations in images or videos, causing performance degradation for the visual perception systems (Garg and Nayar 2005, 2007) across different tasks,e.g., pedestrian detection (Wang et al. 2018), object tracking(Li et al. 2018a), semantic segmentation (Chen et al. 2020),
etc. As a common real-world phenomenon, it is almost
mandatory that an all-weather vision system is equippedwith the deraining capability for usage. A deraining method
processes the rain-corrupted image/video data and removes
the rain streaks, with the intention to achieve good image
quality for the downstream vision tasks.
In many real-time applications that are efficiencysensitive and critical, (e.g., vision-based autonomous driving or navigation), being able to perform deraining efficiently on-chip is of great importance. A deraining algorithm
achieving both high efficiency and high performance (e.g., interms of PSNR, SSIM), while remaining at low overhead, isof great importance for practical usage.

* Qing Guo and Jingyang Sun are co-first authors.
†

Lei Ma and Xiaofei Xie are corresponding authors:
ma.lei@acm.org, xfxie@ntu.edu.sg.
Copyright © 2021, Association for the Advancement of Artificial
Intelligence (www.aaai.org). All rights reserved.

1487

(a)

64

128

256

512

512

512

256

(b)

Selectedregions

pixel-wise
single kernel

Predicted
kernels

Derainedregions
c

R1

R2

R3

R4

R5

R6

pixel-wise
multi-daliated kernels

Figure 2: Pipeline of our EfficientDeRain. (a) represents the pixel-wise image filtering introduced in Sec. 3.1 where each pixel
is processed by an exclusive kernel predicted by a kernel prediction network. (b) we further extend this simple structure andpropose the pixel-wise dilation filtering in Sec. 3.2 to handle multi-scale rain streaks. (c) we show that the predicted kernels canbe adapted to different image contents while recovering the object boundary in the original image.
Although we have witnessed the recent progress for deraining, existing methods mostly focus on studying the physical models of rain and background layers, removing rainstreaks via solving an optimization problem and employing
the power of deep learning and some priors, e.g., the rain
having similar local patterns across the image domain (Wanget al. 2020a; Li et al. 2016; Wei et al. 2019; Hu et al. 2019;
Yang et al. 2019). An essential problem, i.e., the efficiencyof the derain method, however, is largely neglected so far,which brings great limitation for real-time applications.

our proposed method follows a single-stage and does not require either iterative optimization or progressive refinement,
thus leading to high-efficiency deraining. Overall, our main
contributions are three-fold:• We propose the pixel-wise dilation filtering for efficientderaining. A rainy image is filtered by the pixel-wise kernels estimated from a kernel prediction network that can
automatically and efficiently predict the suitable multiscale kernels for each pixel.
• To bridge the gap between synthetic and real data, we further propose the RainMix component for simple yet effective data augmentation, which enables us to train networks
for handling real rainy images

In particular, existing methods (see Sec. 2) often heavily rely on various rain-generating assumptions and rainbackground models, whose goal is to revert such a rainadding step in the deraining process, involving heavy iterative optimization and subsequent refinement steps. However, some caveats of theirs cannot be overlooked: ¶ therain model assumptions adopted by many of the algorithms
may be limited, which do not well represent and reflect thereal-world rain patterns. The models based on these rainmodel assumptions may not perform as strongly under the
real-world scenarios; · many of the existing methods arecomputationally expensive, either requiring complex iterative optimization to find the optimal solution, or constructing
the deraining framework in multiple stages with recurrent or
progressive refinement steps involved.

• We demonstrate the advantage of our method on synthetic and real-world rainy datasets, achieving both highperformance and high-efficiency deraining. As shown in
Fig. 1, our method (i.e., EfDeRain) runs around 88 times
faster than the state-of-the-art method RCDNet (Wanget al. 2020a) with similar performance in terms of PSNR
and SSIM. Furthermore, equipped with the RainMix,
EfDeRain can achieve much better visualization result on
the real rainy images than RCDNet.

2

Related Work

Existing deraining algorithms can be largely categorized
into two groups: video-based deraining, e.g., (Kim, Sim, andKim 2015), and single image deraining. In this section, we
discuss several representative DL-based single-image deraining methods, which are most relevant to ours.
In (Wang et al. 2020a), a rain convolutional dictionary
network (RCD-Net) is proposed for single image deraining, where the rain shapes are encoded by exploiting the in-In this work, we approach the single-image deraining
problem from a different angle, aiming to propose a efficientwhile general derain method. First, our proposed method ismodel-free, which makes no assumptions of how the rain
is generated. As we show in the experimental section, rainmodel assumption is somewhat not mandatory for achieving
high-performance deraining, and sometimes such assumptions can even impair the deraining performance. Second,1488

transy

(1

)w

2

te

sheary

4

-w

)w

3

-w

(1-w)w

y

y

zoom

om
zo

(1

w1

rota

zoomy

Algorithm 1: Learning EfficientDeRain via RainMix
Input: KPN(·), fusion Convolution Conv(·), Lossfunction L, Rainy Images I r , Clean Images
I, real rain streak set R, Operation set
O = {rot, shearx/y , transx/y , zoomx/y }.
Output: Pre-trained Network KPN(·) and Conv(·).
1 Function RainMix(R):2
Sample a rain map Rorg ∼ R;
3
Initialize an empty map Rmix ;4
Sample mixing weights(w1 , w2 , w3 , w4 ) ∼ Dirichlet;
5
for i = 1 to 4 do
6Sample operations (o1 , o2 , o3 ) ∼ O;
7
Combine via o12 = o2 o1 and o123 = o3 o2 o1 ;8
Sample o ∼ {o1 , o12 , o123 };9
Rmix + = wi o(Rorg )10
Sample a blending weight w ∼ Beta;
11
return R = wRorg + (1 − w)Rmix ;12 End function;r
13 for i = 1 to |I | do
14
Generate rain map via R = RainMix(R);15
Sample an image pair via (Ir , I) ∼ (I r , I);
16
Sample X ∼ {Ir , I} and Perform Irm = R + X;
17
Predict pixel-wise kernels via K = KPN(Irm );18
Derain via Eq. (4) and get Î = Conv({Îl });19
Calculate Eq. (5) and do back-propagation;
20
Update parameters of KPN(·) and Conv(·);

w)

sx

(1-

n
tra

zoomx

w

Figure 3: An example of RainMix to generate a rainy image.

trinsic convolutional dictionary learning mechanism, leveraging the proximal gradient technique as the optimization
method to seek the optimal solution. RCD-Net is designed
as multiple stages to iteratively solve the deraining problem.
In (Du et al. 2020), a conditional variational image deraining (CVID) network is proposed for draining, which uses
the conditional variational auto-encoder (CVAE) architecture as the backbone. As a variational method, multiple predictions of the derained images from the input rainy image
can be generated by the learned decoder, and the final singleimage output is obtained by integrating these predictions.
In (Yang et al. 2019), a multi-task network to perform joint
rain detection and removal (JORDER) is proposed to solve
the inverse problem of single-image deraining. Contextual
dilated networks are incorporated to extract regional contextual information so that the learned feature can be invariant to rain streaks. To further process rain streaks with various directions and shapes, a recurrent process is adopted toprogressively remove the rain streaks in stages. In (Li et al.
2018b), a recurrent squeeze-and-excitation context aggregation net (RESCAN) is designed for single-image deraining.
Similarly, the entire procedure is also performed in multiple stages, with a memory enabled RNN architecture to aggregate useful information of earlier stages. In (Wang et al.
2019), a spatial attentive network (SPANet) based on a tworound four-directional RNN architecture is proposed, where
three standard residual blocks are used to extract features,and four spatial attentive blocks are used to identify rainstreaks progressively in four stages.

3
3.1

Methodology

Pixel-wise Image Filtering for Deraining

In this part, we propose an image filtering-based method formodel-free deraining. The rain can be regarded as a kind of
degradation that may cause effects such as occlusion, fog,
motion blur, etc. As a result, it is reasonable to use image
filtering methods to handle it, which can be efficient and effective for various degradations. Specifically, we process an
input rainy image Ir ∈ RH×W 1 via pixel-wise filteringÎ = K ~ Ir ,(1)

where Î ∈ RH×W is the estimated derained image, and ~denotes the pixel-wise filtering operation where each pixel2
is processed by its exclusive kernel and K ∈ RH×W ×K
contains the kernels of all pixels. Specifically, when deraining the p-th pixel of Ir , we have its exclusive kernel, i.e., thevector at p-th position of K, which is reshaped and denoted
as Kp ∈ RK×K . We use p as 2D coordinates for a pixel.
Then, we can predict the derained pixel byXÎ(p) =Kp (t)Ir (q),(2)

Overall, these existing methods share some commonalities. In particular, they often heavily rely on a presumed
rain model to develop their algorithms such as (Yang et al.
2019; Li et al. 2018b; Wang et al. 2020a), or require specific a priori domain knowledge of the rain streaks such as(Wang et al. 2019; Du et al. 2020; Fu et al. 2017). Again,
many of these methods are built on a recurrent or progressiveframework, where later stage deraining can help refine theearly stage results such as (Yang et al. 2019; Li et al. 2018b;
Wang et al. 2019, 2020a). However, we believe that thesestrong prerequisites and constraints can potentially hinder
both universality and efficiency of the derain method and its
real-world deployment. In this work, we originally propose
a deraining method from a different perspective, aiming toaddress both issues.

t,q=p+t
K−1
K−1 K−1
where t ranges from (− K−1
2 , − 2 ) to ( 2 , 2 ).
11489

We exemplify our method via the gray-scale image for better understanding. For
color images, we can handle each channel independently.

PSNR Comparison on Rain100H

PSNR Comparison on Rain1400

32.5

32

27.5

27

22.5

4.8

74.8 144.8

424.8

Time (ms)SSIM Comparison on Rain100H
0.90

17

37

5.9

75.9

145.9

845.9

Time (ms)SSIM Comparison on Rain1400

0.95

0.80

74.8 144.8

Time (ms)

424.8

0.89

75.5 145.5

425.5

75.5 145.5

425.5

Time (ms)SSIM Comparison on SPA

0.97
0.96

0.91

0.75

35
5.5

0.98

0.93

0.85

4.8

39

22

17.5

PSNR Comparison on SPA

41

0.95
5.9

75.9

145.9

Time (ms)

845.9

5.5

Time (ms)Figure 4: Comparing EfDeRain with nine baseline methods on Rain100H (Yang et al. 2017, 2019), Rain1400 (Fu et al. 2017),
and the real SPA dataset (Wang et al. 2019) with PSNR vs. Time and SSIM vs. Time plots.
To realize effective deraining with the simple pixelwise filtering, we should consider the following challenges: ¶ how to estimate spatial-variant, scale-variant,and semantic-aware kernels effectively and efficiently. Rainmay cause streak occlusion, fog, and blur, at different positions with different appearances. For example, rain streaks
could exhibit different scales, directions, and transparency
across the image and semantically related to the image contents, e.g., the scene depth (Hu et al. 2019). As a result,the pixel-wise kernels should be adapted to scene information, the spatial and scale variations of rain streaks. Obviously, hand-craft designed kernels can hardly satisfy such requirements. To address this challenge, we propose the multidilated-kernel prediction network in Sec. 3.2, which takes
the rainy image as the input and predict multi-scale kernels for each pixel via the deep neural network (DNN). ·how to train a powerful deraining DNN to bridge the gap
to real rain with synthetic data. Most of the existing deraining DNNs are trained on the synthetic data. However, there
is still a gap between real and synthetic rain. Bridging thisgap is of great importance for real-world applications. We
propose a simple yet effective rain augmentation method,
denoted as RainMix in Sec. 3.3 to reduce such a gap.

3.2

thickness and strength while preserving the object boundary.
We show a deraining example in Fig. 2, where we validate
our method on six representative regions and observe that: ¶Our method can effectively remove the rain steak while recovering the occluded boundary, as shown in the region one(R1) (see Fig. 2 (c)). · The predicted kernels can adapt to
the rain with different strengths. As shown in Fig. 2 (c), from
R2 to R5, the rain strength gradually becomes weak and our
method can remove all trace effectively. Moreover, according to the visualization of predicted kernels, our network canperceive the positions of rain streak. As a result, the predictkernels assign higher weights to non-rainy pixels and lowers ones to rainy pixels, confirming the effectiveness of our
method. ¸ According to the results on R6, our method does
not harm the original boundary and makes it even sharper.
Multi-dilated image filtering and fusion. To enable ourmethod to handle multi-scale streaks without harming theefficiency, we extend each predicted kernel to three scales
via the idea of dilated convolution (Yu and Koltun 2016).
Intuitively, when the rain streak covers a large region of
the image, the large scale kernel can use related pixels far
from the rain for effective deraining. A simple solution is topredict multi-scale kernels directly, leading to extra parameters and time costs. Alternatively, we propose to extend the
pixel-wise filtering in Eq. (1) to pixel-wise dilation filtering(Yu and Koltun 2016) for convolution layer,XÎl (p) =Kp (t)Ir (q),(4)

Learnable Pixel-wise Dilation Filtering

Kernel prediction network. Inspired by recent works on
image denoising (Bako et al. 2017; Mildenhall et al. 2018),
we propose to estimate the pixel-wise kernels K for deraining by taking the rainy image as input,
K = KPN(Ir ),t,q=p+lt(3)

where l is the dilation factor to control the applying rangeof the same filter. In practical, we consider four scales, i.e.,l = 1, 2, 3, 4. With Eq. (4), we get four derained images, i.e.,Î1 , Î2 , Î3 , and Î4 . Then, we use a convolution layer with thesize 3 × 3 to fuse the four images and obtain the final output.

where KPN(·) denotes a UNet-like network and we show thearchitecture in Fig. 2. By offline training on rainy-clean image pairs, the kernel prediction network can predict spatialvariant kernels that adapt to the rain streaks with different
1490

PSNR: 28.14
SSIM: 0.50

PSNR: 30.18
SSIM: 0.75
Input

Case1

Case2
Ground Truth

PSNR: 31.41
SSIM: 0.88

PSNR: 33.70
SSIM: 0.91

EfDeRain

PSNR: 31.31
SSIM: 0.88

PSNR: 33.63
SSIM: 0.91

RCDNet

PSNR: 30.57
SSIM: 0.84

PSNR: 32.49
SSIM: 0.90
PReNet

PSNR: 43.99
SSIM: 0.99

PSNR: 43.47
SSIM: 0.99

PSNR: 41.58
SSIM: 0.98

PSNR: 40.64
SSIM: 0.98

Case2

PSNR: 48.20
SSIM: 0.99

PSNR: 45.28
SSIM: 0.99

PSNR: 44.46
SSIM: 0.99

PSNR: 41.03
SSIM: 0.98

Cas3

PSNR: 47.68
SSIM: 0.99

PSNR: 41.59
SSIM: 0.98

PSNR: 41.68
SSIM: 0.98

Ground Truth

EfDeRain

RCDNet

PReNetFigure 6: Three visualization results of EfDeRain, RCDNet(Wang et al. 2020a) and SPANet (Wang et al. 2019) on thereal-world SPA dataset. The red arrow shows the main difference between EfDeRain and other two methods.

Loss function. We consider two loss functions for training
the network, i.e., L1 and SSIM. Given the derained image,i.e., Î, and the clean image I as ground truth, we have

distribution (i.e., line 4 and 11 in Algorithm 1). Intuitively,
the multiple random processes simulate the diverse rain appearance patterns in the real world. We give an example of
RainMix for generating a rainy image in Figure 3.

(5)

where we fix λ = 0.2 for all experiments.

3.3

Case1

Input

Figure 5: Two visualization results of EfDeRain, RCDNet(Wang et al. 2020a), and PReNet (Ren et al. 2019) on
Rain100H (Case1) and Rain1400 (Case2). We magnify themain difference to highlight the advantages of our method.

L(Î, I) = kÎ − Ik1 − λ SSIM(Î, I)

PSNR: 37.59
SSIM: 0.95

4

RainMix: Bridging the Gap to Real Rain
4.1

How to reduce the gap of synthetic rainy images to real datais still an open problem. Inspired by recent data augmentation methods (Hendrycks et al. 2020), we explore a novel
solution RainMix to address this challenge. Garg and Nayar(2006) conducted a detailed study about the appearance ofrain streaks and constructed a dataset of real rain streaks byconsidering different lighting and viewing conditions. Eventhen, it is still hard to say the collected real rain is exhaustive,
which covers all possible situations in the real world, since
the rain has diverse appearances due to the influences of various natural factors, such as the wind, the light reflection,and refraction, etc. However, it is reasonable to use thesereal rain streaks to generate more rain streaks through transformations that may occur in the real world, e.g., zooming,
shearing, translation, and rotation. RainMix is originally designed based on this such intuition.
We show our RainMix-based learning algorithm in Algorithm 1. At each training iteration, we generate a rain map
via RainMix and add it to the clean or rainy images, to obtaina new rain image for training the kernel prediction networkand the fusion convolution layer. Our RainMix randomly
samples a rain map from the real rain streak dataset (Gargand Nayar 2006) (i.e., line 2 in Algorithm 1) and performs
three transformations on the rain map via randomly sampled and combined operations (i.e., line 5-9 in Algorithm 1).
Finally, the three transformed rain maps are aggregated via
the weights from Dirichlet distribution and further blended
with the original sampled rain map via the weight from Beta

Experiments

SetupsDatasets. To comprehensively validate and evaluate ourmethod, we conduct the comparison and analysis experiments on 4 popular datasets, including Rain100H (Yanget al. 2017, 2019), Rain1400 (Fu et al. 2017) synthetic
dataset, the recent proposed SPA real rain dataset (Wanget al. 2019), and the real Raindrop dataset (Qian et al. 2018).
Metrics. We employ the commonly used peak signal tonoise ratio (PSNR) and structural similarity (SSIM) as thequantitative evaluation metric for all datasets. In general, thelarger PSNR and SSIM indicate better deraining results.
Baselines. We perform a large-scale evaluation to compare with a total of 14 (=9+5) state-of-the-art deraining
methods, i.e., 9 baselines for the derain streak task (removing rain streak), including rain convolutional dictionary network (RCDNet) (Wang et al. 2020a), conditional variational
image deraining (CVID) (Du et al. 2020), joint rain detection
and removing (JORDERE) (Yang et al. 2019), spatial attentive deraining method (SPANet) (Wang et al. 2019), progressive image deraining network (PReNet) (Ren et al. 2019),
semi-supervised transfer learning for rain removal (SIRR)(Wei et al. 2019), recurrent squeeze-and-excitation context
aggregation net (RESCAN) (Li et al. 2018b), deep detail
network (Fu et al. 2017), and Clear (Fu et al. 2017). Furthermore, for deraindrop task (i.e., removing train drop) on
the Raindrop dataset (Qian et al. 2018), we compare another1491

Figure 7: Three visualization results of four variants of EfDeRain: v1 denotes the pixel-wise filtering based on KPN without
dilation; v2 is the pixel-wise dilation filtering for deraining. v1 and v2 are trained via the L1 loss. v3 and v4 have the samestructure with v2 but are trained via L1 and SSIM loss functions, i.e., Eq. (5). In addition, the v4 uses RainMix for training.

5 state-of-the-art methods (Li et al. 2019) as baselines, including GMM (Li et al. 2016), JORDER (Yang et al. 2017),
DDN (Fu et al. 2017), CGAN (Zhang, Sindagi, and Patel
2019), DID-MDN (Zhang and Patel 2018), and DeRaindrop(Qian et al. 2018). Note that, the time cost of all methods are
one-by-one evaluated on the same PC with the Intel Xeon
CPU (E5-1650) and NVIDIA Quadro P6000 GPU.

the baseline methods, EfDeRain removes the rain streak
while recovering the original details well. In Case2, RCDNet and PReNet remove or destroy the original streak-likedetails, e.g., the fine lines. In contrast, our method preserves
these details while removing the rain effectively, demonstrating that our method could understand the scene betterand predict kernels for different pixels.

4.2

4.3

Comparison on Rain100H&1400 Dataset

Comparison on Real-world SPA Rain Dataset

We further compare our method with 8 baseline methods on
the SPA dataset (Wang et al. 2019), where the rainy image isreal and its ground truth is obtained by human labeling andmulti-frame fusion. As shown in Fig. 4, our method achieves
almost the same PSNR and SSIM with the top method, i.e.,
RCDNet, and outperforms all other baselines while runningover 71 times more efficient than RCDNet.
We also visually compare our method with RCDNet andSPANet in Fig. 6. Obviously, all the results demonstrate that
our method can handle various rain traces with different patterns and achieves better visualization results than RCDNet
and SPANet. In particular, both RCDNet and SPANet fail toremove the wider rain streak in Case2 while our method successfully handles all rain streaks and the obtained derained
image is almost the same with the ground truth.

We compare our method with the 9 baseline methods in
Fig. 4 on Rain100H and Rain1400 datasets. In general, ourmethod, i.e., EfDeRain, achieves the lowest time cost whileobtaining comparable PSNR or SSIM with the top methods on both datasets. More specifically, on the challenging
Rain100H dataset where the rain streaks cover a large portion of the image, we observe that EfDeRain achieves almost
the same PSNR and SSIM with the rank1 method, i.e., RCDNet (Wang et al. 2020a), while being 88 times faster. Compared with other methods, e.g., JORDERE (Yang et al. 2019)
and CVID (Du et al. 2020), our method exhibits clear advantages on both recovery quality and efficiency. For example,
EfDeRain achieves 11. 4% relative PSNR improvement over
CVID while running over 50 times faster. Similarly, in termsof the Rain1400 dataset, our method still achieves the lowest time cost while maintaining the competitive PSNR and
SSIM in line with the state-of-the-art methods, e.g., PReNet
and RCDNet. The main reason is likely to be that the rainstreak of Rain1400 is slighter than Rain100H and a lot of theregions are not covered by the rain. Meanwhile, as shown in
the Fig. 2, our method can not only remove rain streaks but
enhance the object boundary, leading to the negative score of
PSNR and SSIM that evaluate the recovery quality insteadof the enhancement capability.
We further compare the visualization results of EfDeRain
with the state-of-the-art baseline methods, i.e., RCDNet and
PReNet, in Fig. 5 and observe that: ¶ In Case1, EfDeRain
can remove the rain streaks more effectively than the othertwo methods since there are clear rain traces on derained images, i.e., the white streaks near the nose. · Compared with

4.4

Comparison on Real-world Raindrop Dataset

Besides the rainy streak image dataset, we also compare
our method on the deraindrop task to show the generalization capability of our method. We train our network on the
Raindrop dataset (Qian et al. 2018) and compare it with 6
state-of-the-art baseline methods. In particular, the method
DeRaindrop (Qian et al. 2018) is specifically designed forthis problem, where the region of the raindrop is perceived
by an attentive-recurrent network. As shown in Table 1,
our method without changing any architectures or hyperparameters, achieves the second-best results with competitive SSIM to DeRaindrop, and outperforms all other deraining methods, demonstrating both the effectiveness and generality of our method.
1492

PSNR
SSIM

GMM

JORDER

DDN

CGAN

DID-MDN

DeRaindrop

EfDeRaiin

24.58
0.780827.52
0.823925.23
0.836621.35
0.730624.76
0.793031.57
0.902328.48
0.8971

Table 1: Results on Raindrop dataset (Qian et al. 2018). The baseline results are reported by (Li et al. 2019) and (Qian et al.
2018). We highlight the top three results.
EfDeRainw/o dilation(v1)+dilation(v2)+ssim loss(v3)+RainMix(v4)

PSNR
SSIM
Time (ms)30.12
0.88344.41

30.27
0.89186.05

30.35
0.89705.93

31.02
0.9079
5.97

Table 2: Ablation study on Rain100H. We consider 4 variants of EfDeRain: v1 is the pixel-wise filtering based on KPN without
dilation structure; v2 is the pixel-wise dilation filtering for deraining. v1 and v2 are trained based on the L1 loss. v3 and v4 havethe same structure with v2 but are trained via L1 and SSIM loss functions, i.e., Eq. (5). The v4 uses RainMix for training.

4.5

5

Ablation Study

Conclusions

In this paper, we propose a novel model-free deraining
method denoted as EfficientDeRain. Our method can not
only achieve the significantly high performance but runs
over 80 times more efficient than the state-of-the-art method.
Two major contributions benefit to the results: First, we proposed and designed the novel pixel-wise dilation filtering
where each pixel is filtered by multi-scale kernels estimatedfrom a pre-trained kernel prediction network. Second, we
propose a simple yet effective data augmentation method fortraining the network, i.e., RainMix, bridging the gap between
synthesis data and real data. Finally, we perform a largescale evaluation to comprehensively validate our method on
popular and challenging synthesis datasets, i.e., Rain100H
and Rain1400, and real-world datasets, i.e., SPA and Raindrop, all of which demonstrate the advantage of our method
in terms of both efficiency and deraining quality.
In future, we will study the effects of deraining to othercomputer vision tasks, e.g., object segmentation (Guo et al.
2018, 2017b) and object tracking (Guo et al. 2020c,a,
2017a), with the state-of-the-art DNN testing works (Xieet al. 2019a; Du et al. 2019; Xie et al. 2019b; Ma et al.
2018a,b, 2019). We also would like to study the singleimage deraining from the view of adversarial attack methods, e.g., (Guo et al. 2020b; Wang et al. 2020b; Cheng et al.
2020a; Gao et al. 2020; Cheng et al. 2020b; Tian et al. 2020).

We validate the effectiveness of our main contributions,
i.e., pixel-wise dilation filtering (Sec. 3.2) and RainMix(Sec. 3.3), on Rain100H dataset. We also discuss the effectiveness of the loss function in Eq. (5). More specifically,
we develop four variants of our method: the first version(v1) is the pixel-wise filtering based on KPN without thedilation structure; the second version (v2, i.e., Sec. 3.2) denotes the pixel-wise dilation filtering for deraining. v1 andv2 are trained based on the L1 loss. v3 and v4 share the samestructure with v2 but are trained via L1 and SSIM loss functions, i.e., Eq. (5). In addition, the final version (v4) uses
RainMix in Sec. 3.3 for training. As shown in Table 2, the
PSNR and SSIM of the 4 versions gradually increase andreach the highest performance on our final version with the
pixel-wise dilation filtering trained via the RainMix as wellas L1 and SSIM loss functions. This demonstrates that all of
our main contributions are beneficial for effective deraining.
Moreover, we also analyze the time cost of the four variants
and observe that: the proposed dilation filtering only leads
to around 1.5 ms additional costs.
We further validate the advantages of contributions via theRain100H visualization results in Fig. 7 and observe that: ¶In general, our final version can not only remove the heavy
rain streak but recovering the original details, thus achieving the highest PSNR and SSIM scores. · When comparing
the EfDeRain-v1 with v2 (i.e., dilation-enhanced v1), we canconclude that the dilation structure clearly facilitates to remove more rain streaks. For example, in Case 1, the rain
traces in v1 have been obviously suppressed in v2. ¸ The
SSIM loss function helps recover the details but enhances
the rain streak. For example, in Case 1, with the SSIM loss
function(i.e., EfDeRain-v3), the boundary of the sun in v3
becomes much sharper than that in v2 and v1. However, therain streak boundary becomes obvious as well. We observe
similar results in other cases. ¹ By combing dilation structure, SSIM loss function, and RainMix, our final version,i.e., EfDeRain-v4, is able to remove the heavy rain effectively while recovering the details very well.

Acknowledgments
This work was supported in part by JSPS KAKENHI Grant
No.20H04168, 19K24348, 19H04086, and JST-Mirai Program Grant No.JPMJMI18BB and JPMJMI20B8, Japan. It
was also supported in part by Singapore National Cybersecurity R&D Program No. NRF2018NCR-NCR005-0001,
National Satellite of Excellence in Trustworthy Software
System No.NRF2018NCR-NSOE003-0001, NRF Investigatorship No. NRFI06-2020-0022, and the National Natural
Science Foundation of China (NSFC) under Grant 62072334
and Grant 61671325. We gratefully acknowledge the support of NVIDIA AI Tech Center (NVAITC) to our research.
1493

References

Guo, Q.; Sun, S.; Ren, X.; Dong, F.; Gao, B. Z.; and Feng, W.
2018. Frequency-tuned active contour model. Neurocomputing
275: 2307–2316.

Bako, S.; Vogels, T.; McWilliams, B.; Meyer, M.; Novák, J.;
Harvill, A.; Sen, P.; DeRose, T.; and Rousselle, F. 2017. Kernelpredicting convolutional networks for denoising Monte Carlo renderings. ACM Trans. Graph. 36(4): 97:1–97:14.

Guo, Q.; Xie, X.; Juefei-Xu, F.; Ma, L.; Li, Z.; Xue, W.; Feng,W.; and Liu, Y. 2020c. SPARK: Spatial-aware online incremental attack against visual tracking. In Proceedings of the European
Conference on Computer Vision (ECCV).

Chen, Y.-C.; Lin, Y.-Y.; Yang, M.-H.; and Huang, J.-B. 2020. Show,
Match and Segment: Joint Weakly Supervised Learning of Semantic Matching and Object Co-segmentation. IEEE Transactions on
Pattern Analysis and Machine Intelligence .

Hendrycks, D.; Mu, N.; Cubuk, E. D.; Zoph, B.; Gilmer, J.; andLakshminarayanan, B. 2020. AugMix: A Simple Data Processing Method to Improve Robustness and Uncertainty. International
Conference on Learning Representations .

Cheng, Y.; Guo, Q.; Juefei-Xu, F.; Xie, X.; Lin, S.-W.; Lin,W.; Feng, W.; and Liu, Y. 2020a.
Pasadena: Perceptually
Aware and Stealthy Adversarial Denoise Attack. arXiv preprint
arXiv:2007.07097 .

Hu, X.; Fu, C.-W.; Zhu, L.; and Heng, P.-A. 2019. Depthattentional features for single-image rain removal. In Proceedingsof the IEEE Conference on Computer Vision and Pattern Recognition, 8022–8031.

Cheng, Y.; Juefei-Xu, F.; Guo, Q.; Fu, H.; Xie, X.; Lin, S.-W.; Lin,W.; and Liu, Y. 2020b. Adversarial Exposure Attack on Diabetic
Retinopathy Imagery. arXiv preprint arXiv:2009.09231 .

Kim, J.-H.; Sim, J.-Y.; and Kim, C.-S. 2015. Video deraining anddesnowing using temporal correlation and low-rank matrix completion. IEEE Transactions on Image Processing 24(9): 2658–
2670.

Du, X.; Xie, X.; Li, Y.; Ma, L.; Liu, Y.; and Zhao, J. 2019. Deepstellar: Model-based quantitative analysis of stateful deep learning
systems. In Proceedings of the 2019 27th ACM Joint Meeting on
European Software Engineering Conference and Symposium on theFoundations of Software Engineering, 477–487.

Li, F.; Tian, C.; Zuo, W.; Zhang, L.; and Yang, M.-H. 2018a. Learning Spatial-Temporal Regularized Correlation Filters for Visual
Tracking. In CVPR, 4904–4913.

Du, Y.; Xu, J.; Zhen, X.; Cheng, M.-M.; and Shao, L. 2020. Conditional Variational Image Deraining. IEEE Transactions on Image
Processing .

Li, S.; Araujo, I. B.; Ren, W.; Wang, Z.; Tokuda, E. K.; Junior,
R. H.; Cesar-Junior, R.; Zhang, J.; Guo, X.; and Cao, X. 2019. Single Image Deraining: A Comprehensive Benchmark Analysis. Proceedings of the IEEE Conference on Computer Vision and Pattern
Recognition .

Fu, X.; Huang, J.; Ding, X.; Liao, Y.; and Paisley, J. 2017. Clearing
the Skies: A Deep Network Architecture for Single-Image Rain
Removal. IEEE Transactions on Image Processing 26(6): 2944–
2956.

Li, X.; Wu, J.; Lin, Z.; Liu, H.; and Zha, H. 2018b. Recurrent
squeeze-and-excitation context aggregation net for single image
deraining. In Proceedings of the European Conference on Computer Vision (ECCV), 254–269.

Fu, X.; Huang, J.; Zeng, D.; Huang, Y.; Ding, X.; and Paisley, J.
2017. Removing rain from single images via a deep detail network.
In Proceedings of the IEEE Conference on Computer Vision and
Pattern Recognition, 3855–3863.

Li, Y.; Tan, R. T.; Guo, X.; Lu, J.; and Brown, M. S. 2016. Rain
Streak Removal Using Layer Priors. In Proceedings of the IEEE
International Conference on Computer Vision, 2736–2744.

Gao, R.; ; Guo, Q.; Juefei-Xu, F.; Yu, H.; Ren, X.; Feng, W.; andWang, S. 2020. Making Images Undiscoverable from Co-Saliency
Detection. arXiv preprint arXiv:2009.09258 .

Ma, L.; Juefei-Xu, F.; Sun, J.; Chen, C.; Su, T.; Zhang, F.; Xue, M.;
Li, B.; Li, L.; Liu, Y.; Zhao, J.; and Wang, Y. 2018a. DeepGauge:
Multi-Granularity Testing Criteria for Deep Learning Systems. In
The 33rd IEEE/ACM International Conference on Automated Software Engineering (ASE).

Garg, K.; and Nayar, S. K. 2005. When does a camera seerain? In Tenth IEEE International Conference on Computer Vision(ICCV’05) Volume 1, volume 2, 1067–1074. IEEE.
Garg, K.; and Nayar, S. K. 2006. Photorealistic Rendering of Rain
Streaks. In ACM SIGGRAPH 2006 Papers, SIGGRAPH ’06, 996–
1002.

Ma, L.; Juefei-Xu, F.; Xue, M.; Li, B.; Li, L.; Liu, Y.; and Zhao,
J. 2019. DeepCT: Tomographic Combinatorial Testing for Deep
Learning Systems. Proceedings of the IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER).

Garg, K.; and Nayar, S. K. 2007. Vision and rain. International
Journal of Computer Vision 75(1): 3–27.
Guo, Q.; Feng, W.; Zhou, C.; Pun, C.-M.; and Wu, B. 2017a.
Structure-regularized compressive tracking with online data-driven
sampling. IEEE Transactions on Image Processing 26(12): 5692–
5705.

Ma, L.; Zhang, F.; Sun, J.; Xue, M.; Li, B.; Juefei-Xu, F.; Xie, C.;
Li, L.; Liu, Y.; Zhao, J.; and Wang, Y. 2018b. DeepMutation: Mutation Testing of Deep Learning Systems. In The 29th IEEE International Symposium on Software Reliability Engineering (ISSRE).

Guo, Q.; Han, R.; Feng, W.; Chen, Z.; and Wan, L. 2020a. Selective
spatial regularization by reinforcement learned decision making for
object tracking. IEEE Transactions on Image Processing 29: 2999–
3013.

Mildenhall, B.; Barron, J. T.; Chen, J.; Sharlet, D.; Ng, R.; and Carroll, R. 2018. Burst Denoising with Kernel Prediction Networks.
In CVPR, 2502–2510.

Guo, Q.; Juefei-Xu, F.; Xie, X.; Ma, L.; Wang, J.; Yu, B.; Feng,W.; and Liu, Y. 2020b. Watch out! Motion is Blurring the Visionof Your Deep Neural Networks. Advances in Neural Information
Processing Systems 33.

Qian, R.; Tan, R. T.; Yang, W.; Su, J.; and Liu, J. 2018. Attentive Generative Adversarial Network for Raindrop Removal Froma Single Image. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.

Guo, Q.; Sun, S.; Dong, F.; Feng, W.; Gao, B. Z.; and Ma, S.
2017b. Frequency-tuned ACM for biomedical image segmentation. In 2017 IEEE International Conference on Acoustics, Speechand Signal Processing (ICASSP), 821–825. IEEE.

Ren, D.; Zuo, W.; Hu, Q.; Zhu, P.; and Meng, D. 2019. Progressive Image Deraining Networks: A Better and Simpler Baseline.
In Proceedings of the IEEE Conference on Computer Vision and
Pattern Recognition.

1494

Tian, B.; Guo, Q.; Juefei-Xu, F.; Chan, W.; Cheng, Y.; Li, X.; Xie,X.; and Qin, S. 2020. Bias Field Poses a Threat to DNN-based
X-Ray Recognition. arXiv preprint arXiv:2009.09247 .
Wang, H.; Xie, Q.; Zhao, Q.; and Meng, D. 2020a. A Model-driven
Deep Neural Network for Single Image Rain Removal. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern
Recognition, 3103–3112.
Wang, R.; Juefei-Xu, F.; Xie, X.; Ma, L.; Huang, Y.; and Liu, Y.
2020b. Amora: Black-box adversarial morphing attack. In ACM
Multimedia Conference (ACMMM).
Wang, T.; Yang, X.; Xu, K.; Chen, S.; Zhang, Q.; and Lau, R. W.
2019. Spatial attentive single-image deraining with a high quality real rain dataset. In Proceedings of the IEEE Conference on
Computer Vision and Pattern Recognition, 12270–12279.
Wang, X.; Xiao, T.; Jiang, Y.; Shao, S.; Sun, J.; and Shen, C. 2018.
Repulsion Loss: Detecting Pedestrians in a Crowd. In IEEE/CVF
Conference on Computer Vision and Pattern Recognition, 7774–
7783.
Wei, W.; Meng, D.; Zhao, Q.; Xu, Z.; and Wu, Y. 2019. SemiSupervised Transfer Learning for Image Rain Removal. In Proceedings of the IEEE International Conference on Computer Vision.
Xie, X.; Ma, L.; Juefei-Xu, F.; Xue, M.; Chen, H.; Liu, Y.; Zhao,
J.; Li, B.; Yin, J.; and See, S. 2019a. DeepHunter: A CoverageGuided Fuzz Testing Framework for Deep Neural Networks. In
ACM SIGSOFT International Symposium on Software Testing andAnalysis (ISSTA).
Xie, X.; Ma, L.; Wang, H.; Li, Y.; Liu, Y.; and Li, X. 2019b. DiffChaser: Detecting Disagreements for Deep Neural Networks. In
IJCAI, 5772–5778.
Yang, W.; Tan, R. T.; Feng, J.; Guo, Z.; Yan, S.; and Liu, J. 2019.
Joint rain detection and removal from a single image with contextualized deep networks. IEEE transactions on pattern analysis andmachine intelligence 42(6): 1377–1393.
Yang, W.; Tan, R. T.; Feng, J.; Liu, J.; Guo, Z.; and Yan, S. 2017.
Deep Joint Rain Detection and Removal from a Single Image. In
CVPR, 1685–1694.
Yu, F.; and Koltun, V. 2016. Multi-Scale Context Aggregation by
Dilated Convolutions. In International Conference on Learning
Representations.
Zhang, H.; and Patel, V. M. 2018. Density-Aware Single Image Deraining Using a Multi-stream Dense Network. In 2018 IEEE/CVF
Conference on Computer Vision and Pattern Recognition, 695–
704.
Zhang, H.; Sindagi, V.; and Patel, V. M. 2019. Image De-raining
Using a Conditional Generative Adversarial Network. IEEE Transactions on Circuits and Systems for Video Technology 1–1.

1495