RaidaR: A Rich Annotated Image Dataset of Rainy Street Scenes
Wallace Michel Pinto Lira1
Fenggen Yu1
Arezou Fatemi1 *Jiongchao Jin1,2 *Biao Leng2
Rui Ma3,1 †Ali Mahdavi-Amiri1
Hao Zhang1
1
2
3
Simon Fraser University
Beihang University
Jilin University(a) A rainy street scene(b) Image with droplet distortion(c) Semantic segmentation of (b)(d) Instance segmentation of (b)Figure 1: RaidaR is a new annotated dataset to support autonomous driving research. It contains the largest number of rainy street scene
images (58,542) to date — 5,000 of them provide semantic segmentations and 3,658 with object instance segmentations.

Abstract
We introduce RaidaR, a rich annotated image datasetof rainy street scenes, to support autonomous driving research. The new dataset contains the largest number ofrainy images (58,542) to date, 5,000 of which provide semantic segmentations and 3,658 provide object instancesegmentations. The RaidaR images cover a wide rangeof realistic rain-induced artifacts, including fog, droplets,
and road reflections, which can effectively augment existing street scene datasets to improve data-driven machine
perception during rainy weather. To facilitate efficient annotation of a large volume of images, we develop a semiautomatic scheme combining manual segmentation and an
automated processing akin to cross validation, resulting in
10-20 fold reduction on annotation time. We demonstrate
the utility of our new dataset by showing how data augmentation with RaidaR can elevate the accuracy of existing
segmentation algorithms. We also present a novel unpaired
image-to-image translation algorithm for adding/removing
rain artifacts, which directly benefits from RaidaR.

1. Introduction
Autonomous driving under adversarial weather conditions is challenging due to sensors introducing image artifacts and quality degradation [57] that consequently hin∗ Jiongchao Jin and Arezou Fatemi contribute equally to this work.
† Rui Ma is the corresponding author.

der the performance of visual perception algorithms [46, 29,
33, 2, 62]. The most prominent such weather conditions israin. Images captured during rainy weather can be blurry,with partial coverage and non-linear distortion caused byrain streaks or droplets; see Figure 1(b). Even before orafter the rain, gloomy lighting conditions can create difficulties for color perception and delineation, and intensive
use of vehicle headlights and street reflections due to water accumulation near night times, as shown in Figure 1(a),are problematic as well. Machine perception tasks criticalto autonomous driving such as semantic and instance segmentation [62, 49, 3, 7, 20, 61, 32, 51, 15, 31] are mostnegatively impacted by these artifacts from rainy scenes.
In this paper, we introduce RaidaR, a new dataset that isrich in providing street scene images under rainy weather,and it comes with annotations in the form of both semantic
and object instance segmentations; see Figures 1(c-d). Our
goal is to provide sufficient and diverse rainy scene data
to augment existing datasets to improve the performanceof data-driven autonomous driving algorithms. Specifically,
RaidaR represents the largest such dataset to date, consisting of 58,542 rainy images, with 5,000 of them annotatedwith semantic segmentation and 3,658 with instance segmentation. In addition, 4,085 sunny images were also annotated with semantic segmentations.
To acquire RaidaR, we employed a roof-mounted camera platform to collect RGB videos under sunny and rainy
weather conditions in Metro Vancouver, Canada. Among a
total of 77,437 frames extracted from the collected videos,
there are 18,895 sunny and 58,542 rainy images on whichhuman faces and license plates had been identified andName#Lab/#Total #Rainy #SSeg #ISeg
Raincouver [50]326/54K
326
326
0
BDD [55]
10K/120M
5.8K
253
253
nuImages [5]*1.9K/18K
1.3K
5858
ACDC [42]
4K/4K
1K
1K
0
Cityscapes [11]5K/20K
000
KITTI [13]400/15K
000
0
00Apolloscape [16] 147K/147K
RID [23]
2.5K/2.5K
2.5K
00RaidaR
9K/77K 58,542 5,000 3,658
Table 1: Comparing the availability of rainy samples, and their annotations, from well-known street scene image datasets to RaidaR.
#Lab/#Total is the ratio between the number of labelled images(rain or no rain) and the total number of images in the datasets;#Rainy is the number of rainy images; #SSeg is the number ofrainy images with semantic segmentations; and #ISeg is the number of rainy images with instance segmentations. * Only front
camera frames are considered. Weather labels were not originallyavailable, the values shown here are manually curated.

blurred to respect privacy. The captured rainy images covervarious characteristics of rainy weather conditions including fog, road reflection, water droplets, etc. Figure 1 showssome sample images from RaidaR.
In Table 1, we list a set of known datasets. Our main goal
is to include datasets providing real (not synthetic) images
with rainy scenes accompanied with GT segmentations. For
reference, we also added the best known large-scale datasets
including KITTI, Cityscapes, and Apolloscape. We focus
on reporting stats related to rainy images and their annotations. As we can see, the BDD dataset [55], the largest
in terms of total number of images, has the closest numberof rainy images compared to RaidaR, but RaidaR has 10times as many. Only 253 rainy images in BDD have fullframe semantic segmentation, while RaidaR offers thousands. On the other hand, Raincouver [50] was devoted torainy weather, but it is quite small (326 images in total) andits semantic segmentations only emcompass four object categories. In contrast, RaidaR offers 5,000 semantically segmented rainy images on 19 object categories.
Another feature of our new dataset is that more thanhalf (about 35K) of the rainy images have varying degrees of image distortion caused by the presence of water droplets on the camera lens. Data-driven de-raining
and droplet removal have been studied by several recent
works [38, 36, 37, 56], where they mainly work with synthesized images composed using small droplets that were
either produced by spraying water on a transparent shieldin front of the camera lens or by synthesizing droplets
considering their physical properties. In contrast, droplettempered images in RaidaR were acquired under natural
conditions mimicking realistic autonomous driving scenarios. Furthermore, a significant number of these images havebeen annotated. Algorithms trained on such data should
generalize much better on real test data.

To facilitate efficient annotation of a large volume ofimages in RaidaR, we develop a semi-automatic technique
combining manual segmentation and an automated processing akin to cross validation. Specifically, we merge the results from four state-of-the-art segmentation networks withdifferent weights to assess the reliability of each pixel label, where manual intervention is called only when there isa large inconsistency. On average, this approach results in a
10-20 fold speedup over completely manual segmentation.
Our paper makes the following contributions:• We introduce the largest dataset of rainy street scene
images to date, to support research in self-driving. The
images cover a wide range of realistic rain-induced artifacts, including fog, droplets, and road reflections.
The dataset contains thousands of rainy images withsemantic and instance segmentations.
• We develop an efficient semi-automatic scheme forsegmentation annotation, based on cross validation, toreduce manual labeling efforts.
• We demonstrate the utility of our new dataset by showing how data augmentation with RaidaR can elevate
the accuracy of existing segmentation algorithms.
• We present a novel unpaired image-to-image translation algorithm, based on a masked version of GANHopper [25], for adding/removing rain artifacts, which
directly benefits from our annotated dataset.

2. Related Work and Datasets
Here we discuss road datasets for training data-driven
approaches for autonomous vehicles and emphasize theirdifferences in terms of weathering artifacts (e.g., rain) andsemantic/instance segmentation. We then provide a brief
review of road segmentation tools and techniques.

2.1. Road Datasets
There are many available datasets to train autonomous
driving systems [11, 55, 54, 44, 42, 16]. Methods capable of synthesizing weather artifacts have been proposed
along with efforts to capture natural scenes by driving on
the streets at different times and weather conditions.
Datasets to train autonomous cars may also carry various modes and information about the roads. For instance,
they may be composed of imagery datasets to capture visual contents on the roads, along with Lidar datasets that arerich sources of depth and geometry information [27, 53, 10].
Here, we provide an overview of different synthetic and realroad datasets with semantic and instance segmentation.
Synthesizing road datasets can be very useful since they
are flexible in producing desired scenes that are difficult
to capture by driving on the streets. Frameworks such as
CARLA [12] have been designed to generate road data fortraining autonomous cars. SYNTHIA [40], made via Unity,
provides approximately 200K samples, while GTA5 [39]uses footage from the Grand Theft Auto 5 computer gameto generate 25K such frames. Even though data synthesis isa great way to obtain a large amount of reliable segmented
data, it has some limitations as they often fail to capture thecomplex nature of weather and road-specific phenomena.
This may lead to training networks that are not robust in
real-world applications. For instance, these works may failto reproduce realistic droplets on the camera/windshield orroad reflections. Another way for simulating raining effects is to spray water on some transparent shield placed in
front of the camera [36, 37]. However, data captured in thisway is still unlike real-world rainy images which containovercast clouds, mist, and road reflections. Therefore, someefforts have been devoted to collect datasets under various
weather conditions such as Canadian Adverse Driving Conditions Dataset [35], Waymo open dataset [47], Lyft [19],A* 3D [34], Nuscenes [5], etc. However, there is no semantic segmentation ground truth accompanied with thesedatasets. The Rain in Driving (RID) dataset [23] provides
2,496 real rainy images annotated with bounding boxes forselected traffic objects such as car, person, and bicycle.
Hence, it can only be used for evaluating detection insteadof segmentation algorithms.
Datasets with semantic segmentation ground truths exist
but the size of rainy images in these datasets is not considerable. One of the earliest attempts to collect weathered datasets is Raincouver [50] which provides 30 minutesof video recorded in Vancouver in different weather conditions (mostly rainy). This dataset, however, only provides
326 annotated images in which only four classes of objects
are annotated (road, person, vehicle, and other). While thisis a great attempt, Raincouver does not include all the categories necessary for training an autonomous car such as
traffic lights, sidewalks, etc. In CamVid [4], only about 700
images are manually annotated with per-pixel semantic labels. Cityscapes [11], which is the most well-known roaddataset, contains 20K images with coarse and 5K images
with per-pixel semantic annotation. However, most samples
from Cityscapes are captured under normal weather conditions with no rain artifacts. However, there are weatherbased augmentation techniques that have been applied on
Cityscapes dataset, namely Weather Cityscapes [14], andFoggy Cityscapes [43], that employ synthetic rain or fog effects. The main advantages of using such approaches are
the extendibility of the method to other similar datasets andthe possibility of reusing the expensive pixel level semantic
and instance segmentation annotations for generating multimodal samples.
BDD [55] is one of the most complete road datasets
which contains more than 1,100 hours of videos. However,among 10K images with segmentation ground truth, only

253 of them have raining artifacts. Appolloscape [16], another great effort that provides various data forms such as
semantic point cloud, per-pixel semantics, etc. has considered collecting scenes with challenging weather conditions(such as rainy) as a future work. WildDash [58] provides 4Kof diverse images and their semantic segmentation groundtruth, but this dataset is meant to be used as a benchmark forpanoptic, semantic, and instance segmentation. They havenot provided explicit weather classification of their dataset
but after manually curating the dataset, we determined thatit contains only 785 rainy scenes. The recent ACDC dataset[42] provides 4,006 images with high-quality semantic segmentation and the images are evenly distributed between
fog, nighttime, rain, and snow. Although ACDC is valuablefor training and testing semantic segmentation methods onadverse visual conditions, it does not provide as many rainy
images as RaidaR, nor the instance segmentation on images.
Therefore, despite the existence of real (non-synthetic)semantically segmented datasets, they are not as richin terms of rainy images as RaidaR. Consequently, networks trained mainly on normal weather conditions perform
poorly on rainy scenes. Table 1 summarizes some datasetswith ground truth segmentation and compares them withRaidaR. As apparent in the table, RaidaR has the highest
number of rainy images that are accompanied by groundtruth semantic segmentation. In Section 4.1, we show that
adding RaidaR for training segmentation networks will significantly improve the results.

2.2. Segmentation Techniques
Semantic and instance segmentation are key algorithms
when a vision system deals with inference (e.g., self-drivingcars [18]). Great efforts have been devoted to segment road
scenes using deep neural networks [30, 18, 64]. Most ofthese networks are supervised therefore various tools havebeen proposed to help annotators indicate the right label ofa pixel [4, 28]. However, manually labeling all the pixels is
tedious and time consuming.
Semi-automatic techniques have been introduced to easeand speed up this process. Mimicking the process of manual
annotation, a polygon prediction is introduced in [6] where
the polygon vertices are identified and manually adjusted
on a cropped bounding box around a desired object. This
work is later improved by designing a new CNN encoder architecture and employing reinforcement learning and graphneural networks to elevate the accuracy and resolution of theresults [1]. Curve-GCN [24] is the next attempt to improvethese techniques by propagating messages via a graph convolutional network (GCN) to predict location adjustmentsof nodes in an iterative process. However, in these methods, an initial bounding box is needed for each object andthe final results need to undergo a manual adjustment of thepolygon’s vertices to capture fine details (see Section 4).
With similar motivation to reduce the load of manual annotation, Vpred [64] exploits video prediction models’ ability to predict future labels. A joint propagation strategy isalso employed to reduce misalignment in synthesized segmented images. Vpred produces high-quality results (close
to ground truth) for many images especially when the image
is not contaminated with weathering artifacts. However, it isunable to produce reasonable results on rainy scenes when
it is trained on datasets with limited rainy instances. To produce our ground truth, we use Vpred as one of the main
deciders on the label of each pixel as it is one of the mostreliable road segmentation networks. We also cross validate its results against three other recent networks to makesure that a produced label is accurate (see Section 3). For
accurately separating instance segments or when these four
networks are not unanimous, we ask for manual interventions. We show that this method significantly reduces thelabor work involved in segmentation and produces close to
ground truth results (Section 4).

3. RaidaR Dataset
3.1. Collection and Cleaning
Our dataset is composed of various road types including mountains with natural scenery, downtown with traffics,
high-ways, etc. We have also tried to collect images forthe same routes under both sunny and rainy weather conditions to encourage consistency. We mounted our camera on
the roof of a sedan car with a small box around it to stoprain droplets blocking the lens too quickly. The employedcamera was Logitech Ultra HD Webcam with 90 degree extended view recording 30 frames per second. After recording, we manually removed rainy images heavily blockedby droplets and sunny images with massive over-exposures.
After this step, our resulting dataset contains 34,951 rainy
images with small droplets, 23,591 rainy images without
droplets and 18,895 sunny images. The resolution of each
image is 1920*1080. To respect the privacy of people andvehicles appearing in our collected data, we detected humanfaces [22] and license plates [45] and performed a normalized box filter to blur them.

3.2. Ground Truth Generation
To generate our ground truth, we perform a cross validation mechanism to obtain the correct labels for each pixel.
Specifically, we segment each image with four state-of-theart segmentation networks and combine the segmentation
labels of those networks to identify areas for which we were
not able to find reliable labels. The uncertain regions arethen submitted to a manual intervention step to annotate
their right labels. Finally, all other automatically labeledareas are inspected to confirm their accuracy. This process

significantly accelerates the ground truth labeling and reduces the manual work. Based on our labeling process experiment in Section 4, it is 10 to 20 times faster to use ourcross-validation based process than directly labeling all the
pixels on original images.
Cross Validation. To automatically generate ground truth
for semantic segmentation on RaidaR, we use four segmentation networks: Vpred [64], Accel [17], PSP [59] andPSA [60] to generate the initial semantic masks. As different methods may predict different labels for each pixel, wecross validate the labels produced by each method by combining them with different weights and generating a confidence score for all candidate labels of each pixel. Formally, for each semantic mask with label ℓ, we calculate theℓweighted value Si,jfor each pixel (i, j) as the confidence
score of label ℓ:  \label {eq:conf_score} S_{i,j}^\ell = \sum _{k\ in\ \Pi } w_{k}M_{k}^\ell (i,j). 

(1)

Here, Π = {Vpred, Accel, PSP, PSA} and wk is the weightof kth segmentation method; Mkℓ (i, j) is per-pixel binary
mask value (1 if label ℓ has been predicted for pixel (i, j)
by method k and 0 otherwise).
We assign the label with the largest confidence score toeach pixel. As different wk configurations may lead to different results, we discuss how to set proper wk in Section
4.3. We treat pixels with a confidence score higher than athreshold α as reliable and others are treated as unreliable
since they received contradictory labels from the networks.
We set a relatively high threshold α = 0.7 to ensure thecorrectness of cross-validated labels and we highlight theunreliable pixels for manual intervention.
Manual Intervention. After finding the pixels that needmanual intervention, our in-house team annotated the labels. To ease the data annotation process, we offered themasks with regions of uncertain pixels and made a simple
annotation tool. Using this strategy, annotators only needed
to manually label ≈ 24.8% of an image. Moreover, we
asked users to inspect the labels that are automatically generated by the cross validation and correct them in case offalse predictions. Each person needs to spend 5-10 minutes
to fully label one image depending on the size and complexity of the unreliable regions. Based on our experiments,
manually labeling an entire image takes 45-90 minutes.
Instance Annotation. Following the format of BDD andCityscapes, we provide a distinct segment for every vehicle
and pedestrian as they have higher priorities in autonomous
driving. To produce instance segmentation masks, we used
our ground truth semantic segmentation and assigned different colors to detached pedestrian and vehicle segments.
Then, our in-house team manually distinguished instanceswith overlaps.

Test
Train
RaidaRCS
CS + RaidaR
CS + RaidaR + Syn
BDD
BDD + RaidaR
BDD + RaidaR + Syn

CS[64]/[48]0.582/0.603
0.834/0.849
0.714/0.791
0.769/0.813
0.538/0.571
0.603/0.618
0.612/0.615

BDD[64]/[48]0.519/0.612
0.477/0.513
0.547/0.589
0.593/0.631
0.596/0.621
0.638/0.652
0.644/0.654

BDD rainy[64]/[48]0.564/0.612
0.542/0.578
0.582/0.663
0.625/0.672
0.651/0.672
0.676/0.706
0.678/0.714

RaidaR[64]/[48]0.715/0.736
0.703/0.724
0.712/0.734
0.719/0.741
0.622/0.673
0.681/0.725
0.719/0.731

CS[21]/[8]0.318/0.324
0.415/0.407
0.421/0.409
0.434/0.412
0.334/0.315
0.351/0.338
0.366/0.323

BDD[21]/[8]0.303/0.288
0.321/0.297
0.361/0.337
0.369/0.340
0.371/0.362
0.384/0.369
0.397/0.370

BDD rainy[21]/[8]0.398/0.375
0.286/0.278
0.354/0.332
0.366/0.348
0.386/0.370
0.395/0.378
0.400/0.387

RaidaR[21]/[8]0.402/0.391
0.301/0.282
0.373/0.319
0.385/0.351
0.362/0.338
0.392/0.384
0.408/0.393

Table 2: Left: mIoU (averaged by class) for semantic segmentation using Vpred [64] and HMSA [48] on cross-dataset training and testing.
Right: AP for instance segmentation using PointRend [21] and TensorMask [8]. CS stands for Cityscapes.

Statistics. Our 5,000 ground truth rainy images are composed of 1,738 images with droplets, 76 foggy, and 1,214
captured at night that are semantically segmented by 19 labels following the naming convention of Cityscapes. Thisshows the diversity of RaidaR for different raining scenarios. Overall, the semantic masks in RaidaR include 37,126
traffic signs, 25,894 cars, 15,162 vegetation, 12,387 traffic
lights, 9,175 sidewalks, 5,638 roads, 2,015 pedestrians, etc.
In addition, there are 84,602 and 3,977 instance masks forvehicle and pedestrian, respectively.

4. Experiments
In this section, we conduct both quantitative and qualitative experiments to show how challenging our RaidaR
dataset is for the visual perception tasks and its value as acomplement to existing datasets. First, we compare the semantic and instance segmentation results on various crossdataset training and testing configurations. Then, we verify the effectiveness and efficiency of our cross-validation
based label generation scheme for generating the groundtruth semantic labels. We also apply a de-raining methodon our dataset and demonstrate that natural raining artifacts
are not easy to remove and that training a segmentation network using such data is more effective than using a current
de-raining method to obtain proper segmentation.

4.1. Cross-dataset Training and Testing
Semantic Segmentation. The rainy images in our dataset
can raise significant challenges for existing computer vision algorithms. To better position our dataset among theexisting ones, e.g., for the semantic segmentation task, we
compare the results of Vpred segmentation model [64] in
multiple cross-dataset experiments. Moreover, we conduct
a separate experiment using HMSA (Hierarchical MultiScale Attention) [48], the SOTA open source semantic segmentation model on the Cityscapes Benchmark. As HMSA
is not used in our cross-validation based ground truth generation step, comparing the results from HMSA provides amore fair comparison about how well the SOTA segmentation algorithms perform on the experimented datasets.

The standard class-averaged metric Intersection-overUnion (mIoU) values on Cityscapes (CS) [11], BDD [55],and our RaidaR are reported in Table 2 (left) based on various cross-dataset training and testing configurations. Notethat for RaidaR, we use the 5,000 annotated rainy images
and split them by 7:1:2 for training, validation and testing. To verify whether our dataset can indeed complement
the existing dataset such as BDD which also contains rainy
images, we trained new models on the combined training
dataset (BDD + RaidaR). Moreover, we combined BDD +RaidaR with a new synthetic dataset which contains 500
synthetic sunny and 500 synthetic rainy images generated
from RaidaR by our masked image-to-image translation(Section 5) to see whether the performance can be further
boosted. For each model, we also test their performance on
the 253 rainy images extracted from BDD (i.e., BDD rainy).
From Table 2, we can see the models trained on original
non-combined datasets generally perform well on their own
testing dataset. For cross-testing, e.g., the model trained on
CS and tested on BDD, the mIoUs drop significantly which
may be caused by the domain shifts between the datasets.
Similarly, the model trained on RaidaR alone does not perform well on CS or BDD, which are composed of almost exclusively sunny images. On the other hand, the CS-trainedmodel performs better on RaidaR than BDD since BDD ismore diverse and difficult. When combining RaidaR withother datasets such as BDD, performance can be generally improved since RaidaR can complement the existingdatasets for better performance on rainy images. Furthermore, the results on BDD rainy verify RaidaR’s positive
contribution on rainy images. One exception is the CStrained model achieves better performance on CS comparing to the models trained on CS + RaidaR etc. This isalso reasonable since CS only contains sunny images while
adding RaidaR may not help.
Finally, after adding synthetic images into BDD +RaidaR, the performance is generally improved. In fact,
our masked image-to-image translation can be considered
as an effective data augmentation to enhance the existingdatasets. Figure 2 and 3 verify the above observations about
the combined datasets with results obtained using HMSA.

(a) BDD input image(b) HSMA (BDD)(c) HSMA (BDD+RaidaR)(d) HSMA (BDD+RaidaR+Syn)(e) Ground truth

Figure 2: Qualitative comparisons of HMSA trained on different training configurations and tested on BDD. Highlighted regions by theyellow boxes show that after adding RaidaR or RaidaR + Syn, the model can produce finer details (e.g., traffic signs, trees).

(a) RaidaR input image(b) HSMA (RaidaR)(c) HSMA (BDD+RaidaR)(d) HSMA (BDD+RaidaR+Syn)(e) Ground truth

Figure 3: Qualitative comparisons of HMSA trained on different training configurations and tested on RaidaR. Highlighted regions by
the yellow boxes show that the model can produce satisfactory results on rainy images when trained on RaidaR, but inferior results whensimply combined with BDD. After adding the synthetic images (Syn), the model can produce improved results.

Instance Segmentation. To further verify the usefulness of RaidaR and our synthetic data, we also conduct
instance segmentation experiments following the similarcross-dataset training and testing configurations for semantic segmentation. Specifically, we perform PointRend [21]and TensorMask [8], two SOTA instance segmentation algorithms implemented in Detectron2 [52], and calculate
Average Precision (AP) on vehicles and pedestrians. From
Table 2 (right), similar improvements of the AP values from
PointRend and TensorMask can be observed when RaidaRand synthetic images are added to the training data.
Training Details. All of our models have been trained using the open source code available online. For semantic
segmentation, we used available models of Vpred [64] andHMSA [48] which are pre-trained on Cityscapes, and continued training them separately on different datasets, i.e.,
RaidaR, CS + RaidaR, CS + RaidaR+ Syn, BDD, BDD+ RaidaR, BDD + RaidaR + Syn. For instance segmentation, we used available code and pre-trained models on
Detectron2 [52] for PointRend [21] and TensorMask [8],and fine-tuned the networks on different datasets. Specifically, for Cityscapes, we directly used the pre-trained models for both semantic and instance segmentation. For BDD,
we used their train/val/test split with 7,000, 1,000, 2,000
images and trained the models for another about 20 epochs.
RaidaR (3500 out of 5000 rainy images for training) or Syn(500 synthetic sunny and 500 synthetic rainy images) are
combined with CS or BDD and used to fine-tune the models for about 20 epochs.
Overall, the quantitative and qualitative results confirm
our dataset can contribute to existing datasets for increasing their semantic and instance segmentation performanceon rainy images. Note that a simple combination of somedatasets may not be the optimal way to use their fullstrength. We leave more sophisticated explorations of how
our dataset can further contribute to or benefit from other

Test
RaidaR (De-rain) RaidaR
Train
CS
0.705
0.701
RaidaR (De-rain)
0.712
0.707
0.713
0.716
RaidaR
Table 3: mIoU (averaged by class) for Vpred trained and tested on
Cityscapes (CS), de-rained RaidaR, and RaidaR.

datasets to future work.

4.2. Effects of Rain Artifacts for Segmentation.
To evaluate how raining artifacts complicate perceptiontasks (e.g., segmentation), we train Vpred on RaidaR and its
de-rained version. To de-rain RaidaR, we utilize MPRNet[56] on both training and testing sets of RaidaR. We thentrain Vpred on Cityscape, RaidaR (De-rain) and RaidaR.
From Table 3, we can observe that training on Cityscapeand testing on RaidaR produces the worst outcome while
training on RaidaR and testing on RaidaR produces the best
one. This demonstrates that simply de-raining images is
not effective to obtain good segmentation results and ourRaidaR helps the network learn a more robust segmentation
model over images with raining artifacts.

4.3. Cross-Validation based Label Generation
Effectiveness. Our cross-validation based ground truth
generation takes the advantages of existing segmentation algorithms to accelerate the image labeling process. To verify
its effectiveness, we quantitatively evaluated the percentageof pixels labeled as reliable by the automatic cross validation step with different weights wk (Equation 1) on a selection of 2,000 images. We found that using 0.4, 0.3, 0.2, 0.1
for Vpred, Accel, PSP and PSA can generate more reliable
labels than other weight combinations, i.e., 75.2% of the
pixels in an image comparing to 57.1% which uses equal
weights for each method.

(a) RaidaR input image(c) Corss validated masks(b) Vpred’s result(d) Ground truth

Figure 4: Having an input image (a), Vpred alone produces a segmentation (b) that needs improvements. The labels are cross validated with three other segmentation networks (c) where the inconsistent labels are identified by white pixels. Manual intervention
is performed to determine the labels of these pixels and slightly
tweak the other ones to obtain the ground truth (d).

Note that finding the optimal weight setting is not ourgoal. Instead, we mainly use cross validation to reducethe manual effort of annotating the large number of easyto-predict pixels. Using the proposed weight setting, only
24.8% of pixels are left to be manually annotated. In Figure 4, we show segmentation results of only using Vpred,the cross-validation approach and the final ground truth label after manual intervention. It can be observed that thecross-validation approach can predict a reliable segmentation for most parts of the images, leaving some difficult or
error-prone pixels for manual intervention.
To further verify that our cross-validation based approach can generate ground truth similar to the full manual
annotation, we randomly select 200 images from Cityscapesand compare our generated ground truth with theirs. On
average, less than 2.1% of pixels in our ground truth are
different from the manually annotated ground truth from
Cityscapes. Note that the comparison didn’t include the“void” label which relates to the windshield or hood of theego-vehicle as our images are captured using a top-mounted
camera and don’t have such regions.
Efficiency. To quantitatively evaluate how our crossvalidation method increases the efficiency of the groundtruth label annotation, we conduct a preliminary experiment
to compare the labeling time of our process with the time ofannotating from scratch. We asked five users to label 10 images on all original pixels from scratch. We then let thoseusers label the same images again using our cross-validation
based labeling process. After calculating the time spent oneach image, on average, it takes about 8.6 minutes for labeling each image using our labeling process and 53.8 minutesby manually labeling all pixels using the same tool.
Comparison with Polygon-RNN++.

We compare our(a) Input(b) Polygon-RNN++(c) Ours

Figure 5: Given an input rainy image (a), after initializing bounding boxes, Polygon-RNN++ generates (b). As apparent, our mechanism (automatically) generates better details around the droplet
boundary in (c). Note that we keep the droplet region in (c)to indicate that this region needs manual intervention. No postprocessing is applied on either mask.

ground truth generation mechanism with Polygon-RNN++[1] which is a SOTA semi-automatic annotation technique(Curve-GCN’s code [24] was not available at the time ofthis submission). In Polygon-RNN++, users are required todetermine a bounding box to generate a mask for each object (Figure 5 (b)). Ours instead directly generates masks
for all classes without manual intervention for each single
class (Figure 5 (c)). In addition, Polygon-RNN++ does not
fully reproduce thin parts or fine details around the boundary of objects, especially when the image contains weathering artifacts (e.g., fog, droplets). Therefore, it still needs
considerable manual adjustments. In contrast, ours is basedon current state-of-art segmentation algorithms which canaccurately detect edges and details. Based on a small experiment on 10 images, generating a ground truth segmentation
takes about 25 minutes per image using Polygon-RNN++
while our mechanism only needs 9 minutes.

5. Masked Image-to-Image Translation
While applications such as training segmentation or detection networks for autonomous driving systems can directly benefit from our dataset, there are approaches thatuse semantic segmentation as prior to better perform a task,such as image-to-image translation [9, 41]. Here, we benefit from RaidaR segmentation for masked image-to-imagetranslation to convert rainy images to sunny and vice-versa.
This can be useful for improving the quality of images withweathering artifacts or data augmentation for training visual
perception algorithms (Section 4).
Since it is impossible to capture the same road scene under two different weather conditions (e.g., sunny and rainy),
we treat the problem as unpaired image translation. Among
available methods, CycleGAN [63] and UNIT [26] serve as
strong baselines. However, both methods cannot transfer
weathering artifacts successfully as they are only aware of
the color distribution of pixels (Figure 6 (b), (c)). Meanwhile, in a weathered road scene, two semantically different pixels (e.g., road and sky) may attain very similar colors(e.g., gray). To resolve this problem, we introduce a masked
CycleGAN in which semantic masks are fed to CycleGAN

(a) Input(a) Input(b) CycleGAN(c) UNIT(d) Masked CycleGAN

Figure 6: Inputs (a) and the results of CycleGAN (b), Unit (c), andour masked CycleGAN (d). As apparent, our approach is more
successful in preserving the colors of the input image. The colorsof some regions (highlighted by yellow box) are not preserved in
CycleGAN and UNIT due to the lack of semantic awareness.

u

v
...

||u-u’||u’segmentation masks
U: rainy(a)

V: sunny

U: rainy(b)

V: sunny

Figure 7: In CycleGAN, an image is translated from domain U(rainy) to V (sunny) (a). By returning to domain U in a cycle, thesame images should be produced through a reconstruction loss. To
instruct the network about pixels’ semantics, segmentation masks
are provided as extra inputs (b).

100% cat

75% cat, 25% dog

50% cat, 50% dog

25% cat, 75% dog

100% dog

100% rainy

75% rainy, 25% sunny

50% rainy, 50% sunny

25% rainy, 75% sunny

100% sunny

Figure 8: GANHopper translates an image from one domain toanother in several hops (Top). We extend it to masked GANHopper by adding semantic masks for better transferring weathering
artifacts (Bottom).

as additional inputs (Figure 7). To account for semantic segments in CycleGAN, we modify its cycle loss by taking themasks into consideration and assigning higher weights tosemantic regions whose colors should be preserved.
Masked GANHopper. To further improve the translation
results, we extend the masked CycleGAN to GANHopper [25] to benefit from a simple yet effective idea to perform the translation in multiple steps called hops (Figure8). For masked GANHopper, we provide the masks to eachhop and modify the loss function accordingly. Benefited(b) Masked CycleGAN(c) Masked GANHopper

Figure 9: Both masked CycleGAN and masked GANHopper
can produce plausible image translation. In comparison, masked
GANHopper can better preserve the masked region and generate
more delicate artifacts such as shadows in sunny scenes (Top) andreflections in rainy scenes (Bottom).

from RaidaR segmentation, both masked CycleGAN andmasked GANHopper can produce visually plausible results
for translating between sunny and rainy images. Comparing to masked CycleGAN, masked GANHopper can perform better on preserving the masked region and produce
relatively more delicate weathering artifacts as in Figure 9.
Limitation and Discussion. Note that in this section, we
showed an application on how our RaidaR segmentation canbe used as a prior for masked image-to-image translation,
while not targeting at producing perfect image translation
results nor most effective data augmentation for training.
Despite its plausible performance, using masked GANHopper may still produce some artifacts that cause difficulties
for detecting far and fuzzy objects (e.g., a distant car withalready very low confidence score). On the other hand, by
taking the masked region into account when image generation, the translated image can preserve the textures for pixels with important labels (e.g., traffic lights). As more labels
are passed as inputs, masked GANHopper will take slightly
more inference time comparing to the non-masked version.

6. Conclusion and Future Work
We have provided a rainy rich dataset along with its semantic and instance segmentation ground truth. We alsopresented a novel method to generate ground truth segmentation that can reduce the time and work-load needed for labeling pixels. We showed that our dataset can be beneficial
in training segmentation networks especially when the input
is a rainy image. Our dataset can be advantageous in applications including training autonomous driving systems,
droplet removal and image-to-image translation for which
we provided a novel method that provides visually pleasing
results that can be used to enhance the accuracy of segmentation networks. For future work, we intend to extend thevariety and richness of our dataset in terms of road scenes(e.g., different cities), weathering artifacts (e.g., snow) andannotation types (e.g., lane markings).

References[1] David Acuna, Huan Ling, Amlan Kar, and Sanja Fidler. Efficient interactive annotation of segmentation datasets withpolygon-rnn++. In IEEE CVPR, 2018. 3, 7
[2] R. Blin, S. Ainouz, S. Canu, and F. Meriaudeau. Road scenes
analysis in adverse weather conditions by polarizationencoded images and adapted deep learning. In 2019 IEEE
Intelligent Transportation Systems Conference (ITSC), pages
27–32, 2019. 1[3] Hermann Blum, Paul-Edouard Sarlin, Juan Nieto, Roland
Siegwart, and Cesar Cadena. Fishyscapes: A benchmark
for safe semantic segmentation in autonomous driving. In
Proceedings of the IEEE/CVF International Conference on
Computer Vision Workshops, pages 0–0, 2019. 1[4] Gabriel J Brostow, Julien Fauqueur, and Roberto Cipolla.
Semantic object classes in video: A high-definition groundtruth database. Pattern Recognition Letters, 30(2), 2009. 3[5] Holger Caesar, Varun Bankiti, Alex H Lang, Sourabh Vora,
Venice Erin Liong, et al. nuscenes: A multimodal dataset forautonomous driving. In IEEE CVPR, 2020. 2, 3
[6] Lluis Castrejon, Kaustav Kundu, Raquel Urtasun, and Sanja
Fidler. Annotating object instances with a polygon-rnn. In
IEEE CVPR, 2017. 3[7] Bi-ke Chen, Chen Gong, and Jian Yang. Importance-aware
semantic segmentation for autonomous driving system. In
IJCAI, pages 1504–1510, 2017. 1[8] Xinlei Chen, Ross Girshick, Kaiming He, and Piotr Dollár.
Tensormask: A foundation for dense object segmentation,
2019. 5, 6
[9] Anoop Cherian and Alan Sullivan. Sem-gan: Semanticallyconsistent image-to-image translation. In IEEE WACV.
IEEE, 2019. 7[10] Yukyung Choi, Namil Kim, Soonmin Hwang, Kibaek Park,
Jae Shin Yoon, Kyounghwan An, and In So Kweon. Kaist
multi-spectral day/night data set for autonomous and assisted
driving. IEEE Transactions on Intelligent Transportation
Systems, 19(3), 2018. 2[11] Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo
Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe
Franke, Stefan Roth, and Bernt Schiele. The cityscapes
dataset for semantic urban scene understanding. In IEEE
CVPR, 2016. 2, 3, 5[12] Alexey Dosovitskiy, German Ros, Felipe Codevilla, Antonio
Lopez, and Vladlen Koltun. CARLA: An open urban driving
simulator. In Proceedings of the 1st Annual Conference on
Robot Learning, 2017. 2[13] Andreas Geiger, Philip Lenz, and Raquel Urtasun. Are we
ready for autonomous driving? the kitti vision benchmark
suite. In IEEE CVPR, 2012. 2[14] Shirsendu Halder, Jean-François Lalonde, and Raoul
Charette. Physics-based rendering for improving robustness
to rain. pages 10202–10211, 10 2019. 3[15] Namdar Homayounfar, Yuwen Xiong, Justin Liang, WeiChiu Ma, and Raquel Urtasun. Levelset r-cnn: A deep variational method for instance segmentation. In European Conference on Computer Vision, pages 555–571. Springer, 2020.
1[16] Xinyu Huang, Peng Wang, Xinjing Cheng, Dingfu Zhou,
Qichuan Geng, and Ruigang Yang. The apolloscape open
dataset for autonomous driving and its application. IEEE
Trans. Pattern Analysis & Machine Intelligence, 42(10),2019. 2, 3
[17] Samvit Jain, Xin Wang, and Joseph E Gonzalez. Accel: A
corrective fusion network for efficient semantic segmentation on video. In IEEE CVPR, 2019. 4[18] Joel Janai, Fatma Güney, Aseem Behl, Andreas Geiger,
et al. Computer vision for autonomous vehicles: Problems,
datasets and state of the art. Foundations and Trends® inComputer Graphics and Vision, 12(1–3), 2020. 3[19] R. Kesten, M. Usman, J. Houston, T. Pandya, K. Nadhamuni, A. Ferreira, M. Yuan, B. Low, A. Jain, P. Ondruska, S. Omari, S. Shah, A. Kulkarni, A. Kazakova, C.
Tao, L. Platinsky, W. Jiang, and V. Shet. Lyft level 5 perception dataset 2020. https://level5.lyft.com/
dataset/, 2019. 3[20] Alexander Kirillov, Kaiming He, Ross Girshick, Carsten
Rother, and Piotr Dollár. Panoptic segmentation. In IEEE
CVPR, 2019. 1[21] Alexander Kirillov, Yuxin Wu, Kaiming He, and Ross Girshick. Pointrend: Image segmentation as rendering, 2020. 5,
6[22] Jian Li, Yabiao Wang, Changan Wang, Ying Tai, Jianjun
Qian, Jian Yang, Chengjie Wang, Jilin Li, and Feiyue Huang.
Dsfd: dual shot face detector. In IEEE CVPR, 2019. 4[23] Siyuan Li, Iago Breno Araujo, Wenqi Ren, ZhangyangWang, Eric K Tokuda, Roberto Hirata Junior, Roberto CesarJunior, Jiawan Zhang, Xiaojie Guo, and Xiaochun Cao. Single image deraining: A comprehensive benchmark analysis.
In IEEE CVPR, 2019. 2, 3
[24] Huan Ling, Jun Gao, Amlan Kar, Wenzheng Chen, and Sanja
Fidler. Fast interactive object annotation with curve-gcn. In
IEEE CVPR, 2019. 3, 7
[25] Wallace Lira, Johannes Merz, Daniel Ritchies, Daniel
Cohen-Or, and Hao Zhang. GANHopper: Multi-hop gan for
unsupervised image-to-image translation. In ECCV, 2020. 2,
8[26] Ming-Yu Liu, Thomas Breuel, and Jan Kautz. Unsupervised
image-to-image translation networks. In Advances in Neural
Information Processing Systems, 2017. 7[27] Will Maddern, Geoff Pascoe, Chris Linegar, and Paul Newman. 1 Year, 1000km: The Oxford RobotCar Dataset. The
International Journal of Robotics Research (IJRR), 36(1):3–
15, 2017. 2[28] Damien
Mandrioli.
Semantic
segmentation
editor.
https://github.com/
Hitachi-Automotive-And-Industry-Lab/
semantic-segmentation-editor, 2020. 3[29] A. Mehra, M. Mandal, P. Narang, and V. Chamola. Reviewnet: A fast and resource optimized network for enablingsafe autonomous driving in hazy weather conditions. IEEE
Transactions on Intelligent Transportation Systems, pages 1–
11, 2020. 1[30] Shervin Minaee, Yuri Boykov, Fatih Porikli, Antonio Plaza,
Nasser Kehtarnavaz, and Demetri Terzopoulos. Image seg-mentation using deep learning: A survey. arXiv preprint
arXiv:2001.05566, 2020. 3[31] Eslam Mohamed, Mahmoud Ewaisha, Mennatullah Siam,
Hazem Rashed, Senthil Yogamani, and Ahmad El-Sallab.
Instancemotseg: Real-time instance motion segmentation
for autonomous driving. arXiv preprint arXiv:2008.07008,
2020. 1[32] Davy Neven, Bert De Brabandere, Stamatios Georgoulis,
Marc Proesmans, and Luc Van Gool. Fast scene understanding for autonomous driving.
arXiv preprint
arXiv:1708.02550, 2017. 1[33] T. Ort, I. Gilitschenski, and D. Rus. Autonomous navigation in inclement weather based on a localizing ground
penetrating radar. IEEE Robotics and Automation Letters,
5(2):3267–3274, 2020. 1[34] Quang-Hieu Pham, Pierre Sevestre, Ramanpreet Singh
Pahwa, Huijing Zhan, Chun Ho Pang, Yuda Chen, Armin
Mustafa, Vijay Chandrasekhar, and Jie Lin. A* 3d dataset:Towards autonomous driving in challenging environments.
In Proc. IEEE Int. Conf. on Robotics & Automation. 3[35] Matthew Pitropov, Danson Garcia, Jason Rebello, Michael
Smart, et al. Canadian adverse driving conditions dataset.
arXiv preprint arXiv:2001.10117, 2020. 3[36] Horia Porav, Tom Bruls, and Paul Newman. I can see clearly
now: Image restoration via de-raining. In Proc. IEEE Int.
Conf. on Robotics & Automation, 2019. 2, 3[37] Horia Porav, Valentina-Nicoleta Musat, Tom Bruls, and Paul
Newman. Rainy screens: Collecting rainy datasets, indoors,
2020. 2, 3[38] Rui Qian, Robby T Tan, Wenhan Yang, Jiajun Su, and Jiaying Liu. Attentive generative adversarial network for raindrop removal from a single image. In IEEE CVPR, 2018.
2[39] Stephan R. Richter, Vibhav Vineet, Stefan Roth, and Vladlen
Koltun. Playing for data: Ground truth from computer
games. In Bastian Leibe, Jiri Matas, Nicu Sebe, and Max
Welling, editors, Computer Vision – ECCV 2016, pages 102–
118, Cham, 2016. Springer International Publishing. 3[40] German Ros, Laura Sellart, Joanna Materzynska, David
Vazquez, and Antonio M. Lopez. The synthia dataset: Alarge collection of synthetic images for semantic segmentation of urban scenes. In IEEE CVPR, June 2016. 3[41] Pravakar Roy, Nicolai Häni, and Volkan Isler. Semanticsaware image to image translation and domain transfer. arXiv
preprint arXiv:1904.02203, 2019. 7[42] Christos Sakaridis, Dengxin Dai, and Luc Van Gool. Acdc:
The adverse conditions dataset with correspondences for semantic driving scene understanding, 2021. 2, 3
[43] Christos Sakaridis, Dengxin Dai, Simon Hecker, and Luc
Van Gool. Model adaptation with synthetic and real data forsemantic dense foggy scene understanding. In ECCV, 2018.
3[44] Christos Sakaridis, Dengxin Dai, and Luc Van Gool. Guided
curriculum model adaptation and uncertainty-aware evaluation for semantic nighttime image segmentation. pages
7373–7382, 10 2019. 2[45] S. M. Silva and C. R. Jung. License plate detection andrecognition in unconstrained scenarios. In ECCV, Sep 2018.
4[46] D. Steinhauser, P. Held, B. Thöresz, and T. Brandmeier. Towards safe autonomous driving: Challenges of pedestrian detection in rain with automotive radar. In 2020 17th European
Radar Conference (EuRAD), pages 409–412, 2021. 1[47] Pei Sun, Henrik Kretzschmar, Xerxes Dotiwalla, Aurelien
Chouard, Vijaysai Patnaik, Paul Tsui, James Guo, Yin Zhou,
Yuning Chai, Benjamin Caine, et al. Scalability in perception
for autonomous driving: An open dataset benchmark. arXiv
preprint arXiv:1912.04838, 2019. 3[48] Andrew Tao, Karan Sapra, and Bryan Catanzaro. Hierarchical multi-scale attention for semantic segmentation. arXiv
preprint arXiv:2005.10821, 2020. 5, 6
[49] Michael Treml, José Arjona-Medina, Thomas Unterthiner,
Rupesh Durgesh, Felix Friedmann, Peter Schuberth, Andreas Mayr, Martin Heusel, Markus Hofmarcher, Michael
Widrich, et al. Speeding up semantic segmentation for autonomous driving. In MLITS, NIPS Workshop, volume 2,
2016. 1[50] Frederick Tung, Jianhui Chen, Lili Meng, and James J Little.
The raincouver scene parsing benchmark for self-driving inadverse weather and at night. IEEE Robotics and Automation
Letters, 2(4), 2017. 2, 3[51] Kelvin Wong, Shenlong Wang, Mengye Ren, Ming Liang,and Raquel Urtasun. Identifying unknown instances for autonomous driving. In Conference on Robot Learning, pages
384–393. PMLR, 2020. 1[52] Yuxin Wu, Alexander Kirillov, Francisco Massa, Wan-Yen
Lo, and Ross Girshick. Detectron2. https://github.
com/facebookresearch/detectron2, 2019. 6[53] Zhi Yan, Li Sun, Tomas Krajnik, and Yassine Ruichek. EU
long-term dataset with multiple sensors for autonomous driving. In Proceedings of the 2020 IEEE/RSJ International
Conference on Intelligent Robots and Systems (IROS), 2020.
2[54] Senthil Yogamani, Ciarán Hughes, Jonathan Horgan, Ganesh
Sistu, Padraig Varley, Derek O’Dea, Michal Uricár, Stefan Milz, Martin Simon, Karl Amende, et al. Woodscape:A multi-task, multi-camera fisheye dataset for autonomous
driving. arXiv preprint arXiv:1905.01489, 2019. 2[55] Fisher Yu, Haofeng Chen, Xin Wang, Wenqi Xian, Yingying
Chen, Fangchen Liu, Vashisht Madhavan, and Trevor Darrell. Bdd100k: A diverse driving dataset for heterogeneous
multitask learning. arXiv preprint arXiv:1805.04687, 2018.
2, 3, 5[56] Syed Waqas Zamir, Aditya Arora, Salman Khan, Munawar
Hayat, Fahad Shahbaz Khan, Ming-Hsuan Yang, and Ling
Shao. Multi-stage progressive image restoration, 2021. 2, 6
[57] Shizhe Zang, Ming Ding, David Smith, Paul Tyler, Thierry
Rakotoarivelo, and Mohamed Ali Kaafar. The impact of adverse weather conditions on autonomous vehicles: how rain,snow, fog, and hail affect the performance of a self-drivingcar. IEEE Vehicular Technology Magazine, 14(2):103–111,
June 2019. 1[58] Oliver Zendel, Katrin Honauer, Markus Murschitz, Daniel
Steininger, and Gustavo Fernandez Dominguez. Wilddash

- creating hazard-aware benchmarks. In Proceedings of theEuropean Conference on Computer Vision (ECCV), September 2018. 3[59] Hengshuang Zhao, Jianping Shi, Xiaojuan Qi, Xiaogang
Wang, and Jiaya Jia. Pyramid scene parsing network. In
IEEE CVPR, 2017. 4[60] Hengshuang Zhao, Yi Zhang, Shu Liu, Jianping Shi, Chen
Change Loy, Dahua Lin, and Jiaya Jia. Psanet: Point-wisespatial attention network for scene parsing. In ECCV, 2018.
4[61] Dingfu Zhou, Jin Fang, Xibin Song, Liu Liu, Junbo Yin,
Yuchao Dai, Hongdong Li, and Ruigang Yang. Joint 3d
instance segmentation and object detection for autonomous
driving. In Proceedings of the IEEE/CVF Conference on
Computer Vision and Pattern Recognition, pages 1839–
1849, 2020. 1[62] Wei Zhou, Julie Stephany Berrio, Stewart Worrall, and Eduardo Nebot. Automated evaluation of semantic segmentation robustness for autonomous driving. IEEE Transactions on Intelligent Transportation Systems, 21(5):1951–
1963, 2019. 1[63] Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A
Efros. Unpaired image-to-image translation using cycleconsistent adversarial networks. In Proc. ICCV, 2017. 7[64] Yi Zhu, Karan Sapra, Fitsum A Reda, Kevin J Shih, Shawn
Newsam, Andrew Tao, and Bryan Catanzaro. Improving semantic segmentation via video propagation and label relaxation. In IEEE CVPR, 2019. 3, 4, 5, 6