Benchmarking Robustness in Object Detection: Autonomous Driving when Winter is Coming
Authors: Claudio Michaelis
Benjamin Mitzkus
Robert Geirhos
Evgenia Rusak
Oliver Bringmann
Alexander S. Ecker
Matthias Bethge
Wieland Brendel
Venue:  (2020)
Keywords: deep learning
object detection
robustness
neural networks
data augmentation
autonomous driving

Abstract: The ability to detect objects regardless of image distortions or weather conditions is crucial for real-world applications of deep learning like autonomous driving. We here provide an easy-to-use benchmark to assess how object detection models perform when image quality degrades. The three resulting benchmark datasets, termed PASCAL-C, COCO-C and Cityscapes-C, contain a large variety of image corruptions. We show that a range of standard object detection models suffer a severe performance loss on corrupted images (down to 30-60% of the original performance). However, a simple data augmentation trick - stylizing the training images - leads to a substantial increase in robustness across corruption type, severity and dataset. We envision our comprehensive benchmark to track future progress towards building robust object detection models. Benchmark, code and data are available at: (hidden for double blind review)

Under review as a conference paper at ICLR 2020
BENCHMARKING ROBUSTNESS IN OBJECT DETECTION: A AUTONOMOUS DRIVING WHEN WINTER IS COMING
Anonymous authors Paper under double-blind review
ABSTRACT
The ability to detect objects regardless of image corruptions weather conditions is crucial for real-world applications of deep learning like autonomous driving. We here provide an easy-to-use benchmark to assess how object detection models perform when image quality degrades. The three resulting benchmark datasets, termed PASCAL-C, COCO-C and Cityscapes-C. contain a large variety of image corruttions. We show that a range of standard object detection models suffer a severe performance loss corrupted images (down to 30- 30-60% of the original performance). Furthermore, we provide evidence that increased performance on those benchmarks translates into increased robustness towards real-world "natural" distortions such as real-world fog, rain and snow. Using our benchmark we show that corruption robustness scales with performance clean data and that simple data augmentation trick-stylizing the training images-leads to a substantial increase in robustness for both synthetic and natural corruptions all dataset. We envision our comprehensive benchmark to track future progress towards building robust object detection models. Benchmark, code and data available at hhtpp:
clean data
light snow
heavy snow
bird
detection fails
person
Figure I: Mistaking dragon for a bird (left) may be dangerous but missing altogether because of snow (right) means playing with fire. Sadly, this is exactly the fate that an autonomous agent relying on a state-of-the-ar object detection system would suffer. Predictions generated using Faster R-CNN; best viewed on screen.
INTRODUCTION
A day in the near future: Autonomous vehicles are swarming the streets all over the world, tirelessly collecting data. But on this cold November afternoon traffic comes to an abrupt halt as it suddenly begins to snow: winter is coming. Huge snowflakes are falling from the sky and the cameras of autonomous vehicles are no longer able to make sense of their surroundings, triggering immediate emergency brakes. A day later, investigation of this traffic disaster reveals that the unexpectedly large size of the snowflakes was the cause of the chaos: While state-of oftte-of-the-ale the--rtt art vision systems had been trained variety of common weather types, their training data contained hardly any snowflakes of this size...

---

Under review as a conference paper at ICLR 2020
Fog
Raih
Snow
Night
Figure 2:
Expect the unexpected: To ensure safety, autonomous vehicle must be able to recognize objects even in challenging outdoor conditions such as fog, rain, snow and at night
This fictional example highlights the problems that arise when Convolutional Neural Networks (CNNs) encounter settings that were not explicitly part of their training regime. For example, state- of-the-art object detection algorithms such Faster R-CNN (Ren et al. 2015) fail to recognize objects when snow is added to an image (as shown in Figure I even though the objects are still clearly visible to a human eye. At the same time, augmenting the training data with several types of distortions is not a sufficient solution to achieve general robustness against previously unknown corruptions: It has recently been demonstrated that CNNs generalize poorly novel distortion types, despite being trained on a variety of other distortions (Geirhos et al. 2018).
On a more general level, CNNs often fail generalize outside of the training domain or training data distribution. Examples include the failure to generalize to images with uncommon poses of objects
(Alcorn al. 2019) or to cope with small distributional changes (e.g. Zech et al., 2018 Touvron
et al.] (209)). One of the most extreme cases are adversarial examples (Szegedy et al. (013): images
with domain shift so small that is imperceptible for humans yet sufficient fool a DNN. We here
focus on the less extreme but far more common problem of perceptible image distortions like blurry images, noise or natural distortions like snow.
As an example, autonomous vehicles need to be able to cope with wildly varying outdoor conditions such as fog, frost, snow, sand storms, or falling leaves, just to name a few (as visualized in Figure 2 One of the major reasons why autonomous cars have not yet gone mainstream is the inability of their recognition models to function well in adverse weather conditions (Dai & Van Gool 2018). Getting data for unusual weather conditions is hard and while many common environmental conditions can (and have been) modelled, including fog (Sakaridis et al. 2018b), rain (Hospach et al. 2016).. snow (Bernuth et al. 2019) and daytime nighttime transitions (Dai & Van Gool 2018), it impossible to foresee all potential conditions that might occur "in the wild".
If we could build models that are robust every possible image corruption, is be expected that weather changes would not be an issue. However, in order to assess the robustness of models one first needs to define a measure. While testing models on the set of all possible corruption types is impossible. We therefore propose to evaluate models on a diverse range of corruption types that were not part of the training data and demonstrate that this is a useful approximation for predicting performance under natural distortions like rain, snow, fog or the transition between day and night,
More specifically we propose three easy-to-use benchmark datasets termed PASCAL-C, COCO-C and Cityscapes-C to assess distortion robustness in object detection. Each dataset contains versions of the original object detection dataset which are corrupted with 15 distortions, each spanning five levels of severity. This approach follows Hendrycks & Dietterich (2019), who introduced corrupted versions of commonly used classification datasets (ImageNet-C. CIFAR10-C) as standardized benchmarks. After evaluating standard object detection algorithms these benchmark datasets, show how can simple data augmentation technique- aaaniqqu-stylizingg the training images can strongly improve robustness across corruption type, severity and dataset.
1.1 CONTRIBUTIONS
Our contributions can be summarized as follows:
1. We demonstrate that broad range of object detection and instance segmentation models suffer severe performance impairments on corrupted images.
Outdoor hazards have been directly linked increased mortality rates [Lystad & Brown 2018
2

---

Under review as a conference paper at ICLR 2020
2. To quantify this behaviour and to enable tracking future progress, propose the Robust detection Benchmark, consisting of three benchmark datasets termed PASCAL-C, COCO-C & Cityscapes-C.
3. We demonstrate that improved performance on this benchmark of synthetic corruptions corresponds to increased robustness towards real-world "natural" distortions like rain, snow and fog.
4. We use the benchmark to show that corruption robustness scales with performance on clean data and that a simple data augmentation large robustness improvements for all evaluated corruptions without any additional labelling costs or architectural changes.
the training data- leads to
5. We make our benchmark, corruption and stylization code openly available in an easy-to-use fashion:
Benchmark. P data and data analysis are available at https://.....
Our pip installable image corruption library is available at attttps://
Code to stylize arbitrary datasets is provided at httpss//....
1.2 RELATED WORK
Benchmarking corruption robustness Several studies investigate the vulnerability of CNNs to common corruptions. Dodge & Karam (2016) measure the performance of four state-of-the image recognition models on out-of-distribttinn data and show that CNNs are in particular vulnerable to blur and Gaussian noise, Geirhos al. (2018) show that CNN performance drops much faster than human performance for the task of recognizing corrupted images when the perturbation level increases across broad range of corruption types. [Azulay & Weiss (20T8) investigate the lack of invariance of several tfate-ofthhe-rr CNNs to small translations. A benchmark to evaluate the robustness of recognition models against common corruptions was recently introduced by Hendrycks & Dietterich (2019).
Improving corruption robustness One way restore the performance drop on corrupted data is to preprocess the data in order to remove the corruption. Mukherrjee et al. (2018) propose a DNN-based approach restore image quality of rainy and foggy images. Bahhnsn & Moeslund (2018) and Bahnsen et al. (2019) propose algorithms to remove rain from images as a preprocessing step and report subsequent increase in recognition rate. A challenge for these approaches is that noise removal is currently specific to a certain distortion type and thus does not generalize to other types of distortions. Another line of work seeks to enhance the classifier performance by the means of data augmentation, i.e. by directly including corrupted data into the training. Vasiljevic et al. (0066) study the vulnerability of a classifier blurred images and enhance the performance on blurred images by fine-tuning on them. Geirhos et al. (2018) examine the generalization between different corruption types and find that fine-tuning on one corruption type does not enhance performance on other corruption types. In different study, Geirhos et al. (2019) train recognition model on a stylized version of the ImageNet dataset (Russakovsky et al. 2015), reporting increased general robustness against different corruptions a result of a stronger bias towards ignoring textures and focusing on object shape. [Heddrycks & Dietterich (2019) report several methods leading to enhanced performance their corruption benchmark: Histogram Equalization, Multiscale Networks, Adversarial Logit Pairing, Feature Aggregating and Larger Networks.
Evaluating robustness environmental changes in autonomous driving recent years, weather conditions turned out to be central limitation for state-of-the art autonomous driving
systems (Sakaridis et al. 20; Volk et al. 2019 Dai & Van Gool 2018 Chen et al. 2018 Lee el al.
2018). While many specific approaches like modelling weather conditions (Sakaridis et al. | 111bb a
Volk et al. 2019 Bernuth et al. 2019 Hospach et al. 2016; Bernuth et al., 2018)) or collecting
Caesar et al.] 2019) and artificial (Gaidon et al., 2016] Ros et al. 2016, Richter et al.] 2017 JoohssonRRoberson et al. 2017) datasets with
real Wen et 22015 Yu et al. 2018] Che et al. 2019
varying weather conditions, no general solution towards the problem has yet emerged. Radecki
"Our evaluation code to assess performance under corruption has been integrated into one the most widely used detection toolboxes (URL. omitted to keep anonymity during review period).
All URLs omitee keep anonymity for double-bling reviewing
3

---

Under review as a conference paper at ICLR 2020
Gaussian Noise
Shot No
Impo Noise
Defacuu Blur
Glass Blur
Modion Blur
Zoom Blar
Snow
Frost
Tog
Brighnness
Contras:
Elastic Transform
Pixelate
PCG Compression
Figure 3: 15 corruption types from Hendrycks & Dietterich 2019), adapted to corrupt arbitrary images (example: randomly selected PASCAL VOC image, center crop, severity 3), Best viewed on screen.
et al. 2016) experimentally test the performance of various sensors and object recognition and classification models in adverse weather and lighting conditions. [Bernuth al. (2018) report a drop in the performance of a Recurrent Rolling Convolution network trained on the KITTI dataset when the camera images are modified by simulated raindrops on the windshield. Pei al.
introduce VeriVis, framework to evaluate the security and robustness of different object recognition models using real-world image corruptions such as brightness, contrast, rotations. smoothing, blurring and others. Machiraju & Channappayyal (2018) 8) propose a metric to evaluate the degradation of object detection performance of an autonomous vehicle in several adverse weather conditions evaluated on the Virtual KITTI dataset. Building upon Hospach et al. (201)), Volk et al. (2019) study the fragility of an object detection model against rainy images, identify corner cases where the model fails and include images with synthetic rain variations into the training set. They report enhanced performance on real rain images. |Brnuth et al. (2019) model photo-realistic snow and fog conditions to augment real and virtual video streams. They report a significant performance drop of an object detection model when evaluated on corrupted data.
METHODS
2.1 ROBUST DETECTION BENCHMARK
We introduce the Robust Detection Benchmark inspired by the ImageNet-C benchmark for object classification (Hendrycks & Dietterich, 2019) to assess object detection robustness on corrupted images.
Corruption types Following Hendrycks & Dietterich (2019). we provide 15 corruptions on five severity levels each (visualized in Figure 3 assess the effect of broad range of different corruption types on object detection models The corruptions are sorted into four groups: noise, blur, digital important note that the corruption types are not meant to be used as training data augmentation toolbox, but rather to measure model's robustness against previously unseen corruptions. Thus, training should be done without using any of the provided corruptions. For model validation, four separate corruptions are provided (Speckle Noise, Gaussian Blur, Spatter, Saturate). The 15 corruptions described above should only be used to test the final model performance.
and weather groups (as defined by Hendrycks & Dietterich (2019).
'These corruption types were introduced by Hendrycks & Dietterich 2019 and modified by us to work with images of arbitrary dimensions. Our generalized corruptions can be found https: :/:/// /7 and installed via pip3 install
4

---

Under review as a conference paper at ICLR 2020
Benchmark datasets The Robust Detection Benchmark consists of three benchmark datasets: PASCAL-C, COCO-C and Cityscapes-Co C. Among the vast number of available object
detection datasets (Everingham et al.
2010
Geiger et al. 2012 Lin et al. 2014 Cordts et al.
2016, 20huu et al.
2017] et al.
2017 Krasin et al. 2017). we chose to use PASCAL VOC
(Everingham et al. 20TO), MS COCO (Lin a
2014) and Cityscapes (Cordts al. 2016) they
are the most commonly used datasets for general object detection (PASCAL & COCO) and street scenes (Cityscapes). We follow common conventions select the tests splits: VOC2007 test set for PASCAL-C, the COCO 2017 validation set for COCO-C and the Cityscapes validation set for Cityscapes-C.
Metrics Since performance measures differ between the original datasets, the dataset-specific
performance (P) measures are adopted as defined below:
(AP AP(%)
PASCAL VOC
MS COCO & Cityscapes
where A stands for the PASCAL 'Average Precision' metric 50% Intersection over Union (IoU) and AP stands for the COCO 'Average Precision' metric which averages over IoUs between 50% and 95%. On the corrupted data, the benchmark performance is measured in terms of mean performance under corruption (mPC):
N₂
N
mPC
P..E ==1
(1)
N
N
Here, Pe.. is the dataset-specific performance measure evaluated on test data corrupted with corruption under severity level while N₂ = 15 and N, 5 indicate the number of corruptions and severity levels, respectively. In order to measure relative performance degradation under corruption, the relative performance under corruption (rPC) is introduced as defined below:
mPC
IPC
Polean
rPC measures the relative degradation performance on corrupted data compared to clean data.
Submissions Submissions the benchmark should be handed in simple pull request the Rebust Detect i on Benchma r and need to include all three performance measures: clean performance (P (Pelean). mean performance under corruption (mPC) and relative performance under corruption (rPC). While mPC is the metric used to rank models on the Robust Detect i Benchmark, the other measures provide additional insights, they disentangle gains from higher clean performance (as measured by P Pelean) and gains from better generalization performance to corrupted data (as measured by rPC).
Baseline models
We provide baseline results for set of common object detection models including
Faster R-CNN (Ren et al. | 2015), Mask R-CNN (He et al. 2017). Cascade R-CNN (Cai & Vasconcelos/
2018), Cascade Mask R-CNN (Chen al. 20) RetinaNet (Lin et al. (017b) and Hybrid Task
Cascade (Chen et al.
2019a). We use a ResNet50 (He et al. 2016) with Feature Pyramid Networks
(Lin et al. 20) as backbone for all models except for Faster R-CNN where additionally test
ResNet101 (He et al., 2016), ResNeXt1101-3xx4b (Xie et al. (017) and ResNeXt- (Xie et al.)
2017) backbones. We additionally provide results for Faster R-CNN and Mask R-CNN models with deformable convolutions (Dai et al. 2017 Zhu et al. 2018 in Appendix D Models were evaluated
using the mmdetection toolbox (Chen al. 2019b); all models were trained and tested with
standard hyperparameters. The details can be found in Appendix
A
2.2 STYLE TRANSFER AS DATA AUGMENTATION
For image classification, style transfer (Gatys et al. 2016 the method of combining the content of an image with the style of another image has been shown to strongly improve corruption robustness (Geirhos et al. 2019). We here transfer this method to object detection datasets testing two settings: (1) Replacing each training image with stylized version and (2) adding stylized version of each
https://...
5

---

Under review as a conference paper at ICLR 2020
standard
combined
stylized
Figure 4: Training data visualization for COCO and Stylized-COCOO -OCO. The three different training settings are: standard data (top row), stylized data (bottom row) and the concatenation of both (termed 'combined' in plots).
image to the existing dataset. We apply the fast style transfer method AdaIN (Huang & Belongie 2017) with hyperparameter randomly chosen texture information of Kaggle's P Painter by Numbers dataset. Examples for the stylization of COCO images are given in Figure 4 We provide ready-to-use code for the stylization of arbitrary datasets at ht tpt::
to the training data, replacing the original texture with the
2.3 NATURAL DISTORTIONS
Foggy Cityscapes Foggy Cityscapes Sakaridis et al. (2018b) is a version Cityscapes with synthetie fog in three severity levels (given byt he attenuation coefficient B 0.005m 0.01m and 0.02m - ), that was carefully designed to look as realistic as possible. We use Fogy Cityscapes only at test time, testing the same models used for our experiments with the original Cityscapes dataset and report results the same AP metric.
BDD100k BDDIO0% Yu al. (2188) driving dataset consisting of 100 thousand videos of driving scenes recorded in varying conditions including weather changes and different times of the day We use these annotations to perform experiments, on different weather conditions ("clear", "rainy" and "snowy") and on the transition from day to night. Training is performed on what we would consider "clean" " data clear for weather and daytime for time and evaluation is performed on all three splits. We use Faster R-CNN with the same hyper-parameters as in our experiments on COCO. Details of the dataset preparation can be found in Appendix
RESULTS
3.1 IMAGE CORRUPTIONS REDUCE MODEL PERFORMANCE
In order to assess the effect of image corruptions, evaluated set of common object detection models on the three benchmark datasets defined in Section 2 Performance heavily degraded corrupted images (compare Table [T.. While Faster R-CNN retain roughly 60% relative performance (rPC) on the rather simple images in PASCAL VOC, the same model suffers a dramatic reduction to 33% rPC on the Cityscapes dataset, which contains many small objects. With some variations, this effect is present in all tested models and also holds for instance segmentation tasks (for instance segmentation results, please see Appendix D).
3.2 ROBUSTNESS INCREASES WITH BACKBONE CAPACITY
We test variants of Faster R-CNN with different backbones (top of Table I and different head architectures (bottom Table T on COCO. For the models with different backbones, we find that
/tettss:/www/ kaggle. com/
The frame at the 10th second of each video is annotated with additional information including bounding hoxes which for experiments

---

Under review as a conference paper at ICLR 2020
PASCAL VOC
clean P [AP50]
corrupted
relative
model
backbone r50
mPC [AP*0]
rPC [%]
Faster
80.5
48.6
60.4
MS COCO
clean
corrupted mPC [AP]
relative
model
backbone
[AP]
rPC [%]
Faster
r50
36.3
18.2
50.2
Faster
r101
38.5
20.9
54.2
Faster
x101-32x4d
40.1
22.3
55.5
Faster
x101-64x4d
41.3
23.4
56.6
Mask
r50
37.3
Cascade
r50
40.4
20.1
49.7
Cascade Mask
r50
41.2
20.7
50.2
RetinaNet
150
35.6
17.8
50.1
HTC
x101-64x4d
50.6
32.7
64.7
Cityscapes
clean
corrupted mPC [AP]
relative
model
backhone
[AP]
rPC [%]
Faster
r50
36.4
12.2
33.4
Mask
r50
37.5
11.7
31.1
Table I: Object detection performance of various models. Backbones indicated with are ResNet and x ResNeXt. All model names except for RetinaNet and HTC indicate the corresponding model from the R-CNN family. All COCO models were downloaded from the midetect on modelzoo. For all reported quantities: higher is better; square brackets denote metric.
all image corruptions-excepp for the blur types-induce fixed penalty to model performance, independent of the baseline performance on clean data: A mPC A (compare Table and Appendix Figure 10p. Therefore, models with more powerful backbones show a relative performance { In comparison, Mask R R-CNN, Cascade R-CNN and Cascade Mask
improvement under corruption R-CNN which draw their performance increase from more sophisticated head architectures all have roughly the same rPC of Ra 50%. The current state-of-the-arr model Hybrid Task Cascade (Chen) et al., (2099) is in so far exception as it employs combination of stronger backbone, improved head architecture and additional training data to not only outperform the strongest baseline model by 9% AP on clean data but distances itself on corrupted data by a similar margin, achieving leading relative performance under corruption (rPC) of 64.7%. These results indicate that robustness in the tested regime be improved primarily through better image encoding, and better head architectures cannot extract more information if the primary encoding is already sufficiently impaired.
3.3 TRAINING ON STYLIZED DATA IMPROVES ROBUSTNESS
In order to reduce the strong effect of corruptions model performance observed above, we tested whether a simple approach (stylizing the training data) leads to robustness improvement. We evaluate the exact same model (Faster R-CNN) with three different training data schemes (visualized in Figure
standard: the unmodified training data of the respective dataset stylized: the training data is stylized completely concatenation of standard and stylized training data
combined:
The results across our three datasets PASCAL-C, COCO-C and Cityscapes-C are visualized in Figure We observe a similar pattern as reported by Geirhos et al. (2019) for object classification on ImageNet-22 model trained stylized data suffers less from corruptions than the model trained only on the original clean data. However, its performance on clean data is much lower. Combining clean data
stylized and clean data seems achieve the best of both worlds: high performance as well as strongly improved performance under corruption. From the results in Table it can be seen that both stylized and combined training improve the relative performance under corruption
Thii finding further supported by investigating models with deformable convolutions (see Appendix D
7

---

Under review as a conference paper at ICLR 2020
40
80
35
60
8 25
8 25
mAP50 in
40
20-
MAP
20
30-
E 15
15
10-
10.
10
stylized data
combined data
3
corruption severity
corruption severity
corruption severity
(a) PASCAL-C
(b) COCO-C
(c) Cityscapes-C
Figure 5: Figure 5: Training on stylized data improves test performance of Faster R-CNN on corrupted
versions of PASCAL VOC, MS COCO and Cityscapes which include all types of corruptions
shown in Figure 3 Corruption severity 0 denotes clean data. Corruption specific performances are
shown in the appendix (Figures 9
PASCAL VOC
[AP**]
COCO [AP]
Cityscapes [AP]
clean
com..
rel.
clean
corr.
rel.
clean
corr.
rel.
train data
P
mPC
rPC [%]
P
mPC
rPC [%]
P
mPC
rPC [%]
standard
80.5
48.6
60.4
36.3
18.2
50.2
36.4
12.2
33.4
stylized
68.0
50.0
73.5
21.5
14.1
65.6
28.5
14.7
51.5
combined
80.4
56.2
69.9
34.6
20.4
58.9
36.3
17.2
47.4
Table 2:
Object detection performance of Faster R-CNN trained on standard images, stylized images and the combination of both evaluated on standard test sets (test 2007 for PASCAL VOC; val 2017 for MS COCO, val for Cityscapes); higher is better.
(rPC). Combined training yields the highest absolute performance under corruption (mPC) for all three datasets. This pattern is fairly consistent. Detailed results across corruption types are reported in the Appendix (Figure) Figure and Figure 9).
3.4 TRAINING DIRECTLY ON STYLIZED DATA IS BETTER THAN USING STYLIZED DATA ONLY DURING PRE-TRAINING
For comparison reasons, we reimplemented the object detection models from Geirhos et al. (2019) and tested them for corruption robustness. Those models use backbones which are pre-trained with but the object detection models are trained on the standard clean training sets of Pascal VOC and COCO. In contrast we here use backbones trained standard "clean" ImageNet and train using stylized Pascal VOC and COCO. clean data (as reported by Geirhos et al. (2019) but also for corruption robustness (Table 3 albeit less than our approach performing the final training on stylized data (compare to Table
find that stylized pre-training helps not only on
3.5 ROBUSTNESS TO NATURAL DISTORTIONS IS CONNECTED TO SYNTHETIC CORRUPTION ROBUSTNESS
A central question is whether results the robust detection benchmark generalize real-world natural distortions like rain, snow or fog as illustrated Figure We test this using BDD100k (Yu al.) 2018). driving scene dataset with annotations for weather conditions. For first experiment, we train a model only on images that are taken "clear" weather. We also train models on a stylized version of the same images well as the combination of both following the protocol from Section 3.3 We then test these models on images which are annotated to be "clear", "rainy" or "snowy" (see
Note that Geirhos et al. 2019 use Faster R-CNN without Feature Pyramids (FPN), which is why the baseline performance of these models is different from ours
8

---

Under review as a conference paper at ICLR 2020
PASCAL VOC P
MS COCO [AP]
clean
corr.
rel.
clean
corr.
rel.
train data
P
mPC
rPC [%]
P
mPC
rPC [%]
IN
78.9
45.7
57.4
31.8
15.5
48.7
SIN
75.1
48.2
63.6
29.8
15.3
51.3
SIN+IN
78.0
50.6
64.2
31.1
16.0
51.4
SIN+IN ft IN
79.0
48.9
61.4
32.3
16.2
50.1
Table 3: Object detection performance of Faster R-CNN pre-trained on ImageNet (IN), Stylized ImageNet (SIN) and the combination of both evaluated standard test sets (test 2007 for PASCAL VOC; val 2017 for MS COCO); higher is better.
BDD100k [AP]
Weather
Day/Night
clear
rainy
rel.
snowy
rel.
day
night
rel.
train data
P
mPC
rPC [%]
mPC
rPC [%]
P
mPC
rPC [%]
clean
27.8
27.6
99.3
23.6
84.9
30.0
21.5
71.7
stylized
20.9
21.0
100.5
18.7
89.5
24.0
16.8
70.0
combined
27.7
28.0
101.1
24.2
87.4
30.0
22.5
75.0
Table Performance Faster R-CNN across different weather conditions and time changes when trained on standard images, stylized images and the combination of both evaluated on BDDI00k (see Appendix C for dataset details); higher better.
Appendix C for details). We find that these weather changes have little effect on performance on all three models, but that combined training improves the generalization to "rainy" and "snowy" images (Table 4 Weather). It may be important to note that the weather changes of this dataset are often relatively benign (e.g., images annotated rainy often show only wet roads instead of rain).
stronger test is generalization of a model trained on images taken during daytime to images taken night which exhibit a strong appearance change. We find that model trained on images taken during the day performs much worse at night but combined training improves nighttime performance (Table Day/Night and Appendix C
As third test of real-world distortions, we test our approach on Foggy Cityscapes Sakaridis et al. (2018b) which uses fog in three different strengths (given by the attenuation factor 0.005, 0.01 or 0.2m ) as a highly realistic model of natural fog. Fog drastically reduces the performance of standard models trained on Cityscapes which was collected in clear conditions. The reduction almost 50% for the strongest corruption, see Table [ In this strong test for OOD
robustness, stylized training increases relative performance substantially from about 50% to over 70% (Table 5)
Taken together, these results suggest that there is connection between performance on synthetic and natural corruptions. Our approach of combined training with stylized data improves performance in every single case with increasing gains harder conditions.
3.6 PERFORMANCE DEGRADATION DOES NOT SIMPLY SCALE WITH PERTURBATION SIZE
We investigated whether there direct relationship between the impact of corruption on the pixel values of an image and the impact of corruption on model performance. The left of Figure 6 shows the relative performance of Faster R-CNN on the corruptions in PASCAL-C dependent on the perturbation size of each corruption measured in Root Mean Square Error (RMSE). It can be seen that no simple relationship exists, counterintuitively robustness increases to corruption types with higher perturbation size (there is weak positive correlation between rPC and RMSE, This stems from the fact that corruptions like Fog or Brightness alter the image globally (resulting in high RMSE) while leaving local structure unchanged. Corruptions like Impulse Noise alter only few pixels (resulting in low RMSE) but have drastic impact on model performance.
0.45).
To investigate further if classical perceptual image metrics relationship between the perceived image quality of the original and corrupted images measured in structural similarity (SSIM, higher value means more similar, Figure on the right). There is weak
more predictive, look at the

---

Under review as a conference paper at ICLR 2020
Foggy Cityscapes [AP]
0.005
0.01
0.02
clean
corr.
rel.
corr.
rel.
corr.
rel.
train data
P
mPC
rPC [%]
mPC
rPC [%]
mPC
rPC [%]
standard
36.4
30.2
83.0
25.1
69.0
18.7
51.4
stylized
28.5
91.9
86.7
78.9
combined 36.3
32.2
88.7
29.9
82,4
26.2
72.2
Table 5: Object detection performance of Faster R-CNN Foggy Cityscapes when trained on Cityscapes with standard images, stylized images and the combination of both evaluated on the validation set; higher is better; |3 is the attenuation coefficient in m
correlation between rPC and SSIM 0.48). This analysis shows that SSIM better captures the effect of the corruptions on model performance.
100
100
brightness Kag
brightness
Fog
80
80
contrasta fros
contrast show
ipea elastick
(PC[5]]
60
pixelates
shot shotagausiin
snow
pixe lates
shot gaussian impulse
impulse
defocus
amof
defocus
40
naise
gglass
40
blur
20
weather
digital
6
10
1.0
0.8
0.6
0.4
0.2
0.0
RMSE
S5IM
Figure 6: Relative performance under corruption (rPC) function corruption RMSE (left, higher value-greater change in pixel space) and SSIM (right, higher value=higher perceived image quality) evaluated on PASCAL VOC. The dots indicate rPC of Faster R-CNN trained on standard data; the arrows show the performance gained via training on 'combined' data. Corruptions are grouped into four corruption types: noise, blur, weather and digital.
DISCUSSION
We here showed that object detection and instance segmentation models suffer severe performance impairments on corrupted images. This drop in performance has previously been observed in image recognition models (e.g. Geirhos et al. 22088) Hendryyks & Dietterich] 2019). In order track future progress on this important issue, we propose the Robust Detect Benchmark containing three easy-to-use benchmark datasets PASCAL-C, COCO-C and Cityscapes-C. We provide evidence that performance robustness corresponds to model performance on clean data. Apart from providing baselines, we demonstrate how simple data augmentation technique, namely adding stylized copy of the training data in order to reduce model's focus on textural information, leads to strong robustness improvements. On corrupted images, we consistently observe a performance increase (about 16% for PASCAL, 12% for COCO, and 41% for Cityscapes) with small losses on clean data (0-2%). This approach has the benefit that it can be applied any image dataset, requires additional labelling or model tuning and, thus, comes basically for free. At the same time, our benchmark data shows that there is still space for improvement and it is yet to be determined whether the most promising robustness enhancement techniques will require architectural modifications, data augmentation schemes, modifications to the loss function, or a combination of these.
benchmarks predicts performance
natural distortions and show, that
We encourage readers to expand the benchmark with novel corruption types. In order to achieve robust models, testing against a wide variety of different image corruptions is necessary- "too much'. Since benchmark is open source, we welcome new corruption types and look forward to your pull requests to htttp:: //... We envision our comprehensive benchmark to track future progress towards building robust object detection models that can be reliably deployed 'in the wild', eventually enabling them to cope with unexpected weather changes, corruptions of all kinds and, necessary, the occasional dragonfire.
is no
10

---

Under review as a conference paper at ICLR 2020
REFERENCES
Michael A Alcom, Qi Li, Zhitao Gong, Chengfei Wang, Long Mai, Wei-Shinn Ku, and Anh Nguyen. Strike (with) a pose: Neural networks are easily fooled by strange poses of familiar objects. In CVPR, 2019.
Aharon Azulay and Yair Weiss. Why do deep convolutional networks generalize so poorly small image transformations? a805 /2177, 2018.
Chris H. Bahnsen and Thomas B. Moeslund. Rain removal in traffic surveillance: Does it matter? arXiv:1810.12574 2018.
Chris H. Bahnsen, David Vázquez, Antonio M. López, and Thomas B. Moeslund. Learning to remove rain in traffic surveillance by using synthetic data. In VISIGRAPP, 2019.
Alexander Von Bernuth, Georg Volk, and Oliver Bringmamn. Rendering physically correct raindrops on windshields for robustness verification of camera-bass object recognition. Intelligent Vehicles Symposium (IV), pp. 922-927, 2018.
Alexander Von Bernuth, Georg Volk, and Oliver Bringmann. Simulating photo-realistic snow and fog existing images for enhanced CNN training and evaluation. In ITSC, 2019.
Holger Caesar, Varun Bankiti, Alex Lang, Sourabh Vora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom. nuscenes: A multimodal dataset for autonomous driving. arXiv: /027, 2019.
Zhaowei Cai and Nuno Vasconcelos. Cascade R- R-CNN: Delving into high quality object detection. In CVPR, 2018
Zhengping Che, Guangyu Li, Tracy Li, Bo Jiang, Xuefeng Shi, Xinsheng Zhang, Ying Lu, Guobin Wu, Yan Liu, and Jieping Ye. D2-city: A large-scale dashcam video dataset of diverse traffic scenarios. arXiv:1944.01975 2019.
Kai Chen, Jiangmiao Pang, Jiaqi Wang, Yu Xiong, Xiaoxiao Li, Shuyang Sun, Wansen Feng, Ziwei Liu, Jianping Shi, Wanli Ouyang, Chen Change Loy, and Dahua Lin. Hybrid task cascade for instance segmentation. In CVPR, 2019a.
Kai Chen, Jiaqi Wang, Jiangmiao Pang, Yuhang Cao, Yu Xiong, Xiaoxiao Li, Shuyang Sun, Wansen Feng, Ziwei Liu, Jiarui Xu, et al. Mmdetection: Open mmlab detection toolbox and benchmark. arXiv:1906007555 2019b.
Yuhua Chen, Wen Li, Christos Sakaridis, Dengxin Dai, and Lue Van Gool. Domain adaptive faster R-CNN for object detection the wild. In CVPR, 2018.
Marius Cordts, Mohamed Omran, Sebastian Ramos. Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understanding. In CVPR, 2016.
Dengxin Dai and Lue Van Gool. Dark model adaptation: Semantic image segmentation from daytime to nighttime. In ITSC, 2018.
Jifeng Dai, Haozhi Qi, Yuwen Xiong, Yi Li, Guodong Zhang, Han Hu, and Yichen Wei. Deformable convolutional networks. In ICCV, 2017.
Samuel Fuller Dodge and Lina Karam. Understanding how image quality affects deep neural networks. QoMEX, 2016.
Mark Everingham, Luc Van Gool, Christopher K. I. Williams, John Winn, and Andrew Zisserman. The Pascal Visual Object Classes (VOC) Challenge. International Journal of Computer Vision, 2010.
Adrien Gaidon, Qiao Wang, Yohann Cabon, and Elconora Vig. Virtual worlds as proxy for multi- object tracking analysis. In CVPR, 2016.
11

---

Under review as a conference paper at ICLR 2020
Leon A Gatys, Alexander S Ecker, and Matthias Bethge. Image style transfer using convolutional neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2414-2423, 2016.
Andreas Geiger, Philip Lenz, and Raquel Urtasun. Are we ready for autonomous driving? The KITTI vision benchmark suite. In CVPR, 2012.
Robert Geirhos, Carlos RM Temme, Jonas Rauber, Heiko H Schütt, Matthias Bethge, and Felix A Wichmann. Generalisation in humans and deep neural networks. In NeurIPS, 2018.
Robert Geirhos, Patricia Rubisch, Claudio Michaelis, Matthias Bethge, Felix A. Wichmann, and Wieland Brendel. ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness. In ICLR, 2019.
Priya Goyal, Piotr Dollár, Ross Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, Andrew Tulloch, Yangqing Jia, and Kaiming He. Accurate, large minibatch SGD: Training ImageNet in hour. arXiv: 7776707677,2011111
Agrim Gupta, Piotr Dollar, and Ross Girshick. LVIS: A dataset for large vocabulary instance segmentation. In CVPR, 2019.
Kaiming He. Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016.
Kaiming He, Georgia Gkioxari. Piotr Dollár, and Ross Girshick. Mask R-CNN. In ICCV, 2017.
Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common corruptions and perturbations. In ICLR, 2019.
Dennis Hospach, Stefan Müller, Wolfgang Rosenstiel, and Oliver Bringmann. Simulating photo- realistic snow and fog existing images for enhanced CNN training and evaluation. In DATE, 2016.
Xun Huang and Serge Belongie. Arbitrary style transfer in real-time with adaptive instance normal- ization. In ICCV, pp. 1501-1510, 2017.
M. Jobbeonn Charles Barto, Rounak Mehta, Sharath Nittur Sridhar, Karl Rosaen, and Ram Vasudevan. Driving in the matrix: Can virtual worlds replace human-generated annotations for real world tasks? In ICRA, 2017.
Ivan Krasin, Tom Duerig, Neil Alldrin, Vittorio Ferrari, Sami Abu-El-Haija, Alina Kuznetsova, Hassan Rom, Jasper Uijlings, Stefan Popov, Shahab Kamali, Mattco Malloci, Jordi Pont- Tuset, Andreas Veit, Serge Belongie, Victor Gomes, Abhinav Gupta, Chen Sun, Gal Chechik, David Cai, Zheyun Feng, Dhyanesh Narayanan, and Kevin Murphy. Openimages: A public dataset for large-scale multi-label and multi-class image classification. Dalaset available from hulp::/storage.
2017.
Unghui Lee, Jiwon Jung, Seokwoo Jung, and David Hyunchul Shim. Development of a self-driving car that can handle the adverse weather. International journal of automotive technology, 2018.
Tsung-Y Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft COCO: Common Objects in Context. In ECCV, 2014.
Tsung- Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie. Feature Pyramid Networks for Object Detection. In CVPR, 2017a.
Tsung- Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal Loss for Dense Object Detection. ICCV, 2017b.
Reidar Lystad and Benjamin T Brown. "Death is certain, the time is not": mortality and survival in Game of Thrones. Injury epidemiology. 5(1):44, 2018.
Harshitha Machiraju and Sumohana Channappayya. An evaluation metric for object detection
algorithms in autonomous navigation systems and its application to a real-time alerting system. In 25th IEEE International Conference on Image Processing (ICIP), 2018.
12

---

Under review as a conference paper at ICLR 2020
Jashojit Mukherjee, K Praveen, and Venugopala Madumbu. Visual quality enhancement of images under adverse weather conditions. In /TSC, 2018.
Gerhard Neuhold, Tobias Ollmann, Samuel Rota Bulò, and Peter Kontschieder. The mapillary vistas dataset for semantic understanding of street scenes. In ICCV, 2017.
Kexin Pei, Yinzhi Cao, Junfeng Yang, and Suman Jana. Towards practical verification of machine learning: The case of computer vision systems. arXiv:1112.01785, 2017
Peter Radecki, Mark Campbell, and Kevin Matzen. All weather perception: Joint data association,
tracking, and classification for autonomous ground vehicles. CoRR, abs/1605.02196, 2016. URL
http::/aaxiv.. org/ abs /1605/6055 02196
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster R-CNN: Towards real-time object detection with region proposal networks. In NIPS, 2015.
Stephan R. Richter, Zeeshan Hayder, and Vladlen Koltun. Playing for benchmarks. In ICCV, 2017.
German Ros, Laura Sellart, Joanna Materzynska, David Vazquez, and Antonio M. Lopez. The synthia dataset: A large collection of synthetic images for semantic segmentation of urban scenes. In CVPR, 2016.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma. Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision, 115(3): 211-252, 2015.
Christos Sakaridis, Dengxin Dai, Simon Hecker, and Luc Van Gool. Model adaptation with synthetic and real data for semantic dense foggy scene understanding. In ECCV, 2018a.
Christos Sakaridis, Dengxin Dai, and Luc Van Gool. Semantic foggy scene understanding with synthetic data. IJCV, 2018b.
Christian Szegedy, Wojeiech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks. 1rXiv:131..6199, 2013.
Hugo Touvron, Andrea Vedaldi, Matthijs Douze, and Hervé Jégou. Fixing the train-test resolution discrepancy. arXiv:1906. 06423, 2019.
Igor Vasiljevic, Ayan Chakrabarti, and Gregory Shakhnarovich. Examining the impact of blur on recognition by convolutional networks. arXiv:1611.05766 2016.
Georg Volk, Stefan Müller, Alexander von Bernuth, Dennis Hospach, and Oliver Bringmann. Towards robust CNN-based object detection through augmentation with synthetic rain variations. In /TSC, 2019.
Longyin Wen, Dawei Du, Zhaowei Cai, Zhen Lei, Ming-Ching Chang, Honggang Qi, Jongwoo Lim, Ming-Hsuan Yang, and Siwei Lyu. UA-DETRAC: A new benchmark and protocol for multi-object detection and tracking. arXiv:1511.041366 2015.
Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He. Aggregated residual transformations for deep neural networks. In CVPR, 2017.
Fisher Yu, Wenqi Xian, Yingying Chen, Fangchen Liu, Mike Liao, Vashisht Madhavan, and Trevor Darrell. Bdd100k: A diverse driving video database with scalable annotation tooling.
1rXiv:1805.04667, 2018.
John Zech, Mareus Badgeley, Manway Liu, Anthony B Costa, Joseph Titano, and Eric Karl Oermann. Variable generalization performance of a deep learning model to detect pneumonia in chest radiographs: cross-sectional study. PLoS medicine, 15(1 (511))e1002683, 2018
Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Scene parsing through ADE20K dataset. In CVPR, 2017.
Xizhou Zhu, Han Hu, Stephen Lin, and Jifeng Dai. Deformable convnets v2: More deformable, better results. arXiv: 18/ 168, 2018.
13

---

Under review as a conference paper at ICLR 2020
APPENDIX
A IMPLEMENTATION DETAILS: MODEL TRAINING
We train all our models with two images per GPU which corresponds to a batch size of 16 on eight GPUs. On COCO, we resize images so that their short edge is 800 pixels and train for twelve epochs with a starting learning rate of 0.01 which is decreased by a factor of ten after eight and eleven epochs. On PASCAL VOC, images are resized so that their short edge is 600 pixels. Training is done for twelve epochs with a starting learning rate of 0.00125 with a decay step of factor ten after nine epochs. For Cityscapes, we stayed as close possible to the procedure described in (He et al. 2017) rescaling images to a shorter edge size between 800 and 1024 pixels and train for 64 epochs (to match 24k steps at batch size of eight) with an initial learning rate of 0.0025 and decay step of factor ten after 48 epochs. For evaluation, only one scale (1024 pixels) is used. Specifically, we used four GPUs to train the COCO models and one GPU for all other Training with stylized data is done by simply exchanging the dataset folder or adding it to the list of dataset folders to consider. For all further details please refer the config files in our implementation (which we will make available after the end of the anonymous review period).
B CORRUPTING ARBITRARY IMAGES
In the original coruuption benchmark of ImageNet-C (Hendrycks & Dietterich, 2019), two technical aspects are hard-coded: The mnage-dimensions and the number of channels. To allow for different data sets with different image dimensions, several corruption functions are defined independently of each other, such as make_ laike_cifarr c, make Seaae_tiiyylll tiiy innyimagentt....... make wiiagmantett......... _______________ee__ee__________e ion. Additionally, many corruptions expect quadratic images. We have modified the code to resolve these constraints and now all corruptions can be applied to non- quadratic images with varying sizes, which is a necessary prerequisite for adapting the corruption benchmark to the PASCAL VOC and COCO datasets. For the corruption type Frost, crops from provided images of frost are added to the input images. Since images in PASCAL VOC and COCO have arbitrarily large dimensions, we resize the frost images to fit the largest input image dimension necessary. The original corruption benchmark also expects RGB images. Our code now allows for grayscale images [I Both motion. blur and snow relied on the motion-blur functionality of Imagemagick, resulting in an external dependency that could not be resolved by standard Python package managers. For convenience, we reimplemented the motion-blur functionality in Python and removed the dependency on non- non-Python software.
make _es__mmmmmettteee
and
C BDD100K
We the weather annotations present in the BDD100k dataset Yu et al. (2018) to split it in images with clear, rainy and snowy conditions. We disregard all images which are annotated to have any other weather condition (foggy. partly cloudy, overcast and undefined) to make the separation easie?")) I22 We use all images from the training set which are labeled having clear weather conditions for training. For testing, created subsets of the validation
each containing 725 images in clear, rainy or snowy condition [T The sets were created to have the same size which was determined by the category with the least images (rainy). Having same sized test sets important because evaluation under the AP metric leads to lower scores with increasing sequence length Gupta et al. (2019).
In all our experiments, we employ the lincar scaling rule (Goyal learning rate.
2017 select the appropriate
Thhere are approximately 2-3% grayscale images in PASCAL VOC/MS COCO.
12 would have been great to combine the performance on natural fog with the results from Foggy Cityscapes
but as there are only 13 foggy images in the validation set the results cannot he seen as representative in any way 13 We will release the datasets splits httpt::////
14

---

Under review as a conference paper at ICLR 2020
MS COCO
clean
rel.
model
backbone
P [AP]
mPC [AP]
rPC [%]
Mask Cascade Mask
r50
34.2
16.8
49.1
r50
35.7
17.6
49.3
HTC
x101-64x44
43.8
28.1
64.0
Cityscapes
clean P [AP]
corr.
rel.
model
backbone r50
mPC [AP]
rPC [%]
Mask
32.7
10.0
30.5
Table 6: Instance segmentation performance of various models. Backbones indicated with r: ResNet. All model names indicate the corresponding model from the R-CNN family. All models were downloaded from the mmdetect ion modelzoo.
MS COCO
Cityscapes
clean
corr.
rel.
clean
corr.
rel.
train data
[P]
[mPC]
[PCC]
[P]
[mPC]
[rPC]
standard
34.2
16.9
49.4
32.7
10.0
30.5
stylized
20.5
13.2
64.1
23.0
11.3
49.2
combined
32.9
19.0
57.7
32.1
14.9
46.3
Table Instance segmentation performance Mask R-CNN trained on standard images, stylized images and the combination of both evaluated on standard test sets (test 2007 for PASCAL VOC; val 2017 for MS COCO, val for Cityscapes).
D ADDITIONAL RESULTS
D.I INSTANCE SEGMENTATION RESULTS
We evaluated Mask R-CNN and Cascade Mask R-CNN on instance segmentation. The results are very similar those the object detection task with slightly lower relative performance (1%, Table We also trained Mask R-CNN on the stylized datasets finding again very similar trends for the instance segmentation task as for the object detection task (Table 7 On the one hand, this is not very surprising as Mask R-CNN and Faster R-CNN are very similar. On the other hand, the contours of objects can change due to the stylization process, which would expectedly lead poor segmentation performance when training only stylized images. We do not see such an effect but rather find the instance segmentation performance Mask R-CNN to mirror the object detection performance of Faster R-CNN when trained on stylized images.
D.2 DEFORMABLE CONVOLUTIONAL NETWORKS
We tested the effect of deformable convolutions Dai et al. 2017 Zhu et al. 2018) on corruption robustness. Deformable convolutions are a modification of the backbone architecture exchanging some standard convolutions with convolutions that have adaptive filters in the last stages of the encoder. has been shown that deformable convolutions can help on range of tasks like object detection and instance segmentation. This is the case here too as networks with deformable convolutions do not only perform better on clean but also on corrupted images improving relative performance by 6-7% compared to the baselines with standard backbones (See Tables and 9). The effect appears to be the same as for other backbone modifications such as using deeper architectures (See Section in the main paper).
and
IMAGE RIGHTS & ATTRIBUTION
Figure Home Box Office, Inc. (HBO).
15

---

Under review as a conference paper at ICLR 2020
MS COCO
clean
rel.
model
backbone
[AP]
mPC [AP]
rPC [%]
Faster
r50-den
40.0
22.4
56.1
Faster
x x101-64x4d-de
43.4
26.7
61.6
Mask
r50-den
41.1
23.3
56.7
Table 8: Object detection performance of models with deformable convolutions Dai al. 2017). Backbones indicated with r are ResNet, the addition den signifies deformable convolutions in stages c3-c5. All model names indicate the corresponding model from the R-CNN family. All models were downloaded from the mmdetect. in modelzoo.
MS COCO
clean P [AP]
сorr.
rel.
model
backbone
mPC [AP]
rPC
[%]
Mask
r50-den
37.2
20.7
55.7
Table Instance segmentation performance of Mask R-CNN with deformable convolutions DDii
et al. 2017). The backbone indicated with is a ResNet 50, the addition den signifies deformable convolutions in stages c3-c5. The model was downloaded from the mmdetection modelzoo.
16

---

Under review as a conference paper at ICLR 2020
gaussian noise
shot
Impulse noise
MAP50 %
40
40
40
standard data
20
stylized
20
20
combline data
0-
0-
defocus blur
glass blur
motion blur
MAP50 %
60-
60
60
40
40
40
20
20
20
zoom blur
snow
frost
80
80
80
MAP50 in %
60-
60
60
40-
40
40
tog
brightness
contrast
80
60
80
MAP50 in %
60
60
60
40
elastic transform
pixelate
jpeg compression
80
80
80
MAP50 in %
60
40
20
20
20
0-
0-
0-
corruption severity
corruption severity
corruption severity
Figure Results for each corruption type on PASCAL-C.
17

---

Under review as a conference paper at ICLR 2020
gaussian noise
shot
Impulse noise
40
40
40
standard data stylized combined data
MAP in %
30
30
20-
20
20-
10-
10
10
defocus blur
glass blur
motion blur
40
40
40
MAP in %6
30
30-
30
20
20
10-
10-
10
zoom blur
snow
frost
40
MAP in %6
30
30
30
20
20
10-
10
10
tog
brightness
contrast
30
30
30
MAP in %
20
20
20
10-
10-
10-
0-
elastic transform
pixelate
jpeg compression
40
40
40
MAP %6
30
30
30
20
20
20
0-
0-
0-
corruption severity
corruption severity
corruption severity
Figure 8: Results for each corruption type on COCO-C.
18

---

Under review as a conference paper at ICLR 2020
gaussian noise
shot
Impulse noise
40
40
40
standard data
30-
stylized
MAP in %
combined data
20
20
20
10-
10-
defocus blur
glass blur
motion blur
40
40
40
MAP in %6
30
30
30
20
20
20
0-
0-
0-
zoom blur
snow
frost
40
30
30
30
MAP in %
20
20
20
0-
0-
0-
tog
brightness
contrast
30
30
30
MAP in %
20
10-
10
10
0-
0-
elastic transform
pixelate
jpeg compression
40
40
40
30
30
30
MAP %
20
10.
10
10
corruption severity
corruption severity
corruption severity
Figure Results for each corruption type on Cityscapes-C.
19

---

Under review as a conference paper at ICLR 2020
gaussian noise
shot noise
Impulse noise
40
ResNot-50
40
40
ResNet-101
MAP 8P
ResNext:101
20
10-
10
10
0-
0-
0-
defocus blur
glass blur
motion blur
40
40
40
30
30
30
MAP In
0-
0-
0-
zoom blur
snow
frost
40
40
40
MAP in %
30
30
30
20
0-
fog
brightness
contrast
40
40
40
MAP %
30
30
30
20
20
20
10
elastic transform
pixelate
jpeg compression
40
MAP %
30
30
30
20
20
20
10.
10
10
corruption severity
corruption severity
corruption severity
Figure 10: Results for each corruption type using different backbones. Faster R-CNN trained on MS COCO with ResNet-50, ResNet-101 and ResNext-101, 101_x4444 backbones.
20


Reviews:
[{"review_id": "r1gPC72CKH", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents a benchmark for measuring robustness to input image corruption in object detection settings. The paper proposes a benchmark for this task, and proposes a simple data augmentation technique for this task.\n\nStrengths\n1. Understanding the robustness properties of existing vision models is an important problem.\n2. The paper establishes a sensible protocol for the benchmark, where methods are tested upon image perturbations that are not used for training the model.\n3. I like the proposed simple data augmentation procedure and the experimental finding that data augmentation with such a procedure leads to models that are robust to held-out, previously unseen perturbations.\n\nShortcomings:\n1. While the paper proposes a sensible experimental protocol, certain questions remain:\na) Are the set of test time perturbations exhaustive and representative of the perturbations in the real world? The paper doesn't talk about this, or provides any experimental data to establish this. The paper derives them from an earlier paper called \"Benchmarking neural network robustness to common corruptions and perturbations\", and thus I am not even sure if the proposed set of perturbations should be viewed as a contribution of the current paper.\nb) While the paper itself follows good practice by not training on perturbations that are considered at test time, unfortunately, it does not define a clear protocol or characterization as to how future researchers should use the benchmark. I believe setting up such a protocol is going to be difficult and is worthy of more thought and consideration, absence of this weakens the paper, as it leaves the door open for flawed future research.\n\n2. Missing comparisons: Proposed method is interesting, but I wonder if there were a more standard evaluation to test the efficiency of the method, perhaps something like testing if representations learned using such data augmentations were more robust to adversarial perturbations? Or perhaps, comparison against other methods that exist in literature for related tasks, such as methods that study how to make networks robust to adversarial perturbations?\n\nBecause of the aforementioned reasons, I don't view the benchmarking part of the paper as a solid contribution. Similarly, the proposed method is simple and intuitive (which is good), but it will help if there were more comparisons to set the paper in context of related work.", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_thoroughness_in_paper_reading": "I made a quick assessment of this paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "SJgukSgYFH", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\n- key problem or question: assessing / improving the robustness of object detectors to image corruptions (simulated fog, frost, snow, dragonfire...);\n- contributions: 1) a benchmark (obtained by adding image-level corruptions to PASCAL, COCO, and Cityscapes) and an experimental protocol to measure detection robustness , 2) extensive experiments quantifying the severe lack of robustness of multiple state-of-the-art models, 3) experiments showing that data augmentation via style transfer (Geirhos et al, ICLR'19) improves robustness at little cost (at most -2% performance degradation on clean COCO images).\n\nRecommendation: Weak Reject\n\nKey reason: unclear novelty w.r.t. Geirhos et al ICLR 2019, especially due to the lack of specificity to object detection (which is the main goal of the paper).\n- Although the paper from Geirhos et al. study a specific question (texture bias in CNNs), they propose a similar experimental protocol for assessing robustness, where the only differences are i) they use different corruptions (cf. Fig.6 in their paper, vs. Figs. 5, 7, 8, 9 here), ii) they do not have Cityscapes results (but they have PASCAL and COCO results).\n- Furthermore, Geirhos et al 2019 also study the benefits of style-transfer-based data augmentation (this submission uses their technique), they report similar results and conclusions to this submission, including for object detection on PASCAL and COCO (cf. Table 2 in Geirhos et al 2019).\n- What is, in the authors' opinion, the main differentiator of this submission? Does the difference in corruptions and evaluated models combined with the addition of Cityscapes yield new insights compared to Geirhos et al 2019?\n- Beyond this similarity, what is specific to object detection vs. image classification in this submission? Besides the summary evaluation metrics, the corruptions and stylization are global (image-level) and not object-specific. Is the observed lack of robustness due to localization errors, mis-classifications, or other types of detection mistakes (cf. Hoiem et al ECCV'12)? What are the conclusions about detection robustness that differ from the image classification ones? What would be local corruptions that specifically degrade object detection performance (as studied for instance in the adversarial attack community, including physical attacks like Eykholt et al 2018)?\n\nAdditional feedback / questions:\n- The results in section 3.4 / Fig.6 seem counterintuitive: more corruption should yield more degradation (as evidenced in Fig. 5 w.r.t. corruption severity). Is RMSE the right metric? Maybe SSIM would be better (more related to perceptual quality)?\n- The aforementioned remark also raises the point that corruption difficulty night not be related to its intensity / noise level, but more with the (hard to quantify) domain gap w.r.t. clean data. For instance, if some images contain natural fog, fog corruption robustness should be naturally higher, whereas robust to never seen unrealistic dragonfire is expected to be naturally low. Can this relation between corruption and domain gap be somehow assessed? For instance using perceptual distance (using features from intermediate layers) to nearest clean neighbors? Or maybe by correlating with a subjective measure of realism of the corruption assessed globally per corruption type via a human study?\n- Are corruption degradations dataset-specific? Fig. 7,8,9 seem to show different behaviors.\n- How does the proposed benchmark compare to the Robust Vision Challenge (http://www.robustvision.net) proposed at CVPR 2018? How does robustness to corruptions and robustness across datasets correlate? Are they both similar \"out of domain\" robustness measures? How does this \"out of domain\" issue relate to adversarial examples (besides being \"less extreme\")?\n- Could Fig. 5 include error bars / variance across corruption types?\n- rPC is a good metric to compare models but I am not sure it is great to compare robustification methods, because when improving \"clean\" performance you mechanically decrease rPC, hence why combined is worse than stylized (e.g., in Table 2). What would be a better metric?\n- Why does the stylized approach on COCO yields worse mPC (cf. Table 2 and 4), whereas it is expected to improve robustness under corruption (and does on other datasets), esp. since it sacrifices performance on the clean images by a lot?\n- What is the impact of the choice of style images on robustness induced by stylization? Is diversity the most important factor (this can be tested by reducing the number of styles available)? If not, what is and how can it be measured?\n- Do certain corruption (at certain severity levels) result in unrecoverable objects? For instance, dragonfire might completely occlude certain objects on the side, which might be a problem (e.g., there are frequent parked cars on the side in Cityscapes). What is the upper bound after corruption and how can it be measured?\n- What is the human robustness for the new corruptions not present in Geirhos et al 2019? (Also relates to the aforementioned upper bound.)\n- The paper is well written and I enjoyed the multiple pop culture references to Game of Thrones.", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}, {"review_id": "Bkldg_0KKH", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper introduces a  benchmark to assess the performance of object detection models when image quality degrades. Three variants of detection datasets, termed PASCAL-C, COCO-C and Cityscapes-C, are introduced that contain a large variety of image corruptions. The paper shows that standard object detection models suffer a severe performance loss on corrupted images (down to 30–60% of the original performance). Further, this work shows that a simple data augmentation trick of stylizing the training images leads to a substantial increase in robustness across corruption type, severity and dataset.\n\nThe paper is well written and easy to follow. The proposed benchmark is interesting and clearly show the deficiencies of state-of-the-art object detection methods in case of image corruptions or weather conditions. However, my main concern is the novelty in that the proposed approach is just an extension of [1]. [1] introduced corrupted versions of commonly used classification datasets (ImageNet-C, CIFAR10-C) as standardized benchmarks. The different types of corruptions used here for object detection and their sorting into four groups were also introduced originally in [1]. Moreover, the idea to use style transfer as an augmentation to  improve corruption robustness for image classification has been introduced in [2]. Therefore, the only contribution of this paper is to apply the ideas from [1, 2] for object detection. \n\n[1] Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common corruptions and perturbations. In ICLR, 2019.\n[2] Robert Geirhos, Patricia Rubisch, Claudio Michaelis, Matthias Bethge, Felix A. Wichmann, and Wieland Brendel. ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness. In ICLR, 2019.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a benchmark for assessing the impact of image quality degradation (e.g. simulated fog, snow, frost) on the performance of object detection models. The authors introduce corrupted versions of popular object detection datasets, namely PASCAL-C, COCO-C and Cityscapes-C, and an evaluation protocol which reveals that the current models are not robust to such corruptions (losing as much as 60% of the performance). The authors then show that a simple data augmentation scheme significantly improves robustness. The reviewers agree that the manuscript is well written and that the proposed benchmark reveals major drawbacks of current detection models. However, two critical issues with the paper paper remain, namely lack of novelty in light of Geirhos et al., and how to actually use this benchmark in practice. I will hence recommend the rejection of this paper in the current state. Nevertheless, we encourage the authors to address the raised shortcomings (the new experiments reported in the rebuttal are a good starting point). 
Decision: Reject