Benchmarking Robustness in Object Detection: Autonomous Driving when Winter is Coming

The ability to detect objects regardless of image corruptions or weather conditions is crucial for real-world applications of deep learning like autonomous driving . We here provide an easy-to-use benchmark to assess how object detection models perform when image quality degrades . The three resulting benchmark datasets , termed PASCAL-C , COCO-C and Cityscapes-C , contain a large variety of image corruptions . We show that a range of standard object detection models suffer a severe performance loss on corrupted images ( down to 30–60 % of the original performance ) . Furthermore , we provide evidence that increased performance on those benchmarks translates into increased robustness towards real-world “ natural ” distortions such as real-world fog , rain and snow . Using our benchmark we show that corruption robustness scales with performance on clean data and that a simple data augmentation trick—stylizing the training images—leads to a substantial increase in robustness for both synthetic and natural corruptions on all dataset . We envision our comprehensive benchmark to track future progress towards building robust object detection models . Benchmark , code and data are available at https : // .... 1 INTRODUCTION . A day in the near future : Autonomous vehicles are swarming the streets all over the . world , tirelessly collecting data . But on this cold November afternoon traffic comes to an abrupt halt as it suddenly begins to snow : winter is coming . Huge snowflakes are falling from the sky and the cameras of autonomous vehicles are no longer able to make sense of their surroundings , triggering immediate emergency brakes . A day later , an investigation of this traffic disaster reveals that the unexpectedly large size of the snowflakes was the cause of the chaos : While state-of-the-art vision systems had been trained on a variety of common weather types , their training data contained hardly any snowflakes of this size ... This fictional example highlights the problems that arise when Convolutional Neural Networks ( CNNs ) encounter settings that were not explicitly part of their training regime . For example , stateof-the-art object detection algorithms such as Faster R-CNN ( Ren et al. , 2015 ) fail to recognize objects when snow is added to an image ( as shown in Figure 1 ) , even though the objects are still clearly visible to a human eye . At the same time , augmenting the training data with several types of distortions is not a sufficient solution to achieve general robustness against previously unknown corruptions : It has recently been demonstrated that CNNs generalize poorly to novel distortion types , despite being trained on a variety of other distortions ( Geirhos et al. , 2018 ) . On a more general level , CNNs often fail to generalize outside of the training domain or training data distribution . Examples include the failure to generalize to images with uncommon poses of objects ( Alcorn et al. , 2019 ) or to cope with small distributional changes ( e.g . Zech et al. , 2018 ; Touvron et al. , 2019 ) . One of the most extreme cases are adversarial examples ( Szegedy et al. , 2013 ) : images with a domain shift so small that it is imperceptible for humans yet sufficient to fool a DNN . We here focus on the less extreme but far more common problem of perceptible image distortions like blurry images , noise or natural distortions like snow . As an example , autonomous vehicles need to be able to cope with wildly varying outdoor conditions such as fog , frost , snow , sand storms , or falling leaves , just to name a few ( as visualized in Figure 2 ) . One of the major reasons why autonomous cars have not yet gone mainstream is the inability of their recognition models to function well in adverse weather conditions ( Dai & Van Gool , 2018 ) . Getting data for unusual weather conditions is hard and while many common environmental conditions can ( and have been ) modelled , including fog ( Sakaridis et al. , 2018b ) , rain ( Hospach et al. , 2016 ) , snow ( Bernuth et al. , 2019 ) and daytime to nighttime transitions ( Dai & Van Gool , 2018 ) , it is impossible to foresee all potential conditions that might occur “ in the wild ” . If we could build models that are robust to every possible image corruption , is is to be expected that weather changes would not be an issue . However , in order to assess the robustness of models one first needs to define a measure . While testing models on the set of all possible corruption types is impossible . We therefore propose to evaluate models on a diverse range of corruption types that were not part of the training data and demonstrate that this is a useful approximation for predicting performance under natural distortions like rain , snow , fog or the transition between day and night . More specifically we propose three easy-to-use benchmark datasets termed PASCAL-C , COCO-C and Cityscapes-C to assess distortion robustness in object detection . Each dataset contains versions of the original object detection dataset which are corrupted with 15 distortions , each spanning five levels of severity . This approach follows Hendrycks & Dietterich ( 2019 ) , who introduced corrupted versions of commonly used classification datasets ( ImageNet-C , CIFAR10-C ) as standardized benchmarks . After evaluating standard object detection algorithms on these benchmark datasets , we show how a simple data augmentation technique—stylizing the training images—can strongly improve robustness across corruption type , severity and dataset . 1.1 CONTRIBUTIONS . Our contributions can be summarized as follows : 1 . We demonstrate that a broad range of object detection and instance segmentation models suffer severe performance impairments on corrupted images . 1Outdoor hazards have been directly linked to increased mortality rates ( Lystad & Brown , 2018 ) . 2 . To quantify this behaviour and to enable tracking future progress , we propose the Robust Detection Benchmark , consisting of three benchmark datasets termed PASCAL-C , COCO-C & Cityscapes-C. 3 . We demonstrate that improved performance on this benchmark of synthetic corruptions corresponds to increased robustness towards real-world “ natural ” distortions like rain , snow and fog . 4 . We use the benchmark to show that corruption robustness scales with performance on clean data and that a simple data augmentation technique—stylizing the training data—leads to large robustness improvements for all evaluated corruptions without any additional labelling costs or architectural changes . 5 . We make our benchmark , corruption and stylization code openly available in an easy-to-use fashion : • Benchmark , 2 data and data analysis are available at https : // ... 3 • Our pip installable image corruption library is available at https : // ... • Code to stylize arbitrary datasets is provided at https : // ... 1.2 RELATED WORK . Benchmarking corruption robustness Several studies investigate the vulnerability of CNNs to common corruptions . Dodge & Karam ( 2016 ) measure the performance of four state-of-the-art image recognition models on out-of-distribution data and show that CNNs are in particular vulnerable to blur and Gaussian noise . Geirhos et al . ( 2018 ) show that CNN performance drops much faster than human performance for the task of recognizing corrupted images when the perturbation level increases across a broad range of corruption types . Azulay & Weiss ( 2018 ) investigate the lack of invariance of several state-of-the-art CNNs to small translations . A benchmark to evaluate the robustness of recognition models against common corruptions was recently introduced by Hendrycks & Dietterich ( 2019 ) . Improving corruption robustness One way to restore the performance drop on corrupted data is to preprocess the data in order to remove the corruption . Mukherjee et al . ( 2018 ) propose a DNN-based approach to restore image quality of rainy and foggy images . Bahnsen & Moeslund ( 2018 ) and Bahnsen et al . ( 2019 ) propose algorithms to remove rain from images as a preprocessing step and report a subsequent increase in recognition rate . A challenge for these approaches is that noise removal is currently specific to a certain distortion type and thus does not generalize to other types of distortions . Another line of work seeks to enhance the classifier performance by the means of data augmentation , i.e . by directly including corrupted data into the training . Vasiljevic et al . ( 2016 ) study the vulnerability of a classifier to blurred images and enhance the performance on blurred images by fine-tuning on them . Geirhos et al . ( 2018 ) examine the generalization between different corruption types and find that fine-tuning on one corruption type does not enhance performance on other corruption types . In a different study , Geirhos et al . ( 2019 ) train a recognition model on a stylized version of the ImageNet dataset ( Russakovsky et al. , 2015 ) , reporting increased general robustness against different corruptions as a result of a stronger bias towards ignoring textures and focusing on object shape . Hendrycks & Dietterich ( 2019 ) report several methods leading to enhanced performance on their corruption benchmark : Histogram Equalization , Multiscale Networks , Adversarial Logit Pairing , Feature Aggregating and Larger Networks . Evaluating robustness to environmental changes in autonomous driving In recent years , weather conditions turned out to be a central limitation for state-of-the art autonomous driving systems ( Sakaridis et al. , 2018b ; Volk et al. , 2019 ; Dai & Van Gool , 2018 ; Chen et al. , 2018 ; Lee et al. , 2018 ) . While many specific approaches like modelling weather conditions ( Sakaridis et al. , 2018b ; a ; Volk et al. , 2019 ; Bernuth et al. , 2019 ; Hospach et al. , 2016 ; Bernuth et al. , 2018 ) or collecting real ( Wen et al. , 2015 ; Yu et al. , 2018 ; Che et al. , 2019 ; Caesar et al. , 2019 ) and artificial ( Gaidon et al. , 2016 ; Ros et al. , 2016 ; Richter et al. , 2017 ; Johnson-Roberson et al. , 2017 ) datasets with varying weather conditions , no general solution towards the problem has yet emerged . Radecki 2Our evaluation code to assess performance under corruption has been integrated into one of the most widely used detection toolboxes ( URL omitted to keep anonymity during review period ) . 3All URLs omitted to keep anonymity for double-blind reviewing et al . ( 2016 ) experimentally test the performance of various sensors and object recognition and classification models in adverse weather and lighting conditions . Bernuth et al . ( 2018 ) report a drop in the performance of a Recurrent Rolling Convolution network trained on the KITTI dataset when the camera images are modified by simulated raindrops on the windshield . Pei et al . ( 2017 ) introduce VeriVis , a framework to evaluate the security and robustness of different object recognition models using real-world image corruptions such as brightness , contrast , rotations , smoothing , blurring and others . Machiraju & Channappayya ( 2018 ) propose a metric to evaluate the degradation of object detection performance of an autonomous vehicle in several adverse weather conditions evaluated on the Virtual KITTI dataset . Building upon Hospach et al . ( 2016 ) , Volk et al . ( 2019 ) study the fragility of an object detection model against rainy images , identify corner cases where the model fails and include images with synthetic rain variations into the training set . They report enhanced performance on real rain images . Bernuth et al . ( 2019 ) model photo-realistic snow and fog conditions to augment real and virtual video streams . They report a significant performance drop of an object detection model when evaluated on corrupted data .

Summary: This paper introduces a  benchmark to assess the performance of object detection models when image quality degrades. Three variants of detection datasets, termed PASCAL-C, COCO-C and Cityscapes-C, are introduced that contain a large variety of image corruptions. The paper shows that standard object detection models suffer a severe performance loss on corrupted images (down to 30–60% of the original performance). Further, this work shows that a simple data augmentation trick of stylizing the training images leads to a substantial increase in robustness across corruption type, severity and dataset.