Gradient-Free Adversarial Training Against Image Corruption for Learning-based Steering

1 Introduction . Autonomous driving is a complex task that requires many software and hardware components to operate reliably under highly disparate and often unpredictable conditions . In this work , we study “ learning-based steering ” as it contains both perception and control , both critical components for autonomous driving . While on the road , vehicles are going to experience day and night , clear and foggy conditions , sunny and rainy days , as well as bright cityscapes and dark tunnels . All these external factors in conjunction with internal factors of the camera ( e.g. , those associated with hardware ) can lead to quality variations in input data for image-based learning algorithms . One can harden machine learning systems to these degradations by simulating them at training time [ 6 ] . However , an algorithmic tool for analyzing the sensitivity of real-world neural network performance on the properties of ( and corruptions to ) training images is lacking . More importantly , a mechanism to leverage such a sensitivity analysis for improving learning outcomes needs to be developed . In this work , we quantify the influence of image quality on the task of “ learning to steer , ” study how training on degraded and low-quality images can boost robustness to image corruptions , and provide a systematic approach to improve the performance of learning algorithms using quantitative analysis . Image degradations can be simulated by varying attributes such as blur , noise , distortion , color representations ( such as RGB or CMY ) hues , saturation , and intensity values ( HSV ) . However , identifying the correct combination of the simulated corruptions to obtain optimal performance on 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . real data is a difficult—if not impossible—task , as it requires domain transfer and exploring a high dimensional parameterized space . We design a systematic method for measuring the severity of image degradation and predicting the impact of such degradation on model performance . Inspired by the use of image feature variance in sensitivity analysis [ 33 ] , we measure the difference between real-world image distributions and simulated/degraded image distributions using Fréchet Inception Distance ( FID ) . Our results confirm that FID can help predict the performance of a model trained using simulated data and deployed in the real world . Next , we use FID between different simulated datasets as a unified metric to parameterize the severity of various image degradations due to different factors . Borrowing concepts from the adversarial attack literature [ 28 , 35 , 43 ] , we build a scalable training scheme for enhancing the robustness of autonomous driving against various combinations of image degradations , while increasing the overall accuracy of the steering task on clean data . Our proposed method constructs a dataset of adversarially degraded images by applying optimization within the space of possible degradations during training . As shown in Fig . 1 , the method begins by training on a set of real and simulated/degraded images using arbitrary degradation parameters . During each training iteration , the parameters are updated to generate a new degradation set so that the model performance is ( approximately ) minimized . The network is then trained on these adversarially degraded images to promote robustness . A post-training step is applied to further improve the performance on clean data without weakening robustness . Our proposed algorithm uses our FIDbased parameterization to discretize the search space of degradation parameters and accelerates the process of finding optimal parameters . Experiments show that our algorithm improves the performance of “ learning to steer ” up to 97 % in mean accuracy over baselines , and especially improves the performance on datasets contaminated with complex combinations of perturbations ( up to 87 % ) . It additionally boosts the test performance on degradations that are not seen during training , including simulated snow , fog , and frost ( up to 77 % ) . We also compare our approach with other SOTA techniques ( e.g. , data augmentation and adversarial training ) on visual processing tasks such as detection and classification . Our method consistently achieves higher performance . In addition , our method is easy to implement and can be readily integrated with other frameworks such as object detection , classification , regression , etc . Finally , we propose a comprehensive robustness evaluation standard under four different scenarios : clean data , single-perturbation data , multi-perturbation data , and previously unseen data . While state-of-the-art studies usually conduct testing under one or two scenarios ( e.g. , ImageNet-C [ 20 ] ) , our work tests and verifies results under four meaningful scenarios . We plan to release code and datasets for benchmarking “ autonomous driving under perturbations ” using unseen factors such as image corruptions in ImageNet-C [ 20 ] , totaling 480 datasets and 26M images . 2 Related Work . The influence of noise and distortion on real images for learning tasks has been well explored . For example , researchers have examined the impact of optical blur on convolutional neural networks and present a fine-tuning method for recovering lost accuracy using blurred images [ 41 ] . This fine-tuning method resolves lost accuracy when images are distorted instead of blurred [ 49 ] . While these finetuning methods are promising , Dodge and Karam [ 11 ] find that tuning to one type of image quality reduction effect would cause poor generalization to other types of effects . The comparison of image classification between deep neural networks and humans shows similar performance on good-quality images [ 12 ] . However , deep neural networks struggle significantly more than humans on low-quality , distorted , and noisy images . One study shows that adversarial perturbations are more prevalent in the Y channel in the YCbCr color space of images than the other two channels , while perturbations in RGB channels are equally distributed [ 29 ] . Wu et al . [ 42 ] studies the effect of Instagram filters on learning tasks , and introduces a lightweight de-stylization module that predicts parameters used for scaling and shifting feature maps to “ undo ” the changes incurred by filters . Researchers have also explored how to improve the robustness of learning algorithms under various image quality degradations . One recent work provides a novel Bayesian formulation for data augmentation [ 39 ] . Cubuk et al . [ 9 ] proposes an approach to automatically search for improved data augmentation policies . Ghosh et al . [ 17 ] analyzes the performance of convolutional neural networks on quality degradations due to compression loss , noise , blur , and contrast , and introduces a method to improve the learning outcome . Another work [ 21 ] shows that self-supervision techniques can be used to improve model robustness and exceeds the performance of fully supervised methods . A recent method , AugMix [ 22 ] improves model robustness using data augmentation , where transformation compositions are used to create a new dataset that is visually and semantically similar to the original dataset . AugMix is compared to several other augmentation methods , which comprise Cutout [ 10 ] , MixUp [ 45 ] , and CutMix [ 44 ] . Gao et al . [ 14 ] proposes a technique to re-purpose software testing methods to augment the training data of DNNs , with the objective to improve model robustness . A recent work improves model generalizability by first augmenting the training dataset with random perturbations , and then minimizing worst-case loss over the augmented data [ 18 ] . Our work differs from these studies in several regards . First , we simulate adversarial conditions of image factors instead of using commonplace image conditions . Second , we conduct a systematic sensitivity analysis for preparing datasets that are representative of image degradations from multiple factors at various levels . Third , our algorithm can work with the discretized parameter space while generalizing well to the continuous parameter space . Another advantage of our approach is that we can augment the training dataset without the derivatives of the factor parameters , which may not exist or are difficult to compute . 3 Background and Setup . Task . Our target task is end-to-end steering : given a single image as input ( e.g . captured by a front-facing camera on a self-driving car ) , output a steering angle that drives the car safely on the road [ 4 ] . A steering angle of 0 represents the forward direction . We use mean accuracy ( MA ) to evaluate the task since it represents overall performance under a variety of measures ( See Sec . 5 ) . Datasets . We choose four real-world driving corpuses as our datasets : Audi [ 16 ] , Honda [ 31 ] , Waymo [ 37 ] , and SullyChen [ 7 ] . The Audi dataset is the most recent ( 2020 ) ; the Honda dataset has 100+ long-time driving videos ; Waymo includes many environmental conditions such as weather and lightening and is a large dataset ( 390k frames for perception ) ; and the SullyChen dataset focuses on the steering task and has the longest continuous driving image sequence without road branching . The details of these datasets are provided in Appendix A.3 . Other datasets , such as CIFAR-100 , for various computer vision tasks are also used to demonstrate the generalizability of our technique . Basis perturbation . We study nine basis perturbations : blur , noise , distortion , three-color ( RGB ) channels , and hues , saturation , and intensity values ( HSV ) . Blur , noise , and distortion are among the most commonly used perturbations that can directly affect image quality . R , G , B , H , S , V channels are chosen because they are frequently used to represent image color spaces : RGB represent three basic color values of an image , while HSV represent three common metrics of an image . Other color spaces such as HSL or YUV have similar properties , hence are excluded . We use Gaussian blur [ 2 ] ( parameterized by standard deviation ) , additive white Gaussian noise ( AWGN ) ( parameterized by standard deviation ) , and radial distortion [ 47 , 48 ] ( with radial distortion parameters k1 , k2 ) . For representing channel-level perturbations , we use a linear model : denote the value range of one channel C as [ aC , bC ] , in the darker direction , we set v′C = αaC + ( 1− α ) vC , while in the lighter direction , we set v′C = αbC + ( 1− α ) vC , where α is the severity parameter , vC is the pixel value on clean image and v′C is the perturbed pixel value . The default values are aC = 0 and bC = 255 , with two exceptions . We set aV = 10 to exclude a complete dark image on V channel , and bH = 179 according to H channel definition . See examples in Fig . 2 and detailed description in Appendix A.2 . Test scenarios . While other studies usually test in only one or two scenarios [ 20 ] , we test the performance of all methods in four scenarios with increasing complexity . Scenario 1 : Base dataset . Test on the base dataset . Scenario 2 : Single perturbation . Test on the datasets with perturbations , but each dataset only experiences one level of one perturbation ( within blur , noise , distortion , R , G , , B , H , S , V ) . Scenario 3 : Combined perturbations . Test on the datasets with combinations of various perturbations at different levels , and each dataset has different levels of all perturbations . Scenario 4 : Unseen perturbation . Test on the datasets with unseen perturbations at different levels . Specifically , we use “ motion blur ” , “ zoom blur ” , “ pixelate ” , “ jpeg compression ” , “ snow ” , “ frost ” , “ fog ” from ImageNet-C [ 20 ] . We apply the same perturbations/levels to all images within one dataset . ( See details in Appendix A.1 ) .

Summary: The proposed work uses a concept similar to adversarial training to improve robustness against image corruptions. The authors consider different basis perturbations which can account for different complex perturbations and vary their parameters during the min-max procedure to perform adversarial training.  Empirical results are shown on multiple datasets and different tasks such as classification, detection.