Driving through the Lens: Improving Generalization of Learning-based Steering using Simulated Adversarial Examples

To ensure the wide adoption and safety of autonomous driving , vehicles need to be able to drive under various lighting , weather , and visibility conditions in different environments . These external and environmental factors , along with internal factors associated with sensors , can pose significant challenges to perceptual data processing , hence affecting the decision-making of the vehicle . In this work , we address this critical issue by analyzing the sensitivity of the learning algorithm with respect to varying quality in the image input for autonomous driving . Using the results of sensitivity analysis , we further propose an algorithm to improve the overall performance of the task of “ learning to steer ” . The results show that our approach is able to enhance the learning outcomes up to 48 % . A comparative study drawn between our approach and other related techniques , such as data augmentation and adversarial training , confirms the effectiveness of our algorithm as a way to improve the robustness and generalization of neural network training for self-driving cars . 1 INTRODUCTION . Autonomous driving is a complex task that requires many software and hardware components to operate reliably under highly disparate and often unpredictable conditions . While on the road , vehicles are likely to experience day and night , clear and foggy conditions , sunny and rainy days , as well as bright cityscapes and dark tunnels . All these external factors can lead to quality variations in image data , which are then served as inputs to autonomous systems . Adding to the complexity are internal factors of the camera ( e.g. , those associated with hardware ) , which can also result in varying-quality images as input to learning algorithms . One can harden machine learning systems to these degradations by simulating them at train time ( Chao et al. , 2019 ) . However , there currently lacks algorithmic tools for analyzing the sensitivity of real-world neural network performance on the properties of simulated training images and , more importantly , a mechanism to leverage such a sensitivity analysis for improving learning outcomes . In this work , we quantify the influence of varying-quality images on the task of “ learning to steer ” and provide a systematic approach to improve the performance of the learning algorithm based on quantitative analysis . Image degradations can often be simulated at training time by adjusting a combinations of image quality attributes , including blur , noise , distortion , color representations ( such as RGB or CMY ) hues , saturation , and intensity values ( HSV ) , etc . However , identifying the correct combination of simulated attribute parameters to obtain optimal performance on real data during training is a difficult—if not impossible—task , requiring exploration of a high dimensional parameterized space . The first goal of this work is to design a systematic method for studying , predicting , and quantifying the impact of an image degradation on system performance after training . We do this by measuring the similarity between real-world datasets and simulated datasets with degradations using the well known Fréchet Inception Distance ( FID ) . We find that the FID between simulated and real datasets is a good predictor of whether training on simulated data will produce good performance in the real world . We also use FID between different simulated datasets as a unified metric to parameterize the severity of various image quality degradations on the same FID-based scale ( see Section 3 ) . Our second goal is to borrow concepts from the adversarial training literature ( Madry et al. , 2018 ; Shafahi et al. , 2019 ; Xie et al. , 2020 ) to build a scalable training scheme to improve the robustness of autonomous driving systems against multi-faceted image degradations , while increasing the overall accuracy of the steering task for self-driving cars . Our proposed method builds a dataset of adversarially degraded images by apply evolutionary optimization within the space of possible degredations during training . The method begins by training on a combination of real and simulated/degraded data using arbitrary degradation parameters . On each training iteration , the parameters are updated to generate a new degradation combination so that system performance is ( approximately ) minimized . The network is then trained on these adversarially simulated images to promote robustness . Our proposed algorithm speeds up the combinatorial degradation updates by discretizing the search space using our FID-based parameterization . See details in Section 4 . Experiments show that the algorithm improves the task performance of “ learning to steer ” up to 48 % in mean accuracy over strong baselines . We compare our approach with other related techniques , such as data augmentation and adversarial training , and the results show that our method consistently achieves higher performance . The method also improves the performance on datasets contaminated with complex combinations of perturbations ( up to 33 % ) , and additionally boosts performance on degradations that are not seen during training , like simulated snow , fog , and frost ( up to 29 % ) , as discussed in Section 4.2 . For evaluation , we propose a more complete robustness evaluation standard under 4 different scenarios : clean data , single perturbation data , multiple perturbation data , and unseen perturbation data . While previous works usually use one or two scenarios , our work is among the first to test results under all 4 meaningful scenarios that are important for evaluating robustness of algorithms . We also plan to release “ learn to steer under perturbations ” datasets for benchmarking . These datasets will contain a base dataset ; simulated adversarial datasets with multiple levels of image degradation due to either single or multiple image quality attributes ; and simulated adversarial datasets with five levels of combinatorial perturbations due to a different set of unseen factors on images with corruptions in ImageNet-C ( Hendrycks & Dietterich , 2019 ) , totaling about 1.2M images and 120 datasets in all . ( See Section 4.5 . ) 2 RELATED WORK . The influence of the noise and distortion effects on real images on learning tasks has been explored in the last five years . For example , researchers have examined the impact of optical blur on convolutional neural networks and present a fine-tuning method for recovering lost accuracy using blurred images ( Vasiljevic et al. , 2016 ) . This fine-tuning method resolved lost accuracy when images were distorted instead of blurred ( Zhou et al. , 2017 ) . While these fine tuning methods are promising , ( Dodge & Karam , 2017b ) find that tuning to one type of image quality reduction effect would cause poor generalization to other types of quality reduction effects . Comparison on image classification performance between deep neural networks and humans has been done by ( Dodge & Karam , 2017a ) , and found to be similar with images of good quality . However , Deep Neural Networks struggle significantly more than humans on low-quality , distorted or noisy images . Color spaces have also been shown to negatively affect performance of learning models . It was observed that perturbations affect learning performance more in the Y channel of YCbCr color space that is analogous to the YUV color space ( Pestana et al. , 2020 ) . Another work ( Wu et al. , 2020 ) studies the effect of Instagram filters , which mostly change the coloring of an image , on learning tasks . In this work , we study nine common factors characterizing image quality , i.e. , blur , noise , distortion , three-color ( Red-Green-Blue or RGB ) channels , and hues , saturation , and intensity values ( HSV ) . Not only does our study analyze a more comprehensive set of image attributes that could influence the learning-based steering task , but we also parameterize these nine factors into one integrated image-quality space using the Fréchet Inception Distance as the unifiying metric , thus enabling us to conduct sensitivity analysis . Researchers have also explored how to improve the robustness of learning algorithms under various image quality degradations . One recent work ( Tran et al. , 2017 ) provides a novel Bayesian formulation for data augmentation . Cubuk et al . ( 2018 ) proposes an approach to automatically search for improved data augmentation policies . Ghosh et al . ( 2018 ) performs analyses on the performance of Convolutional Neural Networks on quality degradations due to common causes like compression loss , noise , blur , and contrast , etc . and introduces a method to improve the learning outcomes using a master-slave architecture . Another work ( Hendrycks et al. , 2019 ) shows that self-supervision techniques can be used to improve model robustness that exceeds the performance of fully-supervised methods . A new method , also by Hendrycks et al. , tackles model robustness from a data augmentation perspective , where compositions of transformations are used to create new data that is visually and semantically close to the original dataset ( Hendrycks et al. , 2020 ) . Gao et al . ( 2020 ) proposes a technique that re-purposes software testing methods , specifically mutation-based fuzzing , to augment the training data of DNNs , with the objective of enhancing their robustness . A more recent work ( Gong et al. , 2020 ) introduces a solution to improving model generalization by first augmenting training dataset with random perturbations , and then minimizing worst-case loss over the augmented data . Our work differs from these studies in several regards . First , we simulate adversarial conditions of image factors instead of commonplace image conditions . Second , we conduct a systematic sensitivity analysis for preparing datasets that are representative of image degradations due to multiple factors at different levels . Third , our algorithm for improving the generalization of the learning task can work with the discretized parameter space while generalize well to the continuous parameter space . Another advantage of our approach is that we can augment the training dataset without the derivatives of the factor parameters , which may not exist or are difficult to compute . These differences distinguish our approach from the prior arts and other techniques for improving model robustness , such as data augmentation and adversarial training . 3 ANALYZING IMAGE QUALITY FACTORS . 3.1 EXPERIMENT SETUP . Base Dataset . We build our base dataset using the real-world driving dataset ( Chen , 2018 ) , which contains about 63,000 images at the resolution 455 × 256 . There are several good autonomous driving datasets , but not all of them are suitable for the end-to-end learning to steer task . Some datasets ( e.g. , Waymo ( Sun et al. , 2020 ) , KITTI ( Geiger et al. , 2013 ) , Cityscapes ( Cordts et al. , 2016 ) , OxfordRoboCar ( Maddern et al. , 2017 ) , Raincouver ( Tung et al. , 2017 ) , etc ) do not contain steering angle labels . Some other datasets contain steering angle labels ( e.g. , Audi ( Geyer et al. , 2020 ) , Honda ( Ramanishka et al. , 2018 ) , nuScenes ( Caesar et al. , 2019 ) , Ford AV ( Agarwal et al. , 2020 ) , Canadian Adverse Driving Conditions ( Pitropov et al. , 2020 ) , etc ) , but they are not designed for end-to-end learning to steer task specifically , the driving environment may not be suitable for our end-to-end learning to steer task . For example , driving at intersections ( lead to confusion for steering ) , or only a very small subset of data containing image data from turning on a curved road without branches . There are also several well-known simulators like CARLA ( Dosovitskiy et al. , 2017 ) that can generate synthetic dataset , but our work focuses on the real-world driving using real images . Chen ’ s dataset is collected specifically for steering task and has a relatively long continuous driving image sequence on a road without branches and has relatively high turning cases . Image quality factors . We study nine image attributes in this work , i.e. , blur , noise , distortion , three-color ( Red-Green-Blue or RGB ) channels , and hues , saturation , and intensity values ( HSV ) . Blur , noise , and distortion are among the most commonly used and directly affect image quality . R , G , B , H , S , V channels are chosen because both RGB and HSV spaces are frequently used to represent image color spaces : RGB represent three basic color values of an image , while HSV represent three common metrics of an image . Other color spaces such as HSL or YUV have similar properties , hence are excluded from this study to avoid redundant analysis . Learning Algorithm . We choose the model by Bojarski et al . ( 2016 ) as the learning algorithm . The model contains five convolutional layers followed by three dense layers . We select this model , because it has been used to steer an AV successfully in both real world ( Bojarski et al. , 2016 ) and virtual world ( Li et al. , 2019 ) . Evaluation Metric . We use mean accuracy ( MA ) to evaluate our regression task since it can represent the overall performance under different thresholds . We first define the accuracy with respect to a particular threshold τ as accτ = count ( |vpredicted − vactual| < τ ) /n , where n denotes the number of test cases ; vpredicted and vactual indicate the predicted and ground-truth value , respectively . Then , MA is computed as ∑ τ accτ∈T /|T | , where T = { 1.5 , 3.0 , 7.5 , 15 , 30 , 75 } contains empirically selected thresholds of steering angles . Based on MA , we use the maximum MA improvement ( denoted as MMAI ) , the average MA improvement ( denoted as AMAI ) , and mean Corruption Errors ( mCE ) ( Hendrycks & Dietterich , 2019 ) as the evaluation metrics .

Summary: This work proposes a new method to improve the generalization of ML models for the task of vehicle steering using a hybrid of data augmentation and adversarial examples. In a nutshell, the proposed method attempts to increase the accuracy of the model by dynamically adding a selection of candidate datasets during training. Each of these “candidates” is created offline applying a transform (e.g. blur, distortion, and changes in color representation) to the original (base) dataset. During training, the method chooses among the K transformed-datasets those who minimize the mean validation accuracy and based on this selection the steering model is retrained. The approach is evaluated on a driving dataset.