Efficacy of Pixel-Level OOD Detection for Semantic Segmentation

1 INTRODUCTION . Many applications using machine learning ( ML ) may benefit from out of distribution ( OOD ) detection to improve safety . When inputs are determined to be out of distribution , the output of an ML algorithm should not be trusted . A large body of research exists for detecting entire images as OOD for the task of image classification . Image-level OOD detection outputs a classification for the entire image ; this coarse level of detection may be inadequate for many safety critical applications , including autonomous driving . Most of the pixels in an image taken from an onboard camera will be in distribution ( ID ) , i.e . an image of a road scene with cars , people , and roadway—but an unusual object that was not part of the training set may cause only a small number of OOD pixels . Extending the framework to semantic segmentation networks will allow each pixel to have an “ in ” or “ out of ” distribution classification . Applied to autonomous driving , groups of pixels classified as OOD would be considered as unknown objects . Depending on the location of the unknown objects , a planner would then proceed with caution or hand over control to a safety driver . Another application is automatic tagging of images with OOD objects , which would then be sent for human labelling . Figure 1 shows a failure case where OOD detection is beneficial . The two crates are predicted as road . The right image of this figure shows the result of pixel-level OOD detection using one of the proposed methods , which clearly identifies the unusual objects . This paper adapts existing state-of-the-art image-level OOD detection methods to the new task of pixel-level OOD classification and compares their performance on a new dataset designed for this task . In addition to adapting the methods , we address the question of whether the best-performing image-level methods maintain their performance when adapted to the new task . In order to answer this question , we also propose pixel-level OOD detection performance metrics , drawing both on existing image-level OOD detection and semantic segmentation performance metrics . Further , we design two new datasets for pixel-level OOD detection with test images that contain both pixels that are in distribution and pixels that are out of distribution , evaluated with two different network architectures—PSPNet ( Zhao et al. , 2016 ) and DeeplabV3+ ( Chen et al. , 2018 ) . Somewhat surprisingly , our evaluation shows that the best performing pixel-level OOD detection methods were derived from image-level OOD detection methods that were not necessarily the best performing on the image-level OOD detection task . In summary , the contributions of this paper are the following : • adaptation of image-level OOD detection methods to pixel-level OOD detection and their evaluation ; • training and evaluation datasets for pixel-level OOD detection evaluation derived from existing segmentation datasets ; and • a new metric for pixel-level OOD detection , called MaxIoU . 2 ADAPTING OOD METHODS . We use two criteria to select existing image-level OOD detection methods to adapt to pixellevel OOD detection . First , the candidate methods are top performers on image classification datasets . Second , they must be computationally feasible for semantic segmentation . Max Softmax ( Hendrycks & Gimpel , 2016 ) , ODIN ( Liang et al. , 2017 ) , Mahalanobis ( Lee et al. , 2018 ) and Confidence ( DeVries & Taylor , 2018 ) fit both criteria . The Entropy ( Mukhoti & Gal , 2018 ) , Sum of Variances ( VarSum ) ( Kendall et al. , 2015 ) , and Mutual Information ( Mukhoti & Gal , 2018 ) methods do not meet the first criterion , but are included as an existing uncertainty baseline for the pixel-level OOD classification . An example method that does not meet the second criterion is an ensemble method by Vyas et al . ( 2018 ) , with an ensemble of K leave-out classifiers . In general , images and architectures for semantic segmentation are larger than for image classification , and therefore an ensemble method is much less feasible for segmentation than classification due to GPU memory limitations . Generative Adversarial Networks ( GANs ) and Auto-Encoders ( AEs ) are often used in the literature as a method for OOD detection . GANs and AEs are excluded to limit the scope of this work . Table 1 briefly describes the selected image-level OOD detection methods and any architecture modifications necessary to adapt them to OOD detection.1 The Dim reduction modification is an additional penultimate layer that is a 1 × 1 convolution reducing the depth to 32 . DeeplabV3+ has a much larger feature extractor than PSPNet , therefore due to hardware limitation , the Mahalanobis method is not evaluated on this architecture . Each original/adapted method produces a value that can be thresholded to predict whether an image/pixel is OOD . All metrics used in our evaluation are threshold independent , therefore no thresholds are discussed in this section . See Appendix A for more detailed description of each OOD detection method . ODIN has a temperature value hyperparameter that is optimised over a held-out training sample of OOD images . This should be considered when comparing to other methods that do not have access 1Code for each method is attached to the submission . to OOD data prior to evaluation . This unfair comparison has been criticised in the past ( Hendrycks et al. , 2018 ) . Most methods have a input perturbation step ; however , this is removed as it also requires access to OOD data before evaluation . 3 ADAPTING DATASETS . Previous work on image-level OOD detection designates a dataset used for training as the ID dataset , and the dataset for testing as the OOD dataset . This framework is not easily extended to semantic segmentation , as any two datasets may share features with the ID dataset . For example , people may exist in both datasets , but one is indoor scenes and the other is outdoor scenes . Blum et al . ( 2019 ) attempt to get around this issue by inserting animals from the COCO dataset ( Lin et al. , 2014 ) or internet images into the Cityscapes dataset . The lack of realism of the inserted images ( improper scale and hard boundaries ) make the created dataset insufficient for OOD detection . The remainder of this section describes the datasets used in this work and any modifications required . 3.1 TRAINING . The dataset used for training the weights of all networks is the unmodified Cityscapes dataset ( Cordts et al. , 2016 ) . It is the most prominent dataset for training and benchmarking semantic segmentation for road scene images . 3.2 EVALUATION . The diverse SUN dataset ( Xiao et al. , 2010 ) and the India Driving Dataset ( IDD ) ( Varma et al. , 2018 ) are used as the main evaluation datasets . The main motivation for using the SUN dataset is that it has a large variety of scenes ( e.g . street , forest , and conference room ) , as well as a large variety of labels ( e.g . road , door , and vase ) . In total there are 908 scene categories and 3819 label categories . Anonymous label submissions are ignored , as their validity is not confirmed . However , the SUN dataset is not very realistic in terms of what a vehicle camera would see . The IDD dataset is an autonomous vehicle dataset that has all the same labels of Cityscapes with the addition of the autorickshaw class . Although the classes are the same , the instances tend to be OOD , due to different building architecture , road infrastructure , etc . This shift in features leads to higher average values OOD predictions as shown in Figure 2 . Therefore only the car class as ID and auto-rickshaw class as OOD are used for evaluation . The SUN dataset labels have to be modified before the dataset can be used for evaluating OOD detection.2 The approach for modifying this dataset is similar to the open world dataset created by Pham et al . ( 2018 ) . LetC = { person , car , ignore , ... } be the set of labels available in Cityscapes . Let S = { person , roof , chair , ... } be the set of labels available in the SUN dataset . Ambiguous classes 2Code for converting the SUN dataset is will be released publicly upon acceptance . such as “ wall ” ( can appear indoor and outdoor ) or “ path ” ( a sidewalk or a dirt path in the woods ) are sorted into a third set A ⊂ S. The mapM : S → C ∪ { OOD } is defined as : M ( s ) = s if s ∈ C \A ignore if s ∈ A OOD otherwise ( 1 ) For every image in the SUN dataset , each pixel label li gets a new label l′i =M ( li ) . Pixels with the “ ignore ” class are given a weight of 0 during evaluation . All the images in the SUN dataset have various dimensions . To prevent artefacts in the input after resizing , only images that have more than 640 · 640 = 409 , 600 pixels are used . All images are resized to the same size as Cityscapes ( 1024× 2048 ) . The SUN dataset is split into a train set and an evaluation set ( 25 % /75 % split ) . It is stressed that the training set is used only to select the best hyperparameters of the ODIN method . Following previous work on image-level OOD detection , a synthetic dataset of random normally distributed images are used as well . The random normal noise is usually very easily detected by all methods , therefore Perlin noise ( Perlin , 1985 ) images are used as well . Perlin noise is a smooth form of noise that has large ” blobs ” of colour that is much harder to detect and filter . Each of these datasets have the entire image labelled as OOD . All OOD datasets used are mixed with Cityscapes evaluation sets . The resulting ratio of ID to OOD pixels for IDD/Cityscapes and SUN/Cityscapes is about 7.5:1 and 3:1 respectively . Since ODIN requires ODD data for hyperperameter tuning , Cityscapes training set is mixed with a held-out training set of OOD data in the same manner as the evaluation sets . 4 PERFORMANCE METRICS . There are five metrics used to evaluate the performance of each model . The first four listed below are the same metrics usually used by previous works on OOD detection ( Liang et al. , 2017 ; Hendrycks & Gimpel , 2016 ; Lee et al. , 2018 ; DeVries & Taylor , 2018 ) . Since the output of semantic segmentation is so much larger than image classification , the below metrics must be approximated . This is done by using 400 linearly spaced thresholds between 0 and 1 and tracking all true positives ( TP ) , true negatives ( TN ) , false positives ( FP ) , and false negatives ( FN ) for each threshold . Each pixel contributes to one of TP , TN , FP , FN for a given threshold , based on the ground truth and OOD prediction value – accumulated across all images . • AUROC – Area under the receiver operating characteristic ( ROC ) curve . The ROC curve shows the false positive rate ( FPR ) FPFP+TN against the true positive rate ( TPR ) TP TP+FN . The area under this curve is the AUROC metric . • AUPRC – Area under the precision recall ( PR ) curve . The PR curve shows the precision TP TP+FP against the TPR ( or recall ) . The area under this curve is the AUPRC . • FPRatTPR – FPR at 95 % TPR . Extracted from the ROC curve , this metric is the FPR value when the TPR is 95 % . • MaxIoU – Max intersection over union ( IoU ) . IoU is calculated by TPTP+FP+FN . MaxIoU is the maximum IoU over all thresholds . A common metric in the literature , coined detection error ( Liang et al. , 2017 ) , is excluded as it is a linear transformation of the FPRatTPR metric . Therefore , it adds no useful information .

Summary: This paper puts forward a study over out-of-distribution (OOD) detection for semantic segmentation. OOD detection is an active area research which has recently dealt mostly with image level classification. For semantic segmentation the same conclusions might not apply since decisions must be taken for each pixel individually. To this effect the authors propose here to study this task over a set of architectures (PSPNet, DeepLabV3+), outlier datasets (SUN, Indian Driving Dataset, synthetic images) and multiple methods for OOD detection from recent works on image classification.