Efficacy of Pixel-Level OOD Detection for Semantic Segmentation
Authors: Matt Angus
Krzysztof Czarnecki
Rick Salay
Venue:  (2020)
Keywords: Out-of-Distribution Detection
Semantic Segmentation
Deep Learning

Abstract: The detection of out of distribution samples for image classification has been widely researched. Safety critical applications, such as autonomous driving, would benefit from the ability to localise the unusual objects causing the image to be out of distribution. This paper adapts state-of-the-art methods for detecting out of distribution images for image classification to the new task of detecting out of distribution pixels, which can localise the unusual objects. It further experimentally compares the adapted methods on two new datasets derived from existing semantic segmentation datasets using PSPNet and DeeplabV3+ architectures, as well as proposing a new metric for the task. The evaluation shows that the performance ranking of the compared methods does not transfer to the new task and every method performs significantly worse than their image-level counterparts.

Under review as a conference paper at ICLR 2020
EFFICACY OF PIXEL-LEVEL OOD DETECTION FOR SEMANTIC SEGMENTATION
Anonymous authors Paper under double-blind review
ABSTRACT
The detection of out of distribution samples for image classification has been widely researched. Safety critical applications, such autonomous driving, would benefit from the ability to localise the unusual objects causing the image to be out of distribution. This paper adapts state-of-the-ee methods for detecting out of distribution images for image classification to the new task of detecting out distribution pixels, which localise the unusual objects. It further experimen- tally compares the adapted methods on two new datasets derived from existing semantic segmentation datasets using PSPNet and D3eplabV3+ architectures, well as proposing a new metric for the task. The evaluation shows that the per- formance ranking of the compared methods does not transfer to the new task and every method performs significantly worse than their image-level counterparts.
INTRODUCTION
Figure 1: Image from the LostAndFound dataset (Pinggera et al. 2016). where two unlikely objects (storage crates) are almost entirely incorrectly predicted to be road. The Max Softmax method clearly highlights these crates as OOD. (best viewed in colour)
Many applications using machine learning (ML) may benefit from out of distribution (OOD) de- tection improve safety. When inputs are determined to be out of distribution, the output of an ML algorithm should not trusted. A large body of research exists for detecting entire images as OOD for the task of image classification. Image-level OOD detection outputs a classification for the entire image; this coarse level of detection may be inadequate for many safety critical applications, including autonomous driving. Most of the pixels in image taken from onboard camera will be distribution (ID), i.e. an image of a road scene with cars, people, and roadway- but an unusual object that was not part of the training set may cause only a small number of OOD pixels. Extend- ing the framework to semantic segmentation networks will allow each pixel to have an "in" or "out of" distribution classification. Applied to autonomous driving, groups of pixels classified as OOD would be considered as unknown objects. Depending on the location of the unknown objects, planner would then proceed with caution hand over control to safety driver. Another application is automatic tagging of images with OOD objects, which would then be sent for human labelling. Figure shows a failure case where OOD detection beneficial. The two crates are predicted as road. The right image of this figure shows the result of pixel-level OOD detection using one of the proposed methods, which clearly identifies the unusual objects.
This paper adapts existing state-of-the-art image-level OOD detection methods to the new task of pixel-level OOD classification and compares their performance on a new dataset designed for this task. In addition to adapting the methods, address the question of whether the best- perforrero image-level methods maintain their performance when adapted to the new task. In order to answer this question, we also propose pixel-level OOD detection performance metrics, drawing both

---

Under review as a conference paper at ICLR 2020
existing image-level OOD detection and semantic segmentation performance metrics. Further, design two new datasets for pixel-level OOD detection with test images that contain both pixels that are in distribution and pixels that are out of distribution, evaluated with two different network Mnchitectures-PSPNE
(Zhao et al. 2016) and DeeplabV3+ (Chen et al. 2018). Somewhat sur- prisingly, our evaluation shows that the best performing pixel-level OOD detection methods were derived from image-level OOD detection methods that were not necessarily the best performing on the image-level OOD detection task. In summary, the contributions of this paper are the following:
adaptation of image-level OOD detection methods to pixel-level OOD detection and their
evaluation;
training and evaluation datasets for pixel-level OOD detection evaluation derived from ex- isting segmentation datasets; and
a new metric for pixel-level OOD detection, called MaxIoU.
2 ADAPTING OOD METHODS
We use two criteria to select existing image-level OOD detection methods to adapt to pixel- level OOD detection. First, the candidate methods are top performers on image classification datasets. Second, they must be computationally feasible for semantic segmentation. Max Soft- max (Hendrycks & Gimpell 2016), ODIN (Liang et al. 2017), Mahalanobis (Lee et al. 2018) and Confidence (De & Taylor 2018) fit both criteria. The Entropy (Mukhoti & Gal 2018). Sum of Variances (VarSum) (Kendall et al. (015)) and Mutual Information (Mukhoti & Gal 2018) methods do not meet the first criterion, but are included as existing uncertainty baseline for the pixel-level OOD classification. An example method that does not meet the second criterion is an ensemble method by Vyas et al. (2018), with ensemble of K leave-out classifiers. In general, images and architectures for semantic segmentation are larger than for image classification, and therefore an ensemble method is much less feasible for segmentation than classification due GPU memory limitations. Generative Adversarial Networks (GANs) and titto-Enoders (AEs) are often used in the literature as a method for OOD detection. GANs and AEs are excluded limit the scope of this work.
Table briefly describes the selected image-level OOD detection methods and any architecture modifications necessary to adapt them to OOD detection | The Dim reduction modification is an additional penultimate layer that a 1 convolution reducing the depth to 32. DeeplabV3+ has much larger feature extractor than PSPNet, therefore due to hardware limitation, the Mahalanobis method is not evaluated on this architecture. Each original/adapted method produces a value that can be thresholded to predict whether an image/pixel is OOD. All metrics used in our evaluation are threshold independent, therefore no thresholds are discussed in this section. See Appendix A for more detailed description of each OOD detection method.
Method Name
Modifications
Short Description
the maximum predicted prohability after applying the softmax function
Max Softmax
X
augments the Max Softmax hy dividing by a temperature value before applying the softmax function. Higher temperature values make the predictions more uniform.
ODIN
X
models the activations of the penultimate layer over the training set and computes the Mahalanobis distance to the class mcan for each spatial location.
Mahalanobis
Dim reduction
adds second branch to the output that learns the confidence of cach pixel via an extra loss during training.
Confidence
Extra branch
computes the Shannon entropy of the output after the softmax function
Entropy
x
VarSum
Dropout layers
computes the sum variances of the predictions the entropy of the mean prediction minus mean entropy of all pre- dictions over multiple runs
Mutual Information
Dropout layers
Table I: List of methods compared, modifications required, and a short description.
ODIN has a temperature value hyperparameter that is optimised over held-out training sample of O0D images. This should be considered when comparing other methods that do not have access
Code for each method is attached the submission.
2

---

Under review as a conference paper at ICLR 2020
Avg. OOD Prediction
-10-2
6
IDD Es Cityscapes
4
rickshaw bicycle
person
sidewalk
building
traffic sign
rider
sky
truck
car
terrain
pole
traffic light
bus
vegetation motorcycle
wall road fence
Figure 2: Average OOD prediction value of the Entropy method across all images and pixels for each class. The classes are sorted by difference between IDD and Cityscapes from most to least, left to right. The train class is removed as the IDD dataset doesn't 't have any instances.
to OOD data prior to evaluation. This unfair comparison been criticised the past (Hendrycks) et al. 2018). Most methods have input perturbation step; however, this is removed as it also requires access to OOD data before evaluation.
3 ADAPTING DATASETS
Previous work image-level OOD detection designates dataset used for training as the ID dataset, and the dataset for testing as the OOD dataset. This framework is not easily extended to semantic segmentation, any datasets may share features with the ID dataset. For example, people may
exist in both datasets, but one is indoor scenes and the other is outdoor scenes. Blum et al. (2019) attempt get around this issue by inserting animals from the COCO dataset (Lin et al. 2014) or internet images into the Cityscapes dataset. The lack of realism of the inserted images (improper scale and hard boundaries) make the created dataset insufficient for OOD detection. The remainder of this section describes the datasets used in this work and any modifications required.
3.1 TRAINING
The dataset used for training the weights of all networks is the unmodified Cityscapes dataset (Cordts) et al.] 2016). It is the most prominent dataset for training and benchmarking semantic segmentation for road scene images.
3.2 EVALUATION
The diverse SUN dataset (Xiao et al. 2010) and the India Driving Dataset (IDD) (Varma et al. 2018) used the main evaluation datasets. The main motivation for using the SUN dataset is that has a large variety of scenes (e.g. street, forest, and conference room), as well as a large variety of labels (e.g. road, door, and vase). In total there are 908 scene categories and 3819 label categories. Anonymous label submissions are ignored, as their validity is not confirmed. However, the SUN dataset not very realistic in terms of what vehicle camera would see. The IDD dataset is an autonomous vehicle dataset that has all the same labels of Cityscapes with the addition of the auto- rickshaw class. Although the classes are the same, the instances tend to be OOD, due to different building architecture, road infrastructure, etc. This shift in features leads to higher average values OOD predictions as shown in Figure 2 Therefore only the car class as ID and auto-rickshaw class as OOD are used for evaluation.
The SUN dataset labels have to be modified before the dataset can be used for evaluating OOD detection The approach for modifying this dataset is similar to the open world dataset created by Pham et al. (0088) Let C = [person, car, ignore, ...} be the set of labels available Cityscapes. Let {person, roof, chair, }..} be the of labels available in the SUN dataset. Ambiguous classes
"Code for converting the SUN dataset is will he released publicly upon acceptance.
3

---

Under review as a conference paper at ICLR 2020
such "wall" (can appear indoor and outdoor) or "path" (a sidewalk dirt path in the woods) are sorted into a third set C S. The map M CCCU(ODI is defined as:
if E i E A otherwise
M(s) = ignore
OOD
For every image in the SUN dataset, each pixel label 4 gets new label 1 = M(4). Pixels with the "ignore" class are given weight of O during evaluation.
All the images in the SUN dataset have various dimensions. To prevent artefacts in the input after resizing, only images that have more than 640 640 = 409, 600 pixels are used. All images are resized the same size as Cityscupes (1024 2048).
The SUN dataset is split into a train set and an evaluation set (25%/75% split). It is stressed that the training set is used only to select the best hyperparameters of the ODIN method.
Following previous work on image-level OOD detection, synthetic dataset of random normally distributed images are used as well. The random normal noise usually very easily detected by all methods, therefore Perlin noise (Perlin) 1985) images are used as well. Perlin noise is smooth form of noise that has large "blobs" of colour that is much harder to detect and filter. Each of these datasets have the entire image labelled as OOD.
All OOD datasets used are mixed with Cityscapes evaluation sets. The resulting ratio of ID OOD pixels for IDD/Cityscapes and SUN/Cityscapes is about 7.5:1 and 3:1 respectively, Since ODIN requires ODD data for hyperperameter tuning, Cityscapes training set mixed with a held-out training set of OOD data in the same manner the evaluation sets.
PERFORMANCE METRICS
There are five metrics used to evaluate the performance of each model. The first four listed below are same metrics usually used by previous works on OOD detection (Liang et al. 2017 Hendrycks & Gimpel 2016 Lee et al., 2018 DeVries & Taylor 2018). Since the output of semantic segmen- tation is so much larger than image classification, the below metrics must be approximated. This is done by using 400 linearly spaced thresholds between and and tracking all true positives (TP), true negatives (TN), false positives (FP), and false negatives (FN) for each threshold. Each pixel contributes to one of TP, TN FP, FN for given threshold, based the ground truth and OOD prediction value accumulated across all images.
AUROC Area under the receiver operating characteristic (ROC) curve. The ROC curve FP shows the false positive rate (FPR) FP4TN against the true positive rate (TPR) TP4FN. The under this eurve is the AUROC metric.
AUPRC Area under the precision recall (PR) curve. The PR curve shows the precision TP
TF+F against the TPR (or recall). The area under this curve the AUPRC.
FPRaTTPR FPR at 95% TPR. Extracted from the ROC curve, this metric is the FPR
value when the TPR is 95%.
MaxloU - Max intersection over union (loU). IoU is calculated by TP4F74F8 the maximum loU over all thresholds.
TP
MaxloU is
A common metric in the literature, coined detection error (Liang et al. 2017) is excluded as it is linear transformation of the FPRatTPR metric. Therefore, adds useful information.
4.1 MAXIOU
The inspiration for MaxIoU issued from the semantic segmentation community. Mean intersection over union (mloU) is the canonical performance metric for semantic segmentation. MaxIoU is similar, but targets tasks that threshold dependant. Thresholds selected by MaxIoU punish false positives more than AUROC. The optimal threshold generally greater, resulting in fewer positive predictions than for AUROC. To verify that the MaxloU] is complimentary to AUROC, the optimal
4

---

Under review as a conference paper at ICLR 2020
thresholds selected by each were experimentally compared. The mean absolute difference between the threshold selected via Youden index (Youden) 1950) and that chosen for MaxIoU was found to be 0.039.
5 NETWORK ARCHITECTURE
PSPNet (Zhao et al. 2016) and DeeplabV3+ (Chen et al. 2018) network architectures are used with 2015 and Xception (Chollet 2017) feature extractors respectively. The two
the Resnet (He et al.
driving factors for these networks are: near top performance on the Cityscapes benchmark (Cordts) al. 2016) (2.6 and 1.7 less mIoU than the top scoring algorithm) and the final operation before the softmax classification layer being bi-linear upsample. The upsample ensures that any method's OOD prediction will be directly correlated with the pixel prediction from the softmax layer, as most methods rely on the softmax layer. There is clear relationship between any spatial location in the penultimate layer and an output pixel.
The Xception feature extractor is much larger than the Resnet therefore due to hardware limita- tions, the space intensive Mahalanobis method is only evaluated on PSPNet.
EXPERIMENTS
There are two main research questions that this paper focuses on, Each question has an associated experiment.
Do the required modifications to architecture and loss functions negatively affect the semantic segmentation performance of the network?
RQ1:
RQ2: Which OOD detection method performs the best?
To answer RQ1. we evaluate the semantic segmentation performance on Cityscapes using the stan- dard class mean intersection over union (mloU) metric. As long as the performance drop of modified networks is not too large, the modifications do not interfere with the original task. RQ2 is answered by evaluating each method on the datasets described in Section 3
0.8
0.8
AUROC
0.6
AUPRC
0.6
0.4
0.4
Max Softmax
0.2
0.2
S
ODIN
Entrupy
SUN
IDD
Normal
Perlin
SUN
IDD
Normal
Perlin
Mohalanobis
Contidence
VarSum
0.8
0.8
FPRaTTPR
Mutual Info
MaxloU
0.6
0.6
0.4
0.4
0.2
0.2
B
SUN IDD
Normal
Perlin
SUN
GDI
Normal
Perlin
Figure 3: Comparison of methods on different datasets using the PSPNet architecture. The arrow on the y-axis label indicates if larger value is better (1) or smaller value is better ({). Each group of bars are labelled by the dataset used for evaluation.
5

---

Under review as a conference paper at ICLR 2020
AUROC -
0.8
0.8
277777772
0.6
AUPRC
0.6
0.4
0.4
2 Max Softmax
0.2
0.2
ODIN
SUN
IDD
Normal
Perlin
SUN
IDD
Normal Perlin
Entropy
Confidence
VarSum
MaxloU +
0.8
SANNNN
0.8
Murual Info
FPRa(TPR
0.6
0.6
0.4
0.4
0.2
0.2
SUN
IDD
Normal
Perlin
SUN IDD
Normal
Perlin
Figure 4: Comparison of methods different datasets using the V3 architecture. The arrow on the y-axis label indicates if a larger value is better (1) or a smaller value is better ()). Each group of bars are labelled by the dataset used for evaluation.
7
RESULTS AND DISCUSSION
7.1 ARCHITECTURE MODIFICATIONS (RQ1)
Table [ shows the mloU of PSPNet and D3eplabV3+ trained on Cityscapes and evaluated on Cityscapes. The modifications degrade performance only slightly. The maximum performance degradation between the unmodified network and any modified version is small, at 2.4% for PSP- Net and 8.6% for DeeplabV3+ As stated before, Mahalanobis with DeeplabV3+ is too memory intensive, therefore it is not included.
Network
mloU
Network
mloU
PSPNet
0.6721
Deeplab
0.7933
PSPNet + dim reduce
0,6701
Deeplab + dim reduce
PSPNet + dropout 0.6593
DeeplabV3+ + dropout
0.7252
PSPNet confidence
0.6560
Deeplab + confidence
0.7656
Table 2: mloU values for modified PSPNet and Deeplab architectures showing efficacy for original segmentation task. Network modifications are described in Table
7.2 METHOD PERFORMANCE (RQ2)
Figure shows the comparison of the performance of the different methods using the PSPNet ar- chitecture. Each graph shows different metric (c.f. Section 4 for each dataset. Max Softmax, ODIN, and Mahalanobis follow the same trend as their image classification counterparts with the SUN dataset, increasing performance that order. However, for the IDD dataset the order is Mahalanobis, Max Softmax, then ODIN, in increasing performance. For image-level OOD detec- tion, the Confidence method outperforms the Max Softmax baseline. However, for pixel-level OOD detection it is worse. For real datasets, VarSum is worse than Mutual Information, but better on the synthetic datasets.
Across each metric, VarSum has the biggest performance increase from the modified SUN and IDD datasets to the random normal dataset, moving from worst to near top performance. This however not the same for the Perlin noise dataset, as the low frequency noise is not casily filtered. The Confidence method seems to mostly learn to highlight class boundaries, as that where prediction errors are likely to occur. Therefore the prediction loss force lower confidence levels. This makes it less suitable for the random normal and Perlin noise datasets, where the network predicts one single class for the majority of the input.
Figure 4 shows the comparison of the performance of the different methods using the Deeplab architecture. With respect AUROC, AUPRC, and MaxloU on the IDD dataset Mutual Information

---

Under review as a conference paper at ICLR 2020
Figure 5: Comparison of the Mutual Information method on an IDD dataset image, successfully pre- dicted. The top row is using the PSPNet architecture, the bottom row is using the Decplab V3+ archi- tecture. The columns from left right are: input image, ground truth, class prediction (Cityscapes colours), ODD prediction. The OOD prediction is masked to only cars and auto-rickshaws. Best viewed in colour.
has the best performance, followed by ODIN. On datasets and metrics, VarSum has the worst or is near worst performance. Confidence has much better performance with the DeeplabV3+ architec- ture than the PSPNet architecture, especially on the random datasets. The results for DeeplabV34 much less conclusive than PSPNet the relative ordering across the different metrics changes significantly.
The metries values reported in the original works for the image-level OOD detection for all methods with image-level OOD detection counterparts are very close to (~0.95 and above). However, in our experiments, the results are much less than One peculiar result is the significant difference in performance between the Confidence method on Deeplab and PSPNet on the random normal and Perlin noise datasets.
ODIN performss well on all datasets and architectures. The performance of this method is aided by its access OOD data during hyperparameter tuning, however.
7.3 QUALITATIVE DISCUSSION (RQ2)
Figure 5 shows a comparison of the Mutual Information method using both the PSPNet and D3+plabV3+ architecture. Mutual information is one of the worst performing on PSPNet and the best performing on DeeplabV3+. There are two major observations to note. The first is that both networks label different parts of the auto-rickshaw in OOD prediction of all cars from DeeplabV34 to PSPNet, while the OOD prediction of the auto- rickshaw remains approximately the same. Both architectures successfully predict the majority of the pixels correctly; however, the separation is better for D3++
different classes. The second is the increase
Figure 6 shows comparison of the Entropy method using both the PSPNet and Deeplab archi- tecture. The auto-rickshaw is mistaken by both architectures as car. therefore the resulting OOD prediction is similar to other cars in the scene. This failure highlights a more general failure of pixel-level OOD detection methods. When the network is confident about a wrong prediction, the OOD methods are fooled as well.
The drop in performance for pixel-level OOD detection is likely due to features that cause large disruptions the pixel-level, but would not affect an entire image; for example, shadows, occlusion, and far away objects. Figure / shows example of shadows and far away objects in the bottom row. At the end of the road, most pixels are high OOD values as well as the right side the scene, which is in the shade of building. The top row of Figure 7 shows interesting failure case of flooded road being predicted as road with low OOD value.
As be seen in all example outputs, class boundaries vision algorithm was developed, using a series of erosion, dilation and other filters to remove these boundaries. In general performance increased: however, the increase was the order of 10-3
highlighted. A classical computer

---

Under review as a conference paper at ICLR 2020
Figure 6: Comparison of the Entropy method on an IDD dataset image. successfully predicted. The top row is using the PSPNet architecture, the bottom row is using the DecplabV3+ architecture. The columns from left right are: input image, ground truth, class prediction (Cityscapes colours), ODD prediction. The OOD prediction is masked to only cars and auto-rickshaws. Best viewed in colour.
Figure 7: Examples from the SUN (top row) and Cityscapes (bottom row) datasets using PSPNet. The columns from left to right are: input image, ground truth, class prediction (Cityscapes colours), Entropy ODD prediction.
RELATED WORK
To our knowledge, there are very few works researching pixel-level OOD detection. Blum et al. (2119) create dataset by overlaying animals and objects from the COCO dataset (Lin et al. 2014) on top of the Cityscapes dataset (Cordts et al. 2016), this however lacks realism. This new dataset tested with various OOD detection methods. Bevandic al. (2018) train segmentation network with two datasets-on one ID and one OOD dataset. The network learns to classify between the two on per pixel-level. This is compared the Max Softmax baseline. The major flaw in this method is that the network learns its weights from OOD samples.
There
commonalities Active Learning for semantic segmentation (Gorriz et al. 2017 Mackowiak et al.) (2018). These studies attempt to use some form of output uncertainty to choose which images are best for training/labelling in order reduce the number of training examples needed. They produce heat map similar to the OOD detection output. These heat maps are then aggregated across a whole image to produce a score for the entire image.
Pham et al. 2018) create an open world dataset. Known object labels are drawn from the COCO dataset (Lin al. 2014), and labels drawn from the NYU dataset (Silberman et al.] 2012) are relabelled as unknown if the class doesn't exist in COCO. This methodology is very similar to modified SUN dataset Section 3 generic object instance level segmentation algorithm developed, based and is evaluated on the new dataset. This approach splits the image into visually distinct connected regions, but is too slow for real-time applications.
class specific object detector, boundary detector and simulated annealing,
CONCLUSIOON AND FUTURE WORK
Several methods for detecting OOD pixels were adapted from image-level OOD detection, well as a pixel uncertainty estimation. These methods were compared using metrics previously estab- metric that has roots in the semantic segmentation
lished by OOD detection works, as well task. This paper also contributed two new datasets for pixel-level OOD classification derived from semantic segmentation datasets that have common classes but also unique ones.
8

---

Under review as a conference paper at ICLR 2020
There is great room for improvement for pixel-level OOD detection. One shortcoming for all the methods compared in this paper is the ability to distinguish between class boundary pixels and OOD pixels. We tested classical computer vision techniques that could be used to visually fix this problem, but the performance increase was negligible. 'he ODIN and Mahalanobis methods have the best performance with PSPNet and SUN dataset, beating the VarSum, Mutual Information, and Confidence methods by significant margin. However, Mutual Information has the best performance with DeeplabV3+ and the IDD dataset, with the other methods following closely. Therefore the ODIN, Mahalanobis, and Mutual Information methods should be considered the baseline for further research in pixel-Ievel OOD detection.
Understanding the faults of pixel-level OOD detectors is crucial for progress. This would include categorising the failure cases of detector. For example, understanding why a flooded road is not highlighted, and what makes that different to shadows falsely being highlighted.
REFERENCES
Petra Bevandic, Ivan Kreso, Marin Orsic, and Sinisa Segvic. Discriminative out-of-distribution detection for semantic segmentation. CoRR, (bs/1808.00703, 2018. URL htttp://arxiv.
org/abs/1808 1808 07703
Hermann Blum, Paul-Fdouard Sarlin, Juan Nieto, Roland Siegwart, and Cesar Cadena. The fishyscapes benchmark: Measuring blind spots in semantic segmentation, 2019.
Liang- Chich Chen, Yukun Zhu, George Papandreou. Florian Schroff, and Hartwig Adam. Encoder- decoder with atrous separable convolution for semantic image segmentation. ECCV, 2018.
François Chollet. Xception: Deep learning with depthwise separable convolutions. In Proceedings of the IEEE conference computer vision and pultern recognition, pp. 1251-1258, 2017.
Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understanding. CoRR, abs/1604.01688 2016. HRRL/htt://axxiv.oom
1604. 01 665
Terrance DeVries and Graham W Taylor. Learning confidence for out-of-distribution detection in neural networks. arXiv preprint /80..04880. 2018.
Mare Gorriz, Axel Carlier, Emmanuel Faure, and Xavier Giró Nieto. Cost-effective active learning 2017. URL htR/:tpp///arxi..rr//bbs
for melanoma segmentation. CoRR,
1711. 09168
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. CORR, Abs//552..0338. 2015. UR..
0.3385
Dan Hendrycks and Kevin Gimpel. A baseline for detecting misclassified and examples in neural networks. arXiv preprint arXiv:1610.02136 2016.
Dan Hendrycks, Mantas Mazeika, and Thomas G Dietterich. Deep anomaly detection with outlier exposure. arXiv preprint arXiv: 1e:::8812.4444 04606, 2018
Alex Kendall, Vijay Badrinarayanan, and Roberto Cipolla. Bayesian segnet: Model uncertainty in deep convolutional encoder-decoder architectures for scene understanding. arXiv preprint arXiv:1511.02680 2015.
Kimin Lee, Kibok Lee, Honglak Lee, and Jinwoo Shin. A simple unified framework for detecting out-of-distribution samples and adversarial attacks. arXiv preprint
2018
Shiyu Liang, Yixuan Li, and Srikant. Enhancing the reliability out-of-diisrributin image detec- tion in neural networks. arXiv preprint arXiv: 7706022690 2017.
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision, pp. 740-755. Springer, 2014.

---

Under review as a conference paper at ICLR 2020
Radek Mackowiak, Philip Lenz, Omair Ghori, Ferran Diego Oliver Lange, and Carsten Rother. CEREALS cost-effective region-based active learning for semantic segmentation. CoRR, abs/1810.099726 2018. URL http:t wttp://wwwww./////////////////// http:/bbss1110 09726
Jishnu Mukhoti and Yarin Gal. Evaluating bayesian deep learning methods for semantic segmenta- tion. arXiv preprint arXiv:/811.127000 2018.
Ken Perlin. An image synthesizer. SIGGRAPH Comput. Graph., 19(3):287-296, July 1985. ISSN 0097-8930. doi: 10.1145/325165.325544 URL httpttt:////..acee org.////...oo..o....ooo////// 1145/ 3251 615165.22522
Trung Pham, Vijay BG Kumar, Thanh-Toar Do, Gustavo Carneiro, and Ian Reid. Bayesian mantic instance segmentation in open set world. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 3-18, 2018.
Peter Pinggera, Sebastian Ramos, Stefan Gchrig, Uwe Franke, Carsten Rother, and Rudolf Mester. Lost and found: detecting small road hazards for self-driving vehicles. In 2016 JEEE/RSJ Inter- national Conference on Intelligent Robots and Systems (IROS), pp. 1099-1106. IEEE, 2016.
Claude Elwood Shannon. mathematical theory of communication. Bell system technical journal, 27(3):379-422 1948.
Nathan Silberman, Derek Hoiem, Pushmeet Kohli, and Rob Fergus. Indoor segmentation and sup- port inference from rgbd images. In European Conference on Computer Vision, pp. 746-760. Springer, 2012.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: simple way to prevent neural networks from overfitting. The Journal of Machine Learning Research, 15(1):1999-1988 2014.
Girish Varma, Anbumani Subramanian, Anoop M. Namboodiri, Manmohan Chandraker, and C. V. Jawahar. IDD; A dataset for exploring problems of autonomous navigation in unconstrained en- vironments. CoRR, abs/1811..0200 2018. URL http:://///atiit orw./org/aaaooo haap///b///aa abs11111.1111000
Apoorv Vyas, Nataraj Jammalamadaka, Xia Zhu, Dipankar Das, Bharat Kaul, and Theodore Willke. Out-of-distribution detection using ensemble of self supervised leave-out classifiers. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 550-564, 2018.
Jianxiong Xiao, James Hays, Krista A Ehinger, Aude Oliva, and Antonio Torralba. Sun database: Large-scale scene recognition from abbey to zoo. In Computer vision and pattern recognition (CVPR), 2010 IEEE conference on, pp. 3485-3497 3492.. IEEE, 2010.
William J Youden. Index for rating diagnostic tests. Cancer, 3(1):32-35, 1950.
Hengshuang Zhao, Jianping Shi, Xiaojuan Qi, Xiaogang Wang, and Jiaya Jia. Pyramid scene parsing network. CoRR, abs/1612.01 105, 2016. /tt/
A OOD DETECTION METHODS
Each subsection uses notation close to the original works, so that readers can refer to said works for more detail. The appendix also provides additional detail. Notation should not be carried between sections unless explicitly stated. Assume the neural network is a function f. and input image is æ. P. For example fx:;; is the 4t pixel
The set of pixels P will be used with typical subscripts of the result of f(x).
Note that most methods have input perturbation step. They included here for completeness, however, they are not used in any evaluation as requires access to OOD data. This access to OOD data has been criticised unfair comparison to methods that do require access OOD data.
10

---

Under review as a conference paper at ICLR 2020
A.1 VARSUM
At test time, dropout (Srivastava et al., 2014) can be used as estimate for model uncertainty. A multiplier or is randomly chosen for each neuron at test time. (Kendall et al. 2015) use the variation in predictions of a model that uses dropout at test time to compute estimate of model uncertainty. The output is a variance per class, the sum these variances is the estimate of model uncertainty. This is performed per pixel get the value 8; for pixel E P.
A.2 MAX SOFTMAX
Hendrycks & Gimpel (2016) show that the max softmax value can be used to detect OOD examples as baseline for image classification. The softmax function is defined as:
Sy(x) = Σx 6
(2)
Sg(r) = max Sz)
(3)
Where Sz(x) is the max softmax value and ý is used to signify the chosen index.
Given a prediction p; = f(z), the max softmax value for each pixel is os = Sg. (p.). and that value is used to determine if that pixel is OOD.
A.3 ODIN
Liang et al. (017) create a similar method to the max softmax value, dubbed ODIN. This method adds temperature scaling and input preprocessing. The softmax function Equation is modified to include a temperature value T:
S(x;T) S s
(4)
Sy(x; max (x;
(5)
Liang et al. found that perturbing the input in the direction of the gradient influences ID samples more than OOD samples, thus separating II) and OOD examples more. The input preprocessing step is:
5= x
sign Σ V log Sp(f(z);:
(6)
where is hyperparameter chosen from set of 21 evenly spaced values starting at and ending at 0.004. The best temperature value is chosen from predefined set of temperatures (1,2,5, 10, 20, 50, 100, 200, 1000).
The temperature-sealle and preprocessed max softmax score v; = Ss(;; for pixel is used to predict if that pixel is OOD.
A.4 MATIALANOBIS
Lee et al. (2018) use the Mahalanobis distance for detecting OOD samples. The Mahalanobis dis- tance number of standard deviations vector away from the mean, generalised to many dimensions.
Each feature vector at spatial location in the penultimate layer is assumed to be normally dis- tributed. Each distribution is parameterised by the pixel class mean Mei and global class covariance Σg for pixel and class c. The global class covariance is computed for all pixels of a given class, independent of their location. Initial tests showed that using pixel class means and global class variance has better performance than global or pixel class mean and pixel class covariance, therefore they used throughout. The labels resized to match the height and width of the penultimate

---

Under review as a conference paper at ICLR 2020
layer using nearest neighbour interpolation. The two quantities jei and E. computed as follows:
1
Σ f((;)):
Mei
No
3:94 =e
Σ.
1 No
E
(fi(X)) ((x)) Mai)
(8)
where N₂ is the number of examples of class c, and Xj is the 7th example of the training dataset X. St is the output of the 1th layer of the network Here is the second last layer.
Each spatial location has a class distance and minimum distance computed by:
Ma(ar); = v (i(()); - Mos) va.)) )o.) - MMax) M(x): min M.(r):
(10)
This increases the number of matrix multiplications required to compute each pixel distance. Due to hardware memory limitations, a dimensionality reduction layer is needed after the penultimate layer, reducing the depth from 512 to 32 via a 1 convolution. An input preprocessing step is also performed. The new input i is computed by:
f = x 6 sign
Σ M();; iEP
(11)
Instead of a logistic regression layer, the minimum distance is normalised to have zero mean and unit variance. The sigmoid function a is applied to clamp to the interval [0, 1]:
0; = M(();; 1
(12)
where i and are the mean and standard deviation of all (xx computed over the whole training dataset. v is used to determine if pixel i OOD. The prediction values are resized, with bi-linear interpolation, to the original input size.
A.5 CONFIDENCE ESTIMATION
Veiries & Taylor (2018) train secondary branch of an image classification network to output single confidence value. This confidence value is learned via a regularization loss. The loss gives "hints" to the network when the confidence is low, and it penalizes low confidence.
Similar to the image classification method, a secondary branch added to the network f. Therefore C, f()), where is the confidence value and p; is the prediction vector for pixel i E P. The new branch is trained by creating a new prediction pl as:
b; ~ -0.55)
(13)
cf c, b, + - b.)
(14)
pl de p: + c.) y1:
(15)
where is Bernoulli distribution and y is the one hot ground truth vector for pixel The mean negative log likelihood is then applied to the new pl. as well as regularization term is added to force each e (i.e. high confidence):
1
C.
Σ
log(p/)
(16)
P
iEP
1
C. P
log(c:)
(17)
iEP
L L, XLe
(18)
12

---

Under review as a conference paper at ICLR 2020
where L is the total loss that is used to train the network, and À is hyperparameter. À is 0.5 in all experiments.
At test time preprocessing step is applied each pixel, and is computed using the gradients of the Le loss. Note that the adapted Lg sums over all the pixel confidence predictions c; therefore the gradient implicitly sums output pixels well.
sign (V.) Le)
(19)
Let pi, C f(()), then 0ú used determine if pixel OOD.
A.6 ENTROPY
Shannon entropy (Shannon 1948 is an information theoretic concept, that is used to determine how much information a source contains. The entropy equation is H T R:
H(x) Σ
z4 log 24
(20)
Since H was developed for probabilities, must behave like probability distribution meaning:
Σ x =1=11
(21)
(22)
Hendrycks et al. (2018) train an image-level classifier with third outlier dataset that is disjoint from both the training and OOD dataset. An auxiliary loss is added minimising the entropy of predictions of outlier images. The network learns to predict uniform values for all classes. The max softmax value is then used at test time to determine if sample is OOD.
The entropy function is applied to the softmax output, which satisfies the properties in Equations 21
and Let = f(x) then
= softmax P)
(23)
H(S)
(24)
0/ is used to determine if pixel i is OOD.
A.7 MUTUAL INFORMATION
Mukhoti & Gal 2018) develop Bayesian uncertainty estimation methods and evaluation metrics evaluate if higher uncertainty correlated with higher error rates. The estimation can be used for OOD detection as well, similar to the Entropy and VarSum methods. There are two quantities required. Predictive entropy:
H(()))) = Σ p(y k2z) log[p(y = k|x)]]
(25)
and Aleatoric Entropy:
AE(y = T
Σ p(y k)),w,) log[p(y = k|2, wi]]
(26)
R.
In both equations, is the input and is the prediction. wi is the of weights selected by dropout at iteration i. Mutual information is then
Ml(y) [H()) AE (x)
(27)
M][() used determine if pixel is OOD or ID.
13


Reviews:
[{"review_id": "SJguyu66Fr", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "# Summary\n\nThis paper puts forward a study over out-of-distribution (OOD) detection for semantic segmentation. OOD detection is an active area research which has recently dealt mostly with image level classification. For semantic segmentation the same conclusions might not apply since decisions must be taken for each pixel individually. To this effect the authors propose here to study this task over a set of architectures (PSPNet, DeepLabV3+), outlier datasets (SUN, Indian Driving Dataset, synthetic images) and multiple methods for OOD detection from recent works on image classification. \nA major difficulty in OOD works is the definition of a relevant OOD dataset and evaluation setup, and the authors propose here a novel setup for this task by adjusting the SUN and IDD datasets as OOD for Cityscapes. \nThe experimental part is thorough with multiple evaluation metrics and some qualitative examples and discussion. \n\n# Rating\nAlthough the paper studies a meaningful and interesting problem, I would reject the paper for the following reasons (more detailed arguments some lines below):\n1) I find that the proposed dataset setups for OOD detection are not adequate for semantic segmentation and thus alter the reliability of the results.\n2) Other than the evaluation (which has some faults), there is no proposed method addressing this task and its challenges.\n3) The proposed MaxIoU metric proposed is not discussed and compared in more detail against the usual ones for this task.\n\n\n# Strong points\n- This work is dealing with a highly interesting and challenging problem. Indeed there has been few studies in this area and defining proper techniques and evaluation setups is challenging.\n- The authors consider a wide array of methods for evaluation and test them across two architectures and multiple real and synthetic datasets.\n\n# Weak points\n- There are two real datasets considered for OOD evaluation, but I consider there are some flaws in their utilization for OOD detection. First, the SUN dataset is quite different from Cityscapes with a significant domain gap (at least in the visual appearance and distributions of the classes) between the two datasets. Upsampling the SUN images to 5x their size in order to make them compatible to Cityscapes should increase the artefacts even further making it easier to spot the gap between SUN and Cityscapes. This means that there is a risk of having the OOD detector acting merely as domain classifier spotting whenever the domain is different from the one use for training. \nThis argument applies to IDD, although to a smaller extent as the datasets are both automotive, but again there are strong visual differences between samples of the same class across the two datasets.\nThe authors argue that they somehow take this argument into consideration (Figure 2) and select only the non-ID classes to perform evaluation. However in both networks the scene information is mixed into the representation (via pyramid pooling in PSPNet  respectively via a trous convolutions with different dilations in DeepLabV3+). So again, instead of an OOD detector we can end up doing domain classification.\nIt would be useful to see how is the classification performance changing for ID classes, e.g. how is a model trained on Cityscapes scoring for cars and other ID objects in IDD comparing to a model that is trained on IDD for the same classes. A big difference between these scores would correspond to a significant domain gap and in correlation with the OOD performance we might be able to take some more conclusions on the matter. \nAhmed and Courville[i] propose an interesting discussion on this type of problems and propose focusing on semantic anomaly detection, i.e. detecting different classes from the same dataset, to make sure the setting has practical interest. They propose a very basic technique to detect OOD in previous classification setups showing the limitations of previous OOD methods. I encourage the authors to check the arguments stated in that work.\n\n\n- In my opinion, the Fishyscapes work from Blum et al. is unfairly dismissed here by considering only a part of the benchmark for which animals from COCO and internet are inserted over Cityscapes images. The authors argue that this lack of realism of the inserted images make this dataset insufficient for OOD detection. However in the first version of their paper, Blum et al. propose a mix of Foggy Driving, Foggy Zurich, WildDash and Mapillary as dataset for ODD detection, which is similar with the setup proposed here. Furthermore, the latest version of Fishyscapes includes the Lost & Found dataset (mentioned in Fig. 1 here) which is recorded in similar conditions with Cityscapes with the addition of a few small outlier objects used as OOD. This is a relevant dataset and work and I would adjust the critics brought to their work here. That paper has the same objectives and endeavors as the current submission.\n\n\n- Although there are some discussions and experiments on multiple techniques there is not technical contribution mitigating all the limitations of previous OOD methods on classification and the challenges of OOD detection in semantic segmentation. This would have had greatly helped the paper.\n\n- I find that the MaxIoU metric considered here is not sufficiently discussed and analysed to show its utility and the additional perspective it brings when evaluating along with the usual metrics.\n\n## Other less important weak points\n- The choice of dataset in Figure 1 , i.e. Lost & Found, can be misleading. This dataset is not further mentioned and evaluated in the rest of the paper. The image could be replaced with a qualitative results from the rest of the evaluation.\n\n# Suggestions for improving the paper:\n1) The current evaluation setting could have some flaws. I would propose some sanity checks and look at the classification performances over other ID classes, as suggested in the previous section\n\n2) Evaluate on a setting similar to Fishyscapes Lost and Found, in which the dataset does not change much, but there are some novel objects. \n\n3) Include a trivial OOD baseline in the spirit of [i] to show the utility of the proposed datasets for this task by being robust to such baselines.\n\n4) Consider extending the breadth of OOD methods with Deep Prior Networks[ii] which have been shown to perform well on Fishyscapes for OOD detection.\n\n5) Add a qualitative example with OOD detection on Perlin noise images\n\n# References\n[i] F. Ahmed and A. Courville, Detecting semantic anomalies, arxiv 2019 https://arxiv.org/abs/1908.04388\n\n[ii] A. Malinin and M. Gales, Predictive uncertainty estimation via prior networks, NeurIPS 2018\n\n\n\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "HylBaYf6FB", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "- This paper proposes a comparative study of out-of-distribution (OOD) methods for semantic segmentation. To this end, authors extend networks designed for OOD image detection to accommodate for the segmentation task. For evaluation purposes authors create a new dataset and use 3 well-known metrics, as well as a new proposed metric. \n- The paper is in general a bit dense to read and understand, since authors fail to explain many important details. Furthermore, the structure of the paper can be improved (e.g., related work is added at the end of the manuscript). \n- Technical contribution of this work is insufficient. Authors merely employ ODD classification networks to address the pixel-level OOD task. Nevertheless, the motivation of those changes are never detailed. Furthermore, some other important aspects are not explained. For example, typically classification networks are adapted for segmentation adding a decoding path, so that the output result is a map of the same size of the input times number of classes. However, how the probability map for the pixel-level OOD predictions is never explained. \n- Authors also mention that GANs and AEs are excluded to limit the scope of the paper. First, this reason is not convincing. Second, I believe that including all the significant works for the task-at-hand is more relevant. How these methods would work compared to the proposed networks?\n- Unless I miss something, the adapted OOD versions degrade the semantic segmentation performance of original networks. Why not to use the original versions instead?\n- Results are very unclear. Which dataset represents the OOD evaluation? Further, authors talk about results at image-level and pixel-level. Nevertheless, this is not detailed in the experimental section. Fig. 3 reports results of the different models at pixel-level. Where are the results of OOD at image-level? In addition, results are barely interpreted, and authors basically describe the values reported in the figures. I would appreciate a deeper interpretation of the results.\n- I am curious to know why the confidence OOD approach has much better performance with DeeplabV3+ than with PSPNet. While PSPNet is among the worst performing models with confidence (sometimes the worst), Deeplab + confidence is typically top-ranked. Do the authors have any insight on this?\n- There is no evidence that the proposed metric better models the OOD pixel-level performance than other standard metrics. Which are the results on this task achieved by mIOU instead? ", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "BkgE5OCoYr", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper evaluates a variety of existing pixel-wise out-of-distribution detection methods in the task of semantic segmentation of road scenes. To do so, the paper introduces an evaluation protocol and applies it to two datasets (SUN and IDD) and two models (PSPNet and DeepLabV3+).\n\nStrengths:\n- The paper is well written with high quality visuals and plots\n- The paper studies an important problem\n\nWeaknesses:\n- The contribution seems to be rather incremental (evaluating existing methods on 2 dataset) and some related work might be missing\n- Although the analysis is well executed, it is not clear what the community learns from the paper\n\n\nAlthough I enjoyed reading the paper, I'd lean towards rejection of the paper. My main concern are as follows:\n\nIt is not clear what the community learns after reading this paper. Is the difference between different approaches significant? Are the class boundary pixels the biggest problem? The paper would benefit strongly from providing some indications of future steps, e. g. how to improve OOD in semantic segmentation.\n\nI'm missing a discussion between OOD and  and the prior work on uncertainty estimation in semantic segmentation (e. g. https://arxiv.org/pdf/1703.04977.pdf and the follow up works, or https://arxiv.org/pdf/1807.00502.pdf). It seems that uncertainty estimates for semantic segmentation could be applied to the tested scenarios off-the-shelf without the need for additional modifications. Is there any reason the paper did not include approaches for uncertainty estimation in semantic segmentation? In general, it would be useful to connect the tested OOD scenarios to other topics already studied in semantic segmentation such as: uncertainty estimation, outlier detection, and distribution shift in semantic segmentation. A paragraph drawing connections and highlighting the differences would make the paper stronger.\n\nOther comments:\n\nSection 3.2, \"Therefore only the car class as ID...\" I'm not sure I understand why car class is the only one considered ID for IDD. From Fig. 2, it seems that other classes such as bus, traffic light, pole, terrain, etc. could be also considered. Could the authors comment on this?\n\n\"The random normal noise is usually very easily detected by all methods, therefore Perlin noise images are used\". However, when looking at the results Fig 3 and Fig 4 it feels like Normal random noise is harder than Perlin noise. Could the authors comment on this?\n\n\"All OOD datasets used are mixed with Cityscapes evaluation sets\". Why it is important to add Cityscapes images to evaluation set? Wouldn't it be enough to use OOD datasets?\n\nOne suggestion of a plot that could jointly display the information from RQ1 and RQ2 would be to plot both of them in a one scatter plot (e. g. with ID IoU on x-axis and AUROC on the other).\n\nFigures 3 and 4 show results for 6 methods, while Table 2 only displays 3 scenarios for 2 models. Table 3 would benefit from including all 6 models and using the same labels as in Figures. Moreover, it would be interesting expand Table 2 by including the performance of the segmentation on in distribution classes from IID and SUN datasets in addition to Cityscapes results. ", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper studies the problem of out-of-distribution (OOD) detection for semantic segmentation.

Reviewers and AC agree that the problem might be important and interesting, but the paper is not ready to publish in various aspects, e.g.,  incremental contribution and less-motivated/convincing experimental setups/results.

Hence, I recommend rejection.
Decision: Reject