title: Detecting Anything Overlooked in Semantic Segmentation
Creators: name: Chan, Robin
Publisher: name: Bergische Universität Wuppertal
Year: 2022

description: Computer vision is the field that deals with enabling machines to gain an understanding of the content in digital images. This field has seen spectacular advances in recent years through the introduction of deep learning. However, deep learning models are all driven by data and as such they are ill-equipped to handle data samples from categories that they rarely or even never have previously encountered. Consequently, these models are potentially prone to overlook relevant examples. The focus of this dissertation lies on detecting objects that initially have been overlooked in the semantic segmentation of street scenes. Semantic segmentation can be described as pixel-wise image classification problem, which is typically approached by using deep convolutional neural networks (CNNs). These models are usually trained on a closed set of predefined object classes. This procedure, however, could be a source of two types of classification failures. On the one hand, CNNs might overfit their training data, which possibly results in an undesirable bias towards predicting object classes that dominate the dataset in terms of frequency of occurrence. Methods that properly cope with the imbalance problem is crucial in order to prevent the non-detection of naturally underrepresented but particularly relevant instances, such as humans in street scenes. On the other hand, CNNs in real-world applications have to face diversity of the real-world, which necessarily includes previously unseen and semantically unknown object types, i.e. objects from categories on which a model is not trained and thus unable to operate reliably. Detecting and localizing such unknown instances is extremely safety critical in high-stake applications relying on semantic segmentation, especially in automated driving, if they appear as obstacles on the road ahead of the self-driving car. In this work, we present methods addressing both shortcomings. Regarding the problem of class imbalance, we investigate the concept of cost-based decision rules in CNNs and aim to make their ethical difficulties transparent. Furthermore, we introduce and extensively inspect the maximum likelihood decision rule for the reduction of false negative instances of humans as underrepresented class in the semantic segmentation of street scenes. We additionally combine this approach with methods for false positive detection based on aggregated pixel-wise uncertainty metrics. This yields a novel final decision rule, which significantly improves the trade-off between false negative and false positive error rates with respect to infrequent classes. Concerning the problem of unknown objects, we examine their pixel-level identification by thresholding on the softmax entropy as measure for prediction uncertainty of CNNs. We subsequently present a retraining approach using a multi objective loss function that maximizes the entropy of semantic segmentation models on unknown objects while allowing only a marginal drop in original semantic segmentation performance. To this end, we utilize training samples extracted from a large database consisting of images with everyday objects. Compared to pure entropy thresholding, we achieve a clear gain in effectiveness at detecting unknown obstacles on roads, which is additionally enhanced by again combining with methods for false positive detection. Lastly, we create a public benchmark for the pixel-level identification of unknown objects. On two new datasets, consisting of real-world and pixel-annotated images, we provide an in-depth evaluation of multiple state-of-the-art methods using established as well as novel and practically relevant performance metrics.