Venue: NeurIPS 2021 Track Datasets_and_Benchmarks Round2

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: This paper introduces an anomaly(obstacle) segmentation dataset with real-world applications. A detailed benchmark is also included.
strengths: 1. The motivation of an anomaly segmentation dataset is good with real-world applications.
2. The proposed dataset is well constructed and documented.
4. A very comprehensive benchmark is included containing results from many different methods. Detailed settings are also provided.
weaknesses: 1. Although the anomaly(obstacle) categories are more diverse, the overall scale of the dataset is still relatively small compared to other similar datasets.
2. This dataset doesn't provide a training set while the domain shift between the training and test set could greatly affect the models' performance. Since all models are trained on Cityscapes, this paper doesn't show how different training domains may affect models' performance.
correctness: Yes
clarity: Yes
relation to prior work: Yes
additional feedback: None
review_title: Review of SegmentMeIfYouCan

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: This paper introduces two datasets for anomaly and obstacle detection in addition to a well-established evaluation suite for benchmarking models against these datasets. Additionally, the work provides baseline networks from different family of approaches for anomaly detection and explores the impact of different evaluation criteria on perceived performance.
strengths: 1.	The paper tackles anomaly detection, a critical problem in ML evaluation, particularly in real-world settings like autonomous driving.
2.	The experimentation and evaluation is very well detailed and explains how anomaly detection can be explored in a cohesive evaluation environment 
3.	The authors have provided very detailed documentation (including a website), a leaderboard, and evaluation suite to facilitate interaction with the data

weaknesses: 1.	The provided datasets classify anomalies/obstacles based on classes in only one dataset – CityScapes. Different AV and semantic segmentation datasets may have different labels, so something that is an anomaly relative CityScapes may not be an anomaly relative to another dataset. Do the authors have some guidelines for submitting models to this challenge? Do models have to be trained on the CityScapes dataset?
2.	The authors hide the labelled test data from all users, which may limit user’s ability to rapidly prototype models and access data. The authors do provide a limited number of images and annotations for each dataset, which will help future users, but it would be helpful if the authors can detail how they anticipate users to interact with the dataset especially when users need to rapidly benchmark their models (i.e. for deadlines, etc).
3.	The authors mention that RoadObstacle21 contains 85 images in night and in extreme weather. It would be helpful if the authors provide benchmark results on this dataset, even though it has a distribution shift, to showcase how current out of the box methods perform on this data.

correctness: The dataset curation, experimental setup, and evaluation is correct
clarity: The paper is well-written.
relation to prior work: The related work is well detailed and Table 1 gives a good summary of the differences between this dataset and other datasets.
additional feedback: N/A
review_title: A useful dataset for anomaly and obstacle detection for autonomous vehicle settings

final_score: rating: 5: Marginally below acceptance threshold
review_content: summary and contributions: The paper presents a new dataset and benchmark for the localization of anomalies in autonomous driving scenarios. The dataset considers two different tasks, i.e. anomaly segmentation and road obstacle segmentation. The test splits for each of the tasks contain 100 and 327 images, respectively. Pixel precise ground truth annotations are provided for each of the test images. Images for training are not provided. An extensive evaluation of existing methods is performed using pixel-level and component-level metrics. Results indicate room for improvement for future methods when trained on the Cityscapes dataset.

strengths: 1. The paper considers a relevant problem for which existing datasets are not sufficiently large and diverse.
2. The paper is easy to follow and understand.
3. Coverage of related work is adequate. The benchmark contains a diverse set of existing methods and evaluation metrics.
weaknesses: 1. The number of provided images is small. For the anomaly segmentation subset, 100 images are provided. For obstacle detection, 327 images are given. I have concerns that the number of provided images is enough for a thorough method evaluation, especially since the diversity of possible anomalies is immensely large for this particular task.

2. The dataset does not include a training set. The paper also does not contain a clear recommendation on which other datasets should be used for model training. Since objects were defined as anomalous based on the class definitions of the Cityscapes dataset, it seems that using datasets that localize different classes would not be suitable since possible anomalies might occur in the training set of these datasets as well.

3. The task of obstacle detection is not sufficiently detailed. For example, which pixels are included in the evaluation? In Figure 1, there  is a person walking on the road. It is not labeled as an obstacle, nor is it included in the evaluation mask. Does this mean that if an algorithm wrongly detects the human as an obstacle, this is not reflected in the evaluation metric? If this is the case, I think this would bias the evaluation result.
correctness: 1. There is a potentially very large domain shift between the training and the test dataset. Test images contain scenes from very different places (e.g. the desert), the viewpoints of the camera sometimes changes drastically, etc. Therefore, an evaluation on this dataset does not only depend on the used model, but also significantly on the dataset used for training. For example, if the  training dataset does not cover a wide range of domains, an algorithm trained on this dataset could rightfully mark areas as anomalous that are labeled as anomaly-free. I therefore think it is very important to clearly state which training dataset(s) should be used.

2. To compute the number of false positive predictions, one first loops over the predicted components and computes the sIoU with the ground truth. If the sIoU < threshold, the component is marked as a false positive. Does this mean that if a small prediction is fully within a large GT component, this prediction will be marked as a FP since its sIoU with the GT is quite small? If so, I think the definition of FP regions needs to be adapted.
clarity: The paper is easy to follow and understand.
relation to prior work: Prior work is adequatly covered. Compared to other datasets, the proposed one does not rely on artificially generated anomalies. However, it does not contain significantly more images than existing datasets.
additional feedback: 1. For clarity, it would be useful to add a legend to the image in Appendix F5 that explains the color coded class labels.

2. In equation $(1)$, $A(k)$ denotes all GT pixels of the dataset that are not within the GT region $k$. I believe the denominator of the equation should be $|k \cup \hat{K}(k) - A(k)|$ instead of $|k \cup \hat{K}(k)| - |A(k)|$?

General comment: The paper considers an interesting and important problem for which more high quality research datasets are needed. The paper is well written and many recent methods were benchmarked on the proposed dataset. My main concerns are the low number of dataset images as well as the absence of a dedicated training split. In addition, there are several questions in my review above for which it would be great to have an answer to from the authors. In its current form, the weaknesses of the work outweigh its strengths which concludes my paper rating - marginally below acceptance threshold.
review_title: Review for the SegmentMeIfYouCan Dataset and Benchmark.

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: The paper introduces the "SegmentMeIfYouCan" benchmark that contains two datasets for anomaly and obstacle segmentation in road scenes. The first dataset, RoadAnomaly21, can be used to evaluate the capability of semantic segmentation models to segment anomalous objects, i.e., objects not seen during training. The second dataset, RoadObstacle21, is aimed at obstacle segmentation, where a model is supposed to segment all objects on the road, regardless of whether they have been seen before or not.

RoadAnomaly21 contains 100 images of road scenes with pixel-level annotation for three classes: All pixels that are part of one of the 19 CityScapes evaluation classes are labelled as "not anomalous". Pixels in objects that do not belong to a CityScapes class are labelled as "anomalous". Certain parts of the images that the authors do not consider to be main anomalies of interest in the context of street scenes, such as mountains in the image background and street lights, are labelled as "void", meaning they will be ignored in the evaluation.

RoadObstacle21 contains 327 images showing road scenes with a variety of obstacles on the road (e.g., stuffed toys, tree stumps). These objects appear at different distances in the images. In the images, the drivable area is labeled as "not obstacle". All objects that are enclosed by the drivable area are labelled as "obstacle". Pixels outside of the road are labelled as "void" and therefore ignored in the evaluation. RoadObstacle21 also contains 85 images with scenes at night and in extreme wheather. These images are not included in the experiments because of the large domain shift.

For both datasets, the labels are withheld by the authors, to avoid the possibility that methods are overfitted to these specific datasets. However, the authors provide small validation sets that include labels, so that users can check the compatibility of their method with the benchmark implementation.

In their benchmark, the authors evaluate a diverse array of methods on the datasets. Some of these use "auxiliary" out-of-distribution data, while others do not. The authors use pixel-level metrics (e.g., AuPRC) as well component-level metrics (a modified version of IoU) in their benchmark. The performance of the evaluated methods differs wildly, with methods based on generative models and the maximized softmax entropy method performing the best. Overall, the evaluation shows that there is significant room for improvement.
The authors also include evaluations of the methods on other datasets, such as Fishyscapes LostAndFound, to complement their benchmark.
strengths: The proposed benchmark addresses an important task, for which there are not a lot of datasets available. The images in the datasets are very diverse. Each image shows a different scene. (This is in contrast to some other datasets for anomaly segmentation in road scenes, which include images that are almost the same since they are frames taken from a video of a single scene). RoadAnomaly21 and RoadObstacle21 also have a large variety with respect to object types, sizes, and distances.

The differentiation between the pure anomaly segmentation task and the obstacle segmentation task appears to not have been done before.

The experiments and evaluation are very thorough.
weaknesses: The benchmark only contains images for evaluation, making it necessary to train a model on other datasets (usually CityScapes). This leads to a potential domain-shift that complicates the interpretation of results. If a model performs badly on the benchmark (in particular if it predicts a lot of false positives), it is unclear whether the bad performance is a flaw of the anomaly segmentation method or caused by the domain shift. __Edited to add:__ The authors clarified their definition of "anomaly" and emphasized that a certain amount of domain shift is intentional. They want a method to segment only pixels that belong to semantically new classes. Pixels with visually new features that belong to one of the training classes should not be segmented as anomalies. I think this is a valid justification for the present domain shift. On the other hand, the domain shift still makes it difficult to analyse the reason for false positives of a method: It would be unclear whether the false positives are caused by the domain shift or by issues of the method itself.
correctness: On the whole, the dataset construction and evaluation protocol are appropriate. I have two issues, however:

I find the use of the void class problematic. While I understand the motivation (e.g., mountains in the background are not of interest for the task in question), it uses information that would not be available in a real application. The voided areas may contain false positive or negative predictions of the model that could affect the application. I found a particularly illustrative example of my issues with the void class in the presentation (https://segmentmeifyoucan.github.io/segment-me-presentation/#/10) linked on the official benchmark website (https://segmentmeifyoucan.com/): There it is noted that wet spots on the road are ignored in the evaluation. However, the predictions of a model in these spots are very much of interest, as the model might falsely predict obstacles in these regions. Also from a more theoretical standpoint, when using the dataset to evaluate an anomaly segmentation method, a user would want to test whether a model correctly detects all kinds of anomalies, not just a somewhat arbitrary subset. __Edited to add__: The authors addressed this issue in their reply. They explained that they revised the annotations for RoadObstacle21 so that parts of the road that may contain visually new features, but are still drivable (such as wet spots), are marked as obstacle-free. From an application point of view, I still find the void class to be problematic (e.g., semantically ambiguous objects such as rickshaws cannot be ignored in real applications), but I acknowledge that the limitations of the CityScapes annotations and classes cause this problem.

At the beginning of Section 4, "Numerical Experiments", the authors write that for RoadAnomaly21, components smaller than 500 pixels are discarded from the predicted segmentation (50 pixels for RoadObstacle21). This choice is somewhat questionable. It penalizes methods that produce predictions that cover ground truth components well, but consist of a lot of small components. Conversely, it does not penalize methods that predict lots of small false positive regions (that might potentially cover a significant area of the image). __Edited to add:__ In their reply, the authors explain that the removal of small predicted connected components is not part of the evaluation protocol, but an optional choice made for the methods. I think it is legitimate to include such a postprocessing step in a method, so my original issue is sufficiently addressed. There is, however, still a problem, as the choice to remove connected components smaller than 500/50 pixels is based on knowledge of the size of ground truth connected components.
clarity: The paper is well-written and easy to follow.
relation to prior work: The differences of the proposed datasets to other anomaly segmentation datasets in the domain of road scenes are discussed in detail. However, there is no mention of anomaly segmentation datasets in other domains.

In the paper, the authors use the term anomaly segmentation in the context of models that have been trained for semantic segmentation (with a plurality of classes) that should also segment new anomalous classes. However, the term anomaly segmentation is also used for problems where models are trained on data that is not labelled as containing several classes. The "Related Work" section could benefit from a discussion that addresses this difference. This distinction is not just important for the datasets, but also for methods. While I assume that the authors designed their benchmark for the evaluation of methods that have been trained to segment different classes, this is not explicitly stated. If my assumption is correct, this should be made clear in the paper. Otherwise, the "Related Work" section (and the experiments) should also include recent methods that train only on "non-anomalous" data without class distinction. __Edited to add:__ The authors sufficiently addressed this point, as they clarified their definition of "anomaly" in the paper.

The discussion of anomaly segmentation methods that are trained for semantic segmentation is detailed. While ensemble methods are cited in the "Related Work" section, they were not included in the evaluation. This is a bit surprising, since ensemble methods are consistently among the best methods for anomaly detection in a classification setting. __Edited to add:__ I their reply, the authors addressed this point sufficiently. They explain that a problem with ensembles is the associated computational complexity, which is why they are not an established method for semantic segmentation. Additionally, the authors stated that they will perform additional experiments with ensembles so that they can be included in the paper.
additional feedback: In addition to the results on the anomaly segmentation benchmark, it might be a good idea to also report the results of the methods on CityScapes. That way, a reader would get a more complete picture.

I think there is a typo in equation (1). I think it should be $sIoU(k) := \frac{|k\cap\hat K(k)|}{|(k\cup \hat K(k))\backslash\mathcal{A}(k)|}$.

The "Checklist" section at the end of the paper begins with an instructions block. That block can be removed.
review_title: An interesting dataset for road scene anomaly segmentation, but there are some issues

Meta-review: Although reviewers agree on the high relevance of the problem, there are common critiques regarding missing justifications/details, limited data size, and issues related to the lack of a training set. After rebuttal several of the critiques of the reviewers have been carefully discussed and addressed by the authors. Overall the paper achieves the minimum score to be accepted for publication at the NeurIPS data track.
Decision: Accept