ErrorAug: Making Errors to Find Errors in Semantic Segmentation
Authors: Devin Guillory
Dequan Wang
Shun Ishizaka
Kazuki Kozuka
Trevor Darrell
Venue: Submitted to ICLR 2023 (2023)
Keywords: Semantic Segmentation
Uncertainty Quantification
Error Detection

Abstract: In order to develop trustworthy downstream applications for semantic segmentation models, it is important to not only understand the performance of a model on  datasets, but to localize areas where the model may produce errors.
Pixel-wise error prediction of semantic segmentation maps is a challenging problem in which prior work relies on complicated image resynthesis pipelines.
We introduce \it{error augmentation}, a framework which enables us to learn robust error detectors by applying data transformations independently on the predicted segmentation maps.
This approach enables direct prediction of pixel-wise error in semantic segmentation maps, an approach explored as a naive baseline in prior works, to achieve state of the art performance.  
As a proof-of-concept we propose a series of three simple transformations that generate challenging segmentation errors by swapping pixel predictions within a segmentation map.
Our approach outperforms previous methods of error detection for semantic segmentation across all metrics and improves performance by over $7.8\%$ on AUPR-Error.
Additionally, we show that our approach not only generalizes to unseen test examples, but remains reliable despite significant shifts in the target domain.


Under review as a conference paper at ICLR 2023
ERRORAUG: MAKING ERRORS TO FIND ERRORS IN SEMANTIC SEGMENTATION
Anonymous authors Paper under double-blind review
Image
Prediction
Error Detection
SynthCP
ErrorAug
Figure 1:
We propose ErrorAug simple and reliable approach for pixel-wise detection. ErrorAug allows us to artificially generate more examples of errors which are also of higher de- gree of difficulty, ErrorAug improves relative performance of key error detection metries by over 18%/1..2% for in on-domain/outof--dee oomann scenarios versus previous state-of-the art approach Syn- thCP.
ABSTRACT
In order develop trustworthy downstream applications for semantic segmenta- tion models, it is important to not only understand the performance of a model on datasets, but to localize areas where the model may produce errors. Pixel-wise er- ror prediction of semantic segmentation maps is challenging problem in which prior work relies on complicated image resynthesis pipelines. We introduce er- for augmentation, framework which enables us to learn robust error detectors by applying data transformations independently on the predicted segmentation maps. This approach enables direct prediction of pixel-wise error in semantic segmentation maps, an approach explored as naive baseline in prior works, to propose series
achieve state of the art performance. As proof-of-concep of three simple transformations that generate challenging segmentation errors by swapping pixel predictions within segmentation map. Our approuch oulper- forms previous methods of error detection for semantic segmentation across all metrics and improves performance by over 7.8% on AUPR-Error Additionally. we show that our approach not only generalizes to unseen test examples, but re- mains reliable despite significant shifts in the target domain.
INTRODUCTION
Understanding when machine learning models are producing inaccurate predictions is essential for improving the reliability of systems that build upon these models. Recent works in performance prediction have made strides in predicting the performance of classification systems in novel en- vironments Garg et al. (2022); Chen al. (2021); Guillory al. (2021). However for complex computer vision tasks like semantic segmentation, its important to not only identify when model produces and inaccurate prediction but also where the models predictions have failed. For instance, in robotics setting misclassification far away from the action space of the robot may be less rel- evant than one in the immediate pathway. As such the task of pixel-wise error detection becomes strive to produce Al systems that can safely interact with ever-changing real world environments.
increasingly important
We propose Error Augmentation (ErrorAug), process for synthesizing challenging localization errors by applying data transformations on predicted class probabilities independent of any transfor- mations on the input images, step for training high-quality pixel-wise error detectors, In order to demonstrate the effectiveness of this process, we propose three swapping operations that when applied to segmentation map allow treat enror detection as supervised learning task and

---

Under review as a conference paper at ICLR 2023
SemSeg Model
Error Detection Network
é F 0)
Prediction
Image
ErrorAug
Error
ShiftSwap
ClassSwap
MapSwar
F
LBCE(E, é)
Figure 2: Error augmentation allows use directly train an error detection network with supervised learning- Applying ErrorAug over the model's prediction allows to create challenging and diverse set of example errors. Training according to this pipeline leads error detection model which performs reliably on novel examples.
Image
Prediction
Label
SynthCP
ErrorAug
Error
Figure 3: The top illustrates semantic segmentation task where segmentation map is pre- dicted from an image. The difference between the top row prediction and the top row label is the map, depicted in the bottom right. The goal of our approach, ErrorAug, is accurately predict the error map on the bottom right, based on the image and the prediction. We see that despite the model's simplicity, ErrorAug does much better job at locating large misclassified regions, like the moving truck in this example, than the previous state-of-the-as approach SynthCP.
directly train a model to predict pixel-wise error maps for semantic segmentation systems. Most prior works in this space have attempted to directly train error detectors but due to their tendency to overfit in this problem setting, they are most frequently presented as naive baselines to motivate more complicated approaches. In this work show that by including ErrorAng process into previous pipelines we bypass the complications of previous approaches and can produce state-of- the-art results using prior codebases direct prediction implementations. Our training procedure is illustrated in figure 2.
The most prominent prior work this space built upon the success of conditional generative models by using the predicted segmentation map to condition model whose goal it is the recreate the input image Xia al. (2020)Di Biase et al. (2021). The discrepancies between this reconstruction and the original input image are then used to detect errors. These approaches are not only expensive, but are overly- overly-sensitive to the training dataset as the behavior of these generative models on changing domains remains poorly understood. Our approach does not require complicated auxiliary tasks or networks. We dramatically improve upon state-of-the-ar approach, SynthCP, by simplifying the modeling pipeline into single neural network, optimizing with standard binary cross entropy loss, and extending the set of errors observed in training. Figure and illustrate the apple-to-apple comparison between SynthCP (SOTA) and ErrorAug (ours).
Error Augmentation, in the context of eтror detection, can be used to describe any data transforma- tion applied independently to the label predictions in the training pipeline of error detector that allows the derivation of novel target error maps. In this work, we explore three swapping data trans- formations (ShiftSwap, ClassSwap, and MapSwap) which generate different types of challenging
2

---

Under review as a conference paper at ICLR 2023
errors. Shifttwapp involves spatial shifting of predictions in segmentation map and thus induces errors over areas where class boundaries exist. ClassSwap involves a swapping of the prediction probabilities between classes in image, inducing misclassification errors. The last swapping op- eration is meant to induce errors of all types, as it involves swapping the predicted segmentation map of one image with that of another image, MapSwap. Figure 4 illustrates all these operations over the segmentation maps of single image. While some of the changes are drastic and immediately noticeable, others are more subtle but still contribute novel examples for training.
Our work shows that each of these shifts alone is powerful enough to learn a direct prediction model which outperforms prior art. When all shifts are introduced to the same training pipeline we show that we are able to train model that improves on prior art by a margin of >> 7% on error detection on the Cityscapes Cordts et al. (2016) benchmark. More importantly, we demonstrate that this approach is considerably more robust when dealing with instances of domain shift, as demonstrated by outperforming image re-synthesis approaches by > 13% when models trained over the European benchmark of Cityscapes are evaluated over the largely North American dataset, BDDI00K Yu et al. (2020).
2 RELATED WORK
We focus the task of pixel-wise error detection for semantic segmentation which can be de- scribed as identifying erroneously label pixels over objects with known classes. Anomaly detection for semantic segmentation is closely related task that instead focuses identifying pixel of ob- jects with unknown classes Di Biase et al. (2021). Some approaches in prior literature attempt propose solutions to both of these tasks, we instead focus on the pixel-wise error detection tasks and also extend our analysis to error-detection performance over domain shifts. A popular baseline in this space is MSP-CRF Hendrycks al. (2019a), which extends the maximum softmax probability method Hendrycks & Gimpel (2016) for outlier detection. In order to better account for the spa- tial element of pixel-wise error detection, they chose include a conditional random field Lafferty et al. (2001) component which can better account for locality. Other predictive uncertainty- approaches such as MC-Dropout Gal Ghahramani (2016), deep ensembles Lakshminarayanan et al. (2017), and Bayesian SegNet Kendall et al. (2015) have been explored for pixel-wise error detection, but fail to consistently detect misclassification.
More recent works in this space have used conditional generative models to attempt to resynthe- size the original image based on the predicted segmentation map Di Biase et al. (2021); Lis et al. (2019); Xia et al. (2020). A separate module is then developed to determine where the resynthesized image and the original image substantially differ and these regions are estimated as likely contain- ing erroneous predictions. These approaches rely on complicated auxiliary tasks, such as image reconstruction, in order to prevent their error detection model from becoming overly reliant of the predicted probabilities generated by the segmentation model. The approach introduce a substantial amount of computational overhead, and produce methods which do not generalize well to novel domains.
By introducing еттоr augmentation as framework for learning to directly predict error maps for semantic segmentation, we forgo the issues of predictive uncertainty methods which are too reliant on model confidences and do not properly account of spatial structure. As our error augmenta- tion approach is not dependent on a generative model, we are also able to avoid the computational complexity and sensitivity to domain shift observed in image resynthesis approaches.
Our additional focus on evaluating under domain shift stems from several lines of recent works showing that existing model architectures struggle to maintain consistent performance across data shifts where existing model architectures struggle to maintain consistent performance Recht et al. (2019); Koh et al. (2021); Madras & Zemel 1) Peng al. (2018); Bashkirova et al. (2021). Sev- cral works have demonstrated the ability to predict model performance in these new environments Deng & Zheng (2021); Guillory et al. (2021); Rabanser et al. (2019); Garg al. (2022); Chen et al. (2021), however these approaches do not readily translate to the complex task of pixel-wise error detection.
Outside of error detection, the machine learning community has experienced an explosion in re- search concerned with various forms of data augmentation Shorten & Khoshgoftaar (2019) of which
3

---

Under review as a conference paper at ICLR 2023
error augmentation be viewed special case. Data augmentations have been central to the development of statc-of-the-arr methods in self-supervised learning Doersch et al. (2015); Misra & Maaten (2020), semi-supervised learning Sohn et al. (2020), domain adaptation Sun et al. (2020), and uncertainty quantification Hendrycks et al. (2019b). Data augmentation strategies have pri- marily been deployed in one of two ways in prior work. First is as method of insuring model consistency with respect to certain data transformations. Augmentation strategies which aim for model consistency must take care ensure that the data transformations they expose the model to do not alter the underlying task. Other approaches use data augmentations to learn auxiliary tasks. Learning over well motivated auxiliary tasks can lead to improved downstream model performance. The data transformations in error augmentation are only applied to one of the inputs and their ap- plication changes the target labels in a non-symmetric manner. This produces an approach which model consistency is not the goal of the augmentations since they lead to changing set of labels, and the auxiliary task produced by the labels happens to be the same task as the primary task.
ERROR AUGMENTATION
In this section we introduce Error Augmentation, ErrorAug, as a framework for learning more re- liable error detectors for semantic segmentation. ErrorAug contributes two new components the training pipeline for direct error prediction, synthetic segmentation maps and automatically derived error maps.
We first discuss the formulation and motivation of the proposed error detection framework, Erro- rAug. Next, describe three types of error augmentations (ShiftSwap, ClassSwap, MapSwap), all of which are quick and easy to implement, Lastly, wediscuss the neural network architectures used represent the error detectors that we train and how they differ from prior approaches.
3.1 ERROR SYNTHESIS
Our semantic segmentation task consist of L distinct semantic labels. An image x of size w h could be labeled with a semantic segmentation map y E (0, 13 model P, works by operating over image to produce estimate of the semantic segmentation map P(x) = ÿ E {0. Lxwxh, We define an error map e E {0, 1 1| such that
semantic segmentation
ei = max yt,..j * 01,5.
(1)
OSISL
The goal of error detection algorithms is learn model F such that the error map can be esti- mated with F from the image x and predicted segmentation map g. thus = Fzz, f), as described in algorithm 1. Error detection can be formulated as supervised learning problem where the target labels are error maps e and the input instances are images x and predicted segmentation maps ý- Letting C represent loss function, such as binary cross entropy, our error detector F can be learned as minimization of the following equation:
min EBcc(F(2)) e) F
(2)
Following standard procedure in supervised learning, datasets are split into a training subset and test subset with the test subset being used to assess the models performance on previously unseen examples.
Due to high levels of correlation between the segmentation map 9 and the error map e, it is possible to learn direct error detectors F which perform well on the training set, with little comprehension about the image or the label space TL. Prior work uses the auxiliary task of image synthesis to encourage the eπтог detection model to learn non-trival information about the images and/or the segmentation task. Instead of synthesizing images, our framework aims to ensure the error detection model is based examples with less correlation between the segmentation map ÿ and the error map c.
sufficient understanding of the segmentation task by synthesizing challenging
We define Error Augmentation the process of applying data transformation T to the predicted segmentation map ÿ such that generates a new predicted segmentation map g' = =TØ)). Unlike traditional data augmentation strategies, the transformation 7' is intentionally not applied to the input
4

---

Under review as a conference paper at ICLR 2023
Prediction
ShiftSwap
ClassSwap
MapSwap
Label
Figure 4: Error augmentation works by applying data transforms directly the prediction of seg- mentation model. Illustrated above, ShiftSwap, ClassSwap. and MapSwap are the data transforms propose for error augmentation. By comparing the prediction and transformed predictions with the target label map, on the far right, we can see how each transform contribute unique errors to the training set of our error detector.
image æ. As such, not enough for model to be equivariant to transformation T, and the error detection model F must attempt recover the relationship between j' and the input image From the transformed segmentation map j' and the known segmentation map y, a new error map can be derived:
= max yLi + Di.5.d OSiSL
The relationship between T and the new map e' is dependent upon the ground truth label map and can be complex even for simple data transforms. In traditional data augmentation settings, care must be taken when choosing the set of data transforms the relationship between the transformed image x' + Txx and observed label y. For instance, synthetically changing the colors of an observed snake, may lead meaningful change in classi- fication of the observed animal. However, since our input image remains unaltered, as long as the transformed map j/ is valid segmentation map, the derived error map e' is well-defined. We will take advantage of this in later sections by proposing transforms that would be nonsensical if applied to the input image. This formulation allows us to rewrite the minimization objective as operating over ÿ' and e' and allows to arbitrarily scale the number and complexity of training examples by adjust our transformation function T'.
certain transformations may alter
min (FCE(F(x,))),e)
()
The quality of the learned error detector, will largely dependent upon the nature of the trans- formation function 7 and the architecture of the model. In this work we explored three relatively straightforward data transformations and observe they all have a strong positive impact, motivating future work exploration of error augmentation strategies and their relationship with robust machine learning.
3.2 SWAPPING TRANSFORMATIONN FOR ERROR DETECTION
In order better understand the utility of our proposed framework for Error Augmentation, im- plemented three swapping data transformations (ShiftSwap. ClassSwap, MapSwap) which increase the expected delta 8 between the predicted segmentation map Ò and the transformed map j/
1 - r'll
(5)
Our proposed transformations where to designed to generate specific forms of errors, such as mis- classifications and errors object boundaries. In figure ??, we visualize these these three transfor- mations over the same image, illustrating how the prediction maps change and the subsequent new error maps that result from the transformations. For our learned model, Error Aug, we stochasti- cally compose all three transformations. In ablation studies, we investigate the contribution of each one independently.
ShiftSwap is a positional swapping operation which selects spatial window of the predicted label map ý and shifts this window by certain horizontal 6, and vertical 6, directions, overwriting all previous predictions in this new location. For all following swapping operations, we assume that &' is initialized with the values from ÿ. Given a, b and c, as the horizontal and vertical boundaries for the spatial window and assuming that all indices valid, this operation could be represented as follows:
H b
5

---

Under review as a conference paper at ICLR 2023
Algorithm ErrorAug: Making Errors to Find Errors
Given dataset D, semantic segmentation model P, error detection network F for input batch (x y D do
Operate over images w and labels y D ShiftSwap, ClassSwap, MapSwap b Derive error map > Predict error map
y -- EurorAug( P (x)
B -- 1
@ < F(r"
Optimize network F by back-propagating LBCE(E ee8) end for
This approach results in shifted prediction boundaries and encourages the error detection module to learn the correct boundary positions.
semantic swapping operation which arbitrarily chooses two semantic classes in a label map and swaps their predictions throughout at each pixel location in the label map. Given classes p and q as the intended swapping pair, the ClassSwap operation could be represented as follows:
ClassSwap
8p. < Danci
Vy... Up...
Since this approach respects the spatial predictions of the model P it may result in regions with correct segmentation boundaries but incorrect semantic classes.
MapSwap is the last and most drastic transformation propose. It is designed to swap the predicted pixel-level label maps between images. Letting (+)) and x(1) represent two distinct images and g()) and g(و) represent the predicted segmentation maps of these images, respectively. Then MapSwap operation could be represented as follows:
f/10) + 5())
(9)
g + ((0
(10)
This results in segmentation maps which are derived from the input image and thus encode no information about the segmentation models properties over this particular input image P(x).
While this approach has severed the causal relationship between the input image x and the synthetic segmentation map G', it is still likely to produce informative error map e', which is neither all ones or all zeros due to structural similarities between scenes. These structural similarities such as roads being likely to occur near the bottom image and the sky being likely be near the top of an image are illustrated in Figure 4 or through more examples the supplementary materials. This example of data transformation which would be degenerative if applied over the input image & but yields challenge and useful information when employed for error detection.
4 EXPERIMENTS
4.1 DATASET
We intensively evaluate method on semantic segmentation benchmarks for autonomous driving, Cityscapes Cordts et al. (2016) and BDDIOOK Yu et al. (2020).
Cityscapes With 2975 nigh-resolution training images and 500 validation, the Cityscapes dataset, is a large-scale semantic segmentation datasets which prior works have used to validate the quality of error detectors. The dataset consist of images captured from vehicles in various European cities for the task of improving visual systems for self-driving. The pre-defined train IDs contains 19 semantic classes, and also contains out-of-context labeling upon which errors and accuracy are not counted.
BDD100K We look at the semantic segmentation setting for the BDD100K dataset, which consist of 7000 training images, and 1000 validation images for semantic segmentation. The dataset consist

---

Under review as a conference paper at ICLR 2023
Image
Label
Prediction
Enorr
SyntCCP
Error Aug tours)
CitySeapes
BDD JOOKS
Figure 5: MSP, SynthCP, and our ErrorAug are representative algorithms for approaches based on predictive uncertainity, image resynthesis, and direct prediction. We visualize the predicted error maps and the actual error maps for each of these algorithms across multiple images from both the in- distribution Cityscapes dataset and BDD100k which represented shifted target domain. ErrorAug is able to outperform SynthCP and MSP across both domains and the results above illustrate how this improvement is noticeable on the more challenging setting of error detection over domain shift.
of images captured from vehicles throughout cities, highways, and rural landscape in North Amer- ica. In addition to supporting the same 19 semantic classes as Cityscapes, it also contains various attribute annotations for the images, such as weather conditions and environmental settings. As we are interested how our models perform when exposed to novel environmental settings we evaluate the quality of our error detectors trained Cityscapes in the BDD100K dataset.
4.2 SETUP
Implementation We title error detection model ErrorAug because it is trained the proposed set of swapping error augmentations. To build our training dataset for ErrorAug follow the formulation used in Xia et al. (2020), where they use four-fold cross validation in order to address the smaller size of the validation dataset and ensure there exists an held-out set not seen by either the segmentation model or the error detection model. The held-out set for each fourfold cross validation step is annotated with the prediction probabilities for the segmentation model trained this step. The final error detector can then be validated over the held-out test set and an error detector and segmentation model both trained over the entirety of the training set. The ErrorAug transformations used our model are the composed set our three swapping operators (ShiftSwap, ClassSwap, MapSwap), each implement with p chance of being activated.
Architecture Since our approach presents a more challenging error detection problem than those explored in previous works, we require an eиror detection architecture with sufficient capacity. Our architecture consist of two 18 He et al. (2016) branches, one for the image input and one for the predicted label map ÿ. The image ResNet is initialized by prreeinet-retrraieed weights, while 19-channel segmentation map ResNet is trained from scratch with Kaiming Initialization He et al. (2015). Features are extracted from the siamese neural networks, concatenated with each other and then upsampled to allow for direct pixel-wise prediction of errors. Prior direct predic- tion approaches concatenated the segmentation map and image prior to passing it through a neural network, observe that this approach is unable to adequate capture the complex relation between the segmentation map and input image required to learn an adequate detector under the ErrorAug framework, shown in the ablation experiments in table 3.

---

Under review as a conference paper at ICLR 2023
Table I: Experiments on Cityscape Dataset. We detect pixel-wise failures on the results of FCN-8 and beeplab-v2. We report our results of ErrorAug and several baselines error detectors. ErrorAug outperforms all prior approaches across all observed metrics, indicating that when properly trained direct prediction strategies are promising underexplored approach for error detection. ErrorAug offers > 7.5% relative improvement on AUPR-Error over the more computationally intensive prior state-of-the-art SynthCP for both models explored.
FCN-8
Decplab-v2
Method
Arr
Ap-Suc
AUC, 1
FPR95 J
m
Ap-Err
Ap-Suc
AUC
FPR95.
mlnU/ f
MSP
50.31
99.02
91.54
25.34
48.36
99.12
91.68
27.27
MCDropout
49.23
99.02
91.47
25.16
47.85
99.23
92,19
24.68
TCP
48.54
98.82
90.29
32.21
51.15
45.57
98.84
89.14
36.98
58.27
SynhCP
55,53
99.18
92.92
22.47
49.99
99.34
92.98
21.69
ErrorAug (ours)
59.84
99.31
93.94
20.23
53.84
99.40
93.77
20.38
Table 2: Experiments BDD Dataset which models and Error Detectors trained on Cityscapes. We detect pixel-wise failures on the results FCN-8. ErrorAug outperforms all prior approaches across all observed metrics and offers 13.6% relative improvement on AUPR- Error over the more computationally intensive prior state-of-the-art SynthCP. An interesting observation is that the much less sophisticated MSP approach outperforms SynthCP primary metric AUPR-Error BDD Dataset.
Method
Ap-Errt
Ap-Suc t
AUC 1
FPR95 4
mloU
MSP
59.05
94.79
84.52
42.07
SynthCP
58.26
95.34
85.34
39.69
25.82
ErrorAug (ours)
66.21
96.49
88.98
34.58
Baselines To replicate prior work, we choose FCN-8s Shelhamer et al. (2016) and Deeplab- V2 Chen al. (2017) our default semantic segmentation models to predict the error map. The primarily goal of our work is to improve the capability of performing pixel-level error map predic- tion for the given segmentation model. We compare proposed ErrorAug with MSP Hendryeks & Gimpel (2016), MCDropout Gal & Ghahramani (2016), TCP Corbière al. (2019), and Syn- thCP Xia et al. (2020). Following the experimental setup of SynthCP, we also implement a fairly comparable baseline, leveraging regression head to directly predict the pixel-level failures given both input image and the corresponding predicted label map. All these baselines share the same backbone and training strategies. We further ablate the sensitivity of some hyper-parameten in the proposed ErrorAug system.
Metrics In order to evaluate the effectiveness of our approaches, AUPR-Error (denoted as Ap-Err, higher better), AUPR Success (denoted as Ap-Suc, higher is better), AUROC (denoted AUC, higher better), and FPR 95% TPR (denoted FPR95, lower better) as Liu et al. (2019) and Xia et al. (2020). The main metric we concern ourselves with in this work is AUPR-Error, which computes the area under the precision-recall curve by treating errors as the positive class.
the following four metries:
4.3 COMPARISON WITH STATE-OF-THE-ART METHODS
Cityscapes In table we observe that Error substantially outperforms prior methods over all observed metrics. Improving relative performance the main metric of AUPR-ErrON by 7.5%, we show that this model can outperform models with considerably more parameters and a more com- plex evaluation cycle. In figure 5, we visualize some of the results and observe that method does
8

---

Under review as a conference paper at ICLR 2023
Table 3: Ablation Experiments on Cityscape Dataset. We evaluate each proposed swapping oper- ation in our approach as well as variant which uses the proposed architecture to directly predict errors with no swapping. We highlight results which outperform previous state-of-the-art methods and show that each variant of our approach independently outperforms prior methods, with Map- Swap providing the largest improvement. Without error augmentation our proposed architecture still out performs other direct prediction methods such that found the Old Arch from Xia et al. (2020). When exposed to the full set of error augmentations, the performance of the prior architecture suf- fers due to its limited capacity.
Method
Ap-Err f
Ap-Suc 1
AUC f
FPR95
Old Arch NoAug
52.17
99.15
92.55
22.34
EmorAAug-Ol Arch
54.31
99.18
92.85
21.91
EmorAAg-NoAAg
57.63
99.14
92.75
23.72
ErrorAug-MisClass
58.50
99.21
93.30
21.69
ErrorAug-Shift
58.83
99.21
93.25
21.78
ErnorAug-Image
59.39
99.30
93,80
20.86
ErrorAug
59.84
99.31
93.94
20.23
a better job at identifying region-level misclassifications that other approaches. Furthermore, we note that this ability becomes important as contexts shift such in the BDDI00K examples.
BDD100K In table 2, we compare the performance of error detectors on data obtained from new domain (train on Cityscapes, test on BDD100K). We observe an even larger relative improvement of 13.6% in our primary metric AUPR-En Error. Furthermore, we observed that in continues to outperform all other approaches in four of the metrics. An interesting observation, is that parameter-fie MSP approach outperforms the expensive SynthCP approach in this new domain over our primary metric of AUPR-Error. In the supplemental material we visualize the results these approaches.
4.4 ABLATION STUDY OF KEY COMPONENTS
In order understand the impact of different elements of our approach we ablation experiments as reported in table 3. We note that an error detector trained on any of the 3 swapping operations proposed in this paper would outperform all prior work across all reported metries. Additional we observe, that while our new стгог detection architecture would outperform prior approaches over AUPR-Error without using error augmentation. However, it would not improve across all metrics and it's performance is significantly increased when paired with our challenging error augmentation settings. Lastly, we see that when attempting to train direct prediction model which concatenates the segmentation map with the image before input, the model performs worse that if trained without error augmentation. This result indicates the importance of high-capacity models to solve this more challenging direct prediction task.
CONCLUSIONS
We develop framework for learning high-quality and robust error detectors for semantic segmen- tation. Our ErrorAug method performs independent data transformations over predicted semantic segmentation maps and is able to generate arbitrarily many novel and challenging error detection tasks. Using this to directly learn an eтror detection model leads to state-of-the -fte--of-the-rt error detection results for each swapping transformation we explored. ErrorAug produces substantially better error detection models with significantly simpler approach than prior methods. Each augmentation op- eration we explored has consistently improved results, yet we have only investigated the three swap operations reported above. We believe there is ample opportunity for future exploration of further improved data transformations and error detection architectures.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Dina Bashkirova, Dan Hendrycks, Donghyun Kim, Samarth Mishra, Kate Saenko, Kuniaki Saito, Piotr Teterwak, and Ben Usman. Visda-2021 competition universal domain adaptation to improve performance out-of-distribution data. arXiv preprint arXiv:2107.10011 2021.
Liang-Chieh Chen, George Papandreou, lasonas Kokkinos, Kevin Murphy, and Alan Yuille. Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs. IEEE transactions on pattern analysis and machine intelligence, 40(4): 834 848, 2017.
Mayee Chen, Karan Goel, Nimit S Sohoni, Fait Poms, Kayvon Fatahalian, and Christopher Ré. Mandoline: Model evaluation under distribution shift. In International Conference Machine Learning, pp. 1617-1629. PMLR, 2021.
Charles Corbière, Nicolas Thome, Avner Bar-Hen, Matthieu Cord, and Patrick Pérez. Addressing failure prediction by learning model confidence. Advances in Neural Information Processing Systems, 32, 2019.
Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understanding. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3213-3223, 2016.
Weijian Deng and Liang Zheng. Are labels always necessary for classifier accuracy evaluation? In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp- 15069-15078, 2021.
Giancarlo Di Biase, Hermann Blum, Roland Siegwart, and Cesar Cadena. Pixel-wise anomaly detection in complex driving scenes. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 16918-16927, 2021.
Carl Doersch, Abhinav Gupta, and Alexci A Efros. Unsupervised visual representation learning by context prediction. In Proceedings of the IEEE international conference on computer vision, pp. 1432-1430, 2015.
Yarin Gal and Zoubin Ghahramani. Dropout as bayesian approximation: Representing model uncertainty in deep learning. In international conference on machine learning, pp. 1050~1059.
PMLR, 2016.
Saurabh Garg, Sivaraman Balakrishnan, Zachary C Lipton, Behnam Neyshabur, and Hanie Sedghi. Leveraging unlabeled data to predict distrib-iitiittionnn performance. arXiv preprint arXiv:2201.04254 2022.
Devin Guillory, Vaishaal Shankar, Sayna Ebrahimi, Trevor Darrell, and Ludwig Schmidt. Predicting with confidence unseen distributions. In Proceedings of the IEEE/CVF International Confer- ence on Computer Vision, pp. 1134-1144, 2021.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In Proceedings of the IEEE international conference on computer vision, pp. 1026-1034 2015. 2015.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference computer vision and pattern recognition, pp- 770-778, 2016.
Dan Hendrycks and Kevin Gimpel. A baseline for detecting misclassified and examples in neural networks. arXiv preprint 160.0:1610.02116, 2016.
Dan Hendrycks, Steven Basart, Mantas Mazeika, Andy Zou, Joe Kwon, Mohammadreza Mostajabi, Jacob Steinhardt, and Dawn Song. Scaling -ustii-distt detection for worl-world settings. arXiv preprint arXiv: arXiv: 19t1./91111.111 2019a.
10

---

Under review as a conference paper at ICLR 2023
Dan Hendrycks, Norman Mu, Ekin D Cubuk, Barret Zoph, Justin Gilmer, and Balaji Lakshmi- narayanan. Augmix: A simple data processing method to improve robustness and uncertainty. arXiv preprint arX1v:191..02781, 2019b.
Alex Kendall, Vijay Badrinarayanan, and Roberto Cipolla. Bayesian segnet: Model uncertainty in deep convolutional encoder-decoder architectures for scene understanding. arXiv preprint arXiv: 1.0.000000 2015.
Pang Wei Koh, Shiori Sagawa, Henrik Marklund, Sang Michael Xie, Marvin Zhang, Akshay Bal- subramani, Weihua Hu, Michihiro Yasunaga, Richard Lanas Phillips, Irena Gao, et al. Wilds: A benchmark of in-the-wild distribution shifts. In International Conference on Machine Learning, pp. 5637-5664. PMLR, 2021.
John D. Lafferty, Andrew McCallum, and Fernando C. N. Pereira. Conditional random fields: Probabilisticc models for segmenting and labeling sequence data. In Proceedings of the Eighteenth International Conference on Machine Learning, ICML '01, pp. 282-289, San Francisco, CA, USA, 2001. Morgan Kaufmann Publishers Inc. ISBN 1558607781.
Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles, Advances in neural information processing systems, 30, 2017.
Krzysztof Lis, Krishna Nakka, Pascal Fua, and Mathieu Salzmann. Detecting the unexpected via im- age resynthesis. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 2152-2161, 2019.
Fengze Liu, Yingda Xia, Dong Yang, Alan L Yuille, and Daguang Xu. An alarm system for seg- mentation algorithm based on shape model. In Proceedings of the IEEE/CVF International Con- ference on Computer Vision, pp. 10652-10661, 2019.
David Madras and Richard Zemel. Identifying and benchmarking natural out-of-context prediction problems. Advances in Neural Information Processing Systems, 34, 2021.
Ishan Misra and Laurens van der Maaten. Self-supervised learning of pretext-invariant representa- tions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6707-6717, 2020.
Xingchao Peng, Ben Usman, Neela Kaushik, Dequan Wang. Judy Hoffman, and Kate Saenko. Visda: A synthetic-to-ren benchmark for visual domain adaptation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 2021-2026, 2018.
Stephan Rabanser, Stephan Günnemann, and Zachary Lipton. Failing loudly: An empirical study of methods for detecting dataset shift. Advances in Neural Information Processing Systems, 32, 2019.
Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do imagenet classifiers generalize to imagenet? In International Conference on Machine Learning, pp. 5389-5400.
PMLR, 2019.
Evan Shelhamer, Jonathan Long, and Trevor Darrell. Fully convolutional networks for semantic segmentation. IEEE transactions on pattern analysis and machine intelligence, 39(4):440-551, 2016.
Connor Shorten and Taghi M Khoshgoftaar. A survey on image data augmentation for deep learning. Journal of big data, 6(1):1-48, 2019.
Kihyuk Sohn, David Berthelot, Nicholas Carlini, Zizhao Zhang, Han Zhang, Colin A Raffel, Ekin Dogus Cubuk, Alexey Kurakin, and Chun-Liang Li. Fixmatch: Simplifying semi-supervise learning with consistency and confidence. Advances in Neural Information Processing Systems, 33:596-608. 2020.
Yu Sun, Xiaolong Wang, Zhuang Liu, John Miller, Alexei Efros, and Moritz Hardt. Test-time train- ing with sulfvvuuprvi for generalization under distribution shifts. In International Conference Machine Learning, pp. 9229 9248. PMLR, 2020.

---

Under review as a conference paper at ICLR 2023
Yingda Xia, Yi Zhang, Fengze Liu, Wei Shen, and Alan L Yuille. Synthesize then compare: De- tecting failures and anomalies for semantic segmentation. In European Conference on Computer Vision, pp. 145-161. Springer, 2020.
Fisher Yu, Haofeng Chen, Xin Wang, Wenqi Xian, Yingying Chen, Fangchen Liu, Vashisht Madha- van, and Trevor Darrell. Bdd100k: A diverse driving dataset for heterogeneous multitask learn- ing. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp- 2636-2645, 2020.
12


Reviews:
[{"review_id": "whaRuWpCmGs", "reviewer": "Reviewer_G1qP", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "ErrorAug proposes to use data augmentation for direct prediction of errors in semantic segmentation models. By augmenting errors in a model's predictions by swapping classes, shifting predicted mask or swapping segmentation from a different image, ErrorAug is able to train a more robust direct error prediction model. This is shown by showing better transfer of error prediction from Cityscapes to BDD100K, which has much more diverse imagery along with a domain shift from Europe to North America. The paper proposes to use a siamese architecture for direct error prediction instead of concatenating image and label in the channel dimension, which already beats synthesize and compare (SynthCP, ECCV 2020) on area under the precision-recall curve (Ap-Err, error prediction is treated as binary classification) by ~2 points for an FCN-8s model. The proposed data augmentation strategies additionally improve Ap-Err by ~2 points on Cityscapes semantic segmentation using FCN-8s. The paper also shows results using Deeplab-v2 as the base segmentation model. ", "summary_of_the_review": "Overall, ErrorAug shows that data augmentation can make direct error prediction from a model achieve better results than the state of the art method SynthCP, which synthesizes images conditioned on a predicted segmentation map and uses a learned comparison module to predict pixel-wise errors (and detect anomalies / predict IoU, which ErrorAug does not focus on) in a segmentation prediction. This should serve as an improved baseline and help the sub-community move forward. \n\nHowever, ErrorAug is evaluated with at this point quite outdated semantic segmentation models and it is unclear whether state of the art segmentation model errors could be predicted well with these models. Moreover, due to the progress in conditional image synthesis, it is entirely possible that SynthCP, when implemented with newer image synthesis techniques would improve its performance. \n\nDue to this, even though I believe the contribution and idea are valuable, I cannot recommend acceptance of the paper in its current form since the findings from the paper might not hold up when using state of the art methods from 2022. ", "strength_and_weaknesses": "# Strengths\n\n- The simplicity of the proposed method over SynthCP that requires image synthesis in the loop is a big strength\n- The proposed error augmentations are sound and well motivated. They cover class prediction errors and segmentation errors. The proposed map-swap augmentation smartly takes advantage of the fact that autonomous driving scenes often have correlated scene structure\n- While the paper does not suggest this, error augmentation cou\n\n# Weaknesses\n\n- Evaluation follows SynthCP (from 2020) and only uses FCN-8s and Deeplab-v2 as the base segmentation models. At this point in time, these are far from state of the art models, which make much more fine-grained errors in Cityscapes semantic segmentation. It'd be important for the community to understand whether ErrorAug can be used to recover these finer errors in state of the art models such as ViT-Adapter\n\n- The proposed siamese architecture for direct prediction uses double the parameters from the \"OldArch\" baseline (the naming convention could be much better here). It is therefore unclear if it's the siamese architecture or the increased parameter count that improved error prediction performance. A baseline for OldArch with comparable parameters would be ideal\n\n- How does MCDropout and TCP do on BDD transfer? The transfer results do not show these\n\n- The paper compares with SynthCP as is, which uses SPADE (CVPR 2019) as its conditional image synthesis backbone. In the 3 years since, there have been multiple improvements in image synthesis and it is entirely possible that a better conditional image synthesis backbone improves SynthCP's performance considerably", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "Rm-Poa4AVGn", "reviewer": "Reviewer_2JRb", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes an error augmentation neural network to identify the errors made by semantic segmentation models by manipulating the predicted labels fed to the error detection network. \n\nThree transforms are tested - shifting the prediction, swapping a class or completely changing the prediction labels. Along with the RGB image, one of these augmented prediction labels is selected at random and fed to dual-branch error detection NN that yields the final error map.\n\nThe authors claim state-of-the-art performance, with +~7.7 % AUPR error (49.99 vs 53.84) on Cityscapes in-distribution results. Furthermore, even out-of-distribution tests (BDD100K) yield +13.6 % AUPR improvement.", "summary_of_the_review": "Although the paper yields good results on error detection from semantic segmentation, a (mostly) better performing / arguably SOTA method is left out. Disregarding the omission, I believe the augmentation techniques are under-exploited and could benefit from a better investigation. Otherwise, I believe the novelty is fairly limited, since shifting/pasting other labels have been tried and tested for better/domain adapted semantic segmentation.", "strength_and_weaknesses": "### Strengths\n\n- solid performance improvements over predecessor (SynthCP)\n- straightforward error augmentation strategy - ShiftSwap, ClassSwap, and MapSwap involves shifting the image, swapping a class or the whole prediction\n- fairly intuitive insight - if mixing unlabelled samples works for semi-supervised learning [1*] and domain adaptation [2*], variations of the method for error detection should also improve error detection performance\n- can be added on top of any segmentation network\n\n### Weaknesses\n\n- generally poorer results compared to FSNet[3*] -- please cite and compare with\n   - considering the (arguably most relevant metric) AUPR, the method yields 59.84 vs 67.83 on FCN8 ( _+22.15%_ for FSNet compared to +7.8% ErrorAug vs SynthCP) and 53.84 vs 57.84 on DeepLabv2 ( _+15.70%_ for FSNet compared to +7.7% ErrorAug vs SynthCP)\n- ~50% of improvements are due to the architecture input change\n  - even _without any augmentation_ , we are at ~+3.7 over SynthCP -- this means ~half of the error reduction is due to the different way of training the architecture, and only ~4% are due to the augmentation itself\n  - I would speculate that a more powerful architecture and more augmentations would yield better results; nevertheless, the ablation study is frugal (I would add tick marks for each augmentation, I believe they are incremental?) and there are not many options explored (e.g., what happens if all three labels are concatenated, instead of the 1/3 chance of picking one)\n\n\n\n___\n[1*]Tranheden, W., Olsson, V., Pinto, J., & Svensson, L. (2021). Dacs: Domain adaptation via cross-domain mixed sampling. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (pp. 1379-1389).\n\n[2*]Olsson, V., Tranheden, W., Pinto, J., & Svensson, L. (2021). Classmix: Segmentation-based data augmentation for semi-supervised learning. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (pp. 1369-1378).\n\n[3*] Rahman, Q. M., Sünderhauf, N., Corke, P., & Dayoub, F. (2022). Fsnet: A failure detection framework for semantic segmentation. IEEE Robotics and Automation Letters, 7(2), 3030-3037.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "DxvCTKiM3_D", "reviewer": "Reviewer_iFWU", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The work tackles the problem of error detection in semantic segmentation. The naive approach of training a binary pixel-wise classifier on the image and the corresponding semantic map (produced by a segmentation network) is extended by perturbing the semantic map with a three types of straightforward transformations. The experiments indicate that such a binary classifier would surpass previous best methods on standard benchmarks of error detection.", "summary_of_the_review": "Overall, I like this work. However, there is lack of experimentation both in terms of uncertainty estimation and anomaly detection, which needs to be addressed.\n\n**Post-rebuttal comment**\n\nI thank the authors for their response. As before, I like the simplicity of this approach and the consistency improvement over the baseline.\nHowever, I downgrade my rating as I do agree with the other reviewers that the experimentation is a bit too conservative:\n\n* Missing MC-Dropout and TCP in the out-of-distribution setting [R-G1qP]\n* The increase in model capacity is not discussed [R-G1qP]. In fact, as R-2JRb points out, Table 3 suggests that around half of the improvement comes from architectural changes (increasing the number of parameters) rather than from erro augmentation.\n* Missing comparisong to FSNet. I am not sure this would be an apple-to-apple comparsion by just copying the values due to the differences in the segmentation accuracy and the validation set. These points should be discussed, nevertheless.\n* There is not sufficient analysis of this work in the context of anomaly detection and/or uncertainty estimation, as typical in comparable works (e.g. in SynthCP).\n\n", "strength_and_weaknesses": "Strenghts:\n\n- The idea to use data augmentation is straightforward, easily implementable and seems to be more effective than previous works. \n- I like that there are cross-domain experiments (training on Cityscapes, testing on BDD).\n- There is a fair amount of qualitative results and comparisons which improve the presentation quality.\n\nWeaknesses:\n\nI’m not sure where to position this work.\n\nOn one hand, there are uncertainty estimation methods, such as MC-Dropout, that strive to provide calibrated confidence. One could binarise the confidence to obtain an “error detection” method. Although there is one comparison to MC-Dropout, the methodology of training this baseline and its architecture specifics (e.g. one needs to place Dropout layers carefully) are unclear. I am curious if the predictions of the error detection network can be used for uncertainty estimation. For example, how would expected calibration error compare to that of MC-Dropout? I also wonder how this approach compares to test-time augmentation, i.e. augmenting the input image and averaging the semantic maps to obtain calibrated confidence. These would be two off-the-shelf baselines.\n\nOn the other hand, there are anomaly detection methods that focus on identifying out-of-distribution phenomena in the test data. Here, the paper lacks appropriate comparisons as well, since these methods are typically evaluated on anomaly detection benchmarks, e.g. StreetHazards, which is missing in the experiments. Running the method on BDD after training on Cityscapes goes in this direction, but both are the datasets of traffic scenes, after all.\n\nThe notation could be improved (e.g. missing Iverson bracket in Eq. 1, 3)\nThe quality of presentation can be improved significantly. There are numerous typos (e.g. “examples by adjust our […]”, “widiscuss”, etc.), missing cross-references (e.g. Sec. 3.2).\n\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "2w8e51hJCB", "reviewer": "Reviewer_2Ds6", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper introduces a set of data augmentation techniques for predicting dense error maps in semantic segmentation.", "summary_of_the_review": "With better experimental designs (including prioritizing the training process and active fine-tuning) and more comprehensive evaluations (in cross-model and cross-dataset settings), this paper may have a chance to get in. ", "strength_and_weaknesses": "Strengths:\n\n1. The empirical observation that two separate encoding networks (one for the input image and the other for the prediction map) can boost the prediction performance is a plus.\n\nWeaknesses:\n\n1. The paper looks like a rush one with not-hard-to-spot typos on each page.\n\n2. The reviewer is not a fan of augmentation, as it is generally not transferrable to unseen augmentations. To correct the reviewer's biased view, the authors are encouraged to conduct cross-model experiments (e.g., training on FCN-8s / DeepLab v2 and testing on other segmentation models).\n\n3.  How to do boundary handling in ShiftSwap and how to control the similarity level between swapped map and the original one in MapSwap?\n\n4. The primary motivation for developing a failure predictor, either global or local, is to spot hard examples (in an online fashion) to prioritize the training process or to identify catastrophic failures (in an offline fashion) to actively fine-tune the model for improved performance. These results however are not obtained in this paper.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject