Generalize or Detect? Towards Robust Semantic Segmentation Under Multiple Distribution Shifts

Authors: zhitong  gao
bingnan  li
mathieu  salzmann
xuming  he
shanghaitech  university
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-06 14:13:13

Abstract: In open-world scenarios, where both novel classes and domains may exist, an ideal segmentation model should detect anomaly classes for safety and generalize to new domains. However, existing methods often struggle to distinguish between domain-level and semantic-level distribution shifts, leading to poor OOD detection or domain generalization performance. In this work, we aim to equip the model to generalize effectively to covariate-shift regions while precisely identifying semantic-shift regions. To achieve this, we design a novel generative augmentation method to produce coherent images that incorporate both anomaly (or novel) objects and various covariate shifts at both image and object levels. Furthermore, we introduce a training strategy that recalibrates uncertainty specifically for semantic shifts and enhances the feature extractor to align features associated with domain shifts. We validate the effectiveness of our method across benchmarks featuring both semantic and domain shifts. Our method achieves state-of-the-art performance across all benchmarks for both OOD detection and domain generalization. Code is available at https://github.com/gaozhitong/MultiShiftSeg.

Main paper: 
 introduction: 
 Semantic segmentation, a fundamental task in computer vision, has become indispensable in various real-world applications, such as autonomous driving [35] . Recent progress in deep learning-based semantic segmentation has exhibited promising results under the assumption of consistent distributions between the training and testing data. However, these models often falter when faced with distributional shifts. Consequently, research on semantic segmentation under distributional shifts has garnered significant attention in recent years. Some studies approach this challenge from a generalization perspective, aiming to train networks to adapt to data with covariate distribution shifts, such as novel domains [9, 45] . Another line of research focuses on training models to discern (or detect) test data exhibiting semantic distributional shifts, such as anomalies or unfamiliar objects, to ensure reliable predictions [5, 4] . In real-world situation, both types of distribution shifts often occur jointly. This leaves us with the question: Can a model jointly handle both kinds of distribution shift?To address this question, we assess the ability of current domain generalization techniques [9, 45] to detect unknown objects and that of out-of-distribution detection techniques [31, 42, 46] to generalize to unknown domains. Interestingly, we find that models trained using domain generalization techniques, such as domain randomization or whitening transformation, often fail to identify unknown objects, and sometimes even perform worse than the baseline without domain generalization. Furthermore, we observe that models trained using out-of-distribution detection techniques struggle to generalize to unknown domains, exhibiting overly high uncertainty towards objects experiencing domain shifts compared to baseline methods without OOD training. While one intuitive approach is Training for Out-of-distribution (OOD) detection alone [31] yields high uncertainty for both types of shifts, whereas training for domain generalization (DG) alone [9] tends to produce low uncertainty for both. Our method effectively differentiates between the two, generating high uncertainty only for semantic-shift regions. (b) We achieve strong performance in both OOD detection and domaingeneralized semantic segmentation. (c) This is achieved by coherently augmenting original images (first row) with both covariate and semantic shifts (second row).to combine existing anomaly segmentation and domain generalization techniques during training, we note that current domain generalization strategies primarily address image-level shifts, whereas anomaly segmentation focuses on object-level semantic differences. Consequently, the resulting models tend to generalize well to image-level variations, such as changes in weather but struggle with object-level shifts. They often misinterpret any object-level distribution shift as a semantic anomaly, assigning high uncertainty scores to known objects that exhibit covariate changes, such as color variations in cars or changes in pedestrian attire, as demonstrated in Fig. 1 . These experiments underscore the challenge of differentiating and jointly handling different types of distribution shifts.In this work, we jointly study both semantic and covariate distribution shifts. That is, we aim to equip the model to generalize effectively to covariate-shift regions while precisely identifying semantic-shift regions. To achieve this, we design a novel generative augmentation method to produce coherent images that incorporate both anomaly (or novel) objects and various covariate shifts at both image and object levels. Furthermore, we introduce a training strategy that re-calibrates uncertainty specifically for semantic shifts and enhances the feature extractor to align features associated with domain shifts 1 . Specifically, we first introduce a novel data augmentation technique that employs a semantic-to-image generation model to create data that encompasses both covariate and semantic shifts at various levels, allowing the model to learn the essential differences between the shift types. Additionally, we introduce a learnable, semantic-exclusive uncertainty function trained using a relative contrastive loss. We adopt a two-stage training paradigm designed to balance the integration of these enhancements while minimizing their potential interference. A noise-aware training strategy further complements this approach, employing online, pixel-wise selection to mitigate noise in the generated images. Altogether, our approach not only boosts the model's generalization across domain shifts but also ensures a high level of uncertainty in response to semantic shifts.We validate the effectiveness of our method across benchmarks featuring both semantic and domain shifts, including RoadAnomaly [30] , SMIYC [5] , ACDC-POC [12] and MUAD [15] benchmarks. Our results demonstrate that our method achieves state-of-the-art performance across all benchmarks, employing different segmentation backbones for both OOD detection and known class segmentation.In summary, our contributions are: (1) We study semantic segmentation under both semantic and domain shifts, revealing limitations in methods focused on a single shift; (2) We introduce a coherentgenerative augmentation method that augments training data with both shifts; (3) We propose a two-stage, noise-aware training pipeline to optimally leverage augmented data, learning a semanticexclusive uncertainty function while aligning features for domain shifts.Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? Answer: [Yes] Justification: Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 
related work: 
 Anomaly Segmentation (a.k.a. dense out-of-distribution detection). The task aims to detect anomalies or unknown objects by producing pixel-level uncertainty maps. One approach uses generative models to learn training distributions and detect anomalies through reconstruction differences [30, 48] , but this often requires additional networks, resulting in slower inference. Other methods use auxiliary OOD data to train models to distinguish known from unknown instances [46, 6, 31, 42, 36, 18] . Among these, Entropy Maximization [6] uses entire images from COCO [29] as OOD proxies, maximizing softmax entropy on these samples. PEBAL [46] improves upon this by cutting out OOD object instances, pasting them into training images, and using an energy function as the uncertainty score. To reduce artifacts in the pasted OOD region, [52] proposes using a style transfer model to align the pasted region with the background. RPL [31] further regularizes embedding similarity between COCO background pixels and training images. Beyond improvements in OOD proxies and uncertainty functions, recent methods explore the use of the Mask2Former architecture [8] , such as RbA [36] , Mask2Anomaly [42] , and EAM [18] . Our method follows this second approach, generating OOD data with a semantic-to-image model to reduce artifacts and further introducing a learnable uncertainty function to enhance both OOD detection and known class segmentation. It is architecture-agnostic, compatible with both pixel-based and mask-based segmentation backbones.The task aims to train a model on one or more source domains that can perform well on unseen target domains. Existing techniques focus either on introducing specialized model architectures, such as those incorporating normalization [39] or whitening transformations [9, 41, 27] , or on designing domain randomization techniques [45, 50, 23] . Most domain randomization methods rely on image transformation rules or style transfer [45, 50] . Recently, Jia et al. [23] proposed a semantic-to-image model that generates images across diverse domains. Orthogonally, Bi et al. [3] explore architectural changes with Mask2Former. Our approach belongs to the domain randomization category, generating images with both domain and semantic shifts simultaneously to improve the model's ability to distinguish between these shifts.Early works [51, 2] demonstrated the necessity and feasibility of addressing both semantic segmentation under domain shifts and anomaly segmentation. However, these problem settings remain in their early stages (e.g., image-level anomalies) and may not fully capture the true challenges. More recent benchmarks, such as RoadAnomaly [30] , SMIYC [5] , and MUAD [15] , include domain and semantic shifts that better reflect real-world scenarios. Some recent studies [16] have explored the effects of domain shifts on anomaly segmentation benchmarks and proposed a test-time adaptation pipeline to address the problem. In this work, we aim to further bridge the gap by investigating the core challenges of adopting domain generalization techniques and simultaneously enhancing model performance in both areas.This technique is widely used to expand training datasets and prevent overfitting [14, 38, 20, 23, 12] . Unlike rule-based augmentation methods, which focus on image-level changes, generative methods can introduce more object-level variations. Among these works, [12, 33] are the most related to ours, using a text-guided inpainting pipeline to generate anomalies or novel objects. However, this local generation process risks creating inconsistencies between the patch and its background. Additionally, they either focus on generating novel objects within the same domain [12] or use separate pipelines for domain and semantic shifts [33] . In contrast, our method generates multiple distribution shifts in a single process, preserving the global context of the image and ensuring a more natural integration of novel objects.Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes] Justification: Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. Justification:Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
problem formulation and method overview: 
 We consider the problem of semantic segmentation under multiple distribution shifts. Formally, we define the training distribution as P XY in X × Y H×W in , where X = R 3×H×W represents the threedimensional input space of images with H × W pixels, and Y in = [1, C] denotes the semantic label space at each pixel. The test distribution is denoted as Q XY ∈ X × Y H×W test . There are two common types of distribution shifts: covariate shifts-where the input distribution changes (Q X ̸ = P X ) but the label space remains the same -and semantic shifts, which involve alterations to the label space, including the introduction of novel categories (Y test ̸ = Y in ). We consider the possibility of both types of distribution shifts occurring during testing. To distinguish between covariate and semantic shifts, we design a coherent generative-based data augmentation (CG-Aug) pipeline that enriches the training data with realistic and diverse distribution shifts. The pipeline consists of two stages: The first stage uses zero-shot semantic-to-image generation to create a variety of synthetic data, while the second stage automatically filters out low-quality synthetic data. We describe the details of each stage below.Zero-Shot Semantic-to-Image Generation. To generate more realistic and diverse OOD data for segmentation, we propose a generation process that first cut-and-pastes the semantic mask of novel objects to the training labels and then leverages a semantic-to-image generation model to create corresponding augmentation images. By exploiting powerful image generation models, this process is able to produce images with a wide range of covariate shifts and augments the training images with both covariate and semantic shifts in a coherent manner. We detail our process below. . Subsequently, using a pretrained semantic-to-image generative model G : (Y in ∪ Y out ) H×W → R H×W , we generate an augmented image asx aug = G(y aug , t) with y aug = y ⊕ y o , (1) where t is a text prompt and ⊕ denotes the pasting operation. Here we adopt a pretrained Control-Net [53] as G to instantiate the semantic-to-image generation process. Thanks to the powerful prior encoded in Stable Diffusion [43] , this process allows us to generate images with more diverse styles than a task-specific semantic-to-image generation model, therefore creating rich covariate shifts. Moreover, we leverage the text prompt t to produce more diversity in the augmented images by specifying the space, time, and weather, and to enhance the OOD object generation by indicating the class of the pasted objects, via a set of templates (see Appendix A.1 for details).Auto-Filtering. While generative-based augmentation can produce more diverse and realistic distribution shifts than rule-based augmentation, we observed it to often yield inaccurate or noisy rendering for the OOD objects. This might be caused by the fact that these objects appear rarely, or their cut masks are inconsistent with the surroundings. To cope with this, we design an automatic filtering process that identifies generation failures where no object is generated, or a known-category object is incorrectly generated. To achieve this, we leverage pretrained segmentation models to check the region size or its semantic class, and assign a quality score to each generated image. We then filter out the images with low-quality scores (see Appendix A.2 for details).We perform the above image generation process offline before model training, and the resulting synthetic data is used alongside standard augmentation strategies, such as mixup and AnomalyMix [46] , during training. Below, we denote our augmented dataset as D aug = {(x n , y n , x aug n , y aug n )} Nt n=1 .Given the augmented dataset, we aim to train a segmentation model with recalibrated uncertainty output, generating high OOD scores for semantic-shift regions and performing robustly under covariate shifts. To achieve this, we propose a learnable uncertainty function and develop a stagewise learning strategy that initializes the uncertainty function before fine-tuning the entire model.Our training process integrates a relative contrastive loss and a noise-aware data selection scheme, enabling the model to effectively align both the feature space and the OOD output scores. We note that our method is generic and can be applied to pixel-wise models (e.g. DeepLabv3+ [7] ) or mask-wise models (e.g. Mask2Former [8] ).Learnable Uncertainty Function. Suppose we have a neural network with its feature extractor f (x) ∈ R M ×F , where M is the number of pixels (or masks), and F is the feature dimension. We introduce a learnable linear projectionW o ∈ R F ×C , with W o c denotes W o [: c] for short.For a pixel-wise prediction model, we adopt the energy function form and parameterize it into a learnable uncertainty functionEQUATIONFor a mask-based segmentation network, we use the adapted maximum softmax probability (MSP) defined in [8] as the uncertainty function, and parameterize it with W o , leading toEQUATIONHere g(x) ∈ (0, 1) M ×H×W is the sigmoid output of the mask head. For both cases, we initialize the projection function W o as the class weight W in of the pretrained segmentation network. The corresponding uncertainty score corresponds to the original energy score (or MSP score).Relative Contrastive Loss. We train the uncertainty function using a novel relative contrastive loss, which encourages higher uncertainty in unknown-class regions compared to known-class regions, while ensuring that regions with and without covariate shifts exhibit similar levels of uncertainty.Formally, for each batch of data {(x n , y n , x aug n , y aug n )} B n=1 , where B is the batch size, we define the following pixel index sets: Ω in = {i : y(i) ∈ Y in }, representing inlier pixel indices from the original training images; Ω aug = {i : y aug (i) ∈ Y in }, representing inlier pixel indices from the augmented training images (covariate-shift set); and Ω out = {i : y(i) / ∈ Y in } ∪ {i : y aug (i) / ∈ Y in }, representing outlier pixel indices from both original and augmented images (semantic-shift set). Here, y(i) (or y aug (i)) denotes the label of pixel i. Our contrastive loss is defined asEQUATIONwhere τ λ (x) = max(λ -x, 0) is the margin-based contrastive loss, which encourages the input value to exceed λ. The first two terms promote larger uncertainty gaps between unknown-class and knownclass regions, while the third term encourages smaller uncertainty gaps between covariate-shifted and original data. For the third term, we calculate gaps only between pairs of original and augmented images, with m c,i ∈ {0, 1} indicating whether pixel (c, i) is paired in the dataset. The three margin values (λ 1 , λ 2 , λ 3 ) introduce priors on the uncertainty gaps, and are set based on the initial average distance. Our method remains robust across a wide range of margin values (cf. Table 6 ).Compared to existing OOD losses that either maximize uncertainty only for unknown data [31] or supervise known and unknown data separately [46, 42] , our loss supervises the relative distance between them, making it more robust to hyperparameters and simpler to train (cf. Sec.4.5).We now present our two-stage training procedure, which sequentially learns the uncertainty function and the feature extractor f (x) of the segmentation network. Specifically, we first freeze the pretrained segmentation network and learn the semantic-exclusive uncertainty function using the relative contrastive loss defined in Eq. 4. We then fine-tune the feature extractor with both the contrastive and standard segmentation loss to improve the feature representations of both known and OOD classes.Despite the offline filtering process, the generated images may still contain regions that are inconsistent with the label masks. To address this, we introduce a pixel-wise sample selection scheme during training, based on the 'small loss' criterion [1] . Specifically, we compute and rank the cross-entropy loss for each pixel, selecting pixels with smaller losses for backpropagation while ignoring those with larger losses. Formally, our selective cross-entropy loss is defined asEQUATIONwhere p i and y i represent the pixel-wise softmax score and one-hot label, respectively, andη i ∈ {0, 1}indicates whether a pixel is selected for backpropagation. We determine the percentage of selected pixels per batch by visualizing the selection map of a small number of samples, ensuring that visibly incorrect patterns are excluded (see Figure 4 for an example). For models using the Dice loss, such as mask-based ones, we use a similar scheme to remove pixels with a large loss (cf. Appendix A.3).For the original data, which we assume to be noise-free, we set η i = 1 for all pixels. This corresponds to using the standard cross-entropy loss. We denote the segmentation loss for the original data as L in seg and for the generated augmentation data as L aug seg . The overall loss function can be written asEQUATIONHere, β 1 and β 2 ensure that the three loss terms are on the same scale.In summary, our semantic-exclusive uncertainty function, trained through a decoupled parameter training approach and relative contrastive loss, enables the model to fully leverage the generated distribution-shift data. Our noise-aware learning strategy enhances the model's robustness against generation errors. Together, these components of our training pipeline equip the model to effectively learn both domain generalization and accurate OOD detection, ensuring robust performance in dynamic open-world scenarios. 
experiments: 
 In this section, we evaluate our method's performance in jointly handling anomaly segmentation and domain generalization using several datasets that include both domain and semantic shifts: RoadAnomaly [30] , SMIYC [5] , ACDC-POC [12] , and MUAD [15] . We first introduce the datasets in Sec.4.1 and describe the experimental setup in Sec.4.2. The results are presented in Sec.4.3 and Sec.4.4, followed by an ablation study in Sec. 4.5.Following the literature [5, 31, 8] , we train our model on the Cityscapes dataset [11] and evaluate its performance on the test sets described below. Based on the evaluation goals, we divide these datasets into two groups. Examples from each dataset are shown in Fig. 3 .The Road Anomaly dataset [30] includes 60 images of real-world road anomalies such as animals, rocks, and obstacles, featuring various driving conditions Joint Anomaly Segmentation and Domain Generalization Datasets: (a) The ACDC-POC dataset [12] is based on the original ACDC Validation set [44] with generated anomaly objects via inpainting [12] . It contains 200 images with domain shifts including various weather and night scenes. (b) The MUAD dataset [15] is a synthetic dataset containing various driving environments and anomaly objects. We use the challenge test set as in [49] , which contains 240 images with domain shifts at both object and image levels, and anomaly objects such as animals and trash cans. 2These two datasets contain both known-class annotations and unknown object masks, enabling us to evaluate our method jointly for anomaly segmentation and domain generalization.Performance Measure: For evaluation of anomaly segmentation, we use the Area Under the Receiver Operating Characteristics curve (AUROC), the Average Precision (AP), and the False Positive Rate at a True Positive Rate of 95% (FPR95). For evaluation of known class segmentation, we use the mean intersection-over-union (mIoU) and the mean accuracy (mAcc).Implementation Details: We build our method on two segmentation backbones: (a) DeepLabv3+ [7] and (b) Mask2Former [8] . We maintain the network architecture, pretrained models, segmentation loss, and training pipeline the same as in previous work [31, 42] to make a fair comparison. We use the SMIYC validation set for model selection and maintain the same model for evaluation across all test sets. We refer the reader to Appendix A for other training details.We present the performance of our method on anomaly segmentation benchmarks, including Road-Anomaly and SMIYC (RA21 and RO21). As shown in Table 1 , our method achieves state-of-the-art performance on both DeepLabv3+ and Mask2Former-based models. With the same backbone, it outperforms RPL [31] by 3% on RoadAnomaly and 5% on SMIYC, and surpasses Mask2Anomaly [42] by 10% on RoadAnomaly and 3% on SMIYC. Recent methods, M2F-EAM [18] and RbA [36] , use a more powerful Swin Transformer backbone, while ours uses ResNet-50, as Mask2Anomaly. M2F-EAM also uses Mapillary Vistas [37] as additional dataset for training. Despite these unfair comparisons, our method still outperforms both on most metrics, demonstrating its effectiveness. Our method robustly detects anomalies under covariate shifts across five datasets (first five columns) and generated data (last column). The previous method RPL [31] failed to distinguish domain from semantic shifts, producing high uncertainty in both cases.In Fig. 3 , we visualize the uncertainty map output by our method using the DeepLabv3+ architecture.Compared to the previous state-of-the-art method, RPL [31] , our model assigns higher uncertainty scores to anomalous objects and lower uncertainty scores to covariate shifts. This highlights the efficacy of our method in distinguishing between domain shifts and semantic shifts.We then extend our evaluation to the ACDC-POC and MUAD datasets, assessing both anomaly segmentation performance and known-class domain generalization performance. For a comprehensive comparison, we include both previous state-of-the-art OOD detection techniques and domain generalization techniques [9, 45] . Additionally, we trained a DG+OOD combination method by combining naive OOD training with contrastive loss and rule-based data augmentation (denoted as OOD+RuleAug). A DeepLabv3+ model with standard training is used as a baseline method. 3The results are shown in Table 2 , where our model achieves the best results for both out-of-distribution detection and domain generalization, demonstrating its capacity in jointly handling both types of distribution shifts. By comparison, previous methods fall short in either known class segmentation or OOD detection. Specifically, we find that: (a) Previous works that mainly focus on domain generalization (RobustNet [9] , RuleAug [45] ) generally improve the known class segmentation results, but their performance in OOD detection is affected, sometimes worse than the baseline. (b) Previous works that mainly focus on OOD detection (such as PEBAL [46] ) perform poorly We conduct ablation studies to evaluate the design of our components. We begin by analyzing the effectiveness of our proposed modules: the coherent generative-based augmentation (CG-Aug) and our model training strategy. We then proceed with a detailed examination of each module's design.We evaluate the decoupled contributions of our data augmentation and training strategies in Table 3 . Starting with a recent anomaly segmentation method, Mask2Anomaly [42] , we first replace its original OOD data, which utilizes cut-and-pasted COCO images, with our proposed CG-Aug. As shown in Row #2, this substitution results in consistent performance improvements across all datasets. This demonstrates the efficacy of introducing data with both semantic and domain shifts in a coherent way. Next, we replace their training strategy with ours, leading to further gains in performance. This indicates that our training strategy is more effective in leveraging the generated data. Additionally, we present and discuss similar experiments using RPL [31] as a baseline. For more details, please refer to Appendix Table 7 . The second and third methods can be seen as applying [33] to our problem in two ways. Results in Table 4 show that: adding domain shift data significantly improves performance over semantic-shift-only data. Jointly generating DS and SS in one image yields better results than generating them separately. Our method, which generates both DS and SS in one step, achieves the best performance, ensuring more coherence without artifacts and outperforming the two-step approach. We include more comparison results with POC [12] in Appendix C.3. Ablation study of our Training Design We evaluate our training design in Table 5 . In Row #1, we replace our Learnable Uncertainty Function (Learnable-UF) with a fixed energy function. In Row #2, we substitute our Relative Contrastive Loss (RelConLoss) with an absolute contrastive loss [46, 31, 42] , which directly supervises the uncertainty score value rather than the relative gap between two uncertainty scores. In Row #3, we remove the Sample Selection module. Compared to our complete method, presented in the final row, these modifications result in decreased performance in both OOD detection and domain generalization, highlighting the effectiveness of our module design. A visualization of the sample selection process is shown in Figure 4 (a).In Figure 4 (b), we evaluate the effectiveness of our Stage-wise Training pipeline. Starting from a pretrained baseline model, our first stage-fine-tuning only the learnable uncertainty function-doubles the performance on SMIYC-RA/RO datasets, demonstrating that the initial uncertainty function is often sub-optimal and can be significantly improved using fixed features [24] . A second-stage feature fine-tuning further boosts performance. Additionally, our two-stage approach outperforms single-stage fine-tuning with a learnable uncertainty function, showing that training directly with uncalibrated uncertainties can disrupt feature learning and degrade OOD detection performance.We also demonstrate the robustness of our method under a range of hyperparameters (loss margins and selection ratio) in Appendix B.1, and evaluate the impact of generated dataset size in Appendix B.2. 
conclusion: 
 In this work, we have studied semantic segmentation under multiple distribution shifts, finding that prior methods focusing separately on domain generalization and anomaly segmentation may not effectively handle these complex shifts. To tackle this, we have introduced a coherent generative data augmentation approach that enriches training data with both domain and semantic shifts. Additionally, we have proposed a learnable uncertainty function, trained in a stage-wise manner, to fully utilize the data and produce uncertainty scores specifically for semantic shifts. One limitation of our method is its reliance on the quality of the generative model. While we mitigate generation failures through offline autofiltering and online sample selection, some impact remains, such as lower performance for classes the generative model struggles with and potential limitations in scaling up the generated data (see Appendix E for details). 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes]Justification:Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.• The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? Answer: [NA] Justification: Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort. 12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected? Answer: [Yes] Justification: Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets? Answer: [NA] Justification:Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: A.1 Zero-Shot Semantic-to-Image GenerationWe adopt a pretrained semantic-to-image generation model, ControlNet 1.0 [53] , provided by the official GitHub repository 4 , for our generation process. This model is based on Stable Diffusion [43] and fine-tuned on ADE20K [54, 55] . It takes two main inputs: a semantic mask and a text prompt.To obtain our pasted semantic mask, we first convert the masks from Cityscapes labels to ADE20K labels, then overlay this with auxiliary out-of-distribution (OOD) object masks. Specifically, we use the mask labels from ADE20K that belong to the 'thing' categories, excluding those with labels shared with Cityscapes.Our text prompts have two parts: one part specifies the domain shifts, and the other specifies the OOD objects. For domain-shift prompts, we use the template "An image sampled from various stereo video sequences taken by dash cam in {PLACE} in a {WEATHER} {TIME}", where we define PLACE as a set of 100 cities worldwide, WEATHER as ['cloudy', 'rainy', 'snowy', 'foggy', 'clear'], and TIME as ['day', 'night']. Additionally, we improve the OOD object generation by indicating the specific class of pasted objects in the prompt with the template: "There is a {OOD} accidentally staying on the road.", where OOD is the class name of the pasted object. This further contextualizes the generated scene to reflect realistic anomaly scenarios.The generation process can be noisy, particularly when generating image regions for pasted OOD object masks. By nature, these objects are anomalies within the scenes, appearing rarely, and their cut masks may exhibit shapes or poses inconsistent with their surroundings. As a result, the generated objects may deviate significantly in shape from the intended mask, be overlooked (blending into the surroundings), or be incorrectly generated as more common objects within the scene. Such discrepancies make the raw augmented image-label pairs too noisy for direct training.To address these issues, we design an automatic filtering process to identify generation failures, such as cases where no object is generated or a known-category object is incorrectly produced. If the generated object is present and does not belong to a known category, we retain the image while revising the corresponding mask for the generated novel object. Otherwise, we discard the image and regenerate it. To implement this, we use the Segment-Anything Model (SAM), providing the bounding box location of the pasted mask as input to obtain a segmentation. We then compare the SAM output with the original mask, identifying it as a failure case if the Intersection over Union (IoU) is very low (below 0.7). Additionally, we employ a pre-trained segmentation model to produce an uncertainty score for the generated objects, filtering out those with very low uncertainty scores, as these are likely to have been misgenerated into a known-category object. This comprehensive filtering process effectively enhances the quality of the training dataset, making it better suited for effective model training.Following Mask2Anomaly [42] , we train Mask2Former [8] using a combination of dice loss and binary cross-entropy (BCE) loss for the mask prediction head, and cross-entropy loss for the class prediction head. For the dice and BCE losses, we modify the sampling strategy for generated images to implement the sample selection process described in Sec. 3.3. Specifically, we compute the pixel-level BCE loss and select pixels with lower BCE losses for backpropagation in both the dice and BCE loss calculations. Since most generation errors occur at the pixel level, we do not apply sample selection for the mask-wise class prediction in the mask prediction head.We maintain the same model architecture as Mask2Anomaly, which includes a ResNet-50 [19] backbone, a pixel decoder, a Transformer [47] decoder, and a global mask attention mechanism that independently distributes attention between foreground and background. As in Mask2Anomaly, we keep the ResNet backbone frozen while training the remaining model components, and we employ the AdamW [34] optimizer with its default learning rate and scheduler. For our method, the uncertainty loss margins are set to λ 1 = 0.7, λ 2 = 0.5, and λ 3 = 0.2, and we use a selection ratio α = 0.8. The loss weights β 1 and β 2 are set to 10 for both. We use a batch size of 8 for all experiments and train the model on a single NVIDIA A40 48GB GPU.For the DeepLabv3+ backbone, we follow the setup in RPL [31] , using DeepLabv3+ with WideRes-Net38 pretrained by Nvidia. The backbone remains fixed, and only the ASPP layers are fine-tuned. We use the Adam [25] optimizer with a learning rate of 1.0e-6. For our method, the contrastive margins λ 1 , λ 2 , λ 3 are set to 10,5,5, the selection ratio is 0.8. The loss weights are 50 and 10 for β 1 and β 2 respectively. The batch size is set to 8, and all experiments are conducted on two NVIDIA A40 48GB GPUs.As a typical approach to domain generalization, we implement Rule-Based Augmentation as outlined in [45] , using a set of image transformations. Specifically, we apply the following transformations, with their application probabilities indicated in parentheses: color jittering (0.5), Gaussian blur (0.5), random sharpness adjustment (0.5), random contrast adjustment (0.5), random equalization (0.5), random resizing (0.5), random rotation (0.5), random horizontal flipping (0.75), and random cropping (1.0).Loss Margins Our relative contrastive loss 4 includes three terms, each with a margin value λ controlling the distance penalty limits. These margins are set based on the average uncertainty scores from the training set. Specifically, we compute the differences in uncertainty scores between unknown vs. original known data, unknown vs. augmented known data, original known vs. augmented known data, and set the differences as margins for these distance respectively. Moreover, our two-stage training framework first trains the uncertainty function based on the existing model, allowing this function to adapt to different scales. This provides flexibility in parameter setting even without prior knowledge.In Table 6 , we evaluate the model's robustness across a wide range of hyperparameter variations.Our default loss margins are [λ 1 , λ 2 , λ 3 ] = [10, 5, 5] . We start by scaling them by 0.1 and 10 and conduct experiments using (1, 0.5, 0.5) and (100, 50, 50), respectively. As shown in Table 6 (a), the uncertainty function adjusted to these scales with minimal impact on the results. Further analysis, such as changing the second and third parameters individually, showed that while the relative sizes of the three contrastive losses have some impact, the effect remains minor(see Table 6 (b) and Table 6 (c)).Ensuring that the parameters are set within an order of magnitude does not affect the results much. Selection Ratio We determine the selection ratio for our sample selection process by visualizing the selection map of a small batch of data under several choices to ensure that visibly incorrect patterns are removed. To examine the impact of selection ratio to our method, we conducted experiments with selection ratios ranging from 0.6 to 0.9, as detailed in Figure 5 (a). The results show that while including too many pixels (1.0) introduces noise, and including too few (0.6) removes useful regions, the model performance is stable within a wide range (0.7 to 0.9), demonstrating the robustness of the model to this hyperparameter. By default, we generate our distribution-shift dataset at the same size as Cityscapes. To analyze the impact of the generated dataset size, we scale it to 2x and 3x the size of the Cityscapes training set. As shown in Fig. 5 (b), there is a significant improvement from dataset sizes 0 to 1, demonstrating the effectiveness of our generated data, with further gains observed as the dataset size increases.We evaluate the decoupled contributions of our data augmentation and training strategies with RPL [31] in Table 7 . Similar to Table 3 , we first replace its original OOD data, which utilizes cut-andpasted COCO images, with our proposed CG-Aug in Row#2. However, we find the improvement is not as significant. This may be due to certain aspects of RPL's loss and training design being less suitable for our scenario. Firstly, RPL relies on the original network's predictions to supervise a learnable residual part. Since the original network does not generalize well to data with domainshift, this results in imprecise supervision. Secondly, the RPL uncertainty loss focuses solely on increasing uncertainty for unknowns, without adequately addressing the known classes, particularly for augmented images. Additionally, restricting the trainable parameters to a residual block may limit the model's ability to learn more complex patterns, thereby reducing overall effectiveness.Next, we replace their training strategy with ours, leading to consistent performance improvement.Those results demonstrate that effectively utilizing the generated training data with multiple distribution shifts remains an open question. Our work takes a step towards analyzing the shortcomings of existing training designs, offering novel and effective strategies for better handling this data.Performance under Individual Domain Shifts In addition to the main Table 2 , we present the ACDC-POC results with domain-specific splits to provide a more detailed analysis of our method across different types of domain shifts. As shown in Table 8 , our method outperforms previous state-of-the-art methods (RPL [31] and Mask2Anomaly [42] ) across four domains-fog, rain, snow, and night-on most metrics in both OOD detection and known-class segmentation. Per-Class Segmentation Results We evaluated per-class segmentation results and compared them with the baseline DeepLabv3+ [7] model. Results are presented in Table 9 . Our method improves segmentation performance (mIoU) across most categories. However, performance in fence, pole, and traffic sign remains similar (with differences of less than 1%), and performance on vegetation decreases by 3%, likely due to poor generation quality for this class.Table 9 : Per-class segmentation results. We present the segmentation performance (mIoU) for each known class on the ACDC-POC dataset. Compared to the baseline model (DeepLabv3+ [7] ), our method improves performance in most categories. To provide a comprehensive analysis of our method, we complement the main metrics (AP, FPR) with component-wise metrics (sIoU, PPV, F1) on the SMIYC benchmark. The results are summarized in Table 10 . As shown, our method outperforms RPL [31] and Mask2Anomaly [42] across most evaluation metrics.We present additional comparison results of our CG-Aug method against POC [12] across six datasets. Following the experimental setup in POC [12] , we replace the default OOD data (COCO) in Mask2Anomaly [42] with our CG-Aug. As shown in Table 11 , our method outperforms both POC variations on most datasets. Notably, for FS-Static [4] , where OOD objects are introduced through Table 10 : Comprehensive Metric Results on SMIYC. We present the results of our method across pixel-wise metrics (AP, FPR) and component-wise metrics (sIoU, PPV, and F1). Compared to recent methods RPL [31] and Mask2Anomaly [42] , our method achieves superior or comparable results. cut-and-paste, COCO achieves the best performance; however, this setup lacks the ability to reflect realistic OOD objects encountered in real-world scenarios.Table 11 : Comparison of our CG-Aug with POC. Following the experimental setup in POC [12] , we replace the default OOD data (COCO) in Mask2Anomaly [42] with our CG-Aug, and compare with two versions of POC. Our method outperforms both POC variations on most datasets. To assess the effectiveness of our method in domain generalization, we conducted additional comparisons with several recent approaches, including IBN [39] , IterNorm [22] , IW [40] , ISW [10] , ISSA [27] , and CMFormer [3] , on the ACDC [44] dataset. As shown in Table 12 , our method outperforms all ResNet-based methods in the Fog, Rain, and Snow domains, achieving comparable results in the Night domain. Among Mask2Former-based methods, our approach also surpasses ISSA [27] , which similarly uses a ResNet backbone. However, there remains a notable performance gap between our method and CMFormer [3] , likely due to CMFormer's use of the Swin Transformer [32] as the backbone for Mask2Former.Table 12 : Domain generalization performance comparison between our method and other DG methods. Results from other methods are taken from CMFormer [3] . All methods are trained on the Cityscapes [11] dataset and tested on the ACDC [44] In Fig. 6 , we provide additional visualization examples of our generated images (row 2), corresponding selection maps (row 3), and the loss maps used to produce the selection map (row 4). Below each column, we display the weather, time, and location prompts that guide the model in generating diverse covariate shifts, as well as the OOD prompts used to generate objects. As shown, our method effectively generates images with both domain and semantic shifts, with the novel objects blending seamlessly into the background (e.g. pose and lighting). Additionally, our sample selection process effectively filters out some generation errors (highlighted in red boxes). A limitation of our method is its reliance on the quality of the generative model. Although we apply offline auto-filtering and online sample selection to minimize the impact of generation failures during training, some issues may still arise. Specifically, we observe that generation failures typically occur in the following scenarios: (a) remote scenes, (b) small objects, and (c) text-related elements. These limitations highlight the current constraints of generative models and suggest areas for future research. Below, we discuss the impact of generation failures:Impact on Class-Specific Learning: Generation failures can adversely affect specific classes. As shown in Table 9 , we evaluated per-class segmentation results and compared them with the baseline model on Cityscapes. Performance in categories-such as fence, pole, and traffic sign-remains similar (differences of less than 1%), and vegetation shows a 3% decrease, likely due to lower generation quality for these classes.Performance Saturation: We observe that performance tends to saturate with increasing amounts of generated data. Experiments with dataset scaling from 1.0x to 2.0x and 3.0x Cityscapes sizes, as shown in Figure 5 , indicate that while performance improves with larger dataset size, it eventually plateaus. This saturation may result from an interplay between the benefits of additional data and the adverse effects of generation failures.Enhancing OOD detection in autonomous vehicles can significantly improve safety by enabling these systems to better recognize and respond to novel and unexpected situations, thereby reducing the risk of accidents. Improved robustness to domain shifts also contributes to greater resilience and safety across diverse driving scenarios. However, improved OOD detection may lead to an over-reliance on autonomous systems, potentially reducing the vigilance of human drivers or passengers in semiautonomous vehicles. Additionally, unintended biases in OOD detection systems could result in unsafe responses to certain situations, particularly if training data does not sufficiently cover diverse scenarios, potentially compromising safety in rare but critical cases.In this work, we use 'domain shift' and 'covariate shift' interchangeably.Since we use Cityscapes as the training set, the unknown object set differs from that used in[49].For all compared methods, except RuleAug, we used the official pretrained models provided by the respective authors and performed re-inference to obtain the results. For RuleAug, we applied a combination of color jittering, Gaussian blur, etc., as suggested in[45]. For further details, please refer to the Appendix A.5https://github.com/lllyasviel/ControlNet/tree/main 


Decision: This paper studies the problems of domain generalization and anomaly segmentation in tandem, showing that sometimes improving model performance in one direction is detrimental for performance on the other. They propose a new method to tackle both sources of distributional shifts at the same time.

Before the rebuttal, this work had a borderline rating (BR, BA, BA, WA). The main point of the most negative review was the novelty with respect to prior art. The Authors exhaustively commented on this, and the Reviewer found the answer satisfactory enough to increase their score to BA. After the rebuttal, all Reviewers are leaning towards accepting this paper (2 BA and 2 WA).

I agree with the Reviewers on the importance of the problem studied in this paper, and recommend acceptance. The Authors should revise the paper according to the feedback received by the Authors, and further clarify the concepts that were not clear to some of the Reviewers -- for example, the core differences with most related works.

Reviews: review: paper_summary: paper_summary: This paper addresses semantic segmentation in the presence of domain and semantic shifts. To enhance model robustness, the authors propose using a generative model guided by ground truth labels to generate domain-shifted images and further inpaint random negative data. Due to potential noise in the generative process, an additional module is introduced to filter incorrectly filled-in image areas, removing pixels with the highest loss in the generated image.

The segmentation model is trained with a contrastive loss that promotes high uncertainty in negative samples and consistent uncertainty between original and domain-shifted areas.

Training proceeds in two stages. First, an uncertainty function is trained on top of a frozen pretrained semantic segmentation model. Subsequently, both the uncertainty function and the segmentation model are trained simultaneously.
strength_weakness: strength_weakness: 1. The first contribution regarding the presence of domain and semantic shifts has already been addressed in previous works [a] [b].
2. There is limited novelty, as the use of generative models for data augmentation [8] has already been proposed.
3. SOTA results on SMIYC [c] [d] were omitted.

[a] Zendel et al., WildDash - Creating Hazard-Aware Benchmarks, ECCV 2018
[b] Bevandic et al., Simultaneous Semantic Segmentation and Outlier Detection in Presence of Domain Shift, GCPR 2019
[c] Grcic et al., On Advantages of Mask-level Recognition for Outlier-aware Segmentation, CVPRW 2023
[d] Nayal et al. RbA: Segmenting Unknown Regions Rejected by All, ICCV 2023
questions: questions: 1. How is sampling done for Eq. 4?
limitations: limitations: 1. The method is limited by the pretraining of the generative model
paper_summary: paper_summary: This paper aims to tackle both covariate-shift and semantic-shift in semantic segmentation. The idea is to use a generative augmentation method to produce coherent images that incorporate both anomaly objects and various covariate shifts at both image and object levels. The semantic segmentation model is then on the synthetic image for recalibrating uncertainty for semantic shifts and enhances the feature extractor to align features associated with domain shifts. The authors have conducted extensive experiments to show the effectiveness of the proposed method over the state-of-the-art methods.
strength_weakness: strength_weakness: 1. The novelty of the proposed solution is somewhat limited. While the generative-based data augmentation appears reasonable, its novelty is not clearly articulated.

2. The proposed relative contrastive loss involves several hyperparameters, but it is unclear how to effectively tune these parameters in practice.

3. The necessity of the Two-Stage Noise-Aware Training is not clear. Why not use a single-stage training process that trains the uncertainty function and the feature extractor simultaneously?

4. What are conditioned label masks?

5. How can we determine if a pixel has a clean or incorrect label given the augmented images? Equation 6 seems to be used for assessing the accuracy of the pixel label, but why is this an effective strategy? How is α determined?

6. The proposed method involves many hyperparameters, which reduces its applicability in real-world scenarios.
questions: questions: Please see weaknesses.
limitations: limitations: N/A
paper_summary: paper_summary: This work proposes a novel generative pipeline and fine-tuning method for anomaly detection under domain shift. The generative pipeline uses a semantic-map to image model that can leverage the labels from the Cityscapes dataset with some modifications which introduce novel unknown classes. The resulting images have unknown novel objects (semantic shifts) and modified known classes (covariate shifts) while preserving most of the semantic meaning.

Authors use the images augmented with their pipeline to train with a contrastive loss that pushes the representations of known classes together while pushes the novel categories away.
strength_weakness: strength_weakness: Although experiments are quite extensive, it would have been interesting to decouple the contributions of the generative pipeline and the proposed fine-tuning mechanism in the experiments. Many of the methods in table 2 could have used the images augmented with the generative model, and it would be very interesting to see how much can the generated data benefit those models.

Especially because it seems from Table 2 that on the MUAD dataset, the OOD+RuleAug baseline is worse than RPL or Mask2Anomaly and to my understanding, the OOD+RuleAug baseline consists in the training proposed in section 3.3 but replacing the generative data augmentation with a rule-based one from [29]). Thus, leaves the question whether other existing methods like RPL or Mask2Anomaly + the proposed generated data could be better than the training scheme proposed in 3.3. 

I am aware the authors show in table 3 that every proposed component in the training pipeline leads to a drop in accuracy when removed, however, that does not show that the overall proposed pipeline is better than that of previous works if they all used the generated images for training (I would only focus on comparing with RPL / Mask2Anomaly which are the most recent ones).

Moreover, comparing with previous methods, using a contrastive loss was already proposed in prior work e.g. RPL[21]. I think a better discussion of the related work in section 3.3 to highlight the novelty in the components would be very helpful to readers.

In terms of the generative pipeline, I would like to bring to the attention of the authors a preprint which is very much aligned with the proposed pipeline and that it might be worth discussing in the related work: 

[Loiseau *et al.* Reliability in Semantic Segmentation: Can We Use Synthetic Data? ArXiv 2023](https://arxiv.org/pdf/2312.09231)

As a preprint authors are not expected to compare quantitatively but given the similarity and the fact it was published Dec 2023 might be fair to mention it in the related work and discuss the differences with [8] and [34] a bit more extensively.

Last but not least, although the proposed generative pipeline leverages a model that is conditioned on a semantic mask, the generative models still have important limitations and can introduce significant changes that do not align well with the original image. From looking at Fig 5 in the appendix, I could easily spot that in the first image it removes a very large building and replaces it mostly with sky; in the second it does something similar replacing the building at the end with trees; the fifth one replaces traffic lights with street lamps and in the last image modifies the direction sign with a "sign" that does not mean anything. Perhaps for some applications this will not matter, depending on the classes of interest and in the level of semantic detail one might want to accomplish, but in the context of autonomous this kind of modifications might be problematic.

I think the pipeline still has its merit and can be of use to the community but a more clear discussion of the limitations of generative models is needed and perhaps some examples of failure cases would be of interest to add in the appendix.
questions: questions: See my weaknesses section. Especially I'd be interested to know what are the main differences of the proposed training schedule and RPL and Mask2Anomaly and the main differences between the generative pipeline and [8, 34] and [Loiseau *et al.*](https://arxiv.org/pdf/2312.09231).
limitations: limitations: See my last point in the weaknesses section. More discussion on the limitations of the generative pipeline, especially regarding the preservation of semantic details.
paper_summary: paper_summary: The authors propose a method which generalize effectively to covariate-shift regions while precisely identifying semantic-shift regions, i.e., domain generalization and OoD segmentation. They design a novel generative augmentation method to produce coherent images that incorporate both, various covariate shifts and anomaly objects. Moreover, they introduce a training strategy that recalibrates uncertainty specifically for semantic shifts and enhances the feature extractor to align features associated with domain shifts. The approach is compared  across different benchmarks for domain generalization and OoD segmentation (or both).
strength_weakness: strength_weakness: The paper is written in a clear way, but it is also a bit unclean, for example the caption of figure 1c, the reference to figure 2 is missing in the text and there are a few typos (for example "the ResNet backbone are froze" or "AadmW").

The related work section is rather short and as a comparison to anomaly detection and domain generalization it is only mentioned that these methods only focus on their problem, but not on both (in comparison to the authors). However, I miss a methodological comparison at this point to highlight the novelties of the paper.

In the SMIYC benchmark, there are other methods that perform better than the method presented, but there is no comparison with them or an argument of why a comparison does not make sense. In addition, 5 different metrics are used in this benchmark for the evaluation, but the authors only calculate 1, which makes the comparison more difficult. 

The method depends on many hyperparameters, but there are no ablation studies on different values.
questions: questions: The described papers for domain generalization are very limited and rather older, there is certainly newer literature to compare with.

The problem of domain generalization is considered, but the SMIYC validation data is used for model selection, so is it domain adaptation?
limitations: limitations: No limitations section in the paper, but the robustness of the method was demonstrated on various datasets.

Comments: title: Thank you
Answer to the rebuttal
Thank you and Clarification on Hyperparameters
Thank you
Thank you and Further Explanation
Thank you the clarifications.
Reviewer response
Rebuttal by Authors
Rebuttal by Authors
Rebuttal by Authors
Rebuttal by Authors
Rebuttal by Authors
comment: Thank you again for your time and valuable comments. We are pleased to have addressed your concerns and sincerely appreciate your positive feedback on our work.
I find the rebuttal to be thorough enough to increase my score to borderline accept.
Thank you for your reply and for acknowledging our problem setting and proposed method.

Regarding your concern about hyperparameter tuning, as detailed in our responses to weaknesses 2 and 5, the hyperparameters in our method, including loss margins and selection ratios, can be set directly based on training data statistics without the need for further tuning. Additionally, in the attached PDF, we demonstrate the robustness of our method across a wide range of these hyperparameters. Other normal training hyperparameters, like the learning rate, are tuned once per backbone and then kept consistent throughout experiments. We want to emphasize that our method does not require more hyperparameter tuning than previous approaches like RPL and Mask2Anomaly. Developing an OOD training strategy that eliminates the need for hyperparameter tuning is an interesting direction, and we will leave it to future work.
Thank you for increasing the score and for your time and careful review. We are glad to have addressed your concerns with the additional experiments and discussions, and we will incorporate these into the final manuscript.
Thank you for your constructive and detailed feedback. 

- **Regarding the comparisons**: We appreciate your suggestion and will include RbA and M2F-EAM (Grcic et al.) in the revised manuscript. Thank you also for pointing out our typo in the general response, where (the second) Table A1 should be Table A2.
- **Regarding the novelty:** Thank you for acknowledging the contributions of our work. We are pleased that our additional experiments have clarified the advantage of our training pipeline, which is tailored to our specific task and effectively leverages our generated data.
    - **Further experiments with COCO data as OOD**: In response to your further suggestion, we have conducted experiments using COCO data as OOD to assess the efficacy of our proposed training pipeline without the generated data. We note that some training components, such as noise-aware learning and relative contrastive loss, are tailored for scenarios involving the generated domain-shift data. Therefore, in this evaluation, we focus primarily on assessing the effectiveness of our two-stage learnable uncertainty function and the relative contrastive loss applied between the original data and OOD data. **The results in Table C5 show that our training method surpasses RPL and Mask2Anomaly on most metrics,** and the combination with our generative data yields even greater performance improvements.
    - **Comparison with POC:** We appreciate your recognition of the novelty of our generative pipeline. Our work primarily addresses scenarios where both domain shifts and semantic shifts are present, which we believe are common in real-world applications. To address your concerns regarding performance under smaller domain shifts, we have included results on the FS LostAndFound and FS Static datasets, comparing our method with POC using the Mask2Anomaly training pipeline. As shown in Table C6, **our method demonstrates superior performance on FS LostAndFound**, indicating that our generated data closely resembles real OOD scenarios. However, on the FS Static dataset, our performance is lower than that of POC. We note that the FS Static dataset's OOD data is generated using a cut-and-paste technique, which may not reflect real-world OOD distributions. This could explain why simple COCO-pasted data achieves the best results on this dataset.
    
    Overall, we thank you again for your valuable input, which has helped us clarify the components of our design. We will incorporate all these discussions into the revised paper.
    
    Table C5: Performance of our training pipeline using COCO data, compared to previous methods Mask2Anomaly and RPL.
    
    | Backbone | FT | OOD Data | Road Anomaly |  |  | SMIYC -RA21 (val) |  | SMIYC -RO21 (Val) |  |
    | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
    |  |  |  | AUC$\uparrow$ | AP$\uparrow$ | FPR$\downarrow$ | AP$\uparrow$ | FPR$\downarrow$ | AP$\uparrow$ | FPR$\downarrow$ |
    | Mask2Former | Mask2Anomaly | COCO | - | 79.70 | 13.45 | 94.50 | 3.30 | 88.60 | 0.30 |
    |  | Ours | COCO | 95.83 | 80.94 | 29.12 | **97.41** | 1.60 | 92.89 | 0.50 |
    |  | Ours | Ours | **97.94** | **90.17** | **7.54** | 97.31 | **1.04** | **93.24** | **0.14** |
    | DeepLabv3+ | RPL | COCO | 95.72 | 71.61 | 17.74 | 88.55 | 7.18 | **96.91** | **0.09** |
    |  | Ours | COCO | 96.26 | **76.01** | 17.44 | 91.40 | 6.80 | 96.77 | 0.12 |
    |  | Ours | Ours | **96.40** | 74.60 | **16.08** | **93.82** | **3.94** | 95.20 | 0.19 |
    
    Table C6: Comparison of our CG-Augmentation with POC on FS LostAndFound and FS Static, integrated into the Mask2Anomaly training pipeline. COCO results are from the official Mask2Anomaly paper;  POC alt. and POC c. results are from the official POC publication.
    
    |  | FS_LostAndFound (val) |  | FS_Static (val) |  |
    | --- | --- | --- | --- | --- |
    | Mask2Anomlay+ | AP$\uparrow$ | FPR$\downarrow$ | AP$\uparrow$ | FPR$\downarrow$ |
    | COCO | 69.41 | 9.46 | **90.54** | **1.98** |
    | POC alt. | 68.8 | 11.4 | 87.4 | 3.1 |
    | POC c. | 73.0 | **9.2** | 87.0 | 2.1 |
    | CGAug (ours) | **76.56** | 10.17 | 85.70 | 7.16 |
I would like to thank the authors for the response. Since in the real-world applications of OOD segmentation, it is hard or impossible to tune the hyperparameters which somewhat limits the applicability of the method. But the overall, the problem is the interesting and the proposed method is reasonable. Therefore, I will maintain my original rating.
Thank you for the additional experiments and the responses. Given the new experiments and answers regarding all reviewers, I have increased my score.
I would like to thank the authors for their detailed response. After reading their response and other reviewers comments I think the authors did a reasonable job at addressing my questions and the other reviewers comments. Therefore I am inclined to keep my positive score. 

**Missing comparisons:**

 I agree with reviewer LCzB that it would be nice to include RbA as it has a strong performance on SMIYC and is concurrent to Mask2Anomaly and RPL in ICCV 2023 so it might not be fair to exclude it from the comparison. It would also be nice to add Grcic et al., On Advantages of Mask-level Recognition for Outlier-aware Segmentation, against the comparisons. To me this has been sufficiently addressed with (the second) Table A1, which I think should be Table A2 :)

**Novelty of paper:**

 As I see it there are two main contributions in this work, one is the generative pipeline to augment images with both domain and semantic shifts and the other is the training pipeline to leverage that data. Initially the two were entangled as there were no experiments with other methods and their generated data. However, with the additional experiments in Table C1 it is more clear that, although other methods might benefit from the generated data in some cases, the proposed method seems to leverage the generated data better (i.e. Ours with CG-Aug (ours) is better than RPL/Mask2Anomaly with CG-Aug (ours) ). 

There are no experiments with the proposed fine-tuning method and COCO data as OOD which would have been interesting too to understand if the proposed fine-tuning method alone could surpass previous works or if the combination with the new generated data is the key to the improved performance.

Regarding the comparison with POC [8], it would be interesting to discuss that the motivation in [8] is to precisely avoid domain shifts in the evaluation, as this would hinder anomaly segmentation. All datasets used in the comparison in Table 2 of the rebuttal pdf (RoadAnomaly, SMIYC and ACDC-POC) have strong domain shifts with respect to Cityscapes, but datasets with a smaller domain shift such as Lost and Found were not included. The proposed CG-Aug, combines both domain shifts and semantic shifts to improve anomaly segmentation out-of-domain while POC focuses on introducing semantic shifts that minimally modify the image beyond the introduced objects. Both will have its Pro's and Con's and this is where I see the novelty in the generative part.
# 4.Generation Failures
We agree with the reviewer that generative models still have important limitations, and appreciate the reviewer’s detailed examination and careful thought for generation failure cases and impact. Below, we first discuss how we currently deal with generation failures, followed by our analysis of the generation failure pattern, and examine how these failures affect model training. 

- **Noise-aware Learning:** During training, we mitigate the impact of generation failures by using a sample selection strategy (Sec. 3.3.2). We note that for failure cases mentioned by the reviewer, some can be prevented via pixel selection during training. To better illustrate this, we include the loss and selection map corresponding to the images from Fig 5. in the attached PDF. 

- **Generation Failure Cases:** Inspired by the reviewer, we examine our generated image and observe that generation failures typically occur in the following scenarios: (a) Remote scenes. (b) Small objects. (c) Text-related elements. This demonstrates the limitations of the current generative model and we hope they can be addressed in future research.

- **Impact of Generation Failures:**
    - **Impact on Category Learning:** Generation failures may adversely affect specific classes. We evaluated per-class segmentation results and compared them with the baseline model. Results are presented in Table C3.  We find performance in six categories, such as fence, pole, and traffic sign, remains similar (differences less than 1%); and performance on vegetation is worsened by 3%, likely due to poor generation quality for this class.
    - **Performance Saturation:** We observe that performance tends to saturate with increasing generative data. Experiments varying the dataset scale from 1.0x to 2.0x and 3.0x Cityscapes sizes, as shown in Figure 1(b) in the supplementary material, suggest this saturation. This may be due to an interplay between the benefits of additional data and the negative impact of generation failures

We will include more examples of failure cases and a comprehensive discussion of the limitations of generative models in the revised paper and appendix.

Table C4. Per-class segmentation results in mIoU on ACDC-POC dataset.
|  | road | sidewalk | building | wall | fence | pole | traffic light | traffic sign | vegetation | terrain | sky | person | rider | car | truck | bus | train | motorcycle | bicycle |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| DeepLab | 77.57 | 37.70 | 63.55 | 17.46 | **31.22** | 49.42 | 64.24 | 52.78 | **75.75** | 10.58 | 81.12 | 54.17 | 19.37 | 77.89 | **50.76** | **50.74** | 30.33 | 13.77 | 22.05 |
| Ours | **85.75** | **58.16** | **74.80** | **40.79** | 29.43 | **50.81** | **71.69** | **53.01** | 72.82 | **30.21** | **81.35** | **62.77** | **32.17** | **79.99** | 49.41 | 49.33 | **38.08** | **29.19** | **38.52** |
Thank you for the thorough comments and many constructive suggestions. We appreciate the mention of the interesting work by Loiseau et al. [a1] and have discussed our differences, including [8] and [34], in the general response. We address the other concerns below.

# 1. Decouple Contribution
We thank the reviewer for the suggestions and have conducted additional experiments on Mask2Anomaly and RPL. The results are shown in Table C1. Below, we discuss the findings:

-**Mask2Anomaly**: We observe consistent improvements across all datasets, demonstrating the benefits of using our generated data. However, the final performance achieved by Mask2Anomaly + CG-Aug is still lower than the model fine-tuned with our pipeline. This highlights the efficacy of the proposed training design. 

-**RPL**: The improvement is not as significant. This may be due to certain aspects of RPL's loss and training design being less suitable for our scenario. Firstly, RPL relies on the original network's predictions to supervise a learnable residual part. Since the original network does not generalize well to data with domain-shift, this results in imprecise supervision. Secondly, the RPL uncertainty loss focuses solely on increasing uncertainty for unknowns, without adequately addressing the known classes, particularly for augmented images. Additionally, restricting the trainable parameters to a residual block may limit the model's ability to learn more complex patterns, thereby reducing overall effectiveness.

Those results demonstrate that effectively utilizing the generated training data with multiple distribution shifts remains an open question.  Our work takes a step towards analyzing the shortcomings of existing training designs, offering novel and effective strategies for better handling this data.

Table C1: We apply our coherent generative-based augmentation (CG-Aug) to the most recent works, RPL and Mask2Anomaly. 

|  |  |  | RoadAnomaly |  | SMIYC-RA21(val) |  | SMIYC-RO21(val) |  |
|---|---|---|---|---|---|---|---|---|
| Backbone | FT | OOD Data | AP$\uparrow$  | FPR$\downarrow$ | AP$\uparrow$  | FPR$\downarrow$ | AP$\uparrow$  | FPR$\downarrow$ |
| Mask2Former | Mask2Anomaly  | COCO (Default) | 79.70 | 13.45 | 94.50 | 3.30 | 88.6 | 0.30 |
|  | Mask2Anomaly  | CG-Aug (Ours) | 85.47 | 22.38 | 97.96 | 1.55 | 89.80 | 0.12 |
|  | Ours | CG-Aug (Ours) | **90.17** | **7.54** | **97.31** | **1.04** | **93.24** | **0.14** |
| DeepLabv3+ | RPL  | COCO (Default) | 71.61 | 17.74 | 88.55 | 7.18 | **96.91** | **0.09** |
|  | RPL  | CG-Aug (Ours) | 72.46 | 21.85 | 83.50 | 23.88 | 93.30 | 0.51 |
|  | Ours | CG-Aug (Ours) | **74.60** | **16.08** | **93.82** | **3.94** | 95.20 | 0.19 |


# 2. Contrastive Loss

We appreciate the reviewer’s kind suggestion and will add more related work in our model training section (3.3). Below, we discuss the novelty of the proposed contrastive loss.

**Compared with contrastive loss in RPL**: RPL employs their contrastive loss *on a projected feature space*, supervising OoD training with a combination of feature contrastive loss and an additional energy loss to maximize uncertainty scores for unknown class data. In contrast, we calculate our contrastive loss *directly on the uncertainty scores*. This direct supervision allows for more explicit and effective ranking of uncertainties across different data types. Experiments in Table C1 replacing our OOD loss with RPL's showed that RPL's feature contrastive loss alone barely improves performance. Even with combined losses, RPL's performance falls short of ours, demonstrating the efficacy of our direct supervision on uncertainty scores.

**Compared with other OOD Losses**: Existing OOD losses either maximize uncertainty scores solely for unknown data or supervise unknown and known data separately. In contrast, our method supervises the relative distance between OOD and inlier samples. We find this contrastive term to be more robust to hyperparameters and easier to optimize. To demonstrate its efficacy, we replaced the distance-based supervision with value-based supervision, similar to that used in Mask2Anomaly and PEBAL. As shown in Table 3, row #3, this change led to a performance decrease, validating the effectiveness of our loss design.

Table C2: Ablation study of our contrastive loss (In DeepLab v3+). Our loss performs better in both OOD detection and known class segmentation results.
|  | SMIYC-RA21(val) |  | ACDC-POC |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| Loss | AP $\uparrow$ | FPR$\downarrow$ | AP$\uparrow$ | FPR$\downarrow$ | mIoU$\uparrow$ | mAcc$\uparrow$ |
| FeaConLoss  (RPL) | 68.40 | 48.42 | 30.98 | 40.20 | 49.42 | 79.38 |
| FeaConLoss +Energy (RPL) | 86.41 | 9.29 | 76.12 | 1.93 | 51.43 | 83.15 |
| RelConLoss (Ours) | **93.82** | **3.94** | **82.41** | **1.01** | **54.12** | **85.07** |
# Question 1: Domain Generalization Papers
We chose RobustNet (2021)[6] and RuleAug (2023)[29] for our benchmark comparison because they represent two main DG strategies: Constraining the learned feature distributions and domain randomization. Although adding more recent DG techniques could enhance our analysis, many of these works do not provide pretrained models on Cityscapes or are based on different segmentation backbones, making direct evaluation and comparison challenging [25, d5].  

To address the reviewer’s concern, we additionally evaluated a recently published work, CMFormer [d4] (2024), which adopts the Mask2Former architecture for DG in semantic segmentation. Using the provided pretrained model and official code, we performed inference on the ACDC-POC dataset and compared it with our method under the Mask2Former architecture. We note that the comparison is not entirely fair, as CMFormer uses a more powerful Swin Transformer backbone while we use ResNet50 following Mask2Anomaly. As shown in Table A2,  CMFormer non-surprisingly performs better on known class segmentation with a 10-point gap in mIoU and 2 points in mAcc. However,  its OOD segmentation performance is significantly worse, with over 60 points lower in AP and 30 points higher in FPR. This supports our finding that existing DG techniques might overly generalize to all types of distribution shifts, making it difficult to recognize unknown objects, raising safety concerns in autonomous driving scenarios.

| Domain | ACDC-POC |  |  |  |
| --- | --- | --- | --- | --- |
| Method | AP$\uparrow$ | FPR$\downarrow$ | mIoU$\uparrow$ | mAcc$\uparrow$ |
| CMFormer | 27.84 | 31.25 | **60.71** | **85.90** |
| Ours | **90.42** | **0.46** | 51.75 | 83.16 |

# Question 2: Validation
The main difference between domain generalization and domain adaptation techniques is that the former focuses on training one model to generalize to any data with different domain shifts, while the latter typically trains each model for each target domain. Our method belongs to domain generalization as we train one model on Cityscapes and evaluate it on various datasets with different domain shifts. We use the SMIYC validation set for model selection to ensure a fair comparison with other anomaly segmentation techniques in the benchmark, which also use the validation set for model selection.

[d4] Bi, Qi et al. Learning content-enhanced mask transformer for domain generalized urban-scene segmentation. AAAI, 2024.
[d5] Li, Yumeng et al. Intra-Source Style Augmentation for Improved Domain Generalization. WACV, 2023.
Thank you for your time and constructive feedback. We discuss the novelty of our proposed generative-based augmentation in the general response. We address your other concerns below.

# Weakness 2: Hyperparameters for Relative Contrastive Loss

Our relative contrastive loss includes three terms, each with a margin value controlling the distance penalty limits. **These margins are set based on the average uncertainty scores from the training set.** Specifically, we compute the differences in uncertainty scores between unknown vs. original known data, unknown vs. augmented known data, original known vs. augmented known data, and set the differences as margins for these distance respectively.  A histogram of uncertainty scores is provided in the Figure 1 (c) attached PDF for reference.

Moreover, our two-stage training framework first trains the uncertainty function based on the existing model, allowing this function to adapt to different scales. **This provides flexibility in parameter setting even without prior knowledge.** The experimental results, shown in Table 1 of the attached PDF, demonstrate the model's robustness across a wide range of hyperparameter variations. Specifically, maintaining the loss values within the same order of magnitude ensures that variations in parameters do not significantly affect the results.

# Weakness 3: Two-Stage

Thanks for pointing this out. The motivation and benefits of our two-stage training design are clarified below.

- We observe that an initialized model with closed-world pre-training achieves a good feature representation while the initial uncertainty function, such as the energy score, is typically sub-optimal. Directly training the feature extractor with a sub-optimal uncertainty function risks disrupting the well-learned feature representations, which can harm known class segmentation and OOD detection (cf. Figure 4 (a)). 
- Our two-stage training approach addresses this challenge by first optimizing the uncertainty mapping head based on the current feature representations. This allows the subsequent fine-tuning of the feature extractor to be more effective and less disruptive.  Additionally, having a well-initialized uncertainty head before joint training helps minimize task competition between known class segmentation and OOD detection.

- In Figure 4(a), we empirically compare our two-stage training with single-stage training, showing that our approach (noted as 'second stage') outperforms the 'single stage' by nearly 10 points in AP. With a closer look, our first-stage training, which involves training only the uncertainty function already significantly improves the baseline model's performance (from 45% to 85% in AP). This demonstrates the large performance gap between different uncertainty functions under the same feature extractor.

# Weakness 4: Conditioned Label Mask
The term "conditioned label masks" in the context of Line 208 refers to the label masks used to generate images. As explained in Eq. (1), these masks are created by cut-and-pasting the masks of novel objects onto the original training labels.

# Weakness 5: Noise-aware training
- **Sample Selection Mechanism**: We use the 'small loss' criterion to determine whether a pixel has a clean label, a simple and widely used technique in the noisy label learning literature [b1]. Specifically, we calculate and rank the cross-entropy loss for each pixel. Pixels with smaller losses are selected for backpropagation, while those with larger losses are ignored (cf. Eq. (5-6)). This is effective because during training, a model first learns simple and clean patterns before fitting noisy data [b2]; We will include explanations and relevant literature in the paper to aid understanding. A visualization of our sample selection results is illustrated in Figure 4 (b) of the paper. 

- **Determining the Selection Ratio α:** We determine α by visualizing the selection map of a small batch of data under several choices to ensure that visibly incorrect patterns are removed. To address the reviewer's concern, we conducted additional experiments with selection ratios ranging from 0.6 to 0.9, as detailed in Figure 1 (a) of the supplementary PDF. The results show that while including too many pixels (1.0) introduces noise, and including too few (0.6) removes useful regions, the model performance is stable within a wide range (0.7 to 0.9), demonstrating the robustness of the model to this hyperparameter.

# Weakness 6: Hyperparameters & Applicability

We have demonstrated robustness to loss margins and selection ratio in the previous response, supporting the practicality of our approach for real-world applications.

[b1] Jiang, Lu, et al. Mentornet: Learning data-driven curriculum for very deep neural networks on corrupted labels. ICML, 2018.  
[b2] Arpit, Devansh, et al. A closer look at memorization in deep networks. ICML, 2017.
We thank the reviewer for the time and detailed feedback. We address the SMIYC benchmark comparison in the general response and will revise the paper to correct the noted typos. Below, we address your specific concerns.

# Weakness 2: Related Work & Novelty
**Anomaly Segmentation:** Our method falls under discriminative-based anomaly segmentation methods that utilize additional out-of-distribution (OOD) data to train models to distinguish between known and unknown data [3, 30, 21, 26]. We contribute to the construction of generative-based OOD data and training pipelines. The details of our OOD data generation are explained in the general response. Regarding training, previous works [3, 30, 26] directly train models using OOD score functions calculated on logits (e.g., MSP, Energy, Entropy). In contrast, our approach introduces a learnable uncertainty function, initially set as the standard OOD score function and optimized before fine-tuning the feature extractor for joint known class segmentation and OOD detection. This approach decouples the training process to mitigate competition and leverages the meaningful features learned from previous closed-world pretraining.

**Domain Generalization:** Existing methods for domain generalization include instance normalization or whitening [24, 6, 25] and domain-invariant feature learning through domain randomization [33, 29]. Our contribution lies in the latter category, where we propose a generative-based data augmentation technique. Unlike recent works that use generative models for feature [d1] or image randomization [d2], our technique concurrently generates data with both domain shifts and unknown objects. We believe that incorporating various distribution shifts is crucial to avoiding model biases and better handling domain and semantic shifts.

# Weakness 3: Evaluation Metrics

We would like to clarify that we used two metrics, AP and FPR@95, in the presented SMIYC results. These metrics are widely recognized as the **primary evaluation criteria** in the anomaly segmentation literature [1]. It is a common practice in this field to present only the primary metrics in the main paper due to space constraints, as exemplified in [21, d3].

We agree with the reviewer that using multiple metrics provides a more comprehensive evaluation of the method. To address the reviewer’s concern, we have included the results of our method across all five metrics below. As shown in Table A1, our method consistently performs better or on par with RPL/Mask2Anomaly across all evaluation metrics. These additional results will be included in the appendix of the revised paper.

Table A1: Comparison of our methods with two recent works RPL and Mask2Anomaly on the SMIYC benchmark.
| Exp Name | Backbone | SMIYC-RA21 |  |  |  |  | SMIYC - RO21 |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  |  | AP$\uparrow$ | FPR95$\downarrow$ | sIoU$\uparrow$ | PPV$\uparrow$ | F1$\uparrow$ | AP$\uparrow$ | FPR95$\downarrow$ | sIoU$\uparrow$ | PPV$\uparrow$ | F1$\uparrow$ |
| RPL (ICCV 23) | DeepLab v3+ | 83.49 | 11.68 | 49.76 | 29.96 | 30.16 | 85.93 | 0.58 | 52.61 | 56.65 | 56.69 |
| Ours |  | 88.1 | 8.2 | 56.2 | 34.7 | 37.8 | 90.7 | 0.3 | 48.1 | 66.7 | 58.0 |
| Mask2Anomaly (ICCV 23) | Mask2Former | 88.70 | 14.60 | 60.40 | 45.70 | 48.60 | 93.30 | 0.20 | 61.40 | 70.30 | 69.80 |
| Ours |  | 91.9 | 7.9 | 58.7 | 45.8 | 48.7 | 95.3 | 0.1 | 59.4 | 73.5 | 68.7 |

# Weakness 4: Hyperparameters
We thank the reviewer for the suggestion. We have added results ablating the two types of hyperparameters in our method: selection ratio and loss margins. THe results shown in Table 1 of the attached PDF demonstrate the robustness of our model to various hyperparameters. 

[d1] Gong, Rui, et al. Prompting diffusion representations for cross-domain semantic segmentation. BMVC (2024).  
[d2] Jia, Yuru, et al. Domain-Generalizable Semantic Segmentation with Image Diffusion Models and Stylized Semantic Control. ECCV (2024).  
[d3] Grcic et al., On Advantages of Mask-level Recognition for Outlier-aware Segmentation, CVPRW 2023.