Title: Leaving No OOD Instance Behind: Instance-Level OOD Fine-Tuning for Anomaly Segmentation
Venue: NeurIPS 2025
Accepted: yes
Keywords: Computer Vision

Semantic Segmentation

Out-of-distribution Detection

Outlier Exposure
Abstract: Out-of-distribution (OOD) fine-tuning has emerged as a promising approach for anomaly segmentation. Current OOD fine-tuning strategies typically employ global-level objectives, aiming to guide segmentation models to accurately predict a large number of anomaly pixels. However, these strategies often perform poorly on small anomalies. To address this issue, we propose an instance-level OOD fine-tuning framework, dubbed LNOIB (Leaving No OOD Instance Behind). We start by theoretically analyzing why global-level objectives fail to segment small anomalies. Building on this analysis, we introduce a simple yet effective instance-level objective. Moreover, we propose a feature separation objective to explicitly constrain the representations of anomalies, which are prone to be smoothed by their in-distribution (ID) surroundings. LNOIB integrates these objectives to enhance the segmentation of small anomalies and serves as a paradigm adaptable to existing OOD fine-tuning strategies, without introducing additional inference cost. Experimental results show that integrating LNOIB into various OOD fine-tuning strategies yields significant improvements, particularly in component-level results, highlighting its strength in comprehensive anomaly segmentation.


Peer reviews:
comments: SUMMARY:
The paper targets anomaly (OOD) segmentation in autonomous-driving datasets and argues that existing OOD fine-tuning methods optimize global pixel-wise objectives, thereby overlooking small OOD instances. The authors (1) provide a theoretical analysis showing that a dominant large anomaly can mask gradient signal for small ones, (2) propose adding an “instance-level objective” Lins that averages the loss over each OOD instance rather than over all pixels, and (3) introduce a feature-separation loss. The combined framework LNOIB is positioned as a plug-in for three existing fine-tuning methods (Entropy Maximization, PEBAL, Mask2Anomaly).

STRENGTHS AND WEAKNESSES:
Strengths

- Well-motivated focus on small anomalies. Simple, generic objectives that can be inserted into multiple existing pipelines.

- Extensive experiments on five datasets.

- No additional inference cost and tiny drop on ID classes.

Weaknesses

- Novelty is incremental. Instance-wise re-weighting and prototype separation are known ideas in few-shot segmentation and object re-ID; paper does not clearly situate itself w.r.t. those.

- The experiments are reproduced by authors. Why do not follow the benchmark by original papers like M2A?

- The bounds in the theoretical part ignore feature-separation loss. 

- Authors run 3 seeds but only averages are shown; no std/CI or p-values for the headline tables.

- Construction of “OOD instances” relies on copy-paste from COCO with instance labels. Can LNOIB operate when anomaly masks are unavailable?

QUESTIONS:
- How sensitive is performance to imperfect instance masks?

- Can LNOIB operate when anomaly masks are unavailable?

- Provide mean, std over the 3 runs.

LIMITATIONS:
See weaknesses and questions.
TITLE: Official Review
CLARITY: 2
IMPACT: 2
ORIGINALITY: 3
DATE: 2025-07-01

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a fine-tuning strategy for instance-level segmentation in images with small anomalies. The strategy is flexible, and the authors apply it to three segmentation approaches. Along with extensive benchmarks and experiments, the authors provide a theoretical analysis of why prior approaches fail at this task. Finally, some qualitative results are also provided.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n\n- The proposed approach has strong quantitative results across multiple datasets and against numerous baselines.\n- The approach can be applied on top of existing methods to improve segmentation.\n- The paper provides a theoretical analysis of why prior approaches may not work well.\n- The paper is well written and detailed.\n\n## Weaknesses\n\n- The qualitative results are not convincing. For example, consider Figure 4. It is unclear what is ID or OOD in these images. In row 2, the small traffic cones are small anomalies. Is the large one also an anomaly? In the top row, the LNOIB method has some yellow artifact to the right of the anomaly. Does this consistently appear in images segmented using this method? Maybe demonstrating the qualitative performance on different images from the other datasets could convince the readers that the approach practically works well.\n\n- The authors presented means of 3 random seeds as the results, but the variance or confidence bounds are not included.\n\n- Please see my questions for more weaknesses and concerns\n\n## Quality\n\nThe paper is well written and the figures supplement the text well. Overall, the paper is high quality.\n\n## Clarity\n\nThe paper is detailed and the appendix is extensive. The approach is relatively simple and seems clear when considering other approaches in this sub-field.\n\n## Significance\n\nAccording to the results presented in the paper, the approach seems to improve small anomaly segmentation. This is likely significant for many CV applications. However, some of the results seem to be minor improvements with no confidence intervals provided, so it is hard to know whether there is a statistical or practical improvement in some cases (specifically in pixel-level scenarios, component-level seems more obvious).\n\n## Originality\n\nThe fine-tuning strategy and theorems presented seem novel.\n\nQUESTIONS:\n1. Does $L_{pred}$ contain $L_{in}$ twice? It contains $L_{glob}$ which contains $L_{in}$ and $L_{ins}$ which also contains $L_{in}$. I am concerned about the relative weight of the $L_{in}$ ($\\alpha$ becomes pretty meaningless) in the optimization. Is this a problem? Did you notice any increased focused on the in distribution pixels due to this if so?\n\n2. \"the selection of these metrics and datasets aligns with official guidelines\". I think the datasets and metrics seem appropriate, but what official guidelines are you referring to? Being more explicit about this may strengthen your paper and choice of these metrics and datasets.\n\n3. Are the ablation results consistent on the other datasets? Why was the SMIYC-RA chosen for the ablation? \n\n4. Does focusing on improving segmentation on small anomalies hurt segmentation on large anomalies? The quantative results don't seem to clearly distinguish between them.\n\nLIMITATIONS:\nThe limitations are kept as a short paragraph in the appendix. Perhaps they should be moved to the main text of the paper.\n\nNo potential negative societal impacts are discussed. As the paper aims to improve image segmentation for OOD scenarios, these may be worth mentioning even if they are not unique to this paper.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper aims to improve instance level anomaly segmentation (AS) by reconsidering common loss formulations. The paper observes that the commonly used global / pixel prediction objective function de-emphasizes small anomalies. Straightforward theoretical justifications are made to support their observations. Based on this observation a novel instance wise loss is added that is averaged over the number of pixels in the object and is therefore size independent providing an emphasis on detecting each instance. Next, the paper proposes prototype feature matching to emphasize the importance of small instances. The final objective is proposed with 4 subcomponents that are all weighted with 5 hyperparameters. This construction is generalizable across specific forms of loss functions. The paper then provides experimental support for their model showing improved performance on instance and component wise metrics while maintaining and sometimes improving pixel wise metrics. An ablation is also provided to weigh the contributions of each of the proposed losses.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- The paper follows a clear logical flow of arguments and the writing is clear\n- The theoretical support provided is clear and supports the arguments made in the paper\n- Experiments support the aims set out in the construction of the loss\n\nWeaknesses:\n- There are several hyperparameters necessary for weighing each of the components of the loss. While a basic search has been provided over the set of parameters, this is an added complexity in future adoption of this multi-objective loss.\n\n*Note*: I do not closely follow the field of anomaly segmentation and so may not have identified missing reference to related anomaly segmentation methods and am not an expert in the datasets and benchmarks commonly used in this field.\n\nQUESTIONS:\n- What hyperparameter selection strategy or specific values should be considered for each of the loss weights? How likely is this to hinder adoption by the wider community?\n\nLIMITATIONS:\nThe authors address limitations in the supplemental materials.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces a novel framework called LNOIB aimed at improving the detection of out-of-distribution (OOD) instances in anomaly segmentation tasks. It addresses the limitation of existing OOD fine-tuning strategies that often fail to effectively segment small anomalies, proposing an instance-level fine-tuning approach to enhance the detection of both large and small anomalies. The framework is supported by a comprehensive theoretical analysis, which strengthens the soundness of the proposed method.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n1.The paper is technically solid, offering a well-founded theoretical analysis and comprehensive experiments that aim to address key limitations of current out-of-distribution (OOD) strategies in anomaly segmentation.\n\n2.The manuscript is clearly written and well-structured, making complex concepts easier to understand and follow.\n\nWeaknesses:\n\n1.Despite providing detailed theoretical analysis, the instance-level loss proposed in the paper is not entirely novel. Similar instance-level losses have been used in other detection tasks, which limits the originality of this contribution.\n\n2.The authors claim that their feature separation objective significantly improves the detection of small anomalies, but these claims are not well-supported by experimental results or detailed analysis. While feature separation has been shown to improve model performance in other studies (e.g., [1]), the specific impact of the feature separation objective in this work remains insufficiently demonstrated. \n\n[1]Tang J, Lu H, Xu X, et al. An incremental unified framework for small defect inspection[C]//European conference on computer vision. Cham: Springer Nature Switzerland, 2024: 307-324.\n\nQUESTIONS:\n1.Figure 4 is not sufficiently clear. Could the authors provide the ground-truth annotations for the images to facilitate a better understanding of the results? Additionally, it remains unclear whether the introduction of LNOIB leads to an increase in false positives—i.e., misclassifying in-distribution instances as out-of-distribution. More detailed analysis or supporting data is needed to address this concern.\n\n2.While the authors claim that LNOIB enhances the detection of small anomalies, the evidence presented is mainly visual. To substantiate this claim, quantitative metrics—such as performance breakdown by anomaly scale—should be provided. This would offer a more rigorous and comprehensive evaluation.\n\n3.The authors mention that solely adopting L_sem may cause the OOD prototype to align with the prototype of a specific ID category. Could the authors provide specific visual examples to illustrate this point? Additionally, could the authors provide more information about the datasets used, such as the number of ID categories (K) and the similarity between different ID representations? Providing detailed metrics on the performance drop when using L_sem and L_near alone on different datasets would further support the paper’s arguments.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-03"}

comments: SUMMARY:
The paper proposes LNOIB, an instance-level OOD fine-tuning framework for anomaly segmentation (AS). It addresses the limitation of existing global-level OOD fine-tuning methods (e.g., EM, PEBAL, M2A) that neglect small anomalies.

STRENGTHS AND WEAKNESSES:
Strengths:

1. Well-Motivated Problem: Addresses a critical gap—neglect of small anomalies—in OOD fine-tuning for AS.

2. Theoretical Foundation: Lemma 1 and Theorem 1 formally justify why global objectives fail for small anomalies.

3. Comprehensive Experiments: Evaluated on 5 benchmarks (SMIYC, Fishyscapes, Road Anomaly), showing consistent improvements in component-level (sIoU, F1*) and instance-level (iAP) metrics (Tables 1, 5).

4. Practical Design: No added inference cost; compatible with existing methods (EM, PEBAL, M2A).

Weakness:

1. The core idea (instance-level weighting) resembles techniques in long-tail learning (e.g., class-balanced loss). The feature separation objective borrows heavily from metric learning (e.g., prototype alignment). While positioned as a "paradigm," contributions feel incremental over prior OOD fine-tuning works.

2. No analysis of how individual hyperparameters (e.g., $M$, $\tau$) affect results beyond SMIYC-RA (Figure 7).

3. Pixel-level metrics (AuPRC, FPR95) show marginal gains (Table 2), suggesting improvements primarily target small anomalies at the expense of broader robustness.

4. Theorem 2 assumes idealized inter-class separability(\epsilon)—unrealistic for complex urban scenes. Proofs (Appendix A) are sound but lack empirical validation of assumptions.

QUESTIONS:
see weakness

LIMITATIONS:
NA
TITLE: Official Review
CLARITY: 2
IMPACT: 2
ORIGINALITY: 2
DATE: 2025-07-22