Title: Scaling Data-Driven Probabilistic Robustness Analysis for Semantic Segmentation Neural Networks
Venue: NeurIPS 2025
Accepted: yes
Keywords: Robustness Verification

Semantic Segmentation Neural Network

Reachability Analysis
Abstract: Semantic segmentation neural networks (SSNs) are increasingly essential in high-stakes fields such as medical imaging, autonomous driving, and environmental monitoring, where robustness to input uncertainties and adversarial examples is crucial for ensuring safety and reliability. However, traditional probabilistic verification methods struggle to scale effectively with the size and depth of modern SSNs, especially when dealing with their high-dimensional, structured inputs/outputs. As the output dimension increases, these methods tend to become overly conservative, resulting in unnecessarily restrictive safety guarantees. In this work, we propose a probabilistic, data-driven verification algorithm that is architecture-agnostic and scalable, capable of handling the high-dimensional outputs of SSNs without introducing conservative and loose guarantees. We leverage efficient sampling-based reachability analysis to explore the space of possible outputs while maintaining computational feasibility. Our methodology is based on Conformal Inference (CI), which is known for its high data efficiency. However, CI tends to be overly conservative in high-dimensional spaces. To address this, in this paper, we introduce techniques to mitigate these sources of conservatism, enabling us to provide less conservative yet provable guarantees for SSNs. We validate our approach on large segmentation models applied to CamVid, OCTA-500 and Lung\_Segmentation, and Cityscapes datasets, showing that it can offer reliable safety guarantees while lowering the conservatism inherent in traditional methods. We also provide a public GitHub repository for this approach, to support reproducibility.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a three-stage pipeline that delivers pixel-level robustness certificates with finite-sample calibration for full-resolution semantic segmentation networks (SSNs). First, in the naïve conformal reachability stage, a calibration set of size m is used to construct an axis-aligned hyper-rectangle that, with user-defined risk level ϵ, contains the SSN’s logits under an input-uncertainty distribution. Second, in the surrogate + star-set inflation stage, a small ReLU surrogate model g is trained; its exact star-set reach set is computed using NNV and then expanded via a Minkowski sum with an error-bound hyper-rectangle derived through conformal inference, resulting in a significantly tighter reach set. Third, deflation-based PCA is used to address the extremely high dimensionality of SSN outputs: the top-N principal directions are learned, allowing g to operate in a low-dimensional latent space and be lifted back linearly to the original space. The final reach set is projected onto per-pixel logits to determine whether each pixel is robust, non-robust, or uncertain. Experiments on M2NIST, Lung_Segmentation, CamVid, and a CIFAR-100 classifier show that the method certifies ≥99.9% robustness value (RV) at ≥0.999 confidence, with runtimes ranging from 3 seconds to 60 minutes on a single 48 GB GPU.\n\nSTRENGTHS AND WEAKNESSES:\nQuality:\n+ S: Runtime scales to 960 × 720 outputs where deterministic tools have difficulties.\n+ S: Empirical study uses three metrics (RV, RS, RIoU) and reports per-experiment (ℓ, *m*) choices (Table 3).\n- W: The choice of the number of PCA components N is treated as a fixed hyperparameter without a principled selection method, which may affect generality and reproducibility. No ablation on PCA rank N and hyper-parameter sensitivity is unknown.\n\nClarity:\n+ S: Incremental step by step explanation naïve - surrogate - PCA.\n- W: Minor issues—Algorithm 1 lacks a stop-criterion.\n\nSignificance:\n+ S: The method significantly reduces runtime compared to pixel-wise conformal prediction, while scaling to high-dimensional outputs using PCA and star sets\n- W: Evaluation omits standard datasets Cityscapes/ADE20K used in prior CP work  and smoothing papers. At least they should appear in the discussion regarding their applicability due to their importance.\n\nOriginality:\n+ S: Novel integration of CI with star-set reachability via combining these techniques for segmentation.\n+ S: PCA-deflation surrogate training adapts sparse-PCA ideas to verification.\n- W: The individual ingredients (CI, star sets, PCA) exist; novelty stems from synergy more than new theory.\n\nQUESTIONS:\nHyper-parameter sensitivity: Please sweep calibration size *m* and PCA rank *N*; plot RV vs. runtime.  How fast does the guarantee degrade if *m* is cut 10×? How sensitive is the model to m and N?\n\nDataset coverage: Would the model also work on Cityscapes and ADE20K since those are important datasets in the field?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 2, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-25"}

comments: SUMMARY:
The paper presents a probabilistic verification method, based on conformal inference, which is scalable when used in complex semantic segmentation neural networks and lowers conservativism inherent in traditional methods. Experiments are presented with benchmark datasets such as CamVid and Lung segmentation.

STRENGTHS AND WEAKNESSES:
S
- Good theoretical paper for mitigating conservatism in large scale networks such as SSNs.
- Well written, especially section 3, where the reachability technique for robustness verification of SSNs is developed.  

W
- It would be good to see more and detailed experimental results (confidence, runtime) in other real life applications, such as the ones represented by CamVid and Lung segmentation.
- It would be good to explore a bit more the way in which the probabilistic framework could be used in safety-critical systems.

QUESTIONS:
- Please explain and discuss how the values (confidence of guarantee)  in Table 3 relate to the values shown in Figure 1.
- Explain and discuss the differences in average runtimes especially in CamVid and Lung Segmentation shown in Table 3.

LIMITATIONS:
Yes
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-02

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a new certification method to prove the robustness of semantic segmentation models against adversarial perturbations. In contrast to precise methods based on reachability analysis, the authors propose to instead compute probabilistic guarantees using conformal inference. This is inspired by the high output dimensionality of segmentation models (one classification per pixel), which makes them more expensive to certify. The approach is validated on segmentation models up to ResNet18-size for CamVid, Lung_Segmentation, and M2NIST datasets.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\n1. It has been well established that neural networks - and semantic segmentation models in particular - are vulnerable to small perturbations. Provable worst-case guarantees are therefore highly desired, which makes this work a very valuable contribution.\n\n2. To the best of my knowledge, the approach is novel in the context of semantic segmentation. While scalability and practical applicability currently seem limited (see below), I believe that the space of certified robustness methods still requires significant exploration. I therefore appreciate the novel perspective on certified robustness guarantees.\n\n3. The mathematical guarantees appear sound to me, although I am not an expert on conformal predictions and statistics, and therefore may have missed subtle problems.\n\n**Weaknesses**\n\n1. The paper lacks comparisons to prior work and baselines, both in related works and experiments. There are at least two lines of work that can certify the robustness of segmentation models: (i) based on sound reachability analysis [a], and (ii) based on randomized smoothing, which also computes probabilistic certificates [b, c, d]. Even if the proposed approach does not outperform SOTA methods, it would be important to understand the differences in the threat model, precisely what is guaranteed, and the performance in terms of certified accuracy, robust radius, and model and dataset sizes.\n\n2. The evaluation is limited in several aspects:\n - The evaluation is limited to 10/20 test data points. This is insufficient to reach statistically relevant results. Based on the runtimes in Table 3, it should be feasible to run the experiments on at least ~500 - 1000 datapoints in a few days. (2) \n - The threat model considered is non-standard for the adversarial robustness literature. Typically, prior work assumes either $\\ell_p$-norm perturbations or patch attacks. Why was this particular threat model chosen? If sparsity is a requirement, $\\ell_0$ perturbations would be the standard setting that makes the work comparable to prior work. Otherwise, there should be a discussion and justification of the particular threat model.\n\n3. The authors do not provide code for evaluation and extension of the proposed method.\n\n\n**Conclusion**\n\nIn its current form, the paper's weak evaluation and lack of comparison to prior work outweigh its significant contribution and interesting approach. If these two points are addressed, I would be willing to change my rating to acceptance, even if the results fall behind SOTA.\n\n\n[a] Tran, HD. et al. (2021). Robustness Verification of Semantic Segmentation Neural Networks Using Relaxed Reachability. In: Silva, A., Leino, K.R.M. (eds) Computer Aided Verification. CAV 2021\n[b] Scalable Certified Segmentation via Randomized Smoothing. Marc Fischer, Maximilian Baader, Martin Vechev Proceedings of the 38th International Conference on Machine Learning, 2021\n[c] GSmooth: Certified Robustness against Semantic Transformations via Generalized Randomized Smoothing\nZhongkai Hao, Chengyang Ying, Yinpeng Dong, Hang Su, Jian Song, Jun Zhu Proceedings of the 39th International Conference on Machine Learning, 2022\n[d] Adaptive Hierarchical Certification for Segmentation using Randomized Smoothing. Alaa Anani, Tobias Lorenz, Bernt Schiele, Mario Fritz Proceedings of the 41st International Conference on Machine Learning, 2024\n\nQUESTIONS:\n1. Is there any way to get results on Cityscapes or ACDC? Since you mention self-driving cars as motivation, this would complete the paper, make it relevant to practical datasets, and comparable to prior work\n\n2. Can you please compare to the related work listed above [a, b, c, d] - both conceptually and empirically?\n\n3. L53: what do you mean by \"low-conservatism\"?\n\n4. L292: In 2025, a 64x64 pixel dataset cannot really be described as \"high-dimensional\". I understand that the method has limited scalabliliy in terms of image resolution, but please be upfront about it.\n\n5. L294: Same for \"large and complex pre-trained models\" - the largest mode lis ResNet18, which in 2025 cannot be considered large or complex.\n\n6. L297: What is the reason for this particular threat model? Why is the method especially applicable in this setting?\n\nLIMITATIONS:\nThe paper does not address limitations, even though there are many. For example, guarantees are only probabilistic and could fail; it is limited to very small models and datasets; Evaluation does not compare to any baselines; the method is not applicable to the real-time requirements of some of the motivating examples (e.g., self-driving cars); etc.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses the challenge of verifying robustness in Semantic Segmentation Neural Networks (SSNs), particularly under input perturbations and adversarial attacks, where formal verification approaches are often computationally intractable due to the high-dimensional output space of SSNs. The authors propose a scalable, architecture-agnostic probabilistic verification framework that leverages conformal inference (CI) for data-driven reachability analysis with formal probabilistic guarantees.\n\nA naive baseline is introduced that applies CI to SSNs by constructing hyper-rectangular reachsets, which are provably correct but overly conservative. The authors then improve this baseline by: 1. Incorporating surrogate ReLU networks to approximate complex SSNs, 2. Using zonotopic (star-set) representations instead of axis-aligned boxes, 3. Applying a scalable Principal Component Analysis (PCA) technique (via a deflation algorithm) to handle high-dimensional output.\n\nExperiments on benchmark datasets (M2NIST, CamVid, Lung Segmentation) and models (U-Net, BiSeNet, ResNet) demonstrate the practicality of the proposed method, showing its capability to provide provable guarantees with lower conservatism and scalability to large-scale networks.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n\nThe framework is designed to scale to large segmentation models and high-dimensional output spaces, a major bottleneck in prior work on neural network verification.\n\nThe method builds on conformal inference, with sound statistical foundations. The paper offers a formal double-layer probabilistic guarantee, balancing confidence and miscoverage.\n\nThe authors start from a transparent, naive CI-based technique and systematically address its limitations via star-set reachability, surrogate modeling, and dimensionality reduction.\n\nThe use of learning-based PCA (via deflation) to address high output dimensionality in robustness analysis is innovative and effective.\n\nThe experiments target relevant questions (e.g., scalability, conservatism, runtime), are run across diverse datasets and models, and demonstrate superior scalability compared to deterministic methods like αβ-CROWN and NNV.\n\nThe formalization of the probabilistic reachsets via ⟨ϵ, ℓ, m⟩ tuples is precise and instructive, making it easier to interpret coverage and confidence.\n\nWeaknesses\n\nThe key technical building blocks (conformal inference, surrogate modeling, PCA) are individually well-established. The contribution lies in their combination and adaptation to SSNs rather than a fundamentally new technique.\n\nWhile the authors emphasize scalability, there’s no formal analysis of conservativeness versus exactness. The surrogate model approximation is only loosely justified, and the inflation method may introduce nontrivial looseness.\n\nTraining a ReLU surrogate model assumes a sufficiently accurate approximation of the original SSN. But the conditions under which the surrogate is guaranteed to succeed are not discussed, especially for very deep or complex architectures.\n\nThe method is evaluated under a synthetic \"darkening\" attack and UBAA noise models. More standard threat models (e.g., FGSM, PGD, spatial transformations) are not used, which limits generalizability.\n\nWhile pixel-level robustness is central, it's not always clear how the user should interpret “unknown” regions in practice—especially when aggregated over classes or images.\n\nQUESTIONS:\n1. You assume that a small ReLU network can accurately approximate the SSN under the input distribution. What happens when this approximation fails? Could you provide an error bound or confidence estimate for the surrogate's fidelity?\n\n2. The paper emphasizes that the proposed method is less conservative than the naive baseline, but does not quantify this. Can you provide a metric (e.g., volume ratio, false positive rate) to quantify the reduction in conservatism introduced by surrogate-based starsets over hyper-rectangles?\n\n3. You use learning-based PCA for output compression. Does this technique scale to more complex SSNs with class imbalance or highly correlated outputs (e.g., medical segmentation)? Could more sophisticated dimensionality reduction (e.g., autoencoders) offer advantages?\n\n4. These tasks also produce structured, high-dimensional outputs. What changes would be needed for your approach to work on non-semantic segmentation tasks?\n\n5. The framework relies on users tuning calibration size and miscoverage thresholds. Is there a principled way to choose these values to balance runtime and conservativeness, or are they empirically tuned?\n\nLIMITATIONS:\nYes (but partially). The paper discusses the limitations of conformal inference in high-dimensional settings and the lack of deterministic guarantees. However, some critical points are underexplored, including: 1. How approximation errors in the surrogate model affect guarantees; 2. Potential failure of the method when calibration distributions are not representative; 3. No discussion of societal impact, e.g., false security in safety-critical applications like medical imaging.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 2, "CLARITY": 4, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-03"}