Title: Re-coding for Uncertainties: Edge-awareness Semantic Concordance for Resilient Event-RGB Segmentation
Venue: NeurIPS 2025
Accepted: yes
Keywords: event camera

semantic segmentation

event-based segmentation

multi-modality fusion

uncertainty modeling
Abstract: Semantic segmentation has achieved great success in ideal conditions. However, when facing extreme conditions (e.g., insufficient light, fierce camera motion), most existing methods suffer from significant information loss of RGB, severely damaging segmentation results. Several researches exploit the high-speed and high-dynamic event modality as a complement, but event and RGB are naturally heterogeneous, which leads to feature-level mismatch and inferior optimization of existing multi-modality methods. Different from these researches, we delve into the edge secret of both modalities for resilient fusion and propose a novel Edge-awareness Semantic Concordance framework to unify the multi-modality heterogeneous features with latent edge cues. In this framework, we first propose Edge-awareness Latent Re-coding, which obtains uncertainty indicators while realigning event-RGB features into unified semantic space guided by re-coded distribution, and transfers event-RGB distributions into re-coded features by utilizing a pre-established edge dictionary as clues. We then propose Re-coded Consolidation and Uncertainty Optimization, which utilize re-coded edge features and uncertainty indicators to solve the heterogeneous event-RGB fusion issues under extreme conditions. We establish two synthetic and one real-world event-RGB semantic segmentation datasets for extreme scenario comparisons. Experimental results show that our method outperforms the state-of-the-art by a 2.55% mIoU on our proposed DERS-XS, and possesses superior resilience under spatial occlusion. Our code and datasets are publicly available at https://github.com/iCVTEAM/ESC.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes an \"Edge-awareness Semantic Concordance (ESC)\" framework for event-RGB semantic segmentation, specifically targeting extreme conditions like low light and fierce camera motion where traditional RGB-only methods struggle due to information loss. To facilitate reliable evaluation in extreme scenarios, the authors introduce two synthetic (DERS-XS, DSEC-Xtrm) and one real-world (DERS-XR) event-RGB semantic segmentation datasets. Experimental results demonstrate that ESC outperforms state-of-the-art methods.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. The proposed framework is well-structured with distinct, logical modules (ELR, RC, UO) that each contribute to the overall goal of resilient fusion.\n2. The paper introduces a novel approach to multi-modal fusion by leveraging semantic edges as an intermediate commonality.\n3. The experimental results clearly show improved SOTA performance.\n\nWeaknesses:\n1. Unexplained Learnable Noise Embeddings: The paper states that \"learnable noise embeddings\" are used to \"improve fitting ability and enhance learning stability\"  within RC and UO. However, it fails to provide a clear explanation of how these embeddings achieve this, or to cite prior work. There is also a lack of corresponding in ablations.\n2. Bias towards Extreme RGB Degradation: A significant portion of the evaluation appears to deliberately diminish the contribution of the RGB modality, particularly through the use of extremely low-light frames in both synthetic datasets and real-world data. While the goal is to highlight the event modality's role, such severe consistent RGB degradation might not always reflect the real-world extreme conditions. Testing on existing real-world extreme datasets like DSEC-Night (proposed by CMDA [40]) or collecting a more diverse set of truly challenging real-world scenarios would provide a more comprehensive and less-biased validation.\n3. Limited Efficacy in Non-Extreme Real-World Conditions: The proposed method appears to demonstrate its most significant advantages primarily when the RGB modality is severely degraded. For instance, on the real-world DSEC-Semantic dataset (Table 1), which represents more typical, non-extreme conditions, the performance gain over CMNeXt is only marginal (a 2.01% mIoU improvement). \n4. Limited DERS-XR: While DERS-XR is introduced as a real-world dataset for \"extreme conditions\", the qualitative examples shown in Figure 14 suggest that the RGB images are severely degraded, appearing almost completely dark. This level of degradation might deliberately diminish the RGB signal. Furthermore, with only 120 frames used for fine-tuning and 120 for testing, the dataset size is significantly small. \n5. High Computational Cost: While Table 5 shows that the proposed method has a comparable number of parameters to CMNeXt, its FLOPs are significantly higher. This substantial increase in computational cost raises questions about the efficiency of the method. It is unclear whether the performance gains observed are primarily due to the architectural innovations or simply a result of a higher computational budget. A critical aspect to assess would be if the method could maintain its superior performance using a smaller backbone to achieve similar FLOPs and inference speed as CMNeXt.\n\nQUESTIONS:\n1. The authors need to provide a clearer explanation or intuition for how the learnable noise embeddings contribute to improved fitting and stability.\n2. The use of extremely dark RGB inputs may overly emphasize the advantage of the event modality. It’s better to evaluate on more balanced extreme-condition datasets like DSEC-Night to ensure fairer assessment.\n3. The method shows higher FLOPs compared to CMNeXt. The authors need to test the approach with a lighter backbone to better isolate the impact of the proposed components from increased computational cost?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-08"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses the challenge of semantic segmentation under extreme conditions by leveraging heterogeneous Event-RGB inputs. The authors propose a novel framework called Edge-awareness Semantic Concordance (ESC), which utilizes semantic edges as a modality-bridging representation to align RGB and event features in a unified semantic space. The method consists of three main components: Edge-awareness Latent Re-coding (ELR), Re-coded Consolidation (RC), and Uncertainty Optimization (UO). Additionally, the authors construct three new benchmark datasets (DERS-XS, DERS-XR, and DSEC-Xtrm) designed to evaluate performance and robustness under degraded visual conditions. Experimental results demonstrate that the proposed method achieves state-of-the-art performance across multiple metrics and maintains superior robustness under extreme conditions like simulated occlusion and modality degradation.\n\nSTRENGTHS AND WEAKNESSES:\nPros:\n1.The framework introduces semantic edge as a unified representation to bridge RGB and event modalities. The motivation is well illustrated in Fig. 2, with visualizations of edge and event distributions showing high spatial correlation.\n2.The modular design is clear and well-motivated: ELR provides semantic consistency supervision, RC fuses refined edge features across modalities, and UO models the confidence of edge features to enable attention-based adaptive fusion. Together, they form a coherent pipeline for robust multi-modal segmentation.\n3. The experimental evaluation is comprehensive: the authors not only validate the method on the existing popular dataset DSEC-Semantic, but also construct three new and more challenging datasets. The experiments include extreme scenarios such as under-exposure and occlusion, providing a thorough validation of the method's applicability.\n\nCons:\n1.The ELR module uses a VQ-VAE-based edge dictionary to achieve modality alignment. It is unclear whether alternative encoding strategies (e.g., prototype-based clustering) have been considered. Ablation or comparison with other dictionary designs could help validate the generality of the framework.\n2.The recent work ESEG [1] focuses on event-only segmentation using explicit edge semantics. If its detection branch were replaced by RGB input, the pipeline would become highly similar to ESC. A clearer discussion of conceptual and methodological differences between the two works is warranted.\n3.In Appendix B, the DSEC-Semantic event input is built by sampling 50,000 events per sequence rather than using a fixed time window (e.g., 50ms). The rationale for this design choice is not clearly explained. A discussion of the implications and potential performance trade-offs would strengthen the methodological transparency.\n4.While the paper showcases under-exposure scenarios, ESC is proposed to handle a broader range of extreme conditions. Including examples under over-exposure or motion blur would better illustrate the full scope of robustness.\n5.The authors adopt MiT-B2 for RGB and MiT-B1 for event inputs. Combined with multiple attention operations, the overall FLOPs reach 95G, which may hinder deployment efficiency.\n[1] Zhao Y, Lyu G, Li K, et al. ESEG: Event-Based Segmentation Boosted by Explicit Edge-Semantic Guidance[C]//Proceedings of the AAAI Conference on Artificial Intelligence. 2025, 39(10): 10510-10518.\n\nQUESTIONS:\nSee the weaknesses (Cons) for detailed suggestions.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "CLARITY": 4, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-27"}

comments: SUMMARY:
This paper presents ESC that exploits semantic edges to fuse heterogeneous RGB images and event streams for robust semantic segmentation under extreme conditions. ESC first constructs a discrete edge dictionary, then performs ELR to align event- and RGB-derived edge distributions in a shared latent space. Two tailored modules—RC and UO—respectively integrate edge cues into RGB context features and adaptively weight modalities using confidence estimates, yielding resilient predictions when either modality degrades. To enable reliable evaluation, the authors use three benchmark datasets with ground-truth labels in extreme low-light or noisy scenarios.

STRENGTHS AND WEAKNESSES:
The introduction of a shared discrete edge dictionary combined with the Edge-aware Latent Re-coding mechanism is novel and well-motivated. The paper backs this design with solid empirical evidence, including an event-edge correlation analysis and ablations isolating each module’s contribution.

Three benchmark datasets further amplify its impact and should facilitate future research in multimodal segmentation.

---

The study compares ESC mainly with token- or attention-level fusion baselines. Other recent techniques such as cross-modal contrastive pretraining or adaptive modality dropping are not investigated.

The manuscript layout appears misaligned with NeurIPS guidelines: (1) Text embedded in figures is noticeably smaller than the main font, making labels hard to read. (2) Several tables spill over margins or have inconsistent column widths.



[1] SAM-Event-Adapter: Adapting Segment Anything Model for Event-RGB Semantic Segmentation

QUESTIONS:
NONE

LIMITATIONS:
NONE
TITLE: Official Review
CLARITY: 2
IMPACT: 2
ORIGINALITY: 2
DATE: 2025-07-02

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents Edge-awareness Semantic Concordance (ESC), a framework for robust event-RGB semantic segmentation under challenging conditions. ESC leverages a shared discrete edge dictionary learned via a VQ-VAE on semantic edge maps to bridge heterogeneous event and RGB features. It introduces three key modules: (1) Edge-awareness Latent Re-coding (ELR), which aligns event/RGB features into a unified edge-categorical space and produces uncertainty indicators. (2) Re-coded Consolidation (RC), which injects re-coded edge embeddings into image features and (3) Uncertainty Optimization (UO), which dynamically fuses modalities based on confidence/uncertainty maps. The authors also construct two synthetic (DERS-XS, DSEC-Xtrm) and one real-world (DERS-XR) extreme condition datasets to evaluate ESC’s performance. Experiments on these benchmarks demonstrate consistent mIoU improvements of ~2–3% over state-of-the-art fusion methods and greater resilience to spatial occlusion.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. The discrete edge dictionary learned via VQ-VAE provides an effective bridge between event and RGB feature spaces.\n2. The modular design of ELR, RC, and UO is thoughtfully structured, and the comprehensive ablation studies transparently demonstrate the value of each component.\n3. Consistent mIoU improvements (2–3 %) across both synthetic (DERS-XS, DSEC-Xtrm) and real-world (DERS-XR) benchmarks highlight the approach’s robustness under extreme conditions.\n4. The incorporation of uncertainty maps to guide dynamic fusion of event and RGB modalities enhances performance when data quality varies.\n\nWeaknesses:\n1. Dependence on a VQ-VAE–learned discrete edge dictionary adds a non-trivial pretraining step and raises questions about how well the same dictionary transfers to new domains.\n2. The combination of ELR, RC, and UO modules, while effective, increases architectural complexity and may incur higher runtime and memory overhead in practice.\n3. Although parameters and FLOPs are reported in Table 5, inference latency and throughput on real hardware are not provided, making it difficult to fully assess deployment feasibility.\n\nQUESTIONS:\n1. How well does the VQ-VAE–learned edge dictionary transfer across domains? For example, if it's trained on one dataset and appliedunchanged to another, what is the mIoU change? \n2. How sensitive are your results to key hyperparameters such as dictionary size and uncertainty threshold?\n3. Can you report end-to-end inference latency and peak memory usage for ESC on representative hardware?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}