Re-coding for Uncertainties: Edge-awareness Semantic Concordance for Resilient Event-RGB Segmentation
Authors: Nan Bao
Yifan Zhao
Lin Zhu
Jia Li
Venue: NeurIPS 2025 poster ()
Keywords: event camera
semantic segmentation
event-based segmentation
multi-modality fusion
uncertainty modeling
Primary area: deep_learning
TL;DR: A novel multi-modality framework for resilient event-RGB semantic segmentation under extreme conditions.

Abstract: Semantic segmentation has achieved great success in ideal conditions. However, when facing extreme conditions (e.g., insufficient light, fierce camera motion), most existing methods suffer from significant information loss of RGB, severely damaging segmentation results. Several researches exploit the high-speed and high-dynamic event modality as a complement, but event and RGB are naturally heterogeneous, which leads to feature-level mismatch and inferior optimization of existing multi-modality methods. Different from these researches, we delve into the edge secret of both modalities for resilient fusion and propose a novel Edge-awareness Semantic Concordance framework to unify the multi-modality heterogeneous features with latent edge cues. In this framework, we first propose Edge-awareness Latent Re-coding, which obtains uncertainty indicators while realigning event-RGB features into unified semantic space guided by re-coded distribution, and transfers event-RGB distributions into re-coded features by utilizing a pre-established edge dictionary as clues. We then propose Re-coded Consolidation and Uncertainty Optimization, which utilize re-coded edge features and uncertainty indicators to solve the heterogeneous event-RGB fusion issues under extreme conditions. We establish two synthetic and one real-world event-RGB semantic segmentation datasets for extreme scenario comparisons. Experimental results show that our method outperforms the state-of-the-art by a 2.55% mIoU on our proposed DERS-XS, and possesses superior resilience under spatial occlusion. Our code and datasets are publicly available at https://github.com/iCVTEAM/ESC.

Re-coding for Uncertainties: Edge-awareness Semantic Concordance for Resilient Event-RGB Segmentation
Nan Bao', Yifan Zhao', Lin Zhu?, Jia Li
State Key Laboratory of Virtual Reality Technology and Systems, SCSE & QRI, Beihang University Schooo of Computer Science and Technology. Beijing Institute of Technology {nbao, zhaoyf, jiali][buaa. edu. cn, srrrhubbit.edu...
Abstract
Semantic segmentation has achieved great success in ideal conditions. However, when facing extreme conditions (e.g., insufficient light, fierce camera motion), most existing methods suffer from significant information loss of RGB, severely damaging segmentation results. Several researches exploit the high-speed and high-dynamic event modality complement, but event and RGB are naturally heterogeneous, which leads to feature-level mismatch and inferior optimization existing multi-modality methods. Different from these researches, we delve into the edge secret of both modalities for resilient fusion and propose novel Edge-awareness Semantic Concordance framework to unify the multi-modality heterogeneous features with latent edge cues. In this framework, Edge- awgeensssssssss Latent Re-coding, which obtains uncertainty indicators while realigning event-RGB features into unified semantic space guided by re-coded dis- tribution, and transfers event-RGB distributions into re-coded features by utilizing a pre-established edge dictionary clues. We then propose Re-coded Consolidation and Uncertainty Optimization, which utilize re-coded edge features and uncertainty indicators solve the heterogeneous event-RGB fusion issues under extreme conditions. We establish two synthetic and one real-world event-RGB semantic segmentation datasets for extreme scenario comparisons. Experimental results show that our method outperforms the state-of of-the--rt art by 2.55% mloU on our proposed DERS-XS. and possesses superior resilience under spatial occlusion. Our code and datasets are publicly available at https sttps://gite //github....t// CoomiiCVEEM//EE
first propose
Introduction
Being widely used in autonomous driving, medical imaging, geospatial analysis, and industrial inspection, semantic segmentation aims resolve the semantics of visual objects, assigning category labels to each pixel in the image B. When facing extreme conditions due to diversity and complexity the wild, conventional single-RGB semantic segmentation faces challenges of corrupted results, suffering from significant information loss. This has led to the exploration of leveraging information from multiple modalities for semantic segmentation [4 50 51).
We investigate the problem of leveraging event and RGB for semantic segmentation under extreme conditions, focusing on inferior optimization issues in modality imbalance and failure situations. Dynamic vision sensor [24] 6 38), commonly known as event camera, responds to brightness changes and generates events for each pixel asynchronously and independently. This unique mechanism gives it many advantages beyond conventional cameras, such as high dynamic range, high temporal resolution, low latency, and low power consumption [10]. Therefore, event data are widely used
"Correspondence should be addressed to Jia Li, Website: https://cvtean buaa. edu, cn/
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

in tasks that are hard to solve with conventional images alone, such as HDR image reconstruction
BT 35 [3]. motion deblurring 29
17 32 and low-light enhancement [6 23 59 58] As shown
in fig. image suffers from severe information loss under extreme conditions due to low signal-to-
noise ratio, while events clearly show the motion edge of vehicles. becomes feasible to complement the lost information of RGB modality by utilizing event modality.
However, existing event-RGB semantic segmentation meth- ods [50 51 46 do not con- sider the heterogeneous proper- ties of event and RGB modal- ity. Therefore, although the naive fusion strategy has achieved some improvements, is difficult to handle feature- level mismatch and inferior op- timization issues, especially in modality imbalance and fail- ure situations. To overcome the above problems of hetero- geneous event and RGB, we find semantic edge as inter- mediate commonality for both.
Image
Infarmation
Boundary GT
Garlde
Resilient Fasion
Gavile
Events
Edge Senittive
Esoo.....
Rescoded Eddge Disoiaation
Figure I: Edge-awareness Semantic Concordance for event-RGB fusion. RGB suffers from severe information loss under extreme conditions, while events are sensitive to edges in motion, complementing the lost information. Heterogenenus properties of event and RGB lead to feature-level mismatch and inferior optimization of existing methods. Our ESC framework utilizes semantic edge as an intermediate commonality for a more resilient fusion.
Existing studies [2 56 45 have proven that edge- awgeensss is beneficial for RGB segmentation. Intuitively, events highlight edges in motion, and RGB gradients reveal edge cues. Through statistics in section 3.1 we indeed find a strong correlation between events and semantic edge. Semantic edge serves as a bridge, guiding the heterogeneous event and RGB to embed into a unified semantic space. By utilizing information of semantic edge as bridge, we successfully realign the heterogeneous event and RGB into the unified semantic space to jointly optimize their edge information, while consolidating the image contextual information with semantic edge information crucial clue.
In this paper; we propose Edge-awareness Semantic Concordance (ESC), a novel learning framework for event-RGB semantic segmentation. ESC utilizes shared discrete embedding space, creating an edge dictionary containing basic semantic elements from semantic edge. We introduce Edge-awareness Latent Re-coding for discrete latent space modeling and transferring bi- directionally, namely re-coding. The re-coded edge features are utilized for information consolidation, and the re-coded edge distribution enables unified realignment through cross-entropy supervision. Uncertainty indicators are derived from modality distributions for joint optimization. Re-coded Consolidation and Uncertainty Optimization fusion. Prior work (46) assesses event- eeB segmentation using RGB tes----suoolabde DDD17 4. DSEC-Semanti 36) ) leading to potentially unreliable results. To address this, introduce synthetic DERS-XS and real-world DERS-XR XR, featuring low-light RGB, noisy events, and true-labels for extreme scenario comparisons. We further adapt DSEC-Semantic into extreme variant, DSEC-Xtre Xtrm, to mitigate direct dependence of pseudo-labels on original RGB. Experiments on above datasets show that our method achieves better performance and is more resilient under extreme conditions compared to existing multi-modality methods. To the best of our knowledge, this the first work assess model resilience via spatial occlusion evaluation without any fine-tuning.
designed achieve the above processes for resilient datasets (e.g.,
The contributions of our work are summarized as follows:
1) We propose Edge-awareness Semantic Concordance (ESC), novel multi-modaliy, framework that exploits supervision over re-coded distribution to realign heterogeneous event and RGB into unified semantic space, jointly optimizing them based on uncertainties derived from modality distributions.
2) We propose three modules, namely Edge-awareness Latent Re-coding (ELR), Re-coded Con- solidation (RC), and Uncertainty Optimization (UO). ELR re-codes features and distributions bi- directionally, while RC and UO utilize the re-coded features and uncertainties for resilient fusion.
3) We establish two synthetic and one real-world event-RGB semantic segmentation datasets for extreme scenario comparisons. Experimental results show that our method outperforms the state-of- the-art methods and possesses superior resilience under extreme conditions including occlusion.
2

---

2 Related work
2.1 Event-based semantic segmentation
Event data has recently been applied to semantic segmentation tasks. Ev-SegNet [ introduces semantic segmentation for event data by proposing a 6-channel image-like representation and applying
CNN architectures on the DDD17 dataset [4]. EvDistill [40) trains a student network on unlabeled
event data via knowledge distillation from large image-based teacher network. CMDA 443 proposes an unsupervised domain adaption segmentation framework to transfer daytime RGB knowledge to nighttime event domain. ESS (36] leverages labeled images for training on unlabeled event data through unsupervised domain adaptation. EvSegFormer [1]] introduces posterior attention module to utilize prior knowledge from event data, and HPL-ESS [8] proposes hybrid lebuli-la framework to mitigate noisy labels in unsupervised event- event-based segmentation. ESEG [55] is uni-modality event-based segmentation framework that exploits edge semantics to provide explicit guidance toward the regions of interest. ISSAFE [52) leverages events to assist segmentation under accident scenes, and HALSIE SS] features a hybrid dual-encoder scheme with SNN and ANN for efficient segmentation. Recent works demonstrate the feasibility of leveraging event data, while most works do not fully explore its unique characteristics, limiting its advantages over conventional RGB.
2.2 Event-assisted vision tasks
Event data can be utilized for assisting conventional vision tasks due to its high-speed and high- dynamic capacity. Pan et al. [29) propose an event-based double integral model restore sharp video from a single blurry frame with events. Jiang et al. [17) recover sharp videos with events based on a recurrent neural network. Shang et al. (32] utilize events for non-consecutively frames deblurring. Liang et al. [23] and Liu et al. [26) leverage event data to guide low-light video enhancement. Jiang et al. [6 propose joint framework to reconstruct clear images from underexposed frames and event streams. Shi et al. [33] utilize paired images and event streams to estimate monocular depth under night conditions. Li et al. [21] propose an event-based low-light video object segmentation framework. Qi et al. (30) introduce events into neural radiance fields for novel view sharp image rendering. Geng el al. [2] introduce events into visible and infrared fusion task. There are also several cross-modality contrastive pretraining approaches, such as Yang el al. [48], Yao ef al. [49], and Wu et al. (4), aiming to acquire informative and effective pretrained backbones for both event and RGB. Prevailing research proves the effectiveness of event-assisted tasks, while the inferior optimization issue of heterogeneous event and RGB under extreme conditions remains unexplored.
2.3 Inter-modality Fusion
Inter-modality fusion is the core issue of multi-modality tasks. How to obtain better-fused features has become an enduring research topic. Zhang et al. (50] [1] aim to achieve a general cross-modality segmentation model for arbitrary modalities, including event modality. Xie et al. [46] propose a modality recalibration and fusion module to recalibrate and then aggregate events and image features at multiple stages. Other works only focus on fusion techniques without specifying a specific task. Wang et al. [1 detect tokens with less information dynamically and substitute them with aggregated features projected from another modality. Jia et al. [14] introduces noise embeddings into proposed moder-my attention module to improve interaction between features of multi- multi-modality pixel-wise. Zhao et al. (54) utilized extra edge cues for event-RGB stereo. Several approaches have also emerged, including Kim et al. [9 and Lang el al. 20, to address the problem of incomplete modality inputs. In vision-language models, discrete representation learning with shared embedding space is becoming popular. Liu es al. (25) propose a representation learning paradigm that contains a discretized embedding space shared across two different modalities such as vidco and audio. Xia et al. 44 propose a framework that obtains mutual semantic information from different modalities by modality feature reconstruction. Zheng et al. (57) propose iterative learning paradigm for tuning large language models into multi-modality LLM. Zhou et al. [60] draw on the concept of shared latent space and first introduce it into domain adaptation vision task of nighttime optical flow estimation.
Inspired by the above works, we propose an Edge-awareness Semantic Concordance framework to model a shared discrete latent edge space and optimize events and image features into the unified se- mantic space based on the re-coded edge distribution. By edge-awareness latent re-coding, we obtain re-coded edge features and uncertainties, which are utilized for inter-modality resilient optimization.
3

---

3 Method
Naive fusion strategy fails to integrate heterogeneous event and RGB under extreme conditions. We propose an Edge-awareness Semantic Concordance framework to address this. To prove the rationality, we first analyze event edge characteristics in section 3.1 We then establish an edge dictionary as a preliminary in section[3.2 Based on this dictionary. Edge-awareness Latent Re-coding (section 3.3) transforms edge distribution and features bi-directionally, namely re-coding. Re-coded edge distribution is utilized for feature-level unified realignment through supervision. Re-coded Consolidation and Uncertainty Optimization (section [344 and section (355) utilizes re-coded edge features and uncertainties derived from modality distributions for resilient fusion. Since labels from DSEC-Semantic are unreliable, we construct three new datasets for reliable evaluation in section 3.6
3.1 Edge characteristic events
Event camera is bio-inspired sensor that triggers event sig- nals asynchronously when light intensity changes at each pixel. Specifically, as eq. (I) shows, event e = (x. t, Px.x) is triggered when pixel
USEC Semerrem
DERS XSS
DSEEC-Semane
(z. perceives a change in light intensity that reaches threshold θ in the loga- rithmic domain at time I, where Px.t means polarity of light inten- sity change in logarithm domain. Triggered events from lstars tend form an event stream {e, = (xi, to pt) Patarrr Stend
00
26 0F
24 24
Figure 2: Correlation between events and semantic edge. We randomly select 200 event sequenees with dilated boundary map from true-labeled DERS-XS and real-world DSEC-Semantic counting the ratio of edge pixels to all pixels and the ratio of events at edge pixels to all events, respectively. For both datasets, as the area of edge expands, the events ratio always greater than the boundary ratio. This exhibits strong correlation between events and semantic edge under different conditions.
+1, log(!x,) log(Ix.1-A) O
Px.t
11 log(Ix,r) - log(1x;--A)) < -
We demonstrate the correlation between events and semantic edge (i.e., segmentation boundary) through statistics in fig. 2 We randomly select 200 event sequences with dilated boundary map from true-labeled DERS-XS and real-world
counting the ratio of edge pixels to all pixels of the whole plane and the ratio of events falling on edge pixels to all events of the whole sequence. Statistical results show that as the area edge pixels expands, the events ratio is always greater than the boundary ratio for both datasets. This demonstrates that events tend to cluster at areas of semantic edge under different conditions, exhibiting a strong correlation between events and semantic edge, which supports utilization of semantic edge a bridge for heterogeneous event and RGB. Details of statistical process of event-edge correlation with more analyses can be seen in appendix
3.2 Edge dictionary as intermediate semantic clues across modalities
Quantised Indices
(Recosstructin Loss with GD) Reconstmeted Semanii Edge
Semantic Mick GT
Re coded Edge Distrbuttonn
Edge Dictientyy
Nearcat
Voctor Quantisation
Semantic Edge GT
Look-up
Emdhedding Comminment
Edge Embollinngs
Quantised Emisalding
Gradtest Straight-Through
Figure 3: Establishment of edge dictionary. We establish our edge dictionary based on a VQ-VAE architecture. Semantic edge is retrieved from the semantic mask ground truth and leveraged for learning its discrete latent representations as an edge dictionary, which serves as intermediate clues across heterogencous event and RGB,

---

Image Kezs
Edgrasnttrren Latour Recsdiep
Sermanic
Edee
Dieetto
Re- Re-codit
Evara Adya
Imag: Poatee
Reccadid
Edy Consslidare
Cluss
Rocnlcc Eaige Ivstance
Brregg Fdlgo
Image Fdge Dirribution
Opttoiiaatio
Resended LAge Usinbuutce
Senvrce Mesk Predie
Fretnic Eidge
Evezda Edge Diseirttinn
lone Optiiized Femoe
Eledfie
Noddlt
Arrmn Only Training
Figure 4: The overall architecture of our Edge-awareness Semantic Concordance (ESC). ESC contains pre-established edge dictionary and three key modules, namely awareness Latent Re-coding (ELR). Re-coded Consolidation (RC), and Uncertainty Optimization (UO). Based on the pre-trained edge dictionary, ELR transfers edge embeddings into re-coded distribution and modality distribution into re-coded features. RC consolidates edge information with re-coded features. UO jointly optimizes modality edge features with uncertainties. Features from RC and UO are concatenated for final semantic mask prediction.
To utilize semantic edge as intermediate clues, we establish an edge dictionary, which is a discrete latent embedding space derived from semantic edge, containing basic semantic elements of edge and shared by heterogeneous event and RGB. The establishment of our edge dictionary is shown in fig. and based on a VQ-VAE [39 architecture, which is originally used for representation learning and to model shared discrete latent space from inputs.
used by 25 28
We first retrieve semantic edge from semantic mask ground-truth by a mean filter followed by an indicator function. Given semantic mask M {1,..., ,c] H XW boundary map B {0, HxW can be obtained by B = MM (M), where e is number of categories in semantic mask, and is indicator function.
We define edge dictionary as A = [(k, o(k)) E {1, K}), where K is the number of items (i.e. quantised vectors) that edge dictionary contains, and v(-) is the query function as v(k) E R" selects the k-th item of edge dictionary with n n-dimension. As fig. 3.g.3sshh showws the tokenizer fr takes boundary map B as input, producing edge embeddings r fr (B) )))= E R xW' xn, which have downsampled spatial size H' x W and channels. Items in edge dictionary look-up method as r e(k) v(arg (L..... "ww | (k) where K contains the queried indices and I' is the quantised edge embeddings. Reconstructed boundary map B' is obtained by B'
selected by nearest neighbour
(T'), where fr is the detokenizer.
To ensure the edge dictionary contains all basic information of semantic edge, need ensure the boundary map is reconstructed flawlessly while items in edge dictionary are close enough to edge embeddings. Thus, we adopt the training objective with reconstruction loss, embedding loss, and commitment loss as L [ B' + [|e((K) II)
|gg( (K) r where sg(:) means stop gradient, and a is a constant of commitment loss weight. To make the reconstruction loss
propagate back to tokenizer, a gradient traight-through technique adopted, which directly assigns
the gradient from I' to P. Details of edge dictionary training process can be seen in appendix D
3.3 Cross-modality realignment of edge representations via latent re-coding
Re-coding is a key process in our framework, realigning edge representations heterogeneous event and RGB through re-coded distribution, while also producing re-coded edge features for consolidation. Based on the pre-established edge dictionary, our proposed Edge Edge-awareness Latent Re-coding module transfers edge embeddings into re-coded edge categorical prior distribution and modality posterior distribution into re-code re-coded edge features. This section will discuss the latent re-coding operation of two directions mentioned above, and introduce the optimization objective at the end of this section.
Re-coding for edge categorical prior distribution. Given the pre-trained tokenizer fr and the pre-established edge dictionary A, we can re-code any semantic edge B to edge categorical prior
5

---

distribution g(C]B) as one-hot as follows:
(K B)
arg min
[l/r(B) - -((k)ll
(2)
(E(b))RE kK))
where b is the K-dim basis vector with 1 at k-th place.
Re-coding for edge features. We first extract features from inputs. Given image I Rxxx3 and its corresponding event voxel grid [6]] & E HXXW x multi-scaled image feature and events feature be obtained from F - fI( F (E(E), where B is number of voxel grid bins, F and F denotes the backbones. Image edge features are resolved as E = f(x(F²), where fn is an decouple module adopted from [22] as our edge resolver. As event data naturally highlights the edge information, we keep event features directly as events edge features E² = F without additional processing. Then two edge encoders with the same structure are applied respectively to both E and E in order to encode modality edge features into the same unified semantic space. Two MLP-based classification heads are attached after edge encoders for each modality to predict its modality-speciife edge categorical probability distribution p(K]]Z) and p)K]E)). This categorical probability distribution indicates the probability of edge dictionary item number K Kaary classification at each spatial position.
Given probability distributions p( and p(K[E)), we can retrieve image key map K and events key map Ke by
K M arg max p(K = k|M), (EEL,
kE(L...,KK
where K-M E {I,.... [1,....,KH''xI KJ''XW" means the key map of modality M, which can either be image modality I events modality E. By this step, we select the indices of the maximum probability values at each position of the latent space as key map, which can be utilized to query edge dictionary for obtaining re-coded edge features of the specific modality. The image re-coded edge feature r and events re-coded edge feature Te are obtained by
rM = mM o(KM), (I,E,
(4)
where I-M E H xW'>n xn are modality-specific edge embeddings queried by the specific key map.
How to optimize ELR and what are the benefits? We optimize our Edge-awareness Latent Re- coding module by an objective function based on cross-entropy, which narrows the gap between the edge categorical distribution (B with modality edge categorical probability dis- Σ (KC B) (P(pp))))))(X)))E a summation of two bridge the modality gap and realign
tribution p(C]IZ) and p(K[E) as Ledge
cross-entropies. By minimizing this objective function, the image edge feature E with events edge feature E into the same unified semantic space, and make sure the re-coded features IM represent the edge information of events and image correctly.
3.4 Re-coded features for edge information consolidation
Image feature mainly focuses on con- textual information and lack of un-
Re-coded Cunsolidation
Uncertainty Optimization
Add Muhiply
derstanding of edge information. Re-
A
Image Feature
Image
coded features are utilized for edge information consolidation by the Re- coded Consolidation (RC) module. As shown in fig. fig.5oon on n the left, RC takes image feature F", image and events re-coded edge feature I and I as input, outputs a refined feature namely Edge Consolidated Feature +.
Juint
coded Features
Uptimmed
Franur
In RC, we define two learnable noise embeddings Nr E RR and Nv Rn to improve fitting ability and en- hance learning stability. For image feature F and re-coded edge features
Consdiaaual
Foerrs Linrerentics
Exernt Uontsirmoos
Figure 5: RC and UO. The two modules utilize an autention-basor structure with learnable noise embeddings for a resilient fusion.

---

I and T RC applies multi-hear attention operation on vectors at spatial position (h, w), calculates and outputs the consolidated features
as
4 h, = (0i, 02, ... Pmm] Wo +
F
(5)
Φ Softmax (q, d V,,
N.w I.W. The introduction of noise embedding is inspired by [14] and we develop the following theoretical explanation. In the absence of noise, the query (Q2) tends to attend excessively to its own features in the key (K;), thereby suppressing signals from the other source and impeding effective fusion. Introducing noise embeddings mitigates this issue by perturbing the attention space in controlled, learnable manner, encouraging richer and more balanced cross-modality interactions. The main idea of eq. (5) consolidate image feature F with image re-coded edge feature I and events re-coded edge feature I by querying T and I with F" to obtain attention map. The map represents the relevance between F and r", r which decides the amount of edge information consolidated by F.. Refined vectors hi,w constitute together in accordance with their positions as + R Ha Waxin
where Qi - F .... [F +N k. 1% h. 1 b.u. WK,, =F I how I
3.5 Edge-aware uncertainties for joint optimization
Probability values in edge distribution indicate the confident and uncertain areas of image and events. We leverage this confidence and uncertainty information from edge distribution of image and events for resilient fusion by the Uncertainty Optimization (UO) module.
Given edge categorical probability distributions p(K[Z) and p(K[E), we can retrieve confidences and uncertainties of image and events by
p(K k\M), CM max kE(I,-.KK)
(6)
uM -MM ME{I,E)
where C.M, 2 E [0, x|'' denote the confidences and uncertainties of modality M, which can either be image modality I or events modality E.
Confidences and Uncertainties represent spatial reliability of specific modality, which are utilized as indicators in UO. As shown in fig. 5 on the right, UO takes image edge feature E and events edge feature E. input, confidences and uncertainties as indicators, and outputs refined feature namely Joint Optimized Feature V
In UO, we define four learnable noise embeddings N N No N R" enhance fitting capa- bility and improve learning robustness. For feature Ex and E their confidences and uncertainties are CZ 122 and C 21 respectively. UO applies multi-head attention operation on vectors at spatial position (h, w), calculates and outputs the optimized feature V as
Ch.w V
L h,w
k,w
Ch.w
h,w
+
C
hww
M. hw = WMM
MM WoM + E
S.M Softmax (Q, (KMMMK)) / /dd))) V.M
V.M = EE, + N-M where Q.M = E Q K.M = EM, + N K 2, EM W.M. R.
EM W.M. (M,M) E ((I,E) E), (E, The main idea of eq. 7 is to optimize image edge feature E and events edge feature E. based their confidences each spatial position. Multiplied by the uncertainty value 2 moodllity-spectti map represents the self-uncertainty of E and E which decides the amount of complementary information absorbed from the counter modality. The final feature vector is calculated by normal- ized confidence weighted summation of modallieass feature vectors. Optimized vectors L ,,,, constitute together in accordance with their positions V R MaxWaxn
feature exposes its uncertainty to attention map. The
How to optimize our method? Edge consolidate feature and joint optimized feature V are concatenated and input into an MLP--ased classification head for semantic mask prediction. Cross- entropy is utilized for the supervision of semantic mask prediction as Iipred- The final optimization objective function for method is L Lpred + 3- Ledge, where 3 is constant of edge loss weight.
7

---

Simulater 270 Prone Segueneer
Sansiater Low Light Fronnes
I Technique
VZE
CARLA
Siewinnes Autgy Even
Segtemiatio
DERS-XS
(b) DERS-XR
[c) DSFC-Semnnic to DSFC-XXtm
Add Nove
Monsal
Patrnd Rlvent and APS Fnotors
Antosanton
Figure 6: Construction of datasets DERS-X XS, DERS- XR, and DSEC-Xtr for reliable evaluation.
3.6 Constructing datasets for reliable evaluation of event-RGB segmentation
Labels from USEC-Semantic pscudo-labels directly derived from RGB via [37]. They useful for event-only tasks, but not reliable for evaluating event-RGB tasks. Using these labels as ground truth implicitly presupposes that the optimal result of event-RGB segmentation is obtained by RGB-only model, which undermines the unique advantages events. To address the problem, we construct datasets below for reliable evaluation. The construction pipelines of datasets DERS XS, DERS-XR, and DSEC-Xtrm are shown in fig. 6 Details of the datasets can be seen in appendix [
DERS-XS. Dataset of Event-RGB semantic Segmentation under eXtreme conditions Synthetic, abbreviated as DERS-XS, a true-labeled synthetic event-RGB extreme condition semantic segmen- tation dataset. CARLA [9] provides true segmentation labels, and we first use CARLA simulator to simulate 270 frame sequences with segmentation labels of 23 categories, each with 1200 frames, and the size of each frame is 640 360. We then simulate noisy events from the frame sequences using v2e simulator [3] with shot noise parameter of 5.0 Hz. Low-light frames are simulated by attenuating optical signals and adding shot noise in the RAW domain obtained from the ISP unprocessing technique in 77]. Because differences between adjacent frames are small, which is not conducive data diversity, sample data at intervals of 100 frames while discarding other frames. We divide 168 sequences as training set, 12 sequences as validation set, and 90 sequences as test set.
DERS-XR. Dataset of Event-RGB semantic Segmentation under eXtreme conditions Real-world, abbreviated as DERS-XR, is a manually annotated real-world event-RGB extreme condition semantic segmentation dataset. We use DAVIS346 [38] to capture paired APS frames and events under extreme lighting conditions, and manually annotate a subset of 240 frames. Of these, 120 frames are randomly selected for fine-tuning, while the remaining 120 frames are used for testing.
DSEC-Xtrm. DSEC-Xtrm is an extreme condition semantic segmentation dataset synthesized based on DSEC-Semantis [3]. To make use of pseudo-labels from DSEC-Semantic while mitigating their direct dependence on RGB, we apply degradation to the RGB frames. We apply the same low-light image simulation method as DERS-XS to frames and use v2e simulator [13] to generate pure shot noise and add it to events. The degraded frames and events together constitute DSEC-Xtme
4 Experiments
4.1 Implement details
The code implemented by PyTorch. We first train edge dictionary as a separate stage to obtain pre-trained weights of tokenizer and edge dictionary. We utilize pre-trained MiT-B2 backbone and MiT-BI backbone of SegFormer [47] for RGB modality and event modality, respectively. The number of categories is 11. We set the number of items K in edge dictionary as 128, and the dimension of edge embeddings n as 256. The weight of edge dictionary commitment loss a is 0.25, and the weight of edge loss 3 is 0.1. The bins of event voxel grid B is 5. For training, we randomly apply color jitter, horizontal flipping, and gaussian blur to images and randomly resize with scales from 0.5 to 2.0 to images and events and crop the inputs to 256 x 256. For testing, we follow the setting of CMX (50) and CMNeXt (5I), which upsamples the inputs to width and height both divisible by 32 (will be ablated in table More detailed information training settings can be seen in appendix B

---

Table 1: Comparisons DERS-XS, DERS-XR, DSEC- DSEC-Semantic, and DSEC-Xirm.
DERS-XS
DERS-XR
DSEC-Semantic
DSEC-XXtm
Methods
Modality [) [])
mACC(%) mloU(%) MACC(R)] mloU(%) mACC(%) (1]T
SegFormer 47
RGB
62.45
53.21
55.37
51.09
72.91
65.03
41.74
33.88
SegFomeer (E)
47
Event
47.85
37.32
42.03
36.96
47.38
38.59
48.52
37.72
EvSegFormer
15
Event
41.48
31.85
38.60
33.66
44.72
37.13
42.33
34,68
TokenFusion
41
E-RGB
64.88
56.22
54.19
47.72
74.60
67.39
53.04
43.41
CMX (50
E-RGB
71.86
63.12
64.51
59.22
76.18
68.10
51.29
43.95
CMNeXt
E-RGB 73.30
64.55
66.57
60.95
77.50
69.03
52.12
45.16
EISNet 46
E-RGB 69.10
60.68
66.18
61.81
71.60
64.67
56.77
48.76
Ours
E-RGB
75.26
67.10
70.75
65.22
78.61
71.04
59.45
50.87
Reimplemented on DSFC-Semantic using the same dataloader for fair comparison with different training settings from [4fi
including each sequence events count 50000 (HG of 100000), input cropping size of 256 256 (HG 448 x 448). total hatch size of 8), different random resize stratogy, etc.
Events
SegFormer
Fo BoSoyFormer TekeaFusion
CMX
CMNeXI
HISNet
Groundrruhh
F Xe DERS-YS
Dances
Figure Qualitative comparison on DERS-XS, DERS- XR, Semantee,niic, and DSEC-Xtrm.
4.2 Comparisons with state-of-the-art
We compare our model with the current state-of-the-at including RGB-only, event-only, and event- RGB-based methods. For RGB-only, we reimplement SegFormer [47] with MiT-B2 backbone settings, which is a powerful RGB-based semantic segmentation architecture. For event-only, reimplement EvSegFormer 15 and modify the number of input channels of SegFormer to adapt to events input as an events-only comparison method. For event- RGB----Bdd, we reimplement TokenFusion [41, CMX (50, CMNeXt [51], and EISNet [43] For verssoversion SegFormer and EvSegFormer, we follow the setting EvSegFormer and apply 6-channel image [ as their event representation. For TokenFusion, CMX, CM and EISNet, we use 3-bin voxel grid [6]] their event representation. We retrain all methods with the same training settings for fair comparison.
Comparisons on true-labeled synthetic dataset. As shown in table [ our method surpasses all uni-modality and multi-modality methods and outperforms CMNeXt by a 2.55% mloU DERS-XS. As shown in fig. 7h, our method is more stable and robust to the edges of the segmentation results, especially for moving vehicles and pedestrians. This shows that our model effectively leverages edge information from events to compensate for the information loss of RGB under extreme conditions.
Comparisons true-labeled real-world dataset fine-tuning. As shown in table I our meth- ods outperforms EISNet by a 3.41 % mloU on DERS-XR fine-tuning experiments. Results also demonstrate that models trained synthetic DERS-XS be efficiently adapted to real-world data with minimal fine-tuning, further validating the effectiveness DERS-X- -SX. As shown in fig. b our method successfully segments the vehicles, while other methods fail under real- world extreme scenes.
Comparisons on non-extreme DSEC-Semantic. As shown in table I our method outperforms CMNeXt by 2.01% % mIoU on DSEC-Semantee demonstrating its elfectiveness available dataset under real-world, non-extreme conditions, despite the inherent limitations of pseudo- labels. EISNet performs slightly worse, possibly due its sensitivity the input cropping strategy.
publicly
shown in table our method ourperforms EISNet by a 2.1 11 mloU on DSEC-Xtrm. Results also show that our method suffers less performance degradation with degraded inputs. shown in fig. 7e. our method preserves more complete boundaries for vehicles and pedestrians, demonstrating its robustness and resilience.
Comparisons under extreme conditions on degraded DSEC-Xtrm.

---

4.3 Ablation studies and analyses
Resilience study under severe spatial occlu- sion. This study emulates visual degradation due spatial information loss under extreme conditions by applying local masking to the in- puts. masking applied, we mask 100 x 100 area starting at coordinates (350, 200) for RGB and (150, 150) for event. As shown in table
Table 2: Study on spatial occlusion on DERS DERS-XS.
Methods/ /
Apply masking on
None
RGB
Event
E-RGB
TokenFusion 41
36.22
48.44
55.79
48.00
(50)
54.13
62.70
53.73
CMNeXt 5
64.55
54.15
64.07
53.70
EISNet 46
60.68
55.33
59.87
54.47
Ours
67.10
64.34
66.65
63.87
our method suffers less performance degrada-
Table 3: Ablation study architecture DERS-XS.
tion under different settings. As shown in fig. 8
Arch. /ttli/mlo(()))
MParams(M)
wlo masking
E-RGB masking
under E-RGB masking. CMX and CMNeXt fail
ESC
56.875
67.10
63.87
-lo Upsampling
56.875
64.59
61.46
in understanding the semantics the mask area,
wlo RC
56.612
64.29 (4031)
59.53 (-199))
while our method overcomes the problem by
weo UO
56.084
62.53 (-2.06)
58.43 (-3.03)
edge-aware optimization with uncertainty indi-
wlo FLR vodee
38.411
61.35 (-3.24)
56.34 (-5.12)
cators. This demonstrates that method is more resilient than other methods under severe spatial occlusion. Extended experiments with more results can be seen in appendix F and G
Table 4: Ablation study key usage
DERS-XS -XS.
K
16
32
64
128
256
512
K- K-Usage'
16
32
92
97
gACC(%))
92.66
93.12
93.03
93.27
93.19
92.93
mACC(())
74.59
74.77
74.90
75.26
73.91
73.65
Ablation study on ESC architecture. As shown table 3 we ablate our ESC by re- moving modules under no mask setting and E- RGB mask setting on DERS-XS XS. Upsampling, form of data augmentation, is first ablated. Results show that each proposed module con- tributes positively to performance and resilience.
mloU(S()
63.87
66.84
66.64
67.10
66.54
66.11
; K-Usage is the number of dictionary keys used.
Table 5: Ablation study on noise embeddings removal.
Arch. Trch./mlott /mloU DSEC--XS DSEC-Semantic DSE:C-Xtrm
ESC w/o Nx. Ny
66.05
70.86
50.05
ESC w/ Nv.
67.10
71.04
50.87
Table 6: Comp. model complexity DERS-XS.
Ablation study on different K of edge dictio-
Methods
Backbone
#larams(MM
FLONGG)
mloUret
nary. As shown in table when K' is too small,
SegFormer (4]
MrT-62
24,725
25.279
53.21
Seg
the items are insufficient for edge representation; when K is too large, the excess items are under- utilized, leading to confusion in model learning. Both result in performance degradation; thus, we set K = 128 to achieve an optimal trade-off.
EoSegFormerr
[5]
MrT-82
24.740
25.422
31,85
Tokeal
MDT-B2
26,011
34.345
56.22
CMX
2 MiT-B2
65.551
CMNeX1
x MTT-B2
58.687
62.005
64.55
EISNet 16
NET-BO-2 - B2
34,367
67.304
60.68
Ours
MiT-BI + B2
56,875
95.096
67.10
Ablation study on the removal of noise embeddings. As shown in table 5 the removal of noise embeddings leads to a 1.05% and 0.81% mloU drop on DERS- -XR and DSEC-Xtrm, respectively, confirming their contribution to improved fitting stability. The performance drop on DSEC-Semantic minimal (0. 18%), which we attribute to its reliance on the RGB-based pseudo-labels. As the supervision signal has a bias towards RGB, the model naturally relies less event modality. In such cases, the role of noise embeddings in facilitating cross- modality interaction becomes less significant.
Model complexity. Table 66 summarizes the complexity of compared models, where FLOPs are calculated with inputs of 512 x 512. Results show that our method has fewer parameters than CMX and CMNeXt, yet achieves better perfor- mance. The FLOPS of our method are relatively large, primarily due to multiple MLP heads for re-coding and resilient fusion in our framework.
Image
Events
Image Ldde Distr
Everrs Edge Distr
CMX
CMN-Xi
Chrs
GimundTaath
Figure 8: Qualitative study under spatial occlusion.
Conclusions and limitations
Conclusions. In this paper, we propose Edge-awareness Semantic Concordance, multi monlii-mddality framework for event-RGB semantic segmentation. We demonstrate its capability and robustness for handling heterogeneous event and RGB. Results show that our framework outperforms existing event-RGB segmentation methods and possesses superior resilience the case of modality imbalance and failure under extreme conditions. Limitations. Despite the promising results, only the fusion of event and RGB considered so far. Exploring interactions with other visual modalities and designing modules tailored to their specific characteristies continues to be open direction for future research.
10

---

Acknowledgements
This work partially supported by grants from the National Natural Science Foundation China under contracts No. 62132002 and No. 62202010, the Beijing Nova Program (No.20250484786) and the Fundamental Research Funds for the Central Universities.
References
[1] Inigo Alonso and Ana C Murillo. Ey-segnet: Semantic segmentation for event-based cameras. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition Workshops, pages 0-0, 2019.
[2] Bryce Bayer. Color imaging array. United States Patent, no. 3971065, 1976.
[3] Yoshua Bengio, Nicholas Léonard, and Aaron Courville. Estimating propagating gradients through stochastic neurons for conditional computation. arXiv preprint 1oXiv:1308.3422 2013.
[4] Jonathan Binas, Daniel Neil, Shih-Chii Liu, and Tobi Delbruck. Ddd17: End-to tnnd davis driving dataset. arXiv preprint arXiv: /71 1.01110 2017.
[5] Shristi Das Biswas, Adarsh Kosta, Chamika Liyanagedera, Marco Apolinario, and Kaushik Roy. Halsic: Hybrid approach to learning segmentation by simultaneously exploiting image and event modalities. In 2024 JEEE/CVF Winter Conference Applications of Computer Vision (WACV), pages 5952-5962. IEEE, 2024.
[6] Christian Brandli, Raphael Berner, Minhao Yang. Shih-Chii Liu, and Tobi Delbruck. A 240 x 180 130 db latency global shutter spatiotemporal vision sensor. IEEE Journal of Solid-State Circuits, 49(1)):2333 2341, 2014.
Tim Brooks, Ben Mildenhall, Tianfan Xue, Jiawen Chen, Dillon Sharlet, and Jonathan T Barron. Unpro- cossing images for learned raw denoising. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 1 1036-11045, 2019.
[8] Jiaqi Jhen, Jiachen Lu, Xiatian Zhu, and Li Zhang. Generative semantic segmentation. In Proceedings of the JEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 7111-7120, 2023.
[9] Alexey Dosovitskiy, German Ros, Felipe Codevilla, Antonio Lopez, and Vladlen Koltun. CARLA: open urban driving simulator. In Proceedings of the 1st Annual Conference on Robot Learning, pages
1-16, 2017.
[10] Guillermo Gallego, Tobi Delbrück, Garrick Orchard, Chiara Bartolozzi, Brian Taba, Andrea Censi, Stefan Leutenegger, Andrew J Davison, Jörg Conradt, Kostas Danillidis, et al. Event-based vision: survey, IEEE transactions on pattern analysis and machine intelligence, 44(1):154-18) 2020.
[11] Mathias Gehrig, Willem Aarents, Daniel Gehrig, and Davide Scaramuzza. Dsec: A stereo event camera dataset for driving scenarios. IEEE Robotics and Automation Letters, 6(3):4947-4954, 2021.
[12] Mengyue Geng. Lin Zhu, Lizhi Wang, Wei Zhang, Ruiqin Xiong, and Yonghong Tian. Event-based visible and infrared fusion via multi-task collaboration. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 26929-26939. 2024.
[13] Yuhuang Hu, Shih-Chii Liu, and Tobi Delbruck. v2c: From video frames to realistic dvs events. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 3312-322 2021.
[14] Ding Jia, Jianyuan Guo, Kai Han, Han Wu, Chan Zhang, Chang Xu, and Xinghao Chen. Geminifusion: Efficient pixel-wise multimodal fusion for vision transformer. In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024. OpenReview.nek, 2024.
[15] Zexi Jia, Kaichao You, Weihua He, Yang Tian, Yongxiang Feng. Yaoyuan Wang, Xu Jia, Yihang Lou, Jingyi Zhang, Guoqi Li, et Event-based semantic segmentation with posterior attention. IEEE Transactions on Image Processing, 32: 1829 1842, 2023.
[16] Yu Jiang, Yuehang Wang. Siqi Li. Yongji Zhang, Minghao Zhao, and Yue Gao. Event-based low- illumination image enhancement. IEEE Transactions Multimedia, 2023.

---

[17] Zhe Jiang. Yu Zhang, Dongqing Zou, Jimmy Ren, Jiancheng Lv, and Yebin Liu. Learning event-based mo- tion deblurring. In Proceedings the IEEE/CVF Conference Computer Vision and Pattern Recognition. pages 3320-3329, 2020.
[18] Linglin Jing. Yiming Ding, Yunpeng Gao, Zhigang Wang. Xu Yan, Dong Wang, Gerald Schaefer, Hui Fang. Bin Zhao, and Xuelong Li. Hpl-ess: Hybrid leeudo-labeling for unsupervised event- based semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 23772 2024.
[19] Donggeun Kim and Taesup Kim. Missing modality prediction for unpaired multimodal learning via joint embedding of unimodal models. In European Conference on Computer Vision, pages 171-187. Springer, 2024.
[20] Jian Lang. Zhangtao Cheng, Ting Zhong, and Fan Zhou. Retrieval Retrieval-augmentedd dynamic prompt tuning for incomplete multimodal learning. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 18035-18043, 2025.
[21] Hebei Li, Jin Wang, Jiahui Yuan, Yue Li, Wenming Weng, Yansong Peng, Yueyi Zhang, Zhiwei Xiong, and Xiaoyan Sun. Event-assisted low-light video object segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3250-3259, 2024.
[22] Xiangtai Li, Xia Li, Li 7hang, Guangliang Cheng, Jianping Shi, Zhouchen Lin, Shaohua Tan, and Yunhai Tong. Improving semantic segmentation via decoupled hody and edge supervision. In Computer Vision- ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XVII 16, pages 435 Springer, 2020.
[23] Jinxiu Liang, Yixin Yang. Boyu Li, Peiqi Duan, Yong Xu, and Boxin Shi. Coherent event guided low-light video enhancement. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 10615-10625, 2023.
[24] Patrick Lichtsteiner, Christoph Posch, and Tohi Delbruck, A 128 x 128 120 db 15 jus latency asynchronous temporal contrast vision sensor. IEEE journal of solid-state circuits, 43(2):566-576, 2008
[25] Alex Liu, YouYoung Jin, Cheng-I Lai, Andrew Rouditchenko, Aude Oliva, and James Glass. Cross- modal discrete representation learning. In Proceedings the 60th Annual Meeting of the Association for Computational Linguistics ( (Volume 1: Long Pupers), pages 3013-3035, 2022.
[26] Lin Liu, Junfeng An, Jianzhuang Liu, Shankin Yuan, Xiangyu Chen, Wengang Zhou, Houqiang Li, Yan Feng Wang, and Qi Tian. Low-light video enhancement with synthetic event guidance. In Proceedings of the AAAI Conference Artificial Intelligence, pages 1700, 2023.
[27] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In 7th International Conference Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019.
[28] Jia Ning, Chen Li, Zheng Zhang, Chunyu Wang, Zigang Geng, Qi Dai, Kun He, and Han Hu. All in tokens: Unifying output space of visual tasks via soft token. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 19900-19910, 2023.
[29] Liyuan Pan, Cedric Scheerlinck, Xin Yu, Richard Hartley, Miaomiao Liu, and Yuchao Dai. Bringing a blurry frame alive at high frame-rate with an event camera. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 6820 6829, 2019.
[30] Yunshan Qi. Lin Zhu, Yu Zhang, and Jia Li. E2nerf: Event enhanced neural radiance fields from blurry images. Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 13254- 13264, 2023.
[31] Henri Rebecq, René Ranfil, Vladlen Koltun, and Davide Scaramuzza. High speed and high dynamic range video with an event camera. IEEE transactions on pattern analysis and machine intelligence. 43(6):1964 1980, 2019.
[32] Wei Shang, Dongwei Ren, Dongging Zou, Jimmy S Ren, Ping Luo, and Wangmeng Zuo. Bringing events into video deblurring with non-consecutively blurry frames. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4551- 2021.
[33] Peilun Shi, Jiachuan Peng, Jianing Qiu, Xinwei Ju, Frank Po Wen Lo, and Benny Lo. Even: An event- based framework for monocular depth estimation at adverse night conditions. In 2023 IEEE International Conference on Robotics and Biomimetics (ROBIO), pages 1-7. IEEE, 2023.
12

---

[34] Leslie N Smith. Cyclical learning rates for training neural networks. In 2017 IEEE winter conference on applications of computer vixion (WACV), pages 464-472. IEEE, 2017.
[35] Timo Stoffregen. Cedric Scheerlinck, Davide Scaramuzza, Tom Drummond, Nick Barnes, Lindsay Klee- man, and Robert Mahony, Reducing the sim-to-real gap for event cameras. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XXVII 16, pages 534-549. Springer, 2020.
[36] Zhaoning Sun, Nico Messikommer, Daniel Gehrig, and Davide Scaramuzza. Ess: Learning event-based semantic segmentation from still images. In Eumpean Conference on Computer Vixion, pages 341-357. Springer, 2022.
[37] Andrew Tao, Karan Sapra, and Bryan Catanzaro. Hierarchical multi-scale attention for semantic segmenta- tion. arXiv preprint arXiv:2005. 10821, 2020.
[38] Gemma Taverni. Diederik Paul Moeys, Chenghan Li, Celso Cavaco, Vasyl Motsnyi, David San Segundo Bello, and Tobi Delbruck. Front and back illuminated dynamic and active pixel vision sensors comparison. IEEE Transactions on Circuits and Systems II: Express Briefs, 65(5)677-6811 2018.
[39] Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning, Advances in neural information proeesing systems, 30, 2017.
[40] Lin Wang, Yujcong Chac, Sung- Yoon, Tac-Kyun Kim, and Kuk-Jin Yoon. Evdistill: Asynchronous events to end-task learning via bidirectional reconstruction-guider cross-modal knowledge distillation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 608-619, 2021.
[41] Yikai Wang. Xinghao Chen, Lele Cao, Wenbing Huang. Fuchun Sun, and Yunhe Wang. Multimodal token fusion for vision transformers. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 12186-12199 12195, 2022.
[42] Wentao Wu, Xiao Wang, Chenglong L., Bo Jiang, Jin Tang, Bin Luo, and Qi Liu. Cm3ae: A unified rgb frame and voxel-voxel pre-training framework. arXiv preprint arXiv:2504. 12576, 2025.
[43] Ruihao Xia, Chaoqiang Zhao, Meng Zheng, Ziyan Wu, Qiyu Sun, and Yang Tang. CMDA: cross-modality domain adaptation for nighttime semantic segmentation. In IEEE/CVF International Conference on Computer Vision, ICCV 2023, Paris, France, October 1-6, 2023, pages 21515-21524. IEFE, 2023.
[44] Yan Xia, Hai Huang, Jieming Zhu, and Zhou Zhao. Achieving cross modal generalization with multimodal unified representation. Advances in Neural Information Processing Syxtems, 36, 2024.
[45] Xiaoyang Xiao, Yuqian Zhao, Fan Zhang, Biao Luo, Lingli Yu, Baifan Chen, and Chunhua Yang. Baseg: Boundary aware semantic segmentation for autonomous driving. Neural Networks, 157:460-470, 2023.
[46] Bochen Xie. Yongjian Deng, Zhanpeng Shao, and Youfu Li. Eisnet: A multi-modal fusion network for semantic segmentation with events and images. IEEE Transactions on Multimedia, 2024.
[47] Enze Xie, Wenhai Wang, Zhiding Yu, Anima Anandkumar, Jose M Alvarez, and Ping Luo. Segformer: Simple and efficient design for semantic segmentation with transformers. Advances in neural information processing systems, 14:12077-1 2021.
[48] Yan Yang. Liyuan Pan, and Liu Liu. Event camera data pre-training. In Proceedings of the IEEE/CVF international conference on computer vision, pages 10699-10709, 2023.
[49] Bowen Yao, Yongjian Deng, Yuhan Liu, Hao Chen, Youfu Li, and Zhen Yang. aadaerent-adapter: adapter: Adapting segment anything model for event-rgb semantic segmentation. 2024 IEEE International Conference on Robotics and Automation (ICRA). pages 9093 9100. IEEE, 2024.
[50] Jiaming Zhang, Huayao Liu, Kailum Yang, Xinxin Hu, Ruiping Liu, and Rainer Stiefelhagen. Cmx: Cross-modal fusion for rgb-x semantic segmentation with transformers. IEEE Transactions on intelligent transportation systems, 2023.
[51] Jiaming Zhang, Ruiping Liu, Hao Shi, Kailun Yang, Simon Reiß, Kunyu Peng, Haodong Fu, Kaiwei Wang, and Rainer Stiefelhagen. Delivering arbitrary- modal semantic segmentation. In Proceedings of the IEEE/CVF Conference Computer Vision and Puttern Recognition, pages 1 1136-1147, 2023.
[52] Jiaming Zhang, Kailun Yang, and Rainer Stiefelhagen. Issafe: Improving semantic segmentation in accidents by fusing event-based data. In 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 1132- 1139. IEEE, 2021.
13

---

[53] Song Zhang, Yu Zhang, Zhe Jiang, Dongqing Zou, Jimmy Ren, and Bin Zhou. Learning to see in the dark with events. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XVIII 16, pages 666-682. Springer, 2020.
[54] Fengan Zhao, Qianang Zhou, and Junlin Xiong. Edge-guided fusion and motion augmentation for event-image stereo. European Conference on Computer Vision, pages 190-205. Springer, 2024.
[55] Yucheng Zhao, Gengyu Lyu, Ke Li, Zihao Wang, Hao Chen, Zhen Yang. and Yongjian Deng. Eseg: Event-based segmentation boosted by explicit edge-semann seganteitttic guidance. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 10510-100 10518, 2025.
[56] Mingmin Zhen, Jinglu Wang, Lei Zhou, Shiwei Li, Tianwei Shen, Jiaxiang Shang. Tian Fang. and Long Quan. Joint semantic segmentation and boundary detection using iterative pyramid contexts. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13666-13675, 2020.
[57] Sipeng Zheng, Bohan Zhou, Yicheng Feng. Ye Wang. and Zongqing Lu. Unicode: Learning a unified codebook for multimodal large language models. arXiv preprint arXiv:2403.c 09072, 2024.
[58] Chu Zhou, Minggui Teng, Jin Han, Jinxiu Liang, Chao Xu, Gang Cao, and Boxin Shi. Deblurring low-light images with events. International Journal of Computer Vision, (31(5):2244-122)) 2023.
[59] Chu Zhou, Minggui Teng, Jin Han, Chao Xu, and Boxin Shi. Delieve-net: Deblurring low-light images with light streaks and local events. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1155-1164, 2021.
[60] Hanyu Zhou, Yi Chang, Haoyue Liu, Wending Yan, Yuxing Duan, Zhiwei Shi, and Luxin Yan. Exploring the common appearance- pppuurance-bouuddryy adaptation for nighttime optical flow. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024.
[61] Alex Zihao Zhu, Liangzhe Yuan, Kenneth Chaney, and Kostas Danillidis. Unsupervised based optical flow using motion compensation. In Proceedings of the European Conference on Computer Vision (ECCV) Workshops, pages 71 71144,, 2018.
14

---

Appendix
A Details of datasets
Three extreme- extreme-condition datasets are constructed and used for testing in our work, namely DERS-XS, DERS-XR, and DSEC-Xtmm Xtrmm Among them, DERS-XS and DSE are synthetic datasets, and DERS-XR is real-world dataset. In addition, common dataset of normal conditions, namely DSECCSemantic used for testing. By leveraging edge-awareness, our method can effectively obtain the common features of heterogeneous event and RGB under unified semantic space and jointly optimize them. Results show that our method outperforms existing event- segmentation methods and possesses superior resilience in the case of modality imbalance and failure under extreme conditions.
Table 7: Comparison between datasets.
Datasets
#Train
#Validation
#Test
Real/Syn.
True/Pseudo Ibl.
Fine/Course lbl.
Avg. Pixel Val.
Avg. #Events
DERS-XS
2016
144
1080
Synthetic
Truc-label
Fine-label
6.16
70490 88
DERS-XR
120
N/A
120
Real-world
Truc-label
Coarse-label
20.58
12784.70
USEC-Semantic
8082
N/A
2809
Real-world
Pseudo-
Fine-label
75.13
608162.26
DSEC Xtrm
N/A
Synthetic
Pseudo
Fine label
689098.25
A.1 DERS-XS
Our synthetic dataset DERS-XS is constructed based on the CARLA simulator (9) and v2e simulator 13]. containing 270 frame sequences. We first obtain canonical RGB frames with labels from CARLA simulator. For CARLA simulation process, we first load six pre-made maps, namely Town01, Town02, Town03, Town04, Town05, and Town10. Then we apply fifteen different types of weather conditions on each map. For each weather and map combination, we record two sequences of 1200 frames at a frame rate of 20 fps. The spatial size of each frame is 640 x 360.
The CARLA simulated segmentation labels contain 23 categories, and the specific category names are unlabeled, building, fence, other, pedestrian, pole, roadline, road, sidewalk, vegetation, vehicles, wall, traffic sign, sky, ground, bridge, rail track, guard rail, traffic light, static, dynamic, water, and terrain. For experiments, we merge and transform the above 23 categories into 11 categories, in order match the categories setting of DSEC-Semantic [11] 33). The 11 categories are background, building, fence, person, pole, road, sidewalk, vegetation, car, wall, and traffic sign. Categories that do not exist DSEC-Semantic are set to 255 and ignored during the training and testing process.
We then obtain noisy events from v2e simulator based on the CARLA simulated frames. For v2e simulation process, the positive threshold and the negative threshold are both set as 0.2. The 1-std deviation threshold variation is set as 0.05, The cutoff frequency is set as 30, and the leak event rate per pixel is set as 0.1. The shot noise rate is set as 5.0. The refractory period is set as 0.0005.
We implement the Image Signal Processing (ISP) pipeline and its inverse process, referred to as the ISP unprocessing technique in 7 which includes digital gain, white balance, demosaicing, color correction, gamma compression, and tone mapping. By utilizing the inversion of ISP, we first convert CARLA-simulated canonical RGB frames into Bayer-pattern BGGR RAW images [2] As shown in we attenuate the optical signals and add shot noise on RAW images, and then process RAW images by ISP to obtain low-light images.
fig. 9
We sample the 1200-frame sequences the intervals of 100 frames, taking 12 frames from each sequence while discarding the rest. We divide the 270 sequences into three parts, of which the training set has 168 sequences, the validation set has 12 sequences, and the test set has 90 sequences. For training process of DERS-XC XS, we use the training set for training, and the validation set for saving the model with the best validation mloU. The test set used only during the testing process.
A.2 DERS-XR
Our real-world dataset DERS-XR is captured by DAVIS-346 38 with the spatial size of 346 x 260. We use the camera to capture the events with APS frames of 20 fps simultaneously under extreme lighting conditions. We capture total of 27 frame sequences, sampling them at intervals
15

---

Magping
Compression
Altemuate Optical Signals & Adé Shot Noise
Inv. Digital Gain
Digital
Comppeesioo
Mapping
Figure 9: Obtaining low-light RGB images using ISP with its inversion process. We implement the ISP with its inversion process, including digital gain, white balance, demosaic, color correction, gamma compression, and tone mapping. Optical Signals are attenuated, and shot noise added on RAW domain images. The process used to simulate low-light RGB images in DERS- XS and DSEC-XILS Xtrm.
of 100 frames, and finally obtain 240 frames of images with very different contents. We manually annotate semantic labels for the sampled 240 frames, and the annotated categories the same as the transformed 11 categories of DSEC-XS. We randomly select 120 frames for fine-tuning and the remaining 120 frames for testing. For experiments conducted DERS-XR, save the last epoch fine-tuning model for testing.
A.3 DSEC-Xtrm
DSEC-Xur is simulated and converted from the real-world dataset DSEC-Semantic. A sample of DSEC-Semannt and DSEC-Xtrm is shown in fig. 10 DSEEC-Smmntic includes 11 sequences with pseudo-labels of 19 categories and 11 cat- egories. We generate pure noise events by mod- ifying the source codes of v2e simulator and overlaying them to the event sequence DSEC- Semantic to obtain noisy events in DSEC-Xim The shot noise rate tain low-light images, we apply the same ISP and ISP-inversion process, unprocessing RGB images to RAW domain, attenuating the optical signals and adding shot noise on the unprocessed RAW images, and then process the RAW images RGB low-light images by ISP. We sample the 11 sequences at the intervals of frames and discard the first 6 frames of each sequence, fol- lowing the same setting with [36]. The simulated low-light images and noisy events combined with the original 1 11-categories labels of DSEC-Semanti together constitute the final DSEC-Xtrm dataset.
DSEC-Semantic
DSEC-Xtmm
as 10.0 Hz. To ob-
Figure 10: DSFC-Semamtic and DSEC-Xtrm.
A.4 Discussion on datasets
This paper uses the DSEC-Semante and constructs three datasets, each of which has its own properties. Table compares the properties of different datasets, including the number of image-events pair in the training set, validationset and test set, whether the data is real-world or synthetic, whether the label is true pseudo, whether the label is fine coarse, and the average pixel value of images and average number of events each dataset. Based on the different properties of different datasets, we use them in different experimental settings test the model comprehensively.
The DERS-XS only contains synthetic data, but DERS-XS has the largest amount of data with the true fine-grained label, thus DERS-XS can be a standard benchmark for comparative
16

---

Lowe: Noise Level
Higher Noise Level
Diaton bension axt
Lxces lllenddd win Datand Hleasory
Unmre Ulrdded with Dibted Dousttry
Roundary Map
Dvense Diased
Dilated Maps
Salect Randon thdansn turzzooe from
92
Events
Events Blended with Mask
Dazsdary Tholt Batt
Daundory Tbnie Rarte
Figure 11: Details boundary map dilation.
Figure 12: Event-edge statistics under noise levels.
experiments. The DERS-XR a real-world dataset with the most realistic data distribution, however, it is difficult to annotate it accurately, and for this reason, only coarsely annotate small amount of data DERS-XR, XR. Since is difficult to annotate DERS-XR, only small amount of data is annotated, thus only use DERS-XE XR for fine-tuning and testing.
The DSECCSemantic is real-world dataset under normal conditions, however, the labels are pseudo labels based on RGB images only. Although it is used for testing in many works, using DSEC- Semantic as a benchmark for multi-modality semantic segmentation is defective. Thus, we For DSEC- Xtrm, the pseudo
simulate the extreme version DSEC-Xtmi from DSEC-Semantie labels from DSEC-Semantic are no longer a defect for being a benchmark for multi-modality semantic segmentation. Therefore, when testing on the DSEC-Semantic and DSEC-Xtrm, can compare the performance degradation on the two datasets for different methods. The performance degradation DSEC-Semantic and DSEC-Xtrm illustrate the robustness and resilience of different methods.
B Details of training settings
For all datasets, we use AdamW (27 optimizer with a weight decay 0.01, and the learning rate of decoder is 10 times the basic learning rate. For DERS-XS, we train our model on two NVIDIA RTX 3090 GPUs for 300 epochs, and the batch size is on each GPU. The basic learning rate (LR) is x 10-5, which is scheduled by CyclicLR (34) scheduler with maximum learning rate 1.6 LR and triangular cycle of 10 epochs. For DERS XR. we fine-tune our model for 50 epochs on single GPU with batch size of 2, based on the best model trained on DERS-XS -SR. The basic learning rate is x 10-5, which is scheduled by WarmupPolyL.R scheduler with power of 0.9 and warmup epochs of 10. For DSEC-Semantic and DSEC-Xtrm, we follow settings from DERS-XS with epochs of 60. We input events with 50 ms interval for DERS-XX XS and DERS-XR, and count of 50000 events for DSEC-Semantic and XSEC-Xtrm.
C Details of event-edge statistics
For the statistical process, we randomly select 200 event sequences of ms accompanied by their boundary maps from DERS-XX XS as sample. We aim to count the ratio of edge pixels to all pixels of the whole plane, and the ratio of events falling on edge pixels to all events of the whole sequence. As shown in fig. [1 in order make the ratio of edge pixels distributed in larger range, we first dilate the boundary map with 3 x 3 kernel with a random number of iterations in the range 10. We draw scatter plot show the correlation between the two ratios.
As the boundary map dilates, the ratio edge pixels increases, and the ratio of events falling on edge pixels also increases synchronously. Considering the extreme cases, when the ratio of edge pixels is
17

---

O, there is no events falling on edge pixels, then the ratio of events is also 0; when the boundary map dilates to the whole image, the ratio of edge pixels is I, then all events fall on edge pixels, the ratio of events is 1. However, for the cases where the ratio of edge pixels is between O and 1, for most samples, the ratio of events falling on edge pixels is greater than the ratio of edge pixels. This proves the fact that events tend to cluster at the areas of semantic edge. This phenomenon exhibits a strong correlation between events and semantic edge.
We also count the two ratios under two noise levels of events. We use v2e simulator to simulate a version of event sequences that are less noisy than the event sequences of DERS-XS We simultane- ously count the event ratios corresponding to the two noise versions of the event sequences, and draw scatter plots for the lower noise level event sequences and the higher noise level event sequences (i.e. We compare the scatter plots of the two noise levels. As shown in fig. [2] as the noise level increases, the ratio of edge pixels to pixels and the ratio of events falling on edge pixels to all events tend to be equal. The correlation curve for the higher noise level case is still a concave curve, thus the correlation is still maintained even under the influence of high noise. This ensures the resilience of our method in the case of modality imbalance or failure under extreme conditions.
D Details of edge dictionary training process
The training of our edge dictionary is separate stage from the training of the segmentation model. Through this separate training stage, we obtain the pre-trained weights of edge dictionary, which represents the information of the semantic edge. We also obtain the pre-trained weights of the tokenizer, which can be used to embed the semantic edge ground truth into the discrete latent space defined by edge dictionary. The detokenizer only used in the edge dictionary training stage, and is deprecated in the segmentation model training stage.
The tokenizer fr consists of two convolutional layers followed by ReLU, two residual blocks, and a final convolutional layer. The first two convolutional layers downsampled the inputs each with a kernel of 4 x 4 size, stride of (2, 2). and a padding of (1, 1). The two residual blocks keep the spatial scale unchanged, each consists three convolutional layers followed by ReLU with a kernel of 3 x size, stride of (1, 1). and padding of (1, 1). The final convolutional layers adjust the number of channels to n by a convolutional layer with a kernel of 1 x size. For semantic edge B E {0, 1) #fxW the final produced edge embeddings r T= /r( E R xW' xn have the downsampled spatial size H' x W", where H' = # W =
L W
The detokenizer fr- is constructed by changing the downsampled convolutional layers to the trans- posed convolutional layer for upsampling and inverting all the layers in tokenizer. Thus, the detok- enizer consists of convolutional layer followed by ReLU, two residual blocks and two transposed convolutional layers followed by ReLU, and a final convolutional layer to predict the semantic edge. The kernel size, stride, and padding settings of convolutional layers and transposed convolutional layers are the same as the corresponding layers in tokenizer. The detokenizer takes the quantised embeddings I' E R Hf xW' Xn as input, predicts the reconstructed semantic edge B' = frr((T))
We adopt the training objective with reconstruction loss, embedding loss, and commitment loss of VQ-VAE [39] as Tudiet = |B- - 111 + ||u(( - sg( + ai]lsg(n(K)) -Ili, where sg means stop gradient, and a a constant of commitment loss weight, which 0.25 in our work. To make the reconstruction loss propagate back to the tokenizer, gradient straight-through technique [3] is adopted, which directly assigns the gradient from I' to r. We train the edge dictionary with tokenizer and detokenizer for 3000 epochs, based on the data of -ESS-XS and DSEC-Semantic. No additional information is introduced, and the pre-trained weights of tokenizer are introduced into the segmentation training stage only for the supervision in latent space, and not utilized for testing stage.
E
Additional ablation studies
E.1
Edge Dictionary Domain Transferability
Theoretically, the discrete edge dictionary learned by VQ-VAE is expected to have good transferability across datasets. As an intermediate representation, semantic edge exhibits relatively simple and consistent structures, and the latent distributions of semantic edge derived from segmentation labels
18

---

Table 8: Ablation study edge dictionary domain transferahility.
Settings
#ACC(SO]
mACC(S0T
mloU(%)
ESC on DERS-X XS (w/ edge dictionary of DSEC)
93.23
74,96
66.44
ESC on DERS-XS
93.27
75.26
67.10
ESC on DSEC-Semantic (w/ edge dictionary of DERS-XS)
94,91
78.04
70.93
ESC on DSECC-Sematic
94.85
78.61
71.04
ESC on DSEC-Xim (w/ odge dictionary of DERS-XS)
88.57
58.00
50.65
ESC on DSFC-Xim
88,18
59.45
50.87
Table Ablation study on different event sampling strategies.
Settings
gACC(%)
mACC($}T
mloU(%))
ESC on DSEC-SSmande (50 ms)
94.76
78.44
70.83
ESC on DSEC- CeEC-mmmaiicc (100,000 events)
94.87
78.00
70.84
ESC on DSEC-Semantic (50,000 events)
94.85
78.61
71.04
tend to vary only slightly across different datasets. Therefore, we expect the performance degradation under dictionary transfer settings to minimal.
We conduct cross-domain transferability evaluations by (i) using an edge dictionary pretrained on DSEC to evaluate on DERS-XS, and (ii) using dictionary pretrained on DERS-XS evaluate on DSEC-Semantic and DSEC-Xtrm. As shown in table the performance drops are small in all cases. Specifically, the mloU DERS-XS drops by 0.66%, on DSEC-Semantic by 0.10%, and on DSEC-Xtrm by 0.21% %, compared to the original non- non-exchanged dictionary settings. These results suggest that the learned edge dictionary generalizes well across domains, and our method remains robust under moderate domain shifts.
E.2 Event Sampling Strategy
The DSEC-Semanin event input built by sampling fixed number of events per voxel grid rather than a fixed time window, which follows the setting of ESS [36]. In ESS, the event input is built with 100,000 events per voxel grid. We found that 50-ms fixed time window 100,000 fixed number of events is relatively large, which reduces the performance of data preprocessing, and may lead to insufficient edge characteristic representation. After the above trade-offs, we decided use 50,000 events per voxel grid for DSEC-Semantic as our event sampling strategy in our work.
To further demonstrate the impact of different event sampling strategies, conduct experiments DSEC-Semantic with a fixed time window of 50 ms, a fixed number of events of 100,000 per voxel grid, compared with 50,000 events per voxel grid in the main paper. As shown in table (9, different event sampling strategies present comparable results, with mIoU slightly lower (0.20% and 0. 19% respectively) than the fixed number of events of 50,000 in the main paper.
E.3 Deployment Efficiency and FLOPs Ablation
We conduct additional comparative experiments on DERS-X -ttm using smaller backbones (2x MiT- BO), reducing the FLOPs of ESC to be even lower than CMNeXt. In addition, measure the end-to-end inference latency of CMNeXt and our ESC (both standard ESC and reduced variant) on single NVIDIA GeForce RTX 3090 GPU with batch size of 1. All latency measurements are performed with a fixed input size of 512x 512, with each measurement calculating the average execution time over 100 inferences, and repeat times for stability.
As shown in table 10 the reduced ESC variant still outperforms CMNeXt, achieving 49.06% mIoU vs. 45. 16%, despite lower FLOPs (60.658G vs. 62.805G) and significantly fewer parameters (14.184M vs. 58.687M). This suggests that the performance gains stem from architectural design rather than merely increased computational cost. Furthermore, shown in table I0 the reduced ESC variant has an average inference latency of 29.37 ms, which is shorter than that of CMNeXt (29.79 ms), demonstrating its potential for more efficient deployment.
Above results indicate that even with lightweight backbones, our model maintains strong performance with higher inference speed, highlighting the effectiveness of our design beyond raw FLOPs. This reflects favorable orf between efficiency and performance, which is essential for practical deployment in real-world systems.
19

---

Table 10: Ablation study on lighter backhones DSEC--Xtm.
Latency (ms)
Settings
8ACC($9y
mACC($)1
mloU[(O]
#Params(M)
FLOPPs(G)
#1
#2
#3
Avg.
CMNeXt [5]
87.04
52.12
45.16
58.687
62.805
29.75
29.78
29.83
29.79
ESC (Reduced)
88.03
56.31
49.06
14.184
60.658
29.30
29.38
29.43
29.37
ESC (Standard)
88.18
59.45
50.87
56.875
95.086
34.56
34.46
34.78
34.60
Table 11: Extended experiments under severe spatial occlusion.
50 x 50
100
100
150
150
200 x 200
250 x 250
Methods / (%bt
8ACC
mACC
mloU
gACC
mACC
mloU
gACC
mACC
mloU
8ACC
mACC
mloU
gACC
MACC
mloU
TokenFusion
88.03
62.34
52.50
84.92
59.90
48.00
79.63
50.34
43.83
73.25
52.08
39.70
73.27
49.68
37.39
CMX 500
90.53
70.73
57.76
86.43
67.40
53.73
82.24
64.12
49.59
80.21
59.21
45.96
79.33
56.81
(43.99
CMNeXt [S]
91.22
71.91
60.65
87.53
68.71
53.70
81.62
64,30
48.01
77.08
59.51
43.71
74.37
56.42
41.12
EISNet 46
90.43
68.12
57.89
88.20
66.19
54.47
83.83
63.31
49.30
80.99
60.71
45.92
78.83
58.78
43.51
Ours
92.46
74.08
64.53
92.20
72.91
63.87
91.05
70.40
61.32
89.01
66.90
58.02
87.26
64.06
54.95
F Extended experiments under severe spatial occlusion
We extend our experiments under severe spatial occlusion. the main text, we apply masking on RGB and event with mask areas size of 100 x x100. We further conduct experiments with masking areas different sizes. Masking areas of size 50 x 50, 150 x 150, 200 x 200, and 250 x 250 are applied at coordinate (350, 200) for RGB and (150, 150) for event respectively. Both event and RGB are applied masking. The excess part is ignored the masking area exceeds the spatial area.
Table I1 demonstrates the comparison results of extended experiments under severe spatial occlusion. As the size of masking area increases, the performance of methods degrades, and our method consistently outperforms other multi-modality methods on different masking settings. CMX has lower performance than CMNeXt when the masking areas are small, but outperforms CMNeXt when the masking areas become larger. Figure 13 demonstrates the gualitative comparison results of extended experiments on masking settings on DERS-XS, Under different masking settings, although all methods are affected by the modality imbalance and information loss caused by masking, method obtains more reliable information based on uncertainty edge-aware joint optimization and edge consolidation, thus avoiding the misleading information of masking areas much possible.
G More qualitative comparison results
This section is an extension of the qualitative results of the experiments in the main text. Figures are placed at the end of the appendix. Figure demonstrates more qualitative comparison results on DERS-XS -S.. Figure 15] demonstrates more qualitative comparison results on DERS-XR. Figure [6 demonstrates more qualitative comparison results on demonstrates more qualitative comparison results under E-RGB mask setting on DERS-XS.
and DSEC-Xtre Xtrm. Figure
As shown in fig. 14 compared with other methods, our method segment vehicles and pedestrians with complex contours more stably and robustly under extreme conditions. Especially for pedestrians, the contours pedestrians segmented by other methods are not sharp enough, and sometimes even fail to detect the existence of pedestrians. our method can effectively locate pedestrians and segment the contours of pedestrians accurately.
As shown in fig. 15| our method performs well on real-world extreme scenes when fine-tuned with small amount of real-world data. This also confirms that our simulated dataset DERS-XS can effectively provide prior knowledge when the amount of real-world data is small.
As shown in fig. 16. our method can handle Event-RGB semantic segmentation under normal condi- tions well, and when the modality information is lost under extreme conditions, our method still able to identify the contours of ambiguous vehicles and pedestrians and maintains the segmentation performance better than other methods.
As shown in fig. 17 our method can still achieve relatively accurate segmentation results even when the inputs are partially masked. The image edge distribution and events edge distribution indicate their different awareness of edges. The two modalities complement each other and are dynamically fused based on their confidences and uncertainties. The results show that our method is more robust and resilient than other methods in the cases of modality imbalance and failure.
20

---

Image
Evans
Edge Distr.
Edit Thittr
Diatr. FT
TokenPusion
CMX
CMNNeX
FISNet
Qurs
GroandTnuuh
Figure 13: Qualitative results of extended experiments under spatial occlusion DERS-XX XS.
Image
Evants
SepPoraeer
SegFarmenE)) EVSopFormer
TokonFusion
CMX
CMNeX:
EISNet
Ours
GroandTruth
Figure 14: More qualitative comparison results DERS DERS-XS.
Image
Events
SenFormer
Fo EvSoyFormer
TokenFusion
CMX
CMNeXi
FISNet
Ours
GroundTruh
Figure 15: More qualitative comparison results on DERS-XR.
21

---

Imagee
Events
SeyFormer
So EvSeyFormer TokenFusion
CMX
CMNeX:
FiSNet
Ours
GroundTruth
Figure 16: More qualitative comparison results on DSEC-Semantic and DSEC--trm.
Evants
Edice
TokenFusion
CMX
CMNeXi
FiSNet
GroundTrul
10
Figure 17: More qualitative comparison results under E-RGB mask setting DERS-XS.
22

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We convinced that the main claims made in the abstract and introduction do accurately reflect the paper's contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Despite the limited length of the paper, we still briefly discussed one possible limitation of the work in section 5
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sellsspeciiiationn asymptotic approximations only holding locally). ). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
23

---

Answer:
[NA]
Justification:
The results of the paper are all experimental, not theoretical.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof's can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: Technical details of the core method and dataset construction method provided section 3] and implementation details are listed in section 4.1 and appendix B
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, descríbing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e-g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
5. Open access to data and code
24

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification:
The code and datasets are released at http::tt//gtiii http://gthuub..com///cEE//EE/E/
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.co cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run reproduce the results. See the NeurIPS code and data submission guidelines (https
/ "pipsscc/puullc//uuu /otee//guudddeeeedSuuuubmiibia ssionPolicy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: All the training and test details are listed section and appendix
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: Error bars are not reported due to the high computational cost.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
25

---

The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative eтror rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
The type and amount of GPU are reported in section 4.1
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g.. preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics http::///harrrps..ccccppp//////e
Answer: [Yes]
Justification: We have reviewed the NeurlPS Code of Ethnics, and the research conducted in the paper conforms with the NeurIPS Code of Ethnics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g.. if there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: The paper is foundational research and not tied to particular applications currently, so there are no potential societal impacts of the work performed.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
26

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper poses such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
The creators or original owners of assets (e.g., code, data, models), used the paper, are properly credited, and the license and terms of explicitly mentioned and properly respected.
Justification:
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version the asset used and, possible, include URI.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g.. website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms use in package should be provided. For popular datasets, paperswithcone com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
27

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [No]
Justification:
The paper has not released new assets at present.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moddl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Bthies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, not include any information that would break anonymity (if applicable), such the institution conducting the review.
28

---

16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: The core method development in this research does not involve LLMs as any important, original, or non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy (https: /ttpp/////euups..... wwwparipppscccoooeeeeeeeee cc/Conferencees///////e for what should or should be described.
29


Reviews:
[{"review_id": "zwWmB4EI21", "reviewer": "Reviewer_DoyC", "summary": "The paper presents Edge-awareness Semantic Concordance (ESC), a framework for robust event-RGB semantic segmentation under challenging conditions. ESC leverages a shared discrete edge dictionary learned via a VQ-VAE on semantic edge maps to bridge heterogeneous event and RGB features. It introduces three key modules: (1) Edge-awareness Latent Re-coding (ELR), which aligns event/RGB features into a unified edge-categorical space and produces uncertainty indicators. (2) Re-coded Consolidation (RC), which injects re-coded edge embeddings into image features and (3) Uncertainty Optimization (UO), which dynamically fuses modalities based on confidence/uncertainty maps. The authors also construct two synthetic (DERS-XS, DSEC-Xtrm) and one real-world (DERS-XR) extreme condition datasets to evaluate ESC’s performance. Experiments on these benchmarks demonstrate consistent mIoU improvements of ~2–3% over state-of-the-art fusion methods and greater resilience to spatial occlusion.", "questions": "1. How well does the VQ-VAE–learned edge dictionary transfer across domains? For example, if it's trained on one dataset and appliedunchanged to another, what is the mIoU change? \n2. How sensitive are your results to key hyperparameters such as dictionary size and uncertainty threshold?\n3. Can you report end-to-end inference latency and peak memory usage for ESC on representative hardware?", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. The discrete edge dictionary learned via VQ-VAE provides an effective bridge between event and RGB feature spaces.\n2. The modular design of ELR, RC, and UO is thoughtfully structured, and the comprehensive ablation studies transparently demonstrate the value of each component.\n3. Consistent mIoU improvements (2–3 %) across both synthetic (DERS-XS, DSEC-Xtrm) and real-world (DERS-XR) benchmarks highlight the approach’s robustness under extreme conditions.\n4. The incorporation of uncertainty maps to guide dynamic fusion of event and RGB modalities enhances performance when data quality varies.\n\nWeaknesses:\n1. Dependence on a VQ-VAE–learned discrete edge dictionary adds a non-trivial pretraining step and raises questions about how well the same dictionary transfers to new domains.\n2. The combination of ELR, RC, and UO modules, while effective, increases architectural complexity and may incur higher runtime and memory overhead in practice.\n3. Although parameters and FLOPs are reported in Table 5, inference latency and throughput on real hardware are not provided, making it difficult to fully assess deployment feasibility.", "was_revised": true, "final_justification": "The rebuttal addressed all of my earlier concerns with additional experiments and clarifications. I am satisfied with the responses and maintain my 5 (Accept) rating.", "rebuttal": "We are grateful to the reviewer for the detailed and professional review, which reflects a deep understanding of the topic. We sincerely appreciate the positive comments, valuable concerns, and suggestions on our work. Here is our response to the mentioned weaknesses and questions.\n\n---\n\n### **Edge Dictionary Domain Transferability (W1, Q1)**\n\nThank you for the insightful question. Theoretically, the discrete edge dictionary learned by VQ-VAE is **expected to have good transferability across datasets**. As an intermediate representation, semantic edge exhibits relatively simple and consistent structures, and the latent distributions of semantic edge derived from segmentation labels tend to vary only slightly across different datasets. Therefore, we expect the performance degradation under dictionary transfer settings to be minimal.\n\nFollowing the reviewer’s suggestion, we conduct cross-domain evaluations by (i) using an edge dictionary pretrained on DSEC to evaluate on DERS-XS, and (ii) using a dictionary pretrained on DERS-XS to evaluate on DSEC-Semantic and DSEC-Xtrm. As shown in Table R1, the performance drops are small in all cases. Specifically, the mIoU on DERS-XS drops by 0.66%, on DSEC-Semantic by 0.10%, and on DSEC-Xtrm by 0.21%, compared to the original non-exchanged dictionary settings. These results suggest that **the learned edge dictionary generalizes well across domains**, and our method remains robust under moderate domain shifts.\n\n**Table R1: Ablation study on edge dictionary exchange settings.**\n\n| Settings                                       | gACC(%)↑ | mACC(%)↑ | mIoU(%)↑ |\n| ---------------------------------------------- | -------- | -------- | -------- |\n| ESC on DERS-XS (w/ edge dictionary of DSEC)    | 93.23    | 74.96    | 66.44    |\n| ESC on DERS-XS                                 | 93.27    | 75.26    | 67.10    |\n| ESC on DSEC-S. (w/ edge dictionary of DERS-XS) | 94.91    | 78.04    | 70.93    |\n| ESC on DSEC-S.                                 | 94.85    | 78.61    | 71.04    |\n| ESC on DSEC-X. (w/ edge dictionary of DERS-XS) | 88.57    | 58.00    | 50.65    |\n| ESC on DSEC-X.                                 | 88.18    | 59.45    | 50.87    |\n\n---\n\n### **Hyperparameters Sensitivities (Q2)**\n\nThank you for raising this important point. We observe that the model's performance shows noticeable variation under different dictionary size $K$, **but remains robust across a reasonable range**. As shown in Table 4 of the main paper (reproduced here as Table R2), using too small a dictionary leads to insufficient latent edge representations, while an overly large dictionary introduces ambiguity in modality-specific learning, potentially due to increased difficulties in selecting appropriate dictionary entries for modality-specific edge patterns. Both extremes negatively impact performance. The best results are achieved with a moderate dictionary size (e.g., $K=128$), which offers a good balance between representation capacity and stability in codebook learning.\n\n**Table R2: Ablation study on key usage (dictionary size) on DERS-XS.**\n\n| Dictionary Size      | 16    | 32    | 64    | 128   | 256   | 512   |\n| -------------------- | ----- | ----- | ----- | ----- | ----- | ----- |\n| **Dictionary Usage** | 16    | 32    | 64    | 92    | 99    | 97    |\n| **gACC(%)↑**         | 92.66 | 93.12 | 93.03 | 93.27 | 93.19 | 92.93 |\n| **mACC(%)↑**         | 74.59 | 74.77 | 74.90 | 75.26 | 73.91 | 73.65 |\n| **mIoU(%)↑**         | 65.87 | 66.84 | 66.64 | 67.10 | 66.54 | 66.11 |\n\nAs for the uncertainty threshold, our framework does not rely on any manually set threshold as a hyperparameter. Instead, the uncertainty indicators are **dynamically inferred** and represented as continuous values in the range of 0 to 1. These maps serve as **soft attention weights**, allowing the model to adaptively determine the fusion strength between heterogeneous event and RGB.\n\nTo further examine the role of these learned confidence and uncertainty maps, as well as the contribution of the event modality, we conduct additional experiments on DERS-XS by **injecting controlled modifications during the inference phase**, specifically by **zeroing out the confidences** of event modality. As shown in Table R3, this results in a 1.49% drop in mIoU, indicating a noticeable degradation in performance. This ablation underscores the effectiveness of our dynamically learned uncertainty indicators, demonstrating that the model can **adaptively control the fusion strength of each modality based on the input content, without relying on a manually tuned threshold**.\n\n**Table R3: Ablation study on injecting modifications to confidence and uncertainty indicators.**\n\n| Settings         | gACC(%)↑ | mACC(%)↑ | mIoU(%)↑ |\n| ---------------- | -------- | -------- | -------- |\n| ESC (zero conf.) | 92.98    | 73.50    | 65.61    |\n| ESC              | 93.27    | 75.26    | 67.10    |\n\n---\n\n### **End-to-end Inference Latency, Throughput and Peak Memory Usage (W2, W3, Q3)**\n\nCertainly, and thank you for the suggestion. Here are the end-to-end inference latency, inference throughput, and peak memory usage of GPU and CPU in Table R4, averaged over three runs for stability. All experiments are performed on a machine equipped with NVIDIA GeForce RTX 3090 GPUs and an AMD EPYC 7642 48-Core CPU. Inference latency is measured on a single GPU with a batch size of 1, while throughput and memory usage are measured on two GPUs with a batch size of 16 per GPU, and only the peak memory usage of the process on GPU 0 is reported. All measurements are conducted with a fixed input resolution of 512 × 512. For inference latency and throughput, each measurement calculates the average execution time over 100 inferences.\n\nThese results indicate that ESC achieves competitive inference efficiency and moderate memory usage, suggesting good potential for deployment in practical scenarios. Combined with the reported FLOPs and parameter count in Table 5, this further supports the **practical feasibility** of our proposed ESC framework.\n\n**Table R4: Inference latency, throughput, and peak memory usage of GPU and CPU.**\n\n| #    | Inference latency | Inference throughput | peak memory usage (GPU) | peak memory usage (CPU) |\n| ---- | ----------------- | -------------------- | ----------------------- | ----------------------- |\n| 1    | 34.56 ms          | 84.26 samples/sec    | 12169.33 MB             | 1903.26 MB              |\n| 2    | 34.46 ms          | 84.17 samples/sec    | 12169.33 MB             | 1901.64 MB              |\n| 3    | 34.78 ms          | 84.35 samples/sec    | 12169.33 MB             | 1879.25 MB              |\n| Avg. | 34.60 ms          | 84.26 samples/sec    | 12169.33 MB             | 1894.72 MB              |"}, {"review_id": "DYSNl9Pm8q", "reviewer": "Reviewer_BVgg", "summary": "This paper presents ESC that exploits semantic edges to fuse heterogeneous RGB images and event streams for robust semantic segmentation under extreme conditions. ESC first constructs a discrete edge dictionary, then performs ELR to align event- and RGB-derived edge distributions in a shared latent space. Two tailored modules—RC and UO—respectively integrate edge cues into RGB context features and adaptively weight modalities using confidence estimates, yielding resilient predictions when either modality degrades. To enable reliable evaluation, the authors use three benchmark datasets with ground-truth labels in extreme low-light or noisy scenarios.", "questions": "NONE", "limitations": "NONE", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "The introduction of a shared discrete edge dictionary combined with the Edge-aware Latent Re-coding mechanism is novel and well-motivated. The paper backs this design with solid empirical evidence, including an event-edge correlation analysis and ablations isolating each module’s contribution.\n\nThree benchmark datasets further amplify its impact and should facilitate future research in multimodal segmentation.\n\n---\n\nThe study compares ESC mainly with token- or attention-level fusion baselines. Other recent techniques such as cross-modal contrastive pretraining or adaptive modality dropping are not investigated.\n\nThe manuscript layout appears misaligned with NeurIPS guidelines: (1) Text embedded in figures is noticeably smaller than the main font, making labels hard to read. (2) Several tables spill over margins or have inconsistent column widths.\n\n\n\n[1] SAM-Event-Adapter: Adapting Segment Anything Model for Event-RGB Semantic Segmentation", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate the reviewer’s time and effort in thoroughly reviewing our manuscript. We also appreciate the insightful suggestions and provide detailed responses below.\n\n---\n\n### **Other Techniques Investigation and Comparison (W1)**\n\nWe appreciate the reviewer for highlighting this important research direction. Our ESC framework lays emphasis on the **inferior optimization issues of heterogeneous event and RGB in the case of modality imbalance and failure situations**. Motivated by this, our work emphasizes developing more **robust and resilient fusion strategies** under the existing extreme input conditions. Therefore, we center our investigation around recent token-level and attention-level fusion baselines that address similar concerns.\n\nOther techniques suggested by the reviewer, such as cross-modal contrastive pretraining (e.g., Event-Camera-Data-Pre-training [A], SAM-Event-Adapter [B], CM3AE [C]) and adaptive modality dropping (e.g., Missing-Modality-Prediction [D], RAGPT [E]), indeed **provide valuable and complementary perspectives** in the broader multimodal learning field. However, cross-modal contrastive pretraining approaches primarily aim to acquire informative and effective pretrained backbones, while adaptive modality dropping methods mainly tackle issues of incomplete modality inputs. **In contrast**, our focus lies in inferior optimization issues of modality imbalance and failure, which are **prevalent yet underexplored challenges** in real-world scenarios. This distinction reflects different motivations and problem settings, rather than limitations of any particular method.\n\nFor completeness of comparison, we include in Table R5 the reported performance of two representative contrastive pretraining methods ([A], [B]) on the DSEC-Semantic dataset. The results are cited directly from the original papers, as [B] does not release its code, and we rely on their published numbers.\n\nIn the final version of the paper, we will **expand the above discussion and theoretical comparison in the related work section to include these approaches and cite the relevant literature, including [A-E]**. We believe that this will help situate our work more clearly within the multimodal fusion landscape and highlight its distinct contributions.\n\n**Table R5: Comparison on DSEC-Semantic with two cross-modal contrastive pretraining methods.**\n\n|                                    | gACC(%)↑ | mACC(%)↑ | mIoU(%)↑ |\n| ---------------------------------- | -------- | -------- | -------- |\n| Event-Camera-Data-Pre-training [A] | -        | -        | 59.16    |\n| SAM-Event-Adapter [B]              | 93.58    | -        | 69.77    |\n| ESC (Ours)                         | 94.85    | 78.61    | 71.04    |\n\n---\n\n### **Formatting and Layout Issues (W2)**\n\nThanks for your careful review and valuable suggestions. We will carefully adjust the texts embedded in figures for more convenient reading in our final manuscript version. As for tables, we will also trim them neatly in the final version. \n\nIt must also be seriously pointed out that we **DO NOT** violate any NeurIPS style guidelines. **No tables spill over margins or have inconsistent column widths.**\n\n### **References**\n\n[A] Yan Yang, Liyuan Pan, and Liu Liu. \"Event camera data pre-training.\" ICCV, 2023.\n\n[B] Bowen Yao, Yongjian Deng, Yuhan Liu, Hao Chen, Youfu Li, and Zhen Yang. \"Sam-event-adapter: Adapting segment anything model for event-rgb semantic segmentation.\" IEEE ICRA, 2024.\n\n[C] Wentao Wu, Xiao Wang, Chenglong Li, Bo Jiang, Jin Tang, Bin Luo, Qi Liu. \"CM3AE: A Unified RGB Frame and Event-Voxel/-Frame Pre-training Framework.\" ACM MM, 2025.\n\n[D] Donggeun Kim, and Taesup Kim. \"Missing modality prediction for unpaired multimodal learning via joint embedding of unimodal models.\" ECCV, 2024.\n\n[E] Jian Lang, Zhangtao Cheng, Ting Zhong, and Fan Zhou. \"Retrieval-augmented dynamic prompt tuning for incomplete multimodal learning.\" AAAI, 2025."}, {"review_id": "zTSXTXOpuq", "reviewer": "Reviewer_53vx", "summary": "This paper addresses the challenge of semantic segmentation under extreme conditions by leveraging heterogeneous Event-RGB inputs. The authors propose a novel framework called Edge-awareness Semantic Concordance (ESC), which utilizes semantic edges as a modality-bridging representation to align RGB and event features in a unified semantic space. The method consists of three main components: Edge-awareness Latent Re-coding (ELR), Re-coded Consolidation (RC), and Uncertainty Optimization (UO). Additionally, the authors construct three new benchmark datasets (DERS-XS, DERS-XR, and DSEC-Xtrm) designed to evaluate performance and robustness under degraded visual conditions. Experimental results demonstrate that the proposed method achieves state-of-the-art performance across multiple metrics and maintains superior robustness under extreme conditions like simulated occlusion and modality degradation.", "questions": "See the weaknesses (Cons) for detailed suggestions.", "limitations": "Yes", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Pros:\n1.The framework introduces semantic edge as a unified representation to bridge RGB and event modalities. The motivation is well illustrated in Fig. 2, with visualizations of edge and event distributions showing high spatial correlation.\n2.The modular design is clear and well-motivated: ELR provides semantic consistency supervision, RC fuses refined edge features across modalities, and UO models the confidence of edge features to enable attention-based adaptive fusion. Together, they form a coherent pipeline for robust multi-modal segmentation.\n3. The experimental evaluation is comprehensive: the authors not only validate the method on the existing popular dataset DSEC-Semantic, but also construct three new and more challenging datasets. The experiments include extreme scenarios such as under-exposure and occlusion, providing a thorough validation of the method's applicability.\n\nCons:\n1.The ELR module uses a VQ-VAE-based edge dictionary to achieve modality alignment. It is unclear whether alternative encoding strategies (e.g., prototype-based clustering) have been considered. Ablation or comparison with other dictionary designs could help validate the generality of the framework.\n2.The recent work ESEG [1] focuses on event-only segmentation using explicit edge semantics. If its detection branch were replaced by RGB input, the pipeline would become highly similar to ESC. A clearer discussion of conceptual and methodological differences between the two works is warranted.\n3.In Appendix B, the DSEC-Semantic event input is built by sampling 50,000 events per sequence rather than using a fixed time window (e.g., 50ms). The rationale for this design choice is not clearly explained. A discussion of the implications and potential performance trade-offs would strengthen the methodological transparency.\n4.While the paper showcases under-exposure scenarios, ESC is proposed to handle a broader range of extreme conditions. Including examples under over-exposure or motion blur would better illustrate the full scope of robustness.\n5.The authors adopt MiT-B2 for RGB and MiT-B1 for event inputs. Combined with multiple attention operations, the overall FLOPs reach 95G, which may hinder deployment efficiency.\n[1] Zhao Y, Lyu G, Li K, et al. ESEG: Event-Based Segmentation Boosted by Explicit Edge-Semantic Guidance[C]//Proceedings of the AAAI Conference on Artificial Intelligence. 2025, 39(10): 10510-10518.", "was_revised": true, "final_justification": "I appreciate the authors' response, which has addressed most of my concerns regarding the experiments and differences bettween related works.", "rebuttal": "We thank the reviewer for the constructive feedback. The comments have been very helpful in guiding us to improve the manuscript. Please find our detailed responses below.\n\n---\n\n### **Framework Generality and Alternative Dictionary Strategies (W1)**\n\nWe appreciate the reviewer for the valuable suggestion. In our framework, VQ-VAE-based edge dictionary **not only provides a feasible edge encoding method, but also provides a feasible optimization mechanism for modality edge features**, deriving a series of reliable indicators based on confidences and uncertainties to guide the fusion of the heterogeneous event and RGB. The utilization of alternative encoding strategies, such as a prototype-based encoding method, will **have a significant change in the current training paradigm**, for example, the confidence indicator based on classification softmax logit cannot be obtained. This results in different edge encoding strategies not being able to be replaced losslessly in the framework.\n\nDespite this, we still do our best to **re-implement our framework with a simple prototype-based edge encoding method**. We pre-define 24 categories of edge prototypes, which represent 11 categories of semantic boundaries and one non-boundary category for each modality. The prototypes are updated every epoch based on the average feature vectors of each category. In the training and testing stages, the nearest neighbour look-up algorithm is used to match the prototypes with edge features at each spatial position, and the matched prototypes reconstitute the re-coded edge embeddings $\\textbf{Γ}^\\mathcal{I}$ and $\\textbf{Γ} ^\\mathcal{E}$. \n\nAs shown in Table R6, for the prototype-based edge encoding method, the mIoU on DERS-XS drops by 1.20%, on DSEC-Semantic by 0.12%, and on DSEC-Xtrm by 0.36%, compared to the original VQ-VAE-based edge encoding method. These results suggest that **our framework has generality in different edge encoding designs**, while the VQ-VAE-based edge dictionary is uniquely designed in our proposed framework after careful consideration.\n\n**Table R6: Ablation study on prototype-based edge encoding method and VQ-VAE-based edge encoding method.**\n\n|                            | gACC(%)↑ | mACC(%)↑ | mIoU(%)↑ |\n| -------------------------- | -------- | -------- | -------- |\n| ESC on DERS-XS (prototype) | 92.91    | 74.07    | 65.90    |\n| ESC on DERS-XS             | 93.27    | 75.26    | 67.10    |\n| ESC on DSEC-S. (prototype) | 94.79    | 78.84    | 70.91    |\n| ESC on DSEC-S.             | 94.85    | 78.61    | 71.04    |\n| ESC on DSEC-X. (prototype) | 87.98    | 58.17    | 50.50    |\n| ESC on DSEC-X.             | 88.18    | 59.45    | 50.87    |\n\n---\n\n### **Differences Between ESC and ESEG (W2)**\n\nWe appreciate the reviewer for suggesting this paper, and we would like to discuss and clarify any discrepancies between ESEG and our ESC. \n\nBoth ESEG and our ESC utilize edge information to guide the learning process; however, there are several main discrepancies as follows. \n\n(a) **Different tasks.** ESEG is a uni-modality method that relies solely on event input, aiming to **approximate the segmentation performance typically obtained from RGB data**. Our ESC is a multi-modality semantic segmentation method with event and RGB as inputs, which mainly focuses on **the inferior optimization issues in modality imbalance and failure situations**. For this reason, it is acceptable to use RGB-based pseudo-labels for ESEG, while our ESC needs more accurate labels for assessment. \n\n(b) **Different motivations and learning objectives of the utilization of the edge.** ESEG utilizes edge-semantic messages to **explicitly inform the interested region** to the model as guidance, thus ESEG structure introduces direct edge supervision with different semantics of the objects. Our ESC utilizes edge information as an intermediate commonality for heterogeneous event and RGB, which **realigns event and RGB into a unified semantic space and jointly optimizes them** based on the confidence and uncertainty indicators derived from latent edge distributions. \n\n(c) **Different pipelines even if ESEG's detection branch replaced by RGB input.** For ESEG, if the dense-semantic branch is replaced by RGB input, the entire pipeline will become that events only represent edge semantics, and RGB only represent dense semantics. However, **our assumption is that event and RGB both contain edge semantic information**. Therefore, our pipeline realigns event and RGB into the unified semantic space to jointly optimize their edge semantics, with implicit edge information as a crucial clue.\n\nAlthough this work differs from ours in many ways, it is still a key message for us. We are grateful to the reviewer for his keen observation, and we will discuss and cite this work in our related work section with the above analysis.\n\n---\n\n### **Rationale Behind Event Sampling Strategy (W3)**\n\nThe DSEC-Semantic event input is built by sampling a fixed number of events per voxel grid rather than a fixed time window, which **follows the setting of ESS [34]**. In ESS, the event input is built with 100,000 events per voxel grid. We found that a 50-ms fixed time window or 100,000 fixed number of events is relatively large, which reduces the performance of data preprocessing, and may lead to insufficient edge characteristic representation. After the above trade-offs, we decided to use 50,000 events per voxel grid for DSEC-Semantic as our event sampling strategy in our work.\n\nTo further demonstrate the impact of different event sampling strategies, we conduct experiments on DSEC-Semantic with a fixed time window of 50 ms, a fixed number of events of 100,000 per voxel grid, compared with 50,000 events per voxel grid in the main paper. As shown in Table R6, different event sampling strategies present **comparable results**, with mIoU slightly lower (0.20% and 0.19% respectively) than the fixed number of events of 50,000 in the main paper.\n\n**Table R7: Ablation study on different event sampling strategies.**\n\n|                                 | gACC(%)↑ | mACC(%)↑ | mIoU(%)↑ |\n| ------------------------------- | -------- | -------- | -------- |\n| ESC on DSEC-S. (50 ms)          | 94.76    | 78.44    | 70.83    |\n| ESC on DSEC-S. (100,000 events) | 94.87    | 78.00    | 70.84    |\n| ESC on DSEC-S. (50,000 events)  | 94.85    | 78.61    | 71.04    |\n\n---\n\n### **Data of Broader Range Extreme Conditions (W4)**\n\nGiven the specificity of this field and the high cost of precise annotation, there is currently **a lack of publicly available** event-RGB semantic segmentation benchmarks that feature a broader range of extreme real-world conditions with true labels. Existing datasets such as DDD17 and DSEC-Semantic primarily rely on RGB-based pseudo labels and do not explicitly differentiate between different conditions.\n\nTo address this gap, we are currently dedicated to collecting a set of larger-scale, more diverse, and truly challenging real-world scenes data as our future work, which will include challenging scenarios such as underexposure, overexposure, motion blur, and lens flare. Nevertheless, we stress that the datasets employed in our current study are **carefully designed and sufficient to demonstrate the core effectiveness and robustness** of our method.\n\n---\n\n### **Deployment Efficiency and FLOPs Ablation (W5)**\n\nWe appreciate the reviewer's concerns regarding the deployment efficiency. We address this from both a hardware evaluation and an algorithmic ablation perspective.\n\nHardware-wise, we report the end-to-end inference latency, throughput, and peak memory usage of NVIDIA GeForce RTX 3090 GPUs and an AMD EPYC 7642 48-Core CPU, with a fixed input resolution of 512 × 512. As shown in Table R4, our framework achieves **favorable deployment efficiency** on standard hardware.\n\n**Table R4: Inference latency, throughput, and peak memory usage of GPU and CPU.**\n\n| #    | Inference latency | Inference throughput | peak memory usage (GPU) | peak memory usage (CPU) |\n| ---- | ----------------- | -------------------- | ----------------------- | ----------------------- |\n| 1    | 34.56 ms          | 84.26 samples/sec    | 12169.33 MB             | 1903.26 MB              |\n| 2    | 34.46 ms          | 84.17 samples/sec    | 12169.33 MB             | 1901.64 MB              |\n| 3    | 34.78 ms          | 84.35 samples/sec    | 12169.33 MB             | 1879.25 MB              |\n| Avg. | 34.60 ms          | 84.26 samples/sec    | 12169.33 MB             | 1894.72 MB              |\n\nAlgorithm-wise, we further evaluate our framework with lighter backbones (2×MiT-B0). As shown in Table R10-A, even with FLOPs lower than CMNeXt, the reduced ESC variant achieves **better performance** on DSEC-Xtrm compared to CMNeXt **(49.06% vs. 45.16% mIoU)** and a **significantly lower parameter count**. Moreover, as shown in Table R10-B, the average inference latency of the reduced variant is also **slightly lower** than CMNeXt **(29.37 ms vs. 29.79 ms)**. These results highlight the effectiveness of backbone scaling in achieving a favorable balance between deployment efficiency and performance.\n\n**Table R10-A: Ablation study on lighter backbones on DSEC-Xtrm.**\n\n||gACC(%)↑|mACC(%)↑|mIoU(%)↑|#Params(M)|FLOPs(G)|\n|-|-|-|-|-|-|\n|CMNeXt|87.04|52.12|45.16|58.687|62.805|\n|ESC (Reduced)|88.03|56.31|49.06|14.184|60.658|\n|ESC (Standard)|88.18|59.45|50.87|56.875|95.086|\n\n**Table R10-B: Inference Latency comparison on lighter backbones.**\n\n||Latency #1(ms)|Latency #2(ms)|Latency #3(ms)|Avg. Latency(ms)|\n|-|-|-|-|-|\n|CMNeXt|29.75|29.78|29.83|29.79|\n|ESC (Reduced)|29.30|29.38|29.43|29.37|\n|ESC (Standard)|34.56|34.46|34.78|34.60|"}, {"review_id": "EXF0ksUFeg", "reviewer": "Reviewer_tfrK", "summary": "This paper proposes an \"Edge-awareness Semantic Concordance (ESC)\" framework for event-RGB semantic segmentation, specifically targeting extreme conditions like low light and fierce camera motion where traditional RGB-only methods struggle due to information loss. To facilitate reliable evaluation in extreme scenarios, the authors introduce two synthetic (DERS-XS, DSEC-Xtrm) and one real-world (DERS-XR) event-RGB semantic segmentation datasets. Experimental results demonstrate that ESC outperforms state-of-the-art methods.", "questions": "1. The authors need to provide a clearer explanation or intuition for how the learnable noise embeddings contribute to improved fitting and stability.\n2. The use of extremely dark RGB inputs may overly emphasize the advantage of the event modality. It’s better to evaluate on more balanced extreme-condition datasets like DSEC-Night to ensure fairer assessment.\n3. The method shows higher FLOPs compared to CMNeXt. The authors need to test the approach with a lighter backbone to better isolate the impact of the proposed components from increased computational cost?", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. The proposed framework is well-structured with distinct, logical modules (ELR, RC, UO) that each contribute to the overall goal of resilient fusion.\n2. The paper introduces a novel approach to multi-modal fusion by leveraging semantic edges as an intermediate commonality.\n3. The experimental results clearly show improved SOTA performance.\n\nWeaknesses:\n1. Unexplained Learnable Noise Embeddings: The paper states that \"learnable noise embeddings\" are used to \"improve fitting ability and enhance learning stability\"  within RC and UO. However, it fails to provide a clear explanation of how these embeddings achieve this, or to cite prior work. There is also a lack of corresponding in ablations.\n2. Bias towards Extreme RGB Degradation: A significant portion of the evaluation appears to deliberately diminish the contribution of the RGB modality, particularly through the use of extremely low-light frames in both synthetic datasets and real-world data. While the goal is to highlight the event modality's role, such severe consistent RGB degradation might not always reflect the real-world extreme conditions. Testing on existing real-world extreme datasets like DSEC-Night (proposed by CMDA [40]) or collecting a more diverse set of truly challenging real-world scenarios would provide a more comprehensive and less-biased validation.\n3. Limited Efficacy in Non-Extreme Real-World Conditions: The proposed method appears to demonstrate its most significant advantages primarily when the RGB modality is severely degraded. For instance, on the real-world DSEC-Semantic dataset (Table 1), which represents more typical, non-extreme conditions, the performance gain over CMNeXt is only marginal (a 2.01% mIoU improvement). \n4. Limited DERS-XR: While DERS-XR is introduced as a real-world dataset for \"extreme conditions\", the qualitative examples shown in Figure 14 suggest that the RGB images are severely degraded, appearing almost completely dark. This level of degradation might deliberately diminish the RGB signal. Furthermore, with only 120 frames used for fine-tuning and 120 for testing, the dataset size is significantly small. \n5. High Computational Cost: While Table 5 shows that the proposed method has a comparable number of parameters to CMNeXt, its FLOPs are significantly higher. This substantial increase in computational cost raises questions about the efficiency of the method. It is unclear whether the performance gains observed are primarily due to the architectural innovations or simply a result of a higher computational budget. A critical aspect to assess would be if the method could maintain its superior performance using a smaller backbone to achieve similar FLOPs and inference speed as CMNeXt.", "was_revised": true, "final_justification": "As my concerns have been resolved, I have decided to increase my score to 4", "rebuttal": "We appreciate the reviewer’s careful evaluation, including both the comprehensive review and the thoughtful critiques. The following responses aim to address the raised concerns in detail.\n\n---\n\n### **Clarification on Learnable Noise Embeddings (W1, Q1)**\n\nThank you for pointing this out. Here we clearly explain, from both technical and experimental aspects, how learnable noise embeddings are beneficial for improving fitting ability and enhancing learning stability.\n\n**Technical explanation:** \n\n**From an optimization perspective**, the learnable noise embeddings serve to enhance the model's fitting capacity and learning stability by **introducing additional degrees of freedom** into the cross-modality interaction process. This added flexibility allows the model to better adapt to heterogeneous modalities during learning.\n\n**From the viewpoint of the cross-attention mechanism**, in the absence of noise, the query (Q) tends to attend excessively to its own features in the key (K), thereby suppressing signals from the other modality and impeding effective fusion. Introducing noise embeddings mitigates this issue by **perturbing the attention space in a controlled, learnable manner**, encouraging richer and more balanced cross-modal interactions.\n\n**Conceptually**, this mechanism is inspired by **prompt-based learning**, where learnable prompts are used to guide model behavior. In our case, the noise embeddings act as fixed prompts that help regulate and stabilize the cross-modal attention patterns during training.\n\n**Experimental explanation:** \n\nWe conduct an ablation study by removing the learnable noise embeddings on three datasets: DERS-XS, DSEC-Semantic, and DSEC-Xtrm. As shown in Table R8, **the removal of noise embeddings leads to a 1.05% and 0.81% mIoU drop on DERS-XS and DSEC-Xtrm, respectively**, confirming their contribution to improved fitting ability and stability. The performance drop on DSEC-Semantic is minimal (0.18%), which we attribute to its reliance on RGB-based pseudo-labels. As the supervision signal is already biased towards RGB, the model naturally relies less on event modality. In such cases, the role of learnable noise embeddings in facilitating cross-modal interaction becomes less significant.\n\n**Table R8: Ablation study on learnable noise embeddings.**\n\n||gACC(%)↑|mACC(%)↑|mIoU(%)↑|\n|-|-|-|-|\n|ESC on DERS-XS (w/o noise embeddings)|93.12|73.41|66.05|\n|ESC on DERS-XS|93.27|75.26|67.10|\n|ESC on DSEC-S. (w/o noise embeddings)|94.69|79.45|70.86|\n|ESC on DSEC-S.|94.85|78.61|71.04|\n|ESC on DSEC-X. (w/o noise embeddings)|88.22|57.13|50.05|\n|ESC on DSEC-X.|88.18|59.45|50.87|\n\n---\n\n### **Justification of Challenging RGB and DSEC-Night Evaluation (W2, Q2)**\n\nWe would like to clarify that the use of dark or degraded RGB inputs is **intentional and consistent with our primary motivation**, that is, to facilitate resilient semantic segmentation by addressing the core challenge of inferior optimization issues of heterogeneous event and RGB in modality imbalance and failure situations. In this context, using challenging RGB conditions and applying spatial occlusions via local masking is not a source of artificial bias, but a deliberate design choice to evaluate the fusion capabilities under representative adverse scenarios.\n\nThat said, we fully understand the reviewer’s concern about broader real-world applicability. Notably, our method also achieves consistent gains on DSEC-Semantic, a dataset that represents more typical, non-extreme conditions. This demonstrates that the effectiveness of our approach is **not limited to extremely degraded RGB scenarios**.\n\nIn addition, to directly address the reviewer’s suggestion, we include a new evaluation on **DSEC-Night**, which was originally proposed for unsupervised cross-modality domain adaptation by CMDA. As this dataset only contains 150 annotated frames for testing, we adapt it to a supervised setting by splitting the labeled sequences into training and testing sets: 54 frames from `zurich_city_09_a` and `zurich_city_09_b` for training, and 96 frames from `zurich_city_09_c/d/e` for testing. All models are trained for 100 epochs with the same settings for fair comparison. As shown in Table R9, our method outperforms CMX by 6.55%, CMNeXt by 6.28% mIoU, and EISNet by 5.95%, demonstrating the strong efficacy of our method in real-world extreme conditions.\n\n**Table R9: Quantitative comparisons on DSEC-Night.**\n\n||gACC(%)↑|mACC(%)↑|mIoU(%)↑|\n|-|-|-|-|\n|CMX|77.38|43.24|32.85|\n|CMNeXt|77.38|43.62|33.12|\n|EISNet|77.87|44.95|33.46|\n|ESC (Ours)|81.68|48.55|39.40|\n\nLastly, we agree that a more comprehensive evaluation on diverse extreme conditions would be valuable. As part of our ongoing work, we are actively collecting a new, large-scale real-world dataset covering a broader range of challenging scenarios, including underexposure, overexposure, motion blur, and lens flare. Due to the high cost of manual annotation, this dataset could not be completed during the rebuttal period, but we plan to release it in future work.\n\n---\n\n### **Efficacy in Non-Extreme Real-World Conditions (W3)**\n\nWe **respectfully disagree with the characterization of the 2.01% mIoU improvement as marginal**. The significance of a performance gain should be evaluated in the context of task difficulty, dataset characteristics, and the strength of the baseline. Achieving a 2.01% improvement over a strong baseline like CMNeXt is non-trivial, particularly given that DSEC-Semantic represents non-extreme real-world conditions where existing methods already perform well and leave limited room for further improvement.\n\nFor reference, **CMNeXt itself reports only 0.2% and 0.6% mIoU improvements on the MFNet and NYU Depth V2**, respectively. In this context, our 2.01% improvement is both meaningful and practically relevant. Moreover, in real-world deployment scenarios, even modest increases in mIoU can lead to noticeably improved segmentation quality.\n\n---\n\n### **Justification of DERS-XR Dataset (W4)**\n\nWe would like to clarify that the DERS-XR dataset is constructed from real-world captures under naturally occurring extreme conditions. Our intention is not to deliberately diminish the RGB signal, but rather to **faithfully reflect real-world scenarios**. Although the RGB images appear severely degraded, they still retain essential semantic cues for segmentation.\n\nRegarding the dataset scale, we acknowledge that DERS-XR contains a limited number of frames. This limitation is primarily due to the high cost of accurate pixel-level annotation under extreme conditions. The original motivation behind DERS-XR is to provide a benchmark for evaluating model performance under real-world extreme conditions with **reliable ground truth**. Given the scarcity of such labeled data, **DERS-XR fulfills this purpose effectively despite its modest size**.\n\n---\n\n### **Efficiency Performance Trade-off with Lighter Backbones (W5, Q3)**\n\nTo address the reviewer's concern, we conduct **additional comparative experiments** on DERS-Xtrm using smaller backbones (2×MiT-B0), reducing the FLOPs of ESC to be even lower than CMNeXt. In addition, we measure the **end-to-end inference latency** of CMNeXt and our ESC (both standard ESC and reduced variant) on a single NVIDIA GeForce RTX 3090 GPU with a batch size of 1. All latency measurements are performed with a fixed input size of 512 × 512, with each measurement calculating the average execution time over 100 inferences, and we repeat 3 times for stability.\n\nAs shown in Table R10-A, the reduced ESC variant still **outperforms CMNeXt**, achieving 49.06% mIoU vs. 45.16%, despite lower FLOPs (60.658G vs. 62.805G) and significantly fewer parameters (14.184M vs. 58.687M). This suggests that the performance gains stem from architectural design rather than merely an increased computational cost. Furthermore, as shown in Table R10-B, the reduced ESC variant has an average inference latency of 29.37 ms, which is **shorter than**  that of CMNeXt (29.79 ms), demonstrating its potential for more efficient deployment.\n\nAbove results indicate that even with lightweight backbones, our model maintains strong performance with higher inference speed, **highlighting the effectiveness of our design beyond raw FLOPs**. This reflects a favorable trade-off between efficiency and performance, which is essential for practical deployment in real-world systems.\n\nWe hope these additional results and analyses adequately address the reviewer’s concern.\n\n**Table R10-A: Ablation study on lighter backbones on DSEC-Xtrm.**\n\n||gACC(%)↑|mACC(%)↑|mIoU(%)↑|#Params(M)|FLOPs(G)|\n|-|-|-|-|-|-|\n|CMNeXt|87.04|52.12|45.16|58.687|62.805|\n|ESC (Reduced)|88.03|56.31|49.06|14.184|60.658|\n|ESC (Standard)|88.18|59.45|50.87|56.875|95.086|\n\n**Table R10-B: Inference Latency comparison on lighter backbones.**\n\n||Latency #1(ms)|Latency #2(ms)|Latency #3(ms)|Avg. Latency(ms)|\n|-|-|-|-|-|\n|CMNeXt|29.75|29.78|29.83|29.79|\n|ESC (Reduced)|29.30|29.38|29.43|29.37|\n|ESC (Standard)|34.56|34.46|34.78|34.60|"}]

Author rebuttal:

Meta-review: This paper tackles the problem of semantic segmentation in extreme conditions using a novel approach that fuses heterogeneous Event and RGB inputs. The authors introduce the Edge-awareness Semantic Concordance (ESC) framework, which uses semantic edges as a bridge to align features from both modalities within a unified semantic space. The framework's key components are the Edge-awareness Latent Re-coding (ELR), Re-coded Consolidation (RC), and Uncertainty Optimization (UO) modules. To facilitate robust evaluation, the authors also contribute three new benchmark datasets specifically designed for degraded visual conditions. After the rebuttal, the paper received scores of 5, 4, 4, and 3. Importantly, reviewer BVgg, who initially provided the score of 3, did not submit a Final Justification, but explicitly stated in their rebuttal that they "will support this article." Given this positive shift in sentiment,  this paper actually got an average score 4.25, which is around the borderline. The ACs recommended acceptance for this paper. The ACs recommend that the authors integrate their rebuttal into the final version of the paper to further enhance its overall quality. Additionally, the authors should supplement appropriate citations where needed, for example, the discussion of noise embedding in section 3.4 should reference the work geminifusion.
Decision: Accept (poster)