Rethinking RGB-Event Semantic Segmentation with a Novel Bidirectional
  Motion-enhanced Event Representation
Authors: name: Zhen Yao, name: Xiaowen Ying, name: Mooi Choo Chuah

Event cameras capture motion dynamics, offering a unique modality with great
potential in various computer vision tasks. However, RGB-Event fusion faces
three intrinsic misalignments: (i) temporal, (ii) spatial, and (iii) modal
misalignment. Existing voxel grid representations neglect temporal correlations
between consecutive event windows, and their formulation with simple
accumulation of asynchronous and sparse events is incompatible with the
synchronous and dense nature of RGB modality. To tackle these challenges, we
propose a novel event representation, Motion-enhanced Event Tensor (MET), which
transforms sparse event voxels into a dense and temporally coherent form by
leveraging dense optical flows and event temporal features. In addition, we
introduce a Frequency-aware Bidirectional Flow Aggregation Module (BFAM) and a
Temporal Fusion Module (TFM). BFAM leverages the frequency domain and MET to
mitigate modal misalignment, while bidirectional flow aggregation and temporal
fusion mechanisms resolve spatiotemporal misalignment. Experimental results on
two large-scale datasets demonstrate that our framework significantly
outperforms state-of-the-art RGB-Event semantic segmentation approaches. Our
code is available at: https://github.com/zyaocoder/BRENet.