LaSe-E2V: Towards Language-guided Semantic-aware Event-to-Video Reconstruction
Authors: Kanghao Chen
Hangyu Li
Jiazhou Zhou
Zeyu Wang
Lin Wang
Venue: NeurIPS 2024 poster (2024)
Keywords: event camera
video reconstruction
diffusion model
Primary area: machine_vision

Abstract: Event cameras harness advantages such as low latency, high temporal resolution, and high dynamic range (HDR), compared to standard cameras. Due to the distinct imaging paradigm shift, a dominant line of research focuses on event-to-video (E2V) reconstruction to bridge event-based and standard computer vision. However, this task remains challenging due to its inherently ill-posed nature: event cameras only detect the edge and motion information locally. Consequently, the reconstructed videos are often plagued by artifacts and regional blur, primarily caused by the ambiguous semantics of event data. In this paper, we find language naturally conveys abundant semantic information, rendering it stunningly superior in ensuring semantic consistency for E2V reconstruction. Accordingly, we propose a novel framework, called LaSe-E2V,  that can achieve semantic-aware high-quality E2V reconstruction from a language-guided perspective, buttressed by the text-conditional diffusion models. However, due to diffusion models' inherent diversity and randomness, it is hardly possible to directly apply them to achieve spatial and temporal consistency for E2V reconstruction. Thus, we first propose an Event-guided Spatiotemporal Attention (ESA) module to condition the event data to the denoising pipeline effectively. We then introduce an event-aware mask loss to ensure temporal coherence and a noise initialization strategy to enhance spatial consistency. Given the absence of event-text-video paired data, we aggregate existing E2V datasets and generate textual descriptions using the tagging models for training and evaluation. Extensive experiments on three datasets covering diverse challenging scenarios (e.g., fast motion, low light) demonstrate the superiority of our method. Demo videos for the results are attached to the project page.

LaSe-E2V: Towards Language-guided Semantic-aware Event-to-Video Reconstruction
Kanghao Chen'
Jiazhou Zhou' AT Thrust, 2CM Thrust, HKUST(GZZ)
Hangyu Li'
Zeyu Wang Lin Wang
Dept. of CSE, HKUST
kchen8798conneect. hkush- gz edu, en, linwangdust.hr
Project Page: https://vlislwe
Abstract
Event cameras harness advantages such as low latency, high temporal resolution, and high dynamic range (HDR). compared to standard cameras. Due to the distinct imaging paradigm shift, dominant line of research focuses on event-to-video (E2V) reconstruction to bridge event-based and standard computer vision. How- ever, this task remains challenging due to its inherently ill-posed nature: event cameras only detect the edge and motion information locally. Consequently, the reconstructed videos are often plagued by artifacts and regional blur, primarily caused by the ambiguous semantics of event data. this paper, we find language naturally conveys abundant semantic information, rendering stunningly supe- rior in ensuring semantic consistency for E2V reconstruction. Accordingly, we propose novel framework, called LaSe-E2V. that achieve semantic-awan high-quality E2V reconstruction from a language-guides perspective, buttressed by the text-conditional diffusion models. However, due to diffusion models' inherent diversity and randomness, it is hardly possible to directly apply them to achieve spatial and temporal consistency for E2V reconstruction. Thus, first propose Event-guided Spatiotemporal Attention (ESA) module to condition the event data the denoising pipeline effectively. We then introduce event-aware mask loss to ensure temporal coherence and a noise initialization strategy to enhance spatial consistency. Given the absence of event-text tent-deex-- paired data, we aggregate exist- ing E2V datasets and generate textual descriptions using the tagging models for training and evaluation. Extensive experiments on three datasets covering diverse challenging scenarios (e.g., fast motion, low light) demonstrate the superiority of our method. Demo videos for the results are attached to the project page.
1
Introduction
Event cameras are bio-inspired sensors that detect per-pixel intensity changes, producing asyn- chronous event streams [5] with high dynamic range (HDR) and high temporal resolution. They particularly excel capturing the edge information of moving objects, thus discarding the redundant visual information. Such a distinct imaging shift poses challenges for integration with the off off-the-shelf vision algorithms designed for standard cameras. To bridge the event-based and standard computer vision [27, 31, 3, 29], promising way is event-to-video (E2V) reconstruction.
Recently, deep learning has been applied to E2V reconstruction [49, 54, 64], and remarkable perfor- mance is achieved thanks to the availability of synthetic event-video datasets and the development of model architectures. Most existing research emphasizes on the network design [64, 13. 54, 34] or high-quality data synthesis [57, 34]. However, this task remains challenging due to its inherently ill ill-posed nature: event cameras capture edge and motion information locally but neglect semantic
*Corresponding author
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

(a)
"ww persons rislngg through an obstaele course dirt ftels!..."
Aggregated event frames
Event stream
HyperE2VID
LaSe-E2V(Ours)
Semantic ambiguity in structures
Enriched semantic information
Figure I: Comparison of the E2V pipeline between HyperE2VID [13] and LaSe-E2V: The baseline method solely relies on event data, leading to ambiguity local structures. contrast, our approach integrates language descriptions to enrich the semantic information and ensure the video remains coherent with the event stream.
information in regions with no intensity changes (See Fig. (a)). Consequently, the reconstructed videos, e.g., HyperE2VID [13], plagued by artifacts and regional blur, shown in Fig. (b).
To fill this gap, in this paper, we find language naturally conveys abundant semantic information, which is beneficial in enhancing the semantic consistency for the reconstructed video, shown Fig. (c). Intuitively, we propose novel geguuage-uided semantic- aware E2V reconstruction framework, called LaSe-E2V, with the text-conditional diffusion model [52] as the backbone. While many efforts [62, 51, 23] apply diffusion models for images and video generation, adapting them to problem is hardly possible. The reason is that the inherent randomness in diffusion sampling and the diversity objectives generative models may result temporal inconsistency between consecutive frames and spatial inconsistency between the event data and reconstructed videos.
Therefore, we first introduce an Event-guided Spatio-temporal Attention (ESA) module enhance reconstructed video by introducing spatial and temporal event-driven attention layers, respectively (Sec. 3.2). This module not only achieves fine-grained spatial alignment between the events and video (See Fig. (right)) but also maintains temporal smoothness and coherence by adhering to the temporal properties of event cameras. We then introduce an event-aware mask loss to maintain temporal coherence throughout the video by considering the spatial constraints of event data from adjacent frames (Sec. 3.3). Lastly, we propose noise initialization strategy that utilizes accumulated event frames to provide layout guidance and reduce discrepancies between the training and inference stages of the denoising process (Sec. 3.4).
Given the absence of event-text-video paired data, we aggregate existing E2V datasets and employ tagging models to generate textual descriptions, thereby facilitating both training and evaluation processes. Extensive experiments on three widely used real-world datasets demonstrate the superiority of our method in enhancing the quality and visual effects of reconstructed videos, especially its super generalization ability when applied in challenging scenarios, e.g., fast motion, and low light (See Fig. 4 and Fig. 5). In summary, the contributions of our work are three-fold: (I) We explore E2V reconstruction from language-guider perspective, utilizing the text-conditioned diffusion model to effectively address the semantic ambiguities inherent event data. (II) We propose the event guiddd spatio-temporal attention mechanism, event-aware mask loss, and noise initialization strategy to ensure the semantic consistency and apatio-tempore coherency of the reconstructed video. (III) We have rebuilt event-text-videe paired datasets based on existing event datasets with textual descriptions generated from off-the-shelf models [73]. Extensive experiments on three datasets covering diverse scenarios (e.g., fast motion, low light) demonstrate the effectiveness of our framework.
2 Related Works
Event-to-Videe (E2V) Reconstruction. E2V reconstruction falls into two categories: model- based and learning-based methods. Model-based approaches [2, 44, 6. 53] exploit the correlation

---

between events and intensity frames through hand-crafted regularization techniques. However, its reconstruction result comparatively inferior the more recent learning- -based methods [50, 49, 54, 34, 13]. For example, E2VID [50, 49] used an Unet-like network with skip connections and ConvL.STM units to reconstruct videos from long event streams. Following E2VID, SPADE- E2VID [7] extended E2VID to enhance temporal coherence by feeding previously reconstructed frames into a SPADE block. HyperE2VID [13] introduced hypernetworks to generate per-pixel adaptive filters guided by context fusion module. GANs, such as conditional GAN [61] and cycle- consistency GAN [69], are used to address this issue, but often exhibit blurry images with artifacts in antivititted areas. In summary, previous learning- -ased methods can only achieve plausible reconstructed results for this ill- ilo-posed problem because event data solely captures motion information without semantic context information. In this work, explore the possibility of incorporating textual descriptions with semantic awareness for E2V reconstruction.
Texteto-Video Diffusion Models. The success of Diffusion models [45, 52] in generating high- quality images from text prompts has advanced text-to-image (T21) synthesis. Inspired by T2I synthesis, text-to-video (T2V) diffusson models, such [4, 19, 20, 23, 30, 40, 55, 60, 66] adapt T2I synthesis to video. Subsequent developments [23, 4, 18] focused on refining the temporal information interaction, such as optimizing temporal convolution or self-attention modules for motion learning. Meanwhile, one T2V research line focuses on enhancing controllability by incorporating additional conditions, thus addressing the text prompts' ambiguity in motion, content, and spatial structure. For high-level video motion control, studies propose learning LoRA [24] layers for specific motion patterns [18, 75], or utilizing extracted trajectories [68], motion vectors [62], pose sequences [41] to guide the synthesis. For fine-grained spatial structure control, methods like Gen- 1 [14], VideoComposer [62], and others [42, 70] leverage monoeular depth, sketch, and image control models [42, 70] for flexible and controllable video generation [8, 18, 30, 72].
Formally, given a video sample zo. the latent diffusion model (LDM) [52] first encodes it into latent feature 20- = (20). noisy input obtained based on the forward diffusion process by introducing Gaussian noise to the latent representation: ž Varzu + vI agt, where -= 1, ..., T and T denotes the maximum timestep. O₁ II B&) is the coefficient that controls the noise strength. The diffusion model is trained by predicting the noise c through a mean squared error:
4 = ||e c0(2,, 11l
where 0 denotes the parameters of the U-Net of the diffusion model. denotes the addition conditions (text, image, depth, el al.) to control the diffusion process. the inference stage, the generated sample 20 can be obtained from the denoised latent &, using pre-trained decoder 20 D2(2))
These methods achieve fine-grained controllability based on the well-designed condition c, focusing on flexible user instructions and diverse outcomes. In this study, we employ a basic T2V diffusion model with the 3D-UNet [60] architecture as the foundation for language-guil -oguuaee-guide E2V reconstruction. In particular, our framework prioritizes video fidelity by aligning motion details from event data and semantic insights from text, thus outperforming previous models, especially in extreme scenarios like fast motion and low light.
3 The Proposed LaSe-E2V Framework
Event Representation. An event stream & = {ee, Nh consists of N₂ events. Each event et, E E is represented as tuple (Ik, yk. tk, Pk)), which denotes the pixel coordinates, timestamp, and polarity. To make the event stream compatible with the pre-trained diffusion model, we convert & into grid-like event voxel grid V BxX HxW with B time bins using temporal bilinear interpolation [77].
3.1 Overall Pipeline
The goal of our LaSe-E2V is to reconstruct video = { (E.,,,?...,,' E NXCxHXx from the set of event segments { where E is i-th event segment corresponding to the frames. The reconstructed video is expected to retain the motion and edge content from the event data and compensate for the semantic information from the language description. The difficulty of our task lies not only in achieving high visual quality that aligns with the text descriptions but also in maintaining content consistency with the event data. We address the difficulty by integrating the diffusion model
3

---

Fro Frozen we gitls
Text descriation
CLIP text Encoder
LDM
notse
add noice
D
Concat
E
2D
PE PE
2D Attention
3D Attention
x
Spatial Attention
Text
Temporol Attention
ResBlock2D
Event Pre-Processing
Spatial Self-att
Event Spatial
Cross-att atss-aatt
Cross-art att
Temporal Self-att
Event Temporal
Cross-at att
E
Ze
Text
ResBlock3D
Event Maek
Raw
Event Voxel Grid
AttBlock
(
Binerization
Event Encoder E.
Event-aware Mask Loss Im (Eg. 4)
Figure 2: An overview of our proposed LaSe-E2V framework.
with events and text descriptions effectively. As shown in Fig. 2, LaSe-E2V consists of four major components: an image encoder E,, an event encoder Eg, video LDM, and a decoder D,.
Image Encoder E₁. Given a sequence of frames x {x² _.,,..,, x }. we extract the latent representation {2), { ....,""" ,?......."" > with the pre-trained image encoder of LDM [52].
Event Encoder E.. To adapt the event voxel grids to the latent space of E,, we first project into latent representation with an event encoder E,,( (V).. As illustrated in Fig. 2 (a), initially apply lightweight spatial 21) and temporal 3D convolution blocks to the set input voxel (V'), with N segments. These blocks are designed to extract local spatial and temporal feature information, which is then processed through an attention block (AtrBoock) for global temporal modeling. Subse- quently, we concatenate the event latent representation with the noise latent representation along the channel dimension to serve as a rough condition for the model. Finally, we construct the input latent N + xw' where z and 2 denote the noise latent and event feature corresponding to the frame with channel dimensions C² and Ce.
representation as x = {[æ]. 1
Video LDM. We first extend the capabilities of the text-conditional image-based LDM by incorporat- ing temporal layers in the U-Net, following the approach of video diffusion models [14, 23, 55, 62]. As depicted in Fig. 2, our framework presents a video LDM with multiple blocks consisting of 2D spatial convolution, 3D temporal convolution, 2D spatial attention, and 3D temporal attention layers. A pre-trained CLIP [48] text encoder is employed to extract text descriptions conditioning on the attention layers provide semantic information for the reconstructed video. To facilitate fine-grained control of event data, we propose an Event-guuded Spatio-temporal Attention (ESA) module (see Fig. 2 (b) and (c)) following E, convert the event voxel grids to the latent space. The ESA module enhances the control by introducing specific spatial and temporal cross-attenlie in the U-Net. The technical details will be discussed in Sec. 3.2.
Decoder D,. Finally, we apply the pre-trained decoder D, of [52] to convert the estimated latent representation à to video & in the image space.
Event-aware Mask Loss Im To optimize LaSe-E2V, we propose a novel event-aware mask loss Inx in addition to the e-prediction loss (Eq. 1). The details of L., will be described in Sec. 3.3.
In addition, different from previous LDM models [10, 51]. we introduce a noise initialization strategy to alleviate the train-test gap in the inference stage, which will be described in Sec. 3.4.
3.2 Event-guided
Attention (ESA)
Relying solely on the 3D U-Net and feature concatenation is insufficient to reconstruct content- consistent video with fine-grained control from event data, as demonstrated experimentally in Fig. (right). To address this, we propose using the event latent representation condition spatial and temporal attention, ensuring more reliable spatial alignment and temporal consistency (See Tab. 4). Our ESA module is specially designed to enhance the spatio-temporal consistency between events

---

and video in contrast to the original SD [52], which serves as a baseline attention mechanism for integrating conditional input. Our approach differs in attention design. which introduces two distinct attention mechanisms respective to the spatial domain and temporal domain.
Event Spatial Cross-Attention. As illustrated in Fig. 2 (b), the 2D spatial attention layer in the LDM U-Net integrates self-attention mechanism that processes each frame individually and a cross- attention mechanism connecting frames with text embedding, which follows Stable Diffusion [52]. Intuitively, Event data naturally deliver abundant edge information in the spatial space, which is expected to directly constrain the structural information of the reconstructed video. To incorporate event information in the spatial attention module, we concatenate the features from the event latent za. the U-Net intermediate features formulate a event-based spatial attention:
QK?
Zout = Softmar
V Ve-si
(2)
Vd
where W Ke_,- WK 21 Ve-,= [' 2 [+], represents the concatenation
operation on the spatial dimension. This modification ensures that each spatial position in all frames accesses comprehensive information from the corresponding event data, enabling detailed structural
feature control within the spatial attention layers.
Event Temporal Cross-Attention. In addition to the spatial information, event data inherently represents the difference between the adjacent frames, which is a strong constraint. As shown in Fig. 2 (c), to effectively leverage the temporal constraint of the event data, facilitate event features into temporal attention. Specifically, given the intermediate features 2, we first reshape the height and width dimensions into the batch dimension, forming a new hidden state z E R xXxXXxC Then the event latent feature z is employed to interact with the latent feature in the temporal dimension through the temporal attention layer:
QK Vd
Pout Softmax
Ve-ti
where Q
[ Ve- [" 2.]e. [t], represents the concatenation
operation on the temporal dimension.
Previous Frame Conditioning. To ensure consistency for the whole video reconstruction. we an autoregressive way to reconstruct the video, by conditioning the model on the last frame N.N of the previously estimated video clip X. We concatenate the previous frame with the latent representation & on the temporal dimension
condition obtain [ zz] (N++))xC' x H' xW" Due the absence of the previous frame in the first video clip, we randomly drop the previous frame condition by applying Gaussian noise to the frame latent representation. During inference, the first video clip is reconstructed directly from the events and text data without the previous frame. For subsequent video clips, the reconstruction process utilizes the last frame from the previous video clip condition to ensure temporal coherency and smoothness.
3.3 Event-aware Mask Loss
Considering the noise prediction loss can not capture the event constraint on temporal coherence, we introduce an event-aware mask loss to directly supervise the inter-frame difference:
4m = ||-
M) MM
(4)
where M = I(V) indicates the event motion mask obtained by setting value one for event activated area and value zero for non-activated part; 2μ represents the model's estimated clean video latent, which can be obtained by:
2t
v1 anco
V0g
Finally, we combine the noise prediction loss and the event-aware mask loss with the scaling factor X.
1 1 1. + X mm
(6)
5

---

Table 1: Quantitative comparison with state-of-the-art methods on both synthetic and real-world benchmarks. The best and second best results of each metric are highlighted in red and blue, respectively. To align the metric of SSIM, we re-evaluate the previous methods based on their pre-trained models to obtain SSIM".
Dutsess
Matrics
EEVID [50]
[54]
EIVID+ [57]
FireNet- [57]
SPADE-EIVID [7]
SSL-EEVID [46]
ET-Ne: HyperE2VID [13]
La (Ours)
MSEL
0.212
0,131
01070
0.063
0,091
0.046
0.047
0.033
0,823
55IMT
0.502
0.555
0.517
0.364
0.655
DCD
SSIM
0,450
0.490
0.503
0452
0,461
0,415
0.552
0.526
0.629
LPIPS]
0.350
0.320
0.235
0.290
0.337
0.425
0.224
0.212
0.194
MSE]
0.337
0.292
0.152
02118
0,138
0.062
0.107
0.076
0,055
MVSRC
SSIM1
0.206
0.261
0.345
0.297
0.342
0.345
0.300
0:419
0.212
0.266
0.264
0.315
LPIPS]
0.705
0.780
0.514
0.570
0.589
0.993
0.489
0:476
0.461
0.040
0,077
HQF
SSIM]
0.540
0,533
0.643
0.614
0,521
0.295
0.658
0.658
SSIM
0,405
LPIPS]
0,382
0.441
0.253
0.314
0,502
0,498
0.260
0,257
0.254
3.4 Event-aware Noise Initialization
During training, we construct the input latent by adding noise on the clean video latent. The noise schedule leaves some residual signal even at the terminal diffusion timestep T. As result, the diffusion model has a domain gap to generalize the video during the inference time when we sample from random Gaussian noise without any real data signal. To solve this train-test discrepancy problem, during the testing, we obtain the base noise by adding noise on event-accumulated frames using the forward process of DDPM [21]. The noise latent for frame can be expressed as:
2/-- V 2 + VI aTE'
(7)
where aT denotes the diffusion factor and 2 E, (1-1 (1-1+ e') The 1-1 is the last frame of the previous estimated video clip. Intuitively, accumulated event data provides structural information (e.g., edges) in the scene, acting as an additional spatial constraint during the denoising process.
4
Experiments
4.1 Datasets and Implementation Details
Dataset. We train our pipeline using both synthetic and real-world datasets. For synthetic data, following prior arts [50, 34), we generate event and video sequences from the MS-COCO dataset [33] using the v2e [25] event simulator because it ensures stable and high-quality ground truth images. To enrich semantic information, we also utilize the real-world dataset BS-ERGB [58], which includes lk training sequences. For all datasets, we employ the off-the-shelf tagging model RAM [73] to generate language descriptions. Recent work [65] has demonstrated the superiority of RAM compared to other prompting models due to its rich objects and concise description. We evaluate our model on Event Camera Dataset (ECD) [43], Multi Vehicle Stereo Event Camera (MVSEC) dataset [76] and High-Quality Frames (HQF) dataset [57]. RAM is also used to generate tags for these datasets.
Implementation Details. Based on Stable Diffusion 2. bbsee [52], we use a text-guided video diffusion model [51] to initialize our model, pre-trained on large-scale video datasets [1]. For each training video clip, we sample 16 frames and the corresponding event streams, with an interval of 1 I e < 3 frames. The input size is adapted to 256 x 256. Following previous methods [64, 57], the data augmentation strategies include Gaussian noise, random flipping, and random pause. The value À is set to 0.01 for all experiments. The model is trained with the proposed loss across all U-Net parameters, with a batch size of 3 and a learning rate of 5e-5 for 150k steps on 8 NVIDIA V100 GPUs. We also provide an analysis of the computation complexity. See more details in Appendix 5.
Evaluation Metric. The Mean Squared Error (MSE), Structural Similarity (SSIM [63]), and Perceptual Similarity (LPIPS [71]) are used to measure image quality. The SSIM metric raises ambiguity as it involves several hyperparameters that may differ across various codebases. For this reason, reevaluated all compared methods using unified metric, denoted as the SSIM" scores.

---

Mutii be weees engineer
alss be
EVVID
our mr nds.
gineer
our m
aiss De
ginner ing eial
EZVID
qui in
gineer
our m
ams bet
gignne
FireNete
gineen
QUr
MODE bet
aiS Det
gignee ng
ET-Na
niww Takes
gineer
Bell
AIS De
Hype
Br Brgline 1 Busl
gineen
our
Greand Lasc-aa
SIEE DEH
er gineering toe
gineen
our
our
bet neen
aNS Det
ar gineering
our mi ide."
gineeri
our mi
Figure 3: Qualitative comparisons on four sampled sequences from the test datasets. While the
previous approaches suffer from low contrast, blur, and extensive artifacts, LaSe-E2V obtains clear edges with high contrast and preserves the semantic details of the objects
Reference Frame
Figure Qualitative results fast-motion condition from HS-ERGB dataset [58].

---

E2VID
HyperE2211
LaSe- E2V (Our))
Reference
Figure 5: Qualitative results in low light condition from MVSEC dataset [76] Doutooornighht LaSe-E2V performs better to preserve the HDR characteristic of event cameras with higher contrast.
4.2 Comparison with State-of-the-Art Methods
Quantitative results. We compare LaSe-E2V with eight existing learning-based methods [50, 54, 57, 57,7 7, 46, 64, 13J. To ensure fair comparison, we maintain identical experiment settings without any post-processing operations across all methods. As shown in Tab. 1, LaSE-E2V mostly outperforms previous methods. In particular, LaSE-E2V significantly surpasses HyperE2VID by 30% on MSE on the ECD dataset. LaSe-E2V also excels in SSIM and LPIPS mostly. This highlights the superiority of the structural and semantic reconstruction ability of LaSE-E2V.
Qualitative Results. Fig. 3 illustrates the qualitative results reconstructed by LaSe-E2V and previous methods. As shown in the street reconstruction (Column 1), our method reconstructs more details, especially for the and the building. For the bike in Column 2, our method achieves clearer edges and higher contrast, while the previous methods are inclined to exhibit foggy artifacts around the bikes. Although our method achieves superior performance, some artifacts persist for the reconstruction of the text region (e.g.3rd row). This issue arises because it depends on the prior of the pre-trained diffusion model (SD2 [52]), which faces challenges in text generation. The recently released SD3 [15] claims to show improved text generation capabilities, which could potentially address the problem. Please refer to the video demo and the appendix for additional qualitative results, which demonstrates the superiority of our LaSe-E2V on temporal smoothness and consistency.
Results with Fast Motion. In Fig. 4, show sampled reconstructed frames based sequences of HS-ERRB [58] captured by Prophesee Gen4 (1280 x 720) event camera with high resolution and that method is capable of clearly recovering the details for high-speed movement and preserving temporal consistency.
relatively fast motion conditions. We
HDR Results. We test our model on the video sequences in extreme conditions (i.e.low light and fast motion) to further demonstrate the advantages of event cameras and the effectiveness of our framework. As shown in Fig. 5, we sample sequences from MVSEC (outdoor (nigo/2 with relatively low-light conditions in nighty streets. We can see that LaSe-E2V performs better in reconstructing the scene with higher contrast and more clear edge. Compared with HyperE2VID exhibiting foggy artifacts of the whole street, E2VID can reconstruct video without over-exposure or under-exposure.
Quantitative Result on Temporal Consistency. We further evaluate the results based on the temporal quality metrics from VBench [26]. As shown in Tab. 2, the numerical results demonstrate the effectiveness of our approach in maintaining temporal consistency. In particular, our method significantly outperforms others on the subject consistency and background consistency, while achieving comparable performance on motion smoothness.

---

Timestamp
Reference Frame
Figure 6: Qualitative results for color video reconstruction.
text "pavement, floor, juice
w/o text
Ground-Truth
w/o ESA
with ESA
Ground-Truth
Figure 7: Qualitative comparison for the ablation study on text guidance and ESA module.
Table 2: Quantitative comparison on temporal consistency on ECD [43] based on VBench [26].
Mesics
E2VID [50]
FireNet (54)
SPADE E2VID [7]
SSL D2VID (06]
ET Net (64)
1ypert22VID (13]
LaSe-E2V (Ours)
GT (Dmpprical Mar)
Subject
Comisiocy
52.14%
49.6%%
50.5%%
51.89%
55.49%
30.41%
84.25%
29%
Backeround
Comitiacy?
85.28%
#2 78%
82.61%
84.86%
86 85%
83.50%
93.39%
92.69%
Smenthaest
Motion
97668
98.40%
98.1%
95.97%
97.726
37.59%
98.11%
98.67%
4.3
Discussion
Video Editing with Language. In our framework, language serves as supplementary semantic information for E2V reconstruction. We investigate the impact of varying language guidance for video editing, as illustrated in Fig. 8. Utilizing text description from the off-the-shelf tagging model [73]. our method reconstructs a scene with reasonable structure in low-light condi- tions based on descriptors like "night, dark.... Interestingly, when we manu- ally alter the text description to "brighı, day light, ...". the lighting condition in the scene shifts to daylight, revealing clearer details, especially the sky area. This demonstrates our framework's 's abil- ity to modify lighting conditions based on textual descriptions, by effectively modeling light conditions semantic in- formation. this way, with a language- guided perspective, it offers the flexibil- ity to manually adapt the reconstructed video according user preferences.
"night, dark, city street..."
"bright, day light, city street
Reference Frame
Figure Qualitative results for video editing with text.
Color Video Reconstruction. Based on a pre-trained diffusion model [51], our model inherits the capability for colored video generation by training real-world datasets. As illustrated in Fig. our method successfully reconstructs color videos with clear details, although it occasionally misinterprets background semantics due to the absence of event data.
4.4 Ablation Study
We conducted ablation experiments on the LaSe-F2V framework to evaluate the effectiveness of language guidance, event-based attention, event-aware mask loss, and initialization strategy.

---

Language Guidance. To evaluate the effectiveness of language guidance, we conducted ex- periments without text conditions by setting the text input to null during the denoising stage. As shown in Tab. 3, introducing the text description achieves 0.015 improvement on SSIM.
As shown in Fig. (left), when provided with the language description "pave- ment", the model tends to reconstruct the scene more closely to the ground truth, whereas the baseline model randomly generates background which distinct to the ground truth. This demonstrates the importance of semantic information for E2V reconstruction, especially in cases where semantic ambiguity exists in the event data. We also conduct an ablation study on the previous frame condition, as indicated in Tab. 3 (row 2), where performance significantly dropped across all metrics. This underscores the critical role of the previous frame condition in our diffusion pipeline, ensuring temporal consistency within the autoregressive reconstruction process.
Table 3: Ablation study on context conditions.
Event
Text
Frame
MSF.
SSIM
LPIPSL
0.038
0.567
0.199
0.067
0.474
0.258
0.023
0.629
0.194
Event-guided Spatio-temporal Attention. guided spatio-temporal attention (ESA), we with simple channel-wise concatenation for event input. As shown in Tab. 4, performance significantly drops without the attention mech- anism, which demonstrates the effectiveness of ESA in preserving the event control on the video. Fig. 7 illustrates that our model maintains visual content close to the ground- truth, whereas the baseline method loses control over lighting and luminance. We further investigate the effective- ness of the event-aware mask loss. As is shown in Tab. 4. the event-aware loss function achieves clear margin of improvement compared to the baseline method (row 2). Moreover, Tab. (row 3)
To demonstrate the effectiveness of our event- conducted experiments by training the model
Table 4: Ablation study on key components. "EML. denotes the event-aware mask loss. "El" denotes event-based initialization.
ESA
EMI
El
MSE
SSIM
LPIPS
0.105
0.468
0.288
0.042
0.443
0.322
0.043
0,482
0.251
0.023
0.629
0.194
demonstrates the effectiveness of our went-based initialization.
5 Conclusion and Future Work
Conclusion. In this paper, we introduce LaSe-E2V, a language-guicn guided, semantic-aware E2V re- construction method. Leveraging language descriptions that naturally contain abundant semantic information, LaSe-E2V explores (cxt-conditional diffussion models with our proposed attention mech- anism and loss function, thus achieving high-quality, semantic-aware E2V reconstruction. Extensive experiments demonstrate the effectiveness of our innovative framework.
Limitations and Future Work. Despite the promising performance of our method, LaSe-E2V has several limitations. First, the training datasets, comprising synthetic and limited real-world data, are inadequate for optimizing data-intensive diffusion models. Consequently, our method may reconstruct scenes with artifacts differing from the training data. Second, given that LaSe-E2V relies on the diffusion model, multiple denoising steps are required for high-quality videos, slowing the process. Future work can focus on accelerating the inference efficiency based on the recent progress of diffusion models [37, 9, 38, 39].
Broader Impacts. Based on the event camera, our LaSe-E2V enhances the capabilities of various technologies by improving the safety and robustness of intelligent systems. As this technology matures, its integration into everyday devices and systems seems likely, heralding a shift in how visual data is captured and utilized across industries.
Acknowledgments. This work is supported by the Guangzhon-HHKUSSCCZ Joint Funding Program (No. 2024003)0080,, the Guangzhou City, University and Enterprise Joint Fund under Grant No.SS.2002A03J01 and Guangzhou Fundamental and Applied Basic Research (Grant Number: 2024A0444072), This work is partially supported by the CCF-Tencent Rhino -hir Open Research Fund RAGR20230120.
10

---

References
[1] Max Bain, Arsha Nagrani, Güi Varol, and Andrew Zisserman. Frozen in time: A joint vidco and image encoder for end-to-end retricval. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1728-1738, 2021.
[2] Patrick Bardow, Andrew J Davison, and Stefan Leutenegger. Simultaneous optical flow and intensity estimation from an event camera. Proceedings of the IEEF conference on computer vision and pattern recognition, pages 984-892.22 2016.
[3] Anthony Bisulco, Fernando Cladera, Volkan Isler, and Daniel Lee. Fast motion understanding with spatiotemporal neural networks and dynamic vision sensors. In 2021 IEEE International Conference on Robotics and Automation (ICRA), pages 14098- 44084-14144 IEEE, 2021.
[4] Andreas Blattmann, Robin Rombach, Huan Ling, Tim Dockhorn, Scung Wook Kim, Sanja Fidler, and Karsten Kreis, Align your latents: High-resolution video synthesis with latent diffusion models. In Proceedings of the IELE/CVF Conference on Computer Vision and Pattern Recognition, pages 22563- 22575, 2023.
[5] Christian Brandli, Raphael Berner, Minhao Yang, Shih-Chii Liu, and Tobi Delbruck. A 240x 180 130 db latency global shutter spatiotemporal vision sensor. IEEE Journal of Solid-State Circuits, 49(10):2333-234) 2014.
[6] Christian Brandli, Lorenz Muller, and Tobi Delbruck. Real-time, high-speed video decompression using frame-and event-based davis sensor. In 2014 IEFF International Symposium Circuits and Systems (ISCAS), pages 686-689. IERE, 2014.
[7] Pablo Rodrigo Gantier Cadena, Yeqiang Qian, Chunxiang Wang, and Ming Yang. Spade-e2vid: Spatially- adaptive denormalization for event-based video reconstruction. IEEE Transactions on Image Processing. 30:2488-2500, 2021.
[8] Weifeng Chen, Jie Wu, Pan Xie, Hefeng Wu, Jiashi Li. Xin Xia, Xuefeng Xiao, and Liang Lin. Control- a-video: Controllable text-to-video generation with diffusion models. arXiv preprint arXiv:2305.13844 2023.
[9] Yu-Hui Chen, Raman Sarokin, Juhyun Lee, Jiuqiang Tang, Chuo-Ling Chang, Andrei Kulik, and Matthias Grundmann. Speed is all you need: On-device accelcration of large diffusion models via gpu-aware optimizations. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 4650-4654. 2023.
[10] Zuozhuo Dai, Zhenghao Zhang, Yao Yao, Bingxue Qiu, Siyu Zhu, Long Qin, and Weizhi Wang. Ani- mateanything: Fine- siaingrained open domain image animation with motion guidance. arXiv e-prints, pages 2023.
[11] Huiyu Duan, Xiongkuo Min, Sijing Wu, Wei Shen, and Guangtao Zhai. Uniprocessor: A text-induced unified low-level image processor. arXiv preprint arXiv:2407.2090 20928, 2024.
[12] Burak Erean, Onur Eker, Aykut Erdem, and Erkut Erdem. Evreal: Towards comprehensive benchmark and analysis suite for event-based vidco reconstruction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 3942-3951, 2023.
[13] Burak Erean, Onur Fkerr Canherk Saglam, Aykut Erdem, and Erkut Erdem. Hypere2vid: Improving event-based video reconstruction via hypernetworks. IEEE Transactions on Image Processing, 2024.
[14] Patrick Esser, Johnathan Chiu, Parmida Atighehchian, Jonathan Granskog, and Anastasis Germanidis. Structure and content content-guided video synthesis with diffusion models. Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 7356-735,, 2023.
[15] Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et al. Scaling rectified flow transformers for high-resolution image synthesis. In Forty-first International Conference on Machine Learning, 2024.
[16] Kanchana Vaishnavi Gandikota and Paramanand Chandramouli. Text-guided explorable image super- resolution. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pages 25900-25911, 2024.
2024.
[17] Ming Gui, Johannes S Fischer. Ulrich Prestel, Pingchuan Ma. Dmytro Kotovenko, Olga Grebenkova, Stefan Andreas Baumann, Vincent Tao Hu, and Björn Ommer. Depthfm: Fast monocular depth estimation with flow matching, arXiv preprint 4r0iv: 137888 2024.

---

[18] Yuwei Guo, Ceyuan Yang, Anyi Rao. Yaohui Wang. Yu Qiao. Dahua Lin, and Bo Dai. Animated- iff: Animate your personalized text-to-image diffusion models without specific tuning. arXiv preprint S.Xiv:2307.04725. 2023.
[19] William Harvey, Sacid Naderiparizi, Vaden Masrani, Christian Weilhach, and Frank Wood. Flexible diffusion modeling of long videos. Advances in Neural Information Processing Systems, 35:27953-27665, 2022.
[20] Jonathan Ho, William Chan, Chitwan Saharia, Jay Whang, Ruiqi Gao, Alexey Gritsemko, Diederik P Kingma, Ben Poole. Mohammad Norouzi, David Fleet, et al. Imagen video: High definition video generation with diffusion models. arXiv preprint arXiv:2210.023033 2022.
[21] Jonathan Ho, Ajay Jain, and Pieter Abbecl. Denoising diffusion probabílistic models. Advances in neural information processing systems, 33:6840 6851, 2020.
[22] Jonathan Ho and Tim Salimans. Classifier-frce diffusion guidance. arXiv preprint arXiv:200.112598, 2022.
[23] Jonathan Ho, Tim Salimans, Alexey Gritsenko. William Chan, Mohammad Norouzi, and David J Fleet. Video diffusion models. Advances Neural Information Processing Systems, 35:8633 8646, 2022.
[24] Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang. Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint wrx.:220060068.. 2021.
[25] Yuhuang Hu, Shih-Chii Liu, and Tobi Delbruck. v2e: From video frames realistic dvs events. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1312-1321, 2021.
[26] Ziqi Huang, Yinan He, Jiashuo Yu, Fan Zhang, Chenyang Si, Yuming Jiang, Yuanhan Zhang. Tianxing Wu, Qingyang Jin, Nattapol Chanpaisit, et al. Vhench: Comprehensive benchmark suite for video generative models. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 21807-21818, 2024.
[27] Zhuangyi Jiang. Pengfei Xia, Kai Huang, Walter Stechele, Guang Chen, Zhenshan Bing. and Alois Knoll. Mixed trame-levent-drive fast pedestrian detection. In 2019 International Conference on Robotics and Automation (ICRA), pages 8332-88 8338. IARF, 2019.
[28] Bingxin Ke, Anton Obukhov, Shengyu Huang, Nando Metzger, Rodrigo Caye Daudt, and Konrad Schindler. Repurposing diffusion-basee image generators for monocular depth estimation. In Proceedings of the IEEE/CVF Conference Computer Vision and Puttern Recognition, pages 9492-9502, 2024.
[29] Alex Kendall and Roberto Cipolla. Geometric loss functions for camera pose regression with deep learning. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 5974-59999999 5983,
2017.
[30] Levon Khachatryan, Andranik Movsisyan, Vahram Tadevosyan, Roberto Henschel, Zhangyang Wang. Shant Navasardyan, and Humphrey Shi. Text2video-zero: Text-to-image diffusion models are zero-shot video generators. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 15954-15964, 2023.
[31] Jianing Li. Siwei Dong. Zhaofei Yu, Yonghong Tian, and Tiejun Huang. beent-based vision enhanced: joint detection framework in autonomous driving. In 2019 iece international conference on multimedia and expo (ieme), pages 1 1396-1401. IERF, 2019.
[32] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-iman pre-training with frozen image encoders and large language models. In International conference on machine learning, pages 19730-100 19742. PMLR, 2023.
[33] Tsung-Yr Yi Lin, Michael Maire, Serge Belongic, James Hays, Pictro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision- ECCV 2014: 13th European Conference, Zurich, Swirzerland, September 6-12. 2014, Proceedings, Part V 13, pages 740-755. Springer, 2014.
[34] Siying Liu and Pier Luigi Dragotti. Sensing diversity and sparsity models for event generation and video reconstruction from events. IEEE Transactions Pattern Analysis and Machine Intelligence, 2023.
[35] Siying Liu and Pier Luigi Dragotti. Enhanced event-based video reconstruction with motion compensation. arXiv preprint arXiv:2403.11961 2024.
12

---

[36] Xingchao Liu, Xiwen Zhang, Jianzhu Ma. Jian Peng, et al. Instaflow: One step is enough for high- quality diffusion-based text-to-image generation. In The Twelfth International Conference Learning Representations, 2023.
[37] Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Dpm-solver: A fast ode solver for diffusion prohabilistic model sampling in around 10 steps. Advances Neural Information Processing Systems, 35:5775-5787, 2022.
[38] Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Dpm-solver++: Fast solver for guided sampling of diffusion probabilistic models, arXiv preprint arXiv:2211.01095, 2022.
[39] Simian Luo, Yiqin Tan, Longbo Huang, Jian Li, and Hang Zhao. Latent consistency models: Synthesizing high-resolutee images with few-step inference. arXiv preprint arXiv:2310.04377, 2023.
[40] Zhengxiong I Luo, Dayou Chen, Yingya Zhang. Yan Huang, Liang Wang. Yujun Shen, Deli Zhao, Jingren Zhou, and Tieniu Tan. Videofusion: Decomposed diffusion models for high-quality video generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10209- 10218, 2023.
[41] Yue Ma. Yingqing He, Xiaodong Cun, Xintao Wang. Siran Chen. Xiu Li, and Qifeng Chen. Follow your pose: Pose-guided text-to-vidco generation using pose-free videos. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 41 111--4155, 2024.
[42] Chong Mou, Xintao Wang, Liangbin Xie, Yanze Wu, Jian Zhang, Zhongang Qi, and Ying Shan. T2i- adapter: Learning adapters to dig out more controllable ability for texl-to-image diffusion models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 4296-4304, 2024.
[43] Elias Mueggler, Henri Rebeeq, Guillermo Gallego, Tobi Delbruck, and Davide Scaramuzza. The event- camera dataset and simulator: Event-based data for pose estimation, visual odometry, and slam. The International Journal of Robotics Research, 36(2):142 149, 2017.
[44] Gottfried Munda, Christian Reinbacher, and Thomas Pock. Real-time intensity-image reconstruction for event cameras using manifold regularisation. International Journal of Computer Vision, 126( 126(12):1381- 1393, 2018
[45] Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam. Pamela Mishkin, Bob McGrew, llya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. arXiv preprint arXiv:2 11..2/22..7722.. 10741, 2021.
[46] Federico Paredes-Vallés and Guido CHE De Croon. Back to event basics: Self-supervised learning of image reconstruction for event cameras via photometric constancy. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages
2021.
[47] Chenyang Qi, Zhengzhong Tu, Keren Ye, Mauricio Delbracio, Peyman Milanfar, Qifeng Chen, and Hossein Talebi. Tip: Text-driven image processing with semantic and restoration instructions. arXiv preprint arXiv:2312.11595 2023.
[48] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748-8763. PMLR, 2021.
[49] Henri Rebecq, René Ranftl, Vladlen Koltun, and Davide Scaramuzza. Events-to-video: Bringing modern computer vision to event cameras. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pages 3857-3866, 2019.
[50] Henri Rebecq, René Ranfil, Vladlen Koltun, and Davide Scaramuzza. High speed and high dynamic range video with an event camera. IEEE transactions on pattern analysis and machine intelligence, 1900))994--9000 2019.
[51] Weiming Ren, Harry Yang, Ge Zhang, Cong Wei, Xinrun Du, Stephen Huang, and Wenhu Chen. Consisti2v: Enhancing visual consistency for image-to-vided generation. arXiv preprint arXiv:2402.04333 2024.
[52] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björ Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684-10695. 2022.
[53] Cedric Scheerlinck. Nick Barnes, and Robert Mahony. Continuous time intensity estimation using event cameras. In Asian Conference on Computer Vision, pages 308-324. Springer, 2018
13

---

[54] Cedric Scheerlinck, Henri Rebecq. Daniel Gehrig, Nick Barnes, Robert Mahony, and Davide Scaramuzza. Fast image reconstruction with an event camera. In Proceedings the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 156-163, 2020.
[55] Uriel Singer, Adam Polyak, Thomas Hayes, Xi Yin, Jie An, Songyang Zhang, Qiyuan Hu, Harry Yang. Oron Ashual, Oran Gafni, et al. Make-a-video: Text-lo-video generation without text-viden data. arXiv preprint Xr0: 14792, 2022.
[56] Jiaming Song. Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models, arXiv preprint arXiv:2010.0253 2020.
[57] Timo Stoffregen, Cedric Scheerlinck, Davide Scaramuzza, Tom Drummond, Nick Barnes, L Lindsay Klee- man, and Robert Mahony. Reducing the sim-to-real gap for event cameras. In Computer Vicion-ECCV 2020: 16th European Conference, Glasgow, UK. August 23-28, 2020, Proceedings, Part XXVII 16, pages 534-549. Springer. 2020.
[58] Stepan Tulyakov, Daniel Gehrig, Stamatios Georgoulis, Julius Erbach, Mathias Gehrig. Yuanyou Li, and Davide Scaramuzza. Time lens: Event-hased video frame interpolation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 16155-16164, 2021.
[59] Jianyi Wang. Zongsheng Yue, Shangchen Zhou, Kelvin CK Chan, and Chen Change Loy. Exploiting diffusion prior for real-world image super-resolutio. International Journal of Computer Vision, pages 1-21, 2024.
[60] Jiuniu Wang, Hangjie Yuan, Dayou Chen, Yingya Zhang, Xiang Wang, and Shiwei Zhang. Modelscope text-to-video technical report. arXiv preprint 30X.022008.065),, 2023.
[61] Lin Wang. Yo-Sung Ho, Kuk-Jin Yoon, et al. Event-based high dynamic range image and very high frame rate video generation using conditional generative adversarial networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10081-10090, 2019.
[62] Xiang Wang, Hangjie Yuan, Shiwei Zhang, Dayou Chen, Jiuniu Wang, Yingya Zhang, Yujun Shen, Deli Zhao, and Jingren Zhou. Videocomposer: Compositional video synthesis with motion controllability. Advancex in Neural Information Processing Systems, 36, 2024.
[63] Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE transactions image processing, 13(4):600 612, 2004.
[64] Wenming Weng, Yueyi Zhang, and Zhiwei Xiong, Event-based video reconstruction using transformer. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2563-2572, 2021.
[65] Rongyuan Wu, Tao Yang. Lingehen Sun. Zhenggiang Zhang, Shuai Li, and Lei Zhang. Seesr: Towards semantics-aware real-world image super-resolution. In Proceedings of the IEEF/CV conference on computer vision and pattern recognition, pages 25456-25467, 2024.
[66] Ruihan Yang, Prakhar Srivastava, and Stephan Mandt, Diffusion probabilistic modeling for video genera- tion. Entropy, 25(10): 1469, 2023.
[67] Tao Yang, Rongyuan Wu, Peiran Ren, Xuansong Xie, and Lei Zhang. Pixel-aware stable diffusion for realistic image super-resolution and personalized stylization. arXiv preprint arXiv: 14469, 2023.
[68] Shengming Yin, Chenfei Wu, Jian Liang, Jie Shi, Hougiang Li, Gong Ming, and Nan Duan. Drag- nuwa: Fine-grained control in video generation by integrating text, image, and trajectory. arXiv preprint arXiv:2308.0m 08089, 2023.
[69] Lei Yu, Wen Yang, et al. Event-based high frame-rate video reconstruction with novel cycle-event network. In 2020 IEEE International Conference on Image Processing (ICIP), pages 86-90. IEEF, 2020.
[70] Lymin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3836-388 3847, 2023.
[71] Richard Zhang, Phillip Isola, Alexci Efros, Eli Shechtman, and Oliver Wang. The unreasonable elfectiveness of deep features as a perceptual metric. In Proceedings of the IEEE conference on computer vision and puttern recognition, pages 586-595, 2018.
[72] Yabo Zhang, Yuxiang Wei, Dongsheng Jiang, Xiaopeng Zhang, Wangmeng Zuo, and Qi Tian, Controlvideo: Training-free controllable text-to-video generation. arXiv preprint 2i0:22005.1307, 2023.
14

---

[73] Youcai Zhang, Xinyu Huang, Jinyu Ma, Zhaoyang Li, Zhaochuan Luo, Yanchun Xie. Yuzhuo Qin. Tong Luo, Yaqian Li, Shilong Liu, et al. Recognize anything: A strong image tagging model. arXiv preprint arXiv:2306. 035144 2023.
[74] Zelin Zhang, Anthony Yezzi, and Guillermo Gallego. Formulating event-based image reconstruction linear inverse problem with deep regularization using optical flow. IEEE Transactions on Puttern Analysis and Machine Intelligence, 45(7):8372-8333 8389, 2022.
[75] Rui Zhao, Yuchao Gu, Jay Zhangjie Wu, David Junhao Zhang, Jiawei Liu, Weijia Wu, Jussi Keppo, and Mike Zheng Shou, Motiondirector: Motion customization text to vo-video diffusion models. arXiv preprint arXiv:2310.08460 2023.
[76] Alex Zihao Zhu, Dinesh Thakur, Tolga Ōzaslan, Bernd Pfrommer, Vijay Kumar, and Kostas Danilidiss The multivehicle stereo event camera datasct: An event camera dataset for 3d perception. IEEE Robotics and Automation Letters, 3(3):2032-20333 2018.
[77] Alex Zihao Zhu, Liangzhe Yuan. Kenneth Chaney, and Kostas Danillidss, Unsupervised event-based lcarning of optical flow, depth, and egomotion. In Proceedings of the IEFF/CVF Conference Computer Vision and Pottern Recognition, pages 989-997, 2019.
[78] Lin Zhu, Xiao Wang. Yi Chang, Jianing Li, Tiejun Huang, and Yonghong Tian. Event-based video reconstruction potential assisted spiking neural network. In Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3594-3604, 2022.
15

---

Appendix
A. Event voxel representation
Given a event stream Ex = {er I with N events stream, each event et, 8" denotes four- clement tuple (2t, yk tg. Pk), reporting spatial coordinates, timestamp and polarity respectively. The event voxel representation is formulated as follows:
(*(k) = Σ Ps max (0.1 k 1 to (B (1B-1))) IN to
(8)
where to. tu denote the start time and end time of event stream E respectively, k E [0, B 1], B = 5 for experimental setting.
B. Additional datasets and Implementation Details
Datasets. For ECD, we use seven short sequences from this dataset, where the DAVIS240C [5] camera moves with 6-DOF and with increasing speed in six of them. These sequences mostly contain simple office environments with static objects. MVSEC is recorded by synchronized stereo event camera system. Each sequence of MVSEC releases extensive ground-truth reference data for ovaluations. The HQF dataset, recorded by two DAVIS240C [5] cameras, provides high-quality ground truth frames, of which the motion blur is maximally mitigated under preferable exposure. 14 sequences are contained, covering a wider range motions and scene types, including static scenes and motion scenes of slow, medium and fast, indoor and outdoor scenes. Following the training processing, we generate language descriptions for each sequence. For fair comparison, we select the same sequences from the three datasets as those reported in the recent benchmark of EVREAL [12].
Implementation Details. During training, we randomly drop input text prompts with a probability of 0.1 to enable classifier-free guidance [22]. For the reconstruction of the first clips and the accumulation error of the autoregressive pipeline, we randomly drop the first frame as the condition with 0.4 probability. During inference, employ the DDIM sampler [56] with 50 steps and classifier-free free guidance with text guidance scale w = 5 to sample videos.
Evaluation Metrics. The SSIM metric raises ambiguity because it involves several hyper- hyper-paranen parameters that may differ across various codebases. For example, the structural similarity function the skimage package, parameters like paussian_weiieee and sigma are used for spatial weighting of each patch with a Gaussian kernel, aise_sampee_omm wwt.aapllllovrrrraneee covarrincee indicate whether normalize covariances, and K1 and K2 are algorithm-specifee parameters that need to be set. For this reason, we reevaluated all comparison methods by using unified metric, denoted as the SSIM scores.
16

---

MaDe Lab
MaDe Lab
MaDeLaa
MaDe Lab
MiDDo Lib
MaDs Lab
MaDo Lab
MERE Enfrweey
onginstemg
onginniiing
manta.
E2VDD
EWDD+
FicMe+
EI-Nct
Rypa EWVID
Lave-B2V (Uars)
Grous Trufh
Figure 9: Additional qualitative results for three datasets.
C. Additional qualitative results
In Fig. 9, provide the additional qualitative results. For example, shown in column 4 and 5, method presents more accurate reconstructed results, especially for those vocabularies. In column 1, 2 and 3, method reconstructs more details. In the supplementary material, also provide several reconstructed video clips to demonstrate the superiority of LaSe-E2V on temporal smoothness and consistency.
D. Complexity analysis
We provide a detailed computational complexity analysis Tab. 5, which includes recent event- to-video reconstruction methods and related diffusion- based approaches. Our method requires significant amount of inference time due to the 50 denoising steps, in contrast to previous single-step event-to-video methods. However, this is common challenge for all diffusion-basee models, as observed in diffusion-based super-resolution [59, 67, 65] and depth estimation methods [17,28 28). To reduce inference time, some research, e.g., Instaflow [36], already explored decreasing the number of denoising steps, offering a promising direction for further improvement (as a future work) to our framework. Albeit with lower inference speed than conventional E2V methods (but higher than diffusion-basse super-resolution and depth estimation methods), our work brings new ideas and may hopefully inspire future research for event-based vision by incorporating language guidance.
D. More analysis results
Additional comparison methods. We mainly compared with the latest state-of-the-art method, e.g., HyperE2VID [13] in the main paper. By default, method is superior to all the previous methods. We also provide more comparison methods Tab. 6, which further demonstrates the superior performance of our method.
Quantitative results on HS-ERGB dataset. Tab. 7 provides a quantitative comparison based on three sequences (horse_!1) 11, horse_12 horse_13 13) from the HS-ERGB dataset. Existing E2V methods typically fail reconstruct regions without events, leading to significantly worse quantitative results. Although our method may not perfectly reconstruct every detail for reality, it does generate
17

---

Table 5: Complexity comparison on various methods. All tests are conducted on one NVIDIA Tesla 32G-V100 GPU.
Methods
Parameters
Inference time (per frame)
Conventional Event-to- Video
ET-Net [64]
22.18M
0,0124 0.0043
HyperE2VID [13]
10.15M
Diflusion-based Depth Estimation
DepthFM [17]
891M
2.1 s
Marigold [28]
948M
5.2 s
Diffusion-based Supor-Resolution
StableSR [59]
1409M
18.70
PASD [67]
1900M
6.07 s
SeeSR [65]
2284M
7.24 S
Diffusion-based Event-to-Vider
LaSe-E2V (Ours)
1801M
1.09 s
Table 6: Comparison with more event-to-video methods.
ECD
MVSEC SSIM
HQF SSIM
Methods
MSEJ
SSIM
LPIPS
MSEJ
LPIPS
MSEI
LPIPS)
Zhang et ul. (IPAM12022) [74)
0,076
0.519
0.457
EVSNN (CVPR2022) [78]
0.061
0.570
0.362
0.104
0.389
0.538
0.086
0.482
0.433
PA-EVSNN (CVPR2022) [78]
0.046
0.626
0.367
0.107
0.403
0.566
0.061
0.532
CISTA-LSTC (TPAM12023) [34]
0.038
0.585
0.229
0.041
0.563
0.271
FlSw (Arxiv2024) [35]
0.047
0.586
0.225
0.034
0.590
0.257
HyperE2VID (TIP2024) [13]
0.033
0.576
0.212
0.076
0.315
0.476
0.031
0.531
0.257
LaSe-E2V (Ours)
0.023
0.629
0.194
0.055
0.342
0.461
0.034
0.548
0.254
reasonable output that aligns with human preference and is generally close to the distribution of the real scene. Therefore, while results the HS-ERGB dataset may be less significant than those on "constantly moving" datasets (ECD, MVSEC, HQF), our method is still substantially better than baseline methods.
Impact of the prompting model. We also tested BLIP [32] on a sampled sequence (i.e., boxes in HQF) to further evaluate the influence of the prompting model, shown in Tab. 8. BLIP can generate reasonable text prompts in caption-style and show nearly reconstructed performance on MSE (0.025 vs 0.027).
E. Discussion
We begin by discussing the source of text prompts used for event data. Our framework primarily aims to establish a pipeline for reconstructing videos from events using a language-guide approach. This is based on our observation that language naturally conveys rich semantic information, which improves the semantic consistency of the reconstructed video. This form of text guidance resembles text-guided denoising [11, 47] and super-resolution [16] methods, which also use natural language as user-frient user-friendly interface to control the image restoration process. In this work, we do not emphasize specific sources of text prompts, as these can vary depending on the application scenario. For instance, when using DAVIS346 camera, text can conveniently be obtained from APS frames. If Prophesee camera used and only event data is available, tagging-base text prompts derived from models can be employed. Additionally, in complex scenes with limited
event-based
18

---

Table 7: Quantitative comparison of HS-ERGB [58]. total of 497 frames.
Results are conducted on 3 sequences with
Methods
MSE
SSIM
LPIPS
E2VID [50]
0.199
0.382
0.736
HyperE2VID [13]
0.161
0.374
0.745
LaSe-E2V (Ours)
0.078
0.429
0.665
Table 8: Comparisons between different prompting models on boxes of HQF.
Prompting Models
MSEJ
LPIPS SSIM
RAM [73]
0.025
0.557
0.196
BLIP [32]
0.027
0.546
0.207
event data, human intervention can be incorporated interactively to enable user control. Overall, our method offers flexible and generic pogguaee-guided interface for E2V reconstruction.
Regarding the role of text information, serves as a crucial component to activate the semantic prior in the diffusion model and compensate for missing semantic content. In regions with sufficient event data, our approach enhances reconstruction performance by leveraging the semantic priors provided by the text prompts, ensuring high fidelity. For regions with sparse event data, the method relies solely on the text prompts to reconstruct scenes that align with human expectations. While this approach may introduce textures details that differ from the actual ones, it still produces images closer to reality than previous methods. In contrast, prior E2V models typically reconstruct such regions as indistinet haze, far from the true distribution of real images, as shown in Fig. 1. The quantitative comparison Tab. further demonstrates the superiority of our approach.
Admittedly, hallucination is potential side effect introduced by incorporating semantic priors from the diffusion model into our framework. While our proposed techniques effectively reduce hallucinations, completely eliminating them remains a challenge in the diffusion model research community. Extensive experiments (Tab. and Fig. 3) show that our method significantly improves fidelity and reduces hallucinations in scenes with adequate event data. Additionally, approach exhibits superior reconstruction performance even in scenes with insufficient event data (Tab. 7). It is important to note that eliminating hallucinations entirely for regions with sparse event data not feasible. However, our method effectively leverages the semantic priors in the diffusion model produce images closer the true scene distribution, whereas previous E2V methods (e.g.. HyperE2VID) fail in these regions, as demonstrated in Fig.
For future work, given that our method reliably ensures fidelity in regions with sufficient event data, is practical to assign a confidence map based on event density to identify high-confidence regions. For safety-critical applications, decisions could be made based on both the reconstructed video and the confidence map, allowing simultaneous consideration event-based multi-modality models offer a potential source of semantic information, which could be explored in future research. Since these models are trained with large-scale event-image paired data, they have the potential provide prior semantic information absent in event data.
image quality and safety. Moreover,
19

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Our main claims made in the abstract and introduction accurately reflect the paper's contributions and scope. The claims are supported by our experimental results.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitations in Sec.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
20

---

Justification:
Our paper does not include theoretical results
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes)
Justification:
We present all methodology details in Sec. 3.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
21

---

Answer: [Yes]
Justification:
We don't provide open access to data and code in the supplemental material.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips. cc/
oniioicyyyl for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https:
/ //iips.cooo wwtpccccpppccceeeeecec.eccccccct wwwpc//gg//////eee /CooeSubmbii
/ttee//uuudeeeoeeuubimiiii ceobbmissii ssisnnsssn ceiiinoylcc)) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We specify all the training and test details in Sec. 4.1.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: Error bars are not reported because it would be too computationally expensive.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
22

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of enrors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
show in tables or
error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We provide sufficient information in the appendix.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https: /tttpp://neuute http://////c/ttiicccccuiiieeeeee
Answer: [Yes]
Justification: We read the guidance carefully and make sure that our research is conducted in accordance with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative socictal impacts of the work performed?
Answer: [Yes]
Justification: We discuss both potential positive societal impacts and negative societal impacts of the work performed.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
23

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our research poses no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We explicitly mention and respect the creators and original owners.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: Our paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets,
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: Our paper does not research human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: Our paper does not research human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
25


Reviews:
[{"review_id": "hmZWTgV7BR", "reviewer": "Reviewer_fQWS", "summary": "This paper uses abundant semantic information and raw event information to guide the reconstruction of RGB images from event images based on U-Net. Furthermore, this paper introduces event-aware mask loss to ensure temporal coherence and a noise initialization strategy to enhance spatial consistency. Experiments demonstrate that the proposed algorithm has a strong reconstruction performance.", "questions": "1. What are the tagging models mentioned in the paper? There are no specifics and no ablation experiments.\n\n2. The comparison methods in the references are mostly from 2020 and 2021, with only one method from 2024.\n\n3. Other methods are compared on SSIM metrics. Why not give SSIM metrics instead of SSIM* metrics?\n\n4. How do you define spatial consistency? Adding ‘Noise Initialization’ only in the testing phase did not enhance the consistency of the model itself in the spatial dimension.", "limitations": "See Weaknesses. The final rating will be made based on the rebuttal.", "rating": 4, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Constructing ESA utilizes abundant semantic information and raw events to construct the cross attention with the combination of frames and raw events separately to guide event image reconstruction to RGB image.\n\n2. The mask loss is constructed to supervise the reconstructed image from the temporal dimension strongly.\n\n3. Extensive experiments on three datasets covering diverse challenging scenarios (e.g., fast motion, low light) demonstrate the superiority of this method.", "weaknesses": "1. In the ESA module, two modalities of information, raw events and text were used for cross-attention with frames. Still, no ablation experiments were given, which makes it impossible to determine whether the final experimental results of this work are more useful for raw events or text. In particular, raw events were inserted in each U-Net section.\n\n2. In this paper, there are no reported flops as well as parameters, especially relative to some of the second-best methods in Table 1, what is the approximate percent increase in the two parameters?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank the reviewer for valuable comments and suggestions. We address the concerns below:\n\n> **About ablation experiments for ESA on raw events and text.**\n> \n- The reviewer might misunderstand the ESA module. Indeed, the ESA and the text are **separate** parts in our framework. As outlined in Sec. 4.2 (Line 157-183), the ESA module consists of event spatial attention and event temporal attention layers, which compute attention based on the hidden state $z$ and event feature $z_e$, respectively. This way, ESA module facilitates spatio-temporal consistency between event data and video. The text input is integrated separately via a cross-attention layer that follows the original LDM [42], and it is not involved in the ESA module. Since event input is essential for the E2V reconstruction task, we have already provided ablation studies on both ESA and text with event input to demonstrate their individual effectiveness. Hereby, we reiterate the results as follows:\n    - To demonstrate the impact of text description, the ablation study in Tab. 2 (1st row vs. 3rd row) and Fig. 7 (left) compare the E2V results with (w) and without (w/o) text. Please check them.\n    - Tab. 3 (1st row) compares the performance of the baseline trained with simple channel-wise concatenation of events, without the ESA module,  showing a significant performance drop. This ablation result confirms the effectiveness of ESA module.\n\n> **About FLOPs and parameters comparison.**\n> \n- Thanks for the suggestion. Please refer to **Global Response to All Reviewers**.\n\n> **About the tagging models and the corresponding ablation experiments.**\n> \n- In Line 216, we employ the off-the-shelf tagging model RAM [61], which serves as a prompting model to provide text descriptions for the datasets. In fact, a recent work (SeeSR [70]) has also demonstrated the superiority of RAM compared to other models because of **rich objects and concise description**.\n- As suggested, we also tested BLIP[71] on a sampled sequence (i.e., *boxes* in HQF) to further evaluate the influence of the prompting model, as shown in Tab. A-2. BLIP can generate reasonable text prompts in caption-style and show **nearly reconstructed performance** on MSE (0.025 vs 0.027). A detailed discussion will be provided in the camera-ready revision.\n\nTable A-2: Comparisons between different prompting models on *boxes* of HQF.\n\n| Models | MSE | SSIM | LPIPS |\n| --- | --- | --- | --- |\n| RAM [61] | 0.025 | 0.557 | 0.196 |\n| BLIP [71] | 0.027 | 0.546 | 0.207 |\n\n> **Additional comparison methods.**\n> \n- We mainly compared with the latest state-of-the-art method, e.g., HyperE2VID (TIP 2024). By default, our method is superior to the methods published in 2022 and 2023. However, upon the suggestion, we have provided more comparison methods in Tab. A-3, which further demonstrates the superior performance of our method. We will update Tab.1 of main paper in the camera-ready revision.\n\nTable A-3: Comparison with more event-to-video methods.\n\n| Methods | | ECD | | | MVSEC | | | HQF | |\n| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |\n|  | MSE | SSIM | LPIPS | MSE | SSIM | LPIPS | MSE | SSIM | LPIPS |\n| Zhang et.al. (TPAMI2022) [72] | 0.076 | 0.519 | 0.457 | - | - | - | - | - | - |\n| EVSNN (CVPR2022) [73] | 0.061 | 0.570 | 0.362 | 0.104 | 0.389 | 0.538 | 0.086 | 0.482 | 0.433 |\n| PA-EVSNN (CVPR2022) [73] | 0.046 | 0.626 | 0.367 | 0.107 | **0.403** | 0.566 | 0.061 | 0.532 | 0.416 |\n| CISTA-LSTC (TPAMI2023) [74] | 0.038 | 0.585 | 0.229 | - | - | - | 0.041      | 0.563 | 0.271 |\n| CISTA-Flow (Arxiv2024) [75] | 0.047 | 0.586 | 0.225 | - | - | - | 0.034      | **0.590** | 0.257 |\n| HyperE2VID (TIP2024) [12] | 0.033 | 0.576 | 0.212 | 0.076 | 0.315 | 0.476 | **0.031** | 0.531 | 0.257 |\n| LaSe-E2V (OUrs) | **0.023** | **0.629** | **0.194** | **0.055** | 0.342 | **0.461** | 0.034 | 0.548 | **0.254** |\n\n> **About SSIM and SSIM\\* metric.**\n> \n- The SSIM metric raises ambiguity because it involves several hyper-parameters that may differ across various codebases. For example, in the `structural_similarity` function of the **skimage** package, parameters like `gaussian_weights` and `sigma` are used for spatial weighting of each patch with a Gaussian kernel, `use_sample_covariance` indicates whether to normalize covariances, and `K1` and `K2` are algorithm-specific parameters that need to be set. For this reason, we **reevaluated** all comparison methods by using a unified metric, denoted as the SSIM* scores. We will further clarify this point in the revision.\n\n> **How to define spatial consistency and how does noise initialization enhance the consistency?**\n> \n- Spatial consistency denotes the consistency between event data and the reconstructed video. Specifically, events typically occur at the **edges/texture** part of the scene context, and the reconstructed video needs to align the event data in the spatial structure.\n- Noise Initialization mainly focuses on mitigating the **train-test gap** during inference, which is a common challenge in diffusion models. Intuitively, accumulated event data provides structural information (e.g. edges) in the scene, acting as an **additional constraint** during the denoising process.\n\n**Additional Reference:**\n\n[71] Li J, Li D, Savarese S, et al. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models[C]//ICML, 2023.\n\n[72] Zhang Z, Yezzi A J, Gallego G. Formulating event-based image reconstruction as a linear inverse problem with deep regularization using optical flow[J]. TPAMI, 2022, 45(7): 8372-8389.\n\n[73] Zhu L, Wang X, Chang Y, et al. Event-based video reconstruction via potential-assisted spiking neural network[C]//CVPR. 2022.\n\n[74] Liu S, Dragotti P L. Sensing diversity and sparsity models for event generation and video reconstruction from events[J]. TPAMI, 2023, 45(10): 12444-12458.\n\n[75] Liu S, Dragotti P L. Enhanced Event-Based Video Reconstruction with Motion Compensation[J]. arXiv, 2024.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "TxkEhMCKvd", "reviewer": "Reviewer_ueNs", "summary": "This paper addresses the issue of artifacts and regional blur in existing event-to-video (E2V) reconstruction algorithms by leveraging the rich semantic information in language to enhance the semantic consistency of reconstructed videos. The authors propose a language-guided E2V generation model that employs existing text-conditional diffusion models as a framework. They use an Event-guided Spatiotemporal Attention (ESA) module for fine-grained spatial alignment and temporal continuity, an event-aware mask loss for further ensuring temporal consistency, and an event-aware noise initialization to address training-testing discrepancies. Extensive comparative experiments validate the algorithm's performance, and ablation studies demonstrate the effectiveness of each component.", "questions": "Why is it necessary for the event activation regions of adjacent frames to be similar, rather than the regions without events?", "limitations": "The authors discuss the limitations of their work, particularly concerning the required training data and the inference speed of diffusion models, in accordance with official guidelines.", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1 This paper is the first to tackle the event data reconstruction task from a language-guided perspective. This approach could inspire future work in the field.\n\n2 The proposed algorithm achieves optimal performance on nearly all metrics across multiple datasets, demonstrating superior visual effects and validating the algorithm's effectiveness.\n\n3  The paper presents a high-quality body of work, including effective methods and extensive experimental validation.", "weaknesses": "1 The paper lacks a detailed comparison of the algorithm's inference speed and model size. The authors acknowledge the inherent speed limitations of using diffusion models, emphasizing the importance of providing this comparison.\n\n2 In practical applications lacking APS reference frames, obtaining accurate textual information that matches the scene description is difficult or nearly impossible. Using existing text generation models to extract semantic information introduces additional reference, compromising fairness to some extent.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank the reviewer for the valuable suggestions. We address the questions below.\n\n> **Detailed comparison of inference speed and model size.**\n> \n- Please refer to the **Global R*esponse to All Reviewers***. As shown in Tab. A-1, our method requires considerable inference time due to multiple denoising steps, showing a limitation of diffusion models. However, our approach “*could inspire future work in the field*”(***ueNs***) by incorporating language guidance and proves “*it is possible to use diffusion-based models for the event-based video reconstruction task*”(***H8fU***).\n\n> **Textual information is difficult to obtain and compromises fairness.**\n> \n- We respectfully disagree. The textual description is not difficult to obtain for our framework. Please note that our method only requires coarse text descriptions to leverage the capabilities of the pre-trained T2I diffusion model. These descriptions, generated by the tagging model RAM, simply identify objects in a tagging style, such as \"car,\" \"road,\" and \"tree\". These can be easily provided by humans in the absence of APS frames.\n- Also, please note that it is not uncommon to introduce additional priors for event-to-video (E2V) reconstruction task. For instance, Zhang et al. [72] incorporated optical flow to address event-to-image reconstruction as a linear inverse problem. Our work is the first to introduce text as a guiding prior for event-to-video reconstruction, which, as noted by reviewer ***ueNs***, *\"could inspire future work in the field\".*\n\n> **Why is it necessary for the event activation regions of adjacent frames to be similar?**\n> \n- Sorry for the misunderstanding. It should be the regions without events to be similar. The value zero is assigned to the event-activated area, and vice versa. We will rectify Line 196 in the version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "DyHt72GW9l", "reviewer": "Reviewer_H8fU", "summary": "The paper explores the use of Denoising Diffusion Models (DDM) for event-based video reconstruction task. The most important contribution, in my opinion, is the improvement in video quality. The researchers adapted an existing model (ModelScope Text-to-Video) for the  event-based video reconstruction task. They introduced ESA, that modiffies a conditional inpainting technique (presented on High-Resolution Image Synthesis with Latent Diffusion Models). Additionally, they used text conditioning to further enhance video quality.\n\nThe innovative contributions of the paper are two. The fist one is the Event-aware Noise Initialization; This technique enables the use of frame I_(t-1) to reconstruct frame I_(t) during the inference stage (autoregressive).\nThe second one is the Event-aware Mask Loss; This new loss function is designed to improve the temporal consistensy.\n\nIn summary, the contributions are:\n\n1. An existing text-to-video model (diffusion-based) was adapted for the event-based recosntruction task.\n2. The model was adapted to event data using a conditional inpainting technique. Also proposed Event-aware Noise Initialization and Mask Loss to improve video quality.\n3. The result is a new state-of-the-art in event-to-video reconstruction.\n\nHowever, the proposed model has several drawbacks. The DDM \"hallucinates\" content, especially in areas with low or no event data, which can be problematic for applications like object detection in the context of self-driving cars.", "questions": "1) Although traditional metrics (LPIPS, MSE, and SSIM) show an increase in video reconstruction quality, the qualitative results, especially in Figure 3, reveal that the reconstructed frames do not reflect reality in specific areas. These artifacts represent a smaller percentage of the image (in terms of pixels) and are not captured by traditional metrics, how does this affect the overall video quality?\n\n2) In the inference stage, no mention is made of the length (number of frames) that the model can reconstruct. If there is a limit on the number of frames (how many frames and) , how is temporal consistency between reconstructed sections resolved?\n\n3) Despite discussing the model's temporal consistency, no metrics verify this.\n\n4) On line 134, the variable V is introduced, representing the conversion of event data to voxels. However, it is unclear whether V represents a segment with N temporal bins or N segments (the assumption is that V represents N segments). Clarification of this detail is necessary.\n\n5) On line 139, the input latent representation is introduced through the variable \\hat{\\epsilon} . In DDM, \\epsilon variable represents noise, and the noisy input latent representation normaly is denoted with z_t in this case as \\hat{z}^{i}_{t}, since it is the latent image after applying the noise and the representation of the events z^{i}_{e}. I'm not sure if it's correct. \n\n6) The Event-guided Spatio-temporal Attention (ESA) module is very similar to the technique used in SD (High-Resolution Image Synthesis with Latent Diffusion Models) for inpainting. A reference to this would be beneficial.\n\n7) In section 4.2, on line 162, the title \"Event Spatial Attention\" mentions a cross-attention technique. However, the title does not reflect this, causing confusion. It could be changed to \"Event Spatial Cross-Attention\". Similarly, on line 176, the title \"Event Temporal Attention\" maybe should be changed to \"Event Temporal Cross-Attention.\"\n\n7) In equation 8, line 199, the value of \\(\\lambda\\) is not mentioned.\n\n8) No mention is made of the computational cost in the inference stage, nor is the inference time mentioned. Could these data be mentioned?", "limitations": "Due to the DDM's tendency to hallucinate, the model cannot be used in self-driving cars or other computer vision applications where safety is involved.\n\nIt is believed that the model cannot be run in real time, much less on an embedded device.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "As mentioned before, the main contribution of this paper is the improvement in video quality. Also, this paper proves that it is possible to use diffusion-based models for the event-based video reconstruction task.", "weaknesses": "Diffusion-based models tend to \"hallucinate,\" producing some parts on a frames that are far from reality (in the event-based video reconstruction task), which can be problematic for applications like object detection in the context of safety (self-driving cars). \n\nAnother problem is that the proposed model uses text prompts for video generation. Although these text prompts allow some control over the content, they also introduce a lot of ambiguity. This leads to a trial-and-error process (prompt engineering) until the most realistic reconstruction is achieved according to the user.\n\nAdditionally, the proposed model requires very high computational resources. Inference times are not specified in the paper, but it is known that the model does not run in real-time.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank the reviewer for appreciating our work with valuable suggestions. We address the comments below.\n\n> **About the \"hallucination\" issue of diffusion-based models.**\n> \n- Yes, this is indeed a problem for the diffusion models. To address this, we have proposed novel techniques (i.e., ESA module, Event-aware Mask Loss, and Event-aware Noise Initialization) to mitigate the diversity objectives of the diffusion model. The proposed techniques thereby enhance the consistency between events and the video and reduce the hallucination issue.\n- Please be noted that, in static areas without event data, the model tends to reconstruct these regions with hallucination. By incorporating text descriptions, the hallucination issue can be mitigated to some extent, as shown in Fig. 7 (left).\n\n> **Text could introduce ambiguity and lead to a rial-and-error process.**\n> \n- Unlike existing image synthesis pipelines, our method primarily relies on event data, with text descriptions serving as the **supplementary guidance** only when events are too sparse or not trigged. This approach minimizes the potential for ambiguity.\n- Additionally, our method requires only **coarse** text descriptions to effectively leverage the semantic prior in the T2I diffusion model. Most of these descriptions simply identify objects, such as \"car\" or \"road,\" generated by the tagging model RAM, which introduces little ambiguity. As shown in Tab. 1 and Fig. 4, our experiments confirm that this coarse-grained text is effective and does **not** necessitate a trial-and-error process.\n\n> **Computational Resources and Inference Times.**\n> \n- Thanks for the suggestion. We have included a complexity comparison in **Global Response to All Reviewer*s***. As shown in Tab. A-1, our method requires considerable inference time due to multiple denoising steps, showing a limitation of diffusion models. However, our approach “*could inspire future work in the field*”(***ueNs***) by incorporating language guidance and proves “*it is possible to use diffusion-based models for the event-based video reconstruction task*”(***H8fU***).\n\n> **The artifacts in Fig. 3 are not captured by traditional metrics in Tab. 1.**\n> \n- The reviewer might misunderstand Fig.3. Fig. 4 presents the qualitative results corresponding to the quantitative data in Tab. 1, derived from the widely-used ECD, MVSEC, and HQF datasets. These results demonstrate a significant improvement in reducing artifacts. Conversely, Fig. 3 illustrates qualitative results from a **different** dataset (HS-ERGB) that involves fast motion.\n- Additionally, the traditional metrics (MSE, SSIM, and LPIPS) used in Tab. 1 are widely recognized in recent E2V research. These metrics have been proven to effectively assess results at the pixel level (MSE), structure level (SSIM), and in terms of human perception (LPIPS).\n\n> **How is temporal consistency between reconstructed sections resolved?**\n> \n- As mentioned in Line 184, we condition the **frame from the previous section** to enable an **auto-regressive** pipeline for long video reconstruction. The ablation study in Tab. 2 (2nd row vs. 3rd row) has demonstrated the effectiveness of the previous frame condition to ensure the temporal consistency between sections. Theoretically, the auto-regressive pipeline imposes **no limitations** on the number of frames.\n\n> **Metrics for temporal consistency.**\n> \n- Upon the suggestion, we evaluate the results based on the temporal quality metrics from VBench [76]. As shown in Tab. A-4, our method significantly outperforms others on subject consistency and background consistency, while achieving comparable performance on motion smoothness. These results demonstrate the effectiveness of our approach in maintaining temporal consistency. We will include these results in the camera-ready version.\n\nTable A-4: Quantitative Comparison on temporal consistency on ECD based on VBench metrics.\n\n|  | Subject Consistency | Background Consistency | Motion Smoothness |\n| --- | --- | --- | --- |\n| E2VID [40] | 52.14% | 85.26% | 97.62% |\n| FireNet [44] | 49.61% | 82.78% | 98.40% |\n| E2VID+ [48] | 51.83% | 85.33% | 97.56% |\n| FireNet+ [48] | 47.97% | 83.23% | 97.11% |\n| SPADE-E2VID [7] | 50.58% | 82.61% | **98.41%** |\n| SSL-E2VID [37] | 51.89% | 84.86% | 95.97% |\n| ET-Net [54] | 55.49% | 86.85% | 97.72% |\n| HyperE2VID [12] | 50.41% | 83.50% | 97.59% |\n| **LaSe-E2V (Ours)** | **84.25%** | **93.39%** | 98.11% |\n| *GT (Empirical Max)* | *88.29%* | *93.65%* | *98.67%* |\n\n> **Clarification on the Variable \"V\".**\n> \n- Thanks for the question. The **V** in bold represents N segments. We will clarify this in the final version.\n\n> **Clarification on the Variable $\\epsilon$.**\n> \n- Yes, the noisy input latent representation is typically denoted as $z_t$ during the training process, while $\\epsilon$ refers to the sampling noise input during inference. Sorry for the confusion. We will clarify it in the final version.\n\n> **Adding reference to ESA.**\n> \n- Thanks for the suggestion. We will add the reference for this module in the revision. While SD [42] serves as a baseline architecture to incorporate text embedding, we introduce ESA to enhance spatial-temporal consistency between the event data and video by considering the unique characteristics of event data.\n\n> **Adding Cross-Attention to section title of ESA.**\n> \n- Thanks for the insightful comment. We will revise the title in the final version to better reflect the inclusion of cross-attention.\n\n> **About the value of $\\lambda$.**\n> \n- Thanks for the question. The value of $\\lambda$ is set to 0.01 for all the experiments. We will clarify it in the implementation details.\n\n**Additional Reference:**\n\n[76] Huang Z, He Y, Yu J, et al. Vbench: Comprehensive benchmark suite for video generative models[C]/CVPR. 2024.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ejlTnylnA0", "reviewer": "Reviewer_uWG8", "summary": "This paper proposes a language-guided event-based video reconstruction method LaSe-E2V, which introduces the popular language model into event-based imaging tasks. The LaSe-E2V is generally based on a diffusion model. In order to further improve the method, this paper proposes a series of designs, such as event-guided spatio / temporal attention for the fusion between event and previous reconstructed frames, previous frame conditioning for ensuring consistency, event-aware mask loss, and event-aware noise initialization. Experimental results demonstrate the effectiveness of the proposed methods.", "questions": "1. The proposed method requires a description of scenes contained in the event streams to be reconstructed. What if the description of the scene is unknown (For example, a ``blind'' event stream)? Can existing models handle this situation?\n2. The running times and GPU memory costs for inference are absent. Diffusion models are still new for the event-based imaging community, and offering these values can help readers better understand the proposed method and do subsequent work.\n3. It says in Line 228 of the paper that the definition of the SSIM metric has ambiguity. What's the ambiguity?\n4. In the 3rd column of Fig. 4, there are more artifacts in the text part reconstructed by the proposed method than other methods. It's an interesting phenomenon, and it would be better if some explanations were given.\n5. There is a typo ``evet'' in the Line 66 of the manuscript.", "limitations": "Please see the Questions part.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper introduces the popular language model into the event-based imaging community, providing new ideas for subsequent works.\n2. The event-guided spatio / temporal attention provides a new idea for event-image fusion.\n3. The event-aware mask loss and the event-aware noise initialization are specially designed for event-based imaging based on diffusion models.", "weaknesses": "There are some unclear points, please see the Questions part.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank the reviewer for the constructive comments and valuable concerns.\n\n> **Can the model handle the situation when the description of the scene is unknown?**\n> \n- Yes, our model can handle such scenarios. If the text description is unavailable, our framework defaults to a conventional event-to-video approach. As demonstrated in Tab. 2 (row 1), even without text input, our method **still achieves reasonable performance**.\n- Additionally, event-based multi-modality models such as EventBind[77] and EventClip[78] can also be employed to **generate text descriptions directly from the event stream**.\n\n> **About running times and GPU memory costs for inference.**\n> \n- Thanks for the question. We have included a complexity comparison in **Global Response to All Reviewers.** As illustrated in Tab. A-1, our method requires significant inference time due to the multiple denoising steps, showing a known limitation of diffusion models. However, while diffusion models have been employed in tasks like super-resolution and depth estimation, “*they are still new to the event-based imaging community*”(***uWG8***). As such, our approach could hopefully *“provide new ideas for subsequent works”(**uWG8**)* in this field.\n\n> **Ambiguity of SSIM metric.**\n> \n- The SSIM metric raises ambiguity because it involves several hyper-parameters that may differ across various codebases. For example, in the `structural_similarity` function of the **skimage** package, parameters like `gaussian_weights` and `sigma` are used for spatial weighting of each patch with a Gaussian kernel, `use_sample_covariance` indicates whether to normalize covariances, and `K1` and `K2` are algorithm-specific parameters that need to be set.\n\n> **About explanations for the artifact of text reconstruction in Fig.4.**\n> \n- Thanks for the insightful question. Although our method achieves superior overall performance, some artifacts persist in the text part reconstruction. This issue arises because we depend on the prior of the pre-trained diffusion model (SD2 [42]), which also faces challenges in text generation. However, the recently released SD3 [79] claims to show improved text rendering capabilities, which could potentially address this problem.\n\n> **Typo: \"evet\".**\n> \n- Thanks for pointing out the typo. We will correct it in the final version.\n\n**Additional Reference:**\n\n[77] Zhou J, Zheng X, Lyu Y, et al. E-clip: Towards label-efficient event-based open-world understanding by clip[J]. arXiv, 2023.\n\n[78] Wu Z, Liu X, Gilitschenski I. Eventclip: Adapting clip for event-based object recognition[J]. arXiv, 2023.\n\n[79] Esser P, Kulal S, Blattmann A, et al. Scaling rectified flow transformers for high-resolution image synthesis[C]//ICML. 2024.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
### **Global Response to All Reviewers**

We sincerely thank the reviewers for their constructive feedback. We are pleased that the reviewers found our **method to be novel and effective**, the **performance to be strong and with high-quality**, which is believed to **inspire future work** in the community. Below, we address the questions raised and promise to throughly revise the paper accordingly.

> **Complexity analysis**
> 
- Following the reviewers' suggestions, we have provided a detailed computational complexity analysis in Tab. A-1, which includes recent event-to-video reconstruction methods and related diffusion-based approaches. Our method requires a significant amount of inference time due to the 50 denoising steps, in contrast to previous single-step event-to-video methods. However, we kindly note that this is a common challenge for all diffusion-based models, as observed in diffusion-based super-resolution [69,70] and depth estimation methods [66,67]. To reduce inference time, some research, eg., [65], already explored to decrease the number of denoising steps, offering a promising direction for further improvement (as a future work) to our framework.
- Please be noted that previous approaches [54, 12]  often struggle to recover regions *without active events*. In contrast, as demonstrated in Fig.1, our method achieves **holistic, semantic-aware reconstruction**. As acknowledged by reviewers ***ueNs*** and ***uWG8***, this paper aims to *"**provide new ideas for subsequent works**"* and *"**inspire future work in the field**"* by incorporating language guidance.
- As affirmed by reviewer ***H8fU***, this paper demonstrates that "***it is possible to use diffusion-based models for the event-based video reconstruction task***," thereby leveraging the rich semantic priors of large-scale LDM. This approach holds promise for extending to other event-based tasks, including video frame interpolation, deblurring, and denoising.

In summary, albeit with lower inference speed than conventional E2V methods (but higher than diffusion-based super-resolution and depth estimation methods), our work brings new ideas and may hopefully inspire new future research for event-based vision.

Table A-1: Complexity comparison on various methods. All tests are conducted on one NVIDIA Tesla 32G-V100 GPU.

| Methods |  | Parameters | Inference time (per frame) |
| --- | --- | --- | --- |
| Conventional Event-to-Video | ET-Net [54] | 22.18M | 0.0124s |
|  | HyperE2VID [12] | 10.15M | 0.0043s |
| Diffusion-based Depth Estimation | DepthFM [66] | 891M | 2.1s |
|  | Marigold [67] | 948M | 5.2s |
| Diffusion-based Super-Resolution | StableSR [68] | 1409M | 18.70s |
|  | PASD [69] | 1900M | 6.07s |
|  | SeeSR [70] | 2284M | 7.24s |
| Diffusion-based Event-to-Video | **LaSe-E2V (Ours)** | 1801M | 1.09s |

**Additional Reference**:

[65] Liu X, Zhang X, Ma J, et al. Instaflow: One step is enough for high-quality diffusion-based text-to-image generation[C]//ICLR. 2023.

[66] Gui M, Fischer J S, Prestel U, et al. Depthfm: Fast monocular depth estimation with flow matching[J]. arXiv, 2024.

[67] Ke B, Obukhov A, Huang S, et al. Repurposing diffusion-based image generators for monocular depth estimation[C]//CVPR. 2024.

[68] Wang J, Yue Z, Zhou S, et al. Exploiting diffusion prior for real-world image super-resolution[J]. IJCV, 2024: 1-21.

[69] Yang T, Ren P, Xie X, et al. Pixel-aware stable diffusion for realistic image super-resolution and personalized stylization[J]. arXiv, 2023.

[70] Wu R, Yang T, Sun L, et al. Seesr: Towards semantics-aware real-world image super-resolution[C]//CVPR. 2024.

Meta-review: This paper is at borderline with ratings of 1x borderline reject, 1x borderline accept and 2x weak accepts. The reviewers mention that the strengths of this paper is that it is the first to tackle tackle the event data reconstruction task from a language-guided perspective. This approach could inspire future work in the field. The proposed algorithm achieves optimal performance on nearly all metrics across multiple datasets, demonstrating superior visual effects and validating the algorithm's effectiveness. The paper presents a high-quality body of work, including effective methods and extensive experimental validation. This paper proves that it is possible to use diffusion-based models for the event-based video reconstruction task. The event-guided spatio / temporal attention provides a new idea for event-image fusion. The reviewer who gave weak reject mentioned that no ablation given for the ESA module and no reported flops and parameters are given in Tab. 1 of the paper. These concerns are addressed in the authors' rebuttal. On the consideration of the strengths outweighing the weaknesses, the metareviewers decide to accept the paper.
Decision: Accept (poster)