LaSe-E2V: Towards Language-guided Semantic-aware Event-to-Video Reconstruction

Authors: kanghao  chen
hangyu  li
jiazhou  zhou
zeyu  wang
lin  wang
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-14 12:57:53

Abstract: Event cameras harness advantages such as low latency, high temporal resolution, and high dynamic range (HDR), compared to standard cameras. Due to the distinct imaging paradigm shift, a dominant line of research focuses on event-to-video (E2V) reconstruction to bridge event-based and standard computer vision. However, this task remains challenging due to its inherently ill-posed nature: event cameras only detect the edge and motion information locally. Consequently, the reconstructed videos are often plagued by artifacts and regional blur, primarily caused by the ambiguous semantics of event data. In this paper, we find language naturally conveys abundant semantic information, rendering it stunningly superior in ensuring semantic consistency for E2V reconstruction. Accordingly, we propose a novel framework, called LaSe-E2V,  that can achieve semantic-aware high-quality E2V reconstruction from a language-guided perspective, buttressed by the text-conditional diffusion models. However, due to diffusion models' inherent diversity and randomness, it is hardly possible to directly apply them to achieve spatial and temporal consistency for E2V reconstruction. Thus, we first propose an Event-guided Spatiotemporal Attention (ESA) module to condition the event data to the denoising pipeline effectively. We then introduce an event-aware mask loss to ensure temporal coherence and a noise initialization strategy to enhance spatial consistency. Given the absence of event-text-video paired data, we aggregate existing E2V datasets and generate textual descriptions using the tagging models for training and evaluation. Extensive experiments on three datasets covering diverse challenging scenarios (e.g., fast motion, low light) demonstrate the superiority of our method. Demo videos for the results are attached to the project page.

Main paper: 
 introduction: 
 Event cameras are bio-inspired sensors that detect per-pixel intensity changes, producing asynchronous event streams [5] with high dynamic range (HDR) and high temporal resolution. They particularly excel in capturing the edge information of moving objects, thus discarding the redundant visual information. Such a distinct imaging shift poses challenges for integration with the off-the-shelf vision algorithms designed for standard cameras. To bridge the event-based and standard computer vision [27, 31, 3, 29] , a promising way is event-to-video (E2V) reconstruction.Recently, deep learning has been applied to E2V reconstruction [49, 54, 64] , and remarkable performance is achieved thanks to the availability of synthetic event-video datasets and the development of model architectures. Most existing research emphasizes on the network design [64, 13, 54, 34] or high-quality data synthesis [57, 34] . However, this task remains challenging due to its inherently ill-posed nature: event cameras capture edge and motion information locally but neglect semantic Aggregated event frames Event stream Figure 1 : Comparison of the E2V pipeline between HyperE2VID [13] and our LaSe-E2V: The baseline method solely relies on event data, leading to ambiguity in local structures. In contrast, our approach integrates language descriptions to enrich the semantic information and ensure the video remains coherent with the event stream. information in regions with no intensity changes (See Fig. 1 (a) ). Consequently, the reconstructed videos, e.g., HyperE2VID [13] , are plagued by artifacts and regional blur, as shown in Fig. 1 To fill this gap, in this paper, we find language naturally conveys abundant semantic information, which is beneficial in enhancing the semantic consistency for the reconstructed video, as shown in Fig. 1 (c ). Intuitively, we propose a novel language-guided semantic-aware E2V reconstruction framework, called LaSe-E2V, with the text-conditional diffusion model [52] as the backbone. While many efforts [62, 51, 23] apply diffusion models for images and video generation, adapting them to our problem is hardly possible. The reason is that the inherent randomness in diffusion sampling and the diversity objectives of generative models may result in temporal inconsistency between consecutive frames and spatial inconsistency between the event data and reconstructed videos. Therefore, we first introduce an Event-guided Spatio-temporal Attention (ESA) module to enhance reconstructed video by introducing spatial and temporal event-driven attention layers, respectively (Sec. 3.2). This module not only achieves fine-grained spatial alignment between the events and video (See Fig. 7 (right)) but also maintains temporal smoothness and coherence by adhering to the temporal properties of event cameras. We then introduce an event-aware mask loss to maintain temporal coherence throughout the video by considering the spatial constraints of event data from adjacent frames (Sec. 3.3) . Lastly, we propose a noise initialization strategy that utilizes accumulated event frames to provide layout guidance and reduce discrepancies between the training and inference stages of the denoising process (Sec. 3.4) .Given the absence of event-text-video paired data, we aggregate existing E2V datasets and employ tagging models to generate textual descriptions, thereby facilitating both training and evaluation processes. Extensive experiments on three widely used real-world datasets demonstrate the superiority of our method in enhancing the quality and visual effects of reconstructed videos, especially its super generalization ability when applied in challenging scenarios, e.g., fast motion, and low light (See Fig. 4 and Fig. 5 ). In summary, the contributions of our work are three-fold: (I) We explore E2V reconstruction from a language-guided perspective, utilizing the text-conditioned diffusion model to effectively address the semantic ambiguities inherent in event data. (II) We propose the event-guided spatio-temporal attention mechanism, an event-aware mask loss, and a noise initialization strategy to ensure the semantic consistency and spatio-temporal coherency of the reconstructed video. (III) We have rebuilt event-text-video paired datasets based on existing event datasets with textual descriptions generated from off-the-shelf models [73] . Extensive experiments on three datasets covering diverse scenarios (e.g., fast motion, low light) demonstrate the effectiveness of our framework. 
related works: 
 Event-to-Video (E2V) Reconstruction. E2V reconstruction falls into two categories: modelbased and learning-based methods. Model-based approaches [2, 44, 6, 53] exploit the correlation between events and intensity frames through hand-crafted regularization techniques. However, its reconstruction result is comparatively inferior to the more recent learning-based methods [50, 49, 54, 34, 13] . For example, E2VID [50, 49] used an Unet-like network with skip connections and ConvLSTM units to reconstruct videos from long event streams. Following E2VID, SPADE-E2VID [7] extended E2VID to enhance temporal coherence by feeding previously reconstructed frames into a SPADE block. HyperE2VID [13] introduced hypernetworks to generate per-pixel adaptive filters guided by a context fusion module. GANs, such as conditional GAN [61] and cycleconsistency GAN [69] , are used to address this issue, but often exhibit blurry images with artifacts in non-activated areas. In summary, previous learning-based methods can only achieve plausible reconstructed results for this ill-posed problem because event data solely captures motion information without semantic context information. In this work, we explore the possibility of incorporating textual descriptions with semantic awareness for E2V reconstruction.Text-to-Video Diffusion Models. The success of Diffusion models [45, 52] in generating highquality images from text prompts has advanced text-to-image (T2I) synthesis. Inspired by T2I synthesis, text-to-video (T2V) diffusion models, such as [4, 19, 20, 23, 30, 40, 55, 60, 66] adapt T2I synthesis to video. Subsequent developments [23, 4, 18] focused on refining the temporal information interaction, such as optimizing temporal convolution or self-attention modules for motion learning. Meanwhile, one T2V research line focuses on enhancing controllability by incorporating additional conditions, thus addressing the text prompts' ambiguity in motion, content, and spatial structure. For high-level video motion control, studies propose learning LoRA [24] layers for specific motion patterns [18, 75] , or utilizing extracted trajectories [68] , motion vectors [62] , pose sequences [41] to guide the synthesis. For fine-grained spatial structure control, methods like Gen-1 [14] , VideoComposer [62] , and others [42, 70] leverage monocular depth, sketch, and image control models [42, 70] for flexible and controllable video generation [8, 18, 30, 72] .Formally, given a video sample x 0 , the latent diffusion model (LDM) [52] first encodes it into a latent feature z 0 = E I (x 0 ). A noisy input is obtained based on the forward diffusion process by introducing Gaussian noise to the latent representation: z t = √ ᾱt z 0 + √ 1 -ᾱt ϵ, where t = 1, ..., T and T denotes the maximum timestep. ᾱt = t i=1 (1 -β i ) is the coefficient that controls the noise strength. The diffusion model is trained by predicting the noise ϵ through a mean squared error:EQUATIONwhere θ denotes the parameters of the U-Net of the diffusion model. c denotes the addition conditions (text, image, depth, et al.) to control the diffusion process. In the inference stage, the generated sample x0 can be obtained from the denoised latent ẑ0 using a pre-trained decoder x0 = D I (ẑ 0 ).These methods achieve fine-grained controllability based on the well-designed condition c, focusing on flexible user instructions and diverse outcomes. In this study, we employ a basic T2V diffusion model with the 3D-UNet [60] architecture as the foundation for language-guided E2V reconstruction.In particular, our framework prioritizes video fidelity by aligning motion details from event data and semantic insights from text, thus outperforming previous models, especially in extreme scenarios like fast motion and low light.3 The Proposed LaSe-E2V Framework Event Representation. An event stream E = {e t k } Ne k=1 consists of N e events. Each event e t k ∈ E is represented as a tuple (x k , y k , t k , p k ), which denotes the pixel coordinates, timestamp, and polarity. To make the event stream compatible with the pre-trained diffusion model, we convert E into a grid-like event voxel grid V ∈ R B×H×W with B time bins using temporal bilinear interpolation [77] . 
overall pipeline: 
 The goal of our LaSe-E2V is to reconstruct a video x = {x 1 , x2 , ..., xN } ∈ R N ×C×H×W from the set of event segments {E i } N i=1 , where E i is i-th event segment corresponding to the frames. The reconstructed video is expected to retain the motion and edge content from the event data and compensate for the semantic information from the language description. The difficulty of our task lies not only in achieving high visual quality that aligns with the text descriptions but also in maintaining content consistency with the event data. We address the difficulty by integrating the diffusion model Temporal A+en-on with events and text descriptions effectively. As shown in Fig. 2 , LaSe-E2V consists of four major components: an image encoder E I , an event encoder E e , a video LDM, and a decoder D I .𝑥 1 , 𝑥 2 , … , 𝑥 # 𝑥 & 1 , 𝑥 & 2 , … , 𝑥 & 𝑁 ❄ 𝑫 𝑰 𝑧 $ (a) (b) (c) 𝑧 $ Text 𝑧 $ 𝑧 % & , 𝑧 % ' , … , 𝑧 % # 𝑧 (Image Encoder E I . Given a sequence of frames x = {x 1 , x 2 , ..., x N }, we extract the latent representation z = {z 1 , z 2 , ..., z N } with the pre-trained image encoder of LDM [52] .Event Encoder E e . To adapt the event voxel grids to the latent space of E I , we first project it into a latent representation with an event encoder E e (V). As illustrated in Fig. 2 (a), we initially apply lightweight spatial 2D and temporal 3D convolution blocks to the set of input voxelV = {V i } N i=1with N segments. These blocks are designed to extract local spatial and temporal feature information, which is then processed through an attention block (AttBlock) for global temporal modeling. Subsequently, we concatenate the event latent representation with the noise latent representation along the channel dimension to serve as a rough condition for the model. Finally, we construct the input latent representation asz t = {[z i t , z i e ]} N i=1 ∈ R N ×(C ′ +Ce)×H ′ ×W ′, where z i t and z i e denote the noise latent and event feature corresponding to the i frame with channel dimensions C ′ and C e .Video LDM. We first extend the capabilities of the text-conditional image-based LDM by incorporating temporal layers in the U-Net, following the approach of video diffusion models [14, 23, 55, 62] . As depicted in Fig. 2 , our framework presents a video LDM with multiple blocks consisting of 2D spatial convolution, 3D temporal convolution, 2D spatial attention, and 3D temporal attention layers. A pre-trained CLIP [48] text encoder is employed to extract text descriptions conditioning on the attention layers to provide semantic information for the reconstructed video. To facilitate fine-grained control of event data, we propose an Event-guided Spatio-temporal Attention (ESA) module (see Fig. 2 (b) and (c) ) following E e to convert the event voxel grids to the latent space. The ESA module enhances the control by introducing a specific spatial and temporal cross-attention in the U-Net. The technical details will be discussed in Sec. 3.2.Decoder D I . Finally, we apply the pre-trained decoder D I of [52] to convert the estimated latent representation ẑ to video x in the image space.Event-aware Mask Loss l m . To optimize LaSe-E2V, we propose a novel event-aware mask loss l m , in addition to the ϵ-prediction loss (Eq. 1). The details of l m will be described in Sec. 3.3.In addition, different from previous LDM models [10, 51] , we introduce a noise initialization strategy to alleviate the train-test gap in the inference stage, which will be described in Sec. 3.4.Relying solely on the 3D U-Net and feature concatenation is insufficient to reconstruct a contentconsistent video with fine-grained control from event data, as demonstrated experimentally in Fig. 7 (right). To address this, we propose using the event latent representation to condition spatial and temporal attention, ensuring more reliable spatial alignment and temporal consistency (See Tab. 4). Our ESA module is specially designed to enhance the spatio-temporal consistency between events and video in contrast to the original SD [52] , which serves as a baseline attention mechanism for integrating conditional input. Our approach differs in attention design, which introduces two distinct attention mechanisms respective to the spatial domain and temporal domain.Event Spatial Cross-Attention. As illustrated in Fig. 2 (b), the 2D spatial attention layer in the LDM U-Net integrates a self-attention mechanism that processes each frame individually and a crossattention mechanism connecting frames with text embedding, which follows Stable Diffusion [52] . Intuitively, Event data naturally deliver abundant edge information in the spatial space, which is expected to directly constrain the structural information of the reconstructed video. To incorporate event information in the spatial attention module, we concatenate the features from the event latent z i e to the U-Net intermediate features z to formulate a event-based spatial attention:EQUATIONwhereQ = W Q z i , K e-s = W K [z i , z i e ] s , V e-s = W V [z i , z i e ] s .[ * ] s represents the concatenation operation on the spatial dimension. This modification ensures that each spatial position in all frames accesses comprehensive information from the corresponding event data, enabling detailed structural feature control within the spatial attention layers.Event Temporal Cross-Attention. In addition to the spatial information, event data inherently represents the difference between the adjacent frames, which is a strong constraint. As shown in Fig. 2 (c ), to effectively leverage the temporal constraint of the event data, we facilitate event features into temporal attention. Specifically, given the intermediate features z, we first reshape the height and width dimensions into the batch dimension, forming a new hidden state z ∈ R (H×W )×N ×C . Then the event latent feature z i e is employed to interact with the latent feature z in the temporal dimension through the temporal attention layer:EQUATIONwhereQ = W Q zi , K e-t = W K [z i , z i e ] t , V e-t = W V [z i , z i e ] t .[ * ] t represents the concatenation operation on the temporal dimension.Previous Frame Conditioning. To ensure consistency for the whole video reconstruction, we use an autoregressive way to reconstruct the video, by conditioning the model on the last frame xN of the previously estimated video clip x. We concatenate the previous frame with the latent representation ẑ on the temporal dimension as a condition to obtain z = [z 0 0 , z t ] ∈ R (N +1)×C ′ ×H ′ ×W ′ . Due to the absence of the previous frame in the first video clip, we randomly drop the previous frame condition by applying Gaussian noise to the frame latent representation. During inference, the first video clip is reconstructed directly from the events and text data without the previous frame. For subsequent video clips, the reconstruction process utilizes the last frame from the previous video clip as a condition to ensure temporal coherency and smoothness.Considering the noise prediction loss can not capture the event constraint on temporal coherence, we introduce an event-aware mask loss to directly supervise the inter-frame difference:EQUATIONwhere M = I(V) indicates the event motion mask obtained by setting value one for event activated area and value zero for non-activated part; ẑ0 represents the model's estimated clean video latent, which can be obtained by:EQUATIONFinally, we combine the noise prediction loss and the event-aware mask loss with the scaling factor λ. EQUATIONDuring training, we construct the input latent by adding noise on the clean video latent. The noise schedule leaves some residual signal even at the terminal diffusion timestep T . As a result, the diffusion model has a domain gap to generalize the video during the inference time when we sample from random Gaussian noise without any real data signal. To solve this train-test discrepancy problem, during the testing, we obtain the base noise by adding noise on event-accumulated frames using the forward process of DDPM [21] . The noise latent for frame i can be expressed as:EQUATIONwhere α T denotes the diffusion factor and z t e = E I (I -1 + t 0 e i ). The I -1 is the last frame of the previous estimated video clip. Intuitively, accumulated event data provides structural information (e.g., edges) in the scene, acting as an additional spatial constraint during the denoising process. 
datasets and implementation details: 
 Dataset. We train our pipeline using both synthetic and real-world datasets. For synthetic data, following prior arts [50, 34] , we generate event and video sequences from the MS-COCO dataset [33] using the v2e [25] event simulator because it ensures stable and high-quality ground truth images. To enrich semantic information, we also utilize the real-world dataset BS-ERGB [58] , which includes 1k training sequences. For all datasets, we employ the off-the-shelf tagging model RAM [73] to generate language descriptions. Recent work [65] has demonstrated the superiority of RAM compared to other prompting models due to its rich objects and concise description. We evaluate our model on Event Camera Dataset (ECD) [43] , Multi Vehicle Stereo Event Camera (MVSEC) dataset [76] and High-Quality Frames (HQF) dataset [57] . RAM is also used to generate tags for these datasets.Implementation Details. Based on Stable Diffusion 2.1-base [52] , we use a text-guided video diffusion model [51] to initialize our model, pre-trained on large-scale video datasets [1] . For each training video clip, we sample 16 frames and the corresponding event streams, with an interval of 1 ≤ v ≤ 3 frames. The input size is adapted to 256 × 256. Following previous methods [64, 57] , the data augmentation strategies include Gaussian noise, random flipping, and random pause. The value λ is set to 0.01 for all experiments. The model is trained with the proposed loss across all U-Net parameters, with a batch size of 3 and a learning rate of 5e-5 for 150k steps on 8 NVIDIA V100 GPUs. We also provide an analysis of the computation complexity. See more details in Appendix 5.Evaluation Metric. The Mean Squared Error (MSE), Structural Similarity (SSIM [63] ), and Perceptual Similarity (LPIPS [71] ) are used to measure image quality. The SSIM metric raises ambiguity as it involves several hyperparameters that may differ across various codebases. For this reason, we reevaluated all compared methods using a unified metric, denoted as the SSIM * scores. LaSe-E2V performs better to preserve the HDR characteristic of event cameras with higher contrast.Quantitative results. We compare LaSe-E2V with eight existing learning-based methods [50, 54, 57, 57, 7, 46, 64, 13] . To ensure fair comparison, we maintain identical experiment settings without any post-processing operations across all methods. As shown in Tab. 1, LaSE-E2V mostly outperforms previous methods. In particular, LaSE-E2V significantly surpasses HyperE2VID by 30% on MSE on the ECD dataset. LaSe-E2V also excels in SSIM and LPIPS mostly. This highlights the superiority of the structural and semantic reconstruction ability of LaSE-E2V.Qualitative Results. Fig. 3 illustrates the qualitative results reconstructed by our LaSe-E2V and previous methods. As shown in the street reconstruction (Column 1), our method reconstructs more details, especially for the car and the building. For the bike in Column 2, our method achieves clearer edges and higher contrast, while the previous methods are inclined to exhibit foggy artifacts around the bikes. Although our method achieves superior performance, some artifacts persist for the reconstruction of the text region (e.g.3 rd row). This issue arises because it depends on the prior of the pre-trained diffusion model (SD2 [52] ), which faces challenges in text generation. The recently released SD3 [15] claims to show improved text generation capabilities, which could potentially address the problem. Please refer to the video demo and the appendix for additional qualitative results, which demonstrates the superiority of our LaSe-E2V on temporal smoothness and consistency.Results with Fast Motion. In Fig. 4 , we show sampled reconstructed frames based on sequences of HS-ERGB [58] captured by Prophesee Gen4 (1280 × 720) event camera with high resolution and relatively fast motion conditions. We can see that our method is capable of clearly recovering the details for high-speed movement and preserving temporal consistency.. We test our model on the video sequences in extreme conditions (i.e.low light and fast motion) to further demonstrate the advantages of event cameras and the effectiveness of our framework. As shown in Fig. 5 , we sample sequences from MVSEC (outdoor_night2) with relatively low-light conditions in nighty streets. We can see that LaSe-E2V performs better in reconstructing the scene with higher contrast and more clear edge. Compared with HyperE2VID exhibiting foggy artifacts of the whole street, E2VID can reconstruct video without over-exposure or under-exposure.Quantitative Result on Temporal Consistency. We further evaluate the results based on the temporal quality metrics from VBench [26] . As shown in Tab. 2, the numerical results demonstrate the effectiveness of our approach in maintaining temporal consistency. In particular, our method significantly outperforms others on the subject consistency and background consistency, while achieving comparable performance on motion smoothness. Video Editing with Language. In our framework, language serves as supplementary semantic information for E2V reconstruction. We investigate the impact of varying language guidance for video editing, as illustrated in Fig. 8 . Utilizing a text description from the off-the-shelf tagging model [73] ,"night, dark, city street…" "bright, day light, city street …" our method reconstructs a scene with a reasonable structure in low-light conditions based on descriptors like "night, dark, ...". Interestingly, when we manually alter the text description to "bright, day light, ...", the lighting condition in the scene shifts to daylight, revealing clearer details, especially in the sky area. This demonstrates our framework's ability to modify lighting conditions based on textual descriptions, by effectively modeling light conditions as semantic information. In this way, with a languageguided perspective, it offers the flexibility to manually adapt the reconstructed video according to user preferences.Reference FrameColor Video Reconstruction. Based on a pre-trained diffusion model [51] , our model inherits the capability for colored video generation by training on real-world datasets. As illustrated in Fig. 6 , our method successfully reconstructs color videos with clear details, although it occasionally misinterprets background semantics due to the absence of event data.We conducted ablation experiments on the LaSe-E2V framework to evaluate the effectiveness of language guidance, event-based attention, event-aware mask loss, and initialization strategy.Language Guidance. To evaluate the effectiveness of language guidance, we conducted experiments without text conditions by setting the text input to null during the denoising stage.As shown in Tab. 3, introducing the text description achieves a 0.015 improvement on SSIM. As shown in Fig. 7 (left), when provided with the language description "pavement", the model tends to reconstruct the scene more closely to the ground truth, whereas the baseline model randomly generates a background which distinct to the ground truth. This demonstrates the importance of semantic information for E2V reconstruction, especially in cases where semantic ambiguity exists in the event data. We also conduct an ablation study on the previous frame condition, as indicated in Tab. 3 (row 2), where performance significantly dropped across all metrics. This underscores the critical role of the previous frame condition in our diffusion pipeline, ensuring temporal consistency within the autoregressive reconstruction process.Event-guided Spatio-temporal Attention. To demonstrate the effectiveness of our eventguided spatio-temporal attention (ESA), we conducted experiments by training the model with simple channel-wise concatenation for event input. As shown in Tab. 4, performance significantly drops without the attention mechanism, which demonstrates the effectiveness of ESA in preserving the event control on the video. Fig. 7 illustrates that our model maintains visual content close to the ground-truth, whereas the baseline method loses control over lighting and luminance. We further investigate the effectiveness of the event-aware mask loss. As is shown in Tab. 4 , the event-aware loss function achieves a clear margin of improvement compared to the baseline method (row 2). Moreover, Tab. 4 (row 3) demonstrates the effectiveness of our event-based initialization. 
conclusion and future work: 
 Conclusion. In this paper, we introduce LaSe-E2V, a language-guided, semantic-aware E2V reconstruction method. Leveraging language descriptions that naturally contain abundant semantic information, LaSe-E2V explores text-conditional diffusion models with our proposed attention mechanism and loss function, thus achieving high-quality, semantic-aware E2V reconstruction. Extensive experiments demonstrate the effectiveness of our innovative framework.Limitations and Future Work. Despite the promising performance of our method, LaSe-E2V has several limitations. First, the training datasets, comprising synthetic and limited real-world data, are inadequate for optimizing data-intensive diffusion models. Consequently, our method may reconstruct scenes with artifacts differing from the training data. Second, given that LaSe-E2V relies on the diffusion model, multiple denoising steps are required for high-quality videos, slowing the process. Future work can focus on accelerating the inference efficiency based on the recent progress of diffusion models [37, 9, 38, 39] .Broader Impacts. Based on the event camera, our LaSe-E2V enhances the capabilities of various technologies by improving the safety and robustness of intelligent systems. As this technology matures, its integration into everyday devices and systems seems likely, heralding a shift in how visual data is captured and utilized across industries. reasonable output that aligns with human preference and is generally close to the distribution of the real scene. Therefore, while our results on the HS-ERGB dataset may be less significant than those on "constantly moving" datasets (ECD, MVSEC, HQF), our method is still substantially better than baseline methods.Impact of the prompting model. We also tested BLIP [32] on a sampled sequence (i.e., boxes in HQF) to further evaluate the influence of the prompting model, as shown in Tab. 8. BLIP can generate reasonable text prompts in caption-style and show nearly reconstructed performance on MSE (0.025 vs 0.027). 
experiments compute resources: 
 Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?Answer: [Yes]Justification: We provide sufficient information in the appendix.Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). 
code of ethics: 
 Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines? 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes]Justification: We discuss both potential positive societal impacts and negative societal impacts of the work performed.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA]Justification: Our research poses no such risks.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: We explicitly mention and respect the creators and original owners.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. 
 
 Appendix: We begin by discussing the source of text prompts used for event data. Our framework primarily aims to establish a pipeline for reconstructing videos from events using a language-guided approach. This is based on our observation that language naturally conveys rich semantic information, which improves the semantic consistency of the reconstructed video. This form of text guidance resembles text-guided denoising [11, 47] and super-resolution [16] methods, which also use natural language as a user-friendly interface to control the image restoration process. In this work, we do not emphasize the specific sources of text prompts, as these can vary depending on the application scenario. For instance, when using a DAVIS346 camera, text can conveniently be obtained from APS frames. If a Prophesee camera is used and only event data is available, tagging-based text prompts derived from event-based multi-modality models can be employed. Additionally, in complex scenes with limited Regarding the role of text information, it serves as a crucial component to activate the semantic prior in the diffusion model and compensate for missing semantic content. In regions with sufficient event data, our approach enhances reconstruction performance by leveraging the semantic priors provided by the text prompts, ensuring high fidelity. For regions with sparse event data, the method relies solely on the text prompts to reconstruct scenes that align with human expectations. While this approach may introduce textures or details that differ from the actual ones, it still produces images closer to reality than previous methods. In contrast, prior E2V models typically reconstruct such regions as indistinct haze, far from the true distribution of real images, as shown in Fig. 1 . The quantitative comparison in Tab. 7 further demonstrates the superiority of our approach.Admittedly, hallucination is a potential side effect introduced by incorporating semantic priors from the diffusion model into our framework. While our proposed techniques effectively reduce hallucinations, completely eliminating them remains a challenge in the diffusion model research community. Extensive experiments (Tab. 1 and Fig. 3 ) show that our method significantly improves fidelity and reduces hallucinations in scenes with adequate event data. Additionally, our approach exhibits superior reconstruction performance even in scenes with insufficient event data (Tab. 7).It is important to note that eliminating hallucinations entirely for regions with sparse event data is not feasible. However, our method effectively leverages the semantic priors in the diffusion model to produce images closer to the true scene distribution, whereas previous E2V methods (e.g., HyperE2VID) fail in these regions, as demonstrated in Fig. 1 .For future work, given that our method reliably ensures fidelity in regions with sufficient event data, it is practical to assign a confidence map based on event density to identify high-confidence regions.For safety-critical applications, decisions could be made based on both the reconstructed video and the confidence map, allowing simultaneous consideration of image quality and safety. Moreover, event-based multi-modality models offer a potential source of semantic information, which could be explored in future research. Since these models are trained with large-scale event-image paired data, they have the potential to provide prior semantic information absent in event data.• It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified. • For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates). • If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.Justification: We read the guidance carefully and make sure that our research is conducted in accordance with the NeurIPS Code of Ethics.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 


Decision: This paper is at borderline with ratings of 1x borderline reject, 1x borderline accept and 2x weak accepts. The reviewers mention that the strengths of this paper is that it is the first to tackle tackle the event data reconstruction task from a language-guided perspective. This approach could inspire future work in the field. The proposed algorithm achieves optimal performance on nearly all metrics across multiple datasets, demonstrating superior visual effects and validating the algorithm's effectiveness. The paper presents a high-quality body of work, including effective methods and extensive experimental validation. This paper proves that it is possible to use diffusion-based models for the event-based video reconstruction task. The event-guided spatio / temporal attention provides a new idea for event-image fusion. The reviewer who gave weak reject mentioned that no ablation given for the ESA module and no reported flops and parameters are given in Tab. 1 of the paper. These concerns are addressed in the authors' rebuttal. On the consideration of the strengths outweighing the weaknesses, the metareviewers decide to accept the paper.

Reviews: review: paper_summary: paper_summary: This paper uses abundant semantic information and raw event information to guide the reconstruction of RGB images from event images based on U-Net. Furthermore, this paper introduces event-aware mask loss to ensure temporal coherence and a noise initialization strategy to enhance spatial consistency. Experiments demonstrate that the proposed algorithm has a strong reconstruction performance.
strength_weakness: strength_weakness: 1. In the ESA module, two modalities of information, raw events and text were used for cross-attention with frames. Still, no ablation experiments were given, which makes it impossible to determine whether the final experimental results of this work are more useful for raw events or text. In particular, raw events were inserted in each U-Net section.

2. In this paper, there are no reported flops as well as parameters, especially relative to some of the second-best methods in Table 1, what is the approximate percent increase in the two parameters?
questions: questions: 1. What are the tagging models mentioned in the paper? There are no specifics and no ablation experiments.

2. The comparison methods in the references are mostly from 2020 and 2021, with only one method from 2024.

3. Other methods are compared on SSIM metrics. Why not give SSIM metrics instead of SSIM* metrics?

4. How do you define spatial consistency? Adding ‘Noise Initialization’ only in the testing phase did not enhance the consistency of the model itself in the spatial dimension.
limitations: limitations: See Weaknesses. The final rating will be made based on the rebuttal.
paper_summary: paper_summary: This paper addresses the issue of artifacts and regional blur in existing event-to-video (E2V) reconstruction algorithms by leveraging the rich semantic information in language to enhance the semantic consistency of reconstructed videos. The authors propose a language-guided E2V generation model that employs existing text-conditional diffusion models as a framework. They use an Event-guided Spatiotemporal Attention (ESA) module for fine-grained spatial alignment and temporal continuity, an event-aware mask loss for further ensuring temporal consistency, and an event-aware noise initialization to address training-testing discrepancies. Extensive comparative experiments validate the algorithm's performance, and ablation studies demonstrate the effectiveness of each component.
strength_weakness: strength_weakness: 1 The paper lacks a detailed comparison of the algorithm's inference speed and model size. The authors acknowledge the inherent speed limitations of using diffusion models, emphasizing the importance of providing this comparison.

2 In practical applications lacking APS reference frames, obtaining accurate textual information that matches the scene description is difficult or nearly impossible. Using existing text generation models to extract semantic information introduces additional reference, compromising fairness to some extent.
questions: questions: Why is it necessary for the event activation regions of adjacent frames to be similar, rather than the regions without events?
limitations: limitations: The authors discuss the limitations of their work, particularly concerning the required training data and the inference speed of diffusion models, in accordance with official guidelines.
paper_summary: paper_summary: The paper explores the use of Denoising Diffusion Models (DDM) for event-based video reconstruction task. The most important contribution, in my opinion, is the improvement in video quality. The researchers adapted an existing model (ModelScope Text-to-Video) for the  event-based video reconstruction task. They introduced ESA, that modiffies a conditional inpainting technique (presented on High-Resolution Image Synthesis with Latent Diffusion Models). Additionally, they used text conditioning to further enhance video quality.

The innovative contributions of the paper are two. The fist one is the Event-aware Noise Initialization; This technique enables the use of frame I_(t-1) to reconstruct frame I_(t) during the inference stage (autoregressive).
The second one is the Event-aware Mask Loss; This new loss function is designed to improve the temporal consistensy.

In summary, the contributions are:

1. An existing text-to-video model (diffusion-based) was adapted for the event-based recosntruction task.
2. The model was adapted to event data using a conditional inpainting technique. Also proposed Event-aware Noise Initialization and Mask Loss to improve video quality.
3. The result is a new state-of-the-art in event-to-video reconstruction.

However, the proposed model has several drawbacks. The DDM "hallucinates" content, especially in areas with low or no event data, which can be problematic for applications like object detection in the context of self-driving cars.
strength_weakness: strength_weakness: Diffusion-based models tend to "hallucinate," producing some parts on a frames that are far from reality (in the event-based video reconstruction task), which can be problematic for applications like object detection in the context of safety (self-driving cars). 

Another problem is that the proposed model uses text prompts for video generation. Although these text prompts allow some control over the content, they also introduce a lot of ambiguity. This leads to a trial-and-error process (prompt engineering) until the most realistic reconstruction is achieved according to the user.

Additionally, the proposed model requires very high computational resources. Inference times are not specified in the paper, but it is known that the model does not run in real-time.
questions: questions: 1) Although traditional metrics (LPIPS, MSE, and SSIM) show an increase in video reconstruction quality, the qualitative results, especially in Figure 3, reveal that the reconstructed frames do not reflect reality in specific areas. These artifacts represent a smaller percentage of the image (in terms of pixels) and are not captured by traditional metrics, how does this affect the overall video quality?

2) In the inference stage, no mention is made of the length (number of frames) that the model can reconstruct. If there is a limit on the number of frames (how many frames and) , how is temporal consistency between reconstructed sections resolved?

3) Despite discussing the model's temporal consistency, no metrics verify this.

4) On line 134, the variable V is introduced, representing the conversion of event data to voxels. However, it is unclear whether V represents a segment with N temporal bins or N segments (the assumption is that V represents N segments). Clarification of this detail is necessary.

5) On line 139, the input latent representation is introduced through the variable \hat{\epsilon} . In DDM, \epsilon variable represents noise, and the noisy input latent representation normaly is denoted with z_t in this case as \hat{z}^{i}_{t}, since it is the latent image after applying the noise and the representation of the events z^{i}_{e}. I'm not sure if it's correct. 

6) The Event-guided Spatio-temporal Attention (ESA) module is very similar to the technique used in SD (High-Resolution Image Synthesis with Latent Diffusion Models) for inpainting. A reference to this would be beneficial.

7) In section 4.2, on line 162, the title "Event Spatial Attention" mentions a cross-attention technique. However, the title does not reflect this, causing confusion. It could be changed to "Event Spatial Cross-Attention". Similarly, on line 176, the title "Event Temporal Attention" maybe should be changed to "Event Temporal Cross-Attention."

7) In equation 8, line 199, the value of \(\lambda\) is not mentioned.

8) No mention is made of the computational cost in the inference stage, nor is the inference time mentioned. Could these data be mentioned?
limitations: limitations: Due to the DDM's tendency to hallucinate, the model cannot be used in self-driving cars or other computer vision applications where safety is involved.

It is believed that the model cannot be run in real time, much less on an embedded device.
paper_summary: paper_summary: This paper proposes a language-guided event-based video reconstruction method LaSe-E2V, which introduces the popular language model into event-based imaging tasks. The LaSe-E2V is generally based on a diffusion model. In order to further improve the method, this paper proposes a series of designs, such as event-guided spatio / temporal attention for the fusion between event and previous reconstructed frames, previous frame conditioning for ensuring consistency, event-aware mask loss, and event-aware noise initialization. Experimental results demonstrate the effectiveness of the proposed methods.
strength_weakness: strength_weakness: There are some unclear points, please see the Questions part.
questions: questions: 1. The proposed method requires a description of scenes contained in the event streams to be reconstructed. What if the description of the scene is unknown (For example, a ``blind'' event stream)? Can existing models handle this situation?
2. The running times and GPU memory costs for inference are absent. Diffusion models are still new for the event-based imaging community, and offering these values can help readers better understand the proposed method and do subsequent work.
3. It says in Line 228 of the paper that the definition of the SSIM metric has ambiguity. What's the ambiguity?
4. In the 3rd column of Fig. 4, there are more artifacts in the text part reconstructed by the proposed method than other methods. It's an interesting phenomenon, and it would be better if some explanations were given.
5. There is a typo ``evet'' in the Line 66 of the manuscript.
limitations: limitations: Please see the Questions part.

Comments: title: Author response to Reviewer H8fU
Author response to Reviewer ueNs (2/2)
Author response to Reviewer ueNs (1/2)
Author response to Reviewer H8fU (2/2)
Author response to Reviewer H8fU (1/2)
comment: Thank you to the authors for the clarifications and the additional experiments. After thorough discussion, all my doubts and concerns have been addressed. My rating for this work is 6.
Thank the reviewer for the insightful discussion! We will include these points in the camera-ready version.

> **Hallucinations**
> 
- We will manually identify small artifacts that traditional metrics fail to capture. These findings will be discussed in the Limitations section of the camera-ready version.

> **Ambiguity in Text Prompts**
> 
- Considering that our method reliably ensures fidelity in regions with sufficient event data, it is practical to assign a **confidence map** based on event density to identify high-confidence regions. For safety-critical applications, it is feasible to make decisions based on both the reconstructed video and the confidence map. This approach allows for simultaneous consideration of image quality and safety. We will include this limitation and the potential solution in the revision.

> **Computational Resources and Inference Times**
> 
- Sorry for the confusion. The unit in the Tab. A-5 is **megabytes** (MB). **Float32** is used for all the methods.
- Previous E2V methods are based on a recurrent architecture. Theoretically, recurrent models require more memory because they initially allocate enough memory to store the previous states, which potentially enhances the inference speed. However, when we clear the pre-allocated memory before each iteration, we observe that memory usage for HyperE2VID drops to 1372MB. There is also a minor decrease in inference speed due to the time required to reallocate memory. This represents a trade-off between memory usage and time cost, potentially influenced by CUDA tools. We will clarify this in the revision.
Hallucinations

While this paper demonstrates improved video reconstruction quality compared to previous works (such as HyperE2VID), it is important to address the side effect of "hallucinations" in the limitations section. These include small artifacts that traditional metrics like LPIPS, MSE, and SSIM cannot capture.

Ambiguity in Text Prompts

Similar to hallucinations, the potential limitations of introducing "ambiguity" with text prompts in the reconstruction pipeline should be discussed. These effects could lead to unsuitable video reconstructions for safety-critical applications, such as self-driving cars, due to the risk of generating nonexistent objects. This concern should also be highlighted in areas like computational photography, where fidelity is crucial.

Computational Resources and Inference Times

Table A-5 does not specify the units for memory requirements—are they in gigabytes or megabytes? Additionally, what precision is being used—float16, float32, or something else? Moreover, the significant differences in the number of parameters between models do not seem to align with their reported memory consumption. For instance, LaSe-E2V is reported to have 1,801 million parameters, while HyperE2VID has 10.15 million. Yet, LaSe-E2V’s memory consumption (12,139 unknown units) is listed as less than HyperE2VID’s (12,227 unknown units). Please address and correct these discrepancies.
> **Regarding the source of the text prompts for the events**
> 
- Our framework primarily focuses on providing a potential pipeline for reconstructing videos from events with a **language-guided perspective**. This is based on our finding that language naturally conveys abundant semantic information, which enhances the semantic consistency of the reconstructed video (see Lines 39-41). Such text guidance manner is **similar** to text-guided denoising [80,81] and super-resolution [82] methods, which also rely on natural language as a user-friendly interface to control the image restoration process. In this work, we do not focus much on the specific sources of the text prompts, as these can be flexible and generic, varying depending on the application scenarios. For example, if a DAVIS346 camera is used, obtaining text from APS frames is convenient. If a Prophesee camera is used and only event data is available, it is feasible to provide tagging-based text prompts derived from event-based multi-modality models. Additionally, in complex scenes lacking sufficient event data, it is practical to incorporate human intervention in an interactive manner to support human control. Overall, our method provides a language-guided interface for E2V reconstruction that could be **feasible** and **generic**.
- The reviewer might misunderstand the significance of the text prompts. The text is **not only** used to reconstruct regions without events. For regions with event data, our method incorporates text to provide complementary semantic information, ensuring semantic consistency and further improving performance.  For regions without event data, our method can rely on the text to reconstruct a reasonable scene close to the distribution of the real images. The results in Tab. 1 and Tab A-6 have demonstrated the superiority of our method in both scenarios.
- Regarding event-based multi-modality models, they offer an option for providing semantic information, which can be explored in future work. Since these models are trained with large-scale event-image paired data, they have the potential to provide prior semantic information that is absent in event data. Based on these text prompts, the method reconstructs video from event data, while effectively exploiting the semantic prior in the diffusion model to ensure semantic consistency.

Thanks for the insightful question! We will include this discussion in the camera-ready version.
Thanks for bringing up this thoughtful discussion.

> **About the hallucinations issue**
> 
- We respectfully disagree with the reviewer. Please note that all reconstructed results are **not unreliable**. In regions with sufficient event data, our method enhances performance with text guidance while ensuring fidelity. As demonstrated in Tab. 1 and Fig. 4, our method significantly **outperforms** previous methods. We also encourage the reviewer to recheck the demo video provided in our supplementary materials. For regions with insufficient event data, our method relies primarily on text prompts to reconstruct a scene that aligns with human preferences. As demonstrated in Fig. 1, HyperE2VID struggles with **severe artifacts** (haze-like artifacts), typically in the background regions, far from the true distribution of real images. Whereas, our method exhibits **higher-quality** reconstruction results that are closer to the ground truth. The results indicate that our method can subtly leverage the semantic information from language and thus ensure semantic consistency of the reconstructed video. To further demonstrate the effectiveness of our method, we provide a quantitative comparison of the HS-ERGB dataset, which includes larger regions lacking event data, as shown in Tab. A-6. Our method substantially **outperforms** baseline methods with an MSE of 0.083. Apparently, the quantitative results verify the superiority and reliability of our method.
- In this work, we mainly focus on a new research direction of exploring E2V reconstruction from a language-guided perspective, as affirmed by other reviewers: "This approach is expected to "*inspire future work*" (***ueNs***) and "*provide new ideas*" (***uWG8***). We did not focus much on the computation efficiency of the diffusion model, which will be left as a future work. Please be noted that using language guidance does not actually hamper the application value. Such a text-guided manner is also similar to recent methods used in text-guided denoising [80,81] and super-resolution [82], which have provided broad applications for low-level vision in a user-friendly manner. Our research is **application-significant** as the language provides abundant semantic information, beneficial for ensuring the **semantic consistency** of the reconstructed video. The text prompts also provide a way of human intervention to control the reconstruction process.

> **What is the significance of the text in this model?**
> 
- The text information serves as a **crucial** component to **activate** the semantic prior in the diffusion model. As clarified above, our method effectively enhances performance in regions **both** with sufficient and insufficient event data by utilizing text prompts. To simplify text prompts and enhance the model's robustness, we utilize only coarse, tagging-style descriptions for training. Without text prompts, it is challenging to exploit the semantic prior in the diffusion model to ensure semantic consistency. As shown in Tab. 3 (1st row), without the text prompt, our method yields modest results. However, when text information is incorporated, it significantly **improves** the model's ability to leverage the semantic prior in the diffusion model, thereby enhancing performance.

> **Text-related parts of the experiments**
> 
- For the qualitative comparison in Fig. 4, we respectfully suggest the reviewer recheck the supplementary **video** material. It shows not only an improvement in contrast. The previous methods tend to reconstruct misty images with distinct artifacts, especially in the region with insufficient event data. Our method effectively exploits the semantic prior of the diffusion model with the guidance of text prompts, which demonstrates semantic consistency. Additionally, Fig. 1 shows another extreme case. While the previous method (i.e., HyperE2VID) fails in the regions without event data, our method still reconstructs a reasonable scene according to the text guidance, which is closer to the ground truth. Tab. A-6 also shows the quantitative comparison in this dataset (i.e., HS-ERGB) and demonstrates the superiority of our method.
- Regarding the artifacts in the letter “g”, as also noted by reviewer ***uWG8***, these issues arise because our method relies on the prior from the pre-trained diffusion model (SD2 [42]), which also struggles with text rendering. However, the recently released SD3 [79] claims to have improved text rendering capabilities, which could potentially address this problem and improve the performance.
> **Artifacts Not Captured by Traditional Metrics in Figure 3**
> 
- Tab. A-6 provides a quantitative comparison based on three sequences (*horse_11*, *horse_12*, *horse_13*) from the HS-ERGB dataset. Existing E2V methods typically fail to reconstruct regions without events, leading to significantly **worse** quantitative results. Although our method may **not perfectly** reconstruct every detail for reality, it does generate a **reasonable** output that aligns with human preference and is generally **close to the distribution** of the real scene. Therefore, while our results on the HS-ERGB dataset may be less significant than those on "constantly moving" datasets (ECD, MVSEC, HQF), our method is still **substantially better** than baseline methods.
- Regarding the artifacts in the letters, as also noted by reviewer ***uWG8***, these issues arise because our method relies on the pre-trained diffusion model (SD2 [42]), which also struggles with text rendering. However, the recently released SD3 [79] claims to have improved text rendering capabilities, which could potentially address this problem and improve the performance. We believe that the MSE can capture these artifacts at **pixel level**. However, although minor artifact in small regions exists, our method excels in **overall** image quality, since previous methods tend to reconstruct misty areas in front of the poster in Fig. 4 (3rd column)

Table A-6: Quantitative comparison of HS-ERGB. Results are conducted on 3 sequences with a total 497 frames.

|  Methods |  | HS-ERGB |  |
| --- | --- | --- | --- |
|  | MSE | SSIM | LPIPS |
| E2VID | 0.199 | 0.382 | 0.736 |
| HyperE2VID | 0.161 | 0.374 | 0.745 |
| **LaSe-E2V (Ours)** | **0.078** | **0.429** | **0.665** |

> **Reference to ESA**
> 
- Our ESA module is specifically designed to enhance spatio-temporal consistency between events and videos. In contrast, the original SD [42] serves as a baseline attention mechanism for integrating conditional input. Our approach differs in **attention design**. While the original SD relies solely on **simple cross-attention** to incorporate various feature conditions, our ESA module takes into account the unique spatial and temporal characteristics of event data. It introduces two **distinct attention** mechanisms respective to the **spatial** domain and **temporal** domain, which fully leverage the constraints provided by the event data and ensure spatio-temporal consistency.

**Additional Reference:**

[80] Duan H, Min X, Wu S, et al. UniProcessor: A Text-induced Unified Low-level Image Processor[C]//ECCV. 2024.

[81] Qi C, Tu Z, Ye K, et al. Tip: Text-driven image processing with semantic and restoration instructions[C]//ECCV. 2024.

[82] Gandikota K V, Chandramouli P. Text-guided Explorable Image Super-resolution[C]//CVPR 2024.
Thank the reviewer for elaborating on the points. We address the reviewer's concerns below:

> **Hallucinations**
> 
- Hallucination is a potential **side effect** introduced by incorporating semantic priors from the diffusion model into our framework. In this regard, our proposed techniques are demonstrated to be effective in reducing hallucinations, but **not entirely** eliminating them, which is still a challenge in the diffusion model research community. Extensive experimental results (Tab. 1 and Fig. 4) show that our method significantly **improves fidelity** and reduces hallucinations in the scenes with sufficient event data.  Our method also exhibits better reconstruction performance for the scenes even with insufficient event data (Tab. A-6). Please be reminded that it is impossible to totally remove the hallucination effect for regions with insufficient events. However, our method indeed effectively leverages the semantic prior in the diffusion model to reconstruct the image more **closer to the distribution** to the real scene, whereas the previous E2V methods (e.g., HyperE2VID) **fail** in these regions, as shown Fig. 1.
- Our framework mainly focuses on providing a potential pipeline to reconstruct the video from events with the guidance of language. This is based on our finding that language naturally conveys abundant semantic information, which is beneficial in enhancing the **semantic consistency** for the reconstructed video (see Lines 39-41). The text guidance can serve as a form of human intervention through text prompts, which is **similar** to the way of the text-guided denoising [80,81] and super-resolution[82] methods.
- As shown in Fig. 1, it compares the difference between our method and the previous E2V method (i.e., HyperE2VID). For region with insufficient events, although our results still differs from reality in some details, they indeed reconstructs a scene according to human preference. In contrast, the previous E2V method always reconstructs **haze** in these regions. For Fig. 7, we will update the results from previous E2V methods for clearer comparison. Quantitatively, Tab. A-6 also shows the superiority of our method in the scene with insufficient events.

> **Ambiguity in Text Prompts**
> 
- Our framework incorporates text descriptions as complementary information, marking the first instance of allowing language guidance in the E2V pipeline. This approach is believed to "*inspire future work*" (***ueNs***) and "*provide new ideas*" (***uWG8***). While this may introduce some ambiguity, it also offers the flexibility to manually adapt the reconstructed video according to user preferences, as demonstrated in Fig. 8. The text guidance manner is also similar to the way of the text-guided denoising [80,81] and super-resolution[82] methods.
- In regions with sufficient event data, our method improves reconstruction performance by leveraging the semantic priors from the text prompts, ensuring high fidelity. For regions with insufficient event data, the method relies solely on the text prompts to reconstruct a scene that aligns with human preferences. Although this approach may produce textures or details that differ from the real ones, it still generates images that are closer to reality. In contrast, the previous E2V models always reconstruct these regions as indistinct haze, far from the true distribution of real images. Fig. 1 shows the qualitative comparison. Tab. A-6 also shows the quantitative comparison and demonstrates the superiority of our method. We will add more comparisons on these scenes with insufficient events in the revision.

> **Computational Resources and Inference Times**
> 
- Table A-5 provides the GPU memory for different E2V models. It shows similar memory costs among different methods.

Table A-5: GPU memory cost of different E2V methods. All tests are conducted on one NVIDIA Tesla 32G-V100 GPU.

| Methods | GPU memory |
| --- | --- |
| E2VID [40] | 12120 |
| ET-Net [54] | 12218 |
| HyperE2VID [12] | 12227 |
| LaSe-E2V (Ours) | 12139 |
Thank you for your response. My concern still lies with the acquisition of the text. All reviewers mentioned that the issue of hallucinations makes reconstruction unreliable. Event cameras are not good at capturing image details but are instead suitable for high-speed, real-time applications, unreliable reconstruction contradicts the motivation for using event cameras. Additionally, the reconstruction method in this paper is nearly 100 times slower than previous event reconstruction algorithms. If we consider the acquisition of text prompts, the application scenarios become even more limited.

If we do not consider the application and focus solely on the method itself, the paper does not explore the impact of different text prompts on the results. The authors mention that "our method requires only coarse text descriptions" and that "this coarse-grained text is effective and does not necessitate a trial-and-error process." However, if the text information is insignificant, then what is the significance of the text in this model? Is it the case that relying solely on a pre-trained diffusion model would already yield good results? This also contradicts the motivation of the paper.

After reading the concerns raised by other reviewers, I found the text-related parts of the experiments confusing. The qualitative results do not indicate the text used, making it difficult to assess whether the improvements in the results are related to the text. For instance, in Figure 4, the first three scenes show only an improvement in contrast, making it hard to see any other differences. In the second-to-last column, the letter "g" is clearly reconstructed incorrectly. What caused this? It's impossible to determine whether this is related to the text prompt.

Regarding the source of the text prompts for the events, the authors mentioned in their response to me that "These can be easily provided by humans in the absence of APS frames." However, in their response to another reviewer, they stated that "event-based multi-modality models such as EventBind[77] and EventClip[78] can also be employed to generate text descriptions directly from the event stream." For the former, I cannot imagine whether a person would be watching the events to provide a text description or watching the scene to do so. It seems that if a person can directly observe the scene, then reconstruction wouldn't be necessary. If a person is observing the events, the sparsity of the events makes it difficult to provide an accurate description. For the latter, if event-based multi-modality models are used, these models would require the presence of events to function. However, the authors emphasize that the text is meant to enhance areas without events. Can these models provide accurate descriptions for sparse event regions in the absence of events?

As this paper focused on event reconstruction with text as additional guidance, I believe it is essential to thoroughly explore the motivation and role of the text.
Hallucinations:

I understand that the ESA module and mask loss help mitigate the problem of "hallucinations." However, this information does not present anything new regarding hallucinations. Additionally, as shown in Figure 7, a prompt had to be manually adjusted to generate a similar background, but it still differs from reality.


Ambiguity in Text Prompts:

As shown in Figure 7, when no events are triggered in large regions, it becomes necessary to manually generate a text prompt that, according to the user, matches reality. This introduces ambiguity into the process. Furthermore, even when a close image can be generated with the prompt, the textures and forms often differ significantly from the real ones.

Computational Resources and Inference Times:

Table A-1 clarifies this question; however, it would be helpful to include the memory (VRAM) required to run each model.


Artifacts Not Captured by Traditional Metrics in Figure 3:

Figure 3 shows the qualitative results from the HS-ERGB dataset, but no quantitative results are provided for this dataset. The ECD, MVSEC, and HQF datasets might not capture the artifacts (hallucinations) produced by the proposed LaSe-E2V model because the camera is constantly moving in those scenarios. However, in situations like the HS-ERGB dataset, where both the background and the camera are stationary, the proposed model may yield worse quantitative results for the HS-ERGB dataset.

Additionally, if we look at Figure 4, in the third column, there is a poster with letters that are completely distorted by the proposed LaSe-E2V model, making it impossible to read the content. However, since these are small regions relative to the size of the image, traditional metrics (MSE, SSIM, and LPIPS) do not adequately reflect these artifacts.


Temporal Consistency Between Reconstructed Sections:

Information about the autoregressive pipeline helps clarify doubts related to video generation during the inference stage.


Metrics for Temporal Consistency:

Doubts related to temporal consistency have been clarified.


Reference to ESA:

References regarding the ESA module and the main similarities and differences with respect to SD (in inpainting) were not added.


Remaining Questions:

The rest of the questions were answered satisfactorily.
Thanks for your rebuttal. I'll keep my rating.
Hi reviewers,

Thank you for your hard work in reviewing the paper! Please check out the authors' responses and ask any questions you have to help clarify things by Aug 13.

--AC