Title: UniE2F: A Unified Framework for Event-to-Frame Reconstruction with Diffusion Model
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: Event-based Video Frame Reconstruction

Abstract: Event cameras, also known as dynamic vision sensors, record high-frequency pixel intensity changes as discrete ``events''. They offer remarkable advantages, including high dynamic range, high temporal resolution, low power consumption, and absence of motion blur, all while utilizing minimal bandwidth. In this paper, we explore leveraging the generative prior of a pre-trained video diffusion model to reconstruct high-quality video frames from sparse event data. Specifically, we first establish a baseline model by directly applying event data as a condition to synthesize videos. Then, based on the physical correlation between the event stream and video frames, we further introduce the event-based inter-frame residual guidance to enhance the accuracy of video frame reconstruction. Furthermore, we extend our method to video frame interpolation and prediction in a zero-shot manner by modulating the reverse diffusion sampling process, thereby creating a unified event-to-frame reconstruction framework. Experimental results on real-world and synthetic datasets demonstrate that our method significantly outperforms previous approaches both quantitatively and qualitatively. The code will be publicly available.