E-Motion: Future Motion Simulation via Event Sequence Diffusion
Authors: Song Wu
Zhiyu Zhu
Junhui Hou
Guangming Shi
Jinjian Wu
Venue: NeurIPS 2024 poster (2024)
Keywords: Event-based vision
video diffusion model
Primary area: machine_vision
TL;DR: We propose to integrate the strong learning capacity of the video diffusion model with the rich motion information of an event camera as a motion prediction framework.

Abstract: Forecasting a typical object's future motion is a critical task for interpreting and interacting with dynamic environments in computer vision. Event-based sensors, which could capture changes in the scene with exceptional temporal granularity, may potentially offer a unique opportunity to predict future motion with a level of detail and precision previously unachievable. Inspired by that, we propose to integrate the strong learning capacity of the video diffusion model with the rich motion information of an event camera as a motion simulation framework. Specifically, we initially employ pre-trained stable video diffusion models to adapt the event sequence dataset. This process facilitates the transfer of extensive knowledge from RGB videos to an event-centric domain. Moreover, we introduce an alignment mechanism that utilizes reinforcement learning techniques to enhance the reverse generation trajectory of the diffusion model, ensuring improved performance and accuracy. Through extensive testing and validation, we demonstrate the effectiveness of our method in various complex scenarios, showcasing its potential to revolutionize motion flow prediction in computer vision applications such as autonomous vehicle guidance, robotic navigation, and interactive media. Our findings suggest a promising direction for future research in enhancing the interpretative power and predictive accuracy of computer vision systems. The source code is
publicly available at https://github.com/p4r4mount/E-Motion.

E-Motion: Future Motion Simulation via Event Sequence Diffusion
Song Wu Zhiyu Zhu 2, Junhui Hou 2, Guangming Shi
Jiijian Wu
1 Xidian University, City University of Hong Kong ewu_6660stu, xidian edu. cn, zhiyuzhu2-comy cityu. edu eduu.b. hk jh. hwu@hioyu. edu. hk, www.siiiiiiiioooooommooooooomiie @xidi an. edu. en, jinjian.. wu@mail xidian an edu. cn
Abstract
Forecasting a typical object's future motion is critical task for interpreting and interacting with dynamic environments in computer vision. Event-based sensors, which could capture changes in the scene with exceptional temporal granularity, may potentially offer unique opportunity to predict future motion with a level detail and precision previously unachievable. Inspired by that, propose to inte- grate the strong learning capacity of the video diffusion model with the rich motion information of event camera motion simulation framework. Specifically, we initially employ pre-trained stable video diffusion models to adapt the event sequence dataset. This process facilitates the transfer of extensive knowledge from RGB videos to an event-centric domain. Moreover, we introduce an alignment mechanism that utilizes reinforcement learning techniques enhance the reverse generation trajectory of the diffusion model, ensuring improved performance and accuracy. Through extensive testing and validation, demonstrate the effec- tiveness of our method in various complex scenarios, showcasing its potential to revolutionize motion flow prediction in computer vision applications such as tonomous vehicle guidance, rebotic navigation, and interactive media. Our findings suggest promising direction for future research in enhancing the interpretative power and predictive accuracy of computer vision systems. The source code is publicly available at httpt: //github com/ Be-ooontt//--iioorooo
Introduction
Accurately capturing and interpreting dynamic scenes under fluctuating motion and illumination conditions remains an enduring challenge in computer vision [27, 33]. This challenge is particularly pronounced in real-world settings, where subtle variations can dramatically affect the perception and analysis of future motion [60, 11]. Traditional imaging modalities often struggle to capture these nuances, leading to a gap in accurately modeling and predicting motion flow in complex visual environments.
The rapid advancements in deep learning have catalyzed transformative developments in computer vision, particularly in the generative models [45, 16, 18. 25, 46, 48, 49]. Video diffusion models [1, 19, 32, 10, 2], which stand at the forefront of these innovations, leverage stochastic diffusion processes to generate, restore, and accurately manipulate video content. These models, emblematic of the state- of-the-art in temporal data processing, offer refined capabilities for complex video-based tasks [5, 39], underscoring the significant strides made in understanding and interpreting dynamic visual scenes.
"This work was supported in part by National Key Research and Development Program of (hina(203YYFA1008500) in part by NSFC Excellent Young Scientists Fund 62422118, and in part by Hong Kong Innovation and Technology Fund ITSS/14//3, The first two authors contributed to this paper equally. Corresponding author: Jinjian Wu.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

State
Event sampping
RG3 sampiing
Event prddit motio
RGB predict metion
Gouund notio
Predict time
Figure 1: Illustration that the exceptional temporal resolution afforded by event cameras, alongside their distinctive event-driven sensing paradigm, presents a significant opportunity for advancing the precision in predicting future motion trajectories.
Coming into the high temporal resolution field, event data stands as a revolutionary sensing ap- proach [37, 12, 53] that significantly mitigates this gap and consistently captures even very subtle fluctuations. Such strong capacity comes from the unique sensing pattern the event camera, which asynchronously measures the intensity variation in high temporal resolution. allows for the precise detection of miniature illumination changes, providing a rich, granular record that traditional cameras simply cannot offer [67, 68. 7].
To enable the video diffusion model completely and correctly learn concise motion information for the estimation of potential future movement, integrating it with event data to realize an event sequence generation model is potential solution that may inject high-frequency motion information into the video diffusion model. This paper delves into the symbiosis of video diffusion models and high-temporal--esoouu
event data, exploring its potential to redefine motion forecasting in computer vision. We commence by delineating the landscape of video diffusion models and the mechanics of event-based sensing, elucidating their complementary strengths. Subsequently, we introduce novel framework that amalgamates these technologies. aiming to augment the precision of motion flow predictions. Through comprehensive experimentation, we validate the effectiveness of our methodology, demonstrating its superior performance across diverse scenarios. Our findings illuminate the path forward, showcasing the integration of video diffusion models with event data as robust, innovative solution for capturing and interpreting the complexities of dynamic scenes with unparalleled detail and accuracy.
In summary, the contribution of this paper lies in the following three parts:
we make the first attempt to integrate event-sequences with a video diffusion model, resulting in event-sequence diffusion model, which could potentially estimate future object motion, given by certain event prompt;
we propose to align the pre-trained event-sequence diffusion model with the real-world motion via reinforcement learning process, which stabilizes the results generated by the diffusion model and makes them more closely resemble real motion.
integrate test-time prompt augmentation method to make use of high temporal resolution event sequence prompt to enhance the generation performance.
2
Related Work
2.1 Event-based Vision
Event-based vision represents paradigm shift from traditional frame-based imaging, offering dynamic and highly granular approach to capturing visual information. Unlike conventional cameras that record static frames at fixed intervals, the event-based sensor (developed by Lichtsteiner et al. [28] and further elaborated by Posch et al. [36]) asynchronously records intensity changes per pixel and generates signal termed an "event" whenever the intensity surpasses a threshold, thereby providing continuous stream of data that reflects temporal changes with remarkable precision. This

---

method is particularly effective in environments with rapid motion or varying illumination, where traditional cameras suffer from motion blur and latency issues.
Recent advancements in this field have focused on leveraging the high temporal resolution of event data for various applications, including high-speed tracking [57, 30, 65], dynamic scene reconstruction [43, 23, 69], and optical flow estimation [44, 9, 22]. Works by Gallego et al. [31] and Rebecq et al. [38] have been instrumental in demonstrating the utility of event-based data in reconstructing high-speed phenomena and enhancing motion analysis, setting solid foundation for our research.
2.2 Multimodal Diffusion Models
Generative diffusion models, introduced by Sohl-Dickstein et al. [45], represent a class of probabilistic generative models that simulate the gradual transformation of data from a complex distribution into simpler, typically Gaussian distribution, and vice versa. [8, 25, 61, 4]. This process, characterized by series of forward and reverse diffusion steps, has been applied successfully to range of tasks, including image synthesis [20, 34J, restoration [29, 21], and, more recently, temporal data manipulation [32, 15].
The application of diffusion models to video data, as explored by Ho et al. [19] and extended by others, marks significant advancement in the field, offering new pathways for the generation and manipulation of dynamic scenes. These models have shown exceptiooal promise in capturing the temporal continuity and complexity inherent in video data, providing robust framework for tasks such as video prediction and temporal interpolation.
In recent years, the development of multimodal diffusion technology has advanced rapidly. Re- searchers are dedicated to applying the powerful generative capabilities of diffusion to different modalities with unique advantages, such as optical flow and depth. Saxena et al. [40] was the first to apply diffusion models to optical flow and depth estimation. For the characteristics of training data, they introduced infilling, step-rolling, and LI loss during training to mitigate distribution shifts between training and inference. To address the lack of ground truth in datasets, they also used a large amount of synthetic data for self-supervised pretraining, enabling the diffusion model acquire reliable knowledge. Chen et al. [6] utilized the motion information embedded in control signals such edges and depth maps to achieve more precise control over the text-to-vides video (T2V) process. They used pixel residuals and optical flow to extract motion-price -piior information to ensure continuity in video generation. Additionally, they proposed a first-frame generator to integrate semantic information from text and images.
Despite the extensive body of research within the domain of temporal analysis, it is important to acknowledge that the majority of these studies focus primarily on the domain of RGB images and videos. As previously discussed, the superior temporal resolution offered by event data holds significant potential for enhancing the alignment process. Consequently, it is imperative to undertake thorough investigation into the application and adaptation of existing pre-trained diffusion models to the realm of event data.
3 Preliminary
Diffusion Models are class of generative models that simulate the gradual transformation of data from a complex, high-dimensional distribution to simpler, typically Gaussian distribution through a process known as forward diffusion [48, 18, 46, 47]. Conversely, the reverse diffusion process aims to reconstruct the original data distribution from the simpler one. This mechanism is inspired by thermodynamic processes and has been increasingly applied in the field of deep learning for generating high-qualing diverse samples from complex distributions.
The mathematical foundation of diffusion models is rooted in stochastic differential equations (SDEs), which describe the forward and reverse diffusion processes. By score-based formulation [48], the forward process of diffusion model acts
dx = f(x t)dt + g())dw
where x T indicates the state of reconstructed signal, w IR" represents a standard Wiener process (g(t)dw ~ (N(0, g(t) Idw)), f(:,t) _ + R and g(-) IR - IR represent the drift and
3

---

X T Steps
Steps
X
X
U-Net.
U-Net
Gaussian Noise
Replace
Denoiinng 1] Not wi schhottlerr
Denaiiin U-Net schedaler
Rosults
X
Dow Downstream Application
Diffusion Add tstep noise
Event Prompt
Noised Prompt
Segmentation
Event Stream
High Temporal Resolution Event Prompt Denoising
Tracking
Flow Estimation
Figure 2: Inference workflow of the proposed method, where the left upper one indicates the random Gaussian noise, left lower represents the prompted event sequence. We perform steps forward diffusion processing on the event prompt and substitute a portion the diffusion input noise, followed by Steps of conventional denoising.
diffusion coefficients, respectively. Moreover, in the evaluating phase, the reverse (inference) process could be illustrated as iteratively performing the following ODE step:
dx = [(x, t) [
1
(2)
where (x)ogp(x) is usually approximated by a learnable score model Sa (a(x, t). Based on the theoreti- cal formulation of stable video diffusion [1], e.g., variance exploding (VE) diffusion process [48], the reverse process is acted as
N, (X (a: - 0t-1);
dt
where (a(-) is one of parametrization method of So(-), which estimate the clean image from noise latent X, ~ ~Xxo, ofI).
Methods
The endeavor of video diffusion presents multifaceted challenges, notably the simultaneous generation of sequences that not only manifest dynamic, convincing motion but also maintain authentic textures. The utilization of event data, characterized by its intrinsic high temporal resolution, emerges as a strategic solution for the precise modeling of object motion. In contrast to traditional stable video diffusion models, which are typically initialized using singular image textual prompt, our proposed methodology, designated as the Event-Sequence Diffusion Network, capitalizes on succinct sequence of events its conditioning input. This novel approach is illustrated in Fig. 2.
As elaborated Sec. 4.1, we delineate the comprehensive pre-training regimen that facilitates the learning of object motion through the prediction of subsequent events. Nevertheless, due to the intrinsic diversity-generatiin
property of diffusion models, they commonly yield several distinct samples from a single input prompt. Although these variations may each seem feasible, there is no assurance that they align consistently with the actual dynamics real-world motion. Thus, in Sec. 4.2, augment the quality of generation by incorporating alignment process. It employs reinforcement learning techniques to impose a structured regularization on the generative process of the diffusion models, thereby enhancing the fidelity and coherence of the produced sequences.
4.1 Learning Motion Prior via Pretraining on Event Sequences
An initial formatting process is discussed to retain its high-temporal resolution properties effectively and to facilitate integration with pre-existing large-scale models. Furthermore, to mitigate the substantial computational demands associated with video diffusion models, we have implemented prompt sampler designed to enhance the efficiency of information encapsulation derived from event

---

Algorithm Motion Alignment Process
1: Input: Reference model 0', training model 0 and reward model R(x).
2: For = ..., I do
3: Sample reference trajectory { R(xo) and Par
4: For f = .... do
5: Calculate Po based reference trajectory {=o Too }.
6: Optimizing (x)'
7: End for
8: 0' <- 0.
9: End for
10:
Return 0
sequence diffusion frameworks. Subsequently, we will delineate these critical aspects in detailed manner.
Event Representation. To leverage contemporary video diffusion models for event data generation, we adopt a strategy where both event information and corresponding images are concurrently inputted into the video diffusion framework, which then processes adaptively sampled outcomes. Specifically, as illustrated in Fig. 2, for a given event denoted as = {h, w. p. f). we consolidate the event data into voxel grid representation [66], symbolized as E R Bx HXW where B denotes the number of time bins. In order to utilize the rich pre- pre- training information from RGB frames, we set B 3. This approach ensures that the video diffusion model effectively interpret and integrate the high-dimennsonal event data alongside conventional image inputs, facilitating more comprehensive synthesis of dynamic visual content.
Pretraining. The training regimen for our proposed approach is similar with that of diffusion models [48, 24]. This methodology involves estimating the clean image from perturbed samples, as
Spre Ex (x() Ex,12 I||=0
(o 11))
of oftaauun During training, we randomly where X(t) denotes a weighting function, parameterized as taatuu
select set of intermediate states X₁ and apply regularization techniques to guide them towards the accurate estimation of the underlying noise component e. Moreover, to make the diffusion network adaptively capture the object motion with different time windows, in the training process we randomly augment the diffusion network with the voxel from different time ranges. (Please refer to the Appendix Sec. A for more details)
High-temporal Resolution Guided Sampling. Owing to the unique operational mechanism of event cameras, which detect changes in intensity rather than capturing static images, the clarity and definition of recorded objects can significantly diminish if the chosen temporal window is either brief or excessively prolonged, especially when we only feed one prompt, like a standard stable video diffusion network. This characteristic often results in blurred indistinct imagery when the temporal resolution does not align optimally with the scene's 's dynamics.
Inspired by the recent advancement of guided sampling [35], we propose aggregate multiple high- temporal resolution event frames as a test-time prompt. Specifically, as illustrated in Fig. 2, during the inference process,
feed > frames instead of single frame as prompt X."TM xxx For the step T, both X.P and X are initialized with the random Gaussian noise. However, for the step t < т, set X f- X o.C. Subsequently, the noised event prompt X replaces the first 8 random noises in the noise tensor X providing motion priors for the denoising process. (Please refer to the Appendix Algorithm for more details)
x
4.2 Motion Aligmment via Reinforcement Learning
Given the inherent challenges in precisely tailoring diffusion models to fit the entirety of the training data, particularly when considering the disparity between the model's size and the volume of the dataset, is pragmatic to direct potential losses towards regions less perceptible to end-users or higher- level algorithms. Furthermore, the multi-step nature of the diffusion generation process renders the simultancous training of the entire pipeline nearly unfeasible. To address this, we employ a strategy
5

---

of reinforced optimization, conceptualizing the generation process Markov chain. This approach underscores the importance of guiding the diffusion process yield results of superior quality, thereby optimizing the model's performance while accommodating its structural and computational complexities. Optimizing such diffusion model starts with the following equations:
- max J = JP,(x)) R(x)dx,
(5)
where (x(x) indicates the distribution of reconstructed samples under the model weight 0. Although
the diffusion model is a probabilistic model, its weights are generally deterministic. [46] The
randomness generally comes from Gaussian sampling. Thus, we adopt the same measurement as [64]
2(x..0)²
utilize the Gaussian density function N (xxll4, o) ov2mm
measure the (x(x) of
given sample X,, where ju()) indicates the estimated clean latent without adding random noise. We then adopt the policy gradient descent method, i.e., PPO [41, 42]. to optimize the alignment process of these diffusion models. The policy gradient descent optimization process is formulated follows:
VoJ'(x) - E VoPe R(xo) + AVgKC((Prr (Po). Par
(6)
where (CI(--)) indicates the KL-divergence between two distributions. The training process is shown Algorithm 1. Note that due the huge GPU memory and time consumption of the video diffusion process, we distribute the data generation and network training on different GPUs. Please refer to Sec.5 Settings for more details.
Modeling of Reward. To measure the quality of the reconstructed event frame, we utilize the FVD and SSIM as the reward R(-) guide the training process of reinforcement alignment. Moreover, to remove the bias of rewards, we randomly generate M samples given one prompt, forming a pair of samples.
5 Experiments
Figure 3: Qualitative comparison between SOTA methods. The first row each sequence represents the ground truth of the event sequence. The second and third rows respectively depict the results of future event estimation by Sim VP [14] and TAU [50]. The final row represents the results obtained by our method. The complete sequence is shown in Fig.9.
Dataset. In our study, we utilize two large-scale event datasets, i.c., VisEvent [55] and EventVOT dataset [56]. The VisEvent dataset encompasses wide variety of scenes, including 820 RGB-Event video pairs and 371, 128 RGB frames. Moreover, was captured by the DAVIS346 camera [3], with resolutions of 346 x 260 for RGB and events. It addresses diverse environmental conditions and includes 17 distinct attributes such as camera motion, low illumination, and background clutter, facilitating detailed performance analysis under various challenging scenarios.
The EventVOT dataset provides 1,141 high-definition videos with 569,359 frames, making the largest dataset in this domain. It features a resolution of 1280 x 720, encompassing 19 diverse classes of target objects. Compared to earlier datasets such as VOT-DVS, TD-DVS, and Ulster from 2016, and more recent ones like FE108 and COESOT, EventVOT offers unprecedented scale and variety,

---

Ratraa
baiaa
Srpar
tape:
Rotating Umbrella.
(b) Falling Balloon.
ECGO GT
P022
P916
(92)
1pp
0122
Ind
(c) Poor exposure: The car driving out of the tunnel.
(d) Occlusion: person walking to shadow.
Figure 4: More visualization of our method's prediction in various scenarios. The results of the complete sequence along with other methods are presented Fig. 12 and Fig. 13.
Table 1: Quantitative comparison between SOTA methods, where SVD denotes the standard stable video diffusion network. "VID" represents the video and "EVT" indicates the event data. 1 (resp-l) represents the bigger (resp. lower) the better.
Methods
Modal
FVD
MSE
SSIM
LPIPS |
mloU
aloU f
PhydNet [17]
VID
1602.84
0.0295
0.4299
0.6048
0.077
0.348
Sim VP [13]
VID
1347.57
0.0216
0.6261
0.3051
0.264
0.520
TAU [50]
VID
1371.65
0.0240
0.6381
0.3026
0.264
0.529
PredRNNv2 [59]
VID
1266.83
0.0176
0.5550
0.2407
0.275
0.540
SVD [1]
VID
1122.54
0.0246
0.6451
0.3299
0.233
0.506
PredRNNv2 [59]
EVT
1339.05
0.0306
0.6598
0.3388
0.166
0.504
Vi [13]
EVT
1242.25
0.0210
0.7961
0.3371
0.213
0.532
TAU [50]
EVT
1218.03
0.0231
0.7972
0,3354
0.228
0.514
Ours
EVT
1055.25
0.0170
0.7998
0.3123
0.302
0.522
including substantial number videos and a wide array of environmental conditions, aimed at improving the development and evaluation of event-based visual tracking algorithms. The dataset is meticulously annotated and divided into training (841 videos), validation (18 videos), and testing (282 videos) subsets, ensuring comprehensive framework for robust algorithm testing and benchmarking.
Settings. All experiments are conducted on machines with GeForce RTX 3090 GPUs, Intel(R) Core(TM) 10-0 CPU of 2.90GHz, and 64-GB RAM. In the pre-training stage, we employed the ADAM optimizer with the exponential decay rates 31 0.9 and 82 0.999. The total training process 20000 iterations for both kinds of noise experiments. We initialized the learning rate as le-5. We set the batch size to 128 (with 8 gradient accumulation steps).
For the alignment process, due to the fact that it takes quite lot of GPU memory to generate reference trajectory during the training process, we have achieve the trajectory generation and reinforcement alignment in a parallel and distributed manner. Specifically, we utilize 4x RTX3090s to train reinforcement learning alignment processes. Moreover, 4x RTX3090s is utilized to generate training trajectory data. The updating episode of the reinforcement learning process is set at 100 optimization steps. (Please refer Appendix for the detailed illustration of) We also employed the ADAM optimizer with the exponential decay rates 31 0.9 and B2 0.999. We initialized

---

KGB
Segneciatoo
Opnicall
(a) Segmentation and flow estimation results using GT event and RGB frame.
Defras
Datras
Tooddg
Orpcca Vice
Opdccll
(b) Downstream results using our estimated events.
(c) Downstream results using SVD estimated frames.
Figure Visualization results on downstream tasks, where we show tasks of tracking, segmenta- tion, and flow estimation. (a) denotes the ceiling performance of settings (b) and (c).
the learning rate as 2e and set the batch size to 16(with gradient accumulation steps) in all experiments.
Comparison Methods. comprehensively evaluate the performance of the proposed method, we compared different methods, which could be briefly divided into two categories: image-based future frame prediction and event-based future motion prediction. For the image-based future frame prediction, we adopt the original stable video diffusion [1] with some popular time serious forecasting methods, e.g., PredRNN [58]. SP [14], TAU [51]. Moreover, for future event forecasting, we simply stable video diffusion [1] as baseline since other methods make it hard to achieve desirable performance on those datasets.
Metrics. In the evaluation of our proposed model, we employ comprehensive suite of metrics designed to assess the quality. We incorporate the Fréchet Video Distance (FVD) [52] to evaluate the temporal coherence and visual quality of generated video sequences against real video content. The Mean Squared Error (MSE) serves as a fundamental metric, quantifying the average squared difference between the estimated values and the actual values. The Structural Similarity Index Measure (SSIM) employed to assess the visual impact of structural information, brightness, and contrast differences between the generated images and the ground truth. Lastly, the Learned Perceptual Image Patch Similarity (LPIPS) [63] metric is adopted to evaluate the perceptual similarity between the generated and real images. In addition, apply mloU and aloU metrics for the object segmentation task to validate the feasibility of the generated events downstream tasks.
5.1 Quantitative and Qualitative Comparison Results
The experimental outcomes are illustrated in Table 1. It is evident that by applying regularization to event data, our methodology attains performance levels comparable state-of-the-ar (SOTA) methods. Notably, in metrics specific to the event domain, our approach outperforms the comparative

---

Table 2: Quantitative comparison between SOTA methods object tracking. SVD denotes the standard stable video diffusion network. "VID" represents the video and "EVT" indicates the event data. (resp-4) represents the bigger (resp. lower) the better. Rupper denotes comparison of task performance with GT Event, where we average the ratio of different metrics (please refer to the Appendix for the detailed comparisons).
Methods
Modal
AUC
OP50 1
OP75 f
PR 1
Rupper 1
PredRNN [59]
EVT
28.54
22.24
17.24
14.96
0.496
Sim VP [13]
EVT
35.10
32.41
18.87
21.74
0.640
TAU [50]
EVT
34.95
31.85
20.31
24.74
0.669
Ours
EVT
43.77
47.77
27.31
30.99
0,891
GT Event
EVT
47.87
53.30
31.93
34.62
1.000
Table 3: Ablation Study of Pre-training Phase. All models are tested with only feeding single event voxel frame, where "#Prompt" indicates the number of training prompt event data, (111 (1,3) 3) indicates to randomly select to 3 frames as training prompt. 'Fine-tuning' refers to the parameters that are fine-tuned, 'T' indicates the fine-tuning of all temporal attention parameters, and 'S+T' indicates the simultaneous fine-tuning of both spatial and temporal attention parameters. 'CLIP" indicates whether the CLIP features extracted have been fine-tuned for events. 'RGB' refers to the CLIP model pre-trained on RGB data, while 'Event' indicates the CLIP model fine-tuned with event data.
#Prompt
Fine-tuning
CLIP
FVD 4
SSIM t
LPIPS +
mloU f
aloU
u(1,3)
T
RGB
1972.91
0.69513
0.3651
0.266
0.507
u(13))
S+T
RGB
1378.92
0.78496
0.3076
0.252
0,525
1
S+T
RGB
1406.24
0.78374
0.3219
0.268
0.524
1
S+T
Event
1646.78
0.72779
0.3464
method of SVD. The visual results, as shown in Fig. 3. demonstrate that compared to TAU, SimVP, and other methods, our approach can predict longer durations, more precise motion, and generate events that are more stable and closer to the ground truth.
In addition to the standard tests, we further validated our method on the BS-ERGB [54] dataset, which was captured using completely different event camera. As shown in Fig. 4a and Fig. 4b, our method achieves satisfactory prediction results on this dataset, demonstrating its effectiveness. Furthermore, we conducted experiments in extreme scenarios, with the visualization results shown in Fig. 4c and Fig. 4d. Even under stringent lighting conditions, our method is still able to predict human and viewpoint motion with high accuracy, indicating strong robustness.
For downstream task results, we conducted both qualitative and quantitative evaluations on the tasks of target segmentation and object tracking, and performed qualitative evaluations optical flow estimation. Fig. 5 shows the qualitative results on downstream tasks. Due to the motion information priors provided by the high temporal resolution event prompt, our method the tank's 's turning direction, whereas using only single input RGB frame makes this difficult. Since directly comparing cross-modal results does not provide meaningful insights, we defined an upper bound ratio Rupper based on the upper bound of target tracking performance for comparison. Table 2 presents the quantitative comparison results, showing that our method significantly outperforms other methods. For more detailed definitions and results, please refer to the Appendix materials.
accurately predict
5.2 Ablation Study
Fine-tuning Layers. As delineated in Table we conduct empirical validations across various fine-tuning configurations, including temporal-only and joint during the refinement process, we keep the neural network with a similar amount of training weights. Moreover, all methods are trained under the same configuration, excluding the training parameters. The results from these experiments provide critical insights into the optimal configuration settings that enhance the accuracy and efficiency of event-based video generation. Besides, the authors also want to note that it's ineffective to only change some parts of a large generative model, since the diffusion U-Net is trained with the perception of original VAE, CLIP models generation distributions. We also
adjustments. Note that,

---

Table 4: Ablation Study of motion alignment and multi prompt. All models tested with only feeding single event voxel frame. 'EP" denotes denoising using the high temporal resolution event prompt, and 'MA' denotes motion alignment based reinforcement learning.
Method
EP
MA
FVD 4
SSIM 0
LPIPS
mloU t
aloU
A V
X
1378.92
0.78496
0.3076
0.252
0.505
BBC
X
1227.56
0.79077
0.3101
0.295
0.528
V
1119.71
0.79597
0.3246
0.277
0.516
D
V
V
1055.25
0.79981
0.3123
0.302
0.522
have experimentally validated that after changing those modules. The experimental results are shown in Table 3, where feed features from different clip models (Event-traine RGB-trained) to the SVD U-Net. Note that all CLIPs are fed with event voxels. Even further fine-tuning the SVD with plenty of data, the resulting diffusion model with Event-trained is still underperformed.
Testing-time Prompt Augmentation. As detailed in Sec. 4.1, we enhance the test-time prompt by incorporating multiple event frames of high temporal resolution. To thoroughly examine the impact of these hyperparameters and ascertain the efficacy our test-time augmentation strategy, we compare various approaches employing different test-time prompts, as delineated in Table 6, titled "Testing-time". From this comparative analysis, it evident that there is gradual improvement in the neural network's performance as the extent of prompt augmentation increases.
5.3 Discussion
While the proposed method has exhibited preliminary capabilities, it is imperative to address its existing limitations. One significant challenge is the inherent nature of event data; while it boasts high temporal resolution, this type of data typically lacks texture, thereby impeding the effective capture of detailed semantic information. This limitation highlights the difficulty in accurately representing complex visual scenes solely based on event data. Furthermore, there is pressing need for the development of a lossless representation technique that can fully preserve the unique high-temporal attributes of event data, ensuring no critical information is lost during processing. Specific constrained scenarios are visualized and discussed in detail in Appendix Section D.
To address these challenges and enhance the efficacy of the proposed method, future work should focus on several key areas. Firstly, advancing the method's 's ability to interpret texture would mark significant improvement. This could involve integrating additional sensory inputs, e.g., RGB images, or employing more sophisticated data fusion techniques. Secondly, the creation of more comprehensive event sequence dataset essential. Such dataset should encompass a wider variety of scenarios and conditions, thereby providing robust platform for training and testing the improved models. By addressing these aspects, future research can pave the way for more accurate, reliable, and versatile event-based vision systems for object motion forcasting.
6 Conclusion
In this study, we have introduced the Event-Sequence Diffusion Network, a novel approach poised to redefine the landscape of video diffusion. By leveraging event-based data, characterized by its high temporal resolution, our methodology advances the frontiers of motion modeling. enabling only rich in detail but also grounded in the realistic
the generation of video sequences that dynamics of object motion. Our approach stands in stark contrast to traditional video diffusion models that rely on single images textual prompts for initialization. By employing sequences of events as the conditioning input, we ensure a more nuanced and temporally coherent synthesis of video content.
Future work will focus on refining the Event-Seqqeent Diffusion Network, exploring its applicability across broader spectrum of computer vision tasks, and enhancing its efficiency for real-time applications. Moreover, we aim to delve deeper into the interplay between event-based data and diffusion models, seeking unlock new potentials and applications in areas such as autonomous navigation, interactive gaming, and dynamic scene reconstruction.
10

---

References
[1] A. Blattmann, T. Dockhorn, S. Kulal, D. Mendelevitch, M. Kilian, D. Lorenz, Y. Levi, 2. English, V. Voleti, A. Letts, et al. Stable video diffusion: Scaling latent video diffusion models to large datasets. arXiv preprint (rXiv:2311.51227 2023.
[2] A. Blattmann, R. Rombach, H. Ling, T. Dockhorn, S. W. Kim, S. Fidler, and K. Kreis. Align your latents: High-resolution video synthesis with latent diffusion models. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
[3] C. Brandli, R. Berner, M. Yang, S.-C. Liu, and T. Delbruck. A 240 180 130 db μs latency global shutter spatiotemporal vision sensor. IEEE Journal of Solid-State Circuits, 49(10): 2333-2341, 2014.
[4] H. Cao, C. Tan, Z. Gao, Y. Xu, G. Chen, P.-A. Heng, and S. Z. Li. A survey on generative diffusion models. IEEE Transactions on Knowledge and Data Engineering, 2024.
[5] W. Chai, X. Guo, G. Wang, and Y. Lu, Stablevideo: Text-driven consistency-awes aware diffusion video editing. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 23040-23050, 2023.
[6] W. Chen, J. Wu, P. Xie, H. W., X. Xia, X. Xiao, and L. Lin. Control-a-video: Controllable text-to-video generation with diffusion models, 2023.
[7] Z. Chen, Z. Zhu, Y. Zhang, J. Hou, G. Shi, and J. Wu, Segment any events via weighted adaptation of pivotal tokens. arXiv preprint arXiv:2312.16222, 2023.
[8] F.A. Croitoru, V. Hondru, R. T. Ionescu, and M. Shah. Diffusion models in vision: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023.
[9] Z. Ding. Zhao, J. Zhang. T. Gao, Xiong. Z. Yu, and T. Huang. Spatio ppatio-temporal recurrent networks for event-based optical flow estimation. In Proceedings of the AAAI conference artificial intelligence, pages 525-533, 2022.
[10] P. Esser, J. Chiu, P. Atighehchian, J. Granskog, and A. Germanidis. Structure and content- guided video synthesis with diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 7346-7356, 2023.
[11] S. Ettinger. S. Cheng, B. Caine, C. Liu, H. Zhao, S. Pradhan, Y. Chai, B. Sapp. C. R. Qi, Y. Zhou, et al. Large scale interactive motion forecasting for autonomous driving: The waymo open motion dataset. In Proceedings of the IEFF/CVF International Conference Computer Vision, pages 9710-9719, 2021.
[12] G. Gallego, T. Delbrück, G. Orchard, C. Bartolozzi, B. Taba, A. Censi, S. Leutenegger, A. J. Davison, Conradt, K. Daniilidis, et al. Event-based vision: A survey. IEEE transactions on pattern analysis and machine intelligence, 44(1):154-18)) 2020.
[13] Z. Gao, C. Tan, L. Wu, and Z. Li. Simvp: Simpler yet better video prediction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 3170-3180, June 2022.
[14] Z. Gao, C. Tan, L. Wu, and S. Z. Li. Simvp: Simpler yet better video prediction. In Proceedings of the IEEF/CVF Conference Computer Vision and Pattern Recognition, pages 3170-3180, 2022.
[15] S. Ge. S. Nah, G. Liu, T. Poon, A. Tao, B. Catanzaro, D. Jacobs, J.-B. Huang, M.-Y. Liu, and Y. Balaji. Preserve your own correlation: noise prior for video diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 22930- 22941, 2023.
[16] I. Goodfellow, J. Pbaget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Bengio. Generative adversarial networks. Communications of the ACM, (3(11):139-14 2020.
11

---

[17] V. L. Guen and N. Thome. Disentangling physical dynamics from unknown factors for unsuper- vised video prediction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), June 2020.
[18] J. Ho, A. Jain, and P. Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-68511 2020.
[19] J. Ho. W. Chan, C. Saharia, J. Whang. R. Gao, A. Gritsenko, D. P. Kingma, B. Poole, M. Norouzi, D. J. Fleet, al. Imagen video: High definition video generation with diffusion models. arXiv preprint arXiv:2210.02330, 2022.
[20] J. Ho, C. Saharia, W. Chan, J. Fleet, M. Norouzi, and T. Salimans. Cascaded diffusion models for high fidelity image generation. The Journal of Machine Learning Research, 23(1): 2249 2,81, 2022.
[21] J. Hou, Z. Zhu, J. Hou, H. Liu, H. Zeng, and H. Yuan. Global structure-aware diffusion process for low-light image enhancement. Advances in Neural Information Processing Systems, 36, 2024.
[22] L. Hu, R. Zhao, Z. Ding. L. Ma, B. Shi, R. Xiong, and T. Huang. Optical flow estimation for spiking camera. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 17844-17853, 2022.
[23] Y. Jiang, Y. Wang, S. Li, Y. Zhang. M. Zhao, and Y. Gao. Event-based low-illumination image enhancement. IEEE Transactions on Multimedia, 2023.
[24] T. Karras, M. Aittala, T. Aila, and S. Laine. Elucidating the design space of diffusion-based generative models. Advances in Neural Information Processing Systems, 01-26565-26577, 2022.
[25] D. Kingma, T. Salimans, B. Poole, and J. Ho. Variational diffusion models. Advances in neural information processing systems, 22211666227777 2021.
[26] A. Kirillov, E. Mintun, N. Ravi, H. Mao, C. Rolland, L. Gustafson, T. Xiao, S. Whitehead, A. C. Berg, W.-Y. Lo, P. Dollár, and R. Girshick. Segment anything. arXiv:2304.02643, 2023.
[27] M. Liang, B. Yang. R. Hu, Y. Chen, R. Liao, S. Feng, and R. Urtasun. Learning lane graph representations for motion forecasting. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part II 16, pages 541-556. Springer, 2020.
[28] Lichtsteiner, C. Posch, and T. Delbruck. A 128x 128 120 db 15 js latency asynchronous temporal contrast vision sensor. IEEE Journal of Solid-State Circuits, 43(2):566-576, 2008. doi: 10.1 20// 914377.
[29] X. Lin, J. He, Z. Chen, Z. Lyu, B. Fei, B. Dai, W. Ouyang, Y. Qiao, and C. Dong. Diffbir: To- wards blind image restoration with generative diffusion prior. arXiv preprint ar 23X8v:2308.15070 2023.
[30] F. Mahlknecht, D. Gehrig, J. Nash, F. M. Rockenbauer, B. Morrell, J. Delaune, and D. Scara- muzza. Exploring event camera- camera-based odometry for planetary robots. IEEE Roboties and Automation Letters, 7(4): 2022.
[31] A. I. Maqueda, A. Loquercio, G. Gallego, N. García, and D. Scaramuzza. Event-based vision meets deep learning on steering prediction for self-driving cars. In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5419-5427, 2018. doi: 10.1109/CVPR.
[32] K. Mei and V. Patel. Vidm: Video implicit diffusion models. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 91 9117-9125, 2023.
[33] N. Nayakanti, R. AI-Rfou, A. Zhou, K. Goel, K. S. Refaat, and B. Sapp. Wayformer: Motion forecasting via simple & efficient attention networks. 2023 IEFE International Conference on Robotics and Automation (ICRA), pages 2980-2987. IEEE, 2023.
12

---

[34] A. Nichol, P. Dhariwal, A. Ramesh, P. Shyam, P. Mishkin, B. McGrew, I. Sutskever, and M. Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. arXiv preprint 1r 10741, 2021.
[35] J. Pan, H. Yan, J. H. Liew, J. Feng, and V. Y. Tan. Towards accurate guided diffusion sampling through symplectic adjoint method, 2023.
[36] C. Posch, D. Matolin, and R. Wohlgenannt. A qvga 143 db dynamic range frame-free pwm image sensor with lossless pixel-level video compression and time-domain cds. IEEE Journal of Solid-State Circuits, 46(1):259-275, 2011. doi: 10.1 http://SS...20.2088828
[37] H. Rebecq, R. Ranfil, V. Koltun, and D. Scaramuzza. High speed and high dynamic range video with an event camera. IEEE transactions on pattern analysis and machine intelligence, 43(6): 1980-1980, 2019.
[38] H. Rebecq. R. Ranfil, V. Koltun, and D. Scaramuzza. High speed and high dynamic range video with an event camera. IEEE Transactions on Pattern Analysis and Machine Intelligence, 43(6): 1984-1980, 2021. doi: 10.1009/PPMM2200 2963996.........
[39] I.. Ruan, Y. Ma, H. Yang, H. He, B. Liu, J. Fu, N. J. Yuan, Q Jin, and B. Guo. Mm-diffusion: Learning multi-modal diffusion models for joint audio and video generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10219-10228, 2023.
[40] S. Saxena, C. Herrmann, J. Hur, A. Kar, M. Norouzi. D. Sun, and D. J. Fleet. The surprising effectiveness of diffusion models for optical flow and monocular depth estimation. Advances in Neural Information Processing Systems, 36, 2024.
[41] J. Schulman, S. Levine, P. Abbeel, M. Jordan, and P. Moritz. Trust region policy optimization. In International conference on machine learning, pages 1889-1897. PMLR, 2015.
[42] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: 1707.06347 2017.
[43] R. Shaw, S. Catley-Chandar, A. Leonardis, and E. Pérez-Pellitero. Hdr reconstruction from bracketed exposures and events. arXiv preprint arXiv:2203. 14825, 2022.
[44] S. Shiba, Y. Aoki, and G. Gallego. Secrets of event based optical flow. In European Conference on Computer Vision, pages 628 645..5..666 Springer, 2022.
[45] J. Sohl-Dickstein, E. Weiss, N. Maheswaranathan, and S. Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International conference on machine learning, pages 2256-2265. PMLR, 2015.
[46] J. Song, C. Meng, and S. Ermon. Denoising diffusion implicit models. 20X:v22010.02502, 2020.
arXiv preprint
[47] Y. Song and S. Ermon. Generative modeling by estimating gradients of the data distribution. Advances in neural information processing systems, 32, 2019.
[48] Y. Song, J. Sohl-Dickstein, D. P. Kingma, A. Kumar, S. Ermon, and B. Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011.13456, 2020.
[49] Y. Song, P. Dhariwal, M. Chen, and I. Sutskever. Consistency models. arXiv preprint 233.v2230..01669, 2023.
[50] C. Tan, Z. Gao, L. Wu, Y. Xu, J. Xia, S. Li, and S. Z. Li. Temporal attention unit: Towards efficient spatiotemporal predictive learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 18770-18782, June 2023.
[51] C. Tan, Z. Gao, L. Wu, Y. Xu, J. Xia, S. Li, and S. Z. Li. Temporal attention unit: Towards efficient spatiotemporal predictive learning. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pages 18770-18782, 2023.
13

---

[52] Thomas, S. van Steenkiste, K. Kurach, R. Marinier, M. Michalski, and S. Gelly. Fvd: A new metric for video generation. CoRR, 2019.
[53] S. Tulyakov, D. Gehrig, S. Georgoulis, J. Erbach, M. Gehrig, Y. Li, and D. Scaramuzza. Time lens: Event-based video frame interpolation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 16155-1611 2021.
[54] S. Tulyakov, A. Bochicchio, D. Gehrig, Georgoulis, Y. Li, and D. Scaramuzza. Time lens++: Event-based frame interpolation with parametric non-linear flow and multi-scale fusion. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
[55] X. Wang, J. Li, L. Zhu, Z. Zhang. Z. Chen, X. Li, Y. Wang, Y. Tian, and F. Wu. Visevent: Reliable object tracking via collaboration of frame and event flows, 2023.
[56] X. Wang, S. Wang. C. Tang, L. Zhu, B. Jiang, Y. Tian, and J. Tang. Event stream- based visual object tracking: A high-resolution benchmark dataset and novel baseline. arXiv preprint arXiv:2309. 22021119994411,, 2023.
[57] X. Wang. J. Li, L. Zhu, Z. Zhang, Z. Chen, X. Li, Y. Wang, Y. Tian, and F. Wu. Visevent: Reliable object tracking via collaboration of frame and event flows. IEEE Transactions on Cybermetics, 54(3):1997-2010, 2024. doi: 10.1 (09/TCBB.2023.33
[58] Y. Wang, M. Long. J. Wang. Z. Gao, and P. S. Yu. Predrnn: Recurrent neural networks for predictive learning using spatiotemporal Istms. Advances in neural information processing systems, 30, 2017.
[59] Y. Wang, H. Wu, J. Zhang, Z. Gao, J. Wang, P. S. Yu, and M. Long. Predrnn: recurrent neural network for spatiotemporal predictive learning, IEEE Transactions Pattern Analysis and Machine Intelligenc,, 45(2):2208 2552)22208-2225, 2023, doi: 10.1199/PAAM2222.3166555.
[60] Y. Xu, L. Chambon, M. Chen, A. Alahi, M. Cord, P. Perez, et al. Towards motion forecasting with real-world perception inputs: Are end-to-end approaches competitive? In International Conference on Robotics and Automation (ICRA), 2024.
[61] L. Yang, Z. Zhang, Y. Song, Hong, R. Xu, Y. Zhao, W. Zhang, Cui, and M.-H. Yang. Diffusion models: A comprehensive survey of methods and applications. ACM Computing Surveys, 56(4):1-39, 2023.
[62] B. Ye. H. Chang, B. Ma, Shan, and X. Chen. Joint feature learning and relation modeling for tracking: A one-stream framework. In ECCV, 2022.
[63] R. Zhang, P. Isola, A. A. Efros, E. Shechtman, and O. Wang. The unreasonable effectiveness of deep features as a perceptual metric. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 586-595, 2018.
[64] Y. Zhang, E. Tzeng, Y. Du, and D. Kislyuk. Large-scale reinforcement learning for diffusion models, arXiv preprint arXiv:2401. 12244, 2024.
[65] J. Zhao, S. Ji, Z. Cai, Y. Zeng, and Y. Wang. Moving object detection and tracking by event frame from neuromorphic vision sensors. Biomimetics, 7(1):1,, 2022.
[66] A. Z. Zhu, L. Yuan, K. Chaney, and K. Daniilidis. Ev-flownet: Self-supervised optical flow estimation for event-based cameras. arXiv preprint arXiv: 2018.
[67] Z. Zhu, J. Hou, and X. Lyu. Learning graph-embeaded key-event back-tracing for object tracking in event clouds. Advances in Neural Information Processing Systems, 35:7462-7476. 2022.
[68] Z. Zhu, J. Hou, and D. O. Wu. Cross-modal orthogonal high-rank augmentation for rgb- teeaffommr-traccern In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 22045-22055, 2023.
[69] Y. Zou, Y. Zheng, T. Takatani, and Y. Fu. Learning To Reconstruct High Speed and High Dynamic Range Videos From Events. In Proceedings of the IEEF/CVE Conference on Computer Vision and Pattern Recognition, pages 2024-2033, 1021.
14

---

Appendix Overview
In this Appendix material, we provide additional details to complement the content of the paper, including the training and inference details of our method(Sec.A)) an analysis of the results from comparative methods, more experiments on downstream tasks(Sec.B), details related to reinforcement leaming(Sec.C) and limitation of proposed method(Sec.D).
A Training and Inference Details
We provide a comprehensive training pipeline along with detailed parameter specifications, encom- passing the model architecture and fine-tuning parameters. For voxelized event data, we perform preprocessing and data augmentation to enhance the performance and generalization capability the model.
A.1 Architecture
As discussed in Section. 4.1, We employ the temporal U-Net architecture based on Stable Video Diffusion. During training, we preprocess the event sequence into multiple frames of voxel data, which serve conditional inputs the network. Fig. illustrates the overall training procedure, which bears resemblance to SVD. The initial segment of the event sequence serves as the conditional input the U-Net model. For the parameters trained in the U-Net, we only fine-tuned the cross- attention layer that incorporates the event condition. We conducted ablation experiments on different attention layers, as shown Table Simultaneously fine-tuning both spatial and temporal attention layers yielded the best results.
X Steps
Video Encoder
Noised
Shared
Sequence
U-Net
Results
VAE
$
Repeal
Encoder
Attn_ Block
Sputial-T Block
Temporal-T
First frames
S-Atm
Block
Trained params
Condition
CLIP Image Embedding
params
Frozen blocks
Figure 6: Training workflow of the proposed method, where the left upper one indicates the target noised latent and the left lower represents the prompted event sequence. We concatenate the prompt information with denoising latent for noise learning. Moreover, the feature from the CL.IP model is also injected into the diffusion U-Net.
A.2 Datasets and augmentation
During the pre-training phase of the proposed diffusion model, events of poor quality can adversely affect model training. Therefore, we emulate the approach taken in EventSAM [7] by filtering out sequences with severe degradation in VisEvent. Finally, integrated 291 event sequences from the VisEvent dataset and 841 sequences from the EventVOT dataset. We omitted specific sequences from the CeleX-V V sensors due to their absence of polarity information. For the alignment phase, uniformly sourced all data, the same as the training dataset, aligning them in accordance with the pre-training prompts. In the evaluation stage, random selection of 100 sequences from each dataset was employed to rigorously assess the efficacy our proposed methodology.
During training, we randomly crop the Visevent sequences using 256 x 256 kernel and then resize them to 128 128 before feeding them into the network. For the higher-resolution EventVOT dataset, in order to preserve as much motion information possible, randomly select crop kernels ranging
15

---

from 512 to 720, crop the images accordingly. and then resize them to 128 x 128 as well. We divide the events into intervals of 20ms.
After events are converted into voxels, normalization is required. Simple normalization the range of 0 may lead flickering in the sequence. Therefore, when obtaining the sequence, perform a smoother normalization using maximum value normalization. Specifically, dividing each frame of the sequence by the maximum absolute value of the sequence ensures that the voxel values between O and and remain stable.
B Downstream tasks
To validate the reliability of the generated event voxels, we assessed our approach's performance on several classic downstream tasks such object tracking, segmentation, and optical flow estimation.
B.1 Evaluation Methods and Metrics
For the object tracking task, we utilized the OSTrack [62] library to evaluate the event voxels estimated by our method and the RGB frames estimated by SVD [1], focusing on the selected VisEvent test set ensure alignment. Due the significant modality differences between event data and RGB data, directly comparing tracking metrics would be unfair. Therefore, we tested on the ground truth of the selected test sequences separately for each modality, using these as performance upper bounds. The model performance was assessed by analyzing the discrepancies between the obtained results and these upper bounds. We utilized AUC (Area Under the Cur5 OP75 and PR(Precision and Recall) as our base evaluation metrics,upon which we calculated the mean ratio of each metric to its upper bound, obtaining a cross-modality tracking evaluation metric Rapper:
AUC
OP50
OP75
PR PRappee
Rupper Mean( AU Cupper
0500upper
P77
For the object segmentation task, we employed SAM [26] to segment objects in the RGB domain and fine-tuned EventSAM [7] on event sequences for object segmentation in the event domain. Given the absence of segmentation ground truth in the selected dataset and the strong generalization capabilities of the segmentation models their respective domains, we used the sequence ground truth segmentation results as the ground truth for our evaluations. we employ mIOU (mean Intersection over Union) and alOU (average Intersection over Union) as evaluation metrics.
Optical flow estimation is particularly challenging due to the limited and narrowly distributed datasets with optical flow ground truth. Although event-based optical flow estimation has been explored, it often suffers from poor generalization performance. Consequently, we restricted our qualitative experiments simple motion scenarios involving single object better understand the model's performance under controlled conditions. The DDVM [40] was employed to estimate the optical flow for both event and RGB sequences.
B.2 Results Analysis
The quantitative results of the tracking are presented in Table 2. Due to the modality differences, the overall tracking performance of RGB surpasses that of event tracking, as evident from their respective ground truth sequences. However, our method exhibits a higher ratio of upper bounds compared SVD, indicating that our estimated results are closer the ground truth, making the motion estimation more reliable.
Fig. 5 and Fig. 7 depict the visual results of object tracking. In Fig. 5c, due to erroncous motion trajectory prediction by SVD, a significant discrepancy between tracking and labeling is observed. Conversely, our method demonstrates better trajectory estimation, attributed to the high temporal resolution event prompts providing prior information on motion during denoising.
Table 1 presents the quantitative results of object segmentation, where our method achieves the best performance. Additional visual results are provided Fig. 10, illustrating that the event segmentation generated by our method closely resembles the ground truth.
16

---

Optical flow is typically employed to describe object motion. In our work, the emphasis of optical flow estimation lies in the accuracy of motion prediction. Fig. 5 and Fig. illustrate that our method can predict motion more precisely, leading to better optical flow estimation results.
C Reinforcement learning for SVD
C.1 Event voxel normalization based on standard deviation.
In reinforcement learning, the choice of the reward function is crucial factor in determining training outcomes. We selected the FVD and SSIM metrics, which are closest to perceptual results, as our reward Sumntions.Speccficee ((F))(z))) where À equals 2. Upon each model update, reward scores of the data generated from the previous batch are normalized to a standard normal distribution.
the final reward function is defined R(x) (x)
Further, To quantitatively demonstrate the rationale behind our selection of these two metrics, we actually have experimentally tried to utilize a mixture of all metrics to model the reconstruction reward. However, the performance results are much worse than the method we ultimately used, as shown in Table. 5. It may be due to that sometimes the optimization of pixel-level metrics of MSE and PSNR may be contradictory to perceptual metrics. Thus, such a mixture of metrics makes the objective hard to optimize. Meanwhile, due to the target data distribution, it's more plausible to optimize the process on the data manifold (perceptual space) than the raw space.
Table 5: Ablation Study of reward metrics
Reward Metrics
MSE
PSNR
FVD L
FID \
SSIM f
LPIPS +
mixture metrics
0.0240
16.198
1562.66
265.32
0.6674
0.3463
FVD & SSIM
0.0170
17.696
1055.25
243.45
0.7998
0.3123
Directly utilizing spatial metrics as evaluation criteria in the RGB domain is typically feasible because RGB data contains dense spatial information, and the distribution of motion and static scenes is relatively similar. However, this approach is not viable for the event domain, which only records motion information. In Fig. 8b, the samples illustrate this point. As depicted in the rightmost column, when capturing a static scene, the event camera does not generate events, leading to a high original reward score. However, the samples in the first two columns occur in scenarios with significant motion, resulting in relatively high-quality, natural outcomes. Nevertheless, when normalized alongside the results from the first column, even these samples yield negative scores.
Fig. 8a illustrates the distribution of reward scores with respect to sample standard deviation. Typically, higher standard deviations in samples correspond lower scores, indicating poorer performance. Conversely. in relatively static scenes, higher scores are typically obtained. Such outcomes may inadvertently induce an overall trend toward static motion, which not desirable. As illustrated by the green curve in Fig. 8a, after several model parameter updates, the reward score begins to decline, ultimately resulting unsatisfactory outcomes.
To address this issue, we perform normalization of samples based on their standard deviation each time parameters are updated. Specifically, the standardized SCORE,td denoted as SCORE,tt SCORE + 3 (sld(r) sidmin). ). where sidmin represents the minimum value of the total standard deviation of the previous batch of generated samples and equals 30.
C.2 Training Setting and Results Analysis
During the training process, we initialize the training with the pre-trained model obtained from Section 1. The batch size for training is set to 64, and the model is updated every 100 iterations, simultaneously updating the sample pool. Gaussian normalization and standard deviation normaliza- tion are sequentially applied to the sample pool. After standardizing the samples, the training process returns to a positive trajectory. The purple curve in Fig. 8a represents the score curve during training, with convergence observed around the 1000th iteration.
Fig. 11 illustrates the visual results before and after reinforcement learning. Due to the inherent unpredictability and randomness in the generation of results by diffusion, utilizing only the results
17

---

Algorithm 2 Multi Prompt Reverse Process
1: XTm (0, I) ~ I
2: For t = T.... do
3: if 2 then
4: X +- o
04 xx pm
5: X, ree + replace (X, Xrr).
6: Sampling X from X, via Eq. 3.
7: else Sampling x from X via Eq. 3.
8: End for
9:
Return X
Table 6: Ablation study different number of testing time prompts. Here we only compare with the sesting-iimeeee nearest 10 future event voxels. All experimental settings share the model weights.
Metric
1
4
8
12
15
MSE
0.0177
0,0195
0.0170
0.0196
0.0178
0,0182
FVD 4
1170.48
1196.43
1055.25
1137.69
1142.15
1188.01
FID +
242.318
247.803
243.451
243.079
238.369
239.297
SSIM f
0.79349
0.78988
0.79981
0.80718
0.82553
0.83545
LPIPS
0.3272
0.3228
0.3123
0.3228
0.3189
0.3308
mloU
0.287
0.296
0.302
0.302
0.309
0.302
aloU f
0.518
0.527
0.522
0.529
0.525
0,518
generated by the pre-trained model (Fig. 11b) often leads to uncontrollable distortions and defor- mations. However, the results after reinforcement learning (Fig. 11c) tend to be more stable, with motion trends closely resembling real-world scenarios.
Telror
Oppiaal The Separatee
Opec
(a) A tank moving towards the right.
(b) A car moving towards the left.
Figure More results of downstream tasks.
D Limitation
D.1 Limitation scenarios
Although our method achieves good results in most scenarios, it is still significantly limited in the following cases due to the sampling characteristics of event data: (1) Complex background scenarios. Event cameras may capture incomplete textures in certain situations, leading to poorer prediction performance, as shown in Fig. 14a. Complex backgrounds can reduce the clarity of the target object, resulting in worse outcomes, especially in cases where the camera lens is shaking. (2) Heavily overlapped object scenarios. When objects overlap, their motion becomes quite complex, and due to the edge-focused characteristics of event cameras, understanding such motion is
18

---

by M/
Couund Touth Truth
Eatintte
003 2000
1200
1000
Score: 0,950
0.1682
2,008
Score: 1,800
5L0078
800 8.15 0.30
030
(a)
(b)
(c)
Figure 8: Analysis of the Reinforcement Learning Process. Fig. 8a illustrates the reward curves during training, with the purple curve representing the scenario with standard deviation normalization applied, and the green curve representing the scenario without it. 8b displays partial visualization results during the training process, where the first represents the ground truth, and the second row depicts the results estimated by the pre-trained model. Fig. 8c illustrates the distribution of reward scores with respect to standard deviation normalization for all training samples.
Table Comparison methods in terms of parameters and FLOPs.
Methods
Params (M)
FLOPs (G)
PredRNNv2
23.9
48.92
SimVP
58.0
60.61
TAU
44.7
92.50
ours
1521.0
693.92
challenging, shown in Fig. 14b and Fig. 14c. When people overlap, their footsteps often become chaotic, leading to less accurate predictions.
D.2 Computational Resource Scalability
The following Table. compares the computational resources of our method with SOTA methods. The powerful generative capability and high fidelity of diffusion models lead to the cost of substantial computational resource consumption. As shown in the following table, our parameter count and FLOPs significantly exceed those of traditional models. However, we believe this trade-off is necessary because of the powerful learning capability of large models in the real world. Taking the future motion estimation task as an example, our diffusion-bal bbased method significantly surpasses traditional methods in understanding and learning motion.
For salable model size for different inference environments, there indeed are many works indicating that the diffusion model can be applied to quantization or other acceleration techniques for speeding up the inference process. We believe that with the advancement of hardware and acceleration techniques, the inference speed of diffusion models will be significantly improved in the near future.
19

---

Gronnd T39
Storp
TAC
Oue
sal6
(a) A bicycle riding into an obstacle.
Grumt Tron
Saury
TAC
Iepul
tikk
(b) A car driving to the left, with the camera lens rotating follow.
Grond Toun
(c) Two bicycles traveling towards each other and intersecting.
Figure 9: Qualitative comparison between SOTA methods. The first row each sequence represents the ground truth of the event sequence. The second and third rows respectively depict the results of future event estimation by Si [14] and TAU [50]. The final row represents the results obtained by method.
20

---

Oroedd Toon
Ass
Siorg
Our
A
A
TAU
Ingun
1+2
b-4
t+$
EHED
5-12
1-14
1w
tw15
3x20
Grond Trih
Segmottiim
Dalnae
Segnentamn
Grund Torh
Segnettato
.4
Segmoradio
6+18
Grond Trubh
Dosnuve
Tegut
1+2
t-4
1+G
t+5
EeID
9-12
t+34
b-16
t+18
>420
Figure 10: Additional results of Segmentation. The first row of each sequence represents the ground truth of the events, and the second row shows the segmentation results. The last two rows respectively display the results estimated by our method and the corresponding segmentation results.
21

---

Input
153
126
1-12
LF15
1=18
E21
E2
(a) Ground Truth of the Car Sequence
Visualization of the Pre-train model's Estimation Results(without Motion Alignment)
(c) Visualization of the Results after Motion Alignment
Figure 11: Visualization results of Motion Alignment. Fig. 1la shows the ground truth of the car sequence; Fig. lb presents the estimated results using pre-trained model with 9 different random seeds, where several instances resulted in failures: Fig. illustrates the motion alignment results generated by applying reinforcement learning, yielding more stable estimation.
22

---

batrrass
Ispor
1+2
be4
548
5-10
1+12
1-14
tw
I-|8
5420
(a) Rotating Umbrella.
5
5
5
binoss
(b) Falling Balloon.
Figure 12: Visualization of our method's 's prediction in hs-ergb dataset.
23

---

Isput
1+2
be4
58
1-10
1+12
1e14
bx16
[+]8
5+20
(a) Poor exposure: The car driving out the tunnel.
TAU
bitoos
(b) Occlusion: A person walking from bright place to shadow.
Figure 13: Visualization of our method's prediction in various scenarios.
24

---

Em4
0-10
1-12
1-14
I-15
0430
(a) Shaky camera capturing a stationary person.
Ixpot
124
1:10
e:14
1:
1:18
(b) A person running out behind another person.
Iipor
Two people walking side by side.
Figure 14: Visualization of our method's prediction in severely degraded scenarios.
25

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The claims indeed reflect the paper's contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We have described the limitations of our work in Sec
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
26

---

Justification:
No theoretical results
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We provide detailed settings and parameters for the experimental training and testing.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
27

---

Answer: [Yes]
Justification: We provide the source code and information on how to obtain the datasets used in the Appendix materials.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (uttps://nips.. ec/
for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: cenollc/ggiiess...
/ /nips...
sssissii ssissssliccyyyy for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We provide all experimental settings and details in Sec. A.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer:
[NA]
Justification:
We fixed the random seed to reproduce the experimental results.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
28

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We provide the experimental platform details Sec. 5.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: : //tt://eee.../eeeeeeee cccpppp.........c.ooooooo ct/ccc/pp// http/puull/etiiieeeeeeeeee csGuidel ines?
Answer: [Yes]
Justification:
We meticulously reviewed the ethical guidelines.
Guidelines:
The answer NA means that the authors have reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discuss the potential societal impacts in Sec. I and Sec. 6.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
29

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
We do not have such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We have provided proper attribution for every asset used in the paper.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/dataseta has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
30

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We have included the source code in the Appendix material. And we will also release the Pre-train model.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/code/moodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
We do engage in these subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
31


Reviews:
[{"review_id": "GvzCMC5QKZ", "reviewer": "Reviewer_8wZD", "summary": "The paper introduces a novel framework that leverages the high temporal resolution of event-based sensors to predict future motion trajectories with unprecedented detail and precision. The authors propose an integration of video diffusion models with event camera data, resulting in an Event-Sequence Diffusion Network. This network is designed to capture the nuances of dynamic scenes and generate video sequences that are both rich in detail and grounded in realistic motion dynamics.", "questions": "Has the paper addressed how the model performs under various conditions such as different lighting, weather, or in the presence of occlusions?\nDoes the paper provide evidence of the model's ability to generalize beyond the datasets it was trained on? Are there any specific domains or scenarios where the model might underperform?\nAre there discussions on the computational resources required to run the model, and is it scalable for use in resource-constrained environments?\nDoes the paper discuss any potential misuse of the technology, such as in surveillance or other applications that might infringe on individual rights?\nAre there discussions on how the technology could affect different demographic groups differently, potentially exacerbating existing biases?\nAre there any regulatory or compliance issues related to the deployment of such technology, especially in sectors like automotive or robotics?", "limitations": "See above", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "Integration of Event Sequences with Video Diffusion Models: The paper presents the first attempt to combine event sequences with a video diffusion model, creating an event-sequence diffusion model capable of estimating future object motion.\n\nReinforcement Learning for Motion Alignment: The authors propose a method to align the pre-trained event-sequence diffusion model with real-world motion using reinforcement learning techniques, enhancing the fidelity and coherence of the generated motion sequences.\n\nTest-Time Prompt Augmentation: A method is introduced to augment the test-time prompt with high temporal resolution event sequences, which improves the generation performance of the diffusion model.\n\nExtensive Testing and Validation: The authors demonstrate the effectiveness of their approach across various complex scenarios, showcasing its potential for applications in autonomous vehicle guidance, robotic navigation, and interactive media.\n\nPromising Direction for Future Research: The findings suggest a new direction for enhancing the interpretative power and predictive accuracy of computer vision systems, particularly in the context of motion flow prediction.\n\nThe paper's contributions are significant as they push the boundaries of motion estimation in computer vision by harnessing the unique capabilities of event-based sensors and integrating them with advanced diffusion models. The proposed framework opens up new possibilities for accurate prediction of dynamic environments, which is crucial for various real-world applications.", "weaknesses": "Following is my concerns:\nWhile the paper demonstrates strong results in controlled scenarios, it may lack evidence of how well the model generalizes to a broader range of real-world conditions, such as various weather effects or low-light environments.\nThe paper could benefit from a more detailed discussion on the computational efficiency of the proposed model. Including runtime analysis and resource requirements would provide a clearer picture of the model's practicality for real-time applications.\nAlthough the paper acknowledges the high temporal resolution of event data, it could delve deeper into the limitations of such data, such as the lack of texture information, and how this might impact the model's performance in complex visual scenes.\nWhile the paper includes some ablation studies, there could be a more thorough investigation into the contribution of each component of the model. This would provide clearer insights into which aspects are most critical to the model's performance.\nThe paper could address how the model performs under noisy conditions or when outliers are present in the event data. This is particularly important given the sensitivity of diffusion models to input quality.\nThere is an opportunity to discuss the explainability and interpretability of the model's predictions. Understanding the factors that contribute to the model's decisions could be valuable for applications in autonomous systems.\nAlthough the paper touches on potential societal impacts, a more detailed discussion on ethical considerations, such as privacy concerns or the potential for misuse, could be beneficial.\nWhile the paper mentions the availability of source code, providing more detailed instructions on how to reproduce the experiments, including the exact versions of software and hardware used, would enhance the reproducibility of the study.\nThe paper focuses on short-term motion estimation. It could discuss the model's capability or limitations in predicting motion over longer time horizons, which is crucial for some applications.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## **Q1:** Performance in Challenging Visibility\nBenefiting from the unique characteristics of event cameras, our event-based video diffusion framework **can handle future motion estimation issues** to some extent. To further address your concern about algorithm performance on challenging visibility scenes, we conducted experiments across various scenarios, as shown in Fig.1 of the uploaded PDF file. Specifically, we first illustrate the **poor exposure** scene in which a car is passing through a tunnel (Fig.1(a)). The proposed method clearly predicts the car's contour, whereas the contour is difficult to discern from the RGB data and even some frames of the GT event data.\n\nFurthermore, for the **object occlusion**, the experimental results shown in Fig.1(b) illustrate a scenario where a person is passing through an occluding object. Our method successfully estimates the future motion of the object despite the occlusion. Similarly, Fig.1(c) demonstrates that when a bicycle enters an occlusion, our method provides a more accurate prediction of the bicycle's motion after it enters the occlusion.\n## **Q2:** Generalization Ability on Other Datasets\nWe further validate the generalization ability of the proposed algorithm on other datasets, i.e., CRSOT[1], VisEvent([54] in paper), and hs-ergb[2]. The following table presents the test results, the consistent performance of our method indicates strong generalization. Fig.1 and Fig.2 in the uploaded PDF demonstrate that our method outperforms existing methods across various scenarios from these datasets.\n|Datasets|Methods|Scenarios|*FVD*↓|*FID*↓|*SSIM*↑|*LPIPS*↓|\n|----------|-----------|----------------|-----------|-----------|----------|-----------|\n|CRSOT|PredRNNv2|normal|1120.2|252.8|0.809|0.453|\n|CRSOT|SimVP|normal|834.7|202.5|0.829|0.328|\n|CRSOT|TAU|normal|811.7|196.8|0.827|0.353|\n|CRSOT|ours|normal|**780.6**|**192.6**|**0.853**|**0.290**|\n|VisEvent|PredRNNv2|Poor exposure|2321.6|308.8|0.593|0.535|\n|VisEvent|SimVP|Poor exposure|2124.4|248.5|0.642|0.379|\n|VisEvent|TAU|Poor exposure|2125.0|**241.9**|0.651|0.363|\n|VisEvent|ours|Poor exposure|**1638.0**|322.3|**0.696**|**0.291**|\n|hs-ergb|PredRNNv2|close|1675.1|279.0|0.619|0.565|\n|hs-ergb|SimVP|close|1363.9|273.1|0.655|0.458|\n|hs-ergb|TAU|close|1343.3|274.1|0.705|0.395|\n|hs-ergb|ours|close|**1109.7**|**232.3**|**0.726**|**0.290**|\n\n[1] Zhu Y, Wang X, Li C, et al. Crsot: Cross-resolution object tracking using unaligned frame and event cameras[J]. arXiv preprint arXiv:2401.02826, 2024.\n\n[2] Tulyakov, Stepan, et al. \"Time lens: Event-based video frame interpolation.\" Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2021.\n## **Q3:** Limitation \\& Failure Cases\nOur method is relatively limited in the following scenarios:\n\n(1)  **Complex background scenarios.** Event cameras may capture incomplete textures in certain situations, leading to poorer prediction performance, as shown in Fig. 3(a) of the uploaded PDF. Complex backgrounds can reduce the clarity of the target object, resulting in worse outcomes, especially in cases where the camera lens is shaking.\n\n(2)  **Heavily overlapped object scenarios.** When objects overlap, their motion becomes quite complex, and due to the edge-focused characteristics of event cameras, understanding such motion is challenging, as shown in Fig.3(b) and Fig.3(c). When people overlap, their footsteps often become chaotic, leading to less accurate predictions.\n\nHowever, we also note that those scenes (extreme complexity or occlusion) are also very challenging for traditional RGB-based vision. We expect that in future research, a diffusion model that perceives different modalities may help to address those problems.\n## **Q4:** Computational Resource \\& Scalability\nThe following table compares the computational resources of our method with SOTA methods. The powerful generative capability and high fidelity of diffusion models lead to the cost of substantial computational resource consumption. As shown in the following table, our parameter count and FLOPs significantly exceed those of traditional models. However, we believe this trade-off is necessary because of the powerful learning capability of large models in the real world. Taking the future motion estimation task as an example, our diffusion-based method significantly surpasses traditional methods in understanding and learning motion. \n\nFor a salable model size for different inference environments, there indeed are many works [1] indicating that the diffusion model can be applied to quantization or other acceleration techniques for speeding up the inference process. The authors will also add further illustrations in the final version.\n|Methods|Params (M)|FLOPs (G)|\n|---------|----------|---------|\n|PredRNNv2|23.9|48.92|\n|SimVP|58.0|60.61|\n|TAU|44.7|92.50|\n|ours|1521.0|693.92|\n\n[1]So, Junhyuk, et al. \"Temporal dynamic quantization for diffusion models.\" Advances in Neural Information Processing Systems 36 (2023).\n\n## **Q5:** Different demographic groups \\& Potential misuse\nSince the event sensors have a higher dynamic range and without color information, the authors believe it has better equality for different skin color races. Moreover, because it contains less texture information, the individual's privacy can be better protected. We will add further discussions in the final version.\n## **Q6:** Regulatory and Compliance Issues\nFor applying the proposed algorithm on the robotic or automotive platform, such a system must have an event sensor first. Moreover, sufficient computational resources are also necessary for network running. We will add further discussion in the final version of the paper. Thanks for the advice.\n## **Q7:** Source Code\nIn the supplementary materials file provided, we include the hyperparameters and core components required for training the network. We will also make the code open-source in the future to facilitate replication of our methods by other researchers.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Yy8YPYNG7Q", "reviewer": "Reviewer_Yrkm", "summary": "This work focuses on the task of future motion estimation, where the goal is to leverage event-based vision sensors (an alternate modality, compared to traditional vanilla RGB inputs) to predict motion flow in settings useful for robotics and autonomous vehicles. The authors propose a method that leverages stable video diffusion models (pretrained on RGB settings) and adapting them with an event sequence dataset for this specific task. They authors consider two large-scale datasets (VisEvent and EventVOT), showing improvements in FVD and mIoU compared with prior work, and extensions to settings like video object tracking. The paper also discusses ablations of the method, with varying prompts and fine-tuning techniques.", "questions": "Overall, the preliminary rating of the work leans borderline+; the work offers an exploration of a modern set of diffusion tools and related techniques in a relatively underexplored area (which can have some useful applications downstream), but there remain some questions regarding the analysis and broader novelty. If the authors could address the questions and clarification areas listed in the weaknesses section above for the rebuttal, it would be helpful to inform the discussion phase and final rating.", "limitations": "The authors provide a discussion of limitations, future work, and impacts in the paper and supplement.\n\n---\n\n**Post-rebuttal update:** The rebuttal discussion and additional results help to strengthen the initial impression of the work. Given the reviewer consensus, I am maintaining my rating leaning towards acceptance. (And given that I also believe that the reviewers have adequately addressed concerns of a reviewer who has not updated their review, I am upgrading my rating a bit further since I believe the rating for this work falls between 5-6).", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "`+` Motion estimation (both flow and video object tracking) are important sub-tasks for embodied vision applications (robotics, autonomous vehicles).\n\n`+` The proposed framework is a sensible extension on traditional RGB-only stable diffusion models, and represents a good early exploration of incorporating recent techniques to this relatively smaller focused area of research.\n\n`+` The results indicate promising improvements over ablation variations and key baselines for this task, and the authors examine several different downstream tasks (segmentation, tracking and flow estimation).", "weaknesses": "`-` *Additional analysis w.r.t. baselines.* It is unclear why some of the metrics show regression in the ablation analysis and comparison tables. For example, in Table 1, TAU and SimVP outperform on FID and aIoU metrics, while the qualitative visuals seem to indicate a substantially different picture (are there examples for which these prior work show more compelling visual results, and if so, what areas of improvement could be identified by this?). Relatedly, another example is in Table 2/3, where it is unclear why some ablations (e.g., removing motion alignment with RL, one of the core technical listed contributions) show improvements over the full approach. If the authors could expand further on this analysis it would be helpful to understanding the overall value and impact of the work relative to the prior work in this space.\n\n`-` *Novelty beyond specific context of event sequence inputs.* This specific area for video event understanding is relatively niche, so while novelty is present, it is also limited to this specific context. In particular, the broader ideas around incorporating similar signals for video diffusion (inputs and outputs) have been explored previously, e.g. [A1, A2] ([A1] considers diffusion models for optical flow and monocular depth estimation [related tasks], and [A2] specifically looks at incorporating related depth estimation signals). Additional discussion with such methods (beyond the brief note in L545-547 in supplement section C) would be helpful to better contextualize the broader potential impact of the work beyond this specific domain.\n\nReferenced above:\n\n[A1] The Surprising Effectiveness of Diffusion Models for Optical Flow and Monocular Depth Estimation, NeurIPS 2023. ([40] in paper, referenced in supplement section C.)\n\n[A2] Control-A-Video: Controllable Text-to-Video Generation with Diffusion Models, 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## **Q1:** Additional Analysis w.r.t. Baselines\n\nThere are indeed some conflicts between different metrics, e.g., FID, IOU, and FVD, because only FVD can comprehensively evaluate the both spatial and temporal distribution alignment between the generated samples and GTs. FID and IOU can only measure the per-frame (spatial) distributions, which neglects the spatial-temporal consistency. However, through showing those metrics, we want to illustrate a comprehensive evaluation of different methods. To further resolve your concern, we draw a sample metric distribution graph, comparing the proposed method with SimVP, which performs best in terms of aIoU. The results are shown in Fig. 5(a) and Fig. 5(b) of the uploaded PDF file, It can be seen that our method overall outperforms SimVP, but SimVP achieves excessively high scores on certain samples, such as the blank scene shown in Fig. 5(c).\n\nAs shown in Lines **187-190** in the main text, we utilize the FVD and SSIM as metrics to reward the motion alignment reinforcement learning process. Thus, it's plausible that the trained metrics are increased. Moreover, as aforementioned, the FVD is the most principle metric for evaluating the generation results. Further, we actually have experimentally tried to utilize a mixture of all metrics to model the reconstruction reward. However, the performance results are much worse than the method we ultimately used, as shown in the following table. It may be due to that sometimes the optimization of pixel-level metrics of MSE and PSNR may be contradictory to perceptual metrics. Thus, such a mixture of metrics makes the objective hard to optimize. Meanwhile, due to the target data distribution, it's more plausible to optimize the process on the data manifold (perceptual space) than the raw space. We also refer the reviewer to the second reply for the Q2 of **reviewer AE3a**.\n\n**Table: Ablation Study of reward metrics**\n\n| Reward Metrics  | *MSE* ↓ | *PSNR* ↑ | *FVD* ↓ | *FID* ↓ | *SSIM* ↑ | *LPIPS* ↓ |\n| --------------- | ------- | -------- | ------- | ------- | -------- | --------- |\n| mixture metrics | 0.0240  | 16.198   | 1562.66 | 265.32  | 0.6674   | 0.3463    |\n| FVD & SSIM      | 0.0170  | 17.696   | 1055.25 | 243.45  | 0.7998   | 0.3123    |\n\n## **Q2:** Related Work\n\nThanks for indicating such wonderful work. The authors will definitely include more discussions with related works in the main body of the paper in the final version. Moreover, we write some discussions as below shown:\n\n\"In recent years, the development of multimodal diffusion technology has advanced rapidly. Researchers are dedicated to applying the powerful generative capabilities of diffusion to different modalities with unique advantages, such as optical flow and depth. Saxena et al. [A1] were the first to apply diffusion models to optical flow and depth estimation. For the characteristics of training data, they introduced infilling, step-rolling, and L1 loss during training to mitigate distribution shifts between training and inference. To address the lack of ground truth in datasets, they also used a large amount of synthetic data for self-supervised pretraining, enabling the diffusion model to acquire reliable knowledge. Chen et al. [A2] utilized the motion information embedded in control signals such as edges and depth maps to achieve more precise control over the text-to-video (T2V) process. They used pixel residuals and optical flow to extract motion-prior information to ensure continuity in video generation. Additionally, they proposed a first-frame generator to integrate semantic information from text and images. Unlike them, we focus on exploring the rich motion information contained in event data and use it to conditionally achieve more precise control over the generation of future motions. Furthermore, we also investigate the significant role of reinforcement learning in video diffusion and the task of motion estimation.\"\n\n[A1] The Surprising Effectiveness of Diffusion Models for Optical Flow and Monocular Depth Estimation, NeurIPS 2023.\n\n[A2] Control-A-Video: Controllable Text-to-Video Generation with Diffusion Models, 2023.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dLRu9jm5Ic", "reviewer": "Reviewer_AE3a", "summary": "The paper explores video diffusion models on the modality of information captured by event cameras. Stable Video Diffusion model is fine-tuned on an event stream dataset. On top of traditional diffusion set up, additional training using FVD and SSIM losses as rewards in a PPO is done. A method to inject motion priors during inferences further proposed. The method is evaluated in terms of generation quality downstream segmentation and object tracking applications.", "questions": "The main point to address in the rebuttal is the discrepancy between the commonly RGB-based metrics like FID, SSIM, LPIPS, and whether it makes sense to apply them to event data.", "limitations": "The limitations are discussed.", "rating": 5, "confidence": 2, "soundness": 1, "presentation": 3, "contribution": 3, "strengths": "1) The paper explores an interesting problem of predicting future motion based on temporally dense settings offered by event cameras.\n2) The proposal builds on successes of SVD models and adapts to work in  the event stream domain. Additional techniques such as PPO-based optimisation and guided sampling are also interesting.\n3) The writing is sufficient, and the presentation of the results is good.", "weaknesses": "1) While the main motivating point is to benefit from the specifics of event stream data, the processing of it is done by treating it mostly like RGB stream, including VAE and CLIP encoders (B.1 Fig. 5). Does this not abandon the benefits of the event stream data?\n2) The majority of the metrics used are defined and proposed in the RGB space (FID, SSIM, LPIPS). However, they appear to be applied on top of the event stream data. It is not immediately clear whether this is correct or would signal the results in the same way. Moreover, the results appear to be presented alongside the evaluation done on RGB modality, although these are not directly comparable. \n3) The main motivation is \"future motion estimation\"; however, the predictive accuracy is measured in a more perceptual, structural way (SSIM, LPIPS) and not on more \"raw\" metrics like PSNR or MSE.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## **Q1:** Event-specific Design\n\nThe authors want to note that the proposed method **indeed makes event-based designs**. Specifically, during the high temporal resolution guided sampling stage in Section 4.1 of the main text, our method fully leverages the high temporal resolution and flexible sampling capabilities of event cameras. We divided the temporal bin of event data into multiple sections and achieved a high-temporal resolution diffusion process by prompting those high-temporal resolution event representations. \n\nMoreover, due to the large and diverse training datasets and extensive training, those RGB-based modules (VAE and CLIP) have very strong generalization ability. Although it may not outperform some event-based designs on specific datasets, its generalization and robustness make it not fail in most scenes. This is also why we retained the original architecture and adjusted certain weights to adapt the SVD to event data. Besides, the authors also want to note that it's ineffective to only change some parts of a large generative model, since the diffusion U-Net is trained with the perception of original VAE, CLIP models generation distributions. We also have experimentally validated that after changing those modules. The experimental results are shown in the following table, where we feed features from different clip models (Event-trained or RGB-trained) to the SVD U-Net. Note that all CLIPs are fed with event voxels. Even further fine-tuning the SVD with plenty of data, the resulting diffusion model with Event-trained is still underperformed. \n\n\n**Table: Experimental results of SVD fed with different CLIP inputs, where both CLIP models are fairly perceived with event data.**\n| #Prompt | Fine-tuning | CLIP | *FVD* ↓     | *FID* ↓    | *SSIM* ↑    | *LPIPS* ↓  |\n| ------- | ----------- | ------ | ----------- | ---------- | ----------- | ---------- |\n| U(1,3)  | T           | RGB-Trained      | 1972.91     | **210.89** | 0.69513     | 0.3651     |\n| U(1,3)  | S+T         | RGB-Trained     | **1378.92** | 230.18     | **0.78496** | **0.3076** |\n| 1       | S+T         | RGB-Trained      | 1406.24     | 233.58     | 0.78374     | 0.3219     |\n| U(1,3)  | S+T         | Event-Trained      | 1646.78     | 298.88     | 0.72779     | 0.3464     |\n\nMoreover, in the future, the authors will also try to design and adapt more event-specific designs with the diffusion model, with large datasets and an extensive training process.\n\n## **Q2:** RGB-Domain Metrics \\& Raw Data Metrics\n\nThat's a good question for generative models. Here the authors merge those two questions and answer them together. The conclusion goes first that metrics on the **raw space are inappropriate for measuring and training generative models**.\n\nThe authors want to first note that perceptual metrics (or the metrics operated in low-dimensional data manifold) are essential for the generative model. In other words, the measuring and training of the generative model especially for the diffusion model must (or to be more effectively) be carried out on the data distribution manifold (or we can say in the perceptual space) [1]. The underlying reason is quite simple usually the target data distribution is not continuous on the raw space. If we utilize a metric such as MSE (as mentioned in the third question), it will make the generated data approach the GT data in the raw space. However, in the raw space, the data distributed near GT samples, e.g., blurring or noised versions of GT samples are not in the original GT data distribution. The target data distribution contains nearly all clear and meaningful samples. Thus, considering that if we can find the manifold of data, the nearby samples of a target sample must be clear event data showing the content with similar semantic meaning. Then, we should use perceptual metrics, which can act as a measurement tool for semantic and high-level meaning similarity. \n\nMoreover, just as we discussed the aforementioned question, the current RGB-based model has a strong generalization ability, which ensures there with less probability to be failure on the perception in various scenarios. Thus, the authors adopt the aforementioned FVD, FID, and LPIPS as our measure metrics. \n\nFinally, to address your concern, we also show the quantitative results for MSE and PSNR metrics in the table below. Our method also outperforms SOTA methods in terms of pixel-level metrics in the raw space.\n\n\n**Table: Quantitative comparison between SOTA methods.**\n\n| Methods   | Modal | *MSE* ↓    | *PSNR* ↑   | *FVD* ↓     | *SSIM* ↑   | *LPIPS* ↓  | *mIoU* ↑  |\n| --------- | ----- | ---------- | ---------- | ----------- | ---------- | ---------- | --------- |\n| PredRNNv2 | EVT   | 0.0306     | 15.143     | 1339.05     | 0.6598     | 0.3388     | 0.166     |\n| SimVP     | EVT   | 0.0210     | 16.778     | 1242.25     | 0.7961     | 0.3371     | 0.213     |\n| TAU       | EVT   | 0.0231     | 16.364     | 1218.03     | 0.7972     | 0.3354     | 0.228     |\n| Ours      | EVT   | **0.0170** | **17.696** | **1055.25** | **0.7998** | **0.3123** | **0.302** |\n\n[1]Song, Yang, et al. \"Consistency models.\" arXiv preprint arXiv:2303.01469 (2023).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9LFdaXDLWS", "reviewer": "Reviewer_vmNW", "summary": "The paper proposes a novel approach to integrate event-sequences with a video diffusion model for event-based future motion prediction. The authors integrate the learning capacity of video diffusion models with the rich motion information of event cameras to create a motion simulation framework and propose to align the event-sequence diffusion model with the real-world motion via a reinforcement learning process. They demonstrate the effectiveness of their method in various scenarios and highlight its potential in several downstream applications.", "questions": "See Weaknesses.", "limitations": "No", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "(1) The paper makes the first attempt to combines event-based sensors with video diffusion models, offering a unique solution for future motion prediction.\n(2) The paper provides a unique solution to align the pre-trained event-sequence diffusion model with the real-world motion via a reinforcement learning process.\n(3) The paper includes sufficient testing and validation, demonstrating the effectiveness an potential of the proposed method.", "weaknesses": "(1)\tThis paper lacks an deep analysis of the relationship between event cameras and future motion estimation tasks. If only the role of high temporal resolution is emphasized, high-speed cameras are also an alternative, of which the spatial content is richer.\n(2)\tThe proposed solution leans towards image-based techniques and lacks to utilize the characteristics of events.\n(3)\tThe rationale for incorporating reinforcement learning remains unclear to me. I hope the authors can provide a more convincing justification.\n(4)\tSome experiment settings lack explanations. For example, in Table 2, the specific processes corresponding to 'T' and 'S+T' are not clearly described. Although I can infer from the supplementary materials that they correspond to spatial and temporal attention layers, this is difficult for readers to understand in main text.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## **Q1:** Event V.S. High-Speed Camera\n\nThere are basically three reasons for event data outperforming high-speed cameras:\n\n(1) **Data characteristics.** Event cameras record only dynamic intensity changes and have extremely high-temporal resolution, which means they capture a wealth of motion information. Compared to RGB data from evenly spaced exposures and the often difficult-to-obtain optical flow data, event data has a natural advantage in the task of predicting future motion information. Moreover, due to the concise structure of data, events can be sensed in low latency, which is also a strength of event data.\n\n(2) **Measurement requirements.** For capturing high-temporal resolution motion information with a high-speed camera, we usually need to enhance the lighting condition or highlight our target, since the exposure time of a high-speed camera is very short. Otherwise, the captured images will be blurry and dark, making the content difficult to discern.\n\n(3) **Cost.** The acquisition and usage costs of high-speed cameras far exceed those of event cameras. Theoretically, the smallest time interval for data generation by an event camera is 1 $\\mu s$, which corresponds to a maximum temporal resolution of 1e6 frames per second. In practice, many studies e.g.[1,2] have confirmed that event cameras can easily achieve nearly 1,000 frames while maintaining good semantic information. To achieve the same frame rate, the computational and storage costs for high-speed cameras are significantly higher than those for event cameras. Moreover, event cameras have a larger dynamic range compared to conventional cameras, allowing them to capture object motion effectively even under poor lighting conditions, as shown in Fig. 1 of the uploaded PDF file.\n\n[1] Tulyakov, Stepan, et al. \"Time lens: Event-based video frame interpolation.\" Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2021.\n\n[2] Tulyakov, Stepan, et al. \"Time lens++: Event-based frame interpolation with parametric non-linear flow and multi-scale fusion.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.\n\n## **Q2:** Event-based Design\n\nIn fact, our proposed method takes full advantage of the characteristics of event data. Specifically, as detailed in Section 4.1 (Lines **153-165**) of the main text, under the High-Temporal Resolution Guided Sampling section, we exploit the high temporal resolution and flexibility of event data by segmenting the temporal bin into multiple sections, which containing rich motion information from the events. During the reverse diffusion process, we replace pure *Gaussian Noise* with these high-temporal resolution voxels in each denoising phase. This strategy enables us to achieve a high-temporal resolution diffusion process by utilizing these detailed event representations.\n\n## **Q3:** Reinforcement Learning Incorporation\n\nReinforcement learning is a training strategy that stabilizes and enhances the results generated by diffusion. Diffusion models are trained with decoupling steps from a continuously linked probability flow. Since the model cannot accurately estimate the score function at each step, there must be accumulated errors during the reverse diffusion steps. RL-based methods can take the accumulated error into consideration since the reward is directly modeled on the final reconstructions. Furthermore, Figure 10 in our Appendix demonstrates the effectiveness of using reinforcement learning for motion alignment. Notably, as shown in Figure 10(b), even extensively pre-trained video diffusion models can yield unstable and flawed outcomes in challenging tasks like motion prediction. However, after applying motion alignment through RL, our model delivers more stable results (Figure 10(c)), highlighting the importance of incorporating RL-based motion alignment.\n\n## **Q4:** Experiment Settings\n\n Yes, the “T” and \"S\" indeed represent the temporal and spatial layers. We have added relevant annotations to the table. We will carefully review the figures and tables in the main text and make corrections in subsequent versions. Thank you for pointing this out.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
## **General Response**

We thank all reviewers for your time, constructive feedback, and acknowledgment of our work. We believe all concerns have been clearly and directly addressed. Here, we also want to summarize a few key clarifications concerning the contributions of our work.

Our **major** contribution lies in the pioneering integration of event sequence data with video diffusion, i.e., utilizing the high temporal resolution of event data to accurately predict future object motions in various scenarios. 

Specifically, we transform events into 3-channel voxels and fine-tune the spatiotemporal cross-attention layers of the U-net in SVD. During the denoising phase, to fully leverage the high temporal resolution characteristics of event data, we sample events into sub-streams and prompt multiple event voxels to SVD. Compared to the input of a single RGB frame for the original SVD, our method effectively utilizes the motion priors present in event data, leading to more accurate future motion estimation. Furthermore, we introduce motion alignment using reinforcement learning to enhance the stability of both the diffusion training process and the estimations. 

Figures 1, 2, and 4 in the uploaded PDF file, as well as the table in the response to **Reviewer 8wZD**, demonstrate the excellent generalization capability of our method across various scenarios and datasets. 
Additionally, ablation studies in Table 3 of the main paper and the visual results in Figure 10 in the Appendix further substantiate the necessity of reinforcement learning. For motion estimation tasks that demand high accuracy, the absence of reinforcement learning for aligning with real motion leads to unstable and prone-to-failure diffusion-generated results.

We posit that our contributions will pave the way for advancing event-based diffusion and future motion estimation fields. Our method improves the SOTA performance of event-based future motion estimation to a higher level, providing a promising benchmark for this community.

Last but not least, we will make the reviews and author discussion public regardless of the final decision. Besides, we will include the newly added experiments and analysis in the final manuscript/supplementary material.

Meta-review: The paper received mixed reviews, leaning towards acceptance. Reviewers commended the importance and originality of the problem addressed, as well as the extensive experimental evaluation and the high quality of the presentation. However, they also noted concerns regarding the method's generalization abilities and the absence of certain details and discussions. In their rebuttal, the authors thoroughly addressed all these concerns. After considering the paper, the reviews, the rebuttal, and subsequent discussions, the area chair decided to accept the paper. The authors are encouraged to incorporate the additional experiments and discussions from the rebuttal into the camera-ready version of the manuscript.
Decision: Accept (poster)