V2V: Scaling Event-Based Vision through Efficient Video-to-Voxel Simulation
Authors: Hanyue Lou
Jinxiu Liang
Minggui Teng
Yi Wang
Boxin Shi
Venue: NeurIPS 2025 poster ()
Keywords: event-based vision
Primary area: applications
TL;DR: Our V2V module converts videos to voxels efficiently at train time so we can scale up datasets to train better E2VID & EvFlow models.

Abstract: Event-based cameras offer unique advantages such as high temporal resolution, high dynamic range, and low power consumption. However, the massive storage requirements and I/O burdens of existing synthetic data generation pipelines and the scarcity of real data prevent event-based training datasets from scaling up, limiting the development and generalization capabilities of event vision models. To address this challenge, we introduce Video-to-Voxel (V2V), an approach that directly converts conventional video frames into event-based voxel grid representations, bypassing the storage-intensive event stream generation entirely. V2V enables a 150× reduction in storage requirements while supporting on-the-fly parameter randomization for enhanced model robustness. Leveraging this efficiency, we train several video reconstruction and optical flow estimation model architectures on 10,000 diverse videos totaling 52 hours—an order of magnitude larger than existing event datasets, yielding substantial improvements.

V2V: Scaling Event-Based Vision through Efficient Video-to-Voxx Simulation
Hanyue Lou .2 Jinxiu Liang² Minggui Teng'
Yi Wang Boxin Shi!
1 Peking University Shanghai Innovation Institute
National Institute Informatics Shanghai Al Laboratory fhylz, minggui _enng, shiboxin){pku. edu. cn
casherryliangggmail. com com rangyi&pjlab.om org. cn
Abstract
Event-based cameras offer unique advantages such high temporal resolution, high dynamic range, and low power consumption. However, the massive storage requirements and I/O burdens of existing synthetic data generation pipelines and the scarcity of real data prevent event-based training datasets from scaling up, limiting the development and generalization capabilities of event vision models. To address this challenge, we introduce Video-to- Voxel (V2V), an approach that directly converts conventional video frames into event-based voxel grid representations, bypassing the storage-intensive event stream generation entirely. V2V enables a 150x reduction in storage requirements while supporting on-the-fly fly parameter randomization for enhanced model robustness. Leveraging this efficiency, we train several video reconstruction and optical flow estimation model architectures on 10,000 diverse videos totaling 52 hours an order of magnitude larger than existing event datasets, yielding substantial improvements.
Introduction
Event-based cameras are bio-inspired visual sensors that asynchronously record per-pixel intensity changes, offering high temporal resolution and high dynamic range with low power consumption, compared traditional frame-based cameras. These properties make event cameras particularly promising for applications involving high-speed motion, challenging lighting conditions. and resource- constrained environments such as robotics, autonomous driving, and augmented reality.
Despite their theoretical advantages, event cameras have yet to match the practical performance of conventional cameras. This performance gap can be attributed to several factors, with the scarcity of large-scale training data constituting a critical bottleneck. While deep learning approaches have revolutionized frame-based computer vision through decades of dataset collection and massive repositories like ImageNet [4. COCO [19] and WebVid [2], event camera data remains scarce and limited in diversity. The restricted commercial deployment of event cameras creates fundamental chicken-ande -ndeenn-n-egg problem: widespread adoption requires robust algorithms, which in turn depend on diverse training data that can only come from widespread deployment.
To address this data limitation, researchers have developed simulation pipelines that convert conven- tional visuals into synthetic event ones. However, significant challenges remain in both approximating the high temporal dynamics of event cameras and utilizing these simulations efficiently.
Standard video datasets typically have frame rates of around 30 per second (FPS), providing temporal resolution orders of magnitude lower than event cameras. This fundamental mismatch creates a
Hanyue Lou, Minggui Teng and Boxin Shi are with the State Key Laboratory Multimedia Information Processing and the National Engineering Research Center of Visual Technology, School of Computer Science, Peking University. Corresponding author. Code available:
com/HYLZ 2019/V2V
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Data Preparation
Video-to-Event Event-to-Voxel
Training
Avg. SSIM
V2V-E2VID
V2V- HyperEzVID
280 Scenes
Events
Voxels
E2VID+ (Baseline)
Random each scene
279 GB Disk I/
0.66
W C+, C_
Stack
(per epoch)
Samples Variant
0.64
V2V-ETNet
Synthetic
98 GB
279 GB
0.62
10000 Videos
Voxels
V2V-E2VID (Ours)
20 GB Disk I/O
Random each iteration
0.60
(per epoch)
C4, c_
0.58
ETNet
Hyper-
22K Samples
E2VID+
E2VID
Realistic
Video-to-Voxel
X oo Variants
0.56
Figure I: The Video-to- Voxel (V2V) approach enables our training dataset to have more diversity (280->1000) scenes), less storage usage and I/O load (279 GB GB), and more randomly augmented sample variants. V2V-retrained models E2VID 33], ETNet [36] and HyperE2VID [ demonstrate better reconstruction quality, indicated by higher SSIM (structural similarity, averaged over HQF 333 and EVAID (6) on real-workd test datasets.
significant hurdle for data simulation. The field has consequently developed two primary simulation approaches, each with distinct limitations: Video-based simulators like V2E [3] attempt to bridge this temporal gap through frame interpolation techniques; however, they inevitably introduce artifacts and interpolation errors since they attempt to reconstruct information that was never captured in the original footage. Model-based simulators like those in E2VID+ [33] render high frame rate videos from 3D scene models, providing precise control of the temporal resolution but suffering from significant realism limitations. In practice, these methods often resort to simplistic approximations, such randomly flying 2D images against static backgrounds far removed from weal-world scene complexity, illustrated in topleft of Figure
Beyond realism concerns, both approaches encounter severe storage and computational inefficiency. Unlike conventional video data, which benefits from decades of codec optimization, event data compression remains relatively underdeveloped [14 31. Consequently, storing and processing synthetic event data becomes exceedingly resource-intensive. For context, single 10-second sequence from the ESIM-280 dataset occupies approximately 1GB. If we were to scale to 10,000 diverse scenes, substantial I/O bottlenecks during training will emerge. Moreover, since each event stream corresponds to a specific camera configuration, exploring multiple parameter variations (thresholds, noise levels, etc.) multiplies storage requirements by the number configurations.
These technical challenges create a significant barrier to training on large-scale, diverse datasets, leading question: Is explicit event stream synthesis truly necessary for model training?
Our analysis reveals a surprising answer: No. We observe that most event-based deep learning pipelines convert asynchronous events into dense representations (particularly voxel grids) before neural network processing. This creates an opportunity to directly target these representations instead of generating intermediate event streams.
In this paper, we introduce Video-to-Voxel (V2V), principled approximation that efficiently gener- ates discrete voxel representations directly from video frames. This approach bypasses the computa- tionally expensive event generation step while preserving the essential spatial-temporal information needed for learning. The key advantages of our approach are illustrated in Figure
Efficiency. Our approach eliminates the need to store intermediate event streams, drastically saving storage resources (up to 150 times) and enabling the use of substantially larger datasets. Flexibility. Our approach allows on on-the-fly parameter randomization during training, enabling models to learn more robust representations from diverse virtual camera configurations without multiplicative storage requirements.
Scalability. This efficiency enables direct utilization of massive internet-scale video repositories like WebVid (2) creating a training dataset containing 10,000 videos with total duration of 52 hours-an an order of magnitude larger than existing event camera datasets.

---

We validate the effectiveness of the V2V module on two fundamental event-based visual dense prediction tasks: event based video reconstruction and optical flow estimation. particular, our method enables the training of existing models on diverse dataset with 35 times more scene variations than the previous practice, with performance scaling with dataset size. As shown in the right of Figure | this leads to significant improvements in reconstruction quality. These results demonstrate that our V2V pipeline effectively addresses the data scarcity bottleneck in event-based vision, opening new possibilities for developing robust event-based algorithms.
2 Related Works
Synthetic event data. Video-based and model-based event simulators have been proposed to generate synthetic event data. Video-based simulators [33 18 41 44 46] synthesize events from videos, using frame interpolation algorithms [33 or learning-based methods 41 44] to compensate the lack of temporal information due to limited video frame rate. This requries reconstructing intermediate intensity changes that were never captured- an ill-posed problem. Model-based event simulators such as ESIM (28) and PECS [12] render events from synthetic 3D scenes, providing fully precise temporal information. However, the diversity and realism of the scenes are limited. Random scenes with flying 2D
335 objects and elaborate models of city scenes [22] have been used, but complex real-world phenomena, such as with non-rigid movement, fluid dynamics, and complex lighting interactions, remain difficult to model.
33 or 3D
The development of video reconstruc- tion from event streams (E2VID) has been predominantly constrained by available training data. Existing learning-basel E2VID methods use model-based simulated events real events for train- ing. E2VID [29] proposed to use the ESIM simulator for training data generation, and Stoffregen et al. 33 improved model performance by using diversified thresholds in ESIM. FireNet [30], SPADE PPADE-E2VID [3), ETNet (36), Event-Diffusion [17]. HyperE2VID [7.. EVSNN (45) and TFC- SNN 38 all used ESIM data for training. Mostafavi al. [24) used mixture of ESIM data and real data. Gu et al. (IT] used the vidco-based simulator V2E [3], but the high frame rate video inputs were from synthesized scenes with flying images. SSL-E2VID (77) proposed self-supervised training framework so that the model could be trained on real event streams without corresponding ground truth images. The low light methods DVS-Dark (40) NER-Net [20). and NER-Net+ (21
Training data of event-based video reconstruction.
utilize real-world low-light events for domain adaptation.
Training data of event-based optical flow estimation. Event-based optical flow estimation has explored marginally more diverse training strategies but encounters similar limitations. EVFlow 42 trained on FSIM-generated data, while EVFloo++ 3K is trained on ESIM with diversified thresholds.
ADMFlow [2 utilizes a synthetic dataset MDR, rendered by browsing cameras through 53 static virtual 3D scenes and using the V2E [13 toolbox. E-FlowFormer (16) is trained on BlinkFlow, which is composed of 3362 rendered scenes of random moving objects. Zhu et al trained their model on the
real MVSEC dataset 43 with an unsupervised loss, while FloweeFFloNeet [15] and STE-FlowNet (5) used it in a sulf-supervised way. ERAFT [10] is trained on DSEC-Flow, real dataset from 24 DSEC 9 sequences. EEMFlow+ 23 is trained on HREM, real event dataset with 100 scenes.
3 Method
In this section, we will introduce our V2V framework, which utilizes large-scale video datasets for training. Section 3.1] we introduce preliminaries and emphasize the importance of data scalability. In Section 3.2. reveal that intra-bin temporal information can be neglected in training with the discrete voxel representation, greatly improving efficiency. Finally, in Section [33]] we show that on-the-fly video-to- voxel generation provides additional flexibility that improves data diversity.
3.1 Preliminaries
Formulation of event generation model. Event cameras are bio-inspired vision sensors that operate on fundamentally different principles than conventional frame-based cameras. Instead of capturing intensity frames fixed time intervals, event cameras asynchronously report changes in logarithmic brightness at the pixel level. In an ideal event camera, pixel at (z, y) triggers an event
3

---

Different ((ta) estimations
Different M selections
Events fx)
F(5.)
F(,))
F(r.)
Figure 2: Event data's inherent ambiguity: multiple valid image sequences / (fo) can produce identical event streams & under different initial conditions and camera parameters M. indicates the critical importance of diverse training data to establish robust priors for reconstruction tasks.
signal (a, y, p) whenever the change of its logarithmic irradiance exceeds a threshold c:
log (1,)(() - log (z,)( 2
where (t) indicates the pixel's irradiance at time t, and to is the timestamp of the previous event triggered from pixel (z, y). The polarity p e [-1,+1} +1} represents whether the irradiance has decreased increased.
The output of an event camera stream of asynchronous events &, capturing the change of the scene. By summing the positive and negative events triggered between t and t2 on each pixel into an event stack E(t,, t2), we can acquire the relationship between irradiance levels [(t,) and (12))
log [ (12) log [(t)) = c* E(t1,t)),
where (4,
E
P.
(
The inherent ambiguity in event-based vision tasks. This relationship highlights a fundamental challenge in event-based vision which has been widely discussed [29] 34]: events only provide constraints on relative changes in logarithmic irradiance, not absolute intensity values. For example, in the task of event-to-video reconstruction, model is trained to reconstruct series of image frames F(tz)..... Ftt,) based on an event stream This means that it must estimate both an initial irradiance [(to) and mapping M from irradiance [tt,) to pixel values F(t,) that would produce high-quality frames, which are related to camera parameters such as aperture size, exposure time and image signal processer (ISP) configuration interpretations of the same event stream exist, corresponding to different scene priors and camera parameters. To provide models with high-quality prior knowledge, large-scale datasets that resemble real-world distributions are essential.
As demonstrated in Figure 2 multiple valid
However, current model-based simulators cannot achieve scale, diversity, and quality simultaneously, as realistic models are prohibitively expensive to design and simple scenes fall short of real- world complexity. Utilizing existing video datasets offers promising alternative, but requires addressing the significant temporal resolution gap between conventional videos (typically 30 FPS) and event data (microsecond resolution). Our method bridges this gap through novel representation approach that enables effective learning from standard video data while preserving compatibility with real event camera outputs.
3.2 Bridging the temporal resolution gap
The fundamental challenge utilizing conventional videos for event-based learning lies the vast difference in temporal resolution. Event streams operate at microsecond temporal precision, while typical videos provide only 30-60 frames per second. Although frame interpolation algorithms could theoretically increase frame rates, interpolating to event-level frame rates would be computationally prohibitive and introduce compounding artifacts such as ghosting effects [37]. More importantly, such interpolation attempts to reconstruct temporal information that never captured in the original footage, resulting in significant fidelity loss.
Our approach tackles this challenge by fundamentally reconsidering how event data is represented and processed in deep learning pipelines. For deep learning approaches, the sparse, asynchronous nature of event streams necessitates conversion into dense representations that can be fed into conventional neural networks. Among these representations, voxel grids have emerged as the most widely used format. In voxel representations, events are first separated into temporal bins, and each bin of events

---

Low temporal resolution WVideo-based simulation)
Interpolated Event Voxels
Discrete Event Voxels
Different appearance
S imilar appearance
High temporal resolution (Real events)
While synthetic (top) and real (bottom) events exhibit notable differences in interpolated voxels (left) due to microsecond-leww temporal disparities, they demonstrate similarity in discrete voxel representations (ight)-justifynne
Figure 3:
direct video-to-voxel conversion approach.
encoded into frame through some form of accumulation. These frames are then stacked into different channels of the voxel corresponding to their sequence. This organization creates a natural separation between inter-bin temporal information (the relationship between separate bins) and intra- bin temporal information (the precise timing of events within each bin). This separation is crucial to our approach, as it allows us to discard the high-precision intra-bin timing information woocht cannot be reliably simulated from low frame rate videos-we while preserving the essential inter-bin dynamics that can be accurately derived from frame differences.
Interpolated vs. discrete voxel grids. The most commonly used voxel representation is the interpolated event voxel. An interpolated event voxel Vimp with B bins has shape B H W. For each event (t, x. P) timestamp (normalized to [0, I)):
()) E &, its polarity is linearly split to the closest two bins according to its
Vintp (b, 16,,,,0 x,
Σ (R.y..p)EE
p max
b)).
The interpolated event voxel representation is widely used in event-based video reconstruction algorithms [3] 17
27
33 36) and optical flow estimation algorithms [10 22 33 However, needs. The intra-bin temporal information still takes significant effect: as
29
it does not meet the timstamp of an event changes, its weights on the two neighbouring bins continuously changes, resulting in smooth smudge-like like edges in voxels. When the intra-br bin temporal information is absent, the discreteness of the timestamps creates layered-like visual effects in the voxels. The difference is depicted in the left wing of Figure 3
The discrete event voxel, however, meets our needs. It another variant of the voxel representation, also referred to as SBT (24). Each bin in B x H x W discrete event voxel Viise is the summation of events with timestamps between b/ B and (b + 1)/B:
Vaiso (6, (b)
Σ (K.9A--)EE
1
b
b+1 1
In the discrete event voxel, all intra-bin information is discarded, and temporal information is only encoded in inter-bin information. As shown the right side of Figure 3 the discrete voxel represen- tations of video-basee simulated events show similar appearances to real ones with high temporal resolution. One may worry that discarding intra-bin information would hurt the expressiveness of the input data, but our experimental results (m) in Table 2 show that models operating on the interpolated and discrete representations can achieve comparable performance.
For discrete event voxel with B bins, we can efficiently simulate it with just B + 1 video frames F(lo), F(I1)... Flnn). as detailed in the next section. the input video has a frame rate X FPS, then each created voxel will cover physical time of B/X seconds. A potential concern is that the physical time span of real event voxels may much shorter when we want to exploit its high temporal resolution. However, due to the large scale and high diversity of our video datascts,
5

---

V2V Module
V2V-E2VID
V2V-EvFlow
Input Videos
log
log
F(ta
Fl
P
MCG+)
M(G
V2V
V2V
C_
Random Parameters
Al.(s))
| [+c,]]
[+c_]
| c.l
[+c
EvFlow
Output
Voxels
(1)
(2)
LPIPS
Data Flow.
Loss Constraints.
L1: Loss. LPIPS LPIPS Loss. TC: Temporal Consistency Loss.
Figure 4: The V2V module efficiently converts input videos to output voxels with random parameters selected at train time (left). We use the V2V module and lightweight optical flow estimator RAFT to train video reconstruction (middle) and optical flow estimation models (right).
find that our models generalize well: although trained 24/5 FPS synthetic voxels, they achieve superior performance in reconstructing 150 FPS videos from real EVAID events (6]
This representation choice is the cornerstone of our approach, enabling to significantly reduce storage requirements and improve training efficiency by leveraging advanced video compression and decoding techniques.
3.3 Directly converting videos voxels
There exist many simulation algorithms that convert videos to events, such EventGAN V2E [3), DVS-Voltmeter [18] and V2CE [41]. They are usually used in a two-stage pipeline. In the first stage, event camera parameters such thresholds and noise strength are selected. The photoreceptor output voltage of the event sensor is simulated using the logarithm difference between video frames as well random noise. The number of events to triggered on each pixel, equivalent to the values of the corresponding discrete voxel, is calculated according to the thresholds. Event streams then generated by deciding timestamp for each event triggered. In the second stage, events are accumulated into dense representations such as voxels. Additional augmentation such as hot pixels and gaussian noise can be added to the voxels on-the-fly when training, but with less fidelity. For example, in real event cameras, when background noise triggers an event, the brightness change detector is reset, effecting future pixel behaviour, while post-voxe! augmentation noise is independent to the signals.
44]
Our Video-to-Voxel (V2V) approach skips the generation of streams, merging both stages into an efficient on-the-fly process conducted in each training iteration. As a result, the camera parameters single video can be randomly selected in each iteration,
and high-fidelity noise corresponding providing far more sample diversity and improving model robustness. The V2V conversion process conducted in the following steps.
Parameter selection. In each V2V iteration, we first randomly select the event camera parameters: the positive threshold C+ 0, the negative threshold c_ > O, background noise strength d and a hot pixel map H E R W The background noise corresponds to the dark current in the photodiode of event pixels, and the hot pixel map H is a sparse matrix where a small number of randomly selected "hot pixels" have abnormally large values and the other elements are zero [I3].
Initialization. Then we begin the simulation. We use AL to denote the sensor's photoreceptor output voltage, corresponding to the amount of logarithm illumination change on each pixel since their last previous triggered event. The initial value (L(()) is initialized by sampling from the uniform distribution [-c_, C+J., assuming that the "sensor" has already been recording for while.
Sensor simulation. We process the frames F(l)), F ((.),.... F(In) one by one. Given the frame F(I.), we first apply a reverse gamma correction to make approximately linear to T(4,), and calculate its logarithm difference the previous frame log [tt,) - log [(t,-1). Following DVS-

---

Voltmeter [18], which models sensor voltage change as Brownian motion process caused by photon reception randomness, add background noise independently sampled from gaussian distribution (0, of)) for each frame. We also randomly generate hot pixel map H that keeps the same throughout the sequence. For simplicity, we did not include other noise types like leak noise, but they could be easily incorporated into our framework. We add up the signal and noise to acquire the total output voltage:
(T = (14 + (log (t4) log ((44-1)) + N(4) H, N(4) ~N(0 (5)
Then, we calculate the number of positive and negative events triggered, denoted as N+ and N_, and subtract these already- triigeerd changes to prepare for the next imput frame:
(1) max (o,
1) |
C+ (1AA()))
N max (o, (o(1-L(H))
(6)
Ct Ai((.) - c
(7)
Note that, unlike post-voxel augmentation noise, noise added in the V2V process can play an interactive part in the event simulation, which has the potential to lead to higher data fidelity.
Voxel calculation. Finally, we can compute the corresponding discrete event voxel bin:
Vdise (i) ( (i).).
(8)
When c+ and c. are equal, or when the pixel intensity change between two frames monotonic, the equality holds strictly.
Applications. As illustrated in Figure 4 apply the v2V module the video reconstruction (E2VID) and optical flow estimation (EvFlow) tasks by using them to generate voxels from video- based datasets. We predict optical flow used for training from the video frames, using image-based algorithms such as RAFT [32). The V2V module enables us to train efficiently on large-scale diverse datasets with flexible augmentation, boosting the performance of the models is applied to.
Experiments
Dataset
Scenes
Duration
Resolution
Segs
Space
SSIM
Avg.
V2V-Random
Webvid-10K
V2V-Fixed
ESIM-Event
280
47 min
256 256
3421
97.77 GB
0.66
ESIM
FSSM- Voxel
min
256 256
3421
279.00 GB
WebV d-1K
V00
100
33 min
180 x 596
253
0.21 GB
WibVidlk
1000
319 min
180 x 596
2332
1.94 GB
WebVid1Ok
10000
52 hours
180 x 596
22725
19.14 GB
0.62
WebVid 100
URR 25
27
18 min
240 x 180
638
32.2 GB
0.60
MVSEC 43
14
68 min
346 x 260
9063
109.30 GB
ESIM-280
HQF 335
14
11 min
240 180
385
2.74 GB
Scenes
EVAID-R
6
min
954
1350
7.4 GB
100
280
1000
10000
Table 1: Comparison of dataset characteristics. "Seqs" for event datasets represents total frames divided by 40, providing normalized comparison metric.
Figure Effectiveness of proposed parameter randomization across dataset sizes.
4.1 Dataset preperation
We use the WebVid 2 video dataset for model training due to its high aesthetic quality and clean shot cuts. From the 2.5M videos of WebVid we randomly sample 10K, 1K and 100 videos, forming datasets Vid WebVidIK and VedVid In Table 2 we use "V2V-X" represent
V2V framework combined with the WebVid dataset.
The original resolution of WebVid videos is 336 596, and we only use the upper 180 x 596 to avoid the watermarks. For each WebVid training sequence, we use 200 ove-ovelggp frames to create 40 voxels, each with 5 event bins. The average Web video contributes 2 samples.
7

---

We prepare the ESIM-280 dataset using the same configurations as Stoffregen et al. [33], generating 280 scenes composed of 2D flying objects along with camera movement. We also sample non- overlapping sequences with 40 event voxels and ground truth frames, acquiring 3421 sequences ("Seqs" in Table [ with 256 x 256 resolution in total. We refer to the original dataset as ESIM- Events, and the pre-stacked dataset with voxels (in float32 format) instead of events is referred to as ESIM-VoxeL..
The statistics of our WebVid datasets and the ESIM datasets are listed in Table Due to the compact encoding of videos, WebViK10K only uses 6.9% disk space while providing 6.6 more sequences and 35.7x x more scene diversity compared to ESIM- Voxel. It also provides rich real-world priors with its realistic scenes, diverse viewing angles and non-rigid movement such as human motion.
Table 2: Quantitative evaluation of event-based video reconstruction results. Lower (L) MSE and LPIPS values and higher (1) SSIM values are desirable. For each model type, we highlight the best values with green Experiments marked with use the original model weights directly downloaded.
HQF
EVAID
1dx Model
Train Dataset Loss
Eps
MSE
SSIM
LPIPS
MSE
SSIM
LPIPS
Results from different training dataset sizes
E2VID
ESIM-280 &
Alex+TC
500
0.037
0.638
0.256
0.088
0.526
0.469
(b)
E2VID
V2V-100
V+L1+H
8000
0.042
0.626
0.327
0.078
0,594
0.441
(c)
E2VID
V2V-IK
V+L1+H
800
0.037
0,661
0.286
0.072
0.627
0.405
(d)
E2VID
V2V-10K
V+L1+H
80
0.032
0.676
0.265
0.054
0.663
0.371
Results with other model structures
(e)
HyperE2VID
ESIM-280 &
Alex+TC
400
0.032
0.646
0.265
0.071
0.533
0.474
HyperE2VID
V2V-10K
V+LI+H
60
0.035
0.671
0.269
0.055
0.654
0.377
(g) ETNet
ESIM-280 1
Alex+TC
700
0.035
0.642
0.274
0.080
0.541
0.523
(h)
ETNet
V2V-10K
V+L1+H
100
0.039
0.641
0.306
0.056
0.610
0.406
Ablation study: Effect of augmentation flexibility
E2VID
V2V-100-F
V+L1+H
8000
0.049
0.611
0.330
0.073
0.604
0.451
E2VID
V2V-1K-F
V+L1+H
800
0.039
0.640
0.293
0.066
0.617
0.413
(k) E2VID
V2V-10K-F
V+L1+H
80
0.032
0.666
0.264
0.058
0.644
0.378
Ablation study: Interpolated voxel vs. Discrete voxel
E2VIDD(nnp.)
ESIM-280
Alex+TC
500
0.041
0.599
0.306
0.078
0.526
0.478
(m) E2VID(Dise.)
ESIM-280
Alex+TC
500
0.040
0.608
0.296
0.062
0.548
0.450
4.2 Video reconstruction
We retrained the models E2VID 29 ETNet (36] and HyperE2VID 77] on WebVid and ESIM datasets. Then we performed zero-shot model evaluation on the real event datasets HQF [3] and EVAID (6,, with metrics MSE (Mean Square Error), SSIM (Structural Similarity) and LPIPS (Learned Perceptual Image Patch Similarity). Quantitative results (a)-(h in Table[aand qualitative comparisons (Figure 6) show that models trained with the V2V. Web dataset can outperform the original ESIM versions.
Effect of scaling up. To test the effect of scaling up the dataset, we retrained the E2VID model with datasets of different sizes: WebVid- WebVid-1K K and Web 10K. As shown rows (b)-(d) in Table Table2[and and Figure 5] larger datasets produce models with better performance.
Effect of augmentation flexibility. A key advantage of our V2V module is that the same N scenes can be augmented with different threshold configurations in each iteration, yielding N x M sample variants over M training epochs. To validate the importance this feature by ablation, we designed an experiment where each video in the Web Vid dataset was pre-assigned fixed threshold value, meaning that the model could only access N x sample variants across M epochs. In rows (i)-(k) of Table 2 we denote these processed datasets with "-F" (Fixed). The results are visualized in Figure
8

---

and show that disabling the augmentation flexibility causes performance drop, proving that providing different variants in each iteration acts as an effective data augmentatiin technique.
Perceptual loss design. The loss functions of the baseline E 33 is composed of LPIPS loss (from the Alex model) and temporal consistency loss (from ground truth optical flow). We refer to this combination as "Alex+TC". We observe that using the Alex LPIPS loss causes patterned artifacts in the reconstructed videos, while using a VGG version of the LPIPS loss does not. Since VGG is more inclined to supervise high-level features [39], we combine an L1 loss with the VGG LPIPS loss to supplement the supervision of low-level features, Ablation studies can be found in the appendix (Section D
Temporal consistency loss design. Due to the lack of ground truth optical flow from videos, we use the RAFT-Small [32] model to predict optical flow between frames at train time. We observe that adding the temporal consistency loss eliminates flickering in the generated videos, but also introduces dirty-window artefacts: static undesired patterns float on top of the videos, as if filming through dirty window, likely because the model tries to maintain consistency even for unwanted artifacts from previous frames. To mitigate this, only apply temporal consistency loss to the latter half (last 20 frames of a 40- 30-frame sequence), so consistency is only enforced after the reconstruction has had time to stabilize. In Table as "V+LI+HH"
2 we refer to the loss combination of "VGG + L1 +
Conisrstensssteey-eeee "alff"
Ablation on voxel representation. To show that using discrete voxels does not harm performance, we retrained E2VID with the same loss and the same ESIM-280 dataset, but with the interpolated event voxel (Intp.) and the discrete event voxel (Disc.) input representations respectively. Results (I)(mm) in Table show that discrete voxels work comparably to interpolated voxels.
show
Training details. We train the E2VID model with batch size 12, crop size 128 x 128 and constant learning rate 0.0001. For ETNet and HVPD, we also follow their original training protocols. The training epochs ("Eps" in Table 2 are set to keep the total amount of iterations approximately the same across experiments.
V2V parameters. For the event threshold parameters c4 and used in the V2V simulator, we first uniformly sample a threshold e from the range [0.05, 2]. (Following the ESIM simulator, we calculate logarithm images with np. 1og (0. 001 + video/255. 0), so the amount of change between two frames is in the range of [0, 6.908J.) To keep the positive and negative thresholds close, we uniformly sample "threshold ratio" from the range [1, 1.5), assigning (c, rc) to (c+, or (c_, c+) with equal probability. The standard deviation of the background gaussian distribution is uniformly sampled from the range [0, 0.1]. The standard deviation of the hot pixel gaussian distribution is uniformly sampled from the range [o, 10). The fraction of hot pixels is uniformly sampled from the range [o, 0.001] (i.e., 0 to 1)).
More evaluation details (Section A). test results on IJRR 25 and MVSEC 43 (Section B and more ablation studies (Section D are provided in the appendix.
4.3 Optical flow estimation
To demonstrate that the V2V framework can adapt to various tasks, we also applied it to event-based optical flow estimation. Based on the model design EvFlow 42), we trained with the WebVid dataset, using optical flow predicted with RAFT-Large from images as pseudo ground truth. We evaluated the models quantitatively on the MVSEC datasct (see Section A for details) with the dt = 1 [42] setting. For metrics we used the Average Endpoint Error (AEE) and the percentage of pixels with more than 3 Pixel Error (3PE). We provide dense (D) metrics which were averaged over all pixels, and sparse (S) metrics which were calculated over pixels that triggered at least one event. We also show qualitative results of our model on IJRR, HQF and EVAID in Figure 17
In Table Zeros refers to the baseline metrics acquired by predicting zero optical flow. EvFlow+ 33 trained on the ESIM+ dataset. Although EvFlow+ produces dense results, it often predicts hollows with zero flow where events are sparse. Boosted with the real world priors from the WebVid10K dataset, V2V-ExFlow is able to fill in the hollows with reasonable flow predictions. V2V-ExFlow exceeds EvFlow+ on all metrics, especially dense metrics.

---

Events
E
V2V-E2VID (100)
V2V-E2VID (1K)
V2V-E2VID (10K)
Ground Truth
ETNet
V2V-ETNet
HyperE2VID
V2V-HyperE2VID
Figure 6: Qualitative comparison of reconstruction results on the EVAID Traffic sequence. Our
method maintains information from camera motion occurred previously, even after it ceases.
Table Optical flow results on the MVSEC dataset. Lower (L) values of AEE and 3PE are desirable, dense (D) or sparse (S). We highlight the best values of each metric with green
Sequence
indoor ffyingg
indoor_flyie flying2
indoor indoor_flying3
outdoor_de __da_
outdoor_day: day2
AEE
4
D
S
D
S
S
D
D
Zeros
1.772
2.041
2.705
3.463
2.412
2.772
3.985
3.585
2.793
2.562
EvFlow+
0.967
0.752
1.154
0.990
1.184
0.833
2.903
1,445
1.983
1.580
V2V-EvFlow
0.732
0.745
0.870
0.990
0.741
0.758
2.114
1.191
1.627
1.365
3PE (%6 &
D
S
D
S
S
D
D
Zeros
14.5
20.2
36.1
52.5
30.2
38.6
57.1
54.5
32.9
31.7
EyFlowt+
2.7
5.4
2.4
1.6
36.1
12.1
19.8
14.4
V2V-EvFlow
0.5
2.3
0.6
23.6
8.4
15.0
Limitations
Computation resources. Since all training data stored in video format, video decoding is required throughout the training process. Video decoding algorithms are quite CPU-intensive; when accelerated with hardware, they also consume GPUs. In our experiments, the CPU capabilities of our systems were sufficient, and the real efficiency bottleneck was successfully pushed to the 100%-utilized GPU. However, this may be problem on systems with other CPU-GPU combinations.
Event representations. As analyzed in Section 3 the video-to-voxel module is only applicable when the input repressentatio of neural network is that discards all intra-bin temporal informa- tion, such as the discrete voxel. Although the discrete voxel representation can be applied to large range of event-based vision tasks, there still scenarios when non-applicable representations essential, especially if the model to train is unconventional (e.g., spiking neural networks).
Conclusion
We introduce the V2V module, which directly converts videos voxels and significantly reduces the storage and data transfer costs of synthetic event data while providing more variant training samples. Empowered by the V2V module, we scale up training datasets for video reconstruction and optical flow estimation, boosting existing models to exhibit better performance. The V2V module has broad application potential across more model architectures and event-based tasks, which remains to be explored in the future.
Acknowledgments
This work was supported by National Natural Science Foundation of China (Grant No. 62088102 62302019, 621 62136001), Beijing Natural Science Foundation (Grant No. L.233024), and Beijing Municipal Science & Technology Commission, Administrative Commission of Zhongguancun Science Park (Grant No. Z241 2241100003524012,
10

---

References
[1] Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenhin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang. Zhaohai Li, Jianqiang Wan, Pengfei Wang. Wei Ding, Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang. Tianbao Xie, Zesen Cheng, Hang Zhang, Zhibo Yang, Haiyang Xu, and Junyang Lin. Qwen2.5-vl technical report. arXiv preprint arXiv: 13923, 2025.
[2] Max Bain, Arsha Nagrani, Güi Varol, and Andrew Zisserman. Frozen in time: A joint video and image encoder for end-to-end retrieval. In Proc. of IEEE/CVF International Conference on Computer Vision (ICCV), 2021.
[3] Pablo Rodrigo Gantier Cadena, Yeqiang Qian, Chunxiang Wang, and Ming Yang. SPADE-E2VID Spatially- denormalization for event- basdd video reconstruction. IEEE Transactions on Image Processing (TTP), 2021.
[4] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. ImageNet: large-scale hicrarchical image database. In Proc. of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2009.
[5] Ziluo Ding, Rui Zhao, Jiyuan Zhang, Tianxiao Gao, Ruiqin Xiong, Zhaofei Yu, and Tiejun Huang. Spatio- temporal recurrent networks for event-based optical flow estimation. In Proc. of Association for the Advancement of Artificial Intelligence (AAAI), 2022.
[6] Peiqi Duan, Boyu Li, Yixin Yang, Hanyue Lou, Minggui Teng, Yi Ma, and Boxin Shi. EventAid: Benchmarking event-aided image/video enhancement algorithms with real-captured hybrid dataset. IEEE Transactions on Puttern Analysis and Machine Intelligence (TPAMI), 2025.
[7] Burak Ercan, Onur Eker, Canberk Saglam, Aykut Erdem, and Erkut Erdem. HyperE2VID: Improving event-based video reconstruction via hypernetworks. IEEE Transactions Image Processing (TIP), 2024.
[8] Guillermo Gallego, Tobi Delbruck, Garrick Orchard, Chiara Bartolozzi, Brian Taba, Andrea Censi, Stefan Leutenegger, Andrew Davison, Jorg Conradt, Kostas Danilidis, and Davide Scaramuzza, Event-based vision: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), 2022.
[9] Mathias Gehrig, Willem Aarents, Daniel Gehrig, and Davide Scaramuzza. DSFC: A stereo event camera dataset for driving scenarios. In IEEF Robotics and Automation Letters (RAI.), 2021.
[10] Mathias Gehrig, Mario Millhausler, Daniel Gehrig, and Davide Scaramuzza. E-RAFT: Dense optical flow from event cameras. Proc. of International Conference on 3D Vision (3DV), 2021.
[11] Daxin Gu, Jia Li, and Lin Zhu, Learning adaptive parameter representation for event-based video reconstruction. IEEE Signal Processing Letters (SPL), 2024.
[12] Haiqian Han, Jiacheng Lyu, Jianing Li. Henglu Wei, Cheng Li, Yajing Wei. Shu Chen, and Xiangyang Ji. Physical-brae -ased event camera simulator. In Proc. of European Conference Computer Vision (ECCV), 2024.
[13] Yuhang Hu. Shih-Chii Liu, and Tobi Delbruck. v2e: From video frames to realistic DVS events. Proc. of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, 2021.
[14] Bowen Huang and Touradj Fbrahimi. Event data stream compression based on point cloud representation. In Proc. of IEFE International Conference Image Processing (ICIP), 2023.
[15] Chankyu Lee, Adarsh Kumar Kosta, Alex Zihao Zhu, Kenneth Chaney, Kostas Danilidis, and Kaushik Roy. Spike-FlowNer Event- based optical flow estimation with energy-efficien hybrid neural networks. In Proc. of European Conference on Computer Vision (ECCV), 2020.
[16] Yijin Li, Zhaoyang Huang, Shuo Chen, Xiaoyu Shi, Hongsheng Li, Hujun Bao, Zhaopeng Cui, and Guofeng Zhang. BlinkFlow: A dataset to push the limits of bveet-based optical flow estimation. In Proc. of IEEE/RSJ International Conference Intelligent Robots and Systems (IROS), 2023.
[17] Quammin Liang, Xiawu Zheng, Kai Huang, Yan Zhang, Jic Chen, and Yonghong Tian, Event-Diffussoo: Event-based image reconstruction and restoration with diffusion models. In Proc. of ACM International Conference on Multimedia (ACM MM), 2023.
[18] Songnan Lin, Ye Ma, Zhenhua Guo, and Bihan Wen. DVS- Voltmeetmetrr: Stochastic process- based event simulator for dynamic vision sensors. In Proc. European Conference on Computer Vision (ECCV), 2022.
11

---

[19] Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft COCO: Common objects in context. In Proc. of European Conference on Computer Vixion (ECCV). 2014.
[20] Haoyue Liu, Shihan Peng, Lin Zhu, Chang, Hanyu Zhou, and Luxin Yan. Seeing motion nighttime with an event camera. In Proc. of the IEEEJCVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024.
[21] Haoyue Liu, Jinghan Xu, Shihan Peng, Yi Chang, Hanyu Zhou, Yuxing Duan, Lin Zhu, Yonghong Tian, and Luxin Yan. NER-Net+: Seeing motion nighttime with an event camera. IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), 2025.
[22] Xinglong Luo, Kunming Luo, Ao Luo, Zhengning Wang, Ping Tan, and Shuaicheng Liu. Learning optical flow from event camera with rendered dataset. In Proc. of IEEE/CVF International Conference on Computer Vision (ICCV), 2023.
[23] Xinglong Luo, Ao Luo, Zhengning Wang, Chunyu Lin, Bing Zeng, and Shuaicheng Liu. Efficient meshflow and optical flow estimation from event cameras. Proc. of the IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), 2024.
[24] Mohammad Mostafavi, Lin Wang, and Kuk-Jin Yoon. Learning to reconstruct hdr images from events, with applications to depth and flow prediction. International Journal of Computer Vision (IJCV), 2021.
[25] Elias Mueggler. Henri Rebecq, Guillermo Gallego, Tobi Delbruck, and Davide Scaramuzza. The event- camera dataset and simulator: Event-based data for pose estimation, visual odometry, and SLAM. Interna- tional Journal of Robotics Research (IJRR), 2017.
[26] Garrick Orchard, Ajinkya Jayawant, Gregory Cohen, and Nitish Thakor. Converting static image datasets to spiking neuromorphic datasets using saccades. Frontiers in Neuroscience, 2015.
[27] Federico Paredes- Parldes-Valles and Guido C. H. E. de Croon. Back to event basics: Self-supervised learning of image reconstruction for event cameras via photometric constancy. In Proc. of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), 2021.
[28] Henri Rehecq, Daniel Gehrig, and Davide Scaramuzza. ESIM: an open event camera simulator. Proc. of Conference on Robot Learning, 2018
[29] Henri Rebeeq, René Ranftl, Vladlen Koltun, and Davide Scaramuzza. Events-to-video: Bringing modern computer vision to event cameras. Proc. of the IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
[30] Cedric Schcerlinck, Henri Rebecq. Daniel Gehrig, Nick Barnes, Robert Mahony, and Davide Scaramuzza. Fast image reconstruction with an event camera. In Proc. of IEEE Winter Conference Applications of Computer Vision (WACV). 2020.
[31] Tonut Schiopu and Radu Ciprian Bilcu. Lossless compression of event camera frames. IEEE Signal Processing Letters (SPL), 2022.
[32] Zachary Teed and Jia Deng. RAFT: Recurrent all-pairs field transforms for optical flow. In Proc. of European Conference on Computer Vision (ECCV), 2020.
[33] Stoffregen Timo, Cedric Scheerlinck, Davide Scaramuzza, Tom Drummond, Nick Barnes, Lindsay Klee- man, and Robert Mahony, Reducing the sim-to-real gap for event cameras. Proc. of European Conference on Computer Vision (ECCV), 2020.
[34] Jack Tumblin, Amit Agrawal, and Ramesh Raskar. Why i want a gradient camera. In Proc. of the IEEE/CVF Conference Computer Vision and Puttern Recognition (CVPR), 2005.
[35] Zhexiong Wan, Yuchao Dai, and Yuxin Mao. Learning dense and continuous optical flow from an event camera. IEF.E. Transactions on Image Processing (TTP), 2022.
[36] Wenming Weng, Yueyi Zhang, and Zhiwei Xiong. Event-based vidco reconstruction using transformer. In Proc. of IEEE/CVF International Conference on Computer Vision (ICCV), 2021.
[37] Guangyang Wu, Xin Tao, Changlin Li, Wenyi Wang. Xiaohong Liu, and Qingqing Zheng. Perception- oriented video frame interpolation via asymmetric blending. In Proc. of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024.
12

---

[38] Jiajie Yu, Xing Lu, Lijun Guo, Chong Wang. Guoqi Li, and Jiangbo Qian. Event-based video reconstruction via spatial-temporal heterogeneous spiking neural network. IEEE Transactions Circuits and Systems for Video Technology (TCSVT). 2025.
[39] Richard Zhang, Phillip Isola, Alexei A. Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness deep features as perceptual metric. In Proc. of IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2018.
[40] Song Zhang, Yu Zhang, Zhe Jiang, Dongqing Zou, Jimmy Ren, and Bin Zhou. Learning to see the dark with events. In Proc. of European Conference on Computer Vision (ECCV), 2020.
[41] Zhongyang Zhang, Shuyang Cui. Kaidong Chai, Haowen Yu. Subhasis Dasgupta, Upal Mahbub, and Tauhidur Rahman. V2CE: Video to continuous events simulator. In Proc. of IEEE International Conference on Robotics and Automation (ICRA), 2024.
[42] Alex Zhu, Liangzhe Yuan, Kenneth Chaney, and Kostas Daniilidis. EV-FlowNet: Self-supervised optical flow estimation for event-based cameras. In Proceedings of Robotics: Science and Systems, 2018.
[43] Alex Zihao Zhu, Dinesh Thakur, Tolga Ozaslan, Bernd Pfrommer, Vijay Kumar, and Kostas Danilidis. The multivehicle stereo event camera datasct: An event camera dataset for 3D perception. In IEEE Robotics and Automation Letters (RAL). 2018.
[44] Alex Zihao Zhu, Ziyun Wang, Kaung Khant, and Kostas Daniilidis. EventGAN: Leveraging large scale image datasets for event cameras. In Proc. of International Conference Computational Photography (ICCP), 2021.
[45] Lin Zhu, Xiao Wang. Yi Chang. Jianing Li, Tiejun Huang, and Yonghong Tian. Event-based video reconstruction potential-assistee spiking neural network. In Proc. of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
[46] Andreas Ziegler, Daniel Teigland, Jonas Tebbe, Thomas Gossard, and Andreas Zell. Real-time event simulation with frame-basee cameras. Proc. of IEEE International Conference on Robotics and Automation (ICRA), 2023. 2023.
13

---

Appendix
A Evaluation Details
In Table 2 report metrics MSE, SSIM, and LPIPS. There several versions of these metrics that produce different values, causing inconsistency across papers. We use a version that closely reproduces the values in previous works such as E2VID++ 3 and ETNet [36.. Specifically:
MSE. our experiments, Mean Square Error calculated on pixel values of images normalized in the range of [0, 1].
SSIM. Structural Similarity is calculated with skimage metrics structural_ caslllssmmlarrr from the Python Package "scikit-image". The images to be processed are normalized into the range of [0, 1]. The window size is set to the default win_sizen size=i. The data_range parameter is incorrectly set to 2 instead of align with previous works.
LPIPS. The Learned Perceptual Image Patch Similarity is calculated with the code and weights provided in https: //gttuu...... com/ cedric-scheers Sanck//eereptullSineee walllSimillrityy The pa- rameters are set to net "alex" and version="O.. "..".. is noted that using different versions of code and weights will lead to different metric values.
We evaluate the methods on selected sequence cuts of IRR 255 MVSEC 4 HQF 3 and
EVAID (6). Following [3], we use the full HQF sequences, and cut the IJRR and MVSEC sequences with boundaries as listed Table The same MVSEC sequence cuts are used for optical flow evaluation.
Table 4: Cut boundaries for IJRR and MVSEC sequences.
URR
MVSEC
Sequence
Start [s]
End [s]
Sequence
Start [S]
End [s]
boxes 6dof
5.0
20.0
indoor flyingl
10.0
70.0
calibration
5.0
20.0
indoor_ @fyoorrfyynng2
10.0
70.0
dynamic_6on 6dof
5.0
20.0
indoor indoor_flying?
10.0
70.0
office_ jiice_
5.0
12.0
indoor_fo indoor_flying
10.0
19.8
poster_6dot
5.0
20.0
outdoor_de
0.0
60.0
shapes_6dos 6dof
5.0
20.0
outdoor_day2
100.0
160.0
slider_depth
1.0
2.5
The frames provided in the EVAID testing burden, we only crop 5-second segment (up to 750 frames, EVAID has high frame rate) from each the longer sequences. The boundaries these segments are listed Table 5
dataset have relatively better visual quality. To reduce the
Table 5: Cut boundaries for EVAID sequences.
EVAID
Sequence
Start [s]
End [s]
Frames
Sequence
Start [s]
End [s]
Frames
ball
0.0
5.0
500
bear
0.0
2.7
66
box
0.0
5.0
100
building
0.0
5.0
750
outdoor
0.0
1.4
26
playball
25.0
30.0
750
room]
0.0
5.0
750
sculpture
0.0
5.0
750
toy
0.0
5.0
100
traffic
0.0
5.0
500
wall
0.0
5.0
750
did not include the "blocks" and "umbrella" sequences, because the cameras were completely static and no signal events were triggered in the background throughout the sequences. As a result, event-based video reconstruction methods can only wildly guess the backgrounds, which causes metrics to be highly random. We cut from the later part of the "playball" sequence for the same
14

---

Perceptual Loss EZVID WeOVio-1KK
Perceptuel Loss on E2VID + ESIM
0.85
0.50
0.5
5
0.30
TOIVESIN
0.45
0.3
0.40
10
-00
100
2000
100
000
500
Epoch
Epoch
Figure 7: Performance on real data fluctuates training loss decreases.
reason: there was no camera motion at the beginning. We exclude the "room2" sequence because it is highly homogeneous with the "rooml" sequence.
We observe that, although the loss on the synthetic training datasets gradually converges after sufficient epochs, the test performances on real datasets still exhibit considerable instability. Figure shows the train loss and validation loss curves of E2VID in the experiments on WebVidJOK ((d) in Table 9 and ESIM (() in Table 9). The checkpoints achieving the best performance on different datasets are often not the same, and the choice of checkpoint for final testing has a significant impact on performance evaluation.
our study, for all experiments, we perform approximately the same number of validation epochs on real datasets throughout training. For example, for E2VID-ESIM trained over 500 epochs, validation is performed every epochs; for E2VIDD- Web Vid10K trained over 80 epochs, every epoch; for E2VID- D2VID-Webbie Wid trained over 800 epochs, every 10 epochs: and for E2VID- SEEVID-Webbie 10d trained over 8000 epochs, every 100 epochs. Then, we select the checkpoint corresponding to the lowest average LPIPS loss across the HQF and EVAID datasets as the final model for testing.
License. MVSEC 43 is released under the CC BY-SA 4.0 license. IJRR 255 is released under the CC BY-NC-SA 3.0 license. HQF 33 and EVAID did not include a license.
B Training Details
All our experiments conducted single-GPU instances, and the peak GPU memory usage does not exceed 80 GB. We followed the optimizer and learning rate settings of the original models.
E2VID. The E2VID models were trained with batch size of 12. The optimizer is Adam with constant learning rate of 0.0001, no weight decay, and AMSGrad enabled.
ETNet. The ETNet models were trained with batch size of 6. The optimizer is Adam with initial learning rate 0,0002, weight decay rate 0.01, and AMSGrad enabled. We adjust the gamma value of the exponential learning rate scheduler to 0.94 since only train for 100 epochs.
The HyperE2VID models were trained with a batch size of 12. The optimizer is Adam with constant learning rate of 0.001, no weight decay, and AMSGrad enabled.
HyperE2VID.
We trained the models on datasets rendered by ESIM 28 well as the Web Vid (2 dataset. As shown in Figure 8 scenes from the WebVid dataset are far more diverse and realistic than the flying-image style scenes rendered by ESIM, which the source of our models' exceeding performance.
License. WebVid 2 states on its website http://lihuub.oommmmbbbbbbe/ee/e be used for non-commercial purposes.
that the dataset can
C Dataset Quality Discussion
The low quality of the APS frames in IJRR and MVSEC has already been reported by Stoffregen al. [33], who selected relatively good clips from IJRR and MVSEC to evaluate on. However, we observe that even the selected clips are far from ideal.
15

---

(a) WebVid scenes
(b) ESIM scenes
Figure 8: Scenes from the WebVid dataset are much more diverse and realistic than the flying-image style scenes rendered by ESIM
For example, as shown Figure the ground truth frame from RJRR/dynami cetrnamic_6doo. 6dof underex- posed and too dark. Even after adjusting the image, the person's 's leg under the table cannot be seen. The image also suffers from vignetting. Hence, the GT-based metrics E2VID methods will be punished if they generate frames with moderate exposure successfully reconstruct the details under the table, causing the metrics to misleading.
The ground truth frame example from (SSEC/outdors __ay is also underexposed. Moreover, when we adjust the frame be brighter, we can observe that it very noisy and unsuitable to serve as ground truth.
The quality of the event pixels also varies across datasets. In Figure 10 for sequences from MVSEC, URR, HQF, and EVAID, we visualize the total number of events triggered on each pixel within the sequence.
In all DAVIS datasets (MVSEC, URR, HQF), we can observe "hot pixels" (dark red dots) that keep omitting events. We also observe vertical stripe-like patterns, suggesting variations in threshold levels between odd and even pixel columns.
In the MVSEC/indos indoor bactioor__lllli flying sequences, we also observe many "dead pixels" (dark blue particles) that do not trigger any events despite the large camera motion. This may have caused the dirty-window artifacts (fixed patterns that hover top of videos, like looking through dirty window) shared by reconstructions of almost all the methods.
We did not observe these problems in the EVAID dataset, which was captured with Prophesee EKV4 event camera. Hence, we speculate that these issues inherent the DAVIS camera series. While we could simulate corresponding defects during event synthesis to improve model generalization on DAVIS data, the fact that these cameras are no longer available on the market makes backward compatibility less valuable. Therefore, we have decided to look ahead and directly update our testing benchmark to focus on more advanced camera models.
D Ablation Study Results
D.1 Dataset quality
In the WebVid dataset, a large proportion of videos have out-of-focus blur due to large aperture settings, and some videos synthetic CG instead of real videos. We tried to improve dataset quality by filtering them out, but the results were counterintuitive: using the unfiltered version produced better results.
Specifically, from the 2.5M videos of WebVid, we arbitrarily selected subset 32K videos and labeled them with the large vision language model Qwen2.5-VL. [0. For each video, we sample one frame and ask Qwen2.5-VI. if there is any post-production artefact (such as subtitles/CG/...) in the image, if the image is blank, and if the image has out-of-focus motion blur. We exclude
16

---

(a)
(b)
Ground Truth
Adjusted GT
Ours
Events
Figure 9: Frames from IJRR (a) and MVSEC (b) are far from ideal.
MVSEC/indoor flying1
URRidynamic 6dof
HQF/bik bay hdr
EVAIDrraffic
MVSECloutdoor day1
URR/boxes 6dol
HQF/still life
EVAlDroomm
Figure 10: Event count maps from real event data sequences.
all videos with post-production artefacts, blank content, or blurriness. From the remainings, we randomly sampled 10K videos, forming the dataset WebVid10K*
By comparing lines (d) and (n) in Table we that models trained WebVid10f. perform slightly better than on Web Vid10K*. This may be because the filtering decreases dataset diversity, since specific sample types, such daytime outdoor long-shot videos, are less likely to be filtered out, The semantic statistics of the two datasets (annotated using Qwen2.5-VI [T] are listed in Table
Table 6: Semantic statistics of Web Vid datasets.
Description
VedVid10K (%)
WebVid10K (%)
Is real video
100.00
76.83
From outdoor scene
70.63
53.15
From indoor scene
21.33
25.09
daytime
75.53
60.51
Is nighttime
3.44
9.56
Has water
22.22
16.17
Has humans
30.09
34.11
Has sky
46.19
27.59
blank
0.00
3.44
Has defocus blur
0.00
41.10
Has motion blur
0.00
9.32
Contains text
3.76
3.43
To demonstrate the importance of exposure quality, we intentionally degenerate the dynamic range of the video dataset, and observe its effect the model. Specifically, in each iteration, with 80%
17

---

Training Data
Test Results
LDR Degradation
Predict images with higher contrast
Figure 11: Degenerating the dynamic range of training videos makes the model reconstruct images with higher contrast.
E2VID WebVid 10K Alex Temporal Consistency
E2VID WebVid10K VGG Temporal Consistency (Half)
Figure 12: The Alex LPIPS loss introduces arrow-like patterned artefacts when the video degenerates.
probability, we degenerate the video frames with a random scale s between and 3:
Fdegrade Clip((I 127.5) + 127.5, 255).
This increases the contrast of the video, and creates more overexposed and underexposed areas. Our experiments show that the corresponding model indeed learns to reconstruct images in a more high contrast style, shown in Figure The quantitative metries are listed in line (o) of Table 9
D.2 Design of loss function
Alex or VGG for perceptual loss. We observe that using the Alex model for the computation of LPIPS loss introduces a certain type of patterned artefact when events are sparse and the outputs start to degenerate. These patterns are visually unpleasant, and they disappear when we change to using the VGG LPIPS loss (combined with LI loss). In the ablation experiment (p) of Table 9 we train on WebVid10K using the original loss combination, Alex LPIPS + full temporal consistency (with optical flow from RAFT-Smal (32). Figure 12 shows the unpleasant arrow-lik patterns, and the corresponding metrics in the table are also worse than experiment (d), which was trained on our loss design.
Pseudo ground truth optical flow. Model-based event simulators such as ESIM provide ground truth optical flow, which is used to compute temporal consistency loss. When we shift to video-based simulation, ground truth optical flow is no longer accessible. Although we can predict optical flow using the image frames, worry that the errors in prediction may have negative effects training.
In experiment (t) of Table we train the E2VID model using predictions RAFT-Small 322 as optical flow. The experiment (l) of Table 9 has the same setting, except for the optical flow used in the temporal consistency loss ESIM-rendered ground truth. Through comparison, no significant negative influence can be observed.
18

---

Half temporal consistency loss. In this paper, we introduce half temporal consistency loss, i.e.. only computing temporal consistency loss over the last 20 frames of a 40-frame sequence. Table g experiments (r) (s) (1) explore the benefits of using half temporal consistency loss instead of no temporal consistency loss (only Alex LPIPS, (r)) or full temporal consistency loss (Alex+TC-RAFI (0)) Although the half consistency loss experiment (s) do not exceed in the frame-based quantitative metrics, the elimination of flickering and the relief of dirly-window artefacts can observed the reconstructed videos, which can be found in the supplementary video.
D.3 Comparison against the V2E solution.
In order to compare our V2V framework to a "frame interpolation + event simulation" approach, we interpolated the WebVid- 100 dataset by 8x with PerVFI [37] and used the popular V2E simulator [[3] to generate training events. We used the "clean" noise mode. For the thresholds, we linearly scaled the random thresholds in the "fixed thresholds" ablation experiment due the different logarithm mapping of V2E from ESIM (our V2V framework follows ESIM).
In the simulation. video (1066695484 mp4, with original size 1.2MB) produced 14GB of events. This caused our prst-procssiing step to fail due to out-of-memory issues. We discarded this sample, leaving with WehVid-99 dataset. The WebVid-9P Vid-99 dataset (in events) sums 12 GB space, while the original Web Vid-100 videos only sum to 210 MB.
We then stacked the events into interpolated voxel bins. In this stage, we need to select the durations of the voxel bins. the V2V experiments, each voxel corresponds to the events between real frames, and the E2VID models reconstruct 24/5 FPS videos. We tried two binning strategies in our V2E experiments:
In strategy 1, we encode all the events between 1+1 original frames (crossing 7 interpolated frames) into single voxel. This makes the E2VID models reconstruct 24 FPS videos, utilizing the high temporal resolution from frame interpolation. We annotate this as "V2E-1".
In strategy 2, encode all the events between 5+1 original frames (crossing 35 interpolated frames) into a single voxel. This makes the E2VID models reconstruct 24/5 FPS videos, aligning to the V2V experiments. We annotate this "V2E-5".
During training, we used the same augmentation noise (gaussian noise + hot pixels) as the ESIM protocol. For the V2E-5 experiment, we train for 8000 epochs, aligning to the V2V-WebVid-100 experiment. For the V2E-1 experiment, produces 5x more training samples, so we train for 1600 epochs.
The test results are shown Table 7 experiment, showing that training on shorter bins may indeed benefit model performance. However, both V2E experiments produced metrics worse than the WebbVid-10--Fiie uses fixed thresholds for each video. This shows that our V2V method not only exceeds the V2E approach in efficiency, but also produces models with better performance.
The V2V-1 experiment produced better results than the V2V-5
experiment, which also
Table 7: Comparison of V2V solution and V2E approach. Lower (L) MSE and LPIPS values and higher (1) SSIM values are desirable. We highlight the best values with green
HQF
EVAID
Model
Train Dataset
Loss
Eps
MSEJ
SSIMI
LPIPS.
MSEJ
SSIM
LPIPS.
E2VID 12V-100-Fixed
V+LI+TH
8000
0.049
0.611
0.330
0.073
0.604
0.451
F2VID V2V-100
V+I.++TH
8000
0.042
0.626
0.327
0.078
0.594
0.441
E2VID Web V2F-1
V+I.1+TH
1600
0.057
0.524
0.459
0.116
0.484
0.561
E2VID
RebVid99-V2E-S
V+LI+TH
8000
0.077
0.470
0.569
0.117
0.483
0.588
D.4
Finetuning on test datasets.
Our V2V framework produces models with strong zero-shot generalization abilities. In order to show this, we conducted a finetuning experiment on EVAID to show that our zero-shot model can outperform model trained or finetuned with limited real event data.
19

---

Our EVAID test set has 10 sequences. We took 7 sequences (Ball, Box, Building, Outdoor, Playball, Rooml, Toy, Wall) as a train set, and sequences (Bear, Sculpture, Traffic) the new test set.
Zero-shot inference. We directly used the V2V-E2VID model weights, trained on the Web Vid dataset, for zero-shot inference on the EVAID test set.
Finetuning on EVAID. We finetuned the pretrained V2V-E2VID model on the EVAID train set for 100 epochs, with learning rate of 0.0001, observing that the validation loss had converged.
Directly training on EVAID, We trained an E2VID model from scratch on the EVAID train set. We trained it for 2000 epochs with learning rate 0.0001, observing that the validation loss had converged.
The resulting metrics of the three models are shown in Table X Surprisingly, the zero-shot model performed best. Finetuning caused slight performance drop, while training real data from scratch performed worst. This is likely due to the small size of the EVAID dataset, which caused the model overfit to the training sequences and fail generalize the test sequences.
Table 8: E2VID directly trained or finetuned with EVAID. Lower (L) MSE and LPIPS values and higher (1) SSIM values are desirable. We highlight the best values with green
EVAID
Model Train Dataset
Loss
Eps
MSEJ
SSIM
LPIPS
E2VID V2V-10K (Zero-shot) E2VID V2V-10K EVAID (Finetuned)
V+LI+TH
80
0.056
0.661
0.424
V+LI+TH
80 100
0.064
0.645
0.458
E2VID EVAID (From scratch)
V+LI+TH
2000
0.078
0.483
0.658
D.5 Performance gain on downstream tasks
In order to demonstrate the reusability of the V2V weights on downstream tasks, we conducted an experiment on the N-MNIST classification task.
The MNIST dataset is an image-based dataset, corresponding the task of classifying hand-written digits, frequently used as a toy dataset in computer vision. N-MNIST (26) is a corresponding event- based classification dataset, recorded by saccading the original MNIST images with real event camera. We chose this task due to its convenience: the dataset is small and code is simple.
We first trained very simple convolution model on the train split of MNIST (the image version). On the test split (images), the model achieved an accuarcy of 98.768.. 76%..
Then we explored using video reconstruction models in zero-shot pipeline for the N-MNIST task. follows:
The pipeline
Take real event streams from the N-MNIST test split. Stack them to 5*5 event bins.
For each of the 5 voxels (each with 5 bins), the E2VID model reconstruct image frame. This produces 5 reconstructed frames.
Use the last frame as the input to the bmsge MNIST classifier, and predict the digit.
When using the original E2VID model, the resulting accuracy on the N-MNIST test set is 47.46%. When changing to the V2V-E2VID model, the accuracy improves to 63. 63.62%. This is due to the better reconstruction quality of the V2V-E2VID model.
This shows that the quality improvement brought by the V2V framework can be utilized by zero-shot downstream tasks. Note that there are existing classification models that can generalize much better; the purpose of using a weak model is to compare the original E2VID against V2V-E2VID.
E Additional Reconstruction Results
We provide qualitative results of method V2V-E2VID (trained on WebVid10K) in Figure B Figure 14] Figure [5] and Figure [6] Compared to the baseline method E2VID+, our method produces fewer artefacts and better image contrasts.
In Table we report quantitative metries over the datasets EVAID, HQF, IJRR, and MVSEC. Quantitative metrics of all ablation studies are also provided.
20

---

LPIPS
0.506
0.525
0.509
0.465
0.538
0.513
0.492
0.473
0.57
0.491
0.551
0.49
0.494
0.465
0.468
0.48
0.465
0.453
0.497
0.532
0.504
0.49
MVSEC
SSIM
0.34
0.385
0.36
0.406
0.356
0.344
0.37
0.419
0.34
0.36
0.351
0.324
0.345
0.406
0.398
0.386
0.406
0.375
0.368
0.313
0.335
0.324
MSE
0.132
0.076
0.103
0.066
0.096
0.104
0.097
0.07
0.077
0.111
0.103
0.127
0.125
0.066
0.072
0.092
0.066
0.088
0.119
0.165
0.141
0.127
Table 9: Qualitative results of all experiments all datasets. Duplicated lines provided for convenience of comparison.
LPIPS
0.238
0.278
0.251
0.248
0.287
0.275
0.256
0.227
0.231
0.236
0.26
0.258
0.255
0.248
0.238
0.265
0.248
0.268
0.249
0.266
0.255
0.258
IJRR
SSIM
0.556
0.55
0.569
0.562
0.554
0.529
0.539
0.624
0.6
0.59
0.544
0.561
0.57
0.562
0.574
0.56
0.562
0.519
0.584
0.533
0.552
0.561
MSB
0.063
0.067
0.078
0.07
0.077
0.1
0.086
0..06
0.059
0.049
0.085
0.061
0.06
0.07
0.069
0.092
0.07
0.082
0.057
0.083
0.071
0.061
LPIPS
0.499
0.441
0.405
0.371
0.451
0.413
0.378
0.474
0.377
0.523
0.406
0.478
0.45
0.371
0.389
0.44
0.371
0.388
0.457
0.448
0,451
0,478
EVAID
SSIM
0.526
0.594
0.627
0.663
0.604
0.617
0.644
0.533
0.654
0.541
0.526
0.548
0.663
0.645
0.565
0.663
0.604
0.55
0.549
0.541
0.526
MSE
0.088
0.078
0.072
0.054
0.073
0.066
0.058
0.071
0.055
0.08
0.056
Voxel Representation
0.078
0.062
0.054
0.059
0.09
0.054
0.054
0.06
0.072
0.07
0.078
Ablation Dataset quality
V2V Other models
Loss design
LPIPS
EEVID
0.256
0.327
0.286
0.265
0.33
0.293
0.264
0.265
0. 0.269
0.274
0.306
0.306
0.296
0.265
0.275
0.287
0.265
0.249
0. 0.285
0.285
0.276
0.306
HQF
SSIM
V2V
0.638
0.626
0.661
0.676
0.611
0.64
0.666
0.646
0.671
0.642
0.641
0.599
0.608
0.676
0.662
0.647
Ablation
0.676
0.652
0.621
0.608
0.613
0.599
Ablation
MSE
0.037
0.042
0.037
0,032
0.049
0.039
0.032
0.032
0.035
0.035
0.039
0.041
0.04
0,032
0.038
0.034
0.032
0.034
0.041
0.045
0.039
0.041
Epochs
500
8000
800
80
8000
800
80
400
60
700
100
500
500
80
80
80
500
500
500
500
TOXFTC-RAFT
Alex+TC
V+LI+TH
V+LI+TH
V+LI+TH
V+LI+TH
V+LI+TH
V+LI+TH
Alex+TC
V+LI+TH
Alex+TC
V+LI+TH
Alex+TC Alex+TC
V+LI+TH V+LI+TH V+LI+TI
V+LI+TH Alex+TC
Alex+TH-RAF
Alex+TC-GT
Loss
Alex
Training Dataset
Pretrained (ESIM)
Pretrained (ESIM)
WebVid Pretrained (ESIM) Vid
Web Web 1id Web WibVid1Ok-Ie LDR
Vid11K
Web Web Vid100-F
Web Web VidlkFF
Web VidlOOI
ESIM-280-Im ESIM 28--200-DD Disc.
VidlKK
Web Web VidIOK
Web Web Vid1OK
ESIM-280-IM 2nt-Intp.
ESIM 2-0-2800--- Intp.
ESIM 280- 280-Intp. Intp.
ESIM- 4SIM-280-In1E
Intp.
Wid1k
Web
HyperE2VID EVID
Idx Model
E2VID
E2VID
E2VID
E2VID
E2VID
E2VID
E2VID
ETNet
ETNet
E2VID
E2VID
E2VID
E2VID
E2VID
E2VID
E2VID
E2VID
E2VID
E2VID
E2VID
(8)
(D)
6)
(3)
(c)
(m)
(
(1)
(0)
(d)
(p)
11
0
21

---

Qualitative Optical Flow Results
In Figure 17 we provide visualizations of our optical flow predictions. Our retrained model V2V- EyFlowNett produces denser results and fewer black hollows.
G Broader Impacts
Positive impacts. Our V2V module enables scaling up event-based datasets with less storage usage, less data transfer burden, more sample diversity, and greater augmentation flexibility. This will lower the economic barriers for conducting event-based vision research, empower more researchers advance related work, and leverage the event camera's high temporal resolution, high dynamic range, and low power consumption to push the boundaries of machine vision.
Negative Impacts. Video-based training enables models to learn prior knowledge from the video data. As result, they may suffer from biased or discriminatory features present in the video datasets, which calls for future researchers more carefully consider the data used for training.
22

---

E2VID+ (Baseline)
V2V-E2VID-10K (Ours)
Ground Truth
EAA
nU CAR
A RS SNUGKS
AU
ANNIRAA W
SNTGKS
SNONSS
TEBI CIANIUU
BST GIRIBOR
ROLTOIT
200
EVAID box
EVAID building
EVAID/ room1
EVAID / traffic
EVAID / toy
EVAID / wall
Figure 13: Qualitative results on the EVAID dataset.
23

---

E (Baseline)
V2V-E2VID-10K (Ours)
Ground Truth
HQF boxes
HQF
/ dees__aaddd _ass__aad__nly
Resurres Digioring
Dewthert
HQF engineering posters
HQF / whihhhtggttttteeee texture_plante
HQF /
Figure 14: Qualitative results on the HQF dataset.
24

---

+ (Baseline)
V2V-E2VID-10K (Ours)
Ground Truth
IJRR / fexee__6doff
IJRR / calibration
IJRR / office office_zigzeenl zigzag
IJRR / poster_ 6dof
IJRR / blider_depth
Figure 15: Qualitative results on the IJRR dataset.
25

---

E2VID+ (Baseline)
V2V-E2VID-10K (Ours)
Ground Truth
MVSEC/ indoor_flying1
MVSEC indoor indoor_flying2
MVSEC/ indoor __llingg
MVSEC/ outdoor 1
MVSEC / outdoor_day
Figure 16: Qualitative results on the MVSEC dataset.
26

---

Image
Ground Truth
EvF IowNet+
V2V-EwFlowNet
MVSEC/indoo__ _flyinglllll
Image
RAFT
EvFlowNet
V2V-EvFlowNet
EVAID/box
R R/boxes 6dof
HQF/slow hand
Figure 17: Qualitative results of V2V-ExFlow.
27

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We claim to contribute the V2V module, which enables efficient, flexible and scalable video-based event vision model training. These claims are validated by our experiments.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the
results can be expected to generalize to other settings. is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We discuss the limitations of our work in the appendix.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification,
asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how
they scale with dataset size.
applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions favor of transparency play important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: Our main theoretical result is the feasibility of converting videos directly to discrete voxel grids. We conducted an analysis and pointed out that its basis is an event representation that discards intra- bin temporal information.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross-referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
28

---

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main experi- mental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification:
We described the V2V model in detail, and we promise to release corresponding
code.
Guidelines:
The answer NA means that the paper does not include experiments
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, if the contribution is a specific model and empirical evaluation, it may necessary to either make it possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submissions provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how reproduce that algorithm.
(b) If the contribution is primarily architecture clearly and fully.
model architecture, the paper should describe the
(c) If the contribution is new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g.. with open-source dataset instructions for how construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases. in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (c.g.. to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
Open access data and code
Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We promise to release our code on Github after paper acceptance.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines stcttps//nips.cc/pputt
Stl.dess/CodeSubiiise ssiissssiccyyyy.oooo for more details.
While encourage the release of code and data, understand that this might not be possible,
so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: //nips//ipp/// http//pppppppppplc.../. quides/CodeSudle wembbbbmssiiiiii www.misssoooooooo onPolicy) for more details.
29

---

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reprodueible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended. but including URLs data and code is permitted. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits. hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We explained the experiment settings in our paper, and will release experiment configurations with the code for further clarification.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is
necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: Error bars are not reported because it would be too computationally expensive.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors). It should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report 1 1-sigma error bars, but one should state it. The authors should preferably report 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of eπrors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates). If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
Experiments compute resources
Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed reproduce the experi- ments?
Answer: [Yes]
Justification: We provide training details including computer resources in the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud
provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
30

---

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g.. preliminary or failed experiments that didn't 't make into the paper).
Code of ethies
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code Ethics /tttp:://eurr
Answer: [Yes]
Justification: This paper does not propose new datasets and does not involve human subjects.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code Ethics.
If the authors answer No, they should explain the special cireumstances that require a deviation
from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consideration due
to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We discuss the potential societal impacts in the appendix, both positive and negative. Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deploy- ment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to par- ticular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (c.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification: The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Relcased models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been seraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.
12. Licenses for existing assets
31

---

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We cite the original papers of all code and datasets used. We also mention the licenses and terms of use in the appendix. MVSEC [43] is released under the CC BY-SA 4.0 license. IJRR [25] is released under the CC BY-NC-SA 3.0 license. HQF [33] did not include a license. Web Vid (2 states on its website (httss//github.. http://gihuu..coommbbbiiii hot///bbbi///wwwbiiddd that the dataset
can be used for non-commercial purposes. Since research is non-commercial and do not redistribute the datasets, the paper respects the licenses. Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version the asset used and, possible, include a URL..
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.
For existing datasets that are re-packaged. both the original license and the license the derived asset (if it has changed) should be provided.
this information not available online, the authors are encouraged reach out to the asset's
creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: The code related to the V2V module will be released with detailed documentation. Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their sub- missions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human
subjects.
Including this information the supplemental material is fine, but if the main contribution of the paper involves human subjects, then much detail possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were
obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects
32

---

Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applica- ble). such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper descrihe the usage of LLMs if it is important, original, standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification: The original ideas were developed by human mind, and all experiments were conducted with human operation.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs any important, original, or non-standard components. Please refer our L.L.M policy /est:p///eerrppsscc///ofeeeeeeee should or should not be described.
for what
33


Reviews:
[{"review_id": "Lv363QHkOu", "reviewer": "Reviewer_WNjQ", "summary": "Scaling up event-based training dataset is tough because of massive bandwidth/storage requirements of synthetic data pipelines. The authors propose a Video2Voxel approach which converts video frames to event-based voxel grids – bypassing event stream generation.\nEvent cameras emit an asynchronous stream of events $(x, y, t, p)$, where $(x, y)$ are pixel indices, t is a microsecond-accurate timestamp, and $p \\in {+1, –1}$ indicates whether the log-intensity at that pixel increased or decreased beyond a preset threshold. Pixels fire only when that contrast change occurs, so the data are spatially sparse and temporally precise. Frame cameras, by contrast, deliver full RGB / grayscale images at discrete, globally clocked intervals (e.g. 30 fps). Peak bandwidth in an event stream can be high under rapid motion, yet average bandwidth is typically lower than that of a high-speed conventional camera because inactive pixels remain silent. One can also view it as a temporal-gradient sensor [a]: integrating events over time tells us brightness, but only up to an unknown offset at each pixel.\n\nEarlier methods wish to use videos either via (a) video simulation which uses frame interpolation to detect events, this is of limited fidelity because it’s an attempt to fill event reconstruction gaps from data that is not present in low-temporal-resolution videos (b) model simulation, i.e. build a simulation environment in 3D and render event camera streams. While viable, this is tough to use because of the sim2real gap. When synthetic streams are persisted at micro-second resolution the I/O load explodes, whereas on-the-fly processing of real sensors is typically lightweight.\n\nThe paper asks the simple question: “Do I need event streams in the first place?” Many current event-based methods convert async-event into dense representations like voxel grids (essentially an accumulation step for each sparse observation $(x, y, t, p)$). Thus, instead of generating event streams, one can generate voxel accumulations. This has several advantages (a) one can now use all the video data without interpolation (b) it’s more efficient to generate compared to streams, which are anyways accumulated later.\nPaper focuses on dense prediction tasks: event-based video reconstruction and optical flow estimation. The method is to utilize the difference between interpolated and discrete voxel grids – recognizing that interpolated grids generated from videos do not match event cameras compared to discrete voxel grids. Using this key fact, they write a simulator that provides them discrete voxel grids from videos.\n\n[a] Why I want a Gradient Camera, CVPR 2005", "questions": "While V2V convincingly scales training data for voxel-based reconstruction and optical-flow networks, it does not show benefits for any system that leverages an event camera’s fundamental advantage—microsecond-level latency. Tasks that need per-event timing (event-based SLAM, gesture recognition, etc.) remain completely untested. The paper therefore answers an important data-volume question for a subset of event-vision problems, but it sidesteps—and leaves open—whether its video-to-voxel shortcut actually helps the applications for which event cameras were invented. \n\nIn principle the approach could help, say, by augmenting SLAM with the authors’ flow estimator, but we need to see such a system-level demonstration before the claim is convincing. Evaluating only on dense voxel tasks tells us little if their methods leverages the advantages of both internet-scale data and event cameras for the things they are uniquely good at. Can the authors show improvements on at least one latency-critical, event-native task?\n\nI'll switch to an accept if such evidence is provided. The core idea is in itself solid.", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "1. Strength. Motivation, problem statement, and method are all sound. It makes sense to directly generate voxels if methods resort to voxels to accumulate events in the first place when operating on event streams. \n\n2. Strength. Results themselves appear to be sound for reconstruction and optical flow estimations. Efficiency gains are reported and are appreciated. In itself, for these two tasks, the quantitative gains are solid.\n\n3. Weakness. Paper writing leaves much to be desired. Reframe inherent ambiguity in event-based vision tasks. Please cite [a] or other papers to reference the inherent ambiguity, this is a pretty well known fact about event cameras.\n\n4.\tWeakness. Results section is hard to read, and parse. It reads like a technical report, and the onus is on the reader to understand what are the distinct advantages of their approach. It can be re-written to highlight the benefits of their video2event-voxel method.\n\n- Table 2 to is extremely hard to parse, which results are yours and which ones are the baselines. I understand WebVid10K is yours, but reorganize the table to highlight the results and other methodologies (for example ESIM-280-Intp. Seems to be a video interpolation method).\n- Fig 6, Please highlight which methods are yours and which ones are others in the figure/caption.\n\n5. Weakness. Evaluation scope is narrow. So, the method is simple and improves on dense tasks, but what’s missing from the overall picture? Voxel accumulation helps in tasks where an explicit accumulation step is required in the first place – this is limited as any dense binning costs temporal precision but keeps the full spatial grid, which defeats the entire purpose of an event camera in my view. Essentially, the whole point of the event camera was to gain temporal resolution by sacrificing per-frame intensity, and performing an accumulation step is counter to the real benefit of this camera.\n- The real core solution in my view addresses the following question -- can we reuse weights trained on such video scale datasets, and finetune on limited real events data for other event tasks (on, say some sparse task or some system level program which operates on events, say event based SLAM), and get benefits over the limited data regime?", "was_revised": true, "final_justification": "Authors have resolved many of my concerns by providing additional clarifications about existing results and have promised to improve their presentation -- which amounts to a few minor changes. They also provided additional results by testing their synthetic data pipeline in comparison to training on limited real data from scratch and shown their large-scale data-driven method prevails. Lastly, while the additional pipeline experiment is limited in scope and size, it provides some hope that large-scale data-driven priors from videos can be exploited for fast paced event camera pipelines.", "rebuttal": "**Response to Reviewer WNjQ**\n\nThank you for your thorough review and insightful comments. We will address each concern below.\n\n**W3 & W4. Clarity of writing.**\n\nWe will improve the clarity of our writing in the final version. \n\n* We will add citations (including [a]) to our preliminary discussion on the inherent ambiguity of event-based vision, as suggested.\n* We will highlight the \"Ours\" methods in Table 2 and Figure 6 to improve readability.\n\nThe \"ESIM-280-Intp\" in Table 2 refers to a training dataset created from ESIM, composed of 280 scenes, and converted to the Interpolated voxel representation (L259-L261). The corresponding experiments (l) and (m) are also video reconstruction experiments. They are designed to compare the interpolated voxel representation against the discrete voxel representation. We apologize for any confusions.\n\n**W5 & Q. Limitations of the voxel representation.**\n\nWe acknowledge the reviewer's concern on discarding intra-bin timing, which is a limitation we discuss in Section G.\nHowever, we believe that this limitation does not \"defeat the entire purpose of an event camera\".\n\n1. Voxel representations preserve key advantages of event cameras. As detailed in Section 3.2, while discrete voxels discard fine-grained intra-bin timing, they preserve essential inter-bin temporal dynamics. This allows for:\n\n\t* High temporal resolution: By adjusting bin durations, voxel grids can effectively encode high-speed motion. In our work, we successfully reconstruct videos at 150 FPS on the EVAID dataset.\n\t* High dynamic range (HDR): The HDR property of event cameras, which relies on per-pixel logarithmic changes, is fully preserved in the voxel representation.\n\t* Sparsity and efficiency: The principle of sparsity can be maintained by adapting bin durations based on event activity or by processing only local patches with sufficient event density, preserving the low-power potential of event cameras.\n\n2.  Voxel-based methods are widely and successfully used across diverse event-based tasks, including latency-sensitive ones like SLAM. The simplicity and effectiveness of voxel grids have made them a cornerstone of modern event-based vision research. Examples include:\n\n\t* SLAM: [CVPR 2019] Unsupervised Event-based Learning of Optical Flow, Depth, and Egomotion.\n\t* Action recognition: [CVPR 2022] E^2(GO)MOTION: Motion Augmented Event Stream for Egocentric Action Recognition.\n\t* Head pose estimation: [ECCV 2024] Event-based Head Pose Estimation: Benchmark and Method.\n\t* Object detection: [CVPR 2024] Scene Adaptive Sparse Transformer for Event-based Object Detection.\n\t* Eye tracking: [CVPRW 2024] Event-Based Eye Tracking. AIS 2024 Challenge Survey.\n\t* HDR imaging: [CVPR 2023] Learning event guided high dynamic range video reconstruction.\n\t* Image deblurring: [CVPRW 2025] NTIRE 2025 Challenge on Event-Based Image Deblurring: Methods and Results.\n\n\tOur V2V framework can boost models that accepts a voxel-based input, making it broadly applicable. We chose video reconstruction and optical flow as representative examples, and we hope to inspire the community to explore more tasks given task-specific large-scale datasets. \n\n3. While the ultimate goal may be end-to-end spiking neural networks on neuromorphic hardware, the current ecosystem faces challenges with hardware cost and computational limitations. High-performing, GPU-trained voxel-based models can demonstrate the value of event cameras, driving the widespread adoption needed to scale the industry. As the hardware matures, these successes can pave the way for a transition to more neuromorphic approaches.\n\n4. Regarding whether training on synthetic data can improve performance on real data, our intuition is that it can. Existing literature, such as *[CVPR 2020] Video to Events: Recycling Video Datasets for Event Cameras*, suggest that training on synthetic data and fine-tuning on real data often yields the best results. However, this is beyond the scope of our current study and awaits future validation."}, {"review_id": "ThLNnnQMsO", "reviewer": "Reviewer_92jc", "summary": "This paper proposes a method to generate event-based training data from conventional RGB video. The method directly converts RGB images into an event voxel representation, skipping the step of materializing sparse event data. This significantly reduces the storage and memory requirements, as event voxels are a much more compact representation than sparse events. Using this method, the authors generate large-scale event-based training datasets. Training on these larger-scale datasets leads to substantial improvements in model performance.", "questions": "My only request is further explanation/justification for the modeling assumptions in Equation 5. I am basing my positive rating on the assumption that this model is generally sound. I am interested to see the comments of the other reviewers on this point.", "limitations": "The paper is missing a limitations section. A potential discussion point would be issues arising from using lower-speed RGB video to generate training data. E.g., have the authors observed any generalization failures on real event data? Are there types of motion that are not well represented in the generated data?", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 4, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n\nI am overall positive on this paper. The authors have identified one of the key issues with event cameras - smaller-scale datasets hinder model performance, which inhibits further adoption of event cameras. The proposed method can be used to train event camera models on internet-scale data, and hopefully bring event cameras into the realm of modern large-scale deep learning. The results are convincing (especially the qualitative results in Figure 6).\n\nThe comparison between interpolated and discrete event voxels was nicely explained and well-illustrated in the figure.\n\nThe writing and figure quality is excellent.\n\nI appreciate the transparency about the experiment mistake at L287.\n\n**Weaknesses**\n\nThe description of key technical details (in the last half of page 6) could be expanded. In particular, I would like to see more discussion of Equation 5 and justification for its modeling assumptions. E.g., why can we assume that background noise can be modeled as a Gaussian on the output voltage?\n\nThe temporal resolution of the event voxels is tied to the input frame rate. This could be seen either as a feature or a bug - the method is not capable of producing higher-speed outputs, but we also don't have to solve the interpolation problem.", "was_revised": true, "final_justification": "I remain positive on this paper - the authors have addressed my few concerns in the rebuttal.", "rebuttal": "**Response to Reviewer 92jc**\n\nThank you for your positive evaluation and constructive feedback. We will address your concerns below.\n\n**S4. Experiment mistake.**\n\nThank you for your understanding. We have reconducted the experiment with the correct configuration, and we will update the results in the final version of the paper. The test results before and after correction are as follows.\n\n| Idx       | Model | Train Dataset | Loss   | Eps | HQF MSE | HQF SSIM | HQF LPIPS | EVAID MSE | EVAID SSIM | EVAID LPIPS |\n| :-------- | :---- | :------------ | :----- | :-- | :------ | :------- | :-------- | :-------- | :--------- | :---------- |\n| (k-old)   | ETNet | WebVid10K*    | V+L1+H | 100 | 0.036   | 0.662    | 0.279     | 0.081     | 0.586      | 0.432       |\n| (k-fixed) | ETNet | WebVid10K     | V+L1+H | 100 | 0.039   | 0.641    | 0.306     | 0.056     | 0.610      | 0.406       |\n\n**W1 & Q. Justification for noise simulation modeling.**\n\nOur noise model was designed by referencing two widely used simulators, ESIM and DVS-Voltmeter. The Gaussian noise added to the output voltage is adapted from DVS-Voltmeter's model, which simulates the voltage change as a Brownian motion process caused by photon reception randomness. This is more physically plausible than adding noise post-voxelization (as in the ESIM training protocol), as it allows noise to interact with the signal during event generation. For simplicity, we did not include other noise types like leak noise, but they could be easily incorporated into our framework.\n\n**W2 & L1. Adaption to higher temporal resolution.**\n\nWhen applying V2V-trained models to real event streams, the temporal resolution of the outputs can be adjusted by changing the bin durations of the voxels. For example, the EVAID dataset has ground truth frame rates up to 150 FPS (see Table 5), so we set the duration of each bin to 1/150/5 = 1.33 ms. The resulting high frame rate video outputs achieved good results qualitatively (Figure 6) and quantitatively (Table 2).\n\nThis raises a question: Why can the model, trained on 5 FPS synthetic voxels, adapt to 150 FPS real event inference? We believe the answer lies in the vast diversity of our training data. The WebVid dataset contains scenes with a wide range of motion speeds, from rapid action to nearly static shots. Statistically, voxels from long bins in slow scenes can appear similar to voxels from short bins in fast scenes. This diversity allows the model to learn representations that are robust to different bin durations and, therefore, different temporal resolutions.\n\n**L1. Missing limitations.**\n\nDue to the space limit, we moved our limitations section to the appendix (Section G). We apologize for the resulting inconvenience."}, {"review_id": "9I6IXHKQ5Z", "reviewer": "Reviewer_rSE5", "summary": "The paper introduces Video-to-voxel, a method to generate synthetic event voxels from videos. It builds on the idea that we do not need to calculate actual raw events if we use voxel grids as input to our neural network. The authors show results for two tasks, video reconstruction and optical flow estimation.", "questions": "- In Fig. 5/Tab. 1, you are showing a comparison between your method and ESIM-280. How well would a network perform that is trained on WebVid-100, using \"conventional\" video interpolation + event simulation pipeline? The comparison between V2V + WebVid and ESIM-280 (ECCV'20), seems to limit the comparison.\n\n- What intervals are you choosing for the random contrast thresholds c+, c-?\n\n- Other grid representations have shown performance advantages over voxel grids (e.g., Zubic, ICCV'23). If we neglect the intra-bin time differences for other representations (e.g., time maps), would we see similar effects?", "limitations": "Yes, the authors address limitations.", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strength**\n- The idea is simple and tackles an urgent need in event-based vision to enable training on larger-scale datasets.\n- The presentation is very clear.\n- The results are promising.\n\n**Weaknesses**\n- The performance evaluation, compared to a \"frame interpolation + event simulation\" approach, could be performed a bit cleaner. E.g., use the same base dataset (WebVid-X) and use the best available frame interpolation method. I think the idea is good, but the ESIM-280 might not reflect the performance you could achieve with the \"classical\" frame interpolation + event simulation approach.", "was_revised": true, "final_justification": "I appreciate the extended experiments on the WebVid-100 dataset. This improves the evaluation. If I see it correctly, models trained on the same data show comparable performance, between video interpolation + vid2e and v2v, while v2v avoids the costly interpolation. I keep my rating at 5.", "rebuttal": "**Response to Reviewer rSE5**\n\nThank you for your positive feedback and insightful suggestions. We will address your concerns below.\n\n**W1 & Q1. Comparison against existing video-based simulators.**\n\nThank you for your suggestion. In order to compare our V2V framework to a \"frame interpolation + event simulation\" approach, we conducted two new experiments based on the WebVid-100 dataset.\n\n1. Frame interpolation: We interpolated the WebVid-100 videos by 8x using the PerVFI [1] model. We selected this model since it was proposed recently (CVPR 2024) and code is available. If required, we can change it to other models.\n\n\t[1] Perception-Oriented Video Frame Interpolation via Asymmetric Blending. Guangyang Wu, Xin Tao, Changlin Li, Wenyi Wang, Xiaohong Liu, and Qingqing Zheng. In Proc. of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024.\n\n2. Event simulation: We used the popular V2E simulator to generate events. We used the \"clean\" noise mode. For the thresholds, we used the same random thresholds as in the \"fixed thresholds\" ablation experiment. Since the logarithm mapping of V2E is different from the ESIM simulator (our V2V framework follows ESIM), we scaled the thresholds linearly. \n\tIn particular, the logarithm mappings of V2E and ESIM are:\n\n\t```python\n\tdef ESIM_log(x):\n\t\t# The output has range [-6.908, 0], with range length 6.908.\n\t\treturn np.log(0.001 + x / 255.0)\n\n\tdef V2E_log(x):\n\t\t# The output has range [0, 5.545], with range length 5.545.\n\t\tif x <= 20:\n\t\t\treturn (1.0 / 20) * math.log(20) * x\n\t\telse:\n\t\t\treturn np.log(x)\n\t```\n\n\tSo, for a video corresponding to thresholds $c_+$ and $c_-$, we set the V2E thresholds to $c_+ * 5.545 / 6.908$ and $c_- * 5.545 / 6.908$, respectively.\n\n\tIn the simulation, a video (1066695484.mp4, with original size 1.2MB) produced 14GB of events. This caused our post-processing step to fail due to out-of-memory issues. Since the rebuttal stage is time-limited, we decided to discard this video, leaving us with a WebVid-99 dataset. The WebVid-99 dataset (in raw events) sums to 12 GB space, while the original WebVid-100 videos sum to 210 MB.\n\n3. Voxel stacking. We stacked the events into interpolated voxel bins. In this stage, we need to select the durations of the voxel bins. In the V2V experiments, each voxel corresponds to the events between 6 real frames, and the E2VID models reconstruct 24/5 FPS videos. We tried two binning strategies in our V2E experiments:\n\n\t- In strategy 1, we encode all the events between 1+1 original frames (crossing 7 interpolated frames) into a single voxel. This makes the E2VID models reconstruct 24 FPS videos, utilizing the high temporal resolution from frame interpolation. We annotate this as \"V2E-1\".\n\n\t- In strategy 2, we encode all the events between 5+1 original frames (crossing 35 interpolated frames) into a single voxel. This makes the E2VID models reconstruct 24/5 FPS videos, aligning to the V2V experiments. We annotate this as \"V2E-5\".\n\n4. Training and results. During training, we used the same augmentation noise (gaussian noise + hot pixels) as the ESIM protocol. For the V2E-5 experiment, we train for 8000 epochs, aligning to the V2V-WebVid-100 experiment. For the V2E-1 experiment, it produces 5x more training samples, so we train for 1600 epochs.\n\n\tThe test results are shown in the table below. The V2V-1 experiment produced better results than the V2V-5 experiment, showing that training on shorter bins may indeed benefit model performance. However, both V2E experiments produced metrics worse than the \"WebVid-100-Fixed\" experiment, which also uses fixed thresholds for each video. \n\n\t| Idx | Model | Train Dataset    | Loss    | Epochs | HQF MSE | HQF SSIM | HQF LPIPS | EVAID MSE | EVAID SSIM | EVAID LPIPS |\n\t| :-- | :---- | :--------------- | :------ | :----- | :------ | :------- | :-------- | :-------- | :--------- | :---------- |\n\t| (b) | E2VID | WebVid100-Fixed  | V+L1+TH | 8000   | 0.049   | 0.611    | 0.330     | 0.073     | 0.604      | 0.451       |\n\t| (e) | E2VID | WebVid100        | V+L1+TH | 8000   | 0.042   | 0.626    | 0.327     | 0.078     | 0.594      | 0.441       |\n\t| (u) | E2VID | WebVid99-V2E-1   | V+L1+TH | 1600   | 0.057   | 0.524    | 0.459     | 0.116     | 0.484      | 0.561       |\n\t| (v) | E2VID | WebVid99-V2E-5   | V+L1+TH | 8000   | 0.077   | 0.470    | 0.569     | 0.117     | 0.483      | 0.588       |\n\n**Q2. Simulation parameters.**\n\nDetails of the selection of simulation parameters are as follows.\n\n1. We first sample a threshold $c$ from the range [0.05, 2]. Note that we calculate logarithm images with `np.log(0.001 + video/255.0)` (following the ESIM simulator), which means that the amount of change between two frames is in the range of [0, 6.908].\n\n2. When using real event cameras, it is improbable that the positive and negative thresholds are too different, say 0.1 vs 1. So we uniformly sample the \"threshold ratio\" $r$ from the range [1, 1.5]. Then, with 50% probability, we set $c_+ = c, c_- = c * r$; with the other 50% probability, we set $c_+ = c * r, c_- = c$. The parameters $c_+$ and $c_- are the positive and negative thresholds, respectively.\n\n3. The standard deviation of the background gaussian distribution is uniformly sampled from the range [0, 0.1].\n\n4. The standard deviation of the hot pixel gaussian distribution is uniformly sampled from the range [0, 10].\n\n5. The fraction of hot pixels is uniformly sampled from the range [0, 0.001] (i.e., 0 to 0.1%).\n\nThese configuration details will be included in our code release to insure reproducibility.\n\n**Q3. Other grid representations.**\n\nThe idea of using the V2V framework on other representations by neglecting intra-bin differences is interesting. Intuitively, we believe that this could work. However, further experiments would be needed for validation."}, {"review_id": "k32zAv6iKl", "reviewer": "Reviewer_bJ8Q", "summary": "This paper investigates the potential of using video-based voxel generation to significantly enhance the training efficiency of event-to-video (E2V) models. The authors also introduce a large-scale synthetic dataset to facilitate this process.", "questions": "1.\tMy primary concern is that most of the experiments evaluate the joint effect of the V2V method and large-scale synthetic data. How can the effectiveness of the V2V approach itself be isolated and validated independently?\n\n2.\tThe performance gains on the EVAID dataset are significantly greater than those on the HQF dataset. In particular, the improvements in HQF are marginal. The authors should provide a more detailed discussion of the underlying reasons for this discrepancy.", "limitations": "The authors did not discuss the limitations of the techniques in the paper.", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n1. The construction of a large-scale, multi-scene event dataset is valuable, especially given the current limitations in real-world data.\n\n2. The paper addresses critical issues in event frame reconstruction, including storage and computational efficiency, from a novel and interesting perspective.\n\n\nWeaknesses:\n1. Converting video to event voxel representations is similar to the first stage of V2CE. While V2CE does not thoroughly explore this video-to-voxel conversion, a clearer comparison and discussion are necessary to clarify the novelty and contribution of the proposed method.\n\n    [1] Zhang Z, Cui S, Chai K, et al. V2ce: Video to continuous events simulator[C]//2024 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2024: 12455-12461.\n\n2. The paper does not further explore the model design for voxel-to-frame translation, which is unfortunate and significantly limits the technical depth of the work.\n\n3. The proposed dataset is synthetic. While large-scale, the lack of a large, real-world event dataset remains a key bottleneck in the field, and this work does not address that gap.\n\n4. While the authors demonstrate consistent improvements through pretraining on the proposed WebVid synthetic dataset, there is insufficient ablation or analysis to disentangle the benefits of the V2V approach from the benefits of large-scale pretraining.", "was_revised": true, "final_justification": "The author's rebuttal has largely addressed my concerns.", "rebuttal": "**Response to Reviewer bJ8Q**\n\nThank you for your detailed feedback. We will address your concerns below.\n\n**S1. Clarification of contributions.**\n\nThank you for your appreciation of our contributions. However, we would like to clarify that V2V is not a specific dataset, but a general framework for efficient event data simulation, especially for large scale datasets. It can be used to generate synthetic data for a large range of event-based vision tasks, including but not limited to video reconstruction and **optical flow estimation**.\n\n**W1. Similarities to existing simulators.**\n\nIt is indeed common for event simulators to first calculate the amount of events triggered on each pixel -- in other words, the voxel -- before assigning timestamps to the individual events. This paradigm can be found in V2CE, V2E, EventGAN and many other simulators. Our key novelty lies in removing the final, expensive voxel-to-event step. By arguing that the intermediate event stream is unnecessary for many deep learning pipelines, V2V directly generates discrete voxels, fundamentally improving efficiency and avoiding the ill-posed problem of interpolating continuous temporal data. We will add this discussion to our related works section.\n\n**W2. Insuffucient exploration of model design.**\n\nThe focus of our paper is to propose a **general** data framework that could boost model performance accross **different tasks** and **different models**. It would indeed be beneficial to explore specific model designs that can better utilize the V2V framework, but this is beyond the scope of our current work.\n\n**W3. Limitations of synthetic datasets.**\n\nOur V2V framework indeed produces synthetic data, and does not provide large-scale real data. However, compared to existing data synthesization methods, the V2V framework scales up the amount of realistic video-based synthetic data, and also allows for more flexible data augmentation methods. This help boosts the performance and robustness of the models trained, which is the source motivation of pursuing large-scale real datasets.\n\n**W4 & Q1. Disentanglement of V2V framework & data scaling.**\n\nThe key advantages of our V2V framework are efficiency, scalability and flexibility. The efficiency and scalability contribute to the model performance by allowing the use of large-scale video datasets, so their contributions cannot be disentangled from data scaling. The flexibility, however, can be disentangled and verified.\n\nThe flexibility refers to the variety of data augmentation methods applicable when using the V2V framework. For example, in each epoch, a video can be combined with different event thresholds and noise levels, serving as different training samples. Existing event simulators do not support changing thresholds after simulation, so we can evaluate the benefits of the V2V framework by disabling the threshold changing functionality.\n\nSpecifically, to disable the threshold changing functionality, we randomly sampled threshold values for each video, and fixed them throughout the training process. We annotate these ablation experiments as \"V2V-Fixed\" yellow dots in Figure 5. The full methods, with thresholds randomly sampled in each epoch, are annotated as \"V2V-Random\" green dots.\n\nAs visualized in Figure 5 (qualitative results are in Table 2), disabling the threshold changing functionality leads to a performance drop, no matter which scale the training dataset is (100 -> 1K -> 10K). This verifies that the flexibility of the V2V framework contributes to the model performance, disentangled from data scaling.\n\n**Q2. Different performance gains over EVAID and HQF.**\n\nThe more significant gains on EVAID compared to HQF stem from EVAID being a more challenging and higher-quality dataset, offering more room for improvement.\n\n1. EVAID has a higher frame rate (18 ~ 150 FPS) than HQF (17 ~ 26 FPS). This requires models to generalize to higher temporal resolutions and sparser event streams.\n\n2. EVAID is captured with a Prophesee EKV4 event camera, with resolution 720x1280. HQF is captured with DAVIS346, with resolution 346x260. To gain good performance on both EVAID and HQF, the model would need to generalize to higher spatial resolutions, and also generalize to the different noise characteristics.\n\n3. There are long periods in EVAID where the camera is static, and no events are generated in the background (an example begins 01:40 of the supplementary video). This requires the model to \"remember\" the background information reconstructed in the beginning, and distinguish the texture it needs to remember from the noise it should gradually smooth out. \n\nAs a result, existing models meet more challenges when tested on EVAID, while the metrics over HQF are near saturation. Hence, our V2V-retrained versions, with better generalization capabilities, can achieve more significant performance gains over EVAID than HQF.\n\n**L1. Missing limitations.**\n\nDue to the space limit, we moved our limitations section to the appendix (Section G). We apologize for the resulting inconvenience."}]

Author rebuttal:

Meta-review: This paper proposes a method to generate event-based training data from conventional RGB video, by converting a video into (synthetic) event voxel grids. The paper received a consensus of positive reviews (4, 5, 5, 4). The reviewers appreciate the main idea of the work, and appear excited by the possibility of training event camera models on internet-scale data. While they note that the current set of experiments is still limited in scope and size, they find that the experiments added in the rebuttal improved their impression of the work's value, and everyone's overall opinion is positive. The authors also promised to make minor revisions to the manuscript, clarifying the limitations and reorganizing the results for clarity. Based on all this, the AC recommends acceptance.
Decision: Accept (poster)