翻译：Introduction
Video frame prediction (VFP) aims to predict future frames
from previous frames, which has broad applications in autonomous
 driving, robotics planning and weather forecasting.
 Existing VFP methods usually take previous image sequences
 as input to predict a sequence of future frames
with the same framerate as the inputs (simplifed diagram in
Fig. 1). By exploiting various network architectures, the performance
 of VFP has been signifcantly improved. However,
due to the limitations of frame-based cameras in capturing
complex scenes, it is still challenging by only exploiting the previous frames. In Fig. 4 and Fig. 5, even the state-ofthe-art
 VFP methods, their performances deteriorate quickly
as the motion complexity increases. This is mainly due to
the fxed framerate inputs, which limit the ability to capture
 complex dynamics beyond the sampling frequency. Another
 major issue is that they cannot predict future frames at
fexible time intervals, i.e., consecutive in-between frames or
across multiple frames. These factors limit the performance
and real-world applications of existing VFP methods.
The event camera, as a new kind of bio-inspired sensor,
 can capture the asynchronous brightness change of each
pixel. As the event camera captures high-speed motion with
a low data bandwidth and a high temporal resolution with
millisecond accuracy, it provides critical and complementary
 information to images and is widely used in motion estimation
 (Ding et al. 2022; Wan et al. 2023), object tracking
 (Wang et al. 2023) and etc. In this paper, we investigate a
new and minimal setup for event-enhanced video frame prediction,
 where we predict the next frames from a single RGB
image (providing contextual information) and the following
events (providing rich motion information).
Apparently, this task is closely related to video frame interpolation
 (VFI) with events, where the intermediate frames
are interpolated by exploiting two frames and the events inbetween.
 In Fig. 1, subject to the interpolation formulation,
these methods require image and event data after time t to
generate the frame at time t. This violates the causality constraint
 in frame generation and limits their applications in
any practical systems. With this precedent of successfully
introducing events to VFI, we believe that combining events
also has a signifcant effect on improving the usability of
VFP in complex dynamic scenarios.
In this paper, we propose a lightweight network that can
predict a 720P frame within 25ms on an RTX2080Ti GPU.
We frst use two respective encoders to extract pyramid features
 from the input reference image and event representation.
 To complementarily utilize the characteristics of image
and event data, we design a symmetrical cross-modal attention
 module to augment these two features. Then we refne
the synthesized feature and optical fow in a coarse-to-fne
joint estimation way. To resolve the holes arising from forward
 warping, we present an inpainting module that can repair
 the holes without bringing lots of extra computation.
Finally, we adopt a weighted fusion to output the fnal frame
prediction from the synthesized and warped frames. Thanks
to the sparse events that can be divided into multiple time
segments, the training data we can use covers various motion
ranges and time intervals. Therefore, by adjusting the end
times of input events, our model can predict high-framerate
frames as well as frames for a long time, whereas the framebased
 VFP models cannot because their predicted framerates
 need to be consistent with the input framerates. We conduct
 experiments on both synthetic and real datasets, and the
PSNR is improved by over 3.5dB on GoPro compared to the
state-of-the-art frame-based and event-based VFP methods,
which demonstrates the effectiveness of our model in solving
 the VFP problem.
Our main contributions are summarized as follows:
1) We introduce a minimal practical confguration to introduce
 events for the VFP tasks, i.e., predicting future
frames from a single image and events.
2) We propose a lightweight model with symmetrical crossattention
 augmentation and hole inpainting module,
which can predict a future frame from a single image and
events within real-time requirements.
3) Experiments on both synthetic and real-captured datasets
prove the effectiveness and effciency of our approach in
predicting fexible future video frames.