Continuous Space-Time Video Super-Resolution via Event Camera

## Abstract

Abstract Continuous space-time video super-resolution (C-STVSR) aims to simultaneously enhance video resolution and frame rate at an arbitrary scale.
Recently, implicit neural representation (INR) has been applied to video restoration, representing videos as implicit fields that can be decoded at an arbitrary scale.
However, the highly ill-posed nature of C-STVSR limits the effectiveness of current INR-based methods: they assume linear motion between frames and use interpolation or feature warping to generate features at arbitrary spatiotemporal positions with two consecutive frames.
This restrains C-STVSR from capturing rapid and nonlinear motion and long-term dependencies ( involving more than two frames ) in complex dynamic scenes.
In this paper, we propose a novel C-STVSR framework, called HR-INR , which captures both h olistic dependencies and r egional motions based on INR.
It is assisted by an event camera – a novel sensor renowned for its high temporal resolution and low latency.
To fully utilize the rich temporal information from events, we design a feature extraction consisting of (1) a regional event feature extractor – taking events as inputs via the proposed event temporal pyramid representation to capture the regional nonlinear motion and (2) a holistic event-frame feature extractor for long-term dependence and continuity motion.
We then propose a novel INR-based decoder with spatiotemporal embeddings to capture long-term dependencies with a larger temporal perception field.
We validate the effectiveness and generalization of our method on four datasets (both simulated and real data), showing the superiority of our method.

## 1 Introduction

The real world’s visual information, e.g., edge and object motion, is continuous, spanning both time and space dimensions.
However, the limited I/O bandwidth and sensor size of modern systems [, , ] confines us to record videos at low frame rates and fixed resolutions.
This limitation has profound repercussions across various computer vision applications, e.g., encompassing immersive experiences in virtual reality [, ], traffic analysis in autonomous driving [, ].
To address this limitation, recent research works [, ] have explored restoring videos with continuous resolutions and frame rates, referred to as Continuous Space-Time Video Super-Resolution (C-STVSR).

Recently, implicit neural representation (INR) has been applied to video restoration: it represents images or videos as neural fields that can be decoded at any resolution with a pointwise MLP decoder [, , , , , ].
One of the seminal INR approaches is LIIF [], which is designed for arbitrary-scale image SR.
This line of research soon extended to the video domain.
In this context of C-STVSR, VideoINR [] employs a fixed STVSR model [] that extracts features from two consecutive video frames.
Then, it introduces a temporal INR to generate inverse backward warping optical flow [] to warp features.
Lastly, it employs a spatial INR, similar to LIIF, to decode the RGB frame with arbitrary resolution.
Building upon this, MoTIF [] improves VideoINR by using forward motion estimation, reducing gaps and holes in the temporal INR, which are typically caused by the randomness and discontinuities associated with backward warping [, ].
These methods depend solely on two successive RGB frames, rendering the task of predicting inter-frame motions ill-posed.
Consequently,, it becomes challenging to accurately capture highly dynamic motion (e.g., regional high-speed or nonlinear movements) and to model long-term dependencies that extend across more than four frames.

Figure: Figure 1: With event data as guidance, our method (HR-INR) takes in videos with low frame rates and resolution (a) and produces continuous space-time videos with arbitrary frame rate and resolution (b). As shown in (c), our method is able to recover the rotation of the bicycle wheels, which is unachievable by the prior method VideoINR [].

Motivation and Contributions.
Event cameras are bio-inspired sensors, known for their high temporal resolution and low latency ($<1us$) [, , ].
Recent research has demonstrated the potential of events in guiding various video super-resolution (VSR) [, ] and video frame interpolation (VFI) tasks [, , , ].
However, utilizing events to facilitate joint video super-resolution and frame interpolation is a challenging area yet to be explored.

This paper introduces HR-INR, a novel INR-based method that leverages events for jointly guiding VSR and VFI.
It adeptly captures regional, rapid motion and holistic, long-range motion dependencies, as shown in Fig. 1.
To capture the regional motion, we propose Temporal Pyramid Representation (TPR) to construct a time-series pyramid structure around the pivotal timestamp of events (Sec. 3.1).
Different from the evenly divided timeline representations, like voxel grids [, , ], time surfaces [], time moments [, ] and symmetric cumulative [], TPR offers finer temporal granularity with less complexity and effectively captures rapid motion changes, as shown in Fig. 2 (a).
Additionally, our method is capable of processing multiple frames and their associated events, which empowers it to estimate holistic, long-range motion that involves more than two frames.

Accordingly, we design two specialized feature extractors: the regional event feature extractor (RE) and the holistic event-frame feature extractor (HE), see Sec. 3.2.
Both extractors are grounded in the Swin-Transformer architecture [, , ], renowned for its efficacy and efficiency in video enhancement tasks.
RE is a lightweight network designed specifically for extracting local information from our event TPR.
Meanwhile, HE employs a more sophisticated approach, utilizing long-term and multi-scale fusion strategies to integrate both events and frames.
Consequently, our training and inference strategy requires HE to be invoked only once for multi-frame interpolation.
Subsequently, the extracted features from RE and HE are fused as the output of the feature extraction module.

After fusing the regional and holistic features, we propose a novel INR-based spatial-temporal decoding module (Sec. 3.3).
Our motivation is to avoid gaps and holes typically found in optical flow-based warping and multi-frame fusion, as identified in previous research [, , , ].
To accomplish this, we propose an implicit temporal embedding designed to transform timestamps into focused attention vectors on long-term features.
This approach ensures attention is also given to long-distance dependencies, which is crucial for effectively modeling long-term temporal dependencies.
Subsequently, inspired by LIIF [], we employ spatial embedding to achieve arbitrary up-sampling in the spatial dimension.

We conducted experiments on two simulated and two real-world datasets. The results validate the superiority of our method and its excellent generalization capabilities on real-world datasets.
Our approach is the first event-based method to achieve continuous space-time video super-resolution, surpassing frame-based methods, as shown in Fig. 1. It also excels in individual VSR and VFI metrics compared to previous event-based methods.

## 2 Related Works

Space-time Video Super-Resolution
aims to enhance the resolution and frame rate of a video simultaneously [, , , ]. In comparison to two-stage solutions, where VFI [, , , , ] and VSR [, , , , , ] methods are applied sequentially, simultaneous space-time video super-resolution reduces cumulative errors and leverages the natural relations between VFI and VSR methods.
Zooming Slow-Mo [] uses temporal interpolation to generate missing frames and aligns temporal information using a deformable ConvLSTM network.
Similarly, TMNet [] extracts short-term and long-term motion cues in videos by modulating convolution kernels.
However, these methods cannot simultaneously achieve spatiotemporal resolution across arbitrary scales.

INR for VFI and VSR
have achieved space-time video super-resolution with arbitrary resolutions [, ] by learning videos implicit neural representations (INRs).
These methods primarily estimate optical flows from two consecutive frames to warp features into arbitrary space-time coordinates, which are then decoded using MLP layers.
However, by relying on just two consecutive frames, these methods cannot inherently model long-term motions (involving three or more frames) and fail to accurately capture local, inter-frame, non-linear motions due to missing inter-frame information.

Event-guided VFI and VSR
seek to boost performance by incorporating the biologically inspired event cameras [].
Previous works have demonstrated the potential of event-guided VFI, which mainly focus on modeling non-linear motion with events [, , , , , , ].
EFI [] exclusively adopts the synthesis approach for intermediate frame generation.
TimeLens [] and TimeLens++ [] employ events to model nonlinear motion correlations, integrating both synthesis and warping-centric approaches.
Building on these advancements, CBM-Net [], introduces a motion field to handle complex movements.
However, while these VFI methods utilize events to capture inter-frame motion, they fail to establish long-term dependencies beyond two frames and support simultaneous VSR.
The realm of event-guided video super-resolution has also been explored.
E-VSR [] highlights that high-frequency temporal information from events is beneficial to recovering high-frequency spatial information.
Like our work, EG-VSR [] employs events to comprehend INR, allowing for video ups-sampling with arbitrary scale.
However, the INR of EG-VSR cannot interpolate frames.
Contrasting these methods, we pioneer using events to enable concurrent VSR and VFI across arbitrary spatial-temporal scales, i.e.,, C-STVSR.

Video Long-term Dependence Modeling is a crucial aspect of VSR and VFI.
For instance, BasicVSR [] and BasicVSR++[] enhance VSR performance by processing multi-frame inputs through an alignment module to model long-term motion correlations.
Similarly, in the VFI domain, many methods [, , ] employ RNN or LSTM to model sequences of frames, capturing long-term dependencies effectively.
Furthermore, Zooming Slow-Mo [], TMNet [], and RSST [] leverage multi-frame inputs in the joint task of VSR and VFI, showcasing the importance of integrating multiple frames for improved modeling of video dynamics.
However, current C-STVSR methods [, ], and event-based VFI methods [, , , ], primarily rely on estimating optical flow between two consecutive frames.
Therefore, they are challenging to handle multi-frames as inputs, inherently undermining their capability to model long-term dependencies.

Figure: Figure 2: Overview of our framework. The inputs are multi-frame images and their corresponding events. The output is a video with enhanced frame rates and resolutions. Firstly, events proximate to a particular time point are transformed into Temporal Pyramid Representations (TPR) to capture motion at a more granular temporal level (a). Secondly, TPRs, the comprehensive set of multi-frames and events, are directed into the feature extraction part (b). Within this part, the Regional Events Feature Extractor and the Holistic Events Feature Extractor process the input separately. Lastly, the resulting features are then fused and inputted into an INR-based spatiotemporal decoding part (c). Within this part, a temporal embedding is executed to capture features at a specific timestamp $t$, followed by spatial embedding with an up-sampling factor $s$ and decoding, culminating in the generation of frames at the desired resolution.

## 3 Proposed Framework

Our proposed HR-INR framework is depicted in Fig. 2.
The inputs of this framework are RGB frames $\boldsymbol{I}_{in}=\{I_{in}\}_{i}^{N_{in}}\in R^{N_{in}\times H\times W\times
3}$ and associated events $\boldsymbol{E}$.
$H$ and $W$ denote the spatial resolution of frames and events.
$3$ means three channels of RGB.
$N_{in}$ denotes the input number of frames.
Furthermore, the framework outputs a video with an arbitrary frame rate and spatial resolution.
In particular, we consider the output video as $\boldsymbol{I}_{out}=\{I_{out}\}_{i}^{N_{out}}$, consists of $N_{out}$ frames, each with a resolution of $(s\times H)\times(s\times W)$, where $s$ represents the up-sampling scale greater than $1$.
For the output $N_{out}$ frames, we denote the time corresponding to each frame as $\boldsymbol{T}=\{t\}_{i}^{N_{out}}$.
For convenience, we also record the up-sampling scale $s$ and the time $\boldsymbol{T}$ as a part of inputs.
Therefore, the mapping function $f_{hr}(.)$ of C-STVSR can be described by Eq. 1.

$$ $\boldsymbol{I}_{out}=f_{hr}\left(\boldsymbol{I}_{in},\boldsymbol{E},s, \boldsymbol{T}\right)$ (1) $$

Our framework comprises three main components:
First, Sec. 3.1 presents the event temporal pyramid representation (TPR), capturing regional dynamic motion and edges.
Second, Sec. 3.2 elaborates on the feature extraction process using regional and holistic feature extractors.
Third, Sec. 3.3 describes the INR-based spatiotemporal decoding.

Input Frames and Events:
Our input frames, $\boldsymbol{I}_{in}$, consist of multiple frames with timestamps normalized to the $[0,1]$ interval. We consider the events, $\boldsymbol{E}$, occurring within this time range.
Each event point can be represented by $(x,y,t,p)$, signifying a change in pixel intensity at coordinates $(x,y)$ at time $t$; here, $p$ is $+1$ for increased brightness and $-1$ for a decrease.
Benefiting from the event camera’s high temporal resolution and low latency ($<1us$), these event points are effectively considered continuous along the timeline.
For a detailed exposition of the principles of event generation, please refer to the Suppl. Mat..

### 3.1 Temporal Pyramid Representation

Firstly, we represent the event stream $\boldsymbol{E}$ into a frame-like form that the network can process.
To capture holistic motion, we partition all events during $[0,1]$ of the timeline into $M$ equal intervals using a voxel grid [, ], denotes as $\boldsymbol{E}_{v}$ with dimensions $M\times H\times W$.
In practice, event representation methods like voxel grid [, ] and its extended structure, symmetric cumulative event representation [], achieve time granularity by uniformly dividing time into intervals with resolutions of $1/M$.

However, for C-STVSR, capturing intricate motion and edges requires a finer granularity.
Merely increasing $M$ to enhance detail sharply raises computational costs; for instance, capturing $1/1000$ second intervals within a second necessitates expanding $M$ to $1000$, which is computationally impractical.
To address this, we introduce Temporal Pyramid Representation (TPR), leveraging the high temporal resolution of events while reducing representation complexity.

Our idea: The core of TPR is constructing a temporal pyramid where each successive layer’s duration is $1/r,(r>1)$ of the preceding one, leading to exponentially finer time granularity with additional layers.
For instance, as illustrated in Fig. 2 (a), around any given time $t$, we define a surrounding time window $\Delta t$ and select an attenuation factor $r$.
At the pyramid’s $L$-th level, the events are within the time span of $[t-\Delta t/r^{L},t+\Delta t/r^{L}]$.
Each layer is further segmented into $M_{p}$ intervals, represented using a voxel grid.
Accordingly, for an $L$-th layer, each layer contains $M_{p}$ moments within the TPR, and its finest time granularity, denoted by $\delta_{t}$, is as delineated in the Eq. 2:

$$ $\delta_{t}=\frac{2\times\Delta t}{M_{p}\times r^{L}}$ (2) $$

Therefore, for any time $t$, we construct the corresponding TPR $E_{p}$ with shape $L\times M_{p}\times H\times W$.
We record the TPRs at all target timestamp as $\boldsymbol{E_{p}}=\{E_{p}\}_{i}^{N_{out}}$.

Discussion:
The time granularity $\delta_{t}$ of TPR exponentially improves with the increase in layers, $L$.
For an attenuation factor of $r=3$ and a goal to detect motions as brief as $1/1000$ of a second within a $1s$ window ($2\times\Delta t=1$), we require only $7$ TPR layers with each layer divided into $2$ intervals ($M_{p}=2$).
Consequently, a TPR with dimensions $7\times 2\times H\times W$ suffices to discern motion down to $1/1000s$.
Based on the above representation, we obtained $\boldsymbol{E}_{v}$, encapsulating holistic motion, and $\boldsymbol{E}_{p}$, which focuses on regional edges and motion.

### 3.2 Holistic-Regional Feature Extraction

This module aims to extract features from regional TPRs $\boldsymbol{E_{p}}$, and the frame $\boldsymbol{I_{in}}$ and the holistic events $\boldsymbol{E_{v}}$ for INR-based spatial-temporal decoding.
Accordingly, we introduce:
(1) the regional event feature extractor, $f_{re}$, for dynamic motion and edges detail capture.
(2) the holistic event-frame feature extractor, $f_{he}$, for long-term motion dependencies modeling across time and space.

Figure: Figure 3: Holistic event-frame feature extractor. The down-sample module will halve the resolution. The up-sample module will double the resolution. The encoder and decoder have the same structure as Swin-Transformer [, , ].

Regional Event Feature Extraction:
The input of $f_{re}$ is TPR $E_{p}\in R^{L\times M_{p}\times H\times W}$ in each timestamp $t$.
Given that $f_{re}$ is invoked $N_{out}$ times, its design needs to strike a balance between efficiency and the ability to model inter-level relationships to ensure that it can capture precise regional motion and edge details effectively.
Initially, our method applies a convolution layer to increase the dimensions of the feature.
Then, we use four Swin Transformer Encoder Blocks (STEB) [, , ] to model the relationship between different pyramid levels.
The STEB enjoys a large view field and a multi-head attention mechanism that proficiently models distant dependencies and conveys edge information across various pyramid levels, effectively capturing short-term motion.
Notably, STEB is optimized with fewer parameters, boosting computational efficiency.

Holistic Event-Frame Feature Extraction:
The inputs of $f_{he}$ is $\boldsymbol{I_{in}}$ and $\boldsymbol{E_{v}}$, as shown in Fig. 3.
First, a convolutional layer processes both frames and events to increase dimensions.
Motivated by the events feature manifest between successive frames to provide inter-frame motion information.
We adjusted the first dimension of the event feature to $N_{in}-1$ to account for inter-frame motion losses.
We then incorporate the STEB to facilitate interactions across varied levels and spatial domains.
To minimize computational overhead and expand the receptive field, we integrated a down-sampling module between STEBs, forming a multi-scale encoder.
Each down-sampling iteration halves the resolution while maintaining the channel dimensions.
After three iterations, the feature resolution reduces to $1/8$ of its initial size, enlarging the receptive field.
We then employ a Swin Transformer Decoder Block [, , ] to fuse features at matching resolutions.
Each of the first three STEBs is followed by an upsampling process, which doubles the resolution while maintaining the channel count.
Ultimately, this process outputs the feature $F_{g}$.
For more details, see Suppl. Mat..

Notably, to output $N_{out}$ frames, the holistic event-frame feature extractor $f_{he}$ is called once, capturing the comprehensive feature $F^{g}$.
Subsequently, for each time $t$, $f_{re}$ extracts regional features $F_{t}^{l}$ from each TPR $E_{p}\in\boldsymbol{E_{p}}$.
For each regional feature $F_{t}^{l}$, we use addition and Cov$1\times 1$ operation as fusion function $f_{fu}$ to fuse with holistic feature $F^{g}$ to obtain the output $R_{t}$.
For each time $t$, the whole process can be described by Eq. 3, where $E_{p}\in\boldsymbol{E_{p}}$ is the TPR at the specific timestamp $t$.

$$ $\begin{split}R_{t}=&f_{fu}\left(F^{g},F_{t}^{l}\right);F^{g}=f_{he}\left( \boldsymbol{I_{in}},\boldsymbol{E_{v}}\right);F_{t}^{l}=f_{re}\left(E_{p} \right)\end{split}$ (3) $$

Figure: Figure 4: Temporal embedding. Given the input time $t\in[0,1]$, the output is the temporal attention $E_{t}$ derived from a two-layer MLP. (b) presents a visualization of the trained $E_{t}$ during $[0,1]$ on real-world dataset [].

### 3.3 INR-based Spatial-Temporal Decoding

In this section, we employ INR-based spatial-temporal decoding to effectively retrieve RGB frames at any desired time and resolution. To achieve this, we leverage a temporal INR to generate features at any timestamp and a spatial INR to upscale the features to any spatial resolution.

Temporal Embedding:
We utilize learned temporal embedding as attention vectors to aggregate the fused feature in the channel dimension in a time-specific manner.
At a given timestamp $t$, we first use a two-layer MLP to increase its dimension to $C_{t}$, resulting in a temporal attention vector (as illustrated in Fig. 4 (a)).
The visualization of the learned temporal attention is depicted in Fig. 4 (b), exhibiting variations across both time and channel dimensions.
This attention vector is then multiplied directly with $R_{t}$ to generate the temporal embedded feature $R_{ts}$.
This temporal INR allows for a larger temporal perception field without the need for estimating optical flows.
Next, a $1\times 1$ convolution is applied to compress $R_{ts}$ to $C_{ts}$ dimensions, reducing the complexity of the spatial embedding and decoding.

Spatial Embedding and Decoding:
To upscale the temporal embedded features to any desired spatial scale, we utilize a similar approach to previous works [, , ].
We query the four nearest neighbors in the temporal embedded feature for each spatial coordinate and concatenate these with their distances for spatial embedding.
A four-layer MLP decoder computes the RGB values, which are then aggregated through area-weighted interpolation for arbitrary-scale super-resolution.
Similar to [], we use the Charbonnier loss [] as the fundamental loss function for training.

Figure: Figure 5: 7-skip frame interpolation visualization results in Adobe240 dataset []. Our method (c) more effectively captures the rotating wheels compared to the VideoINR (b), which tends to show noticeable holes and gaps. Green circles highlight obvious holes and gaps.

## 4 Experiments

To facilitate a comprehensive comparison between the frame-base [, ] and event-guided methods [, , , ], we employed two simulated datasets [, ] and two real-world datasets [, ] for our experiments
(1) Adobe240 dataset [] includes 133 videos, each with $720\times 1280$ resolution and a $240fps$ frame rate.
Following established protocols [, , ], the dataset is divided into 100 training, 16 validation, and 17 testing subsets.
We employed the widely-used event simulation method vid2e [], which accounts for real noise distribution, enhancing our model’s robustness and generalization capabilities.
(2) GoPro dataset [] featuring the same resolution and frame-rate with Adobe240 dataset, comprises 22 training and 11 test videos.
Given its compact size, previous studies [, ] have primarily employed the test set for quantitative analysis.
We follow this to be consistent with established practice.
For datasets (1) and (2) during 7-skip frame interpolation, we input 4 frames where each pair of adjacent frames is separated by 7 frames, allowing a total of 25 frames to serve as GTs.
(3) BS-ERGB [] recorded using a spectroscope, includes real-world paired events and frames. Previous studies, e.g., TimeLens++[], initially pre-trained on the Vime90k simulation dataset[] before fine-tuning on BS-ERGB.
In contrast, we opted to pre-train our model on the Adobe240 dataset before fine-tuning.
Notably, the Vime90k dataset is larger in scale than the Adobe240 dataset.
During fine-tuning, we also used perceptual loss [] with weight $0.1$ to be consistent with previous methods [, ] for fair comparison.
(4) CED [] is a real-world dataset in VSR. To fairly compare the previous research [, ], only this data set is used for training for VSR comparison.

Implementation Details:
Our model is trained using Pytorch [], employing the Adam optimizer [].
Referring to the VideoINR [], our training consists of two stages.
(1) Train frame interpolation under a fixed spatial up-sampling ($4\times$), over 70 epochs, starting with a learning rate of $5e-4$.
(2) Train frame interpolation under random space upsampling rate in $\mathcal{U}(1,8)$, spanning 30 epochs with the learning rate $5e-5$.
We randomly choose 20 frames from a pool of 25 frames as ground truth.
Data augmentation is implemented via Random Crop, extracting $512\times 512$ areas from frames and events, with the input resolution dynamically determined by a randomly chosen upsampling ratio $s$.
To optimize memory usage and accelerate speed, we implemented the mixed precision strategy [, ].
All experiments are performed on an NVIDIA A800 computing card.
Please refer to the Suppl. Mat. for more details.

Evaluation:
To ensure the fairness, we adopted PSNR [], SSIM [], and LPIPS [] as quantitative evaluation metrics.
Aligning with prior works [, ] for consistency, we use only the $Y$-channel for GoPro and Adobe datasets and all three RGB channels for BS-ERGB and CED datasets.

**Table 1: Quantitative metrics (PSNR/SSIM) with 7-skip VFI and $4\times$ VSR. Center Average remain consistent with previous work [, ].**
| VFI | VSR | Params ($M$) |  | GoPro-Center | GoPro-Average | Adobe-Center | Adobe-Average |
| --- | --- | --- | --- | --- | --- | --- | --- |
|  | Bicubic | 19.8 |  | 27.04/0.7937 | 26.06/0.7720 | 26.09/0.7435 | 25.29/0.7279 |
| Su-SloMo [] | EDVR [] | 19.8+20.7 |  | 28.24/0.8322 | 26.30/0.7960 | 27.25/0.7972 | 25.95/0.7682 |
|  | BasicVSR [] | 19.8+6.3 |  | 28.23/0.8308 | 26.36/0.7977 | 27.28/0.7961 | 25.94/0.7679 |
| \hdashline | Bicubic | 29.2 |  | 26.50/0.7791 | 25.41/0.7554 | 25.57/0.7324 | 24.72/0.7114 |
| QVI [] | EDVR [] | 29.2+20.7 |  | 27.43/0.8081 | 25.55/0.7739 | 26.40/0.7692 | 25.09/0.7406 |
|  | BasicVSR [] | 29.2+6.3 |  | 27.44/0.8070 | 26.27/0.7955 | 26.43/0.7682 | 25.20/0.7421 |
| \hdashline | Bicubic | 24.0 |  | 26.92/0.7911 | 26.11/0.7740 | 26.01/0.7461 | 25.40/0.7321 |
| DAIN [] | EDVR [] | 24.0+20.7 |  | 28.01/0.8239 | 26.37/0.7964 | 27.06/0.7895 | 26.01/0.7703 |
|  | BasicVSR [] | 24.0+6.3 |  | 28.00/0.8227 | 26.46/0.7966 | 27.07/0.7890 | 26.23/0.7725 |
| Zooming Slow Mo [] | 11.10 |  | 30.69/0.8847 | -/- | 30.26/0.8821 | -/- |  |
| TMNet [] | 12.26 |  | 30.14/0.8692 | 28.83/0.8514 | 29.41/0.8524 | 28.30/0.8354 |  |
| Video INR-fixed [] | 11.31 |  | 30.73/0.8850 | -/- | 30.21/0.8805 | -/- |  |
| Video INR [] | 11.31 |  | 30.26/0.8792 | 29.41/0.8669 | 29.92/0.8746 | 29.27/0.8651 |  |
| MoTIF [] | 12.55 |  | 31.04/0.8877 | 30.04/0.8773 | 30.63/0.8839 | 29.82/0.8750 |  |
| HR-INR (Ours) | 8.27 |  | 31.97/0.9298 | 32.13/0.9371 | 31.26/0.9246 | 31.11/0.9216 |  |

**Table 2: More quantitative comparisons using PSNR/SSIM on the GoPro dataset.**
| Temporal<br>Scale | Spatial<br>Scale | Su-SloMo []<br>+ LIIF [] | DAIN []<br>+ LIIF [] | TMNet [] | Video INR [] | MoTIF [] | Ours |
| --- | --- | --- | --- | --- | --- | --- | --- |
| 12$\times$ | 4$\times$ | 25.07/0.7491 | 25.14/0.7497 | 26.38/0.7931 | 27.32/0.8141 | 27.77/0.8230 | 28.87/0.8854 |
| 12$\times$ | 6$\times$ | 22.91/0.6783 | 22.92/0.6785 | - | 24.68/0.7358 | 26.78/0.7908 | 27.14/0.8173 |
| 16$\times$ | 4$\times$ | 24.42/0.7296 | 24.20/0.7244 | 24.72/0.7526 | 25.81/0.7739 | 25.98/0.7758 | 27.29/0.8556 |
| 16$\times$ | 6$\times$ | 23.28/0.6883 | 22.80/0.6722 | - | 23.86/0.7123 | 25.34/0.7527 | 26.09/0.7954 |
| 6$\times$ | 1$\times$ | - | - | - | 32.34/0.9545 | 34.77/0.9696 | 38.53/0.9735 |

### 4.1 Comparison Experiments

Space-time Super-resolution:
We conduct space-time super-resolution comparison experiments on the Adobe240 and GoPro datasets.
We categorized the comparison methods into three groups.
(I) Frame-based cascade methods: VFI methods, e.g., Super SloMo [] and DAIN [] followed by VSR methods, e.g., EDVR [] and BasicVSR [].
(II) Fixed STVSR methods: e.g., Zooming Slow-Mo [] and TMNet [].
(III) Frame-based C-STVSR methods: VideoINR [] and MoTIF [].
The numerical comparison is presented in Tab. 1. It is evident that C-STVSR methods consistently outperform cascade and fixed STVSR methods.Our method achieves the highest performance in both datasets with the smallest model size.
In the GoPro dataset, our method improves the center frame by 0.93 $dB$ and 0.0415 SSIM, and on average by 2.09 $dB$ and 0.0598 SSIM compared to the best method, MoTIF [].
Similarly, in the Adobe240 dataset, our method outperforms MoTIF [] by 0.63 $dB$ and 0.0407 SSIM for the -center, and on -average by 1.29 $dB$ and 0.0466 SSIM.
It is worth noting that the difference between our method and other methods is more pronounced for the -average frames than the -center frame.
This observation suggests that our method demonstrates enhanced temporal stability and superior adaptability across varying interpolation intervals, an advantage not shared by previous methods [, ].
For more analysis, please refer to Sec. 4.2 and Fig. 8 in appendix.

**Table 3: Temporal super-resolution results, i.e.,, VFI, on BS-ERGB dataset [].**
|  |  |  | 1-skip | 3-skip |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Methods | Params ($M$) | Event | PSNR $\uparrow$ | SSIM $\uparrow$ | LPIPS $\downarrow$ | PSNR $\uparrow$ | SSIM $\uparrow$ | LPIPS $\downarrow$ |
| FLAVR [] | - | ✗ | 25.95 | - | 0.086 | 20.90 | - | 0.151 |
| DAIN [] | 24.0 | ✗ | 25.20 | - | 0.067 | 21.40 | - | 0.113 |
| Super SloMo [] | 19.8 | ✗ | - | - | - | 22.48 | - | 0.115 |
| QVI [] | 29.2 | ✗ | - | - | - | 23.20 | - | 0.110 |
| TimeLens [] | 72.2 | ✓ | 28.36 | - | 0.026 | 27.58 | - | 0.031 |
| TimeLens++ [] | 53.9 | ✓ | 28.56 | - | 0.022 | 27.63 | - | 0.026 |
| CBMNet [] | 15.4 | ✓ | 29.32 | 0.815 | - | 28.46 | 0.806 | - |
| CBMNet-Large [] | 22.2 | ✓ | 29.43 | 0.816 | - | 28.59 | 0.808 | - |
| HR-INR (Our) | 8.3 | ✓ | 29.66 | 0.828 | 0.011 | 28.59 | 0.814 | 0.021 |

Tab. 2 presents additional comparative experiments for arbitrary spatial and temporal super-resolution. Our method consistently outperforms other methods, even in extreme space-time upsample scales, such as $16\times$ temporal upscale and $6\times$ spatial upscale.
The visualization results, Fig. 1 (c) and Fig. 5, also demonstrate that our method effectively models regional nonlinear motion, e.g., the wheel rotation — a capability not achieved by previous method VideoINR [].
For more visualization results, please refer to the Suppl. Mat.

Figure: Figure 6: 3-skip frame interpolation visualization results in BS-ERGB dataset []. Our method accurately captures local non-linear motion (e.g., balls in (1.d) and (2.d)), surpassing TimeLens [], which exhibits ghosting and holes (green circles). A yellow arrow shows TimeLens’s inaccurate ball positioning.

Separate Comparison of Event-based VFI and VSR:
We also compared our method with previous approaches in separate VFI [, , ] and VSR [, ] tasks.
The results can be seen in Tab. 3 and Tab. 4 respectively.
In the VFI task, our method surpasses TimeLens++ [] by 1.1 $dB$ for 1-skip and 0.17 $dB$ for 3-skip VFI, and CBMNet [] by 0.23 $dB$ for 1-skip.
Additionally, our model’s size is merely $1/7$ and $1/3$ that of TimeLens++ [] and CBMNet [], respectively.
In the visualization aspect, our approach excels in modeling non-linear motion and long-term dependencies, as evident in Fig. 6 in appendix.
It precisely forecasts the positions of small balls (Fig. 6 (1.d)) and the soccer ball (Fig. 6 (2.d)) at intermediate timestamps, outperforming previous methods.

For the VSR task, our method outperforms EG-VSR [] by 1.03 $dB$ and 0.447 SSIM for 4 $\times$ super-resolution, and by 3.02 $dB$ and 0.0134 SSIM for 2 $\times$ super-resolution.
Fig. 7 demonstrates our model’s effective VSR performance on the real-dataset CED, highlighting its ability to reproduce sharp edges and robustness to noise.
For more visualization results, see the Suppl. Mat.

**Table 4: Spatial super-resolution results on CED dataset. * denotes the values obtained from the official pre-trained models.**
|  |  |  | $4\times$ | $2\times$ |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| Methods | Params ($M$) | Events | PSNR $\uparrow$ | SSIM $\uparrow$ | PSNR $\uparrow$ | SSIM $\uparrow$ |
| DUF [] | 1.90 | ✗ | 24.43 | 0.8177 | 31.83 | 0.9183 |
| TDAN [] | 1.97 | ✗ | 27.88 | 0.8231 | 33.74 | 0.9398 |
| SOF [] | 1.00 | ✗ | 27.00 | 0.8050 | 31.84 | 0.9226 |
| RBPN [] | 12.18 | ✗ | 29.80 | 0.8975 | 36.66 | 0.9754 |
| BasicVSR++ [] | 7.30 | ✗ | 14.76 | 0.1641 | - | - |
| VideoINR* [] | 11.31 | ✗ | 25.53 | 0.7871 | 26.77 | 0.7938 |
| E-VSR [] | 412.42 | ✓ | 30.15 | 0.9052 | 37.32 | 0.9783 |
| EG-VSR [] | 2.45 | ✓ | 31.12 | 0.9211 | 38.69 | 0.9771 |
| HR-INR (Our) | 8.27 | ✓ | 32.15 | 0.9658 | 42.01 | 0.9905 |

Figure: Figure 7: $4\times$ video super-resolution visualization results in CED dataset [].

**Table 5: Ablation studies in Adobe-Average [] ($4\times$ and 7-skip). The † symbol marks the line for comparison with other lines.**
| Case | Events | TPR | Temporal<br>Embedding | Temporal<br>Dim ($C_{t}$) | Input<br>Frames | PSNR $\uparrow$ | SSIM $\uparrow$ |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Case#1 | ✗ | ✗ | Learning | 640 | 4 | 26.84 ( $-4.27$) | 0.8366 ( $-0.0850$) |
| Case#2 | ✓ | ✗ | Learning | 640 | 4 | 29.69 ( $-1.41$) | 0.8974 ( $-0.0242$) |
| Case#3 † | ✓ | ✓ | Learning | 640 | 4 | 31.11 | 0.9216 |
| Case#4 | ✓ | ✓ | Sinusoid | 640 | 4 | 30.42 ( $-0.69$) | 0.9120 ( $-0.0096$) |
| Case#5 | ✓ | ✓ | Learning | 320 | 4 | 28.44 ( $-2.67$) | 0.8700 ( $-0.0516$) |
| Case#6 | ✓ | ✓ | Learning | 640 | 2 | 30.41 ( $-0.70$) | 0.9151 ( $-0.0065$) |
| Case#7 | ✓ | ✓ | Learning | 640 | 3 | 30.72 ( $-0.39$) | 0.9174 ( $-0.0042$)- |

**Table 6: Ablation studies for TPR levels and moments in Adobe-Average [] ($4\times$ and 7-skip). The † symbol marks the line for comparison with other lines.**
| Case | TPR Level ($L$) | TPR Moments ($M_{p}$) | PSNR $\uparrow$ | SSIM $\uparrow$ |
| --- | --- | --- | --- | --- |
| Case#1 | 3 | 3 | 29.93 ( $-1.18$) | 0.9011 ( $-0.0205$) |
| Case#2 | 5 | 3 | 30.32 ( $-0.79$) | 0.9165 ( $-0.0051$) |
| Case#3 | 7 | 3 | 30.78 ( $-0.33$) | 0.9187 ( $-0.0029$) |
| Case#4 † | 7 | 9 | 31.11 | 0.9216 |
| Case#5 | 7 | 18 | 31.18 ( $+0.07$) | 0.9228 ( $+0.0012$) |

**Table 7: Ablation in BS-ERGB (TimeLens++) dataset [].**
|  | 1-skip | 3-skip |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| TPR | PSNR $\uparrow$ | SSIM $\uparrow$ | LPIPS $\downarrow$ | PSNR $\uparrow$ | SSIM $\uparrow$ | LPIPS $\downarrow$ |
| ✗ | 28.25 | 0.8187 | 0.018 | 26.65 | 0.7867 | 0.039 |
| ✓ | 29.66 ( $+1.41$) | 0.8281 ( $+0.0094$) | 0.011 ( $-0.007$) | 28.59 ( $+1.94$) | 0.8140 ( $+0.0273$) | 0.021 ( $-0.018$) |

### 4.2 Ablation and Analytical Studies

Ablation and analytical studies conducted on the Adobe240 [] and BS-ERGB [] datasets unveiled several critical insights.
On the Adobe240 dataset, we executed simultaneous 7-skip VFI and $4\times$ VSR tests, as shown in Tab. 5 and Tab. 6, while on the BS-ERGB dataset, 1-skip and 3-skip VFI were performed in Tab. 7.
Events Gain:
Tab. 5-Case#1 shows that with events input replaced as zero and unchanged network architecture, PSNR and SSIM significantly drop, highlighting the importance of events for temporal motion learning.
Adding events alone substantially raised PSNR by $2.85dB$ and SSIM by $0.06$.
Event TPR Enhancement:
Incorporating event TPR enhances performance, with PSNR and SSIM increasing with layer count, peaking at seven layers, as shown in Tab. 5-Case#2 and Tab. 6.
However, expanding ETPR moments from three to seven only improves PSNR by $0.33dB$, with minimal gains beyond that, suggesting a plateau in performance benefits.
The TPR also demonstrates enhancement on the BS-ERGB dataset, as shown in Tab. 7, yielding an increase of $0.92dB$ for 1-skip and an improvement of $1.13dB$ for 3-skip. This indicates that the benefits of TPR become more pronounced with the increase in the number of skips.
Time Embedding Method: Table 5-Case#3#4 shows Sinusoid embedding’s results. It’s outperformed by learning-based methods, confirmed by previous research [, , , ], due to their superior high-frequency information capture.
Temporal Dimension Impact:
The INR temporal dimension significantly influences performance. Lowering the dimensions from 640 to 320 degrades performance in Tab. 5-Case#3#5, suggesting a reduction in temporal detail capture. Conversely, expanding the dimension to 960 poses instability risks (e.g., $NAN$ errors).
This highlights the need to balance dimensionality and training stability.
Input Frames: Tab. 5-Case#3#6#7 illustrates the impact of varying input frame counts on the final results. We observed a performance decrease of 0.70 $dB$ and 0.39 $dB$ when inputting two and three frames, respectively, compared to four frames. This indicates a clear advantage of multi-frame inputs in modeling longer-term dependencies. Moreover, even with two frames, our method also outperforms previous works [, ].

## 5 Conclusion

Our work introduced the first event-guided continuous space-time video super-resolution method. The main contributions are:
(I) Event temporal pyramid representation for capturing short-term dynamic motion;
(II) A feature extraction process combining holistic and regional features to manage motion dependencies;
(III) A spatiotemporal decoding process based on implicit neural representation, avoiding traditional optical flow and achieving stable frame interpolation through temporal-spatial embedding.
Limitations and Future Work: Training a robust C-STVSR network requires extensive time and computational resources. Addressing these challenges remains an important area for future research.
Due to space limitations, please refer more details, analysis and visualization results in the appendices.

## Appendix Overview

The appendix of this paper consists of five sections:

- 1.
(Sec. A) More Details about the Event Generation Model:
This section introduces the event generation model, providing insights into why events are effective for C-STVSR.
- 2.
(Sec. B) More Details about the Network Structure:
Detailed information on the network architecture is provided, including comprehensive descriptions of the inputs, outputs, and intermediate processes of each module.
- 3.
(Sec. C) More Details about the Experimental Settings:
This section describes additional experimental settings to ensure the accurate replication of our method.
- 4.
(Sec. D) Additional Experimental Results and Analysis :
Further experimental analysis and results are offered, demonstrating the superiority of our approach.
- 5.
(Sec. E) Additional Visualization Results:
This section presents more visual materials, including images and videos.

## Appendix A More Details about Event Generation Model

Events are discrete points capturing the positive and negative changes in pixel brightness.
Their generation hinges on brightness alterations within the logarithmic domain.
Specifically, an event point $e=(x,y,t,p)$ is triggered and logged upon meeting certain criteria.
Suppose $L(x,y,t)$ represents the brightness at point $(x,y)$ at any given time $t$.
The event is recorded if the absolute difference $\Delta L=log\left(L(x,y,t)\right)-L\left(x,y,t-\Delta t\right)$ surpasses a predetermined threshold $C$, as formulated as Eq. 4.

$$ $\displaystyle p=\left\{\begin{aligned} +1,&\Delta L>C\\ -1,&\Delta L<-C\end{aligned}\right.$ (4) $$

Utilizing Eq. 4, the processing pipeline for a specified pixel at coordinates $(x,y)$ at any given time $t$ and $t^{\prime}$ can be delineated by Eq. 5.

$$ $\displaystyle L(x,y,t)=I(x,y,t^{\prime})\times\exp({C\int_{t^{\prime}}^{t}p~{} dt})$ (5) $$

Utilizing Eq. 5, coupled with corresponding events, the intensity frame at a given time enables the computation of intensity frames for alternate times, facilitating video frame interpolation.
However, events typically contain noise, and employing Eq. 5 directly cannot produce optimal outcomes [].
Numerous studies [, , ] have demonstrated the effectiveness of employing neural networks to guide event-based modeling in robust frame interpolation.
Additionally, the high temporal resolution of events aids in high spatial resolution conversion, a finding corroborated by prior research [, ].
In conclusion, comprehending the event generation model reveals its substantial benefits for frame interpolation and VSR tasks, making it a natural guide for continuous space-time video super-resolution (C-STVSR) tasks.

## Appendix B More Details about Network Structure

Owing to space constraints in the main text, this section provides a more detailed account of the network’s results.
The description begins with the Regional Event Feature Extraction component, covering its preprocessing steps and the architecture of the Swin Transformer Encoder Blocks (STEB).
Subsequently, the Holistic Event-Frame Feature Extraction is detailed, which, besides STEB, incorporates the Swin Transformer Decoder Block (STDB).
In summary, this section elaborates on the preprocessing for both feature extraction components and details the structures of STEB and STDB.

### B.1 Regional Event Feature Extraction:

This section delves into Regional Event Feature Extraction, denoted as $f_{re}$.
$f_{re}$ receives Temporal Pyramid Representation (TPR) $E_{p}$ corresponding to timestamp $t$, formatted as $R^{L\times M_{p}\times H\times W}$.
Here, $L$ signifies the number of TPR layers, while $M_{p}$ indicates the count of moments within each layer.
$f_{re}$ involves two primary stages: preprocessing and feature extraction via STEB.
Specifically, this preprocessing first increases its dimension to $C_{r}$ through a $1\times 1$ convolution operation.
This preprocessing alters the TPR’s shape to $L\times C_{r}\times H\times W$.
Feature extraction is then performed using STEB, where both the input and output retain their shape throughout the process.

Swin Transformer Encoder Blocks (STEB):
The STEB structure, proven effective in video super-resolution and frame interpolation [, , ], is adopted in our design.
The input and output shapes of STEB are represented as $L\times C\times H\times W$ for clarity.
Initially, for a window size of $M\times M$, specifically $4\times 4$ in our implementation, the input is partitioned into disjoint windows of $(M\times M)\times N\times(H/M)\times(W/M)\times C$ dimensions.
Then, each window is compressed to form a feature map of shape $(M\times M)\times(N\times H\times W/M^{2})\times C$.
Following this, Layer Normalization [] and window-based multi-head self-attention [, ] are computed for each window, succeeded by further transformation via another Layer Normalization and a Multi-Layer Perception.
Shifted window-based multi-head self-attention [, ] is then employed to establish cross-window connections.
After applying one STEB structure, a second STEB is introduced with an identical configuration, except the input feature window is offset by $(M/2)\times(M/2)$.
In total, four STEBs are utilized for comprehensive feature extraction in the Regional Event Feature Extraction.

### B.2 Holistic Event-Frame Feature Extraction:

This section details the Holistic Event-Frame Feature Extraction ($f_{he}$) module.
The $f_{he}$ module processes inputs $\boldsymbol{I_{in}}$ and $\boldsymbol{E_{v}}$ to generate the output $F^{g}$.
Input $\boldsymbol{I_{in}}$ has a shape of $N\times 3\times H\times W$, whereas $\boldsymbol{E_{v}}$ is structured as $M\times H\times W$, with $M$ indicating the count of moments derived from the events.
The output, $F^{g}$, has a structure of $C_{g}\times H\times W$.
Initially, $f_{he}$ transforms $\boldsymbol{I_{in}}$ and $\boldsymbol{E_{v}}$.
$\boldsymbol{I_{in}}$ and $\boldsymbol{E_{v}}$ are transformed into features with dimensions $N\times C\times H\times W$ and $N-1\times C\times H\times W$ using $1\times 1$ convolution and reshaping, followed by processing through a STEB module.
The process then employs a combination of downsampling and STEB to extract features across multiple scales, each possessing a distinct view field.
Notably, STEB maintains the same resolution and channel count in both its input and output.
Next, the Swin Transformer Decoder (STDB) is utilized for fusing and decoding the dual-modal features.
Each STDB block receives three types of inputs: the preceding STDB’s output and the outputs from the $\boldsymbol{I_{in}}$ and $\boldsymbol{E_{v}}$ encoder sections, all at a matching resolution.
In practice, the outputs from the $\boldsymbol{I_{in}}$ and $\boldsymbol{E_{v}}$ encoders are concatenated, transformed into queries via an MLP layer, and then fused utilizing a multi-head attention mechanism.
Notably, the first STDB module lacks a preceding STDB output, hence defaulting this input to zero.
The final output, $F^{g}$, is derived from the last STDB’s output, post reshaping and $1\times 1$ convolution.
In summary, $f_{he}$ employs a sequence of intricate yet efficacious transformation and fusion procedures, designed to extract multifaceted features from inputs $\boldsymbol{I_{in}}$ and $\boldsymbol{E_{v}}$ for subsequent tasks in video super-resolution and frame interpolation.

## Appendix C More Details about Experiments Setting

### C.1 More Details about Datasets

To facilitate a comprehensive comparison between the frame-base [, ] and event-guided methods [, , , ], we employed four datasets: two simulated (Adobe240 [] and GoPro []) and two real-world (HS-ERGB [] and CED []) for our experiments.

1) Adobe240 Dataset []: This dataset comprises 133 videos, each with a resolution of $720\times 1280$ and a frame rate of 240.
We follow [, , ] to split this dataset into 100 training, 16 validation, and 17 test sets.
Upon frame extraction, we employed the widely-used event simulation method vid2e [], which accounts for real noise distribution, ensuring robust neural network training with enhanced generalization.
In generating the inputs and ground truth of the network, we adopted and extended the previous works [, ] to accommodate multi-frame input.
Specifically, input and ground truth (GT) frames are selected via sliding windows.
We define the window size as $W$, the number of input frames as $N_{in}$, and the interval as $S$.
They interrelate as:
$W=(N_{in}-1)*(S+1)+1$
For instance, with 4 frames input at 7-frame intervals, 25 frames are chosen.
The 1st, 9th, 17th, and 25th frames become the input after down-sampling, while frames 1-25 serve as the GT.
We adopted two strategies in line with prior works [, ]: I) A fixed magnification set at $4\times$ the input resolution, and II) A variable enlargement strategy, wherein the scaling factor is governed by a $\mathcal{U}(1,8)$ distribution.

2) GoPro Dataset []: Both the GoPro and Adobe240 datasets share a resolution of $720\times 1280$ and a frame rate of 240 $fps$.
However, the GoPro dataset is more compact, encompassing 22 training videos and 11 for testing.
Owing to this, prior research [, ] predominantly utilized the GoPro test set for quantitative evaluations, neglecting its training set.
In the interest of fairness, we adopted the same approach.

3) BS-ERGB []: This dataset, captured with a spectroscope, comprises paired events and RGB frames from real-world scenarios. It has a resolution of $970\times 625$ with RGB frames captured at 28 fps.
Of the 123 videos in the dataset, 47 are allocated for training, 19 for validation, and 26 for testing.
Each video contains between 100 to 600 frames.
Despite being a real-world dataset, its size is inadequate for standalone training of a frame interpolation network.
In previous work, e.g.TimeLens++ [], the Vime90k [] simulation dataset was initially utilized for pre-training, followed by fine-tuning using this real-world dataset.
We, on the other hand, opted for a model pre-trained on Adobe240 for our fine-tuning.

4) CED dataset []: This is another real-world dataset in SR research where both frames and events exhibit a resolution of $346\times 260$. Following the previous research [], we conducted preprocessing on this dataset.

### C.2 More Details about Implementation Details

Our experiments encompass four datasets, compared against a variety of benchmark methods.
To guarantee fairness in these comparisons, we provide additional details regarding our experimental approach.
For the Adobe240 and GoPro datasets, our training and testing protocols adhere to methods established in prior research [, ].
Specifically, our approach diverges from prior studies in handling the smaller-scale BS-ERGB dataset. Unlike previous methods that pre-trained on the larger Vime90k dataset before fine-tuning on BS-ERGB, we opted for a different strategy.
Our method involves pre-training on the Adobe240 dataset. During this phase, we input four video frames, with a gap of seven frames between each pair. From a total of 25 frames, 20 are randomly chosen as the ground truth.
In the pre-training phase, our focus is solely on modules related to frame insertion, omitting any upsampling procedures.
Consequently, the spatial embedding aspect is excluded, and the decoding is performed using only a four-layer MLP decoder.
For the super-resolution experiments on the CED dataset, we deviated from pre-training and instead trained our model exclusively on the CED dataset.
A more detailed comparison of the $2\times$ VSR is shown in Tab. 8.
This approach was chosen to maintain a fair comparison with other works [, ].

## Appendix D More Experiments Results and Analysis

### D.1 Stability of VFI in Various Timestamps:

Figure: Figure 8: Timestamps and corresponding PSNR for each frame during $4\times$ VSR and 7-skip VFI on the Adobe240 dataset [].

Figure: Figure 9: $4\times$ VSR and 7-skip VFI visualization results. Please refer to the supplementary material for the video of this case.

Our method demonstrates temporal stability, accurately estimating motion states at each time point during frame interpolation.
Specifically, whether in the GoPro-Average and Adobe-Average of Tab. 1, or the 12-skip and 16-skip tests of Tab. 2, our method significantly outperforms previous methods by at least $1.2dB$ with $4\times$ VSR.

Fig. 8 shows the relationship between timestamps and PSNR, validating the greater stability of our method compared to VideoINR [], especially around the $0.5s$ mark, where VideoINR experiences a notable decrease.
This observation is also reflected in the visualization results of Fig. 1 and Fig. 5 in main paper and Fig. 6 and Fig. 9 in appendix.
In Fig. 6 and Fig. 9, whether using real-world or simulated data, our method demonstrates superior temporal consistency, exhibiting fewer artifacts compared to previous methods such as VideoINR [] and TimeLens [].
This showcases the real-world effectiveness of our method’s temporal stability.

The main reason is that optical flow-based methods [, , ] suffer from instability in flow estimation at time far from the reference frames (at 0 0 and $1$ timestamp), which impacts the motion estimation.
In contrast, this limitation does not affect our method and maintains higher stability throughout the temporal sequence.

### D.2 Video Super-resolution with 2 × 2\times 2 × in CED dataset

**Table 8: Quantitative results (PSNR/SSIM) of the proposed our framework and other methods on the CED dataset for $\times 2$. Because the official training code is not available, * denoted values were acquired from the pre-trained model that the authors have released.**
| Clip Name | TDAN [] | SOF [] | RBPN[] | VideoINR []* | E-VSR [] | EG-VSR [] | Ours |
| --- | --- | --- | --- | --- | --- | --- | --- |
| people_dynamic_wave | 35.83 / 0.9540 | 33.32 / 0.9360 | 40.07 / 0.9868 | 27.47 / 0.8229 | 41.08 / 0.9891 | 38.78 / 0.9794 | 41.50 / 0.9901 |
| indoors_foosball_2 | 32.12 / 0.9339 | 30.86 / 0.9253 | 34.15 / 0.9739 | 26.03 / 0.7766 | 34.77 / 0.9775 | 38.68 / 0.9750 | 42.17 / 0.9904 |
| simple_wires_2 | 31.57 / 0.9466 | 30.12 / 0.9326 | 33.83 / 0.9739 | 26.77 / 0.8321 | 34.44 / 0.9773 | 38.67 / 0.9815 | 42.21 / 0.9922 |
| people_dynamic_dancing | 35.73 / 0.9566 | 32.93 / 0.9388 | 39.56 / 0.9869 | 27.36 / 0.8202 | 40.49 / 0.9891 | 39.06 / 0.9798 | 42.02 / 0.9913 |
| people_dynamic_jumping | 35.42 / 0.9536 | 32.79 / 0.9347 | 39.44 / 0.9859 | 27.24 / 0.8183 | 40.32 / 0.9880 | 38.93 / 0.9792 | 42.09 / 0.9916 |
| simple_fruit_fast | 37.75 / 0.9440 | 37.22 / 0.9390 | 40.33 / 0.9782 | 27.21 / 0.8456 | 40.80 / 0.9801 | 41.96 / 0.9821 | 43.96 / 0.9912 |
| outdoor_jumping_infrared_2 | 28.91 / 0.9062 | 26.67 / 0.8746 | 30.36 / 0.9648 | 26.88 / 0.8226 | 30.70 / 0.9698 | 38.03 / 0.9755 | 42.68 / 0.9902 |
| simple_carpet_fast | 32.54 / 0.9006 | 31.83 / 0.8774 | 34.91 / 0.9502 | 24.21 / 0.5909 | 35.16 / 0.9536 | 36.14 / 0.9635 | 39.80 / 0.9853 |
| people_dynamic_armroll | 35.55 / 0.9541 | 32.79 / 0.9345 | 40.05 / 0.9878 | 27.26 / 0.8193 | 41.00 / 0.9898 | 38.84 / 0.9787 | 41.99 / 0.9915 |
| indoors_kitchen_2 | 30.67 / 0.9323 | 29.61 / 0.9192 | 31.51 / 0.9551 | 26.44 / 0.7502 | 31.79 / 0.9586 | 37.68 / 0.9726 | 41.61 / 0.9901 |
| people_dynamic_sitting | 35.09 / 0.9561 | 32.13 / 0.9367 | 39.03 / 0.9862 | 27.63 / 0.8230 | 39.97 / 0.9884 | 38.86 / 0.9810 | 41.99 / 0.9917 |
| average PSNR/SSIM | 33.74 / 0.9398 | 31.84 / 0.9226 | 36.66 / 0.9754 | 26.77 / 0.7938 | 37.32 / 0.9783 | 38.69 / 0.9771 | 42.01 / 0.9905 |

In Tab. 8, our proposed method demonstrates the highest performance across various clips in the CED dataset, significantly outperforming previous methods, including EG-VSR. Our approach consistently achieves higher PSNR and SSIM values, indicating superior visual quality and reconstruction accuracy. For instance, on average, our method improves PSNR by 3.32 $dB$ and SSIM by 0.0134 compared to EG-VSR. These results highlight the effectiveness of our framework in handling complex scenes, showcasing its robustness and reliability in video super-resolution tasks.

### D.3 Holistic-Regional feature visualization

Fig. 10 (j, l, n) showcases TPR’s regional features, emphasizing its focus on local motion and highlighting its capability to capture local dynamics as a novel event representational form. Specifically, in a scenario where the camera is fixed while filming a basketball player, the background information is displayed in the Holistic Feature, whereas the motion information (such as the moving basketball) is emphasized in the Regional Feature. This indicates that our method can effectively capture different types of information through both Holistic and Regional features.
For more cases, please refer to the example figure below and the videos in the supplementary materials.

Figure: Figure 10: Feature visualization on real data []: (h) shows the holistic feature, $F^{g}$, derived from multiple frames and events; (j, l, n) depict the regional features ($F_{t}^{g}$), highlighting the capability to capture local motion.

### D.4 Time embedding feature visualization

Figure: Figure 11: Visualization of Temporal Embedding. Figure (a) shows the original visualization of Temporal Embedding, while figure (b) displays the results after sorting. The sorting is based on the size of the first element, arranged in ascending order. We use MLP decoding, where the order is not crucial. However, to more clearly demonstrate the outcomes of Temporal Embedding learning, we have chosen to present the sorted results.

Fig.11 illustrates the visualization of time embedding features. Compared to traditional sine-cosine embedding features, the learning-based approach performs better, as shown in Tab. 5. The visualized learning-based embedding features not only demonstrate the capability to learn periodic positional representations but also provide a richer expression of exposure information.

### D.5 Bad case analysis

We have observed that our method has certain limitations in some cases (Fig. 9). For example, when restoring color information, although our model can accurately reconstruct the contours of objects, the color information is often distorted or missing. This issue primarily arises due to the lack of color information in the event stream. We believe that with future advancements in color event technology, this problem will be effectively addressed.

### D.6 Inference time analysis

In Fig. 11, we analyze the inference times of three different methods. Both our method and VideoINR achieve an average frame time of less than 100 $ms$ for $34\times$ frame interpolation. In contrast, TimeLens [] has an average frame time of 187 $ms$, which is more than double that of our method. The tests were conducted on a high-performance computer, and each method was tested 30 times, with the final inference time being the average of these 30 trials.

Figure: Figure 12: Comparison of total and average time for $34\times$ frame interpolation by different methods. Our method takes less time than TimeLens [], but slightly more time than VideoINR [].

## Appendix E More Visualization Results

Additional videos have been included in the supplementary materials to provide a more comprehensive demonstration of our method’s visual results.
Below, we enumerate these videos and briefly describe their key features.
We then present more visualizations to demonstrate the generalization of our method on real data.

- •
1-Adobe240: This video contains the following five clips.
–
IMG_0013-7skip4xsr-Cyclist: In this video, the camera and the people in the background are in motion, creating a complex scene. Our method successfully recovers the locally moving bicycle, demonstrating exceptional video frame interpolation and super-resolution capabilities.
–
IMG_0037-7skip4xsr-
TrafficIntersectionManyCars: The video demonstrates a camera with slight movement, capturing a busy intersection bustling with vehicles. Our method is capable of accurately recovering vehicles in motion within the scene, including the intricate details of rotating tires.
–
IMG_0037a-7skip-
MovingForegroundAndBackground: The video includes both distant and close-up elements. In the close-up scenes, the comparative methods resulted in significant deformations and distortions.
–
IMG_0045-7skip-
PortraitSculpture: This video demonstrates the effects under significant camera movement. When the camera moves rapidly, frame-based methods tend to underperform.
–
IMG_0175-7skip4xsr-
LawnAndCar: The same scene occurs when the camera moves violently.
–
IMG_0175-7skip4xsr-
TreeComplexTexture: This video captures leaves, demonstrating that methods based on optical flow tend to fail in the presence of complex textures.
- •
2-TimeLensPP-Ours-1: This video shows the performance of our method on real-world data sets and the visualization of features. Demonstrates that we effectively capture local motion.
- •
3-Our-vs-Timelens: This video shows the results of comparing our method with Timelens [].

Figure: Figure 13: More visualization results on real-world data set [].

Figure: Figure 14: More visualization results on real-world data set [].

Figure: Figure 15: More visualization results on real-world data set [].

Figure: Figure 16: More visualization results on real-world data set [].

Figure: Figure 17: More visualization results on real-world data set [].

Figure: Figure 18: More visualization results on real-world data set [].

Figure: Figure 19: More visualization results on real-world data set [].

Figure: Figure 20: More visualization results on real-world data set [].

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 6
presentation: 3
questions: Questions:

Considering that this paper does not commit to open-source code, do the authors have any plans to release the code?
soundness: 3
strengths: Strengths

1.The authors’ novel approach of integrating an event camera for high-resolution temporal information is commendable. Event cameras capture finer temporal details than traditional cameras, making them particularly effective for dynamic scenes with nonlinear motion. This approach allows the method to move beyond traditional frame-based video super-resolution limitations.

2.The model is tested on a variety of datasets, both simulated and real, highlighting its robustness and generalization ability. The results indicate HR-INR’s superiority over prior methods, even achieving better performance with much fewer parameters than other models.
summary: Summary: 

This paper addresses the challenging task of Continuous Space-Time Video Super-Resolution (C-STVSR), which aims to enhance video resolution and frame rate at arbitrary scales. Experiments demonstrate HR-INR’s effectiveness across four datasets, achieving state-of-the-art results.
weaknesses: Weaknesses:

1.It appears that this paper achieves better performance and accomplishes more tasks with fewer parameters (e.g., one-tenth of TimeLens’s parameters). Can the authors analyze whether their method has surpassed existing methods in both super-resolution and interpolation tasks? Are there any limitations?

2.In the experimental setup, the authors used Adobe24 + Vid2e to generate events for pretraining and tested on other datasets. If I’m not mistaken, the training data for other methods, aside from the CED dataset, is different from that used in this paper. Although the authors emphasize that TimeLens used a larger training set, it seems that event density, which affects the performance of event-based tasks, was not standardized across methods.

3.For the analysis of the Temporal Representation Pyramid (TRP), is there actually motion within 1/1000s in the current datasets?

---

confidence: 5
contribution: 2
initial_rating: 3
presentation: 2
questions: 1. Why does Figure 8 compare the proposed method with the sub-SOTA method, TimeLens, rather than the more recent SOTA method, such as CBMNet?
soundness: 2
strengths: 1. In the STVSR task, the paper effectively addresses the limitations of RGB-based methods in handling highly dynamic motion by introducing event signals, which helps improve performance.
2. Compared to existing event representation methods, such as voxel grids, time surfaces, and time moments, this paper proposes an innovative event representation method, the Temporal Pyramid Representation (TPR).
summary: The paper addresses Continuous Space-Time Video Super-Resolution (C-STVSR), which aims to upscale both resolution and frame rate in videos at arbitrary scales. Existing methods that use implicit neural representation (INR) struggle with capturing rapid, nonlinear motion, and long-term dependencies due to their reliance on linear motion assumptions and limited frame usage. To address these limitations, the authors propose HR-INR, a C-STVSR framework enhanced by event  signals, which provide high temporal resolution. The framework includes a regional event feature extractor that captures nonlinear motion through an event temporal pyramid and a holistic event-frame feature extractor for long-term dependencies. Additionally, an INR-based decoder with spatiotemporal embeddings is introduced to handle long-term dependencies across a large temporal scope. Results on both synthetic and real-world datasets demonstrate the proposed method’s effectiveness and generalization capabilities.
weaknesses: 1. The paper claims that compared to existing RGB-based methods such as VideoINR and MoTIF, the use of event signals can address highly dynamic motion, which is intuitive. However, the assertion that extending the input window to four frames solves the long-term dependency issue lacks explanation and experimental validation. For instance, it would be helpful to clarify the specific scenarios where a four-frame window is essential, and a two-frame setup would be insufficient. A suggestion would be to visualize Temporal Attention on the input images in Figure 4.
2. While the introduction of event signals may address highly dynamic motion, the paper does not discuss any potential challenges introduced by event signals. The scientific problem being addressed also remains unclear. This paper seems to be engineering-oriented without scientific rigor. 
3. The paper claims its approach avoids the gaps and holes commonly associated with optical flow methods. However, in comparison with the optical flow-based VideoINR, no explanation or experimental evidence demonstrates how this approach mitigates these issues, or why VideoINR cannot mitigate them. For example, in Figure 13, the HR-INR method exhibits blurring in frame 000080 and noticeable holes in the hand region of frame 000160, indicating that gaps and holes persist in certain cases. As a result, the novelty of this work relative to previous studies is not established.
 
4. The contributions of the paper are not clearly defined. It is recommended that the authors refine and explicitly summarize the key contributions of the work in the writing.

Overall, I think 
1. the proposed method lacks a clearly defined contribution to the STVSR task. It seems to be too engineering-oriented. 
2. the experimental results are insufficient to demonstrate a clear advantage over RGB-based methods.

---

confidence: 5
contribution: 3
initial_rating: 6
presentation: 3
questions: It appears that the proposed method can handle a longer sequence of events as input. In comparison, what is the length of the event input used by other methods in this study? Given that a long event sequence may not be suitable for other methods to provide optimal results, testing different sequence lengths would be beneficial.
soundness: 3
strengths: 1.	The proposed temporal pyramid representation captures motion information at multiple pyramid levels, allowing the subsequent feature extraction module to access a range of detail from sharp but sparse features to blurrier yet denser details.
2.	Two specialized feature extractors for events and frames are introduced, each designed to capture information at distinct temporal resolutions.
3.	The spatial-temporal decoding module, which integrates temporal and spatial embeddings, enhances the method’s temporal and spatial consistency, outperforming existing space-time video super-resolution techniques.
summary: This paper presents a space-time video super-resolution method using an event camera. The proposed approach utilizes events and video inputs, employing an event temporal pyramid representation, a comprehensive event-frame feature extractor, and spatiotemporal decoding to produce high-frame-rate, high-resolution videos. Experimental results demonstrate that this method significantly outperforms existing state-of-the-art techniques, particularly in terms of temporal consistency, as shown in the supplementary videos.
weaknesses: 1.	The feature extractors appear to be based on a Swin-Transformer and U-Net (Swin-Unet) combination, which raises concerns about novelty; additional ablation studies would help validate their effectiveness.
2.	Given that this is a space-time video super-resolution approach, including one or two video metrics to assess quality and temporal consistency would provide a more comprehensive evaluation.
3.	The visual comparison results include only one or two baseline method, while more methods are compared in the tabular results.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: * What is the upper bound of the number of FPS? This should be compared with the other VFI methods. Besides, the performance of setting different numbers of FPS should also be analyzed.
* In the abstract, the authors mention that continuous space-time video super-resolution (C-STVSR) aims to simultaneously enhance video resolution and frame rate at an arbitrary scale. Therefore, can the video be SR in any scale using the proposed method? Please show an example with a large scale (\eg, 8x 16x).
soundness: 3
strengths: * The idea of simultaneously VSR and VFI using events is new. And the proposed method achieve the state-of-the-art (SOTA) performance on not only VSR but also VFI.
* The idea about using implicit neural representation (INR) could be new.
summary: This paper proposes an event-guided method for simultaneously enhancing video resolution and frame rate at an arbitrary scale. It captures both holistic dependencies and regional motions based on INR. Experimental results show the superiority of the proposed method.
weaknesses: * Tech

  * The authors say they use INR for C-STVSR. However, as far as I know, INR is something like NeRF (\ie, using a neural network to represent something such as 3D/2D objects). However, in this paper the author do not define the INR in a clear manner. I cannot find where the INR is used. It seems that the term INR in this paper is about some feature layers used in the designed network architecture, not representations. It is much more like "designing a new network structure/proposing new modules" instead of propose a new INR. There are lots of works about event representation (for example,  PhasedLSTM [a],  MatrixLSTM [b] , NEST [c]). However, the authors do not mention, discuss, or compare with them.
  * How to define the regional and holistic events? It seems that they are just two handcrafted terms without clear meaning. The motivation of defining such terms is not clear.
  * It seems that the other event-based VFI methods can remove the motion blur. However, from Figure 1 we can see that the motion blur still exists.

* Experiments

  * The comparisons may be not that comprehensive enough. For example, the authors should compare with pipelines such as "SOTA event-based VFI method + SOTA event-based VSR method" (\eg,   CBMNet + EG-VSR and other combinations) in Table 1. Only in this way it can show that it achieves the SOTA performance on C-STVSR. It seems that the compared baselines do not involve such kinds of pipelines.
  * Why not evaluate on the GoPro and Adobe dataset for either VSR or VFI? It seems that the authors only evaluate the performance of C-STVSR on this two datasets. Besides, for VSR, why not evaluate the proposed method on ALPIX-VSR dataset in [d]?

* Writing

  * The writing quality is not that good. For example, the conference names in Ref are not unified (\eg, for CVPR, the authors use both "2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) " and " Proceedings of the IEEE/CVF conference on computer vision and pattern recognition").

  [a] Phased LSTM: Accelerating recurrent network training for long or event-based sequences 

  [b] M.: A differentiable recurrent surface for asynchronous event-based data

  [c] NEST: Neural Event Stack for Event-based Image Enhancement 

  [d] Learning Spatial-Temporal Implicit Neural Representations for Event-Guided Video Super-Resolution

Meta-review: ## Summary 

The paper presents a novel Continuous Space-Time Video Super-Resolution (C-STVSR) framework, HR-INR, which enhances video resolution and frame rate at arbitrary scales. The framework uses event signals, event temporal pyramid representation, event-frame feature extractor, and spatiotemporal decoding to capture both holistic dependencies and regional motions, outperforming existing techniques.

## Strength

* Integrates an event camera for high-resolution temporal information, enhancing its effectiveness for dynamic scenes with nonlinear motion.
* Tests on various datasets, demonstrating its robustness and generalization ability.
* Proposes Temporal Pyramid Representation (TPR) as an innovative event representation method and enhances the method's temporal and spatial consistency with the spatial-temporal decoding module.
* Achieves state-of-the-art performance on both VSR and VFI using events.

## Weaknesses

* The experimental setup used in the study is different from other methods, suggesting that event density, which affects event-based tasks, is not standardized across methods.
* The Temporal Representation Pyramid (TRP) analysis is also unclear, with no clear explanation or experimental validation for the use of event signals.
* The paper's approach to optical flow methods is not well-defined, with gaps and holes persisting in certain cases, indicating the novelty of the work.
* The paper's technical aspects include unclear definitions of the Inverse Neural Network (INR) and the use of regional and holistic events.

## Conclusion
First of all, thanks to the authors for the enormous effort to provide new results and comparative analysis to the paper and try to answer all the doubts of the reviewers. However, all the new content should be summarized in the main manuscript and have a second review including the improvements suggested by the reviewers.`

Decision: Reject