Asymmetric Event-Guided Video Super-Resolution

Authors: zeyu  xiao
dachun  kai
yueyi  zhang
xiaoyan  sun
zhiwei  xiong
Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-07 16:25:23

Abstract: Event cameras are novel bio-inspired cameras that record asynchronous events with high temporal resolution and dynamic range. Leveraging the auxiliary temporal information recorded by event cameras holds great promise for the task of video super-resolution (VSR). However, existing event-guided VSR methods assume that the event and RGB cameras are strictly calibrated (e.g., pixel-level sensor designs in DAVIS 240/346). This assumption proves limiting in emerging high-resolution devices, such as dual-lens smartphones and unmanned aerial vehicles, where such precise calibration is typically unavailable. To unlock more event-guided application scenarios, we propose to perform the task of asymmetric event-guided VSR for the first time, and we propose an Asymmetric Event-guided VSR Network (AsEVSRN) for this new task. AsEVSRN incorporates two specialized designs for leveraging the asymmetric event stream in VSR. Firstly, the content hallucination module dynamically enhances event and RGB information by exploiting their complementary nature, thereby adaptively boosting representational capacity. Secondly, the event-enhanced bidirectional recurrent cells align and propagate temporal features fused with features from content-hallucinated frames. Within the bidirectional recurrent cells, event-enhanced flow is employed for simultaneous utilization and fusion of temporal information at both the feature and pixel levels. Comprehensive experimental results affirm that our method consistently produces superior results both quantitatively and qualitatively. Code will be released.

Main paper: 
 introduction: 
 High-resolution (HR) videos are attracting increasing attention in both academia and industry, and have been already widely used in modern society, especially for the multimedia field. Video superresolution (VSR) stands as a foundational task within the domains of computer vision to generate HR videos. The primary goal of VSR is to enhance visual quality by reconstructing an HR video from a low-resolution (LR) observation. VSR has garnered substantial attention and popularity due to its diverse applications, encompassing areas such as video surveillance [1, 77] , high-definition television [12] , and satellite imagery [8, 38, 62, 64] . In contrast to single-image super-resolution, which primarily addresses spatial dimensions, VSR uniquely exploits both spatial and temporal dependencies. Advanced VSR methods focus on harnessing temporal information through various techniques such as sliding windows [16, 24, 29, 30, 37, 57, 60, 74] and recurrent structures [5-7, 27, 28, 48, 73] . Recently, with the rapid development of Transformers in computer vision, several attempts have been made to exploit Transformers for better recovering missing details in LR sequences [31, 34, 43, 51] . However, effectively modeling and harnessing temporal relationships continues to pose an open and formidable challenge in the task of VSR.Event cameras represent an innovative class of bio-inspired sensors capable of asynchronously detecting intensity changes in individual pixels at the microsecond level [47] . These sensors can generate asynchronous event data, amounting to millions of events per second while maintaining robustness in HDR lighting conditions. Therefore, recent event-guided VSR methods have been proposed [19, 21, 22, 36, 67] to leverage the advantages of the events for VSR, comparing favorably to RGB-only methods. In practice, however, these event-guided VSR methods assume strict alignment between images and events. To substantiate this assumption, [19] leverages the CED dataset [49] , which comprises aligned images and events sourced from DAVIS346. Similarly, [36] introduces the well-aligned ALPIX-VSR dataset for event-guided VSR.In practice, however, in our daily imaging systems, especially in edge devices like dual-lens (or more) smartphones and unmanned aerial vehicles [11] , deploying strictly aligned event and RGB cameras is challenging, let alone leveraging data collected from aligned cameras for downstream tasks. It is crucial to develop an asymmetric VSR algorithm based on asymmetric stereo events and RGB cameras to address this issue. However, utilizing information from different modalities and dealing with non-aligned data pose significant challenges for this task.In this paper, to solve this new task, we propose the Asymmetric Event-guided VSR Network (AsEVSRN). AsEVSRN is the first endto-end learning-based network that can generally be applied to super-resolve an LR video using an asymmetric event camera. Our proposed AsEVSRN introduces two key components to leverage asymmetric event streams for VSR. Firstly, our proposed content hallucination (CH) module dynamically enhances both event and RGB information by exploiting their complementary characteristics, thereby adaptively boosting representational capacity. Specifically, we adopt a dual-branch architecture to fuse event and RGB information adaptively and employ a dynamic convolution for dynamic enhancement of representational capacity. Secondly, drawing inspiration from successful practices in existing VSR methods [6] , we design the event-enhanced bidirectional recurrent cells. The eventenhanced bidirectional recurrent cells align and propagate temporal features, integrating them with features extracted from contenthallucinated frames. Due to misalignment between the event stream and RGB frames, direct utilization for temporal fusion and propagation is not feasible. Therefore, our proposed event-enhanced bidirectional recurrent cells first pre-align event information with RGB views using a deformable convolution, enabling simultaneous utilization and fusion of temporal information at both the feature and pixel levels. We conduct experiments using event-RGB stereo data. Through extensive experimentation, we have quantitatively and qualitatively demonstrated the effectiveness of AsEVSRN.In summary, our contributions can be summarized as follows:(1) We propose AsEVSRN for super-resolving RGB frames with the guidance of the asymmetric event data. To the best of our knowledge, this is the first event-guided VSR method based on asymmetric event and RGB cameras.(2) We propose the CH module dynamically to enhance event and RGB information by leveraging their complementary nature, thereby adaptively boosting representational capacity.(3) We propose the event-enhanced bidirectional recurrent cells to align and propagate temporal features fused with features from content-hallucinated frames. Within these recurrent cells, eventenhanced flow facilitates simultaneous utilization and fusion of temporal information at both feature and pixel levels.(4) Extensive experiments demonstrate that our proposed AsEVSRN is superior to the existing advanced potential methods. 
related work: 
 Video Super-Resolution. Existing RGB-only VSR methods enhance LR frames using temporal information via sliding windows [63, 65, 66, 69] and recurrent structures [6, 7] . Sliding-window techniques like 3DSRNet [24] , TDAN [57] , and EDVR [60] recover HR frames by predicting dynamic offsets and sampling convolution kernels from adjacent LR frames. They employ methods such as 3D convolution [24] , optical flow estimation [25, 55] , and deformable convolution [57, 60] to align temporal features. However, capturing long-range temporal features remains challenging for these approaches. To address this, recurrent structures-based methods [6, 13, 15, 55, 73] have been developed to model long-range temporal dependencies by utilizing hidden states to connect video frames. For example, BasicVSR [6] uses a bidirectional recurrent structure that merges forward and backward propagation features, resulting in significant performance gains. Vision Transformerbased methods [3, 34, 51, 75] have also achieved remarkable success in VSR. This paper focuses on event-guided VSR, which integrates event cameras to enhance VSR performance.Event-Guided VSR. Event-guided VSR emerges as a pivotal application, leveraging the high-frame-rate motion details captured by event cameras. In event-guided VSR, consecutive frames and event data are utilized to generate HR frames. Various approaches have emerged in this domain. Jing et al. [19] propose a two-stage method that leverages events to interpolate the LR video, resulting in a high-frequency video that is then used to reconstruct HR frames. Kai et al. [22] introduce a bidirectional VSR framework, which harnesses nonlinear motion information from events to aid temporal alignment and incorporates a bidirectional cross-modal synthesis module to enhance the model's robustness to lighting variations. Lu et al. [36] present a joint framework that learns implicit neural representations from both RGB frames and events, enabling arbitrary-scale VSR. However, these methods presume strict alignment between the event stream and RGB images, posing practical challenges in real-world applications. For instance, in edge devices like dual-lens smartphones and unmanned aerial vehicles [11] , acquiring strictly aligned event and RGB cameras can be challenging. To tackle this issue, we propose the first framework for asymmetric event-guided VSR.Event-Guided Video Restoration. Event cameras have the unique capability to measure intensity changes at each pixel independently with microsecond accuracy, making them valuable for various video restoration tasks. One of the notable advantages of event cameras is their ability to provide motion information within the exposure time, which serves as a natural motion cue for deblurring [18, 50, 52, 53, 76] . In the context of video frame interpolation, the integration of event cameras has sparked innovations, such as TimeLens [59] . Subsequently, there has been a growing emphasis on designing interaction modules facilitating the exchange of information between event data and RGB frames, ultimately enhancing the performance of event-based video frame interpolation [42, 58, 68] The information from the left event stream and the right RGB frame stream are fed to content hallucination module, aiming to highlight valuable information while mitigating interference from misaligned data from different modalities. Then the hallucinated event and image features are fed to bidirectional recurrent cells for further alignment and aggregation. Finally, the bidirectional features are fed to the upsampling module to generate the super-resolved results. For simplicity, the upsampling residual operations are omitted in the figure . in correcting rolling shutter artifacts in consecutive global shutter frames [10, 35, 61, 78] . They have also been applied to tasks such as high-dynamic-range imaging [40, 46, 72, 80] , deraining [53] , and low-illumination enhancement [17] , showcasing their versatility across various domains. In this paper, to the best of our knowledge, we propose the first event-guided VSR method based on asymmetric event and RGB cameras. 
method 3.1 overview: 
 Given a multi-modal stereo camera capturing system, where the left camera is an event camera and the right one is a normal LR RGB camera, our goal is to reconstruct consecutive HR clear right frames Î𝑅= {. . . , Î𝑅 𝑡 -1 , Î𝑅 𝑡 , Î𝑅 𝑡 +1 , . . . } ( Î𝑅 ∈ R 𝑇 ×𝑠𝐻 ×𝑠𝑊 ×3 ) using the captured right LR frames I 𝑅 = {. . . , 𝐼 𝑅 𝑡 -1 , 𝐼 𝑅 𝑡 , 𝐼 𝑅 𝑡 +1 , . . . } (I 𝑅 ∈ R 𝑇 ×𝐻 ×𝑊 ×3) and the corresponding left event streams E 𝐿 = {. . . , 𝐸 𝐿 𝑡 -1 , 𝐸 𝐿 𝑡 , 𝐸 𝐿 𝑡 +1 , . . . } triggered within 𝑇 . 𝐸 𝐿 𝑡 denotes the left event stream at the time stamp 𝑡. Î𝑅 should be close to the ground truthI 𝑅,𝐺𝑇 = {. . . , 𝐼 𝑅,𝐺𝑇 𝑡 -1 , 𝐼 𝑅,𝐺𝑇 𝑡 , 𝐼 𝑅,𝐺𝑇 𝑡 +1 , . . . } (I 𝑅,𝐺𝑇 ∈ R 𝑇 ×𝑠𝐻 ×𝑠𝑊 ×3). 𝑇 , 𝐻 , and 𝑊 are the frame number, height, and width, respectively. 𝑠 is the upscaling factor and 𝑇 = {. . . , 𝑡 -1, 𝑡, 𝑡 + 1, . . . }. Note that, the camera system can consist of an event camera on the left and an RGB camera on the right, or vice versa with the RGB camera on the left and the event camera on the right.Due to the fact that the event streams are not convenient for observation and processing by convolutional neural networks because of their sparse, irregular and unstructured properties, we convert event streams E 𝐿 into voxel grids V 𝐿 ∈ R 𝑇 ×𝐻 ×𝑊 ×𝐵 using temporal bilinear transformation, suitable for convolutional neural networks [36, 68] EQUATIONwhere 𝑡 0 and 𝑡 𝑁 𝑒 denote the start time and the end time of the event stream, and 𝑁 𝑒 denotes the number of the event data. 𝑡 𝑘 is the event firing timestamp, and 𝑝 𝑘 is the polarity indicating the sign of illumination changes, respectively. The index 𝑘 spans from 0 to 𝐵 -1, with 𝐵 set as 5 in our experiments.Figure 1 shows an overview of the proposed AsEVSRN. AsEVSRN employs bidirectional recurrent cells 𝐹 𝑓 and 𝐹 𝑏 akin to the scheme proposed in [6] . However, it introduces novel elements such as extra inputs and specialized modules to harness event streams, setting it apart from prior approaches. The left event streams and the right LR frames are first converted into the feature domain using the feature encoders (𝑓 𝑉 𝐸𝑛 and 𝑓 𝐼 𝐸𝑛 ), following which they are directed into the CH module, denoted as 𝑓 𝐶𝐻 (•). This step aims to accentuate valuable information while simultaneously mitigating interference originating from misaligned data across distinct modalities. The above process can be denoted asEQUATIONEQUATIONThen the hallucinated event and image features (𝑭 𝐸,𝐶𝐻 and 𝑭 𝐼,𝐶𝐻 ) are fed to recurrent cells, and for a time step 𝑡, each recurrent cell 𝐹 𝑓 or 𝐹 𝑏 not only takes the hallucinated event and image features at the current time step, but also the corresponding features (ℎ cell propagates the resulting features ℎ { 𝑓 ,𝑏 } 𝑡 to the next cell. The above process can be denoted asEQUATIONTo generate a super-resolved output Î𝑅 𝑡 at timestamp 𝑡, the upsampling module 𝑈 incorporates multiple convolutional layers along with pixel-shuffle operations. This module takes intermediate features ℎ 𝑓 ,𝑏 𝑡 and the LR frame 𝐼 𝑅 𝑡 as inputs, yielding the final super-resolved frame Î𝑅EQUATIONwhere [•, •] is the concatenation operation, and (•) ↑ 𝑠 is the bilinear upsampling operation with a scaling factor of 𝑠.In scenarios involving LR scenes, both event streams and LR images inherently contain noise in the form of missing details and artifacts, respectively. To tackle this challenge and leverage the complementary nature of information across modalities, we propose the CH module (see Figure 2 ). Specifically, the CH module adopts a dual-branch structure (i.e., RGB2Event, and Event2RGB branches), enabling the simultaneous hallucination of representations for two modal features. Given the left event feature 𝐹 𝐸 𝑡 and the right frame feature 𝐹 𝐼 𝑡 , the proposed CH module initially employs an adaptive estimation process to derive dynamic filters of high-level contextual information independently for each modality branch. Subsequently, these dynamic filters are utilized to enhance the features of the corresponding modality, facilitating cross-modal feature refinement and integration for improved representation learning. We haveEQUATIONwhere 𝜙 3 (•) denotes the convolution operation, P(•) denotes the adaptive average pooling operation, A(•) denotes the ReLU activation operation, and ⊛ is the depth-wise convolution. Using a learned dynamic filter from one modality to modulate the feature representation of another, the proposed CH module enhances valuable information while mitigating interference. This module facilitates the refinement and integration of features across modalities, thereby promoting more effective representation learning in the task of event-guided VSR.Conv ReLU Conv ReLU Conv Conv * + * + RGB2Event Event2RGB 𝐹 𝑡 𝐸 𝐹In VSR, exploiting temporal information is crucial, particularly in asymmetric event-guided VSR scenarios. The bidirectional propagation scheme has been widely acknowledged for its effectiveness [6, 7, 54] ; therefore, we adopt it in AsEVSRN. Specifically, we draw inspiration from the bidirectional recurrent cells employed in BasicVSR to implement the utility of bidirectional propagation.In each recurrent cell 𝐹 𝑓 (•) and 𝐹 𝑏 (•), a flow estimation network (e.g., SpyNet [44] ) is typically employed to estimate the optical flow between the LR frame 𝐼 𝑅 𝑡 at the current time step and 𝐼 𝑅 𝑡 ±1 at the previous or the next time steps for alignment and further processing. The optical flow estimation can be denoted asEQUATIONHowever, under the practical setting of asymmetric event-guided VSR, the conventional approach described above fails to fully exploit the asymmetric information from events. Considering that the event information can effectively assist in aligning and fusing multiple frames [9, 18, 33, 42, 59] , we propose the event-enhanced bidirectional recurrent cells.Although the event stream contains crucial temporal information and cues for alignment, asymmetric events may adversely affect the alignment of RGB frames [4, 26, 27] . Therefore, we first employ deformable convolution to align the event stream with the RGB frames in feature space. Specifically, taking the forward recurrent cell as an example (see Figure 3 ), the content-hallucinated features is then employed for flow refinement. On one hand, it is used to mitigate the influence of low-resolution frames on flow estimation. On the other hand, the motion information from events is utilized to further optimize the optical flow 𝑂 where 𝑅𝑒𝑠𝐵(•) is the residual block.To obtain the temporally aggregated feature ℎ 𝑓 𝑡 , we utilize the refined optical flow to warp the RGB feature through a warping operation. Then, we concatenate the warped result with 𝐼 𝑅 𝑡 and feed it into a residual block. This design effectively leverages information from both the feature domain and the pixel domain [68] , enhancing the recurrent cell's representation capability. Formally, we haveEQUATIONwhere Warp(•, •) denotes the warping operation. The event-enhanced backward recurrent cell can be obtained using a similar method to obtain ℎ 𝑏 𝑡 . 
experiments 4.1 experimental settings: 
 Datasets. We train and evaluate our proposed AsEVSRN on KITTI 2012 [2] and KITTI 2015 [39] Without loss of generality, we transform the left view of KITTI 2012 and KITTI 2015 into event data, while keeping the right view unchanged. This creates asymmetric event-RGB inputs. We first utilize the pre-trained RIFE interpolation model [14] to generate additional left-view frames at a 4× higher frame rate. Then we use the event camera simulator ESIM [45] , to simulate events from the interpolated high-frame-rate left videos. Training Settings. During the training stage, we follow the division of the training sets of the KITTI 2012 and KITTI 2015 datasets. We utilize bicubic downsampling by a factor of 4 on the left and right view video frames to obtain LR frames. In other words, we set 𝑠 = 4. The proposed AsEVSRN aims to learn the mapping relationship from low-resolution frames to high-resolution frames. Given the ground-truth frame I 𝑅,𝐺𝑇 and the super-resolved results Î𝑅 generated by our proposed AsEVSRN, we adopt the simple but effective Charbonnier loss [60] to train it from scratch, which can be described as:EQUATION)where 𝜀 is set to 1𝑒 -6 in our experiments. Following previous works, we use a pre-trained SpyNet to estimate optical flow in the event-enhanced bidirectional recurrent cells. We utilize the Adam optimizer with parameters 𝛽 1 = 0.9 and 𝛽 2 = 0.999, and we utilize the Cosine Annealing scheduler for optimization. Each minibatch consists of 6 samples. The input patch size is set to 64 × 64. Experiments are conducted using PyTorch [85] on two NVIDIA 3090 GPUs. We fix the weights of the pre-trained SpyNet in the first 5K iterations, and the total number of iterations is 300K.Inference Settings. During the testing stage, we follow the division of the training sets of the KITTI 2012 and KITTI 2015 datasets. To quantitatively evaluate the reconstructed HR videos, we choose Peak Signal-to-Noise Ratio (PSNR) and Structural Similarity Index (SSIM) on Y channel as metrics. The temporal consistency can be analyzed by evaluating the estimated optical flow of the reconstructed HR videos and the extracted temporal profiles.We compare the proposed AsEVSRN with a wide range of potential methods that could be used to address asymmetric event-guided VSR, aiming to explore as many diverse and rich approaches as possible. (1) Single image SR (SISR) methods: Bicubic, SwinIR [32] , and SRFormer [79] . Specifically, we process each LR frame sequentially through the SISR network for reconstruction, and then assemble the reconstructed frames to form an HR video. (2) VSR methods: VSRNet [23] , DUF [20] , TOF [71] , EDVR [60] , BasicVSR [6] , Ba-sicVSR++ [7] , PSRT [51] , TTVSR [34] , and IART [70] . In particular, we exclude the event stream and solely feed the LR video frames into the VSR network for reconstruction, resulting in the final reconstructed video output. (3) Event-guided VSR methods: as this direction is relatively less explored, we compare our AsEVSRN with EGVSR and EBVSR to provide a thorough evaluation. It is important to note that for fair comparison, we retrain all these methods on the KITTI 2012 and KITTI 2015 datasets using their publicly released codes. We refrain from using fine-tuning or directly utilizing pre-trained models on Vimeo90K [71] or REDS [41] .Quantitative Results. Table 1 shows the quantitative comparisons on various testsets in terms of PSNR and SSIM. From the table, we can draw several conclusions. Firstly, VSR methods generally outperform SISR methods in terms of PSNR and SSIM. This indicates that the temporal information provided by video sequences helps in achieving better reconstruction quality compared to SISR methods. For instance, BasicVSR and its variants consistently outperform SwinIR, which is a representative leading SISR method. Secondly, event-guided VSR shows potential for achieving higher performance compared to traditional VSR methods. For instance, methods like EBVSR, which incorporates event information, exhibit higher PSNR and SSIM values than some traditional VSR methods like Ba-sicVSR++ and TTVSR. For example, for EBVSR compared to TTVSR on the KITTI 2012 dataset, there is a PSNR increase of 0.15 dB and a SSIM increase of 0.0014. On the KITTI 2015 dataset, EBVSR shows a PSNR increase of 0.20 dB and an SSIM increase of 0.0030 compared to TTVSR. Lastly, our proposed AsEVSRN demonstrates superior performance compared to both traditional VSR methods and other event-guided VSR methods. Specifically, AsEVSRN achieves the highest PSNR and SSIM values among all methods evaluated on both KITTI 2012 and KITTI 2015 datasets. For instance, AsEVSRN achieves a PSNR of 31.17 dB on KITTI 2012, outperforming all other baseline methods.Computational Efficiency. We compare AsEVSRN to other methods in terms of the number of parameters and the runtime. Results are listed in Table 1 . Comparing the number of parameters and runtime among different methods, we observe that AsEVSRN achieves competitive performance with fewer parameters and comparable runtime. Specifically, AsEVSRN has 9.648M parameters and a runtime of 49.37 ms, while EBVSR, which has a similar performance, requires 12.151M parameters and a runtime of 46.76 ms. Despite having fewer parameters, AsEVSRN achieves higher PSNR and SSIM scores compared to EBVSR. This indicates that AsEVSRN effectively utilizes parameter efficiency to improve reconstruction quality, demonstrating its superiority in terms of performancecomplexity trade-off. Therefore, even under similar computational constraints, AsEVSRN outperforms other methods in terms of PSNR and SSIM on both KITTI 2012 and KITTI 2015 datasets.Qualitative Results. In Figure 4 , we present visual comparisons between the results obtained by our proposed AsEVSRN and those of other competing baselines on the KITTI 2012 dataset. It is evident from these visual comparisons that our proposed AsEVSRN method outperforms the baseline methods, yielding superior qualitative results characterized by more accurate details and significantly reduced blurring artifacts. For instance, in the restoration of fine texture details, our AsEVSRN excels in reconstructing the intricate brick patterns on the rooftops, while artifacts are visibly present in the results produced by other methods. Furthermore, AsEVSRN demonstrates a clearer restoration of textural details compared to other methods, which often generate blurry results. This is particularly noticeable in regions with complex textures and high-frequency details, where our method successfully recovers fine structures without introducing unwanted noise or smoothing effects. These qualitative improvements underscore the efficacy of our proposed architecture in enhancing the visual quality of super-resolved video content.Temporal Consistency. To evaluate the temporal consistency of the super-resolved video clips, we estimate the optical flow on the KITTI 2012 dataset using the advanced RAFT algorithm [56] . Specifically, we compute the optical flow between consecutive frames to assess the motion coherence of the reconstructed sequences. As illustrated in Figure 5 , the optical flow estimated from the results produced by our proposed AsEVSRN is remarkably close to that obtained from the original HR frames. This observation substantiates the superior temporal consistency and motion accuracy of our method, demonstrating its effectiveness in generating high-quality, temporally coherent super-resolved video content.To analyze the effectiveness of the proposed AsEVSRN, we conduct the following experiments on KITTI 2012.Effectiveness of Two Modules in AsEVSRN. The CH module and the event-enhanced bidirectional recurrent cells are two core modules in AsEVSRN. We analyze the performance of each component by removing different modules and replacing them with residual blocks of equivalent parameter amount. Table 2 presents the effectiveness of the CH module and the event-enhanced bidirectional recurrent cells in terms of PSNR and SSIM. Each method is evaluated with different combinations of these modules. Firstly, when excluding the CH module (AsEVSRN-w/o-𝐹 𝐶𝐻 ), the PSNR is 30. RGB-to-Event and Event-to-RGB fusion, along with dynamic filtering in the CH module enhances performance, as evidenced by the highest PSNR and SSIM values compared to the configurations with individual components excluded. In Figure 6 , we also visualize the results after removing different parts in the CH module. The removal of dynamic convolution resulted in an overall deterioration in performance. Furthermore, eliminating the RGB2event branch lead to a degradation in the reconstructed details, while removing the event2RGB branch caused the results to become blurrier. These observations align with the findings presented in Table 3 .Investigation of the Event-Enhanced Bidirectional Recurrent Cells. The event-enhanced bidirectional recurrent cells aim at leveraging the event information for RGB feature fusion and propagation. To showcase its effectiveness, we design and analyze several variants: 4 presents the effectiveness of different designs in the event-enhanced bidirectional recurrent cells based on PSNR and SSIM metrics. The results show that each design variation contributes to improving performance, with the complete cell achieving the highest PSNR of 31.17 dB and SSIM of 0.9052 compared to its variants without specific components.Embedding the Components of the AsEVSRN into Existing Baseline Methods. Table 5 presents the results after integrating two important components into BasicVSR and EBVSR. We observe that incorporating these components leads to improvements in both PSNR and SSIM metrics for both methods. Specifically, BasicVSR with the integrated components achieves a PSNR of 30.92 dB and SSIM of 0.9001, while EBVSR with the integrated components achieves a PSNR of 31.03 dB and SSIM of 0.9029. These results indicate that the integration of the components enhances the performance of both BasicVSR and EBVSR methods, demonstrating the effectiveness of the proposed components.While our AsEVSRN demonstrates promising results, there are still challenges that need to be addressed. For instance, we encounter difficulties in reconstructing small objects, as elaborated in the supplementary material. These small objects often require finer detail recovery, which can be challenging due to the limited amount of information available at lower resolutions. 
conclusion: 
 In this paper, we address the challenge of performing asymmetric event-guided VSR for the first time, introducing AsEVSRN tailored specifically for this novel task. The AsEVSRN leverages two specialized designs: a content hallucination module that dynamically enhances event and RGB information, boosting representational capacity, and event-enhanced bidirectional recurrent cells that align and propagate temporal features fused with content-hallucinated frames. These cells employ event-enhanced flow for the simultaneous utilization and fusion of temporal information at both the feature and pixel levels. The AsEVSRN consistently generates superior results both quantitatively and qualitatively. 
 
 Appendix:  


Decision: This paper got mixed reviews. Although two reviewers had negative comments on the paper, most of their comments were about some missing baselines and some details about the paper, which had been partially addressed in the authors' rebuttal. By taking all reviewers' comments into consideration, I would like to recommend acceptance of the paper.

Reviews: review: paper_summary: paper_summary: The paper introduces an innovative video super-resolution technique that eliminates the need to align color and event streams beforehand. In other words, no prior calibration is necessary to transform event frames to align with color frames. The authors adopt a strategy of individually hallucinating both streams, leveraging on their complementary characteristics. They also put forward the use of bidirectional recurrent cells to align and propagate temporal features, integrating them with features derived from content-hallucinated frames.
strength_weakness: strength_weakness: Compared to existing work, the proposed method requires no prior calibration of the event camera and RGB camera for the video super-resolution task. I believe this is a key strength of the work, enhancing its practicality for real-world applications.

The paper makes solid technical contributions with the AsEVSRN model, which uses a Content Hallucination (CH) module to dynamically enhance both event and RGB information.

The approach significantly outperforms all existing methods in Single Image Super-Resolution (SISR), Video Super-Resolution (VSR), and Event-guided VSR, marking a significant contribution. Additionally, the paper includes strong evaluations, both qualitative and in terms of ablation studies, to establish the effectiveness of the approach.
limitations: limitations: This approach may lack generalizability, as the authors have not demonstrated how their model performs when the alignment between the event camera and color camera is changed from the training set. As an augmentation method, random transformations could be applied to alter the alignment between color and event streams to evaluate robustness.

The authors mentioned they used 3090 GPUs for their training and inference; however, Table 1 states that the latency reported is measured on a 1080 Ti GPU. The authors need to clarify which GPU was used to measure the latency of which model.

Authors may consider moving equation 13 to the Method section, as it represents the loss function used for training the VSR model.

The discussion of limitations was moved to supplementary material, but including a concise version in the main paper would aid readers in better understanding the method.

The authors have not reported the latency for the preprocessing task, which converts the event streams to voxel grids. This omission may give readers a clouded understanding of the real-time performance of the system. Additionally, it is important to verify whether the run-time of the compared baselines, specifically Event-guided VSR, includes the latency for preprocessing to the required event representation.
paper_summary: paper_summary: The paper introduces the new task of Asymmetric Event-Guided Video Super-Resolution, focusing on leveraging asynchronous event cameras for video super-resolution (VSR) tasks. The main motivation is to address the limitations of existing event-guided VSR methods that assume strict alignment between event and RGB cameras, which is often not feasible in high-resolution devices like dual-lens smartphones and unmanned aerial vehicles. To tackle this challenge, the authors propose an Asymmetric Event-guided VSR Network (AsEVSRN) that incorporates two key components: a content hallucination module to enhance event and RGB information and event-enhanced bidirectional recurrent cells for aligning and propagating temporal features. Experimental results demonstrate the superiority of AsEVSRN in producing high-quality results. The key contributions of the paper include the proposal of AsEVSRN for asymmetric event-guided VSR, the introduction of the content hallucination module, and the design of event-enhanced bidirectional recurrent cells for temporal fusion.
strength_weakness: strength_weakness: - Introduce a new task: Asymmetric Event-Guided Video Super-Resolution.
- Propose AsEVSRN, including two novel components.
- Demonstrate better performance of AsEVSRN compared to traditional VSR methods and other event-guided VSR methods, 
- Showcase good performance-complexity trade-off.
- Effectively integrate components into existing baseline methods, leading to improvements in both PSNR and SSIM metrics.
limitations: limitations: ## No video results.
Although some frames are shown in the paper to demonstrate temporal consistency, it is hard to get a sense of from a single frame. Instead, we perceive temporal consistency naturally from videos. However, I did not see any video in the supp. A suggestion for rebuttal is embedding a video in the Acrobat PDF, without using web links.
## Not convincing/unfair quantitative results.
In Table 1, BasicVSR, BasicVSR++ and TTVSR have fewer parameters and less than a half running time. It is unfair for such a comparison. What about aligning the runtime to 49.37 for BasicVSR, BasicVSR++ and TTVSR to see how they perform? Similar for EGVSR and EBVSR.
## Motivation justification
How challenging to align asynchronous event stream with video RGB frames? I did not see any illustration on this. If it is simple, this task seems to be unnecessary.
## Request to add a baseline
Following the motivation justification, it seems that one can just use accumulated events in an adjacent frame time window (e.g. LNES) as a simple alignment to RGB frames. How effective is this baseline?
paper_summary: paper_summary: This work introduces a new video super-resolution (VSR) method leveraging the capabilities of asymmetric event cameras paired with RGB cameras. This new approach, named Asymmetric Event-guided VSR Network (AsEVSRN), does not require strict alignment between the event and RGB cameras, which broadens the applicability of VSR technology in practical scenarios like smartphones and drones. The technique utilizes a content hallucination module to dynamically enhance the information from both event and RGB data and employs event-enhanced bidirectional recurrent cells to manage temporal features effectively. This approach allows for superior handling of video data with high dynamic range and motion, delivering enhanced resolution and detail.
strength_weakness: strength_weakness: - By integrating event-enhanced bidirectional recurrent cells, the method improves the temporal consistency of the super-resolved video. 
- The proposed AsEVSRN demonstrates superior quantitative and qualitative performance over existing VSR methods.
limitations: limitations: - Figure 1 shows the illustration of the asymmetric event and RGB camera system. Maybe the definition of asymmetric is a little bit ambiguous. I note that the feature fusion in Figure 2 is adopted with the features of the event and RGB at the same time, e.g. t or t+1. How to determine the concept of asymmetric. 

- As for the method section, the content hallucination module fuses the features of the event and RGB through a depthwise convolution. However, how do you perform depthwise convolution between two features rather than features and kernels in Eq (6), where K and F seem to be features? 

- The design of the entire framework is less explainable. It is more module accumulation and lacks a core motivation to design the framework like Figure 1.

- In Table 1, the proposed method, AsEVSRN, outperforms EBVSR on the two datasets with fewer parameters (9.648 vs. 12.151). However, it costs more running time (49.37). Which part of the entire framework is more time-consuming? And whether this part mainly improves the PSNR or SSIM. More details need to be explained. 

- For the qualitative evolution, e.g. Figure 4 and Figure 5, it is not obvious that the proposed method achieves better results compared to other methods in terms of generation quality. Take a picture as an example, ‘ours’ looks not better than SwinIR in Figure 5. Therefore, I am also suspicious of the results for video super resolution.