Event-Guided Rolling Shutter Correction with Time-Aware Cross-Attentions

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-03-30 14:30:03

Abstract: Many consumer cameras with rolling shutter (RS) CMOS would suffer undesired distortion and artifacts, particularly when objects experiences fast motion. The neuromorphic event camera, with high temporal resolution events, could bring much benefit to the RS correction process. In this work, we explore the characteristics of RS images and event data for the design of the rolling shutter correction (RSC) model. Specifically, the relationship between RS images and event data is modeled by incorporating time encoding to the computation of cross-attention in transformer encoder to achieve time-aware multi-modal information fusion. 
Features from RS images enhanced by event data are adopted as keys and values in transformer decoder, providing source for appearance, while features from event data enhanced by RS images are adopted as queries, providing spatial transition information. 
By embedding the time information of the desired GS image into the query, the transformer with deformable attention is capable of producing the target GS image.
To enhance the model's generalization ability, we propose to further self-supervise the model by cycling between time coordinate systems corresponding to RS images and GS images.
Extensive evaluations over both synthetic and real datasets demonstrate that the proposed method performs favorably against state-of-the-art approaches.

Main paper: 
 related works 2.1 rgb-based rsc methods: 
 Multiple images methods mostly predict optical flow between two consecutive images according linear motion assumption. Fan et al. [2, 4] proposed to calculate optical flow in stage one, and then obtain the final GS image with warped image in stage two, which is similar to the [9] structure. DSUN [15] and SUNet [3] calculate cost volume for multi-scale pyramid features respectively, and distort and fuse the features to restore GS image. JCD [31] simultaneously processes RSC and deblurring tasks by using deformable attention module to fuse bidirectional warped features. Naor et al. [17] regarded the RS correction problem as a temporal upsampling problem and proposed additional supervision at the x-t level. Fan et al. [5] proposed a single-stage architecture which combines appearance refinement and undistortion motion estimation for efficient RS correction. Zhong et al. [30] adopted a novel setup where two RS images are scanned in different directions (from top to bottom and from bottom to top) simultaneously, extracting symmetrical information for GS recovery. Building upon the same setup, a bidirectional warping module was designed to restore original RS images in [19] , followed by self-supervised training based on temporal consistency.The rich temporal information of events enables them to perform many image enhancement tasks, such as deblurring [10, 14, 18, 20-22, 27, 29] , frame interpolation [7, 12, 23, 24, 26] , and RSC [1, 32] . REDNet [27] achieves self-supervised deblurring by unfolding a blurry image into multiple sharp images and introducing optical flow constraints between each sharp image. TimeReplayer [7] uses a structure similar to SuperSloMo to interpolate an image from two input images, and one input image and the resulting image can be extrapolated to another input image, achieving self-supervised results. EVDI [29] uses two blurry images taken at different times to restore a sharp image at the same timestamp, achieving selfsupervised results. These self-supervised methods mostly rely on either a single blurry image or multiple sharp images. TimeLens [24] is the first to use both the motion information and the physical properties of event , and it employs two branches for warping and synthesis to perform frame interpolation. EvUnroll [32] uses a similar two-branch structure to handle the RSC problem, reflecting the different time relationships of each row in RS by changing the timemap during warping branch. EvShutter [1] captures the events between RS and GS and estimates the optical flow between RS and GS without using the linear motion assumption. 
method: 
 Here we first briefly introduce preliminary knowledge about event representation and formation of rolling shutter images. Then we will explain how spatio-temporal relationship among events, RS images and GS images could be leveraged in the encoder-decoder framework for event-assisted rolling shutter correction. Finally, the supervision imposed on the model training is presented, which also include both supervised losses and the proposed self-supervised loss.In an event camera, when the intensity variance of a certain pixel between consecutive time stamps reaches a threshold 𝐶, an event with a certain polarity would triggered. Typically, a stream of events over a period of time can be represented as:EQUATIONwhere 𝑡 is the triggered time stamp, (𝑥, 𝑦) represents the triggered pixel position, and 𝑝 represents its polarity. A popular way to represent event set is to transform those events into voxel grids [33] as shown below:EQUATIONwhere𝑡 ★ 𝑖 △ = 𝐵-1 △𝑇 (𝑡 𝑖 -𝑡 0 )represents the normalized time slices. In voxel grids, events are accumulated in adjacent time slices such that each time slice could contain information at different times. Compared with other event representation such as record the quantity and the last moment of the event, voxel grid contains more detailed temporal information.A rolling shutter image can be treated as a stack of rows of a global shutter image with each one exposed at different time stamps. Even a small object motion or camera motion would cause distortion in the rolling shutter image. Hence a rolling shutter image can be formulated as below:EQUATIONwhere 𝑡 𝑟 denotes the readout time between consecutive rows; 𝑀 denotes the number of rows; 𝑡 + (ℎ -𝑀/2)𝑡 𝑟 means the exposure time of the ℎ-th row. Since GS are sharp images captured with short exposure time, here the temporal information is represented by the mid-exposure moment rather than the start and end of exposure.As shown in Fig. 2 , the overall model is an encoder-decoder based framework composed of transformers. In encoder, we employ transformers with the proposed Time-Aware Multi-head Attention to model the spatio-temporal relationship among RS images, events and GS images to obtain multimodal features for later GS image decoding. In decoder, we used DCA to extract the appearance information of RS features based on the timestamp of GS, utilizing the motion information of the event features. The network takes a stream of events and an RS image with its exposure parameters as inputs, outputting GS images at any time within the time interval.Events record almost all continuous brightness change during the exposure time, from the beginning time stamp of the first row of a RS image to the end time stamp of its last row. Such brightness change could be caused by either motion or dynamic lighting. In this task we expect to explore the value of motion information associated with events in the entire exposure period. The encoder starts with a few convolutional layers for shallow feature extraction of the RS frame, while for event data group convolution operation is conducted to retain time ordering information. Simply concatenating representations of two modalities might not be suitable because the features of two modalities are neither spatially aligned nor temporally synchronized. This would cause significant burden to the network in later information fusion and feature enhancement. To fully make use of motion information with event data for RSC, we first resort to powerful transformers to enhance features of both RS frame and events with help of each other. We design a novel Time-aware Multi-head Attention (TMA) for the transformer layer Inspired by the positional encodings in most transformers [25] , here we propose to use time encodings in the transformer and leverage them to promote mutual enhancement. As for features of the RS frame F 𝑟𝑠 ∈ R 𝐵×𝐶 ×𝐻 ×𝑊 , its rows correspond to a set of consecutive readout time stamps. Therefore we can directly assign a row-dependent time encodings to RS features. Since each column has the same readout time, we simply denote the time encoding for RS frame as ) , where 𝑠𝑖𝑛𝑢𝑠𝑜𝑖𝑑 represents the sinusoidal positional encoding, represented as: sinusoid(𝑝) [2𝑖] = 𝑠𝑖𝑛(𝑝/10000 2𝑖/𝐶 ),T ℎ,𝑤 𝑟𝑠 = 𝑠𝑖𝑛𝑢𝑠𝑜𝑖𝑑 ( ℎ 𝐻 ), ℎ ∈ [0, 𝐻 ), 𝑤 ∈ [0,𝑊EQUATIONwhere 𝑖 = (0, ..., 𝐶/2 -1).While as for the voxel-grid-like representation of event data F 𝑒𝑣 ∈ R 𝐵×𝐶 ×𝐻 ×𝑊 , groups along channel correspond to consecutive temporal intervals. Therefore, we divide the event features into 𝑁 groups along channel, with each one corresponding to certain temporal range. Group-channel-wise time encodings are assigned to event features, which can be denoted as T ℎ,𝑤,𝑖 𝑒𝑣 = 𝑠𝑖𝑛𝑢𝑠𝑜𝑖𝑑 ( 𝑖 𝑁 ) and T ℎ,𝑤 𝑒𝑣 = 𝑐𝑜𝑛𝑐𝑎𝑡 ( [T ℎ,𝑤,𝑖 𝑒𝑣 ]), 𝑖 = (0, ..., 𝑁 -1). To promote interaction between RS features and event features, time encodings of each modality are combined with the corresponding modality features in computation of cross-attention. Inspired by [16] , query or key features are concatenated with time encodings and the cross-attention could be computed as below:EQUATIONHowever, it is not proper to directly compute the product of 𝑇 𝑟𝑠 and 𝑇 𝑒𝑣 because each time encoding vector for RS frame corresponds to one timestamp while each time encoding vector for event data corresponds to multiple timestamps. Hence we take inspiration from multi-head cross-attention and present time-aware multi-head attention to address this issue. Here we will take the case of RS features as queries and event features as keys and values as an example to explain how Time-aware Multi-head Attention (TMA) works. Since each group of 𝐶/𝑁 channels in the time encodings for event features corresponds to a certain temporal range, the dimension of time encodings for RS features are also set to 𝐶/𝑁 . In this way time encodings of two modalities could be easily compared and their similarity could also be measured through product. The 𝑖-th group event time encodings is T ℎ,𝑤,𝑖 𝑟𝑠 = 𝑠𝑖𝑛𝑢𝑠𝑜𝑖𝑑 ( 𝑖 𝑁 ) ∈ R 𝐵×𝐶/𝑁 ×𝐻 ×𝑊 , and the 𝑖-th group RS time encodings is T ℎ,𝑤,𝑖 𝑟𝑠 = 𝑠𝑖𝑛𝑢𝑠𝑜𝑖𝑑 ( ℎ 𝐻 ) ∈ R 𝐵×𝐶/𝑁 ×𝐻 ×𝑊 . In this way, both RS and event features are divided into sub-features similar to the heads in standard multi-head attention, we can compute the product of RS and event time encodings in a time-wise way within each head. Correspondingly, RS features and event features are also divided into 𝑁 sub-features which are denoted as F 𝑖 𝑟𝑠 and F 𝑖 𝑒𝑣 . Then as shown in Fig 3 , time-aware multi-head attention can be represented as:EQUATIONwhereEQUATION𝑄 𝑖 = F 𝑟𝑠 𝑊 𝑄 𝑖 , 𝐾 𝑖 = F 𝑒𝑣 𝑊 𝐾 𝑖 , 𝑉 𝑖 = F 𝑒𝑣 𝑊 𝑉 𝑖 , 𝑑 = 𝐶/𝑁 .As for the case of event features as queries and RS as keys and values, the computation is similar. In summary, by dividing temporal encodings and features for RS frame and event data into multi-heads, TMA is able to effectively integrate temporal relationship between the two modalities into the cross-attention for cross-modal enhancement .In the encoding stage, information associated with RS frame and event data is sufficiently explored and fused. The decoder aims at predicting the desired GS image in the middle exposure period or at any specified time stamp based on the enhanced features from encoder. Since there is often significant misalignment between RS and GS frames, we employ deformable cross-attention in the decoder to expand receptive fields and accelerate model convergence. The queries of the deformable cross-attention consist of two components, i.e., enhanced event features from the encoder and the time encoding for desired GS time stamp. The enhanced RS features from the encoder are used as the keys and values of the attention. In this way the desired time and rich motion information provided by event features work as guidance and complement to remove spatial distortion in RS features. Then a GS image is recovered from the spatially corrected RS features with another few convolutional layers. The supervision for our RSC framework include both supervised reconstruction loss and a proposed self-supervised loss. We first introduce the proposed self-supervised loss. As shown in Fig. 1 , given a pair of RS and GS images 𝐼 𝑎 and 𝐼 𝑏 , can we determine which one is RS image and which one is GS image? Obviously, we cannot. Without a prior assumption like "lines must remain straight", we cannot determine if an object appears curved due to rolling shutter effect or if it was originally shaped in that way. Actually, we could have two different time reference systems based on either GS 𝑡 𝑖 𝑔𝑠 or RS 𝑡 𝑖 𝑟𝑠 . In the GS time reference system, each row in GS image 𝐼 𝑎 has the same exposure time while each row in RS image 𝐼 𝑏 has different time as shown below:EQUATIONwhere 𝑡 𝑎 (𝑏 ),𝑖 𝑔𝑠 represents the 𝑖-th row's exposure time with respect to GS time reference system, 𝑖 ∈ [1, ..., 𝐻 ], 𝛿 denotes the readout time delay between successive rows in 𝐼 𝑏 .In the RS time reference system, each row in 𝐼 𝑏 has the same exposure time while the one in 𝐼 𝑎 has different time as shown below:EQUATIONwhere 𝑡 𝑎 (𝑏 ),𝑖 𝑟𝑠 represents the 𝑖-th row's exposure time with respect to GS time reference system. The relation between the two time reference systems can be represented in the transform as shown in Eq. 9 and Fig 4 .EQUATIONwhere 𝑏 represents the inception. The choice of time reference system would affect the use of event data because every event is represented with a quadruplet (𝑡, 𝑥, 𝑦, 𝑝) including its time stamp, position, and polarity. By transforming the time value in the event quadruplets, we convert events from one temporal coordinate system to another, which are respectively denoted as 𝑒 and 𝑒 ′ as shown below.𝑒 (𝑡 𝑔𝑠 , 𝑥, 𝑦, 𝑝) = 𝑒 ′ (𝑡 𝑟𝑠 , 𝑥, 𝑦, 𝑝).Assume we have a pre-trained rolling shutter correction model 𝑓 𝑅𝑆𝐶 . By introducing the idea of cycling transform, we could construct a self-supervision loss. Firstly, in the GS time reference system, the RS image 𝐼 𝑎 with RS and GS timestamp is sent to the RSC model to predict a GS image 𝐼 𝑜1 .EQUATIONBy switching to RS time reference system and considering 𝐼 𝑜1 as the input "RS image", we can apply the RSC model again to recover a "GS image" with help of events {𝑒}.EQUATIONThe recovered "GS image" 𝐼 𝑜2 in RS time reference system should be identical to the original RS image 𝐼 𝑟𝑠 in GS time reference system. Hence the self-supervised loss function could be computed as follows:EQUATION)We provide an x-t plot in the supplementary materials for a better understanding of self-supervised learning.Finally, our loss function consists of a Charbonnier loss [? ] to reconstruct GT, a perceptual loss [11] , and a self-supervised loss.L = L 𝑒𝑟𝑟𝑜𝑟 + 𝛼 L 𝑝ℎ𝑜𝑡𝑜 + 𝛽L 𝑠𝑒𝑙 𝑓 (14) 
experiments: 
 In Section 4.1, we validate the effectiveness of the design module through ablation experiments. Subsequently, in Sections 4.2 and 4.3, we will compare with existing methods on RSC benchmark datasets. Finally, we will examine the effectiveness of self-supervised effectiveness on different methods and both synthetic and real datasets. To examine the effectiveness of TMA, we experiment with another simpler time encoding without the design of multi-head attention focusing on events at different time periods. In this approach, RS time encoding is generated based on the exposure time of each row with the same size as the RS feature (𝑡 𝑟𝑠 , F 𝑟𝑠 ∈ R 𝐵×𝐶 ×𝐻 ×𝑊 ), and event time encoding is generated by the mid-exposure time of RS with the same size as the event feature as well (𝑡 𝑒𝑣 , F 𝑒𝑣 ∈ R 𝐵×𝐶 ×𝐻 ×𝑊 ). And their product is add to the product of query and key as Eq. 5.To validate the importance of receptive field in the RSC task, we conduct a comparative experiment where we replace deformable attention with normal attention. Additionally, we provide a reference experiment with SwinIR to verify if the encoder-decoder structure is better. For SwinIR, we directly concatenate RS, event, and timemap together as the input of the original SwinIR architecture.As shown in Tab. 1, compared with original SwinIR architecture, our time enconding baesd encoder-decoder structure performance better. This is beacuse without query based decoder, it's hard to build difficult to establish the temporal relationship between RS, GS, and events. Just concatenating all these as inputs can't fully integrate the temporal information of the two modalities. Without deformable attention, there is noticeable decrease in PSNR. This is mainly due to two reasons. First, the network lacks a sufficient receptive field, resulting in failure to recover in scenes with large RS offsets, such as high-speed motion, leading to residual RS artifacts. Additionally, the use of regular cross-attention does not effectively exploit the characteristics of each modality, and the motion information of events is not fully utilized. In addition to its effectiveness, using DCA can also lead to reduced computational complexity and accelerated training. Using a different time encoding instead of TMA will also result in a certain decrease in PSNR. That is because this type of encoding of RS represents only a small time interval at each position while event time encoding represents the entire exposure duration. When performing cross-modality event RS EvUnroll Ours GT fusion, directly fusing these two can not extract precise temporal information. In scenarios with high requirement on time accuracy, such as high-speed motion, it may lead to local blur in the restored output. Gev-RS collects GS videos by using a high-speed camera at 5700 fps. Then GS videos are fed into the event simulator V2E [8] to obtain event data, and generate RS images based on Eq. 3 with 640 × 360 resolution.In Fig. 5 , we compare our method with EvUnroll [32] on Gev-RS dataset. The resulting images in the figure are all GS images at the intermediate exposure time of RS images. In the first case, it can be clearly seen that Evunroll produces artifacts when the foreground and background are close, such as the region between the metal box and the train. Additionally, it also failed to accurately reproduce the reflection of the lamp post in the window. This is caused by event overlap between the foreground and background. Our approach effectively integrates the two modalities, enabling model to distinguish whether the event is generated by the foreground or background. As a result, our method can avoid the occurrence of similar artifacts. The second example is highly challenging, involving complex scenes with moving objects. Both the small green and white fence in the foreground are difficult targets for restoration. While EvUnroll's recovery of the white fence appears slightly blurred and the green fence exhibits disjointed segments, our method can accurately capture the displacement of small objects to restore the green fence and also recover the white fence without being affected by high-speed motion. Finally, as shown in Tab. 2, we outperform EvUnroll by 1.37 dB in PSNR, 0.040 in terms of LPIPS [28] and far superior to other image based methods. Fastec-RS collects GS images at 2400 fps with 640 × 480 resolution, and then simulate event data and RS image in the same way as Gev-RS. In Fig. 6 , We compare our method with EvUnroll [32] . When an object moves too close to the camera, causing significant relative movement, EvUnroll produces noticeable artifacts. Even though evunroll utilizes a dual-branch structure, its warping branch fails under significant displacements, while the synthesis branch lacks detailed handling of temporal information in events. Consequently, when a large number of events occur within a short period due to high-speed motion, the accumulation of unnecessary events leads to blurring. In our method, each head of the TMA focuses on event information at different time frames, enabling us to better address the challenges presented by highly dynamic movements.As shown in Tab. 3, Our method outperform EvUnroll by 1.16 dB in PSNR but only exhibits only a slight superiority over EvShutter [1] . More results compared with image-based methods on Gev-RS and Fastec-RS will be shown in Supp. Mat.. To validate the effectiveness of self-supervised learning, we conducted experiments on both the test set of a synthetic dataset and a real-world dataset.As shown in Fig. 7 , the left image is the result of training with self-supervised function, while the right image is reversed. With self-supervised learning, the model learned partial data distribution of the test set and removed the artifacts caused by incomplete restoration, leading to better performance and a 0.49 dB improvement in PSNR. Additionally, we applied the self-supervised loss to EvUnroll, resulting in a 1.33 dB boost, as shown in Tab. 4. This demonstrates the broad applicability of our self-supervised function, which can be applied to different multi-modality methods.Furthermore, in the Fig. 1 and Fig. 8 , we present the results on a real dataset collected by EvUnroll. The yellow dashed line is the GS reference line drawn based on the intermediate event. The initial results with ghosting effects were not good, possibly due to dataset gaps, but after the self-supervised learning, the image quality gradually improved. In the case of building, the boundary the building restored by EvUnroll is quite blurry, as analyzed earlier, primarily due to the limited use of detailed temporal information. Moreover, the restored windows exhibit ghosting and distortion in the glass. Our method outperforms EvUnroll in edge restoration, but it still exhibits ghosting in the windows. Nevertheless, the addition of self-supervised learning results in remarkably clear window restoration. In the next case, self-supervised learning eliminates the direct artifacts between the stick and the base, and prevents the occurrence of puppet distortion in the results of evunroll. It is noteworthy that fine-tuning from the weights pretrained on the synthetic data on real data typically requires just a few hundred iterations.Both the results on real-world and synthetic datasets demonstrate the effectiveness of the proposed self-supervised loss. To validate the effectiveness of Time-Aware Attention, GS images at different timestamps are generated. As shown in Fig. 9 , we present two scenarios, with one featuring a person occluded by trees and the other with text occluded by a bus. Despite occlusions of text and persons by the foreground in the RS images, these areas are effectively recovered. Additionally, we conduct a quatitative comparison with EvUnroll across different timestamps in Tab. 5. It can be observed that performance decreases as the timestamp 𝑡 approaches the start or end of the RS exposure time (𝑡 = 0.1, 0.9). This's because at these moments, the temporal span between RS and GS at the bottom/top of the image is too large (nearly the whole RS exposure time), leading to poor local reconstruction. In contrast, at the mid-exposure time (𝑡 = 0.5), the temporal span is relatively smaller (half of the whole RS exposure time). The larger the difference is, the more challenging the RSC task is. As shown in Tab. 5, our method outperforms EvUnroll at all timestamps. 
conclusion: 
 In this work, we propose a encoder-decoder framework for the RSC task. In encoder, we design time encoding as a bridge for the interaction of two modalities, and skillfully integrate it with multi-head attention to make more intricate use of temporal information. In decoder, we use the existing DCA module to integrate motion information of event and appearance information of RS, and employed the GS time encoding as a query for decoding. Additionally, we develop self-supervised loss specific to the RSC task and confirm its effectiveness on both synthetic and real datasets. Our method outperforms existing methods in both qualitative and quantitative performance.  
: 
 distortion-free GS image, various rolling shutter correction (RSC) techniques are developed.The task of rolling shutter correction is challenging because it is difficult to estimate the transformation between RS and GS images especially when rolling shutter is coupled with both object motion and camera motion. It is a highly ill-posed problem for RSC from a single image due to the absence of motion information. Hence, there have been several works [5, 15, 31] proposed to first estimate motion fields between consecutive frames and compute correction fields based on them, which is followed by applying warping operation to the RS images for GS image recovery.However, most of them exhibit poor performance when confronted with complex motion because of simple motion assumption during exposure.Event camera, a bio-inspired sensor that outputs event signal with high temporal resolution (in the order of µs) by asynchronously measuring brightness change at each pixel [6] , could provide much complementary information for the RSC task. By synergizing a RGB camera and an event camera, the triggered events are able to record how intensity at each pixel evolves during exposure of RGB camera. They carry abundant information about motion happened in the scene. Hence, making full use of events could bring much benefit to the process of rolling shutter correction. There have been a few attempts along this direction. Zhou et al. [32] present a two-branch approach to leverage events for RSC. For the synthesis branch rowwise readout time offsets are injected into attention and fused with RS frame to restore a GS image, while for the warping branch events join the estimation of flow between RS and GS images and obtain another GS image. The two estimated GS images are further combined to give the final GS restoration result. In [1] a transformation is conducted to encode only events between readout time of RS and GS such that row-wise motion can be directly estimated without constant linear motion assumption. It also takes two-branch framework and uses the fusion result as the final prediction. However, the way that they use the events and row-wise readout time for correcting rolling shutter effect , i.e., either directly injecting time offsets to event representation or selecting only events between RS and GS as input, might not be optimal. Involving row-wise readout time in the interaction between RS images and events could help explore more potential of events for the RSC task.In this work, we present an event-assisted encoder-decoder framework for rolling shutter correction. In the encoding stage, the RS frame and event data are first respectively sent to a separate convolutional layer for shallow feature extraction. They are then separately fed to several transformer layers which are equipped with the proposed time-aware multi-head attention for cross-modal feature enhancement. Since time encodings play different roles in the feature enhancement for each modality, different forms of time encodings are exploited. For the RS image, row-wise time encodings are used because each row in the RS image correspond to the same exposure time; while for event data, channel-wise time encodings event RS EvUnroll Ours Ours(u) are used due to the use of voxel grid representation. We utilize this characteristic to design time encoding and combine it with multi-head attention to better integrate the information from both modality. In the decoding stage, we produce the corresponding GS image at the desired time stamp. Due to the significant image displacement in the RSC task, directly reconstructing the image from features can be challenging. To address this, we utilize Deformable Cross Attention (DCA) [34] to expand receptive field. By taking both event features and GS time encoding as the queries and RS features as keys, DCA can effectively leverage the motion information with events and appearance information with RS frame to reconstruct a GS image. Moreover, to improve the model's generalization ability, we designed an self-supervised method with single RS image and event specifically for the RSC task. This method enables us to fine-tune the model on test datasets and real datasets. In summary, our contributions are as follows:• We proposed a novel architecture specifically designed for the characteristics of the RSC task. • We proposed a self-supervised loss tailored for the RSC task to enhance the model's generalization and improve its performance on real-world datasets. • Our method outperforms existing methods in both qualitative and quantitative performance. 
 
 Appendix:  


Decision: Based on the summaries and final ratings provided by the reviewers, it is clear that while there are some concerns, the paper also has significant strengths and innovative aspects, and the author's rebuttal also effectively addressed these concerns. The decision to accept this paper is supported by the following considerations:

1. The paper presents an innovative approach to rolling shutter correction using an event-guided framework with a novel temporal encoding method, which is a valuable contribution to the field.

2. The self-supervised loss function is a creative addition that enhances the model's performance and generalization capabilities.

3. The authors have addressed the concerns raised by the reviewers in their rebuttal, which indicates their willingness to improve the work further.

Despite the majority of the reviewers giving negative scores, I did not find compelling reasons in their comments to reject this article. Moreover, the reviewers requested the authors to supplement many experiments. Although the authors are not obliged to provide additional experimental results, their rebuttal effectively addressed these concerns. Therefore, I still lean towards accepting this manuscript.

Reviews: review: paper_summary: paper_summary: This paper proposes an event-guided rolling shutter correction framework using an encoder-decoder transformer. In the encoder, the authors incorporate time encoding to connect the two modalities, cleverly integrating it with multi-head attention to better utilize temporal information. For the decoder, the authors use the DCA module to merge the motion data from events with the appearance data from RS, and apply GS time encoding as the query for decoding. Besides, a self-supervised loss for the rolling shutter correction task is designed to improve the performance and enhance the generalization in real-world scenarios. The results show that the proposed method outperforms all competitors in two simulated datasets (i.e., Gev-RS and Fastec-RS).
strength_weakness: strength_weakness: 1. As far as I know, this work is an early attempt to address the rolling shutter correction problem using an event camera, and it indeed has potential applications in real mobile devices.
2. The authors provide comprehensive experiments that demonstrate the capability to resolve undesired distortions and artifacts caused by rolling shutter effects during fast motion.
limitations: limitations: 1. Although event cameras have asynchronous and high temporal resolution characteristics, excessively long time windows in high-speed scenarios can still lead to motion blur. Therefore, determining the appropriate length of the time window for different scenarios is crucial. In real-world applications, finding a reasonable time window may not always be feasible. The authors do not provide an analysis of the impact of different time windows on rolling shutter correction results.
2. In high-speed scenarios, the resulting motion blur might further complicate the issue, potentially making it difficult to distinguish between distortions caused by motion blur and those caused by rolling shutter artifacts. It might be more meaningful to study this problem in conjunction with deblurring tasks.
3. The authors chose PSNR and SSIM as metrics, which may not be effective in assessing distortions caused by rolling shutter effects. Other factors, such as background influences, might have a greater impact on these metrics than the rolling shutter distortions themselves. Although the authors present some effective visual results, it is crucial to establish a quantitative metric specifically for rolling shutter distortion.
4. The authors propose a self-supervised loss function as an innovation, but the various weight parameters within the loss function could significantly impact the results. The authors did not provide ablation studies to explore these effects.
5. The paper has some areas for improvement in writing, such as the usage of definite and indefinite articles like "a" and "an" in the conclusion.
paper_summary: paper_summary: 1 This paper presents an event-guided rolling shutter (RS) correction method that corrects image distortion and artifacts caused by rolling shutter through a time-aware cross-attention mechanism. 

2 A novel temporal encoding is introduced to better integrate temporal information. 

3 A self-supervised loss is proposed for fine-tuning the model on real-world data. 

4 The performance is qualitatively and quantitatively satisfactory.
strength_weakness: strength_weakness: 1 The proposed temporal encoding is innovative, and the self-supervised loss that extends from it is also interesting. 

2 The experiments are comprehensive, thoroughly validating the effectiveness of the method. 

3 The manuscript is easy to understand and well-structured.
limitations: limitations: 1 There are some formatting issues, such as the text in Figure 4 being displayed incorrectly in some PDF readers. L523 lacks a reference citation. 

2 The term "Deformable" typically refers specifically to Deformable Convolution Network. It may be misleading in this context of the paper. 

3 In the Supp, unsupervised training is mentioned. Does this mean the training is conducted solely with self-supervised loss? If so, it would be best to provide a qualitative and quantitative analysis of the final training results. In fact, the self-supervised loss can serve three purposes: joint training with supervised loss, fine-tuning on the test set, and unsupervised training. It would be best to clarify the usage of the self-supervised loss in the experiments.

4 In some tables, the best results are highlighted in bold, while others are not. There should be consistency in these.
paper_summary: paper_summary: This paper introduces an advanced encoder-decoder framework that uses high-temporal-resolution event cameras to correct distortions in rolling shutter (RS) images. By incorporating time encoding into cross-attention mechanisms within transformers, the method effectively fuses RS images and event data, leading to superior GS image reconstruction. The approach also includes a self-supervised learning component to enhance generalization. Extensive evaluations on both synthetic and real datasets demonstrate significant improvements over existing methods in terms of PSNR, SSIM, and perceptual quality.
strength_weakness: strength_weakness: 1.	Innovative use of event cameras leverages high temporal resolution to capture fine-grained motion details critical for correcting rolling shutter distortions.
2.	Incorporates time encoding in transformers for effective fusion of RS images and event data, improving alignment and accuracy.
3.	Encoder-Decoder Architecture: Utilizes transformers for feature enhancement and deformable cross-attention for handling large displacements, resulting in high-quality GS image reconstruction.
4.	Introduces a self-supervised loss function that improves generalization and performance on new datasets without extensive manual labeling.
limitations: limitations: 1. This work involves training with both supervised and self-supervised information. Would it be more reasonable to call it semi-supervised?
2. The abbreviation GS should be defined at its first occurrence (line 22).
3. Minor typo: the star symbol (line 274) is inconsistent with Equation (2).
4. The improvement value should be 0.39dB (Line 771) according to the Table.4. 
5. The performance improvement of the self-supervised loss function in Table 4 is not obvious, especially regarding whether SSIM. All of the results are trained under the same random seed?
6. Does the self-supervision loss term work on all datasets? If not, could you please provide examples or references of failures and explain why it doesn't work?
paper_summary: paper_summary: This paper presents an encoder-decoder framework for the RSC task. In the encoder, a multi-head attention for enhanced temporal information processing is proposed. The decoder utilizes the existing DCA module to merge motion and appearance information. Additionally, a self-supervised loss specific to the RSC task is devised and confirmed effective on both synthetic and real datasets.
strength_weakness: strength_weakness: 1. the results on real-world and synthetic datasets demonstrate the effectiveness of the proposed method.
2. The authors have written the paper in an organized manner, making it easy to follow their arguments and understand the presented concepts.
limitations: limitations: 1. The proposed Time-awareness Multi-head Attention and self-supervised loss may lack distinctive innovations within the domain of event-based vision, as it appears to amalgamate existing methodologies.

2. Will the checkpoints (ckpts) required for replicating the paper's results be released as open-source?

3. The comparative method only includes EvUnroll. The comparison seems insufficient due to the absence of a comprehensive evaluation against state-of-the-art methods.