SkipVSR: Adaptive Patch Routing for Video Super-Resolution with Inter-Frame Mask

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-09 04:41:23

Abstract: Deep neural networks have revealed enormous potential in video super-resolution (VSR), yet the expensive computational expense limits their deployment on resource-limited devices and actual scenarios, especially for restoring multiple frames simultaneously. Existing VSR models contain considerable redundant filter, which drag down the inference efficiency. To accelerate the inference of VSR models, we propose a scalable method based on adaptive patch routing to achieve more practical speedup. Specifically, we design a confidence estimator to predict the aggregation performance of each block for adjacent patch information, which learns to dynamically perform block skipping, i.e., choose which basic blocks of a VSR network to execute during inference so as to reduce total computation to the maximum extent without degrading reconstruction accuracy dramatically. However, we observe that skipping error would be amplified as the hidden states propagate along with recurrent networks. To alleviate the issue, we design Temporal feature distillation to guarantee the performance. This proposal essentially proposes an adaptive routing scheme for each patch. Extensive experiments demonstrate that our method can not only accelerate inference but also provide strong quantitative and qualitative results with the learned strategies. Built upon an BasicVSR model, our method achieves a speedup of 20% on average, going as high as 50% for some images, while even maintaining competitive performance on REDS4.

Main paper: 
 introduction: 
 Video Super-Resolution (VSR) aims to reconstruct a high-resolution video from its corresponding low-resolution counterpart by filling in missing details, which has been widely used in remote sensing, satellite imagery, surveillance and security. As the development of Deep Neural Networks (DNNs), plenty of DNN-based methods are proposed for VSR. Although existing CNN-based video superresolution works have been made promising performance over the past few years, the video super-resolution in real application for smartphones and TV monitors need huge computation cost and unable to respond in real time. That is because recovering video always requires the integration of multi-frames, which usually come at a high computational cost and high memory footprint. That means, it is necessary to design an accelerate strategy for video super-resolution networks to be more applicable.More important, most of the deep learning models perform inference in a static manner. Both the computational graph and the network parameters are fixed once trained, which may limit their representation power, efficiency and interpretability. Therefore, it's an issue that deserves to be explored.Figure 1 : The heatmap about patch PSNR index of two frames in REDS dataset. The PSNR is calculated between lowresolution image and high resolution image super-resolved by pre-trained BasicVSR [2] model. A darker color indicates a higher PSNR value, suggesting better results and easier recovery. On the other hand, lighter colors indicate lower PSNR values, implying greater difficulty in recovering the image. It shows that images with complex content or intricate textures tend to have lighter colors in the heat map, indicating a higher level of difficulty in achieving satisfactory recovery results. Conversely, images with simpler content or smoother textures exhibit darker colors, indicating a lower level of difficulty in the recovery process.To alleviate above issue, we introduce a new direction for effective and efficient VSR. To accelerate the inference process and train once for all scenes with different resource constraints, we design an adaptive patch routing scheme for the VSR task for the first time.Currently, VSR methods can be mainly classified into two categories: parallel models and recurrent models. The parallel methods simultaneously enhance the recovery of all frames, and the recurrent model incorporates temporal dependencies by considering video features in the sequential manner, which means that the restoration for each frame depends on the information obtained from previous frames. Both proposals have their advantages and trade-offs, recurrent networks are widely used in VSR models to extract temporal features, consisting of residual blocks (e.g., BasicVSR [2] has 60 residual blocks). Therefore, we focus on the acceleration strategy for recurrent model and choose classical model BasicVSR [2] as our backbone.In the field of single image super-resolution, there have been advancements in adapting subnetwork to accommodate varying computational demands in different regions. For instance, ClassSR [15] utilizes a classification scheme based on PSNR index to assign patch regions to pretained models with different sizes; APE [23] introduces an adaptive patch exiting strategy for single image superresolution, which estimates. while dynamic networks have been explored in single image super-resolution tasks, the considerations in VSR are different due to the need to incorporate temporal and spatial information from multiple frames.To implement this idea, we need to address two challenges: 1) how to automatically find the effective routing for each image patches in video under different consumption Settings; and 2) how to effectively solve the significant drop in performance caused by error accumulation.To explore the optimal path in our proposal, we introduce a decision mechanism after each residual block to determine whether next residual block shold be exexuted or skipped. For precise prediction, we develop a confidence estimator to give an guidance for execution. During training, we utilize intermediate feature difference as surpervision to constrain the training; while inference, we can obtain the optimal effective sub-network by adjusting the threshold. By this way, we can achieve once-for-all model. Furthermore, we have observed that when performing block skipping, there is an inevitable introduction of errors in the feature dimensions. In contrast to single image super-resolution, VSR tasks involve the fusion of features from previous frames. Thus, these error will accumulate layer by layer, which can lead to the model collapse. Therefore, we adopt temporal feature alignment, which involves leveraging a pre-trained model to correct the fratures at specific layers and maintain the consistency of features across frames.In summary, our contributions can be concluded as : (1) In this paper, we proposed a novel framework based on Ba-sicVSR to accelerate VSR networks. To the best of our knowledge, we are one of the first to design a adaptive acceleration strategy for VSR.(2) The patch-specific subnetwork is guided by the decision of the confidence estimator. Meanwhile, a GT constraint is also introduced to accurate prediction.(3) Extensive experimental results demonstrate that our proposal can obtain great performance and efficiency trade-off, while achieving once-for-all scalable model. 
related work 2.1 video super-resolution: 
 VSR models can leverage additional information from neighboring LR frames for restoration, as demonstrated in previous works such as [3, 5, 9, 10, 17, 26, 27, 29, 31, 32] . These methods exploit different techniques for alignment and fusion of LR frames.Earlier VSR methods, such as [1, 21, 27] , estimate the optical flow between LR frames and perform spatial warping for alignment. Then, more recent proposals have adopted iplicit alignment strategies. Instead of image-level motion alignment, TDAN [22] and EDVR [24] operate alignment at the feature level, aligning features from different frames by deformable convolutional [4] layers. EDVR further improves upon TDAN by introducing coarse-to-fine deformable alignment and a new spatial-temporal attention fusion module. RSDN [9] utilizes a recurrent detail-structural block and a hidden state adaptation module to mitigate the impact of appearance changes and error accumulation. Recently, BasicVSR [2] discovered that bidirectional propagation coupled with a simple optical flow-based feature alignment can enhance performance. Likewise, in the work by Yi et al. [28] , a bidirectional propagation framework is employed to exploit LR frames and estimate hidden states from the past, present, and future frames. To address the issue of huge consumption, Xiao et al. [25] proposed a space-time knowledge distillation scheme for VSR model compression. However, these VSR methods often require high computational costs, impeding their application on resource-limited devices. In contrast, our work focuses on designing an adpative patch routing strategy to expedite the inference of VSR models. By acceleratig the patch routing process, we aim to reduce computational requirements while maintaining restoration performance.Deep neural networks(DNNs) are playing an important role in video super-resolution area, as shown in section 2.1. However, most of the deep learning models perform inference in a static manner, i.e., both the computational graph and the network parameters are fixed once trained, which may limit their representation power, efficiency and interpretability. Dynamic networks [7] , as opposed to static ones, can adapt their structures or parameters to the input during inference, and therefore enjoy favorable properties that are absent in static models.Motivate by the fact that different regions may have diverse computational demands, there are some methods about dynamic network in single image super-resolution. ClassSR [15] classifies the patch region as three kinds according to the PSNR of each path, and then deploy pretrained models with different sizes for restoring them. This method requires evaluating and counting the PSNR value of the whole dataset and manually classifying them, which is inconvenient. Then APE [23] proposes an adaptive patch exiting strategy for single image super-resolution, which estimates the exiting layer directly based on the degree of path recovery. These methods have greatly achieved performance and efficiency tradeoff. However, the temporal and spatial information between multi frames must be considered in video super-resolution task.Since we cannot directly apply a general dynamic inference strategy for VSR, we explore the properties of VSR networks and develop an adaptive patch skipping scheme in this paper. 
method 3.1 basicvsr: 
 The BasicVSR [2] model is currently considered the state-of-the-art model, and it is chosen as the backbone network for the approach in this chapter. Now, let's provide a brief introduction to BasicVSR [2] .EQUATIONAs shown in Figure 2 , BasicVSR [2] is a classic bidirectional recurrent network composed of three modules: the backward feature fusion module (represented by the red module 𝐹 𝑏 in Figure 2 The processing of these modules can be represented by the equations in (1) . Finally, the upsampling module 𝑈 combines the outputs of 𝐹 𝑏 and 𝐹 𝑓 to reconstruct the super-resolution image 𝑆𝑅 𝑡 corresponding to the current frame.As shown in Figure 2 (b), both 𝐹 𝑏 and 𝐹 𝑓 modules have the same structure, consisting of motion estimation 𝑆, spatial transformation 𝑊 , and residual blocks 𝑅 𝑓 /𝑅 𝑏 . In this chapter, the redundancy in the residual blocks 𝑅 𝑓 /𝑅 𝑏 of the forward and backward recurrent feature fusion modules is explored and analyzed. By dynamically adjusting the inference process and reducing unnecessary computations, the efficiency of the model is improved while maintaining image quality.For a video sequence 𝑋 𝑇 ∈ R 𝑇 ×𝐻 ×𝑊 ×3 consisting of T frames of low-resolution images, the corresponding high-resolution video image sequence 𝑆𝑅 𝑇 ∈ R 𝑇 ×𝑠𝐻 ×𝑠𝑊 ×3 is obtained through model reconstruction, where s represents the image upscaling factor, and H, W, and 3 represent the height, width, and number of channels of the input frames, respectively.In the forward feature fusion module, we take the dynamic inference of the residual block group 𝑅 as an example to explain. The residual block group 𝑅 performs feature fusion between adjacent frames in the order from the first frame to the last frame.EQUATIONFor the t-th frame, the input to the residual block group 𝑅 is the current frame image 𝑥 𝑖 and the spatially compensated feature ℎ 𝑡 of the reference frame, and the output is the fused image feature ℎ 𝑡 . This can be expressed as Equation ( 2). The module 𝑅 consists of M residual blocks, and the fusion operates sequentially on the input features, resulting in the output features ℎ 𝑡 𝑚 of the m-th fusion module, with the input features ℎ 𝑡 𝑚-1 .EQUATIONIn a video sequence, adjacent frames exhibit high similarity, known as inter-frame redundancy. Performing complete multilayer fusion operations on all input features in the fusion module may lead to resource wastage. Therefore, this chapter proposes an adaptive mask prediction module that generates a binary mask 𝑀 𝑡 𝑚 (where m denotes the m-th residual block in R and t denotes the t-th frame) based on the similarity between the input frame and the reference frame features. This mask predicts whether to perform full fusion operations or partial fusion operations to reduce computation. The process is as follows:EQUATIONHere, ⊙ denotes element-wise multiplication.In addition, in order to mitigate the issue of error accumulation when recursively propagating and fusing features of adjacent frames in BasicVSR [2] , this chapter maps features at specific layers to a high-dimensional space and imposes constraints during training to ensure the accuracy of high-frequency details. For detailed explanation, please refer to Section 3.4.This section provides a detailed explanation of the computation process for generating the adaptive mask 𝑴 𝑡 𝑚 based on inter-frame redundancy.The inter-frame redundancy refers to the similarity between the input features received by the current fusion module. In this chapter, the difference between the current frame feature ℎ 𝑡 𝑚 and the We design a confidence estimator to evaluate the importance of each residual blocks. If the confidence value is below a given threshold, the patches will be not excuted by next residual block. Finally, the SR patches are merged to the output image.reference frame feature ℎ 𝑡 -1 𝑚 is used as the inter-frame redundancy △𝑇 :EQUATIONA small adaptive mask generator is employed, which takes the inter-frame redundancy △𝑇 as input and generates the corresponding fusion confidence score Ŝ𝑖 :EQUATIONThe generator consists of a fully connected layer and a global average pooling layer. Here, 𝑔 represents the global average pooling operation, and 𝑊 and 𝑏 are the weights and biases of the fully connected layer. The same generator is shared across all layers (separate generators are used for the forward and backward fusion modules), and it is jointly trained with the backbone network.As the modules deepen, the fusion effect on image patches reaches a bottleneck, and the feature maps exhibit minimal changes. To ensure the correctness of the adaptive mask generation, the explicit variation 𝑆 𝑖 between the current layer feature ℎ 𝑆 𝑚,𝑡 of the dynamic network 𝑆 and the target layer feature ℎ 𝑇 𝑡𝑎𝑟𝑔𝑒𝑡,𝑡 of the original network 𝑇 is used as supervision for the generator:EQUATIONwhere 𝜃 represents the Tanh activation function, and a target layer is selected every 6 layers. During the training phase, the confidence score 𝑆 𝑖 is used to constrain the generator by applying an 𝐿 2 loss between the generated fusion confidence Ŝ𝑖 and the supervised variation 𝑆 𝑖 :𝐿 𝑖 = || Ŝ𝑖 -𝑆 𝑖 || 2 2 (8)During the inference phase, the method uses the obtained fusion confidence scores to generate a binary mask by applying a threshold:EQUATIONThe calculated binary mask 𝑴 𝑡 𝑚,𝑛 is then used in Equation ( 4) for further computation.Based on the structure design of BasicVSR [2] , the restoration of the t-th frame image is achieved based on the (t-1)-th frame and the (t+1)-th frame. If there is a small deviation in generating the hidden layer features ℎ 𝑓 and ℎ 𝑏 in the dynamic network, this deviation will be propagated frame by frame and amplified during the network inference, leading to model collapse and inability to reconstruct subsequent frames.To address this issue, this chapter aligns specific high-dimensional features of the dynamic network 𝑆 with the original model 𝑇 to correct the errors and maintain model accuracy.Inspired by spatial attention map distillation, this chapter proposes to generate high-dimensional features through spatial feature mapping F . Here, 𝐻 𝑇 𝑡 ∈ R 𝐶 ×𝑊 ×𝐻 and 𝐻 𝑆 𝑡 ∈ R 𝐶 ×𝑊 ×𝐻 represent the feature maps of the t-th frame image in the original network and the dynamic network, respectively. Here, 𝐶, 𝐻 , and 𝑊 represent the number of channels, height, and width of the feature maps, respectively. The mapping function F can be defined as one of the following three operations:EQUATIONwhere 𝐻 𝑡,𝑖 represents the i-th slice in the feature channel dimension. F 2 sum (𝐻 𝑡 ) is used as the mapping function, which gives more weight to high-frequency details and describes scene details more clearly and accurately through global computation.During the training of the dynamic network, specific high-dimensional feature maps 𝑄 𝑆 𝑡 and 𝑄 𝑇 𝑡 are selected from a certain interval of layers in the dynamic network and the original network, and their differences are constrained by the 𝐿 2 loss to ensure consistency between the dynamic network features and the original network features, preserving high-frequency details. The error correction constraint is as follows:𝐿 error = 1 𝑇 𝑇 ∑︁ 𝑡 =1 ||F 2 sum (𝐻 S 𝑡 ) -F 2 sum (𝐻 T 𝑡 )|| 2 2 (11)Here, 𝐿 error represents the 𝐿 2 norm distance, and 𝑇 is the number of frames in the input sequence.According to previous methods EDVR [? ] and BasicVSR [? ], this chapter employs the Charbonnier loss as the reconstruction loss to constrain the training of the backbone network:𝐿 rec = √︁ ||𝑆𝑅 𝑡 -𝐻𝑅 𝑡 || 2 + 𝜖 2 (12)Here, 𝜖 is set to 1𝑒 -6 . 𝑆𝑅 𝑡 and 𝐻𝑅 𝑡 denote the high-resolution image reconstructed for the t-th frame and the corresponding ground truth high-resolution image, respectively. Based on this design, the overall training loss function of the dynamic network includes the reconstruction loss, the adaptive mask generation loss, and the error correction constraint, as follows:𝐿 = 𝐿 rec + 𝜆 1 𝐿 mask + 𝜆 2 𝐿 error (13)where 𝜆 1 and 𝜆 2 are weight coefficients, and 𝐿 mask and 𝐿 error represent the constraints for adaptive mask training and error correction, respectively. 
experiments 4.1 experiments settings: 
 We adopts widely-used datasets for training: REDS [19] and Vimeo-90K [27] . For REDS, following, we use the REDS4 dataset as our test set. We additionally define REDSval4 as our validation set. The remianing clips are used for training. We use Vid4 [16] , UDM10 [29] , and Vimeo-90K-T [27] as test sets along with Vimeo-90K [27] . We train and test our models with 4× downsampling using two degradations -Bicubic (BI) and Blur Downsampling (BD) as BasicVSR did. For BI, the MATLAB function "imresize" is used for downsampling. For BD, we blur the HR images by a Guassian filter with 𝜃 = 1.6, followed by a subsampling every four pixels.We use pre-trained SpyNet [20] as our flow estimation, and fix the parameters of the flow estimator are fixed. The patch size of input LR frames [18] is 48 × 48. We adopt Adam [14] optimizer, 𝜃 = 10 -8 , Cosine Annealing scheme and we use 𝐿 1 loss as loss function. We pretrain BasicVSR as VSR network backbone. Experiments are conducted on a server with Pytorch 1.10 and V100 GPUs.The batch size is 16 and the LR patch size is 48. We use Adam optimizer, where 𝛽 1 is set to 0.9 and 𝛽 2 is set to 0.999. During testing, we first split LR images into patches of size 48 with stride 46 unless otherwise specified. Then the LR patches are super-resolved in parallel, and the parallel size can be tuned to fit the computational resource. Finally, the SR patches are merged to obtain the complete SR images by weighting overlapping areas. The Peak Signal-to-Noise Ratio (PSNR) and Structural Similarity (SSIM) calculated on RGB channels are adopted as the evaluation metrics to measure SR performance. We use FLOPs to evaluate the computational cost and the practical running time is benchmarked on NVIDIA V100 GPUs.Performance Results. To evaluate the effectiveness and applicability of our proposed method, we conduct experiments by applying it to the classical method BasicSR [2] and compare iy with stateof-the-art and representative VSR netwroks. For a fair comparison, we not only adopt the model trained with our adaptive skipping strategy, but also compare models with varing scale sizes. Specifically, we set different confidence thresholds during the inference process.We evaluate the performance of SkipVSR on various datasets, as shown in Table 1 . It shows that VSR networks with SkipVSR can achieve comparable performance compared to original SR networks in terms of PSNR and SSIM. This comparison validates that our adaptive patch routing can provide insight into the diminishing return and role of individual layers. Efficiency Results As for the efficiency of SkipVSR, Table 2 shows the performance under different computational costs as BasicVSR [2] . Although our method add lightweight decisive network, it only consists of several MLP layers and its FLOPs are negligible. It illustrated that our method can reduce the computational cost of Table 1 : Quantitative comparisons of state-of-the-art methods and SkipVSR. To compare the performance of SkipVSR under the same computation constriants, the confidence threshold of SkipVSR is set 0. Parameters, FLOPs, PSNR and SSIM on various representative datasets with scaling factors ×4 are reported in the table. The changes in PSNR and FLOPs indexes corresponding to BasicVSR and SkipSR are marked in red and blue, respectively. Note that all the efficiency indexes (Params and FLOPs) are measured under the setting of LR images as 180 × 320 resolution on all scales.Params (M) FLOPs (G) Runtime (ms) REDS4 [19] Vimeo-90K-T [27] Vid4 [16] UDM10 [29] Vimeo-90K-T [27] We apply SkipVSR to BasicVSR with scaling factors ×4 on REDS4 [19] dataset. For a fair comparison, we adopt BasicVSR [2] and EDVR [24] as label. Average FLOPs of all 180 × 320 LR patches and PSNR/SSIM calculated on the full image are reported.original VSR networks while maintaining the performance. More important, our proposal enables to meet different computational resource scenarios based on training once. For example, our model only requires 43% of the original model to obtain recovery results. The computational cost of BasicVSR is significantly reduced by our method, and the computational costs of head and tail stay the same.We show the performance-efficiency tradeoff results in Fig. 4 to demonstrate the scalability of SkipvSR. By controlling the confidence threshold during inference, SkipVSR can achieve scalable performance-efficency trade-off. Therefore, we can deploy one SkipVSR network on platforms with different computational resources. For the device with low computational resource, we can decrease the confidence threshold to get lower performance and faster inferences speed.Visual Results Figure 6 shows the qualitative comparison of our method against the original VSR networks and with different confidence threshold. As we can see, SkipVSR can achieve same or even better visual results compared with original VSR networks. Figure 5-(b) visualizes the length of adaptive routing on each patch. As can be seen, the confidence selector network selects fewer blocks in the path for smooth regions. As for patches in complicated regions, the confidence selector network allocates more blocks. This is consistent with the motivation of applying appropriate networks to various difficulties. More important, we also visualize the number of patches executed by each block as figure 5 -(a), and we can see that it is not the case that the later layers execute fewer blocks, which verifies the fact that we perform confidence per layer not exit once is correct.The Validation of Components in SkipVSR. We conduct an ablation study to demonstrate of our SkipVSR by progressively adding components. The result are shown in table 3 . VSR 1 adopts skip blocks uniformly while VSR 2 adopts confidence estimator to guide which block should be skipped. Comparing VSR 1 and VSR 2 , it is indicated that our strategy is superior to other skip strategy and provide a promising direction for VSR inference speed-up. That is because confidence estimator can extract all information contained in feature channels and analyze its restoration performance. As we can see, SSL 3 and SSL 4 outperform other methods. That is because introducing the temporal feature alignment can mitigate error accumulation owing to layer skipping. Comparing VSR 3 and VSR 4 , we can see that With the guarantee of temporal feature alignment, adding a confidence estimation module does not affect the overall performance either. Variants of skip strategy To validate our adaptive patch routing strategy, we conduct experiments on REDS4 for BasicVSR with different skip strategies. We compare our approach with three empirical policies inllustrated in Table 4 : skip blocks uniformly, shallow and deep aggressively execute shallow and deep layers respectively. It shows that using our strategy can substantially reduce the computation cost while maintaining performance better. For a more fair comparison, we set the confidence upper limits with 0.95 and 0.94. It is indicated that while confidence threshold set as 0.94, SkipVSR the final feature difference between original model and SkipVSR as follow:EQUATION)and the intermediate feature difference as equation ??. As shown in table 5 , it shows that only adapting the metric indexes calculated at the end of whole model can not provide great guidance the confidence estimators for each intermediate layer, causing a failure of the whole model to converge. The PSNR index obtained by upsampling the intermediate features can be a good instruction for model recovery, which achieve 30.62dB PSNR. However, the resource consumption caused by upsampling each layer is huge. As a trade-off, we use the intermediate features as confidence estimator GT, which give an enough and clear guidance efficiently. getting 30.60dB with 58ms.There are several limitations in our work. 1) The skipping operation only performs on the block-wise level. Future work should take more fine-grained skipping into consideration. 2) More lighter weight temporal feature alignment strategies can be explored, allowing more accurate alignment of features with less time and resource consumption. We believe that our research is inspiring and the above limitations should be addressed in future works.  
conclusion: 
 In this paper, we propose an adaptive patch routing strategy to speed up the inference of the VSR model in resource-limited situations. Specifically, the key idea is using a confidence estimator constrained by a feature difference loss between current layers and target layers to obtain the binary decision, which provides a execution signal for each patch in each images to indicate its recovery importance. What's more, temporal feature alignment is designed for keeping the accuracy and performance of target layers. We apply SkipVSR on the BasicVSR, and extensive experiments show that our proposal can achieve superior trade-off between performance and efficiency.  
 
 Appendix:  


Decision: This paper presents SkipVSR, an adaptive acceleration strategy for video super-resolution that aims to reduce computational costs while maintaining performance. The method demonstrates promising results with a 20% average speedup compared to the baseline, and offers scalability for different speed-performance trade-offs. However, reviewers noted concerns about limited generalizability beyond BasicVSR, insufficient comparisons with recent works. Despite these limitations, the paper's innovative approach to VSR acceleration and its potential practical applications make it an accept.

Additional comment form SAC: Despite its strengths, the submission lacks multi-modal, multi-media aspects; its focus on "video super-resolution" is inherently single media. Therefore, we recommended that its is accepted as a "Poster"

Reviews: review: paper_summary: paper_summary: The paper introduces SkipVSR, a method that speeds up video super resolution by deciding which parts of the video data need more processing and which do not. This approach helps save computing resources without losing much quality in the video output, especialy for devices with limited processing power, like smartphones and tablets. The method builds on an existing model called BasicVSR and modifies it to only work on important parts of the video, making the processing faster and efficient. Experiments suggests that SkipVSR can achieve 20% average speedup of the super-resolution process comparing to the baseline.
strength_weakness: strength_weakness: 1. This paper proposes using a confidence estimator to dynamically manage the execution of network blocks during the super-resolution process. This approach effectively balances performance with computational cost.

2. The paper assesses the method against several benchmarks, including REDS4 and Vimeo-90K, demonstrating its effectiveness and robustness across various scenarios and datasets. The performance results indicate that SkipVSR can achieve significant efficiency gains.
limitations: limitations: 1. The contributions of this work are somewhat limited, mainly offering incremental improvements on the existing BasicVSR framework, which has already established the fundamental structure for video super-resolution.

2. The presentation of the paper is unclear. The notation used is inconsistent, for example, $h^f_t$ denotes the frame number using subscripts in section 3.1, whereas $h^t_m$ uses superscripts for the same purpose in section 3.2. This inconsistent notation makes the logic in section 3 hard to follow. Additionally, the descriptions and labels in Figure 3 do not match the text, with missing parts (a) and (b) that are referenced in the caption. It is also unclear where the confidence estimator fits within the diagram shown in Figure 3. It would be better if the authors can clarify the main logic of Section 3 with consistent notations.
paper_summary: paper_summary: The paper introduces an adaptive and efficient acceleration strategy called SkipVSR, which is based on the BasicVSR model. It is claimed to be the first method to design an adaptive acceleration strategy for video super-resolution (VSR) tasks. The paper describes the novel aspects of its patch-predicting algorithm, which includes a confidence estimator module to predict the importance of each patch in a video frame, and a feature alignment module to correct errors and maintain accuracy when skipping blocks in a recurrent VSR network. The experiments demonstrate that SkipVSR can significantly reduce the computational cost of VSR models while maintaining performance. Additionally, SkipVSR is scalable and adjustable, allowing it to be configured for different speed-performance trade-offs based on various computational budgets and confidence thresholds.
strength_weakness: strength_weakness: 1. The paper provides a strong motivation for reducing the computational cost of VSR models. While VSR can be highly beneficial for mobile devices, previous models can cause significant delays and power consumption.

2. The adjustable confidence threshold offers configuration options suitable for different devices, enhancing the practicality of the proposed method.

3. It appears that implementing this method on previous VSR models with similar architectures would be relatively easy， as it involves additional modules only.

4. The temporal feature alignment module utilizes the original network with superior feature representations to constrain the lighter adaptive network, which demonstrates a general concept of knowledge distillation.
limitations: limitations: 1. The trade-off presented in Table 2 shows a noticeable decline in performance. When the inference speed is doubled, the PSNR drops by about 1.5dB across each dataset. Under certain configurations, the effectiveness of the method is not convincing. For example, the RRN method in Table 1 achieves a PSNR of 38.96 with a runtime of 45ms, whereas the threshold 0.93 SkipVSR has a similar runtime of 49ms but only achieves a PSNR of 38.46 on the UDM10 dataset. The performance gap is considerable and should be explained by the authors.

2. Moreover, since SkipVSR is based on BasicVSR, it is important to comprehensively compare the customized model with the base model to prove the effectiveness of the customization. For instance, one could reduce the number of layers in the BasicVSR model to achieve a similar runtime as different SkipVSR configurations and then compare their performance. The existing results only show a slight performance decline without skipping operations (threshold=0), which is due to the added modules. Therefore, the experiment is insufficient.

3. As mentioned in the paper, the content of frames can significantly influence the frame-selecting module. Conducting experiments on different video content types (such as sports, films, etc.) would be more convincing than using standard datasets alone. The performance of this framework may vary across different video categories.

4. From the previous experiments, it appears that SkipVSR is capable of adapting to different trade-offs between performance and speed through simple configuration. However, the performance of the framework is not necessarily superior to the BasicVSR model.

5. Several mistakes should be addressed. Table 1 does not contain elements marked in red or blue as described.
paper_summary: paper_summary: The paper presents SkipVSR, a novel approach for accelerating video super-resolution (VSR) through adaptive patch routing. By dynamically skipping certain blocks during inference based on a confidence estimator, SkipVSR aims to reduce computational cost while maintaining high reconstruction accuracy. The method incorporates temporal feature distillation to address error amplification in recurrent networks. The proposed approach is evaluated using the BasicVSR model and demonstrates a 20% average speedup with comparable performance on various datasets.
strength_weakness: strength_weakness: - **Well-Structured and Clear:** The paper is well-structured and clearly written. 

- **Computational Efficiency**: The proposed method significantly reduces computational costs, making it feasible for real-time applications

- **Honest Discussion of Limitations**: The authors provide a candid discussion of the method's limitations, demonstrating a balanced perspective.
limitations: limitations: - **Insufficient evaluation:** There is a lack of comparisons with recent relevant works beyond 2022, such as those in references [1-3].

- **Limited Generality**: The method is evaluated primarily on BasicVSR; its generalizability to other VSR models [4] or different tasks is not thoroughly explored.

- **Lack of related works**: The paper introduce a decision mechanism after each residual block to determine whether next residual block should be executed or skipped. It would benefit from a discussion on adaptive or dynamic inference, referencing works like [5-6].



[1] Structured sparsity learning for efficient video super-resolution. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.

[2] Multi-scale video super-resolution transformer with polynomial approximation. IEEE Transactions on Circuits and Systems for Video Technology (2023).

[3] Multi-frequency representation enhancement with privilege information for video super-resolution. Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.

[4] Compression-aware video super-resolution. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.

[5] Lgvit: Dynamic early exiting for accelerating vision transformer. Proceedings of the 31st ACM International Conference on Multimedia. 2023.

[6] SAVSR: Arbitrary-Scale Video Super-Resolution via a Learned Scale-Adaptive Network. Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 38. No. 4. 2024.



### Minor points

- In Line 13, "which drag down" should be "...drags..."
- In Line 130, "...exexuted..." should be "...executed..."
- In Line 732, "The result are shown in table 3" should be "The results are shown in Table 3"

- In Line 511, "According to previous methods EDVR [? ] and BasicVSR [? ],", which lacks reference labels.
paper_summary: paper_summary: This paper introduces a method to accelerate VSR networks by selectively bypassing certain neural network blocks, thereby reducing inter-frame redundancy. The framework efficiently determines the patch routing by leveraging feature similarity. To address the issue of error accumulation, the authors have introduced a temporal feature alignment strategy, which involves incorporating an additional loss term to minimize the distance between the dynamic network and the full networks. The findings of this study indicate that the framework significantly enhances the speed of the VSR process while maintaining satisfactory video quality.
strength_weakness: strength_weakness: 1. The task (accelerating VSR network) is significant and the proposed approach is valid.
2. The design of the framework is intuitive and easy to follow
3. The high scalability of the framework is quite beneficial in the context of VSR acceleration.
4. Comprehensive evaluations have been done to demonstrate the effectiveness of the proposed approach
limitations: limitations: 1. The proposed method has only been tested with BasicVSR, yet the framework appears to have broader applicability. It would be helpful if the authors could elaborate on how SkipVSR might be adapted to work with other VSR networks.

2. Inter-frame dynamics are a critical factor in video super-resolution. Could the authors detail how well SkipVSR handles videos that exhibit varying levels of temporal dynamics? This area seems to lack sufficient experimentation and discussion in the current study.

3. Typos and formatting errors:
(1) L24: "Temporal" miscapitalized
(2) PSNR: please provide the full name of this term
(3) L511: wrong references
(4) Table 1: numeric changes are not marked in red or blue as noted in the caption.