Dual-view Pyramid Network for Video Frame Interpolation

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-08 14:57:39

Abstract: Video frame interpolation is a critical component of video streaming, a vibrant research area dealing with requests of both service providers and users. However, most existing methods cannot handle changing video resolutions while improving user perceptual quality. We aim to unleash the multifaceted knowledge yielded by the hierarchical views at multiple scales in a pyramid network. Specifically, we build a dual-view pyramid network by introducing pyramidal dual-view correspondence matching. It compels each scale to actively seek knowledge in view of both the current scale and a coarser scale, conducting robust correspondence matching by considering neighboring scales. Meanwhile, an auxiliary multi-scale collaborative supervision is devised to enforce the exchange of knowledge among current scale and a finer scale and thus reduce error propagation from coarse to fine scales. Based on the robust video dynamic caption of pyramidal dual-view correspondence matching, we further develop a pyramidal refinement module that formulates frame refinement as progressive latent representation generations by developing flow-guided cross-scale attention for feature fusion among neighboring frames. The proposed method achieves favorable performance on several benchmarks of varying video resolutions with better user perceptual quality and a relatively compact model size.

Main paper: 
 introduction: 
 Video frame interpolation analyzes video dynamics to synthesize intermediate frames between reference frames [1] . As a critical video processing technique, it has various applications in multimedia systems, such as video compression [14, 40] , video on demand streaming [29] and video editing [3, 4, 15, 34] . Recent years have witnessed emergence of approaches leveraging deep learning algorithms for video frame interpolation [8, 12, 43] . However, video frame interpolation is still challenged by video dynamics that involve complex occlusions and large motions [11, 12] . One way deployed by existing video frame interpolation methods to understand video dynamics is exploring motion cues like optical flow among reference frames. Based on optical flow estimates between reference frames and the intermediate frame to be synthesized, reference frames at two time steps are sampled via warping and fused to generate interpolation results at a time step inbetween [1] . Advances in optical flow estimation [2, 17, 36, 38] have been leveraged to handle complex video dynamics in recent video frame interpolation methods [11, 19] . To mitigate degradation of interpolation quality due to deficient quality of optical flow estimation, asymmetric [26] and multiple flow fields estimations [7, 11, 21] has been considered. But video frame interpolation for varying resolution videos remains difficult, where larger motions need to be handled with longer-range correlation analysis while avoiding error propagation in this process [24] .To tackle these difficulties, we propose to capture long-range correlations among reference frames and intermediate frames to be synthesized with pyramidal dual-view correspondence matching. Our goal is to build a pyramid network that is capable of handling complex occlusions and motions while preserving this capability for varying video resolutions through unleashing the multifaceted knowledge yielded by the hierarchical views at multiple scales. We approach this by introducing dual-view correspondence matching in a pyramidal network architecture that estimates multiple bidirectional flow fields and their corresponding weights from coarse to fine, where each scale considers neighboring scales for correspondence matching. This enables knowledge in view of both the current scale and a coarser scale coalesce into more robust correspondence matching for analyzing long-range correlation. Unlike some other multimedia tasks such as optical flow estimation [45] and video object segmentation [10] , the correspondence matching in video frame interpolation is subject to the intermediate frame which is not available for the computation of correspondence matrix. As a result, calibration of correspondence matching for this unknown frame plays a critical role. Therefore the proposed pyramidal architecture allows the dual-view correspondence matching to be gradually calibrated according to the intermediate frame to be synthesized through estimates of multiple bidirectional flow fields and their corresponding weights across scales. As the calibration is operated in RGB color space through the generation of intermediate frames by directly sampling reference frames with estimates of multiple bidirectional flow fields and their corresponding weights across scales, the quality of supervision offered by downscaled ground truth intermediate frames at coarser scales may not be sufficient to calibrate the dual-view correspondence matching. For this reason, we introduce an auxiliary multi-scale collaborative supervision to enforce the exchange of knowledge among current scale and a finer scale, thus reducing error propagation across scales.Compared with existing attention-based video frame interpolation methods [24, 43] that incorporate long-range correlation by leveraging the capability of Transformer blocks for long-range connectivity, we pay attention to the multifaceted knowledge yielded by the hierarchical views of a pyramidal network architecture. Establishing robust dense correspondences among frames via the proposed pyramidal dual-view correspondence matching instead of Transformer blocks exhibits three potential advantages. Firstly, pyramidal network architecture disseminates the burden of establishing long-range connectivity across scales, thus it facilitates the design of a compact model, gradually building global connectivity across scales. Secondly, pyramidal dual-view correspondence matching can utilize a broad range of supervisions from multiscale knowledge which are critical for model generalization among varying video resolutions. Thirdly, we can enhance representation capability of the proposed pyramidal network architecture by further incorporating latent representations of reference frames and intermediate frames to be synthesized. This is achieved by developing a refinement module, which only considers a relatively small neighborhood to efficiently refine the synthesis of each pixel utilizing the guidance of video dynamics captured by the pyramidal dual-view correspondence matching. To take advantage of the robust video dynamics captured by pyramidal dual-view correspondence matching in constructing latent representations, we introduce a pyramidal refinement module that formulates frame refinement as progressive latent representation generations. Latent representations of reference frames are adaptively transformed to establish the latent representation of the intermediate frame to be synthesized by our proposed flow-guided cross-scale attention for feature fusion among neighboring frames. We show the proposed Dual-view Pyramid (DvP) network and its enhanced version DvP+ with our pyramidal refinement module are able to handle changing video resolutions while improving user perceptual quality. The main contributions of this paper are summarized as follows:• We introduce pyramidal dual-view correspondence matching for video frame interpolation, which enables efficient capture of long-range correlations to handle complex video dynamics along with large motion by seeking knowledge in view of both the current scale and a coarser scale. • We propose an auxiliary multi-scale collaborative supervision for effective learning of pyramidal dual-view correspondence matching, which reduces error propagation across scales to improve video frame interpolation quality for varying video resolutions. • We combine designs above in building Dual-view Pyramid (DvP) network for video frame interpolation, and introduce a pyramidal refinement module to build DvP+ that enhances representation capacity using latent representation generations through flow-guided cross-scale attention. • We demonstrate that DvP network and its enhanced version achieve favorable video frame interpolation performance on several benchmarks with video resolutions ranging from 480p to 4K, while keeping a relatively compact model size. 
related works 2.1 video frame interpolation: 
 Video frame interpolation is a classical task in video processing [1, 42] that attracts increasing attention due to the emergence of streaming services and a growing demand for better video systems. Existing methods leverage deep neural networks to improve the performance of video frame interpolation on several benchmarks [7, 8, 11, 12] . The majority of these approaches belong to the flow-based paradigm [19, 22, 32] that estimates flow fields among frames to synthesize intermediate frames and thus rely on high quality flow estimates to mitigate corruption in output, and kernelbased paradigm [18, 23] that predicts adaptive convolution kernels to generate intermediate frames and may be vulnerable to blurry output by lacking explicit flow guidance. Recently, progress has been made by improving quality of flow estimation and compensate imperfect estimates. This includes generating asymmetric or multiple flow estimates [7, 11, 21, 26] , deploying sophisticated enhancement, leveraging Transformer-based architecture [24, 43] and latent diffusion model [8] . For example, ST-MFNet [7] introduces compensation with a multi-branch structure and a 3D convolutionbased enhancement network. However, these methods are still deficient to handle large motion in complex video dynamics especially regarding improvement of perceptual quality [8] , and are limited by computational burden due to deployment of heavy enhancement module, Transformer or latent diffusion. As a result, we propose to explore effective analysis of video dynamics via more robust correspondence matching approach, unleash the multifaceted knowledge yielded by multiple scales in a pyramid network.Establishing spatiotemporal correspondences among frames has been widely studied for various multimedia applications. For example, correspondence matching is involved in binocular stereo to determine the disparity of pixels in stereo pair [30] , and in finegrained action recognition to facilitate the capture of contextual details from multiple perspectives [37] . This technique is also applied in optical flow estimation [45] and video object segmentation [10] to estimate the motion of scene contents across frames. Acquiring high quality correspondence matching is nontrivial, due to the illposedness of the task. To this end, Patchmatch Stereo++ introduces deep-learning-based continuous disparity optimization to tackle edge ambiguity [30] , MVFlow extracts pre-computed information in video compression as a robust prior to mitigate ambiguities in optical flow estimation [45] . The ambiguities for correspondence matching in video frame interpolation is distinguished by the absence of intermediate frame to compute correspondences. BMBC [25] and XVFI [32] devised respectively bilateral cost volume and complementary flow reversal for correspondence matching in video frame interpolation. These approaches are limited by linear motion assumption, motivating ABME [26] and NCM [14] being developed to handle asymmetric motions and inaccurate flows. However, video frame interpolation for varying resolution videos still challenges correspondence matching due to larger motions that requires longer-range correlation analysis while avoiding error propagation in this process. In this paper, we disseminate the burden of establishing long-range connectivity across scales by introducing pyramidal dual-view correspondence matching, mitigating error propagation through knowledge exchange while gradually building global connectivity across scales. 
methods: 
 Figure 2 shows an overview of the proposed approach. The pyramidal video frame interpolation network composed of two subnetworks. Given four consecutive reference frames {𝐼 -1 , 𝐼 0 , 𝐼 1 , 𝐼 2 }, the first subnetwork 𝐻 1 establishes robust dense correspondences among reference frames and intermediate frame 𝐼 𝑡 , 𝑡 ∈ (0, 1) to be synthesized between 𝐼 0 and 𝐼 1 with pyramidal dual-view correspondence matching and an auxiliary multi-scale collaborative supervision, and the second subnetwork 𝐻 2 progressively constructs latent representations to refine the synthesis of each pixel of 𝐼 𝑡 through a pyramidal refinement module with the guidance of video dynamics generated by 𝐻 1 .To capture complex video dynamics among {𝐼 -1 , 𝐼 0 , 𝐼 EQUATIONwhere 𝐵(•) denotes the bilinear interpolation kernel, and 𝑢 𝑘,𝑚 0→𝑡 is parameterized with estimates of flow fields 𝑢 𝑘,𝑚 0→-1 , 𝑢 𝑘,𝑚 0→1 using the quadratic motion model [41] to accommodate nonlinear motion as:EQUATION)By analogy, we compute the estimate of 𝑢 𝑘 𝑡 →1 [𝒑 𝑡 ] via weighted sampling of {𝑢 𝑘,𝑚 1→𝑡 } 𝑀 𝑚=1 , and the estimate of intermediate frame Ĩ𝑘𝑡 [𝒑 𝑡 ] via weighted sampling of {𝐼 𝑘 0 , 𝐼 𝑘 1 }. Such weighted sampling warps and fuses pixels direcly in the RGB color space, thus the estimation results are directly affected by the quality of estimates of candidate pairs of flow fields and their corresponding weights. However, acquiring estimates of sufficient quality is challenged by video dynamics that involve complex and large motions [11, 12] . To tackle this challenge, we propose pyramidal network with dual-view correspondence matching to improve video frame interpolation quality of complex video dynamics with varying video resolutions.Based on the observation that large motion is easier to be captured at coarser scales while small object may only be observed at finer scales in a pyramidal network, the core idea of pyramidal dual-view correspondence matching is to simultaneously acquire knowledge in view of both the current scale and a coarser scale from coarse to fine. In detail, let 𝐻 1 utilizes a 𝐾-level feature encoder 𝑓 [13] to extract pyramidal features {𝐹 𝑘 𝑖 } 𝐾 𝑘=1 , 𝑖 ∈ {0, -1, 1} of reference frames, to enable multifaceted knowledge yielded by the hierarchical views at multiple scales to coalesce into more robust correspondence matching for analyzing long-range correlation of complex video dynamics, we compute the correspondences with reference frames 𝐼 𝑖 , 𝐼 𝑗 for a pixel through search grids at current scale and a coarser scale 𝑐 𝑘 𝑖→𝑗 (𝒒 𝑖 , 𝜹 𝑘 , 𝜹 𝑘+1 ) as follows:EQUATION(𝐹 𝑘 𝑖 [𝒒 𝑖 ] • 𝑆 𝑘+1 (𝐹 𝑘 𝑗 ) [𝐵(𝒒 𝑖 /2 + 𝑀 ∑︁ 𝑚=1 𝑤 𝑘,𝑚 𝑖 [𝒒 𝑖 ]𝑢 𝑘,𝑚 𝑖→𝑗 [𝒒 𝑖 ]/2 + 𝜹 𝑘+1 )])where • and ⊙ refer to vector dot product and channel concatenation respectively, 𝜹 𝑘 , 𝜹 𝑘+1 denotes the displacements in the correlation search grid of size 𝐷 conduct bilinear resizing to yield features with the same spatial dimension of 𝐹 𝑘+1 . Different from correspondence analysis through residual-based single connectivity [21] or Transformer-based dense connectivity [24, 43] , the proposed matching allows knowledge of correspondences with different ranges being accessible and adaptable across scales via dual-view-based sparse connectivity, so a compact model is capable to handle complex scene dynamics to meet the need of video frame interpolation for varying video resolutions.𝑢 𝑡→1 2 𝐼 -1 𝐼 0 𝐼 1 𝐼 2 መ 𝐼 𝑡 𝑓 Η 1 4 Η 1 3 Η 1 2 = Η 1 Η 2 Output ሚ 𝐼 𝑡 2 = ሚ 𝐼 𝑡 {𝐹 0 𝑘 } 𝑘=1 4 {𝐹 1 𝑘 } 𝑘=1 4 {𝐹 -1 𝑘 } 𝑘=1Noting that the correspondence matching in video frame interpolation is subject to the intermediate frame, which is different from correspondence matching subject to only reference frames [36] . We propose to gradually calibrate the dual-view correspondence matching across scales according to the intermediate frame to be synthesized during training by generating its estimate with direct sampling based on multiple bidirectional flow fields and their corresponding weights from coarse to fine.As the calibration is operated in RGB color space through the generation of intermediate frames by directly sampling, we develop auxiliary multi-scale collaborative learning across scales to improve the quality of supervision at coarser scales for better calibration of the proposed pyramidal dual-view correspondence matching by exchanging knowledge, and thus reducing error propagation across scales.To formulate the loss function, denote intermediate motion estimations between the intermediate frame 𝐼 𝑡 to be synthesized and the input reference frames {𝐼 0 , 𝐼 1 } for level 𝑘 as:EQUATIONEQUATIONwhere a smaller 𝑘 represents a finer level. Considering subnetwork 𝐻 1 being composed of 𝐾 -1 nested estimators {𝐻 𝑘 1 } 𝐾 𝑘=2 with the finest-scale estimator being 𝐻 2 1 = 𝐻 1 , we conduct estimation of {𝑢 𝑘 𝑡 →0 } 𝐾 𝑘=2 with the proposed pyramidal dual-view correspondence matching to establish robust correspondence among frames, using feature pyramids generated in a similar way as LiteFlowNet [13] . We define the intermediate flow estimator for level 𝑘 as for test time. To this end, the proposed auxiliary multi-scale collaborative supervision jointly learns teacher 𝐻 𝑇 1 using the regular input-privileged information pairs, computes the teacher output 𝑦 𝑇 𝑡 →𝑖,𝑛 = 𝑢 𝑇 𝑡 →𝑖,𝑛 , 𝑖 ∈ {0, 1}, and learns student 𝐻 𝑘 1 using both the regular input-privileged information pairs and the regular inputteacher output pairs with imitation parameter 𝜆 𝐼 𝑀 ∈ [0, 1]. Accordingly, the loss of auxiliary multi-scale collaborative supervision for sample 𝑛 is computed as:𝐻 𝑘 1 = 𝐷 𝑘 • 𝐷 𝑘+1 • • • • • 𝐷 𝐾 • 𝑓EQUATIONwhere 𝐿 𝐼 𝑀,𝑇 ,𝑘 𝑛 is the imitation loss for the errors of student 𝐻 𝑘 We formulate the imitation loss by considering the error distribution of teachers' predictions. A teacher's prediction is adaptively chosen according to its reliability that is evaluated using the empirical photometric reprojection error. We define the reliability 𝑟 is defined analogously to 𝑟 𝑘 𝑡 →0,𝑛 . The imitation loss is computed as:EQUATIONwhere bilinear resizing 𝑆 𝑘 (•) is used to ensure that the spatial dimension of teacher is the same as that of the student 𝐻 𝑘 1 . Let 𝐿 𝐼 𝑁 𝑛 denote the interpolation loss of the second subnetwork, the total loss for the first subnetwork that composed of a pyramidal network with dual-view correspondence matching is formulated as follows.EQUATIONEQUATIONwhere we use features of frames 𝜙 (•) extracted from 𝑐𝑜𝑛𝑣4_3 of a pretrained VGG16 [33] to construct the perceptual loss with parameter 𝜆 𝑉 𝐺𝐺 . Since 𝐻 1 estimates intermediate frame via direct sampling in color space, its capability of representing complex dynamics may be deficient due to low quality reference frames and flow estimates, causing degradation of video frame interpolation quality. To address this limitation, we develop subnetwork 𝐻 2 that progressively constructs latent representations to refine the estimate of 𝐼 𝑡 with direct sampling. 𝐻 2 is formed by a pyramidal refinement module that deploys the guidance of video dynamics yielded by 𝐻 1 . Instead of using a convolution-based pyramidal network with each scale operating under guidance of the intermediate flow estimation at current scale [26] , we propose to improve the capability of capturing long-range correlations among reference frames and intermediate frame estimates in latent space with flow-guided cross-scale attention. Specifically, to efficiently extract spatiotemporal dynamics of scene contents among reference frames with a relatively small neighborhood, each scale of the refinement module takes as input both warped information guided by intermediate flow estimates of current scale and a coarser scale. As shown in Figure 3 , the extracted scene contents are adaptively transformed according to intermediate flow guidance, based on which we formulate frame refinement as progressive latent representation generations from coarse to fine where attention-based feature fusion among neighboring frames is conducted with Swin transformer blocks [20] .We deploy an interpolation loss similar to equation ( 10) to supervise the second subnetwork, enforcing smaller difference between estimates of intermediate frame refined by the pyramidal refinement module Î𝑡 instead of Ĩ𝑡 and the ground truth intermediate frame to be synthesized. 
experiments 4.1 experimental setup: 
 The proposed method is implemented with Pytorch [27] , using publicly available datasets for training and testing. Our models are trained on samples generated using a mixture of 64, 612 sequences of 7 frames from the Vimeo90K-Septuplet training dataset [42] and 15, 029 sequences of 25 frames from the Adobe240 dataset [35, 41] . For a sequence from the Vimeo90K-Septuplet training dataset, we learn to interpolate the 4 𝑡ℎ frame using the 1 𝑠𝑡 , 3 𝑟𝑑 , 5 𝑡ℎ and 7 𝑡ℎ frames. For a sequence from the Adobe240 dataset, we take the 1 𝑠𝑡 , 9 𝑡ℎ , 17 𝑡ℎ and 25 𝑡ℎ frames of each sequence as inputs and one among the 10 𝑡ℎ to 16 𝑡ℎ frames randomly as target to synthesize. We crop 448 × 256 patches and conduct image flipping and sequence reversal randomly for data augmentation. The training utilizes the Adam optimizer [16] and a batch size of 4, starting with a learning rate of 10 -4 which is reduced by half once for every 2×10 5 iterations, until convergence with around 6 × 10 5 iterations. We use PSNR, SSIM [39] and LPIPS [44] as metrics to quantitively evaluate the performance and additionally deploy tOF [6] as metric to quantitively evaluate temporal consistency for multiple frame interpolation, following [22, 32] . Higher PSNR and SSIM values along with lower LPIPS and tOF values indicate better results.We evaluate the performance for single frame interpolation on 2, 849 sequences of 7 frames with a spatial resolution of 480p from the DAVIS dataset [28] as [7, 41] , 1, 164 sequences with spatial resolution ranging from 640 × 368 to 1280 × 720 from the SNU-FILM dataset that are divided into 4 settings, Easy, Medium, Hard and Extreme, with varying difficulties defined by the temporal gap between input frames as [5] , 8 sequences of 100 frames with a spatial resolution of 4K from the Xiph-4K dataset [22] as [24, 43] . To evaluate the performance for multiple frame interpolation, we utilize the X-TEST benchmark that consists of 15 scenes of 4K video resolution to interpolate 7 intermediate frames in-between reference frames as [32] .For single frame video interpolation, we compare the proposed approach against the state-of-the-art methods, including EQVI [19] , ABME [26] , M2M-VFI [11] , ST-MFNet [7] , SVMV [21] , BiFormer [24] , EMA-VFI [43] and LDMVFI [8] . For multiple frame interpolation, we additionally included XVFI [32] for comparisons. Our enhanced version DvP+ is different from DvP by deploying the proposed pyramidal refinement module. Tables 2 and 3 show quantitative results for single frame interpolation of varying video resolutions on the SNU-FILM [5] , DAVIS [28] and Xiph-4K [22] dataset. Our approach achieves the best perceptual quality in terms of LPIPS [44] on all datasets, while achieving competitive PSNR and SSIM with a relatively compact model size. Compared with previous stateof-the-art scheme ST-MFNet [7] that achieves the best PSNR and SSIM score on most datasets, our DvP model reduces the average LPIPS by 34.03% with 68.57% less number of parameters, our DvP+ model reduces the average LPIPS by 44.50% with 60.95% less number of parameters. Especially, our DvP and DvP+ models obtain Table 1 : Quantitative comparisons of video frame interpolation results on the X-Test [32] dataset. The best and the second-best results are marked in bold and with an underline, respectively. [7] on the challenging Xiph-4K [22] dataset that composed of 4K videos, showing the capability of the proposed approach to more effectively capturing complex and large motions in highresolution videos. This is likely due to the proposed approach is capable of improving the quality of multiple intermediate flow estimation by leveraging the multifaceted knowledge yielded by the hierarchical views at multiple scales in a pyramid network. Compared with recent perceptually-oriented video frame interpolation scheme LDMVFI [8] that explores the power of latent diffusion model [9, 31] in synthesizing perceptually-optimized frames, our DvP+ model achieves average LPIPS reduction by 21.87% with number of parameters over 50 times less than that of LDMVFI [8] , and our DvP model achieves average LPIPS reduction by 5.86% with number of parameters over 65 times less than that of LDMVFI [8] . This demonstrates that the proposed pyramidal refinement module is able to efficiently deploy the representation capability of latent space to facilitate frame synthesis in color space. The results of quantitative comparisons are consists with the visual perceptual comparisons in Figure 1 that contains complex mixture of camera and object motions, and in Figures 4 that involves moving objects with finer details. Our approach generates more visually pleasing results with less artifacts and blurry details. Tables 1 shows quantitative results for multiple frame interpolation of 4K video resolution on the X-Test [32] dataset. Our approach achieves the best perceptual quality in terms of LPIPS [44] and best temporal consistency in terms of tOF [6] on all datasets, while achieving the best PSNR and SSIM. Compared with previous stateof-the-art scheme EMA-VFI [43] and BiFormer [24] that achieve respectively the best PSNR and SSIM score, our DvP model reduces the average LPIPS by 51.04% and average tOF by 36.80 with 1.89 dB PSNR gain on average, our DvP+ model reduces the average LPIPS by 55.36% average tOF by 48.33 with 1.97 dB PSNR gain on average. The results demonstrate that the proposed approach can be better extended to multiple frame interpolation of high video resolution for favorable performance, illustrating the proposed scheme for To understand the effectiveness of the proposed pyramidal dualview correspondence matching, the auxiliary multi-scale collaborative supervision and the pyramidal refinement module, we perform an ablation study by gradually adding these components on two baselines. The results are summarized in Table 4 , where the proposed model without and with a refinement module are compared with corresponding variants in the top 5 rows and bottom 3 rows respectively. The baseline without a refinement module, as shown in line 1, computes the correspondences between neighboring frames for each scale in view of current scale, using correlations among image features at that scale. The baseline with a refinement module, as shown in line 6, constructs each scale of a pyramidal module by taking warped inputs guided by flow estimates of current scale. Effectiveness of pyramidal dual-view correspondence matching. For evaluation, the proposed dual-view correspondence matching is gradually involved from the coarest level 6 to finer levels in a pyramidal network architecture, yielding results of 3 variants as shown in line 2 to 4 of Table 4 . Compared with the baseline in line 1 of Table 4 , the three variants deployed the proposed dualview correspondence matching to seek knowledge in view of both the current scale and a coarser scale instead of in view of only current scale. As such, during decoding of video dynamics, there is correspondence information among neighboring frames from both current and coaser levels for each level. At the expense of a slight increase in the numbers of model parameters, Line 1 to 4 of Table 4 show that the video frame interpolation quality of the pyramidal network architecture increases as dual-view correspondence matching is leveraged in more levels, especially for high video resolution, i.e., 2.22 dB PSNR increase and 26.3% LPIPS drop on X-Test. Accordingly, Figure 5 shows the network yields estimates of intermediate frame and flow of lower quality when dropping auxiliary multi-scale collaborative supervision and pyramidal dual-view correspondence matching, implying the proposed approach being critical for more robust correspondence matching by considering neighboring scales to handle large and complex motions. 4 , both cross-scale flow guidance and attention-based feature fusion contribute to improvements of video frame interpolation quality. Such improvements is more obvious on X-Test, implying the eminence of enlarging receptive field for high resolution videos. 
conclusion: 
 In this paper, we propose a pyramidal dual-view correspondence matching algorithm with an auxiliary multi-scale collaborative supervision for video frame interpolation, which can capture complex video dynamics, utilizing the multifaceted knowledge in a pyramid network. The representation capacity of the network can be effectively enhanced with the proposed pyramidal refinement module where we introduce flow-guided cross-scale attention to generate latent representations through efficient feature transform and fusion. Experiments show the superiority of our model in achieving favorable performance, especially perceptual quality.  
dual-view pyramid network for video frame interpolation: 
 Anonymous Author(s) Submission Id: 4636(a) Overlay (b) EQVI [19] (c) ABME [26] (d) M2M-VFI [11] (e) ST-MFNet [7] (f) SVMV [21] (g) BiFormer [24] (h) EMA-VFI [43] (i) LDMVFI [8] (j) DvP (k) DvP+ (l) GT Figure 1 : An example of video frame interpolation on DAVIS [28] dataset, containing complex mixture of camera and bicycle riding motions as shown by the overlay of reference frames in (a). Previous methods yield blur or distortions at the limb of the rider and in the background as shown in (b-i), while the proposed DvP and DvP+ model synthesize clear contents (j-k).Video frame interpolation is a critical component of video streaming, a vibrant research area dealing with requests of both service providers and users. However, most existing methods cannot handle changing video resolutions while improving user perceptual quality. We aim to unleash the multifaceted knowledge yielded by the hierarchical views at multiple scales in a pyramid network. Specifically, we build a dual-view pyramid network by introducing pyramidal dual-view correspondence matching. It compels each scale to actively seek knowledge in view of both the current scale and a coarser scale, conducting robust correspondence matching by considering neighboring scales. Meanwhile, an auxiliary multiscale collaborative supervision is devised to enforce the exchange of knowledge among current scale and a finer scale and thus reduce error propagation from coarse to fine scales. Based on the robust video dynamic caption of pyramidal dual-view correspondence matching, we further develop a pyramidal refinement module that formulates frame refinement as progressive latent representation generations by developing flow-guided cross-scale attention for feature fusion among neighboring frames. The proposed method 
 
 Appendix:  


Decision: In this submission, the authors propose the Dual-view Pyramid (DvP) network for improving the video frame interpolation with a pyramidal network to enhance the quality of the interpolated frames that utilizes dual-view correspondence matching along with multi-scale collaborative supervision mechanism. 

The reviewers agree that there are several strengths in  the submission ( novelty, experimental results, improved performance). They also identify several limitations (some parts of the paper are hard to follow, need for perceptual quality evaluation, results with broader range of datasets, poor readability, marginal improvement in performance, etc). Given the strengths and the weaknesses of the submission, it is recommended to accept the paper assuming tha the readability issues (and any other fixable issues) will be addressed in the camera ready version.

Reviews: review: paper_summary: paper_summary: The paper proposes a Dual-view Pyramid (DvP) network designed to improve video frame interpolation. It introduces a pyramidal network that utilizes dual-view correspondence matching and a multi-scale collaborative supervision mechanism to enhance the quality of interpolated frames, especially in videos with varying resolutions and complex motions.
strength_weakness: strength_weakness: 1) The introduction of dual-view correspondence matching in a pyramidal network is a novel idea and the experiments demonstrate that it tackles the challenge of long-range correlation and error propagation effectively.
2) The approach facilitates the design of a relatively compact model with fewer parameters than the SoTA, which is beneficial for real-world applications where computational resources may be limited.
3) It demonstrates favorable performance on benchmarks with varying video resolutions, indicating the robustness of the method.
limitations: limitations: 1) The method section, including the equations and modules in the figures is dense and difficult to follow. It would benefit from clearer explanations and figures to illustrate the proposed method and its main modules.
2) These metrics (PSNR, SSIM, LPIPS), while useful, do not fully capture perceptual quality, especially in complex motion scenarios. The paper could benefit from more diverse evaluation metrics, including user studies or perceptual quality assessments.
3) Test the model on a broader range of datasets and real-world scenarios to better understand its generalization capabilities.
paper_summary: paper_summary: The proposed method builds a dual-view pyramid network by introducing pyramidal dual-view correspondence matching. The method achieves favorable video frame interpolation performance on several benchmarks with video resolutions ranging from 480p to 4K, while keeping a model size within 10M.
strength_weakness: strength_weakness: Favorable video frame interpolation performance on several benchmarks with video resolutions ranging from 480p to 4K, while keeping a model size within 10M.
limitations: limitations: 1. The author claims that video frame interpolation for varying resolution videos remains difficult, where larger motions need to be handled with longer-range correlation analysis while avoiding error propagation in this process, but they do not discuss the framework differences between the proposed method and other extreme video frame compression methods.
2. Teacher-student and multi-scale supervision strategies are commonly used in video frame interpolation, like RIFE[1] and IFRNet[2]. The author seems to have changed the original method to 4 frames input.
3. The experimental performance of the proposed method is compared to SOTA methods, but there is not much comparison between the method with 4 frames of input. It is recommended to display the methods with 2 frames of input and the methods with 4 frames of input separately for a fair comparison. The author can add the comparisons with QVI[3], FLAVR[4], VFIT[5], JNMR[6], MA-CSPA[7].

[1] Huang, Zhewei, Tianyuan Zhang, Wen Heng, Boxin Shi, and Shuchang Zhou. "Real-time intermediate flow estimation for video frame interpolation." In European Conference on Computer Vision, pp. 624-642. Cham: Springer Nature Switzerland, 2022.
[2] Kong, Lingtong, Boyuan Jiang, Donghao Luo, Wenqing Chu, Xiaoming Huang, Ying Tai, Chengjie Wang, and Jie Yang. "Ifrnet: Intermediate feature refines network for efficient frame interpolation." In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1969-1978. 2022.
[3] X. Xu, L. Siyao, W. Sun, Q. Yin, and M.-H. Yang, “Quadratic video interpolation,” in Proc. Adv. Neural Inf. Process. Syst., vol. 32, 2019, pp. 1–10.
[4] T. Kalluri, D. Pathak, M. Chandraker, and D. Tran, “FLAVR: Flow-agnostic video representations for fast frame interpolation,” in Proc. IEEE/CVF Winter Conf. Appl. Comput. Vis. (WACV), Jan. 2023, pp. 2070–2081.
[5] Z. Shi, X. Xu, X. Liu, J. Chen, and M.-H. Yang, “Video frame interpolation transformer,” in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit., Jun. 2022, pp. 17482–17491.
[6] M. Liu, C. Xu, C. Yao, C. Lin, and Y. Zhao. “Jnmr: Joint non-linear motion regression for video frame interpolation.” IEEE Transactions on Image Processing, 2023, pp. 5283–5295.
[7] K. Zhou, W. Li, X. Han, and J. Lu, “Exploring motion ambiguity and alignment for high-quality video frame interpolation,” in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), Jun. 2023, pp. 22169–22179.
paper_summary: paper_summary: In this paper, a video frame interpolation method based on dual-view pyramid network is proposed. By introducing dual-view corresponding matching and auxiliary multi-scale collaborative supervision, the performance of video frame interpolation is improved, especially for complex video dynamics.
strength_weakness: strength_weakness: - Introducing pyramidal dual-view correspondence matching to enable robust correspondence between reference frames and the intermediate frame to be synthesized, which captures long-range correlations in the video dynamics.
- Using an auxiliary multi-scale collaborative supervision to calibrate the correspondence matching across scales, which helps reduce error propagation.
limitations: limitations: - The performance is not good enough. On the FILM (Easy) data set, there is still a big gap compared with the best method, which is worth discussing and further improving.
- In introduction part, logical relations are chaotic, and nouns are listed without emphasis, for example, the last paragraph does not clearly describe the key technical details of the proposed method.
- Poor Readability. The equations and sentences in the method are too long to read. At the same time, the symbols lack clear descriptions. The overall logic of the article is poor, and the narrative is not coherent.
- The author provides limited visual results, which are crucial for understanding the performance and perceptual quality of the proposed method.