Title: Surface-Aware Feed-Forward Quadratic Gaussian for Frame Interpolation with Large Motion
Venue: NeurIPS 2025
Accepted: yes
Keywords: video synthesis; large motion; multimedia;
Abstract: Motion in the real world takes place in 3D space.
Existing Frame Interpolation methods often estimate global receptive fields in 2D frame space. 
Due to the limitations of 2D space, these global receptive fields are limited, which makes it difficult to match object correspondences between frames, resulting in sub-optimal performance when handling large-motion scenarios.
In this paper, we introduce a novel pipeline for exploring object correspondences based on differential surface theory.
The differential surface coordinate system provides a better representation of the real world, enabling effective exploration of object correspondences.
Specifically, the pipeline first transforms an input pair of video frames from the image coordinate system to the differential surface coordinate system.
Subsequently, within this coordinate system, object correspondences are explored based on surface geometric properties and the surface uniqueness theorem.
Experimental findings showcase that our method attains state-of-the-art performance across large motion benchmarks. 
Our method demonstrates the state-of-the-art performance on these VFI subsets with large motion.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors tackle the task of video frame interpolation by predicting Quadratic Gaussians from the two input frames (along with relative pose). They then project the estimated Gaussians on the target view (with pose given by interpolation under a constant velocity assumption). Performance is measured across a wide variety of datasets, showing a consistent improvement over existing methods.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- The proposed approach is elegant and well motivated\n- Evaluation is extensive and covers a wide range of datasets, with varying settings and difficulty levels.\nAblation experiments are well structured and well motivates the pipeline architecture.\n\nWeaknesses: \n- Lacking comparison to prior works [1,2], which both predict regular Gaussians and poses \n- The connection between Thm 1 and the method is not very clear. For example, it is not explained what “fundamental forms” are. Please add some sentences to draw the connection to surface curvature and normals.\n- Including eq.7 (or similar) from the QGS paper would be helpful to make the background more self contained and make it easier to keep track of notations.\n\n\n[1] Smart, Brandon, et al. \"Splatt3r: Zero-shot gaussian splatting from uncalibrated image pairs.\" arXiv preprint arXiv:2408.13912 (2024).\n[2] Ye, Botao, et al. \"No pose, no problem: Surprisingly simple 3d gaussian splats from sparse unposed images.\" arXiv preprint arXiv:2410.24207 (2024).\n\nQUESTIONS:\n- How is the method able to handle dynamic motions, such as the skateboarding person in Figure 6? A discussion around this in the paper would be informative.\n- How does the proposed method relate to [1, 2] (see s&w section)?\n\nLIMITATIONS:\nI wonder what is meant by “the limitations is that our definition of large motion and the proposed ideas are somewhat naive”. Could you explain further?", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-26"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nTo address the challenges of current video frame interpolation (VFI) methods in handling large motion scenarios, this paper introduces a differential surface representation to model videos, aiming to capture frame-level object correspondences and geometric constraints. Specifically, the authors propose a Surface-Aware Feed-Forward Quadratic Gaussian (SA-FFQG) framework that maps video frames to 3D surfaces, thereby mitigating the limitations of local correspondences. Experimental results demonstrate strong performance on benchmarks with large motion magnitudes.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n\n1. Novelty and Potential Impact: The proposed Feed-Forward Quadratic Gaussian framework exhibits significant novelty in the VFI task and has the potential to inspire future research directions.\n\n2. Pioneering Use of Gaussian Splatting: As the first work to apply Gaussian Splatting in VFI, the method achieves competitive performance compared to prior state-of-the-art approaches.\n\nWeaknesses\n\n1. Clarity of Theoretical Contributions: As a non-expert in Gaussian Splatting, I found the mathematical details in Equations 4-8 challenging to follow. Specifically, I am unsure whether the superscript t in Equations 7 and 8 is necessary, as both equations appear to involve interpolations between two frames. Further clarification would enhance the accessibility of the work.\n\n2. Computational Efficiency: Given the use of a pre-trained VGGT model, the paper does not explicitly report the total parameter count. Table 2 indicates that the proposed method is not outperformed in terms of speed, which may limit its practical applicability in real-time scenarios such as gaming. \n\n3. Generalization to Non-Rigid Motion: Theorem 1 is derived under the assumption of rigid motion, whereas real-world scenarios often involve non-rigid deformations. It remains unclear how this assumption affects the method's generalization capability and practical performance.\n\n4. Incomplete Baseline Comparison: The experiments lack a comparison with the current SOTA method VFIMamba on large motion benchmarks, which weakens the claim of superiority.\n\n5. Limited Visualization: The paper provides few video examples and lacks qualitative visualizations, which undermines the credibility of the reported results.\n\nQUESTIONS:\nQuestions\n\n1. Intuitive Explanation for Global Matching: Could the authors provide more intuitive evidence or visualizations to illustrate why Quadratic Gaussian Splatting (QGS) enables better global correspondences compared to traditional methods?\n\n2. Comparison with VFIMamba: Please include a comparison with VFIMamba to validate the method's performance against the current state-of-the-art.\n\n3. Visual Validation: Additional video visualizations, especially for challenging scenarios with large motions and non-rigid objects, are essential to demonstrate the method's generalization ability.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work presents a new frame interpolation model based on modeling a generalized notion of pixel, named Quadratic Gaussians (QG). Specifically, the model employs foundation 3D reconstruction model (e.g.., DUSt3R) to estimate the 3D coordinates (or Gaussian means) and the camera poses of given video frames, then additionally employ a head network to predict Gaussian splatting parameters (such as the opacity, covariances, etc.) based on the internal features and the outputs of the 3D reconstruction model. Finally, the surface properties such as normals and curvatures are derived in a rule-based manner following their mathematical definition (as given in Equation 4). These features, along with the predicted Gaussian splatting parameters, are rasterized to 2D planes to render the final image (i.e., the interpolation target frame).\n\nThe experiments are performed in several standard VFI benchmarks, and the proposed method achieves comparable or slightly enhanced performance with respect to recent baseline models.\n\nSTRENGTHS AND WEAKNESSES:\n- Strength of the paper includes clear presentation and straightforward model design\n\n- The method directly relies on the outputs of foundation 3D reconstruction model (e.g., DUSt3R), which provides camera parameters and initial 3D coordinates of the pixel points. However, these foundation models are known to be underperforming in videos with dynamic objects, which can potentially harm the generalization performance of the proposed method in various real-world videos\n\n-  The training objective includes reconstructing the surface normal and curvature, however, the details for obtaining ground truth for these features are missing in the current manuscript\n\nQUESTIONS:\n- As noted previously, DUSt3R is known not to generalize well on videos capturing dynamic scenes. Is it possible to employ the dynamics-aware variants, such as the recent CUT3R (Wang et al., 2025)? What would be the impact on the performance on the model, if such variants are employed instead of DUSt3R?\n\n- In the current design, the camera pose of the interpolation target is estimated via linearly interpolating the camera poses predicted by the 3D foundation model. Is there any failure case due to noisy camera pose estimation results?\n\nLIMITATIONS:\nThe proposed design is adapt to the 2 source frame - 1 target frame design for VFI. However, recent foundation 3D reconstruction models (which the proposed method heavily relies on) mainly perform multi-frame inferences to obtain more robust results when affected by complex dynamics. It may not be trivial to generalize the proposed method for these foundation 3D models.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces a novel approach to video frame interpolation by representing video frames as 3D differential surfaces. The authors introduce a new framework, taking inspiration by Abena et al. [22] and Zhang et al. [23], to map frames to Quadratic Guassian responsible for global surface-level alignment unlike previous methods for better object correspondence alignment. Surface properties are rasterized onto the interpolated camera plane and a custom loss function enforces proper alignment therein. Several experiments underline the effectiveness of the proposed method both quantitavely and qualitatively.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. It is the first paper to explore differential geometry in large motion video frame interpolation by treating frames as 3D surfaces.\n2. The architecture itself is modular making it easier to extend and change the method in follow-up work making it more accessible to the entire audience.\n3. Experimental results support the reason for considering differential geometry. The presented results highlight cases where the proposed method outperforms previous SOTA models.\n\nWeaknesses:\n1. Diffusion models, specifically DDPM, are mentioned in the beginning of the paper, however, they are not used in comparison. Possibly due to being time consuming and being challenging for real-time inference (I assume so by reading your Related work section). Yes, in general they are not real-time worthy but as your method fails to be real-time as well it is not common practice to remove good performing diffusion models from comparison because of that exact reason. \"Video Interpolation with Diffusion Models\" by Jain et al. is a paper I would like to highlight that is also able to perform very crisp interpolation of two frames.\n2. In Table 1 you compare methods that have not been trained on the same dataset. It would however be very interesting to see how your model would perform if it was also only trained on the Vimeo-90K dataset in comparison and mention that as training on two datasets theoretically allows for a wider variety of data that the other methods did not have access to. I think it would be fair to neglect the methods SGM-VFI and VFIMamba-S in that case as I see only EMA-VFI-S to be your closer competitor ranking second or first more often than the other two methods combined (It is still fine to keep it in the supplementary material). This is the main reason for the score but I am happy to adjust it if relevant comparisons can be shown.\n3. Only 2 visual comparisons are made which make it hard to judge. Of course in these the model outperforms the others but it would be nice to see a more extensive comparison.\n4. The results on the anonymized GitHub repository and the paper itself do not match. Either there was some compression affecting the result or there is a bigger problem as the shirt of the skater for example is very well readable in the paper images but in the individual images in the gif on the website they are super blurry and not readable at all.\n5. This is a weakness that applies to any frame interpolation method more or less: Usually $t \\ \\in \\ (0,1)$ is set as 0.5 the entire time. However, as you have slow-motion videos or videos with 1000fps at hand would it not be helpful to train the network with varying $t$? The problem that arises from setting $t = 0.5$ the entire time occurs when you want to generate a sequence of images between a starting and an ending image. The quality always relies on the resulting mid-frame generated whereas choosing a free $t \\ \\in \\ (0,1)$ could in theory help supervise smaller movements and preserve more details as well as making the method more robust due to requiring only the starting and ending image for interpolating at any $t$. Did you just follow this way because it is established? (I do understand that this makes it a lot easier when comparing but it makes it harder for the later inference use.)\n\nQUESTIONS:\n1. How was the $\\alpha$ loss weighting factor chosen? What would happen if it was set to a very large value or a very small value? how would results change?\n2. Why have you decided to train on two datasets and compare with methods that are only trained on one dataset? Could you elaborate on this and provide more insights?\n3. Out of curiosity, how well does your model generalize to larger resolutions or how much retraining it would require (interesting as far as FILM is able to generalize quite well for larger resolutions and only requires very little continued training on larger resolutions to hand frame interpolation fairly well give it is only trained on small resolutions)?\n\nLIMITATIONS:\nI highly appreciate the effort and openness for doing a limitations section and mentioning weaknesses. However, this could be more specific and explicit with limiting examples rather than keeping it fairly general (or at least a rewording would make limitations more clear in my opinion). For example: an inherent limitation of the method is the ability to infer geometry from appearance which is not possible in every case, especially in very fine detailed repetitive patterns in rather small areas.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-03"}