Title: Controllable Human-centric Keyframe Interpolation with Generative Prior
Venue: NeurIPS 2025
Accepted: yes
Keywords: Keyframe Interpolation; Generative Inbetweening; Video Diffusion Models
Abstract: Existing interpolation methods use pre‑trained video diffusion priors to generate intermediate frames between sparsely sampled keyframes. In the absence of 3D geometric guidance, these methods struggle to produce plausible results for complex, articulated human motions and offer limited control over the synthesized dynamics. In this paper, we introduce PoseFuse3D Keyframe Interpolator (PoseFuse3D-KI), a novel framework that integrates 3D human guidance signals into the diffusion process for Controllable Human-centric Keyframe Interpolation (CHKI). To provide rich spatial and structural cues for interpolation, our PoseFuse3D, a 3D‑informed control model, features a novel SMPL‑X encoder that encodes and aggregates 3D geometry and shape into the 2D latent conditioning space, alongside a fusion network that integrates these 3D cues with 2D pose embeddings. For evaluation, we build CHKI-Video, a new dataset annotated with both 2D poses and 3D SMPL‑X parameters. We show that PoseFuse3D-KI consistently outperforms state-of-the-art baselines on CHKI-Video, achieving a 9\% improvement in PSNR and a 38\% reduction in LPIPS. Comprehensive ablations demonstrate that our PoseFuse3D model improves interpolation fidelity.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nMain goal of the paper is to propose a system for performing frame interpolation focused on human videos. This task is particularly challenging as human motion tends to be complex and highly articulated – which leads to a lot of ambiguity making it hard to model. This proposed solution is to reduce this ambiguity with per-frame 3D and 2D human poses (or control signals). The work introduced a framework to condition a pre-train video diffusion model on such poses and also introduces a dataset focused on human centric key frame interpolation withaccompanying annotations for 2D and 3D human poses (SMPL-X parameters).\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\n* The core-contribution of the paper, PoseFuse3D seems novel and technically sound for adding 3D and 2D control signals to for interpolation. Overall, the idea of resolving motion ambiguity with per-frame 3D poses seems to be valuable and can be used for future work\n\n* The CHKI dataset appears to be a high-quality contribution. Descriptions of the data collection, filtering and annotation pipeline seem to indicate care and rigor in the dataset preparation, making it likely to be useful to the community.\n\n* Paper is well written and the experiments are conducted thoughtfully (except one detail as highlighted in the weakness section). I appreciate the care taken to standardize the backbone (i.e use SVD backbone when comparing against FCVG), making it easier to identify the improvements due this framework.\n\n**Weaknesses**\n\n* It is not clear if the groundtruth intermediate frames are used to extract line features for FCVG when evaluating against FCVG in Table 1 and 2. If not, this is a weakness in the evaluation as motion ambiguity should be removed for FCVG as well for a fair comparison to this method. \n\n* In line with the previous point, I think the approach suggested in section 1.4 of the supplementary material is a critical evaluation criterion for this method. Current method relies on extracting the control signals from the groundtruth intermediate frames. However keyframe interpolation as a problem relies on just the start and end frames and doesn’t not have access to the intermediate frames. So, to support the claim of a keyframe interpolation method, this work should show performance on a typical keyframe interpolation task.\n\nQUESTIONS:\n1. Can you report numbers on an approach similar to what you describe in section 1.4 of the supplementary material? This will help address point 2 of the weaknesses.\n\n2. Can you clarify if lines extracted from groundtruth intermediate frames are used for evaluating FCVG (instead of the interpolations from the start and end frames). If not, can FCVG be evaluated in such a set-up? This will help address point 1 of the weakness.\n\nLIMITATIONS:\nyes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-28"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces PoseFuse3D-KI for controllable human-centric keyframe interpolation. The core of this framework is the PoseFuse3D control model, which integrates 3D human guidance signals (from SMPL-X models) and 2D pose information into a video diffusion model to generate intermediate frames. The key innovation is an SMPL-X encoder that processes 3D model data directly, rather than relying solely on 2D renderings, to better handle complex motions and occlusions. To evaluate their method, the authors built a new dataset, CHKI-Video, containing over 180,000 frames from 2,614 video clips, annotated with 2D poses and 3D SMPL-X parameters. Experiments show that PoseFuse3D-KI outperforms existing methods on this new benchmark.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\n1. The proposed SMPL-X encoder, which directly processes 3D model information instead of just its 2D renderings, is a novel approach to preserving geometric details, especially for occluded parts.\n\n2. The paper contributes the CHKI-Video dataset, a valuable resource for the specific task of controllable human-centric interpolation, as it includes both 2D pose and 3D SMPL-X annotations which are missing in many existing interpolation datasets.\n\n**Weaknesses**\n\n1. My major concern about this paper is the novelty claimed in \"our encoder processes features directly in 3D and integrates projected features through feature aggregation\". Although the claimed novelty mainly focus on the effectiveness of directly utilizing 3D SMPL information, the experiments in Table 1 fails to support this claim: the performance gain of VE+SE over VE/VE+DN is marginal. The overall framework can be viewed as an combination of two established research lines: diffusion-based video interpolation and pose-guided video generation, e.g., Champ. Although the authors want to emphasize the contribution of direct learning from SMPL parameters instead of visualizations in Champ, the experiments does not support such claim. Besides, there is an important ablation study missing: only using SE. If only using SE is way worse than only using VE+DN, it will indicates that the proposed modules is not effective at all compared to Champ (which could be regarded as VE+DN).\n\n2. As we know 2D pose detector is generally more accurate than SMPL pose estimator. The paper relies on the SMPLer-X tool for automated 3D model fitting, supplemented by \"manual verification\". However, it lacks a detailed description of this verification process, such as the scope of manual checks or the final error rate. The quality and accuracy of the underlying SMPL-X annotations are critical for training a robust 3D-aware model, and any systematic errors from the automated tool could compromise the model's performance and the validity of the evaluation.\n\n3. The paper creates a new dataset with 2,614 videos by sourcing videos from Pexels, but it does not discuss or compare its contribution to other recent datasets like HumanVid [a] or MiraData [b], which also use Pexels videos for video-related tasks. Both mentioned datasets have a scale larger than this submission. A discussion of why a new dataset was necessary instead of augmenting existing ones would strengthen the justification for this contribution. For example, HumanVid [a] already curated human-centric 20K videos from Pexels, will this dataset shares a large portion of videos with this submission?\n\n\n[a] Wang, Zhenzhi, Yixuan Li, Yanhong Zeng, Youqing Fang, Yuwei Guo, Wenran Liu, Jing Tan et al. \"Humanvid: Demystifying training data for camera-controllable human image animation.\" In The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track. 2024.\n\n[b] Ju, Xuan, Yiming Gao, Zhaoyang Zhang, Ziyang Yuan, Xintao Wang, Ailing Zeng, Yu Xiong, Qiang Xu, and Ying Shan. \"Miradata: A large-scale video dataset with long durations and structured captions.\" Advances in Neural Information Processing Systems 37 (2024): 48955-48970.\n\nQUESTIONS:\nPlease see the weaknesses. My major concern about this paper is that the experiments fail to support its main contribution: \"effectively extracts control signals from 3D SMPL-X and fuses 2D signals, allowing precise and informative control\", considering that 2D signals are well-studied in Champ [50].\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces PoseFuse3D-KI, a controllable human-centric keyframe interpolation framework that integrates 3D human guidance signals into the diffusion process for human-centric keyframe interpolation (CHKI). This novel framework aims to improve interpolation fidelity, especially for articulated human motion, by leveraging 3D SMPL-X models alongside 2D poses for precise spatial and structural control. The paper proposes a new dataset, CHKI-Video, annotated with both 2D poses and 3D SMPL-X parameters, and demonstrates the effectiveness of the model through extensive experiments, achieving state-of-the-art performance on keyframe interpolation tasks.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. Novel Contribution: The integration of 3D human guidance with 2D poses for keyframe interpolation is a significant improvement over existing methods, addressing the challenges of complex human motion and occlusions. In addition, a new dataset is presented in this paper.\n2. Comprehensive Evaluation: The introduction of a new dataset, CHKI-Video, specifically designed for evaluating controllable human-centric keyframe interpolation, strengthens the paper's contributions and allows for fair comparison with existing methods.\n3. State-of-the-art Performance: The proposed PoseFuse3D-KI framework outperforms existing techniques, showing improvements in PSNR and reductions in LPIPS, indicating its effectiveness in generating high-quality human-centric interpolations.\n\nWeaknesses:\n\n1. Limited Discussion on Scalability: While the paper presents impressive results, there is limited discussion about how well the method scales with different video lengths or more complex human actions, particularly in large-scale applications.\n2. Reproducibility Concerns: The code and data are not currently available, and while the authors plan to release them, this limits the reproducibility of the results.\n3. Computational Efficiency: Although the paper provides implementation details, there is no clear discussion about the computational efficiency of the framework or how it scales with increasing complexity or dataset size. \n4. Potential Overfitting: While the method shows strong performance on the CHKI-Video dataset, it would be useful to see how the model generalizes to other datasets or human motion tasks to avoid potential overfitting.\n5. Limited Practical Expandability: The reliance on 3D motion information (e.g., SMPL-X parameters) may hinder the method’s applicability in real-world scenarios where such annotations are difficult or expensive to obtain. The lack of fallback strategies for 2D-only input scenarios limits the applicability of this framework.\n\nQUESTIONS:\n1.\tHow does the framework perform on a wider range of human actions beyond sports and everyday activities covered in CHKI-Video? Could the method be generalized to other domains, such as robotics or virtual character animation?\n2.\tWhat is the maximum temporal gap (in terms of frames or seconds) that the proposed method can effectively interpolate?\n3.\tGiven that the model combines 3D and 2D signals, how does the performance degrade in scenarios with extreme occlusion or very fast motion? In other words, how does the model address the change in background after a large change in character position?\n4.\tRegarding the 3D auxiliary information aspect, does the proposed method require the SMPLX parameters for all frames, or only for the first and last frames? If the former is the case, can it be argued that the method actually draws on some information from ground truth rather than being truly effective? \n5.\tIn some cases where human SMPLX parameters are not available, does the method proposed in this paper still maintain a high level? In other words, did the authors consider scenarios where 3D information is not available?\n6.\tHas the performance been evaluated using SSIM or other structural similarity metrics? Including such results could offer a more complete view of visual quality.\n7.\tHow much GPU resources are required in the training phase of the proposed method? What about the inference time?\n\nLIMITATIONS:\n1.\tDependency on Specific Datasets: The results presented are based on the CHKI-Video dataset, which may limit the generalizability of the findings. The method's performance on other datasets or real-world video sources remains untested.\n2.\tDependency on SMPLX Parameters: The method proposed by the authors, although capable of generating higher quality videos, is overly reliant on 3D information and lacks extensibility.\n3.\tFuture Work on Data and Code Availability: Although the authors plan to release the code and data after acceptance, the current lack of access limits the paper's ability to be fully reproducible and verifiable by the research community.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

comments: SUMMARY:
This paper aims to improve the performance of Human-centric Keyframe Interpolation (HKI) by proposing PoseFuse3D, which introduces 3D-informed control signals SMPL (rendering and 3D information) to guide HKI.

STRENGTHS AND WEAKNESSES:
Strengths：
1. Existing keyframe interpolation methods cannot effectively address human scenarios, making the scenario addressed in this paper meaningful.
2. The introduction of SMPL-X compensates for the insufficient 3D information in DWPose.
3. It is innovative to utilize both the rendering information of SMPL-X and propose an SMPL-X Encoder to directly encode its 3D information.
4. The results of this paper are sota.

Weaknesses:

1. Introducing human studies would better evaluate the performance of generative interpolation models.
2. I found that most cases provided by the authors target a single person, but when extending to many humans, such as a group of people (more than 5), can PoseFuse3D-KI still work? Can SMPL still estimate accurately?
3. How does the performance fare when humans are not positioned at the center of the frame?
4. Is there any insight into the design of the SMPL-X Encoder?
5. Do you need to consider copyright when using Pixels data?

QUESTIONS:
see weaknesses.

LIMITATIONS:
yes.
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-03