Controllable Human-centric Keyframe Interpolation with Generative Prior
Authors: Zujin Guo
Size Wu
Zhongang Cai
Wei Li
Chen Change Loy
Venue: NeurIPS 2025 poster ()
Keywords: Keyframe Interpolation; Generative Inbetweening; Video Diffusion Models
Primary area: applications

Abstract: Existing interpolation methods use pre‑trained video diffusion priors to generate intermediate frames between sparsely sampled keyframes. In the absence of 3D geometric guidance, these methods struggle to produce plausible results for complex, articulated human motions and offer limited control over the synthesized dynamics. In this paper, we introduce PoseFuse3D Keyframe Interpolator (PoseFuse3D-KI), a novel framework that integrates 3D human guidance signals into the diffusion process for Controllable Human-centric Keyframe Interpolation (CHKI). To provide rich spatial and structural cues for interpolation, our PoseFuse3D, a 3D‑informed control model, features a novel SMPL‑X encoder that encodes and aggregates 3D geometry and shape into the 2D latent conditioning space, alongside a fusion network that integrates these 3D cues with 2D pose embeddings. For evaluation, we build CHKI-Video, a new dataset annotated with both 2D poses and 3D SMPL‑X parameters. We show that PoseFuse3D-KI consistently outperforms state-of-the-art baselines on CHKI-Video, achieving a 9\% improvement in PSNR and a 38\% reduction in LPIPS. Comprehensive ablations demonstrate that our PoseFuse3D model improves interpolation fidelity.

Controllable Human-centric Keyframe Interpolation with Generative Prior
Zujin Guo'
Size Wu'
Zhongang Cai² Wei Li
Chen Change Loy'
S-Laa, Nanyang Technological University
SSnsseTime Research
https://gseanccat.gttee httphh...c/oooeett////aaauuuesee KI
Abstract
Existing interpolation methods use pre-trained video diffusion priors to generate intermediate frames hetween sparsely sampled keyframes. In the absence of 3D ge- ometric guidance, these methods struggle to produce plausible results for complex, articulated human motions and offer limited control over the synthesized dynamics. In this paper, we introduce PoseFuse3D Keyframe Interpolator (PoseFuse3D-KI] a novel framework that integrates 3D human guidance signals into the diffusion pro- cess for Controllable Human-cenmn Keyframe Interpolation (CHKI). To provide rich spatial and structural cues for interpolation, our PoseFuse3D, 3D-informed control model, features novel SMPII-X encoder that transforms 3D geometry and shape into the 2D latent conditioning space, alongside fusion network that integrates these 3D cues with 2D pose embeddings. For evaluation, we build Vide-,ideo,
dataset annotated with both 2D poses and 3D SMPL-X pa- rameters. We show that PoseFuse3D-KI consistently outperforms state-of-the-ar baselines on CHKI-Video, achieving 9% improvement in PSNR and 38% reduc- tion in LPIPS. Comprehensive ablations demonstrate that our PoseFuse3D model improves interpolation fidelity.
Introduction
Frame interpolation aims to generate new frames between two consecutive video frames to improve temporal smoothness. Traditional interpolation methods [12 46 [7 assume small, simple motion over short time spans. These methods are challenged when the input frames are widely separated, known as keyframe interpolation or generative inbetweening [56]
11], where the motion between them becomes complex and ambiguous. This challenge is magnified in numan--cnttre videos, where articulated body movements encompass diverse poses and shapes. With human subjects being prevalent in today's 's video content, there is growing need for interpolation methods to handle large temporal gaps and intricate human motion while offering plausible results.
Human-centric keyframe interpolation remains challenging for current methods. Recent ap- proaches [I]] 41 leverage generative priors from image-to-video (I2V) models to bridge the temporal gap. These methods condition the interpolation solely on the input keyframes without inter- mediate guidance. Consequently, they often struggle resolve motion ambiguities and accurately capture the complex articulated dynamics of human motion. For instance, when keyframes involve large occlusions or non-rigid joint movements, these methods often produce implausible or distorted interpolations (Figure [[)) due to insufficient intermediate guidance. FCVG [56] has explored interpolation keyframes with 2D skeletons as control signals for human subjects. However, 2D lines
cannot convey full body shape and geometry, leading to unrealistic results (see Figure Ib)) These
Corresponding authors
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

(b)
(c)
Start Keyfrume
Interpolated Keyframes (4)
End Keyframe
Figure 1: Keyframe Interpolation with Different Strategies. (a) Interpolation using I2V models without intermediate guidance often yields implausible or distorted frames, especially under large motion or occlusion. (b) Skeleton-guided interpolation offers structural cues but lacks geometric detail, resulting unrealistic body shape and appearance. (c) Our PuseFUse3D-KI employs dense human-centric guidance, enabling temporally coherent and visually plausible interpolations.
methods lack fine-grained control over the interpolation process, limiting their ability to produce flexible, high-fidelity human-centric interpolations.
In this study, we investigate the integration of 3D human conditions into the human-centric keyframe interpolation pipeline. Drawing inspiration from recent advances in human animation [55 54 we propose to integrate 2D human poses 448 with 3D SMPL-X models [30] as intermediate control signals. These signals provide precise guidance for complex articulated motions: 2D poses offer concise representations of human joint poses, while 3D models capture rich spatial geometry. How- ever, effectively processing these control signals poses challenges. First, common practice renders 3D human models into 2D proxies (e.g., colored surface, normals, depth maps) before encoding, leading to substantial loss of spatial information in occluded regions. Therefore, we need to develop dedicated encoder that preserves occluded 3D details when converting models into control signals. Second, fusing signals with different information content and granularity is nontrivial. This necessi- tates designing appropriate neural architectures that can accurately extract 3D cues and harmonize them with 2D poses into a unified, informative control input.
To this end, we introduce PoseFuse3D Keyframe Interpolator, termed as PoseFuse3D- -KI, novel pose-control framework for controllable human-centric keyframe interpolation (Figure Our framework is unique in its 3D-informed control model PoseFuse3D, which comprises three jointly trained modules. The first derives control features from visualized conditions; the second encodes and aggregates 3D body geometry into 2D image conditioning features; and the third combines outputs from the first two modules into a unified control signal for interpolation. In contrast to Champ (55), which fuses 3D features from rendered visualizations, encoder processes features directly in 3D and integrates projected features through feature aggregation.
To evaluate the proposed PoseFuse3IDKK KI, we have also built high-quality video dataset for Control- lable Human-centric Keyframe Interpolation (CHKI). Most existing interpolation datasets, such as SportsSlomo [6), target small temporal gaps, lack annotations for 2D poses or 3D human models, and offer limited human-centric motion diversity. Therefore, we introduce CHKI- CHKI-Video, a new dataset for systematic evaluation of CHKI algorithms. CHKI-Video comprises 2,614 high-quality video clips of over 180K frames sourced from SportsSlomo [6] and Pexels [ website that hosts high-quality stock videos. Each frame is carefully annotated with bounding boxes, segmentation masks, 2D human poses, and SMPLLXX parameters, using state-of-the-art tools 4 20 34 45 supplemented by manual verification. From this collection, we derive a benchmark specifically for the CHKI task. We hope this benchmark will help improve controlled keyframe interpolation techniques with its high-quality videos and diverse examples.
Our contributions are threefold: (i) We present PoseFuseeD-KI, an effective interpolation framework for human-centric keyframe interpolation, characterized by a novel pose-control model, PoseFuse3I.. effectively extracts control signals from 3D SMPI--X and fuses 2D signals, allowing precise and informative control. (ii) For evaluation, construct CHKI-Videw benchmark dataset with

---

comprehensive cuman-ccettre annotations, which are absent in existing interpolation benchmarks. (iii) Through extensive experiments. we demonstrate that PoseFuse3D-K delivers of-te-of-the-are performance on our CHKI-Vider benchmark with an improvement of 1.85 dB in PSNR and reduction of 0.0796 in LPIPS.
2 Related Work
Frame Interpolation. Traditional frame interpolation methods are primarily designed for temporally
adjacent frames and rely on either direct synthesis using convolutional networks 8 or motion
representations such as dynamic kernels (28 29
32
7 9 [2] and optical flows [16 23 21
37
14
12
extend this task more challenging keyframe interpolation scenarios by leveraging the generative priors of image-to-vided diffusion models B 39
17
44 50. Recent advances [1
35 [3]. These methods combine temporal forward and reverse denoising predictions in a unified process to enable interpolation. However, such approaches struggle when faced with complex articulated motions or ambiguous transitions from human keyframes. To alleviate motion ambiguity in interpolation, FCVG [56] introduces 2D matched lines control signals. However, this control signal lacks the 3D geometric context required for plausible human-centric interpolation, resulting in unrealistic body shape and appearance.
Pose-Guided Human Animation. Recent advances in pose-guided human animation
33
harness the power of diffusion models and have achieved remarkable success in generating videos from single reference image. These methods offer flexible and precise control by incorporating enriched conditioning signals and increasingly sophisticated control mechanisms. 2D human poses, such as OpenPose
51
40
(5] and DWPose [48], are widely used existing works [55] 54 33 [1], but they are limited in capturing fine-grained geometry and motion dynamics. To address this, recent works [53] 54 integrate 3D human parametric models [27 30,, which offer realistic body representation through blend shapes and skinning, resulting in more accurate and expressive human animations. Specifically, Champ [55] renders 3D human models into 2D proxies (c.g., normals, depth, and semantic maps) and combines them with 2D pose visualizations
control input. It operates directly on these visualizations and unifies them using a simple summation operation. To incorporate adapt ControlNet [51] for their customized control networks. Some methods (55) introduce task-specific pose guiders but often require retraining the majority of denoiser parameters. ControlNeXt (33) improves control efficiency by encoding conditions with a lightweight convolutional network and injecting them via cross- normalization after the first denoising block. tuning only a minimal subset of parameters. This efficient mechanism enables robust conditioning over large-scale pre-trained video generators 33
control signals into diffusion models, many approaches 44 45
51
54
In this paper, demonstrate the advantage of combining 2D poses (DWPose 48] with 3D human models (SMPL-X (30)) through comprehensive experiments. We present a novel pose control model that extracts unified, 3D-informed control features provide precise guidance. Instead of relying on rendered normals and depth maps, the model directly extracts explicit 3D information from human models in 3D space, preserving richer control signals. It adopts a ControlNext-inspiree strategy to control video diffusion models for keyframe interpolation.
Method
Given human-centrie keyframe-pair Tu, IN E Hx with timesteps {0, N), Controllable Human-centric Keyframe Interpolation (CHKI) is formulated as:
(i,) (To, IN, (C)
where G denotes an interpolation model guided by control signals {C.). In this work, we aim to address CHKI by proposing an effective controllable interpolation framework, POS This framework integrates 3D-aware human-centric signals into a pre-trained Video Diffusion Model (VDM) through our 3D-informed control model PoseFuse3D, as illustrated in Figure a)..
3.1 PoseFuse3D
PoseFuse3I is 3D-informed control model that provides 3D human structure and geometry guidance for plausible human interpolation. This 3D-informed guidance is injected into the base diffusion
3

---

Visualicatoon
PeseFusc3D
Condition Fusion Modul
Curr Enooder
SMILXX
3D Juini Iado
3D Coord Infe
2D Goood Iudo
Corv Esoskr
Chnabiio Plaki
Choe
Set- Alannkon
Jee
Nere
Teapueal Salf-Ansster
20 Malagiii
SMPL-X Paseniis (8,).s)
sviPt X y
Beoadr
X2
Coes
pM
ML
8
Norm
Vensa
VAI Dooakr
Trapoesi
CavzD Reallskk
Douncerphe
(4i7
(4) PoseFusc3D-KI Framework
(b) SMPL-X Encuder
Figure 2: Model Architecture. Our PoseFuse3D-KI framework, as shown in (a), comprises video diffusion model (VDM) and a novel control model, PoseFuse3D. The PoseFuse3D model extracts rich features from both 3D and 2D control signals and fuses them into unified representation to guide the VDM. The key component of PoseFuse3D is the SMPL-X encoder as illustrated in (b), which provides explicit 3D signal features. Specifically, the SMPL-X encoder first extracts 3D information from the SMPL-X model with 2D correspondences via projection. The 3D and 2D information is then encoded in parallel. With features of 2D correspondences, 3D information is aggregated onto the 2D image plane using attention mechanisms. The aggregated features are subsequently processed to produce the final feature 53D
model after the first denoising block via cross- noss-normalization 33 Internally, PoseFuse3D comprises three jointly trained components: a visual encoding module that derives control features from 3D SMPL-X (30 renderings and 2D DWPose 48] visualizations; a SMPL-X encoder that directly embeds 3D humans and aggregates them into image conditioning maps; and a fusion module that integrates encoding streams into unified control tensor to guide interpolation in the base VDM.
Visual Encoding. This visual encoding module extracts conditional features from visualized control images, maintaining natural pixel-level alignment with the controlled video latents. To enhance the control signals, we incorporate visualizations from both DWPose and SMPL-X. While SMPL-X renderings provide rich human surface details, their keypoint information is indirect, mixed with other vertices and mesh faces. Therefore, we add DWPose visualizations emphasize the skeletal keypoint layout for robust pose understanding. This combination was also demonstrated to be effective in prior work 55 Specifically, our visual encoding module employs two parallel convolutional encoders to capture comprehensive pose information. One encoder processes DWPose visualizations to capture compact pose information, while the other handles SMPL-X renderings that retain 3D cues such as occlusion boundaries and projected shapes. Notably, enrich semantic detail, we use SMPL-X Colored Surface (54) during rendering, which assigns a unique color to each vertex. The resulting feature maps, E2D and F&D, are then passed to the fusion module for unified conditioning.
SMPL-X Encoder. Although 3D model renderings offer aligned image maps for conditioning in VDMs, the rendering operation discards parts of 3D information, particularly in occluded regions. This results in implausible interpolation for keyframes of large human motion. For example, in the first case of Figure 3 all variants relying solely on rendered controls fail to interpolate the occluded arms with correct spatial position. To enhance controllability with direct 3D information. we introduce SMPL-X encoder (Figure (])) that processes the SMPL-X model in 3D space and transforms it into an image conditioning feature 53D Specifically, a SMPL-X model [30] is parameterized with (0, B, v)), corresponding to pose, shape, and expression. We obtain structural information by forwarding these parameters into the SMPI.-X model to generate vertex and joint coordinates p&D j3D in 3D space and obtain their corresponding 2D coordinates p2D J2D through

---

projection:
p3D _3D SMPLX v),
(2)
p2D , Projection( (pD jaD),
where the joint coordinates jaP, J2n correspond to the pose parameter that indicates joint rotations. Notably, the projection step establishes correspondence between the 3D) space and the 2D image plane, making it possible to retain 3D spatial structure while producing image conditioning maps. Next, the raw 3D coordinates and joint features. These joint features are refined through temporal residual block that fuses them with pose information into expressive joint-level representations V To aggregate these 3D features into 2D image control maps for joints and vertices, we employ separate attention mechanisms:
processed using an MI.P to produce point-wise vertex features VP
0 Joint K", VVV), OP (Qn KP, VP),
(5)
where Q Bx the flattened d-dimension feature of a standard 2D meshgrid extracted by convolutional encoder. Finally, the outputs O and OP concatenated and passed through downsampling block to produce the final SMPL-X control representation s&D, which serves as an informative and compact image embedding of the underlying 3D human structure.
The condition fusion module combines control features from both 2D and 3D signals into unified representation to guide keyframe interpolation. For robust feature representation, we introduce coarse-to-fin fusion strategy that progressively integrates rich geometric information from the 3D features into the compact 2D pose features. Specifically, we adopt two attention- based fusion blocks to perform this integration, where each block contains three attention layers for progressive refinement. The first layer is self-attention module that processes the 3D features by operating on the sum of rendering encoding E&D and SMPL-X features 5&D The second layer performs cross attention, aligning the 3D features with the 2D encoding E²D through spatially localized interaction scheme. Notably, we adapt the shifted handow-partition restrict attention computation to adjacent regions, enhancing local alignment. The third layer applies temporal self-attention to capture temporal dynamic correlations in the fused representation. We use the second fusion block output the final control signal, which is injected into the base interpolation engine to provide greigga structure-aware guidance during synthesis.
Condition Fusion.
strategy [26 50 to
3.2
VDMs as Base Interpolation Engine
To supply generative priors for human-centre keyframe interpolation, we adapt pre-trained Video Diffusion Models (VDMs) under the latent diffusion framework [35]. VDMs perform the diffusion process in VAE-encoded latent space, conditioning video synthesis input frames. Our main experiments investigate Wan2.1 [39 as the base model for keyframe interpolation.
Wan2.1 consists of scaled-up DiT- denoiser [3 and a causal 2D-VAE that performs spa- tiotemporal compression. It employs the flow matching strategy 24 10 for the diffusion process. formulates the forward diffusion process as a linear interpolation between the clean video latent and the noise e, which adds the noise by: z; = (1 t)z + fc. the backward process, the denoiser fo iteratively refines conditioned on the first frame To. The training objective is formulated as:
L= en
() y11))-
Since the latent space is unevenly compressed in time, where the target objective y dt that combine temporally forward and reverse denoising paths are noise fusion strategies [11, [56]
ineffective. Instead, we adapt Wan2. I for keyframe interpolation in one unified denoising process. Specifically, we condition Wan2. on both endpoint frames To, Tv. and apply LoRA tuning to its input patch embeddings. Additional details are provided in the supplementary material.
The CHKI- Video Dataset
which are designed for interpolating temporally adjacent frames, are not suitable for the CHKI task. To address this gap, we built the CHKI-Video dataset in three consecutive stages. More details can be found in the supplementary material.
Existing interpolation datasets 46
6
36
38
5

---

Stage 1: Dataset Collection. To create a dataset with challenging and diverse human motion, we curate video clips from SportSlomo 66] and Pexels (.. While SportsSlomo provides challenging human-centric videos, its exclusive focus on sports limits the diversity of activities. To enhance diversity versus the reality [49], we compile a list of keywords, spanning everyday activities to high-intensity actions. We use these keywords to retrieve videos from Pexels, complementing the sports videos. The curated videos are then downsampled to eliminate frame redundancy unnecessary for keyframe interpolation.
Stage 2: Pre-annotation Processing. We first perform general filtering for the low-quality videos according to brightness changes and assessed scores 43 Then, we use Grounding-Diin DINO [25] and SAM2 [34 detect, segment, and track human instances in each video. We discard any video of more than three people or that is shorter than 20 consecutive frames to ensure sufficient temporal span for keyframe interpolation. After automated processing, we manually review and filter detections in complex sports scenarios.
Stage 3: Human-centric Annotation. Building on the accurate human detections obtained in Stage we annotate each clip for precise human-centric information. First, we employ Sapiens [20] to extract 2D human keypoints and perform whole-body detection to filter out clips with incomplete human figures. This ensures our dataset remains strictly human-centric. Finally, we apply SMPLer-X A) leveraging its high re- re-projection accuracy to fit detailed SMPL-X models for human images and produce reliable 3D body parameters for each frame.
As result, CHKI-Video comprises 2,614 video clips of over 180K frames, carefully annotated with human bounding boxes, masks, 21) keypoints and 3D parametric models. To prepare the train and test split, follow the original division for SportsSlomo [6] videos and distribute the Pexels videos according to their keyword frequencies to maintain balanced coverage of all motion categories.
5 Experiments
We present quantitative and qualitative results in Sec. 5.1 to validate the effectiveness of our 3D control strategy in PoseFuse3D. We compare our interpolation performance against state-of offte-of-the-art the-art methods in Sec. 5.2 and analyze the scalability across temporal gaps in Sec. [.3] We further assess the in-the-wild interpolation capability in Sec. where ground-truth control signals are not available. Finally, Sec. 5.5 provides a detailed ablation study to justify our model design.
Implementation Details. We fine-tune PoseFuse3D-KK KI on the SHKI-Vider training split for 70k iterations. Specifically, we fine-tune our 3D-informed control model PoseFuse3D, and employ LoRA on the input patch embeddings, modules. During training, we randomly sample 25 consecutive frames from video clips and process them to resolution of 512 320. To maximize largest annotated human bounding box. Given the ratio from the target input size, we first crop the maximum scale of the image with the largest box the center. The cropped image is then resized to match the target resolution. More implementation details are provided in the supplementary material.
well the value and output projections of the VDM's attention
content, crop each clip around its
Evaluation Protocols. Unless otherwise noted, all methods are evaluated on the CHKI- Video test set using ground-truth annotations for controllable interpolation. With motion ambiguity limited by the ground truth controls, we adopt the standard interpolation metrics of PSNR and LPIPS computed on the whole images for evaluation. To quantify performance specifically on human regions, we further leverage the annotated human boxes and masks, yielding PSN LPIPS PSNRmask and LPIPS mask metrics. Notably, we apply binary dilation to expand and smooth the masks before computing the metrics, preventing potential artifacts along mask boundaries.
5.1
3D Control Strategy in PoseFuse3ID
Setup. We evaluate the effectiveness of the 3D-informed control design in PoseFuse3D via compar- isons among 3D control strategies, including 'VE', 'VE+DN' and 'VE+SE'. Specifically, VE refers to the visual encoding on the SMPL-X colored surface rendering, removing the SMPL-X encoder from the full design of our control model. VE+DN extends VE by incorporating depth and normal renderings, using two additional encoders identical to the one used in VE. VE+SE represents our proposed strategy in PosDDuse3D that directly encodes 3D information via the SMPL-X encoder

---

Table I: Comparisons of 3D Control Strategies. This table presents quantitative results of different 3D control strategies. The PSNR and LPIPS metrics are calculated for the whole image, as well as for the human-centrie parts via annotated boxes or masks. Best results are highlighted with boldface.
Evaluation Metrics
Method
Backbone
3D Control Strategy
PSNR1
PSNR ? maNR
LPIPSI
LPIPS mack
FCVG
SVD SVD
N.A.
20.42
18.05
16.91
0.2100
0.0899
0.0606
PoseFuse3D
VE-SVD
20.96
18.56
17.47
0.1975
0.0835
0.0553
PoscFusc3D
Wan2.1-12V
VE
21.91
19.13
17.87
0.1400
0.0682
0.0484
PoseFuse3D
Wan2. Wan2.1-12V
VE+DN
22.07
19.12
17.80
0.1363
0.0667
0,0473
PoseFuse3D
Wan2.1-122 I-12V
VE+SE (Ours)
22.14
19.30
18.01
0.1330
0.0653
0.0464
Overlaid Input
FCVG
VE-SVD
VE+DN
VE+SE (Ours)
Ground Truth
Figure 3: Qualitative Results of Different 3D Control Strategies. We use red circles to highlight
regions where the 3D controls and our strategy significantly improve the interpolation quality.
and integrates it with VE. Experiments for these strategies use Wan2.1-12V 39 as the interpolation backbone. For efficiency, they are trained for 40K iterations.
To assess the necessity of 3D information, we compare against FCVG [56], which is conditioned solely on 2D signals. For fair comparison, we create a variant of by replacing the backbone with SVD 3 We adapt SVD for interpolation with the temporal forward and reverse denoising path fusion strategy used in FCVG. This variant referred to as VE-SVD for case of analysis.
Results. We provide quantitative comparisons of 3D control strategies in Table In comparisons between FCVG and VE-SVD, we find that adding 3D control improves the interpolation performance. VE-SVD outperforms FCVG across metrics with more than 0.50 dB increase on all the PSNRs, indicating the improvements on both whole-image and buman-centric levels. Moreover, our study highlights the importance of explicit 3D information. VE+DN and VE+SE, which incorporate depth and normal maps or direct SMPL-X information, outperform the simpler strategy VE. VE+DN and VE+SE show clear improvements in perceptual quality as reflected by the LPIPS metrics. Notably, our VE+SE strategy, which directly encodes information in 3D space, delivers the best performance, achieving the lowest LPIPS of 0.0653 and the highest PSNR of 22.14 dB.
Visualizations. Besides our quantitative analysis. we perform qualitative comparison of the 3D control strategies, as shown in Figure 3 We find that incorporating 3D controls better preserves human shape during interpolation. Take the Tennis case in Figure example, methods with 3D control strategies interpolate the player's body close to the ground truth, whereas FCVG exhibits noticeable distortion. Moreover, our VE+SE, which directly encodes 3D information from SMPL-X, proves effective in handling occluded human motion. In both the Skateboarding (1*t row) and Jumping (2nd row) cases in Figure we can observe that our VE+SE strategy produces plausible results for the occluded arms, demonstrating its ad advantage in complex scenarios.
7

---

Table 2: Comparisons with State-of-the-art Interpolation Methods.
Metrics
Methods
PSNR1
PSNRwas
PSNR
LPIPS)
LPIPS
L
HA1
GIMM-VFI 12
20.29
16.36
14.93
0.1954
0.1187
0.0860
0.9146
GI 41
15.81
13.04
12.03
0.3364
0.1672
0.1146
0.8954
Wan2.1 KI (Ours)
15.96
14.62
0.2081
0.1208
0.0868
0.9180
FCVG 56
20.42
18.05
0.0899
0.0606
0.9187
PoscFuse3D-KI (Ours)
22.27
19.49
18.24
0.1304
0.0636
0.0450
0,9189
Table 3: Results across Temporal Gaps.
Metrics
Temporal Gap
PSNR
PSN
PSNRmakk
LPIPS
LPIPS
LPIPS
---rrame
23.86
21.64
20.70
0.1144
0.0508
0.0332
48- 48-frame
21.46
19.48
18.57
0.1566
0.0644
0.0432
64-frame
20.37
18.34
17.45
0.1882
0.0755
0.0514
96-frame
19.44
17.50
16.66
0.2208
0.0836
0.0577
5.2 Benchmark Results
Setup. We compare PoseFuse3D-KG KI against several advanced interpolation methods on our CHKI- Video dataset. The main comparison is with FCVG [56]. which also enables intermediate control during interpolation. For broader coverage, we also include the keyframe interpolation method GI [4] and the traditional video frame interpolation method GIMM-VFI [12]. We also include Wan2.1-KI an adaptation of Wan2.1 (39 for keyframe interpolation, following the strategy in Sec. 3.2 Since not all baselines support ground truth controls, additionally compute the Human Anatomy (HA) 53
score to assess the quality of human synthesis and support fair comparison.
Results. We report quantitative results of PoseFuse3D-KI on the CHKI-Video benchmark in Table 2 Our method delivers state-of-the-ar performance on human-centric keyframe interpolation. On whole-image metrics, it boosts PSNR by 1.85 dB and lowers LPIPS by 0.0796 in comparison with the state-of-the-arr method FCVG [56]. Crucially, it also outperforms on numan--cntris metrics compared with other methods, achieving PSNRabox of 19.49 dB, LPIPS mask of 0.045, and an HA score of 0.9189. This indicates that our method produces plausible, high-fidelity human interpolations that closely follow the ground-truth dynamics, demonstrating the effectiveness of our method.
Visualizations. For qualitative evaluation, we qualitatively compare PoseFuse3D-KG with other advanced methods, as illustrated in Figure E Our approach delivers more accurate human inter- polations, faithfully following real-world motions and preserving body shape. For example, in the 2nd 'Fencing" case and 4th "Stunt Bike' case, only PoseFuse3D-Ki correctly interpolates leg and arm movements while maintaining consistent shapes. Moreover, our method naturally handles the occluded human motion well with 3D-informed control. We observe that method interpolates correct spatial positions for the occluded legs (1st case) and arms (3nd case), achieving significant improvements over FCVG, Furthermore, although the control-free ffrrrl--free keyframe interpolation methods GI and Wan2. 1-KI occasionally generate undistorted humans, they often generate implausible motion that violates real-world dynamics, observed in 1st, 2nd and 4th cases.
5.3 Scalability Across Temporal Gaps
Setup. We evaluate the scalability of PoseFuse3D-E DI in interpolating frames across different temporal gaps. The temporal gap is defined the number of frames between the input keyframes, ranging from 24 to 96 frames. Evaluations are conducted on the CHKI-Vide Video test set using PSNR and LPIPS as quantitative metrics.
We provide quantitative results PoseFuse3D-KK across different temporal gaps in Table As common trend observed in most interpolation methods, the interpolation performance gradually declines as the interval between keyframes increases. Nevertheless, PoseFuseeD-R maintains strong performance over wide range of temporal settings. Notably, it achieves a PSNR of 23.86 dB and an LPIPS of 044 at 24 frames, and PSNR of 21.46 dB and an LPIPS of 0.1566 at 48 frames. Even at large gap of 96 frames, where the task becomes substantially more challenging,
Results.
8

---

Keylrame
Wanz. KKI
End Keylrame
use
Ground
Keyframe
Wanzi EN
FO
Keylrame
(Ours)
GINNSEYI
Start Keytrame
Wan2
End Keyframe
3D (Ours)
Ground Truth
GIMMEVFI
Keytrame
Wan2.1KKi
FCVG
End Keyframe
Posel use KI (Ours)
Ground Truth
Figure 4: Qualitative Comparisons with State-of-The-Art Methods.
PoscFuse3D-KK KI remains competitive, achieving PSNR of 19.44 dB and LPIPS of 0.2208. These results demonstrate the robustness of PosscFuse3D-K under long-range controllable interpolation.
5.4 In-the-wild Interpolation Results
Our PoseFuse3D-KI framework be readily applied to interpolate in-the-wild human-centric keyframes without ground-truth control signals.
Setup. We assess the in-the-will wild interpolation capability of PoscFuse3D-KI on the CHKI-Vides dataset by discarding the ground-trute annotations. The comparison is conducted against the state-of- the-art method FCVG [56]. Since ground-truth controls are unavailable in this setting, employ simple strategy that linearly interpolates human body joints to generate intermediate control signals. Details of this pipeline provided in the supplementary material. For performance measurement, we compute the same metrics as described in Sec. 5.2
Results. We report quantitative results of PoseFuse3ID-K under the in-the-wild interpolation setting Table Our method achieves state-of-the-art performance and delivers significant improvements over FCVG [56). On whole-imane metrics, it boosts PSNR by 4.5% and reduces LPIPS by 22.1 %

---

Table 4: Comparisons with FCVG on In-the-wild Interpolation.
Metrics
Methods
PSNRT
PSNR
PSNRmss 1
LPIPS
LPIPS
LPIPS
HA1
FCVG 56
18.47
15.31 15.75
14.08
0.2607
0.1321 0.1194
0.0921
0,9284
M-seFuse3D-K. (Ours)
19.30
14.46
0.2031
0.0859
0.9289
Table 5: Ablation Study on Visual Encoding and Fusion Module.
Model Variants
Evaluation Metries
Visual Encoding Fusion Module
PSNR
PSN f mask
LPIPS]
p |
LPIPS |
Conv-Ene (2D)
N.A.
20.29
16.94
16.10
0.1990
0.1096
0,0836
Dual Conv-Ene
Sum
20.50
17.20
16.36
0.1953
0.1059
0,0806
Dual Conv-Enc
Non-TSA
20.30
16.96
16.12
0.1956
0. 1053
0.0790
Dual Conv-Enc
Non-Wi WP
20.46
17.19
16.37
0.1938
0.1038
0,0775
Dual Conv-Enc Full
20.55
17.30
16.48
0.1927
0.1031
0.0773
compared with FCVG. More importantly, PoseFuse3I-KI attains the best results on perceptual human-centric metrics, with an LPIPSmast of 0.0859 and an HA score of 0.9289. These results demonstrate that PoseFuse3D-KK effectively handles in-the-wild interpolation.
5.5 Ablation Study
In this section, we ablate the visual encoding and fusion modules of PoseFuse3D in Table [ For efficiency, we use SVD the backbone and process video clips into 9 consecutive frames of 256 x 256. We use 3x temporally downsampled videos from the CHKI Video test set for evaluation.
3D Visual Encoding, Our visual encoding module includes two convolutional encoders for 2D and 3D control maps, respectively. We denote the variant including this entire module 'Dual Conv-Enc', and the one using only the 2D encoder as "Conv-Enc (2D)', Removing the 3D visual encoding leads performance drop of 0.26 dB on both PSNR and PSNRmask highlighting the importance of 3D visual encoding.
Fusion Module. In PoseFuse3D, condition features fused through carefully designed fusion module. To validate its effectiveness, we replace it with a simple summation operation (55), denoted 'Sum' in Table This change leads significant performance drop, particularly in perceptual quality, with an increase of 0.0033 in LPIPS These results demonstrate the fusion module's contribution to providing informative control for high-quality interpolation.
Window-Partition Strategy. PoseFuse3D employs a cross-attention layer with a shifted window- partition strategy to fuse features across neighboring windows. To validate this design, we remove the window partitioning, denoted as Non- WP. This results notable drops of 0.11 dB in both PSN and PSNRmask. indicating that the window-partition strategy enhances controlled interpolation.
Temporal Attention in Fusion Module. To justify the efficacy of the temporal self-attention (TSA) layer in the fusion module, we conduct experiments excluding the TSA layer (Non-7SA). This removal causes increases of 0.0022 and 0.0017 in LPIPS and LPIIS demonstrating the crucial role of the temporal self-attention layers in the fusion module.
6 Conclusion
We propose PoseFuse3D-KL controllable human-centric keyframe interpolation framework powered by novel 3D-informed control model, PoseFuse3D. PoseFuse3D embeds rich spatial geometry from 3D human signals together with 2D poses into unified control feature, enabling the generation of more plausible and realistic intermediate frames. For evaluation, construct CHKI-Vider dataset with comprehensive human-centrie annotations. Extensive experiments on the benchmark demonstrate that PoseFuse3D-KI outperforms previous interpolation methods with 9% improvement in PSNR and a 38% reduction in LPIPS.
10

---

This study is supported under the RIE2020 Industry Alignment Fund Industry Collaboration Projects (IAF-ICP) Funding Initiative, as well as cash and in-kind contribution from the industry partner(s).
Acknowledgement.
References
[1] Pexels. URL https::/www.pxxellls. comvvideos// 2025.
[2] Wenbo Bao, Wei-Sheng Lai, Chao Ma, Xiaoyun Zhang, Zhiyong Gao, and Ming-Hsuan Yang. Depth-aware video frame interpolation. In CVPR, 2019.
[3] Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Do- minik Lorenz, Yam Levi, Zion English, Vikram Voleti, Adam Letts, al. Stable Video Diffusion: Scaling latent video diffusion models to large datasets. arXiv preprint arXiv:2311.1512) 2023.
[4] Zhongang Cai, Wanqi Yin, Ailing Zeng, Chen Wei, Qingping Sun, Wang Yanjun, Hui En Pang, Haiyi Mei, Mingyuan Zhang, Lei Zhang, et al. SMPLer-X: Scaling up expressive human pose and shape estimation. In NeurlPS, 2023.
[5] Zhe Cao, Tomas Simon, Shih-En Wei, and Yaser Sheikh. Realtime multi-person 2d pose estimation using part affinity fields. In CVPR, 2017.
[6] Jiaben Chen and Huaizu Jiang. Sportsslomo: new benchmark and baselines for human-centric video frame interpolation. In CVPR, 2024.
[7] Xianhang Cheng and Zhenzhong Chen. Multiple video frame interpolation via enhanced deformable separable convolution. IEEE TPAMI, 2021.
[8] Myungsub Choi, Heewon Kim, Bohyung Han, Ning Xu, and Kyoung Mu Lee. Channel attention all you need for video frame interpolation. AAAI, 2020.
[9] Tianyu Ding, Luming Liang, Zhihui Zhu, and llya Zharkov. CDFI: Compression-driven network design for frame interpolation. In CVPR, 2021.
[10] Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederie Boesel, et al. Scaling rectified flow transform- ers for high-resolution image synthesis. In ICML, 2024.
[11] Haiwen Feng, Zheng Ding, Zhihao Xia, Simon Niklaus, Victoria Abrevaya, Michael J Black, and Xuaner Zhang. Explorative inbetweening time and space. In ECCV, 2024.
[12] Zujin Guo, Wei Li, and Chen Change Loy. Generalizable implicit motion modeling for video frame interpolation. In NeurIPS, 2024.
[13] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In NeurlPS, 2020.
[14] Xinyu Hou, Liming Jiang, Rui Shao, and Chen Change Loy. Video infilling with rich motion prior. In BMVC, 2023.
[15] Li Hu. Animate Anyone: Consistent and controllable image-to-video synthesis for character animation. CVPR, 2024.
[16] Zhewei Huang, Tianyuan Zhang, Wen Heng, Boxin Shi, and Shuchang Zhou. Real-time intermediate flow estimation for video frame interpolation. ECCV, 2022.
[17] Huaizu Jiang, Deqing Sun, Varun Jampani, Ming-Hsuan Yang, Erik Learned-Mille and Jan Kautz. Super SloMo: High quality estimation of multiple intermediate frames for video interpolation. In CVPR, 2018.
[18] Xuan Ju, Yiming Gao, Zhaoyang Zhang, Ziyang Yuan, Xintao Wang, Ailing Zeng, Yu Xiong, Qiang Xu, and Ying Shan. MiraData: A large-scale video dataset with long durations and structured captions. In NeurIPS, 2024.
11

---

[19] Tarun Kalluri, Deepak Pathak, Manmohan Chandraker, and Du Tran. FLAVR: Flow-agnostic video representations for fast frame interpolation. WACV, 2023.
[20] Rawal Khirodkar, Timur Bagautdinov, Julieta Martinez, Su Zhaoen, Austin James, Peter Seledník, Stuart Anderson, and Shunsuke Saito. Sapiens: Foundation for human vision models. In ECCV. Springer, 2024.
[21] Lingtong Kong, Boyuan Jiang, Donghao Luo, Wenqing Chu, Xiaoming Huang, Ying Tai, Chengjie Wang, and Jie Yang. IFRNet: Intermediate feature refine network for efficient frame interpolation. In CVPR, 2022.
[22] Hyeongmin Lee, Taeoh Kim, Tae-Young Chung, Daehyun Pak, Yuseok Ban, and Sangyoun Lee. AdaCoF: Adaptive collaboration of flows for video frame interpolation. In CVPR, 2020.
[23] Zhen Li, Zuo-Liang Zhu, Ling- Ling-Hao Han, Qibin Hou, Chun-Le Guo, and Ming-Ming Cheng. AMT: All-pairs multi-field transforms for efficient frame interpolation. In CVPR, 2023.
[24] Yaron Lipman, Ricky TQ Chen, Heli Ben-Hamu, Maximilian Nickel, and Matthew Le. Flow matching for generative modeling. In ICLR, 2022.
[25] Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang. Qing Jiang, Chunyuan Li, Jianwei Yang, Hang Su, et al. Grounding DINO: Marrying dino with grounded pre-training for open-set object detection. ECCV, 2024.
[26] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In ICCV, 2021.
[27] Matthew Loper, Naureen Mahmood, Javier Romero, Gerard Pons-Moll, and Michael J Black. SMPL: skinned multi-person linear model. ACM TOG, 2015.
[28]
Simon Niklaus, Long Mai, and Feng Liu. Video frame interpolation via adaptive convolution. In CVPR, 2017.
[29] Simon Niklaus, Long Mai, and Feng Liu. Video frame interpolation via adaptive separable convolution. In ICCV, 2017.
[30] Georgios Pavlakos, Vasileios Choutas, Nima Ghorbani, Timo Bolkart, Ahmed AA Osman, Dimitrios Tzionas, and Michael Black. Expressive body capture: 3d hands, face, and body from a single image. In CVPR, 2019.
[31] William Peebles and Saining Xie. Scalable diffusion models with transformers. In ICCV, 2023.
[32] Tomer Peleg, Pablo Szekely, Doron Sabo, and Omry Sendik. Im-net for high resolution video frame interpolation. In CVPR, 2019.
[33] Bohao Peng, Jian Wang, Yuechen Zhang, Wenbo Li, Ming-Chanp Yang, and Jiaya Jia. Con- trolNeXt: Powerful and efficient control for image and video generation. arXiv preprint arXiv:2408.0m 06070, 2024.
[34] Nikhila Ravi, Valentin Gabeur, Yuan-Ting Hu, Ronghang Hu, Chaitanya Ryali, Tengyu Ma, Haitham Khedr, Roman Rädle, Chloe Rolland, Laura Gustafson, et al. SAM 2: Segment anything in images and videos. In ICLR, 2025.
[35] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High-resolution image synthesis with latent diffusion models. In CVPR, 2022.
[36] Hyeonjun Sim, Jihyong Oh, and Munchurl Kim. XVFI: extreme video frame interpolation. In ICCV, 2021.
[37] Li Siyao, Shiyu Zhao, Weijiang Yu, Wenxiu Sun, Dimitris Metaxas, Chen Change Loy, and Ziwei Liu. Deep animation video interpolation in the wild. In CVPR, 2021.
[38] Alexandros Stergiou. Lavib: A large-scale video interpolation benchmark. In NeurIPS, 2024.
12

---

[39] Ang Wang, Baole Ai, Bin Wen, Chaojie Mao, Chen- Wei Xie, Di Chen, Feiwu Yu, Haiming Zhao, Jianxiao Yang. Jianyuan Zeng. et al. Wan: Open and advanced large-scale video generative models. arXiv preprini 2rXiv22503.03314 2025.
[40] Tan Wang. Linjie Li, Kevin Lin, Yuanhao Zhai, Chung-Ching Lin, Zhengyuan Yang, Hanwang Zhang, Zicheng Liu, and Lijuan Wang. DisCo: Disentangled control for realistic human dance generation. In CVPR, 2024.
[41] Xiaojuan Wang, Boyang Zhou, Brian Curless, Ira Kemelmacher-Shlizeemie Aleksander Holyn- ski, and Steven M Seitz. Generative Inbetweening: Adapting image-to-video models for keyframe interpolation. In ICLR, 2025.
[42] Zhenzhi Wang, Yixuan Li, Yanhong Zeng, Youqing Fang, Yuwei Guo, Wenran Liu, Jing Tan, Kai Chen, Tianfan Xue, Bo Dai, et al. Human Vid: Demystifying training data for camera- controllable human image animation. In NeurlPS, 2024.
[43] Haoning Wu, Erli Zhang, Liang Liao, Chaofeng Chen, Jingwen Hou, Annan Wang, Wenxiu Sun, Qiong Yan, and Weisi Lin. Exploring video quality assessment on user generated contents from aesthetic and technical perspectives. ICCV, 2023.
[44] Xiangyu Xu, Li Siyao, Wenxiu Sun, Qian Yin, and Ming-Hsuan Yang. Quadratic video interpolation. In NeurlPS, 2019.
[45] Zhongcong Xu, Jianfeng Zhang, Jun Hao Liew, Hanshu Yan, Wei Liu, Chenxu Zhang, Jiashi Feng, and Mike Zheng Shou. Magicanimate: Temporally consistent human image animation using diffusion model. In CVPR, 2024.
[46] Tianfan Xue, Baian Chen, Jiajun Wu, Donglai Wei, and William T Freeman. Video enhancement with task-oriented flow. IJC,, 2019.
[47] Serin Yang, Taesung Kwon, and Jong Chul Ye. ViBiDSampler: Enhancing video interpolation using bidirectional diffusion sampler. In ICLR, 2025.
[48] Zhendong Yang, Ailing Zeng, Chun Yuan, and Yu Li. Effective whole-body pose estimation with two-stages distillation. In ICCV, 2023.
[49] Mengying Yu and Aixin Sun. Dataset versus reality: Understanding model performance from the perspective of information need. JASIST, 2023.
[50] Guozhen Zhang, Yuhan Zhu, Haonan Wang. Youxin Chen, Gangshan Wu, and Limin Wang. Ex- tracting motion and appearance via inter-frame attention for efficient video frame interpolation. In CVPR, 2023.
[51] Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. In ICCV, 2023.
[52] Kaifeng Zhao, Gen Li, and Siyu Tang. DartControl: A diffusion- based autoregressive motion model for real-time text-driven motion control. In ICLR, 2024.
[53] Dian Zheng, Ziqi Huang, Hongbo Liu, Kai Zou, Yinan He, Fan Zhang, Yuanhan Zhang, Jingwen He, Wei-Shi Zheng, Yu Qiao, et al. VBench-2.0; Advancing video generation benchmark suite for intrinsic faithfulness. arXiv preprint arXiv:2503.2175 2025.
[54] Jingkai Zhou, Benzhi Wang, Weihua Chen, Jingqi Bai, Dongyang Li, Aixi Zhang, Hao Xu, Mingyang Yang, and Fan Wang. RealisDance: Equip controllable character animation with realistic hands. arXiv preprint wwXii:2409.06200 2024.
[55] Shenhao Zhu, Junming Leo Chen, Zuozhuo Dai, Zilong Dong, Yinghui Xu, Xun Cao, Yao Yao, Hao Zhu, and Siyu Zhu. Champ: Controllable and consistent human image animation with 3d parametric guidance. In ECCV, 2024.
[56] Tianyi Zhu, Dongwei Ren, Qilong Wang, Xiaohe Wu, and Wangmeng Zuo. Generative inbetweening through frame-wise conditions-driven video generation. In CVPR, 2025.
13

---

NeurIPS Paper Checklist
The checklist is designed encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No] or [NA]. .
[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justificatin right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.
The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation.
While "[Yes] " is generally preferable to "[No]" ", is perfectly acceptable to answer "[No] "provided proper justification is given (c.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). "[No] or "[NA] is not grounds for rejection. While the questions are phrased in binary way, we acknowledge that the true answer often more nuanced, so please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
general, answering
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS Paper Checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only the provided macros for your answers.
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We have made clear claims in the abstract and introduction that accurately reflect paper's 's contributions the studied problem. Please refer to the abstract and introduction for more detailed information.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes]
14

---

Justification:
We have discussions on the limitations in the supplementary materials.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approxinations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proof'
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer:
[NA]
Justification: Our work does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: We provide the implementation details in the main text and training hyperpa- rameters in the supplementary materials.
Guidelines:
15

---

The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Justification: We are unable to provide our code upon submission, but releasing the code to the public in the future is our plan.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ //nips.ccppulic//guue http:///cc/giiee///deuuubbiiiiii
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
16

---

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We provide the general training information and evaluation settings in the main text. The full implementation details and training hyperparameters presented in the supplementary materials.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification:
This paper does not report error bars.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We provide the type and the number of GPUs used in experiments.
Guidelines:
17

---

The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer:
[Yes]
Justification:
We carefully follow the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- cration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer:
[Yes]
Justification:
We discuss the potential impacts the supplementary materials.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (c.g., disinformation, generating fake profiles, surveillance), fairness considerations deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Question:
Answer: [NA]
18

---

Justification:
The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging. and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: our paper, we have cited the related papers and dataset sources.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, if possible, include URL..
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g.. website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of in the package should be provided. For popular datasets, thcors thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and the documentation provided alongside the assets?
Answer: [No]
Justification: Upon submission, do not provide the source code and the dataset. But will make them public in the future after acceptance.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/mooel as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
19

---

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: Our paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification: Our paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification: The core method development in this research does not involve LLMs as any important, original, or non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs any important, original, non-standard components.
Please refer to our L.LM policy hatoge//nearris.cc/oooffeeeeeeee for what should or should not be described.
20

---

7 Appendix
In this section, as referenced in the main text, we first provide detailed descriptions of our method and
dataset Sec. 7.1 Sec. 7.2 and Sec. 7.3 We then present additional experiments on our approach, including in- in-the-wir will interpolation results(Sec. 7.4.. extended benchmarking on the FCVG [56] test set (Sec. 7.5p. user study 77.6 and ablation studies on the control signals (Sec. (7.7). as well as the SMPL encoder (Sec. 778). We also provide suppiementary visualizations in Sec. 7.9 and analyze
model runtime in Sec. 7.10 Finally, we discuss the limitations and broader impact of our method in
Sec. 7.11 and Sec. 7.12
7.1 Wan2.1 for Keyframe Interpolation
We adapt the Wan2.1 39 Image-to-Video model (14B, 480P) as the keyframe interpolator Wan2. along single temporal forward diffusion path. Specifically, we insert zero- zero-padeel arro-padding between the input keyframes to construct a full-length video sequence. This sequence is then encoded into latents using the VAE encoder of 1an2.1. The resulting latent representation is concatenated with a noisy latent and a latent mask, and passed to the denoising network for prediction. In parallel, the input keyframes are encoded using the image CLIP encoder to produce condition tokens, which guide the denoising process through cross-attention mechanisms. To accommodate changes both the latent inputs and attention layer inputs, we perform parameter-effiient LoRA fine-tuning on the input embedding layer and on the value and output projection matrices of the attention layers.
7.2 CHKI-Video: Detailed Construction Stages
Our dataset is specifically designed for the Controllable Human-centrr Keyframe Interpolation (CHKI) task, emphasizing complex human motions, human-centre annotations, and distant keyframe inputs. To this end, we carefully control the data collection and annotation process, sourcing videos from task-relevant datasets (6] or the internet using specifically curated keywords, rather than relying on large-scale collections
[18 42
that may include irrelevant content.
Stage 1: Dataset Collection. We begin by collecting video clips from SportSlomo (6) which are temporally downsampled to 60 fps due to the large motions typically present in sports scenarios, making them more challenging for keyframe interpolation. To enhance dataset diversity versus reality [49], we additionally crawl high-quality stock videos from the Pexels website. We compile a list of keywords representing fundamental human movements such as 'Walking', 'Kicking,, 'Throwing', "Catching', and 'Climbing', to cover a broad range of human activities. For each keyword, we collect 100 unique videos with resolutions above 720p and durations under 30 seconds. These keywords are grouped into three motion categories: arm motion, leg motion, and general motion, ensuring balanced labels for subsequent train-test splitting. To match the motion characteristies of the SportSlomo videos, downsample the collected stock videos based on their optical flow scores, ensuring the flow score distributions are aligned.
Stage 2: Pre-annotation Processing. To ensure the quality of the collected videos, we use to obtain both technical and overall quality scores, and compute brightness change scores between adjacent frames. Videos falling below the bottom 5th percentile in any of these metries are filtered out. Given the importance of accurate human detection for downstream keypoint and SMPL-X annotation, we design a robust detection pipeline. We combine Grounding-DINO 25 with SAM2 44 to achieve reliable human detection. For challenging sports scenes, we prioritize videos with prominent foreground humans and relatively static or blurred backgrounds, striking balance between annotation complexity and scenario diversity. Additionally, we exclude videos containing more than three people or fewer than 20 frames to maintain clean motion patterns and ensure sufficient temporal coverage. All sports videos are manually reviewed to verify compliance with these criteria and to confirm the accuracy of the human detections.
DOVER 4
Stage 3: Human-centric Annotation. We perform frame- wise human-centro annotations for all video clips based on the detections in the previous stage. First, we use Sapiens [20] to estimate whole-body keypoints. To ensure the dataset remains strictly human-centric, we perform bodle-body detection based on these keypoints. Specifically, we extract the keypoints into DWPose to better define the human figure. We merge all head keypoints into a single point, as significant motion rarely occurs in that region. A whole-body detection is considered valid if it contains fewer than three
21

---

Start Ceytro//Kyffamee
Interpolated Saltrols/Kyframeess
End Control/Keyframe
Figure 5: Qualitative Results of In-the-wild Control and Keyframe Interpolation.
invalid keypoints, using keypoint score threshold of 0.3. We further filter video clips to retain only those with more than 20 consecutive valid frames. Finally, we apply SMPLer-X [4), which provides high re-projection accuracy, to fit detailed SMPL-X models to each frame and generate reliable 3D body parameters.
7.3 Implementation Details
We fine-tune the entire PoseFuse D--KI framework in an end-to-end manner using the AdamW optimizer with learning rate of 8 x 10 5, The fine-tuning is applied to our 3D-informed control model, PoseFuse3D, with additional LoRA adaptation on the input patch embeddings, as well as the value and output projections of the VDM's attention modules. Both the LoRA rank and LoRA alpha are set to 32. For implementation, we leverage Fully Sharded Data Parallel (FSDP) across GPUs.
7.4 In-the-wild Interpolation
Our PoseFuse3D-KK framework can be readily applied to interpolate in-the-wild human- keyframes. In this subsection, we present simple pipeline that uses linear interpolation for human body joints. Given a human-centric keyframe pair To. IN, we first employ a 3D human model estimator, such SMPLer-X 4. fit SMPL-X models (30 for each keyframe input. Leveraging the strong human body priors from SMPL.-X we linearly interpolate the SMPL-XX parameters to generate intermediate 3D human models, which serve control signals for interpolation. We then extract 2D DWPose keypoints from the 2D projections of the interpolated SMPL-X models. With these steps, all necessary guidance inputs for PoscFuse3D-KI are prepared and can be directly used for keyframe interpolation. F shows the results of the interpolated SMPL-X models and the corresponding video frames. While this pipeline offers a straightforward and efficient approach for generating intermediate poses, it is limited in its ability to handle complex human motion. In particular, it struggles to capture realistic dynamics such as acceleration or multi-step actions. To address these limitations, we could explore text-to-motion models [22] part of future work, which generate intermediate SMPL--X models from textual descriptions, offering a more flexible and semantically rich alternative to lincar interpolation.
22

---

Table 6: Benchmark Results on FCVG-Test-HC.
Metrics
Methods
PSNR1
PSNRma P
LPIPS]
L
HA1
GIMM-VH 12
23.61
21.25
20.28
0.1324
0.0759
0.0587
0.9459
GI 4
17.21
15.20
14.36
0.2701
0.1422
0.1045
0.9438
Wan2,1 KI (Ours)
21.50
19.03
18.17
0.1553
0.0915
0.0704
0.9312
FCVG 56
22.49
21.03
20.69
0.1738
0.0734
0.0493
0.9241
PoseFuse3D (Ours)
24.84
22.97
22.26
0.0460
0.0340
0.9245
COMM-VTI
Sort Keyfomme
End Keyfhane
PoseFusDD&K
Ground Truhh
CUMM-VHI
Keyfnime
Won2.1
PCVG
End Trane
PosePuRe3D XI
Ground Trust
GIML-VFI
Stort Keytrome
Want 1-K1
FCVG
Keyframe
Figure 6: Qualitative Comparisons FCVG-Test-HC..
7.5
Evaluation on Additional Benchmark
Setup. We conduct an additional evaluation on the from FCVG [56]. We first perform human detection and extract human-centric video clips from the test set. The extracted videos are then annotated using the same processing pipeline detailed in Sec. 772 This results in FCVG-Test-HIC, curated human-centric subset of 54 clips suitable for CHKT benchmarking. The FCVG-Test- HC benchmark relatively casier than CHKI-Video. primarily consisting of human-centric clips with limited motion rather than more challenging scenarios such sports and dancing. Other benchmarking settings follow those described the main paper.
Results. We present quantitative comparisons the FCVG-Test- HC benchmark in Table Our PoseFuse3D-Ki outperforms other methods for human-centir keyframe interpolation. Compared with the previous tfate-of-the--- method FCVG [56]. our method achieves 7.6% improvement in PSNRmuk and a 31% % reduction in LPIPS We observe that methods achieve higher PSNR and lower LPIPS scores the FCVG-Test-HC benchmark compared to the CHKI- dataset, indicating that FCVG-Test-HC is an easier benchmark for interpolation. This aligns with our earlier observation during dataset construction, where CCVG-Test-HHl primarily consists of human-centric
23

---

Table 7: User Study.
Methods
GIMM-VFI 12
GI
[41
FCVG [56)
PoseFuse3D-Ko (Ours)
User Preference (%)
1.75
5.13
6.63
86.50
Table 8: Ablation on the Visual Encoding.
Evaluation Metrics
Model Variant
PSNRT
PoN 1
pmN
LPIPS]
LPIPS
LPIPS
Non-Viz Vis
19.63
16.02
14.69
0.2097
0.1232
0,0889
Non-2D
21.71
18.65
17.28
0,1438
0.0738
0.0531
Full
22.14
19.30
18.01
0.1330
0.0653
0.0464
clips with limited motion. Interestingly, methods with fewer learned priors tend to achieve higher HA scores in this setting. For instance, the traditional interpolation method GIMM-VFI [12) records the highest Human Anatomy (HA) score. This is likely because such methods rely more heavily on the input keyframes. While this reliance leads to motion artifacts under large movement, it better preserves human textures from inputs when the motion between keyframes small.
Visualizations. We qualitatively compare PoseFuse3D-KK with other advanced methods on the FCVG-Test-HC benchmark, shown in Figure Consistent with our findings in the Benchmark Results section of the main paper, our method achieves robust human-centric interpolation, accurately follows real-world dynamics, and effectively preserves human body shape. For instance, method generates plausible interpolations of complex body movements while maintaining the correct leg structure and posture in the last 'Breaking Dance' case.
7.6 User Study
Setup. To further assess the perceptual quality of our controllable human-centris interpolation, we conducted user study involving 20 interpolation scenarios sampled from both the CHKI-Video test set and FCVG-Test-HC. We compared PoseFuse-K KI with three representative state-of-the-ar methods: GIMM-VFI [2], GI [1], and FCVG [56]. A total of 40 participants were invited to take part in the study, where each participant was asked to select their preferred interpolation result.
Our method received strong majority user preferences (86.5%), consistently outperforming all baselines. These results further demonstrate the perceptual effectiveness of PoseFuse3ID-KI.
Results. We report the user preference of the study in Table 7
7.7 Ablation Study on Visual Encoding
The core of our framework is the control module, PoseFuse3D. As detailed in the Method section of the main paper, it includes encoding visualizations from both SMPL-X and DWPose [48]. To evaluate the importance of encoding these visualizations and explore whether 2D visual cues can be omitted, we conduct an ablation study on the visual encoding component of PoseFuse3D.
Necessity of Encoding Visualizations. In PoseFuse3D, we encode visualizations of control signals. Since they preserve natural pixel-level alignment with the video latent, thereby providing direct control signals on the pixel plane. To assess its necessity, we ablate all visual encoding components in PoseFuse3D and rely solely on the SMPL-X encoded information as the control representation. We refer to this variant as 'Non- Vis'. In Table g this modification results in a significant performance degradation, with a 4.32 dB drop in PSNRmack and 0.0425 increase in LPIPS masks underscoring the critical role of encoding visualizations in achieving high-fidelity results.
Importance of Encoding 2D Visualization. The visual encoding module of PoseFuse3D integrates 2D DWPose visualizations with rendered SMPL-X images. The 2D DWPose visualizations emphasize skeletal keypoint layouts, contributing to robust pose understanding. To assess its importance, we exclude the encoding of 2D visualizations, denoting this variant as 'Non-2D', This leads to drop of 0.65 dB in PSN and 13% increase LPIPSatox: demonstrating the significance of encoding 2D visualizations in the visual encoding module.
24

---

Table 9: Ablation on SMPL.-X Encoder.
Evaluation Metrics
Model Variant
PSNRT
PSNR
maN
LPIPS./
pho
LPIPS musk
Non-JA
22.07
19.24
17.95
0.1348
0.0659
0.0466
Non- VA
22.15
19.27
17.99
0.1374
0.0667
0.0470
Full
22.14
19.30
18.01
0.1330
0.0653
0.0464
Start Keyframe
Joctumn bontagor
Wank 1-K1
FCNG
End Keyframe
PoseFrseeDD
Ground Trudi
Keyfromc
PCVG
End Keylhame
PosdFuxcDD&d
Ground Truls
Figure 7: Additional Qualitative Results CHKI-Video.
7.8 Ablation on SMPL-X Encoder
We conduct an additional ablation study on the SMPL-X encoder justify our design.
Joint Aggregation. The SMPL-X encoder extracts joint motion and position features from 3D space and projects them onto the 2D image plane via an attention mechanism, providing spatial human body motion cues. To assess the impact of this design, we remove the joint aggregation module, denoted 'Non-JA'. As shown in Table 99] this leads to 0.06 dB drop in both PSNR and P emphasizing the importance of 3D joint aggregation for accurate body control representation.
Vertex Aggregation. We also apply separate attention mechanism aggregate vertex information into the 2D image plane, To examine its necessity, we remove the vertex attention module. We denote this variant 'Non-VA'. As reported in Table this leads to noticeable degradation in performance across all LPIPS scores, including 0.0040 rise in LPIPS and 0.0014 increase in LPIPS These results demonstrate the significance of incorporating 3D vertex information for effective control representation from SMPL-X.
7.9 Supplementary Visualizations
Additional Qualitative Comparisons. We present additional qualitative comparisons with other interpolation methods in Figure 7 Our PoseFuse3D-KK framework consistently produces more
plausible interpolations, closely capturing real- world dynamics observed in the ground truth.
Robustness across Corner Cases. PoseFuse3D-KI exhibits robust interpolation performance across diverse corner cases. As shown in Figure g we evaluate scenarios such as off-center subjects, camera motion, multiple persons, virtual animation, and roboties videos. Our method consistently produces controllable interpolations with plausible and temporally coherent motion.
25

---

Table 10: Runtime Comparison.
Methods
G1 41
FCVG [56
PoseFuse3D-K KI (Ours)
Runtime (8)
523
(5)
(c)
(d)
(c)
Start Keytrame
Interpolated Keyframes
End Keyframe
Figure 8: Qualitative Results Corner Cases. (a) Person off-center. (b) Camera motion. (c)
Multiple persons. (d) Virtual animation. (e) Robotics video.
7.10 Runtime Comparison
To assess computational efficiency, we compare the runtime of PoseFuse3D-KG with existing diffusion- based baselines. The comparison measures the time required interpolate 25 frames a resolution of 1024 x 576, using the same GPU equipped with 80 GB memory. As summarized in Table I0 PoseFuse3D-KI exhibits the shortest runtime, confirming its efficiency advantage over competing methods.
7.11 Limitations
There several known limitations method. First, PoseFuse3D-K KI relies accurate SMPL-X estimations to generate reliable 3D control signals. Therefore, it inherits the limitations of the 3D human model estimators, where inaccurate predictions Additionally, our method, while offering strong control via 3D and 2D fusion, still depends on the base diffusion model's generative priors. As result, output quality is influenced by the model's learned behavior and inherits its high GPU memory demands. Finally, our method does not explicitly model human-object interactions, which may lead to artifacts or misaligned object motion in scenarios involving close interaction with external objects.
degrade the quality of interpolated results.
7.12 Broader Impacts
Our proposed method, PoseFuse3D-KL. enables accurate and controllable human-centric keyframe interpolation, with applications in areas such as human animation and video generation. By integrating explicit 3D information from human models and 2D pose cues, our framework supports 3D-informed and semantically meaningful guidance for interpolating realistic human motion across frames. This technique not only enriches creative workflows but also opens new opportunities for research in human motion understanding and video synthesis. While powerful, our method shares common limitations of generative models and may pose risks if misused to produce manipulated or deceptive human videos, highlighting the importance of responsible use and ethical safeguards.
26


Reviews:
[{"review_id": "5iqvVynSHh", "reviewer": "Reviewer_9e7q", "summary": "This paper aims to improve the performance of Human-centric Keyframe Interpolation (HKI) by proposing PoseFuse3D, which introduces 3D-informed control signals SMPL (rendering and 3D information) to guide HKI.", "questions": "see weaknesses.", "limitations": "yes.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths：\n1. Existing keyframe interpolation methods cannot effectively address human scenarios, making the scenario addressed in this paper meaningful.\n2. The introduction of SMPL-X compensates for the insufficient 3D information in DWPose.\n3. It is innovative to utilize both the rendering information of SMPL-X and propose an SMPL-X Encoder to directly encode its 3D information.\n4. The results of this paper are sota.\n\nWeaknesses:\n\n1. Introducing human studies would better evaluate the performance of generative interpolation models.\n2. I found that most cases provided by the authors target a single person, but when extending to many humans, such as a group of people (more than 5), can PoseFuse3D-KI still work? Can SMPL still estimate accurately?\n3. How does the performance fare when humans are not positioned at the center of the frame?\n4. Is there any insight into the design of the SMPL-X Encoder?\n5. Do you need to consider copyright when using Pixels data?", "was_revised": true, "final_justification": "In the rebuttal, the authors have: 1) addressed my concerns regarding data openness and licensing, 2) provided further explanation of the design insights behind the SMPL-X Encoder, and 3) offered clarifications regarding multi-person scenarios, scenarios where people are not at the center, and scenarios with camera movement. I believe my concerns have been satisfactorily resolved. I consider this paper to provide valuable insights for the field of human-centric generative video interpolation, and therefore I maintain my score as accept.", "rebuttal": "> **Q1**: **[Weakness 1]** Introducing human studies would better evaluate the performance of generative interpolation models.\n> \n\n**A1**: Thank you for the valuable suggestion to include a human user study. To further evaluate the perceptual quality of our method, we conducted a user study involving 40 participants across 20 interpolation scenarios. Participants were asked to choose their preferred output among the results generated by the baseline methods and our proposed PoseFuse3D-KI. The summarized results are as follows:\n\n| Methods | GIMM-VFI | GI | FCVG | PoseFuse3D-KI (Ours) |\n| --- | --- | --- | --- | --- |\n| User Preference (%) | 1.75 | 5.13 | 6.63 | **86.50** |\n\nOur method received a strong majority of user preferences (86.5%), consistently outperforming all baselines. These results further demonstrate the perceptual effectiveness of PoseFuse3D-KI.\n\n> **Q2**: **[Weakness 2]** I found that most cases provided by the authors target a single person, but when extending to many humans, such as a group of people (more than 5), can PoseFuse3D-KI still work? Can SMPL still estimate accurately?\n> \n\n**A2**: Thank you for the valuable question. As our method focuses on the plausible interpolation of human images with complex motion, we limit the number of humans in our training dataset. However, we have qualitatively validated the effectiveness of PoseFuse3D-KI in test cases involving more than five people. Unfortunately, due to limitations of the discussion format, we are currently unable to include the qualitative results here. These examples will be included in the supplementary materials of the revised version of the paper.\n\n> **Q3**: **[Weakness 3]** How does the performance fare when humans are not positioned at the center of the frame?\n> \n\n**A3**: We appreciate the reviewer for raising this question. Our method generalizes well to scenarios where humans are not centered in the frame, maintaining consistent and plausible interpolation performance. Such cases are included in the test set of our proposed dataset and are part of the quantitative evaluations presented in the paper. We will include visualizations of these examples in the revised version of the paper.\n\n> **Q4**: **[Weakness 4]** Is there any insight into the design of the SMPL-X Encoder?\n> \n\n**A4**: The SMPL-X encoder is introduced to supply additional spatial information, particularly in occluded regions where rendering operations result in information loss. While the rendering process transforms the 3D model into 2D images, it only preserves visible (non-occluded) information during projection. However, occluded information is crucial for modeling motion and generating plausible interpolations. Discarding it often leads to implausible results, as illustrated in Figure 3 of the main paper. To address this, the SMPL-X encoder directly processes 3D information and uses attention mechanisms to aggregate it into the image space, effectively preserving important cues that are lost during rendering.\n\n> **Q5**: **[Weakness 5]** Do you need to consider copyright when using Pexels data?\n> \n\n**A5**: Thank you for raising this important question. All Pexels content is provided under the Pexels license, which allows free use. The videos we collected are modified and used purely for research purposes, not distributed on other platforms or sold. Therefore, our usage complies fully with Pexels’ licensing terms."}, {"review_id": "JKSEVHEJWO", "reviewer": "Reviewer_Ayns", "summary": "The paper introduces PoseFuse3D-KI, a controllable human-centric keyframe interpolation framework that integrates 3D human guidance signals into the diffusion process for human-centric keyframe interpolation (CHKI). This novel framework aims to improve interpolation fidelity, especially for articulated human motion, by leveraging 3D SMPL-X models alongside 2D poses for precise spatial and structural control. The paper proposes a new dataset, CHKI-Video, annotated with both 2D poses and 3D SMPL-X parameters, and demonstrates the effectiveness of the model through extensive experiments, achieving state-of-the-art performance on keyframe interpolation tasks.", "questions": "1.\tHow does the framework perform on a wider range of human actions beyond sports and everyday activities covered in CHKI-Video? Could the method be generalized to other domains, such as robotics or virtual character animation?\n2.\tWhat is the maximum temporal gap (in terms of frames or seconds) that the proposed method can effectively interpolate?\n3.\tGiven that the model combines 3D and 2D signals, how does the performance degrade in scenarios with extreme occlusion or very fast motion? In other words, how does the model address the change in background after a large change in character position?\n4.\tRegarding the 3D auxiliary information aspect, does the proposed method require the SMPLX parameters for all frames, or only for the first and last frames? If the former is the case, can it be argued that the method actually draws on some information from ground truth rather than being truly effective? \n5.\tIn some cases where human SMPLX parameters are not available, does the method proposed in this paper still maintain a high level? In other words, did the authors consider scenarios where 3D information is not available?\n6.\tHas the performance been evaluated using SSIM or other structural similarity metrics? Including such results could offer a more complete view of visual quality.\n7.\tHow much GPU resources are required in the training phase of the proposed method? What about the inference time?", "limitations": "1.\tDependency on Specific Datasets: The results presented are based on the CHKI-Video dataset, which may limit the generalizability of the findings. The method's performance on other datasets or real-world video sources remains untested.\n2.\tDependency on SMPLX Parameters: The method proposed by the authors, although capable of generating higher quality videos, is overly reliant on 3D information and lacks extensibility.\n3.\tFuture Work on Data and Code Availability: Although the authors plan to release the code and data after acceptance, the current lack of access limits the paper's ability to be fully reproducible and verifiable by the research community.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. Novel Contribution: The integration of 3D human guidance with 2D poses for keyframe interpolation is a significant improvement over existing methods, addressing the challenges of complex human motion and occlusions. In addition, a new dataset is presented in this paper.\n2. Comprehensive Evaluation: The introduction of a new dataset, CHKI-Video, specifically designed for evaluating controllable human-centric keyframe interpolation, strengthens the paper's contributions and allows for fair comparison with existing methods.\n3. State-of-the-art Performance: The proposed PoseFuse3D-KI framework outperforms existing techniques, showing improvements in PSNR and reductions in LPIPS, indicating its effectiveness in generating high-quality human-centric interpolations.\n\nWeaknesses:\n\n1. Limited Discussion on Scalability: While the paper presents impressive results, there is limited discussion about how well the method scales with different video lengths or more complex human actions, particularly in large-scale applications.\n2. Reproducibility Concerns: The code and data are not currently available, and while the authors plan to release them, this limits the reproducibility of the results.\n3. Computational Efficiency: Although the paper provides implementation details, there is no clear discussion about the computational efficiency of the framework or how it scales with increasing complexity or dataset size. \n4. Potential Overfitting: While the method shows strong performance on the CHKI-Video dataset, it would be useful to see how the model generalizes to other datasets or human motion tasks to avoid potential overfitting.\n5. Limited Practical Expandability: The reliance on 3D motion information (e.g., SMPL-X parameters) may hinder the method’s applicability in real-world scenarios where such annotations are difficult or expensive to obtain. The lack of fallback strategies for 2D-only input scenarios limits the applicability of this framework.", "was_revised": true, "final_justification": "The author provides an answer to the question of how to obtain SMPL-X parameter information for intermediate frames, resolving my previous main concerns. The interpolation algorithm is a viable solution. Considering the content of the entire paper, I believe that this article has no major shortcomings overall, its contributions are clear, and it is worthy of consideration for acceptance.", "rebuttal": "Thank you for the constructive comments. Please find the following for our response.\n\n> **Q1**: **[Weakness 1]** Limited Discussion on Scalability: … how well the method scales with different video lengths or more complex human actions, particularly in large-scale applications. **[Question 2]** What is the maximum temporal gap (in terms of frames or seconds) that the proposed method can effectively interpolate?\n> \n\n**A1**: Thank you for your suggestion to evaluate the scalability of our method across different video lengths and temporal gaps. We have conducted additional evaluations of PoseFuse3D-KI on varying temporal gaps, and the results are summarized in the table below:\n\n|  | 24 frames | 48 frames | 72 frames | 96 frames |\n| --- | --- | --- | --- | --- |\n| PSNR$\\uparrow$ | 23.86 | 21.46 | 20.37 | 19.44 |\n| PSNR$_{mask}$$\\uparrow$ | 15.93 | 13.80 | 12.68 | 11.89 |\n| LPIPS$\\downarrow$ | 0.1144 | 0.1566 | 0.1882 | 0.2208 |\n| LPIPS$_{mask}$$\\downarrow$ | 0.0332 | 0.0432 | 0.0514 | 0.0577 |\n\nAs with most interpolation methods, performance decreases as the temporal gap increases. Nevertheless, PoseFuse3D-KI maintains **competitive performance across a wide range of gaps**, especially for 24 and 48 frames. At 96 frames, while performance degradation is observed (e.g., PSNR drops below 20 dB and LPIPS exceeds 0.2000), the method still provides reasonable results. These results demonstrate the robustness of PoseFuse3D-KI in long-range controllable interpolation settings.\n\n> **Q2**: **[Weakness 2]** Reproducibility Concerns: The code and data are not currently available, … limits the reproducibility of the results. **[Limitation 3]** Future Work on Data and Code Availability: …, the current lack of access limits the paper's ability to be fully reproducible and verifiable by the research community.\n> \n\n**A2**: We fully agree that open-sourcing code and data is important for the research community. We will try to make our database and codebase publicly available as soon as possible.\n\n> **Q3**: **[Weakness 3]** Computational Efficiency: … discussion about the computational efficiency of the framework or how it scales with increasing complexity or dataset size.\n> \n\n**A3**: Thank you for your valuable feedback regarding computational efficiency. We have conducted a runtime comparison between our method and existing baselines. Specifically, we measured the time required to interpolate 25 frames at a resolution of $1024 \\times 576$ on an NVIDIA A800 GPU. The results are shown below:\n\n| Methods | Runtime (s) |\n| --- | --- |\n| GI | 975 |\n| FCVG | 523 |\n| PoseFuse3D-KI (Ours) | **212** |\n\nOur method achieves the fastest runtime among the evaluated approaches, indicating comparatively superior computational efficiency.\n\n> **Q4**: **[Weakness 4]** Potential Overfitting: …, it would be useful to see how the model generalizes to other datasets or human motion tasks to avoid potential overfitting. **[Limitation 1]** Dependency on Specific Datasets: …The method's performance on other datasets or real-world video sources remains untested.\n> \n\n**A4**: We appreciate the reviewer for this valuable suggestion. To further evaluate generalizability, we conducted both qualitative and quantitative experiments on the FCVG test set, as described in Section 1.5 of the supplementary material. The results, shown in Table 1 and Figure 2, demonstrate that PoseFuse3D-KI consistently outperforms all baseline methods in terms of quantitative metrics and produces robust human-centric interpolation in challenging cases, such as the `Breaking Dance' case in Figure 2. These results support the generalization capability of our method beyond the CHKI-Video dataset.\n\n> **Q5**: **[Weakness 5]** Limited Practical Expandability: The reliance on 3D motion information (e.g., SMPL-X parameters) may hinder the method’s applicability in real-world scenarios … The lack of fallback strategies for 2D-only input scenarios limits the applicability of this framework. **[Question 5]** In some cases where human SMPLX parameters are not available … In other words, did the authors consider scenarios where 3D information is not available? **[Limitation 2]** Dependency on SMPLX Parameters: The method proposed by the authors, although capable of generating higher quality videos, is overly reliant on 3D information and lacks extensibility.\n> \n\n**A5**: Thank you for the insightful suggestion. We address scenarios where only start and end frames are available, without ground-truth 3D annotations, as **in-the-wild interpolation**, described in Section 1.4 of the supplementary material. In this setting, we estimate SMPL-X parameters from the input frames and apply linear joint interpolation to generate intermediate SMPL-X controls. Figure 1 in the supplementary material illustrates the qualitative results, showing the effectiveness of this strategy. Additionally, we conducted quantitative comparisons with FCVG under this setting. The results are shown below:\n\n| Methods | PSNR$\\uparrow$ | PSNR$_{bbox}\\uparrow$ | PSNR$_{mask}\\uparrow$ | LPIPS$\\downarrow$ | LPIPS$_{bbox}\\downarrow$ | LPIPS$_{mask}\\downarrow$ | HA$\\uparrow$ |\n| --- | --- | --- | --- | --- | --- | --- | --- |\n| FCVG | 18.47 | 10.54 | 9.31 | 0.2607 | 0.1321 | 0.0921 | 0.9284 |\n| PoseFuse3D-KI | **19.30** | **10.98** | **9.69** | **0.2031** | **0.1194** | **0.0859** | **0.9289** |\n\nOur method consistently outperforms FCVG across all metrics, demonstrating its **effectiveness for the in-the-wild interpolation** setting. Furthermore, our strategy has the potential to incorporate advanced techniques such as text-to-motion models for generating intermediate human model controls, enabling more flexible and effective interpolation for complex motion.\n\n> **Q6**: **[Question 4]** Regarding the 3D auxiliary information aspect, does the proposed method require the SMPLX parameters for all frames, or only for the first and last frames? If the former is the case, can it be argued that the method actually draws on some information from ground truth rather than being truly effective?\n> \n\n**A6**: Our PoseFuse3D-KI framework requires intermediate SMPL-X controls to guide the interpolation process. However, these controls do **not** need to be ground-truth. As described in Section 1.4 of the supplementary material and elaborated in **A5**, the intermediate SMPL-X parameters can be obtained via simple interpolation or possibly generated using text-to-motion models. We emphasize that the method **remains effective** even in the absence of ground-truth 3D annotations. This is further supported by the quantitative results we present in **A5** under the in-the-wild interpolation setting. This demonstrates the strong performance of our method even when only the first and last frames are available.\n\n> **Q7**: **[Question 1]** How does the framework perform on a wider range of human actions … Could the method be generalized to other domains, such as robotics or virtual character animation?\n> \n\n**A7**: While PoseFuse3D-KI is designed to perform plausible interpolation for real-world human motions with complex dynamics, it can also be applied to other video domains such as robotics and virtual character animation. In our tests, it has demonstrated promising results in these settings. However, due to limitations of the discussion format, we are currently unable to include the qualitative results here. These examples will be added to the supplementary materials in the revised version of the paper.\n\n> **Q8**: **[Question 3]** … how does the model address the change in background after a large change in character position?\n> \n\n**A8**: We would like to clarify that PoseFuse3D-KI is designed to address controllable **human-centric** interpolation, with a focus on plausibly interpolating foreground humans with control signals. For the background, our framework relies on the underlying adapted video diffusion model to synthesize the in-between frames, without explicit control signals. While this approach works well in many cases, we acknowledge that it may lead to artifacts under challenging conditions such as large background changes. Addressing this limitation is an interesting direction for future research and could extend the applicability of our method to more general video interpolation tasks. We will include this limitation in the revised version of our paper.\n\n> **Q9**: **[Question 6]** Has the performance been evaluated using SSIM or other structural similarity metrics? Including such results could offer a more complete view of visual quality.\n> \n\n**A9**: Thank you for the valuable suggestion. We have computed SSIM scores as additional evaluation metrics to complement Table 2 in the main paper. The results are presented below:\n\n| Methods | SSIM$\\uparrow$ | SSIM$_{bbox}$$\\uparrow$ | SSIM$_{mask}$$\\uparrow$ |\n| --- | --- | --- | --- |\n| GIMM-VFI | 0.7776 | 0.6070 | 0.4985 |\n| GI | 0.5875 | 0.4378 | 0.3483 |\n| Wan2.1-KI | 0.7460 | 0.5852 | 0.4853 |\n| FCVG | 0.7429 | 0.6735 | 0.6210 |\n| PoseFuse3D-KI | **0.8049** | **0.7308** | **0.6802** |\n\nWe observe that PoseFuse3D-KI achieves the highest SSIM scores across all image regions, further demonstrating its strong controllable interpolation performance and visual quality. We will include this table in the revised version of our paper for a more complete view of the interpolation quality.\n\n> **Q10**: **[Question 7]** How much GPU resources are required in the training phase of the proposed method? What about the inference time?\n> \n\n**A10**: PoseFuse3D-KI is trained using 4 NVIDIA A100 GPUs, each with 80 GB of memory. During inference, it takes 212 seconds to interpolate 25 frames at a resolution of $1024 \\times 576$ on an NVIDIA A800 GPU. A more detailed discussion on runtime performance is provided in **A3**."}, {"review_id": "x7tOqon9kH", "reviewer": "Reviewer_KLZa", "summary": "This paper introduces PoseFuse3D-KI for controllable human-centric keyframe interpolation. The core of this framework is the PoseFuse3D control model, which integrates 3D human guidance signals (from SMPL-X models) and 2D pose information into a video diffusion model to generate intermediate frames. The key innovation is an SMPL-X encoder that processes 3D model data directly, rather than relying solely on 2D renderings, to better handle complex motions and occlusions. To evaluate their method, the authors built a new dataset, CHKI-Video, containing over 180,000 frames from 2,614 video clips, annotated with 2D poses and 3D SMPL-X parameters. Experiments show that PoseFuse3D-KI outperforms existing methods on this new benchmark.", "questions": "Please see the weaknesses. My major concern about this paper is that the experiments fail to support its main contribution: \"effectively extracts control signals from 3D SMPL-X and fuses 2D signals, allowing precise and informative control\", considering that 2D signals are well-studied in Champ [50].", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "**Strengths**\n\n1. The proposed SMPL-X encoder, which directly processes 3D model information instead of just its 2D renderings, is a novel approach to preserving geometric details, especially for occluded parts.\n\n2. The paper contributes the CHKI-Video dataset, a valuable resource for the specific task of controllable human-centric interpolation, as it includes both 2D pose and 3D SMPL-X annotations which are missing in many existing interpolation datasets.\n\n**Weaknesses**\n\n1. My major concern about this paper is the novelty claimed in \"our encoder processes features directly in 3D and integrates projected features through feature aggregation\". Although the claimed novelty mainly focus on the effectiveness of directly utilizing 3D SMPL information, the experiments in Table 1 fails to support this claim: the performance gain of VE+SE over VE/VE+DN is marginal. The overall framework can be viewed as an combination of two established research lines: diffusion-based video interpolation and pose-guided video generation, e.g., Champ. Although the authors want to emphasize the contribution of direct learning from SMPL parameters instead of visualizations in Champ, the experiments does not support such claim. Besides, there is an important ablation study missing: only using SE. If only using SE is way worse than only using VE+DN, it will indicates that the proposed modules is not effective at all compared to Champ (which could be regarded as VE+DN).\n\n2. As we know 2D pose detector is generally more accurate than SMPL pose estimator. The paper relies on the SMPLer-X tool for automated 3D model fitting, supplemented by \"manual verification\". However, it lacks a detailed description of this verification process, such as the scope of manual checks or the final error rate. The quality and accuracy of the underlying SMPL-X annotations are critical for training a robust 3D-aware model, and any systematic errors from the automated tool could compromise the model's performance and the validity of the evaluation.\n\n3. The paper creates a new dataset with 2,614 videos by sourcing videos from Pexels, but it does not discuss or compare its contribution to other recent datasets like HumanVid [a] or MiraData [b], which also use Pexels videos for video-related tasks. Both mentioned datasets have a scale larger than this submission. A discussion of why a new dataset was necessary instead of augmenting existing ones would strengthen the justification for this contribution. For example, HumanVid [a] already curated human-centric 20K videos from Pexels, will this dataset shares a large portion of videos with this submission?\n\n\n[a] Wang, Zhenzhi, Yixuan Li, Yanhong Zeng, Youqing Fang, Yuwei Guo, Wenran Liu, Jing Tan et al. \"Humanvid: Demystifying training data for camera-controllable human image animation.\" In The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track. 2024.\n\n[b] Ju, Xuan, Yiming Gao, Zhaoyang Zhang, Ziyang Yuan, Xintao Wang, Ailing Zeng, Yu Xiong, Qiang Xu, and Ying Shan. \"Miradata: A large-scale video dataset with long durations and structured captions.\" Advances in Neural Information Processing Systems 37 (2024): 48955-48970.", "was_revised": true, "final_justification": "There are still some concerns about the small performance gain of VE+SE (the proposed method) over VE+DN (baseline method). The author also not fully justify the necessity of the proposed dataset compared to existing ones and the quality control of SMPL annotation process. Yet, this paper has provided extensive experiments and ablation studies to analyze the proposed method, which could be useful for the community to further explore in this direction. So I think it is acceptable to recommend the exposure of this paper to the community. I have increased the rating from borderline reject to be borderline accept. But I am not willing to defend acceptance for this paper against other reviewers, instead, I think the final recommendation of this paper could be achieved by aggregation the rating of all reviewers.", "rebuttal": "Thank you for the constructive comments. Please find the following for our response.\n\n> **Q1**: **[Weakness 1]** Although the claimed novelty mainly focus on the effectiveness of directly utilizing 3D SMPL information, the experiments in Table 1 fails to support this claim: the performance gain of VE+SE over VE/VE+DN is marginal. … Although the authors want to emphasize the contribution of direct learning from SMPL parameters instead of visualizations in Champ, the experiments does not support such claim.\n> \n\n**A1**: We respectfully disagree with the reviewer. We argue that our results support the effectiveness of our method. Specifically, in Table 1 of the main paper, our 'VE+SE' consistently outperforms the 'VE+DN' baseline across all quantitative metrics, with a notable **0.21 dB improvement** on PSNR$_{mask}$, which focuses on the human foreground. More importantly, our SMPL-X encoder is explicitly designed to model **occluded body parts,** information typically discarded in the rendering process used in visualization. This advantage is evident in the qualitative results. For example, as shown in Figure 3 of the main paper, only our strategy `VE+SE' correctly handles the challenging occlusions, such as the occluded arm interpolation in Skateboarding ($1^{st}$ row) and Jumping ($2^{nd}$ row). Therefore, with both quantitative and qualitative results, we claim that our results demonstrate the effectiveness of our proposed SMPL-X encoder.\n\n> **Q2**: **[Weakness 1]** Besides, there is an important ablation study missing: only using SE. If only using SE is way worse than only using VE+DN, it will indicates that the proposed modules is not effective at all compared to Champ (which could be regarded as VE+DN).\n> \n\n**A2**: Thank you for your suggestion to include this ablation study. We would like to clarify that we have already included an experiment using only SE in Section 1.6 of the supplementary material. \nThis variant, denoted as 'Non-Vis', is used to **ablate the importance of visual encoding** in PoseFuse3D-KI. The corresponding results are provided in Table 2 of the supplementary material.\n\nFurthermore, we would like to emphasize that it is inappropriate to directly compare 'Non-Vis' with 'VE+DN', as our proposed method is designed as an integrated framework. While the SMPL-X encoder is responsible for preserving **occluded information**, our visual encoder incorporates 3D human rendering to provide direct shape and appearance cues. Therefore, a more meaningful comparison is between our method 'VE+SE' and 'VE+DN', whose results are shown in Table 1 and Figure 3 of the main paper. As discussed in A1, these comparisons demonstrate the effectiveness of our proposed SMPL-X encoder in modeling occluded human motion.\n\n> **Q3**: **[Weakness 2]** As we know 2D pose detector is generally more accurate than SMPL pose estimator. The paper relies on the SMPLer-X tool for automated 3D model fitting, supplemented by \"manual verification\". However, it lacks a detailed description of this verification process, such as the scope of manual checks or the final error rate. The quality and accuracy of the underlying SMPL-X annotations are critical for training a robust 3D-aware model, and any systematic errors from the automated tool could compromise the model's performance and the validity of the evaluation.\n> \n\n**A3**: As described in Section 4 of the main paper and Section 1.4 of the supplementary material, our annotation pipeline involves both 2D human keypoint annotation using Sapiens and 3D SMPL-X model fitting using SMPLer-X. To ensure annotation quality, we conduct **manual verification at two stages**: 1) Pre-annotation verification is applied to the outputs of the human detection stage to filter out low-quality detections before proceeding with annotation. 2) Post-annotation verification is performed on the annotated data to correct or discard erroneous fits. Thanks to the high re-projection accuracy of SMPLer-X, most fitted SMPL-X models closely align with the image human body motion evidence upon projection, which supports their use as reliable supervision in our human-centric keyframe interpolation task. Furthermore, the effectiveness of the resulting dataset is validated through both **quantitative and qualitative** results presented in our paper. We believe this demonstrates that the SMPL-X annotations are of sufficiently high quality for training and evaluation.\n\n> **Q4**: **[Weakness 3]** The paper creates a new dataset with 2,614 videos by sourcing videos from Pexels, but it does not discuss or compare its contribution to other recent datasets like HumanVid [a] or MiraData [b], which also use Pexels videos for video-related tasks. Both mentioned datasets have a scale larger than this submission. A discussion of why a new dataset was necessary instead of augmenting existing ones would strengthen the justification for this contribution. For example, HumanVid [a] already curated human-centric 20K videos from Pexels, will this dataset shares a large portion of videos with this submission?\n> \n\n**A4**: Thank you for the suggestion to clarify our motivation for creating a new dataset. Our dataset is specifically designed for the Controllable Human-centric Keyframe Interpolation (CHKI) task, with a focus on complex human motions, human-centric annotations, and distant keyframe inputs. These features are not directly addressed by existing datasets. As described in Section 4 of the main paper and Section 1.4 of the supplementary material, our dataset is sourced from both the SportsSlomo dataset and the Pexels website. Specifically, to ensure motion complexity and diversity, we begin with videos from the **SportsSlomo** dataset, which contains challenging scenarios. Then, we further expand the dataset in diversity by retrieving additional videos from **Pexels** using a curated list of motion-related keywords. This targeted video collection strategy provides more precise control over the dataset content, which is crucial for training and evaluation in the CHKI setting. While there may be some overlap in Pexels-sourced content with datasets such as HumanVid and MiraData, our dataset is independently curated with a distinct task focus. Unlike those datasets, our design prioritizes motion complexity, keyframe temporal sparsity, and annotation consistency specific to CHKI. We appreciate the reviewer bringing up these related datasets and will cite both HumanVid and MiraData in the revised version, along with a discussion highlighting the distinctions and motivation behind our dataset construction."}, {"review_id": "iJVnv2zs8H", "reviewer": "Reviewer_CXZP", "summary": "Main goal of the paper is to propose a system for performing frame interpolation focused on human videos. This task is particularly challenging as human motion tends to be complex and highly articulated – which leads to a lot of ambiguity making it hard to model. This proposed solution is to reduce this ambiguity with per-frame 3D and 2D human poses (or control signals). The work introduced a framework to condition a pre-train video diffusion model on such poses and also introduces a dataset focused on human centric key frame interpolation withaccompanying annotations for 2D and 3D human poses (SMPL-X parameters).", "questions": "1. Can you report numbers on an approach similar to what you describe in section 1.4 of the supplementary material? This will help address point 2 of the weaknesses.\n\n2. Can you clarify if lines extracted from groundtruth intermediate frames are used for evaluating FCVG (instead of the interpolations from the start and end frames). If not, can FCVG be evaluated in such a set-up? This will help address point 1 of the weakness.", "limitations": "yes.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n\n* The core-contribution of the paper, PoseFuse3D seems novel and technically sound for adding 3D and 2D control signals to for interpolation. Overall, the idea of resolving motion ambiguity with per-frame 3D poses seems to be valuable and can be used for future work\n\n* The CHKI dataset appears to be a high-quality contribution. Descriptions of the data collection, filtering and annotation pipeline seem to indicate care and rigor in the dataset preparation, making it likely to be useful to the community.\n\n* Paper is well written and the experiments are conducted thoughtfully (except one detail as highlighted in the weakness section). I appreciate the care taken to standardize the backbone (i.e use SVD backbone when comparing against FCVG), making it easier to identify the improvements due this framework.\n\n**Weaknesses**\n\n* It is not clear if the groundtruth intermediate frames are used to extract line features for FCVG when evaluating against FCVG in Table 1 and 2. If not, this is a weakness in the evaluation as motion ambiguity should be removed for FCVG as well for a fair comparison to this method. \n\n* In line with the previous point, I think the approach suggested in section 1.4 of the supplementary material is a critical evaluation criterion for this method. Current method relies on extracting the control signals from the groundtruth intermediate frames. However keyframe interpolation as a problem relies on just the start and end frames and doesn’t not have access to the intermediate frames. So, to support the claim of a keyframe interpolation method, this work should show performance on a typical keyframe interpolation task.", "was_revised": true, "final_justification": "The paper solves an important sub-problem in keyframe interpolation, performs rigorous experiments and has novel contributions to make that can be utilized by other future work as well. Through their rebuttal, authors sufficiently address the weaknesses I could initially identify, hence i raise my rating to an Accept.", "rebuttal": "We really appreciate your constructive comments. Please find the following for our response.\n\n> **Q1**: **[Weakness 1]** It is not clear if the groundtruth intermediate frames are used to extract line features for FCVG when evaluating against FCVG in Table 1 and 2. If not, this is a weakness in the evaluation as motion ambiguity should be removed for FCVG as well for a fair comparison to this method.\n> \n\n**A1**: As described in the `Evaluation Protocol' paragraph (Sec. 5) of our main paper, we would like to clarify that intermediate ground-truth control signals are leveraged for evaluating all the controllable interpolation methods, including FCVG. We will revise the paper to clarify this point more explicitly in the updated version.\n\n> **Q2**: **[Weakness 2]** In line with the previous point, I think the approach suggested in section 1.4 of the supplementary material is a critical evaluation criterion for this method. Current method relies on extracting the control signals from the groundtruth intermediate frames. However keyframe interpolation as a problem relies on just the start and end frames and doesn’t not have access to the intermediate frames. So, to support the claim of a keyframe interpolation method, this work should show performance on a typical keyframe interpolation task.\n> \n\n**A2**:  We fully agree that it is important to evaluate interpolation methods in scenarios where no ground-truth intermediate control signals are available. We define this setting as `in-the-wild interpolation' in our supplementary material. In response to your suggestion, we conducted additional evaluations on the CHKI-Video test set for both FCVG and our proposed method, using only the start and end frames as inputs, without any intermediate information. The results are summarized in the table below:\n\n| Methods | PSNR$\\uparrow$ | PSNR$_{bbox}\\uparrow$ | PSNR$_{mask}\\uparrow$ | LPIPS$\\downarrow$ | LPIPS$_{bbox}\\downarrow$ | LPIPS$_{mask}\\downarrow$ | HA$\\uparrow$ |\n| --- | --- | --- | --- | --- | --- | --- | --- |\n| FCVG | 18.47 | 10.54 | 9.31 | 0.2607 | 0.1321 | 0.0921 | 0.9284 |\n| PoseFuse3D-KI | **19.30** | **10.98** | **9.69** | **0.2031** | **0.1194** | **0.0859** | **0.9289** |\n\nWe observe that under this setting, PoseFuse3D-KI consistently outperforms FCVG across all metrics, demonstrating the robustness and effectiveness of our method for in-the-wild interpolation. We will add this table in the updated version of our supplementary material.\n\n> **Q3**: **[Question 1]** Can you report numbers on an approach similar to what you describe in section 1.4 of the supplementary material? This will help address point 2 of the weaknesses.\n> \n\n**A3:** Thanks for your suggestion, we have conducted quantitative evaluations for the in-the-wild interpolation setting (described in section 1.4 of the supplementary material). The resulting table is presented in **A2**.\n\n> **Q4: [Question 2]** Can you clarify if lines extracted from groundtruth intermediate frames are used for evaluating FCVG (instead of the interpolations from the start and end frames). If not, can FCVG be evaluated in such a set-up? This will help address point 1 of the weakness.\n> \n\n**A4**: All evaluations of FCVG presented in both the main paper and the supplementary material were conducted with intermediate ground-truth control signals. We will revise the paper to clarify this point more explicitly in the updated version."}]

Author rebuttal:

Meta-review: Strengths
* Video keyframe interpolation technique focusing on humans by making use of the 2D-rasterized DWPose and SMPL-X, and 3D features from the specially-designed SMPL-X encoder, which maps 3D and 2D-projected SMPL pose information to a conditioning feature
* Dataset collection and curation strategy

Weaknesses
* Just another variant of the ControlNet application on top of an off-the-shelf video generation model
* Uses simple linear pose interpolation between keyframes, completely ignoring the difficulty of the pose interpolation/in-betweening problem and previous papers on this topic

The contributions are relatively incremental and weak, but may be worth sharing with the community as another idea for improving human video generations, under the condition that the authors will incorporate feedback from reviewers for the final revision.

Therefore, I agree with reviewers to accept this paper.
Decision: Accept (poster)