Stable Part Diffusion 4D: Multi-View RGB and Kinematic Parts Video
  Generation
Authors: name: Hao Zhang; name: Chun-Han Yao; name: Simon Donné; name: Narendra Ahuja; name: Varun Jampani
Published: 2025-09-12 20:39:43

Abstract: We present Stable Part Diffusion 4D (SP4D), a framework for generating paired
RGB and kinematic part videos from monocular inputs. Unlike conventional part
segmentation methods that rely on appearance-based semantic cues, SP4D learns
to produce kinematic parts - structural components aligned with object
articulation and consistent across views and time. SP4D adopts a dual-branch
diffusion model that jointly synthesizes RGB frames and corresponding part
segmentation maps. To simplify the architecture and flexibly enable different
part counts, we introduce a spatial color encoding scheme that maps part masks
to continuous RGB-like images. This encoding allows the segmentation branch to
share the latent VAE from the RGB branch, while enabling part segmentation to
be recovered via straightforward post-processing. A Bidirectional Diffusion
Fusion (BiDiFuse) module enhances cross-branch consistency, supported by a
contrastive part consistency loss to promote spatial and temporal alignment of
part predictions. We demonstrate that the generated 2D part maps can be lifted
to 3D to derive skeletal structures and harmonic skinning weights with few
manual adjustments. To train and evaluate SP4D, we construct KinematicParts20K,
a curated dataset of over 20K rigged objects selected and processed from
Objaverse XL (Deitke et al., 2023), each paired with multi-view RGB and part
video sequences. Experiments show that SP4D generalizes strongly to diverse
scenarios, including real-world videos, novel generated objects, and rare
articulated poses, producing kinematic-aware outputs suitable for downstream
animation and motion-related tasks.