Title: Stroke3D: Lifting 2D strokes into rigged 3D model via latent diffusion models
Primary area: generative models
Keywords: Diffusion model, 3D generation, Graph diffusion

Abstract: Rigged 3D assets are fundamental to 3D deformation and animation. However,
existing 3D generation methods face challenges in generating animatable geometry,
while rigging techniques lack fine-grained structural control over skeleton
creation. To address these limitations, we introduce Stroke3D, a novel framework
that directly generates rigged meshes from user inputs: 2D drawn strokes and a
descriptive text prompt. Our approach pioneers a two-stage pipeline that separates
the generation into: 1) Controllable Skeleton Generation, we employ a
Skeletal Graph VAE (Sk-VAE) to encode the skeleton’s graph structure into a latent
space, where a Skeletal Graph DiT (Sk-DiT) generates a skeletal embedding.
The generation process is conditioned on both the text for semantics and the 2D
strokes for explicit structural control, with the VAE’s decoder reconstructing the
final high-quality 3D skeleton; and 2) Enhanced Mesh Synthesis via TextuRig
and SKA-DPO, where we then synthesize a textured mesh conditioned on the
generated skeleton. For this stage, we first enhance an existing skeleton-to-mesh
model by augmenting its training data with TextuRig—a dataset of textured and
rigged meshes with captions, curated from Objaverse-XL. Additionally, we employ
a preference optimization strategy, SKA-DPO, guided by a skeleton-mesh
alignment score, to further improve geometric fidelity. Together, our framework
enables a more intuitive workflow for creating ready-to-animate 3D content. To
the best of our knowledge, our work is the first to generate rigged 3D meshes
conditioned on user-drawn 2D strokes. Extensive experiments demonstrate that
Stroke3D produces plausible skeletons and high-quality meshes.