Title: How Animals Dance (When You're Not Looking)
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: animal dance video generation; choreography pattern; long video generation

Abstract: We present a framework for generating music-synchronized, choreography aware animal dance videos. Our framework introduces choreography patterns---structured sequences of motion beats that define the long-range structure of a dance---as a novel high-level control signal for dance video generation. These patterns can be automatically estimated from human dance videos. Starting from a few keyframes representing distinct animal poses, generated via text-to-image prompting or GPT-4o, we formulate dance synthesis as a graph optimization problem that seeks the optimal keyframe structure to satisfy a specified choreography pattern of beats. We also introduce an approach for mirrored pose image generation, essential for capturing symmetry in dance. In-between frames are synthesized using an video  diffusion model. With as few as six input keyframes,  our method can produce up to 30 seconds dance videos across a wide range of animals and music tracks.