Title: SceneAdapt: Scene-aware Adaptation of Human Motion Diffusion
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: human motion generation, diffusion model adaptation, scene-aware human motion

Abstract: Human motion is inherently diverse and semantically rich, while also shaped by the surrounding scene. However, existing motion generation approaches address either motion semantics or scene-awareness in isolation, since constructing large-scale datasets with both rich text-motion coverage and precise scene interactions is extremely challenging. In this work, we introduce \textbf{SceneAdapt}, a framework that injects scene awareness into text-conditioned motion models by leveraging disjoint scene–motion and text–motion datasets through two adaptation stages: inbetweening and scene-aware inbetweening. The key idea is to use motion inbetweening, learnable without text, as a proxy task to bridge two distinct datasets, thereby injecting scene-awareness. In the first stage, we introduce keyframing layers that modulate motion latents for inbetweening while preserving the latent manifold. In the second stage, we add a scene-conditioning layer that injects scene geometry by adaptively querying local context through cross-attention. Experimental results show that \textbf{SceneAdapt} effectively injects scene awareness into text-to-motion models, and we further analyze the mechanisms through which this awareness emerges. Code and models will be released.

TL;DR: We adapt human motion diffusion for scene-aware text conditioned generation with only scene-motion paired dataset.