ISDrama: Immersive Spatial Drama Generation through Multimodal Prompting
Authors: name: Yu Zhang, name: Wenxiang Guo, name: Changhao Pan, name: Zhiyuan Zhu, name: Tao Jin, name: Zhou Zhao

Multimodal immersive spatial drama generation focuses on creating continuous
multi-speaker binaural speech with dramatic prosody based on multimodal
prompts, with potential applications in AR, VR, and others. This task requires
simultaneous modeling of spatial information and dramatic prosody based on
multimodal inputs, with high data collection costs. To the best of our
knowledge, our work is the first attempt to address these challenges. We
construct MRSDrama, the first multimodal recorded spatial drama dataset,
containing binaural drama audios, scripts, videos, geometric poses, and textual
prompts. Then, we propose ISDrama, the first immersive spatial drama generation
model through multimodal prompting. ISDrama comprises these primary components:
1) Multimodal Pose Encoder, based on contrastive learning, considering the
Doppler effect caused by moving speakers to extract unified pose information
from multimodal prompts. 2) Immersive Drama Transformer, a flow-based
mamba-transformer model that generates high-quality drama, incorporating
Drama-MOE to select proper experts for enhanced prosody and pose control. We
also design a context-consistent classifier-free guidance strategy to
coherently generate complete drama. Experimental results show that ISDrama
outperforms baseline models on objective and subjective metrics. The demos and
dataset are available at https://aaronz345.github.io/ISDramaDemo.