AdaViewPlanner: Adapting Video Diffusion Models for Viewpoint Planning
  in 4D Scenes
Authors: name: Yu Li, name: Menghan Xia, name: Gongye Liu, name: Jianhong Bai, name: Xintao Wang, name: Conglang Zhang, name: Yuxuan Lin, name: Ruihang Chu, name: Pengfei Wan, name: Yujiu Yang
Organization: Kling Team

Recent Text-to-Video (T2V) models have demonstrated powerful capability in
visual simulation of real-world geometry and physical laws, indicating its
potential as implicit world models. Inspired by this, we explore the
feasibility of leveraging the video generation prior for viewpoint planning
from given 4D scenes, since videos internally accompany dynamic scenes with
natural viewpoints. To this end, we propose a two-stage paradigm to adapt
pre-trained T2V models for viewpoint prediction, in a compatible manner. First,
we inject the 4D scene representation into the pre-trained T2V model via an
adaptive learning branch, where the 4D scene is viewpoint-agnostic and the
conditional generated video embeds the viewpoints visually. Then, we formulate
viewpoint extraction as a hybrid-condition guided camera extrinsic denoising
process. Specifically, a camera extrinsic diffusion branch is further
introduced onto the pre-trained T2V model, by taking the generated video and 4D
scene as input. Experimental results show the superiority of our proposed
method over existing competitors, and ablation studies validate the
effectiveness of our key technical designs. To some extent, this work proves
the potential of video generation models toward 4D interaction in real world.