MVLLaVA: An Intelligent Agent for Unified and Flexible Novel View
  Synthesis
Authors: name: Hanyu Jiang; name: Jian Xue; name: Xing Lan; name: Guohong Hu; name: Ke Lu
Published: 2024-09-11 09:25:37

Abstract: This paper introduces MVLLaVA, an intelligent agent designed for novel view
synthesis tasks. MVLLaVA integrates multiple multi-view diffusion models with a
large multimodal model, LLaVA, enabling it to handle a wide range of tasks
efficiently. MVLLaVA represents a versatile and unified platform that adapts to
diverse input types, including a single image, a descriptive caption, or a
specific change in viewing azimuth, guided by language instructions for
viewpoint generation. We carefully craft task-specific instruction templates,
which are subsequently used to fine-tune LLaVA. As a result, MVLLaVA acquires
the capability to generate novel view images based on user instructions,
demonstrating its flexibility across diverse tasks. Experiments are conducted
to validate the effectiveness of MVLLaVA, demonstrating its robust performance
and versatility in tackling diverse novel view synthesis challenges.