LDM3D-VR: Latent Diffusion Model for 3D VR
Authors: name: Gabriela Ben Melech Stan; name: Diana Wofk; name: Estelle Aflalo; name: Shao-Yen Tseng; name: Zhipeng Cai; name: Michael Paulitsch; name: Vasudev Lal
Published: 2023-11-06 16:12:10

Abstract: Latent diffusion models have proven to be state-of-the-art in the creation
and manipulation of visual outputs. However, as far as we know, the generation
of depth maps jointly with RGB is still limited. We introduce LDM3D-VR, a suite
of diffusion models targeting virtual reality development that includes
LDM3D-pano and LDM3D-SR. These models enable the generation of panoramic RGBD
based on textual prompts and the upscaling of low-resolution inputs to
high-resolution RGBD, respectively. Our models are fine-tuned from existing
pretrained models on datasets containing panoramic/high-resolution RGB images,
depth maps and captions. Both models are evaluated in comparison to existing
related methods.