LDM3D: Latent Diffusion Model for 3D
Authors: name: Gabriela Ben Melech Stan; name: Diana Wofk; name: Scottie Fox; name: Alex Redden; name: Will Saxton; name: Jean Yu; name: Estelle Aflalo; name: Shao-Yen Tseng; name: Fabio Nonato; name: Matthias Muller; name: Vasudev Lal
Published: 2023-05-18 10:15:06

Abstract: This research paper proposes a Latent Diffusion Model for 3D (LDM3D) that
generates both image and depth map data from a given text prompt, allowing
users to generate RGBD images from text prompts. The LDM3D model is fine-tuned
on a dataset of tuples containing an RGB image, depth map and caption, and
validated through extensive experiments. We also develop an application called
DepthFusion, which uses the generated RGB images and depth maps to create
immersive and interactive 360-degree-view experiences using TouchDesigner. This
technology has the potential to transform a wide range of industries, from
entertainment and gaming to architecture and design. Overall, this paper
presents a significant contribution to the field of generative AI and computer
vision, and showcases the potential of LDM3D and DepthFusion to revolutionize
content creation and digital experiences. A short video summarizing the
approach can be found at https://t.ly/tdi2.