Latent Radiance Fields with 3D-aware 2D Representations
Authors: name: Chaoyi Zhou, name: Xi Liu, name: Feng Luo, name: Siyu Huang

Latent 3D reconstruction has shown great promise in empowering 3D semantic
understanding and 3D generation by distilling 2D features into the 3D space.
However, existing approaches struggle with the domain gap between 2D feature
space and 3D representations, resulting in degraded rendering performance. To
address this challenge, we propose a novel framework that integrates 3D
awareness into the 2D latent space. The framework consists of three stages: (1)
a correspondence-aware autoencoding method that enhances the 3D consistency of
2D latent representations, (2) a latent radiance field (LRF) that lifts these
3D-aware 2D representations into 3D space, and (3) a VAE-Radiance Field
(VAE-RF) alignment strategy that improves image decoding from the rendered 2D
representations. Extensive experiments demonstrate that our method outperforms
the state-of-the-art latent 3D reconstruction approaches in terms of synthesis
performance and cross-dataset generalizability across diverse indoor and
outdoor scenes. To our knowledge, this is the first work showing the radiance
field representations constructed from 2D latent representations can yield
photorealistic 3D reconstruction performance.