Latent Radiance Fields with 3D-aware 2D Representations
Authors: Chaoyi Zhou
Xi Liu
Feng Luo
Siyu Huang
Venue: ICLR 2025 Poster (2025)
Keywords: 3D Gaussian Splatting
3D-aware Representation
Primary area: applications to computer vision, audio, language, and other modalities
TL;DR: To our knowledge, this is the first work demonstrating that radiance field representations in the latent space can achieve decent 3D reconstruction performance across various settings including indoor and unbounded outdoor scenes.

Abstract: Latent 3D reconstruction has shown great promise in empowering 3D semantic understanding and 3D generation by distilling 2D features into the 3D space. However, existing approaches struggle with the domain gap between 2D feature space and 3D representations, resulting in degraded rendering performance. To address this challenge, we propose a novel framework that integrates 3D awareness into the 2D latent space. The framework consists of three stages: (1) a correspondence-aware autoencoding method that enhances the 3D consistency of 2D latent representations, (2) a latent radiance field (LRF) that lifts these 3D-aware 2D representations into 3D space, and (3) a VAE-Radiance Field (VAE-RF) alignment strategy that improves image decoding from the rendered 2D representations. Extensive experiments demonstrate that our method outperforms the state-of-the-art latent 3D reconstruction approaches in terms of synthesis performance and cross-dataset generalizability across diverse indoor and outdoor scenes. To our knowledge, this is the first work showing the radiance field representations constructed from 2D latent representations can yield photorealistic 3D reconstruction performance.

Published as a conference paper at ICLR 2025
LATENT RADIANCE FIELDS WITH 3D-AWARE 2D REP- RESENTATIONS
Chaoyi Zhou", Xi Liu, Feng Luo, Siyu Huang!
Visual Computing Division
School of Computing Clemson University
{chaoyiz, xi9, luofeng, siyuhh@clemson. edu
axXiv:2502.096133 [cs.CV] 13 Feb 2025
Groundtruth
3DGS Latent Space
Latent Radiance Fields (ours)
Figure 1: This work novelly enables the radiance field representations on the latent space of VAE, achieving photorealistic 3D reconstruction performance on unbounded outdoor scenes.
ABSTRACT
Latent 3D reconstruction has shown great promise in empowering 3D semantic understanding and 3D generation by distilling 2D features into the 3D space. However, existing approaches struggle with the domain gap between 2D feature space and 3D representations, resulting in degraded rendering performance. To address this challenge, we propose a novel framework that integrates 3D aware- ness into the 2D latent space. The framework consists of three stages: (1) a correspondence autoencoding method that enhances the 3D consistency of 2D latent representations, (2) a latent radiance field (LRF) that lifts these 3D- aware 2D representations into 3D space, and (3) VAE-Radiance Field (VAE-RF) alignment strategy that improves image decoding from the rendered 2D repre- sentations. Extensive experiments demonstrate that our method outperforms the state-of-the-art latent 3D reconstruction approaches in terms of synthesis perfor- mance and cross- datsset generalizability across diverse indoor and outdoor scenes. To our knowledge, this is the first work showing the radiance field representations constructed from 2D latent representations yield photorealistic 3DD reconstruc- tion performance. The project page latent-radiance field. git.ub..t....o.
INTRODUCTION
Recently, significant advancement in radiance field representation, such as Neural Radiance Fields (NeRF) (Mildenhall et al., 2020) and 3D Gaussian Splatting (3DGS) (Kerbl et al., 2023), have been made for fast and high-quality 3D reconstruction and novel view synthesis (NVS). As demonstrated
Equal contribution
Corresponding author: Siyu Huang

---

Published as a conference paper at ICLR 2025
by Stable Diffusion Models (Rombach et al., 2021), optimizing in the 2D latent space instead of the image space can significantly boost generation efficiency. Mcanwhile, a 3D-consistent latent space and photorealistic decoding capability can benefit many tasks such text-to-3D generation, latent NVS, few-shot NVS, efficient NVS, 3D latent diffusion model, and 3D semantic understanding. To empower 3D semantic understanding, researchers have explored latent 3D reconstruction methods, such as Feature 3DGS (Zhou et al.. 2024a), to distill 2D semantic features into 3D space for novel view semantic segmentation. However, there are significant domain gaps between the 2D feature space and 3D representations, arising from the lack of consistent 3D spatial structure information, which hinders the direct feeding of 2D features into the 3D representations. The 2D feature extrac- tors cannot effectively perceive the 3D structures behind the inputs images since the training images are presented to the network in an unstructured way and the training objective does not include 3D consistency. Therefore, the loss of 3D awareness is inevitable.
Few previous work attempt to bridge the gap between the 2D features and 3D representations. By focusing on better 3D scene understanding, Feature 3DGS (Zhou et al., 2024a) proposes to distill a feature field from 2D semantic features by leveraging the view-independent approach while cannot model the view-dependent visual properties. Another line of work improves the latent field in the context of 3D generation task. Latent-NeRF (Metzer et al., 2022) and ED-NeRF (Park et al., 2023) includes additional per-scene refinement layer to enhance the latent rendering quality, while introducing more computation cost and exhibiting the generalizabilty.
To bridge the gap between the 2D latent space and 3D representations, we observe main chal- lenges: Firstly, the massive view-dependent high-frequency noise in the 2D latent space causes the inconsistent geometry and unstable optimization. Moreover, data distribution shift by applying RGB-based NVS methods to latent features also prohibits the photorealistic rendering. To tackle with these two issues, our key insight is embed 3D awareness into the latent space, while maxi- mumly preserving the representation ability of autoencoders without introducing any additional lay- ers. We propose a novel framework that builds a latent radiance field (LRF) based on the 3D-aware 2D representations. Specifically, it consists of three stages. Firstly, we introduce correspondence- aware autoencoding method to improve the 3D awaresness of the VAE's latent space, making the 2D representations follow the geometry consistency. Then, we build the LRF to represent 3D scenes from the 3D-aware 2D representations, lifting the 3D-aware 2D representations into the 3D space. Finally, we introduce distribution shift caused by NVS and boost the performance of image decoding from the rendered 2D representations. In together, the created 3D-aware latent space and LRFs can be smoothly injected into existing NVS or 3D generation pipelines without further fine-tuning, achieving high-quality and photorealistic synthesis results.
AXE-RRaiaanee Field VAE-RF) alignment method to further mitigate the data
To the best of our knowledge, this is the first work demonstrating that radiance field representa- tions constructed in the latent space, with the injection of 3D awareness, can achieve photorealistic 3D reconstruction performance across various settings including indoor and unbounded outdoor scenes. Extensive NVS, 3D generation, and few-shot novel view synthesis experiments show that our method outperforms existing methods with respect to its high-quality synthesis and cross-dataset generalizability, as shown in Fig. and the following sections. In summary, main contributions of this work include:
We introduce a novel framework to integrate 3D awareness into 2D representation learning, in- cluding a correspondence- aware autoencoding method and VAE-Radiance (VAE-RF) field align- ment to enable high-quality 3D reconstruction in latent space.
We propose the latent radiance field (LRF) to effectively elevate the 3D-aware 2D representations into 3D latent fields. It represents the first step towards constructing radiance field representations directly in the latent space for 3D reconstruction tasks.
We conduct extensive experiments to show that our method achieves superior fidelity and cross- dataset generalizability across NVS, few-shot NVS, and 3D generation tasks.
2 RELATED WORK
Injecting 3D priors into 2D representations. While many existing works focus on incorporat- ing 2D features into 3D representations, which improves performance in downstream tasks such as
2

---

Published as a conference paper at ICLR 2025
scene understanding (Zhi et al., 2021; Ha & Song, 2022; Qin et al., 2023; Shi et al., 2023; Zhou et al., 2024a; Cen et al., 2023; Gu et al., 2024; Guo et al.. 2024), less attention has been paid to the opposite direction: leveraging 3D knowledge to enhance 2D features, which benefit challenging tasks that require 3D understanding while the perceived information is limited such as monocular depth estimation (Stan al., 2023; Bhat et al., 2023; Piccinelli et al., 2024; Chatterjee et al., 2024; Moon et al., 2023) and semantic segmentation (Wang et al., 2023; Sun et al., 2024). Studies such as (Bachmann al., 2022; Zhou al., 2024a) utilize 3D priors from multi-view and geometric information to improve the Masked Autoencoders (He et al., 2021), achieving better performance on downstream tasks of segmentation and detection. However, directly injecting the geometry con- straints into the pre-trained feature extractors is harmful for the self-supervised 2D representation and heavily relying on pre-trained feature extractors poses potential limitations for performance and requires significant computational resources. In contrast, our method does not require any addi- tional per-scene refinement module, serving as an efficient and generalizable approach for injecting 3D priors into 2D representations.
Radiance field representations on images and features. Neural Radiance Fields (NeRF) (Milden- hall et al., 2020) and 3D Gaussian Splatting (3DGS) (Mildenhall al., 2020) are benchmark ra- diance field representation methods for the NVS task. NeRF represents 3D scenes and renders photorealistic novel views based on the representation capacity of neural networks. 3DGS employs a set of 3D Gaussian primitives to represent 3D scenes, and a fast differentiable rasterizer to enable more efficient rendering while keeping the photorealism of novel views. However, the distillation of the 2D features into the 3D representations remains challenging, mainly due to the significant geometric inconsistency in the feature maps caused by massive high-frequency information. There- fore, some recent literature (Zhou et al., 2024a; Kobayashi et al., 2022; Siddiqui et al., 2023; Fan et al., 2022; Kerr et al., 2023) propose alternative solutions by leveraging the geometry information from the RGB space to help the 3D reconstructions of 2D features. Fit3D (Yue et al., 2024) builds a huge amount of 3D representation dataset as the superivsion for the pre-trained feature extractor fine-tuning: however, without considering the compatibility of the 3D representation and 2D feature space, they also require a customized decoder to ensure the performance in the downstream tasks. All the methods mentioned above all rely the per-scene optimization with additional modules, while our method bridging the gap between 2D feature space and 3D representation with an efficient method.
correspondence-anee
Text-to-3D generation with 2D priors. Despite the impressive 3D generation capabilities demon- strated by many existing 2D generative prior-guided works (Tang al., 2023; Poole al., 2022; Wu et al., 2023; Zhou et al., 2024b; Jain et al., 2022; Michel et al., 2021), performing paok-propagation of the Score Distillation Sampling (SDS) loss (Poole et al.. 2022) on images is computationally intensive and time-consuming Latent diffusion models (LDMs) offer more efficient solutions by operating in the latent space. However, the vastly different distribution of the latent space means that directly utilizing the latent representations for NVS leads to degraded rendering performance. To our knowledge, only few works attempt to overcome this challenging task. Latent-NeRI (Met- zer et al., 2022) employs per-scene refinement layer to map the rendered latent to RGB space as an additional constraint for training the NeRF representations. ED-NeRF (Park et al., 2023) introduces more complex refinement module by initializing from a set of specific layers in Variational Au- toencoder (VAB). Although these per-scene refinement modules effectively mitigate the artifacts in the rendering results, they require resource-consuming optimization for each scene, and lack gen- eralization ability to novel views or scenes. Moreover, the smoothness introduced by the neural networks hinders the reconstruction of high-frequency signals on the 2D features. On the contrary, our method requires no additional efforts for lifting the 2D features to the 3D radiance field repre- sentations, such that it can be injected into any existing NVS or text-to-3D frameworks smoothly and efficiently.
3 PRELIMINARIES
Variational autoencoder. A variational autoencoder (VAE) (Kingma, 2013) is generative model that represents nigh-dimensional data distributions in a lower-dimensional latent space. The encoder maps the input data x to latent variable by estimating the parameters of posterior distribution qo (z)x). The posterior is typically assumed to follow the Gaussian distribution. parameterized by a mean (x) and variance (x)). The latent variable 2 is sampled from this posterior distribution,
3

---

Published as a conference paper at ICLR 2025
ie., (z(z)) (zz (z(xx, (xx))) The decoder reconstructs the input x by mapping back to the data space through the likelihood p=(x)z). The learning objective of is:
LVAe (0, o; X) = E.g ( log pe [)) - KL g 2 [())
(1)
3D Gaussian Splatting. 3DGS (Kerbl et al., 2023) is efficient NVS framework that uses a set of 3D Gaussian primitives to represent a scene explicitly. Each Gaussian primitive has a position vector μ E R 3D covariance matrix Σ E 33x3 opacity E I, and spherical harmonics (SH) coefficient e E R& (Ramamoorthi & Hanrahan, 2001) representing the view dependent colors.
G(z) = = e (--
(2)
where Σ RSST RT, S denotes the scaling matrix and R is the rotation matrix. Then, raster- ization (Zwicker et al., 2001) can transform the 3D Gaussian spheres to the 2D camera plane to calculate the 2D covariance matrix in the camera space
Σ -WEWTTJ JT,
(3)
where W is the perspective transformation matrix and J is Jacobin of the projection matrix. For every pixel, the Gaussians are traversed in depth order from the image plane, and their pixel colors e; are combined through alpha compositing, forming pixel color C
C Σ GO II (1 (1-a)). j=1
iEN
4 METHOD
In this work, we propose a method to achieve 3D-aware 2D representations and enable 3D recon- struction the latent space. We base our method on the widely used Variational Autoencoder (VAE) from Latent Diffusion models (Metzer et al., 2022). To enhance the 3D awareness of both encoder and decoder of the VAF, we present a three-stage pipeline as illustrated in Fig. 2. The first stage focuses improving the 3D awaresness of the VAE's encoder through novel correspondence- aware constraint on the latent space, making the 2D representations follow the geometry consis- tency (Sec. 4): The second stage builds a latent radiance field (LRF) to represent 3D scenes from the 3D-aware 2D representations (Sec. 4.2); The third stage further introduces a VAE-Radiance Field (VAE-RF) alignment method to boost the reconstruction performance (Sec. 4.3). In together, our LRF enables 3D reconstruction on the 2D latent space instead of the image space. It can render high-quality and photorealistic novel views, even for the unbounded scenes (Sec. 5). More details of our method are discussed in the following sections.
4.1 CORRESPONDECE-AWAN AUTOENCODING
The first stage of our method is incorporating the -eoretry-awwreness into the autoencoding pro- cess. Given muilt-view images I = {4) (I, E #x*xx))). the VAE encoder extracts 2D representations Z = { Z2 E H' x4 in low-dimensional latent space while the semantic information can be preserved effectively. However, as shown in Fig. most of exist- ing NVS frameworks fail to reconstruct the photo-realistic images from the rendered latents. It is mainly because the VAE encoding process significantly damages the multi-view consistency within the original image space, since the latent space presents massive high-frequency noises to compress the original RGB space into a compact latent space (see Fig. 3). This brings severe challenges for reconstructing the 2D latent representations in the 3D space.
Correspondence consistency on the latent space. To address the above issue and enable ef- fective latent 3D reconstruction, are inspired by the multi-view correspondence consistency which serves as the foundation principle for modeling the natural 3D world. Specifically, points 2; R² in image I; and points æj TR² in another image I, are considered correspondences if they are connected by the fundamental matrix F,j 3x333 satisfying the multi-view geometry constraint (Schönberger Frahm, 2016):
ac F Fijo:
(5)
4

---

Published as a conference paper at ICLR 2025
Stuge-I: Correspondence aware Autoencoding (Soc. 4.f)
Stuge- Hi: Latent Radiance Field, LRP (Sec 4.2)
Stuge-HII: CAE-Radiance Field Aligmment 4.3)
Encode
Render
Decoci
Figure 2: An illustration of our pipeline for creating a latent radiance field in conjunction with 3D-aware 2D representation fine-tuning. Firstly Stage-I, we inject 3D awareness into the VAE's encoder through applying novel correspondence consistency constraint on the latent space, making the 2D representations follow the geometry consistency. Then in Stage-II, we create the latent radiance field (IRF) to represent 3D scenes based on the 3D>-aware 2D) representations. Finally in Stage-III, we introduce a VAE-Radiance Field alignment method to enhance the performance of image decoding from the rendered latent space.
Eq. 5 tells that pair of correspondence points on the image space should be close to each other, so that the consistent geometry can be ensured during the optimization in the 3D space; otherwise, the artifacts and redundant geometry representation due to the local optimal will damage the quality of the 3D reconstruction and novel view synthesize. Motivated by this, we propose an computationally efficient strategy that incorporates the correspondence consistency into the autoencoder training. Specifically, set of multi-view images I { (- (4 E ## x3) are fed into the autoencoder
to extract the latent representations Z { | Z; E =='''"x44) and the correspondence
consistency loss the latent space is computed by
K
Leorres
> -
(6)
A= (ER((()
where refers to the the latent pixel in the Z; and z; is the corresponding latent pixel in the neigh- bouring latent Zy. L.ooees ensures that the encoded features follow the correspondence consistency derived from the multi-view images, where Xıj is the weight based on the average pose error (APE) calculated from the Frobenius norm between the two camera poses of images I; and Ij to weight the accurate pose distance to represent the view-dependant latent codes. The detail of calculating Xi can be found in Appendix A.1 By injecting the latent correspondence consistency into the standard VAE training, our VAE training objective is:
Lsugel = LVAE + X Leorres + Xa Krey.
LAV is original VAE traning objective for VAE in Eq. 1. freg = KL. (x) Goriginal Z x)) enforces the fine-tuned 2D representations being close those of the pre-trained VAE, preserving the representation capability of the finet-tuned autoencoder. This new learning objective ensures that the compact latent space of VAE preserves the multi-view geometric consistency, such that is compatible with existing NVS frameworks such as 3DGS.
Insight into latent correspondence consistency. The maximum degree of the spherical harmonics is always set as 3 in NVS methods for the efficiency and robustness in the modeling the view- dependant information. To be more specific, the lower degree terms is aim to mostly capture low- frequency information such as albedo for the scene while the higher degrees are tended to model the high-frequency, view dependent information such as the lightning. For the latent space, the latent code can be considered as the combination of the base value and high frequency noise. Due to such a compact representation. the amount of the noise can be greatly increase compared to the RGB space, creating more difficulties for the SH coefficients to model the information from different views. When maximum degree is fixed, it is easier for SH coefficients to reach the global optimal instead of locally over-fitting Fortunately, with our Loones: the high frequency noise be
5

---

Published as a conference paper at ICLR 2025
Image
VAE latent
Finetuned latent
VAE latent FFT
Finetuned latent FFT
Figure 3: A visualization of latent spaces of original and our fine-tuned VAEs. Our method ensures an accurate geometry in the latent space while removing certain amount of high-frequency noises.
effectively removed while the high-quality image generative ability can still be preserved, leading to more stable process of the optimization and consistent geometry representation. Fig. shows that the correspondence-wware encoding can significantly remove the high frequency noises in the 2D latent space and the visualization of applying Fast Fourier transform also showing less high- frequency noise in latent space achieved by our encoder, resulting effective approach to lifting the 2D features into the 3D latent fields.
4.2 LATENT RADIANCE FIELD
Based on the 3D-aware 2D representation fine-tuning discussed Sec. 4.1, we create 3D represen- tations directly in the 2D latent space of VAE, namely the latent radiance field (LRF). We take 3DGS (Kerbl et al., 2023) as an example of radiance field representations to discuss our LRF
By following 3DGS, set of latent 3D Gaussians is formulated as
(11, 8, R, a, SH,);)
where 14 3 the 3D mean of the Gaussian, S diag(s) E RRa3 the Gaussian scale, R E 3x3 its orientation, E R a per-Gaussian opacity, and SH, models the dieeddepnndant latent in the 3D latent space. By following the differentiable rasterization process of 3DGS, we rasterize the 2D latent representations using point-based a-blending as follows:
Z
Σ 2:0; II (1))),, SEN
where / is set of ordered Gaussians overlapping the pixel, z, E R is the siew-dependent latent code of each Gaussian, where dim is the number of the latent dimension of the feature. and a; is given by evaluating 2D Gaussian with covariance E multiplied with learned per-point opacity.
x3) be set of multi-view images of scene with corresponding Let I [ (I: E R #xxx33
camera parameters. Let 2 { : Zz E Hx be a corresponding set of latents from the VAE encoder. The rasterization function renders set of latent Gaussians into 2D latent representation according to the camera pose P., Then, we optimize the latent Gaussian parameters, to optimally represent latent Z:
G= arg min (...R..a,SHH
Σ e (r(G,.))). 1=1
(10)
where L is pixel-wise loss combined with a D-SSIM term. Notably, we do not need im- pose additional geometric consistency constraints introduced by previous literature (Yue et al., 2024; Kobayashi et al., 2022: Zhou et al., 2024a), as our correspondence-awiit autoencoder fine-tuning en- sures geometrically consistent 2D representations in the 3D space. Therefore, our LRF reconstructs the 2D latent representations as radiance field representation directly, and enables an efficient rendering of the 2D latent representations for novel views.
4.3 VAE-RADIANCE FIELD ALIGNMENT
Although the correspoondence-aveee awaeee autoencoding introduced in Sec. 4.1 improves the 3D con- sistency of VAE latent space, the LRF distribution p(2Nvs) still shifted from the VAE latent

---

Published as a conference paper at ICLR 2025
Table 1:
Our method outperforms the image and latent space NVS baselines on most settings and
metrics, from object-level to unbounded outdoor scenes. Latent- NeRnF denotes we adapt it to NVS.
Image Space
Latent Space Feature-GS
Dataset
Metric
3DGS/8
Mip-Splating/8
3DGS-VAE
Latent-NeRF"
3DGS-LRF (Ours)
PSNR
16.93
24.89
25.04
18.50
21.09
26.26
MVImgNet
SSIM
0.561
0.799
0.824
0.709
0.772
0.863
LPIPS
0.466
0.328
0.250
0.403
0.372
0,178
PSNR f
9.98
19.68
19.07
18.31
16.48
20.00
NeRF-LLFF
SSIM
0.110
0.484
0.493
0.457
0.415
0.541
LPIPS
0,631
0.513
0.364
0.387
0.539
0.289
PSNR
14.03
21.81
20.57
18.16
16.60
22.45
D13DV-10K
SSIM
0,352
0.609
0.595
0.530
0.449
0.667
LPIPS
0,541
0.451
0.346
0.432
0.602
0.197
PSNR
14.79
22.38
19.44
15.93
17.13
20.83
Mip-NeRF360
SSIM
0.273
0.502
0.404
0.312
0.337
0.469
LPIPS
0.586
0.521
0.432
0.537
0.642
0.328
distribution p( (VAE) due to the non-linearity in neural rendering, resulting in performance decrease when we decode LRF rendering results back to images through the VAE decoder.
We further propose to fine-tune the VAE decoder under the radiance field guidance to address this issue. With the LRF built Sec. 4.2, we can reconstruct LRFs from a large amount of scenes to generate a latent-image paired dataset. This dataset consists of the 2D latent representations
2 [ Z R "'''''W'x'" x4) rendered by LRFs and the corresponding ground truth images
I = { (Ig E 3 Notably, we also include the training views of LRFs in this dataset, since key feature of existing NVS methods is to overfit the training views. The training objective of our VAE-RF alignment decoder fine-tuning is:
LsnagelTI
Avrain | ([rri))))) - Irain || + Anovel [D( - Lnvell Il
(11)
where D-)) is the decoder, Zurin and Znovet the latent codes of the training views and novel views, respectively. 7 refer to the corresponding ground truth images. Anovel and Anovel are the weighting coefficient that balances the contributions of the training and novel views. Both of the weights set to 0.5 to ensure that the decoder learns not only to decode effectively from the training views but also generalize and perform well the novel views. Eq. 11 effectively minimizes the distribution mismatch between the VAE latent space and the LRF rendering space. After decoder fine-tuning, high-quality images can be reconstructed from the LRF rendering of either training or novel views. The fine-tuned autoencoder enhances 3D reconstruction and generation by providing geometry- aware 2D latent space as well as radiance field-compatible autoencoder.
EXPERIMENTS
5.1 LATENT 3D RECONSTRUCTION
We first evaluate LRF on four real-world datasets, including MVIngNet (Yu et al., 2023), NeRF- LLFF (Mildenhall al., 2019), MipNeRF330 (Barron al., 2022), and DL3DV-10K Ling et al. (2024), to demonstrate the effectiveness of our approach for latent 3D reconstruction. Among these datasets, DL3DV serves an in-distribution dataset, where the training set is used for model train- ing, and the test set is used for evaluation. In contrast, MVImgNet, LLFF, and Mip-NeRF360 are dut-of-distribution datasets, as they have never been used in the training process. We follow the standard train and test split in 3DGS and Srpp-Spaattnnneettt et al., 2023; Yu et al., 2024).
Fig. shows that our method significantly improves the capability of the 2D latent representations for 3D reconstruction task. Our approach mitigates the artifacts such ghosting, color distortion, blurring, and texture warping caused by 3D inconsistency. While the latent and image space ap- proaches share the same input resolution, our rendering results present clearer visual details, richer textures, and more high-frequency information.

---

Published as a conference paper at ICLR 2025
NeRF-LLFF DL3DV-10K
Groundruth
Ours
3DGS VAE
MpllSinatt
Groundrruth
Ours
3DGS-VAL VAE
Mip-Splatting88
Groundrruth
Ours
JAGS-VAE
Mip-NeRF336
Mip-Splatting/8
Feature
3DGS#
Creundrruth
Ours
3DUS- VAB
Mip-Splatting/
Figure 4: A visual comparison of rendering results. Our method can not only render high-quality images for in-distribution dataset (DL3DV-10K), but also shows great generalization ability across different datasets.
8

---

Published as a conference paper at ICLR 2025
Ours
Dreamfusion
Ours
Dreamfusion
Ours
GSGEN
lego
vase with pink flowers
A DSLR photo of tray of sushi containing pugs
Visual comparison of different text-to-3D generation methods. Our model enables the generation of more view-consistent results.
Figure 5:
As shown in Table 1, LRF achieves the state-of-the-art performance across all datasets in terms of metrics of PSNR. SSIM, and LPIPS. These results underscore the effectiveness of our approach in fine-tuning latent space representations support novel view synthesis. This demonstrates that our fine-tuning approach not only ef- fectively reduces the geometry information loss caused by 3D-inconsistent 2D representations but also preserves
Table 2:
A comparison of different
methods on LLFF dataset using views.
Method
PSNR
SSIM f LPIPS
3DGS-VAL VAE
13.06
0.283
0,570
3DGS
13.79
0.331
0.468
Mip-Splatting
13.70
0.315
0.486
Ours
15.51
0.379
0.465
perceptual and textural information NVS outputs. Compared the original VAE model, our fine- tuning approach significantly enhances 3D-consistnncy in the 2D latent representations by enforcing correspondence points to be consistent, resulting in superior latent NVS performance across all metrics. "Image Space" means that we input images to 3DGS with the same resolution the la- tent representations, then render output images with the same resolution as before latent encoding. Since render high-resolution images from low-resolution training images, to avoid unfair com- parisons caused by aliasing, we also compare our method with the Mip-Splatting (Yu et al., 2024) which is specialized super-resolution rendering. Compared with these image space methods, latent reconstruction method still achieves better performance on most of the datasets, highlighting its potential for future work in efficient 3D representation learning.
We also show the generalizability by performing the synthesis of few-shot novel views in NeRF- LLFF dataset. We follow the same experimental configurations in the previous work (Li al., 2024: Liu et al., 2024). And we keep the same input resolution for all the methods. As shown in Table our method outperforms the other image-space approaches in the sparse-view setting.
5.2 TEXT-TO-3D GENERATION
We evaluate our method for the state of art text-to-3l) generation framework in both latent and image space. We leverage the GSGEN (Chen al., 2024) and Dreamfusion (Poole al., 2022) as the image space generation framework, while we use Latent-NeRF (Metzer et al., 2022) as the latent space method. GSGEN optimized the 512x512 image space. Dreamfusion optimized in the 800x800 image space. Latent-NeRF is optimized in the 128 x 128 latent space and then reconstruct images resolution of 1024x 1024. By following the prompts evaluated in these two works, we generate 3D objects and render them from multiple views. The text prompts fed into the GSGEN are more complicated considering it the state of the art generation method.
As shown in Fig. 5, our method can boost the performance under extremely complicated text prompts, achieve complex geometry while preserving the multi-view consistency. Moreover, our encoder model can significantly enhance the high-frequency details such as the texture of the fried chicken. Besides, our approach compatible with the diffusion model operating within the origi- nal VAE latent space. Without necessitating any fine-tuning of the diffusion U-Net parameters, the diffusion process remains capable of accurately denoising the 2D latent representations provided by fine-tuned VAE, according the text guidance. Furthermore, the VAE-RF alignment in decoder

---

Published as a conference paper at ICLR 2025
Table 3: We ablate correspondence-aware autoencoding and VAE-radiance field aligned decoder
fine-tuning on DL3DV-IOK dataset to reveal their necessity in latent 3D reconstruction
VAE Encoder fine-tuned
Decoder fine-tuned
PSNR
SSIMT
LPIPS!
20.57
0.595
0.346
21.16
0.620
0.282
21.73
0.645
0.208
22.45
0.667
0.197
No fine-tuning
Encoder fine-tuning
Groundtruth
Decoder fine-tuning
Ours
Figure A qualitative study the effect of different fine-tuning stages for view synthesis results.
fine-tuning also facilitates the reconstruction of rendered latent representations, improving the image quality after VAE decoding.
5.3 ABLATION STUDY
We conduct ablation studies on two major components of our three-stage framework, the correspondence-awarc autoencoding and the VAE-RF aligned decoder fine-tuning, assess their contributions to overall performance. The quantitative results shown in Table 3 indicate that both components contribute to performance improvement. Notably, the decoder presents more signifi- cant impact on the results, as it directly influences the reconstruction of images from the latent space, thereby leading stronger performance gains. Although the encoder does directly act on image reconstruction, it enhances geometric consistency of 2D representations, which also contributes to reconstruction.
performance improvement
The qualitative results are shown in Fig. 6. The encoder fine-tuning allows the 3D latent space to capture more precise geometry, reduce blurriness in the synthesized images, and recover finer details. Additionally, the decoder fine-tuning further refines the results by rectifying inaccuracies and preserving perceptual and textural fidelity. Together, these modules synergistically contribute to significant improvements in the overall pipeline.
CONCLUSION
This paper introduces the Latent Radiance Field (LRF), which to our knowledge, is the first work to construct radiance field representations directly in the 2D latent space for 3D reconstruction. We present a novel framework for incorporating 3D awareness into 2D representation learning, featur- ing a correspondence-awarr autoencoding method and VAE-Radiance Field (VAE-RF) alignment strategy to bridge the domain gap between the 2D latent space and the natural 3D space, thereby significantly enhancing the visual quality of our LRF. Future work will focus incorporating method with more compact 3D representations, efficient NVS, few-shot NVS in latent space, as well exploring its application with potential 3D latent diffusion models.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENT
The authors thank Minghui Xu for beneficial discussions. This work is partially supported by the AIM for Composites, an Energy Frontier Research Center funded by the U.S. Department of Energy (DOE), Office of Science, Basic Energy Sciences (BES), under Award # DE-SC0023389 and by the US National Science Foundation (NSF; Grant Number MTM2-2025541 01A-2242812). The authors acknowledge research support from Clemson University with generous allotment of com- putation time on the Palmetto cluster.
REFERENCES
Roman Bachmann, David Mizrahi, Andrei Atanov, and Amir Zamir. MultiMAF: Multi-modal multi- task masked autoencoders. 2022.
Jonathan T. Barron, Ben Mildenhall, Dor Verbin, Pratul P. Srinivasan, and Peter Hedman. Mip-nerf 360: Unbounded anti-aliased neural radiance fields. CVPR, 2022.
Shariq Farooq Bhat, Reiner Birkl, Diana Wofk, Peter Wonka, and Matthias Müller. Zoedepth: Zero- shot transfer by combining relative and metric depth. arXiv preprint arXiv:2302. /2288, 2023.
Jiazhong Cen, Jiemin Fang, Chen Yang, Lingxi Xie, Xiaopeng Zhang, Wei Shen, and Qi Tian. Segment any 3d gaussians. arXiv preprint arXiv:2312.0088 2023.
Agneet Chatterjee, Tejas Gokhale, Chitta Baral, and Yezhou Yang. On the robustness of language guidance for low-level vision tasks: Findings from depth estimation. In Proceedings of the IEEE/CVF Conference Computer Vision and Puttern Recognition (CVPR), pp. 2794-2803, June 2024.
Zilong Chen, Feng Wang, Yikai Wang, and Huaping Liu. Text-to-3d using gaussian splatting, 2024. URL https::/ /atsiv///rriiv..// org/ abs/2305 16585.
Zhiwen Fan, Peihao Wang, Xinyu Gong, Yifan Jiang, Dejia Xu, and Zhangyang Wang. Nerf-sos: Any-view self-supervised object segmentation from complex real-world scenes. arXiv e-prints, pp. arXiv-2209, 2022.
Michael Grupp. evo: Python package for the evaluation of odometry and slam. //github. com Mi /tt//Michhhha http:/iihhaelGGrppeeeeee 2017.
https:
Qiao Gu, Zhaoyang Lv, Duncan Frost, Simon Green, Julian Straub, and Chris Sweeney. Egolifter: Open-world 3d segmentation for egocentric perception. arXiv preprint arXiv:2403.18111, 2024.
Jun Guo, Xiaojian Ma. Yue Fan, Huaping Liu, and Qing Li. Semantic gaussians: Open-vocabulary scene understanding with 3d gaussian splatting, 2024.
Huy Ha and Shuran Song. Semantic abstraction: Open-worid 3D scene understanding from 2D vision-language models. Proceedings of the 2022 Conference on Robot Learning, 2022.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. arXiv:2111.06377 2021.
Ajay Jain, Ben Mildenhall, Jonathan T. Barron, Pieter Abbecl, and Ben Poole. Zero-shot text-guided object generation with dream fields. 2022.
Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, and George Drettakis. 3d gaussian splat- ting for real-time radiance field rendering. ACM Transactions on Graphics, 42(4), July 2023. URL httts::///epppp.. sam. inria. wttfffggrff//fffffm htwpa//--aa..iiia.....aaaaaaaaaa wwwps//dddggssuaaaeeeeeeess inn- an- spl splatting/.
Justin Kerr, Chung Min Kim, Ken Goldberg, Angjoo Kanazawa, and Matthew Tancik. Lerf: Language embedded radiance fields. In International Conference on Computer Vision (ICCV), 2023.
Diederik Kingma, Auto-encoding variational bayes. arXiv preprint arXiv: 2013.
11

---

Published as a conference paper at ICLR 2025
Sosuke Kobayashi, Eiichi Matsumoto, and Vincent Sitzmann. Decomposing nerf for editing via feature field distillation. In Advances in Neural Information Processing Systems, volume 35, 2022. URL pettss:/arrin
org/ wttp://////...../22222
pdf.
Jiahe Li, Jiawei Zhang, Xiao Bai, Jin Zheng, Xin Ning, Jun Zhou, and Lin Gu. Dngaussian: Op- timizing sparse-view 3d gaussian radiance fields with global-local depth normalization. arXiv preprint arXiv:2403.06912 2024.
Lu Ling, Yichen Sheng, Zhi Tu, Wentian Zhao, Cheng Xin, Kun Wan, Lantao Yu, Qianyu Guo, Zixun Yu, Yawen Lu, et al. DI3dv-10k: A large-scale scene dataset for deep learning- basg 3d vision. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 22160-22169, 2024.
Xi Liu, Chaoyi Zhou, and Siyu Huang. 3dgs-enhancer: Enhancing unbounded 3d gaussian splatting with view-consistent 2d diffusion priors. In Advances in Neural Information Processing Systems (NeurIPS), 2024.
Gal Metzer, Elad Richardson, Or Patashnik, Raja Giryes, and Daniel Cohen-Or. Latent-nerf for shape-guided generation of 3d shapes and textures. arXiv preprint arXiv:2211.07600, 2022.
Oscar Michel, Roi Bar-On, Richard Liu, Sagie Benaim, and Rana Hanocka. arXiv preprint arXiv:2112.0322// 2021.
Ben Mildenhall, Pratul P. Srinivasan, Rodrigo Ortiz-Cayon, Nima Khademi Kalantari, Ravi Ra- mamoorthi, Ren Ng, and Abhishek Kar. Local light field fusion: Practical view synthesis with prescriptive sampling guidelines. ACM Transactions on Graphics (TOG), 2019.
Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes neural radiance fields for view synthesis. In ECCV, 2020.
Jacho Moon, Juan Luis Gonzalez Bello, Byeongjun Kwon, and Munchurl Kim. From-ground to-objects: Coarse-to-fine self- cufl-super monocular depth estimation of dynamic objects with ground contact prior. arXiv preprint 23Xiv:2312.10118 2023.
Jangho Park, Gihyun Kwon, and Jong Chul Ye. Ed-nerf: Efficient text-guided editing of 3d scene using latent space nerf. arXiv preprint arXiv:2310.02712, 2023.
Luigi Piccinelli, Yung- Hsu Yang. Christos Sakaridis, Mattia Segu, Siyuan Li, Luc Van Gool, and Fisher Yu. UniDepth: Universal monocular metric depth estimation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), 2024.
Ben Poole, Ajay Jain, Jonathan T. Barron, and Ben Mildenhall. Dreamfusion: Text-to-3d using 2d diffusion. arXiv, 2022.
Minghan Qin, Wanhua Li, Jiawei Zhou, Haoqian Wang, and Hanspeter Pfister. Langsplat: 3d lan- guage gaussian splatting. arXiv preprint arXiv:2312. 16084, 2023.
Ravi Ramamoorthi and Pat Hanrahan. An efficient representation for irradiance environment maps. In Proceedings of the 28th annual conference on Computer graphics and interactive techniques, pp. 497-500, 2001.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models, 2021.
Johannes Lutz Schönberger and Jan-Michael Frahm. Structure-frood fromee-foooooooo momtmn revisited. In Confer- ence on Computer Vision and Pattern Recognition (CVPR), 2016.
Jin-Chuan Shi, Miao Wang, Hao-Bin Duan, and Shao-Hua Guan. Language embedded 3d gaussians for open-vocabulary scene understanding. arXiv preprint 0rXiv:2311.18A4 18482, 2023.
Yawar Siddiqui, Lorenzo Porzi, Samuel Rota Bulò, Norman Müller, Matthias Nießner, Angela Dai, and Peter Kontschieder. Panoptic lifting for 3d scene understanding with neural fields. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp- 9043-9052, June 2023.
12

---

Published as a conference paper at ICLR 2025
Gabriela Ben Melech Stan, Diana Wofk, Scottie Fox, Alex Redden, Will Saxton, Jean Yu, Estelle Aflalo, Shao- Yen Tseng, Fabio Nonato, Matthias Muller, et al. Ldm3d: Latent diffusion model for 3d. arXiv preprint arXi::2305. 2023.
Boyuan Sun, Yuqi Yang, Le Zhang, Ming-Ming Cheng, and Qibin Hou. Corrmatch: Label propaga- tion via correlation matching for semi-supervised semantic segmentation. IEEE Computer Vision and Pattern Recognition (CVPR), 2024.
Jiaxiang Tang, Jiawei Ren, Hang Zhou, Ziwei Liu, and Gang Zeng. Dreamgaussian: Generative gaussian splatting for efficient 3d content creation. arXiv preprint 2tti:2300.1666550 2023.
Peng Wang, Shijie Wang, Junyang Lin, Shuai Bai, Xiaohuan Zhou, Jingren Zhou, Xinggang Wang, and Chang Zhou. One-peace: Exploring one general representation model toward unlimited modalities. arXiv preprint arXiv:2305. 112 2023.
Rundi Wu, Ben Mildenhall, Philipp Henzler, Keunhong Park, Ruiqi Gao, Daniel Watson, Pratul P. Srinivasan, Dor Verbin, Jonathan T. Barron, Ben Poole, and Aleksander Holynski. Reconfusion: 3d reconstruction with diffusion priors. arXiv, 2023.
Xianggang Yu, Mutian Xu, Yidan Zhang, Haolin Liu, Chongjie Ye, Yushuang Wu, Zizheng Yan, Tianyou Liang, Guanying Chen, Shuguang Cui, and Xiaoguang Han. Mvimgnet: A large-scale dataset of multi-view images. In CVPR, 2023.
Zehao Yu, Anpei Chen, Binbin Huang, Torsten Sattler, and Andreas Geiger. Mip-splatting: Alias- free 3d gaussian splatting. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp. 19447-19456, June 2024.
Yuanwen Yue, Anurag Das, Francis Engelmann, Siyu Tang, and Jan Eric Lenssen. Improving 2D Feature Representations by 3D-Aware Fine-Tuning. In European Conference on Computer Vision (ECCV), 2024.
Shuaifeng Zhi, Tristan Laidlow, Stefan Leutenegger, and Andrew J. Davison. In-place scene la- belling and understanding with implicit scene representation. 2021.
Shijie Zhou, Haoran Chang, Sicheng Jiang, Zhiwen Fan, Zehao Zhu, Dejia Xu, Pradyumna Chari, Suya You, Zhangyang Wang, and Achuta Kadambi. Feature 3dgs: Supercharging 3d gaussian splatting to enable distilled feature fields. In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition, pp. 11676221655, 2024a.
Shijie Zhou, Zhiwen Fan, Dejia Xu, Haoran Chang, Pradyumna Chari, Suya Bharadwaj. Tejas You, Zhangyang Wang, and Achuta Kadambi. Dreamscene360: Unconstrained text-to-3d scene gen- eration with panoramic gaussian splatting. arXiv preprint arXiv:2404.06903 2024b.
Matthias Zwicker, Hanspeter Pfister, Jeroen Van Baar, and Markus Gross. Surface splatting. In
Proceedings of the 28th annual conference on Computer graphics and interactive techniques, pp- 371-378, 2001.
13

---

Published as a conference paper at ICLR 2025
A APPENDIX
A.1 DETAILS OF CALCULATING THE WEIGHT
To compute Azj. first calculate the Absolute Pose Error (APE) for each pose pair using the for- mula: Eij P,- P.. where P. and Py are the different camera poses respectively. After obtaining Eig, the APE is calculated as:
APE,j = Eijj - Taxalle;
(12)
where 1xx is the identity matrix F and represents the Frobenius norm. In each iteration, the APE values normalized across all image pairs to derive the weights Xıj- Xij E. APEg where represents each image pair within one iterations. This normalization ensures they reflect the relative contributions of each pose error in a consistent manner. This method is implemented based on the APE computation approach in the evo library (Grupp, 2017).
APE;;
A.2 DETAILS DATASET
We create a correspondence pair dataset based on the training set of DL3DV-105 (Ling et al., 2024) dataset fine-tune our VAE encoder. We randomly sample 784 scenes and extract correspondence pairs from the multi-view images by using COLMAP. The correspondence points for each scene will be pre-computed before the model fine-tuning process. We use sequential matcher with the number of overlapping images set to 10 and the number of quadratic overlaps set to 1. Such over- lapping searching strategy ensures our model not only learns from easy and dense correspondence, but also from challenging cases among far-view image pairs, adding great robutness for our model. The ability to remain consistency in large view difference particularly necessary for the outdoor unbounded reconstruction. Moreover, we set the minimum number of inliers and minimum ratio of inliers 15 and 0.25 with the loop detection to make sure the extracted correspondence is accurate enough. We also train the same number of latent 3D Gaussian splatting scenes from the DL3DV- 10K datasets to create paired dataset of images and rendered latents, which are used for Stage-III decoder fine-tuning.
A.3 IMPLEMENTATION DETAILS
For Stage-I, employ the pre-trained VAE model (f 8, KL), from LDM model zoo as the backbone VAE model. We fine-tune the VAE on NVIDIA A A100-80GB GPUs for around one day, by using the correspondence pair dataset with image resolution of 512x512, the base learning rate of 4.5e 06, and the default optimizer. For Stage-III, we fine-tune the decoder on the image-latent dataset with NVIDIA A A100-80GB GPUs for around one day.
In the implementation of LRF, we normalize the latent input to the radiance field using the scale of all input views stabilize radiance field optimization, and apply denormalization during rendering. During the VAE encoding stage, we start the discriminator at step 501 for better image quality, and we set K Lwiight 1.0 10 and Dweight 0.5. For the decoder training, we use the same configuration as the original VAE, except K Teeight = 0 to ensure only the decoder was optimized.
A.4 IMAGE RECONSTRUCTION PERFORMANCE
To verify that our approach does not degrade the performance on downstream tasks, we evaluate the image reconstruction performance of our fine-tuned VAE by calculating PSNR between the original images and the reconstructed images. As shown in Table 4, adding the correspondence consistency constraint to inject 3D awareness and applying a regularization loss to keep the latent space close to the original latent space perform minimal impact on the VAE's reconstruction performance. This sures that our VAE model can still be effectively used in conjunction with other pre-trained models, such the Stable Diffusion model, without any fine-tuning.
14

---

Published as a conference paper at ICLR 2025
Table Evaluation of PSNR for images reconstructed by VAEs on NeRF-LLFF, DL3DV-10K, and Mip-NeRF360 datasets.
Method
Metric
NeRF-LLFF
DL3DV-10K
Mip-NeRF360
VAE
PSNR 1 PSNR 1
23.47 23.59
24.59 23.25
24.54
Our-VAE
24.24
A.5 MORE IMAGE GENERATION RESULTS
Fig. 7 demonstrates that VAE model generate 3D objects guided by text prompts without any fine-tuning of the diffusion model. Moreover, Fig. 8 shows that our VAE can also improve the GSGEN (Chen et al., 2024) to achieve better 3D generations with complicated text prompts.
A.6 EFFICIENCY ANALYSIS
Table demonstrates that our method reduces input resolutions, model storage space, and GPU usage for photorealistic NVS, which particularly useful cases with limited communication bandwidth and storage. For instance, some individuals may not have GPUs with large memories, where our method is an efficient solution for them to run photorealistic NVS algorithms.
Table Efficiency comparison of different image- space and latent-space NVS methods
Method
resolution
Training Time
GPU
Storage
Rendering FPS
Decoding FPS
PSNR
SSIM
LPIPS
512x572
GB
mia
3DGS VAE
64x64
4.8 min
GB
25097 MB
20.57
0.595
0.346
Latem-NaRF
642664
27.2 min
10 GB
390.50 MR
0.09
18:16
0.530
0.422
Ours
61x61
3.9 min
GB
96.42 MB
180
20
22.45
0.667
0.197
A.7 MORE EXPERIMENTAL RESULTS
To demonstrate the effectiveness and generalizability of our method for 3D latent reconstruction, we show more NVS and generation results on four datasets covering indoor scenes, outdoor scenes, and object-level scenes. As shown in Fig. 9, 10, 11 and 12, method yields significant improvement image quality.
15

---

Published as a conference paper at ICLR 2025
Ours
Dreamfusion
Ours
Dreamfusion
Ours
Latent-NeRF
An ice cream
A temple
lego man
A small saguaro cactus planted in clay pot
hamburger
stack of pancakes cov- ered in maple syrup
Figure 7: Samples for text-to-3D generation the image and latent space.
Ours
GSGEN
Ours
GSGEN
Ours
GSGEN
A DSLR photo of a tray of sushi zoomed DSLR photo of containing pugs
zoomed DSLR photo of a plate of fried chicken and waffles
cake in shape of a train
Figure 8: More samples for text-to-3D generation on the image space.
16

---

Published as a conference paper at ICLR 2025
Groundtrath
3DUS-VAE
Mip Splatting
Feature GS
3DGS
A
Figure 9: More NVS results on the DL3DV-10K dataset.
17

---

Published as a conference paper at ICLR 2025
Groundtruth
Ours
3DGS VAE
Mip Splalting
Feature-GS
3DGS
Figure 10: More NVS results on the NeRF-LLFF dataset.
18

---

Published as a conference paper at ICLR 2025
Groandrruth
Ours
3DGS-VAL VAF
Mip-Splatting
Feature-GS
3DGS
Ur
Figure 11: More NVS results the Mip-NeRF360 dataset.
19

---

Published as a conference paper at ICLR 2025
Groundrruth
Ouurs
BDGG VAR
Mii-SSllating
Feature- GS
3DGS
Figure 12: More NVS results on the MVImgNet dataset.
20


Reviews:
[{"review_id": "Jj91xpvaZ7", "reviewer": "Reviewer_duK5", "summary": "This submission tries to resolve the problem of 3D reconstruction in the latent space via a 3-stage idea. The first stage focuses on improving the 3D awareness of the VAE's encoder via a correspondence-aware constraint on the latent space; the second stage builds a latent radiance field (LRF) to represent 3D scenes from the 3D-aware 2D representations; the last stage further introduces a VAE-Radiance Field (VAE-RF) alignment method to boost the reconstruction performance. The results generated by this pipeline out-perform the results generated by many state-of-the-arts.", "questions": "I'm willing to raise my score if questions below is answered:\n\n1. How is \\lambda_{ij} computed in equation (6), section 4.1? Basically, how to compute the average pose error and how it contributes to the weight \\lambda_{ij}?\n2. Since equation (6) becomes multi-objective optimization, does this change largely increase the training convergence time? Did you experience any convergence issues?\n3. How is the inference speed of this pipeline?\n4. The idea is kind of similar to CaesarNeRF: Calibrated Semantic Representation for Few-shot Generalizable Neural Rendering: https://haidongz-usc.github.io/project/caesarnerf, which also uses calibrated image feature in each 2d latent, could you cite and compare?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "There are many innovations in this work, but I think the best part is the introduction of the 3d awareness into the 2D representation training. In this part, especially the correspondence aware autoencoding is the key to the success of this overall idea.", "weaknesses": "There are still some weaknesses prevented me from giving a higher score, especially, the details of how to compute each component of the pipeline. Please see my questions below. In addition, some related references are missing.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "HsHF9JRlrD", "reviewer": "Reviewer_nTxk", "summary": "The author introduces pixel-to-pixel correspondences across different viewpoints to help the VAE learn a 2D latent space with 3D awareness. Using 3D Gaussian Splatting (3DGS), they perform 3D reconstruction and rendering in the latent space to obtain 2D latent representations from specified camera poses. The rendered results are then decoded back into image space by the decoder to obtain RGB images.\n\nExperimental results demonstrate that the resulting 2D latent space possesses a certain level of 3D perception capability and outperforms existing methods when decoding to higher-resolution images.", "questions": "- Could the author please provide further clarification on the motivation and applicable scenarios for this work? \n- In the experimental section, it would be beneficial to employ fairer comparison methods; using low-resolution training for comparison models is not advisable. \n- Please consider using more recent models to compare the text-to-3D generation capabilities of this work.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The author is committed to integrating 3D awareness into the 2D latent space, and the results show a significant degree of success in this endeavor. Additionally, using 3D Gaussian Splatting (3DGS) in modeling the latent space is an intriguing idea.", "weaknesses": "The motivation of this paper is somewhat unclear. Is the author aiming to improve reconstruction accuracy, enhance rendering speed, reduce storage space, or achieve some other application? It appears that none of these goals have been fully addressed.\n\n**Reconstruction Accuracy**: When training the comparison methods, the author down-scaled the RGB images to the same resolution as the latent representation before training, which may be considered unfair. The VAE used by the author has been exposed to high-resolution images, while the comparison methods have not. This discrepancy could reduce the reconstruction performance of the comparison methods and impact the paper's credibility.\n\n**Rendering Speed**: In Section 5.1, the author reports the training times for Stage 1 and Stage 3, but not for Stage 2 or for inference time. Therefore, it is challenging to conclude that the proposed method has a faster rendering or training speed compared to other methods.\n\n**Storage Space Reduction**: In Section 5.1, the author mentions the need to \"train the same number of latent 3D Gaussian splatting scenes... for Stage-III,\" indicating that Stage 2 is scene-specific. Compared to 3DGS, this does not seem to save much storage space.\n\n**Other Applications**: In Section 5.3, the author suggests that their work can be used for text-to-3D generation. However, the two methods used for comparison are relatively outdated. It is recommended to compare the method with more recent approaches, such as IPDREAMER [1], to make the claim more convincing.\n\n[1] Zeng, Bohan, et al. \"Ipdreamer: Appearance-controllable 3d object generation with image prompts.\" arXiv preprint arXiv:2310.05375 (2023).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "MIY9RH8BAN", "reviewer": "Reviewer_RdaN", "summary": "This paper introduces a framework for constructing radiance field representations in latent space, aiming to bridge the domain gap between 2D feature space and 3D representations. The authors propose a three-stage pipeline: (1) a correspondence-aware autoencoding method that enforces 3D consistency in latent space through correspondence constraints, (2) a latent radiance field (LRF) that lifts these 3D-aware 2D representations into 3D space, and (3) a VAE-Radiance Field alignment strategy that improves image decoding from rendered 2D representations.\n\nThe key technical contribution is the integration of 3D awareness into 2D representation learning without requiring additional per-scene refinement modules. The authors adapt the 3D Gaussian Splatting framework to operate in latent space, using spherical harmonics to model view-dependent effects. They demonstrate their method's effectiveness on both novel view synthesis and text-to-3D generation tasks across various datasets including MVImgNet, NeRF-LLFF, MipNeRF360, and DL3DV-10K. The authors claim their approach is the first to achieve photorealistic 3D reconstruction performance directly from latent representations while maintaining cross-dataset generalizability.\n\nThe work represents an attempt to make latent 3D reconstruction more practical by addressing the geometric consistency issues in existing approaches. The framework is designed to be compatible with existing novel view synthesis and 3D generation pipelines without requiring additional fine-tuning.", "questions": "- How to handle the scale mismatch between COLMAP and latent space? Please clarify: 1) the exact mapping strategy from pixel to latent correspondences; 2) how multiple pixel correspondences within one latent cell are aggregated\n- What's the correspondence filtering pipeline? In particular: 1) thresholds used for COLMAP matching 2) any additional filtering criteria in latent space 3) how outlier correspondences are handled.\n- During VAE-RF alignment (Section 4.3), how to: 1) balance the training/novel view losses 2) prevent overfitting during decoder fine-tuning.\n- Regarding the resolution setup, why choose this specific resolution comparison protocol? Would the method's advantages hold at equal resolutions?", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "- The paper follows a standard pipeline structure addressing latent space 3D reconstruction. The method section breaks down into three components: correspondence-aware encoding, latent radiance field construction, and VAE alignment. The ablation study provides basic validation of these components, though more comprehensive analysis would be beneficial.\n- While building heavily on existing techniques, the paper demonstrates competent engineering in combining different elements into a working system. The adaptation of correspondence constraints and 3DGS to latent space shows reasonable technical implementation. The provided implementation details outline the basic approach.\n- The evaluation includes tests on multiple datasets (MVImgNet, NeRF-LLFF, MipNeRF360, DL3DV-10K), attempting to demonstrate applicability across different scenarios. While the cross-dataset evaluation has limitations, it provides basic evidence of generalization capability. The inclusion of both novel view synthesis and text-to-3D generation shows the method's potential utility, though more thorough evaluations are needed.\n- The method functions without per-scene refinement modules, which could be advantageous compared to some previous approaches.", "weaknesses": "- The paper fails to provide compelling justification for operating in latent space. While previous works like Latent-NeRF (for text-to-3D generation) established initial groundwork, this paper does not clearly demonstrate additional benefits of its approach. The motivation for operating in latent space remains questionable. The paper shows modest improvements in PSNR/SSIM metrics but doesn't address fundamental questions: What are the computational advantages over image-space methods? How does memory consumption compare? Why is the added complexity of latent space operations justified? The authors should conduct a thorough efficiency analysis, measuring training time, inference speed, and memory usage against image-space baselines. Without such evidence, the practical value of the latent space approach is hard to justify.\n- Section 4.1's correspondence mechanism has several fundamental issues. Most critically, the paper fails to address the scale mismatch between COLMAP's pixel-level correspondences and the VAE's latent space. Given that the VAE operates at a lower resolution (likely 8x or 16x downsampled) with larger receptive fields, how are pixel-level correspondences meaningfully mapped to latent features? This mapping is non-trivial: a single latent code typically corresponds to a large receptive field in pixel space, making precise correspondence matching questionable. The paper should answer: How are multiple pixel correspondences within one latent cell handled? How does the receptive field size affect correspondence accuracy? Additionally, basic details are missing: the correspondence filtering criteria, quality metrics, and robustness to matching errors. The use of L1 distance for latent features (Eq. 6) needs justification, especially given the coarse nature of latent correspondences. These technical gaps raise serious concerns about the method's fundamental soundness.\n- The use of spherical harmonics in latent space (Eq. 8) is puzzling. Given that the features are already in a learned latent space, why introduce SH basis functions? A direct learnable decoder or simpler view-dependent representation might suffice. Similarly, the VAE-RF alignment stage seems unnecessarily complex - the authors may quantify the alleged distribution shift and explore simpler alternatives. These design choices add complexity without clear benefits.\n- The experimental setup has a fundamental flaw: image-space methods are handicapped by low-resolution inputs while the proposed method has access to high-resolution data. This creates an artificial advantage for the proposed method. A fair comparison requires either: testing at matched resolutions, or demonstrating specific benefits under computational constraints. The ablation study skips crucial experiments on correspondence quality, loss function components, and architectural variations. These missing comparisons make it difficult to assess the true value of each component.\n- The paper sidesteps important practical concerns. Where are the failure cases? How does the method handle challenging scenes with varying illumination or complex geometry? The text-to-3D generation results lack comparisons with current state-of-the-art methods. The claim of \"photorealistic reconstruction\" needs validation through proper user studies or established perceptual metrics. Testing on more diverse, challenging scenarios would better demonstrate real-world applicability.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "uDSIPpLhjx", "reviewer": "Reviewer_Cwtt", "summary": "This paper target at Latent 3D reconstruction, and address the domain gap between 2D feature space and 3D representations. They proposed propose a novel framework that comprise (1) a correspondence-aware autoencoding method, (2) a latent radiance field (LRF), and (3) a VAE-Radiance Field (VAE-RF) alignment strategy.", "questions": "I think the strict view-inconsistency can not be solved fundamentally due to the emplyment of VAE Decoder. But it coule be possible to showcase more view consistent results and evlaute the 3D consistency of proposeld methods. \n\nThe author provide more details about the experiment settings, especially the view sampling strategy and the number of views used in the experiment. I hope see whether the proposed method can be applied to a sparse-view setting.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "Strengths*:\nWith the proposeld framework, this paper enhances the 3D consistency of 2D latent representations as well as effectively mitigated the gap between the 2D latent space and the natural 3D space.", "weaknesses": "Weaknesses*:\n\n1. Compared with feature-GS, it is a good improvement to add a correspondence-aware constraint during VAE encoder finetuning to improve its 3D awareness. However, this approach still cannot guarantee strict multi-view consistency of the encoded multiview features. As a result, after constructing the LRF, there may be a blurred radiance field with significant detail losses. Although the LRF is 3D consistent, the final decoded features may still exhibit noticeable flickering effects due to the lack of view consistency of decoder.\n\n\n2. I think this paper still needs optimization during the LRF stage instead of using a total feed forward method. When compared with 3DGS and Mip-Splatting, the authors only train them in latent space resolution (8 times lower than the image resolution), which yields very bad visual results. I suggest that the authors consider training the competing methods at full resolution. This paper may not work better than Mip-Splatting when trained with full resolution and very dense view, but it would be interesting to see if it outperforms Mip-Splatting in a sparse-view setting. This might be achieved because of generation capability of Stable Diffusion VAE used in this paper.\n\n3. The author didn't provide a detailed explanation of experiment settings. For example, I wonder how views are sampled in the experiment during training. And how many views are used as input and evaluation repectively. This is important for me to fully evaluate this paper. \n\n4. Object change in final rendering. As shown in Fig 1, the building in the final rendering is different from the ground truth image. One is in red, the other one is in white. This lead to my concern of identity preserving capability of purposed method. I think this is a problem that needs to be addressed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)