RefinedFields: Radiance Fields Refinement for Planar Scene Representations
Authors: Karim Kassab
Antoine Schnepf
Jean-Yves Franceschi
Laurent Caraffa
Jeremie Mary
Valerie Gouet-Brunet
Venue: Accepted by TMLR ()

Abstract: Planar scene representations have recently witnessed increased interests for modeling scenes from images, as their lightweight planar structure enables compatibility with image-based models. Notably, K-Planes have gained particular attention as they extend planar scene representations to support in-the-wild scenes, in addition to object-level scenes. However, their visual quality has recently lagged behind that of state-of-the-art techniques. To reduce this gap, we propose RefinedFields, a method that leverages pre-trained networks to refine K-Planes scene representations via optimization guidance using an alternating training procedure. We carry out extensive experiments and verify the merit of our method on synthetic data and real tourism photo collections. RefinedFields enhances rendered scenes with richer details and improves upon its base representation on the task of novel view synthesis. Our project page can be found at https://refinedfields.github.io .

Published in Transactions on Machine Learning Research (05/2025)
RefinedFields: Radiance Fields Refinement for Planar Scene Representations
Karim Kassab
k.kasab@critee.com com
Criteo AT Lab, Paris, Fronce LASTIG, Universilé Gustave Biffel, IGN-ENSG, F-94160 Saint-Mande
Antoine Schnopf
a. a.schmepf@eriteo..om com
Criteo AI Lab, Paris, Prance Université Côte d' d'Azur, CNRS, 18S, France
Jean- Yves Franceschi
bycjo..ranceschii@errrcc.ocm
Criteo Al Lab, Paris, Prance
Laurent Caraffa
leurent.caraffioBign..f
LASTIG, Université Gustave Biffel, IGN-ENSG, F-94160 Saint- Mandé
Jeremie Mary
j.mar@@critoo.omm
Criteo Al Lab, Poris, Prance
Valerio Gouet-Brunct
galeric.gouctBign..
LASTIG, Université Gustave Eiffel, IGN-ENSG, F-94160
Reviewed on OpenReview: https: openrevieew. net/ pet/foone forum? S6JpSSSS
Abstract
Planar scene representations have recently witnessed increased interests for modeling scenes from images, as their lightweight planar structure enables compatibility with image-based models. Notably, K-Planes have gained particular attention as they extend planar scene representations to support in-the-wild scenes, in addition to object-level scenes. However, their visual quality has recently lagged behind that of state-of-the-art techniques. To reduce this gap, we propose RefinedFields, a method that leverages pre-trained networks to refine K-Planes scene representations via optimization guidance using an alternating training pro- codure. We carry out extensive experiments and verify the merit of our method on syuthetic data and real lourism photo collections. RefinedFields enhances rendered scenes with richer details and improves upon its base representation on the task of novel view synthesis. Our project page can be found itttp::/rreiinefffi htw.eeeeeeeddddddiiiiiib........ github.io.
1 Introduction
To draw novel objects and views, humans often rely on blend of cognition and intuition, where the latter is built large prior acquired from long-term continuous exploration of the visual world. Nevertheless, ablating one of these two elements results in catastrophic representations. On the one hand, humans find it particularly difficult to draw a bicycle based solely on this preconceived prior (Gimini, 2016). However, once one photograph is observed, drawing novel views of a bicycle becomes straightforward. On the other hand, drawing monuments and complex objects based solely on observed images, and with no preconceived notions of geometry and physics, is also non-trivial. In computer vision, recent methods tackling object generation and novel view synthesis generally focus either the former the latter ablation.

---

Published in Transactions on Machine Learning Research (05/2025)
PONT
XII PONT. MAX MENS XII PONT E-
MALYMIIAATE COMMENDAT
AALYRITATE COMMENNDT
VIRGINEM
DOMINI MOCCEXAN
ORNAFIT
DONINI MOCCAXN
OR ORWANI
KEDICU EDEICE
ES VS X
(a) RefinedFields Rendering
(b) K-Plancs
(c) RefinedFields
(d) GT
Figure 1: Qualitative Results. Given images of the Trevi fountain from Phototourism (Jin et al., 2020), as well as a pre-trained model (Rombach et al., 2022), our method leverages the pre-trained model and refines K-Planes with finer details that are under-represented when optimizing the same K-Planes on the images
alone.
The first class of methods approaches novel view synthesis by learning scenes through rigorous cognition, as in dense observations of captured images. Although classic methodologies like structure-from-mooion (Hart- ley & Zisserman, 2004) and image- mmsge-based rendering (Shum et al., 2008) have previously tackled this problem, the field has recently seen substantial advancements thanks to stereo reconstruction techniques (Wang et al., 2024), as well as implicit neural representation ion methods (Mildenhall et al., 2020) which this paper focuses on. Particularly, planar scene representations such as Tri-Plancs (Chan et al., 2022) have recently gained significant attention (Shue et al., 2023; Lan et al., 2024) for their lightweight planar structure that allows for seamless integration with image- based models. Recent methods extend NeRFs (Martin (Martin-Brualla et al., 2021) and Tri-Planes (Fridovich-Keil et al., 2023, K-Planes) to support learning from unconstrained "in- the-wild" photo collections by enabling robustness against illumination variations and transient occluders. These representations however, do not learn any prior across scenes they are trained from scratch for each scene. This means that these representations are learned in a closed-world setting, where the information scope is limited to the training set at hand. Despite the wide interest in planar scene representations, their visual quality currently falls short behind recent state-of-the-ar methods (Kulhanek et al., 2024; Xu et al., 2024). Our goal is to integrate pre-trained networks into the training framework of K-Planes, with an aim to improve their rendering performance.
The second class of methods tackles novel view synthesis and object generation by learning and leveraging priors over images and scenes, reminiscent of drawing from insights and intuition. These methods have also recently witnessed accelerated advancements. Recent works leverage pre-trained networks to achieve Object Generation (OG) (Poole et al., 2023; Metzer et al., 2023) and Novel View Syuthesis (NVS) (Yu et al., 2021; Jain et al., 2021; Liu et al., 2023; Kyriaziy al., 2023). Particularly, Liu et al. (2023) achieve NVS from single images only by simply fine-tuning a pro-trained latent diffusion model (Rombach et al., 2022). This proves pivotal significance related to large-scale pre-trained vision models, as it shows that, although trained 2D data, these models learn a rich geometric 3D prior about the visual world. Nevertheless, as these pre-trained models alone have no explicit multi-view goometric constraints, their use for 3D applications is usually prone to geometric issues (e.g. geometric inconsistencies, multi-face Janus problem, content drift issues (Shi et al., 2023, Figure 1)). This class of methods has not yet. been explored to enhance implicit models representing in-the-wild scenes, as leveraging priors over such representations not evident.
RefinedFields proposal. Our work builds on the previous discourse and aims to enhance planar scene modeling by leveraging pre-trained networks. To this end, we present alternating training algorithm that
2

---

Published in Transactions on Machine Learning Research (05/2025)
integrates novel scene refining stage, which aims to propose a better optimization initialization by utilizing pre-trained model. We adopt K-Planes (Fridovich-Keil et al., 2023) as base scene representation, which extend Tri-Planes to support in-the-wild scenes, in addition to object-level scenes. Liu et al. (2024) highlight that features in planar scene representations resemble projected scone images, a finding we also corroborate in Appendix B, and exploit to improve upon K-Planes. Specifically, RefinedFields refines planar scene representations by projecting them onto the space of representations inferable by a pre-trained network, which pushes K-Planes features more closely resemble real-world images. To do so, we build on the seamless integration of K-Planes with image-based based models and present an alternating training procedure that iteratively switches between optimizing K-Planes representation on images from particular dataset, and fine-tuning a pre-trained network to output a new conditioning leading to a refined version of this K- Planes representation. Overall, this procedure guides the optimization of a particular scene, by leveraging not only the training dataset at hand but also the rich prior lying within the weights of the pre-trained model, which first for in-the-wild scene modeling.
We conduct extensive quantitative and qualitative evaluations of RefinedFields. We show that our method improves upon K-Planes with richer details in scene renderings. We prove via ablation studies that this added value indeed comes from the finc-tuned prior of the pre-trained network. Figure 1 illustrates the improvements our method showcases on the Trevi fountain scene from Phototourism (Jin et al., 2020).
2 Related Work
RefinedFields achieves geometrically consistent novel view synthesis, that can also be applied in-the-wild, by leveraging K-Planes, and a large-scale pre-trained network. Our method is the first to satisfy all of these attributes, as summarized in Table 1. In this section, we develop the various preceding works from which our method takes inspiration.
Neural representations. Neural rendering (Tewari et al., 2020) has seen significant advancements since the introduction of Neural Radiance Fields (Mildenhall et al., 2020, NeRF). At its core, NeRF learns a scene by fitting the weights of neural network posed images of said scene. This subsequently enables the reconstruction of the scene thanks to volume rendering (Kajiya & Herzen, 1984). Subsequent to NeRFs, various scene representations have been introduced (Chen et al., 2022a; Müller et al., 2022; Kerbl et al., 2023). Particularly, Chan et al. (2022) introduce Tri-Planes, a planar scene representation serving as a middle ground between implicit and explicit representations, enabling a faster learning of scenes. Tri-Planes have been widely adopted in recent works (Shuc et al., 2023; Lan et al., 2024; Wang et al., 2023b), as their planar structure enables a seamless integration with image-based models.
Neural representations in-the-wild. Subsequent to NeRFs, several techniques emerged to extend the NeRF setup to "in-the-wild" unconstrained photo collections (Jin al., 2020, Phototourism) plagued by illumination variations and transient occluders. This added variability makes learning a scene particularly challenging, surfaces can exhibit significant visual disparities across views. NeRF-W MMrrtin-rruii et al., 2021) addresses the challenge of novel view synthesis in-the-wild (NVS-W) by modeling scene lighting through appearance embeddings, and transient occluders through an additional transient head. Splatfacto-V W (Xu et al., 2024) proposes a Nerfstudio (Tancik et al., 2023) implementation of 3D Gaussian Splatting (Kerbl et al., 2023, 3DGS) that extends standard 3DGS to support in-the-wil wild scene modeling via appearance embeddings. WildGaussians (Kulhanek et al., 2024) also proposes using 3DGS with appearance embeddings for in-the-wild scene modeling, but additionally predicts uncertainty masks to exclude transient occluders from the loss computation. It is important to note that although this work adopts a pre-trained network, it is solely used for the computation of the loss masks the data, and not a prior on the scene representation. This approach is orthogonal to our approach of leveraging a prior to directly improve upon specific scene representations, as it acts on the data and not the scene representation itself. Kridovich-Keil el al. (2023) present K-Planes, which modify and extend Tri-Planes (Chan et al., 2022) to in-the-wild scenes thanks to Icarnable appearance embeddings, similarly to Martin-Brualla et al. (2021). Our work aims to improve upon K-Planes representations by extending their training beyond closod-world setups using pre-trained networks. Note that other works also tackle NVS-W: Chen et al. (2022b) model scene lighting through
3

---

Published in Transactions on Machine Learning Research (05/2025)
Table 1: Related work overview. RefinedFields leverages a pre-trained prior (Rombach et al., 2022) on the scene representation to refine K-Planes, our underlying scene representation, utilized for novel view synthesis in the wild (NVS-W). NNo exhaustive, other works with characteristics similar to NeRF-W exist. Refer to Section for details.
In-the-wild scene modeling
Underlying repmsentation
Geometric consistency
No 3D supervision
Pre-trained prior
Task
NFD (Shne et al., 2023)
x
X
Tri- Tri-Planes
3DGen (Gupta et al., 2023)
x
X
Tri-Planes
OG
Latent-NeRF (Metzer et al., 2023)
/
X
NeRF
DrcamFusion (Poole et al., 2023)
/
X
NeRF
NoRF (Mildenhall ct al., 2020)
X
X
NcRF
RealFusion (Melas-Kyriazi et al., 2023)
/
X
NcRF
DiffusioNeRRF (Wynn & Turrukhhmmbeov, 2023)
X
/
X
NeRF
NerIDiff (Gu et al., 2023)
/
X
Tri-Planes
NVS
3DMM (Watson et al., 2023)
/
X
Zero-1-to-3 (Liu ct al., 2023)
X
X
NeRF-W (Martin- Brualla al., 2021)
X
V
NeRF
WildCaussians (Kulhanck et al., 2024)
X /
7
3DGS
Splatfacto-W (Xu et al., 2024)
X 5 7
3DGS
NVS-Wi
K-Planes (Fridovich-Keil et al., 2023)
X 7
K-Planes
RefinedFields (ours)
5
K-Planes
appearance embeddings and transient occluders through transient embeddings, Yang al. (2023) leverage interactive information across rays mimic the perception of humans in-the-wild, and Chen et al. (2024) separate static and transient components by utilizing cauristics-guider segmentation. However, these works limit their training and evaluations to downscaled versions of Phototourism, which makes their results not directly comparable with ours.
Priors in neural representations. The integration of pre-trained priors for downstream tasks has emerged as a prominent trend, as they enable the effective incorporation of extrinsic knowledge into di- verse applications. For neural representations, priors have been utilized for few-shot scene modeling (Yu et al., 2021; Jain et al., 2021; Deng et al., 2023; Gao et al., 2024) as well as generative tasks (Shue et al., 2023; Poole et al., 2023; Watson et al., 2023; Gao et al., 2024) for object generation and novel view syn- thesis. In this realm, denoising diffusion probabilistic models (Ho et al., 2020; Rombach et al., 2022) have recently gained particular attention for their application as plug-and aagg-and-layy priors (Graikos et al., 2022). They have been utilized in various domains such as super-resolution (Wang et al., 2023a) and more specifically novel view synthesis. Liu al. (2023) fine-tune Stable Diffusion (Rombach et al., 2022), pre-trained la- tent diffusion model for 2D images, to learn camera. controls over 3D dataset and thus performing NVS by generalizing to other objects. These results hold paramount value, they highlight the rich 3D prior Icarned by Stable Diffusion, even though it has only been trained on 2D images. This however comes with geometric inconsistency issues across views, as a pre-trained model alone has no explicit multi-view geo- metric constraints. In this work, we aim to leverage a pre-trained model to enhance a volumetric scene ropresentation, which inherently adheres to consistency constraints, hence mitigating multi-view consistency issues. The utilization of such priors enhance in-the-wit wild scene representation has been an unexplored area of research.
4

---

Published in Transactions on Machine Learning Research (05/2025)
Scene Fitting
repeat times
(iny) MLP
Volume rendering
Catting
11
P P
P,BxNNNxx
Ra
I
I
relting
i x
Scene Refining
repeat N2 limes
U--Net
reshage
Dw
N0,1
n
P,UxNNXX
LoRAa
Figure 2: Scene learning procedure. The K-Plancs Py, the MLP with trainable paramcters a, and the appearance embeddings are learned during scene fitting. The LoRA parameters e well as the decoder Du are learned during scene refining. The pre-trained U-Net is frozen. Assets in violet and underlined are intermediate results. At each iteration, new planes P. are inferred and assigned to Py, which are then corrected by scene fitting.
3 Method
To guide the optimization of planar scene representations with extrinsic signals, we learn a scene through two altornating stages, as illustrated in Figure 2. Scene fitting optimizes our K-Planes representation P, to reproduce the images the training set, as traditionally done in neural rendering techniques. Scene refining finetunes pre-trained network to this K-Planes representation, and then infers a new one Pe, which will subsequently be corrected by scene fitting. The main idea behind this that we use our 3D implicit model P- for optimizing the scene to the available information in the training set and adhere to essential geometric constraints, and then project this scene representation on the set of scenes inferable by the pre-trained network, making it closer to natural images. In this section, we detail each stage and elucidate the intuition behind our method.
3.1
Scene Fitting
The goal at this stage is to fit scene, adhering to pre-defined geometric constraints, from posed RGB images. To fit the scene, we adopt the K-Planes representation Srridovich-Keil et al., 2023). As such, this stage corresponds to optimizing K-Planes to fit a scene, from which we adapt the code.
K-Planes are compact 3D model representations applicable static scenes, "in-the-wild" scenes (scenes with varying appearances), and dynamic soenes. These models allow for fast training and rendering, while maintaining low-memory usage. K-Planes model d-dimensional scene with k (2) planes, which represent the combinations of every pair of dimensions. This structure makes K-Planes compatible with multitude of neural network architectures, and more particularly inage-specialized network architectures. This enables K-Planes inference by minimally tweaking image architectures. For static 3D scene, At = and the planes represent the xy, æz, and yz planes. These planes, each of size N x N x C, encapsulate features representing the density and view-dependent colors of the scene.
The K-Planes model P, originally randomly initialized. The first goal of scene fitting then correct this random initialization to fit the training set.. Note that the first iteration of scene fitting is especially
5

---

Published in Transactions on Machine Learning Research (05/2025)
particular, since it is starting with randomly initialized scene, opposed to proposed scene, describe in Section 3.2.
To render the 3D scene from K-Planes, as done by Mildenhall et al. (2020) and Fridovich-Keil et al. (2023),
we cast rays from the desired camera position through the coordinate space of the scene, on which we sample 3D points. We decode the corresponding RGB color for each 3D point q (i, by normalizing [0, N)
and projecting it onto the k 3 planes, denoted as Pl P
(aa)
(m=) p(a)),
f (q) = (P(()) (1) (q))
(1)
where h E H {ay, 22, (q) projects q onto pM and &b denotes bilincar interpolation on regular 2D grid.
These features are then aggregated using the Hadamard product to produce single feature vector of size M:
f(g) II (q) GCH
(2)
To decode these features, we adopt the hybrid formulation K-Planes (Fridovich- Keil et al., 2023). Two small Multi-Layer Perceptrons (MLPs), go and gRGB, map the aggregated features as follows:
(q)), f(g) - g. ()(())))
c(q. d) gRGB (f(g).
where (p) (sin(2"xp), cps(2"Tp),.... sin(22-1Tp)) "io Tp), cos(24-1 xp)) is the positional embedding of p. Jo maps the K-Planes features into density a and additional features f. Subsequently, JRGB maps f and the posttionally-encodeed view directions 7(d) into view-dependent RGB colors. This enforces densities to be independent of view directions.
These decoded RGB colors are then used to render the final image thanks to ray marching and integrals from classical volume rendering (Kajiya & Herzen, 1984), that are practically estimated using quadrature:
N
((r)
Σ 71(1 - exp(-o,6))cc
i=1
(4)
Σ 05,0
T. = exp
where (r) is the expected color, T; is the accumulated transmillance along the ray, and 6, = 4441 1, is the distance between adjacent samples.
3.2 Scene Refining
This section presents the core of our method, which consists of proposing better optimization initializations for the scene fitting stage. Given a fitted scene representation P,, this stage consists learning this fitted implicit representation and proposing a new refined representation P.. Formally, this stage consists of projecting our K-Planes P., on the set Q of K-Planes inferable by low-rank fine-tuning of the pre-trained model:
P = arg min [P, PP12
(5)
PEQ
As K-Plance feature channels show similar structure to images (Appendix B), this projection pushes the K-Planes to be even more similar in structure to real images, more particularly to orthogonal projections of the scene on the planes. Figures 5 to 7 illustrate a comparison between feature planes at the end of our optimization and those of standard K-Planes. RefinedFields leads to feature planes exhibiting sharper details, which ultimately lead more refined details in scene renderings, as proven by our experiments (Figure 3 and Table 3).

---

Published in Transactions on Machine Learning Research (05/2025)
Algorithm Alternating training algorithm.
Input: Nopochs N1. N2, N, C, n, c, I (Tey, Cs), Ra, Dw: SD de optimizer
2: f standard-gaussinn
n,.))
3: Py f -aaaaarrrdguuusnnnMM
4: for Nopochs steps do
5: // scene fitting
for N1 steps do
7, o opiimizeetsttep(Cnciin (P:::
end for
// refoning
10:
for N₂ steps do
11:
Ps + Dw
12:
w, Ø - tet-opmmeerrsteppCeeeieeee (P Py.,
13: end for
14: P. P.
15: end for
To provide scene refining with a rich prior, we employ a large-scale pre-trained latent diffusion model, as these networks exhibit great performances as priors for downstream tasks, and share similar properties to our planar representation, both in terms of shape and distribution (Appendix B). More particularly, we adopt Stable Diffusion (Rombach et al., 2022, SD) for its proven performances for downstream 3D (Liu et al., 2023) and 2D (Wang et al., 2023a) tasks. Thus, we integrate the U-Net SD, and the decoder D into our pipeline, and treat the K-Planes 3C-channel N N images. We also replace the last layer of the decoder D. with randomly initialized convolutional layer (with no bias), to take into account the shape of the K-Planes. We then finc-tune the pro-trained model using the fitted K-Plancs P,,, and infer refined K-Planes P D (TD((r)) where x ~ NV, I) is sampled once at the beginning of the training, Nole that this is different from the multi-step generation process of diffusion model inference, as we only apply the inference at the last time-step of our diffusion model. This key as our goal here is not learn distributions over scenes and sample them for generation, but to adapt the pre-trained network and leverage the information already learned within its weights to infer K-Planes closest to representing the scene at hand.
To achieve the fine-tuning of our pre-trained network, significant challenge presents itself: due to the sheer size of Stable Diffusion, it would be too costly to fine-tune all of its trainable parameters. Moreover, as we only want to modulate priors embedded into the pre-trained network, we look for an alternative to doing full fine-tuning. To circumvent these constraints, we adopt Low-Rank Adaptation (Hu et al., 2022, LoRA), a simple yet effective parameter-effcient fine-tuning method that has proven great transfer capabilities across modalities and tasks (Fan et al., 2023; Lee et al., 2023; Zeng & Lee, 2023). LoRA's relatively minimal design works directly weight tensors, which means that it can be seamlessly applied to most model architectures. Furthermore, LoRA does not add any additional cost at inference, thanks to its structural re- parameterization design. To achieve this, Hu et al. (2022) inject trainable low-rank decomposition matrices into each layer of a frozen pre-trained model. Let Wo, bo be the frozen pre-trained weights and biases, and x be the input. Fino-tuning a frozen lincar layer f(x) = Wa + by comes down to learning the low-rank decomposition weights AW =BA:
f(x) - (Wo AW)x w
(6)
where Wo. AW E dddk B B Rdxr; A E Rexk; and the rank r & min(d,
Thus, to implement scene refining, we fine-tune the LoRA parameters é modulating the pre-trained U-Net, as well as the decoder's parameters w, on the fitted scene P,, Subsequently, we query the U-Net with Gaussian noise x, decode intermediary output latent with D and infer Pa that is reshaped into a new refined scene P.. Finally, P. is proposed to scene fitting as an improved initialization to be optimized. For in-depth inspection of the feature planes, we refer the reader to Appendix B.

---

Published in Transactions on Machine Learning Research (05/2025)
3.3 Training
We define alternating training procedure rotaling between scene fitting and scene refining, as described above, and as illustrated in Figure 2.
For scene fitting, we train the K-Planes model as proposed by Fridovich-Keil et al. (2023). We use spatial total variation regularization to encourage smooth gradients. This is applied over all the spatial dimensions of each plane in the representation:
Crv(P) = C
E(Ip' - P | -P))-113)
For scenes with varying lighting conditions (e.g. in-the-wild scenes as in the Phototourism dataset (Jin et al., 2020)), an M-dimensional appearance vector e, is additionally optimized for each image. This vector is then passed as input to the ML. color decoder gRGB at the rendering step Ra. Hence, the training objective for scene fitting is written as:
min Littingg 9.7
A [R - - Icll2 - ATVLTv(Px), -
(8)
where Ra represents the K-Planes rendering procedure (i.e. ray marching, feature decoding via a small MLP with trainable parameters a, and volume rendering), P., are the K-Planes with trainable parameters yy and Ic is a ground truth RGB image with camera position C.
As for the scene refining phase, we optimize the decoder parameters w as well as the LoRA parameters o, modulating the frozen U-Net weights, on the fitted scene Py, Thus, the fine fine-tuning objective for scene refining is written as:
min Lrefiing wid
D (DD (z)( - PP
(9)
where x (), is fixed during scene refining, SD is the frozen Stable Diffusion model modulated by LoRA with trainable parameters Φ, and D the latent K-Planes decoder. After this optimization, P is reassigned (SD, and passed to scene fitting. Note that, thanks the alternating nature of our training and the absence of bias in the decoder's convolutional layers, this optimization does not overfit the model to produce exactly P.,, which is key as it would lead to resuming scene fitting from exactly the same point.
At the end of the alternating training procedure, we save the refined and corrected representation P, for rendering and testing. We refer the reader to Algorithm 1 for an overview of our training procedure.
4 Experiments
We start by assessing RefinedFields via an experiment on a case study. We then evaluate RefinedFields on synthetic scenes (Mildenhall al., 2020) and real-world Phototourism (Jin et al., 2020) scenes, where we showcase the improvements our method exhibits relative to our K-Planes base representation. Quantitative results can be found in Tables 2 and 3, where report for cach experiment the Pcak Signal-to-Noise Ratio (PSNR) for pixel-level similarity, the Structural Similarity Index Measure (SSIM) for structural-level similarity, and the Learned Perceptual Image Patch Similarity (Zhang et al., 2018, LPIPS) for perceptual similarity. RefinedFields demonstrates an improved performance compared to K-Plancs on the task novel view synthesis. For a further look, experimental details including hyperparameters and more dataset details can be found Appendix A. Additional qualitative results on synthetic and real-world scenes are available in Appendix C.
4.1 Datasets
We evaluate our method similarly to prior work (Mildenhall et al., 2020; Martin-Brualaa et al., 2021: Fridovich-Ken et al., 2023) in novel view synthesis, by adopting the Real Synthetic 360 dataset (Mildenhall et al., 2020) for synthetic scenes and the same three scenes of cultural monuments from the Phototourism
8

---

Published in Transactions on Machine Learning Research (05/2025)
RefinedFields optimization
GT
K-Planes
Figure 3: Case study. Qualitative results on the Lego scene from the NeRF synthetic dataset (Mildenhall et al., 2020) showcasing the optimization progression on RefinedFields, and a comparison with the ground truth and K-Planes. The training set constrained to 50% of its initial size for both RefinedFields and K-Planes. RefinedFields refines the K-Planes representation enabling the proper reconstruction details in the scene. At the end of optimization, the Mcan Squared Error (MSE) for RefinedFields 3.46 10-4. while the one for K-Planes is 4.36 10-1
Table 2: Quantitative results. Results on static synthetic scenes (Mildenhall et al., 2020). The bold and underlined entries respectively indicate the best and second-best results. Dashes denote values that were not reported in prior work. Our method outperforms K-Planes, our main baseline, on the task of novel view synthesis for synthetic scenes.
PSNR (f)
Chair
Drums
Ficus
Hotdog
Lego
Materials
Mic
Ship
Mean
NeRF (Mildenhall al., 2020)
33.00
30.13
36.18
31.00
TensoRP (Chen et al., 2022a)
35.76
26.01
33.99
37.41
36.46
30.12
34.61
30.77
33.14
Plenoxela (Fridovich-Keel Keil al., 2022)
33.98
25.35
31.83
36.43
34.10
29.14
33.26
29.62
31.71
INGP (Miller et al., 2022)
35.00
26.02
33.51
37.40
36.39
29.78
36.22
31.10
33.18
K-Planes (Pridovich-Keil et sl., 2023)
34.98
25.68
31.4 44
36.75
35.81
29.48
34.10
30.76
32.37
K-Planes-SS (Fridovich-Ker) Keil et al., 2023)
33.61
25.27
30.92
35.88
35.09
28.83
33.01
30.04
31.58
RefinedFields (ours)
35.77
25.94
32.45
37.08
36.47
29.39
34.77
31.41
32.91
SSIM
Chair
Drums
Ficus
Hotdog
Lego
Materials
Mic
Ship
Moan
NeRF (Mildenhall et al., 2020)
0.967
0.925
0.964
0.974
0.961
0.949
0.980
0.856
0.947
TensoRF (Chen et al., 2022a)
0.985
0.937
0.982
0.982
0.983
0.952
0.988
0.895
0.963
Plenoxels (Fridovich-Keell al., 2022)
0.977
0.933
0.976
0.980
0.975
0.949
0.985
0.890
0.958
INGP (Miller et al., 2022)
K-Planes (Fridovich-Keil et sl., 2023)
0.983
0.938
0.975
0.982
0.982
0.950
0.988
0.897
0.962
K-Planes-SS (Fridovich-Kan Keil et al., 2023)
0.974
01.932
0.971
0.977
0.978
0.943
0.983
0.887
0.956
RefineFFeelds (ours)
0.985
0.937
0.980
0.981
0.984
0.945
0.989
0.903
0.963
dataset (Jin et al., 2020) for real-world scenes: tional dataset details can be found in Appendix A.1.
Brandenburg Gate, Sacré Coeur, and Trevi Fountain. Addi-
4.2 Implementation Details
For fair comparison, we take similar experimental settings in scene fitting Fridovich-Keil et al. (2023). However, due to the nature of our scene refining pipeline, we limit the implementation in our case single- scale K-Planes of 512 512 resolution, in contrast to the multi-scale approach taken by Reiivvich-Kiil et al.
9

---

Published in Transactions on Machine Learning Research (05/2025)
(2023) where N (64, 128, 256, 512). The number of channels in each plane remains the same (C 32). Moreover, throughout all the experiments, we consider the hybrid implementation of K-Planes, where plane features are decoded into colors and densities by small MLP. As for the scene refining pipeline, we apply no modification to the U-Net in Stable Diffusion. Yet, we replace the last layer of the decoder Du with new convolutional layer (without bias) to account for the shape of the K-Planes. Hence, the dimensions used in For an in-depth look at our frameworks
scene refining (Figure 2) are: N = 512, C 32, n = 64, and and hyperparameter settings, we refer the reader to Appendices A.2 and A.3.
4.3 Evaluations
Baselines. For both synthetic and in-the-wild scenes, we primarily compare our method to our K-Plancs baseline, highlighting the improvements it brings to planar scene representations. Note that, for a fair comparison, and to assess the added value of our refining pipeline with respect to our base representation (Section 4.2), we also include single-scale ablation of K-Planes with N = 512 (dubbed K-Planes-SS). In order to provide a more comprehensive perspective, we also illustrate the NVS performances of other recent works that do not employ planar scene representations for synthetic (Mildenhall et al., 2020; Chen et al., 2022a; Fridovich-Keit et al., 2022; Müller et al., 2022) and in-the-wild (Mildenhall et al., 2020; Martin-Brualle et al., 2021; Xu et al., 2024; Kulhanek el al., 2024) scenes.
Comparisons. We start by testing our method against K-Planes on a case study consisting of the Lego scene from the NeRF synthetic dataset. Here, we train both methods on half of the training set as to deliberately produce a lower-quality fitted scene. As illustrated in Figure 3 our method refines K-Planes and exhibits better quantitative and qualitative results thanks to our scene refining pipeline.
We then apply our method to learn synthetic and in-the-wild scenes. In this case, RefinedFields improves upon not only K-Planes-SS but also K-Planes on the task of NVS (Tables and 3). As a result, enhances the performance of planar scene representations and brings them closer to recent works, which highlights the value of scene refining. Particularly, for synthetic scenes, RefinedFields improves upon K-Planes and sometimes even outperforms other state-of-the-art methods. For in-the-wild scenes, RefinedFields also improves K- Planes performances in notable way. However, recent state-of-the-art methods based on Gaussian Splatting architectures (Kerbl et al., 2023) continue lo outperform the refined planar representations in terms of overall quality. Figures and 4 show qualitative comparisons of RefinedFields with K-Planes, showing the visual improvements brought by our refining pipeline, which brings finer details to monuments in the Phototourism scenes. Further qualitative results on synthetic and in-the- wild scenes can be found in Appendix C. We also present an inspection of K-Planes features learned by our method and K-Planes in Appendix B.
It is important to note that we do not compare our NVS metrics with other recent works (Chen et al., 2022b; Yang et al., 2023; Zhang et al., 2025). This is because these methods are trained and evaluated on downscaled versions of the Phototourism dataset, which reduces the prominence of fine details in the ground truth for monument structures. As their NVS metrics are computed relative to these ground truth, their results are not comparable to ours. Although other works do compare NVS metrics of various methods across different resolutions, we refrain from this practice as it is not an accurate comparison.
As presented, RefinedFields utilizes an alternating training procedure and pre-trained prior to refine scene representations, leading to richer details in rendered images. While our method demonstrates promising results, this however comes with training time increase as compared to our base representation, as our K-Planes feature projection via alternating training leads to the repeated fine-tuning of both scene fitting and scene refining pipelines, which takes overall around 80 hours on a single NVIDIA A100 GPU. We leave the optimization of training time for future work.
4.4 Ablations
To justify our choices and explore further, we compare our in-the-wild results (Table 3) to results from two main ablations of our method. RofinedFieldsnoFFnnee innttunnn is a variation of our method without LoRA fino-tuning. Here, we consider the same exact pipeline (frozen U-Net, same decoder configuration). except that we don't 't modulate the weights of the frozen U-Net with LoRA. This means that the prior is kept intact
10

---

Published in Transactions on Machine Learning Research (05/2025)
Table 3: Quantitative results. Results on three real-world datasets from Phototourism (Jin et al., 2020).
Our method shows notable improvements compared K-Plancs on the task of NVS-W.
Resuuls from public implementation (Aoi, 2022) reproduced by Fridovich-Kie et al. (2023).
Brandenburg Gate
Sacré Coeur
Trevi Fountain
PSNR (*)
SSIM (1)
LPISS (4)
PSNR (1)
SSIM (1)
LPIPS (4)
PSNR (t)
SSIM (1)
LPIPS (4)
NeRF
18.90
0.5159
0:231
13.60
0.7155
0.291
16.14
0.6007
0.366
NeIt--W
19.17
18.61
Splatfacto-V W
26.87
0.9320
0.124
22.53
0.8760
0.158
22,66
0.7690
0.221
WildCaussins
27.77
0.9270
0.133
22.58
0.8550)
0.177
23.63
0. 0.7680
0.228
K-Planes
25. 19
0.8785
0.221
20.61
0.7735
0.265
22.67
0. 0.7199
0.317
K-Plancs-S8
0.8622
0.242
19.86
0.312
21.30
0.8627
0.355
Redined ields-ed Finettuin
25.39
0.8834
0.206
21.41
0.8058
0.239
22.54
0. 0324
0.291
MeanedPields-oPPPP
25.42
0.8822
0.214
21.17
0.7978
0.248
22, 16
0.7251
0.291
RelinedFields (ours)
26.64
0.8869
0:206
22.26
0.8116
0.228
23.42
0. 0.79999
0.284
K-Planes
RefinedFiie iee
GT
NEAW
Brandenburg Gate
Sacré Cocur
Trevi Fountain
Figure 4: Qualitative results. Results three scenes from Phototourism (Jin et al., 2020). Our method refines K-Planes and leads richer and finer details in scene renderings.
and fine-tuning is done. This is to assess the role that LoRA finetuning of the pre-trained prior plays in our pipeline. Note that the decoder is still fine-tuned in this ablation, additionally freezing it would lead to random constant optimization initialization Pe. which would defeat the purpose of the refining stage. RefinedFields-noPrion ablates the prior of Stable Diffusion by randomly re-initializing all U-Net weights while leaving all other elements of the scene-refining pipeline intact. This ablation done to evaluate the importance of the prior, and to verify that the observed refinements are not entirely resulting from alternate training. Note that ablating the entire scene refining pipeline leads back the K-Planes-SS setting. As illustrated in Table 3, consistently obtaín worse results during the ablation study compared to full model, thus demonstrating the value of the pre-trained prior and of LoRA finetuning. This proves the importance of extending scene learning beyond closed-worl settings, as our K-Planes projection ion is done via an optimization conditioning coming from a large image prior that a reasonably sized training set cannot fally capture, especially in-the-wild.
5 Conclusion
In this paper, introduce RefinedFields, method that refines K-Planes representations by using pre- trained prior and an alternate training procodure. Extensive experiments show that RefinedFields exhibits notable improvements on the task of novel view senth compared to its K-Planes baseline. In concluding this study, several avenues of future work emerge as we consider this work to be a first sepping-stons in improving planar scene representations via conditioning with extrinsic signals. This includes the exploration of approaches to achieve this conditioning other than optimization guidance, and the application of scene refining on other representations.
11

---

Published in Transactions on Machine Learning Research (05/2025)
Impact Statement
This paper presents work that enhances the construction of high-quality neural representations. As such, the risks associated with our work parallel those of other neural rendering papers. This includes but is not limited to privacy and security concerns, as our method is trained on a dataset of publicly captured images, where privacy-sensitin
information (e.g. human faces, license plate numbers) could be present. Hence, similarly to other neural rendering approaches, there a risk that such data could end up in the trained model if the employed datasets are not properly filtered before use. Furthermore, Diffusion as prior, it inherits any problematic biases and limitations this model may have.
work utilizes Stable
Acknowledgments
This work granted accoss to the HPC resources of IDRIS under the allocation 2023-AD011014261 made by GENCI We thank Loic Landrieu, Vicky Kalogeiton and Thibaut Issenhuth for inspiring discussions and valuable feedback.
References
Al Aoi. nerf pl. http:://ttttbcccce////eee///tttp
2022. Accessed: 2023-10-25.
Fric R. Chan, Connor 7. Lin, Matthew A. Chan, Koki Nagano, Boxiao Pan, Shalini De Mello, Orazio Gallo, Leonidas J. Guibas, Jonathan Tremblay, Sameh Khamis, Tero Karras, and Gordon Wetzstein. Efficient Geometry-Aware 3D Generative Adversarial Networks. In Proceedings of the IEEE/CVP Conference on Computer Vision and Pattern Recognition (CVPR), pp. 16123 -11133, June 2022.
Anpei Chen, Zexiang Xu, Andreas Geiger, Jingyi Yu, and Hao Su. TensoRF: Tensorial Radiance Fields. In European Conference on Computer Vision (ECCV), 2022a.
Jiahao Chen, Yipeng Qin, Lingjee Liu, Jiangbo Lu, and Guanbin Li. NeRF-HuGS: Improved Neural Radiance Fields in Non-static Scenes Using Heuristics-Guided Segmentation. In Proceedings of the JEFE/CW /CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 19436 19446, June 2024.
Xingyu Chen, Qi Zhang, Xiaoyu Li, Yue Chen, Ying Feng, Xuan Wang, and Jue Wang. Hallucinated Neural Radiance Fields in the Wild. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pallern Recognition (CVPR), pp. 22943-12953 June 2022b.
Congyue Deng, Chiyu Max Jiang, Charles R. Qi, Xinchen Yan, Yin Zhou, Leonidas Guibas, and Dragomir Anguelov. Nerdi: Single-view nerf synthesis with language-guided diffusion general image priors. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 20637 2064-,00777 June 2023.
Ying Fan, Olivia Watkins, Yuqing Du, Hao Liu, Moonkyung Ryu, Craig Boutilier, Pieter Abbeel, Mohammad Ghavanzadeh, Kangwook Lee, and Kimin Lee. Dpok: Reinforcement learning for fine-tuning text-to-image diffusion models. arXiv preprint arXiv:2305. 16381, 2023.
Sara Fridovich-Ked Keil, Alex Yu, Matthew Tancik, Qinhong Chen, Benjamin Recht, and Angjoo Kanazawa. Plenoxcls: Radiance Fields Without Neural Networks. In Proceedings of the IEEE/CVP Conference Computer Vision and Pallern Recognition (CVPR), pp. 5501 5510, June 2022.
Sara Fridovich-Keil, Giacomo Meanti, Frederík Rahbæek Warburg, Benjamin Recht, and Angjoo Kanazawa. K-Planes: Explicit Radiance Fields in Space, Time, and Appearance. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp. 12479-12488, June 2023.
Ruiqi Gao, Aleksander Holynski, Philipp Henzler, Arthur Brussee, Ricardo Martin Brualla, Pratul P. Srini- vasan, Jonathan T. Barron, and Ben Poole. CAT3d: Create anything in 3d with multi-view diffusion models. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. URL https://openrevier wttp:/ww.ee//fuoeeeeeeeee net//forummit-T721119.....
12

---

Published in Transactions on Machine Learning Research (05/2025)
Gianluca Gimini. Velocipedia. Accessed: 2023-10-25.
http::/www...g...........aaaaaaa 2016.
Alexandros Graikos, Nikolay Malkin, Nebojsa Jojic, and Dimitris Samaras. Diffusion Models Plug-and- Play Priors. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (ods.), Advances in Newral Information Processing Systems, volume 35, pp.
Curran Associates, Inc., 2022.
Jiatao Gu, Alex Trevithick, Kai-En Lin, Josh Susskind, Christian Theobalt, Lingjie Liu, and Ravi Ra- mamoorthi. NerfDiff: Singlo-image View Synthosis with NeRF-guided Distillation from 3D-aware Diffu- sion. In International Conference on Machine Leurning, 2023.
Anchit Gupla, Wenhan Xiong, Yixin Nie, Ian Jones, and Barlas Oguz. 3dgen: Triplane latent diffusion for textured mesh generation. arXiv preprint arXiv:2909.0537 2023.
Richard Hartley and Andrew Zisserman. Multiple View Geometry in Computer Vision. Cambridge University Press, 2 edition, 2004. doi: 10.1017/CB097705118110
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising Diffusion Probabilistic Models. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 6840 6851. Curran Associates, Inc., 2020.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-Rank Adaptation of Large Language Models. In International Conference Learning Representations, 2022.
Ajay Jain, Matthew Tancik, and Picter Abbeel. Putting nerf on diet: Semantically consistent few-shot view synthesis. In Proceedings of the IEEE/CVF International Conference on Compuler Vision (ICCV), pp. 5885 5894, October 2021.
Yuhe Jin, Dmytro Mishkin, Anastasiia Mishchuk, Jiri Matas, Pascal Fua, Kwang Moo Yi, and Eduard Trulls. Image Matching Across Wide Baselines: From Paper to Practice. International Journal of Compuler Vision, (29(2):517-)) 547, oct 2020. doi: 10.1007/s11263--
James Kajiya and Brian Von Herzen. Ray tracing volume densities. Proceedings of the 11th annaual conference on Computer graphics and interactive techniques, 1984.
Bernhard Kerbl, Georgios Kopanas, Thomas Leimküüler, and George Drettakis. 3D Gaussian Splatting for Real-Time Radiance Field Rendering. ACM Transactions Graphics, 42(4), July 2023.
Jonas Kulhanek, Songyou Peng, Zuzana Kukelova, Mare Pollefeys, and Torsten Sattler. WildGaussians: 3D Gaussian Splatting In the Wild. In The Thirty-eighth Anrual Conference Neural Information Processing Systems, 2024.
Yushi Lan, Fangzhou Hong, Shuai Yang, Shangchen Zhou, Xuyi Meng, Bo Dai, Xingang Pan, and Chen Change Loy. LN3Diff: Scalable Latent Neural Fields Diffusion for Speedy 3D Generation. In ECCV, 2024.
Ariel N Lee, Cole J Hunter, and Nataniel Ruiz. Platypus: Quick, cheap, and powerful refinement of llms. arXiv preprint Xrx:22230 07317, 2023.
Ruoshi Liu, Rundi Wu, Basile Van Hoorick, Pavel Tokmakov, Sergey Zakharov, and Carl Vondrick. Zero-l- to-3: Zero-shot One Image to 3D Object. In Proceedings of the IEEE/CVE International Conference on Computer Vision (ICCV), pp. 9298 9309, October 2023.
Ying-Tiae Tian Liu, Yuan-Chen Guo, Guan Luo, Heyi Sun, Wei Yin, and Song-Hai Zhang. PI3D: Efficient Text-to-3D Generation with Pseudo-Image Diffusion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp- 19914,19924, June 2024.
13

---

Published in Transactions on Machine Learning Research (05/2025)
Ricardo Martin-Bruale Noha Radwan, Mchdi S. M. Sajjadi, Jonathan T. Barron, Alexey Dosovitskiy, and Daniel Duckworth. NeRF in the Wild: Neural Radiance Fields for Unconstrained Photo Collections. Tn Proceedings of the LEEE/CVE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 7210-7219, June 2021.
Luke Melas-Kyriazii Iro Laina, Christian Rupprecht, and Andrea Vedaldi. RealFusion: 360deg Reconstruc- tion of Any Object From a Single Image. In Proceedings of the IEEE/CVF Conference on Compuler Vision and Pattern Recognition (CVPR), pp. 8446 8455, June 2023.
Gal Metzer, Elad Richardson, Or Patashnik, Raja Giryes, and Daniel Cohen-Or. Latent-NeRF for Shape- Guided Generation of 3D Shapes and Textures. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 12663 12663-12673, June 2023.
Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng. NeRF: Representing Scenes Neural Radiance Fields for View Synthesis. In ECCV, 2020.
Thomas Müller. tiny-cuda-nn, 4 2021. URL attpp:///giiie htttp:///gtthhh..c///aaaaattt
Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller. Instant Neural Graphics Primitives with a Multircsolution Hash Encoding. ACM Trans. Graph., 11044:10221111::15, July 2022. doi: 10.1145/ 3528223.3530121,
Adam Pasuke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. PyTorch: An Imperative Style, High-Performacee Deep Learning Library. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d' Alché-Buc, E. Fox, and R. Garnett (eds.), Advances in Neurul Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
Ben Poole, Ajay Jain, Jonathan T. Barron, and Ben Mildenhall. DreamFusion: Text-to-3D using 2D Diffusion. In The Eleventh International Conference on Learning Representations, 2023.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. Hish-Resolttion Image Synthesis With Latent Diffusion Models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognátion (CVPR), pp. 10684-10695, June 2022.
Johannes Lutz Schönberger and Jan-Michael Frahm. Structure-from-mooss revisited. In Conference Compater Vision and Pallern Recognition (CVPR), 2016.
Yichun Shi, Peng Wang, Jianglong Ye, Mai Long, Kejie Li, and Xiao Yang. Mvdream: Multi-view diffusion for 3d generation, 2023.
J. Ryan Shue, Eric Ryan Chan, Ryan Po, Zachary Ankner, Jiajun Wu, and Gordon Wetzstein. 3D Neural Field Generation Using Triplanc Diffusion. In Proceedings of the IEEE/CT Conference Computer Vision and Pallern Recognilion (CVPR), pp. 20875 20886, June 2023.
H.Y. Shum, S.C. Chan, and S.B. Kang. Image-Basse Rendering. Springer US 2008. ISBN 9780387326689.
Matthew Tancik, Ethan Weber, Evonne Ng, Ruilong Li, Brent Yi, Justin Kerr, Terrance Wang, Alexan- der Kristoffersen, Jake Austin, Kanyar Salahi, Abhik Ahuja, David McAllister, and Angjoo Kanazawa. Nerfstudio: A Modular Framework for Neural Radiance Field Development. In ACM SIGGRAPH 2023 Conference Proceedings, SIGGRAPH '23, 2023.
Ayush Tewari, Ohad Fried, Justus Thies, Vincent. Sitzmann, Stephen Lombardi, Kalyan Sunkavalli, Ricardo Martin- Martin-Brualla, Tomas Simon, Jason Saragih, Matthias Nießner, Rohit Pandey, Sean Fanello, Gordon Wetzstein, Jun-Yan Zhu, Christian Thcobalt, Mancesh Agrawala, Eli Shechtman, Dan B. Goldman, and Michael Zollhöfer. State of the Art on Neural Rendering. Computer Graphics Forum (Eurographics '20): State of the Art Reports, 39(2):701 727, May 2020.
14

---

Published in Transactions on Machine Learning Research (05/2025)
Patrick von Platen, Suraj Patil, Anton Lozhkov, Pedro Cuenca, Nathan Lambert, Kashif Rasul, Mishig Davaadorj, and Thomas Wolf. Diffusers: State-of-the-arl diffusion models, 2022. Accessed: 2023-10-25.
Jianyi Wang, Zongsheng Yue, Shangchen Zhou, Kelvin CK Chan, and Chen Change Loy. Exploiting Diffusion Prior for Real-World Image Super-Resohttin. In arXiv preprint or vrXim23355000111 2023a.
Shuzhe Wang, Vincent, Leroy, Yohann Cabon, Boris Chidlovskii, and Jerome Revaud. Dust3r: Geometric
3d vision made easy. Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 20697-20709, June 2024.
Tengfei Wang, Bo Zhang, Ting Zhang, Shuyang Gu, Jianmin Bao, Tadas Baltrusaitis, Jingjing Shen, Dong Chen, Fang Wen, Qifeng Chen, and Baining Guo. RODIN: A Generalive Model for Sculpting 3D Digital Avatars Using Diffusion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 4563 4573, June 2023b.
Daniel Watson, William Chan, Ricardo Martin Brualla, Jonathan Ho, Andrea Tagliasacchi, and Mohammad Norouzi. Novel View Synthesis with Diffusion Models. In The Eleventh International Conference on Learning Representations, 2023.
Jamie Wynn and Daniyar Turmukhambetov. DiffusioNeRP:: Regularizing Neural Radiance Fields With Denoising Diffusion Models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 4180 4189, June 2023.
Congrong Xu, Justín Kerr, and Angjoo Kanazawa. Splatfacto-w: A nerfstudio implementation of gaussian splatting for unconstrained photo collections. Xiv preprint arXiv:2407. 12306, 2024.
Yifan Yang, Shuhai Zhang, Zixiong Huang, Yubing Zhang, and Mingkui Tan. Cross- Neural Radiance Fields for Novel- View Synthesis from Unconstrained Image Collections. In Proceedings of the IEEE/CVP International Conference on Computer Vision (ICCV), pp- 15901-15911 October 2023.
Alex Yu, Vickie Ye, Matthew Tancik, and Angjoo Kanazawa. pixelnerf: Neural radiance fields from one or few images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognation (CVPR), pp. 4578-4587 June 2021.
Yuchen Zeng and Kangwook Lee. The expressive power of low-rank adaptation. arXiv preprint arXiv:2310. 17513, 2023.
Dongbin Zhang, Chuming Wang, Weitao Wang, Peihao Li, Minghan Qin, and Haoqian Wang. Gaussian in the wild: 3d gaussian splatting for unconstrained image collections. In European Conference Compuler Vision, pp. 341-359. Springer, 2025.
Richard Zhang, Phillip Isola, Alexei A. Efros, Eli Shechtman, and Oliver Wang. The Unreasonable Effee-- tiveness of Deep Features as Perceptual Metric. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition (CVPR), June 2018.
15

---

Published in Transactions on Machine Learning Research (05/2025)
A Experimental Details
A.1 Datasets
Synthetic datasct. For synthetic renderings, we adopt the Real Synthetic 360 datasct from NeRF (Mildenhall et. al., 2020). This dataset consists of eight. path-traced scenes containing objects exhibiting complicated geometry and realistic non-Lamberiinn materials. Each image is coupled with its corresponding camera paramcters. Consistently with prior work, 100 images are used for training cach scene and 200 images are used for testing. All images at 800 800 pixels.
In-the-wild dataset. For in-the-wild rendcrings, we adopt the Phototourism dataset (Jin et al., 2020) which is commonly used for in-the-wild tasks. This dataset consists of multitudes of images of touristic landmarks gathered from the internet. Thus, these images are naturally plagued by visual discrepancies, notably illumination variation and transient occluders. Camera parameters are estimated using COLMAP (Schönberger Frahm, 2016). All images are normalized to [0,1]. We adopt three scenes from Phototurism: Brandenburg Gale (1363 images); Sacré Coeur (1179 images), and Trevi Fountain (3191 images). Testing is done on standard set that is free of transient occluders.
A.2 Frameworks
We make use of multiple frameworks to implement our method. Our Python source code (tested on version 3.7.16), based on PyTorch (Paszke al., 2019) (tested on version 1.13.1) and CUDA (tested on version 11.6), is publicly available as open source. We also utilize Diffusers (von Platen et al., 2022) and Stable Diffusion (tested version 1-5, main revision). K-Plancs also adopt the tingcudann framework (Miller, 2021). We run all experiments on single NVIDIA A100 GPU.
A.3 Hyperparameters
A summary of our hyperparameters for synthetic well as in-the-wild scenes can be found in Table 4.
B Feature Planes Inspection
In this section, we present a visual inspection of K-Planes feature planes within different contexts. Figures 5 to 7 each correspond to one out of the three orthogonal planes. Each element in Figures 5 to presents single feature plane, picked randomly from the C' feature planes. Figures 5a, 6a and 7a represent the state of the planes at an intermediate stage of the RefinedFields optimization process. Figures 5b, 6b and 7b represent the state of the planes at the end of the RefinedFields optimization. Figures 5c, 6c and 7c represent the planes at the end of the K-Planes-S SS optimization (no refining is done in this case).
Two noteworthy observations emerge. First, as seen in Figures 7, K-Planes feature planes are very similar in structure to real images. In fact, these feature planes depict orthogonal projections of the planes. These findings are especially compelling, as they justify the appropriate choice of Stable Diffusion the pre-trained prior for the refining stage, and provide insight onto the quantitative and qualitative results showcased by our method. Second, a comparison between columns [5b, 6b and 7b] and [5c, 6c and 7c] highlights the impact scene refining has on the feature plancs themselves, as planes 5b, 6b and 7b exhibit details that are more similar structure to the scene, and that are sharper than planes 5c, Ge and 7c.
onto the
C Supplementary Results
We present below additional qualitative results for the-hhhldd scenes (Figures 8 to 10) and synthetic scenes (Figures 11 and 12).
16

---

Published in Transactions on Machine Learning Research (05/2025)
(a) RefinedFields (In-Process)
(b) RefinedFields (Final)
(c) K-Planes
Figure 5: Feature planes inspection. Visualization of the (æy) K-Planes feature planes during the RefinedFields optimization process (5a), at the end of the RefinedFields optimization (ōb), and a comparison with vanilla K-Plancs-SS (5c). Feature planes within the (zy) K-Plancs are picked randomly.
17

---

Published in Transactions on Machine Learning Research (05/2025)
(a) RefinedFields (In- (n-Process)
(b) RefinedFields (Final)
(c) K-Planes
Figure 6: Feature planes inspection. Visualization of the (æz) K-Planes feature planes during the RefinedFields optimization process (6a), at the end of the RefineFFields optimization (6b), and a comparison with vanilla K-Planes-SS (6c). Feature planes within the (æz) K-Planes are picked randomly.
18

---

Published in Transactions on Machine Learning Research (05/2025)
(a) RefinedFields (In-Process)
(b) RefinedFields (Final)
(c) K-Planes
Figure Feature planes inspection. Visualization of the (yz) K-Planes feature planes during the RefinedFields optimization process (7a), at the end of the RefineFFields optimization (7b), and comparison with vanilla K. K-Planes-SS (7c). Feature planes within the (yz) K-Planes are picked randomly.
19

---

Published in Transactions on Machine Learning Research (05/2025)
(a) K-Planes
(b) RefinedFields
(c) GT
Figure 8: Qualitative results. Results on the Brandenburg Gate scene from Phototourism (Jin et al., 2020).
11I
MAN
K-Planes
(b) RefineFFields
(c) GT
Figure 9: Qualitative results. Results on the Sacré Coeur scene from Photolourism (Jin et al., 2020).
CENEE MAGNNEEE COMMENN MIDICI
L DEGINEEM COMMENND
MONIEIIC MONIFICO OR LRDOCCKS KOOO
(a) K-Planes
(b) RefinedFields
(c) GT
Figure 10: Qualitative results. Results on the Trevi Fountain scene from Phototourism (Jin et al., 2020).
20

---

Published in Transactions on Machine Learning Research (05/2025)
Figure 11: Qualitative results. RefinedFields results the NeRF Synthetic scenes.
Figure 12: Ground Truth Renderings. Ground truth images from the NeRF Synthetic dataset.
21

---

Published in Transactions on Machine Learning Research (05/2025)
Table 4: Hyperparameters. A summary of the hyperparameters used to train our model. Appearance optimizations only apply for in-the-wild training. Note that this parameter taken differently from Fridovich- Keil et al. (2023), as we only work with single- scale planes. We consider the highest plane resolution from the multi-scale approach taken by Fridovich-Ken et al. (2023).
Hyperparameter
Value
200 (syathetic)
20 (Sacré Coeur)
Epochs (Nopocha)
20 (Brandenburg Gate)
10 (Trevi Fountain)
Fitting iterations (N1)
30000
Refining iterations (N2)
3000
Batch size
4096
Optimizer
Adam Warmup Cosine
Schodulor
K-Planes Learning Rate
0.01
LoRA Learning rate
0.0001
LoRA rank (r)
SD latent resolution
SD channel dimension
SD prompt
""
Number of planes
K-Planes resolution"
512
K-Planes channel dimension
32
Epochs Appearance Optimization Appearance embeddings dimension
10
32
0.1 (Socré Cocur)
Appearance learning rate
0.1 (Trevi Fountoin)
0.001 (Brandenbury Gate)
A Appearance batch size
512
22


Reviews:
[{"review_id": "AsBgEOpb0O", "reviewer": "Reviewer_GN77", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**:\n\n1.This idea is interesting and presents a novel approach to leveraging prior information for novel view synthesis. It utilizes only existing pre-trained models without requiring additional training data, making it both efficient and scalable for future applications.\n\n2.The paper is well-written, and the experiments effectively demonstrate the method's effectiveness.\n\n**Weaknesses**:\n\n1.The pre-trained diffusion model is not trained in the feature plane domain, and the reviewer is unsure what kind of prior is used in the proposed method. Some toy experiments or additional discussion may be needed to clarify this point.\n\n2.In the ablation studies, \"RefinedFields-noFinetuning\" refers to the method without LoRA on U-Net. However, it is unclear whether the decoder is still fine-tuned. Clarification on this aspect would be helpful.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces RefinedFields, a novel method that enhances  the quality of the K-Planes representation by integrating pre-trained networks. The key contribution is a training scheme that iterates between scene fitting (optimizing K-Planes with training images) and scene refining (leveraging a pre-trained model to enhance the representation). In this way, the method can leverage the prior information in the pre-trained diffusion model to regularize the K-Planes representation. This approach enables richer details in novel view synthesis (NVS) and improves in-the-wild scene modeling. Extensive experiments demonstrate that RefinedFields significantly outperforms K-Planes.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please refer to weaknesses.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "exT0oUd5YI", "reviewer": "Reviewer_etk7", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\n\n- The proposed approach guides volumetric scene modeling by leveraging not only the training dataset at hand but also the rich prior lying within the weights of the pre-trained model.\n- The paper is well-written and the results are promising, with an evaluation on synthetic and real scenes.\n\n### Weaknesses\n\n- The proposed method is really interesting, however its scope seems limited only to K-Planes. Can RefinedFields be applied also to other models?\n\n- The proposed method increases marginally and partially K-Planes results (Table 2), but what is the computational overhead? How do you choose the number of fitting and refining iterations? \n\n- (Minor) Best results should be highlighed in Table 3 and the arrows in the metrics header are missing.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper extends K-Planes leveraging pre-trained generative models as a rich prior to refine planar scenes representations. The training procedure iteratively switches between optimizing a K-Planess representation on images from a particular dataset (scene fitting), and fine-tuning a pre-trained network to output a new conditioning leading to a refined version of this K-Planes representation (scene refining).", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please see the weaknesses.", "broader_impact_concerns": "No concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "JxHnLHazNY", "reviewer": "Reviewer_ML5J", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\nThe idea of refining triplanes with pretrained video models is interesting.\nThe paper is well-motivated and easy-to-follow.\n\nWeaknesses:\nThe authors claimed that “This class of methods has not yet been explored for in-the-wild scene modeling, as leveraging priors over representations modeling unconstrained scenes is not evident.” I don’t think this is correct, with the advance of multi-view diffusion models and video generation models, there are a lot of works that use pretrained models to handle novel view synthesis for in-the-wild scenes, such as CAT3D [1]\n\nAnother line of work is Dust3R-based methods, which also achieve very high-quality reconstruction results with a feedforward transformer method. These models are trained on several real-world datasets and are assumed to be able to generalize to in-the-wild images.\n\nThe results of the proposed model are not good enough: even with refining, the results are still lagging behind existing models such as Splatfacto-W and WildGaussians\n\nCurrently, the method is quite simple and involves two quite straightforward stages, I am not quite sure the technical contribution of the methods.\n\n[1] CAT3D: Create Anything in 3D with Multi-View Diffusion Models", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper presents a method to leverage pretrained Stable Diffusion to refine the K-planes representations optimized from a set of input images. Specifically, the authors alternate between training using input images and refining it with pretrained diffusion models to push K-planes features to more closely resemble real-world images. The authors experimented to validate that their methods can improve the vanilla K-Planes models.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Better clarify the main contributions of the proposed method. Right now it is a simply combination of two stages.\n\nCompare with methods such as CAT3D to demonstrate the effectiveness of the method\n\nImprove quality: the authors might consider applying the same technique to refine more advanced methods, such as WildGaussians, by parametrizing the Gaussians into pixel splat format and compatible with pretrained diffusion models", "broader_impact_concerns": "The authors might need to discuss the influence of the proposed method and generative AI on the society.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: While the reviewers acknowledged the interest of the core idea behind the work and the clarity of the paper, they expressed some concerns about the technical contributions and its scope, the relationship with some existing methods, and some aspects of the empirical results. The authors' feedback clarified most of these points, leading to two reviewers leaning to accept, but one leaning to reject. The main argument for rejection is the limited contributions of the work. Considering that this argument goes against the TMLR rules for rejection based on novelty, and that the paper is supported by the other two reviewers, this AE recommends acceptance.
Decision: Accept (poster)