IllumiNeRF: 3D Relighting Without Inverse Rendering
Authors: Xiaoming Zhao
Pratul P. Srinivasan
Dor Verbin
Keunhong Park
Ricardo Martin Brualla
Philipp Henzler
Venue: NeurIPS 2024 poster (2024)
Keywords: 3D Relighting; NeRF; Diffusion Model
Primary area: machine_vision

Abstract: Existing methods for relightable view synthesis --- using a set of images of an object under unknown lighting to recover a 3D representation that can be rendered from novel viewpoints under a target illumination --- are based on inverse rendering, and attempt to disentangle the object geometry, materials, and lighting that explain the input images. Furthermore, this typically involves optimization through differentiable Monte Carlo rendering, which is brittle and computationally-expensive. In this work, we propose a simpler approach: we first relight each input image using an image diffusion model conditioned on target environment lighting and estimated object geometry. We then reconstruct a Neural Radiance Field (NeRF) with these relit images, from which we render novel views under the target lighting. We demonstrate that this strategy is surprisingly competitive and achieves state-of-the-art results on multiple relighting benchmarks. Please see our project page at [illuminerf.github.io](illuminerf.github.io).

IllumiNeR:: 3D Relighting Without Inverse Rendering
Xiaoming Zhao 1,3+ Pratul P. Srinivasan
Dor Verbin
Keunhong Park
Ricardo Martin-Brualla'
Philipp Henzler'
"Google Research
Google DeepMind
University of Illinois Trbana-Champiin
Input: images poses + novel illumination
GOLDEN CURRY
Output: 3D reconstruction under novel illumination
Figure I: Given of posed input images under an unknown lighting (four exemplar images from
the set are shown on top), IllunNeRF produces high-quality novel views (bottom) relit under a target lighting (illustrated chrome balls). Inputs obtained from the Stanford-ORB dataset [27].
Abstract
Existing methods for relightable view synthesis - using set of images of an object under unknown lighting to recover a 3D representation that can be rendered from novel viewpoints under a target illumination - are based on inverse ren- dering, and attempt disentangle the object geometry, materials, and lighting that explain the input images. Furthermore, this typically involves optimization through differentiable Monte Carlo rendering, which is brittle and computationally- expensive. In this work, we propose a simpler approach: we first relight each input image using an image diffusion model conditioned lighting and estimated object geometry. We then reconstruct a Neural Radiance Field (NeRF) with these relit images, from which we render novel views under the target lighting. We demonstrate that this strategy is surprisingly competitive and achieves state-of-the-ar results on multiple relighting benchmarks. Please see our project page at illuminert github. io.
target environment
Introduction
Capturing an object's appearance so that it can be accurately rendered in novel environments is central problem in computer vision whose solution would democratize 3D content creation for augmented and virtual reality, photography, filmmaking, and game development. Recent advances in view synthesis [36] have made impressive progress in reconstructing 3D representation that can be rendered from novel viewpoints, using just set of observed images. However, those methods
Work done intern at Google.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

typically only recover the appearance of the object under the captured illumination, and relightable view synthesis - rendering novel views of the captured object under arbitrary target environments remains challenging.
Recent methods for recovering relightable 3D representations treat this task inverse rendering, and attempt to estimate the geometry, materials, and illumination that jointly explain the input images using physically-based rendering methods. These approaches typically involve gradient- based optimization through differentiable Monte Carlo rendering procedures, which are noisy and Moreover, the inverse rendering optimization problem is brittle and inherently ambiguous; many potential sets of geometry, materials, and lighting can explain the imput images, but many of these incorrect explanations produce obviously implausible renderings when rendered under novel unobserved illumination.
We propose different approach that avoids inverse rendering and instead leverages generative image model fine-tuned for the task of relighting. Given a set of images viewing an object and a desired target illumination, we use single-image 2D Relighting Diffusion Model that outputs relit images of the object under the target illumination. Due to the ambiguous nature of the problem, each sample of the generative model encodes a different explanation of the object's 's materials, geometry and the input illumination. However, as opposed to optimization-baaed inverse rendering, such samples are all plausible relit images since they are the output of the trained diffusion model.
Instead of attempting to recover a single explanation of the underlying object's 's appearance, we sample multiple plausible relit images for each observed viewpoint, and treat the underlying explanations as samples of unobserved latent variables. To recover a final consistent 3D representation of the relit object, we use the full set of sampled relit images from all viewpoints to train "latent NeRF" that reconciles all the samples into a single 3D representation, which can be rendered to produce plausible relit images from novel viewpoints.
The key contribution of our work is a new paradigm for relightable 3D reconstruction that replaces 3D inverse rendering with: generating samples with single-image 2D Relighting Diffusion Model followed by distilling these samples into a 3D latent NeRF representation. We demonstrate that this strategy is surprisingly competitive and outperforms existing most 3D inverse rendering baselines on the TensoIR [23] and Stanford-ORB [27] relighting and view synthesis benchmarks.
2 Related Work
Our work addresses the task of relightable 3D reconstruction by using a model as a generative prior for single-image relighting. It is closely related to prior work in relightable 3D reconstruction, inverse rendering, and single-imag relighting. Below, we review these lines of work and discuss how they relate to proposed approach.
diffusion
Relightable 3D Reconstruction The goal of relightable 3D reconstruction is to reconstruct a 3D representation of an object that can be relit by novel illumination conditions and rendered from novel camera poses. In scenarios where an object is observed under multiple lighting conditions [12]. it is trivial to render its appearance under novel illumination that a linear combination of the observed lighting conditions, due to the linear behavior of light. This approach is generally limited to laboratory capture scenarios where it is possible to observe an object under lighting basis.
In more casual capture scenarios, the object is observed under just a single or a small handful of lighting conditions. Existing works typically address this setting using methods based inverse rendering that explicitly factor an object's is appearance into the underlying 3D geometry, object material properties, and lighting that jointly explain the observed images. State-of-the-an approaches to 3D inverse rendering [9. 10, 17, 23, 26, 33, 38, 46, 47] generally utilize the following strategy: they start with neural field representation of 3D geometry (typically volume density as in NeRF [36], hybrid volume-surien representations as in NeuS [57] and VoISDF [59], or meshes extracted from neural field representations) from the input images, equip the model with representation of surface materials (e.g. apatially-varying BRDF parameters) and lighting. and jointly optimize these factors through differentiable physics-based rendering procedure [40]. While methods may differ in their choice of geometry, material, and lighting representations, and employ different techniques to accelerate the evaluation of the rendering integral, they generally all follow this same high- level inverse rendering strategy. Unfortunately, even if the geometry is known, inverse rendering

---

is notoriously ambiguous problem [43, 52] and many combinations of materials and lighting can explain an object's appearance. However, not all of these combinations are plausible, and incorrect factorizations that explain observed images under one lighting condition may produce glaring artifacts when rendered under different lighting. Furthermore, differentiable physics- based as thousands of samples are needed for Monte Carlo estimates of the rendering integral. typically requires custom implementations [2, 3, 22, 28, 32, 35, 54], and the resulting inverse rendering loss landscape is non-smooth and difficult to optimize effectively with gradient descent [14].
rendering is computationally-expnnsstt
Single Image Relighting Instead of using inverse rendering to recover object material parameters which can be relit with physically- based rendering techniques, we train diffusion model that can directly sample from the distribution of relit images conditioned on a target lighting condition. This diffusion model is essentially a generative single-image relighting model. Early single image relighting techniques employed optimization- based inverse rendering [4]. Subsequent methods trained deep convolutional neural networks output image geometry, materials, and lighting [0), or in some cases, to directly output relit images [48, 7, 8].
Most related to method are few recent works that have trained diffusion models for single image relighting. LightIt [25] trains a model similar to ControlNet [63] to relight outdoor images under arbitrary sun positions conditioned
input normals and shading. DiffissionLihht [41] estimates the lighting of an image by using a ControlNet to inpaint the color pixels of a chrome ball in the middle of the scene, from which environment map be recovered.
Most similar to our work the concurrent method of DiLightNet [61] that focuses on single image relighting. DiLLighNeet uses a Control [63] approach to condition single-image relighting diffusion model on target environment map. Dil iighttt uses a set of "radiance cues" [15] renderings of the object's geometry (obtained from an off-the-shelf monocular depth network) with various roughness levels under the target environment illumination - as conditioning. Our method instead focuses on 3D relighting, where multiple of images of object are available. It uses similar single-image relighting diffusion model conditioned on radiance cues. Unlike DiLightNet which uses geometry from monocular depth estimation render radiance cues, we use geometry estimated from the input views using a state-of-the-as art surface reconstruction method [56]. This allows our model to better model complex light transport effects such interreflections caused by occluded geometry.
3 Method
3.1 Problem Formulation
Given a dataset of images of an object and corresponding camera poses D = [4, )=)) the general goal of relightable 3D reconstruction is to estimate a model with parameters 0 that when rendered, produces relit versions of the dataset under unobserved target illumination L"" This can be expressed as:
0- = ar argmax p(D) (D),
(I)
where Dj- A (relight L", T. 0), T. ) illumination L using model 0. Note that Eq. (1) only maximizes the likelihood of the original given poses after relighting. However, by using view synthesis, we can then turn the collection of relit images into 3D representation which can be rendered from arbitrary poses. For brevity, we therefore omit the implicit dependence of DT in 6.
a relit version of the original dataset under target
This relighting problem has traditionally been solved by using inverse rendering. Inverse rendering techniques do not maximize the probability of the relit renderings, but instead recover a single point estimate of the most likely scene geometry G, materials M, and lighting T. (note that this is the "source" lighting condition for the observed images) that together explain the input dataset, and then use physically-based rendering to relight this factorized explanation under the target lighting. Inverse rendering seeks recover 0if (G*, MM), where:
G* S.MM, ,",,L* argmax MG L(D) = argmax Seninaxp(D|G... G.M.L
(2)
G.M.L
3

---

Target L
(b) Lotent NeRE
p Matenal latent
(d) Radiance CIRs
(e) Kelighting Diffisionn Model
[2) Passs
N Images
Figure 2: Overview, Given set of images I and camera poses in (a), run NeRF to extract the 3D geometry as in (b). Based on this geometry and a target light shown (c), we create radiance cues for each given input view in (d). Next, we independently relight each input image using single-image Relighting Diffusion Model illustrated in (e) and sample S possible solutions for each given view displayed in (f). Finally, distill the relit of images into 3D representation through a Latent NeRF optimization as in (g) and (h).
The first data likelihood term is computed by physics-based rendering of the estimated model and the second prior term is often factorized into separate handcrafted priors geometry, materials, and lighting [23, 33, 43].
relighting approach based on inverse rendering then renders each image in corresponding to camera pose using the recovered geometry and materials, illuminated by the target lighting L resulting in relight(DD L π, 0BR). This approach has three main issues. First, the differentiable rendering procedures used expensive. Second, it requires careful modeling of light transport which is cumbersome and existing differentiable renderers do not account for many types of lighting and material effects seen in the real world. Third, there are often ambiguities between M and L, meaning that any errors in their decomposition may apparent in the relit data. is quite difficult design effective handcrafted priors on geometry, materials, and lighting, so inverse rendering procedures frequently recover explanations that have high data likelihood (are able to render the observed data) but produce clearly incorrect results when re-rendered under different illumination.
compute the gradient of the likelihood term computationally-
3.2
Model Overview
We propose an approach that attempts to maximize the probability of relit images in Eq. (1) without using explicit physically-han latent variable Z that can be thought of as implicitly representing the input images' lighting along with the object's material and geometry parameters. We can write the likelihood of the relit data
model of the object's lighting materials. First, let us introduce
D(DT(D)
2DD)dZ - 1
Introducing these latent variables lets us consider all relit renderings in the dataset, DJ (T. as conditionally independent, since the rendering under the target lighting L is deterministic given the object's geometry and materials. This enables writing the likelihood as:
(T(.m.m)))) m.).
/ II D(D) Z,, D.)
D(D"ID)
(4)
latent NeRF
latent prior
We propose to model this with latent NeRF model, as used by Martin Biriiin--Bulllaa et al. [34] that is able render novel views under the target illumination for any sampled latent vector. We describe this model in Sec. 3.3. We train this NeRF model by generating large quantity of sampled relit images with the same target lighting but with different (unknown) latent vectors using a Relighting Diffussin Model which we will describe in Sec. 3.4. In this way, the latent NeRF model effectively distills large dataset of relit images sampled by the diffusion model into single representation that can render novel views of the object under the target lighting for any sampled latent.

---

(a) Samples from Relighting Diffusion Model
(b) Latent NeRF Renderings
Figure 3: Relit samples latent NeRF. (a) Samples of the Relighting Diffusion Model (Sec. 3.4) for the same target environment map, and (b) renderings from the optimized Latent NeRF (Sec. 3.3) for fixed value of the latent. The diffusion samples correspond different latent explanations of the scene and our latent NeRF optimization is able to effectively optimize these latent variables along with the NeRF model's parameters to produce consistent renderings for each latent explanation.
3.3 Latent NeRF Model
We wish model the distribution in Eq. (4) in manner that lets render images that correspond relit views of the object for any sampled latent Z. We choose to model this with latent code NeRF 3D representation, inspired by prior works that condition NeRFs on latent codes to represent sources of variation such as the time of day during capture [34]. This latent NeRF optimizes set of latent codes that are used condition the view-dependent color function represented by the NeRF, enabling it to render novel views of the relit object under the target illumination for any sampled latent code. our implementation, the latent NeRF's geometry does not depend on the latent code, so the latent code may be interpreted as only representing the object's material properties.
To optimize the parameters of the latent NeRF model, maximize the log-likelihood, which by using Eq. (4), can be written as the following maximization problem:
argmax log (DD argmax log /
N II
(PD D.) p(ZiD)dZ.
(5)
Because integrating over all possible latents Z is intractable, and replace the integral with the maximum a posteriori (MAP) estimate of
heuristic inference strategy
8 argmax max Z
log p(D Z, D.) Σ
log log p(2
(6)
By assuming a Gaussian model over the data given the materials, the first term in Eq. (6) is reconstruction loss over the images. However, since do not have access to the true latent vector Z. we assume uniform prior over them, turning the second term in Eq. (6) into constant. In practice, similar prior work on NeRFs optimized to generate new views given dataset containing images with varying appearance, we rely the NeRF model to resolve any mismatches in the appearance of different images [34]. See Fig. 3 for illustrations. The minimization of the negative aegliielloood can then be written as:
0+ arg min min
ID' lntent-NeRF(O, Z, 1.).
0
3.4 Relighting Diffusion Model
In order to train the latent NeRF model described in Sec. 3.3, we use a Relighting Diffusion Model (RDM) to generate S samples for each viewpoint from p(D("),) In other words, given input image and target lighting LT, the single-image RDM samples S images corresponding to relit
5

---

Diftuse
Roughness 0.34
Roughness 0.13
Roughness 0.05
Figure 4: Example radiance cues for view of the 'hotdog' scene.
versions of Dj that have high likelihood given the target light L" We then associate each sample E {1....,S) with its own latent code Z, and sum over all samples when training the latent NeRF (Eq. (7)).
Our RDM is implemented as an image denoising diffusion model that is conditioned by the input image and target lighting. To encode the target lighting, we use image-space radiance cues [15, 44, 61], visualized in Fig. 4. These radiance cues are generated by using simple shading model to render handful of images of the object's estimated geometry under the target lighting. This procedure is designed to provide information about the effects of specularities, shadows, and global illumination, without requiring the diffusion network to learn these effects from scratch. In our experiments, we use four different pre-defined materials to render radiance cues: one diffuse material with a pure white albedo, and three purely-specular materials with roughness values {0.05, 0.13, 0.34). We use GGX [55] as the shading model. For more details, please refer to Sec. A.2.
The RDM architecture consists of pretrained latent image diffusion model, similar to StableDiffu- sion [45], and uses a ControlNet [63] based approach condition on the radiance cues. Please refer to Sec. A.3 for more architecture details.
Experiments
4.1 Experimental Setup
Relighting Dataset We render objects from Objaverse [13] under varying poses and illuminations. For each object, we randomly sample 4 poses, and render each under different lighting conditions. We represent the lighting as HDR environment maps, and randomly sample from dataset of 509 environment maps from Polyhaven [60]. For more details, see Sec. A.4.
Evaluation Datasets We evaluate our method two datasets: TensoIR [23], synthetic benchmark,
and Stanford-ORB [27], a real-world benchmark. TensolR contains renderings of four synthetic objects rendered under six lighting conditions. Following [23],
the training split of 100 renderings with "sunset" lighting as input {1.). We then evaluate on 200 poses, each of which has renderings under five different environment maps, i.e., "bridge", "city", "fireplace", "forest", and "night", for a total of 4000 renderings. Stanford-ORe -RB is a real-world benchmark for inverse rendering on data captured in the wild. contains 14 objects with various materials and captures each object under three different lighting settings, resulting in 42 (object, lighting) pairs. For the task of relighting. we are given images of an object under a single lighting condition and follow the benchmark protocol to evaluate relit images of the object under the two target lighting settings.
Baselines We compare method to several existing inverse rendering approaches. On both benchmarks, we compare to NeRFactor [65] and InvRender [66]. On the synthetic benchmark, we additionally compare to TensoIR [23], the current top-performing approach on that benchmark. For the Stanford-ORB benchmark, we additionally compare to PhySG [62], NVDiffRec [38], NeRD [10], NVDiffRecMC [17], and Neural-PBIR [47].
Our Model Inferenee At inference time, the ideal embedding vector Z that best corresponds the actual material is unknown. One approach to find this vector is to optimize Z to match a subset of the test images (as in [34]). However, to ensure fair comparison, we avoid this optimization. Instead, we set Z = 0 for all views when rendering test images.

---

Table 1: TensoIR benchmark [23]. We evaluate four objects. Each object has five target lightings. each of which is associated with 200 poses, resulting in evaluating 4000 renderings in total. Running time for baselines are copied from [23]. Our time is (geometry optimization on GPU) (diffusion
sampling on TPU) + C (latent NeRF optimization on GPU). Best and 2nd-best are highlighted.
PSNRT
SSIMT
LPIPS]
Wall-clock Time!
Device RTX
NcRFactor [65]
23,383
0,908
0.131
> 100
InvRender [66]
23.973
0.901
0.101
h
RTX 2080 Ti RTX 2080 Ti
TensolR [23]
28.580
0.944
0.081
5 h
Ours
29.709
0.947
0.072
0.75 h 0.75 h
16 A 40GB + a TPUV5 a A100 40GB + a TPUv5
Ours (single GPU)
29.245
0.946
0.073
2h+ 1 1h1h 2
Ground-trui
Ours
TensolR
Ground-truth
Ours
TensoIR
Figure 5: Qualitative results on TensoIR. Renderings from all approaches have been rescaled with respect the ground-truth truth as mentioned in Eq. (4.1). Unlike TensolR, our method faithfully recovers specular highlights and colors as indicated in red.
Evaluation Metrics For both benchmarks, we evaluate the quality of 3D relighting by reporting image metrics for rendered images. We report PSNR, SSIM [58], and LPIPS- VGG [64] on low dynamic range (LDR) images. Additionally, we report PSNR on high dynamic range (HDR) images on Stanford- ORBB following the benchmark protocol, denoted as PSNR-H while the PSNR LDR images marked as PSNR-L. For approaches that do not produce HDR renderings, including ours, we convert the LDR renderings to linear values by using the inverse of the sRGB tone mapping curve. Due to the inherent ambiguities for the relighting task, we follow prior works [23, 27] and apply channel-wise scale factor to RGB channels to match the ground truth image before computing metrics. Following established evaluation practices Stanford-ORE we compute the scale per output image individually whereas for TensolR we compute a global scale factor that is used for all output images."
4.2 Benchmarking
Unless otherwise specified, all results are produced using S = 16 samples (see Sec. 3.4) and make use of 16 A100 40GB GPUs (batch size 214 rays for NeRF optimization). We also provide results on single A100 40GB GPU (batch size of 213 for NeRF optimization).
We report quantitative results on the TensoIR benchmark in Tab. and show qualitative examples in Fig. 5. We significantly outperform all competitors quantitatively on all metrics with comparable or improved wall-clock time. Visually our method is capable of recovering specular highlights whereas prior methods struggle to model these.
Similarly, report results Stanford-ORe -RBB in Tab. 2 and Fig. 6. Our proposed approach quantita- tively improves upon all baselines, except those of Neural-PBIR [47], indicating the effectiveness of IllunNeRR in real world scenarios. Note that although Neural-PBIR achieves better metrics than us, Fig. 6 shows that their relighting results are mostly diffuse, even for highly-glossy objects, and that they lack many of the strong specular highlights that method is able to recover. This behavior of their model may explain their better metrics despite worse qualitative performance for specular highlights, because the illumination maps provided by Stanford-ORB do not correspond to the
'Please refer to https: / thub. httpa//ooooo.aaaaaaaaoooaaaaaaaa test py3136 and http://twwt..aaaaaaaaaaaaaaaaaaa
py#L112

---

GT
Ours
Neural PBIR
NVDiffReeMC
InvRender
NeRFactor
NeRD
NVDiiReee
PC
Figure 6: Qualitative results on Stanford-ORB. Renderings from all approaches have been rescaled with respect to the ground-truth mentioned in Sec. 4.1. Areas where our approach performs well are highlighted. Our approach produces high-quality renderings with plausible specular reflections.
incident illumination at the object's location, since they were captured using a light probe which was moved for each image in the dataset [27]. This means that even given perfect materials and geometry, the images relit by any method cannot match with the true captured images, which most noticeable in specular highlights. This mismatch penalizes methods like ours, which recover such specularities, over ones that recover mostly diffuse appearance with no apparent specular highlights [52]. For more detailed discussion see Sec. B.
We also provide qualitative results for different latent codes in Fig. 7. These results demonstrate that the optimized latent codes effectively capture various plausible explanations of the materials.

---

Table 2: Stanford-ORB benchmark [27]. We evaluate 14 objects, each of which was captured under three different lightings. For each (object, lighting) pair, we evaluate renderings of the same object under the other two lightings, resulting evaluating 836 renderings. fdenotes models trained with the ground-truth 3D scans and pseudo materials optimized from light-box captures. Best and 2nd-best are highlighted.
PSNR-H HT
PSNR-L1
SSIM1
LPIPS
NVDiffRecMC [17]f
25.08
32.28
0.974
0.027
NVDiffRec [38]
24.93
32.42
0.975
0.027
PhySG [62]
21.81
28.
0.960
0.055
NVDiffRec [38]
22.91
29.72
0.963
0.039
NeRD [10]
23.29
29.65
0,957
0.059
NcRFactor [65]
23.54
30.38
0.969
0.048
InvRender [66]
23.76
30.83
0.970
0,046
NVDiffReecC [17]
24.43
31.60
0.972
0.036
Neural-PBIR [47]
26.01
33.26
0.979
0.023
Ours
25.42
32.62
0.976
0.027
Ours (single GPU)
25.56
32.74
0.976
0.027
Sample
NeRF Rendering
Sample
Rendering
RDM Sample 3
NeRF Rendering
Figure 7: Renderings from various latents. Each column shows 1) Relighting Diffusion Model (RDM) sample and 2) two latent NeRF renderings using the sample's 's latent code. The diffusion samples are selected uniformly from all N (#views) x S (#samples per view) diffusion generations. Each row shows results from the same object and lighting with latent codes capturing various plausible explanations of the materials.
4.3 Ablations
We evaluate ablations of our model on TensolR's hotdog scene in Tab. 3, and visualize them in Fig. 8. We reach the following conclusions: 1) The latent NeRF model is essential: optimizing a standard NeRF cannot reconcile variations across views, even if we only generate a single sample per viewpoint for optimization (S 1). 2) More diffusion samples help: by increasing S. the number of samples from the RDM per viewpoint, we observe consistent improvements across almost all metrics. This corroborates our intuition that using an increased number of samples helps the latent NeRF effectively fit the target distribution (Eq. (4)) in more stable way.

---

Table 3: Ablations. We conduct ablation studies on the Hotdog scene from TensoIR [23]. We evaluate renderings of 200 novel test camera poses, each under five target environment map lighting conditions, resulting in evaluating 1000 renderings in total.
Best is highlighted.
S
Latent
PSNRT
SSIM
LPIPS
24.957
0.921
0.099
1
/
26.321
0.925
0.097
4
/
27.409
0,936
0.087
16
/
27.950
0.939
0.082
Souppe
Sauppes
No Latent, S 5-1 1
w/ Latent, S I
w/ Latent, 4
w/ Latent, S 16
Ground-truth
Figure 8: Using standard NeRF instead of latent NeRF model is unable to reconcile training samples with different underlying latent explanations. Using a latent NeRF model significantly increases the accuracy of rendered specular appearance, and increasing the number of samples S from the RDM used to train the latent NeRF model further increases the quality of the output renderings.
4.4 Limitations
Our model relies on high quality geometry estimated by UniSDF [56] (see Sec. A.I) to provide sufficiently good radiance cues for conditioning the RDM (Sec. 3.4). Any missing structure will lead our model to miss specular reflections, as seen on the top left of the salt can result in Fig. 6's second column. Errors in geometry also affect the quality of synthesized novel views, e.g. the missing thin branches from the plant in Fig. 5 or fine details of the cactus (column 4) in Fig. 6. Note that our RDM, trained on high-quality synthetic geometry, will inherently improve with future advances in geometry reconstruction. Our approach is not suited for real-time relighting, as it requires generating new samples with the RDM and optimizing a NeRF for any new target lighting condition.
Conclusion
We have proposed new paradigm for the task of relightable 3D reconstruction. Instead of decom- posing an object's appearance into lighting and material factors and then relighting the object with physically-basee rendering,
single-image Relighting Diffusion Model (RDM) to sample a varied collection of proposed relit images given a target illumination, and distill these samples into single consistent 3D latent NeRF representation. This 3D representation synthesize novel views of the object under the target lighting. Perhaps surprisingly. this paradigm consistently outperforms existing inverse rendering methods on synthetic and real-world object relighting benchmarks. This new paradigm's success is likely due to the RDM's ability to generate a large number of proposals for the new relit images. This is in contrast to prior works based on inverse rendering, which first estimates a single material model and then uses it for relighting, since errors in material estimation may propagate to the relit images. We believe that this paradigm may be used to improve data capture, material and lighting estimation, and that it may be used to do so robustly on real-world data.
be rendered to
10

---

Acknowledgements
We would like to thank Ben Poole and Ruiqi Gao for insightful discussions. We thank Yunzhi Zhang and Zhengfei Kuang for providing their qualitative results for the Stanford-ORB [27] baseline, and Haian Jin for the TensoIR [23] baseline results. We are also grateful to Abhijit Kundu and Henna Nandwani for their infrastructure support.
References
[1] H. Aanæs, R. R. Jensen, G. Vogiatzis, E. Tola, and A. B. Dahl. Large-Scale Data for Multiple- View Mulw Stereopsis. IJCV, 2016.
[2] Bangaru, M. Gharbi, T.MM. Li, F. Luan, K. Sunkavalli, M. Hasan, S. Bi, Z. Xu, Bernstein, and Durand, Differentiable Rendering of Neural SDFs through Reparameterization. SIGGRAPH Asia, 2022. 3
[3] S. Bangaru, L. Wu, T.-M. Li, J. Munkberg, G. Bersstein, J. Kelley, F. Durand, A. Lefohn, and Y. He. SLANG.D: Fast, Modular and Differentiable Shader Programming. ACM TOG, 2023. 3
[4] J. T. Barron and J. Malik. Shape, Illumination, and Reflectance from Shading. TPAMI, 2014. 3
[5] J. T. Barron, B. Mildenhall, D. Verbin, P. P. Srinivasan, and P. Hedman. Mip-NeRF 360: Unbounded Anti-Aliased Neural Radiance Fields. In CVPR, 2021. 14
[6] J. T. Barron, B. Mildenhall, D. Verbin, P. P. Srinivasan, and P. Hedman. Zip-NeRF: Anti- Aliased Biid-Based Neural Radiance Fields. In ICCV, 2023. 14
[7] A. Bhattad and D. A. Forsyth. Cut-and-Passe Object Insertion by Enabling Deep Image Prior for Reshading. In 3DV, 2022.
[8] A. Bhattad, Soole, and D. Forsyth. StyLitGAN: image-Based Relighting via Latent Control, In CVPR, 2024.
[9] S. Bi, Z. Xu, P. Srinivasan, B. Mildenhall, K. Sunkavalli, M. Hašan. Y. Hold-Geoffroy D. Kriegman, and
R. Ramamoorthi, Neural Reflectance Fields for Appearance Acquisition. ArXiv, 2020.
[10] M. Boss, R. Braun, V. Jampani, T. Barron, Liu, and H. P. Lensch. NeRD: Neural Reflectance
Decomposition from Image Collections. ICCV, 2021. 2, 6, 9, 17
[11] Bradbury, R. Frostig, Hawkins, M. Johnson, Leary, D. Maclaurin, G. Necula, A. Paszke,
J. VanderPlas, S. Wanderman-Millne, and Q. Zhang. JAX: Composable Transformations of Python+NumPy Programs, 2018 URL http: thub. com/ hoon//oool//aaao 14, 15
[12] P. Debevec, T. Hawkins, C. Tchou, H.-P. Duiker, W. Sarokin, and M. Sagar. Acquiring the Reflectance Field of a Human Face. In ACM CGIT, 2000. 2
[13] M. Deitke, D. Schwenk, J. Salvador, L. Weihs, O. Michel, E. VanderBilt, L. Schmidt, K. Ehsani, A. Kemb- havi, and A. Farhadi. Objaverse: A Universe of Annotated 3D Objects. In CVPR, 2023. 6, 16
[14] M. Fischer and T. Ritschel. Reduced Differentiable Path Tracing. In CVPR, 2023. 3
[15] D. Gao, G. Chen, Y. Dong, P. Peers, K. Xu, and X. Tong. Deferred Neural Lighting: Free-viewpoint Relighting from Unstructured Photographs. ACM TOG, 2020. 3,6
[16] K. Greff, F. Belletti, L. Beyer, C. Doersch, Y. Du, D. Duckworth, D. Fleet, D. Gnanapragussa, F. Golemo, C. Herrmann, T. Kipf, Kundu, D. Lagun, Laradji, H.-T. D. Liu, H. Meyer, Y. Miao, D. Nowrouzezahrai, Rebain, S. Sabour, M. S. Sajjadi, Sela. V. Sitzmann, A. Stone, D. Sun, S. Vora, Z. Wang. T. Wu, K. M. Yi, F. Zhong, and A. Tagliasacchi. Kubric: a scalable dataset generator. In CVPR, 2022. 14
C. Oztireli, E. Pot, Radwan,
[17] J. Hasselgren, N. Hofmman, and J. Munkberg. Shape. Light, and Material Decomposition from Images using Monte Carlo Rendering and Denoising. In NeurlPS, 2022. 2, 6, 9, 17
[18] J. Heek, A. Levskaya, A. Oliver, M. Ritter, B. Rondepierre, A. Steiner. and M. van Zee. Flax: A Neural Network Library and Ecosystem for JAX, 2023, URL httpt://tttthtt com/ pooolllllla.. 16
[19] D. Hendrycks and K. Gimpel. Gaussian Eror Linear Units (GELUs). ArXiv, 2016. 16

---

[20] J. Ho and T. Salimans, Classifier-Free Diffusion Guidance. ArXiv, 2022. 16
[21] J. Ho, A. Jain, and P. Abbeel. Denoising Diffusion Probabilistic Models. In NeulPS, 2020. 16
[22] W. Jakob, S. Speierer, N. Roussel, and D. Vicini. Dr.jit: A Just-In-Time Compiler for Differentiable Rendering. ACM TOG, 2022.
[23] H. Jin, L. Liu, P. Xu, X. Zhang, S. Han, S. Bi, X. Zhou, Z. Xu, and H. Su. TensoIR: Tensorial Inverse Rendering. In CVPR, 2023. 2, 4, 6, 7, 10, 11
[24] D. P. Kingma and J. Ba. Adam: A Method for Stochastic Optimization. ArXiv, 2014. 14
[25] P. Koesis, J. Philip, K. Sunkavalli, M. Nießner, and Y. Hold-Geoffroy. Lightlt: Illumination Modeling and Control for Diffusion Models. In CVPR, 2024. 3
[26] Z. Kuang, K. Olszewski, M. Chai, Z. Huang, P. Achlioptas, and S. Tulyakov. NeROIC: Neural Rendering of Objects from Online Image Collections. ACM TOG, 2022, 2
[27] 7. Kuang, Y. Zhang, H.-X. Yu, S. Agarwala, E. Wu, J. Wu, et al. Stanford-ORB: A Real-World 3D Object Inverse Rendering Benchmark. In NeurlPS, 2023. 1, 2, 6, 7, 8, 9, 11,
[28] T-MM. Li, M. Aittala, F. Durand, and J. Lehtinen. Differentiable Monte Carlo Ray Tracing through Edge Sampling. ACM TOG, 2018.
[29] 7. Li, 7. Xu, R. Ramamoorhh,, K. Sunkavalli, and M. Chandraker. Learning reconstruct shape and spatially-varying reflectance from a single image. ACM TOG, 2018. 3
[30] Z. Li, M. Shafiei, Ramamoorthi, Sunkavalli, and M. Chandraker. Inverse Rendering for Complex Indoor Ccenes: Shape, Spatially-varying Lighting and SVBRDF from a Single Image. CVPR, 2020.
[31] E. Lorensen and E. Cline. Marching Cubes: A High Resolution 3D Surface Construction Algorithm.
In ACM CGIT, 1987. 14
[32] G. Loubet, Holzschuch, and W. Jakoh. Reparameterizing Discontinuous Integrands for Differentiable Rendering. ACM TOG, 2019.
[33] A. Mai, D. Verbin, F. Kuester, and S. Fridovich- Keil. Neural Microfacet Fields for Inverse Rendering. In ICCV, 2023. 2,
[34] R. Martin-Brualla, N. Radwan, M. S. M. Sajjadi, J. T. Barron, A. Dosovitskiy, and D. Duckworth. NeRF in the Wild: Neural Radiance Fields for Unconstrained Photo Collections. In CVPR, 2021. 4, 5,
[35] Mcrlin Himicr-David and Sébastien Speierer and Benoît Ruiz and Wenzel Jakob. Radiative backpropaga- tion: An adjoint method for lightming-aat fast differentiable rendering, ACM TOG, 2020. 3
[36] B. Mildenhall, P. P. Srinivasan, M. Tancik, J. T. Barron, R. Ramamoorthi, and R. Ng. NeRF: Representing Scenes Neural Radiance Fields for View Synthesis. In ECCV, 2020. 1.2
[37] T. Müller, A. Evans, C. Schied, and A. Keller. Instant neural graphics primitives with multiresolution hash encoding. ACM TOG, 2022. 14
[38] J. Munkberg, Hasselgren, T. Shen, J. Gao, W. Chen, A. Evans, T. Müller, and S. Fidler. Extracting Triangular 3D Models, Materials, and Lighting From Images. In CVPR, 2022. 2. 6, 9, 17
[39] M. Oechsle, S. Peng, and Geiger. UNISURF: Unifying Neural Implicit Surfaces and Radiance Fields for Multi- View Reconstruction. In ICCV, 2021. 14
[40] M. Pharr, W. Jakob, and G. Humphreys. Physically Based Rendering: From Theory to Implementation. MIT Press, 2023.
[41] P. Phongthawee, W. Chinchuthakun, N. Sinsunthithet, Raj, V. Jampani, Khungurn, and S. Suwa- janakorn. Diffusionlight: Light probes for free by painting a chrome ball. In CVPR, 2024.
[42] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin,
J. Clark, G. Krueger, and I. Sutskever. Learning Transferable Visual Models From Natural Language Supervision. In /CML, 2021. 15, 16
[43] R. Ramamoorthi and P. Hanrahan. A Signal- Processing Framework for Inverse Rendering. In ACM CGIT. 2001. 3, 4
12

---

[44] P. Ren, J. Wang, J. M. Snyder, X. Tong, and B. Guo. Pocket Reflectometry. ACM TOG, 2011. 6
[45] R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer. High-Resolution Image Synthesis with Latent Diffusion Models. In CVPR, 2022. 6, 15
[46] P. P. Srinivasan, B. Deng, X. Zhang, M. Tancik, B. Mildenhall, and J. T. Barron. Nerv: Neural reflectance and visibility fields for relighting and view synthesis. In CVPR, 2021. 2
[47] C. Sun, G. Cai, Z. Li, K. Yan, C. Zhang, C. S. Marshall, J.BB. Huang, S. Zhao, and Z. Dong. Neural-PBIR Reconstruction of Shape, Material, and Illumination. In ICCV, 2023. 2, 6, 7,9
[48] T. Sun, J. T.. Barron, Y-T. Tsai, Z. Xu, X. Yu, G. Fyffe, C. Rhemann, J. Busch, P. Debevec, and R. Ra- mamoorthi. Single Image Portrait Relighting. ACM TOG, 2019. 3
[49] J. Tang, Z. Chen, X. Chen, T. Wang, G. Zeng, and Z. Liu. LGM: Large Multi-Vies Gaussian Model for Nigh-Resolution 3D Content Creation. ArXiv, 2024. 16
[50] The Blender Foundation. Blender 2.93. URL httph:htp//// //www//wwww///// blender hoande..rgg/...... 14
[51] D. Verbin, P. Hedman, B. Mildenhall, T. E. Zickler, J. T. Barron, and P. P. Srinivasan. Ref-NeRF: Structured View-Dependent Appearance for Neural Radiance Fields. In CVPR, 2022. 14
[52] D. Verbin, B. Mildenhall, P. Hedman, J. T. Barron, T. Zickler, and P. P. Srinivasan. Eclipse: Disambiguating Illumination and Materials using Unintended Shadows. In CVPR, 2024. 3, 8
[53] D. Verbin, P. P. Srinivasan, Hedman, B. Mildenhall, B. Attal, R. Szeliski, and J. T. Barron. NeRF-Casting: Improved View-Dependent Appearance with Consistent Reflections. ArXiv, 2024. 14
[54] D. Vicini, S. Speierer. and W. Jakob. Differentiable Signed Distance Function Rendering. ACM TOG, 2022. 3
[55] B. Walter, S. Marschner, H. Li, and K. E. Torrance. Microfacet Models for Refraction through Rough Surfaces. In Rendering Techniques, 2007. 6, 14
[56] F. Wang, M.-J Rakotosaona, M. Niemeyer, R. Szeliski, M. Pollefeys, and E. Tombari. UniSDF: Unifying Neural Representations for High-Fidelity 3D Reconstruction of Complex Scenes with Reflections. In NeurIPS, 2024. 3, 10, 14
[57] P. Wang. L. Liu, Y. Liu, C. Theobalt. T. Komura, and W. Wang. NeuS: Learning Neural Implicit Surfaces by Volume Rendering for Multi-view Reconstruction. In NewrIPS, 2021.
[58] Z. Wang, A. C. Bovik, H. R. Sheikh, and E. Simoncelli. Image quality assessment: from error visibility structural similarity. TIP, 2004. 7
[59] L. Yariv, J. Gu, Y. Kasten, and Y. Lipman. Volume Rendering of Neural Implicit Surfaces. In Neur!PS, 2021. 2
[60] G. Zaal, R. Tuytel. R. Cilliers, J. R. Cock. A. Mischok, S. Majboroda, D. Savva, and J. Burger. Polyhaven: a Curated Public Asset Library for Visual Effects Artists and Game Designers, 2021. 6, 17
[61] C. Zeng, Y. Dong, P. Peers, Y. Kong, H. Wu, and X. Tong. DiLightNet: Fine-grained Lighting Control for Diffusion-based based Image Generation. In SIGGRAPH. 2024. 3, 6, 16
[62] K. Zhang, F. Luan, Q Wang, K. Bala, and N. Snavely. PhySG: Inverse Rendering with Spherical Gaussians for Physics-based Material Editing and Relighting. In CVPR, 2021. 6, 9. 17
[63] I. Zhang, A. Rao, and M. Agrawala. Adding Conditional Control to Text-to-Image Diffusion Models. In ICCV, 2023. 3, 6, 15, 16
[64] R. Zhang, P. Isola, A. A. Efros, E. Shechtman, and O. Wang. The Unreasonable Effectiveness of Deep Features as Perceptual Metric. In CVPR, 2018.
[65] X. Zhang, P. P. Srinivasan, B. Deng, P. E. Debevee, W. T. Freeman, and J. T. Barron. Nerfactor. In ACM TOG, 2021. 6, 7, 7,9. 14, 17
[66] Y. Zhang, J. Sun, X. H. He, H. Fu, R. Jia, and X. Zhou. Modeling Indirect Illumination for Inverse Rendering. In CVPR, 2022. 6, 6,7,911
13

---

Appendix IllumiNeRF: 3D Relighting Without Inverse Rendering
This appendix is organized as follows:
1. Sec. A provides more implementation details;
2. Sec. B details the inconsistent illumination issue on Stanford-OPe
A Additional Implementation Details
A.1 Latent NeRF Model and Geometry Estimator
We use JAX [11] [] implement both the geometry estimator and Latent NeRF model as UniSDF [56]. state-of-the-ans volume rendering approach based on a signed distance function (SDF). The advantage of using UniSDF is that it enables easily extracting mesh from the SDF, which into a standard rendering engine such as Blender [50] in order to compute radiance cues. Additionally, UniSDF decouples geometry from appearance, allowing to fix the weights related to geometry and only optimize for weights that model the appearance. Note that future NeRF/SDF approaches with improved geometric reconstruction can be seamlessly integrated in our method.
then import
Our parameterization of the UniSDF model similar to the one used the original paper for the DTU dataset [1], with four key changes. First, reduce the number of rounds of proposal sampling (as introduced by mip-NeRF 360 [5]) from two to one, using 64 proposal samples. Second, we use the asymmetrie predicted normal loss from NeRF NeRF-Casting [53]:
Cp Σ (Aiws xnlA++ AwWailnn - -X1i11 Aa Nn. - m((l2),
(SI)
where wi is the volume rendering weight of the i-th sample, V denotes the noo-ggaddeett operator, n, and n/ are the i-th sample's density normals and predicted normals respectively (see [51]), and we set X = A22 = 10-3, À 10-2. Third, like NeRI-Casting [53]. we use an additional hash grid encoding [37] with 15 scales between resolution of 32 and 4096, used only for outputting predicted normals. Fourth, we further encourage the local smoothness of the predicted normals n' by using
smoothness loss similar to [39, 65]:
L. = Ao Σ willn'(x + n2xxx)1?
(S2)
where x; is the 3D position of the i-th sample, and E ~ NN(0, is an isotropic Gaussian random variable used to perturb the sample locations. We set \ 0.1 and 0.01.
We find that these modifications result better and smoother geometry necessary for our model's ability to relight objects with specular highlights.
Finally, to incorporate the GLO embeddings, we utilize MLP predict element-wise scale and shift value to be applied to the 'bottleneck' feature of UniSDE, similar to AffineGLOO in Zip-NeRF [6].
For both geometry estimation and latent NeRF optimization, utilize the Adam [24] optimizer with 3 = 0.9, 32 = 0.99, and = x 10-15, 15, We decay our learning rate logarithmically from 5 x 10-3 to x 10 over 25k training iterations with cosine-scheduled warmup in the first 500 steps.
A.2 Radiance Cues
Geometry To extract radiance cues we first optimize UniSDF [56] on the input images. After optimization, we convert the SDF representation to mesh using marching cubes [31] with threshold set to be zero.
We use Blender Cycles [50], physically- -based path-tracer render the radiance cues. We run Blender via the Kubric python wrapper [16], and we use the estimated geometry with the predefined materials based on the GGX material model [55], described in Sec. 3.4.
Rendering
14

---

(a) w/o smoothness.
(b) w/ smoothness.
Figure S1: Effects of shading normal smoothing function.
Emply String
Latent Noises
Frozen Decoder
Frazen Buse
Diffision Model UNet
Relit Image
ConyNet 4
Trainable Copy
ComoNer
Rase Diffision
Model UNet
ZeroCanv
Encoder
Middle Blocks
Radiance Cues
CoavNet 2
ConvÑel
Given Image Mask
Figure $2: Schematics of our ControlNet-based diffusion model.
Shading Normals In order to produce smoothly-varyine specular highlights which look realistic, we need the normals used for shading to be smooth. By default, Blender computes normals for shading based on the input geometry, which may be noisy. To mitigate this, we can feed the predicted normals n' described in Sec. A.I to Blender and enable its shading normal smoothing function which applics to the predicted normals, and uses them for shading. However, over-smoothnes may harm the photorealism of the rendered shadows. See Fig. S1 for qualitative comparison radiance cues rendered without enabling the shading normal smoothing (Fig. Sla) and with the feature enabled exploit hybrid strategy: utilize radiance
(Fig. S1b). In implementation, smoothness for the diffuse material and use radiance cues with smoothness for the specular materials. Concretely, our final radiance cues composed of the first rendering in Fig. Sla and the right three ones in Fig. S1b.
without
A.3 Relighting Diffusson Model
We implement relighting diffusion model in JAX [11]. We illustrate the architecture of the model for inference in Fig. S2. We build upon a ext--to-mmage latent diffusion model which is similar to the model of Rombach et al. [45]. It denoises gaussian noise of size 64 x 64 x and decodes the output latent features into relit image of size 512 x 512 x 3. The model was not conditioned on text input, receiving only empty strings via CLIP text encoder [42]. During training the base model is frozen.
Following ControlNet [63], we create a trainable copy of the base diffusion model's UNet encoder and middle blocks and append them with ZeroConv-ben based blocks the frozen base model. The
15

---

Table S1: Fig. S2's ConvNet Structure. Con- volution layer's definition is represented as (ker- nel size, stride, padding). We use SiLU [19] as the activation function between layers. Layer 8 uses zero initialization while the other layers use Flax's 's [18] default initialization', In our imple- mentation, we have H = W = 512.
Table S2: Fig. S2's ConvNet 3 Structure. Con- volution layer's definition is represented as (ker- nel size, stride, padding). We use SiLU [19] as the activation function between layers. Layer 5 uses zero initialization while the other layers uses Flax [18] default initialization', In our im- plementation, we have H = W = 512.
Index
Layer
Output Shape
(input)
H x W x
(3. 1)
H x W x 16
2-1
(3, 11 1)
H W x 1
Index
Layer
Output Shape
2-2
(3,2, 1)
H/2 x W/2 x 32
0 (imput)
H x W x 12
3-1
1.)
H/2 X W72 x 32
(3. 1)
H W
3-2
(3.2.1)
H/4x W/4 64
2-1
(3. 1)
H W x 16
4-1
1,,
H/4x W/4 64
2-2
23
H/2 x W/2 32
4-2
(3.2.1)
H/8 X W/8 X 128
3-1
13,
H/2 x W/2 32
5-1
1,
H/8 X W78 128
3-2
(3.2,
H/4x W/4 x 96
5-2
13,2,
H/16 W/16 256
4-1
(3, 1,
H/4 X W74 x 96
6-1
(3, 1,
H/16 X W/16 256
4-2
13,
H/8 X W/8 256
6-2
13.2,1)
H/32 X W/32 x 512
5
1,,
H/8 X W/8 x 320
7-1
(3.
H/32 W/32 512
7-2
1),2,
H/64 W/64 512
8
13,
H/64 X W/64 x 1024
9
flatten
(H/64 x W764) x 1024
given masked image and radiance cues are first fed through ConvNet (see Fig. 4 in [61] for details) and ConvNet (see Tab. S2). The resulting output is added to the output of the latent noise, which is fed through ConvNet 4. ConvNet 4 consists of single convolution layer with kernel size 3, stride 1, padding 1, and 320 output channels. Given that the trainable copy was designed for tokenized text input, the masked image is first fed through ConvNet (see Tab. S1) to generate representative emheddings. To ensure compatibility between the output of ConvNet (size 64) and the CLIP [42] encoder's 's text output shape, zero-valued tensors are appended, increasing the size 77.
We train the diffusion model using an approach similar to ControlNet [63], with a large dataset of synthetic objects rendered under multiple lighting conditions. Each training example for fine-tuning consists of a pair of images that view the same object with the same camera parameters, illuminated by two different environment map (see Sec. A.4). We fine-tune the diffusion model to predict one of these two images, given the other image as well as the corresponding radiance cues rendered using the synthetic object's 's geometry. Note that for synthetic objects, do not need to estimate the geometry G nor to enable the Blender normal smoothing function to compute the radiance cues since we already have the ground-truth meshes and the normals from synthetic objects are smooth enough. We fine-tune the base model for 150k steps using batch size of 512 examples and a learning rate of 10-4, which is linearly warmed up from 0 over the first 1 steps. The fine-tuning takes around 2 days on 32 TPUv5 chips. Besides, we always use the empty string as the text input to effectively make the fine-tuned model image-based.
At inference time, we use the DPPM scheduler [21] without classifier-free guidance [20] to produce samples at 512 512 resolution.
A.4 Training Data Processing
We use Objaverse [13] as the synthetic dataset. To filter out low-quality objects, we use the list from [49] to get our initial set of 156.330 ones.* By additionally removing (emmi))ransppreen ones, we have a final set of 152,649 objects. If the object only contains geometry, we manually assign a homogeneous texture (ShaderVode
with a color uniformly sampled from [0. 1 Further, if the object does not have the material information, we assign it Blender Glossy BSDF
haccper//gttabb.co//g/////////// http:///wwooo.oooooaaa/a///////a 22#
'https://githur http htw.aa/www....aao.....eeeeeeeeea
16

---

Figure S3: Stanford-ORB's per-image illuminations from teapot teapot_ scene002: inconsistent sun and tripod shape/location.
Table S3: Issues illuminations of Stanford-ORB [27]. We create two sets of reference renderings with the ground-truth geometry and material: 1) using fixed illumination to render evaluation images for a same (object, lighting) pair; and 2) using the per-image illumination provided by the benchmark. We then evaluate each approach's 's renderings with respect to the two sets of reference renderings respectively. We also list each approach's 's illumination selection in the second column. For each row, better performance between the two evaluations is highlighted Apparently and consistently, the numerical results favor matched illumination selection.
Illumination
Reference Fixed Illumination
Reference Per-image Illumination
Selection
PS PSNR-L1 SSIMT LPIPS,
PSNR-H PSNK-LT
SSIM
LPIPS
PhySG [62]
Per-image
22.55
28.05
0.959
0.056
22.71
28.19
0.959
0.055
NVDiffRec [38]
Per-image
23.47
29.35
0.960
0.037
23.71
29.60
0.960
0,037
NeRD [10]
Per-image
24.05
30.20
0.968
0.053
24.22
30.36
0.969
0,053
NeRFactor [65]
Per-image
24.38
30.70
0.970
0.049
24.55
30.85
0,970
0,048
InvRender [66]
Per-image
24.50
30.75
0.970
0.047
24.68
30.93
0.971
0,046
NVDiffRecMC [17]
Per-image
25.17
31.19
0.970
0.037
25.45
31.53
0,970
0,036
Ours
Fixed
26.29
32.45
0.973
0.029
26.00
32.11
0.973
0.029
Ours (single GPU)
Fixed
26.34
32.53
0.974
0.029
26.05
32.17
0.973
0.029
Real Images
26.25
32.69
0.975
0.024
26.73
33.27
0.977
0.025
material (ShaderlodeBsdffl.ossss) whose roughness value is uniformly sampled from [0.02, 0.5] and base color is set to be the same as the homogeneous texture. The mixing factor between the specular and diffuse materials (Shaderloode::xxShadeer is uniformaly sampled from [0. 1].
As we discussed in Sec. A.3, our diffuuion training requires image pairs under different lightings. For this, we select 509 equirectangular environment maps from [60]. For each object, we sample four camera poses sphere centered around it. For each camera, we randomly sample two environment maps and augment them with random horizontal shift, vertical flip, and RGB channel shuffle. We then use Blender's s Cycle path tracer to render an image of resolution 512 x 512 with 512 samples per pixel for each environment map using a camera whose focal length set to 512.
B Stanford-ORB Illumination Issues
Stanford-ORB provides estimated per-image illumination via moving light probe for each image, see Fig. S3 for an example. Ideally, fixed illumination per object would match reality, but aligning the object and light probe is challenging. This limitation in the Stanford-ORL benchmark can significantly affect results, especially in areas with specular highlights as demonstrated in Tab. S3. Consequently, there is no "correct" way to do relighting in Stanford-ORB. our results use fixed illumination, while competitors use per-image illumination.
We rendered the ground truth geometry and materials, obtained by Stanford- ORB in a controlled stu- dio environment (see [27]'s Sec.3.2.2). Each view was rendered under fixed illumination (consistent environment map) and per-image illumination (unique environment map per view). Fig. S4 shows both renderings alongside the corresponding real image. Note the significant variations, especially in areas with specular highlights (see marked regions and PSNR).
17

---

32.62
33.48
29.87
31124
28.19 33.38
32.50
24.42
30.72
31.50
Figure S4: For each object, from left to right, we show 1) reference rendering w/ fixed illumination: 2) reference rendering per-image illumination" (see Tab. S3 caption for details); and 3) the real captured image. We show PSNR-L between 1) vs. 3) and 2) vs. 3) respectively. Different illumination settings vary significantly.
We computed metrics for each method using both reference renderings as ground truth whose quantative results are in Tab. S3. Our method excels under fixed illumination but performs worse with per-image illumination. Competitors show the opposite trend, doing better with per-image illumination. Neural-PBIR did not release code or complete results, thus it missing from the table.
18

---

NeurIPS Paper Checklist
The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility. transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and precede the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No], or [NA].
[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justifiiaton right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.
The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation.
While "[Yes] " is generally preferable to "[No]", ", it is perfectly acceptable to answer "[No] provided proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering "[No] " or "[NA] is not grounds for rejection. While the questions are phrased in binary way, we acknowledge that the true answer often more nuanced, so please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS paper checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
The claims about performance are verified by an extensive quantitative and qualitative analysis described in the experiments section.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes]
19

---

Justification: We discuss the limitations of our method in the limitations section. Most notably our method bakes in material and target lighting into the NeRF, i.e. for each target light our method needs to optimize new NeRF model.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well vell-specification asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faireess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [NA]
Justification:
We do not have any theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We attempted to provide all necessary information to reproduce the results. We provide detailed insights into dataset and radiance cue generation, model architectures, hyperparameters and model training in the supplemental.
20

---

Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-soneee open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Justification: We have not made the code model weights available online, however, the Objaverse dataset is publicly available as well as the datasets required for the Stanford-ORB and TensoIR benchmarks.
Guidelines:
The answer NA means that paper does not include experiments requiring code. Please see the NeurIPS code and data submission guidelines (https://nips.cm cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https:
/ //nips.co. wttpcccccppcccceeeeeee.ec. gwwgge/dd/ee/eeee h /CodeSubmissiiiiieecee sssssssonolllo sesiinnyocc)) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
21

---

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We provide information about model training, hyperparameters, optimizer and learning rates in the appendix. For the evaluation we use the data splits defined by the benchmarks.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: We evaluate our method against the most popular benchmarks which do not report enror bars.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the eтror bar is the standard deviation or the standard error of the mean.
OK to report I-sigma error bars, but one should state The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: For both diffusion model training and NeRF optimization we provide the compute requirements in the appendix.
22

---

Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compule required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https: httpp/////////p.....ee// http://wrrrs..cpppiiiiiicc ctw.iiiiiiheeelleecccccccecacaee ines?
Answer: [Yes]
Justification: To the best of our knowledge follow the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
Broader Impacts
Question: Does the paper discuss both potential positive socictal impacts and negative societal impacts of the work performed?
Answer: [No]
Justification: We propose novel paradigm for existing task - 3D relighting. If reviewers believe our method could cause potential harm, we are happy to include a statement.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (c.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
23

---

Answer:
[No]
Justification:
We do not plan to release our trained models.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging. and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We cite the works that provide the assets used for data set generation, as well the use of pre- trained model weights in the appendix.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. If this information is available online, the authors are encouraged reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and the documentation provided alongside the assets?
Answer: [Yes]
Justification: Yes, we explain how we render 3D assets in the appendix section. Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and Research with Human Subjects

---

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
We did not conduct research with humans or crowdsourced any tasks.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification: No studies were undertaken.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
25


Reviews:
[{"review_id": "Hio0z34DLT", "reviewer": "Reviewer_Ma7D", "summary": "The paper proposes a new method called illumiNeRF for 3D relighting — given a set of posed input images under an unknown lighting, illumiNeRF produces novel views relit under a target lighting. Most of existing methods use inverse rendering to first recover the material and lighting in the given images, and apply new lighting to the object. The authors point out there is inherent ambiguity in decomposing materials and lighting. Instead, they propose a probabilistic model of representing multiple possible materials as latents, and train a latent NeRF using a relighting diffusion model. The authors compare their method with existing baselines and outperform most of them.", "questions": "It would be great to show 3D renderings with different latent explanations in Figure 4(b).", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- The problem formulation in sec 3.1 is explained clearly. The authors point out the inherent ambiguity in inverse rendering problems when decomposing materials and lighting.\n- The probabilistic model of representing multiple possible materials as latents in NeRF is very interesting.", "weaknesses": "- Although during training, multiple materials are considered as different Z when optimizing the latent-NeRF, Z is manually set to 0 at test time, which corresponds to a specific explanation. However, Z=0 does not seem to represent the most likely material because the authors assume a uniform prior over Z according to line 163.\n- From Figure 6, the results lose a lot of details compared to other baselines like Neural-PBIR. For example, the method lost details of the spikes in the cactus sample. In table 2, Neural-PBIR also outperforms the proposed method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## Why is Z = 0?\n\nIt is unclear how we could use the optimal Z that best matches the actual material besides optimizing Z using the test set images, which does not seem fair.  We found that setting Z to 0 yields good results across both our synthetic and real-world benchmarks (see Tab. 1 and 2, and Fig. 5 and 6). We'll clarify this in the final version of our paper.\n\nWe provide more qualitative results for different latent codes in Fig. 1 (see PDF). These results demonstrate that the optimized latent codes effectively capture various plausible explanations of the materials.\n\n## Result quality in Fig. 6\n\nOur method consistently surpasses all inverse rendering approaches in capturing high-frequency details, except for Neural-PBIR. Notably, for the salt can (column 2), our method clearly recovers the ingredient list, which most baselines miss. Similar improvements are evident for the toy car (column 6) and gnome (column 7).\n\nCompared to Neural-PBIR, our results often match or surpass its fine details, see the teapot (column 1), salt can (column 2), toy car (column 6), and gnome (column 7). The cactus (column 4) is an exception, due to UniSDF's inability to produce high quality geometry, as noted in the main paper (L251).\n\nOur method's results will inherently improve with advances in geometry reconstruction, as our RDM is trained on perfect synthetic geometry. Neural-PBIR's code is unavailable, but its isolated geometry reconstruction stage (Sec. 3.1 in [39]) would allow us to directly leverage better geometry.\n\n### About more examples for Fig. 4 (b)\n\nWe provided more examples in Fig. 1 of the rebuttal PDF. We are happy to include these in a final version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "e80PkyxNM3", "reviewer": "Reviewer_hgwb", "summary": "The paper proposes a method to relight NeRF representations from a set of multi-view input images. The paper first trains a NeRF model and recovers geometry using UniSDF from the input images. Then, from the mesh, they obtain radiance cues which are used to condition a diffusion model to sample relit images of the scene under a target lighting. Since these relit images are samples from a distribution of plausible relit images, the sampled results are used to train a latent code conditioned NeRF model, and the lighting of the scene can be manipulated by walking through the latent space. The core contribution of the paper is to use the relit images rather than trying to explicitly model and learn the lighting and material properties.", "questions": "Many questions are discussed in the weaknesses, specifically:\n1. How sensitive is the paper to the UniSDF reconstruction pipeline? Are there better methods which exist? Why does the paper only work for objects?\n2. How does the normalization to ground-truth images for visualization of relit images actually affect the quality?", "limitations": "Some limitations are discussed in the paper, but there are additional ones which should be discussed in more detail, such as the long reconstruction time to get new lighting conditions.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "1. The paper shows that instead of recovering the material properties, it is enough to relight the training images and optimize the appearance to reach state-of-the-art results. The comparisons demonstrate that the results are state-of-the-art, which is important for improving the task of object relighting.\n2. The diffusion model trained is described in detail and seems novel for the task of relighting. Details on the radiance cues and the architecture and training process of the diffusion model are described in detail. The model can be trained on synthetic data and applied to real-world data.\n3. The lighting is multi-view consistent with the help of the latent-NeRF model\n4. The presentation of the paper is clear, description of the method is high-quality, and comparisons are detailed", "weaknesses": "1. The significance of using diffusion models to relight images, which are then used to train a NeRF, is limited to the context of application to NeRFs. Concurrent works have very similar methods in the context of Gaussian splatting (https://repo-sam.inria.fr/fungraph/generative-radiance-field-relighting/) - what is the benefit of this approach over those? Some discussion should be made on this.\n2. The pipeline seems brittle, requiring many steps to go right and limited by the sum of limitations of many different technologies. The quality of relighting is not only limited to the quality of the diffusion model output but also depends on the quality of the UniSDF mesh reconstruction, which is required to create the radiance cues. This means that a good capture is a necessary prerequisite to relight with this method, and one could simply relight the extracted mesh to achieve a similar effect.\n3. I find the abstract a bit misleading: “we first relight each input image using an image diffusion model conditioned on lighting and then reconstruct a Neural Radiance Field (NeRF) with these relit images, from which we render novel views under the target lighting.” This description doesn’t accurately represent what actually happens, as it leaves out the most crucial part of the pipeline (reconstruction + conditioning) and instead suggests that the NeRF can be directly reconstructed from the relit images (Compare figure 2 of the paper).\n4. The choice of UniSDF is neither justified nor are alternatives discussed. The part describing this aspect of the approach is relegated to the appendix. Consequently, there is no ablation study to determine if other reconstruction methods might work better.\n5. With respect to figure 6: It makes sense to highlight the same area for all depicted approaches. For instance, the blocks and the ball have no highlights, but other approaches seem to perform better in these areas. Additionally, the normalization technique used to match the ground truth images is not ablated. Does this make a difference in the reconstructed quality? I notice that many of the other methods are darker than I’d expect and I wonder if this introduces some artifacts which hurt these methods and promote the proposed method.\n6. It is unclear why the paper limits itself only to objects. UniSDF clearly shows good performance for scenes. The authors don’t discuss this limitation in the paper.\n7. Changing the target light even slightly requires a wait of 45 minutes for a final rendering. Other approaches, as referenced above, seem to not require that much time.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## Method is limited to NeRF\n\nOur approach introduces a novel 3D relighting paradigm, employing a Relighting Diffusion Model (RDM) as prior to optimize a relit 3D representation. This approach is adaptable to various 3D representations like NeRFs or Gaussian Splats, since both can be conditioned on latent embeddings. For example, Wild Gaussians [a] (Sec 3.2) uses latent embeddings for a similar purpose in a different context (modeling lighting variations in internet photo collections with Gaussian Splatting). \n\n[a] Kulhanek et al., WildGaussians: 3D Gaussian Splatting in the Wild. ArXiv 2024\n\n## Advantages over concurrent work\n\n1. The concurrent work focuses on forward-facing indoor scenes, while ours addresses full 360 degree 3D object-centric relighting.\n2. Our method can handle intricate environment map-based lighting, whereas the concurrent work is limited to single-source directional illumination, a subset of what IllumiNeRF supports.\n3. The concurrent work’s diffusion model relights images only to 18 known light directions in the dataset (Sec. 3.2).\n4. Their latent code is per-view, while ours is per-sample (See Eq. 3 and Fig. 8)\n\nNote that EGSR (July 3-5) took place after the NeurIPS submission deadline (May 22).\n\n## The pipeline seems brittle\n\nOur presented numerical metrics and qualitative visualizations either produce SOTA results (Tab. 1 and Fig. 5) or highly compelling results (Tab. 2 and Fig. 6). \n\nWhile we acknowledge that our approach relies on high-quality geometry estimates from UniSDF (L249), it is crucial to note that our diffusion model's training relies solely on synthetic data. This design choice completely decouples the training process from the inference pipeline depicted in Fig. 2. Consequently, our method will benefit from future improvements in geometry reconstruction as well as improvements with regard to the relighting model or the latent NeRF optimization. \n\n## Simply relighting the extracted mesh\n\nWe are unsure exactly what the reviewer is suggesting. \n\nIf the reviewer suggests simply relighting the output from UniSDF, it's important to emphasize that UniSDF generates only geometric information and outgoing radiance, not material properties. Therefore, direct relighting of the mesh is not possible. \n\nIf the reviewer proposes to perform inverse rendering based on the geometry predicted by UniSDF, we would like to clarify that one of the main benefits of our method is that it does not require inverse rendering (L42 - 46).\n\n\n## Abstract is misleading\n\nThank you for the suggestion! We will modify the final version of the abstract to include reconstruction and conditioning.\n\n## Use of UniSDF is not justified\n\nUniSDF is a SOTA method for surface/SDF-based NeRF approaches (L387) on the Shiny Blender benchmark [43] (see MAE in [47]’s Tab. 1). We build on top of it, as other non-surface methods would make it harder to obtain high-quality geometry for the radiance cues computation (L249, 388 - 390). Our framework can easily be applied to any NeRF/SDF approaches that recover better geometry in the future.\n\n## About highlights in Fig. 6\n\nWe will integrate the changes as suggested.\n\n## About normalization for Fig. 6\n\nThe qualitative baseline results were kindly provided by the authors of Stanford-ORB [21]. Additionally, we utilized the benchmark's official code (https://github.com/StanfordORB/Stanford-ORB/blob/962ea6d2cc/scripts/test.py/#L36) to perform the normalization.\n\n## Why focus on objects?\n\nGood question. There are multiple reasons for focusing on objects:\n1. Object-centric 3D relighting is vital in various downstream applications, e.g., AR/VR, robotics, and game development.\n2. Scene-centric lighting is a much harder task to solve. It is hard to set up realistic illumination for scenes. Conditioning on lighting is non-trivial as lighting representations such as environment maps are not easily applicable to scenes. \n3. Established synthetic and real-world object-centric benchmarks like TensoIR and Stanford-ORB enable clear evaluation and comparison.\n4. Large-scale object-centric synthetic datasets like Objaverse [10] are readily available for training the relighting diffusion model.\n\n## Efficiency of method\nWe acknowledge that our approach currently prioritizes quality over real-time performance, as generating new samples with the RDM and optimizing a NeRF for each new lighting condition is computationally intensive (L252-254).\n\nHowever, we believe this focus on quality is crucial as a first step. Our extensive evaluations and thorough numerical experiments demonstrate competitive results, offering a \"promising alternative for relighting\" (mZi1). We hope our work \"inspires future works using diffusion models to hallucinate lighting\" (q4ex), with a gradual focus on improving efficiency.\n\nWe see parallels in the NeRF community, where the initial MLP-based NeRF [30], despite its impressive results, was computationally demanding. Subsequent research, like instant-NGP [31] and Gaussian Splatting, significantly improved efficiency. We envision a similar trajectory for our approach.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "rbOjuVQkYB", "reviewer": "Reviewer_q4ex", "summary": "The paper proposes a method for novel view synthesis with novel lighting given multi-view observation of an object. The pipeline is composed of a learned single-image-based relighting model based on image diffusion models, and a latent NeRF model that reconstructs the appearance of the relit object by reconciling relit images across views and samples. The relighting diffusion model (RDM) is conditioned on radiance cues generated from the geometry and target lighting under samples of roughness values, using a finetuned ControlNet architecture. The method is evaluated on both synthetic and real-world scenes, against baseline methods (mostly object-centric inverse rendering models), with both qualitative and quantitative results. Ablation study is also provided on design choices (e.g. latent/no latent NeRF, and number of diffusion samples).", "questions": "[1] Quality of the sampled results from RDM. Why are some results look overly specular (e.g. the bread in the hotdog scene in Fig. 4). And if more examples can be provided on results from RDM?\n\n[2] Reconciliation by the latent NeRF. Does the Latent NeRF cost details and specularity in sampled results from RDM when reconciling the results into a NeRF representation? Also if there are view-dependent effects in the sampled results, does the latent NeRF properly preserve those effects? Again, more examples are to be provided in order to further examine the behavior of the two modules beyond one single scene (and especially for synthetic scenes, where 'true' ground truth of relit objects is available). There are additional results on Standford-ORB scenes in the Supp but analysis is needed in the main paper to support the method using those results.\n\n[3] Additional ablation. Is it helpful to use more or fewer roughness levels in generating radiance cues?\n\n[4] In designing baselines, why not comparing against ALL methods in both synthetic and real-world settings?", "limitations": "NA", "rating": 4, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "[1] An alternative approach to object-centric relighting without optimization-based inverse rendering. The main novelty of the method which distinguishes it from previous methods of the task is, the method leverages a learned image-based diffusion model to directly predict the relit images, and then reconstructs 3D representation of the relit object for novel view synthesis. This approach opens up a new venue for solving the task besides baseline methods. The diffusion model is conditioned on 'radiance cues' as an input lighting proxy, which itself is not new in a learning-based inverse rendering setting, but the method is able to retarget the lighting proxy as condition for a diffusion model, which might inspire future works using diffusion models to hallucinate lighting.\n\n[2] The design of the Latent NeRF to reconcile sampled relit results from the diffusion model. The paper demonstrates that the latent NeRF is able to better reconcile sampled results from a diffusion model while preserving the details and decent specular highlights.\n\n[3] Extensive evaluation. The paper is able to include extensive evaluation on both real-world and synthetic datasets on various metrics.", "weaknesses": "[1] Effectiveness of the latent NeRF. The paper claims that the latent NeRF is able to reconcile difference in the sampled results from RDM. However, insufficient analysis is provided for justification. For instance, in Fig. 4, results in (a) seem to present more specularity than what is actually present in the hotdog scene (in other words, RDM seems to produce unrealistic relit images), and results from the latent NeRF in (b) seems to be an averaged version of the samples, with most of the specularity averaged out, arriving at a mostly diffuse appearance except for areas where specularity all agree among sampled. As a result, in Fig. 6, when compared to inverse rendering methods like Neural-PBIR, despite that the proposed method is able to produce better specularity in highlighted areas (thanks to explicitly conditioning on radiance cues), the results of the proposed method look 'dull', where shadows and high-frequency details seems to be lacking. This casts questions into the performance of both RDM to produce consistent and photorealistic relit images, as well as the ability of the latent NeRF to successfully reconcile inconsistency in the sampled results without sacrificing details and shadows.\n\n[2] Claim on better secularity in the results is questionable. The paper explains that, when compared to Neural-PBIR the proposed method is in second place numerically because the approximate lighting ground truth from the light probe unfairly favors Neural-PBIR despite the proposed method produced more convincing specularity. However this is largely unjustified, as (1) it is not clear how much weight is on the specular pixels in the overall metrics (2) whether the proposed method actually produces better specularity in the absence of true ground truth of relit objects. Moreover as mentioned in [1], the proposed method seems to lose more details (e.g. in the spikes of the cactus, or details in the ball). In this case, even if the methods excels in specularity, this is not the single most important property that is accounted for in those metrics, or visually, when evaluating the quality of the results.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## Fig. 4.(a)’s generations are unrealistic\n\"More specularity than what is actually present in the hotdog scene\" does not imply unrealistic results. Without known source lighting, the RDM samples from the entire distribution of plausible relighting outcomes. We showcased random samples from this distribution, some of which exhibit higher specularity on the hotdog. However, due to ambiguities between materials and lighting (L127), even these seemingly less-likely images are plausible. The subsequent latent-NeRF optimization process focuses on finding the most probable results.\n## Averaged results in Fig. 4.(b)\nThis outcome is to be expected: with a sufficiently large sample size, we accurately capture the underlying material distributions. Latent-NeRF maintains consistent appearance across all samples while mitigating inconsistencies. As shown in Fig. 4(b), it preserves the plate's specular effects seen in most samples from Fig. 4(a) while reducing the bread's glossiness, keeping mild specular effects on the sauces.\n## Lack of detail in Fig. 6\nOur method surpasses all inverse rendering approaches in capturing high-frequency details, except for Neural-PBIR. Notably, for the salt can (column 2), our method clearly recovers the ingredient list, which most baselines miss. Similar improvements are evident for the toy car (column 6) and gnome (column 7).\n\nCompared to Neural-PBIR, our results often match or surpass its fine details, see the teapot (column 1), salt can (column 2), toy car (column 6), and gnome (column 7). The cactus (column 4) is an exception, due to UniSDF's inability to produce high geometric quality, as noted in the main paper (L251).\n\nOur method's results will inherently improve with advances in geometry reconstruction, as our RDM trains on perfect synthetic geometry. Neural-PBIR's code is unavailable, but its isolated geometry reconstruction stage (Sec. 3.1 in [39]) would allow us to directly leverage better geometry.\n\n## Our results lacking shadows in Fig. 6\n\nOur method consistently produces shadows that accurately match the ground truth. Although Neural-PBIR shows more pronounced shadows on the toy car (column 6), these deviate significantly from the ground truth. \n\nShadows depend on geometry and illumination, modeled through our radiance cues, leading to realistic shadow placement (Fig. 4.(a)). This accuracy is maintained through Latent-NeRF optimization (Fig. 4.(b)).\n\n\n## Consistency of samples from RDM\n\nTo clarify, we do not claim that our RDM generates \"consistent\" relit images. As our RDM performs single-image relighting, we anticipate diverse relit images across samples for the same view, as well as across different views (see Fig. 4.(a)). The role of the latent-NeRF is to reconcile these inconsistencies into a cohesive 3D relit result.\n\n\n## Questioning claim that light probe unfairly favors Neural-PBIR\nWe appreciate this valid concern and would like to clarify. \n\nStanford-ORB provides estimated **per-image** illumination (L235 and [21]’s Fig. 4 and 5), moving a light probe for each image (PDF’s Fig. 2). Ideally, fixed illumination per object would match reality, but aligning the object and light probe is challenging. This limitation in the Stanford-ORB benchmark can significantly affect results, especially in areas with specular highlights (PDF’s Tab. 1). Consequently, there is no “correct” way to do relighting in Stanford-ORB: our results use fixed illumination, while competitors use per-image illumination.\n\nWe rendered the ground truth geometry and materials, obtained by Stanford-ORB in a controlled studio environment ([21]’s Sec. 3.2.2). Each view was rendered under **fixed** illumination (consistent environment map) and **per-image** illumination (unique environment map per view). Fig. 3 in the PDF shows both renderings alongside the corresponding real image. Note the significant variations, especially in areas with specular highlights (see marked regions and PSNR).\n\nWe computed metrics for each method using both reference renderings as ground truth (Tab. 1 in PDF). Our method excels under fixed illumination but performs worse with per-image illumination. Competitors show the opposite trend, doing better with per-image illumination. \n\nNeural-PBIR did not release code or full results, limiting comparisons to 8 images in [39]’s Fig. 10, where it also performs better with per-image setup. We outperform Neural-PBIR in most metrics:\n|8 images|PSNR-H $\\uparrow$|PSNR-L $\\uparrow$|SSIM$\\uparrow$|LPIPS $\\downarrow$| \n|----|-:|-:|-:|-:|\n|[fixed] Neural-PBIR |24.61|30.70|0.966|0.032| \n|[fixed] Ours |25.19|31.33|0.968|0.030| \n||\n|[per-image] Neural-PBIR |25.03|30.97|0.966|0.032| \n|[per-image] Ours |25.04|30.76|0.967|0.030| \n\nComputational limitations prevented us from employing per-image illumination, as it would have required optimizing a latent-NeRF for each per-image illumination. Given the trends observed for competitors, we believe our method would demonstrate improved performance if this were feasible.\n\n## More qualitative results for Fig. 4\n\nSee Fig. 1 in the PDF.\n\n## About view-dependent effects\n\nOur latent-NeRF can maintain view-dependent properties corresponding to each generation. Please refer to the cactus (4th row), pepsi (5th row), grogu (6th row), and teapot (7th row) scenes in Fig. 1 in the PDF.\n\n## Ablation on #roughnesses\n\nWe empirically found that using four radiance cues yielded strong results (Tab. 1 & 2, Fig. 5 & 6). Given the computational cost of training the RDM (L452), we did not conduct further ablations. However, we are happy to provide additional ablation studies in the final version if the reviewer would like us to.\n\n## Why not compare against all baselines?\n\nWe used the official benchmark results from [17] and [21] to ensure authenticity. We also obtained genuine qualitative results directly from the benchmark.\n\nThe only exception is Neural-PBIR, as its code is not publicly available, preventing us from applying it to the TensoIR dataset.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "KaB813xktr", "reviewer": "Reviewer_mZi1", "summary": "This paper introduces a new paradigm for the relightable 3D reconstruction: leverage a Relighting Diffusion Model to generate relit images and distill them into a latent NeRF. The proposed paradigm demonstrates superior performance compared to existing methods, which are based on inverse rendering, on both sythetic and real-world datasets.", "questions": "- How many samples ($S$) were used during the experiments presented in Table 1 and Table 2? \n\n- As this method models the probability of relit images and fixes $Z=0$ during inference, it would be interesting to investigate how the performance varies by sampling different $Z$ values.\n\n- Typo in Line $446$", "limitations": "Limitations have been well discussed.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- The writing is clear and the content is well-structured. The authors provide comprehensive implementation details.\n\n- The paper addresses a long-standing challenge by introducing a novel paradigm, which diverges from conventional inverse rendering methods. This approach offers a promising alternative for relighting.\n\n- Thorough numerical experiments showcase the effectiveness of the proposed method. The results are convincing.", "weaknesses": "One major weakness of this method is its efficiency. Unlike traditional methods that can directly relight a 3D scene once the inverse rendering is performed, the proposed approach requires optimizing a NeRF for each new target lighting condition. Furthermore, the computation of the latent NeRF involves a significant time investment (0.75 hours using 16 A100 GPUs). It would be beneficial to discuss potential solutions for this issue.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## About efficiency\nWe acknowledge that our approach currently prioritizes quality over real-time performance, as generating new samples with the RDM and optimizing a NeRF for each new lighting condition is computationally intensive (L252-254).\n\nA potential solution to mitigate this limitation is to adopt an amortization strategy, similar to [a]. Specifically, we could train a single latent-NeRF conditioned on both latent codes and a latent representation of the given lighting. This design would allow us to train a single latent-NeRF model capable of handling multiple target illuminations. As a result, when presented with a new lighting condition, the trained model could directly generate the 3D relit version without requiring further optimization.\n\n[a] Lorraine et al., ATT3D: Amortized Text-to-3D Object Synthesis. ICCV 2023.\n\n## Number of samples used for Tab. 1 and 2\n\nWe use 16 samples per view for all benchmarking evaluations.\n\n## Why is Z = 0?\n\nIt is unclear how we could use the optimal Z that best matches the actual material besides optimizing Z using the test set images, which does not seem fair.  We found that setting Z to 0 yields good results across both our synthetic and real-world benchmarks (see Tab. 1 and 2, and Fig. 5 and 6). We'll clarify this in the final version of our paper.\n\nWe provide more qualitative results for different latent codes in Fig. 1 (see PDF). These results demonstrate that the optimized latent codes effectively capture various plausible explanations of the materials.\n\n\n## Typo\n\nThanks a lot for pointing this out. We will correct it.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank all reviewers for their time and effort spent reviewing the paper.

We are grateful to hear that the reviewers agree that our work “addresses a long-standing challenge by introducing a novel paradigm" (mZi1).

We also appreciate the praise for our "probabilistic model of representing multiple possible materials as latents in NeRF" (Ma7D), the recognition of our work's importance for "improving the task of object relighting" (hgwb), and the potential to "inspire future works using diffusion models to hallucinate lighting" (q4ex).

Furthermore, we are pleased reviewers recognize our work’s “extensive evaluation" (q4ex) and find "the results are convincing" (mZi1).

We are also delighted that all reviewers gave the highest rating for the presentation, describing it as "clear" (hgwb), "well-structured" (mZi1), and "explained clearly" (Ma7D).

Below, we address each reviewer's questions.

Meta-review: All reviewers agree that the idea of leveraging a learned image-based diffusion model to predict relit images for latent NeRF optimization and relighting is novel and interesting. The method also demonstrated superior rendering quality compared to prior inverse rendering methods (except Neural-PBIR) on the TensoIR and Stanford-ORB benchmarks. However, the reviewers also raised concerns about the effectiveness of the learned latent NeRF, noting that the diffusion model seems unreliable compared to dense view conditioning for material estimation, and the overall pipeline appears brittle. AC believes that the diffusion-based solution is fresh and valuable to the community and thus recommends acceptance.
Decision: Accept (poster)