IllumiNeRF: 3D Relighting Without Inverse Rendering

Authors: xiaoming  zhao
pratul p srinivasan
dor  verbin
keunhong  park
ricardo  martin-brualla
philipp  henzler
google  research
google  deepmind
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-03 00:53:18

Abstract: Existing methods for relightable view synthesis --- using a set of images of an object under unknown lighting to recover a 3D representation that can be rendered from novel viewpoints under a target illumination --- are based on inverse rendering, and attempt to disentangle the object geometry, materials, and lighting that explain the input images. Furthermore, this typically involves optimization through differentiable Monte Carlo rendering, which is brittle and computationally-expensive. In this work, we propose a simpler approach: we first relight each input image using an image diffusion model conditioned on target environment lighting and estimated object geometry. We then reconstruct a Neural Radiance Field (NeRF) with these relit images, from which we render novel views under the target lighting. We demonstrate that this strategy is surprisingly competitive and achieves state-of-the-art results on multiple relighting benchmarks. Please see our project page at [illuminerf.github.io](illuminerf.github.io).

Main paper: 
 introduction: 
 Capturing an object's appearance so that it can be accurately rendered in novel environments is a central problem in computer vision whose solution would democratize 3D content creation for augmented and virtual reality, photography, filmmaking, and game development. Recent advances in view synthesis [36] have made impressive progress in reconstructing a 3D representation that can be rendered from novel viewpoints, using just a set of observed images. However, those methodsThe key contribution of our work is a new paradigm for relightable 3D reconstruction that replaces 3D inverse rendering with: generating samples with a single-image 2D Relighting Diffusion Model followed by distilling these samples into a 3D latent NeRF representation. We demonstrate that this strategy is surprisingly competitive and outperforms existing most 3D inverse rendering baselines on the TensoIR [23] and Stanford-ORB [27] relighting and view synthesis benchmarks. 
related work: 
 Our work addresses the task of relightable 3D reconstruction by using a lighting-conditioned diffusion model as a generative prior for single-image relighting. It is closely related to prior work in relightable 3D reconstruction, inverse rendering, and single-image relighting. Below, we review these lines of work and discuss how they relate to our proposed approach.Relightable 3D Reconstruction The goal of relightable 3D reconstruction is to reconstruct a 3D representation of an object that can be relit by novel illumination conditions and rendered from novel camera poses. In scenarios where an object is observed under multiple lighting conditions [12] , it is trivial to render its appearance under novel illumination that is a linear combination of the observed lighting conditions, due to the linear behavior of light. This approach is generally limited to laboratory capture scenarios where it is possible to observe an object under a lighting basis.In more casual capture scenarios, the object is observed under just a single or a small handful of lighting conditions. Existing works typically address this setting using methods based on inverse rendering that explicitly factor an object's appearance into the underlying 3D geometry, object material properties, and lighting that jointly explain the observed images. State-of-the-art approaches to 3D inverse rendering [9, 10, 17, 23, 26, 33, 38, 46, 47] generally utilize the following strategy: they start with a neural field representation of 3D geometry (typically volume density as in NeRF [36] , hybrid volume-surface representations as in NeuS [57] and VolSDF [59] , or meshes extracted from neural field representations) from the input images, equip the model with a representation of surface materials (e.g. spatially-varying BRDF parameters) and lighting, and jointly optimize these factors through a differentiable physics-based rendering procedure [40] . While methods may differ in their choice of geometry, material, and lighting representations, and employ different techniques to accelerate the evaluation of the rendering integral, they generally all follow this same highlevel inverse rendering strategy. Unfortunately, even if the geometry is known, inverse rendering is a notoriously ambiguous problem [43, 52] and many combinations of materials and lighting can explain an object's appearance. However, not all of these combinations are plausible, and incorrect factorizations that explain observed images under one lighting condition may produce glaring artifacts when rendered under different lighting. Furthermore, differentiable physics-based rendering is computationally-expensive as thousands of samples are needed for Monte Carlo estimates of the rendering integral, typically requires custom implementations [2, 3, 22, 28, 32, 35, 54] , and the resulting inverse rendering loss landscape is non-smooth and difficult to optimize effectively with gradient descent [14] .Single Image Relighting Instead of using inverse rendering to recover object material parameters which can be relit with physically-based rendering techniques, we train a diffusion model that can directly sample from the distribution of relit images conditioned on a target lighting condition. This diffusion model is essentially a generative single-image relighting model. Early single image relighting techniques employed optimization-based inverse rendering [4] . Subsequent methods trained deep convolutional neural networks to output image geometry, materials, and lighting [29, 30] , or in some cases, to directly output relit images [48, 7, 8] .Most related to our method are a few recent works that have trained diffusion models for single image relighting. LightIt [25] trains a model similar to ControlNet [63] to relight outdoor images under arbitrary sun positions conditioned on input normals and shading. DiffusionLight [41] estimates the lighting of an image by using a ControlNet to inpaint the color pixels of a chrome ball in the middle of the scene, from which an environment map can be recovered.Most similar to our work is the concurrent method of DiLightNet [61] that focuses on single image relighting. DiLightNet uses a ControlNet-based [63] approach to condition a single-image relighting diffusion model on a target environment map. DiLightNet uses a set of "radiance cues" [15] renderings of the object's geometry (obtained from an off-the-shelf monocular depth network) with various roughness levels under the target environment illumination -as conditioning. Our method instead focuses on 3D relighting, where multiple of images of an object are available. It uses a similar single-image relighting diffusion model conditioned on radiance cues. Unlike DiLightNet which uses geometry from monocular depth estimation to render radiance cues, we use geometry estimated from the input views using a state-of-the-art surface reconstruction method [56] . This allows our model to better model complex light transport effects such as interreflections caused by occluded geometry. 
problem formulation: 
 Given a dataset of images of an object and corresponding camera poses D = {(I i , π i )} N i=1 , the general goal of relightable 3D reconstruction is to estimate a model with parameters θ that when rendered, produces relit versions of the dataset under unobserved target illumination L T . This can be expressed as:EQUATIONwhereD T θ ≜ relight(D, L T , π i , θ), π i N i=1is a relit version of the original dataset under target illumination L T using model θ. Note that Eq. ( 1) only maximizes the likelihood of the original given poses after relighting. However, by using view synthesis, we can then turn the collection of relit images into a 3D representation which can be rendered from arbitrary poses. For brevity, we therefore omit the implicit dependence of D T in θ.This relighting problem has traditionally been solved by using inverse rendering. Inverse rendering techniques do not maximize the probability of the relit renderings, but instead recover a single point estimate of the most likely scene geometry G, materials M , and lighting L (note that this is the "source" lighting condition for the observed images) that together explain the input dataset, and then use physically-based rendering to relight this factorized explanation under the target lighting. Inverse rendering seeks to recover θ IR = (G ⋆ , M ⋆ ), where: The first data likelihood term is computed by physics-based rendering of the estimated model and the second prior term is often factorized into separate handcrafted priors on geometry, materials, and lighting [23, 33, 43] .EQUATIONA relighting approach based on inverse rendering then renders each image I in D corresponding to camera pose π using the recovered geometry and materials, illuminated by the target lighting L T , resulting in relight(D, L T , π, θ IR ). This approach has three main issues. First, the differentiable rendering procedures used to compute the gradient of the likelihood term are computationallyexpensive. Second, it requires careful modeling of light transport which is cumbersome and existing differentiable renderers do not account for many types of lighting and material effects seen in the real world. Third, there are often ambiguities between M and L, meaning that any errors in their decomposition may be apparent in the relit data. It is quite difficult to design effective handcrafted priors on geometry, materials, and lighting, so inverse rendering procedures frequently recover explanations that have a high data likelihood (are able to render the observed data) but produce clearly incorrect results when re-rendered under different illumination.We propose an approach that attempts to maximize the probability of relit images in Eq. (1) without using an explicit physically-based model of the object's lighting or materials. First, let us introduce a latent variable Z that can be thought of as implicitly representing the input images' lighting along with the object's material and geometry parameters. We can write the likelihood of the relit data as:EQUATIONIntroducing these latent variables lets us consider all relit renderings in the dataset, D T i ≜ (I T i , π i ), as conditionally independent, since the rendering under the target lighting L T is deterministic given the object's geometry and materials. This enables writing the likelihood as:EQUATIONWe propose to model this with a latent NeRF model, as used by Martin-Brualla et al. [34] that is able to render novel views under the target illumination for any sampled latent vector. We describe this model in Sec. for a fixed value of the latent. The diffusion samples correspond to different latent explanations of the scene and our latent NeRF optimization is able to effectively optimize these latent variables along with the NeRF model's parameters to produce consistent renderings for each latent explanation.We wish to model the distribution in Eq. ( 4) in a manner that lets us render images that correspond to relit views of the object for any sampled latent Z. We choose to model this with a latent code NeRF 3D representation, inspired by prior works that condition NeRFs on latent codes to represent sources of variation such as the time of day during capture [34] . This latent NeRF optimizes a set of latent codes that are used to condition the view-dependent color function represented by the NeRF, enabling it to render novel views of the relit object under the target illumination for any sampled latent code.In our implementation, the latent NeRF's geometry does not depend on the latent code, so the latent code may be interpreted as only representing the object's material properties.To optimize the parameters θ of the latent NeRF model, we maximize the log-likelihood, which by using Eq. ( 4), can be written as the following maximization problem:EQUATIONBecause integrating over all possible latents Z is intractable, we use a heuristic inference strategy and replace the integral with the maximum a posteriori (MAP) estimate of Z:EQUATIONBy assuming a Gaussian model over the data given the materials, the first term in Eq. ( 6) is a reconstruction loss over the images. However, since we do not have access to the true latent vector Z, we assume a uniform prior over them, turning the second term in Eq. ( 6) into a constant. In practice, similar to prior work on NeRFs optimized to generate new views given a dataset containing images with varying appearance, we rely on the NeRF model to resolve any mismatches in the appearance of different images [34] . See Fig. 3 for illustrations. The minimization of the negative log-likelihood can then be written as:EQUATION)In order to train the latent NeRF model described in Sec. versions of D i that have a high likelihood given the new target light L T . We then associate each sample s ∈ {1, . . . , S} with its own latent code Z i,s and sum over all samples when training the latent NeRF (Eq. ( 7)).Our RDM is implemented as an image denoising diffusion model that is conditioned by the input image and target lighting. To encode the target lighting, we use image-space radiance cues [15, 44, 61] , visualized in Fig. 4 . These radiance cues are generated by using a simple shading model to render a handful of images of the object's estimated geometry under the target lighting. This procedure is designed to provide information about the effects of specularities, shadows, and global illumination, without requiring the diffusion network to learn these effects from scratch. In our experiments, we use four different pre-defined materials to render radiance cues: one diffuse material with a pure white albedo, and three purely-specular materials with roughness values {0.05, 0.13, 0.34}. We use GGX [55] as the shading model. For more details, please refer to Sec. A.2.The RDM architecture consists of a pretrained latent image diffusion model, similar to StableDiffusion [45] , and uses a ControlNet [63] based approach to condition on the radiance cues. Please refer to Sec. A.3 for more architecture details. 
experimental setup: 
 Relighting Dataset We render objects from Objaverse [13] under varying poses and illuminations.For each object, we randomly sample 4 poses, and render each under 4 different lighting conditions. We represent the lighting as HDR environment maps, and randomly sample from a dataset of 509 environment maps from Polyhaven [60] . For more details, see Sec. A.4.We evaluate our method on two datasets: TensoIR [23] , a synthetic benchmark, and Stanford-ORB [27] , a real-world benchmark. TensoIR contains renderings of four synthetic objects rendered under six lighting conditions. Following [23] , we use the training split of 100 renderings with "sunset" lighting as input {I i }. We then evaluate on 200 poses, each of which has renderings under five different environment maps, i.e., "bridge", "city", "fireplace", "forest", and "night", for a total of 4000 renderings. Stanford-ORB is a real-world benchmark for inverse rendering on data captured in the wild. It contains 14 objects with various materials and captures each object under three different lighting settings, resulting in 42 (object, lighting) pairs. For the task of relighting, we are given images of an object under a single lighting condition and follow the benchmark protocol to evaluate relit images of the object under the two target lighting settings.Baselines We compare our method to several existing inverse rendering approaches. On both benchmarks, we compare to NeRFactor [65] and InvRender [66] . On the synthetic benchmark, we additionally compare to TensoIR [23] , the current top-performing approach on that benchmark. For the Stanford-ORB benchmark, we additionally compare to PhySG [62] , NVDiffRec [38] , NeRD [10] , NVDiffRecMC [17] , and Neural-PBIR [47] .Our Model Inference At inference time, the ideal embedding vector Z that best corresponds to the actual material is unknown. One approach to find this vector is to optimize Z to match a subset of the test set images (as in [34] ). However, to ensure a fair comparison, we avoid this optimization. Instead, we set Z = 0 for all views when rendering test images. Evaluation Metrics For both benchmarks, we evaluate the quality of 3D relighting by reporting image metrics for rendered images. We report PSNR, SSIM [58] , and LPIPS-VGG [64] on low dynamic range (LDR) images. Additionally, we report PSNR on high dynamic range (HDR) images on Stanford-ORB following the benchmark protocol, denoted as PSNR-H while the PSNR on LDR images is marked as PSNR-L. For approaches that do not produce HDR renderings, including ours, we convert the LDR renderings to linear values by using the inverse of the sRGB tone mapping curve. Due to the inherent ambiguities for the relighting task, we follow prior works [23, 27] and apply a channel-wise scale factor to RGB channels to match the ground truth image before computing metrics. Following established evaluation practices on Stanford-ORB, we compute the scale per output image individually whereas for TensoIR we compute a global scale factor that is used for all output images. 2Unless otherwise specified, all results are produced using S = 16 samples (see Sec. 3.4) and make use of 16 A100 40GB GPUs (batch size of 2 14 rays for NeRF optimization). We also provide results on a single A100 40GB GPU (batch size of 2 13 for NeRF optimization).We report quantitative results on the TensoIR benchmark in Tab. 1, and show qualitative examples in Fig. 5 . We significantly outperform all competitors quantitatively on all metrics with comparable or improved wall-clock time. Visually our method is capable of recovering specular highlights whereas prior methods struggle to model these.Similarly, we report results on Stanford-ORB in Tab. 2 and Fig. 6 . Our proposed approach quantitatively improves upon all baselines, except those of Neural-PBIR [47] , indicating the effectiveness of IllumiNeRF in real world scenarios. Note that although Neural-PBIR achieves better metrics than us, Fig. 6 shows that their relighting results are mostly diffuse, even for highly-glossy objects, and that they lack many of the strong specular highlights that our method is able to recover. This behavior of their model may explain their better metrics despite worse qualitative performance for specular highlights, because the illumination maps provided by Stanford-ORB do not correspond to the Renderings from all approaches have been rescaled with respect to the ground-truth as mentioned in Sec. 4.1. Areas where our approach performs well are highlighted. Our approach produces high-quality renderings with plausible specular reflections. incident illumination at the object's location, since they were captured using a light probe which was moved for each image in the dataset [27] . This means that even given perfect materials and geometry, the images relit by any method cannot match with the true captured images, which is most noticeable in specular highlights. This mismatch penalizes methods like ours, which recover such specularities, over ones that recover mostly diffuse appearance with no apparent specular highlights [52] . For a more detailed discussion see Sec. B.We also provide qualitative results for different latent codes in Fig. 7 . These results demonstrate that the optimized latent codes effectively capture various plausible explanations of the materials. Each row shows results from the same object and lighting with latent codes capturing various plausible explanations of the materials.We evaluate ablations of our model on TensoIR's hotdog scene in Tab. 3, and visualize them in Fig. 8 . We reach the following conclusions: 1) The latent NeRF model is essential: optimizing a standard NeRF cannot reconcile variations across views, even if we only generate a single sample per viewpoint for optimization (S = 1). 2) More diffusion samples help: by increasing S, the number of samples from the RDM per viewpoint, we observe consistent improvements across almost all metrics. This corroborates our intuition that using an increased number of samples helps the latent NeRF effectively fit the target distribution (Eq. ( 4)) in a more stable way. Our model relies on high quality geometry estimated by UniSDF [56] (see Sec. A.1) to provide sufficiently good radiance cues for conditioning the RDM (Sec. 3.4) . Any missing structure will lead our model to miss specular reflections, as seen on the top left of the salt can result in Fig. 6 's second column. Errors in geometry also affect the quality of synthesized novel views, e.g. the missing thin branches from the plant in Fig. 5 or fine details of the cactus (column 4) in Fig. 6 . Note that our RDM, trained on high-quality synthetic geometry, will inherently improve with future advances in geometry reconstruction. Our approach is not suited for real-time relighting, as it requires generating new samples with the RDM and optimizing a NeRF for any new target lighting condition. 
conclusion: 
 We have proposed a new paradigm for the task of relightable 3D reconstruction. Instead of decomposing an object's appearance into lighting and material factors and then relighting the object with physically-based rendering, we use a single-image Relighting Diffusion Model (RDM) to sample a varied collection of proposed relit images given a target illumination, and distill these samples into a single consistent 3D latent NeRF representation. This 3D representation can be rendered to synthesize novel views of the object under the target lighting. Perhaps surprisingly, this paradigm consistently outperforms existing inverse rendering methods on synthetic and real-world object relighting benchmarks. This new paradigm's success is likely due to the RDM's ability to generate a large number of proposals for the new relit images. This is in contrast to prior works based on inverse rendering, which first estimates a single material model and then uses it for relighting, since errors in material estimation may propagate to the relit images. We believe that this paradigm may be used to improve data capture, material and lighting estimation, and that it may be used to do so robustly on real-world data. 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)? Answer: [NA] Justification: We did not conduct research with humans or crowdsourced any tasks. Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: This appendix is organized as follows:1. Sec. A provides more implementation details; 2. Sec. B details the inconsistent illumination issue on Stanford-ORB.A.1 Latent NeRF Model and Geometry EstimatorWe use JAX [11] to implement both the geometry estimator and Latent NeRF model as UniSDF [56] , a state-of-the-art volume rendering approach based on a signed distance function (SDF). The advantage of using UniSDF is that it enables easily extracting a mesh from the SDF, which we can then import into a standard rendering engine such as Blender [50] in order to compute radiance cues. Additionally, UniSDF decouples geometry from appearance, allowing us to fix the weights related to geometry and only optimize for weights that model the appearance. Note that future NeRF/SDF approaches with improved geometric reconstruction can be seamlessly integrated in our method.Our parameterization of the UniSDF model is similar to the one used in the original paper for the DTU dataset [1] , with four key changes. First, we reduce the number of rounds of proposal sampling (as introduced by mip-NeRF 360 [5] ) from two to one, using 64 proposal samples. Second, we use the asymmetric predicted normal loss from NeRF-Casting [53] :EQUATIONwhere ω i is the volume rendering weight of the i-th sample, ∇ denotes the stop-gradient operator, n i and n ′ i are the i-th sample's density normals and predicted normals respectively (see [51] ), and we set λ 1 = λ 2 = 10 -3 , λ 3 = 10 -2 . Third, like NeRF-Casting [53] , we use an additional hash grid encoding [37] with 15 scales between a resolution of 32 and 4096, used only for outputting predicted normals. Fourth, we further encourage the local smoothness of the predicted normals n ′ by using a smoothness loss similar to [39, 65] :L s = λ 4 i ω i ∥n ′ (x i + ε) -n ′ (x i )∥ 2 , (S2)where x i is the 3D position of the i-th sample, and ε ∼ N (0, σ 2 I) is an isotropic Gaussian random variable used to perturb the sample locations. We set λ 4 = 0.1 and σ = 0.01.We find that these modifications result in better and smoother geometry necessary for our model's ability to relight objects with specular highlights.Finally, to incorporate the GLO embeddings, we utilize an MLP to predict an element-wise scale and shift value to be applied to the 'bottleneck' feature of UniSDF, similar to AffineGLO in Zip-NeRF [6] .For both geometry estimation and latent NeRF optimization, we utilize the Adam [24] optimizer with β 1 = 0.9, β 2 = 0.99, and ε = 1 × 10 -15 . We decay our learning rate logarithmically from 5 × 10 -3 to 5 × 10 -4 over 25k training iterations with cosine-scheduled warmup in the first 500 steps.Geometry To extract radiance cues we first optimize UniSDF [56] on the input images. After optimization, we convert the SDF representation to a mesh using marching cubes [31] with threshold set to be zero.Rendering We use Blender Cycles [50] , a physically-based path-tracer to render the radiance cues. We run Blender via the Kubric python wrapper [16] , and we use the estimated geometry with the predefined materials based on the GGX material model [55] , as described in Sec. 3.4.Table S1 : Fig. S2 's ConvNet 1 Structure. Convolution layer's definition is represented as (kernel size, stride, padding). We use SiLU [19] as the activation function between layers. Layer 8 uses zero initialization while the other layers use Flax's [18] default initialization 3 . In our implementation, we have H = W = 512. . We use SiLU [19] as the activation function between layers. Layer 5 uses zero initialization while the other layers uses Flax [18] default initialization 3 . In our implementation, we have H = W = 512.Index Layer Output Shape 0 (input) - H × W × 4 1 (3, 1, 1) H × W × 16 2-1 (3, 1, 1) H × W × 16 2-2 (3, 2, 1) H/2 × W/2 × 32 3-1 (3, 1, 1) H/2 × W/2 × 32 3-2 (3,Index Layer Output Shape 0 (input) - H × W × 12 1 (3, 1, 1) H × W × 16 2-1 (3, 1, 1) H × W × 16 2-2 (3, 2, 1) H/2 × W/2 × 32 3-1 (3, 1, 1) H/2 × W/2 × 32 3-2 (3, 2, 1) H/4 × W/4 × 96 4-1 (3, 1, 1) H/4 × W/4 × 96 4-2 (3, 2, 1) H/8 × W/8 × 256 5 (3, 1, 1) H/8 × W/8 × 320given masked image and radiance cues are first fed through ConvNet 2 (see Fig. 4 in [61] for details) and ConvNet 3 (see Tab. S2). The resulting output is added to the output of the latent noise, which is fed through ConvNet 4. ConvNet 4 consists of a single convolution layer with kernel size 3, stride 1, padding 1, and 320 output channels. Given that the trainable copy was designed for tokenized text input, the masked image is first fed through ConvNet 1 (see Tab. S1) to generate representative embeddings. To ensure compatibility between the output of ConvNet 1 (size 64) and the CLIP [42] encoder's text output shape, zero-valued tensors are appended, increasing the size to 77.We train the diffusion model using an approach similar to ControlNet [63] , with a large dataset of synthetic objects rendered under multiple lighting conditions. Each training example for fine-tuning consists of a pair of images that view the same object with the same camera parameters, illuminated by two different environment map (see Sec. A.4). We fine-tune the diffusion model to predict one of these two images, given the other image as well as the corresponding radiance cues rendered using the synthetic object's geometry. Note that for synthetic objects, we do not need to estimate the geometry G nor to enable the Blender normal smoothing function to compute the radiance cues since we already have the ground-truth meshes and the normals from synthetic objects are smooth enough. We fine-tune the base model for 150k steps using batch size of 512 examples and a learning rate of 10 -4 , which is linearly warmed up from 0 over the first 1k steps. The fine-tuning takes around 2 days on 32 TPUv5 chips. Besides, we always use the empty string as the text input to effectively make the fine-tuned model image-based.At inference time, we use the DPPM scheduler [21] without classifier-free guidance [20] to produce samples at 512 × 512 resolution.We use Objaverse [13] as the synthetic dataset. To filter out low-quality objects, we use the list from [49] to get our initial set of 156,330 ones. 4 By additionally removing (semi-)transparent ones, we have a final set of 152,649 objects. If the object only contains geometry, we manually assign a homogeneous texture (ShaderNodeBsdfDiffuse) with a color uniformly sampled from [0, 1] 3 . Further, if the object does not have the material information, we assign it a Blender Glossy BSDF Table S3 : Issues on illuminations of Stanford-ORB [27] . We create two sets of reference renderings with the ground-truth geometry and material: 1) using a fixed illumination to render all evaluation images for a same (object, lighting) pair; and 2) using the per-image illumination provided by the benchmark. We then evaluate each approach's renderings with respect to the two sets of reference renderings respectively. We also list each approach's illumination selection in the second column.For each row, better performance between the two evaluations is highlighted . Apparently and consistently, the numerical results favor matched illumination selection. material (ShaderNodeBsdfGlossy), whose roughness value is uniformly sampled from [0.02, 0.5] and base color is set to be the same as the homogeneous texture. The mixing factor between the specular and diffuse materials (ShaderNodeMixShader) is uniformaly sampled from [0, 1].As we discussed in Sec. A.3, our diffusion training requires image pairs under different lightings. For this, we select 509 equirectangular environment maps from [60] . For each object, we sample four camera poses on a sphere centered around it. For each camera, we randomly sample two environment maps and augment them with random horizontal shift, vertical flip, and RGB channel shuffle. We then use Blender's Cycle path tracer to render an image of resolution 512 × 512 with 512 samples per pixel for each environment map using a camera whose focal length is set to be 512.Stanford-ORB provides estimated per-image illumination via moving a light probe for each image, see Fig. S3 for an example. Ideally, fixed illumination per object would match reality, but aligning the object and light probe is challenging. This limitation in the Stanford-ORB benchmark can significantly affect results, especially in areas with specular highlights as demonstrated in Tab. S3. Consequently, there is no "correct" way to do relighting in Stanford-ORB: our results use fixed illumination, while competitors use per-image illumination.We rendered the ground truth geometry and materials, obtained by Stanford-ORB in a controlled studio environment (see [27] 's Sec.3.2.2). Each view was rendered under fixed illumination (consistent environment map) and per-image illumination (unique environment map per view). Fig. S4 shows both renderings alongside the corresponding real image. Note the significant variations, especially in areas with specular highlights (see marked regions and PSNR). 2) reference rendering w/ per-image illumination" (see Tab. S3 caption for details); and 3) the real captured image. We show PSNR-L between 1) vs. 3) and 2) vs. 3) respectively. Different illumination settings vary significantly.We computed metrics for each method using both reference renderings as ground truth whose quantative results are in Tab. S3. Our method excels under fixed illumination but performs worse with per-image illumination. Competitors show the opposite trend, doing better with per-image illumination. Neural-PBIR did not release code or complete results, thus it is missing from the table.Guidelines:• The answer NA means that the paper does not include experiments.• If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not. • If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable. • Depending on the contribution, reproducibility can be accomplished in various ways.For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed. • While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. , with an open-source dataset or instructions for how to construct the dataset). (d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility.In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results. Justification: We propose a novel paradigm for an existing task -3D relighting. If reviewers believe our method could cause potential harm, we are happy to include a statement. Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).Please refer to https://github.com/StanfordORB/Stanford-ORB/blob/962ea6d2cc/scripts/ test.py#L36 and https://github.com/Haian-Jin/TensoIR/blob/2a7a4d00/renderer.py#L12.https://github.com/google/flax/blob/144486b5fa7b3dfb/flax/core/nn/linear.py#L27https://github.com/ashawkey/objaverse_filter/tree/dc9e7cd0df8626f30df02bb 


Decision: All reviewers agree that the idea of leveraging a learned image-based diffusion model to predict relit images for latent NeRF optimization and relighting is novel and interesting. The method also demonstrated superior rendering quality compared to prior inverse rendering methods (except Neural-PBIR) on the TensoIR and Stanford-ORB benchmarks. However, the reviewers also raised concerns about the effectiveness of the learned latent NeRF, noting that the diffusion model seems unreliable compared to dense view conditioning for material estimation, and the overall pipeline appears brittle. AC believes that the diffusion-based solution is fresh and valuable to the community and thus recommends acceptance.

Reviews: review: paper_summary: paper_summary: The paper proposes a new method called illumiNeRF for 3D relighting — given a set of posed input images under an unknown lighting, illumiNeRF produces novel views relit under a target lighting. Most of existing methods use inverse rendering to first recover the material and lighting in the given images, and apply new lighting to the object. The authors point out there is inherent ambiguity in decomposing materials and lighting. Instead, they propose a probabilistic model of representing multiple possible materials as latents, and train a latent NeRF using a relighting diffusion model. The authors compare their method with existing baselines and outperform most of them.
strength_weakness: strength_weakness: - Although during training, multiple materials are considered as different Z when optimizing the latent-NeRF, Z is manually set to 0 at test time, which corresponds to a specific explanation. However, Z=0 does not seem to represent the most likely material because the authors assume a uniform prior over Z according to line 163.
- From Figure 6, the results lose a lot of details compared to other baselines like Neural-PBIR. For example, the method lost details of the spikes in the cactus sample. In table 2, Neural-PBIR also outperforms the proposed method.
questions: questions: It would be great to show 3D renderings with different latent explanations in Figure 4(b).
limitations: limitations: Yes
paper_summary: paper_summary: The paper proposes a method to relight NeRF representations from a set of multi-view input images. The paper first trains a NeRF model and recovers geometry using UniSDF from the input images. Then, from the mesh, they obtain radiance cues which are used to condition a diffusion model to sample relit images of the scene under a target lighting. Since these relit images are samples from a distribution of plausible relit images, the sampled results are used to train a latent code conditioned NeRF model, and the lighting of the scene can be manipulated by walking through the latent space. The core contribution of the paper is to use the relit images rather than trying to explicitly model and learn the lighting and material properties.
strength_weakness: strength_weakness: 1. The significance of using diffusion models to relight images, which are then used to train a NeRF, is limited to the context of application to NeRFs. Concurrent works have very similar methods in the context of Gaussian splatting (https://repo-sam.inria.fr/fungraph/generative-radiance-field-relighting/) - what is the benefit of this approach over those? Some discussion should be made on this.
2. The pipeline seems brittle, requiring many steps to go right and limited by the sum of limitations of many different technologies. The quality of relighting is not only limited to the quality of the diffusion model output but also depends on the quality of the UniSDF mesh reconstruction, which is required to create the radiance cues. This means that a good capture is a necessary prerequisite to relight with this method, and one could simply relight the extracted mesh to achieve a similar effect.
3. I find the abstract a bit misleading: “we first relight each input image using an image diffusion model conditioned on lighting and then reconstruct a Neural Radiance Field (NeRF) with these relit images, from which we render novel views under the target lighting.” This description doesn’t accurately represent what actually happens, as it leaves out the most crucial part of the pipeline (reconstruction + conditioning) and instead suggests that the NeRF can be directly reconstructed from the relit images (Compare figure 2 of the paper).
4. The choice of UniSDF is neither justified nor are alternatives discussed. The part describing this aspect of the approach is relegated to the appendix. Consequently, there is no ablation study to determine if other reconstruction methods might work better.
5. With respect to figure 6: It makes sense to highlight the same area for all depicted approaches. For instance, the blocks and the ball have no highlights, but other approaches seem to perform better in these areas. Additionally, the normalization technique used to match the ground truth images is not ablated. Does this make a difference in the reconstructed quality? I notice that many of the other methods are darker than I’d expect and I wonder if this introduces some artifacts which hurt these methods and promote the proposed method.
6. It is unclear why the paper limits itself only to objects. UniSDF clearly shows good performance for scenes. The authors don’t discuss this limitation in the paper.
7. Changing the target light even slightly requires a wait of 45 minutes for a final rendering. Other approaches, as referenced above, seem to not require that much time.
questions: questions: Many questions are discussed in the weaknesses, specifically:
1. How sensitive is the paper to the UniSDF reconstruction pipeline? Are there better methods which exist? Why does the paper only work for objects?
2. How does the normalization to ground-truth images for visualization of relit images actually affect the quality?
limitations: limitations: Some limitations are discussed in the paper, but there are additional ones which should be discussed in more detail, such as the long reconstruction time to get new lighting conditions.
paper_summary: paper_summary: The paper proposes a method for novel view synthesis with novel lighting given multi-view observation of an object. The pipeline is composed of a learned single-image-based relighting model based on image diffusion models, and a latent NeRF model that reconstructs the appearance of the relit object by reconciling relit images across views and samples. The relighting diffusion model (RDM) is conditioned on radiance cues generated from the geometry and target lighting under samples of roughness values, using a finetuned ControlNet architecture. The method is evaluated on both synthetic and real-world scenes, against baseline methods (mostly object-centric inverse rendering models), with both qualitative and quantitative results. Ablation study is also provided on design choices (e.g. latent/no latent NeRF, and number of diffusion samples).
strength_weakness: strength_weakness: [1] Effectiveness of the latent NeRF. The paper claims that the latent NeRF is able to reconcile difference in the sampled results from RDM. However, insufficient analysis is provided for justification. For instance, in Fig. 4, results in (a) seem to present more specularity than what is actually present in the hotdog scene (in other words, RDM seems to produce unrealistic relit images), and results from the latent NeRF in (b) seems to be an averaged version of the samples, with most of the specularity averaged out, arriving at a mostly diffuse appearance except for areas where specularity all agree among sampled. As a result, in Fig. 6, when compared to inverse rendering methods like Neural-PBIR, despite that the proposed method is able to produce better specularity in highlighted areas (thanks to explicitly conditioning on radiance cues), the results of the proposed method look 'dull', where shadows and high-frequency details seems to be lacking. This casts questions into the performance of both RDM to produce consistent and photorealistic relit images, as well as the ability of the latent NeRF to successfully reconcile inconsistency in the sampled results without sacrificing details and shadows.

[2] Claim on better secularity in the results is questionable. The paper explains that, when compared to Neural-PBIR the proposed method is in second place numerically because the approximate lighting ground truth from the light probe unfairly favors Neural-PBIR despite the proposed method produced more convincing specularity. However this is largely unjustified, as (1) it is not clear how much weight is on the specular pixels in the overall metrics (2) whether the proposed method actually produces better specularity in the absence of true ground truth of relit objects. Moreover as mentioned in [1], the proposed method seems to lose more details (e.g. in the spikes of the cactus, or details in the ball). In this case, even if the methods excels in specularity, this is not the single most important property that is accounted for in those metrics, or visually, when evaluating the quality of the results.
questions: questions: [1] Quality of the sampled results from RDM. Why are some results look overly specular (e.g. the bread in the hotdog scene in Fig. 4). And if more examples can be provided on results from RDM?

[2] Reconciliation by the latent NeRF. Does the Latent NeRF cost details and specularity in sampled results from RDM when reconciling the results into a NeRF representation? Also if there are view-dependent effects in the sampled results, does the latent NeRF properly preserve those effects? Again, more examples are to be provided in order to further examine the behavior of the two modules beyond one single scene (and especially for synthetic scenes, where 'true' ground truth of relit objects is available). There are additional results on Standford-ORB scenes in the Supp but analysis is needed in the main paper to support the method using those results.

[3] Additional ablation. Is it helpful to use more or fewer roughness levels in generating radiance cues?

[4] In designing baselines, why not comparing against ALL methods in both synthetic and real-world settings?
limitations: limitations: NA
paper_summary: paper_summary: This paper introduces a new paradigm for the relightable 3D reconstruction: leverage a Relighting Diffusion Model to generate relit images and distill them into a latent NeRF. The proposed paradigm demonstrates superior performance compared to existing methods, which are based on inverse rendering, on both sythetic and real-world datasets.
strength_weakness: strength_weakness: One major weakness of this method is its efficiency. Unlike traditional methods that can directly relight a 3D scene once the inverse rendering is performed, the proposed approach requires optimizing a NeRF for each new target lighting condition. Furthermore, the computation of the latent NeRF involves a significant time investment (0.75 hours using 16 A100 GPUs). It would be beneficial to discuss potential solutions for this issue.
questions: questions: - How many samples ($S$) were used during the experiments presented in Table 1 and Table 2? 

- As this method models the probability of relit images and fixes $Z=0$ during inference, it would be interesting to investigate how the performance varies by sampling different $Z$ values.

- Typo in Line $446$
limitations: limitations: Limitations have been well discussed.

Comments: title: Discussion
comment: We thank the reviewer for their time in reading our rebuttal and providing feedback.

We are encouraged that the reviewer recognizes the quality improvements and novel contributions introduced by our method. 

We would like to clarify that every existing 3D relighting baseline also requires geometry reconstruction. 

Our current method did not focus on improving speed, but rather introducing a novel 3D relighting paradigm. It was designed as a general framework that can be further developed to substantially improve speed in the future, e.g. by using Gaussian Splatting representation instead of NeRF as well as using a faster diffusion sampler.
Thank you for the detailed rebuttal. It has addressed the majority of my concerns, and I do see improvement over prior work in terms of quality and acknowledge the novel contributions. However, I do feel as though the computational drawback and reliance on geometry reconstruction limit the applicability of the method in its current state enough such that it might not be presenting significant or impactful contributions at this time. I remain slightly positive on the paper.
We thank the reviewer for their time in reading our rebuttal and providing the feedback.

We are pleased that the reviewer appreciates the concept and formulation of our method. 

Our primary objective was to introduce a novel paradigm with the potential to replace hand-crafted inverse rendering techniques. Although our current method may not significantly outperform all existing baselines, we believe it lays a strong foundation for future advancements in the field.

We now address each question below:

## Response to Point [1]

### **Diffusion model input**

There seems to be a misunderstanding with respect to the number of views used as input to our relighting diffusion model (RDM). The RDM model (L168 - 172)  is a **single-image/view** model. It has never been exposed to “multiview images” (see Eq. (4)).

We also agree that it would be interesting to further improve the RDM model by conditioning on multiple views.

### **Unrealistic relit images | little to no chance that the bun is actually of specular material**

We want to emphasize that our RDM models the entire distribution of plausible materials given a single observation of an object without known source lighting.

While material roughness is still ambiguous when the source lighting is unknown, we agree that the samples with specular hot dog buns are less likely, and would hope that a future multi-view RDM model could improve this.

### **Different appearances for Hotdog’s bun between main paper and rebuttal**

For Fig. 4(a) we chose to show a wide spectrum of plausible samples generated by the diffusion model. 

For the rebuttal we chose to show more samples, which included less specular buns. 

This further demonstrates the wide range of possible combinations between lighting and material. The final reconstructions recover the most likely materials (filter out the shiny bun samples) and do not exhibit any specular highlights for the bun, see Fig. 4(b).

## Response to Point [2]

### **On the benchmark with Stanford-ORB**
We agree that in the presence of noisy GT lighting the analysis does not favor any method and will rephrase the related language.

## Response to Point [3]

We show qualitatively that our method is the only one that somewhat-consistently recovers correct specularities on this dataset, while all other methods exhibit no specular highlights, apart from Neural-PBIR for a single example (the ball in the rightmost column of Figure 6). For the task of relighting this is perhaps one of the most important features. 

Unfortunately, Neural-PBIR only provides 8 qualitative result images (no source code is currently available), hence it is hard to properly compare our results with their approach, and we are happy to clarify our claims in the paper and limit them to only refer to specular highlights. 

We also admit that our method sometimes misses details due to the UniSDF reconstruction, and are happy to clarify this in a revised version.


## Overall

We are happy that the reviewer recognizes the value of our proposed 3D relighting paradigm, which we consider the core contribution of our work (L42 and 256). 

Our method significantly outperforms all baselines except Neural-PBIR. However, due to limited results, unavailable code, and challenges with Stanford-ORB, it's difficult to determine which method is superior. 

We also agree that each component of our pipeline has the potential for further improvements. Since the components are independent of each other, improving one should improve overall performance. 

We are optimistic that future research will address these areas and build upon our framework.
I would like to thank the reviewers for their rebuttal to address my concerns. I agree with the responses to some of the minor concerns including ablation on roughness, and evaluation settings, but would insist to differ on the rest of the major issues. Specifically,

[1] Unrealistic relit images from the hot dog scene in Fig. 4. The trained image diffusion model is supposed to make correct estimation on the material of the scene (and produce convincing images under novel lighting with correct materials true to the input images). However, assuming the input images are typical of the popular hot dog scene where the bun is mostly diffuse, some samples are unreasonably specular on those regions. I do agree with the author that there is inherent ambiguity between materials and lighting in inverse rendering, which sometimes lead to different plausible combinations of materials and lighting. However, (a) the diffusion model is supposed to make good estimation on the materials based on the input images, instead of producing results of diverse levels of specularity; (b) given multiview observation of the hot dog scene as input, where specularity is NEVER observed on any regions of the bun, there is minimal ambiguity between materials and lighting. There is little to no chance that the bun is actually of specular material while none of the multi-view input images suggest any specularity on the bun. In general, the ambiguity is more likely with single or few images, and more often in the case of baked shadows in materials, and less or not likely at all when there is sufficient observation (as is the setting with this paper) and no specularity is present in any view. In conclusion, I would incline to insist that the samples where the bun is specular are not plausible given the input images, and the trained diffusion model does not do a great job in estimating the true materials in some cases.

Interestingly, in the additional 3 samples of the hot dog scene in the rebuttal PDF, the bun is not specular, while the bun is specular in sample #1, 2, 4, 5, of the 5 samples in Fig. 4(a). It is interesting to hear more from the authors on this difference to make a more informed conclusion on the behavior of the diffusion model.

[2] On the benchmark with Standford-ORB. The authors included additional discussion on the capturing setting of the lighting GT with the dataset, indicating due to the light prob is not collocated with the object, the ground truth lighting is not accurate. I agree on this aspect but insist that due to this issue, the errors in lighting ground truth do not necessarily favor one method or the other. It is simply not reliable for ANY method. In conclusion, I would think it is not justified that the result from this method actually outperforms Neural-PBIR in the presence of the noisy GT lighting, and recommend the authors to rephrase the related language.

[3] On the comparison of results against Neural-PBIR. I would conclude that the proposed method and Neural-PBIR both hit and miss in some aspects over the provided samples, w.r.t. level of details, correctness of specularity and shadows. None is overwhelmingly superior to the other. As a result, the performance of the proposed method is on par with Neural-PBIR over all without significant and meaningful improvement.

Overall, I like the idea and the formulation of the method a lot, but believe there is no significant improvement in the results when compared with baselines. Drawbacks of the method and the paper are also pronounced: long and brittle pipeline (as suggested by Reviewer hgwb), modules (particularly RDM) not sufficiently evaluated, and unjustified claims on the performance. I would refrain from accepting the paper until hearing more from the authors and reviewers.
Dear reviewer q4ex

We received diverse ratings for this submission, and you were the only one with a negative initial review. Please review the rebuttal and the comments from other reviewers and join the discussion as soon as possible. Thank you!

Your AC