ROGR: Relightable 3D Objects using Generative Relighting
Authors: Jiapeng Tang
Matthew Jacob Levine
Dor Verbin
Stephan J. Garbin
Matthias Nießner
Ricardo Martin Brualla
Pratul P. Srinivasan
Philipp Henzler
Venue: NeurIPS 2025 spotlight ()
Keywords: Reflectance modeling; Relighting; Neural Radiance Fields; Diffusion Models
Primary area: applications
TL;DR: We introduce ROGR, a novel approach that reconstructs a relightable 3D model of an object captured from multiple views, driven by a generative relighting model.

Abstract: We introduce ROGR, a novel approach that reconstructs a relightable 3D model of an object captured from multiple views, driven by a generative relighting model that simulates the effects of placing the object under novel environment illuminations. Our method samples the appearance of the object under multiple lighting environments, creating a dataset that is used to train a lighting-conditioned Neural Radiance Field (NeRF) that outputs the object's appearance under any input environmental lighting.  The lighting-conditioned NeRF uses a novel dual-branch architecture to encode the general lighting effects and specularities separately. The optimized lighting-conditioned NeRF enables efficient feed-forward relighting under arbitrary environment maps without requiring per-illumination optimization or light transport simulation. We evaluate our approach on the established TensoIR and Stanford-ORB datasets, where it improves upon the state-of-the-art on most metrics,  and showcase our approach on real-world object captures.

ROGR: Relightable 3D Objects using Generative Relighting
Jiapeng Tang 133+
Matthew Levine
Dor Verbin Stephan J. Garbin'
Matthias Nießner
Ricardo Martin-Bruas
Pratul P. Srinivasan
Philipp Henzler!
Google Research 2 Google Deepmind
3 Technical University of Munich
Input: posed images
target illumination
Output: 3D relit object under novel illumination
Figure I: Given set of posed images under unknown illumination (top), our method reconstructs relightable neural radiance field (bottom), that can be rendered under any novel environment map without further optimization, on-the-fly relighting and novel view synthesis.
Abstract
We introduce ROGR, novel approach that reconstructs relightable 3D model of an object captured from multiple views, driven by a generative relighting model that simulates the effects of placing the object under novel environment illumina- tions. Our method samples the appearance environments, creating dataset that is used to train ral Radiance Field (NeRF) that outputs the object's appearance under any input environmental lighting. The righting-codiition architecture to encode the general lighting effects and specularities separately. The optimized righting-conditionn under arbitrary environment maps without requiring per-illumination optimization light transport simulation. We evaluate approach the established TensoIR and Stanford-ORB datasets, where it improves upon the state-of-the-arr on most metries, and showcase our approach on weal-world object captures. Project Page
the object under multiple lighting
Neu-
NeRF uses novel dual- branch
NeRF enables efficient feed-forward relighting
Introduction
Inserting real-world objects into environments is long-standing problem in computer graphics [1, 2], with numerous applications in the movie and gaming industries. While recent years have seen
"qqual contribution.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

significant progress in 3D object reconstruction for view synthesis using radiance fields [3, 4], these techniques represent an object illuminated by single fixed environment and they do not enable changing the appearance of the object due to changes in the lighting. This work extends 3D object reconstruction to enable rendering the object under arbitrary illumination.
A typical approach for reconstructing relightable 3D representations from images is inverse rendering: optimizing the material and lighting parameters that together explain the captured images. This particularly challenging for real-world captures as it is brittle and sensitive to mismatches between the real world's physical light transport and the simplified lighting and material models used during optimization. Furthermore, due to the problem's 's inherent ambiguities, object properties recovered by inverse rendering often appear implausible and unrealistic when viewed under novel lighting.
At the same time, recent relighting diffusion models [5, 6, 7] have demonstrated impressive capa- bilities for generating realistic images of objects under arbitrary illumination. However, they only generate a single relit image at a time, which results in inconsistent relighting results when applied to sequence of viewpoints. While these inconsistently-relii samples can be reconstructed into single 3D representation [7], optimizing new 3D representation for each new target lighting is tedious and precludes interactive use cases.
We propose a strategy for distilling samples from a relighting diffusion model into a relightable 3D Neural Radiance Field (NeRF) that can be rendered from arbitrary viewpoints under arbitrary novel environment illumination. Given images of an object, we first use a multi-view diffusion network to generate niew-consistent relit images under wide array of environment illuminations. We then use these multi-view nulti-illumination samples to train a novel NeRF architecture that predicts outgoing view-dependent color conditioned a target illumination.
We evaluate the efficiacy of our method on the task of 3D relighting using both synthetic and real- world benchmarks. Our approach achieves state-of-the-art results but also demonstrates significantly improved test-time performance compared to prior work. These performance gains are due to our generalizable ferwadd relightable NeRF.
2 Related Work
Inverse Rendering for Relighting. Recovering relightable 3D representations of objects from images a longstanding goal computer vision and graphics. The prevalent approach is inverse rendering: decomposing the object's appearance into its underlying geometry, illumination, and material parameters, and relighting the object by simulating the physical interaction of new target illumination with the recovered geometry and materials [8, 9, 10, 11].
Modern methods for reconstructing relightable 3D representations with inverse rendering use dif- ferentiable rendering techniques [12, 13] to optimize mesh [14, 15], distance field [16], or volumet- ric [17, 18, 19, 20, 21, 22, 23, 24] representations of object geometry and material parameters. While these inverse rendering approaches can be effective, they come with significant limitations: errors in estimating an object's geometry and materials can produce unrealistic appearance when the object is relit under a new illumination and physically- relighting involves cmmputationalll-exenssiii Monte Carlo simulation of global illumination, which can be too slow for interactive use cases.
Precomputed Radiance Transfer and Light Stages. Early approaches for interactive relighting in the field of computer graphics proposed the idea of Precomputed Radiance Transfer (PRT) [25]. As appearance behaves linearly with respect to lighting, an object or scene's 's appearance can be precomputed under set of basis lightings and these can be linearly combined to produce appearance under any desired target lighting condition. While PRT-based techniques enable interactive relighting. they struggle with the memory demands of storing the precomputed transfer matrices. To enable rendering under novel illuminations from arbitrary viewpoints, PRT-based methods need to store a full transfer matrix for each 3D point in the scene. These memory requirements are prohibitive even for modestly sized scenes and environment maps, so a large body of work focuses on compressing these PRT matrices [26, 27].
One-light-at-a-timee (OLAT) captures in light stages [28] can be thought as directly capturing basis vectors of real object's 's radiance transfer matrices. Since each image captured by the light stage is of the object illuminated by single element of standard lighting basis, they can be linearly combined to reproduce the object's appearance under any desired environment illumination.

---

Posed images
Multi Mluli-illumination dataset
Multi-viw
relighting model
Environment maps
Figure 2: Multi-view Relight Diffusion. Our multi-view relighting diffusion model takes input N posed images illuminated with a consistent, but unknown illumination, represented by camera raymaps and the source pixel values, and an enviromment map per image that has been rotated to the camera pose. The diffusion model generates images of the same object from the same poses, but lit by an input environment map. To generate multi-lllmmination dataset, repeat this relighting process times with M environment maps.
Direct Relighting without Inverse Rendering, In the deep learning many methods have trained networks to directly output relit images. Early techniques utilized standard convolutional neural networks [29, 30] and more recent approaches are based on powerful diffusion models [5, 7, 6, 31]. These generative relighting models have produced impressive results for image relighting, but they cannot be easily used for 3D relighting the relit appearance is often not consistent across views. IllumiNeRF [7]. MIS [31], and Neural Gaffer [6] perform 3D relighting by reconciling samples from 2D image relighting diffusion model into a single NeRF. However, the relit 3D representation needs to be optimized separately for each novel target illumination. Recent work on 3D reconstruction from images of object with varying illumination [32] uses multi-view diffusion model to relight them to have consistent illumination. However, this approach does not allow relighting using a new unseen illumination condition, so it does not allow for generalization. And does not utilize the high-frequent details in environment maps. Zeng et al. [33] and RNG [34] utilize shadow and highlight cues conditioning signals for NeRF. In contrast, features of incident light from the specular reflection direction. Additionally, in contrast to all of these approaches, our proposed method trains generalizable relightable NeRF that can be rendered under any target illumination without additional optimization. A concurrent work, RelitLRM [35], directly recovers 3D geometry and appearance but does not guarantee consistent geometry across environment maps. In contrast, our method reconstructs constant geometry under varying illumination. DiffusionRenderer [36] employs a video diffusion model for relighting but lacks 3D consistency, while our approach enforces it by explicitly modeling geometry and rendering, Moreover, model enables fast inference, avoiding the long sampling times typical diffusion-based generation.
explicitly use the encoded
Method
Given a set of N posed images D = (L, W of an object, where I; the i-th image and т; its pose (including camera extrinsic and intrinsic parameters), we are interested in learning the parameters 0 of relighting function fg. that allows rendering the object from any viewpoint ı illuminated by any lighting F. (e.g. an environment map) to produce new image / = fo (F, r). In order learn the parameters 0 of this transformation, we propose to generate dataset of pairs of posed images with their corresponding illumination to train fo in supervised manner. In Sec. 3.1 we will describe how to generate such paired data using a generative relighting diffusion model, and in Sec. 3.2 we explain how to optimize fg, which we implement using rightin--coditionen relightable radiance field.
3.1 Multi- Diffusion Relighting
Our goal is to train generative multi-view diffusion model relight our given posed images D, which are jointly captured under the same unknown lighting. The diffusion model provides samples from the distribution of possible images Dg relit by the target illumination E:
p(Dg E).
(I)
3

---

Our work crucially relies on multi-view diffusion relighting model to provide consistent relit images, which can then be distilled into relightable 3D model. This is in contrast with prior work on relighting using diffusion models [7, 6] which independently relights single images and results in ambiguities that must resolved at the 3D reconstruction stage.
Our network architecture is inspired by multi-view diffusion architectures such as CAT3D [37], which start with pretrained 2D image diffusion models and inflate them by adding cross-attention layers to process multiple views. We adapt such a scheme for the task of relighting and show our architecture in Fig. 2. Since we use a latent diffusion model (LDM), we first map all images and environment maps into the latent space of the original 2D diffusion model. In particular, the environment map encoded in similar manner to Neural Gaffer [6]: We use two separate latents corresponding to high dynamic range (HDR) and low dynamic range (LDR). This representation captures both bright details like direct light sources, as well as relatively dim sources like diffuse objects. Like Neural Gaffer, we also use standard tone mapping for the LDR environment map [38, 39]. and for the HDR encoding we apply logarithmic tone mapping followed by normalization to [0, 1] by subtracting the minimal value and dividing by the maximal value. Additionally, for each image 4; in our dataset, we apply a 3D rotation to the environment map to align it with the corresponding camera pose TÚ. We combine the HDR and LDR encodings of the environment map with the encoded image and the ray map of the pose of each image by concatenating them, and then apply self-attention. Details of the network architecture for our multi-view relighting diffusion models provided in Fig. of the supplementary material.
3.2 Relightable Neural Radiance Field
Our diffusion model g generates consistent relit images given target lighting environment. Our end goal is to obtain a 3D representation of the object that can modify the illumination of the scene without additional per-illumination optimization. To do this, we first use the multi-view relighting model to create a new dataset Drelit for each object by taking the N images in the original dataset D and relighting them using collection of M environment maps & (Ei,..,, Eas). The new relit dataset of an object can be written as:
Dreli = (g( TH, E,:
I,,,,,, M},
(2)
where g(4i, nú, Eÿ) is the diffusion model's estimate for the ith image I; (whose pose is T.) lit by the jth environment map E,. Dreta contains N M images.
We then use the multi-illuminaan illumination dataset Deeta to train a NeRF. Since we wish to fit the NeRF model to our dataset with varying lighting so that it generalizes to novel lighting environments outside of our training set of illuminants &, care must be taken when designing the model's 's architecture. We use NeRF-Casting [40] as the base NeRF model since it efficiently captures view-dependent appearance, and we modify it to allow for conditioning on the illumination. Crucially, we encode the environment maps using two types of conditioning signals: general conditioning and specular conditioning. The general conditioning encodes the entire environment map into an embedding that is fed the appearance MLP and is designed as a general-purpose, low-frequency signal for relighting, while the specular conditioning only encodes incident light coming from the specular direction. Its goal is to improve the model's capacity to capture high-frequence reflections (similar to prior work on reconstructing specular reflections in NeRF [41]). The conditioning signals are illustrated in Fig. 3.
capacity
In order to encode the specular appearance of camera ray, NeRF-Casting [40] casts small set of reflected rays, traces them through the NeRF's geometry, and volume renders a feature vector that encodes the scene content observed by these reflected rays. In our setting, we provide the two conditioning signals by concatenating them to the feature vector f of each ray, which is mapped by NeRF-Casting's 's MLP to the ray's specular color component.
General Conditioning, Our general conditioning signal encodes the full environment map, and is therefore a complete deseription of the lighting of the object. To do this, we train an transformer- based encoder which maps the environment map into vector. While the idea of using a per-lighting that our embeddings are not optimized to fit to individual images, but they are instead parameterized as a learned mapping from the environment maps. This enables us to render the scene under novel illumination at inference time without requiring any additional training.
encoding is similar to the approach of NeRF-in-the- Wild [42], a crucial difference

---

To encode the illumination features from the environment map, we utilize a transformer encoder with self-attention. Our transformer is trained from scratch, and its architecture is based on ViT-S8 of DINO [43], followed by a single matrix multiplication W to produce a compact 128-dimensional embedding vector:
preneral W ViT(E)
Specular Conditioning. Although the gen- eral conditioning vector theoretically contains all information necessary for relighting, we found it to be insufficient for reconstruct- ing and rendering high-frequency specular highlights. Our specular conditioning is de- signed to fix that by explicitly encoding in- eident light from the specular reflection di- rection, similar to the encoding used in prior work for reconstructing reflective objects and scenes [41, 40, 44]. Instead of only sampling the environment map value the reflection direction, we use a series of blur kernels cen- tered around the reflection direction sim- ulate the effects of materials with different roughnesses. The i-th component of this con- ditioning vector can be written as:
Environment map
Rellezzion direction
Camera
ray
Transformer encoder
fgeneral
Burface
Multi scale Gaussian blur
pepecular
Prefiltered environment maps
pspecutar = =/6 Fiw""GG(w' wr. oz ddw (4)
Figure 3: Lighting conditioning signals. We use combination of two lighting conditioning signals to train the NeRF our generated multi-illumination dataset. The general lighting encoding fecneral is used for encoding the full environment map in single em- bedding, and is obtained using transformer encoder applied the entire sphere of incident radiance. The specular encoding pspecular is composed of the envi- ronment map value, as well prefiltered versions of the environment map, queried at the reflection di- rection w,, which the direction of the camera ray reflected about the surface normal vector. Combining these two conditioning signals provides the NeRF with all the information necessary for relighting dif- fuse materials as well as shiny ones, which exhibit strong reflections.
where G(w'; o,) is Gaussian blur kernel around w, with width os, and W, is the view direction reflected about the surface normal. For efficiency, we preprocess the environment map by blurring it at all directions using Eq. 4, and then query at the reflection direction w during the NeRF optimization stage.
Following NeRF- Casting [40]. our relightable NeRF takes input the 3D coordinates, ray direction, gen- eral conditioning, and specular conditioning. The geometry MLP predicts density, rough- ness, normals, and geometry features, while the color MI.P outputs the RGB values.
Network Predictions.
Experimental Setup
4.1 Datasets
Training datasets. To train multi-view relighting diffuusio,, we use a dataset 400k synthetic 3D objects, including 100k from Objaverse [46]. Each object is rendered in 64 views x 16 HDR illuminations, with environments sampled from Polyhaven [47] (590 maps) and augmented via random up-axis rotations.
Evaluation datasets. For relighting evaluations, we used two datasets: TensoIR [45] and Stanford- ORB [48]. TensoIR is a synthetic benchmark, which contains renderings of four objects under six lighting conditions. We use the train split of 100 views with "sunset" lighting condition as inputs for relightable NeRF. We then evaluate 200 novel views under other five environment maps, including "bridge", "fireplace", "forest", "city", and "night". total, we have 4,000 renderings evaluation metric calculation. Stanford-ORB is real-world benchmarks by data capture in the wild. It has 14 objects composed of various materials. Each object captured under three distinct lighting conditions, producing total of 42 (object, lighting) combinations. Following its evaluation protocol,
5

---

Ground Truth
Ours
lllumiNeRF
Neural Gaffer
R3DGS
Neural PBIR
NeRO
TensorIR
Figure 4: Qualitative comparisons on TensoIR [45]. All renderings are rescaled to the image resolution of the ground truth. Compared to previous works, our method recovers more plausible specular highlights and more accurate colors as indicated with the red box.
we use images of an object under single lighting condition and evaluate novel views under the two target lighting settings.
4.2 Implementation details
Multi-View Diffusion model. We fine-tune model starting from pre-trained latent image generation model, as described in [49]. The multi-view denoiser is derived from the CAT3D network architecture [37], with modifications to the input channel dimensions align with our relighting task. The inputs to our model are images with resolution 512 x 512 which are encoded to 64 x 64 x latents. We chose the number of views to be 64. The model was trained on 128 TPU v5 chips using learning rate of 10-4, with total batch size of 128 for 360k iterations. After training, we generate the multi-lllumination dataset by running relighting diffusion inference 111 environment maps.
Relightable NeRF model. We train our NeRF on 8 H100s for 500k steps. We use a 512 512 resolution environment map the target illumination. We sample each reflection rays 3 times; once using a point sample on the full resolution environment map, and then using Gaussian kernels of sizes 20 x 20 and 40 40 pixels in radius with a; values of 10 and 20 respectively (see Fig. 3). In order to maximize the number of Illumination conditions we use for training, we make several reductions to the size of model relative the NeRF- EaRFi-Cstngg architecture. We lower the batch size to 1,000 and increase the number training steps to 500,000. We also decrease the size of the "bottleneck" vector b in both the geometry and appearance MLPs relative NeRF-Casting. Please refer to supplementary material for more details on the base architecture.
4.3 Baselines
We compare method against existing inverse rendering methods including NeRFactor [50]. InvRender [51], PhySG [52], NeRD [19], NVDiffRecMC [14], NVDiffRec [15], Neural-PBIR [53].
NeRO TensooRR [45], and recent single-view relighting diffusion methods IllumiNeRF [7] and
Neural Gaffer [6]. We also include the most recent Gaussian splatting-base based inverse rendering method
R3DG [54].
4.4 Evaluation Metrics
We evaluate the relighting rendering quality by PSNR, SSIM [55], and LPIPS-VGG [56].
5
Results
Our method the tor-perforning technique on existing relighting benchmarks for synthetic and real- world objects. Furthermore, it can render images from novel viewpoints under novel illuminations at

---

Method
PSNR-H f
PSNR-L
SSIM 1
LPIPS J
NVDiffRecMC [57]
25.08
32.28
0.974
0.027
NVDiffRee [15]
24.93
32.42
0.975
0.027
PhySG [52]
21.81
28.11
0.960
0.055
NVDiffRee [1.5]
22.91
29.72
0.963
0.039
NeRD [19]
23.29
29.65
0.957
0.059
NeRFactor [50]
23.54
30.38
0,969
0.048
InvRender [5]]
23.76
30.83
0,970
0,046
NVDiffRecMC [57]
24.43
31.60
0.972
0,036
Neural-PBIR [53]
26.01
33.26
0.979
0.023
R3DG [54]
21.25
27.50
0.962
0.063
Neural Gaffer [6]
23.16
29.94
0.966
0.047
IllumiNeRF [7]
25.56
32.74
0.976
0.027
Ours
26.21
32.91
0.980
0.027
Table StandfordORB benchmark [48]. We evaluate fourteen objects captured in the real world. Each object was captured in three different lightings. For each object-lighting pair, we evaluate novel view renderings under the other two lighting. The benchmark contains 840 renderings in total. denotes models trained with the ground-truth 3D scans and pseudo materials optimized from light-box captures. Best and 2nd-best are highlighted.
interactive speeds (0.384 seconds per frame). The only other methods that achieves similar relighting and rendering speeds are ones based on Gaussian splatting combined with inverse rendering, such as R3DG (0.415 seconds per frame), but the quality of these methods is significantly lower than of generative methods or inverse rendering methods that are based on NeRF.
5.1 TensoIR benchmark
In Fig. our method achieves state- of-the-art performance on the TensoIR benchmark, improving upon the ability of prior works to capture specularities in the reflective "hot dog" and "ficus" scenes while accurately capturing diffuse appearance from global illumination in the "lego" and "armadillo" scenes. The superiority of our method can also be verified by quantitative results in Tab. 1.
Method
PSNR t
SSIM
LPIPS 4
NeRFactor [50]
23.38
0.908
0.131
InvRender [51]
23.97
0.901
0.101
TensoIR [45]
28.58
0.944
0.081
Neural-PBIR [53]
27.09
0.925
0.085
NeRO [44]
27.00
0.935
0.074
R3DG [54]
29.05
0.937
0.080
NeuralGaffer [6]
27.30
0.918
0.122
IllumiNeRF [7]
29.71
0.947
0.072
5.2 Stanford-ORB benchmark
Ours
30.74
0.950
0.069
In Fig. 5, our method also achieves state- of-the-art performance on the Stanford- effective
Table I: TensoIR benchmark [45]. We evaluate all four
objects in the benchmark, each under five novel lightings. Each object is rendered from 200 views for novel view evaluation under each lighting. Thus, we have 4,000 ren- derings in total for quantitative evaluation. Best and
ORB dataset, and is reflective objects like the "baking" and "ball" scenes where consistent reflections clearly show up in the reconstruction. We provide quantitative comparisons 2nd-best are highlighted.
Tab. 2, where our method outperforms others in PSNR-H and SSIM, and acheive second-best results PSNR-L and LPIPS. As discussed IllumiNeRF [7], our results are also qualitatively superior to those of Meural-PBII [53], but they are quantitatively due to the mostly-diffuse renderings of Neural-PBIR.
5.3 Real-world Objects
Finally, we demonstrate the ability of our method to relight "in-the-wild" captures of real objects with spatially-varying material properties under natural lighting Fig. 1. Since we have no ground truth images for real-world relighting evaluation, we only show qualitative results. Our method captures convincing specularities on wood and metal parts of the model sewing machine, as well accurate shadows cast by the arm of the sewing machine.
7

---

GT
Ours
IllomiNeRF
Neural
R3DGS
InvRender NVDifRReeMC Neural-
nDn
Figure 5: Qualitative comparisons on Stanford-ORB [48]. Renderings from all methods are rescaled to the image resolution of the ground truth. Compared to previous work, our method produces high-fidelity renderings with more faithful specular reflections highlighted in the red boxes.
Method
PSNR 1
SSIM
LPIPS
(a) No blurring in specular conditioning
31.35
0.90
0.080
(h) No specular conditioning
30.00
0.88
0.079
(c) No general conditioning
21.59
0.70
0.130
(d) Per-image appearance embeddings
19.12
0.62
0.160
(e) 128 128 environment map
29.26
0.89
0,082
(f) 64 x 64 environment map Our full model, 64-view, 111 envmaps
27.69
0.86
0.110
31.88
0.91
0.075
Table 3: Ablation studies on the "hotdog" scene from TensoIR [45]. See the text and Fig. 6 for corresponding qualitative results and additional explanations.
Best and 2nd-best are highlighted.
5.4 Ablation Studies
We next perform an ablation study of the different components of our method, done on the "hotdog" scene of the TensoIR benchmark, chosen since has the most interesting materials in that dataset. Each ablation is reported row of Tab. 3 and its corresponding column in Fig. 6.
Multi-scale specular conditioning. Our multi-scale specular conditioning features, which are computed by blurring the environment map using multiple kernel sizes as in Eq. 4, are provided to the model. Each scale designed to approximate the incident light averaged over set of directions corresponding to particular surface roughness. When we skip this blurring operation and use the

---

Ground truth
Our model
(a)
(b)
(d)
(1)
Figure 6: Qualitative ablation studies. We compare a ground truth image and the relighting results of our model with in high-quality images with slightly lower metrics, whereas (b) removing the specular conditioning altogether greatly reduces the accuracy of specular highlights. (c) We observe that our model produces significant artifacts when the general conditioning is removed or (d) replaced with per- image appearance embeddings; and that providing environment maps at lower resolutions of (e)
ablations (a)-(f) (a) Using specular conditioning without blurring results
128 x 128 or (f) 64 64 tends to blur specular highlights and introduce rendering artifacts.
Method
PSNR
SSIM
IPIPS +
Ours full model, view, 111 envmaps
31.04
0.86
0.082
Ours full model, view, 111 envmaps
31.86
0.90
0.077
Our full model, 64-view, 10 envmaps
29.12
0.82
0.086
Our full model, 64-view, 50 envmaps
31.78
0.88
0.079
Our full model, 64-view, 111 envmaps
31.88
0.91
0.075
Our full model, 64-view, 150 envmaps
31.90
0.92
0.075
Table 4: Ablation studies of number of views and illulimantions the "hotdog" scene from TensoIR [45]. Best and 2nd-best are highlighted.
environment map values directly, our model can still represent highly specular regions (see Fig. 6(a)), but struggles more with rough or diffuse surfaces, leading to drop in reconstruction metrics.
Specular conditioning, Next we remove the specular conditioning signal altogether, which leads to another small drop in reconstruction metries as well qualitative drop in the accuracy of rendered specular highlights.
General conditioning. Removing the general conditioning signal from our network results in significant artifacts and poor reconstruction metrics, since the general conditioning is the main mechanism for providing the target illumination to the relighting model.
Per-image appearance embeddings. An alternative to our conditioning signals is a per-image appearance embedding vector, similar the GLO codes in NeRF-in-thhe Wild [42]. While this allows the model to be trained on multiple illuminations, it does not generalize to new unseen lights. Additionally, unlike our model, found that the embedding vectors do not scale well to large number of illumination conditions, resulting in significantly worse qualitative and quantitative results.
Environment map resolution. Our full model uses conditioning signals based on an environment map of resolution 512 x 512. Computing the conditioning signals from environment maps of size 128 x 128 results in loss of detail in the specular highlights. Further lowering the resolution to x 64 results in rendering artifacts even for diffuse surfaces.
Number of views. Our full model learns the joint distribution of 64-view relighting. To analyze the impact the number views in the diffusion model, we compare relighting novel view synthesis results using 4, 16, and 64-view diffusion on the hotdog scene of TensolR dataset. As shown in Tab. 4, the 64-view diffusion model consistently outperforms the others across all metrics. This indicates that jointly denoising more views Icads to more consistent and higher-quality relit images.

---

Figure 7: Normal map rendering of our relightable NeRF on objects from the TensoIR dataset.
Number of environment maps. We also study the effect of environment map count on relightable NeRF training by using 10, 50, 111, and 150 illuminations. As shown in Tab. 4, more illuminations generally improve relighting performance, which saturates around 111 maps.
Normal map visualization. As shown in Fig. 7, our relightable neural radiance fields can render high-quality normal maps, which are comparable to prior inverse rendering techniques like TensolR, well novel view synthesis approaches like NeRF- NeRF-Casting which explicitly encourage geometry to be more surface-like.
5.5 Limitations
While our method improves 3D object relighting by achieving more accurate specularities and fast inference, it can still be further improved several aspects. First, although our relighting diffusion model is trained on objects with materials varying in diffuse albedo and roughness, which are the main sources of variation in real-life materials, we did not train on objects exhibiting phenomena such as subsurface scattering, refraction, volumetric effects. Expanding these complex materials would improve robustness and generalizability to real data. Second, we use environment maps define lighting conditions, which assumes that the light sources are infinitely far away from the object. Exploring more general lighting models containing near-field illumination components could enhance realism diverse illumination scenarios. Finally, our approach focuses on object-centric scenes, and extending it to large-scale scene relighting would be an exciting direction for future research.
training data to include
6 Conclusion
This paper introduces novel method for 3D object relighting, enabling fast, feed-forward relighting during inference. By modeling virtual light stage with generative multi-diffusion model, create a diverse dataset of multi-lllmiintionn images. This dataset serves as a prior to train a light- conditioned Neural Radiance Field (NeRF) model, which subsequently learns render the object under arbitrary target illumination conditions. Existing methods for 3D relighting often rely on inverse rendering techniques, constrained by shading models limited specific material types, they directly generate relit NeRFs, embedding the lighting within the model itself. This requires retraining the NeRF for each new lighting condition. In contrast, our proposed model exhibits generalization across diverse lighting conditions at inference, facilitating efficient feed-forward relighting. Experimental results demonstrate the effectiveness of our method in relighting complex real-world objects with high fidelity.
Acknowledgement. We would like to thank Xiaoming Zhao, Rundi Wu, Songyou Peng, Ruiqi Gao, Stan Szymanowicez, Hadi Alzayer, Ben Poole, Aleksander Holynski, Jason Zhang, Jonathan T. Barron, Peter Hedman, Alex Trevithick, and Jiahui Lei for their valuable contributions. We also extend our gratitude to Shlomi Fruchter, Kevin Murphy, Mohammad Babaeizadeh, Han Zhang and Amir Hertz for training the base text-to-image latent diffusion model.
10

---

References
[1] Paul Debevec. Rendering synthetic objects into real scenes: Bridging traditional and image-based graphics with global illumination and high dynamic range photography. Acm siggraph 2008 classes, pages 1-10. 2008.
[2] Paul Debevec, Tim Hawkins, Chris Tchou, Haarm-Piee Duiker, Westley Sarokin, and Mark Sagar. Acquiring the reflectance field of human face. In Proceddings of the 27th annual conference on Computer graphics and interactive techniques, pages 145-156, 2000.
[3] Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. NeRF: Representing scenes neural radiance fields for view synthesis. Communications of the ACM, 65(1)999-106, 2021.
[4] Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühhlr, and George Drettakis. gaussian splatting for real-time radiance field rendering. 2023.
[5] Chong Zeng, Yue Dong, Picter Peers, Youkang Kong, Hongzhi Wu, and Xin Tong. Dilightnet: Fine-grained lighting control for diffusion-based image generation. In ACM SIGGRAPH 2024 Conference Papers, pages 1-12, 2024.
[6] Haian Jin, Yuan Li. Fujun Luan, Yuanbo Xiangli, Sai Bi, Kai Zhang, Zexiang Xu, Jin Sun, and Noah Snavely. Neural gaffer: Relighting any object via diffusion. In Advances in Neural Information Processing Systems, 2024.
[7] Xiaoming Zhao, Pratul Srinivasan, Verbin, Keunhong Park, Ricardo Martin Brualla, and Philipp Henzler. Illuminerf: 3d relighting without inverse rendering. arXiv preprint arXiv:2406.0633 2024.
[8] Ravi Ramamoorthi and Pat Hanrahan. signal processing framework for inverse rendering. SIGGRAPH, 2001.
[9] Yizhou Yu, Paul Debevec, Jitendra Malik, and Tim Hawkins. Inverse global illumination: Recovering reflectance models of real scenes from photographs. In SIGGRAPH, 1999.
[10] Yoichi Sato, Mark D Wheeler, and Katsushi Ikeuchi. Object shape and reflectance modeling from observation. In Proceedings the 24th annual conference on Computer graphics and interactive techniques, pages 379-387, 1997.
[11] Hendrik PA Lensch, Jan Kautz, Michael Goesele, Wolfgang Heidrich, and Hans-Peter Seidel. Image- based reconstruction of spatial appearance and geometric detail. ACM Transactions on Grapkics (TOG), 22(2):234 257, 2003.
[12] Tzu-Mao Li, Miika Aittala, Frédo Durand, and Jaakko Lehtinen. Differentiable monte carlo ray tracing through edge sampling. ACM Trans. Graph. (Proc. SIGGRAPH Asia), 37(6):222 11, 2018.
[13] Merlin Nimier- David, Delio Vicini, Tizian Zeltner, and Wenzel Jakob. Mitsuba 2: A retargetable forward and inverse renderer. Transoctions Graphics (Proeedings of SIGGRAPH Asia), 38(6), December 2019.
[14] Jon Hasselgren, Nikolai Hofmann, and Jacob Munkberg. Shape, Light, and Material Decomposition from Images using Monte Carlo Rendering and Denoising. arXiv:2206.03325 2022.
[15] Jacob Munkberg, Jon Hasselgren, Tianchang Shen, Jun Gao. Wenzheng Chen, Alex Evans, Thomas Müller, and Sanja Fidler. Extracting triangular 3d models, materials, and lighting from images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9280-8290 2022.
[16] Kai Zhang, Fujun Luan, Zhengqi Li, and Noah Snavely. Iron: Inverse rendering by optimizing neural sdfs and materials from photometric images. In CVPR, 2022.
[17] Sai Bi, Zexiang Xu, Pratul P. Srinivasan. Ben Mildenhall, Kalyan Sunkavalli, Miloλa Haλąan, Yannick Hold-Geoffroy, David Kriegman, and Ravi Ramamoorthi. Neural reflectance fields for appearance acquisition. arXiv, 2020.
[18] Pratul P. Srinivasan, Boyang Deng, Xiuming Zhang, Matthew Tancik, Ben Mildenhall, and Jonathan Barron. NeRV: Neural reflectance and visibility fields for relighting and view synthesis. In CVPR, 2021.
[19] Mark Boss, Raphael Braun, Varun Jampani, Jonathan T Barron, Ce Liu, and Hendrik Lensch. Nerd: Neural reflectance decomposition from image collections. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 12684- 12694, 2021.

---

[20] Xiuming Zhang, Pratul P. Srinivasan, Boyang Deng, Paul Debevec, William T. Freeman, and Jonathan T. Barron. NeRFactor: Neural factorization of shape and reflectance under unknown illumination. In SIGGRAPH Asia, 2021.
[21] Alexander Mai, Dor Verhin, Falko Kuester, and Sara Fridovich-Keil. Neural mierofacet fields for inverse rendering. In ICCV, 2023.
[22] Haian Jin, Isabella Liu, Peijia Xu, Xiaoshuai Zhang, Songfang Han, Sai Bi, Xiaowei Zhou, Zexiang Xu, and Hao Su. Tensoir: Tensorial inverse rendering. In CVPR, 2023.
[23] Benjamin Attal, Dor Verbin, Ben Mildenhall, Peter Hedman, Jonathan T Barron, Matthew OāAŽToole, and Pratul P Srinivasan. Flash cache: Reducing bias in radiance cache hased inverse rendering. In European Conference on Computer Vision, pages 20-36. Springer, 2024.
[24] Xiang Feng, Chang Yu, Zoubin Bi, Yintong Shang, Feng Gao, Hongzhi Wu, Kun Zhou, Chenfanfu Jiang, and Yin Yang. Arm: Appearance reconstruction model for relightable 3d generation. arXiv preprint arXiv:2411. 10825, 2024.
[25] Peter-Pike Sloan, Jan Kautz, and John Snyder. Precomputed radiance transfer for real-time rendering in dynamic, low- frequenc lighting environments. In SIGGRAPH, 2002.
[26] Ren Ng. Ravi Ramamoorthi, and Pat Hanrahan. All frequency shadows using non-linear wavelet lighting approximation. In SIGGRAPH, 2003.
[27] Peter-Pike Sloan, Jesse Hall. John Hart, and John Snyder. Clustered principal components for precomputed radiance transfer. In SIGGRAPH, 2003.
[28] Paul Debevec, Tim Hawkins, Chris Tehou, Haarm-Pice Pieter Duiker, Westley Sarokin, and Mark Sagar, Acquiring the reflectance field of human face. In SIGGRAPH, 2000.
[29] Zexiang Xu, Kalyan Sunkavalli, Sunil Hadap, and Ravi Ramamoorthi. Deep image-based relighting from optimal sparse samples. ACM Transactions Graphicx (TOG), 37(4):126, 2018.
[30] Tiancheng Sun, Jonathan T Barron, Yun-Ta Tsai, Zexiang Xu, Xueming Yu, Graham Fyffe, Christoph Rhemann, Jay Busch, Paul E Dehevec, and Ravi Ramamoorthi. Single image portrait relighting. ACM Transactions on Graphics (TOG), 38(4):79-1, 2019.
[31] Yohan Poirier-Ginter Alban Gauthier, Julien Phillip, J-F Lalonde, and George Drettakis. A diffusion approach to radiance field relighting using multi-ill iitiillluinnniinn synthesis. In Computer Graphics Forum, volume 43, page e15147, Wiley Online Library, 2024.
[32] Hadi Alzayer. Philipp Henzler. Jonathan T Barron, Jia- Bin Huang. Pratul P Srinivasan, and Dor Verbin. Generative multiview relighting for 3D reconstruction under extreme illumination variation. arXiv preprint arXiv:2412.15211, 2024.
[33] Chong Zeng. Guojun Chen, Yue Dong. Pieter Peers, Hongzhi Wu, and Xin Tong. Relighting neural radiance fields with shadow and highlight hints. In ACM SIGGRAPH 2023 Conference Proceedings, pages 1-11, 2023.
[34] Jiahui Fan, Fujun Luan, Jian Yang, Miloš Hašan, and Beibei Wang. Rng: Relightable neural gaussians. arXiv preprint arXiv:2409. 2024.
[35] Tianyuan Zhang, Zhengfei Kuang, Haian Jin, Zexiang Xu, Sai Bi, Hao Tan, Zhang, Yiwei Hu, Milos Hasan, William T Freeman, et al. Relitimm: Generative relightable radiance for large reconstruction models. arXiv preprint 241.:24100002333 2024.
[36] Ruofan Liang, Zan Gojeic, Huan Ling, Jacob Munkberg, Jon Hasselgren, Chih-Hao Lin, Jun Gao, Alexander Keller, Nandita Vijaykumar, Sanja Fidler, et al. Diffusion renderer: Neural inverse and forward rendering with video difffusion models. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 26069-26080, 2025.
[37] Ruiqi Gao, Alcksander Holynski, Philipp Henzler, Arthur Brussce, Ricardo Martin- Bautin, Pratul P. Srinivasan, Jonathan T. Barron, and Ben Poole. Cat3d: Create anything in 3d with multi-view diffusion models. Advances in Neural Information Processing Systems, 2024.
[38] Samuel W Hasinoff, Dillon Sharlet, Ryan Geiss, Andrew Adams, Jonathan T Barron, Florian Kainz, Jiawen Chen, and Marc Levoy. Burst photography for high dynamic range and low-light imaging on mobile cameras. ACM Transactions on Graphics (ToG), 15(6):1-12, 2016.
12

---

[39] Paul Debevec, Erik Reinhard, Greg Ward, and Sumanta Pattanaik. High dynamic range imaging. In ACM SIGGRAPH 2004 Course Notes, pages 14-es. 2004.
[40] Dor Verbin, Pratul P Srinivasan, Peter Hedman, Ben Mildenhall, Benjamin Attal, Richard Szeliski, and Jonathan T Barron. Nerf-casting: Improved view-dependent appearance with consistent reflections. In SIGGRAPH Asia 2024 Conference Papers, pages 1-10, 2024.
[41] Dor Verbin, Peter Hedman, Ben Mildenhall, Todd Zickler, Jonathan Barron, and Pratul P. Srinivasan. Ref-NeRF: Structured view-dependent appearance for neural radiance fields. CVPR, 2022.
[42] Ricardo Bartin-Brualll, Noha Radwan, Mehdi S. Sajjadi, Jonathan T. Barron, Alexey Dosovitskiy, and Daniel Duckworth. NeRF in the Wild: Neural Radiance Fields for Unconstrained Photo Collections. In CVPR. 2021.
[43] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the IEEE/CVF international conference computer vision, pages 9650-9660, 2021.
[44] Yuan Liu, Peng Wang. Cheng Lin, Xiaoxiao Long, Jiepeng Wang. Lingjie Liu, Taku Komura, and Wenping Wang. Nero: Neural geometry and brdf reconstruction of reflective objects from multiview images. 2023.
[45] Haian Jin, Isabella Liu, Peijia Xu, Xiaoshuai Zhang, Songfang Han, Sai Bi, Xiaowei Zhou, Zexiang Xu, and Hao Su. Tensoir: Tensorial inverse rendering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 165 174, 2023.
[46] Matt Deitke, Dustin Schwenk, Jordi Salvador, Luca Weihs, Oscar Michel, Eli VanderBilt, Ludwig Schmidt, Kiana Ehsani, Aniruddha Kembhavi, and Ali Farhadi. Objaverse: A universe of annotated 3d objects. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13142- 13153, 2023.
[47] Greg Zaal, Rob Tuytel, Rico Cilliers, James Ray Coek, Andreas Mischok, Sergej Majboroda, Dimitrios Savva, and Jurita Burger. Polyhaven: curated public asset library for visual effects artists and game designers, 2021.
[48] Zhengfei Kuang, Yunzhi Zhang. Hong-Xing Yu, Samir Agarwala, Elliott Wu, Jiajun Wu, et al. Stanford orb: real-world 3d object inverse rendering benehmark. Advances in Neural Information Processing Systems, 36, 2024.
[49] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684- 1064-, 2022.
[50] Xiuming Zhang, Pratul P Srinivasan, Boyang Deng, Paul Debevec, William T Freeman, and Jonathan T Barron. Nerfactor: Neural factorization of shape and reflectance under an unknown illumination. ACM Transactions on Graphics (ToG), 40(6):1-18, 2021.
[51] Yuanqing Zhang, Jiaming Sun, Xingyi He, Huan Fu, Rongfei Jia, and Xiaowei Zhou. Modeling indirect illumination for inverse rendering. In Proceedings of the IEEF//VV Conference on Computer Vision and Pattern Recognition, pages 18643-18652, 2022.
[52] Kai Zhang, Fujun Luan, Qianqian Wang, Kavita Bala, and Noah Snavely. PhySG: Inverse rendering with spherical gaussians for physics-based material editing and relighting. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1463-5462, 2021.
[53] Cheng Sun, Guangyan Cai, Zhenggin Li, Kai Yan, Cheng Zhang, Carl Marshall, Jia- Bin Huang, Shuang Zhao, and Zhao Dong. Neural-pbin pbir: reconstruction of shape, material, and illumination. In Proceedings the IEEE/CVF International Conference on Computer Vision. pages 1846-18056, 2023.
[54] Jian Gao, Chun Gu, Youtian Lin, Zhihao Li, Hao Zhu, Xun Cao, Li Zhang, and Yao Yao. Relightable gaussians: Realistic point cloud relighting with brdf decomposition and ray tracing. In European Conference on Computer Vision, pages 73 89. Springer, 2025.
[55] Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Ecro Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE transactions on image processing, 13(4):600-61) 2004.
[56] Richard Zhang, Phillip Isola, Alexii A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as perceptual metric. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 586 595,59952220 2018.
13

---

[57] Jon Hasselgren, Nikolai Hofmann, and Jacob Munkberg. Shape, light, and material decomposition from images using monte carlo rendering and denoising. Advances in Neural Information Processing Systems, 35:22856-22869, 2022.
[58] James Bradbury. Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclau- rin, George Necula, Adam Paszke, Jake VanderPlas, Skye Wanderman- Minne, et al. Jax: composable transformations of python+ numpy programs. 2018.
[59] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748 8763. PmLR, 2021.
[60] Jonathan Ho and Tim Salimans. Classificr-free diffusion guidance. arXiv preprint arX1v:2207.12598, 2022.
[61] Jiaming Song. Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02500, 2020.
[62] Jiaxiang Tang, Zhaoxi Chen, Xiaokang Chen, Tengfei Wang, Gang Zeng, and Ziwei Liu. Lgm: Large multi-view gaussian model for high-rcsolution 3d content creation. In European Conference Computer Vision, pages 18... Springer, 2024.
[63] Johannes L Schonberger and Jan-Michael Frahm. Structure-from motion revisited. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 4104-4111 2016.
[64] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead. Alexander C Berg, Wan- Yen Lo, et al. Segment anything. In Proceedings of JEFF/CVF international conference on computer vision, pages 4015-4026, 2023.
14

---

Appendix - ROGR: Relightable 3D Objects using Generative Relighting
A Supplementary Webpage and Videos
We suggest readers check the website in supplementary material. We provide more video renderings, including ablation studies and result comparisons on TensoIR [22]. StanfordORB [48], and real-world objects.
B Additional Implementation Details
B.1 Multi-view Relighting Diffusion
We implement our nult relighting diffusion model using JAX [58]. It is initialized from pre-trained latent diffusion model for text-to-image generation, similar to StableDiffusion [49]. Our model denoises multiple noisy latents of size 64 x 64 x 8 and decodes them into multi-view relit images size 512 512 x 3. Since our model not conditioned on text prompt, we only feed empty strings to the CLIP text encoder [59].
LDR
HDR
STC img
ray map
noise img
Src view
Src view 2
Src view
:
Src view N
2D UNet 3D Attention Layers
Relit output
Relit view 1
Relit view 2
Relit view 3
Relit view N
Figure 8: Multi-view Relighiting Diffusion Models. For each view, we concatenate noisy image latents, raymaps containing pose information, source image latents, HDR and LDR environment latents as inputs, and feed them into multi-view denoiser network that implemented by 2D UNet with additional 3D Attention layers. 2D UNet individually processes the latent feature from each view. 3D Attention layers flatten multi-vies image latents into ID sequence, and then perform self-attention to exchange information across different pixels and views.
15

---

In Fig. 8, we provide the detailed inputs of our relighting diffusion model. While our diffusion model receives and produces image latents, we depict them as images for better clarity. We encode RGB images under source lighting, HDR, and LDR target environment maps into latents of size 64 x 64 x 8. The raymaps consist of ray origins and ray directions corresponding to image pixels. We downscale them from the original image resolution 512 x 512 x 6 to the latent size 64 x 64 x 6. We concatenate the source image latent, HDR, LDR environment latents, raymaps, and noisy latents along the channel dimension to form new feature of size 64 64 x 38, and then feed it into multi-view diffusion network to produce clean target latents of size 64 x 64 8. Our denoiser network is based on CAT3D [37]. Please refer to Fig. 7 of CAT3D for the network architecture details.
During training. we use the DDPM schedule, with beta values that linearly increase from 8.5 x 10-4 to 1.2 x 10 2 over 1024 steps. We use noise prediction as our diffusion objective. The model was trained on 128 TPU v5 chips using a learning rate of 10-1, with a total batch size of 128 for 360k iterations and 10K warm-up steps. We adopted a progressive training scheme, where we first trained 4-view diffusion model for 300k steps, and then fine-tuned it for 16-view diffusion for 15k steps, and finally fine-tuned it for 64-view diffusion for 45 steps. We keep the learning rate as 10 when we fine-tune the model to relight the large number of views. We enable classifier-free guidance (CFG) [60] by randomly dropping the HDR and LDR environment maps with a probability of 0.1. During inference, we use the DDIM schedule [61] with 50 sampling steps and the classifier-fres weight is set 3.0.
B.2 Relightable NeRF
Our relightable NeRF is mainly based NeRF-Casting [40], which implicitly learnt the accumulated reflection features to model accurate and detailed reflections. Instead, we choose to explicitly learn these from given enviroment maps. Given single ray with origin and direction d, we sample N points (5) along the ray and use multi rustlitrs hash grid and MLP to encode x(9) into density 7(1), roughness (()), and surface normal (() Then, they are alpha composited to compute a single expected termination point X. von Mishes distribution (vMF) width k, and surface normal n. Next, we construct a reflection cone by reflecting around the micro-suee surface obtain vMF distribution over reflected rays vM (d', R).. We sample K reflected rays with location o' and (i) along each reflection ray. N' points x; are
d'z We cast these rays and sample N' points x; encoded into N' densities. Based on location o' and d', we can query illumination information
from environment maps and encode them with x) into features r through an MLP. These features
f are alpha-composed along each ray to get per-ray reflection feature p()) j. which can be further averaged into a single reflection feature f. Finally, f, bottleneck geometry feature b(O, mixing coefficients g(t) and view direction are feed into color decoder to predict the color value of x(1)
C Data
C.1
Training data preprocessing
We Objaverse [46] and an internal dataset containing high-quality 3D assets our training data. The Objaverse dataset is released under the Apache-2.0 license.
To ensure high-quality renderings for training, we filter out low-quality 3D assets from Objaverse using the object list provided in [62], and further exclude
objects, as our focus is on reflective and shiny surfaces. For 3D assets lacking texture or material information, assign uniform color sampled from [o, 1] as texture. We randomly sample three values from [0, 1] as the diffuse, roughness, and metallic terms of the material model.
Our relighting diffusion model requires multi-view images under diverse lighting conditions. To this end, we use 590 equirectangular environment maps from [47]. For each object, we randomly sample 64 camera poses on a sphere centered around the object. The camera distance ranges from [0.5, 2.0]. For each camera view, we randomly select 16 environment maps, augmenting each with a random horizontal shift. Then we use Blender's Cycles path tracer to render images at a resolution of 512x512, with 512 samples per pixel.
16

---

C.2 Evaluation benchmarks.
We use TensoIR [22] and StanfordORB [48] datasets for relighting evaluations. TensoIR under the MIT license. StanfordORB does not specify a license in the official GitHub repository.
C.3 Real-world data capture
We capture real-world objects to evaluate relighting model using Sony DSLR camera. Each object is placed on a table, and a handheld camera is used to record an image sequence by moving around the object. Each sequence contains approximately 160 to 300 frames. Camera intrinsics and extrinsics are estimated using COLMAP[63] for NeRF training. For foreground segmentation, we apply the pre-trained SAM [64] model. We will release our captured dataset upon publication under a non commercmallaa academic license.
C.4 Evaluation metrics
We evaluate the relighting rendering quality by PSNR, SSIM [55], and LPIPS-VGG [56] for low dynamic range (LDR) images. On the Stanford-ORB benchmark, we also compute PSNR for high dynamic range (HDR) images, denoted PSNR-H. while PSNR for LDR images is referred as PSNR-L. For methods that only produce LDR images, we apply the inverse process of the sRGB tone mapping transform the outputs into linear values. To address ambiguities in the relighting task, we align the predicted outputs with the ground truth images by applying channel-wise scale factors prior to metric computation. For Stanford- ORBB, these scale factors are determined separately for each output image. For TensoIR, a single global scale factor is calculated and uniformly applied all output images.
D
Social Impacts
Our work presents algorithm for photorealistic object relighting, which can provide immersive experience for VR/AR products. It has the potential to simplify labor-intensive workflows in 3D content creation and empower artists in the shopping. film, and gaming industries. It can also be used to augment 3D/multi -iew datasets with diverse lighting conditions, potentially benefiting downstream tasks that rely large-scale photorealistic 3D/multivview training data. However, this technology also carries potential risks. Misuse of the relighting capability might enable the creation of fraudulent harmful visual content. Additionally, our diffusion models require significant computational resources for training, which could bring environmental concerns due to high electricity consumption.
17

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
See abstract and Sec. 1.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: See Sec. 5.5 of the supplementary material.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
18

---

Justification:
No theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: See Sec. and Sec. of the supplementary material, and video results from website.
Guidelines:
The answer NA means that the paper does include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
19

---

Answer:
[No]
Justification:
reproducible, they We have not cleaned and released our data and code.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines ptttps://nipscoo cc/
public guwdes/id/eeeoe Code coonisssionin siiooooissssocii for more details.
While we encourage the release of code and data, we understand that this might not be possible, so åÄIJNoâÄI is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: //nips cc/ cc/publi c/ gu des /CodeSubm ss onPol icy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
See Sec. 4 and Sec. B of the supplementary material.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: We did report error bars statistical significance in our experiments due to the high computational cost of our model.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
20

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
See the Sec. and Sec. B of the supplementary material.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: /neuriiss cc cttppublcc/thihcccuuuuu lnes?
Answer: [Yes]
Justification:
We reviewer the NeurlPS Code of Ethics and preserve anonymity.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: See the Sec. D of the supplementary material.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
21

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [No]
Justification:
We have not cleaned and released our data and code.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: See Sec. C of the supplementary material.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms of use the package should be provided. For popular datasets, papers aw! thcode com da set. has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
22

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmoel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question:
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
23

---

Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification:
The LLM is used only for writing, editing, or formatting purposes in paper.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs any important, original, non-standard components. / ////urrripsp wcc.ooneeeenccs//////ococ 2025/
Please refer to our LLM policy (https: :
LLM) for what should or should not be described.
24


Reviews:
[{"review_id": "icLJ9ELTRe", "reviewer": "Reviewer_hTMe", "summary": "This paper proposes a method to reconstruct a relightable 3D model from multi-view images taken under the same illumination. The authors first use a multi-view diffusion model to generate a dataset consisting of the same object relighted using various environment maps (a total of 111 illuminations, to be specific). They then employ the generated dataset to train a relightable NeRF casting-based model, taking novel environment maps as conditions, which supports fast rendering (less than 1 second per frame). Specificaly, they propose to introduce conditions on the novel environment map to the relightable NeRF-Casting model in General(via DINO feature) and Specular(blur the environment map by by roughness and query corresponding reflection direction) ways to model both diffuse and specular effects during relighting. Experiments demonstrate improvements compared to previous works.", "questions": "Main concerns:\n* Since the paper highlight the specular high light as visual comparison, I would be curious about its comparison with methods that are more dedicated to the reflective objects like [1] and [2] , which are mentioned in the paper but not included in any comparisons. Specifically, for [2] the target illumination need to be provided in a different way by a rendering of the object (which can be either obtained from GT image or the relighted diffusion model as proposed)\n* Additional analysis of the geometry / normal map quality of the relightable NeRF model would be appreciated to evaluate its capability in reconstruction. \n* It would be appreciated to include analysis of the quality of the Multi-view Relighting Diffusion results, to give a hint on the supervision signal quality. e.g. comparing the relighting from the relightable rendering, the diffusion model, and the GT. (Although it might only be possible in the original camera view) \n\nOther minor concerns:\n* Ablation study or discussion on choosing the hyper-parameter of specular blur kernel size would be appreciated.\n* It is not very clear how many images are used to trained the relightable model. In the paper it mentioned running diffusion inference on 111 enthronement maps. So suppose N camera poses are provided, does it means a total of $N \\times 111$ images are used for training relitable NeRF model? Any specific selection on choosing 111 environment maps or why choosing this number?\n* In Fig.5 the cactus model of Neural-PBIR doesn’t seems to be the same as their paper and degraded too much. Including Neural-PBIR in TensoIR benchmark would also be appreciated.\n\n[1] Nero: Neural geometry and brdf reconstruction of reflective objects from multiview images.\n\n[2] Generative Multiview Relighting for 3D Reconstruction under Extreme Illumination Variation\n\nOverall the current evaluation is fair, and would be stronger with more evaluation.", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\nQuality: The paper is technically sound. The road map to first relight the object jointly (and therefore more consist) through diffusion model and then train a relightable NeRF model is well discussed and evaluated.\nClarity: The paper is clearly written with clear structure. Providing enough detail along with its supplementary. \nOriginality: Regarding the method, the multi-view relight diffusion model and relightable modification to the NeRF-Casting shows originality. \n\nWeaknesses:\nSignificance: The experiment results shows noticeable improvement compared to previous works.  As an end product, the relightable NeRF model can be useful as it can be rendered efficiently. However, as the method claims to be excel in specular highlight, additional comparisons with methods being more dedicated to such effect would be appreciated.", "was_revised": true, "final_justification": "The authors’ rebuttal provides satisfying additional information, including meaningful experiments, thorough ablation studies, and extra technical details. I will keep my accept recommendation for this paper.", "rebuttal": "Q1: Comparisons with NeRO, Neural-PBIR, and Generative Multi-view Relighting\n\nThe table below shows a comparison of our method with NeRO and Neural-PBIR on the TensoIR dataset. We will include these results in our revised version.\nAs pointed out by the reviewer, the setting for Generative Multi-view Relighting (GMR) is different than ours: we aim to relight objects from an unseen environment map, whereas GMR expects an observed image from the target lighting. Furthermore, GMR did not release their source code. \n\n| Method/Metrics | PSNR ↑  | SSIM ↑| LPIPS ↓  |\n|----------------------|--------|--------|--------|\n|Neural-PBIR |27.09|0.925|0.085|\n| NeRO| 27.00 | 0.935|0.074|\n| Ours | **30.74**|**0.950**|**0.069**| \n\nQ2: Additional analysis of the geometry / normal map quality of the relightable NeRF.\n\nThe quality of our method’s normals is comparable to prior inverse rendering techniques like TensoIR, as well as novel view synthesis approaches like NeRF-Casting which explicitly encourage geometry to be more surface-like. We will add normal visualizations to our final supplemental appendix.\n\nQ3: Analysis of the quality of the multi-view relight diffusion results. \n\nAs suggested, we compare the output of the multi-view relighting model with renderings from our full pipeline which includes the relightable NeRF. Note that the metrics are computed over the test views of the TensoIR dataset. While the NeRF results only use the relit training images, meaning that our pipeline never observed the test images, the evaluation of the relighting model required providing the test views to the relighting model as input. Even though our pipeline does relighting and novel view synthesis, it still performs only slightly worse than a model that gets access to the test views and only needs to relight them.\n\n| Method/Metrics | PSNR ↑  | SSIM ↑| LPIPS ↓  |\n|----------------------|--------|--------|--------|\n| NeRF output  | 31.88 | 0.91 | 0.075 |\n| Relighting model output  |**32.32**|**0.97**|**0.074**|\n\n\nQ4: Discussion on choosing the hyper-parameter of specular blur kernel size.\n\n​A 2D Gaussian kernel with a radius of 20 pixels (resulting in a 40 × 40 kernel) corresponds to an angular range of (40 / 512) × 360° = 28.125° centered around the reflection ray.\nA 2D Gaussian kernel with a radius of 40 pixels, aggregates lighting information within an angular range of (80 / 512) × 360° = 56.25°centered around the reflection ray.\nBased on empirical findings, this hyperparameter selection captures the light transport behavior of most reflective surface materials. \n\n\nQ5: Number of images used to train the relightable NeRF.\n\nGiven N cameras and M environment maps, we use a total of NxM images for training the relightable NeRF.\n\nQ6: The reason for choosing 111 environment maps.\n\nWe chose roughly 100 environment maps because our method’s quality increases with the number of environment maps and starts saturating around that point. In the table below we present relighting metrics on the hotdog scene from TensoIR for a varying number of environment maps used.\n| Method/Metrics | PSNR ↑  | SSIM ↑| LPIPS ↓  |\n|----------------------|--------|--------|--------|\n| 10  | 29.12  | 0.82 | 0.086 |\n| 50 | 31.78 | 0.88 | 0.079 |\n|111 | 31.88 | 0.91 | 0.075 |\n|150| **31.90**  |**0.92**|**0.075**|\n\nQ7: Cactus result from Neural-PBIR in Fig.5\n\nThank you for noticing this! We will fix the cactus visualization issue of Neural-PBIR in Fig.5."}, {"review_id": "3lchlhrCU9", "reviewer": "Reviewer_3veB", "summary": "This paper proposes ROGR, a method that takes inputs from multiple views of an object and reconstructs a lighting-conditioned Neural Radiance Field (NeRF) that can be relighted under different lighting conditions. The two main components are: a multi-view diffusion model and a relightable neural radiance field with general and specular conditioning to better encode lighting. Experiments are conducted on TensoIR and Stanford-ORB datasets, where ROGR demonstrates state-of-the-art performance on relighting image quality.", "questions": "1. One of the baselines that should be included in the experiment section (maybe as an ablation study) is: using the same multi-view relighting diffusion model + original NeRF and doing the per-illumination optimization. In other words, for each target lighting condition (env map), use the multi-view relighting model to generate relit images, and then use the original NeRF to do 3D reconstruction on those images. It is reasonable to believe ROGR has an advantage over such a baseline in efficiency, but the quality differences are unclear. \n\n2. While ROGR clearly outperforms others on TensoIR, its performance in Standford-ORB is fairly close to others. Can the authors explain the performance gap more here? \n\n3. I'm a bit surprised to see that the number of views plays a small role in the performance gain, as shown in Table 4. I would imagine it to be very important in obtaining high-quality 3D reconstruction and therefore the relit images. Did I miss something here? \n\nOverall, I'm fairly positive about this paper. If authors could answer questions and clarify points in the questions above, I would raise my score.", "limitations": "There is no discussion about the limitations.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "### Strengths\n- The main idea of obtaining 3D relightable objects without per-illumination optimization makes sense. The proposed solution with a multi-view 2D relighting diffusion model and a relightable neural radiance field seems to be a reasonable solution to the idea. \n- The performance of the relighting is solid: ROGR achieves state-of-the-art performance on widely-used benchmarks, competing favorably against other top methods. At the same time, it is fairly efficient. \n- The proposed 'General Conditioning' and 'Specular Conditioning' could be a good contribution to NeRF community to encode rich lighting information in 3D scenes. It might be useful for other 3D neural rendering models as well. \n\n### Weaknesses\n- Of the two components proposed, the multi-view 2D relighting diffusion model is heavily borrowed from existing work (e.g., CAT3D and Neural Gaffer) in various parts. There could be an argument here about the relatively weak technical novelty. \n- While the overall metric about the relighting quality is good in the experiment section, there are some missing pieces that require additional experiments. See below for questions about the experiment.", "was_revised": true, "final_justification": "After reviewing the authors response and other reviews, I updated my score to Accept. This paper proposes a solid approach to address the multiview relighting problem. It is a good contribution to the field.", "rebuttal": "Q1: Additional ablation study on per-illumination optimization using multi-view relit images and the original NeRF-Casting method.\n\nThank you for the suggestion! See the results of the proposed ablation study in the table below, where we optimize NeRF-Casting directly for each illumination. We will add it to our final revision. While our method performs on par with NeRF-Casting, it has the obvious advantage that it performs relighting in a feed-forward manner and does not require optimization per illumination.\n\nThe experimental setup is the same as Table 3 in our paper, which is from the hotdog scene from TensoIR. See the text and Fig. 6 for corresponding qualitative results and additional explanations. \n\n| Method/Metrics | PSNR ↑  | SSIM ↑| LPIPS ↓  |\n|----------------------|--------|--------|--------|\n| per-illumation optimization | 31.91 | 0.93 | 0.07|\n| Ours | 31.88 | 0.91| 0.075 |\n\nQ2: The performance gap between TensoIR and Stanford-ORB.\n\nThe Stanford-ORB benchmark provides illumination captured using a light probe that was repositioned for each image in the dataset, and is not colocated with the object. As such, their reported “environment maps” don’t capture the actual lighting condition at the object's position. The dataset's inaccuracies mean that any metrics derived from this benchmark should be interpreted with caution. This was also pointed out by IllumiNeRF [1] (Section 4.2) and Eclipse [2] (Section 5 and Section S1). This issue causes mismatches between the provided images and the relighting results obtained by every method, which effectively introduces noise to the metrics. This explains why it’s easier to see our method’s quality boost in the TensoIR data compared with Stanford-ORB.\n\n[1] IllumiNeRF: 3D Relighting Without Inverse Rendering\n\n[2] Eclipse: Disambiguating Illumination and Materials using Unintended Shadows\n\n\nQ3: Ablation study on the effect of varying the number of views in diffusion models for relightable NeRF reconstruction.\n\nGood idea! The table below shows the effect of increasing the number of views input to the diffusion model. For less reflective objects, the inconsistency from 4-view diffusion was already small enough that adding more views did not have a major impact and the NeRF model could reconcile the inconsistency. We observed qualitatively that increasing the number of views for shinier objects was more important, such as the car and sewing machine objects shown in the supplemental “in the wild” capture setup. \n\n| Method/Metrics | PSNR ↑  | SSIM ↑| LPIPS ↓  |\n|----------------------|--------|--------|--------|\n| 4-view | 31.04  | 0.86 | 0.082 |\n| 16-view | 31.86  | 0.90 | 0.077 |\n| 64-view |**31.88**|**0.91**|**0.075**|\n\nQ4: No discussion about limitations \n\nWe provided a limitation discussion in the supplementary material. If the reviewer would like us to, we can move it into our main paper in the final version of the paper."}, {"review_id": "YoFKqSECzm", "reviewer": "Reviewer_XvDU", "summary": "By utilizing multi-view diffusion to generate multi-view images of the same object under various environmental light sources, a NeRF conditioned on these light sources can be trained. This enables rapid, high-quality feedforward relighting in entirely new lighting conditions. The model was evaluated on the TensoIR and Stanfold-ORB datasets, outperforming state-of-the-art methods across multiple metrics.", "questions": "Kindly refer to the [Weaknesses].", "limitations": "Yes.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "### Strengths\n* The motivation for fine-tuning a multi-view diffusion conditioned on environmental light maps to generate multi-view, multi-light source images for a single object is highly reasonable.\n* Experimental validation demonstrates that the rendering results predicted by ROGR reflect the input environmental light map and exhibit clear specular phenomena.\n* The paper is well-written and easy to follow.\n\n### Weaknesses\n* The paper lacks experiments and analysis on fine-tuning the multi-view diffusion. The performance and generalization ability of Relightable NeRF largely depend on the generation quality, viewpoint consistency, and strict adherence to lighting conditions in the multi-view diffusion. It is recommended to include quantitative metrics to evaluate the multi-view diffusion in the relighting task, thus verifying that the proposed method can generate high-quality multi-view images under varying lighting conditions.\n* There is some confusion regarding the reconstruction of \"shadow\" and \"specular.\" In L247-248, the authors claim that the method captures convincing specular reflections and shadows. However, neither the relighting multi-view diffusion nor NeRF-Casting appears to have any strategies or designs that aid in accurately reconstructing shadows. As for \"specular,\" while the authors provide persuasive experimental results on \"specular\"/\"glossy\" surfaces, there is some doubt as to whether the model can be applied to reconstruct objects with purely specular or metallic surfaces. Shadows and specular highlights are highly sensitive to the distribution of high HDR values in the environmental light source and viewpoint variations. Could the authors provide theoretical analysis or experimental evidence explaining how the model reconstructs \"shadow\" and \"specular\"?\n* Recent works, such as RELITLRM[1] and DIFFUSIONRENDERER[2], can achieve \"feed-forward rendering or reconstruction in new environmental light sources,\" even without requiring dense view inputs or scene-specific optimization. These methods can achieve results similar to this work on both synthetic and real datasets. What are the advantages of this work over these relightable feed-forward 3D models? It is suggested to include a discussion of these methods in the related works section.\n\n[1] RelitLRM: Generative Relightable Radiance for Large Reconstruction Models. ICLR2025\n\n[2] DIFFUSIONRENDERER: Neural Inverse and Forward Rendering with Video Diffusion Models. CVPR2025", "was_revised": true, "final_justification": "The authors’ response and additional results have largely addressed my concerns. I will increase my score accordingly. Introducing generative models to tackle challenges such as shadows and specular highlights under complex illumination is a promising and worthwhile research direction.", "rebuttal": "Q1: Quantitative metrics to evaluate the multi-view diffusion in the relighting task.\n\nGood idea! The table below shows the effect of increasing the number of views input to the diffusion model. For less reflective objects, the inconsistency from 4-view diffusion was already small enough that adding more views did not have a major impact and the NeRF model could reconcile the inconsistency. We observed qualitatively that increasing the number of views for shinier objects was more important, such as the car and sewing machine objects shown in the supplemental “in the wild” capture setup. \n\n| Method/Metrics | PSNR ↑  | SSIM ↑| LPIPS ↓  |\n|----------------------|--------|--------|--------|\n| 4-view | 31.04  | 0.86 | 0.082 |\n| 16-view | 31.86  | 0.90 | 0.077 |\n| 64-view |**31.88**|**0.91**|**0.075**|\n\n\n\n\nQ2: Can the method handle shadows and specular highlights? What strategies are used to handle them?\n\nShadows and reflections are two challenging phenomena that our model is designed to handle in somewhat different ways. Shadows are strongly lighting-dependent, but they are not view-dependent. In order for our model to render correct shadows, the conditioning signals (and in particular the general conditioning) must encode the lighting in a way that allows the NeRF model to render them. This is enabled by our multi-view relighting model being trained to output correct shadows. Our model’s ability to handle shadows correctly is demonstrated in our supplementary material—see for example the lego scene with the rotating environment map at the top of the main page (index.html) or the car scene of StanfordORB (in stanford_orb.html).\n\nSpecular highlights are even more challenging since they are both strongly lighting-dependent and view-dependent. For rendering specular highlights, the diffusion model must be able to correctly relight the input images, and the NeRF must be able to use the conditioning signals (and in particular the specular conditioning) to create fast variations in appearance as the view direction changes. The strong dependence on view direction is achieved through the specular conditioning being concatenated to NeRF-Casting’s reflection feature, which was originally designed to enable rendering highly specular environments. We show that this pipeline is indeed effective in our results—see for example the hotdog scene at the top of the main page (index.html) and under additional lights in the TensoIR page (tensoir.html), and multiple objects like teapot and pepsi from StanfordORB (in stanford_orb.html).\n\nQ3: Discussion with RelitLRM and DIFFUSIONRENDERER.\nRelitLRM is a concurrent work to ours and directly recovers the 3D geometry and appearance of an object. While their method produces similar results to ours, their method cannot guarantee to produce the same geometry for different environment maps. Our two-stage strategy enables reconstruction of constant geometry under different illumination conditions. \n\nCompared to DiffusionRenderer, which uses a video diffusion model to generate relighting results, our method enforces 3D consistency by design, since we explicitly model the geometry and rendering, whereas DiffusionRenderer does not guarantee 3D consistency. Additionally, our model enables fast inference as it avoids the long sampling times typically required by diffusion-based generation."}, {"review_id": "Dsjzajj5iH", "reviewer": "Reviewer_xrU1", "summary": "The paper proposes a method to reconstruct a relightable 3D model of an object. The authors first construct a dataset of posed images under different illumination conditions based on a multi-view diffusion model. Then, they train a NeRF model conditioned on illumination with this dataset, such that the model can render images under novel lighting by conditioning on unseen environment maps.", "questions": "* Line 80: There might be a typo?  It’s unclear why the second sentence begins with “However”, the transition feels abrupt.\n* Line 137: Looks like the conditioning signals are illustrated in Figure 3 instead of Figure 2.\n* In section 3.2, the new relit dataset D_relit is for a single target object. Is that correct? Might be more clear to emphasize this in section 3.2\n* It would be helpful if the authors explicitly stated the input and output of the NeRF model. I assume it’s something like NeRF(xyz, ray direction, general conditioning, specular conditioning), where the two conditioning signals are trained with D_relit to learn to handle unseen environment maps, is this correct? If so, I would suggest making this overall idea clearer in Section 3.2.", "limitations": "Yes. It would be nice if the authors can provide a short overview of the limitations in the main paper.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n* The ablation study is thorough and investigates the effects of key design choices.\n* The experiments include comparisons with many existing methods and demonstrate the effectiveness of the proposed approach.\n\nConcerns:\n* The Method section contains a lot of information in text, however, I find some parts somewhat hard to follow. For example, it might be easier for the readers to understand if some of the concepts in section 3.1 are visualized in Figure 2. The current figure 2 is not very informative. More clarity-related questions are listed below. \n* It would be nice if Figure 6 can be larger, currently it is hard to tell the difference between a,b,e, and the proposed model. \n* The paper would be more insightful if the authors discussed why the baselines fail or why their method performs better in the Results section. This would help the community better understand the core contributions.", "was_revised": true, "final_justification": "This paper proposes a pipeline that combines multi-view diffusion with a NeRF model to achieve relightable neural rendering. While the multi-view diffusion component is largely based on existing works, the proposed pipeline demonstrates improved performance over multiple existing methods. The authors have addressed several concerns and are willing to polish the clarity in the final version. Therefore, I am leaning towards accepting this paper.", "rebuttal": "Q1: Visualize more concepts in Section 3.1 and make Figure 6 larger.\n\nThank you for the suggestion. We will update Figure 2 and integrate the concepts of Section 3.1 into it. We will also update Figure 6 with improved visibility for differences. Note that compared to (b) and (e), our final model produces more realistic specular highlights in the regions marked by the inset (specifically note the “ketchup”). Although (a) appears visually similar, our method achieves slightly better quantitative metrics as shown in Table 3.\n\nQ2: Add discussion in the results section—why baselines fail or why our method performs better.\n\nMethods based on inverse rendering such as NeRFactor, TensoIR, and R3DGS rely on accurate decomposition of geometry and materials, making them susceptible to compounded errors that lead to unrealistic relighting under novel illumination. In contrast, our method bypasses this decomposition by directly generating relit images.\nCompared to generative relighting methods like IlluminNeRF and Neural Gaffer, our approach leverages multi-view relighting diffusion, which produces more view-consistent outputs for relightable NeRF optimization. Additionally, our environment map-conditioned relightable NeRF more effectively utilizes lighting information for radiance prediction, resulting in higher-fidelity and physically-plausible relighting. We will add the result discussion into the revision.\n\nQ3 : Limitation discussion.\n\nWe provided a limitation discussion in the supplementary material, but we are happy to move it into our main paper in the final revision.\n\nQ4: Use of ‘However’ in Line 80\n\nThank you. We will revise \"[...] generalization. However [...]\" to \"[...] generalization, and [...]\".\n\nQ5: Figure reference issue in Line 137\n\nThank you for pointing this out. We will fix the sentence to: “The conditioning signals are illustrated in Fig. 3.”\n\nQ6: Clarification on the relit dataset\n\nCorrect. We will emphasize in Section 3.2 that the relit dataset D_{\\text{relit}} is constructed for a single target object.\n\nQ7: Inputs and outputs of the NeRF model.\n\nThe inputs to our relightable NeRF include 3D coordinates (x,y,z), ray direction, general conditioning, and specular conditioning. The geometry MLP predicts roughness, normals, and density, while the color MLP outputs the RGB values. We will explicitly describe these components in Section 3.2 to improve clarity."}]

Author rebuttal:

Meta-review: This paper was reviewed by 4 experts in the field. After authors’ feedback and internal discussion, all reviewers agreed that this is solid work and should accept it (rating: 5, 5, 5, 5).

Specifically, all reviewers agreed that the paper is well written, easy to follow and illustration is very clear. The experimental results are also very solid, with several comparisons with most of the recent methods on various benchmarks. The provided supplementary material also shows the effectiveness of the solution over the previous. The proposed solution is simple, novel and effective. Area chairs also agreed that this is high-quality work.

Considering all these, the decision is to recommend the paper for acceptance to NeurIPS 2025. We will also recommend this work for a spotlight presentation, for the following two reasons: 1) Excellent performance across thorough experiments. 2) Significant contributions on the relighting of 3D objects, a topic of substantial recent interest.

At last, we recommend the authors carefully read all reviewers’ final feedback and revise the manuscript as suggested in the final camera-ready version. We congratulate the authors on the acceptance of their paper!
Decision: Accept (spotlight)