Neural Gaffer: Relighting Any Object via Diffusion
Authors: Haian Jin
Yuan Li
Fujun Luan
Yuanbo Xiangli
Sai Bi
Kai Zhang
Zexiang Xu
Jin Sun
Noah Snavely
Venue: NeurIPS 2024 poster (2024)
Keywords: relighting
single-image relighting
diffusion models
3D relighting.
Primary area: machine_vision
TL;DR: We proposed an 2D relighting diffusion model that can relight any object image under any environmental lighting, which can be used as a prior in many 2D and 3D tasks, such as text-based relighting, object insertion, and relighting 3D representation.

Abstract: Single-image relighting is a challenging task that involves reasoning about the complex interplay between geometry, materials, and lighting. Many prior methods either support only specific categories of images, such as portraits, or require special capture conditions, like using a flashlight. Alternatively, some methods explicitly decompose a scene into intrinsic components, such as normals and BRDFs, which can be inaccurate or under-expressive. In this work, we propose a novel end-to-end 2D relighting diffusion model, called Neural Gaffer, that takes a single image of any object and can synthesize an accurate, high-quality relit image under any novel environmental lighting condition, simply by conditioning an image generator on a target environment map, without an explicit scene decomposition. Our method builds on a pre-trained diffusion model, and fine-tunes it on a synthetic relighting dataset, revealing and harnessing the inherent understanding of lighting present in the diffusion model. We evaluate our model on both synthetic and in-the-wild Internet imagery and demonstrate its advantages in terms of generalization and accuracy. Moreover, by combining with other generative methods, our model enables many downstream 2D tasks, such as text-based relighting and object insertion. Our model can also operate as a strong relighting prior for 3D tasks, such as relighting a radiance field.

Neural Gaffer: Relighting Any Object via Diffusion
Haian Jin
Yuan Li
Fujun Luan
Yuanbo Xiangli
Sai Bi
Kai Zhang
Zexiang Xu²
Jin Sun
Noah Snavely'
Commel Tech, Coreel University *Zhejiang University 3 Adobe Research *University of Georgia
Project Website:
hat.oll-gggafrgggggb.bbb.b io/
Abstract
Single-image relighting challenging task that involves reasoning about the complex interplay between geometry, materials, and lighting. Many prior methods either support only specific categories of images, such as portraits, or require special capture conditions, like using flashlight. Alternatively, some methods explicitly decompose a scene into intrinsic components, such as normals and BRDFs, which be inaccurate or under-expressive In this work, we propose novel end-to-end 2D relighting diffusion model, called Neural Gaffer that takes a single image of any object and synthesize accurate, high-quality relit image under any novel environmental lighting condition, simply by conditioning an image generator on target environment map, without explicit scene decomposition. Our method builds on pre-trained diffusion model, and fine-tunes it on a synthetic relighting dataset, revealing and harnessing the inherent understanding of lighting present in the diffuusin model. We evaluate our model on both synthetic and in-the- wild Internet imagery and demonstrate its advantages in terms of generalization and accuracy. Moreover, by combining with other generative methods, our model enables many downstream 2D tasks, such as text-based relighting and object insertion. Our model can also operate as a strong relighting prior for 3D tasks, such as relighting radiance field.
Introduction
Lighting plays key role in visual world, serving shape our interpretation and interaction with 3D space. The interplay of light and shadows can highlight textures, reveal contours, and enhance the perception of shape and form. In many cases, there is a desire to relight an image that is, to modify it as if it were captured under different lighting conditions. Such relighting capability enables photo enhancement (e.g., relighting portraits 18. 67)), facilitates consistent lighting across various scenes for filmmaking [59] 22 and supports
of the fundamental elements that
the seamless integration of virtual objects into real-world environments 38 19
However, relighting single images is a challenging task because it involves the complex interplay between geometry, materials, and illumination. Hence, many classical model-based inverse rendering approaches aim to explicitly recover shape, material properties, and lighting from input images 2 75: 47
33 87), that they then modify these components and rerender the image. These methods often require mueti-view inputs and can suffer from: 1) model limitations that prevent faithful estimation of complex light, materials, and shape in real-world scenes, and 2) ambiguities in determining these factors without strong data-driven priors.
To circumvent these issues, image- image-based relighting techniques 18 76 avoid explicit scene reconstruction and focus on high-quality image synthesis. However, some image-based methods
In film and television crews, gaffer responsible for managing lighting, including associated resources such as labor, lighting instruments, and electrical equipment
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Condition
In a bustling city square
Atop windswepp cliff overlooking the ocean
Input Image
Figure 1: Single-image relighting results on real data. Neural Gaffer supports single-image relighting for various input images under diverse lighting conditions, using either image-conditioned input (i.e.. an environment map) lext-conditioned input (i.e., a description of the target lighting). These results demonstrate capability to adapt to diverse lighting scenarios while preserving visual fidelity of the original objects. Our relighting results remain consistent with the lighting rotating. Please see the supplementary webpage for additional video results produced from real input images.
model's
18 58 76
require inputs under multiple lighting conditions even sophisticated capture
setups (e.g., a light stage [18), while recent deep learning- based techniques 67 single-view relighting but often work only on specific categories like portraits.
51 89
enable
In this paper, we propose novel calegory-agnostic approach for single-view relighting that tackles the challenges mentioned above. We introduce an end-to-end 2D relighting diffusion model, named Neural Gaffer, that takes single image of any object input and synthesizes an accurate and high-quality relit image under any environmental lighting condition, specified as high-dynamic range (HDR) environment map. In contrast to previous learning baaningg-basdd single-view models that are trained for specific categories, we leverage powerful diffusion models, trained on diverse data, and enable relighting of objects from arbitrary categories. Unlike prior model-based approaches, our diffusion-based data-driven framework enables the model to learn physical priors from an object- centric synthetic dataset featuring physically based materials and High Dynamic Range (HDR) environment maps. This facilitates more accurate lighting effects compared to traditional methods.
Our model exhibits superior generalization and accuracy on both synthetic and real-world images (See Fig. M. It also seamlessly integrates with other generative methods for various 2D image editing tasks, such as object insertion. Neural Gaffer also can serve as a robust relighting prior for neural radiance fields, facilitating novel two-stage 3D relighting pipeline. Hence, our diffusion-bast solution can relight any object under any environmental lighting condition for both 2D and 3D tasks. By harnessing the power of diffusion models, our approach generalizes effectively across diverse scenes and lighting conditions, overcoming the limitations of prior model- and image- based methods.
Related Work
Diffusion models. Recently, diffusion-based image generation models 226 49 have dominated
visual content generation tasks. Stable Diffusion [60] not only generates impressive results in the text-to-image setting, but also enables wide range of image applications. ControlNet [84] adds trainable neural network modules to the U-Net structure of diffusion models to generate images

---

conditioned on edges, depth, human pose, etc. Diffusion models can also change other aspects of
an input image 45 9. such as image style, via image-to-ine to-ia models [52]. ControlCom [80] and
AnyDoor [14] are image composition models that can blend a foreground object into a target image.
Huang et al. provide a comprehensive review of diffusion- -hased image editing [28].
Recent work has also shown that, despite being trained on 2D images, other kinds of information
can be unlocked from diffusion models. A notable example is Zero-I-to-3 [42), which uncovers 3D reasoning capabilities in Stable Diffusion by fine-tuning it on view synthesis task using synthetic data, and conditioning the model on viewpoint. Similar to the spirit of Zero- 1-to- and its follow-up work (such as 63 27 D, our work unlocks the latent capability of diffusion models for relighting.
Single-image relighting. Relighting imagery is a difficult task that requires an (explicit or implicit) understanding of geometry, materials, lighting, and their interaction through light transport. Single- image relighting is especially challenging due to its ill- ill-posedness; there are infinite explanations of a single image in terms of underlying factors. Classic single-image methods, like SIRFS [2) focus on explicit estimation of scene properties, via optimization with hand-crafted priors. More recent learning-basee methods can learn such priors from data, but either focus on specific image types,
such as outdoor scenes [73 69; 78 40, 211. portraits [53] 71 64| 67
48 51
23 56]. or human
Single-image relighting methods for general objects, with realistic appearance, materials, and lighting. remain
bodies [I 37 29], or require special capture conditions like camera flash 33]
elusive. In addition, most recent single-image relighting methods [5T] 39 61 [2]
still explicitly
decompose an image into components, then reconstruct it with modified lighting. This strategy allows the components to be explicitly controlled, but also limits their realism based on the representational ability of the chosen components, such as the Disney BSDF [I] or Phong reflection model [TO] In contrast, similar to [67 79), we achieve object relighting without an explicit scene decomposition. By training an maage-hased and dehhting-conditoone diffusion model, we can relight any given object from a single photo.
3D relighting and inverse rendering. Inverse rendering methods involve estimating 3D models that can be relit under different lighting conditions, typically from multiple images. Recent advances in 3D, such as NeRF 46 and 3D Gaussian Splatting 32 have shown remarkable novel view synthesis results. However, these representations are not themselves relightable as the physical light transport is baked into the radiance field. Efforts have been made toward creating relightable 3D representations. such PhySG 833. NeRFactor [87]. TensolR (30) and others 6
8 7
5 82
88
35
66 77: 81).
These methods often rely on pure optimization to minimize rendering loss, sometimes incorporating hand-crafted material and/or lighting regularization terms (e.g., smoothness or sparsity) to prevent overfitting. However, these approaches often struggle with specular highlights and glossy materials. We show that proposed Neural Gaffer method can be applied in an inverse rendering framework, serving as a powerful prior and regularizer. This is because our model has been trained on a large dataset encompassing diverse lighting and material conditions, effectively learning these priors from high-quality data. Our model improves 3D relighting quality, while baselines tend to produce artifacts due to the baking of lighting information into the color field.
3 Method
Neural Gaffer is a generalizable image-based based 2D diffusion model for relighting general objects; we finetune pretrained image- image-conditioned diffusion model [42] with synthetic relighting dataset to fulfill this goal. In Sec. 3.1] we describe how we construct a large-scale high-quality relighting dataset from Objaverse (20). Then, in Sec. 3.2 we discuss the design and training of 2D diffusion model to support zero-shot innll--image relighting. Finally, we showcase applications of our model in downstream tasks, including 2D object insertion and 3D radiance field relighting tasks.
3.1 RelitObjaverse Dataset
Capturing ground-truth lighting for real object images is both time-consuming and challenging. Existing real-world relighting datasets [41] [36] [7] are constrained to a very limited number of objects. To address this limitation, we opt render a large-scale synthetic object relighting dataset.
We use Objaverse (0 as our data source, which comprises about 800K synthetic 3D object models of varying quality. To avoid 3D object models with poor geometry texture, we developed a series of
3

---

filtering mechanisms. Specifically, we first utilize a classifier trained on a manually annotated subset categorize and filter out the 3D objects with low geometry and texture quality. In addition, we also design set of filtering rules based on 3D object attributes, such as geometry bounding boxes, BRDF attributes, and file size, eventually yielding a subset (of ~90K objects) of the original Objaverse with high quality. To ensure abundant lighting variations, we collected 1,870 HDR environment maps from the Internet, and augmented them with horizontal flipping and rotation, resulting ~30K different environment maps in total.
We render images with the Cycles renderer from Blender 17 During rendering, we randomly sample 12 camera poses for each object. For each viewpoint, we render 16 images under different environment maps. In contrast existing datasets that only render with environment maps [16] 62), we udditionally render an image under a randomly set area light source to increase lighting diversity. In total, for each object, there are 12 (16 + 1) = 204 images rendered under different lighting conditions and from different viewpoints. Our final synthetic relighting dataset has about 18.4M 512 512 rendered images with ground-truct lighting. We will release our full relighting dataset, named the RelitObjaverse dataset, upon acceptance to facilitate reproducibility.
3.2 Lighting-conditionee 2D Relighting Diffusion Model
Given a single object image E R W x3 and a new lighting condition E HR''' x3 (represented HDR environment map), our goal is learn model that can synthesize new image á with correct appearance changes under the new lighting, while preserving the object's identity:
XW'x3
iF [ιx E).
(1)
We fine-tune an image-to-image diffusion model using our rendered data. In particular, we adopt the pre-trained Zero-1-to- model [42] and discard its components and weights relevant to novel-view generation, repurposing the model for relighting. To incorporate lighting conditions,
extend the
input latents of the diffusion model, concatenating the encoded input image and the environment map with the denoising latent. We introduce two design decisions that are critical for the effective integration of lighting into the latent diffusion model.
Environment map rotation. In general, each pixel of an environment map E corresponds to 3D direction in the absolute world coordinate frame, whereas our relighting is performed in image space, that is, locked to the camera's 's coordinate frame. Therefore, need way for user to specify the relationship between the environment map and camera coordinate frames. One option is to concatenate the relative 3D rotation the embedding, similar to how Zero- 1-to-3 42 the relative camera pose for the novel view synthesis task. However, we observe that such a design makes it harder for the network to learn the relighting task, perhaps because it needs to interpret the complex relationship between 3D rotation parameters and the desired rotated lighting. Our solution instead rotate the target environment map to align with the target camera coordinate frame, prior to feeding it to the model. After rotation, each pixel of the rotated environment map corresponds to a fixed lighting direction in the target camera coordinate frame.
encodes
HDR-LDR conditioning. Pixel values in the HDR environment map have theoretical range of [0, +oo), and large pixel values can lead to unstable diffusion training. To rescale it to [0, 1.0], naive idea is to divide it by its maximum value, but this leads to a loss of total energy information. Also, if the map has a very bright region (e.g., the sun), then most of the other values will approach 0 and the environ- ment map will be too dark after division, causing loss of lighting detail. To solve this, we split the HDR environment map E into two maps: Es is an LDR representation of E, computed via standard tope-ng [57] 24). and Eμ is a nonlinearly nor- malized representation of E, computed by a logarith- mic mapping and divided by the new maximum value. E, encodes lighting detail low-intensity regions, which is useful for generating relighting details(such as reflection details). F, maintains the lighting in- formation across the initial spectrum. Inputting Et
Tarpet Ensironment Map
Imput
LOR Map
Normallaed HOR Nap
CUP
Encoder
Encoder
Encoder
embedding
Conditioring
C
Denoiser
Hoisy Latent
Decoder
Delt
Figure 2: Model architecture. Neural Gaffer is an img2img latent diffusion model conditioned on the input image and rotated lighting maps.
4

---

STAGE1
STAGE
Render
grad.
grad.
Input Poses
Encodar
HCC
Pretrained NeRF
Sampled Pose
Add
Geometry Appearance
Noana Gullrr
Nodial Guller
Noley Istents
Figure 3: Relighting a 3D neural radiance field. Given an input NeRF and a target environmental lighting, in Stage 1, we Neural Gaffer to predict relit images each predefined camera viewpoint. We then tune the appearance field overfit the multi-view relighting predictions with reconstruction loss. In Stage 2, further refine the appearance of coarsely relit radiance field via the diffusion guidance loss. Using this pipeline, we can relight a NeRF model in minutes.
and En together to the diffusion model means that network can reason effectively about the full original energy spectrum. Additionally, design helps to achieve balanced exposure in the generated output, preventing areas from appearing too dark or too washed out after normalization due to extreme brightness.
Diffusion model architecture and training. Our latent diffusion model comprises an encoder &, denoiser U-Net co, and a decoder 'D. During each training iteration, given the paired input-target images x and gg, and the target lighting condition E, we (1) rotate into the target camera coordinate frame based on specified lighting direction, forming a rotated map É; (2) transform É into an LDR enviromment map Ez and normalized HDR environment map En; (3) and finally, encode E,, Én and the input image r with the encoder E. The concatenation of the encoded latent maps are depicts 2D relighting system. We fine-tune the
used condition the denoising U-Net eg. Fig- 2
latent diffusion model with the following objective:
min E.g.E(
&,
where [1, 1000] is the diffusion time step, E = E())) represents the union of encoded latents for x, E, and En, and c(x) represents the CLIP embedding of the input image r. Example 2D relighting results from our trained model, applied to real images, shown in Fig.
3.3 Relighting 3D radiance field with diffusion prior guidance
We can utilize Neural Gaffer as a prior for 3D relighting. In particular, given a 3D neural radiance field consisting of density field G. and deeedeeendentt appearance field G. 46), we consider the task of relighting using our 2D relighting diffusion model as prior. We find that simply relighting the input NeRF with the popular Score Distillation Sampling (SDS) method [55] leads to over-saturated and unnatural results, as shown Fig. We instead propose two-stage pipeline (Fig. B to achieve high-quality 3D relighting in minutes. In the first stage, we use our 2D) relighting diffusion model to relight input views and overfit the original NeRF's appearance field, giving us coarsely relit renderings at sampled viewpoints. In the second stage, our model is used to further refine the appearance details with diffusion loss and an exponential annealed multi-step scheduling.
Stage 1: Coarse relighting with reconstruction loss. Given a NeRF with parameters u the multi- view input images Finput (2 and the target lighting Euagett we first use our diffusion model to generate multi-view relighting prediction images &reli [ where á; f(x,, Emsge).
Then, we freeze the density field G. of the NeRF, and only optimize the appearance field of the input NeRF with reconstruction loss Lrecun. aiming to minimize the reconstruction error between rendered image x = x(V, Tinput) and the relit image Trelit at pose Tinpur
Lrecon. (1)) [x((,), Timpur xneltt) 1|2.
After this stage, get initial blurry and noisy relighting result.
Stage 2: Detail refinement with diffusion guidance loss. The results from the first stage can be blurry and exhibit noise because the multi-view relighting predictions results may be inconsistent.
5

---

We further refine the relit NeRF's appearance in Stage 2. We found that directly using the SDS loss [55] for refinement leads to noisy and over-saturated results (see Fig. S). Inspired by [68] 74 151. we treat the blurry rendered images from Stage 1 as an intermediate denoising result. We first render an image å under the sampled camera pose Tsampled- We encode and perturb it to noisy latent ( (*) with an intermediate noise level t", and initiate the multi-step DDIM [65] denoising process for & (ā, E*) from t*, yielding a latent sample Eg. This latent is decoded to an image (teit(t)) = D(6).
which we use to supervise the rendering with the following diffusion guidance loss:
(b) = will f "rei((" )ll + w2Lp( (t)))
(4)
where Lp is the perceptual distance LPIPS [66] and w2 and w2 are scale factors to balance the two terms. We exponentially decrease t* during optimization.
4 Experiments
4.1
Implementation details
checkpoint and discard its original linear projection layer for image embedding and pose. We only fine-tune the UNet of the diffusion model and freeze other parts. During fine-tuning, we use a reduced image size of 256 x 256 and a total batch size of 1024. Both the LDR and normalized HDR environment maps are resized to 256 x 256. We use Adam W [43) and set the learning rate to 10-4 4 for training. We fine-tune our model for 80K iterations on 8 A6000 GPUs for 5 days.
Diffusion model training. We fine-tune our model starting from Zero123's
42
3D relighting. Given a neural radiance field reconstructed from multi-view inputs (we use Ten- soRF (I3). we first freeze its density field and then optimize the input NeRF's appearance field for 2,500 iterations in Stage 1, with a ray batch size of 4096. In Stage 2, we set the initial timestep t" to decrease from 0.4 to 0.05 exponentially. W1 and W2 in Eqn. 4are set to be 0.5 for all of our test cases. We optimize the results from Stage for 500 iterations, and for each iteration, we sample one view to compute the diffusion guidance loss in Eqn. Stage 1 takes about 5 minutes and Stage 2 takes about 3 minutes on a single A6000 GPU.
Downstream 2D tasks. Our model supports downstream 2D image editing tasks, such as text- based relighting and object insertion. For text-based relighting, given text prompt, we first use Tex12Light [15] to generate an HDR panorama, and, using that as an environment map, relight images using our diffusion model. Fig. M (right two columns) shows example results; more are on our project webpage. For object insertion, given an object image and target (perspective) scene image, we segment the foreground object with SAM 34, estimate lighting from the target background image with DiffusionLight [54], perform relighting, and finally generate a shadow attached to the object using [50]. Our workflow can preserve the object's identity and achieve high-quality visual quality.
4.2 Baselines and metrics
We use common image metrics including PNSR, SSIM, and LPIPS [6] to evaluate relighting results quantitatively. In addition, since there exists an ambiguity between lighting intensity and object color, we also compute channel- prediction rescaled to align the average intensity with the ground-truth. We compute metrics for both the raw and the channel-aligned results for quantitative evaluation.
results as in [87 88), where each channel of the relighting
For the single-image relighting task, there are very few prior works that can handle general objects under natural illumination. We compare to a very recent work DightttNet [79], which trains a lighting- conditioned ControlNet (84] to relight images, which is achieved by conditioning its ControlNet with a set of rendered images using predefined materials, estimated geometry of the input image, and target lighting. Per our request, the authors ran their model on our validation data. We select 48 high-quality objects from Objaverse as validation objects, which are unseen during training. We render each object under different camera poses. For each camera, we randomly sample 12 unseen environment maps to render the target relit images, and one additional environment map to render the input. We also compare with IC-Light [85). Since our lighting conditioning is different from its setting (they use background images), we only provide qualitative comparisons in our appendix and project webpage. For relighting 3D radiance fields, we compare with two 3D inverse rendering methods: TensoIR [30] and NVDIFFREC-MC (25). We select 6 objects from the NeRF-Synthetic

---

dataset [46] and Objaverse 20 for testing. For each object, 100 camera poses are sampled to generate training images and 20 camera poses are sampled for testing images. Each pose is rendered under four distinct, previously unseen environmental lighting conditions. One lighting condition is designated for training images, while the remaining three are reserved for relighting testing.
4.3 Results analysis
Comparing single-image relighting with DiLightNet [79]. As shown in Fig. 4 we evaluate across diverse set of unseen objects and unseen target lighting conditions. Our approach showcases superior fidelity to the intended lighting settings, maintaining consistent color accuracy and preserving intricate details. Compared with DiLightNet [79], our model can generate more accurate highlights, shadows, and high-frequent reflections. These results demonstrate our model's ability accurately understand the underlying physical properties of the object in the input image, such as geometry and materials, and faithfully reproduce the desired relighting appearance. Quantitative results are Tab.
Single-image relighting on real data. We evaluate our single-image relighting model on a set of real- world input photos (Fig. [. under either image-conditioned (environment map) or text-conditioned (text description of the desired target lighting) target illumination. As shown in the figure, our model effectively adapts diverse lighting conditions and produces realistic, high-fidelity relighting results under novel illumination. In addition, our relighting results are consistent and stable with the lighting condition rotating. Please refer our webpage to get more video results of more real data.
Input Image DiLightNet
Ours
Ground-truth Target Lighting
Figure 4: Single-image relighting comparison with DiLightNet (79 under diverse lighting, Our method demonstrates superior fidelity to the target lighting, maintains more consistent color and detail, and can generate more accurate highlights, shadows, and high- frequet reflections.
7

---

Table 1: Single-image relighting quantitative comparison.
Table 3D relighting quantitative comparison.
Relighting SSIM
Chamec-aligne Relightng
Method
PSNR
SSIM
LPIPS 1
PSNR
LPIPS
PSNR
SSIM
LPIPS
Ours
29.11
0.930
0.039
Ours
26.706
0,927
0.941
29.829
0,339
0,028
TensoIR
25.80
0.878
0.106
DiLighaNet
0.050
NVDIFFREC-MC
25
28.25
0.915
0.082
Object insertion. Our diffusion model exhibits versatility when combined with other generative
models for downstream 2D tasks, like object insertion. In Fig. 5 we take foreground image containing the object of interest and insert the object into background image depicting the desired target lighting condition. Our method surpasses baseline approaches by better preserving the identity of the inserted objects and synthesizing more natural illumination, seamlessly integrating objects into the background scene.
Foreground Image
Background Image
Est. Lighting & Relighting
Simply Copying
Anyr Results
Our Results
Figure 5: Object insertion. Our diffusion model can be applied to object insertion. Compared with Any-
Door 14 our method better preserves the identity of the inserted object and achieves higher-quality results.
Input Views
WVDIFFREC-MC
TensolR
Ground-truth
Target Lighting
Figure 6: Relighting 3D objects. We take multi-view images and a target lighting condition as input and compare our method's relighting results to NVDIFFREC-MC (25) and TensoIR [30]. (The shown images are channcl-aligned results.) Lacking strong priors, the basclines tend miss specular highlights (NVDIFFREC- MC), bake the original highlights (TensolR), or produce artifacts (NVDIFFREC-M) and TensoIR). In contrast, our method implicitly learns to model the complex interplay of light and materials from the training process of our relighting diffusion model, resulting in more accurate relighting results with fewer artifacts. Video comparisons available
project webpage.
Relighting 3D objects. Fig. showw the results of our two-stage method for relighting 3D objects. Given an input radiance (ield(NeRF) representing a 3D object, we first tune NeRF for coarse relighting in Stage 1, followed by finer relighting result in Stage 2 that improves fidelity. We also compared to
conventional inverse rendering methods such as NVDIFFREC-MC (5 and TensoIR (30] reporting
quantitative results in Tab. 2 and showing qualitative comparisons in Fig 6 and Fig
13

---

4.4
Ablation Study
Input image Figure 7: Ablations on the lighting conditioning designs for our single-image relighting diffusion model.
w/o Rotation
w/o LDR map w/o HDR map
Ours Ground-truth
Ablation studies for alternate lighting conditioning designs for our diffusion model.
As ablations, we evaluated different lighting condition- ing designs for our diffusion model: 1) only inputting one environment map rather than both the rotated LDR and normalized HDR map, and 2) controlling the light- ing direction by adding the relative 3D rotation between the default environment map coordinate frame and the target camera frame to the U-Net CLIP embedding branch without rotating the environment maps explicitly. Due to limited computational resources, when performing ablations for the model design, we train the different models with 2 A6000 GPUs for 48K iterations. We provide quantitative comparison between them in Tab. 3 and a qualitative comparison in Fig. 7 showcasing the effectiveness of our design. As shown in Fig 7 controlling the lighting direction by only adding the relative 3D lighting rotation to the U-Net CLIP embedding branch can't get the relighting results with correct lighting directions for all testing cases. Without inputting the LDR map, the model fails to generate the correct reflection for the first test case. Without the HDR map. it cannot produce the correct specular highlights for the third test case. Only when both the LDR and HDR maps are provided can the model accurately reason about the full original energy spectrum and generate correct relighting results for the second test case. Our full model achieves the best results across different ablation experiments.
Table 3: Ablations on diffusion model designs.
Method
PSNR
SSIM 1
LPIPS 1
Ours
26.052
0.923
0.035
map
25,503
0.920
0.045
Ours w/o HDR map
25.822
0.922
0.036
Ours w/o rotation
24.455
0.915
0.051
blation studies for relighting 3D radiance fields. As shown in Fig. g if we directly relight the input neural radiance field using SDS loss (55) without two-stage pipeline, cannot accurately reproduce highlights and reflections in the first scene, generate precise shadows in the
stage Relight w/ SDS loss wro stage 1
Input
w/o stage 2 Refine w/ SDS loss
Curs
Ground-truth
Figure 8: Ablations on methods for relighting 3D radiance fields. Our full model achieves the most accurate and visually realistic relighting results among all variants. Please zoom to see detailed differences across ablation variants.

---

VAE (res-512) Input Image Target EnvMap Pred Image Figure 9: Limitation Analysis. The left side shows
Image mecoding-decoolee results using the VAE [33 our base diffusion model 22) These results indicate that the VAE struggles to preserve identity for objects with fine details at low resolution (256 x 256), which in turn affects the performance of our model on such objects. On the right side, we present several failed cases of method in Objaverse instances.
Input Image VAE Ires-256)
second sofa scene, or remove shadows the third hotdog scene. Stage I ensures overall relighting accuracy, while Stage 2 helps reduce noise and leads to more detailed results. Without Stage 1, we cannot generate the correct specular highlight for the first scene, and the details of the shadows on the sofa scene's backrest become noisy and less accurate. Without Stage 2, the appearance of the relit scenes becomes noisier and blurrier, especially in the sofa scene. Replacing our diffusion guidance loss (Eqn. in Stage 2 with an SDS loss results in unnatural, noisy, and over-saturated relighting. Our full model consistently produces the most accurate and visually realistic results across all test variants. Please zoom in to see detailed differences across ablation variants.
5
Limitations
While our model can achieve overall consistent relighting results under rotating or changing lighting conditions, our results can still exhibit minor inconsistencies since the model is generative.
Given the high resource demands data preprocessing (specifically, rotating the HDR environment map) and model training, and considering our limited university resources, we trained the model at lower image resolution of 256 x 256. In Fig. 9 we use VAE [33] in our base diffusion model [I 42 to encode input images into latent maps, and then directly decode them. We found that VAE struggles to preserve identity for objects with fine details even from latent maps encoded from the input images this resolution, which in turn results in many relighting failure cases this resolution. Finetuning our model at a higher resolution will greatly help solve this issue.
We observe two key failure cases when relighting Objaverse Instances, as shown on the right side of Fig. 9 First, our model struggles with color ambiguity in diffuse objects, leading to inaccuracies in relit colors (first row). This issue arises from inherent ambiguities in the input image, where color could be attributed to either material or lighting. Including the background of the input image may help the model better infer the lighting conditions of input images and address this challenge. We give more discussions in the Appendix. [ Second, while the model generates low- low-frequency highlights well, it often misses high-frequency reflections in relit images (second row).
In addition, we discuss our limitations regarding portrait relighting in the Appendix.
6 Conclusion
We proposed an end-to-end 2D relighting diffusion model, Neural Gaffer, that addresses the long- standing challenge of single-image relighting through data-driven framework. Our model synthe- sizes accurate and high-quality relighted images from single input image under any environmental lighting condition, demonstrating superior generalization and accuracy on both synthetic and real- world datasets. Neural Gaffer not only enhances various 2D tasks, such as text-based relighting and object insertion but also serves as a strong relighting prior for 3D tasks, enabling a simplified two-stage 3D relighting pipeline. By overcoming the limitations and ambiguities of traditional model-based inverse rendering approaches, our diffusion-based solution extends the applicability of relighting techniques to broad range of practical applications.
10

---

Acknowledgments. We thank Minghua Liu, Chao Xu, et al from Prof. Hao Su's group at UCSD for assisting with the Objaverse filtering process. We thank Xinrui Liu at Cornell for helping with data capture and shadow generation for the object insertion application. We thank Chong Zeng at Zhejiang University for helping run his model (DiLLighNeet on our validation data
This work was done while Haian Jin was a full-time student at Cornell. The selection of data and the generation of all figures and results was led by Cornell University. This work was funded in part by the National Science Foundation (IIS-221 1259). Jin Sun is partly supported by a gift from Google.
References
[1] Stable diffusion image variations- a hugging face space by lambdalabs.
[2] Jonathan T. Barron and Jitendra Malik. Shape, illumination, and reflectance from shading. Trans. Pattern Analysis and Machine Intelligence, 2015.
[3] Sai Bi, Stephen Lombardi, Shunsuke Saito, Tomas Simon, Shih-En Wei, Kevyn Mcphail, Ravi Ramamoorthi, Yaser Sheikh, and Jason Saragih. Deep relightable appearance models for animatable faces. ACM Transactions on Graphics (TOG), 405 2021.
[4] Sai Bi, Zexiang Xu, Pratul Srinivasan, Ben Mildenhall, Kalyan Sunkavalli, Miloš Hašan, Yannick Hold-Geoffrng David Kriegman, and Ravi Ramamoorthi. Neural reflectance fields for appearance acquisition. arXiv preprint arXiv:2008. 2000.0034.... 2020.
[5] Sai Bi, Zexiang Xu, Kalyan Sunkavalli, Miloš Hašan, Yannick Hold-Geoffroy, David Kriegman, and Ravi Ramamoorthi, Deep reflectance volumes: Relightable reconstructions from multi-view photometric images. In Proc. European Conf. Computer Vision (ECCV), pages 294-311. Springer, 2020.
[6] Mark Boss, Raphael Braun, Varun Jampani. Jonathan T Barron, Ce Liu, and Hendrik Lensch. Nerd: Neural reflectance decomposition from image collections. In Proc. Int. Conf. on Computer
Vision (ICCV). pages 1 11684-12694, 2021.
[7] Mark Boss, Andreas Engelhardt, Abhishek Kar, Yuanzhen Li, Deqing Sun, Jonathan Barron, Hendrik Lensch, and Varun Jampani. Samurai: Shape and material from unconstrained real- world arbitrary image collections. Neural Information Processing Systems, 35:26389-26400
2022.
[8] Mark Boss, Varun Jampani, Raphael Braun, Ce Liu, Jonathan Barron, and Hendrik Lensch. Neural-pil: Neural pre-integrated lighting for reflectance decomposition. Neural Information Processing Systems, 34: 14691-1007704 2021.
[9] Tim Brooks, Aleksander Holynski, and Alexei A Efros. Instructpix2pix: Learning to follow image editing instructions. In Proc. Computer Vision and Pattern Recognition (CVPR), pages 18392-18402, 2023.
[10] Phong Bui-Tuong. Illumination for computer generated pictures. CACM. 1975.
[11]
Brent Burley and Walt Disney Animation Studios. Physically- shading at disney. In Acm Siggraph, volume 2012, pages 1-7. vol. 2012, 2012.
[12] Yuanhao Cai, Hao Bian, Jing Lin, Haoqian Wang, Radu Timofte, and Yulun Zhang, Retinex-
former: One-stage retinex-based transformer for low-light image enhancement. In ICCV, 2023.
[13] Anpei Chen, Zexiang Xu, Andreas Geiger, Jingyi Yu, and Hao Su. Tensorf: Tensorial radiance fields. In European Conference on Computer Vision (ECCV), 2022. [14] Xi Chen, Lianghua Huang, Yu Liu, Yujun Shen, Deli Zhao, and Hengshuang Zhao. Anydoor: Zero-shot object-level image customization, 2024.
[15] Zhaoxi Chen, Guangcong Wang, and Ziwei Liu. Text2light: Zero-shot text-driven hdr panorama generation, 2023.
[16] Jasmine Collins, Shubham Goel, Kenan Deng, Achleshwar Luthra, Leon Xu, Erhan Gundogdu, Xi Zhang. Tomas F. Yago Vicente, Thomas Dideriksen, Himanshu Arora, Matthiuu Guillaumin, and Jitendra Malik. Abo: Dataset and benchmarks for real-world 3d object understanding. 2022. [17] Blender Online Community. Blender 3D modelling and rendering package. Blender Foundation, Stichting Blender Foundation, Amsterdam, 2018
[18] Paul Debevec, Tim Hawkins, Chris Tehou, Haarm-Pieter Duiker, Westley Sarokin, and Mark Sagar. Acquiring the reflectance field of human face. In Proceedings of the 27th annual conference on Computer graphics and interactive techniques, pages 145-156, 2000.
[19] Paul E. Debevec. Rendering synthetic objects into real scenes: bridging traditional and image- based graphics with global illumination and high dynamic range photography. Proceedings of the 25th annual conference on Computer graphics and interactive techniques, 1998.
11

---

[20] Matt Deitke. Dustin Schwenk, Jordi Salvador, Luca Weihs, Oscar Michel, Eli VanderBilt, Ludwig Schmidt, Kiana Ehsani, Aniruddha Kembhavi, and Ali Farhadi. Objaverse: A universe
of annotated 3d objects. arXiv preprint arXiv:22/ 22000551 2022.
[21]
David Griffiths, Tobias Ritschel, and Julien Philip. Outcast: Outdoor single-maae relighting
with cast shadows. Computer Graphics Forum, (2):11999119) 2022.
[22]
Kaiwen Guo, Peter Lincoln, Philip L. Davidson, Jay Busch, Xueming Yu, Matt Whalen, Geoff Harvey, Sergio Orts, Rohit Pandey, Jason Dourgarian, Danhang Tang, Anastasia Tkach, Adarsh Kowdle, Emily Cooper, Mingsong Dou, S. Fanello, Graham Fyffe, Christoph Rhemann, Jonathan Taylor, Paul E. Dehevec, and Shahram Izadi. The relightables. ACM Transactions on Graphics (TOG), 38:1 - 19, 2019.
[23] Yuxuan Han, Zhibo Wang. and Feng Xu. Learning 3d morphable face reflectance model from low-cost data, 2023.
[24] Samuel W. Hasinoff, Dillon Sharlet, Ryan Geiss, Andrew Adams, Jonathan T. Barron, Florian Kainz, Jiawen Chen, and Mare Levoy. Burst photography for high dynamic range and low-light
imaging on mobile cameras. ACM Trans. Graph., 35(6), dec 2016.
[25] Jon Hasselgren, Nikolai Hofmann, and Jacob Munkberg. Shape, Light, and Material De- composition from Images using Monte Carlo Rendering and Denoising. arXiv:2206.03380, 2022.
[26] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Neural
Information Processing Systems, 33:6840-663 2020.
[27] Hanzhe Hu, Zhizhuo Zhou, Varun Jampani, and Shubham Tulsiani. Mvd-fusion: Single-view 3d via depth-consistent multi-view generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages
2024.
[28] Yi Huang, Jiancheng Huang, Yifan Liu, Mingfu Yan, Jiaxi Lv, Jianzhuang Liu, Wei Xiong, He Zhang, Shifeng Chen, and Liangliang Cao. Diffusion model-basee image editing: A survey, preprinne arXiv:2402. 1752., 2024. arXiv preprint
[29]
Chaonan Ji, Tao Yu, Kaiwen Guo, Jingxin Liu, and Yebin Liu. Geometry- aware single-image full-body human relighting. In ECCV, page 388-405 2022.
[30]
Haian Jin, Isabella Liu, Peijia Xu, Xiaoshuai Zhang, Songfang Han, Sai Bi, Xiaowei Zhou, Zexiang Xu, and Hao Su. Tensoir: Tensorial inverse rendering. In Proc. Computer Vision and
Puttern Recognition (CVPR), pages 165-174, 2023.
[31]
Yoshihiro Kanamori and Yuki Endo. Relighting humans: occlusion- aware inverse rendering for full-body human images. ACM Trans. Graph., 37(6), 2018.
[32] Bernhard Kerbl, Georgios Kopanas. Thomas Leimkühler, and George Drettakis. 3d gaussian
splatting for real-time radiance field rendering. ACM Transactions on Graphics, 1214))1-14, 2023.
[33] Diederik P Kingma and Max Welling. Auto-encoding variational bayes, 2022.
[34]
Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg, Wan-Yen Lo, Piotr Dollár, and Ross Girshick. Segment anything, 2023.
[35] Zhengfei Kuang, Kyle Olszewski, Menglei Chai, Zeng Huang, Panos Achlioptas, and Sergey Tulyakov. Neroic: Neural rendering of objects from online image collections. ACM Transactions on Graphics (TOG), -14) 2022.
[36] Zhengfei Kuang, Yunzhi Zhang, Hong-Xing Yu, Samir Agarwala, Shangzhe Wu, and Jiajun Wu. Stanford-orb: A real-world 3d object inverse rendering benchmark, 2024.
[37] Manuel Lagunas, Xin Sun, Jimei Yang, Ruben Villegas, Jianming Zhang, Zhixin Shu, Belen Masia, and Diego Gutierrez. Single-image full-body human relighting. Eurographics Symposium on Rendering- DL-only Track, 2021.
[38] Zhengqin Li, Jia Shi, Sai Bi, Rui Zhu, Kalyan Sunkavalli, Milovs Havsan, Zexiang Xu, Ravi Ramamoorthi, and Manmohan Chandraker. Physically-basee editing of indoor scene lighting from a single image. ArXiv, abs/2205.093332 2022.
[39] Zhengqin Li, Zexiang Xu, Ravi Ramamoorthi, Kalyan Sunkavalli, and Manmohan Chandraker. Learning to reconstruct shape and ppatiall-vvayinng reflectance from single image. ACM Transactions on Graphics (TOG), 37:1- 11, 2018.
[40] Andrew Liu, Shiry Ginosar, Tinghui Zhou, Alexei A. Efros, and Noah Snavely. Learning to factorize and relight a city, 2020.
[41] Isabella Liu, Linghao Chen, Ziyang Fu, Liwen Wu, Haian Jin, Zhong Li, Chin Ming Ryan Wong,
Yi Xu, Ravi Ramamoorthi, Zexiang Xu, and Hao Su. Openillumination: A multi-illuminttion dataset for inverse rendering evaluation on real objects, 2024.
[42] Ruoshi Liu, Rundi Wu, Basile Van Hoorick, Pavel Tokmakov, Sergey Zakharov, and Carl Vondrick. Zero- 1 3:o-3: Zero-shot one image 3d object, 2023.
12

---

[43] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization, 2019,
[44] Abhimitra Meka, Christian Haene, Rohit Pandey, Michael Zollhöfer, Sean Fanello, Graham Fyffe, Adarsh Kowdle, Xueming Yu. Jay Busch, Jason Dourgarian, et al. Deep reflectance fields: high Nigh-qqulity facial reflectance field inference from color gradient illumination. ACM Transactions on Graphics (TOG), 38(4): 1-12, 2019.
[45] Chenlin Meng, Yutong He, Yang Song, Jiaming Song, Jiajun Wu, Jun-Yan Zhu, and Stefano Ermon. Sdedit: Guided image synthesis and editing with stochastic differential equations. In International Conference on Learning Representations, 2021.
[46] Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. In ECCV, 2020.
[47] Giljoo Nam, Joo Ho Lee, Diego Gutierrez, and Min H Kim. Practical svbrdf acquisition of 3d objects with unstructured flash photography. ACM Transactions on Graphics (10G),37(6)111 2018.
[48] Thomas Nestmeyer, Françoisss Lalonde, lain Matthews, and Andreas Lehrmann. Learning physics-guided face relighting under directional light. In Proc. Computer Vision and Pattern Recognition (CVPR), pages 1124-5133. 2020.
[49] Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In International conference on machine learning, pages 8162-811 8171. PMLR, 2021. [50] Li Niu, Wenyan Cong, Liu Liu, Yan Hong, Bo Zhang, Jing Liang, and Liqing Zhang. Making images real again: A comprehensive survey on deep image composition. arXiv preprint arXiv:2106. I4490, 2021.
[51] Rohit Pandey, Sergio Orts Escolano, Chloe Legendre, Christian Haene, Sofien Bouaziz, Christoph Rhemann, Paul Debevec, and Sean Fanello. Total relighting: learning to relight portraits for background replacement. ACM Trans. Graph.. 40(4), 2021.
[52] Gaurav Parmar, Krishna Kumar Singh, Richard Zhang, Yijun Li, Jingwan Lu, and Jun- Yan
Zhu. Zero-shot image-to-imagee translation. In ACM SIGGRAPH 2023 Conference Proceedings, pages 1-11, 2023.
[53] Pieter Peers, Naoki Tamura, Wojeiech Matusik, and Paul Debevec. Post-production facial performance relighting using reflectance transfer. ACM Trans. Graph., 26(3), 2007.
[54] Pakkapon Phongthawee, Worameth Chinchuthakun, Nontaphat Sinsunthithet. Amit Raj. Varun Jampani, Pramook Khungurn, and Supasorn Suwajanakorn. Diffusionlight: Light probes for free by painting chrome ball. In ArXiv, 2023.
[55] Ben Poole, Ajay Jain, Jonathan T. Barron, and Ben Mildenhall. Dreamfusion: Text-to-3d using
2d diffusion. arXiv, 2022.
[56] Anurag Ranjan, Kwang Moo Yi, Jen-Hao Rick Chang, and Oncel Tuzel. Facelit: Neural 3d relightable faces. In Proc. Computer Vision and Pattern Recognition (CVPR), pages 8619-8628,
2023.
[57] Erik Reinhard, Greg Ward, Sumanta Pattanaik, and Paul Debevec. High Dynamic Range Imaging: Acquisition, Display, and Image- Lighting (The Morgan Kaufmann Series in Computer Graphics). Morgan Kaufmann Publishers Inc., San Francisco, CA. USA, 2005.
[58] Peiran Ren, Yue Dong, Stephen Lin, Xin Tong, and Baining Guo. Image based relighting using neural networks. ACM Transactions Christian Richardt, Carsten Stoll, Neil A. Dodgson, Hans-Peter Seidel, and Christian Theobalt. Coherent spatiotemporal filtering, upsampling and rendering of RGBZ videos. Computer Graphics Forum (Proceedings of Eurographics). 31(2), May 2012.
Graphics (ToG), 34(4):1-12, 2015.
[59]
[60] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björ Ommer. High- resolution image synthesis with latent diffusion models. In Proc. Computer Vision and Pattern Recognition ( (CVPR), pages 10684-10695, 2022.
[61] Shen Sang and M. Chandraker. Single-shot neural relighting and sybrdf estimation. In ECCV, 2020.
[62] Jian Shi, Yue Dong, Hao Su, and Stella X. Yu. Learning non-lambertian object intrinsics across shapenet categories, 2016.
[63] Ruoxi Shi, Hansheng Chen, Zhuoyang Zhang, Minghua Liu, Chao Xu, Xinyue Wei, Linghao Chen, Chong Zeng, and Hao Su. Zero 1oool2+++:: a single image to consistent multi-view diffusion base model, 2023.
[64] Zhixin Shu, Ersin Yumer, Sunil Hadap, Kalyan Sunkavalli, Eli Shechtman, and Dimitris Samaras. Neural face editing with intrinsic image disentangling. In Proc. Computer Vision and Pattern Recognition (CVPR), pages 5541-5550, 2017.
[65] Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models, 2022.
13

---

[66] Pratul P Srinivasan, Boyang Deng, Xiuming Zhang, Matthew Tancik. Ben Mildenhall, and Jonathan T Barron. Nerv: Neural reflectance and visibility fields for relighting and view synthesis. In Proc. Computer Vision and Puttern Recognition (CVPR), pages 7495-7504, 2021. [67] Tiancheng Sun, Jonathan T Barron, Yun-Ta Tsai, Zexiang Xu, Xueming Yu, Graham Fyffe, Christoph Rhemann, Jay Busch, Paul Debevec, and Ravi Ramamoorthi. Single image portrait relighting. ACM Trans. Graph., 38(4), 2019.
[68] Jiaxiang Tang, Jiawei Ren, Hang Zhou, Ziwei Liu, and Gang Zeng. Dreamgaussian: Generative gaussian splatting for efficient 3d content creation. arXiv preprint arXiv:2309. /6653, 2023. [69] Murat Türe, Mustafa Ege Çiklabakkal, Aykut Erdem, Erkut Erdem, Pinar Satilmıs, and Ah- met Oguz Akyüz. From noon to sunset: Interactive rendering, relighting, and recolouring of landscape photographs by modifying solar position. In Computer Graphics Forum, pages 500-515. Wiley Online Library, 2021.
[70] Benjamin Ummenhofer, Sanskar Agrawal, Rene Sepulveda, Yixing Lao, Kai Zhang, Tianhang Cheng, Stephan Richter, Shenlong Wang, and German Ros. Objects with lighting: A real-world dataset for evaluating reconstruction and rendering for object relighting, 2024.
[71] Yang Wang. Lei Zhang, Zicheng Liu, Gang Hua, Zhen Wen, Zhengyou Zhang, and Dimitris Samaras. Face relighting from a single image under arbitrary unknown lighting conditions.
Trans. Pattern Analysis and Machine Intelligence, 31 196.-198844444 2008
[72] Andreas Wenger, Andrew Gardner, Chris Tchou, Jonas Unger, Tim Hawkins, and Paul Debevec. Performance relighting and reflectance transformation with time-multiplexdd illumination. ACM
Trunsuctions on Graphics (TOG), 24(3):756-764 2005.
[73] Jung-Hse Mang-Hsuan Wu and Suguru Saito. Interactive relighting in single low-dynamic range images.
ACM Trans. Graph. 36(2), 2017.
[74] Rundi Wu, Ben Mildenhall, Philipp Henzler, Keunhong Park, Ruiqi Gao, Daniel Watson, Pratul P. Srinivasan, Dor Verbin, Jonathan T. Barron, Ben Poole, and Aleksander Holynski. Reconfusion: 3d reconstruction with diffusion priors. arXiv, 2023.
[75] Rui Xia, Yue Dong, Pieter Peers, and Xin Tong. Recovering shape and spatially-varying surface reflectance under unknown illumination. ACM Transactions on Graphics (TOG), 35
2016.
[76] Zexiang Xu, Kalyan Sunkavalli, Sunil Hadap, and Ravi Ramamoorthi. Deep image-based relighting from optimal sparse samples. ACM Transactions on Graphics (ToG), 171,) 2018.
[77] Yao Yao, Jingyang Zhang, Jingbo Liu, Yihang Qu, Tian Fang. David McKinnon, Yanghai Tsin, and Long Quan. Neilf: Neural incident light field for physically-base material estimation. In
European Conference on Computer Vision, pages 700-716. Springer, 2022.
[78] Ye Yu, Abhimitra Meka, Mohamed Elgharib, Hans-Peter Seidel, Christian Theobalt, and William AP Smith. Self-supervised outdoor scene relighting. In ECCV, pages 84-101, 2020.
[79] Chong Zeng. Yue Dong, Pieter Peers, Youkang Kong, Hongzhi Wu, and Xin Tong. Dilightnet: Fine-grained lighting control for diffusion- based image generation, 2024.
[80] Bo Zhang, Yuxuan Duan, Jun Lan, Yan Hong. Huijia Zhu, Weiqiang Wang, and Li Niu. Control- com: Controllable image composition using diffusion model. arXiv preprint arXiv:2308. 10040, 2023.
[81] Jingyang Zhang, Yao Yao, Shiwei Li, Jingbo Liu, Tian Fang, David McKinnon, Yanghai Tsin, and Long Quan. Neilf++: Inter-reflectable light fields for geometry and material estimation. In Proc. Int. Conf. on Computer Vision (ICCV), pages 3601-3610, 2023.
S [82] Kai Zhang, Fujun Luan, Zhengqi Li, and Noah Snavely. Iron: Inverse rendering by optimizing neural sdfs and materials from photometric images. In Proc. Computer Vision and Pattern Recognition (CVPR), pages 5565-5574, 2022.
[83] Kai Zhang, Fujun Luan, Qianqian Wang, Kavita Bala, and Noah Snavely. Physg: Inverse rendering with spherical gaussians for physics-based material editing and relighting. In Proc. Computer Vision and Pattern Recognition (CVPR), pages 5453- 5462, 2021.
[84] Lvmin Zhang, Anyi Rao, and Mancesh Agrawala. Adding conditional control to sot-tt--mmaee diffusion models, 2023.
[85]
Lvmin Zhang, Anyi Rao, and Mancesh Agrawala. Ic-light github page. 2024.
[86]
Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreason- able effectiveness of deep features perceptual metric. In Proc. Computer Vision and Pattern Recognition (CVPR), 2018
[87] Xiuming Zhang, Pratul P Srinivasan, Boyang Deng, Paul Debevec, William T Freeman, and Jonathan Barron. Nerfactor: Neural factorization of shape and reflectance under an unknown illumination. ACM Transactions Graphics (ToG), 40(6):1-18, 2021.
14

---

[88] Yuanqing Zhang, Jiaming Sun, Xingyi He, Huan Fu, Rongfei Jia, and Xiaowei Zhou. Modeling indirect illumination for inverse rendering. Proc. Computer Vision and Pattern Recognition (CVPR), 2022.
[89] Hao Zhou. Sunil Hadap, Kalyan Sunkavalli, and David W Jacobs. Deep single-image portrait relighting. In Proceedings of the IEEF/CVF international conference on computer vision, pages 7194-7202, 2019.
15

---

A Comparison with IC-Light
In this section, we conduct a comparison with a recent single-image relighting method, IC-Light In Fig. [0 we present a comparative analysis of relighting results, focusing on the consistency of specular highlights in comparison to IC-Light. Our method consistently adjusts the specular highlights in accordance with the rotation of the lighting, ensuring a coherent relighting effect. In contrast, LCght demonstrates inconsistent movement of highlights, with certain regions showing minimal changes. For additional video comparisons, please visit our supplemental webpage.
Target Lightnn
IC-Light
Ours
Rotating Lighting
Figure 10: Comparison of single-image relighting with IC-Light [85). In this experiment, we compare relighting results and evaluate the consistency of specular highlights against IC-Light (85). Each column represents uniformly rotated version the same environment map image, progressing from left to right. Our method consistently adjusts highlights, faithfully reproducing the object under novel lighting conditions. In contrast, LC-Light (85) exhibits inconsistent highlight movement, with certain regions changing just slightly, Further comparisons available videos provided on the supplemental webpage.
B Portrait Relighting Results
Our model is fine-tuned object data, therefore, while it is category-agnostic, it might not achieve the highest quality results on specific domains-foo instance, for the case of portrait relighting, methods that specifically trained for portrait photos may produce higher quality results. We show our results on the portrait in Fig. of the appendix.
Input Image
Relft
Image
Image
Figure 11: Human portrait relighting results. Overall, in most cases, our method shows good generalization to human portraits (see the second row of the figure above). However, there instances where it produces suboptimal results. For example, the first row, the model fails remove the shadow the neck in the input image.
failure case shown in the last row, the model does not preserve critical portrait details, such as the shape of the mouth. This issue may stem from the low-resolution limitations discussed in Sec. Further fine-tuning with additional portrait data and at a higher resolution could enhance the model's performance on portrait images.
16

---

C Inherent color ambiguity
Input Image
Pred Image
GT relit Image
Input Image
Pred Image
GT relit Image
Input Image 2
Pred Image 2 Target Lighting
Input Image 2 Pred Image 2 Target Lighting
Figure 12: Color ambiguity analysis. We found that our data-driven model can handle the inherent color ambiguity to some extent, especially when the objects' materials in the input image are relatively specular. Because the diffusion model can infer the lighting of the input image through the reflections or specular highlights
Color ambiguity is an inherent issue the single-imas relighting task. That said, we found that our data-driven model can handle this problem some extent, especially when the objects' materials in the input image are relatively specular. Because the diffusion model can infer the lighting of the input image through the reflections specular highlights. In Fig. 12 we show two such examples: render each object with two different environment maps and relight them with the same environment map. In both cases, the input images have different colors and want to relight them under the same target lighting. It turned out that both input images can be relit to the results that match the ground truth results, which indicates model has some ability to handle the color ambiguity.
As for the general diffuse object, we think our model could learn an implicit prior over common colors that show up in object albedos, as well as common colors that occur in lighting, and use that to help resolve this ambiguity. However, this may fail sometimes, as we discussed in See One potential solution could be also inputting the background of the input image to the diffusion model since the diffusion model can learn to infer the lighting of the input image from the background. We leave this future work.
17

---

D
More results of relighting 3D objects
Input Images
NVDIFFREC-MC TensoIR
Ours
GT Relighting
Figure 13: More results of relighting 3D objects. We present additional 3D relighting comparison results, which include all of our testing objects. The results show that methods achieve better qualitative relighting outcomes for all the lested objects
18

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The main focus of this paper is to train lighting- aware image-conditioned diffusion model; it aligns with our title.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: Please the Limitation Section.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
19

---

Justification: [NA]
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification:
We have included enough details to reproduce our results.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
20

---

Answer: [Yes]
Justification:
We will release all code and models once upon acceptance.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.cm cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https guwdes/ddeeeeee Cooeeeoodeesssseeeeee //nips.com / cassiicnncccyyyc for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification:
we have described in details our experimental setups in the experiments section.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: As each experiment takes significant amount of compute resource, don't have enough computing resources to complete the significance test.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
21

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Yes, provide both data creation and model training.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics http://arrrp.cccooooalc/ccccccaa
Answer: [Yes]
Justification:
The paper is written own and follows the honest codes.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [No]
Justification: Please refer to the main paper.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
22

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We have cited all previous papers that lead to current paper's presentation.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
23

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: No new assets were introduced.
Guidelines:
The answer NA means that the paper does not release new assets,
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
24


Reviews:
[{"review_id": "c4su0GzybC", "reviewer": "Reviewer_NcDc", "summary": "The paper proposes a novel method for single-image relighting, which takes an image of an object and a target environmental map as inputs. The authors fine-tune Stable Diffusion on a synthetic relighting dataset to output relit images, conditioning on both the input object image and the target environmental map. The authors show their method outperforms existing baselines. Additionally, the trained relighting model can be applied to downstream tasks such as relighting a neural radiance field and object insertion.", "questions": "The problem of inverse rendering with a single image is inherently ambiguous. For example, the object color in the input image could come from either the material or the lighting. I was wondering about the authors' thoughts on this problem in the context of Neural Gaffer. When relighting an object, there could be multiple possible outputs depending on the decomposition of the material and lighting in the input. Is the probabilistic model of Neural Gaffer able to model this perplexity?", "limitations": "Yes", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- I check the video results in the supplementary video. The visual results are impressive.\n- The authors have shown several downstream applications using their trained relighting model, including text-based relighting and object insertion.\n- The authors have conducted extensive ablation studies to prove the effectiveness of their proposed method.", "weaknesses": "I don’t have many complaints about the paper. I list several potential improvements below:\n\n- In the 3D relighting experiments, it seems unfair to compare with inverse rendering methods such as Nvdiffrec-mc and TensoIR, as they can apply any lighting to the object once the material is recovered, while Neural Gaffer needs optimize for every lighting. On the other hand, I think Neural Gaffer should be combined with these inverse rendering methods and provide priors when recovering material and lighting.\n- The extrinsic information is injected by rotating the environmental map. However, it seems intrinsic information is not considered, which means there is an assumed fixed FOV. This could introduce biases in downstream applications and limit the input views in 3D relighting.\n- The quantitive comparison with IC-Light is missing.\n- The generated image resolution is limited to 256x256.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful comments and valuable suggestions. We will revise our paper based on your feedback. Here are our responses to your comments:\n\n**1. Inherent color ambiguity**\n\n​\tColor ambiguity is an inherent issue in the single-image relighting task.\n\n​\tThat said, we found that our data-driven model can handle this problem to some extent, especially when the objects’ materials in the input image are relatively specular. Because the diffusion model can infer the lighting of the input image through the reflections or specular highlights. In Fig.1 of the rebuttal file, we show two such examples: we render each object with two different environment maps and relight them with the same environment map. In both cases, the input images have different colors and we want to relight them under the same target lighting. It turned out that both input images can be relit to the results that match the ground truth results, which indicates our model has some ability to handle the color ambiguity.\n\n​\t As for the general diffuse object, we think our model could learn an implicit prior over common colors that show up in object albedos, as well as common colors that occur in lighting, and use that to help resolve this ambiguity. However, this may fail sometimes, as shown in the first example of Fig.7 in the rebuttal file. One potential solution could be also inputting the background of the input image to the diffusion model since the diffusion model can learn to infer the lighting of the input image from the background. We leave this as future work.\n\n**2. Neural Gaffer versus other inverse rendering work and using our model for materials recovering**\n\n* Although our method must re-optimize the 3D representation for each new lighting, we would like to emphasize that the re-optimization process is relatively fast: in our implementation, the two-stage 3D relighting process takes just a few minutes on a single GPU.\n* The reviewer suggests that we combine Neural Gaffer with inverse rendering methods. That is a good idea, but our paper intentionally sets out to avoid explicit inverse rendering and decomposition, and explore whether we can successfully build systems using an implicit relighting capability. We first achieve this goal in single-image relighting task, and then we follow the same motivation and build our 3D relighting pipeline. It turns out that such a new pipeline works very well and in theory it has better data-driven and representation potential. Since such a new pipeline is essentially different from previous BRDF reconstruction-based methods and has no similar previous baselines to compare with, we can only compare this pipeline with previous reconstruction-based methods that also work on 3D relighting, such as TensoIR and Nvidiffrec-MC.\n* Thank you for your suggestion about using our model as prior for recovering materials. As we noted in our paper, “*our model can also operate as a strong relighting prior for 3D tasks*” (L 18). Using our model to improve materials recovering is for sure another interesting application. Due to the limited time we have, in Fig. 2 of the rebuttal PDF, we show a simple experiment where we use our model as data prior for recovering materials, demonstrating an improvement in albedo reconstruction quality. In particular, given input images with unknown lighting, we relight each of the input views under three different target environment maps, then use TensoIR to reconstruct BRDFs from the multi-illumination relit images (we use TensoIR because it supports multi-illumination input images). Results show that both NVDIFFREC-MC and the original TensoIR bake shadows present in the original images into the albedo map; in contrast, combining TensoIR with our diffusion prior yields albedo maps with reduced baking of shadows. Note that the current experiment is very simple and has the potential to improve for best quality. We leave this as one of the future work.\n\n**3. Camera Intrinsics are not considered** \n\n* Although the training data was rendered with fixed camera intrinsics, the model shows good generalization ability to cameras with different intrinsics when we test our 2D relighting model using real images from the Internet, which have unknown FOV (as shown in Fig. 1 of the main paper and our supplemental webpage). Training our diffusion model using images rendered with different intrinsics may further improve its performance. We thank the reviewer for the suggestion as an important future direction for scaling up the system.\n* Note that for the 3D relighting application, we first reconstruct a radiance field using the multi-view inputs and then relight the reconstructed radiance field. Since the reconstructed radiance field allows us to do novel view synthesis, we can render novel views with any camera intrinsics, and therefore for the 3D relighting task we can always render novel views with the fixed intrinsics used during training.\n\n**4. Quantitive comparison with IC-Light** \n\n​\tSince IC-light assumes a different lighting input from our method (background image versus environment map), we feel it may not be fair for us to compare with it in the main paper because we have more input lighting information, which is why we only compare with it qualitatively on real data in the appendix and the supplementary videos, in which we have shown an obvious advantage over it. But if the reviewer still thinks quantitive comparison with IC-Light is important, we will provide it in our revised paper. \n\n**5. The image resolution is limited**\n\n​\tWe have realized that the relatively low image resolution is one of our methods’s key limitations and mentioned this point in our limitations section. The main reason we don’t try higher resolutions is that we only have limited CPUs to preprocess the data and limited GPUs to train the model.\n\n​\tIn theory, our methods can support higher resolutions. We will release all code to encourage future work to improve this point.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "kos6Enh35V", "reviewer": "Reviewer_rkJr", "summary": "Neural Gaffer presents an approach to object-centric image relighting using diffusion models. The method adapts a pre-trained diffusion model and fine-tunes it on a synthetic dataset designed for relighting tasks. The main feature is its ability to condition the diffusion process on target environment maps, allowing for control over lighting effects.", "questions": "1) Why weren't datasets like OpenIllumination, Objects with Lighting, or Stanford ORB used for evaluation?\n\n2) Have you explored the necessity of full environment maps for relighting?\n\n3) How well does your method generalize to full scenes, beyond individual objects?\n\n4) Given recent findings on the inconsistency of PSNR and SSIM with human perception for lighting tasks, have you considered user study?\n\n5) Why were comparisons with recent lighting-aware object compositing methods (e.g., ControlCom, Intrinsic Harmonization) not included?\n\n6) Have you considered comparing your method with existing 3D object compositing and lighting estimation approaches?\n\n7) Why wasn't Mitsuba + NeuS used as a baseline for 3D relighting, given its reported strength in recent literature?\n\n8) Can you provide a more detailed analysis of failure cases, including examples from Objaverse instances?\n\n9) Can you provide a simple color histogram matching baseline? \n\n10) Comparison with DiLightNet: DiLightNet offers full image generation with background handling, while Neural Gaffer focuses on object relighting. This raises several points: \n\n- Background consistency: How does Neural Gaffer address the background when relighting objects?\n- Evaluation scope: Are quantitative evaluations done on the full scene or just the object region? This impacts the interpretation of results.\n- Lighting control: DiLightNet allows full-scene control. How comprehensive is Neural Gaffer's approach in comparison?\n- User input method: DiLightNet uses radiance hints, Neural Gaffer uses environment maps. How do these compare in terms of user-friendliness and control/precision?\n- Shadow and Indirect lighting effects quality: DiLightNet's shadows and indirect effects appear more convincing from their project page. Can you comment on this difference? Can you provide a user study comparing the perceived lighting quality between Neural Gaffer and DiLightNet? \n\n11) How sensitive is your method to the resolution of input environment maps?", "limitations": "Somewhat but not fully. See my weakness 8.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1) Simple yet effective approach: The paper presents a straightforward fine-tuning method for object relighting, similar to zero-1-2-3 shot learning. This simplicity is a strength, demonstrating that complex relighting can be achieved without overly complicated techniques.\n\n2) Powerful data-driven learning: The supervised conditional diffusion model effectively learns to relight objects, highlighting the potential of data-driven approaches in capturing intricate lighting interactions.\n\n3) Competitive results: Based on the presented figures, the method appears to outperform the recent DiLightNet in some aspects. However, this comparison raises some evaluation questions (see questions section for details).", "weaknesses": "1) Real-world evaluation: The model is fine-tuned on a synthetic relighting dataset, which might not fully capture the complexity of real-world lighting scenarios. Real-world evaluation is necessary, and there are datasets capturing these effects. The paper is currently missing this evaluation, and there are datasets available for such evaluation [1] OpenIllumination [2] Objects with Lighting or [3] Stanford ORB dataset. These papers have been cited but it is surprising to not see an evaluation of these datasets.\n\n2) Reliance of Environment map: Do you need to supply the environment map for relighting? There is a missing baseline that shows what happens if you condition the target lighting image without a full environment map (only image crops). The Diffusion Light Probe (CVPR 2024) paper indicates that diffusion models are capable of inpainting reliable environment maps and they seem to be implicitly encoded within the model. This baseline will justify why a full environment map is required or necessary for this task.\n\n3) Generalization to scenes: The extent to which the method generalizes to scenes -- not just objects -- is unclear. Evaluating the MIT-multi illumination dataset could shed light on this. The current reliance on explicit environment maps makes it harder to perform on these scenes, but it would be interesting to see if, without explicit environment maps (like suggested above), can you learn to relight and compare on scenes.\n\n4) Evaluation metrics: Recent studies show that PSNR, SSIM, etc. are not consistent with human evaluation. See \"Towards a Perceptual Evaluation Framework for Lighting Estimation\" (CVPR 2024). These metrics don't tell us much about whether the method is promising as such. A thorough evaluation via user studies or the metrics as defined in the recent paper is currently missing from the paper.\n\n5) Unrealistic results and missing comparisons: The object insertion results look unrealistic, with incorrect shadows that don't match the lighting conditions. Several relevant lighting-aware compositing methods are missing from the comparisons, such as ControlCom [Zhang et al., arXiv 2023], Intrinsic Harmonization [Carega et al., SIGGRAPH 2023], Reshading [Bhattad and Forsyth, 3DV 2022], and ARShadowGAN [CVPR 2020]. The comparison to AnyDoor doesn't make sense as it's not lighting-aware. Including these comparisons would provide a better evaluation of the method's performance against current state-of-the-art techniques.\n\n6) Further, as the papers use off-the-shelf methods to estimate environmental maps (text2light), why not compare with existing 3D object compositing with lighting estimation methods to get a sense of how the proposed methods compare to these tasks -- see Garon et al (CVPR 2019), StyleLight (Wang et al; ECCV 2022) and similar papers? Rendering objaverse objects using lighting estimated from the mentioned or similar methods would help understand the gaps between explicit environment map prediction methods.\n\n7) 3D relighting evaluation: For the 3D relighting setting, according to the Objects with Lighting 3DV 2024 paper, Mitsuba + NeuS is a stronger baseline compared to TensorIR, which is currently missing in the paper.\n\n8) Failure analysis: The paper mentions in the limitations section that the approach might not work for portrait relighting, but it would be interesting to see the kind of failures the diffusion model makes. The current setup lacks experiments in this direction to see what are these failures to encourage future research. Further, the current papers also do not provide any failure examples from Objaverse instances. Is the method perfect on all unseen objects -- detailed analysis is missing as to what objects the proposed methods perform best or worse on. Such analysis helps scope out limitations of the current methods instead of shallow limitations provided in Appendix D.\n\n9) Lack of comparison with simple color matching baselines: The paper doesn't include a comparison with straightforward color adjustment techniques, such as RGB histogram matching between the inserted object and the target scene. This omission raises questions about how much of the method's perceived success in relighting is due to sophisticated light interaction modeling versus simple color transformations. A comparison with such a baseline would help quantify the added value of the diffusion model approach over a simpler method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your detailed comments and insight suggestions. We will revise our paper based on your feedback. Here are our responses to your comments:\n\n**1. Evaluating our relighting model on real-world dataset** (in response to weaknesses 1 and question 1)\n\nWe evaluate our diffusion model on in-the-wild real data as shown in Fig. 1 of the main paper and the video results in the supplemental material. We note that other reviewers stated that these results are impressive (reviewer **NcDc,** **YxGp)** and convincing (reviewer **C74A**). We have provide more real-world example in our rebuttal file.\n\nWe didn’t evaluate our diffusion model on “Objects with Lighting” and “Standford-ORB” because they only provide relit target images under different camera poses from the input image, which can’t be used for evaluating the single-image relighting task.\n\nWe didn’t evaluate our diffusion model on “OpenIllumination” because it is a light-stage relighting dataset with different lighting distribution as our methods’s assumption. (Our relighting diffusion model is designed for and trained with general environment maps.) But we are glad to provide our results in our revised paper if the reviewer still wants to see them.\n\n**2. Necessity of full environment maps for relighting** (in response to weaknesses 2 and question 2)\n\n1. As stated in the abstract, the main focus of our diffusion model is *“taking a single image of any object and synthesizing an accurate, high-quality relit image under any novel* ***environmental lighting condition***. Our task is accurate single-image relighting with a user-defined target environment map, which is a common setting for the single-image relighting task. Relighting the image input without a full environment map is a different task and not the main focus of our work. Therefore, we consider comparison with such a baseline outside the scope of our work.\n2. That said, our method can be easily combined with other methods to enable relighting without a full environment map. In fact, we have used the paper the reviewer mentions (“The Diffusion Light Probe” (CVPR 2024)) in our object insertion application (see L 230), enabling single-image relighting conditioned on a scene image. \n\n**3. Generalization to Scenes** (in response to weaknesses 3 and question 3)\n\n1. We want to re-emphasize that the main focus of this paper is object-level single-image relighting (as indicated by the title — “Neural Gaffer: Relighting Any **Object** via Diffusion”) — hence, scene-level relighting is not the research scope of this work. Scene-level and object-level relighting are two distinct tasks with different potential methodology designs, and we leave scene-level single-image relighting to future work.\n2. Although our model was only trained with synthetic object data, it still shows a degree of generalization ability on scene-level data: As shown in Fig.5 of the rebuttal PDF, we sample some scene images from the MIT multi-illumination dataset (mentioned by the reviewer) and use our diffusion model to relight them with some target environment maps and get some reasonable results: our model can generate shadows in the scenes as shown in the second column of Fig.5 in the rebuttal PDF, and generate reasonable highlights in the scene as shown in the second example of Fig.5 in the rebuttal PDF. We can’t compare with ground truth on this scene dataset because our model was trained with environment maps as conditions, but the MIT-multi illumination dataset doesn’t provide ground truth environment maps. Thank you for your suggestions on how to test our methods without explicit full environment maps. We describe why we need a full environment map for relighting in the previous subsection of this rebuttal reply.\n\n**4.Evaluation metrics and why we don’t conduct user study** (in response to weaknesses 4 and question 4)\n\n1. The paper the you mentioned studies lighting estimation, which is a different task from relighting. Its conclusion may not be applicable to the relighting task.\n2. In addition to the PSNR and SSIM metrics you describe, we also compute LPIPS metrics, which are based on human perception and can better reflect visual quality.\n3. When ground truth results are available, comparing different methods via a user study is not a common way to evaluate accuracy in the relighting community. Most recent relighting papers only evaluate their methods by computing commonly used metrics, such as PSNR, SSIM and LPIPS. For example, all papers (over 10 recent popular works) that have been compared in the recent real dataset benchmarks mentioned by the reviewers (Stanford-ORB, Openillumination, and Objects with Lighting) all evaluate their methods by computing PSNR, SSIM, and LPIPS without a user study. These benchmarks themselves only compare different methods on their dataset by computing quantitative metrics without a user study.\n4. We appreciate your suggestion regarding the user study. However, due to the limited time for the rebuttal period, we are unable to finish user studies. If you still think a user study is critical for evaluation, please let us know. We will attempt to complete a user study before the end of the discussion period.\n\n\n----\nUnfinished. Please keep reading the comments.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "TQYERDblIs", "reviewer": "Reviewer_YxGp", "summary": "The paper introduces Neural Gaffer, an end-to-end 2D relighting diffusion model designed for single-image relighting without the need for explicit scene decomposition. Neural Gaffer can synthesize high-quality relit images of any object under novel environmental lighting conditions by conditioning on a target environment map. The model builds on a pre-trained diffusion model, fine-tuning it on a synthetic relighting dataset. The advantages in generalization and accuracy through evaluations on both synthetic and in-the-wild Internet imagery are shown in the paper. Neural Gaffer can be combined with other generative methods for various downstream 2D tasks like objection insertion. The video results presented in the paper are of high quality.", "questions": "The paper does not demonstrate how the method behaves when the target object is not centred and has complex backgrounds or varied lighting conditions. How does the method perform in such scenarios, especially with objects that have high-frequency texture details?\n\nIt appears that the method may struggle with scenes containing multiple objects. Can the authors provide further evaluation or examples to show how the method handles multiple objects in a single image?\n\nWhile the real-world object examples are good, they are not particularly impressive. Can the authors provide more compelling examples involving faces, humans, or animals under varied lighting conditions to better demonstrate the generalizability of the method? While it's understood that portrait lighting might not be comparable to those methods specifically trained on portraits, it would be good to see the generalizability of the method.", "limitations": "The authors have discussed the limitations.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1)  Neural Gaffer performs single-image relighting without the need for explicit scene decomposition into intrinsic components like normals and BRDFs. This provides an avenue for relighting without collecting expensive relighting real-world datasets.\n\n2)  The model can generate relit images of various objects under different environmental lighting conditions based on a target environment map. The method takes a single image as an input.\n\n3) The method can be applied to real-world objects with high-quality relighting results and perform various downstream tasks such as object insertion.", "weaknesses": "1) In case of the real-world object scenarios, the object may not be always centred and may have complex backgrounds and lighting to start with. The paper does not demonstrate how would the method behave in such cases. How about the objects with high-frequency texture details?\n\n2) Related to 1) there might be multiple objects in a scene. From the results, it seems that the method cannot handle multiple objects from a single image.\n\n3)  The real-world object examples shown in the paper and the video are good but not impressive. It would be more compelling to show faces, humans, animals etc under the lighting conditions to show the generalizability of the method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful comments and valuable suggestions. We will revise our paper based on your feedback. Here are our responses to your comments:\n\n**1. How the method performs when the target is not centered and has a complex background or varied lighting conditions, especially with objects that have high-frequency texture details**\n\n​\tWe have implemented an automatic preprocessing script that can first detect the object to be relit using Segment Anything (SAM), segment it out from the background, and finally move the segmented foreground object to the center of the input image based on its bounding box. Indeed, many objects in our real data results are not originally centered (we show centered images in the paper for better visualization, but our preprocessing script doesn’t need centered objects). We will release this preprocessing code along with our full code release.\n\n​\tWe have tested our model on real data with complex backgrounds or varied lighting conditions (as shown in the left column of the supplementary webpage, Fig. 1 in the main paper, and Fig. 3, 4 in the rebuttal file). With the aid of our preprocessing method, we can handle real data with complex backgrounds or varied lighting conditions. In the rebuttal PDF, we show additional examples with high-frequency surface texture details (e.g., sheep and dogs in Fig. 3 of the rebuttal file), and observe high-quality relit results with texture details.\n\n**2. Handling multiple objects**\n\n​\tIn our main paper, we did assume there is only a single object present when evaluating various methods, which is a common assumption in most previous object-centric 2D relighting work. And most of our training images only contain one object because we train our model on the filtered Objaverse dataset. \n\n​\tHowever, we find that our method isn’t limited to just single objects. In Fig. 4 of the rebuttal file, we show that our methods still show good generalization ability when tested on real images containing multiple objects.\n\n**3. More real-world testing examples(animals and human portraits).**\n\n​\tAs shown in the Fig. 3 of the rebuttal PDF, our methods show good generalization on real-world animal examples, achieving good relit results with high-frequency surface texture details kept.\n\n​\tAs shown in the Fig. 6 of the rebuttal PDF, our methods show good generalization on human portraits in most cases (see the examples of the first two columns in the Fig. 6 of the rebuttal PDF). However, as we mentioned in the limitations section of the main paper since our model is trained on object data and we didn’t specially train it with portrait data, sometimes it might not achieve high-quality results when handling human portraits. We show a failure case example in the last columns in Fig. 6 of the rebuttal PDF, where we found that our model failed to keep the facial details of the portrait image, such as the shape of the mouth. We believe that further finetuning our model with more portrait data can help our model achieve better performance on human portrait data.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "CpBjI95Arn", "reviewer": "Reviewer_C74A", "summary": "This paper presents a method for relighting objects observed from a single image. While existing approaches rely on specific capture condition using flashlight illumination or portrait captures, or require to explicitly decompose the scene into geometry and reflectance, the proposed method aims to generate images of a given objects under novel illumination conditions for arbitrary environmental lighting conditions. The authors show that this is possible by relying on a generative diffusion method that is conditioned on the environmental map. The method relies on a pre-trained diffusion model that is fine-tuned on a synthetic relighting dataset to learn the conditioning. The approach is evaluated qualitatively and quantitatively on single-object images. Relying on a conditional diffusion model for relighting, the authors also show additional conditioning on text for relighting.", "questions": "See questions regarding trends in the quantitative evaluations and the qualitative results that do not seem to match.", "limitations": "All major limitations are addressed. The only open limitation not addressed in the manuscript is the runtime. The authors should address and comment on the runtime for their diffusion model.", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "This work presents a simple (this is a good thing) and effective method for relighting from a single image. The method relies on synthetic supervision with a novel Blender-rendered dataset that uses Objeverse as input model source. The authors went a long way by collecting diverse HDR environment maps from the Internet that were augmented to produce a large synthetic relighting dataset of almost 20M rendered images with ground truth lighting maps. Overall, the method offers a number of intriguing benefits listed as follows:\n\n* Conditional image-to-image diffusion model: The method inherits a conditional Zero-1-to-3 model that is extended in its input latents to a rotated environment map with the camera coordinate frame, allowing for image-to-image relighting in a consistent frame. While, given enough training data, the method is effective in relighting, the approach also enjoys the benefits of existing diffusion architectures with various types of conditioning. The authors demonstrate this effectively with their image conditioning. \n\n* Relighting 3D radiance fields: The proposed method is evaluated as a prior for 3D relighting of a neural radiance field. Specifically, the authors propose to use diffusion-based relighting as a coarse reconstruction loss (predicting a coarse relit scene during the NeRF optimization) and a detail refinement loss where the NeRF appearance is further refined.\n\n* Qualitative evaluation: The evaluations presented qualitatively in the main manuscript and the supplemental material in the form of supplemental videos are visually plausible and convincing. \n\n* Quantitative evaluations: The method is adequately ablated and quantitatively compared to single image relighting methods, 3D radiance field relighting with reasonable margins on the test sets. This validates the method as an effective approach.", "weaknesses": "What makes the method exciting, at first glance, is also one of the major weaknesses: the technical novelty. The paper piggy-backs on an existing generative method, the Zero-1-to-3 model, that is with a few variations used for relighting. While the simplicity is something that is desired, it also makes it challenging for the reader to derive deeper insights from this work. We learn that pre-trained diffusion-models, when just given enough and the right synthetic data, can allow for plausible novel view synthesis with artefacts that are improved over existing methods. However, the recent work by \n\nChong Zeng, Yue Dong, Pieter Peers, Youkang Kong, Hongzhi Wu, and Xin Tong. Dilightnet: Fine-grained lighting control for diffusion-based image generation, 2024.\n\nin a way also does show the exactly same, although the technical approach is different. Overall, the technical contribution of the approach is rather incremental (although the method is effective). As such, I am torn on this work. While the technical contribution is not near other work at NeurIPS, the method is effective and likely of high impact. \n\nA further qualm I have is regarding the results compared to NVDIFFREC. While the margins are not substantially different, the results in Fig. 6 seem to indicate differently. It seems as if these results are cherry-picked.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful comments and valuable suggestions. We will revise our paper based on your feedback. Here are our responses to your comments:\n\n**1. Technical contribution is incremental considering DiLightNet**\n\n* Although our method and DiLightNet both approach single-image-relighting using diffusion models, our method is fundamentally different from it and provides novel insights to the community. Dilightnet first estimates a coarse mesh from the single image input using an off-the-shelf depth estimator, then uses Blender to render the estimated mesh with several predefined BRDF materials and target lighting conditions, and finally uses the rendered image maps as the relighting conditions to its diffusion model, which means it still involves some explicit geometry, BRDF and lighting modeling in its relighting process, which can be inaccurate or under-expressive. **In contrast, our work is purely end-to-end and shows that relighting for general objects can be totally data-driven without any explicit physically-based decomposition or modeling. To our knowledge, this is a new insight that does not appear in prior work, including Dilightnet**. And as pointed out by Reviewer YxGp, our work “*provides an avenue for relighting without collecting expensive relighting real-world datasets*”.\n* While our method is simple, the technical details, like how HDR environment maps are represented and encoded to be used as the conditions for diffusion models, are non-obvious and important to get right. We have also conducted sufficient ablation studies to validate the effectiveness of our designs.\n* In contrast to DiLightNet, which only explores 2D relighting, we also demonstrate how to apply our relighting diffusion model as data prior to downstream tasks such as 3D relighting. We emphasize that our work is also the first to show that 3D relighting can be achieved without an explicit inverse rendering framework that solves BRDFs. Our 3D relighting pipeline is fundamentally different from any previous inverse rendering-based 3D relighting work and achieves better visual quality. It has the potential to be a new paradigm of 3D relighting.\n* We believe that our simple but effective design is an advantage, making our model robust, more user-friendly (because our model is end-to-end), and easier to scale up (because the training of our model doesn’t rely on any explicit BRDF or geometry information). We are glad to see that all reviewers agree that our methods are effective. In particular, Reviewer rkJr also agrees that our “simplicity is a strength”.\n\n**2. The 3D relighting results may be cherry-picked**\n\n* To demonstrate that the results were not cherry-picked, we present additional 3D relighting comparison results in Figure 8 of the rebuttal PDF, which includes all of our testing objects. The results show that our methods achieve better qualitative relighting outcomes for all the tested objects. We will provide more results in our final revised paper.\n* Regarding the questions about the \"trends in the quantitative evaluations and the qualitative results that do not seem to match,\" we believe this difference arises because per-pixel-based PSNR does not accurately reflect the visual quality of our diffusion-based relighting method. As shown in Table 3 of our main paper, although our PSNR is only slightly higher than that of NVDIFFREC-MC (29.01 versus 28.25), our LPIPS loss is significantly lower (0.040 versus 0.082). Recent research suggests that human perception-based LPIPS loss is a better indicator of visual quality, which explains the superior visual quality we observe in Figure 6 of our paper.\n* To further illustrate previous point, please refer to Figure 9 in the rebuttal PDF. In that figure, our PSNR is only 0.8 dB higher than NVDIFFREC-MC in the first example and just 0.2 dB higher in the second example. Despite these small differences in PSNR, our method demonstrates much better visual quality. Our LPIPS loss is consistently much lower than the baselines in both examples, reinforcing the idea that human perception-based LPIPS loss aligns more closely with visual quality trends. \n\n**3. Runtime for our diffusion model**\n\nBased on our local test, it takes ~0.5 seconds to relight one image with one A6000 GPU.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
Dear Reviewers,

Thank you for dedicating your time to review our paper and offering insightful feedback. We sincerely appreciate your efforts to help enhance the quality of our research. We are also pleased to note that all reviewers were supportive of our work:

(a)Recognize our methods are effective and have high-quality results (NcDc, rkJr,  YxGp,  C74A)

(c) Praise our methods outperform the existing methods(NcDc, rkJr, C74A)

(b)Acknowledge our extensive ablation study to prove the effectiveness of the proposed method.(NcDc, C74A)

(c) Praise our methods enable many downstream applications(NcDc, YxGp)

(d)Recognize our methods are simple and effective, which is our strength ( rkJr, C74A)

(e) Acknowledge our methods show good generalization ability on real world data, and have impressive (NcDc, YxGp) and convincing (C74A) results

(f) Acknowledge our methos show potwerful data-driven potential ( rkJr) and provide an avenue for relighting( YxGp)



We thank all the reviewers for your insightful suggestions. Since we have shown a lot of new results as required by reviewers in the rebuttal PDF file. Please zoom in for better visualization when reading it.

Meta-review: The paper proposed a simple yet effective approach for single-image relighting with impressive results on relighting tasks by finetuning image-based diffusion models. The paper further showed applications for relighting and object insertion on real-world examples with adequate evaluations and ablation studies to verify the effectiveness of the proposed approach, demonstrating that end-to-end diffusion-based method can achieve the relighting.  After discussing with SAC, the AC agrees with the opinions of the majority of the reviewers and recommends accepting the paper.

Please add the suggested comparisons (full comparison to NVdiffrec-mc) and evaluations (multiple objects/scenes/human portraits, and better metrics) into the paper.
Decision: Accept (poster)