Neural Gaffer: Relighting Any Object via Diffusion

Authors: haian  jin
yuan  li
fujun  luan
yuanbo  xiangli
sai  bi
kai  zhang
zexiang  xu
jin  sun
noah  snavely
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-05 20:32:23

Abstract: Single-image relighting is a challenging task that involves reasoning about the complex interplay between geometry, materials, and lighting. Many prior methods either support only specific categories of images, such as portraits, or require special capture conditions, like using a flashlight. Alternatively, some methods explicitly decompose a scene into intrinsic components, such as normals and BRDFs, which can be inaccurate or under-expressive. In this work, we propose a novel end-to-end 2D relighting diffusion model, called Neural Gaffer, that takes a single image of any object and can synthesize an accurate, high-quality relit image under any novel environmental lighting condition, simply by conditioning an image generator on a target environment map, without an explicit scene decomposition. Our method builds on a pre-trained diffusion model, and fine-tunes it on a synthetic relighting dataset, revealing and harnessing the inherent understanding of lighting present in the diffusion model. We evaluate our model on both synthetic and in-the-wild Internet imagery and demonstrate its advantages in terms of generalization and accuracy. Moreover, by combining with other generative methods, our model enables many downstream 2D tasks, such as text-based relighting and object insertion. Our model can also operate as a strong relighting prior for 3D tasks, such as relighting a radiance field.

Main paper: 
 introduction: 
 Lighting plays a key role in our visual world, serving as one of the fundamental elements that shape our interpretation and interaction with 3D space. The interplay of light and shadows can highlight textures, reveal contours, and enhance the perception of shape and form. In many cases, there is a desire to relight an image-that is, to modify it as if it were captured under different lighting conditions. Such a relighting capability enables photo enhancement (e.g., relighting portraits [18; 67] ), facilitates consistent lighting across various scenes for filmmaking [59; 22] , and supports the seamless integration of virtual objects into real-world environments [38; 19] . However, relighting single images is a challenging task because it involves the complex interplay between geometry, materials, and illumination. Hence, many classical model-based inverse rendering approaches aim to explicitly recover shape, material properties, and lighting from input images [2; 75; 47; 83; 87] , so that they can then modify these components and rerender the image. These methods often require multi-view inputs and can suffer from: 1) model limitations that prevent faithful estimation of complex light, materials, and shape in real-world scenes, and 2) ambiguities in determining these factors without strong data-driven priors.To circumvent these issues, image-based relighting techniques [18; 72; 76] avoid explicit scene reconstruction and focus on high-quality image synthesis. However, some image-based methodsIn a bustling city square Atop a windswept cliff overlooking the ocean Figure 1 : Single-image relighting results on real data. Neural Gaffer supports single-image relighting for various input images under diverse lighting conditions, using either image-conditioned input (i.e., an environment map) or text-conditioned input (i.e., a description of the target lighting). These results demonstrate our model's capability to adapt to diverse lighting scenarios while preserving the visual fidelity of the original objects. Our relighting results remain consistent with the lighting rotating. Please see the supplementary webpage for additional video results produced from real input images. [18; 58; 76; 44; 3] require inputs under multiple lighting conditions or even sophisticated capture setups (e.g., a light stage [18] ), while recent deep learning-based techniques [67; 51; 89] enable single-view relighting but often work only on specific categories like portraits.In this paper, we propose a novel category-agnostic approach for single-view relighting that tackles the challenges mentioned above. We introduce an end-to-end 2D relighting diffusion model, named Neural Gaffer, that takes a single image of any object as input and synthesizes an accurate and high-quality relit image under any environmental lighting condition, specified as a high-dynamic range (HDR) environment map. In contrast to previous learning-based single-view models that are trained for specific categories, we leverage powerful diffusion models, trained on diverse data, and enable relighting of objects from arbitrary categories. Unlike prior model-based approaches, our diffusion-based data-driven framework enables the model to learn physical priors from an objectcentric synthetic dataset featuring physically-based materials and High Dynamic Range (HDR) environment maps. This facilitates more accurate lighting effects compared to traditional methods.Our model exhibits superior generalization and accuracy on both synthetic and real-world images (See Fig. 1 ). It also seamlessly integrates with other generative methods for various 2D image editing tasks, such as object insertion. Neural Gaffer also can serve as a robust relighting prior for neural radiance fields, facilitating a novel two-stage 3D relighting pipeline. Hence, our diffusion-based solution can relight any object under any environmental lighting condition for both 2D and 3D tasks. By harnessing the power of diffusion models, our approach generalizes effectively across diverse scenes and lighting conditions, overcoming the limitations of prior model-based and image-based methods. 
related work: 
 Diffusion models. Recently, diffusion-based image generation models [26; 49] have dominated visual content generation tasks. Stable Diffusion [60] not only generates impressive results in the text-to-image setting, but also enables a wide range of image applications. ControlNet [84] adds trainable neural network modules to the U-Net structure of diffusion models to generate images conditioned on edges, depth, human pose, etc. Diffusion models can also change other aspects of an input image [45; 9] , such as image style, via image-to-image models [52] . ControlCom [80] and AnyDoor [14] are image composition models that can blend a foreground object into a target image. Huang et al. provide a comprehensive review of diffusion-based image editing [28] .Recent work has also shown that, despite being trained on 2D images, other kinds of information can be unlocked from diffusion models. A notable example is Zero-1-to-3 [42] , which uncovers 3D reasoning capabilities in Stable Diffusion by fine-tuning it on a view synthesis task using synthetic data, and conditioning the model on viewpoint. Similar to the spirit of Zero-1-to-3 and its follow-up work (such as [63; 27] ), our work unlocks the latent capability of diffusion models for relighting.Single-image relighting. Relighting imagery is a difficult task that requires an (explicit or implicit) understanding of geometry, materials, lighting, and their interaction through light transport. Singleimage relighting is especially challenging due to its ill-posedness; there are infinite explanations of a single image in terms of underlying factors. Classic single-image methods, like SIRFS [2] , focus on explicit estimation of scene properties, via optimization with hand-crafted priors. More recent learning-based methods can learn such priors from data, but either focus on specific image types, such as outdoor scenes [73; 69; 78; 40; 21] , portraits [53; 71; 64; 67; 48; 51; 23; 56] , or human bodies [31; 37; 29] , or require special capture conditions like camera flash [39; 61] . Single-image relighting methods for general objects, with realistic appearance, materials, and lighting, remain elusive. In addition, most recent single-image relighting methods [51; 39; 61; 12] still explicitly decompose an image into components, then reconstruct it with modified lighting. This strategy allows the components to be explicitly controlled, but also limits their realism based on the representational ability of the chosen components, such as the Disney BSDF [11] or Phong reflection model [10] . In contrast, similar to [67; 79] , we achieve object relighting without an explicit scene decomposition.By training an image-based and lighting-conditioned diffusion model, we can relight any given object from a single photo.3D relighting and inverse rendering. Inverse rendering methods involve estimating 3D models that can be relit under different lighting conditions, typically from multiple images. Recent advances in 3D, such as NeRF [46] and 3D Gaussian Splatting [32] , have shown remarkable novel view synthesis results. However, these representations are not themselves relightable as the physical light transport is baked into the radiance field. Efforts have been made toward creating relightable 3D representations, such as PhySG [83] , NeRFactor [87] , TensoIR [30] and others [6; 8; 7; 5; 4; 82; 88; 35; 66; 77; 81] . These methods often rely on pure optimization to minimize a rendering loss, sometimes incorporating hand-crafted material and/or lighting regularization terms (e.g., smoothness or sparsity) to prevent overfitting. However, these approaches often struggle with specular highlights and glossy materials. We show that our proposed Neural Gaffer method can be applied in an inverse rendering framework, serving as a powerful prior and regularizer. This is because our model has been trained on a large dataset encompassing diverse lighting and material conditions, effectively learning these priors from high-quality data. Our model improves 3D relighting quality, while baselines tend to produce artifacts due to the baking of lighting information into the color field. 
method: 
 Neural Gaffer is a generalizable image-based 2D diffusion model for relighting general objects; we finetune a pretrained image-conditioned diffusion model [42] with a synthetic relighting dataset to fulfill this goal. In Sec. 3.1, we describe how we construct a large-scale high-quality relighting dataset from Objaverse [20] . Then, in Sec. 3.2, we discuss the design and training of a 2D diffusion model to support zero-shot single-image relighting. Finally, we showcase applications of our model in downstream tasks, including 2D object insertion and 3D radiance field relighting tasks.Capturing ground-truth lighting for real object images is both time-consuming and challenging. Existing real-world relighting datasets [41; 36; 70] are constrained to a very limited number of objects. To address this limitation, we opt to render a large-scale synthetic object relighting dataset.We use Objaverse [20] as our data source, which comprises about 800K synthetic 3D object models of varying quality. To avoid 3D object models with poor geometry or texture, we developed a series of filtering mechanisms. Specifically, we first utilize a classifier trained on a manually annotated subset to categorize and filter out the 3D objects with low geometry and texture quality. In addition, we also design a set of filtering rules based on 3D object attributes, such as geometry bounding boxes, BRDF attributes, and file size, eventually yielding a subset (of ∼90K objects) of the original Objaverse with high quality. To ensure abundant lighting variations, we collected 1,870 HDR environment maps from the Internet, and augmented them with horizontal flipping and rotation, resulting in ∼30K different environment maps in total.We render images with the Cycles renderer from Blender [17] . During rendering, we randomly sample 12 camera poses for each object. For each viewpoint, we render 16 images under different environment maps. In contrast to existing datasets that only render with environment maps [16; 62] , we additionally render an image under a randomly set area light source to increase lighting diversity. In total, for each object, there are 12 × (16 + 1) = 204 images rendered under different lighting conditions and from different viewpoints. Our final synthetic relighting dataset has about 18.4M 512 × 512 rendered images with ground-truth lighting. We will release our full relighting dataset, named the RelitObjaverse dataset, upon acceptance to facilitate reproducibility.Given a single object image x ∈ R H×W ×3 and a new lighting condition E ∈ R H ′ ×W ′ ×3 (represented as an HDR environment map), our goal is to learn a model f that can synthesize a new image xE with correct appearance changes under the new lighting, while preserving the object's identity:xE = f (x, E).(1) We fine-tune an image-to-image diffusion model using our rendered data. In particular, we adopt the pre-trained Zero-1-to-3 model [42] and discard its components and weights relevant to novel-view generation, repurposing the model for relighting. To incorporate lighting conditions, we extend the input latents of the diffusion model, concatenating the encoded input image and the environment map with the denoising latent. We introduce two design decisions that are critical for the effective integration of lighting into the latent diffusion model. Environment map rotation. In general, each pixel of an environment map E corresponds to a 3D direction in the absolute world coordinate frame, whereas our relighting is performed in image space, that is, locked to the camera's coordinate frame. Therefore, we need a way for a user to specify the relationship between the environment map and camera coordinate frames. One option is to concatenate the relative 3D rotation to the embedding, similar to how Zero-1-to-3 [42] encodes the relative camera pose for the novel view synthesis task. However, we observe that such a design makes it harder for the network to learn the relighting task, perhaps because it needs to interpret the complex relationship between 3D rotation parameters and the desired rotated lighting. Our solution is instead to rotate the target environment map to align with the target camera coordinate frame, prior to feeding it to the model. After rotation, each pixel of the rotated environment map corresponds to a fixed lighting direction in the target camera coordinate frame. HDR-LDR conditioning. Pixel values in the HDR environment map have a theoretical range of [0, +∞), and large pixel values can lead to unstable diffusion training. To rescale it to [0, 1.0], a naive idea is to divide it by its maximum value, but this leads to a loss of total energy information. Also, if the map has a very bright region (e.g., the sun), then most of the other values will approach 0 and the environment map will be too dark after division, causing a loss of lighting detail. To solve this, we split the HDR environment map E into two maps: E L is an LDR representation of E, computed via standard tone-mapping [57; 24] , and E H is a nonlinearly normalized representation of E, computed by a logarithmic mapping and divided by the new maximum value. E L encodes lighting detail in low-intensity regions, which is useful for generating relighting details(such as reflection details). E H maintains the lighting information across the initial spectrum. EQUATIONWe can utilize Neural Gaffer as a prior for 3D relighting. In particular, given a 3D neural radiance field consisting of a density field G σ and a view-dependent appearance field G a [46] , we consider the task of relighting it using our 2D relighting diffusion model as a prior. We find that simply relighting the input NeRF with the popular Score Distillation Sampling (SDS) method [55] leads to over-saturated and unnatural results, as shown in Fig. 8 . We instead propose a two-stage pipeline (Fig. 3 ) to achieve high-quality 3D relighting in minutes. In the first stage, we use our 2D relighting diffusion model to relight input views and overfit the original NeRF's appearance field, giving us coarsely relit renderings at sampled viewpoints. In the second stage, our model is used to further refine the appearance details with a diffusion loss and an exponential annealed multi-step scheduling.Stage 1: Coarse relighting with reconstruction loss. Given a NeRF with parameters ψ, the multiview input images x input = {x i } N i=1 and the target lighting E target , we first use our diffusion model f to generate multi-view relighting prediction images xrelit = {x i } N i=1 , where xi = f (x i , E target ). Then, we freeze the density field G σ of the NeRF, and only optimize the appearance field of the input NeRF with reconstruction loss L recon. , aiming to minimize the reconstruction error between a rendered image x = x(ψ, π input ) and the relit image x relit at pose π input :EQUATIONAfter this stage, we get an initial blurry and noisy relighting result.Stage 2: Detail refinement with diffusion guidance loss. The results from the first stage can be blurry and can exhibit noise because the multi-view relighting predictions results may be inconsistent.We further refine the relit NeRF's appearance in Stage 2. We found that directly using the SDS loss [55] for refinement leads to noisy and over-saturated results (see Fig. 8 ). Inspired by [68; 74; 45] , we treat the blurry rendered images from Stage 1 as an intermediate denoising result. We first render an image x under the sampled camera pose π sampled . We encode and perturb it to a noisy latent ẑt (x, t ⋆ ) with an intermediate noise level t ⋆ , and initiate the multi-step DDIM [65] denoising process for ẑt (x, t ⋆ ) from t ⋆ , yielding a latent sample ẑ0 . This latent is decoded to an image xrelit (t ⋆ ) = D( ẑ0 ), which we use to supervise the rendering with the following diffusion guidance loss:EQUATIONwhere L p is the perceptual distance LPIPS [86] and w 1 and w 2 are scale factors to balance the two terms. We exponentially decrease t ⋆ during optimization.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof? Answer: [NA]• It should be clear whether the error bar is the standard deviation or the standard error of the mean. Justification: The paper is written on our own and follows the honest codes.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
implementation details: 
 Diffusion model training. We fine-tune our model starting from Zero123's [42] checkpoint and discard its original linear projection layer for image embedding and pose. We only fine-tune the UNet of the diffusion model and freeze other parts. During fine-tuning, we use a reduced image size of 256 × 256 and a total batch size of 1024. Both the LDR and normalized HDR environment maps are resized to 256 × 256. We use AdamW [43] and set the learning rate to 10 -4 for training. We fine-tune our model for 80K iterations on 8 A6000 GPUs for 5 days.3D relighting. Given a neural radiance field reconstructed from multi-view inputs (we use Ten-soRF [13] ), we first freeze its density field and then optimize the input NeRF's appearance field for 2,500 iterations in Stage 1, with a ray batch size of 4096. In Stage 2, we set the initial timestep t ⋆ to decrease from 0.4 to 0.05 exponentially. w 1 and w 2 in Eqn. 4 are set to be 0.5 for all of our test cases. We optimize the results from Stage 1 for 500 iterations, and for each iteration, we sample one view to compute the diffusion guidance loss in Eqn. 4. Stage 1 takes about 5 minutes and Stage 2 takes about 3 minutes on a single A6000 GPU.Downstream 2D tasks. Our model supports downstream 2D image editing tasks, such as textbased relighting and object insertion. For text-based relighting, given a text prompt, we first use Text2Light [15] to generate an HDR panorama, and, using that as an environment map, relight images using our diffusion model. Fig. 1 (right two columns) shows example results; more are on our project webpage. For object insertion, given an object image and a target (perspective) scene image, we segment the foreground object with SAM [34] , estimate lighting from the target background image with DiffusionLight [54] , perform relighting, and finally generate a shadow attached to the object using [50] . Our workflow can preserve the object's identity and achieve high-quality visual quality.We use common image metrics including PNSR, SSIM, and LPIPS [86] to evaluate relighting results quantitatively. In addition, since there exists an ambiguity between lighting intensity and object color, we also compute channel-aligned results as in [87; 88] , where each channel of the relighting prediction is rescaled to align the average intensity with the ground-truth. We compute metrics for both the raw and the channel-aligned results for quantitative evaluation.For the single-image relighting task, there are very few prior works that can handle general objects under natural illumination. We compare to a very recent work DiLightNet [79] , which trains a lightingconditioned ControlNet [84] to relight images, which is achieved by conditioning its ControlNet with a set of rendered images using predefined materials, estimated geometry of the input image, and target lighting. Per our request, the authors ran their model on our validation data. We select 48 high-quality objects from Objaverse as validation objects, which are unseen during training. We render each object under 4 different camera poses. For each camera, we randomly sample 12 unseen environment maps to render the target relit images, and one additional environment map to render the input. We also compare with IC-Light [85] . Since our lighting conditioning is different from its setting (they use background images), we only provide qualitative comparisons in our appendix and project webpage. For relighting 3D radiance fields, we compare with two 3D inverse rendering methods: TensoIR [30] and NVDIFFREC-MC [25] . We select 6 objects from the NeRF-Synthetic dataset [46] and Objaverse [20] for testing. For each object, 100 camera poses are sampled to generate training images and 20 camera poses are sampled for testing images. Each pose is rendered under four distinct, previously unseen environmental lighting conditions. One lighting condition is designated for training images, while the remaining three are reserved for relighting testing.Comparing single-image relighting with DiLightNet [79] . As shown in Fig. 4 , we evaluate across a diverse set of unseen objects and unseen target lighting conditions. Our approach showcases superior fidelity to the intended lighting settings, maintaining consistent color accuracy and preserving intricate details. Compared with DiLightNet [79] , our model can generate more accurate highlights, shadows, and high-frequent reflections. These results demonstrate our model's ability to accurately understand the underlying physical properties of the object in the input image, such as geometry and materials, and faithfully reproduce the desired relighting appearance. Quantitative results are in Tab. 1.Single-image relighting on real data. We evaluate our single-image relighting model on a set of realworld input photos (Fig. 1 ), under either image-conditioned (environment map) or text-conditioned (text description of the desired target lighting) target illumination. As shown in the figure, our model effectively adapts to diverse lighting conditions and produces realistic, high-fidelity relighting results under novel illumination. In addition, our relighting results are consistent and stable with the lighting condition rotating. Please refer to our webpage to get more video results of more real data.Ground-truth Target Lighting Object insertion. Our diffusion model exhibits versatility when combined with other generative models for downstream 2D tasks, like object insertion. In Fig. 5 , we take a foreground image containing the object of interest and insert the object into a background image depicting the desired target lighting condition. Our method surpasses baseline approaches by better preserving the identity of the inserted objects and synthesizing more natural illumination, seamlessly integrating objects into the background scene.AnyDoor ResultsFigure 5 : Object insertion. Our diffusion model can be applied to object insertion. Compared with Any-Door [14] , our method better preserves the identity of the inserted object and achieves higher-quality results.Target LightingFigure 6 : Relighting 3D objects. We take multi-view images and a target lighting condition as input and compare our method's relighting results to NVDIFFREC-MC [25] and TensoIR [30] . (The shown images are channel-aligned results.) Lacking strong priors, the baselines tend to miss specular highlights (NVDIFFREC-MC), bake the original highlights (TensoIR), or produce artifacts (NVDIFFREC-MC and TensoIR). In contrast, our method implicitly learns to model the complex interplay of light and materials from the training process of our relighting diffusion model, resulting in more accurate relighting results with fewer artifacts. Video comparisons are available on our project webpage.Relighting 3D objects. Fig. 6 shows the results of our two-stage method for relighting 3D objects. Given an input radiance field(NeRF) representing a 3D object, we first tune NeRF for coarse relighting in Stage 1, followed by a finer relighting result in Stage 2 that improves fidelity. We also compared to conventional inverse rendering methods such as NVDIFFREC-MC [25] and TensoIR [30] , reporting quantitative results in Tab. 2 and showing qualitative comparisons in Fig. 6 and Fig. Ablation studies for alternate lighting conditioning designs for our diffusion model. As ablations, we evaluated different lighting conditioning designs for our diffusion model: 1) only inputting one environment map rather than both the rotated LDR and normalized HDR map, and 2) controlling the lighting direction by adding the relative 3D rotation between the default environment map coordinate frame and the target camera frame to the U-Net CLIP embedding branch without rotating the environment maps explicitly. Due to limited computational resources, when performing ablations for the model design, we train the different models with 2 A6000 GPUs for 48K iterations. We provide a quantitative comparison between them in Tab. 13 
limitations: 
 While our model can achieve overall consistent relighting results under rotating or changing lighting conditions, our results can still exhibit minor inconsistencies since the model is generative.Given the high resource demands of data preprocessing (specifically, rotating the HDR environment map) and model training, and considering our limited university resources, we trained the model at a lower image resolution of 256 × 256. In Fig. 9 , we use VAE [33] in our base diffusion model [1; 42] to encode input images into latent maps, and then directly decode them. We found that VAE struggles to preserve identity for objects with fine details even from latent maps encoded from the input images at this resolution, which in turn results in many relighting failure cases at this resolution. Finetuning our model at a higher resolution will greatly help solve this issue.We observe two key failure cases when relighting Objaverse Instances, as shown on the right side of Fig. 9 . First, our model struggles with color ambiguity in diffuse objects, leading to inaccuracies in relit colors (first row). This issue arises from inherent ambiguities in the input image, where color could be attributed to either material or lighting. Including the background of the input image may help the model better infer the lighting conditions of input images and address this challenge. We give more discussions in the Appendix. C. Second, while the model generates low-frequency highlights well, it often misses high-frequency reflections in relit images (second row).In addition, we discuss our limitations regarding portrait relighting in the Appendix. B. 
conclusion: 
 We proposed an end-to-end 2D relighting diffusion model, Neural Gaffer, that addresses the longstanding challenge of single-image relighting through a data-driven framework. Our model synthesizes accurate and high-quality relighted images from a single input image under any environmental lighting condition, demonstrating superior generalization and accuracy on both synthetic and realworld datasets. Neural Gaffer not only enhances various 2D tasks, such as text-based relighting and object insertion but also serves as a strong relighting prior for 3D tasks, enabling a simplified two-stage 3D relighting pipeline. By overcoming the limitations and ambiguities of traditional model-based inverse rendering approaches, our diffusion-based solution extends the applicability of relighting techniques to a broad range of practical applications. 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [No] Justification: Please refer to the main paper.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA]Justification: [NA] Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes]Justification: We have cited all previous papers that lead to our current paper's presentation.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. 
 
 Appendix: In this section, we conduct a comparison with a recent single-image relighting method, IC-Light [85] . In Fig. 10 , we present a comparative analysis of relighting results, focusing on the consistency of specular highlights in comparison to IC-Light. Our method consistently adjusts the specular highlights in accordance with the rotation of the lighting, ensuring a coherent relighting effect. In contrast, IC-Light demonstrates inconsistent movement of highlights, with certain regions showing minimal changes. For additional video comparisons, please visit our supplemental webpage. Figure 10 : Comparison of single-image relighting with IC-Light [85] . In this experiment, we compare relighting results and evaluate the consistency of specular highlights against IC-Light [85] . Each column represents a uniformly rotated version of the same environment map image, progressing from left to right. Our method consistently adjusts highlights, faithfully reproducing the object under novel lighting conditions. In contrast, IC-Light [85] exhibits inconsistent highlight movement, with certain regions changing just slightly. Further comparisons are available in videos provided on the supplemental webpage.Our model is fine-tuned on object data, therefore, while it is category-agnostic, it might not achieve the highest quality results on specific domains-for instance, for the case of portrait relighting, methods that are specifically trained for portrait photos may produce higher quality results. We show our results on the portrait in Fig. 11 of the appendix. Color ambiguity is an inherent issue in the single-image relighting task. That said, we found that our data-driven model can handle this problem to some extent, especially when the objects' materials in the input image are relatively specular. Because the diffusion model can infer the lighting of the input image through the reflections or specular highlights. In Fig. 12 , we show two such examples: we render each object with two different environment maps and relight them with the same environment map. In both cases, the input images have different colors and we want to relight them under the same target lighting. It turned out that both input images can be relit to the results that match the ground truth results, which indicates our model has some ability to handle the color ambiguity.As for the general diffuse object, we think our model could learn an implicit prior over common colors that show up in object albedos, as well as common colors that occur in lighting, and use that to help resolve this ambiguity. However, this may fail sometimes, as we discussed in Sec.5. One potential solution could be also inputting the background of the input image to the diffusion model since the diffusion model can learn to infer the lighting of the input image from the background. We leave this as future work. Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.In film and television crews, the gaffer is responsible for managing lighting, including associated resources such as labor, lighting instruments, and electrical equipment 


Decision: The paper proposed a simple yet effective approach for single-image relighting with impressive results on relighting tasks by finetuning image-based diffusion models. The paper further showed applications for relighting and object insertion on real-world examples with adequate evaluations and ablation studies to verify the effectiveness of the proposed approach, demonstrating that end-to-end diffusion-based method can achieve the relighting.  After discussing with SAC, the AC agrees with the opinions of the majority of the reviewers and recommends accepting the paper.

Please add the suggested comparisons (full comparison to NVdiffrec-mc) and evaluations (multiple objects/scenes/human portraits, and better metrics) into the paper.

Reviews: review: paper_summary: paper_summary: The paper proposes a novel method for single-image relighting, which takes an image of an object and a target environmental map as inputs. The authors fine-tune Stable Diffusion on a synthetic relighting dataset to output relit images, conditioning on both the input object image and the target environmental map. The authors show their method outperforms existing baselines. Additionally, the trained relighting model can be applied to downstream tasks such as relighting a neural radiance field and object insertion.
strength_weakness: strength_weakness: I don’t have many complaints about the paper. I list several potential improvements below:

- In the 3D relighting experiments, it seems unfair to compare with inverse rendering methods such as Nvdiffrec-mc and TensoIR, as they can apply any lighting to the object once the material is recovered, while Neural Gaffer needs optimize for every lighting. On the other hand, I think Neural Gaffer should be combined with these inverse rendering methods and provide priors when recovering material and lighting.
- The extrinsic information is injected by rotating the environmental map. However, it seems intrinsic information is not considered, which means there is an assumed fixed FOV. This could introduce biases in downstream applications and limit the input views in 3D relighting.
- The quantitive comparison with IC-Light is missing.
- The generated image resolution is limited to 256x256.
questions: questions: The problem of inverse rendering with a single image is inherently ambiguous. For example, the object color in the input image could come from either the material or the lighting. I was wondering about the authors' thoughts on this problem in the context of Neural Gaffer. When relighting an object, there could be multiple possible outputs depending on the decomposition of the material and lighting in the input. Is the probabilistic model of Neural Gaffer able to model this perplexity?
limitations: limitations: Yes
paper_summary: paper_summary: Neural Gaffer presents an approach to object-centric image relighting using diffusion models. The method adapts a pre-trained diffusion model and fine-tunes it on a synthetic dataset designed for relighting tasks. The main feature is its ability to condition the diffusion process on target environment maps, allowing for control over lighting effects.
strength_weakness: strength_weakness: 1) Real-world evaluation: The model is fine-tuned on a synthetic relighting dataset, which might not fully capture the complexity of real-world lighting scenarios. Real-world evaluation is necessary, and there are datasets capturing these effects. The paper is currently missing this evaluation, and there are datasets available for such evaluation [1] OpenIllumination [2] Objects with Lighting or [3] Stanford ORB dataset. These papers have been cited but it is surprising to not see an evaluation of these datasets.

2) Reliance of Environment map: Do you need to supply the environment map for relighting? There is a missing baseline that shows what happens if you condition the target lighting image without a full environment map (only image crops). The Diffusion Light Probe (CVPR 2024) paper indicates that diffusion models are capable of inpainting reliable environment maps and they seem to be implicitly encoded within the model. This baseline will justify why a full environment map is required or necessary for this task.

3) Generalization to scenes: The extent to which the method generalizes to scenes -- not just objects -- is unclear. Evaluating the MIT-multi illumination dataset could shed light on this. The current reliance on explicit environment maps makes it harder to perform on these scenes, but it would be interesting to see if, without explicit environment maps (like suggested above), can you learn to relight and compare on scenes.

4) Evaluation metrics: Recent studies show that PSNR, SSIM, etc. are not consistent with human evaluation. See "Towards a Perceptual Evaluation Framework for Lighting Estimation" (CVPR 2024). These metrics don't tell us much about whether the method is promising as such. A thorough evaluation via user studies or the metrics as defined in the recent paper is currently missing from the paper.

5) Unrealistic results and missing comparisons: The object insertion results look unrealistic, with incorrect shadows that don't match the lighting conditions. Several relevant lighting-aware compositing methods are missing from the comparisons, such as ControlCom [Zhang et al., arXiv 2023], Intrinsic Harmonization [Carega et al., SIGGRAPH 2023], Reshading [Bhattad and Forsyth, 3DV 2022], and ARShadowGAN [CVPR 2020]. The comparison to AnyDoor doesn't make sense as it's not lighting-aware. Including these comparisons would provide a better evaluation of the method's performance against current state-of-the-art techniques.

6) Further, as the papers use off-the-shelf methods to estimate environmental maps (text2light), why not compare with existing 3D object compositing with lighting estimation methods to get a sense of how the proposed methods compare to these tasks -- see Garon et al (CVPR 2019), StyleLight (Wang et al; ECCV 2022) and similar papers? Rendering objaverse objects using lighting estimated from the mentioned or similar methods would help understand the gaps between explicit environment map prediction methods.

7) 3D relighting evaluation: For the 3D relighting setting, according to the Objects with Lighting 3DV 2024 paper, Mitsuba + NeuS is a stronger baseline compared to TensorIR, which is currently missing in the paper.

8) Failure analysis: The paper mentions in the limitations section that the approach might not work for portrait relighting, but it would be interesting to see the kind of failures the diffusion model makes. The current setup lacks experiments in this direction to see what are these failures to encourage future research. Further, the current papers also do not provide any failure examples from Objaverse instances. Is the method perfect on all unseen objects -- detailed analysis is missing as to what objects the proposed methods perform best or worse on. Such analysis helps scope out limitations of the current methods instead of shallow limitations provided in Appendix D.

9) Lack of comparison with simple color matching baselines: The paper doesn't include a comparison with straightforward color adjustment techniques, such as RGB histogram matching between the inserted object and the target scene. This omission raises questions about how much of the method's perceived success in relighting is due to sophisticated light interaction modeling versus simple color transformations. A comparison with such a baseline would help quantify the added value of the diffusion model approach over a simpler method.
questions: questions: 1) Why weren't datasets like OpenIllumination, Objects with Lighting, or Stanford ORB used for evaluation?

2) Have you explored the necessity of full environment maps for relighting?

3) How well does your method generalize to full scenes, beyond individual objects?

4) Given recent findings on the inconsistency of PSNR and SSIM with human perception for lighting tasks, have you considered user study?

5) Why were comparisons with recent lighting-aware object compositing methods (e.g., ControlCom, Intrinsic Harmonization) not included?

6) Have you considered comparing your method with existing 3D object compositing and lighting estimation approaches?

7) Why wasn't Mitsuba + NeuS used as a baseline for 3D relighting, given its reported strength in recent literature?

8) Can you provide a more detailed analysis of failure cases, including examples from Objaverse instances?

9) Can you provide a simple color histogram matching baseline? 

10) Comparison with DiLightNet: DiLightNet offers full image generation with background handling, while Neural Gaffer focuses on object relighting. This raises several points: 

- Background consistency: How does Neural Gaffer address the background when relighting objects?
- Evaluation scope: Are quantitative evaluations done on the full scene or just the object region? This impacts the interpretation of results.
- Lighting control: DiLightNet allows full-scene control. How comprehensive is Neural Gaffer's approach in comparison?
- User input method: DiLightNet uses radiance hints, Neural Gaffer uses environment maps. How do these compare in terms of user-friendliness and control/precision?
- Shadow and Indirect lighting effects quality: DiLightNet's shadows and indirect effects appear more convincing from their project page. Can you comment on this difference? Can you provide a user study comparing the perceived lighting quality between Neural Gaffer and DiLightNet? 

11) How sensitive is your method to the resolution of input environment maps?
limitations: limitations: Somewhat but not fully. See my weakness 8.
paper_summary: paper_summary: The paper introduces Neural Gaffer, an end-to-end 2D relighting diffusion model designed for single-image relighting without the need for explicit scene decomposition. Neural Gaffer can synthesize high-quality relit images of any object under novel environmental lighting conditions by conditioning on a target environment map. The model builds on a pre-trained diffusion model, fine-tuning it on a synthetic relighting dataset. The advantages in generalization and accuracy through evaluations on both synthetic and in-the-wild Internet imagery are shown in the paper. Neural Gaffer can be combined with other generative methods for various downstream 2D tasks like objection insertion. The video results presented in the paper are of high quality.
strength_weakness: strength_weakness: 1) In case of the real-world object scenarios, the object may not be always centred and may have complex backgrounds and lighting to start with. The paper does not demonstrate how would the method behave in such cases. How about the objects with high-frequency texture details?

2) Related to 1) there might be multiple objects in a scene. From the results, it seems that the method cannot handle multiple objects from a single image.

3)  The real-world object examples shown in the paper and the video are good but not impressive. It would be more compelling to show faces, humans, animals etc under the lighting conditions to show the generalizability of the method.
questions: questions: The paper does not demonstrate how the method behaves when the target object is not centred and has complex backgrounds or varied lighting conditions. How does the method perform in such scenarios, especially with objects that have high-frequency texture details?

It appears that the method may struggle with scenes containing multiple objects. Can the authors provide further evaluation or examples to show how the method handles multiple objects in a single image?

While the real-world object examples are good, they are not particularly impressive. Can the authors provide more compelling examples involving faces, humans, or animals under varied lighting conditions to better demonstrate the generalizability of the method? While it's understood that portrait lighting might not be comparable to those methods specifically trained on portraits, it would be good to see the generalizability of the method.
limitations: limitations: The authors have discussed the limitations.
paper_summary: paper_summary: This paper presents a method for relighting objects observed from a single image. While existing approaches rely on specific capture condition using flashlight illumination or portrait captures, or require to explicitly decompose the scene into geometry and reflectance, the proposed method aims to generate images of a given objects under novel illumination conditions for arbitrary environmental lighting conditions. The authors show that this is possible by relying on a generative diffusion method that is conditioned on the environmental map. The method relies on a pre-trained diffusion model that is fine-tuned on a synthetic relighting dataset to learn the conditioning. The approach is evaluated qualitatively and quantitatively on single-object images. Relying on a conditional diffusion model for relighting, the authors also show additional conditioning on text for relighting.
strength_weakness: strength_weakness: What makes the method exciting, at first glance, is also one of the major weaknesses: the technical novelty. The paper piggy-backs on an existing generative method, the Zero-1-to-3 model, that is with a few variations used for relighting. While the simplicity is something that is desired, it also makes it challenging for the reader to derive deeper insights from this work. We learn that pre-trained diffusion-models, when just given enough and the right synthetic data, can allow for plausible novel view synthesis with artefacts that are improved over existing methods. However, the recent work by 

Chong Zeng, Yue Dong, Pieter Peers, Youkang Kong, Hongzhi Wu, and Xin Tong. Dilightnet: Fine-grained lighting control for diffusion-based image generation, 2024.

in a way also does show the exactly same, although the technical approach is different. Overall, the technical contribution of the approach is rather incremental (although the method is effective). As such, I am torn on this work. While the technical contribution is not near other work at NeurIPS, the method is effective and likely of high impact. 

A further qualm I have is regarding the results compared to NVDIFFREC. While the margins are not substantially different, the results in Fig. 6 seem to indicate differently. It seems as if these results are cherry-picked.
questions: questions: See questions regarding trends in the quantitative evaluations and the qualitative results that do not seem to match.
limitations: limitations: All major limitations are addressed. The only open limitation not addressed in the manuscript is the runtime. The authors should address and comment on the runtime for their diffusion model.

Comments: title: Response to rebuttal
comment: Thank you for your time and feedback. We will revise our paper based on our discussions.
Thank you very much! I'm happy to see a simple baseline and user study. While we still have disagreements about the need for environment maps and real-world evaluations (which were my top two concerns in the original review), I believe these issues can be addressed in future work. Because many of my other concerns and questions were addressed, I am raising my rating to borderline accept. I appreciate all the efforts made by the authors. I hope this discussion will be included in the revision.
**4. Questions of the user study**

a. First, we would like to clarify that no author stated, "*a user study cannot be conducted for this task*." The original statement was, *"However, due to the limited time during the rebuttal period, we are unable to complete a user study. If you still believe a user study is critical for evaluation, please let us know. We will attempt to complete a user study before the end of the discussion period."* And we have now conducted the user study during the discussion period.

b. In addition, we would like to further explain why we didn’t do a user study in the main paper: Our paper primarily focuses on relighting. As we said, most recent relighting papers, including the three relighting benchmarks you mentioned(Stanford-ORB, Objects with Lighting, and OpenIllumination), do not conduct user studies when ground truth (GT) data is available. Object compositing is a distinct task from relighting, so its standard practices do not necessarily apply to the relighting community.

We appreciate your sharing of the recent paper https://arxiv.org/abs/2312.04334. According to its abstract, the paper aims to *"provide a new perceptual framework to help evaluate future **lighting estimation algorithms.** "* The caption of Figure 5, which you mentioned, reads, *"Agreement between the observer scores and the metric scores (columns) for all the **lighting estimation methods** ."* Therefore, we believe the main findings of that paper are directly related to lighting estimation and may not be directly applicable to all lighting-related tasks, such as our relighting task.

c. We have now conducted a user study during the discussion period. We randomly sampled 22 objects from our 2D relighting test dataset and randomly selected a target pose and lighting condition. In our user study form, we presented the original input image, the GT relit image, and the relighting results of three different methods (our method, the baseline DiLightNet, and simple color matching) in random order. We then asked users to select the method most similar to the GT image and the second most similar.

We received 43 responses from different volunteers.

As shown below, most users (over 73.9%) found our results to be the most similar to the GT images and very few users found our results least similar to GT, demonstrating a clear advantage of our method over others according to the user study.

**User study**

|  | Our Method | DiLightNet | Simple color matching |
| --- | --- | --- | --- |
| most similar to GT | 73.9% | 21.9% | 4.2% |
| second most similar to GT | 24.2% | 41.1% | 34.6% |
| least similar to GT | 1.9% | 37.0% | 61.2% |
Thanks for your replies. Here are our responses to your remained questions.

1. **Suggestions for Real-World Evaluations**

We appreciate your suggestion to use off-the-shelf models to recover environment maps from the relit images and to use the recovered light maps for relighting and comparison with the ground truth (GT).

However, we would like to emphasize that relighting results are highly dependent on the input target lighting. To our knowledge, there is currently no off-the-shelf lighting estimation model that guarantees accurate and reliable lighting estimation. Therefore, we believe that using estimated lighting for relighting and then comparing it with GT would not provide a fair and reasonable evaluation for either our method or the baselines. If the relit results do not align with the GT, it would be difficult to determine whether the discrepancy is due to the relighting model or the inaccurate lighting estimation.

2. **Re Re Necessity of environment maps**
    
    We would like to reiterate that our task is defined as relighting a single image using a full target environment map, which is a common and well-established task setting in many previous single-image relighting papers. While the idea of using incomplete environment maps as a relighting condition is interesting and could be explored in future work, it represents a different task that lies beyond the scope of this paper, and should not be a ablation study.
    
3.  **Re Re Simple color matching baselines**
    
    In our rebuttal, we mentioned that we did not include a comparison with simple color histogram matching baselines due to limited time and space in the rebuttal PDF (we had fitted 9 figures on a single page), and the object insertion task in which the simple color histogram matching was suggested to compare was not our primary focus and core contribution. In addition, we did commit to including this comparison in the revised paper.
    
    We have now conducted further evaluations using simple color histogram matching baselines to better answer your questions:
    
    a. We tested the simple color histogram matching baseline on our 2D relighting dataset. This involved computing the average color ratio between the target relighting background and the input image, then rescaling each RGB channel of the input image accordingly. We then computed metrics such as PSNR, SSIM, and LPIPS, as shown below. 

    b.  We also included the simple color histogram matching baseline in our user studies. (Please refer to the next reply for more details.)
    
|  | PSNR↑ | SSIM↑ | LPIPS↓ |
| --- | --- | --- | --- |
| Our Method | 26.706 | 0.927 | 0.041 |
| DiLightNet | 24.823 | 0.918 | 0.056 |
| Simple color matching | 21.894 | 0.912 | 0.072 |

Both quantitative comparisons and user studies demonstrate that simple color matching performs poorly compared to both our method and the baseline, DiLightNet. This result is intuitive, as relighting involves not only color matching but also complex appearance changes, such as shadow generation, highlight creation, and other nuanced effects. We have illustrated these challenging scenarios in Figures 1 and 4 of the main paper and on our supplementary website.
Thanks for the detailed response. Some of my concerns are addressed and I appreciate the author's effort. Some questions remain: 

1) One way to evaluate for real-world evaluations might be to use off-the-shelf models to recover environment maps from the relit images -- like DiffusionLight (CVPR 2024) or StyleLight (ECCV 2022). Use these recovered maps as a guide and relight objects and compare them with the GT. 

2) Re Necessity of environment maps: My question was about conducting an ablation study to determine if environment maps are really necessary, or if the task can be accomplished using only target image crops. This is because DiffusionLight demonstrates that Stable Diffusion has a strong grasp of environment maps. 

3) Re Simple color matching baselines: I don't think there is a satisfactory response to this question. I'm curious to know what would happen if authors simply tried to match the RGB values of the object to the target scene. The easiest way to do this would be to apply a correction coefficient to the object that needs to be relit for each RGB channel. This coefficient would be computed as the average color ratio of the target scene to the source scene. This method provides a good baseline for accounting for global color changes and indicates how much of the result is a result of learning to relight, and how much might be the method finding a shortcut to match the overall color distribution of the target scene.

4) I disagree with the authors who claim that a user study cannot be conducted for this task. In general, for most relighting tasks, there are no reliable test sets with actual ground truth. Also note that these are simulated ground truths, based on approximations from the chosen rendering engines, and should not be assumed to be the actual ground truth. 

The authors also mention that in the relighting community, it is not common to evaluate accuracy by comparing different methods through a user study when ground truth results are available. However, in object compositing literature, having a user study is still a standard practice even though simulated GT results are available. In addition, the paper already referenced in my review https://arxiv.org/abs/2312.04334 shows that the LPIPS metric is also not reliable contrary to the authors' argument that they adhere to human perception. Please see Fig 5 of their paper. This finding is not only applicable to lighting estimation methods but can also be used to assess how these metrics fare in general for lighting-related tasks. Relying on the argument that previous approaches used certain metrics is not sound when there is now evidence showing that these metrics are unreliable.
Dear Reviewer YxGp,

We would like to express our deepest gratitude for the time and effort you have dedicated to reviewing our work and for offering so many insightful questions. We greatly appreciate your recognition of its effectiveness and high-quality visual results, and praise that *"it provides an avenue for relighting without collecting expensive relighting real-world datasets."*

As the discussion period will close on August 13th, we kindly ask whether our responses have sufficiently addressed your concerns. If there are any remaining issues or points that require further clarification, please let us know. We are eager to provide any necessary support and continue the dialogue.

Thank you once again for your valuable time and expertise.
Dear Reviewer NcDc,

We would like to express our deepest gratitude for the time and effort you have dedicated to reviewing our work and for offering so many potential improvement suggestions. We greatly appreciate your recognition of its effectiveness, impressive visual results, and extensive ablation studies.

As the discussion period will close on August 13th, we kindly ask whether our responses have sufficiently addressed your concerns. If there are any remaining issues or points that require further clarification, please let us know. We are eager to provide any necessary support and continue the dialogue.

Thank you once again for your valuable time and expertise.
Dear Reviewer rkjr,

We would like to express our deepest gratitude for the time and effort you have dedicated to reviewing our work and for offering so many detailed and insightful questions. We greatly appreciate your recognition of its effectiveness, competitive results, and powerful data-driven potential.

As the discussion period will close on August 13th, we kindly ask whether our responses have sufficiently addressed your concerns. If there are any remaining issues or points that require further clarification, please let us know. We are eager to provide any necessary support and continue the dialogue.

Thank you once again for your valuable time and expertise.
Dear Reviewer C74A,

We would like to express our deepest gratitude for the time and effort you have dedicated to reviewing our work and for offering such insightful questions. We greatly appreciate your recognition of its effectiveness, potential high impact, convincing visual results, and thorough evaluations.

As the discussion period will close on August 13th, we kindly ask whether our responses have sufficiently addressed your concerns. If there are any remaining issues or points that require further clarification, please let us know. We are eager to provide any necessary support and continue the dialogue.

Thank you once again for your valuable time and expertise.
**9.How sensitive is the method to environment map resolution** (in response to question 11) 

​	We always resize the target relighting environment map to the resolution required by our diffusion model (256 * 256) before inputting them into our diffusion model. We implemented an energy-preserving resizing function to do this. Therefore, our model is not sensitive to the environment map resolution theoretically. We have also tested conditioning on environment maps with different original resolutions locally (such as 512 $\times$ 1024, 1024 $\times$  2048, and 2048 $\times$  4096), and our local results don’t show obvious differences in diffusion relighting results.
**8. Questions related to comparison with DiLightNet** (in response to question 10)

Before we answer your questions, we want to first point out that DiLightNet generates the final image by first generating a relit foreground image with its diffusion model (which is exactly what our relighting model can also do), and then generates the background separately. It generates the background in two ways:

(1)When environment maps are given, DiLightNet will render a background image from the target environment map, and then use the foreground mask to composite the relit foreground image and rendered background image to get the final image.
We have done the same thing in our paper. As shown in Fig.1 & 4 of the main paper and the supplementary webpage, we also synthesize the background for our relit image and generate a full image with the background.

(2) When environment maps are not given, DlightNet just uses a pre-trained diffusion-based inpainting model(the stable-diffusion-2-inpainting model [Stability AI 2022a]) to inpaint a background for the relit foreground image. Although we didn’t do this in our paper, using a diffusion-based inpainting model to inpaint the background is very easy for our methods to do.

In short, our method can achieve the same full image generation with the background as DiLightNet can.

- **Question:** *Background consistency: How does Neural Gaffer address the background when relighting objects?*
    
    **Answer:** As mentioned at the beginning of this reply subsection, we render a background image from the target environment map. Please refer to the previous paragraphs for more details.
    
- **Question:** *Evaluation scope: Are quantitative evaluations done on the full scene or just the object region? This impacts the interpretation of results*.
    
    **Answer:** When conducting quantitative evaluations, we first use the GT foreground masks obtained during rendering to mask out all background pixels of the testing image and set background pixels to be all 1 (purely white pixels). Then we compute the metrics. This means that we only compare metrics on the relit foreground, which is fair for both methods.
    
- **Question:** *Lighting control: DiLightNet allows full-scene control. How comprehensive is Neural Gaffer's approach in comparison*?
    
    **Answer:** We are not sure what full-scene control means here. Could you please give us more explanation for this question?
    
- **Question:** *User input method: DiLightNet uses radiance hints, Neural Gaffer uses environment maps. How do these compare in terms of user-friendliness and control/precision?*
    
    **Answer:**
    

	First, DiLightNet uses radiance hints as its diffusion input, not as the user inputs. Radiance hints are attained by first estimating a coarse mesh of the single image input and using Blender to render the estimated mesh with some predefined and fixed BRDF under the target lighting. To relight an image with DiLightNet, users will first need to specify the lighting information used to render the radiance hints. So the real user inputs of *DiLightNet* are the lighting information, i.e. environment map, etc. Therefore, our method has a similar user input as DiLightNet.

	Second, our methods are more user-friendly. This is because our method is purely end-to-end — users only need to specify the target environment map, and our relighting diffusion will output the relighting diffusion directly. On the contrary, to relight a single image with DiLightNet, users need to first run a monocular depth estimator to attain the mesh, (which may fail) and then need to use Blender to render the radiance hints and use them as the diffusion inputs. This whole process is more complicated and tricky than our methods.

- **Question:** *Shadow and Indirect lighting effects quality: DiLightNet's shadows and indirect effects appear more convincing from their project page. Can you comment on this difference? Can you provide a user study comparing the perceived lighting quality between Neural Gaffer and DiLightNet?*
    
    **Answer:**
    

	DiLightNet’s results may look good separately but they are not accurate when compared with the ground truth relighting results, as we show in the Fig. 4 of our main paper. In addition to visual quality, relighting should also achieve accuracy. As we find in Table 1 of our main paper, our methods achieve more accurate relighting results while maintaining high visual quality.

	We explain why we didn’t perform a user study in the previous subsection of this rebuttal reply.

---

Unfinished. Please keep reading the following comments
**5. Questions related to object insertion** (in response to weaknesses 5，6，9 and questions 5，6，9)

​	We sincerely appreciate your detailed and insightful suggestions regarding object insertion. Before addressing each of your questions, we would like to clarify that the primary focus of this paper is on relighting, rather than object insertion. In this work, we introduce a robust 2D relighting diffusion model, which can serve as relighting data prior for various downstream tasks and applications in both 2D and 3D. To illustrate its potential, we developed a straightforward object insertion pipeline as one of the 2D application examples. However, we do not consider object insertion to be the core contribution or the main focus of our work.

​a. Provide more baselines for object insertion comparison, including a simple color histogram matching baseline

Due to limited time and space in the rebuttal PDF, and because object insertion is not the primary focus and core contribution of this work, we have not provided additional comparisons at this time. We did not compare our methods with the histogram-matching baseline because it is too simple and has not been compared in recent work. However, we are willing to include these baselines you mentioned in our revised paper.


b. Comparison with existing 3D object compositing approaches

We used DiffusionLight (CVPR 2024) to estimate the environmental lighting of the scene image in our object insertion results, rather than the Text2Light method you mentioned. Since 3D object compositing and lighting estimation are not the tasks of this paper, we did not consider comparing against the methods you mentioned. In theory, our methods could be combined with the lighting estimation techniques you suggested, potentially leading to improved performance in the object insertion task. We appreciate your suggestion and will consider exploring these design choices in future work, but we emphasize that understanding the gaps between different lighting estimation methods is beyond the scope of this paper.

**6. Compare with “NeuS+ Mitsuba” in the 3D relighting task** (in response to weaknesses 7 and question 7)

1. We have tried to compare with the SOTA methods in our 3D relighting experiments. Specifically, we chose Nvdiffrec-mc and TensoIR for comparison in our paper. According to the Stanford-ORB benchmarks, Nvdiffrec-mc achieves SOTA results among all tested methods. Similarly, TensoIR attains SOTA results based on the OpenIllumination benchmarks.
2. We did not compare our methods with Neus+Mitsuba because it is not a commonly used baseline in the field of 3D relighting, and we were not aware of it initially. To our knowledge, Neus+Mitsuba has only been compared in the paper "Objects with Lighting," and we have not seen other recent 3D relighting work that includes this baseline. 
3. During the rebuttal period, we tested Neus+Mitsuba on our testing dataset using its official code. The results showed that its performance metrics are as follows: PSNR: 26.47, SSIM: 0.912, and LPIPS: 0.073. Therefore, our methods outperform Neus+Mitsuba on all metrics. 

**7. Detailed Analysis of Failure Cases** (in response to weaknesses 8 and question 8)

Thanks for your suggestions. We have provided a more detailed analysis of the failure cases below:

a. Portrait Relighting Analysis

Overall, our method demonstrates good generalization to human portraits in most instances (see the first two rows of Fig. 6 in the rebuttal PDF). However, we observed that it sometimes struggles to preserve facial details. For instance, in the failure case shown in the last row of Fig. 6, our model failed to retain crucial details of the portrait, such as the shape of the mouth. We believe that further fine-tuning our model with additional portrait data could improve its performance on such images.

b. Limitations in Objaverse Instances

Color Ambiguity: Our model sometimes fails to resolve inherent color ambiguities, particularly when relighting diffuse objects. As shown in the first row of Fig. 7 in the rebuttal PDF, the model struggles to produce accurate color in the relit results. This issue arises from the inherent color ambiguity, where the color in the input image could be attributed to either the material or the lighting, making it a challenging problem. We believe that including the background of the input image in the diffusion model could help mitigate this issue, as the model could learn to infer the lighting conditions from the background. We consider this a potential avenue for future work.

High-Frequency Reflection Generation: Our model also struggles to generate high-frequency relighting reflections. For example, in the second row of Fig. 7 in the rebuttal PDF, while the model accurately generates low-frequency highlights (as indicated by the red box), it fails to produce high-frequency reflections (as indicated by the green box).

---
Unfinished. Please keep reading the following comments.