Title: Neural Gaffer: Relighting Any Object via Diffusion
Venue: NeurIPS 2024
Accepted: yes
Keywords: relighting

single-image relighting

diffusion models

3D relighting.
Abstract: Single-image relighting is a challenging task that involves reasoning about the complex interplay between geometry, materials, and lighting. Many prior methods either support only specific categories of images, such as portraits, or require special capture conditions, like using a flashlight. Alternatively, some methods explicitly decompose a scene into intrinsic components, such as normals and BRDFs, which can be inaccurate or under-expressive. In this work, we propose a novel end-to-end 2D relighting diffusion model, called Neural Gaffer, that takes a single image of any object and can synthesize an accurate, high-quality relit image under any novel environmental lighting condition, simply by conditioning an image generator on a target environment map, without an explicit scene decomposition. Our method builds on a pre-trained diffusion model, and fine-tunes it on a synthetic relighting dataset, revealing and harnessing the inherent understanding of lighting present in the diffusion model. We evaluate our model on both synthetic and in-the-wild Internet imagery and demonstrate its advantages in terms of generalization and accuracy. Moreover, by combining with other generative methods, our model enables many downstream 2D tasks, such as text-based relighting and object insertion. Our model can also operate as a strong relighting prior for 3D tasks, such as relighting a radiance field.

Neural Gaffer: Relighting Any Object via Diffusion

Abstract

Single-image relighting is a challenging task that involves reasoning about the complex interplay between geometry, materials, and lighting. Many prior methods either support only specific categories of images, such as portraits, or require special capture conditions, like using a flashlight. Alternatively, some methods explicitly decompose a scene into intrinsic components, such as normals and BRDFs, which can be inaccurate or under-expressive. In this work, we propose a novel end-to-end 2D relighting diffusion model, called Neural Gaffer 1 , that takes a single image of any object and can synthesize an accurate, high-quality relit image under any novel environmental lighting condition, simply by conditioning an image generator on a target environment map, without an explicit scene decomposition. Our method builds on a pre-trained diffusion model, and fine-tunes it on a synthetic relighting dataset, revealing and harnessing the inherent understanding of lighting present in the diffusion model. We evaluate our model on both synthetic and in-the-wild Internet imagery and demonstrate its advantages in terms of generalization and accuracy. Moreover, by combining with other generative methods, our model enables many downstream 2D tasks, such as text-based relighting and object insertion. Our model can also operate as a strong relighting prior for 3D tasks, such as relighting a radiance field.

1 Introduction

Lighting plays a key role in our visual world, serving as one of the fundamental elements that shape our interpretation and interaction with 3D space. The interplay of light and shadows can highlight textures, reveal contours, and enhance the perception of shape and form. In many cases, there is a desire to relight an image-that is, to modify it as if it were captured under different lighting conditions. Such a relighting capability enables photo enhancement (e.g., relighting portraits [18; 67]), facilitates consistent lighting across various scenes for filmmaking [59; 22], and supports the seamless integration of virtual objects into real-world environments [38; 19].

However, relighting single images is a challenging task because it involves the complex interplay between geometry, materials, and illumination. Hence, many classical model-based inverse rendering approaches aim to explicitly recover shape, material properties, and lighting from input images [2; 75; 47; 83; 87], so that they can then modify these components and rerender the image. These methods often require multi-view inputs and can suffer from: 1) model limitations that prevent faithful estimation of complex light, materials, and shape in real-world scenes, and 2) ambiguities in determining these factors without strong data-driven priors.

To circumvent these issues, image-based relighting techniques [18; 72; 76] avoid explicit scene reconstruction and focus on high-quality image synthesis. However, some image-based methods

Condition

In a bustling

Atop a windswept

cliff overlooking

the ocean

[18; 58; 76; 44; 3] require inputs under multiple lighting conditions or even sophisticated capture setups (e.g., a light stage [18]), while recent deep learning-based techniques [67; 51; 89] enable single-view relighting but often work only on specific categories like portraits.

In this paper, we propose a novel category-agnostic approach for single-view relighting that tackles the challenges mentioned above. We introduce an end-to-end 2D relighting diffusion model, named Neural Gaffer , that takes a single image of any object as input and synthesizes an accurate and high-quality relit image under any environmental lighting condition, specified as a high-dynamic range (HDR) environment map. In contrast to previous learning-based single-view models that are trained for specific categories, we leverage powerful diffusion models, trained on diverse data, and enable relighting of objects from arbitrary categories. Unlike prior model-based approaches, our diffusion-based data-driven framework enables the model to learn physical priors from an objectcentric synthetic dataset featuring physically-based materials and High Dynamic Range (HDR) environment maps. This facilitates more accurate lighting effects compared to traditional methods.

Our model exhibits superior generalization and accuracy on both synthetic and real-world images (See Fig. 1). It also seamlessly integrates with other generative methods for various 2D image editing tasks, such as object insertion. Neural Gaffer also can serve as a robust relighting prior for neural radiance fields, facilitating a novel two-stage 3D relighting pipeline. Hence, our diffusion-based solution can relight any object under any environmental lighting condition for both 2D and 3D tasks. By harnessing the power of diffusion models, our approach generalizes effectively across diverse scenes and lighting conditions, overcoming the limitations of prior model-based and image-based methods.

2 Related Work

Diffusion models. Recently, diffusion-based image generation models [26; 49] have dominated visual content generation tasks. Stable Diffusion [60] not only generates impressive results in the text-to-image setting, but also enables a wide range of image applications. ControlNet [84] adds trainable neural network modules to the U-Net structure of diffusion models to generate images

conditioned on edges, depth, human pose, etc. Diffusion models can also change other aspects of an input image [45; 9], such as image style, via image-to-image models [52]. ControlCom [80] and AnyDoor [14] are image composition models that can blend a foreground object into a target image. Huang et al. provide a comprehensive review of diffusion-based image editing [28].

Recent work has also shown that, despite being trained on 2D images, other kinds of information can be unlocked from diffusion models. A notable example is Zero-1-to-3 [42], which uncovers 3D reasoning capabilities in Stable Diffusion by fine-tuning it on a view synthesis task using synthetic data, and conditioning the model on viewpoint. Similar to the spirit of Zero-1-to-3 and its follow-up work (such as [63; 27]), our work unlocks the latent capability of diffusion models for relighting.

Single-image relighting. Relighting imagery is a difficult task that requires an (explicit or implicit) understanding of geometry, materials, lighting, and their interaction through light transport. Singleimage relighting is especially challenging due to its ill-posedness; there are infinite explanations of a single image in terms of underlying factors. Classic single-image methods, like SIRFS [2], focus on explicit estimation of scene properties, via optimization with hand-crafted priors. More recent learning-based methods can learn such priors from data, but either focus on specific image types, such as outdoor scenes [73; 69; 78; 40; 21], portraits [53; 71; 64; 67; 48; 51; 23; 56], or human bodies [31; 37; 29], or require special capture conditions like camera flash [39; 61]. Single-image relighting methods for general objects, with realistic appearance, materials, and lighting, remain elusive. In addition, most recent single-image relighting methods [51; 39; 61; 12] still explicitly decompose an image into components, then reconstruct it with modified lighting. This strategy allows the components to be explicitly controlled, but also limits their realism based on the representational ability of the chosen components, such as the Disney BSDF [11] or Phong reflection model [10]. In contrast, similar to [67; 79], we achieve object relighting without an explicit scene decomposition. By training an image-based and lighting-conditioned diffusion model, we can relight any given object from a single photo.

3D relighting and inverse rendering. Inverse rendering methods involve estimating 3D models that can be relit under different lighting conditions, typically from multiple images. Recent advances in 3D, such as NeRF [46] and 3D Gaussian Splatting [32], have shown remarkable novel view synthesis results. However, these representations are not themselves relightable as the physical light transport is baked into the radiance field. Efforts have been made toward creating relightable 3D representations, such as PhySG [83], NeRFactor [87], TensoIR [30] and others [6; 8; 7; 5; 4; 82; 88; 35; 66; 77; 81]. These methods often rely on pure optimization to minimize a rendering loss, sometimes incorporating hand-crafted material and/or lighting regularization terms (e.g., smoothness or sparsity) to prevent overfitting. However, these approaches often struggle with specular highlights and glossy materials. We show that our proposed Neural Gaffer method can be applied in an inverse rendering framework, serving as a powerful prior and regularizer. This is because our model has been trained on a large dataset encompassing diverse lighting and material conditions, effectively learning these priors from high-quality data. Our model improves 3D relighting quality, while baselines tend to produce artifacts due to the baking of lighting information into the color field.

3 Method

Neural Gaffer is a generalizable image-based 2D diffusion model for relighting general objects; we finetune a pretrained image-conditioned diffusion model [42] with a synthetic relighting dataset to fulfill this goal. In Sec. 3.1, we describe how we construct a large-scale high-quality relighting dataset from Objaverse [20]. Then, in Sec. 3.2, we discuss the design and training of a 2D diffusion model to support zero-shot single-image relighting. Finally, we showcase applications of our model in downstream tasks, including 2D object insertion and 3D radiance field relighting tasks.

3.1 RelitObjaverse Dataset

Capturing ground-truth lighting for real object images is both time-consuming and challenging. Existing real-world relighting datasets [41; 36; 70] are constrained to a very limited number of objects. To address this limitation, we opt to render a large-scale synthetic object relighting dataset.

We use Objaverse [20] as our data source, which comprises about 800K synthetic 3D object models of varying quality. To avoid 3D object models with poor geometry or texture, we developed a series of

filtering mechanisms. Specifically, we first utilize a classifier trained on a manually annotated subset to categorize and filter out the 3D objects with low geometry and texture quality. In addition, we also design a set of filtering rules based on 3D object attributes, such as geometry bounding boxes, BRDF attributes, and file size, eventually yielding a subset (of ∼ 90K objects) of the original Objaverse with high quality. To ensure abundant lighting variations, we collected 1,870 HDR environment maps from the Internet, and augmented them with horizontal flipping and rotation, resulting in ∼ 30K different environment maps in total.

We render images with the Cycles renderer from Blender [17]. During rendering, we randomly sample 12 camera poses for each object. For each viewpoint, we render 16 images under different environment maps. In contrast to existing datasets that only render with environment maps [16; 62], we additionally render an image under a randomly set area light source to increase lighting diversity. In total, for each object, there are 12 × (16 + 1) = 204 images rendered under different lighting conditions and from different viewpoints. Our final synthetic relighting dataset has about 18.4M 512 × 512 rendered images with ground-truth lighting. We will release our full relighting dataset, named the RelitObjaverse dataset, upon acceptance to facilitate reproducibility.

3.2 Lighting-conditioned 2D Relighting Diffusion Model

Given a single object image x ∈ R H × W × 3 and a new lighting condition E ∈ R H ′ × W ′ × 3 (represented as an HDR environment map), our goal is to learn a model f that can synthesize a new image ˆ x E with correct appearance changes under the new lighting, while preserving the object's identity:

We fine-tune an image-to-image diffusion model using our rendered data. In particular, we adopt the pre-trained Zero-1-to-3 model [42] and discard its components and weights relevant to novel-view generation, repurposing the model for relighting. To incorporate lighting conditions, we extend the input latents of the diffusion model, concatenating the encoded input image and the environment map with the denoising latent. We introduce two design decisions that are critical for the effective integration of lighting into the latent diffusion model.

Environment map rotation. In general, each pixel of an environment map E corresponds to a 3D direction in the absolute world coordinate frame, whereas our relighting is performed in image space, that is, locked to the camera's coordinate frame. Therefore, we need a way for a user to specify the relationship between the environment map and camera coordinate frames. One option is to concatenate the relative 3D rotation to the embedding, similar to how Zero-1-to-3 [42] encodes the relative camera pose for the novel view synthesis task. However, we observe that such a design makes it harder for the network to learn the relighting task, perhaps because it needs to interpret the complex relationship between 3D rotation parameters and the desired rotated lighting. Our solution is instead to rotate the target environment map to align with the target camera coordinate frame, prior to feeding it to the model. After rotation, each pixel of the rotated environment map corresponds to a fixed lighting direction in the target camera coordinate frame.

HDR-LDR conditioning. Pixel values in the HDR environment map have a theoretical range of [0 , + ∞ ) , and large pixel values can lead to unstable diffusion training. To rescale it to [0 , 1 . 0] , a naive idea is to divide it by its maximum value, but this leads to a loss of total energy information. Also, if the map has a very bright region (e.g., the sun), then most of the other values will approach 0 and the environment map will be too dark after division, causing a loss of lighting detail. To solve this, we split the HDR environment map E into two maps: E L is an LDR representation of E , computed via standard tone-mapping [57; 24], and E H is a nonlinearly normalized representation of E , computed by a logarithmic mapping and divided by the new maximum value. E L encodes lighting detail in low-intensity regions, which is useful for generating relighting details(such as reflection details). E H maintains the lighting information across the initial spectrum. Inputting E L

LDR Map

Normalized

HDR Map

Noisy Latent

Encoder

Encoder

Encoder

CLIP

embedding

Conditioning

Decoder

Target Environment Map

Denoiser

Input Poses

…

Render

𝐿

$%&'(

Input Images HDR maps LDR maps

STAGE 1

∇

no grad.

Pretrained NeRF

Geometry

Appearance

Neural Gaffer

Noisy latents

Sampled Pose

Encoder

Add

noise

Neural Gaffer

Render

𝐿

)!**

∇

no grad.

STAGE 2

HDR map LDR map

and E H together to the diffusion model means that the network can reason effectively about the full original energy spectrum. Additionally, our design helps to achieve a balanced exposure in the generated output, preventing areas from appearing too dark or too washed out after normalization due to extreme brightness.

Diffusion model architecture and training. Our latent diffusion model comprises an encoder E , a denoiser U-Net ϵ θ , and a decoder D . During each training iteration, given the paired input-target images x and ˆ x E , and the target lighting condition E , we (1) rotate E into the target camera coordinate frame based on the specified lighting direction, forming a rotated map ˆ E ; (2) transform ˆ E into an LDR environment map ˆ E L and a normalized HDR environment map ˆ E H ; (3) and finally, encode ˆ E L , ˆ E H and the input image x with the encoder E . The concatenation of the encoded latent maps are used to condition the denoising U-Net ϵ θ . Fig. 2 depicts our 2D relighting system. We fine-tune the latent diffusion model with the following objective:

where t ∼ [1 , 1000] is the diffusion time step, ˆ E = {E ( x ) , E ( ˆ E L ) , E ( ˆ E H ) } represents the union of encoded latents for x , ˆ E L and ˆ E H , and c ( x ) represents the CLIP embedding of the input image x . Example 2D relighting results from our trained model, applied to real images, are shown in Fig. 1.

3.3 Relighting a 3D radiance field with diffusion prior guidance

We can utilize Neural Gaffer as a prior for 3D relighting. In particular, given a 3D neural radiance field consisting of a density field G σ and a view-dependent appearance field G a [46], we consider the task of relighting it using our 2D relighting diffusion model as a prior. We find that simply relighting the input NeRF with the popular Score Distillation Sampling (SDS) method [55] leads to over-saturated and unnatural results, as shown in Fig. 8. We instead propose a two-stage pipeline (Fig. 3) to achieve high-quality 3D relighting in minutes. In the first stage, we use our 2D relighting diffusion model to relight input views and overfit the original NeRF's appearance field, giving us coarsely relit renderings at sampled viewpoints. In the second stage, our model is used to further refine the appearance details with a diffusion loss and an exponential annealed multi-step scheduling.

Stage 1: Coarse relighting with reconstruction loss. Given a NeRF with parameters ψ , the multiview input images x input = { x i } N i =1 and the target lighting E target, we first use our diffusion model f to generate multi-view relighting prediction images ˆ x relit = { ˆ x i } N i =1 , where ˆ x i = f ( x i , E target ) .

Then, we freeze the density field G σ of the NeRF, and only optimize the appearance field of the input NeRF with reconstruction loss L recon., aiming to minimize the reconstruction error between a rendered image x = x ( ψ, π input ) and the relit image x relit at pose π input :

After this stage, we get an initial blurry and noisy relighting result.

Stage 2: Detail refinement with diffusion guidance loss. The results from the first stage can be blurry and can exhibit noise because the multi-view relighting predictions results may be inconsistent.

We further refine the relit NeRF's appearance in Stage 2. We found that directly using the SDS loss [55] for refinement leads to noisy and over-saturated results (see Fig. 8). Inspired by [68; 74; 45], we treat the blurry rendered images from Stage 1 as an intermediate denoising result. We first render an image ˆ x under the sampled camera pose π sampled. Weencode and perturb it to a noisy latent ˆ z t (ˆ x, t ⋆ ) with an intermediate noise level t ⋆ , and initiate the multi-step DDIM [65] denoising process for ˆ z t (ˆ x, t ⋆ ) from t ⋆ , yielding a latent sample ˆ z 0 . This latent is decoded to an image ˆ x relit ( t ⋆ ) = D ( ˆ z 0 ) , which we use to supervise the rendering with the following diffusion guidance loss:

where L p is the perceptual distance LPIPS [86] and w 1 and w 2 are scale factors to balance the two terms. We exponentially decrease t ⋆ during optimization.

4 Experiments

4.1 Implementation details

Diffusion model training. We fine-tune our model starting from Zero123's [42] checkpoint and discard its original linear projection layer for image embedding and pose. We only fine-tune the UNet of the diffusion model and freeze other parts. During fine-tuning, we use a reduced image size of 256 × 256 and a total batch size of 1024. Both the LDR and normalized HDR environment maps are resized to 256 × 256. We use AdamW [43] and set the learning rate to 10 -4 for training. We fine-tune our model for 80K iterations on 8 A6000 GPUs for 5 days.

3D relighting. Given a neural radiance field reconstructed from multi-view inputs (we use TensoRF [13]), we first freeze its density field and then optimize the input NeRF's appearance field for 2,500 iterations in Stage 1, with a ray batch size of 4096. In Stage 2, we set the initial timestep t ⋆ to decrease from 0.4 to 0.05 exponentially. w 1 and w 2 in Eqn. 4 are set to be 0.5 for all of our test cases. We optimize the results from Stage 1 for 500 iterations, and for each iteration, we sample one view to compute the diffusion guidance loss in Eqn. 4. Stage 1 takes about 5 minutes and Stage 2 takes about 3 minutes on a single A6000 GPU.

Downstream 2D tasks. Our model supports downstream 2D image editing tasks, such as textbased relighting and object insertion. For text-based relighting, given a text prompt, we first use Text2Light [15] to generate an HDR panorama, and, using that as an environment map, relight images using our diffusion model. Fig. 1 (right two columns) shows example results; more are on our project webpage. For object insertion, given an object image and a target (perspective) scene image, we segment the foreground object with SAM [34], estimate lighting from the target background image with DiffusionLight [54], perform relighting, and finally generate a shadow attached to the object using [50]. Our workflow can preserve the object's identity and achieve high-quality visual quality.

4.2 Baselines and metrics

We use common image metrics including PNSR, SSIM, and LPIPS [86] to evaluate relighting results quantitatively. In addition, since there exists an ambiguity between lighting intensity and object color, we also compute channel-aligned results as in [87; 88], where each channel of the relighting prediction is rescaled to align the average intensity with the ground-truth. We compute metrics for both the raw and the channel-aligned results for quantitative evaluation.

For the single-image relighting task, there are very few prior works that can handle general objects under natural illumination. We compare to a very recent work DiLightNet [79], which trains a lightingconditioned ControlNet [84] to relight images, which is achieved by conditioning its ControlNet with a set of rendered images using predefined materials, estimated geometry of the input image, and target lighting. Per our request, the authors ran their model on our validation data. We select 48 high-quality objects from Objaverse as validation objects, which are unseen during training. We render each object under 4 different camera poses. For each camera, we randomly sample 12 unseen environment maps to render the target relit images, and one additional environment map to render the input. We also compare with IC-Light [85]. Since our lighting conditioning is different from its setting (they use background images), we only provide qualitative comparisons in our appendix and project webpage. For relighting 3D radiance fields, we compare with two 3D inverse rendering methods: TensoIR [30] and NVDIFFREC-MC [25]. We select 6 objects from the NeRF-Synthetic

dataset [46] and Objaverse [20] for testing. For each object, 100 camera poses are sampled to generate training images and 20 camera poses are sampled for testing images. Each pose is rendered under four distinct, previously unseen environmental lighting conditions. One lighting condition is designated for training images, while the remaining three are reserved for relighting testing.

4.3 Results analysis

Comparing single-image relighting with DiLightNet [79]. As shown in Fig. 4, we evaluate across a diverse set of unseen objects and unseen target lighting conditions. Our approach showcases superior fidelity to the intended lighting settings, maintaining consistent color accuracy and preserving intricate details. Compared with DiLightNet [79], our model can generate more accurate highlights, shadows, and high-frequent reflections. These results demonstrate our model's ability to accurately understand the underlying physical properties of the object in the input image, such as geometry and materials, and faithfully reproduce the desired relighting appearance. Quantitative results are in Tab. 1.

Single-image relighting on real data. We evaluate our single-image relighting model on a set of realworld input photos (Fig. 1), under either image-conditioned (environment map) or text-conditioned (text description of the desired target lighting) target illumination. As shown in the figure, our model effectively adapts to diverse lighting conditions and produces realistic, high-fidelity relighting results under novel illumination. In addition, our relighting results are consistent and stable with the lighting condition rotating. Please refer to our webpage to get more video results of more real data.

DiLightNet

Ours

Ground-truth

Target Lighting

O

Object insertion. Our diffusion model exhibits versatility when combined with other generative models for downstream 2D tasks, like object insertion. In Fig. 5, we take a foreground image containing the object of interest and insert the object into a background image depicting the desired target lighting condition. Our method surpasses baseline approaches by better preserving the identity of the inserted objects and synthesizing more natural illumination, seamlessly integrating objects into the background scene.

Background Image

Est. Lighting & Relighting

Our Results

Simply Copying

AnyDoor Results

Ground-truth

NVDIFFREC-MC

TensoIR

Ours

Input Views

Target Lighting

《《

7

Relighting 3D objects. Fig. 6 shows the results of our two-stage method for relighting 3D objects. Given an input radiance field(NeRF) representing a 3D object, we first tune NeRF for coarse relighting in Stage 1, followed by a finer relighting result in Stage 2 that improves fidelity. We also compared to conventional inverse rendering methods such as NVDIFFREC-MC [25] and TensoIR [30], reporting quantitative results in Tab. 2 and showing qualitative comparisons in Fig. 6 and Fig. 13.

4.4 Ablation Study

Ours

Ground-truth

w/o HDR map

w/o LDR map

Input image

w/o Rotation

Ablation studies for alternate lighting conditioning designs for our diffusion model.

As ablations, we evaluated different lighting conditioning designs for our diffusion model: 1) only inputting one environment map rather than both the rotated LDR and normalized HDR map, and 2) controlling the lighting direction by adding the relative 3D rotation between the default environment map coordinate frame and the target camera frame to the U-Net CLIP embedding branch without rotating the environment maps explicitly.

Due to limited computational resources, when performing ablations for the model design, we train the different models with 2 A6000 GPUs for 48K iterations. We provide a quantitative comparison between them in Tab. 3 and a qualitative comparison in Fig. 7, showcasing the effectiveness of our design. As shown in Fig 7, controlling the lighting direction by only adding the relative 3D lighting rotation to the U-Net CLIP embedding branch can't get the relighting results with correct lighting directions for all testing cases. Without inputting the LDR map, the model fails to generate the correct reflection for the first test case. Without the HDR map, it cannot produce the correct specular highlights for the third test case. Only when both the LDR and HDR maps are provided can the model accurately reason about the full original energy spectrum and generate correct relighting results for the second test case. Our full model achieves the best results across different ablation experiments.

Ablation studies for relighting 3D radiance fields. As shown in Fig. 8, if we directly relight the input neural radiance field using an SDS loss [55] without our two-stage pipeline, we cannot accurately reproduce highlights and reflections in the first scene, generate precise shadows in the

w/o stage 2

Input

Ground-truth

w/o stage 1, 2 &

Relight w/ SDS loss

Ours

w/o stage 1

second sofa scene, or remove shadows in the third hotdog scene. Stage 1 ensures overall relighting accuracy, while Stage 2 helps reduce noise and leads to more detailed results. Without Stage 1, we cannot generate the correct specular highlight for the first scene, and the details of the shadows on the sofa scene's backrest become noisy and less accurate. Without Stage 2, the appearance of the relit scenes becomes noisier and blurrier, especially in the sofa scene. Replacing our diffusion guidance loss (Eqn. 4) in Stage 2 with an SDS loss results in unnatural, noisy, and over-saturated relighting. Our full model consistently produces the most accurate and visually realistic results across all test variants. Please zoom in to see detailed differences across ablation variants.

5 Limitations

While our model can achieve overall consistent relighting results under rotating or changing lighting conditions, our results can still exhibit minor inconsistencies since the model is generative.

Given the high resource demands of data preprocessing (specifically, rotating the HDR environment map) and model training, and considering our limited university resources, we trained the model at a lower image resolution of 256 × 256 . In Fig. 9, we use V AE [33] in our base diffusion model [1; 42] to encode input images into latent maps, and then directly decode them. We found that V AE struggles to preserve identity for objects with fine details even from latent maps encoded from the input images at this resolution, which in turn results in many relighting failure cases at this resolution. Finetuning our model at a higher resolution will greatly help solve this issue.

We observe two key failure cases when relighting Objaverse Instances, as shown on the right side of Fig. 9. First, our model struggles with color ambiguity in diffuse objects, leading to inaccuracies in relit colors (first row). This issue arises from inherent ambiguities in the input image, where color could be attributed to either material or lighting. Including the background of the input image may help the model better infer the lighting conditions of input images and address this challenge. We give more discussions in the Appendix. C. Second, while the model generates low-frequency highlights well, it often misses high-frequency reflections in relit images (second row).

In addition, we discuss our limitations regarding portrait relighting in the Appendix. B.

6 Conclusion

We proposed an end-to-end 2D relighting diffusion model, Neural Gaffer, that addresses the longstanding challenge of single-image relighting through a data-driven framework. Our model synthesizes accurate and high-quality relighted images from a single input image under any environmental lighting condition, demonstrating superior generalization and accuracy on both synthetic and realworld datasets. Neural Gaffer not only enhances various 2D tasks, such as text-based relighting and object insertion but also serves as a strong relighting prior for 3D tasks, enabling a simplified two-stage 3D relighting pipeline. By overcoming the limitations and ambiguities of traditional model-based inverse rendering approaches, our diffusion-based solution extends the applicability of relighting techniques to a broad range of practical applications.

Acknowledgments. We thank Minghua Liu, Chao Xu, et al from Prof. Hao Su's group at UCSD for assisting with the Objaverse filtering process. We thank Xinrui Liu at Cornell for helping with data capture and shadow generation for the object insertion application. We thank Chong Zeng at Zhejiang University for helping run his model (DiLightNet) on our validation data

This work was done while Haian Jin was a full-time student at Cornell. The selection of data and the generation of all figures and results was led by Cornell University. This work was funded in part by the National Science Foundation (IIS-2211259). Jin Sun is partly supported by a gift from Google.

References

Stable diffusion image variations - a hugging face space by lambdalabs.

Jonathan T. Barron and Jitendra Malik. Shape, illumination, and reflectance from shading. Trans. Pattern Analysis and Machine Intelligence , 2015.

Sai Bi, Stephen Lombardi, Shunsuke Saito, Tomas Simon, Shih-En Wei, Kevyn Mcphail, Ravi Ramamoorthi, Yaser Sheikh, and Jason Saragih. Deep relightable appearance models for animatable faces. ACM Transactions on Graphics (TOG) , 40(4):1-15, 2021.

Sai Bi, Zexiang Xu, Pratul Srinivasan, Ben Mildenhall, Kalyan Sunkavalli, Miloš Hašan, Yannick Hold-Geoffroy, David Kriegman, and Ravi Ramamoorthi. Neural reflectance fields for appearance acquisition. arXiv preprint arXiv:2008.03824 , 2020.

Sai Bi, Zexiang Xu, Kalyan Sunkavalli, Miloš Hašan, Yannick Hold-Geoffroy, David Kriegman, and Ravi Ramamoorthi. Deep reflectance volumes: Relightable reconstructions from multi-view photometric images. In Proc. European Conf. on Computer Vision (ECCV) , pages 294-311. Springer, 2020.

Mark Boss, Raphael Braun, Varun Jampani, Jonathan T Barron, Ce Liu, and Hendrik Lensch. Nerd: Neural reflectance decomposition from image collections. In Proc. Int. Conf. on Computer Vision (ICCV) , pages 12684-12694, 2021.

Mark Boss, Andreas Engelhardt, Abhishek Kar, Yuanzhen Li, Deqing Sun, Jonathan Barron, Hendrik Lensch, and Varun Jampani. Samurai: Shape and material from unconstrained realworld arbitrary image collections. Neural Information Processing Systems , 35:26389-26403, 2022.

Mark Boss, Varun Jampani, Raphael Braun, Ce Liu, Jonathan Barron, and Hendrik Lensch. Neural-pil: Neural pre-integrated lighting for reflectance decomposition. Neural Information Processing Systems , 34:10691-10704, 2021.

Tim Brooks, Aleksander Holynski, and Alexei A Efros. Instructpix2pix: Learning to follow image editing instructions. In Proc. Computer Vision and Pattern Recognition (CVPR) , pages 18392-18402, 2023.

Phong Bui-Tuong. Illumination for computer generated pictures. CACM , 1975.

Brent Burley and Walt Disney Animation Studios. Physically-based shading at disney. In Acm Siggraph , volume 2012, pages 1-7. vol. 2012, 2012.

Yuanhao Cai, Hao Bian, Jing Lin, Haoqian Wang, Radu Timofte, and Yulun Zhang. Retinexformer: One-stage retinex-based transformer for low-light image enhancement. In ICCV , 2023.

Anpei Chen, Zexiang Xu, Andreas Geiger, Jingyi Yu, and Hao Su. Tensorf: Tensorial radiance fields. In European Conference on Computer Vision (ECCV) , 2022.

Xi Chen, Lianghua Huang, Yu Liu, Yujun Shen, Deli Zhao, and Hengshuang Zhao. Anydoor: Zero-shot object-level image customization, 2024.

Zhaoxi Chen, Guangcong Wang, and Ziwei Liu. Text2light: Zero-shot text-driven hdr panorama generation, 2023.

Jasmine Collins, Shubham Goel, Kenan Deng, Achleshwar Luthra, Leon Xu, Erhan Gundogdu, Xi Zhang, Tomas F. Yago Vicente, Thomas Dideriksen, Himanshu Arora, Matthieu Guillaumin, and Jitendra Malik. Abo: Dataset and benchmarks for real-world 3d object understanding, 2022.

Blender Online Community. Blender - a 3D modelling and rendering package . Blender Foundation, Stichting Blender Foundation, Amsterdam, 2018.

Paul Debevec, Tim Hawkins, Chris Tchou, Haarm-Pieter Duiker, Westley Sarokin, and Mark Sagar. Acquiring the reflectance field of a human face. In Proceedings of the 27th annual conference on Computer graphics and interactive techniques , pages 145-156, 2000.

Paul E. Debevec. Rendering synthetic objects into real scenes: bridging traditional and imagebased graphics with global illumination and high dynamic range photography. Proceedings of the 25th annual conference on Computer graphics and interactive techniques , 1998.

Matt Deitke, Dustin Schwenk, Jordi Salvador, Luca Weihs, Oscar Michel, Eli VanderBilt, Ludwig Schmidt, Kiana Ehsani, Aniruddha Kembhavi, and Ali Farhadi. Objaverse: A universe of annotated 3d objects. arXiv preprint arXiv:2212.08051 , 2022.

David Griffiths, Tobias Ritschel, and Julien Philip. Outcast: Outdoor single-image relighting with cast shadows. Computer Graphics Forum , 41(2):179-193, 2022.

Kaiwen Guo, Peter Lincoln, Philip L. Davidson, Jay Busch, Xueming Yu, Matt Whalen, Geoff Harvey, Sergio Orts, Rohit Pandey, Jason Dourgarian, Danhang Tang, Anastasia Tkach, Adarsh Kowdle, Emily Cooper, Mingsong Dou, S. Fanello, Graham Fyffe, Christoph Rhemann, Jonathan Taylor, Paul E. Debevec, and Shahram Izadi. The relightables. ACM Transactions on Graphics (TOG) , 38:1 - 19, 2019.

Yuxuan Han, Zhibo Wang, and Feng Xu. Learning a 3d morphable face reflectance model from low-cost data, 2023.

Samuel W. Hasinoff, Dillon Sharlet, Ryan Geiss, Andrew Adams, Jonathan T. Barron, Florian Kainz, Jiawen Chen, and Marc Levoy. Burst photography for high dynamic range and low-light imaging on mobile cameras. ACM Trans. Graph. , 35(6), dec 2016.

Jon Hasselgren, Nikolai Hofmann, and Jacob Munkberg. Shape, Light, and Material Decomposition from Images using Monte Carlo Rendering and Denoising. arXiv:2206.03380 , 2022.

Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Neural Information Processing Systems , 33:6840-6851, 2020.

Hanzhe Hu, Zhizhuo Zhou, Varun Jampani, and Shubham Tulsiani. Mvd-fusion: Single-view 3d via depth-consistent multi-view generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 9698-9707, 2024.

Yi Huang, Jiancheng Huang, Yifan Liu, Mingfu Yan, Jiaxi Lv, Jianzhuang Liu, Wei Xiong, He Zhang, Shifeng Chen, and Liangliang Cao. Diffusion model-based image editing: A survey. arXiv preprint arXiv:2402.17525 , 2024.

Chaonan Ji, Tao Yu, Kaiwen Guo, Jingxin Liu, and Yebin Liu. Geometry-aware single-image full-body human relighting. In ECCV , page 388-405, 2022.

Haian Jin, Isabella Liu, Peijia Xu, Xiaoshuai Zhang, Songfang Han, Sai Bi, Xiaowei Zhou, Zexiang Xu, and Hao Su. Tensoir: Tensorial inverse rendering. In Proc. Computer Vision and Pattern Recognition (CVPR) , pages 165-174, 2023.

Yoshihiro Kanamori and Yuki Endo. Relighting humans: occlusion-aware inverse rendering for full-body human images. ACM Trans. Graph. , 37(6), 2018.

Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, and George Drettakis. 3d gaussian splatting for real-time radiance field rendering. ACM Transactions on Graphics , 42(4):1-14, 2023.

Diederik P Kingma and Max Welling. Auto-encoding variational bayes, 2022.

Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg, Wan-Yen Lo, Piotr Dollár, and Ross Girshick. Segment anything, 2023.

Zhengfei Kuang, Kyle Olszewski, Menglei Chai, Zeng Huang, Panos Achlioptas, and Sergey Tulyakov. Neroic: Neural rendering of objects from online image collections. ACMTransactions on Graphics (TOG) , 41(4):1-12, 2022.

Zhengfei Kuang, Yunzhi Zhang, Hong-Xing Yu, Samir Agarwala, Shangzhe Wu, and Jiajun Wu. Stanford-orb: A real-world 3d object inverse rendering benchmark, 2024.

Manuel Lagunas, Xin Sun, Jimei Yang, Ruben Villegas, Jianming Zhang, Zhixin Shu, Belen Masia, and Diego Gutierrez. Single-image full-body human relighting. Eurographics Symposium on Rendering - DL-only Track , 2021.

Zhengqin Li, Jia Shi, Sai Bi, Rui Zhu, Kalyan Sunkavalli, Milovs Havsan, Zexiang Xu, Ravi Ramamoorthi, and Manmohan Chandraker. Physically-based editing of indoor scene lighting from a single image. ArXiv , abs/2205.09343, 2022.

Zhengqin Li, Zexiang Xu, Ravi Ramamoorthi, Kalyan Sunkavalli, and Manmohan Chandraker. Learning to reconstruct shape and spatially-varying reflectance from a single image. ACM Transactions on Graphics (TOG) , 37:1 - 11, 2018.

Andrew Liu, Shiry Ginosar, Tinghui Zhou, Alexei A. Efros, and Noah Snavely. Learning to factorize and relight a city, 2020.

Isabella Liu, Linghao Chen, Ziyang Fu, Liwen Wu, Haian Jin, Zhong Li, Chin Ming Ryan Wong, Yi Xu, Ravi Ramamoorthi, Zexiang Xu, and Hao Su. Openillumination: A multi-illumination dataset for inverse rendering evaluation on real objects, 2024.

Ruoshi Liu, Rundi Wu, Basile Van Hoorick, Pavel Tokmakov, Sergey Zakharov, and Carl Vondrick. Zero-1-to-3: Zero-shot one image to 3d object, 2023.

Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization, 2019.

Abhimitra Meka, Christian Haene, Rohit Pandey, Michael Zollhöfer, Sean Fanello, Graham Fyffe, Adarsh Kowdle, Xueming Yu, Jay Busch, Jason Dourgarian, et al. Deep reflectance fields: high-quality facial reflectance field inference from color gradient illumination. ACM Transactions on Graphics (TOG) , 38(4):1-12, 2019.

Chenlin Meng, Yutong He, Yang Song, Jiaming Song, Jiajun Wu, Jun-Yan Zhu, and Stefano Ermon. Sdedit: Guided image synthesis and editing with stochastic differential equations. In International Conference on Learning Representations , 2021.

Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. In ECCV , 2020.

Giljoo Nam, Joo Ho Lee, Diego Gutierrez, and Min H Kim. Practical svbrdf acquisition of 3d objects with unstructured flash photography. ACMTransactions on Graphics (TOG) , 37(6):1-12, 2018.

Thomas Nestmeyer, Jean-François Lalonde, Iain Matthews, and Andreas Lehrmann. Learning physics-guided face relighting under directional light. In Proc. Computer Vision and Pattern Recognition (CVPR) , pages 5124-5133, 2020.

Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In International conference on machine learning , pages 8162-8171. PMLR, 2021.

Li Niu, Wenyan Cong, Liu Liu, Yan Hong, Bo Zhang, Jing Liang, and Liqing Zhang. Making images real again: A comprehensive survey on deep image composition. arXiv preprint arXiv:2106.14490 , 2021.

Rohit Pandey, Sergio Orts Escolano, Chloe Legendre, Christian Haene, Sofien Bouaziz, Christoph Rhemann, Paul Debevec, and Sean Fanello. Total relighting: learning to relight portraits for background replacement. ACM Trans. Graph. , 40(4), 2021.

Gaurav Parmar, Krishna Kumar Singh, Richard Zhang, Yijun Li, Jingwan Lu, and Jun-Yan Zhu. Zero-shot image-to-image translation. In ACM SIGGRAPH 2023 Conference Proceedings , pages 1-11, 2023.

Pieter Peers, Naoki Tamura, Wojciech Matusik, and Paul Debevec. Post-production facial performance relighting using reflectance transfer. ACM Trans. Graph. , 26(3), 2007.

Pakkapon Phongthawee, Worameth Chinchuthakun, Nontaphat Sinsunthithet, Amit Raj, Varun Jampani, Pramook Khungurn, and Supasorn Suwajanakorn. Diffusionlight: Light probes for free by painting a chrome ball. In ArXiv , 2023.

Ben Poole, Ajay Jain, Jonathan T. Barron, and Ben Mildenhall. Dreamfusion: Text-to-3d using 2d diffusion. arXiv , 2022.

Anurag Ranjan, Kwang Moo Yi, Jen-Hao Rick Chang, and Oncel Tuzel. Facelit: Neural 3d relightable faces. In Proc. Computer Vision and Pattern Recognition (CVPR) , pages 8619-8628, 2023.

Erik Reinhard, Greg Ward, Sumanta Pattanaik, and Paul Debevec. High Dynamic Range Imaging: Acquisition, Display, and Image-Based Lighting (The Morgan Kaufmann Series in Computer Graphics) . Morgan Kaufmann Publishers Inc., San Francisco, CA, USA, 2005.

Peiran Ren, Yue Dong, Stephen Lin, Xin Tong, and Baining Guo. Image based relighting using neural networks. ACM Transactions on Graphics (ToG) , 34(4):1-12, 2015.

Christian Richardt, Carsten Stoll, Neil A. Dodgson, Hans-Peter Seidel, and Christian Theobalt. Coherent spatiotemporal filtering, upsampling and rendering of RGBZ videos. Computer Graphics Forum (Proceedings of Eurographics) , 31(2), May 2012.

Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. Highresolution image synthesis with latent diffusion models. In Proc. Computer Vision and Pattern Recognition (CVPR) , pages 10684-10695, 2022.

Shen Sang and M. Chandraker. Single-shot neural relighting and svbrdf estimation. In ECCV , 2020.

Jian Shi, Yue Dong, Hao Su, and Stella X. Yu. Learning non-lambertian object intrinsics across shapenet categories, 2016.

Ruoxi Shi, Hansheng Chen, Zhuoyang Zhang, Minghua Liu, Chao Xu, Xinyue Wei, Linghao Chen, Chong Zeng, and Hao Su. Zero123++: a single image to consistent multi-view diffusion base model, 2023.

Zhixin Shu, Ersin Yumer, Sunil Hadap, Kalyan Sunkavalli, Eli Shechtman, and Dimitris Samaras. Neural face editing with intrinsic image disentangling. In Proc. Computer Vision and Pattern Recognition (CVPR) , pages 5541-5550, 2017.

Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models, 2022.

Pratul P Srinivasan, Boyang Deng, Xiuming Zhang, Matthew Tancik, Ben Mildenhall, and Jonathan T Barron. Nerv: Neural reflectance and visibility fields for relighting and view synthesis. In Proc. Computer Vision and Pattern Recognition (CVPR) , pages 7495-7504, 2021.

Tiancheng Sun, Jonathan T Barron, Yun-Ta Tsai, Zexiang Xu, Xueming Yu, Graham Fyffe, Christoph Rhemann, Jay Busch, Paul Debevec, and Ravi Ramamoorthi. Single image portrait relighting. ACM Trans. Graph. , 38(4), 2019.

Jiaxiang Tang, Jiawei Ren, Hang Zhou, Ziwei Liu, and Gang Zeng. Dreamgaussian: Generative gaussian splatting for efficient 3d content creation. arXiv preprint arXiv:2309.16653 , 2023.

Murat Türe, Mustafa Ege Çıklabakkal, Aykut Erdem, Erkut Erdem, Pinar Satılmı¸ s, and Ahmet Oguz Akyüz. From noon to sunset: Interactive rendering, relighting, and recolouring of landscape photographs by modifying solar position. In Computer Graphics Forum , pages 500-515. Wiley Online Library, 2021.

Benjamin Ummenhofer, Sanskar Agrawal, Rene Sepulveda, Yixing Lao, Kai Zhang, Tianhang Cheng, Stephan Richter, Shenlong Wang, and German Ros. Objects with lighting: A real-world dataset for evaluating reconstruction and rendering for object relighting, 2024.

Yang Wang, Lei Zhang, Zicheng Liu, Gang Hua, Zhen Wen, Zhengyou Zhang, and Dimitris Samaras. Face relighting from a single image under arbitrary unknown lighting conditions. Trans. Pattern Analysis and Machine Intelligence , 31(11):1968-1984, 2008.

Andreas Wenger, Andrew Gardner, Chris Tchou, Jonas Unger, Tim Hawkins, and Paul Debevec. Performance relighting and reflectance transformation with time-multiplexed illumination. ACM Transactions on Graphics (TOG) , 24(3):756-764, 2005.

Jung-Hsuan Wu and Suguru Saito. Interactive relighting in single low-dynamic range images. ACM Trans. Graph. , 36(2), 2017.

Rundi Wu, Ben Mildenhall, Philipp Henzler, Keunhong Park, Ruiqi Gao, Daniel Watson, Pratul P. Srinivasan, Dor Verbin, Jonathan T. Barron, Ben Poole, and Aleksander Holynski. Reconfusion: 3d reconstruction with diffusion priors. arXiv , 2023.

Rui Xia, Yue Dong, Pieter Peers, and Xin Tong. Recovering shape and spatially-varying surface reflectance under unknown illumination. ACM Transactions on Graphics (TOG) , 35(6):1-12, 2016.

Zexiang Xu, Kalyan Sunkavalli, Sunil Hadap, and Ravi Ramamoorthi. Deep image-based relighting from optimal sparse samples. ACM Transactions on Graphics (ToG) , 37(4):1-13, 2018.

Yao Yao, Jingyang Zhang, Jingbo Liu, Yihang Qu, Tian Fang, David McKinnon, Yanghai Tsin, and Long Quan. Neilf: Neural incident light field for physically-based material estimation. In European Conference on Computer Vision , pages 700-716. Springer, 2022.

Ye Yu, Abhimitra Meka, Mohamed Elgharib, Hans-Peter Seidel, Christian Theobalt, and William AP Smith. Self-supervised outdoor scene relighting. In ECCV , pages 84-101, 2020.

Chong Zeng, Yue Dong, Pieter Peers, Youkang Kong, Hongzhi Wu, and Xin Tong. Dilightnet: Fine-grained lighting control for diffusion-based image generation, 2024.

Bo Zhang, Yuxuan Duan, Jun Lan, Yan Hong, Huijia Zhu, Weiqiang Wang, and Li Niu. Controlcom: Controllable image composition using diffusion model. arXiv preprint arXiv:2308.10040 , 2023.

Jingyang Zhang, Yao Yao, Shiwei Li, Jingbo Liu, Tian Fang, David McKinnon, Yanghai Tsin, and Long Quan. Neilf++: Inter-reflectable light fields for geometry and material estimation. In Proc. Int. Conf. on Computer Vision (ICCV) , pages 3601-3610, 2023.

Kai Zhang, Fujun Luan, Zhengqi Li, and Noah Snavely. Iron: Inverse rendering by optimizing neural sdfs and materials from photometric images. In Proc. Computer Vision and Pattern Recognition (CVPR) , pages 5565-5574, 2022.

Kai Zhang, Fujun Luan, Qianqian Wang, Kavita Bala, and Noah Snavely. Physg: Inverse rendering with spherical gaussians for physics-based material editing and relighting. In Proc. Computer Vision and Pattern Recognition (CVPR) , pages 5453-5462, 2021.

Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models, 2023.

Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Ic-light github page, 2024.

Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In Proc. Computer Vision and Pattern Recognition (CVPR) , 2018.

Xiuming Zhang, Pratul P Srinivasan, Boyang Deng, Paul Debevec, William T Freeman, and Jonathan T Barron. Nerfactor: Neural factorization of shape and reflectance under an unknown illumination. ACM Transactions on Graphics (ToG) , 40(6):1-18, 2021.

Yuanqing Zhang, Jiaming Sun, Xingyi He, Huan Fu, Rongfei Jia, and Xiaowei Zhou. Modeling indirect illumination for inverse rendering. In Proc. Computer Vision and Pattern Recognition (CVPR) , 2022.

Hao Zhou, Sunil Hadap, Kalyan Sunkavalli, and David W Jacobs. Deep single-image portrait relighting. In Proceedings of the IEEE/CVF international conference on computer vision , pages 7194-7202, 2019.

A Comparison with IC-Light

In this section, we conduct a comparison with a recent single-image relighting method, IC-Light [85]. In Fig. 10, we present a comparative analysis of relighting results, focusing on the consistency of specular highlights in comparison to IC-Light. Our method consistently adjusts the specular highlights in accordance with the rotation of the lighting, ensuring a coherent relighting effect. In contrast, IC-Light demonstrates inconsistent movement of highlights, with certain regions showing minimal changes. For additional video comparisons, please visit our supplemental webpage.

Target

Lighting

IC-Light

Ours

B Portrait Relighting Results

Our model is fine-tuned on object data, therefore, while it is category-agnostic, it might not achieve the highest quality results on specific domains-for instance, for the case of portrait relighting, methods that are specifically trained for portrait photos may produce higher quality results. We show our results on the portrait in Fig. 11 of the appendix.

C Inherent color ambiguity

GT relit Image

Target Lighting

GT relit Image

Pred Image 2

Pred Image 1

Target Lighting

Pred Image 2

Pred Image 1

Color ambiguity is an inherent issue in the single-image relighting task. That said, we found that our data-driven model can handle this problem to some extent, especially when the objects' materials in the input image are relatively specular. Because the diffusion model can infer the lighting of the input image through the reflections or specular highlights. In Fig. 12, we show two such examples: we render each object with two different environment maps and relight them with the same environment map. In both cases, the input images have different colors and we want to relight them under the same target lighting. It turned out that both input images can be relit to the results that match the ground truth results, which indicates our model has some ability to handle the color ambiguity.

As for the general diffuse object, we think our model could learn an implicit prior over common colors that show up in object albedos, as well as common colors that occur in lighting, and use that to help resolve this ambiguity. However, this may fail sometimes, as we discussed in Sec.5. One potential solution could be also inputting the background of the input image to the diffusion model since the diffusion model can learn to infer the lighting of the input image from the background. We leave this as future work.

D More results of relighting 3D objects

Input Images

TensoIR

NVDIFFREC-MC

Ours

GT Relighting

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The main focus of this paper is to train a lighting-aware image-conditioned diffusion model; it aligns with our title.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: Please see the Limitation Section.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: [NA]

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We have included enough details to reproduce our results.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: We will release all code and models once upon acceptance.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: we have described in details our experimental setups in the experiments section. Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [No]

Justification: As each experiment takes significant amount of compute resource, we don't have enough computing resources to complete the significance test.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: Yes, provide both data creation and model training.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: The paper is written on our own and follows the honest codes.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [No]

Justification: Please refer to the main paper.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: [NA]

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We have cited all previous papers that lead to our current paper's presentation.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: No new assets were introduced.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: [NA]

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: [NA]

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a method for relighting objects observed from a single image. While existing approaches rely on specific capture condition using flashlight illumination or portrait captures, or require to explicitly decompose the scene into geometry and reflectance, the proposed method aims to generate images of a given objects under novel illumination conditions for arbitrary environmental lighting conditions. The authors show that this is possible by relying on a generative diffusion method that is conditioned on the environmental map. The method relies on a pre-trained diffusion model that is fine-tuned on a synthetic relighting dataset to learn the conditioning. The approach is evaluated qualitatively and quantitatively on single-object images. Relying on a conditional diffusion model for relighting, the authors also show additional conditioning on text for relighting.\n\nSTRENGTHS:\nThis work presents a simple (this is a good thing) and effective method for relighting from a single image. The method relies on synthetic supervision with a novel Blender-rendered dataset that uses Objeverse as input model source. The authors went a long way by collecting diverse HDR environment maps from the Internet that were augmented to produce a large synthetic relighting dataset of almost 20M rendered images with ground truth lighting maps. Overall, the method offers a number of intriguing benefits listed as follows:\n\n* Conditional image-to-image diffusion model: The method inherits a conditional Zero-1-to-3 model that is extended in its input latents to a rotated environment map with the camera coordinate frame, allowing for image-to-image relighting in a consistent frame. While, given enough training data, the method is effective in relighting, the approach also enjoys the benefits of existing diffusion architectures with various types of conditioning. The authors demonstrate this effectively with their image conditioning. \n\n* Relighting 3D radiance fields: The proposed method is evaluated as a prior for 3D relighting of a neural radiance field. Specifically, the authors propose to use diffusion-based relighting as a coarse reconstruction loss (predicting a coarse relit scene during the NeRF optimization) and a detail refinement loss where the NeRF appearance is further refined.\n\n* Qualitative evaluation: The evaluations presented qualitatively in the main manuscript and the supplemental material in the form of supplemental videos are visually plausible and convincing. \n\n* Quantitative evaluations: The method is adequately ablated and quantitatively compared to single image relighting methods, 3D radiance field relighting with reasonable margins on the test sets. This validates the method as an effective approach.\n\nWEAKNESSES:\nWhat makes the method exciting, at first glance, is also one of the major weaknesses: the technical novelty. The paper piggy-backs on an existing generative method, the Zero-1-to-3 model, that is with a few variations used for relighting. While the simplicity is something that is desired, it also makes it challenging for the reader to derive deeper insights from this work. We learn that pre-trained diffusion-models, when just given enough and the right synthetic data, can allow for plausible novel view synthesis with artefacts that are improved over existing methods. However, the recent work by \n\nChong Zeng, Yue Dong, Pieter Peers, Youkang Kong, Hongzhi Wu, and Xin Tong. Dilightnet: Fine-grained lighting control for diffusion-based image generation, 2024.\n\nin a way also does show the exactly same, although the technical approach is different. Overall, the technical contribution of the approach is rather incremental (although the method is effective). As such, I am torn on this work. While the technical contribution is not near other work at NeurIPS, the method is effective and likely of high impact. \n\nA further qualm I have is regarding the results compared to NVDIFFREC. While the margins are not substantially different, the results in Fig. 6 seem to indicate differently. It seems as if these results are cherry-picked.\n\nQUESTIONS:\nSee questions regarding trends in the quantitative evaluations and the qualitative results that do not seem to match.\n\nLIMITATIONS:\nAll major limitations are addressed. The only open limitation not addressed in the manuscript is the runtime. The authors should address and comment on the runtime for their diffusion model.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces Neural Gaffer, an end-to-end 2D relighting diffusion model designed for single-image relighting without the need for explicit scene decomposition. Neural Gaffer can synthesize high-quality relit images of any object under novel environmental lighting conditions by conditioning on a target environment map. The model builds on a pre-trained diffusion model, fine-tuning it on a synthetic relighting dataset. The advantages in generalization and accuracy through evaluations on both synthetic and in-the-wild Internet imagery are shown in the paper. Neural Gaffer can be combined with other generative methods for various downstream 2D tasks like objection insertion. The video results presented in the paper are of high quality.\n\nSTRENGTHS:\n1)  Neural Gaffer performs single-image relighting without the need for explicit scene decomposition into intrinsic components like normals and BRDFs. This provides an avenue for relighting without collecting expensive relighting real-world datasets.\n\n2)  The model can generate relit images of various objects under different environmental lighting conditions based on a target environment map. The method takes a single image as an input.\n\n3) The method can be applied to real-world objects with high-quality relighting results and perform various downstream tasks such as object insertion.\n\nWEAKNESSES:\n1) In case of the real-world object scenarios, the object may not be always centred and may have complex backgrounds and lighting to start with. The paper does not demonstrate how would the method behave in such cases. How about the objects with high-frequency texture details?\n\n2) Related to 1) there might be multiple objects in a scene. From the results, it seems that the method cannot handle multiple objects from a single image.\n\n3)  The real-world object examples shown in the paper and the video are good but not impressive. It would be more compelling to show faces, humans, animals etc under the lighting conditions to show the generalizability of the method.\n\nQUESTIONS:\nThe paper does not demonstrate how the method behaves when the target object is not centred and has complex backgrounds or varied lighting conditions. How does the method perform in such scenarios, especially with objects that have high-frequency texture details?\n\nIt appears that the method may struggle with scenes containing multiple objects. Can the authors provide further evaluation or examples to show how the method handles multiple objects in a single image?\n\nWhile the real-world object examples are good, they are not particularly impressive. Can the authors provide more compelling examples involving faces, humans, or animals under varied lighting conditions to better demonstrate the generalizability of the method? While it's understood that portrait lighting might not be comparable to those methods specifically trained on portraits, it would be good to see the generalizability of the method.\n\nLIMITATIONS:\nThe authors have discussed the limitations.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-11"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nNeural Gaffer presents an approach to object-centric image relighting using diffusion models. The method adapts a pre-trained diffusion model and fine-tunes it on a synthetic dataset designed for relighting tasks. The main feature is its ability to condition the diffusion process on target environment maps, allowing for control over lighting effects.\n\nSTRENGTHS:\n1) Simple yet effective approach: The paper presents a straightforward fine-tuning method for object relighting, similar to zero-1-2-3 shot learning. This simplicity is a strength, demonstrating that complex relighting can be achieved without overly complicated techniques.\n\n2) Powerful data-driven learning: The supervised conditional diffusion model effectively learns to relight objects, highlighting the potential of data-driven approaches in capturing intricate lighting interactions.\n\n3) Competitive results: Based on the presented figures, the method appears to outperform the recent DiLightNet in some aspects. However, this comparison raises some evaluation questions (see questions section for details).\n\nWEAKNESSES:\n1) Real-world evaluation: The model is fine-tuned on a synthetic relighting dataset, which might not fully capture the complexity of real-world lighting scenarios. Real-world evaluation is necessary, and there are datasets capturing these effects. The paper is currently missing this evaluation, and there are datasets available for such evaluation [1] OpenIllumination [2] Objects with Lighting or [3] Stanford ORB dataset. These papers have been cited but it is surprising to not see an evaluation of these datasets.\n\n2) Reliance of Environment map: Do you need to supply the environment map for relighting? There is a missing baseline that shows what happens if you condition the target lighting image without a full environment map (only image crops). The Diffusion Light Probe (CVPR 2024) paper indicates that diffusion models are capable of inpainting reliable environment maps and they seem to be implicitly encoded within the model. This baseline will justify why a full environment map is required or necessary for this task.\n\n3) Generalization to scenes: The extent to which the method generalizes to scenes -- not just objects -- is unclear. Evaluating the MIT-multi illumination dataset could shed light on this. The current reliance on explicit environment maps makes it harder to perform on these scenes, but it would be interesting to see if, without explicit environment maps (like suggested above), can you learn to relight and compare on scenes.\n\n4) Evaluation metrics: Recent studies show that PSNR, SSIM, etc. are not consistent with human evaluation. See \"Towards a Perceptual Evaluation Framework for Lighting Estimation\" (CVPR 2024). These metrics don't tell us much about whether the method is promising as such. A thorough evaluation via user studies or the metrics as defined in the recent paper is currently missing from the paper.\n\n5) Unrealistic results and missing comparisons: The object insertion results look unrealistic, with incorrect shadows that don't match the lighting conditions. Several relevant lighting-aware compositing methods are missing from the comparisons, such as ControlCom [Zhang et al., arXiv 2023], Intrinsic Harmonization [Carega et al., SIGGRAPH 2023], Reshading [Bhattad and Forsyth, 3DV 2022], and ARShadowGAN [CVPR 2020]. The comparison to AnyDoor doesn't make sense as it's not lighting-aware. Including these comparisons would provide a better evaluation of the method's performance against current state-of-the-art techniques.\n\n6) Further, as the papers use off-the-shelf methods to estimate environmental maps (text2light), why not compare with existing 3D object compositing with lighting estimation methods to get a sense of how the proposed methods compare to these tasks -- see Garon et al (CVPR 2019), StyleLight (Wang et al; ECCV 2022) and similar papers? Rendering objaverse objects using lighting estimated from the mentioned or similar methods would help understand the gaps between explicit environment map prediction methods.\n\n7) 3D relighting evaluation: For the 3D relighting setting, according to the Objects with Lighting 3DV 2024 paper, Mitsuba + NeuS is a stronger baseline compared to TensorIR, which is currently missing in the paper.\n\n8) Failure analysis: The paper mentions in the limitations section that the approach might not work for portrait relighting, but it would be interesting to see the kind of failures the diffusion model makes. The current setup lacks experiments in this direction to see what are these failures to encourage future research. Further, the current papers also do not provide any failure examples from Objaverse instances. Is the method perfect on all unseen objects -- detailed analysis is missing as to what objects the proposed methods perform best or worse on. Such analysis helps scope out limitations of the current methods instead of shallow limitations provided in Appendix D.\n\n9) Lack of comparison with simple color matching baselines: The paper doesn't include a comparison with straightforward color adjustment techniques, such as RGB histogram matching between the inserted object and the target scene. This omission raises questions about how much of the method's perceived success in relighting is due to sophisticated light interaction modeling versus simple color transformations. A comparison with such a baseline would help quantify the added value of the diffusion model approach over a simpler method.\n\nQUESTIONS:\n1) Why weren't datasets like OpenIllumination, Objects with Lighting, or Stanford ORB used for evaluation?\n\n2) Have you explored the necessity of full environment maps for relighting?\n\n3) How well does your method generalize to full scenes, beyond individual objects?\n\n4) Given recent findings on the inconsistency of PSNR and SSIM with human perception for lighting tasks, have you considered user study?\n\n5) Why were comparisons with recent lighting-aware object compositing methods (e.g., ControlCom, Intrinsic Harmonization) not included?\n\n6) Have you considered comparing your method with existing 3D object compositing and lighting estimation approaches?\n\n7) Why wasn't Mitsuba + NeuS used as a baseline for 3D relighting, given its reported strength in recent literature?\n\n8) Can you provide a more detailed analysis of failure cases, including examples from Objaverse instances?\n\n9) Can you provide a simple color histogram matching baseline? \n\n10) Comparison with DiLightNet: DiLightNet offers full image generation with background handling, while Neural Gaffer focuses on object relighting. This raises several points: \n\n- Background consistency: How does Neural Gaffer address the background when relighting objects?\n- Evaluation scope: Are quantitative evaluations done on the full scene or just the object region? This impacts the interpretation of results.\n- Lighting control: DiLightNet allows full-scene control. How comprehensive is Neural Gaffer's approach in comparison?\n- User input method: DiLightNet uses radiance hints, Neural Gaffer uses environment maps. How do these compare in terms of user-friendliness and control/precision?\n- Shadow and Indirect lighting effects quality: DiLightNet's shadows and indirect effects appear more convincing from their project page. Can you comment on this difference? Can you provide a user study comparing the perceived lighting quality between Neural Gaffer and DiLightNet? \n\n11) How sensitive is your method to the resolution of input environment maps?\n\nLIMITATIONS:\nSomewhat but not fully. See my weakness 8.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-13"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a novel method for single-image relighting, which takes an image of an object and a target environmental map as inputs. The authors fine-tune Stable Diffusion on a synthetic relighting dataset to output relit images, conditioning on both the input object image and the target environmental map. The authors show their method outperforms existing baselines. Additionally, the trained relighting model can be applied to downstream tasks such as relighting a neural radiance field and object insertion.\n\nSTRENGTHS:\n- I check the video results in the supplementary video. The visual results are impressive.\n- The authors have shown several downstream applications using their trained relighting model, including text-based relighting and object insertion.\n- The authors have conducted extensive ablation studies to prove the effectiveness of their proposed method.\n\nWEAKNESSES:\nI don’t have many complaints about the paper. I list several potential improvements below:\n\n- In the 3D relighting experiments, it seems unfair to compare with inverse rendering methods such as Nvdiffrec-mc and TensoIR, as they can apply any lighting to the object once the material is recovered, while Neural Gaffer needs optimize for every lighting. On the other hand, I think Neural Gaffer should be combined with these inverse rendering methods and provide priors when recovering material and lighting.\n- The extrinsic information is injected by rotating the environmental map. However, it seems intrinsic information is not considered, which means there is an assumed fixed FOV. This could introduce biases in downstream applications and limit the input views in 3D relighting.\n- The quantitive comparison with IC-Light is missing.\n- The generated image resolution is limited to 256x256.\n\nQUESTIONS:\nThe problem of inverse rendering with a single image is inherently ambiguous. For example, the object color in the input image could come from either the material or the lighting. I was wondering about the authors' thoughts on this problem in the context of Neural Gaffer. When relighting an object, there could be multiple possible outputs depending on the decomposition of the material and lighting in the input. Is the probabilistic model of Neural Gaffer able to model this perplexity?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-13"}