Scaling In-the-Wild Training for Diffusion-based Illumination Harmonization and Editing by Imposing Consistent Light Transport
Authors: Lvmin Zhang
Anyi Rao
Maneesh Agrawala
Venue: ICLR 2025 Oral (2025)
Keywords: diffusion model
illumination editing
image editing
Primary area: applications to computer vision, audio, language, and other modalities
TL;DR: Diffusion-based image illumination harmonization and editing model

Abstract: Diffusion-based image generators are becoming unique methods for illumination harmonization and editing. The current bottleneck in scaling up the training of diffusion-based illumination editing models is mainly in the difficulty of preserving the underlying image details and maintaining intrinsic properties, such as albedos, unchanged. Without appropriate constraints, directly training the latest large image models with complex, varied, or in-the-wild data is likely to produce a structure-guided random image generator, rather than achieving the intended goal of precise illumination manipulation. We propose Imposing Consistent Light (IC-Light) transport during training, rooted in the physical principle that the linear blending of an object's appearances under different illumination conditions is consistent with its appearance under mixed illumination. This consistency allows for stable and scalable illumination learning, uniform handling of various data sources, and facilitates a physically grounded model behavior that modifies only the illumination of images while keeping other intrinsic properties unchanged. Based on this method, we can scale up the training of diffusion-based illumination editing models to large data quantities (> 10 million), across all available data types (real light stages, rendered samples, in-the-wild synthetic augmentations, etc), and using strong backbones (SDXL, Flux, etc). We also demonstrate that this approach reduces uncertainties and mitigates artifacts such as mismatched materials or altered albedos.

Published as a conference paper at ICLR 2025
SCALING IN-THE-WILD TRAINING FOR DIFFUSION- BASED ILLUMINATION HARMONIZATION AND EDITING BY IMPOSING CONSISTENT LIGHT TRANSPORT
Lvmin Zhang', Anyi Rao", Maneesh Agrawala' Stanford University, "Hong Kong University of Science and Technology {1 Livinn, @eanessh@css stanford. edu, anyi raolust hk
ABSTRACT
Diffusion-basee image generators are becoming unique methods for illumination harmonization and editing. The current bottleneck in scaling up the training of diffusion-based illumination editing models is mainly in the difficulty of preserving the underlying image details and maintaining intrinsic properties, such as albedos, unchanged. Without appropriate constraints, directly training the latest large image models with complex, varied, or in-the-wild data is likely to produce a structure-guided random image generator, rather than achieving the intended goal of precise illumination manipulation. We propose Imposing Consistent Light (IC- Light) transport during training, rooted in the physical principle that the linear blending of an object's appearances under different illumination conditions is consistent with its appearance under mixed illumination. This consistency allows for stable and scalable illumination learning, uniform handling of various data sources, and facilitates physically grounded model behavior that modifies only the illumination of images while keeping other intrinsic properties unchanged. Based this method, we can scale up the training of diffusion-based illumination editing models to large data quantities (>>10 10 million), across all available data types (real light stages, rendered samples, in-the-wild synthetic augmentations, etc.), and using strong backbones (SDXL, Flux, etc.). We also demonstrate that this approach reduces uncertainties and mitigates artifacts such as mismatched materials or altered albedos.
INTRODUCTION
Editing the illumination in images is fundamental task in deep learning and image editing. Classic computer graphics methods often model the appearance images using physical illumination models. More recently, large diffusion- flexible paradigms in this area, handling a wider range of "in-the-wild" lighting effects beyond simply changing the distribution of light sources, e-g., generating backlighting rim light, adding special effects like glow, glare, or the Tyndall effect, simulating shadows cast through tree shade or venetian blinds, and even manipulating human-drawn, composed, artistic, non-photorealistic lighting conditions. These applications also provide tools for artists and designers to modify the foreground or background (e.g.. product images, commercial posters, etc.) while maintaining harmonious illumination. These illumination editing applications with generative image models hold unique industrial value for visual content creation and manipulation.
image generators have introduced unique applications and
Diffusion beasue illumination editing methods also present new opportunities and considerations for scaling up training and utilizing stronger backbones. Yet, training an illumination editing model at larger scales and with more diversity is more challenging than it seems. The first challenge lies in maintaining the desired model behavior to ensure proper illumination manipulation rather than deviating into unintended random behaviors. As the dataset size and diversity increase, the mapping and distribution of the learning objective can become ambiguous and uncertain. Without appropriate constraints, the training may produce structure- structure-guided random image generator, resulting in outputs that do not align with the desired illumination editing requirements. This deviation occurs when the model fails to learn a mapping corresponding illumination modification, instead introducing

---

Published as a conference paper at ICLR 2025
sunlight through the blinds, near window blinds"
sunlight from the left side, beach"
Input
input
Input
magic golden lit, forest"
input
neo punk, city night"
Figure 1: Text-conditioned illumination modifying with background generation (Flux. Iede.. We demonstrate the typical use case this approach: users give an object image and illumination description, and our method generates corresponding object appearances and backgrounds.
arbitrary changes to the images, driven by dataset local minima or pretrained model default behaviors without proper alignments.
The second challenge is to preserve the underlying image details and intrinsic properties, such as albedo or reflectance colors, when modifying the illumination. Due the stochastic nature of diffusion algorithms and the encoding-decoding processes of latent spaces, diffusion-based image generators inherently tend to introduce randomness into image contents, making it difficult to retain fine-grained details. Furthermore, effective illumination editing requires the model to have thorough understanding of the scene to correctly adjust elements like shadows, highlights, and specular reflections. For instance, if the original image contains hard shadows, the model must first remove these existing shadows before adding new light sources and appropriate shadows. Preserving image details and intrinsic properties thus requires not only content generation but also discriminative and decomposition capabilities from the model to analyze image constituents. This necessitates careful design of training objectives and constraints to guide the learning process effectively.
In this paper, we propose method to Impose Consistent Light (IC-Light) transport during training, grounded in the physical principle of light transport independence the linear blending of object's appearances under different illumination conditions is consistent with its appearance under mixed illumination. By enforeing this consistency, we introduce strong, physically rrooted constraint that ensures the model modifies only the illumination aspects of an image while preserving other intrinsic properties such as albedo and fine image details. This approach enables stable, scalable training over 10 million diverse samples, including real photos from light stages, rendered images, and te-lhe-wid images with synthetic illumination augmentations. Our method demonstrates improved precision in illumination editing, redueing uncertainties and mitigating artifacts, without altering the underlying appearance details.
This method allows achieve maximized setup: expanding the dataset to over 10 million images, adopting stronger backbones like SDXL and Flux, and utilizing all available types of data sources, including real photos captured from light stages, rendered images, and in-the-wil wild natural artistic
2

---

Published as a conference paper at ICLR 2025
images with synthetic illumination augmentations. We provide experimental evidence to validate that increasing the training scale and diversifying data sources enhance model robustness and performance in various illumination-related downstream tasks.
Ablation experiments demonstrate that applying the IC-Light method during training improves the accuracy of illumination editing in preserving intrinsic properties like albedo and image details. Furthermore, when compared to alternative models trained on smaller or more structured datasets, our approach generalizes wider variety of illumination distributions, egg,, rim lighting, backlighting, magic glowing, sunset halo, elc. We also showcase the method's ability to handle more in-the-wild illumination scenarios, including artistic and composed lighting effects. Additionally, we explore further applications, such as generating normal maps, and discuss the differences between this approach and typical mainstream geometry estimation models.
In summary, (1) we propose IC-Light, method for scaling up the training of diffusion- -aased illumination editing models by imposing consistent light transport, ensuring precise illumination modification while preserving intrinsic image details; (2) we provide pretrained illumination editing models to facilitate illumination editing applications in content creation and manipulation across diverse domains; (3) we present extensive experiments to validate the scalability and performance of this approach, showing its difference from alternative methods in handling diverse illumination conditions; (4) present additional applications, such normal map generation and artistic lighting manipulation, further showcasing the versatility and robustness of our method in real-world, in-the- wild scenarios.
RELATED WORK
Editing with Deep Neural Networks Learning-based methods have become important baselines in image relighting over the past decade. Sun et al. (2019) utilized deep neural networks to learn prior knowledge from light stage data. Nestmeyer et al. (2020) further enhanced the neural network's capability in relighting by modeling physical priors in the neural network training process. Pandey et al. (2021) made use of high dynamic range (HDR) lighting maps (Dehevec, 2008) to train relighting models by explicitly optimizing the Phong model prior. Various baselines have been proposed to improve the efficiency, performance, and rationale of illumination modeling (Zhou al., 2019; Sengupta et al., 2021; Hou et al., 2021; 2022; Wang al., 2023b; Zhou et al., 2023). Switchlight (Kim et al., 2024) is state-of- of-the-art relighting method trained with physically co-designed neural networks for foreground relighting. 3D facial modeling (Shu et al., 2017) or intrinsic images (Barron & Malik, 2014; Sengupta et al., 2018) have also been demonstrated to be effective in portrait relighting. Illumination stylization can also facilitate portrait relighting (Shih et al., 2014).
Image Illumination
Diffusion Models for Appearance and Illumination Manipulation Recently, due to the develop- ment of text-to-image diffusion models (Dhariwal & Nichol, 2021; Ho et al., 2020; Sohl-Dickstein et al., 2015; Song et al., 2021), a wide range of tasks in image processing have seen significant advancements (Ho & Salimans, 2021; Ho et al., 2022; Kawar et al., 2022a;b; Ramesh et al., 2022; Rombach et al., 2022: Saharia al., 2022: Zhang et al., 2023a; Wang et al., 2022). More specifically, image editing (Alaluf et al., 2023; Brack et al., 2023; Brooks et al., 2023; Cao et al., 2023; Fu et al., 2023; Han et al., 2024; Couairon et al., 2022; Hertz et al., 2022; Meng al., 2021; Mokady
et al., 2022; Song et al., 2023: 2024; Tumanyan et al., 2023; Miyake et al., 2023; Zhang et al.,
al., 2023; Wallace al., 2022) and paired image-to-image translation (Kwon & Ye, 2022; Nie et al., 2023; Sasaki et al., 2021: Wang al., 2022; Zhang et al.. 2023a; Zhao al., 2022) have shown that fine-tuning pretrained diffusion models is an effective approach for manipulating the appearance of images. Text-to-image models have also been proven effective in depth estimation (Ke al., 2024), normal estimation (Fu et al., 2024), and 3D construction (Anciukevičius et al., 2023; Chen et al., 2023a;b; Li et al., 2024; Lin et al., 2023; Liu et al., 2023a; 2024b; 2023b; Tang et al., 2023; Metzer et al., 2022: Wang et al., 2023c; Poole et al., 2022; Wang et al., 2023a; Xu et al., 2023). Relightful Harmonization (Ren et al., 2024) is a state-of-te thrt approach for manipulating the illumination of image foregrounds using background conditions. More recently, DilightNet (Zeng al., 2024), FlashTex (Deng et al., 2024), Lightlt (Kocsis et al., 2024), and NeuralGaffer (Jin et al., 2024) have been proposed to manipulate object appearances mainly based on 3D rendering, NeRF representations, and synthetic data.
2023b; Wu & De la Torre, 2023; Huberman--piegelllaa
3

---

Published as a conference paper at ICLR 2025
Light Stage Methods A light stage (Debevec et al., 2000) is facility used to capture the appearance of real-world objects under different illumination conditions. Major recent progress has been made towards efficiency in light patterns and neural representations of different illuminations (Fyffe & Debevec, 2015; Ghosh et al., 201 Meka et al., 2019). Human portrait processing has been considered an important direction in light stage research (Hou al., 2021; 2022; Nestmeyer et al., 2020; Pandey et al., 2021; Sun et al., 2019; 2020; Yeh et al., 2022: Zhang et al., 2021; 2020b). Wang et al. (2023b) pointed out that the sun can also be used as special type of light stage. Calian et al. (2018) proposed modeling "light probes" for facial relighting. Sengupta et al. (2021) discussed that some world light sources can also be viewed as light stage, like different images on desk monitor devices. Sevastopolsky et al. (2020) is another attempt to build an easier facility for more effective capturing of the light stage using a smartphone camera. Various illumination models (Debevec et al., 2000; Dorsey et al., 1995; Wenger et al., 2005) also depend on light stage measurements.
Intrinsic Images Intrinsic imaging and decomposition is long standing problem in image pro- cessing. Early approaches are often closely connected to image smoothing research like, e.g., L0 (Xu al., 2011), LI (Bi et al., 2015), RTV (Xu et al., 2012), WLS (Farbman et al., 2008). EAP (Zhang et al., 2020a), etc. With datasets like Intrinsic Images in the Wild (IIW (Bell et al., 2014)) and MPI-Sintel (Butler et al., 2012). learning-
approaches are adopted. Narihira al. (2015) trains a linear classifier contextual cues from local image patches. Zhou et al. (2015) employs multi-stream network using both local and global inputs. Zoran et al. (2015) leverages local and global contextual information for pairwise classification. Nestmeyer & Gehler (2017) predicts a dense reflectance layer via a CNN. Narihira& & Yu (2015) and Eigen & Fergus (2015) train CNNs to directly predict albedo and shading. Kim et al. (2016) uses a joint model to learn depth maps and intrinsic images with coupled activations. Shi et al. (2017) trains an encoder-decoder CNN to predict intrinsic layers using synthetic images. Barron & Malik (2012; 2020) presents computational approach for estimating environment light from shapes. Fan et al. (2018) presents a learning- method that can make use of both image-space supervision and sparse supervision. Koesis al. (2023) propose to formulate the task of single view appearance decomposition as probabilistic problem, and uses diffusion model to give multiple possible decomposition. IntrinsicAnything (Xi et al., 2024) is a more recent work to estimate intrinsic images using pretrained diffusion models. Dnfrinsie (Luo et al., 2024) fine-tune diffusion models to jointly predict multiple intrinsic modalities (albedo, illumination, and surface geometry) from input images.
intrinsic decomposition
3 METHOD
3.1 IN-THE-WILD DATA DISTRIBUTION OF ILLUMINATION
As shown in Fig. 2, we model the distribution of illumination effects with multiple available types of data sources: arbitrary images, 3d data, and light stage images. These distributions allow capture of diverse and complex real-world lighting scenarios, e.g., back light, rim light, glow, etc. For simplicity, we process all data to a common format. Every appearance image I E Rhxxwx3 is paired with a 32px environment map L R32x32x3 a foreground mask M E kgx an optional background image B E 1g and an optional degradation image Ia E gxxux3
In-the-wild image augmentation We use data augmentation to convert an arbitrary image into paired illumination training data of images with same intrinsics (e.g., albedo) but different illumination appearances. Each sample includes one appearance for input condition, another appearance for output objective, and other meta data like environment maps. The images for output objective are high-quality in-the-wild images, whereas the images for input conditions contain randomized augmentations and degradations to enhance the robustness and generalization of the diffusion model.
Specifically, we first extract environment maps for an arbitrary image by randomly choosing between two methods: We either use the method of Phongthawee et al. (2023) or a custom environment-froo from-- normal method detailed in the supplementary material. We detect foreground mask with Zheng et al. (2024) and generate background images with a distill-accelerated (Luo et al., 2023) Stable Diffusion inpainting model. We detect prompts using the method of Xiao et al. (2023) or by using existing image prompts if the image is from text-image datasets. We then generate a "degradation appearance" that shares the same intrinsic albedo as the original image, but has completely altered illuminations;
4

---

Published as a conference paper at ICLR 2025
In- the wild image augmentation
3D data (objaverse renderer)
Light stage (one-light- time data)
arbitrary image
random shadow
random degradation
appearance
albedo
normal
light
light 2
light N
Common data format for leaming light transport consistency
young woman stands urben environment, with warm sunlight casting dramatic shadows across her face. The light source namow gop, creating sharp contrasts between shadows and highlights...
from
appearance
environment
prompt
mask M
background (oppionaal)
degradation Infoptional)
I
L
Figure 2: Dataset collection. We visualize various sources of the dataset and the components used during the training. Data from multiple sources are unified into common format for neural network training. * Only "in-the-wild data augmentation" part has degradation images.
specifically we extract image albedo by randomly applying 6 albedo extraction methods in the supplementary materials. Then, we synthesize soft shading images using random normal estimation methods, and synthesiz hard shadows with random shadow materials. Finally, we add random level of specular reflection random areas. See also the supplemental materials for full details. The shading images are 20k high-quality shadow materials purchased from several online image stocks, and 500k generated materials using Flux LoRA trained on those 20k purchased samples. We filtered 50M images to finalize 6M images by comparing the CLIP Vision similarity to key words "beautiful lighting", "light", and "illumination" to remove images unrelated to illumination.
rendering data We render Objaverse (Deitke al., 2023) using a method similar to G-buffer Objaverse (Zuo et al., 2024). The difference is that we use image-based rendering pipeline written PyTorch for faster speed. We use random environment maps obtained from previous "in-the-wild image augmentation". and use the method of Xiao al. (2023) to detect prompts. We do not generate degradation images but directly use random unpaired environment map to render an altered appearance as Id. The scale of this portion of our dataset is finalized at 4M images.
Light Stage We use multiple light stage datasets from Mnichelson (2006), Liu et al. (2024a), and an internal dataset with 20k light stage appearances. We pre-render all One-Light-ALe At-a--tt--Tme (OLAT) data into the same aforementioned format. We use random environment maps obtained from previous fin-the-wldd image augmentation", use the method of Xiao et al. (2023) to detect prompts, and use random unpaired environment map render altered appearances Ia-
3.2 IMPOSING CONSISTENT LIGHT TRANSPORT
Our goal is to learn robust and generalized model to handle in-the-wild illumination patterns. Nevertheless, learning the large-scale, complicated, and noisy data is challenging. Without well- suited regularization and constraints, the model can easily degrade to random behaviors that do not correspond to the intended illumination editing. Our solution is to Impose Consistent Light (IC-Light) transport during training, rooted in the physical principle that the linear blending of object's appearances under different illumination conditions is consistent with its appearance under a mixed illumination condition.

---

Published as a conference paper at ICLR 2025
degradation la
degradation la
c(L
af a(L
L
E
scheduler
text &estep
L
11
3
scheduler
L
8
texc Sisepp
L
VAE
VAE
loss
loss
(a) Vanilla image- conditioned diffusion
Leaminng light transport consistency
Learning objective. We visualize the learning objective of vanilla monge-conditione diffusion model for illumination learning, and the learning objective for light transport consistency. The VAES are frozen. Although this architecture is typical for UNet-based diffusion models, the same method also applies for (latent) diffusion transformers.
Figure 3:
Vanilla objective We start with a vanilla image-conditioned diffusion model to learn the illumina- tion without special constraints. As shown in Fig. 3-(a), taking a typical Stable Diffusion UNet as an example, we manipulate the UNet architecture add 4 channels to the input convolution layer to receive Ia. the randomly relighted appearance of the target object or the degradation image. We reshape any 32 32 HDRI environment light source image to 3072 numbers and train MLP from scratch with projection 3072 + 4096 - 4096 - 4096 + 2304 (activated by Leaky RELU) and reshape the 2304 output numbers into a 768 embedding that can be directly received as a prompt embedding input with 3 tokens and 768 channels by SD 1.5. Given the target relighted image Ir latent diffusion algorithms first encode J as a latent image e(Tr). and then progressively add noise the latent image produce noisy latent (TI)). where represents the number of times noise is added. Given the set of conditions including time step t, illumination condition L, as well as the input degradation Ia, image diffusion algorithms learn the network ò predict the noise with
Lyanilla = ||c 5( L, =,,)
(1)
where e is diffusion target (noise v-target for eps/v-prediction model, flow target for flow match); Cvanilla is the cost function. Learning this objective allows for basic image relighting functionality with diffusion models. Besides, train hackground-conditiones model, B to Ia (and fill the extra channel with all zeros if some part of the dataset do not have backgrounds). Nevertheless, since the illumination data is challenging and noisy, this single objective will often lead to random model behaviors, e.g., color mismatch, incorrect details, etc.
concatenate
Light transport consistency In computational photography, light transport theory demonstrates that, considering arbitrary appearance r and the correlated environment illumination L, matrix T always exists so that
F = TL,
(2)
where T can be seen T E [X 332x3333 in data format. The "*" indicates images in raw high-dynamic range. Real-world measurements (Debevec et al., 2000) validate that can always be represented with single matrix, without any non-linear transforms. Because of this linearity, light transport explains appearance merging that
(3) where L,, L2 are two arbitrary environment illumination maps. This intuitively shows that the mixture of object's appearances under separate illuminations (e.g.. L1, L2) is equivalent to the appearance under merged illumination (e.g., +..) This phenomenon is also validated by real-world measurements, e.g., (Haeberli, 1992), and we attach related validating examples in the supplementary materials.
Ei T(L, L2) = Ii.:
In this paper, observe that the appearance I in Eq. (3) can be replaced by arbitrary diffusion targets thanks to its linearity. For instance, consider a simple k-diffusion epsilon target at sigma-space step at estimated noise éz (conditioned L), and noisy image I,,, the estimated clean appearance can

---

Published as a conference paper at ICLR 2025
be written as In (Io, /L)//r, By applying Eq. (3) I +1 I, + ن will also have ACL, = EL, t where is a constant scaling factor (usually À = 2, see also Appendix. A). As result, the term in Eq. (3) can be replaced by any diffusion targets, e-g-, eps pps-prediction, v-prediction, flow match, etc., as long as the target itself is linear and first-order.
The core idea of light transport consisteney is to guarantee Eq. (3) during the diffusion training process so as to constrain the model to only modify image illumination without changing other intrinsic properties (i.e., keep the internal light transport T unchanged). This can be achieved 11 is L2 norm). Using the aforementioned
by minimizing | ( + (igg )l (where || conversion, we can write it as a loss function for eps- preeirediition model || - (ez, + <L2)
For practical implementation, considering that most diffusion models are not pixel diffusion models trained on HDR images, an conversion is needed for latent diffusion or LDR pixel diffusion. We use a simple learnable Multi-Layer Perceptron (MLP) o(.:)) :) to learn an implicit adaptation among potentials data domains (LDR, HDR, Latent), by replacing the sum term in Eq. 3. Taking eps- prediction as an example, we have the final form of light transport consistency
Leensistency
M @ (CL,+L, Ó(EL, (L,)))
where ():.)) is 5-layer MLP with hidden state 128, and input/output same as latent channels for different models, and @ is pixel-wise multiplying with foreground mask M (resized to same size latent images). During training,
synthesize L1, L2 by generating random 4 masks from uniform distribution and resize the masks to same shape as L, then, we view the masked area as L and unmasked areas as L2. ensuring L L1 L2. This loss function can be fully expanded
Coonsistency
M @ (e - (E II )., tt, L1, e (La)))()))))))) 8(E L2, E2,
(5)
where each component is visualized in Fig 3-(b).
Joint learning objective The final learning objective can be written as
L Avanilla SlomsiseancyCcoosisstenn
(6)
where L is the merged objective, and we use Avanilla = 1.0, Acousistoncy = 0.1 as default weights.
4
EXPERIMENT
4.1
EXPERIMENTAL DETAILS
We use the AdamW optimizer with the learning rate of le-5 for training the entire framework. The pretrained Stable Diffusion models used are SD 1.5, SDXL, and Flux..10-dev. The training was conducted 8 H100 80GB NVLink GPUs We select the largest possible batch size for each model. For the SD 1.5 version of the model, the training took 100 hours. For the SDXL model, we first trained 512 resolution for 80 hours, then fine-tuned 1024 resolution for 60 hours.
The training process for the Flux model was more complex due to the large size of the Flux model. We adopted multi-stage training strategy, where separately trained the double-stream and single- stream parts of the model, using gradient freezing to freeze some parts of the gradient graph. This allowed us to train in larger batch sizes. More training details can be found in the supplementary material.
We use scheduled probability balance the multiple training datasets, the in-the-wild image data and 3D rendering data appeared with equal probability during the initial stage of training. As training iterations increased, the probability of light stage data appearing in each batch increases. This allowed us to leverage a smaller portion of high-quality light stage data to improve the final model performance. In the beginning stages of training, the probabilities of in-the-wild data and 3D data were both 0.5, with light stage being 0.0. After 100,000 iterations, the probabilities were adjusted to 0.35 for in-the-wild data, 0.35 for 3D data, and 0.3 for light stage data. These probabilities were adjusted linearly throughout the training process.
4.2 ABLATIVE STUDY
We conducted an ablative study to understand the importance of different components. We first resume the model from training but removing the in-the-wild image augmentation data. As seen in
7

---

Published as a conference paper at ICLR 2025
Input
Removing in- the wild data
Removing light transport consistency
Full method
Figure 4: Ablative Study. We present results by removing the light transport consistency or in the wild data. More results are in the supplementary material. Results in this figure are from Stable Diffusion 1.5 version of our model. Prompts are "toy in room, studio lighting", and "a handsome man, neon city".
Fig. 4, removing in-the-wild data severely impacted the model's generalization capability, especially for complex images like portraits. For example, hats portraits that weren't 't present in the training data would often be rendered in incorrect colors (e.g.. changing from yellow to black).
We also experimented with removing the light transport consistency. Without this constraint, the model's ability to generate consistent illumination and retain intrinsic properties such as albedo (reflectance color) significantly decreased. For example, the red and blue differences vanished in some images, and noticeable issues with color saturation are observed in the output.
The full method, which combines multiple data sources and enforees light transport consistency, produces a well-balanced model capable of generalizing across a range of scenarios. It also retains mne-grained image details and intrinsic properties, such as albedo, while reducing errors in output images. More examples are in the supplementary materials.
4.3 ADDITIONAL APPLICATIONS
As shown in the Fig. 5, we demonstrate additional applications such as using background conditions for illumination harmonization. Trained extra channels for background conditions, our model can perform illumination generation conditioned solely on a background image without relying on environment maps. Additionally, our model supports different base models such SD1.5, SDXL, and Flux, and the capabilities of these models are reflected in the generated results.
Multiple inferences from our method generate consistent appearances, which can be blended into normal maps. Specifically, for each object. we treat the global average of all relighted appearances as the albedo (diffuse color) and divide the independent relighted appearances It, by the albedo to
obtain shading maps SL, as SL. Iz, where L; is the i-th light source. The two vertical shading maps Stp and St... are averaged form the green channel of the normal map N, and the two horizontal shading maps Stan and St are averaged to form the red channel of the normal map as
Seap S2 Nucd S2n Smaa Finally, the blue channel is padded to ensure the normal
N Wereen
8

---

Published as a conference paper at ICLR 2025
Condition
"3
Ingut
Shading from relightngg
"ligh-qualit image a statue"
(a) B Background-conditioned conditioned Model
professinaal studio and orange-spolighte
Input
SD
SDXL
Flux
(b) Altemative base diffusion models
Normal estimation
Figure 5: Additional applications. We show that this model supports more types of inputs like background conditions, and various base models like SDI.5/DDXL/Flxx We also show that multiple inferences of this method yields consistent appearances that can be merged into normal maps.
map N is unit vector for each pixel as Nolue - N
11-
N green and this process generates
complete normal map that can be used for further rendering. We visualize examples in Fig. 5-(c) by transforming each channel by N_ We also point out that this normal extraction is an empirical 2
method since the neural models are not optimized to approximate light stage ground truths or 3D normal maps. The extraction normal maps relies completely on the model's 's capability to produce consistent appearances in different illumination conditions.
4.4
QUANTITATIVE EVALUATION
We conducted quantitative comparisons using metrics such as Peak Signal-to-Noise Ratio (PSNR), Structural Similarity Index (SSIM), and Learned Perceptual Image Patch Similarity (LPIPS). We extracted subset of 50,000 unseen 3D rendering data samples from the dataset for evaluation, ensuring that the model had not encountered these samples during training.
The tested methods SwitchLight (Kim al., 2024), DiLightNet (Zeng et al., 2024), and variants of our method without certain components (e.g., without light transport consistency, without augmentation data, without 3D data, and without light stage data). As shown in Table our method outperforms others in terms of LPIPS, indicating superior perceptual quality. Models trained only on 3D data achieved the highest PSNR, but this is likely due to evaluation bias towards the rendering data (since this test only use 3D rendering data). The full method, which com- bines multiple data sources, achieved a balance between perceptual quality and performance.
Table 1: Quantitative tests of ablative architectures and alternative methods.
Method
PSNR
SSIM
LPIPS
SwitchLight
18.45
0.7024
0.3245
DiLightNet
21.78
0.8013
0.1721
w/o LTC
20.32
0.7542
0.1927
w/o aug. data
23.95
0.8723
0.1115
w/o 3d data
22.10
0.8041
0.1298
w/o light stage
23.70
0.8501
0.1077
Ours
23.72
0.8513
0.1025
4.5 VISUAL COMPARISON
We also conducted visual comparisons with previous methods. As seen in Fig. 6, compared to Relightful Harmonization (Ren et al., 2024), this model demonstrates higher robustness to shadows, thanks to larger and more diverse training dataset. SwitchLight (Kim et al., 2024) and this model produces competitive relighting results. The quality of the normal maps this method a bit more detailed due the methods of merging and deriving shadow from multiple appearances. Addition-

---

Published as a conference paper at ICLR 2025
Ingut (w/ bg)
Relighttul Harmonizaaion
Ours
Input (w/ bg)
Relighthul Harmonization
Ours
Switchl Light (normal)
Input
SwitchLight
Owrs
SwitnhLightt (normal)
Ours (oormal)
Ours (normal)
Inpot
Geowizard
Ours (normal)
Input
DSINE
Ours (normal)
Figure 6: Visual Comparison. The Relightful Harmonization results are taken from official paper Ren al. (2024) while other methods use official code bases or services.
ally, the normal maps produced by this model exhibit higher quality for human than alternatives GeoWizard (Fu et al., 2024) and DSINE (Bae & Davison, 2024).
CONCLUSION
In this paper, we propose an approach for scaling up the training of diffusion-based illumination editing models by imposing consistent light transport (IC-Light). Our method ensures desired illumination manipulation while preserving intrinsic image properties, such as albedo and fine details. This is achieved through light transport consistency, a physically grounded constraint that helps stabilize training across diverse data sources, including stage captures. We demonstrate through extensive experiments ablation studies that this approach reduces uncertainties, prevents artifacts, and improves model generalization to various illumination conditions. Additionally, our method supports a range of applications, such as background- relighting and normal map generation, and it scales to large datasets and strong model backbones like SDXL and Flux. The results validate a robust method well-suited for industrial and creative applications in mage-based illumination editing.
images, 3D rendered data, and light
ACKNOWLEDGMENTS
This work was partially supported by the Brown Institute for Media Innovation and by Google through their affiliation with Stanford Institute for Human-cemtered Artificial Intelligence (HAI).
REFERENCES
Yuval Alaluf, Daniel Garibi, Or Patashnik, Hadar Averbuch-Eie Elor, and Daniel Cohen-Or Cross-image attention for zero-shot appearance transfer. arXiv preprint arXiv:2311.00 01.22311103333 1.03335, 2023.
10

---

Published as a conference paper at ICLR 2025
Titas Anciukevičius, Zexiang Xu, Matthew Fisher, Paul Henderson, Hakan Bilen, Niloy J Mitra, and Paul Guerrero. Renderdiffusion: Image diffusion for 3d reconstruction, inpainting and generation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 12608-12618 8, 2023.
Gwangbin Bae and Andrew J. Davison. Rethinking inductive biases for surface normal estimation. In /EEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024.
Jonathan T. Barron and Jitendra Malik. Shape, albedo, and illumination from single image of an unknown object. In 2012 IEEE Conference on Computer Vision and Pattern Recognition, pp. 334-341, 2012.
Jonathan T Barron and Jitendra Malik. Shape, illumination, and reflectance from shading. IEEE fransactions on pattern analysis and machine intelligence, 3708) 2014.
Jonathan T. Barron and Jitendra Malik. Shape, illumination, and reflectance from shading, 2020.
S. Bell, K. Bala, and N. Snavely. Intrinsic images in the wild. ACM Transactions on Graphics (TOG), 33(4), 2014.
S. Bi, X. Han, and Y. Yu. An L1 image transform for edge-preserving smoothing and scene-level intrinsic decomposition. ACM Transactions Graphics (TOG), 34(4):78, 2015.
Manuel Brack, Felix Friedrich, Dominik Hintersdorf, Lukas Struppek, Patrick Schramowski, and Kristian Kersting. Sega: Instructing diffusion using semantic dimensions. arXiv preprint 10Xiv:2301.12247, 2023.
Tim Brooks, Aleksander Holynski, and Alexei A. Efros. Instructpix2pix: Learning to follow image editing instructions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 18392-18402, June 2023.
J. Butler, J. Wulff, G. B. Stanley, and M. J. Black. A naturalistic open source movie for optical flow evaluation. A. Fitzgibbon et al. (Eds.) (ed.), European Conf. on Computer Vision (ECCV), Part IV, LNCS 7577, pp. 611 6-25. Springer-Verlatd October 2012.
Dan A Calian, -ranooras Lalonde, Paulo Gotardo, Tomas Simon, Iain Matthews, and Kenny Mitchell. From faces to outdoor light probes. In Computer Gruphics Forum, volume 37, pp. 51-61. Wiley Online Library, 2018 8.
Mingdeng Cao, Xintao Wang, Zhongang Qi, Ying Shan, Xiaohu Qie, and Yinqiang Zheng. Masactrl: Tuning-fre free mutual self-attention control for consistent image synthesis and editing. arXiv preprint arXiv:2304.08465, 2023.
A. X. Chang, T. Funkhouser, L. Guibas, P. Hanrahan, Q. Huang, Z. Li, S. Savarese, M. Savva, S. Song, H. Su, al. Shapenet: An information-rich 3d model repository. arXiv preprint arXiv:1512.03012 2015.
Hansheng Chen, Jiatao Gu, Anpei Chen, Wei Tian, Zhuowen Tu, Lingjie Liu, and Hao Su. Single- stage diffusion nerf: unified approach to 3d generation and reconstruction. arXiv preprint arXiv:2304.067 14, 2023a.
Rui Chen, Yongwei Chen, Ningxin Jiao, and Kui Jia. Fantasiaãd: Disentangling geometry and appear- ance for high-quality text-to-3d content creation. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), October 2023b.
Guillaume Couairon, Jakob Verbeek, Holger Schwenk, and Matthieu Cord. Diffedit: Diffusion-based semantic image editing with mask guidance. arXiv preprint arXiv:2210.11427, 2022.
Paul Debevec. Rendering synthetic objects into real scenes: Bridging traditional and image-based graphics with global illumination and high dynamic range photography. In Acm siggraph 2008 classes, pp. 1-10. 2008
Paul Debevec, Tim Hawkins, Chris Tchou, Haarm- Duiker, Westley Sarokin, and Mark Sagar. Acquiring the reflectance field of human face. In Proceedings of the 27th annual conference on Computer graphics and interactive techniques, pp. 145-156, 2000.
11

---

Published as a conference paper at ICLR 2025
Matt Deitke, Dustin Schwenk, Jordi Salvador, Luca Weihs, Oscar Michel, Eli VanderBilt, Ludwig Schmidt, Kiana Ehsani, Aniruddha Kembhavi, and Ali Farhadi. Objaverse: A universe of annotated 3d objects. In CVPR, pp. 13 11442-111,,, 2023.
Kangle Deng, Timothy Omernick, Alexander Weiss, Deva Ramanan, Jun- Yan Zhu, Tinghui Zhou, and Maneesh Agrawala. Flashtex: Fast relightable mesh texturing with lightcontrolnet. In European Conference on Computer Vision (ECCV), 2024.
Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances in Neural Information Processing Systems, 34:8780-88 877844 2021.
Julie Dorsey, James Arvo, and Donald Greenberg. Interactive design of complex time dependent lighting. IEEE Computer Graphics and Applications, 15(2):26-36, 1995.
D. Eigen and R. Fergus. Predicting depth, surface normals and semantic labels with a common multi-scale convolutional architecture. In International Conference Computer Vision (ICCV), 2015.
pp.
Qingnan Fan. Jiaolong Yang. Gang Hua, Baoquan Chen, and David Wipf. Revisiting deep intrinsic image decompositions. 2018.
Zeev Farbman, Raanan Fattal, Dani Lischinski, and Richard Szeliski. Edge-preserving decomposi- tions for multi-scale tone and detail manipulation. ACM Trans. Graph., 27(3):1-10, August 2008. ISSN 0730-0301.
Tsu-Jui Fu, Wenze Hu, Xianzhi Du, William Yang Wang, Yinfei Yang, and Zhe Gan. Guid- ing instruction-based image editing via multimodal large language models. arXiv preprint arXiv:2309. 171022 2023.
Xiao Fu, Wei Yin, Mu Hu, Kaixuan Wang, Yuexin Ma, Ping Tan, Shaojie Shen, Dahua Lin, and Xiaoxiao Long. Geowizard: Unleashing the diffusion priors for 3d geometry estimation from single image. arxiv, 2024.
Graham Fyffe and Paul Debevec. Single-shot reflectance measurement from polarized color gradient illumination. 2015 IEEE International Conference on Computational Photography (ICCP), pp- 1-10. IEEE, 2015.
Abhijeet Ghosh, Graham Fyffe, Borom Tunwattanapong, Jay Busch, Xueming Yu, and Paul Dehevec. Multiview face capture using polarized spherical gradient illumination. In Proceedings of the 2011 1 SIGGRAPH Asia Conference, pp. 1-10, 2011.
Paul Haeberli. Synthetic lighting for photography, 1992.
Ligong Han, Song Wen, Qi Chen, Zhixing Zhang, Kunpeng Song, Mengwei Ren, Ruijiang Gao, Anastasis Stathopoulos, Xiaoxiao He, Yuxiao Chen, et Proxedit: Improving tuning-free real image editing with proximal guidance. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 4291-4301, 2024.
Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or -O. Prompt- to-prompt image editing with cross attention control. arXiv preprint arXiv:2208.0166 2022.
Jonathan Ho and Tim Salimans. Classificr-free difffusion guidance. In NeurIPS 2021 Workshop on Deep Generative Models and Downstream Applications, 2021.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 2020.
Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David J Fleet. Video diffusion models. 2022.
Andrew Hou, Ze Zhang, Michel Sarkis, Ning Bi, Yiying Tong, and Xiaoming Liu. Towards high fidelity face relighting with realistic shadows. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 44719-14728, 2021.
12

---

Published as a conference paper at ICLR 2025
Andrew Hou, Michel Sarkis, Ning Bi, Yiying Tong, and Xiaoming Liu. Face relighting with geometrically consistent shadows. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.
2022.
Inbar Hubermann-Siccelllla,, Vladimir Kulikov, and Tomer Michaeli. An edit friendly ddpm noise space: Inversion and manipulations. arXiv preprint AxXi::2304066440 2023.
Haian Jin, Yuan Li, Fujun Luan, Yuanbo Xiangli, Sai Bi, Kai Zhang. Zexiang Xu, Jin Sun, and Noah Snavely. Neural gaffer: Relighting any object via diffusion, 2024.
Bahjat Kawar, Roy Ganz, and Michael Elad. Enhancing tiffusion-based image synthesis with robust classifier guidance. arXiv preprint arXiv:2208. 08664., 2022a.
Bahjat Kawar, Jiaming Song, Stefano Ermon, and Michael Elad. Jpeg artifact correction using denoising diffusion restoration models. arXiv preprint arXiv:2209. 11888, 2022b.
Bingxin Ke, Anton Obukhov, Shengyu Huang, Nando Metzger, Rodrigo Caye Daudt, and Konrad Schindler. Repurposing diffusion-based image generators for monocular depth estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024.
Hoon Kim, Minje Jang, Wonjun Yoon, Jisoo Lee, Donghyun Na, and Sanghyun Woo. Switchlight: Co-design of physics-driven architecture and pre-training framework for human portrait relighting, 2024.
S. Kim, K. Park, K. Sohn, and S. Lin. Unified depth prediction and intrinsic image decomposition from single image via joint convolutional neural fields. In European Conference on Computer Vision (ECCV), pp. 143-159, 2016.
Peter Kocsis, Vincent Sitzmann, and Matthias Nießner. Intrinsic image diffusion for indoor single- view material estimation, 2023.
Peter Kocsis, Julien Philip, Kalyan Sunkavalli, Matthias Nicßner, and Yannick Hold-Geoffroy, Lightit: Illumination modeling and control for diffusion models. In CVPR, 2024.
Gihyun Kwon and Jong Chul Ye. Diffusion-based image translation using disentangled style and content representation. arXiv preprint arXiv:2209. 15264, 2022.
Jiahao Li, Hao Tan, Kai Zhang, Zexiang Xu, Fujun Luan, Yinghao Xu, Yicong Hong, Kalyan Sunkavalli, Greg Shakhnarovich, and Sai Bi. Instant3d: Fast text-to-3d with sparse-view generation and large reconstruction model. In International Conference on Learning Representations, 2024.
-ssnn Lin, Jun Gao. Luming Tang, Towaki Takikawa, Xiaohui Zeng, Xun Huang, Karsten Kreis, Sanja Fidler, Ming- Yu Liu, and Tsung- Yi Lin. Magic3d: High-resolution text-to-3d content creation. In IEEE Conference on Computer Vision and Pattern Recognition ( (CVPR), 2023.
Isabella Liu, Linghao Chen, Ziyang Fu, Liwen Wu, Haian Jin, Zhong Li, Chin Ming Ryan Wong, Yi Xu, Ravi Ramamoorthi. Zexiang Xu, and Hao Su. Openillumination: A multi-illumination dataset for inverse rendering evaluation on real objects, 2024a.
Minghua Liu, Ruoxi Shi, Linghao Chen, Zhuoyang Zhang, Chao Xu, Xinyue Wei, Hansheng Chen, Chong Zeng, Jiayuan Gu, and Hao Su. One-2-3-45++: Fast single image to 3d objects with consistent multi-view generation and 3d diffusion. arXiv preprint arXiv:2311.07885, 2023a.
Minghua Liu, Chao Xu, Haian Jin, Linghao Chen, Mukund Varma T, Zexiang Xu, and Hao Su. One-2-3-45: Any single image to 3d mesh in 45 seconds without per-shape optimization. Advances in Neural Information Processing Systems, 36, 2024b.
Yuan Liu, Cheng Lin, Zijiao Zeng, Xiaoxiao Long, Lingjie Liu, Taku Komura, and Wenping Wang. Syncdreamer: Generating multiview-consisteed images from a single-view image. arXiv preprint 3oXiv:2300.033553 2023b.
Jundan Luo, Duygu Ceylan, Jac Shin Yoon, Nanxuan Zhao, Julien Philip, Anna Frühstück, Wenbin Li, Christian Richardt, and Tuanfeng Y. Wang. IntrinsicDiffusion: Joint intrinsic layers from latent diffusion models. In SIGGRAPH 2024 Conference Papers, 2024.
13

---

Published as a conference paper at ICLR 2025
Simian Luo, Yiqin Tan, Suraj Patil, Daniel Gu, Patrick von Platen, Apolinário Passos, Longbo Huang, Jian Li, and Hang Zhao. Lem-lora: A universal dable-diffusion acceleration module, 2023.
Abhimitra Meka, Christian Haene, Rohit Pandey, Michael Zollhöfer, Sean Fanello, Graham Fyffe, Adarsh Kowdle, Xueming Yu, Jay Busch, Jason Dourgarian, et al. Deep reflectance fields: high- quality facial reflectance field inference from color gradient illumination. ACM Transactions 2019.
Gruphics (TOG),
Chenlin Meng, Yutong He, Yang Song, Jiaming Song, Jiajun Wu, Jun- Yan Zhu, and Stefano Ermon. Sdedit: Guided image synthesis and editing with stochastic differential equations. International Conference on Learning Representations, 2021.
Gal Metzer, Elad Richardson, Or Patashnik, Raja Giryes, and Daniel Cohen-Or. Latent-nerf for shape-guided generation of 3d shapes and textures. arXiv preprini arXiv:2211.07600 2022.
Daiki Miyake, Akihiro Iohara, Yu Saito, and Toshiyuki Tanaka. Negative-prompt inversion: Fast image inversion for editing with text-guided diffusion models. arXiv preprint arXiv::3305.16887, 2023.
Mnichelson. Light stage data gallery, wgg.ict..s...duddattaiiiiia www.../d//ddaaaighsstgeee.. 2006.
Ron Mokady, Amir Hertz, Kfrr Aberman, Yael Pritch, and Daniel Cohen-Or. Null-text inversion for editing real images using guided diffusion models. arXiv preprint urXiv:2211.09794, 2022.
M. M. Takuya Narihira and S. X. Yu. Direct intrinsics: Learning albedo-shading decomposition by convolutional regression. In IEEE International Conference Computer Vision (CVPR), pp. 2992-3000, 2015.
T. Narihira, M. Maire, and S. X. Yu. Learning lightness from human judgement on relative reflectance. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2965-299999 2973,, 2015.
T. Nestmeyer and P. V. Gehler. Reflectance adaptive filtering improves intrinsic image estimation. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 6789-6798, 2017.
Thomas Nestmeyer, Jean- Cean-François Lalonde, lain Matthews, and Andreas Lehrmann. Learning physics-guides face relighting under directional light. In IEEE Conf. Comput. Vis. Pattern Recog., pp. 5124-5133, 2020.
Shen Nie, Hanzhong Allan Guo, Cheng Lu, Yuhao Zhou, Chenyu Zheng, and Chongxuan Li. The blessing of randomness: Sde beats ode in general diffusion- image editing. arXiv preprint arXiv:2311.014100 2023.
Rohit Pandey, Sergio Orts Escolano, Chloe Legendre, Christian Haene, Sofien Bouaziz, Christoph Rhemann, Paul Debevec, and Sean Fanello. Total relighting: learning to relight portraits for background replacement. ACM Transactions on Graphics (TOG), 40(4):1-22 2021.
Pakkapon Phongthawee, Worameth Chinchuthakun, Nontaphat Sinsunthithet, Amit Raj, Varun Jampani, Pramook Khungurn, and Supasorn Suwajanakorn. Diffusionlight: Light probes for free by painting a chrome ball. In ArXiv, 2023.
Ben Poole, Ajay Jain, Jonathan T Barron, and Ben Mildenhall. Dreamfusion: Text-to-3d using 2d diffusion. arXiv preprint arXiv:2209. 14988, 2022.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with clip latents. arXiv preprint arXiv:2204.06125, 2022.
Mengwei Ren, Wei Xiong, Jae Shin Yoon, Zhixin Shu, Jianming Zhang, HyunJoon Jung, Guido Gerig, and He Zhang. Relightful harmonization: Lighting-aware portrait background replacement. CVPR, 2024.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Confer- on Computer Vision and Pattern Recognition (CVPR), pp. 10684-10695. June 2022.
14

---

Published as a conference paper at ICLR 2025
Chitwan Saharia, William Chan, Huiwen Chang, Chris Lee, Jonathan Ho, Tim Salimans, David Fleet, and Mohammad Norouzi. Palette: smage-to-image diffusion models. In ACM SIGGRAPH 2022 Conference Proceedings. pp. 1-10, 2022.
Hiroshi Sasaki, Chris Willcocks, and Toby Breckon. with denoising diffusion probabilistic models. arXiv preprini 1rXiv:2004005355 2021.
Unit-ddpm: Unpaired image translation
Soumyadip Sengupta, Angjoo Kanazawa, Carlos D Castillo, and David W Jacobs. Sfsnet: Learning shape, reflectance and illuminance of faces in the wild'. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 6296 6306663352220 2018.
Soumyadip Sengupta, Brian Curless, Ira Sennelmacher-Shizeemean and Steven M Seitz. A light stage on every desk. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 2420-2429, 2021.
Artem Sevastopolsky, Savva Ignatiev, Gonzalo Ferrer, Evgeny Burnaev, and Victor Lempitsky. Relightable 3d head portraits from smartphone video. arXiv preprins arXiv:2012.0996, 2020.
J. Shi, Y. Dong, H. Su, and S. X. Yu. Learning non-lambertian object intrinsics across shapenet categories. In JEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1685- 1694, 2017.
YiChang Shih, Sylvain Paris, Connelly Barnes, William Freeman, and Frédo Durand. Style transfer for headshot portraits. ACM Trans. Graph., 33(4):1-14, 2014.
Zhixin Shu, Sunil Hadap, Eli Shechtman, Kalyan Sunkavalli, Sylvain Paris, and Dimitris Samaras. Portrait lighting transfer using a mass transport approach. ACM Trans. Graph., 36(4):1, 2017.
Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. International Conference on Machine Learning. pp. 2256-2265. PMLR, 2015.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffuusion implicit models. 2021.
Yizhi Song. Zhifei Zhang. Zhe Lin, Scott Cohen, Brian Price, Jianming Zhang, Soo Ye Kim, and Daniel Aliaga. Objectstitch: Object compositing with diffusion model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 18310-18319, 2023.
Yizhi Song, Zhifei Zhang, Zhe Lin, Scott Cohen, Brian Price, Jianming Zhang, Soo Ye Kim, He Zhang, Wei Xiong, and Daniel Aliaga. Imprint: Generative object compositing by learning pdentity- representation. arXiv preprint urXiv:2403.11770 2024.
Tiancheng Sun, Jonathan T Barron, Yun-Ta Tsai, Zexiang Xu, Xueming Yu, Graham Fyffe, Christoph Rhemann, Jay Busch, Paul Debevec, and Ravi Ramamoorthi. Single image portrait relighting. ACM Transactions on Graphics (TOG), 38(4):1-12, 2019.
Tiancheng Sun, Zexiang Xu, Xiuming Zhang, Sean Fanello, Christoph Rhemann, Paul Debevec, Yun-Ta Tsai, Jonathan T Barron, and Ravi Ramamoorthi. Light stage super-resolution: Continuous high-frequency relighting. ACM Transactions on Graphics (TOG), 39(6):1-12, 2020.
Junshu Tang, Tengfei Wang, Bo Zhang, Ting Zhang, Ran Yi, Lizhuang Ma, and Dong Chen. Make- it-3d: High-fidelity 3d creation from a single image with diffusiin prior. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 22819-2288 12-9922222, October 2023.
Narek Tumanyan, Michal Geyer, Shai Bagon, and Tali Dekel. Plug-and-play diffusion features for text-driven image-to-image translation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1921-1930, 2023.
Bram Wallace, Akash Gokul, and Nikhil Naik. Edict: Exact diffusion inversion via coupled transfor- mations. arXiv preprint arXiv:2211. 12446. 2022.
Tengfei Wang, Ting Zhang, Bo Zhang, Hao Ouyang, Dong Chen, Qifeng Chen, and Fang Wen. Pretraining is all you need for image-to-image translation. 2022.
15

---

Published as a conference paper at ICLR 2025
Tengfei Wang, Bo Zhang, Ting Zhang. Shuyang Gu, Jianmin Bao, Tadas Baltrusaitis, Jingjing Shen, Dong Chen, Fang Wen, Qifeng Chen, et al. Rodin: A generative model for sculpting 3d digital avatars using diffusion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 4563-4573, 2023a.
Yifan Wang, Aleksander Holynski, Xiuming Zhang, and Xuaner Zhang. Sunstage: Portrait recon- struction and relighting using the sun light stage. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 20792-20802, 2023b.
Zhengyi Wang, Cheng Lu, Yikai Wang, Fan Bao, Chongxuan Li, Hang Su, and Jun Zhu. Prolific- dreamer: High-fidelity and diverse text-to-3d generation with variational score distillation. In Thirty-seventh Conference on Neural Information Processing Systems, 2023c.
Andreas Wenger, Andrew Gardner, Chris Tchou, Jonas Unger, Tim Hawkins, and Paul Debevec. Performance relighting and reflectance transformation with iimemuutiplexed illumination. ACM Trans. Graph., 24(3):556-744 2005.
Chen Henry Wu and Fernando De la Torre. A latent space of stochastic diffusion models for zero- shot image editing and guidance. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 7378-7387, 2023.
Chen Xi, Peng Sida, Yang Dongchen, Liu Yuan, Pan Bowen, Lv Chengfei, and Zhou. Xiaowei. Intrinsicanything: Learning diffusion priors for inverse rendering under unknown illumination. arxiv: 2404. 1153,, 2024.
Bin Xiao, Haiping Wu, Weijian Xu, Xiyang Dai, Houdong Hu, Yumao Lu, Michael Zeng. Ce Liu, and Lu Yuan. Florence-2: Advancing a unified representation for variety of vision tasks. arXiv preprint arXivv2311..00 11.26222.,, 2023.
Li Xu, Cewu Lu, Yi Xu, and Jiaya Jia. Image smoothing via 10 gradient minimization. ACM Transactions on Graphics (SIGGRAPH Asia), 2011.
Li Xu, Qiong Yan, Yang Xia, and Jiaya Jia. Structure extraction from texture via natural variation measure. ACM Transactions on Graphics (SIGGRAPH Asia), 2012.
Yinghao Xu, Hao Tan, Fujun Luan, Sai Bi, Peng Wang, Jiahao Li, Zifan Shi, Kalyan Sunkavalli, Gordon Wetzstein, Zexiang Xu, and Kai Zhang. Dmv3d: Denoising multi-view diffusion using 3d large reconstruction model, 2023.
Yu-Y Ying Ych, Koki Nagano, Samch Khamis, Jan Kautz, Ming- Yu Liu, and Ting-Chun Wang. Learning to relight portrait images via virtual light stage and wynthetic-to-real adaptation. ACM Trans. Graph., 2022.
Chong Zeng, Yue Dong, Pieter Peers, Youkang Kong, Hongzhi Wu, and Xin Tong. Dilightnet: Fine-grained lighting control for diffusion-based image generation. In ACM SIGGRAPH 2024 Conference Papers, 2024.
Longwen Zhang, Qixuan Zhang, Minye Wu, Jingyi Yu, and Lan Xu. Neural video portrait relighting in real-time via consistency modeling. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 802-812, 2021.
Lvmin Zhang, Chengze Li, Yi JI, Chunping Liu, and Tien tsin Wong. Erasing appearance preservation in optimization-basee smoothing. In European Conference on Computer Vision (ECCV). 2020a.
Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. In Proceedings of the IEEE/CVF International Conference Computer Vision pp. 3836-3847, 2023a.
Xuaner Zhang. Jonathan T Barron, Yun-Ta Tsai, Rohit Pandey, Xiuming Zhang, Ren Ng, and David Jacobs. Portrait shadow manipulation. ACM Transactions Graphics (TOG), 39(4):8--1, 2020b.
Zhixing Zhang, Ligong Han, Amab Ghosh, Dimitris N Metaxas, and Jian Ren. Sine: Single image editing with text-to- toxt-to-image diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6027-6037, 2023b.
16

---

Published as a conference paper at ICLR 2025
Min Zhao, Fan Bao, Chongxuan Li, and Jun Zhu. Egsde: Unpaired image-to-image translation via energy-guided stochastic differential equations. Advances in Neural Information Processing Systems, 35:3609-3666 -6239, 2022.
Peng Zheng, Dehong Gao, Deng-Ping Ping-Ping Fan, Li Liu, Jorma Laaksonen, Wanli Ouyang, and Nicu Sebe. Bilateral reference for high-resolution dichotomous image segmentation. CAAI Artificial Intelligence Research, 3:9150038, 2024.
Hao Zhou, Sunil Hadap, Kalyan Sunkavalli, and David W Jacobs. Deep single-image portrait relighting. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 1194-7202, 2019.
T. Zhou, P. Krahenbuhl, and A. A. Efros. Learning data-driven reflectance priors for intrinsic image decomposition. In IEEE International Conference on Computer Vision (ICCV), pp. 3469-3477, 2015.
Taotao Zhou, Kai He, Di Wu, Teng Xu, Qixuan Zhang, Kuixiang Shao, Wenzheng Chen, Lan Xu, and Jingyi Yu. Relightable neural human assets from multi-view gradient illuminations. Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 4315-4327, 2023.
D. Zoran, P. Isola, D. Krishnan, and W. T. Freeman. Learning ordinal relationships for mid-level vision. IEEE International Conference on Computer Vision (ICCV), pp. 388-396, 2015.
Qi Zuo, Xiaodong Gu, Yuan Dong. Zhengyi Zhao, Weihao Yuan, Lingteng Qiu, Liefeng Bo, and Zilong Dong. High-fidelity 3d textured shapes generation by sparse encoding and adversarial decoding. In European Conference on Computer Vision, 2024.
A SCALING FACTOR IN DIFFUSION OBJECTIVES
We observed that certain diffusion objectives, such as e-prediction, introduce an extra scaling factor such that
A = In, + Iiz:
(7)
where À corresponds to the number of blended elements (e.g., == 2 in this case). However, other diffusion objectives, such as Po-prediction, remain unaffected. Similar to the HDR/LDR scaling, this scaling implicitly learned by the MLP.
For example, when blending X = 2 objectives:
Nl = =I,, + -iai
(8)
we have:
1 CLi 11. - ELi X
II. EL2
+
(9)
σ:
0t
XL NELI+11 = -1 Eli
1b.
EL22
(10)
XcLtt1 CLi + CL2
(11)
Thus, the MLP implicitly scales the data by factor of 1/2. Notably, whether we use (L, 1h, Ib or not makes no difference. Even ideal cases where all these values are available and applied, with ground truth noise and À 2, we still obtain:
ML = -L,,- + IL2
(12)
leading to:
11 -E1t42 141 ET1 x
1ha EL2
(13)
σt
11at112 X
1he f 1he flg
(14)
di
A(L1,L22 + OTE) XEL,tL, (II, + toe) + (Il, + 0,e) - (EL, +ELa),
(15)
I, + II, + 20, dELI+L2 = =I,, + o,et Ix, + +o, - (EL, + +cL2);
(16)
dCL,tL, CLi CL2
(17)
17

---

Published as a conference paper at ICLR 2025
This confirms that using It or (Igh } does not affect the final result, and the MLP implicitly learns to scale the data by 1/x. Notably, this scaling effect is only observed in certain objectives such as e-prediction, while other objectives like zo-prediction remain unaffected.
18


Reviews:
[{"review_id": "fzlPq4SlXx", "reviewer": "Reviewer_Tpnw", "summary": "This paper introduces an extremely well-generalized illumination control model for recent diffusion models. Illumination control is essential for numerous artistic effects and is key to driving the mode of generated imagery. Recent methods struggle to capture this control due to the lack of large-scale real-world datasets. The key idea of the paper is to use the linearity of lighting as a constraint, i.e., the blending of two images of the same scene under two different illuminations is the same as blending the two illuminations and applying them to the scene. The authors collect a high-quality dataset from various sources using numerous recent pre-trained models to generate a large-scale paired dataset. The effectiveness of the idea is extremely well-demonstrated both qualitatively and quantitatively. The resulting images have extremely consistent materials, even so consistent that high-quality normal maps can be extracted using multi-light generation.", "questions": "It would be helpful for easier understanding to better describe the followings:\n* More details would be helpful on the synthesis of L1 and L2 (L.319), probably even some visuals in the supplementary. \n* More qualitative samples from the dataset with all the methods used to generate the synthetic components.", "limitations": "", "rating": 10, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "This is an extremely well-written and well-motivated paper; I believe this is a really good example of a highlight paper. Key points I have considered during reviewing the paper:\n* The paper proposes a well-working solution to an extremely challenging task. Lighting and material are only observed in a highly intertwined manner, making it difficult to control only one without changing the other. Previous methods tried to solve this problem with paired synthetic datasets, albeit falling into a domain gap. \n* The proposed dataset provides a really important insight, which can be important for the community: using pre-trained models and synthetic degradations can still provide helpful signals. \n * The light transport consistency is a simple but very effective way to enforce consistency. However, the idea is not new, earlier methods required multi-light datasets to utilize a similar constraint. This paper shows a way how to realize it with in-the-wild samples. \n* The approach is extremely well-documented together with practical insights and considerations.", "weaknesses": "This is an extremely good paper overall; I just have minor comments, which might help improve the writing quality.\n* It would be great to highlight more which datasets are used for which parts of the training. Potential cause for confusion: the pipeline figure shows that the training required I_d, although I_d is used only for in-the-wild samples, but then it is not clear where the other components of the dataset are used. \n* Some additional related works, which could be discussed:\n  * [Shape-form-shading](https://www2.eecs.berkeley.edu/Pubs/TechRpts/2013/EECS-2013-117.pdf) - how it relates to the current normal extraction\n  *  [LightIt](https://peter-kocsis.github.io/LightIt/) - illumination control with a generated paired dataset, where similarly an automatically relit image is used as input during training to avoid domain gap. \n* It would be good to highlight the best metrics with bold in Tab. 1.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6dgxECsgkK", "reviewer": "Reviewer_2MNV", "summary": "This paper proposes a physically inspired relighting scheme for training a lighting-aware generative model. The idea of imposing consistent light (IC-Light) is based on the observation of the real world; i.e., an object’s appearance under different illumination conditions is consistent with its appearance under mixed illumination. To this end, the authors train a generative model conditioned by lighting environment maps and an additional light consistency loss. In order to train the model effectively, the authors introduce several strategies for data collection, including rendering intrinsic components from a synthetic dataset, augmenting the existing dataset with random shadows, and separating the foreground and background. They manage to collect more than 10 million images for final training. The results of the proposed method appear to be astonishing. The authors also demonstrate side applications such as image harmonization and surface normal estimation, both of which are object-centered.", "questions": "- Although the IC-Light scheme works very well for foreground relighting, would it perform equally well for general scene-level relighting? For instance, could the proposed method achieve effects similar to turning on or off a visible light source within a scene, as shown in the paper \"Physically-Based Editing of Indoor Scene Lighting from a Single Image\" by Li et al., ECCV 2022?\n\n- In Figure 1, the authors demonstrate impressive lighting effects. I am curious whether the method consistently provides such high-quality, precise lighting effects, or if it requires careful tuning of the prompt and seed to achieve this result.", "limitations": "", "rating": 10, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "- The results are very strong. The performance of the proposed IC-Light is astonishing, and it can be further improved with a stronger generative model as the backbone.\n\n- The method is sound. Although the fundamental observation (the intrinsic component is always invariant to lighting) is not new, the authors are able to effectively leverage this observation in the training of generative models. They thoroughly explain all the details and intuition behind their method.\n\n- The contribution is significant, impacting not only the vision and graphics communities but also holding great commercial potential for the photography and movie industries.", "weaknesses": "- The fundamental observation of this paper is common knowledge in the computational photography and low-level vision communities. Unfortunately, I found that the authors overlooked a large portion of the literature in these fields, especially in terms of intrinsic image decomposition. I noticed that the authors use several IID methods to generate the intrinsic components and mention them in the supplemental materials; I think the authors should also list them in the main paper. \n\n- The quantitative experiments are somewhat limited (including details on the rendering data, e.g., how distinct they are from those in training dataset). Additionally, the numerical and visual comparisons could be improved by including methods like Neural Gaffer.\n\n- Some figures in the paper should be better explained:  For Figure 4 and 5-(c), what is the prompt/condition for the background/lighting?  \n\n- L. 427 and L. 431: \" shadow maps\" should be shading maps.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "SrwSVrjuuY", "reviewer": "Reviewer_Nsxu", "summary": "This paper proposes a method to generate illumination and background from given foreground reference images and text prompts. The paper's key contribution is to use a new consistency loss, which makes the foreground appearance consistent under different lighting conditions. The paper argues that thanks to the new loss functions, the fine-tuning of diffusion models can be scaled up to over 10 million samples. Experimental results show both qualitatively and quantitatively superior image editing results compared to existing methods.", "questions": "I consider that the proposed method and the resulting tool will be immidiately useful for many users. I would like to ask the authors how it will be released.", "limitations": "", "rating": 10, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "### Loss design\nThe design of the new loss function is reasonable. Adding the appearance consistency under a mixed lighting environment is somewhat inspired by the mixing strategies in augmentation and regularization in neural network training, while using that regularization for image illumination editing would be new to me.\n\n### Large-scale training\nThe paper fine-tunes major image generation models with over 10 million samples, resulting in an immediately useful tool for image editing.", "weaknesses": "### Unclearness of the releases of the data and implementation\nFrom the paper, it is unclear whether the authors will release the data and implementations or intend to provide a black box web tool for users.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Y3adtYGnLV", "reviewer": "Reviewer_djxF", "summary": "The paper presents a diffusion-based method for image illumination control. Compared to vanilla T2I diffusion models, the proposed model additionally inputs a degraded or altered lighting image as a reference and an HDR environment map as lighting condition, and outputs the image under the target lighting. The key of the method is the incorporation of lighting consistency loss and the use of a variety of data, including 3d renderings, processed in-the-wild data, and OLAT light-stage data. The proposed method scales well on both large-scale data and different base models and advances other methods both qualitatively and quantitatively. The authors also provided additional applications, like background-conditioned illumination control and normal estimation.", "questions": "I also have some small and less important questions:\n\nImplementation details:\n1. The reliability of DiffusionLight depends on the FOV of the image, is there any special handling on small FOV images?\n2. To estimate environment maps from in-the-wild images, the original image is divided by albedo to obtain a shading map, and then projected to form the environment map: how to handle visibility/occlusion? e.g. on a concave object. Also, if the original image itself is already over-exposed, it will produce a wrong environment map even with HDR.\n3. L242, the authors use CLIP for lighting filtering. However, as the text prompt is not enough for the lighting description, I wonder if the CLIP really has a good embedding for these lighting-related keywords. How effective does this filter work? Will something like an aesthetic score help here?\n4. L246, do you mean “image-space” rendering? Does this image-based rendering pipeline consider GI? E.g. self-occlusion and inter-reflection. If not, is this the cause of some wrong GI effects?\n5. For background conditioned inference, when only providing background, is env map L set to 0? Or is an environment map estimated from the background condition?\n6. For normal estimation, when generating a shadow map, is 3 channels averaged? I can see some color in Fig 5 (c)’s shadow maps (which should be monocular), but it’s understandable for diffusion models. Besides, will specular highlights degrade this estimation? I'm also a little confused about the division by two in equation 8 and the way of ambient computation (using the average of four images instead of using the minimum value of all images). Can you clarify this?\n\nExtra results:\n1. Normal quantitative results: how well does the normal align with a real 3D normal map? Or what does the shape look like after running a normal integration on the estimated normal map?\n2. How well is the temporal consistency when continuously changing the environment map L during inference?\n\nExtra question:\n1. Do you try to also impose lighting consistency on foreground and background?\n\nI would also be happy to raise my scores if the authors provide answers or results regarding these questions.", "limitations": "", "rating": 10, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "The paper is in general very good, with many strengths:\n1. The data preparation pipeline is very solid\n    1. The authors assembled data from diverse sources for training, significantly enhancing the model's generalizability.\n    2. For in-the-wild data, the authors utilized various intrinsic estimation methods, making the pipeline robust and avoiding inductive bias from any single method.\n2. The proposed lighting consistency loss is novel and effectively prevents training degradation.\n3. The training strategy demonstrates excellent scalability, is applicable to various diffusion base models and is capable of handling large-scale in-the-wild images.\n4. The method achieves state-of-the-art image quality among diffusion-based lighting control methods.\n5. The method also supports alternative lighting control interfaces, such as background condition.\n6. The lighting consistency can be validated through the normal estimation application.", "weaknesses": "Some small weaknesses exist in the paper:\n\nFormulation:\n1. Only linear HDR images have light transport additivity, but natural images and images generated by diffusion models are usually tone-mapped LDR images, which do not have additivity. How is this misalignment handled?\n2. There is material ambiguity in the single input image as different material-lighting combinations can yield a similar appearance. Hence I wonder how much the prompt and seed affect the model’s material estimation / final lighting result.\n\nImplementation details:\n1. Lighting consistency loss:\n    1. To add the lighting consistency regularization during the training of latent diffusion models, the authors replaced the sum of two RGB-space images with an MLP that operates in latent space. The MLP works as a soft proxy, but does this MLP really work as intended without any other regularization?\n    2. Also, the loss is derived using x_0 prediction, which is not very stable and can have low SNR when t is large, is there any special design regarding this issue?\n2. Normal estimation:\n    1. The formulas for normal estimation are not correct. Equation 8 should be minus rather than average. It is also not proper to call these two directions N_green and N_red. They should be defined in the world space with range [-1, 1] and visualized as (N + 1) / 2.\n    2. It is better to use a consistent coordinate for the normal maps. In Fig (6), the man in the middle row has a different normal map specification from the examples on the last row, with R and G representing opposite directions in these two examples.\n\nI would raise my score if the above problems were explained.\n\nBesides, in the image results:\n1. Some specular highlight seems to be baked in or show a strong correlation to the input image. E.g. the specular highlight on that car in Fig. 1, and on the helmet in Fig. 4\n2. Some images have wrong global illumination (GI) effects, e.g. the ice-cream example on supp material page 6, has two shadows in opposite directions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Oral)