DreamLight: Towards Harmonious and Consistent Image Relighting
Authors: Yong Liu
Wenpeng Xiao
Qianqian Wang
Junlin Chen
Shiyin Wang
Yitong Wang
Xinglong Wu
Yansong Tang
Venue: NeurIPS 2025 poster ()
Keywords: Image Composition
Image Relighting
Image Harmonization
Primary area: applications

Abstract: We introduce a model named DreamLight for universal image relighting in this work, which can seamlessly composite subjects into a new background while maintaining aesthetic uniformity in terms of lighting and color tone. The background can be specified by natural images (image-based relighting) or generated from unlimited text prompts (text-based relighting). Existing studies primarily focus on image-based relighting, while with scant exploration into text-based scenarios. Some works employ intricate disentanglement pipeline designs relying on environment maps to provide relevant information, which grapples with the expensive data cost required for intrinsic decomposition and light source. Other methods take this task as an image translation problem and perform pixel-level transformation with autoencoder architecture. While these methods have achieved decent harmonization effects, they struggle to generate realistic and natural light interaction effects between the foreground and background. To alleviate these challenges, we reorganize the input data into a unified format and leverage the semantic prior provided by the pretrained diffusion model to facilitate the generation of natural results. Moreover, we propose a Position-Guided Light Adapter (PGLA) that condenses light information from different directions in the background into designed light query embeddings, and modulates the foreground with direction-biased masked attention. In addition, we present a post-processing module named Spectral Foreground Fixer (SFF) to adaptively reorganize different frequency components of subject and relighted background, which helps enhance the consistency of foreground appearance. Extensive comparisons and user study demonstrate that our DreamLight achieves remarkable relighting performance.

DreamLight: Towards Harmonious and Consistent Image Relighting
Yong Liu'; Wenpeng Xiao² Qianqian Wang", Junlin Chen², Shiyin Wang",
Yitong Wang', Xinglong Wu Yansong Tang
TTinnhuu Shenzhen International Graduate School, Tsinghua University ByteeDance Inc.
liuyong230maile. tsinghua.edee edu. en, tang- yawgyognnnngg..eeee
edu.cn
Abstract
We introduce a model named DreamLight for universal image relighting in this work, which can seamlessly composite subjects into a new background while main- taining aesthetic uniformity in terms of lighting and color tone. The background can be specified by natural images (image-based relighting) or generated from unlimited text prompts (text-based relighting). Existing studies primarily focus magee-ased relighting, while with scant exploration into text-based scenarios. Some works employ intricate disentanglement pipeline designs relying on environ- ment maps to provide relevant information, which grapples with the expensive data cost required for intrinsic decomposition and light source. Other methods take this task an image translation problem and perform pixel-level transformation with autoencoder architecture. While these methods have achieved decent harmoniza- tion effects, they struggle to generate realistic and natural light interaction effects between the foreground and background. To alleviate these challenges, we reorga- nize the input data into unified format and leverage the semantic prior provided by the pretrained diffusion model to facilitate the generation of natural results. Moreover, propose Position-Guidee Light Adapter (PGLA) that condenses light information from different directions in the background into designed light query embeddings, and modulates the foreground with direction-biased masked attention. In addition, we present post-processing module named Spectral Fore- ground Fixer (SFF) to adaptively reorganize different frequency components of subject and relighted background, which helps enhance the consistency of fore- ground appearance. Extensive comparisons and user study demonstrate that our DreamLight achieves remarkable relighting performance.
Introduction
With the rapid development of generative models in recent years [ image composition has received increasing attention owing to its capacity for controlled generation 5 6 However, since the implanted foreground and the background originate from different sources, this discrepancy can easily lead to an unrealistic perception of the composite image. In this paper, we focus on a challenging task named universal image relighting, which aims seamlessly composite subject into a new background while maintaining realism and aesthetic uniformity in terms of lighting and color tone. The background can either be specified by provided natural images (image- based relighting), or be created from unlimited text prompts (text-based relighting). This task ensures that users and digital elements coexist naturally within any environment and has wide application in virtual reality and intelligent editing for the film and ad advertising industries [8] 9 [0 12 13.
"qqual Contribution Conresponding author
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

A natural photo, the bockground is beach, sond, tree
In the field, strong ofternoon Light,
rutns of terrible war, 4h
Our DreamLight is unified image relighting model capable of performing relighting guided by arbitrary text or natural background images without any other prior such as HDR maps. The top left of each set of figures is the original picture of the foreground, the bottom left is the given condition, and the right is the generated result.
Figure 1:
Early rescarches about relighting tend to focus on the image condition, e.g., image harmonization and portrait relighting, while with scant exploration into text-based scenarios. Portrait relighting mainly take the idea of physics-guided design that explicitly models the image intrinsics and formation physics. They decompose the input images into several components and leverage phased pipeline to incrementally learn image components such as surface normals and albedo maps. Some harmonization methods [18 19] also take similar disentanglement idea. Despite the promising results achieved by these approaches, obtaining training data with pairs of high-quality relighting images and their corresponding intrinsic attributes from light stage is expensive and difficult. Most harmonization methods (00 21 22 [3] take this task as an image-to-image translation paradigm and propose to perform pixel-level transformation based on autoencoder architecture. However, they lack object semantic guidance and tend to overlook distinct variations in illumination. Recently, some exploit the powerful semantic modeling ability of generative diffusion models [ to enhance relighting effect, but their light source relies on the given environment maps and such maps are not always feasible to acquire in real world. IC-Light [26] is proposed to address both image-based and text-based relighting for natural images without introducing other signal guidance. It imposes light alignment training strategy to enhance the light consistence. However, IC-Light takes two separate models for different conditions and does not tailor the structure. Simply concatenating foreground, background, and noise limits the model's understanding of baggrouundbbbkkruuuddd interactions, leading to severe color bleeding and foreground distortions in some scenarios.
methods [14 15
16. 17
works [24 g 25
To address the above challenges, we present model named DreamLight that can perform both mmage-based and text-based relighting. In DreamLight, to generate natural interaction effects of light and color tone between the foreground and the background, we propose Dosition-Guidee Light Adapter (PGLA), which condenses light information from different directions in the background into several groups of light query embeddings and selectively modulates the foreground area with direction-biased masked attention. In addition, we present an effective Spectral Foreground Fixer (SFF) as a post-processing module to enhance the consistency of foreground appearance and avoid subject distortion. Based on the wavelet transform, SEF is trained to learn dynamic calibration coefficients for high-frequency textures of input foreground and relighted low-frequency light. By adaptively reorganize these information, SFF can output stunningly consistent foreground. Besides, to facilitate the training of our model, we develop different kinds of data generation processes, e.g., 3D rendering and training relighting lora, to produce diverse training samples.
We have performed extensive quantitative and qualitative comparisons. Experiment results demon- strate that our DreamLight exhibits superior generalization and performance on the universal relighting of natural images. Related ablations also prove the rationality and effectiveness of the proposed designs. Furthermore, we observe that thanks to the unified learning text and image conditions in single model, our DreamLight generate results with the guidance of both conditions.

---

Our contributions can be summarized as follows:
We propose a model named DreamLight for universal image relighting, which can seamlessly composite a subject into new background with either image or text conditions. We also develop high-quality data generation process to benefit the training of our model.
We introduce a Position-Guided Light Adapter (PGLA),, which is designed to enable fore- ground elements at different positions to interact with background light from various direc- tions in a tendentious manner for generating more natural lighting effects.
We propose a Spectral Foreground Fixer (SFF) module to adaptively reorganize different fre- quency components of the input and relighted subjects, which helps enhance the consistency of foreground textures.
2 Related Work
Image-based Relighting:
There are two principal sets of related image- based relighting methods.
The first is portrait relighting approaches [15] [4] 16 [2 28 29 30, [ 17 [1]. They mainly take the idea of physics-guide normals, albedo, and a set of diffuse and specular maps with ground truth supervision. To achieve that, some methods rely on the paired training data acquired with the light stage system [32 and target HDR environment map as the external light source. However, the dependence on light stage data and HDR maps incurs substantial data collection cost and significantly limits their implementation in real-world situations, where obtaining HDR maps may not always be feasible. Some methods [33] 31 employ multi-stage frameworks. As a result, the accuracy and performance of these systems hinge on the precision of cach individual stage. This makes the entire process complex and susceptible to errors that could propagate throughout these intermediate steps. The second image harmonization methods that aim to match the color statistics of the foreground object with those of the background 38) These methods tend take this task as an end-to-end image-to-image translation paradigm, where the network is trained to predict harmonized image from the input composite. Some works [39 40 collect pixel-aligned paired data by color transfer or altering foreground color in real images with pre-designed or learned augmentations. While these methods have achieved decent harmonization effects, they struggle to generate realistic and natural light interaction effects between the foreground and the background. Recently, some works [24] 8 (25) exploit generative models [1] to enhance relighting effect, but most of them still rely on the given environment maps for lighting guidance, which limits the potential application scenarios. LCghight (26) proposes to perform pure natural image relighting by imposing light alignment training strategy and achieves excellent performance for scenarios with strong lighting. However, its simple network design limits the model's performance and is prone to generating severe color bleeding. To alleviate above issues, we propose Dream Light designed for natural images without any additional prior source. A position-guided light adapter is introduced to boost the reasonable interaction between subjects and backgrounds, thus contributing to generating natural and harmonious lighting effect.
model design that typically involve the intermediate prediction of surface
for natural composition [8 20 22
19 35 36
37
Text-based Relighting: Currently, there is relatively less research focused on text- relighting. Although Lasagna [4] takes text as input, the text serves to indicate the light direction rather than to describe the target background. The pioneer IC-Light (26 utilizes two separate models to handle image-based based and text- based relighting. Actually, some text-guided inpainting methods 42 43) can achieve similar effect. However, they tend to preserve the original lighting and color of the subject, which may result in unnatural results. Conversely,
DreamLight achieves powerful relighting performance of natural images using a single model for both image-based and text-based situations.
3 Method
3.1 Overview
Figure SouureBillustinn Pillustrates the pipeline of our DreamLight. It takes triplet of foreground image, background image, and text prompt as input. For image-based relighting, the text prompt set to "blend these two images". For text-based relighting, the background image is designed as an all-black image. The green and brown lines show the specific processes of image-based and text-based relighting, respectively. In detail, we first utilize pretrained segmentation model [26 44) to extract the subject
3

---

"Blend these two images" "Silent road, storry sky, night"
Seg. Model
UNet
Spectral
Foreground
Foreground
Fixer
Light Query piotiia
Rackground
Position-
General Path
Guided
Image Resed
Light
Test based Puth
CLIP
f0
Adapter
Concatenation
Encoder
coeffieient map
Low freq. Enhancement
Figure 2: Pipeline of our DreamLight It takes the foreground image, background image, and text prompt as input. The masked foreground and background images are encoded by pretrained VAE and concatenated with the random noise serve as the input of UNet. The Position-Guided Light Adapter is proposed to selectively inject background light from different directions into the foreground at different locations for more natural relighting results. The Spectral Foreground Fixer is utilized to enhance the consistency of the subject.
region and remove the original background from the input image. Following 45 26
the masked subject and background are encoded by VAE and then concatenated with the random noise to serve as the input for diffusion UNet. For image-based relighting, Position-Guided Light Adapter (PGLA) is proposed to selectively inject background light information from different directions into the foreground with direction- biased masked attention. Finally, Spectral Foreground Fixer (SFF) is utilized to enhance the consistency of the foreground.
3.2 Position-Guided Light Adapter
Although simply concatenating the background with random noise can convey some information about environment lighting, imposes pixel-aligned light prior and overlooks the natural interaction between the subject and light from various directions the background, thus resulting in undesirable relighting results some scenes. To alleviate this problem, we propose the Eesition-Guided Light Adapter (PGLA), which enhances the foreground's response to light sources from different directions in background while reducing potential unreasonable light alignment. This is achieved by additional encoding and organization of the background light information.
Specifically, the overall pipeline of our PGLA is based the process of IP-Adapter [46]. We first utilize a CLIP image encoder to encode the target background image into feature map fa e XC where C' indicate the embedding dimension. H. W denote the spatial size of feature map. To mitigate the potential noise effects of background textures on light information extraction, we perform low-frequency enhancement operation the encoded background features, which helps emphasize the high-level lighting and color tones information. The process can be formulated as:
g = Gaussian(HL W.
fui FF)(f)) 9.
fur =
+
where FFT and IFFT are Fourier transform and Fourier inverse transform. g denotes the filtering coefficient map. is cutoff frequency and means element-wise product. Conv and ReLU operations are utilized to update the features the spectral domain for efficient global interaction.
Then, we restructure the background features and encode light information from different direc- tions into predefined light queries fo. which are randomly initialized learnable embeddings. Con- sidering that the condition is vanilla 2D natural background images, we assume that the light sources can be split into four basic directions: left, right, top, and down. Thus, we leverage four sets of query embeddings to selectively extract light information from background features.

---

To achieve that, we propose a direction- biased masked attention. As shown Figure 3
Query Fimbeddings
Spatial Background
Tokens
we use cross attention mech- anism to condense the information of background into the light query. In de- tail, we initialize four sets of light query and design them to separately perceive the background light sources of the four directions, Le.., left, right, top, and down. Taking the query s corresponding example, generate
left donry
grigh
zap
fe
fe
fdown
Pirection-biasee
Attention weight
the left light a coefficient map that decays from left to right with the same size as the back- ground feature, as shown by the "left decay map" in Figure 33 It is then flat- tened and multiplied with the initial at- tention weight of corresponding regions to modulate the cross attention, making the query pleft focus more on the infor- mation on the left side of the background feature. Similarly for the light query re- sponsible for the other directions. Be- sides, we concatenate the light query and background features as Key and Value to ensure the interaction between different group of queries, which contributes to the overall harmonization.
Cross Attn
Closs Aftn
Cross Altn
fe
KV
KV
KV
fM
Figure 3: Process of direction- biased masked attention. The upper figure the design of the attention mask in cross attention. Here the darker color indicates the ten- dency toward O. To allow different groups of query to per- ceive different directions of light, we generate coefficient maps with different attenuation directions and multiply them to the original attention weight. For ease of un- derstanding, we note the transformation of the left decay map in the figure. Besides, the attention weight between different queries has not been modified for overall light information harmonization.
Having these condensed light queries, the next step to inject their informa-
tion into the foreground area of the latent features in UNet with similar masked attention. That is, we adjust the attention weight of condition cross attention so that different regions of the foreground object have a stronger response to nearby light sources. In addition to the exchange of Q and KV, we additionally add a mask to the background area to ensure that the background is not changed. Furthermore, as shown in Figure we only inject these light prior in the middle and up block of the UNet. This is due the fact that feature changes in down blocks may have an impact on the overall semantics 47 We only need to make changes to the object's 's light based on its overall representation, which helps to avoid the potential distortion problems caused by extra information injection.
With such designs, the combined subject elements can perform selective interaction with the back- ground produce more natural relighting results. Related experiments Section 4.3 also justify the rationality of our designs.
3.3 Spectral Foreground Fixer
Ieffusion-basede methods tend face the problem of foreground distortion, especially in small and detailed areas such as face and text. On the one hand, the latent space of large-scale pre-trained models tends to embellish the foreground, which can lead to inconsistency and variation with the input. On the other hand, the encoding process of VAE may cause information loss in small regions with high information density, Icading to difficulties in maintaining the texture of the subject. Therefore, propose a Spectral Foreground Fixer (SFF) to address this challenge. This module is based on the assumption that the high- high-frequency components of an image correspond to the pixels varying drastically, such as object boundaries and textures, while the low-frequency components correspond to the general semantic information such as the color and light.
As shown in Figure 4 we utilize Wavelet Transform to extract the components of the input foreground image and the initial predicted results. It can seen that the high-frequency part maintains the details and textures, while low-frequency part indicate rough colors and tones. Then we combine the high-frequency part of input foreground image with the low-frequency component of the initial relighting result and feed them into a Modulator, which takes
and low-frequency
5

---

Foreground- BarkkoouddBBkkkgouuddd Composition
LQ
Tranform
Correction parameter
High-Free
Low--Free
Initial Relighting Result
Model Prediction
Modiulator
HO -axiiuu +
Wavelet
Final Relighting Resalt
Comection parameter f
Migh--Pae
Lonssbine
HQm
Iupur Subject
Figure 4: Process of Spectral Foreground Fixer. The modulator is utilized to predict set of coefficients for modulating the textures of foreground with the light background.
image-to-image generation paradigm and is trained predict set of coefficients reorganize these information. The reorganization process can be formulated as:
a, 3 MM(HQin LQqut);
HQ -QQn
B,
l'o HQ'.. LQuut:
where HQ and LQ denote the figh-frequency and low-loowoeee cow-frequency part extracted by Wavelet Transform, respectively. .M is the modulator. I xx'xx3 and B E IR 12 x x3 are the predicted modulation coefficients. a utilized to control the influence degree foreground texture and 3 contains the balance information for harmonizing the foreground. Compared directly reorganize the foreground texture and background semantics, such design helps to output more consistent and natural results, avoiding artifacts from forced combination. Finally, replace the foreground region of initial relighting images by the predicted Lout according to the foreground mask.
The modulator M is individually trained in self-supervised manner. Specifically, we perform random color transformation on arbitrary natural images to obtain pseudo pairs of relighting data, where the transformed image is input and original image serves target. Then extract the high- frequency component of the transformed image and the low-frequency component of the original image inputs to the modulator for modulation coefficient prediction. The modulator is expected to adaptively combine the high-frequency and low-frequency parts from different source and avoid potential color noise artifacts. We utilize MSE and perceptual loss (48] supervise the learning of the modulator. Furthermore, to promote training stability and improve coordination, the supervision applied on both the predicted high-frequency part HQQ' and entire output image Lou.-
3.4 Data Generation
We design data generation pipeline to facilitate the training of our model. Our data has three sources. Firstly, we construct pairs by training a relighting ominicontrol [49 lora in bootstrapping manner, ie., training incorporating results into training set continue training. The initial set consists of 100 image pairs collected from time-lapse photography videos and self-photographed photos. After each training, the model is used to relight vanilla images. High quality pairs are selected and for continue training. We will open source this relighting lora to
incorporated into the training benefit community. Secondly, we utilize available 3D assets (50) to render number of consistent images with lighting different color and directions. We construct an automatically rendering pipeline on 3D Arnold Renderer, and generate various lighting effects with random light sources and HDR images for corresponding pairs. Finally, to enhance data diversity, we also process vanilla images with IC-Light (26) and filter out high-quality synthetic data pairs with aesthetic score [51]. The prompts are generated through a two-step process: GPT-4 [52 initially brainstorms over 200 fundamental scenarios, which are then tailored by LLaVA 53 according to the main subjects present in the images. Totally, the quantities of the three types data are about 600k, 150k, and 300k, respectively. Please see the supplementary material for detailed analysis about the training data.

---

(a)
(b)
Foreground
Background
INR
PCT
IC-Lighht
(c)
Foreground
Background
Ours
Foreground
Background
Ours
Figure 5: Image-based relighting results. (a) and (b) demonstrate the comparisons with popular available image-based relighting methods, i.e., Harmonizer (38), INR [21], PCT [20], and IC- Light (26). (c) shows the generalizability of DreamLight foregrounds of different categories and various styles of images. Please see the supplementary material for more qualitative results.
4 Experiment
4.1
Implementation Details
Our model is implemented in PyTorc [54] using A100 at 512 x 512 resolutions. The main model and fixer model trained separately. The main model is trained end-to-end with the batch size of 512. The learning rate is set to 5e-5. We leverage StableDiiflusion-y1. [ as the base generative model and CLIP-H (55) the encoder for position-guided light adapter. Following (26), we takes RMBG-1.4 as the segmentation model for extracting the region of subject. The spectral foreground fixer is finetuned spectral filter is set to 5 in default. The number of light query is set to 4 for each direction. The evaluation benchmark contains 600 high-quality image pairs rendered by Arnold Renderer from real objects. For image-basse based relighting, we take the popular metrics of standard Peak Signal-to-Now Ratio (PSNR). Structural Similarity (SSIM), Learned Perceptual Image Patch Similarity (LPIPS) (56), and image similarity score calculated by CLIP (55) (CLIP-IS) to verify the effectiveness of our method. For text-based relighting, we utilize image-text matching CLIP score, aesthetic score (51) and Image Reward (IR) [57] score to assess the plausibility of the generated results. IR score is calculated by text-to-image human preference evaluation models trained on large-scale datasets of human preference choices. Aesthetic score is linear model trained on image quality rating pairs of real images. Please refer to the supplementary material for more details and illustrations.
the VAE model of StableDiffusion-V
The cutoff frequency of
4.2 Main Results
Qualitative Comparison: In Figure 5 and Figure 6 we present relighting results of our method and the comparison with existing methods. Our DreamL-ghht achieves excellent performance of both image-based and text-based relighting in a single model. More relighting results can be found in the supplementary materials. From (a) and (b) in Figure 5 we can see that our method only harmonizes the lighting of the foreground and background, but also more effectively models the interaction between light sources and objects within images. In addition human and natural backgrounds, we illustrate the generalization ability of our model for foregrounds of different categories and other stylistic images in (c). In Figure we compare our method with IC-Light [26] and existing state-of-te ttate-of-the-an the-art art prompt-guided inpainting methods since existing studies have paid limited attention to relighting based on given prompts. Although inpainting-base capable of generating backgrounds that are in accordance with text prompts, they exhibit propensity towards maintaining the color and lighting of the foreground unchanged, leading an incongruity with the background. ICght,h in contrast, grapples with issues of excessive color variations in
methods are

---

Prompt: Sunset over sea
Prompt: Chicago jazz club Lights
Prompt: An abstract, digital-ort inspired backdrop
Foreground
PowerPaint-vl
PowerPaint-v22
BrushNet
IC-Light
Ours
Figure Results of text-based relighting. PowerPaint 43) and BrushNet [42] are powerful inpainting methods that can perform prompt-guided inpainting. To better illustrate, we demarcate the foreground. outpainting methods, and relighting methods with dashed lines.
Method
PSNRT
SSIM
LPIPS
CLIPPIST
Method PP.VI
CL.IP Score Aesthetic Scoref Score
INR 22
T6.79
0.694
0.231
0.872
0.605
5.58
1.87
PCT 20
18.45
0.714
0.206
0.895
PP-V2
43
0.627
5.97
2.50
TH 19
20.66
0.762
0.177
0.896
BrushNet 42
0.613
5.73
1.96
LCghight
26
20.27
0.771
0.181
0.889
IC-Light
26
0.629
6.04
2.25
Ours
22.15
0.783
0.158
0.908
Ours
0.644
6.32
3.47
Table Quantitative results comparison
Table Quantitative results about text-based
about image-based relighting.
relighting. PP denotes PowerPoint.
the generated images as well distortions of the foreground. Our approach produce relighting results with more natural light. Additionally, the bottom row illustrates the capacity of our method to maintain consistency for the subject. We can observe that all other methods generate results with significant facial distortion, whereas ours ensures excellent subject consistency.
Quantitative Comparison: Table and Table 2 display the evaluation metrics of different methods about image-based and text-based relighting, respectively. Table I Iindicates that our approach can yield more consistent and harmonious outcomes when provided with target background image. Results in Table that our DreamLight exhibits advantages in terms of vision-text compatibility, aesthetic appeal, and rationality. Results of user study reported in the supplementary materials.
4.3 Case Study
Dusition-Guided Light Adapter: In Figure 7 we conduct visualization comparison of the ablations about the PGLAA. As depicted in Figure 7 the model struggles to learn the light interaction between the foreground and background without any prior im- position (W/o adapter). When applying vanilla IP- Adapter [46), which performs arbitrary interaction between subject and background, information from diverse directions in the background interferes with each other, thereby affecting the final relighting performance. Through utilization of direction-biased masked attention, model selectively transmits background lighting information, enabling the foreground to acquire lighting that is harmo- nious with the background (the last two columns). The design of low-frequency enhancement further discards irrelevant information exist in the background textures, thus contributing robust training
Method w/o adapter
PSNR1
SSIM
I.IPSSJ.
CLIP-IST
18.58
0.732
0.221
0.865
vanilla IP.A.
20.23
0.752
0.184
0.891
w/o Filter
21.81
0.778
0.162
0.903
PGLA (Ours)
22.15
0.783
0.158
0.908
Table 3: Results of different light adapter de- signs. "IP.A." means IP-Adapter [46]. Filter is the spectral filter used to enhance the low- frequency component of background.

---

Foreground
Background
Wi adapler
Vanilla adapter
Wio low-troq
PGLA
Figure 7: Visualization comparisons with different light adapter design strategies.
Input Subject
Wio Fixer
ControlNet
Direct Composition
Ours
Figure 8: Visualization comparisons with different foreground fixer strategies. For the of observation, we accentuate the changes in the facial region on the right side of each relighting result. The green box above showcases the magnified facial region of the relighted results. The red box below presents the magnified facial region of the input foreground image. Best viewed zoom-in.
of the model and promoting generation of more natural and consistent results. Quantitative results in Jalso demonstrate the rationality of our designs.
Spectral Foreground Fixer: Figure 8 shows the qual-
Method W/o fixer
PSNRT
SSIMF
CLIP-IST
AST
itative analysis about the proposed SFF. The introduc- tion of ControlNet (58) to provide subject information
19.31
0.632
0.846
5.31
ControlNet
19.69
0.674
0.868
5.46
fails to alleviate the issue of foreground distortion, as distortions often occur on small areas and such design struggles to avoid the problem of encoding information loss. Our SFF achieves robust foreground preservation effects, as also substantiated by the quantitative results Table As mentioned above, SFF primarily works critical small regions. While the refinement of these regions is important for visual quality, has limited impact on global metries. Thus, to better test the refinement effect, Table we crop small face regions for metric calculation.
SFF (Ours)
25.53
0.831
0,946
6.84
Table 4: Different foreground fix strategies.
Handle Both Conditions: We observe that thanks to the unified learning of text and image conditions a single model, our DreamLight can generate results with the guidance of both con- ditions. As shown in Figure 9 our method can maintain the structure and elements of the given background while making additional adjustments based on text prompts. Note that this ability is emergent and our model does not undergo training for such situation.
Text prompt: at night
Foreground
Background
Oulput
Figure 9: Visualization results of our DreamLight condi- tioned on both text prompt and background image.
Conclusion
In this paper, we present DreamLight that can perform both image-based based and text-based relighting in a single model. In addition to extending application scenarios, our model emerges with the ability to handle both conditions simultaneously. By performing tendentious interaction between foreground and background in Dasition-Guides Light Adapter (PGLA), our model can achieve a natural and harmonious relighting effect. In addition, the Spectral Foreground Fixer (SFF) greatly enhance the consistency of the subject by adaptively leveraging information of different frequency bands. To train the model, we also develop a high-quality data generation pipeline. Experiment results have demonstrated that our Dreaml achieves excellent relighting performance and superior generalization ability for natural images. We hope this work could inspire more related researches and potential practical applications.

---

Acknowledgements
This work was supported by Guangdong Natural Science Funds for Distinguished Young Scholar (No. 2025B15115020122.
References
[1] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björ Ommer. High- resolution image synthesis with latent diffusion models. In CVPR. pages 10684-10695, 2022.
[2] William Peebles and Saining Xie. Scalable diffusion models with transformers. Proceedings of the IEEE/CVF International Conference on Computer Vision, pages (195-4205, 2023.
[3] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusson probabilistic models. NeurIPS, pages 6840-6851, 2020.
[4] Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprini arXiv:2010.02502 2020.
[5] Xi Chen, Lianghua Huang, Yu Liu, Yujun Shen, Deli Zhao, and Hengshuang Zhao. Anydoor: Zero-shot object-level image customization. In CVPR, pages 6593-66966660 2024.
[6] Lianghua Huang, Di Chen, Yu Liu, Yujun Shen, Deli Zhao, and Jingren Zhou. Com- poser: Creative and controllable image synthesis with composable conditions. arXiv preprint arXiv:2302.09978, 2023.
[7] Binxin Yang, Shuyang Gu, Bo Zhang, Ting Zhang, Xuejin Chen, Xiaoyan Sun, Dong Chen, and Fang Wen. Paint by example: Exemplar-base based image editing with diffusion models. In CVPR, pages 18381-18391, 2023.
[8] Mengwei Ren, Wei Xiong, Jac Shin Yoon, Zhixin Shu, Jianming Zhang, HyunJoon Jung, Guido Gerig, and He Zhang. Relightful harmonization: Lighting-and aware portrait background replacement. In CVPR, pages 6452-6462, 2024.
[9] Anand Bhattad, James Soole, and DA Forsyth. Stylitgan: Image-based relighting via latent control. In CVPR, pages 4231 4241- 2024.
[10] Yong Liu, Ran Yu, Fei Yin, Xinyuan Zhao Wei Zhao, Weihao Xia, Jiahao Wang, Yitong Wang, Yansong Tang, and Yujiu Yang. Learning high-quality dynamic memory for video object segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2025.
[11] Zhuoyan Luo, Yicheng Xiao, Yong Liu, Shuyan Li, Yitong Wang, Yansong Tang, Xiu Li, and Yujiu Yang. Soc: Semantic-assisted object cluster for referring vidco object segmentation. Advances in Neural Information Processing Systems, 36:26425-2641 2023.
[12] Kunyang Han, Yong Liu, Jun Hao Liew, Henghui Ding, Jiajun Liu, Yitong Wang, Yansong Tang, Yujiu Yang. Jiashi Feng, Yao Zhao, al. Global knowledge calibration for fast open- vocabulaay segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 797-807, 2023.
[13] Yong Liu, Sule Bai, Guanbin Li, Yitong Wang, and Yansong Tang. Open-vocabulary seg- mentation with semantic-assisted calibration. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3491-3500, 2024.
[14] Hoon Kim, Minje Jang, Wonjun Yoon, Jisoo Lee, Donghyun Na, and Sanghyun Woo. Switch- light: Co-design of physics -hisicc--rivee architecture and pre-training framework for human portrait relighting. In CVPR, pages 25096-25106, 2024.
[15] Yiqun Mei, He Zhang. Xuaner Zhang, Jianming Zhang, Zhixin Shu, Yilin Wang, Zijun Wei, Shi Yan, HyunJoon Jung, and Vishal M Patel. Lightpainter: interactive portrait relighting with frechand scribble. In CVPR, pages 195-205, 2023.
10

---

[16] Yiqun Mei, Yu Zeng, He Zhang, Zhixin Shu, Xuaner Zhang, Sai Bi, Jianming Zhang, HyunJoon Jung. and Vishal M Patel. Holo-relighting: Controllable volumetric portrait relighting from single image. In CVPR, pages 2263-4273, 2024.
[17] Rohit Pandey, Sergio Orts-Escolano, Chloe Legendre, Christian Haene, Sofien Bouaziz, Christoph Rhemann, Paul E Debevec, and Sean Ryan Fanello. Total relighting: learning to relight portraits for background replacement. ACM Trans. Graph., pages 43-1, 2021.
[18] Zonghui Guo, Haiyong Zheng, Yufeng Jiang, Zhaorui Gu, and Bing Zheng. Intrinsic image harmonization. In Proceedings of the ieee/cvf conference on computer vision and pattern recognition, pages 16367-16376, 2021.
[19] Chris Careaga, S Mahdi H Miangoleh, and Yağız Aksoy. Intrinsic harmonization for illumination-avee aware image compositing. In SIGGRAPH Asia, pages 1-10, 2023.
[20] Julian Jorge Andrade Guerreiro, Mitsuru Nakazawa, and Bjöm Stenger. Pet-net: Full resolution image harmonization using pixel-wise color transformations. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 5917-5926, 2023.
[21] Jianqi Chen, Yilan Zhang, Zhengxia Zou, Keyan Chen, and Zhenwei Shi. Dense pixel-to-pixel harmonization via continuous image representation. IEEE Transactions on Circuits and Systems for Video Technology, 2023.
[22] Ben Xue, Shenghui Ran, Quan Chen, Rongfei Jia. Binqiang Zhao, and Xing Tang. Deef: Deep comprehensible color filter learning framework for high-resolution image harmonization. In ECCV, pages 300-316, 2022.
[23] Wenyan Cong, Xinhao Tao, Li Niu, Jing Liang, Xuesong Gao, Qihao Sun, and Liqing Zhang. High-resolution image harmonization via collaborative dual transformations. In CVPR, pages 18470-18479, 2022.
[24] Haian Jin, Yuan Li, Fujun Luan, Yuanbo Xiangli, Sai Bi, Kai Zhang, Zexiang Xu, Jin Sun, and Noah Snavely. Neural gaffer: Relighting any object via diffusion. arXiv preprini 24Xi:22066.05200 2024.
[25] Chong Zeng, Yue Dong, Pieter Peers, Youkang Kong, Hongzhi Wu, and Xin Tong. Dilightnet: Fine- aene-graindd lighting control for diffusion- based image generation. In SIGGRAPH, pages 1-12, 2024.
[26] Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Scaling in-the-wild training for diffusion- based illumination harmonization and editing by imposing consistent light transport. In ICLR, 2025.
[27] Thomas Nestmeyer, Jean- Françorançois Lalonde, Iain Matthews, and Andreas Lehrmann. Learning physics- soysics-guided face relighting under directional light. In CVPR, pages 5124-5133, 2020.
[28] Puntawat Ponglertnapakorn, Nontawat Tritrong, and Supasorn Suwajanakor. Difareli: Diffu- sion face relighting. In ICCV, pages 26646222677. 2023.
[29] Tiancheng Sun, Jonathan Barron, Yun-Ta Tsai, Zexiang Xu, Xueming Yu, Graham Fyffe, Christoph Rhemann, Jay Busch, Paul Debevec, and Ravi Ramamoorthi. Single image portrait relighting. ACM Transactions on Graphics (TOG), 38(4):1-22 2019.
[30] Longwen Zhang, Qixuan Zhang, Minye Wu, Jingyi Yu, and Lan Xu. Neural video portrait relighting in real-time via consistency modeling. In ICCV, pages 802-812 2021.
[31] Yazhou Xing, Yu Li, Xintao Wang, Ye Zhu, and Qifeng Chen. Composite photograph harmo- nization with complete background cues. In ACM MM, pages 2296-2304, 2022.
[32] Paul Debevec, Tim Hawkins, Chris Tchou, Haarm-Pieter Duiker, Westley Sarokin, and Mark Sagar. Acquiring the reflectance field of human face. In SIGGRAPH, pages 145-156, 2000.
[33] Anand Bhattad and David Forsyth. Cut-and- pattte object insertion by enabling deep image prior for reshading. In 3DV, pages 332-341, 2022.

---

[34] Ke Wang, Michaël Gharbi, He Zhang. Zhihao Xia, and Eli Shechtman. Semi-supervised parametric real-world image harmonization. In CVPR. pages 5927-5936, 2023.
[35] Zonghui Guo, Dongsheng Guo, Haiyong Zheng, Zhaorui Gu, Bing Zheng, and Junyu Dong. Image harmonization with transformer. In ICCV, pages 14870-14879) 2021.
[36] Yifan Jiang. He Zhang, Jianming Zhang, Yilin Wang. Zhe Lin, Kalyan Sunkavalli, Simon Chen, Sohrab Amirghodsi, Sarah Kong, and Zhangyang Wang. Ssh: A self-supervised framework for image harmonization. In ICCV, pages 4832-4841, 2021.
[37] Linfeng Tan, Jiangtong Li, Li Niu, and Liqing Zhang. Deep image harmonization in dual color spaces. In Proceedings of the 31st ACM International Conference on Multimedia, pages 2159-21667, 2023.
[38] Zhanghan Ke, Chunyi Sun, Lei Zhu, Ke Xu, and Rynson WH Lau. Harmonizer: Learning to perform white-box image and video harmonization. In ECCV, pages 690-706, 2022.
[39] Wenyan Cong, Jianfu Zhang, Li Niu, Liu Liu, Zhixin Ling, Weiyuan Li, and Liqing Zhang. Dovenet: Deep image harmonization via domain verification. In CVPR, pages 8394-8403, 2020.
[40] Yi-Hsuan Tsai, Xiaohui Shen, Zhe Lin, Kalyan Sunkavalli, Xin Lu, and Ming Haung Yang. Deep image harmonization. In CVPR, pages 3789-3797, 2017.
[41] Dina Bashkirova, Arijit Ray, Rupayan Mallick, Sarah Adel Bargal, Jianming Zhang, Ranjay Krishna, and Kate Saenko. Lasagna: Layered score distillation for disentangled object relighting. arXiv preprint urXiv:23/2.00833 2023.
[42] Xuan Ju, Xian Liu, Xintao Wang, Yuxuan Bian, Ying Shan, and Qiang Xu. Brushnet: plug-and-play image inpainting model with decomposed dual-branch diffusion. arXiv preprint arXiv:2403. 06976, 2024.
[43] Junhao Zhuang. Yanhong Zeng, Wenran Liu, Chun Yuan, and Kai Chen. A task is worth one word: Learning with task prompts for high-quality versatile image inpainting, arXiv preprint arXiv:2312.03355 12.05994, 2023.
[44] Yong Liu, Cairong Zhang, Yitong Wang, Jiahao Wang, Yujiu Yang, and Yansong Tang. Universal segmentation at arbitrary granularity with language instruction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3459-3469, 2024.
[45] Tim Brooks, Aleksander Holynski, and Alexei A Efros. Instructpix2pix: Learning to follow image editing instructions. In CVPR, pages 1 18392-18402, 2023.
[46] Hu Ye, Jun Zhang, Sibo Liu, Xiao Han, and Wei Yang. Ip-adapter: Text compatible image prompt adapter for text-to-image diffusion models. arXiv preprint rxXi::2308.0671,, 2023.
[47] Mingdeng Cao, Xintao Wang, Zhongang Qi, Ying Shan, Xiaohu Qie, and Yinqiang Zheng. Masactrl: Tuning-free mutual self-attention control for consistent image synthesis and editing. In ICCV, pages 22560-22570, 2023.
[48] Justin Johnson, Alexandre Alahi, and Li Fei-Fei. Perceptual losses for real-time style transfer and super-resolution. In ECCV, pages 114-711,2111
[49] Zhenxiong Tan, Songhua Liu, Xingyi Yang, Qiaochu Xue, and Xinchao Wang. Ominicontrol: Minimal and universal control for diffusion transformer. arXiv preprint arXiv:2411.15098, 2024.
[50] Matt Deitke, Dustin Schwenk, Jordi Salvador, Luca Weihs, Osear Michel, Eli VanderBilt, Ludwig Schmidt, Kiana Ehsani, Aniruddha Kembhavi, and Ali Farhadi. Objaverse: A universe of annotated 3d objects. In CVPR, pages 13142-13153, 2023.
[51] Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion- 5b: An open large-scale dataset for training next generation image-text models. NeurlPS, pages 15278-25294 2022.
12

---

[52] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, 1lge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023.
[53] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. NeurIPS, 2024.
[54] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. NeurIPS, 2019.
[55] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In ICML, pages 8748-88 8763, 2021.
[56] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The un- reasonable effectiveness of deep features as a perceptual metric. CVPR, pages 586-595, 2018.
[57] Jiazheng Xu, Xiao Liu, Yuchen Wu, Yuxuan Tong, Qinkai Li, Ming Ding, Jie Tang, and Yuxiao Dong. Imagereward: Learning and evaluating human preferences for text-to-image generation. NeurlPS, 2024.
[58] Lymin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. In ICCV, pages 3836-3847, 2023.
13

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We clearly provide our motivation and contribution in these parts and the claims accurately reflect them.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the potential limitations in the supplementary materials.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
14

---

We aim to address a practical and applicable task named unified image relighting rather than proposing theoretical results.
Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We have provided our implementation details in the main paper and supple- mentary materials. We will open source our code and models.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
15

---

Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Justification: Codes and models need to be reviewed by organization before they made available. We will make them public as soon as possible.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://niis.cc
Soolic//guides/Codesbbbmsss wtt.:eeewuumiisiioooooooogoo for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ wwtps/cccppclgguuiiiaeeeeec
Saaeee//CoSSSubisss..S.. www.ssssiicgcyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We have claimed the corresponding information the main paper and supple- mentary materials.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: We have demonstrated the generation results of different seeds in the supple- mentary materials.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
16

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We have claimed the compute resources.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics http://ppt/pppppp.......p//a//aa
Answer: [Yes]
Justification:
We have followed the NeurIPS code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- cration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We have discussed the potential broader impacts in the supplementary materi-
als.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
17

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: Our work is under the CC-BY 4.0 license.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
18

---

If assets are released, the license, copyright information, and terms of use in the thcode has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
package should be provided. For popular datasets, paperswi
latasets.paperssi
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
We do not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moode as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: Our work does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: Our work does involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
19

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology. scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification:
We only use LLMs to benefit writing.
Guidelines:
The answer NA means that the method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy battps://neurre http///rriisccc/oooffeereee/e//p Cc/Coonferences/22255/LI. for what should or should be described.
20


Reviews:
[{"review_id": "ecCXwR3MhM", "reviewer": "Reviewer_X4Rx", "summary": "This paper proposes a new method called DreamLight to perform both image-based and text-based relighting, given a foreground image and a background image. Two key components are introduced: the position-guided light adapter and the spectral foreground fixer. The authors also generate their own relighting data for the training of the model. Experiments are conducted on the generated data, and DreamLight is compared against methods like IC-Light.", "questions": "1. The proposed method is evaluated on an author-generated dataset. What's the performance of the method on existing datasets that are used as benchmarks for relighting, such as TensoIR (https://github.com/Haian-Jin/TensoIR)?\n\n2. Related to Q1, most results shown in this paper are portraits. How does the method perform on relighting general objects with different materials?  \n\n3. The design of the direction-biased masked attention is pretty interesting. On the other hand, I don't have a good understanding of why it is working well. Why is it important to have four fixed directions? What happens if you need more complex lighting patterns? Or maybe I'm missing something here. \n\n4. While Table 4 shows the performance trade-off of adding the fixer to the relighting pipeline, what about the time cost? \n\n5. Line 214: The modulator M is trained with pseudo pairs of relighting data. Why not use actual relighting data to train it? \n\n\nOverall, the proposed method has interesting designs to solve a challenging task, while there are some missing pieces in fully understanding the effectiveness of the method. If the authors can address the questions mentioned above, I can raise my score.", "limitations": "No discussion on the limitations.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "### Strengths\n1. The idea of having a unified approach for both image-based and text-based relighting makes sense. The proposed method, DreamLight, seems to be a reasonable solution to this task. \n2. The proposed Position-Guided Light Adapter is a novel design that is tailored to model lighting information in an effective format. See below for additional questions on this part for clarification. \n\n### Weaknesses\n1. One major weakness is the lack of experiments on the existing relighting datasets. Currently, the method is trained and tested on the author-generated data. It will be more convincing if the method is also tested on public benchmarks. \n2. While the Position-Guided Light Adapter is directly designed for the relighting task, the second proposed component, Spectral Foreground Fixer, seems to be less tailored for relighting. For example, some lighting effects are actually high-frequency such as specular reflection. It is unclear if the proposed fixer can handle such a relighting situation. \n3. Some clarification is needed to fully understand the effectiveness of the DreamLight method (see questions below). \n\n### Typo\n- Table 2, 'PP denotes PowerPoint'--it should be PowerPaint.", "was_revised": true, "final_justification": "After reviewing the rebuttal, I'm satisfied with the authors responses. Overall, this is an interesting approach for relighting with some novel designs that can inspire other work. I recommend the acceptance of this paper.", "rebuttal": "We sincerely thank your insightful comments and the appreciation of our work. To the concerns expressed in Weaknesses and Questions, our response is as follows:\n\n### **W1&Q1: Evaluation on public datasets.**\nThanks for your valuable advice! \nOur model and the compared works all rely on pure natural images rather than HDR maps for providing light information.\nTo the best of our knowledge, there are indeed no widely used public benchmarks for relighting based on **purely natural background images**.\nThe benchmark  you recommended (TensorIR) also **relies on HDR panoramas** to provide light information rather than mere 2D natural background images.\n\nTo enable testing on this dataset, we render the corresponding 2D background images of each view from the HDR maps based on the relevant information (FOV, camera extrinsic parameters) provided by the dataset.\nWe calculate the metrics on all test views with the scenarios in \"high_res_envmaps_1k\" folder  and the results are shown in the table below.\nIn fact, we find that the locally rendered background images (just patches in the scenarios) in this way are actually unable to provide sufficient light information (there is a huge gap compared with the information that HDR panorama maps can offer). Therefore, there is a discrepancy between the prediction results of the evaluated models and the ground truth provided by the dataset, leading to generally low overall scores. However, we observe that under such background conditions, previous methods also expose some problems, such as color bleeding of ICLight and weak light changes in INR. Generally, our model performs better in most cases.\n\n\n| Method  | PSNR | SSIM   | LPIPS |\n|--------|--------|--------|--------|\n| INR | 17.61   | 0.683 | 0.327  | \n| PCT | 18.59   | 0.705  | 0.296  | \n| IH |  19.13  | 0.726  | 0.214  | \n| ICLight | 18.45    | 0.684   | 0.236 | \n| Ours | 20.86    | 0.783   | 0.175| \n\n\nFurthermore, we understand your potential concerns, yet our test set is indeed not deliberately cherry-picked. When creating the test set, manual intervention only aimed to ensure diverse scenes (for evaluating generalization ability) and natural ground-truth results, without referring to the outcomes of our model. Additionally, we will make our test set publicly available to benefit the community.\n\n\n\n\n### **Q2: Relighting effect on general objects with different materials.**\nThanks for your great question!\nDue to the restrictions of the official rules for this rebuttal, we are unfortunately unable to provide relevant images of relighting results for demonstration. The reason why the paper mainly presents portraits is that some of the comparison methods are mainly trained on portrait data. Actually, our model has great generalization ability for general objects with different materials. \n\nOn the one hand, as shown in Figure 5 of the supplementary materials, our training data contains, and even includes a large amount of, samples of general objects. Specifically, the ratio of portrait data to non-portrait data is about 3:2, which ensures the effectiveness of our model in relighting general objects.\nOn the other hand, the benchmark (TensoIR) mentioned in Q1 is related to general objects of different materials, and the relevant results demonstrate the superiority of our method over previous approaches in handling general objects. In addition, we have also constructed a new test set consisting of 100 images of pure general objects (without portraits) and compared the performance. The results below also prove the effectiveness of our method.\n\n| Method  | PSNR | SSIM   | LPIPS | CLIP-IS   |\n|--------|--------|--------|--------|-------|\n| INR |  20.87    | 0.722   | 0.227   | 0.881|\n| PCT | 21.48    | 0.733  | 0.204   | 0.893 |\n| IH |     23.29     |  0.797 | 0.175  | 0.896 |\n| ICLight | 23.61  |  0.805 |  0.156  | 0.902 |\n| Ours | 24.57  | 0.816  | 0.132   | 0.917|\n\n### **W2: Spectral Foreground Fixer seems to be less tailored for relighting.**\nThanks for your insightful question. \nSFF is  designed to  preserve details of the foreground subject (such as the identity of small human faces) while minimizing the impact of reorganizing frequency components on relighting effects. \nThus its design also takes into account factors related to relighting effects.\nSpecifically, directly combining the high-frequency components of the input foreground with the low-frequency components of the initial relighting result may cause artifacts and biased lighting in some scenarios.\nTo generate more consistent and natural results, we propose to train a modulator to adaptively adjust different parts. \nThis enables SFF to have good generalization ability for diverse light. \n\nHigh-frequency light effects (such as specular reflections) often appear in regions with low information density (e.g., non-facial regions). Since SFF can adaptively adjust the influence degree of the high-frequency components, it will make more use of the relighting information in such cases, thus achieving desirable results.\nWe have tested such cases where smooth object surfaces are exposed to intense light.\nThe results show that the relighting outcomes are satisfactory, and SFF does not have a negative impact on such scenes.\nHowever, in some cases (such as when there are high-frequency light spots on a small face), SFF may affect the relighting and soften the light effect. This is because  there exists a trade-off between them. Users can control this by adjusting the kernel size of the wavelet transform. How to better improve both aspects simultaneously for such challenging scenarios is also an important direction for our future work.\n\n\n### **W3&Q3: Further explanation about PGLA.**\nThe reason for setting \"up, down, left, right\" light directions is that the light information is provided by a 2D background image. We assume that the light sources in a 2D image are mainly in the horizontal and vertical directions. Therefore, the four directions (up, down, left, right) can form the \"basis\" for the light directions in a 2D image. \n\nBesides, the biased perception of each direction changes continuously (for example, the decay map responsible for leftward light gradually changes from 1 at the far left of the image to 0 at the far right). \nThe foreground also interacts with all basic direction light features under the guidance of position.\nThus, other types of light patterns can be automatically perceived through the combination of information from the up, down, left, and right directions. \nFor instance, a light source in the upper-left corner corresponds to strong perceptions of leftward and upward light. \nWe wonder if this explanation is clearer. If you have any further doubts, we are happy to provide further clarification.\n\n### **W3&Q4: Time cost of SFF.**\nThanks for your valuable advice! \nWe provide the inference cost (FPS and memory utilization) of the proposed modules in the table below.\nAll the value are obtained on a single H20 GPU.\nFor comparison, we have also listed the relevant cost of ICLight.\n|  | Baseline | + PGLA   | + SFF (final model) | IC-Light   |\n|--------|:--------:|:--------:|:--------:|:--------:|\n| FPS |  0.585    | 0.491   | 0.411   | 0.303|\n| Memory (G) |  4.33    | 5.85   | 9.37   | 6.43|\n\nWe will update these results in the manuscripts.\n\n### **W3&Q5: Training data of SFF.**\nThanks for your great question! \nWe do not use actual relighting data because they contain samples where foreground details have changed. Specifically, a large amount of relighting training data is generated by models. Although it undergoes screening, there are still cases where foreground details have altered. Using such data has adverse effects on SFF, causing its learning objectives to deviate. Therefore, we directly use a self-supervised manner to construct pseudo pairs with strongly consistent details for training SFF.\n\n### **Typo**\nThanks! We have carefully checked the manuscript and corrected the typos.\n\nThanks again for your valuable time and comments. We hope that our response can address your questions, and if you still have any concerns, we would be pleased to discuss them further with you."}, {"review_id": "Ojd5aU1MJM", "reviewer": "Reviewer_R5Ls", "summary": "This paper proposes DreamLight, a method for relighting images based on either a text prompt or a background image. It introduces two main components: (1) the **Position-Guided Light Adapter (PGLA)**, a modified IP-Adapter designed to model lighting from four primary directions (left, right, up, down), and (2) the **Spectral Foreground Fixer (SFF)**, which enhances detail preservation by injecting high-frequency components from the original input into the relit result, improving areas like faces or text.", "questions": "## 1. Clarify Modulator Implementation in SFF\n\nIn Line 245, the paper states that SFF is fine-tuned on the VAE of Stable Diffusion 1.5. However, it’s unclear how the Modulator is implemented, given that it is supposed to take two inputs high-frequency foreground ($HQ_{in}$) and low-frequency relighting result ($LQ_{out}$) and output two modulation parameters ($\\alpha$, $\\beta$), whereas the VAE has only one single input and output which a single image.\n\n**Question:** Could you clarify how the Modulator is structured and trained in this context?\n\n## 2. Text Prompt Used in Image-Based Relighting\n\nSince the model is based on Stable Diffusion 1.5, a text prompt is required at inference. However, the paper does not clearly state what prompt is used during image-based relighting. Figure 2 shows “Blend these two images,” but this is not explicitly mentioned in the text.\n\n**Question:** What text prompt is used during inference for image-based relighting, and is it the same as during training? If not, how is the prompt selected, and how sensitive is performance to prompt choice?\n\n## 3. Lack of Reflections or Subject-Light Interactions in Background\nSupplementary Figure 1 shows a dog above a reflective surface (wet floor or water), but no reflection is present.\n\n**Question:** Does the current pipeline, particularly SFF, suppress interactions such as reflections or shadows in the background? Would disabling SFF allow more light or reflection effects to propagate naturally?", "limitations": "The authors addressed some limitations in the supplementary material. However, one additional limitation worth mentioning is that the model does not handle physical interactions between the foreground and background, such as reflections or shadows, adapting to background surfaces. This could limit realism in certain scenarios.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "## Strength: \n- **Quality:**  The method is technically sound and well-executed, with clear contributions\n- **Originality:** Both the Position-Guided Light Adapter (PGLA) and Spectral Foreground Fixer (SFF) present novel ideas tailored to the relighting task.\n- **Significance:** The model effectively preserves foreground identity (e.g., face details), a common weakness in prior work. This makes it a strong candidate for real-world relighting applications and downstream tasks.\n- **Clarity (high-level)**: The overall pipeline is well-structured and easy to follow.\n- **Qualitative result:** The visual comparisons clearly demonstrate improved foreground consistency and lighting harmony over prior methods.\n## Weakness:\n- **Clarity (technical detail):**  Some implementation aspects, such as the implementation of the Modulator in SFF, are underexplained, limiting reproducibility.\n- **Quantitative Evaluation:** The benchmark used for evaluation is internally created. While IC-Light did not release its test set, this raises concerns about potential cherry-picking or a lack of generalization.\n- **Ablation gap:**  The ablation study on PGLA evaluates the impact of the spectral filter but omits a crucial variant: applying the filter without the light queries (i.e., IP-Adapter with filtering). This makes it difficult to disentangle the contributions of each component.", "was_revised": true, "final_justification": "The proposed technique demonstrates clear improvements in preserving fine details during relighting compared to prior methods, which making it worthy of acceptance.\n\nThe authors have adequately addressed my concerns regarding evaluation, ablation studies, and implementation details in their rebuttal. I have no remaining concerns.", "rebuttal": "We sincerely thank your insightful comments and the appreciation of our work. To the concerns expressed in Weaknesses and Questions, our response is as follows:\n\n### **W1&Q1: Implementation details of SFF.**\nThanks for your valuable advice!  Specifically, we combine the high-frequency foreground $HQ _ {in}$ and low-frequency relighting result $LQ _ {out}$ to form a single image as the input. To generate two modulation parameters, we modify the output channels of the last layer of the VAE to 6, and during initialization, the new channels copy the weights of original channels.\nWe will revise the description of this part in the manuscript to make it clearer.\n\n### **W2: Evaluation dataset.**\nThanks for your valuable question! \nWe understand your potential concerns. \nHowever,  there are indeed no widely used public benchmarks for relighting based on purely natural images. \nExisting public test sets are mostly based on environment maps.\nTo enable testing on such dataset, we render the corresponding 2D background images of each view from the HDR panorama maps based on the relevant information (FOV, camera extrinsic parameters) provided by the dataset.\nTake TensoIR as an example, we calculate the metrics on all test views with the scenarios of provided HDR maps  and the results are shown in the table below.\nIn fact, the locally rendered background images (just patches in the scenarios) in this way are actually unable to provide sufficient light information (there is a huge gap compared with the information that HDR panorama maps can offer). Therefore, there is a discrepancy between the prediction results of the evaluated models and the ground truth provided by the dataset. However, we observe that under such background conditions, previous methods also expose some problems, such as color bleeding of ICLight and weak light changes in INR. Generally, our model performs better in most cases.\n\n\n| Method  | PSNR | SSIM   | LPIPS |\n|--------|--------|--------|--------|\n| INR | 17.61   | 0.683 | 0.327  | \n| PCT | 18.59   | 0.705  | 0.296  | \n| IH |  19.13  | 0.726  | 0.214  | \n| ICLight | 18.45    | 0.684   | 0.236 | \n| Ours | 20.86    | 0.783   | 0.175| \n\n\nBesides, our test set is indeed not deliberately cherry-picked. When creating the test set, manual intervention only aimed to ensure diverse scenes (for evaluating generalization ability) and natural ground-truth results, without referring to the outcomes of our model. Additionally, we will make our test set publicly available to benefit the community.\n\n### **W3: Ablation gap.**\nThanks for your helpful suggestion!\nWe have experimented with the ablation of IP-Adapter with filtering.  Due to the restrictions of the official rules for this rebuttal, we are unfortunately unable to provide visual images for demonstration. Therefore, we present the results of quantitative comparisons below and describe the visualization comparison through text. Specifically, as shown in the results in the table below, applying the filter without the light queries (i.e., IP-Adapter with filtering) does not bring significant performance improvement. Visualization results also show that the relighting effects of IP-Adapter with filtering are similar to those of the vanilla IP-Adapter, with no obvious light interaction effects.\nThis is because although filtering is introduced to enhance the representation of high-level light information in the background, IP-Adapter with filtering, like the vanilla IP-Adapter, performs unguided interactions between the subject and the background, resulting in light tending to have a general effect. In contrast, through the utilization of direction-biased masked attention and light queries, our PGLA selectively transmits background lighting information, enabling the foreground to acquire lighting that is harmonious with the background.\n\n| Method  | PSNR | SSIM   | LPIPS | CLIP-IS   |\n|--------|--------|--------|--------|-------|\n| w/o adapter |  18.58   | 0.732   | 0.221   | 0.865 |\n| vanilla IP-Adapter | 20.23  | 0.752   | 0.184    | 0.891 |\n| IP-Adapter with filtering |  20.31 | 0.759 | 0.181    | 0.893 |\n| w/o Filter | 21.81   | 0.778   | 0.162   | 0.903 |\n| PGLA (Ours) | 22.15  | 0.783 | 0.158 | 0.908 |\n\n### **Q2: Text prompt.**\nThanks for your valuable question! As mentioned in L118 in the main paper, the text prompt is set to \"blend these two images\" for image-based relighting. This is the same for both training and inference by default. We will make the corresponding descriptions clearer.\nIn addition, our model also has the generalization ability for other text prompts, such as \"natural light\". However, since \"blend these two images\" is used during training, this prompt yields the best results in most inference cases.\n\n### **Q3: Reflections or subject-light interactions in background.**\nGreat question! In fact, this is somewhat related to the definition of the task. For image-based relighting, we follow the definition of this setting in previous works: only modifying the foreground region to adapt to the background while keeping the background region unchanged. \nThe training objectives and corresponding training data are also organized in this setting.\nHence, in examples like the one in Figure 1 of the supplementary materials you mentioned, there is no reflection of the dog in the background. In contrast, for text-based relighting, since the background is also generated by the model, there are interaction effects between the foreground and background (such as shadows), as demonstrated in the examples in Figure 3 of the supplementary materials. \nOur pipeline and the proposed SFF do not suppress interactions such as reflections or shadows in the background region.\n\nBesides, we believe this question highly insightful. If the background region in image-based relighting could also exhibit reasonable interaction effects, the overall result would be more natural and realistic. However, this also poses additional challenges, such as how to obtain such training data in large quantities. This will be a direction for our further optimization in the future.\n\nThanks again for your valuable time and insightful comments. We hope that our response can address your questions, and if you still have any concerns, we would be pleased to discuss them further with you."}, {"review_id": "qksSO6LD5E", "reviewer": "Reviewer_Vqyj", "summary": "This paper introduces DreamLight, a universal image relighting framework that harmonizes foreground subjects with new backgrounds specified by images or text. It leverages pretrained diffusion models and proposes the Position-Guided Light Adapter (PGLA) to extract directional lighting from the background via masked attention, improving natural light interaction. A Spectral Foreground Fixer (SFF) further enhances appearance consistency.", "questions": "- Could you please clarify how your model performs in scenes with more than four dominant light directions? Are there any plans to extend the PGLA mechanism to support more flexible or adaptive directional queries?\n\n- How sensitive is the Spectral Foreground Fixer to the quality and realism of its synthetic training data? Have you evaluated its effectiveness on real-world relighting scenarios?\n\n- Would you be able to provide more details on the runtime performance, memory requirements, and inference speed of your method compared to existing baselines?\n\n- Have you encountered any limitations or failure cases when the background contains strong lighting contrasts or complex shadows? If so, how does your method address these challenges?", "limitations": "See Questions", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths: \n- Writing is clear and figures support the technical explanations. The Position-Guided Light Adapter (PGLA) introduces a novel direction-biased masked attention mechanism that explicitly models multi-directional lighting cues, improving relighting quality. The Spectral Foreground Fixer (SFF) effectively addresses foreground distortion by adaptively combining high-frequency foreground details with low-frequency lighting from the background, enhancing visual fidelity.\n\n\nWeaknesses: \n- The directional light queries are limited to four fixed directions, which may restrict the model’s ability to handle complex lighting environments with multiple arbitrary light sources.\n- The SFF training relies on synthetic color transformations that may not fully capture real-world lighting variations; additional evaluation on real lighting conditions would strengthen the claims.\n- Computational cost and inference speed are not thoroughly discussed, which matters for practical deployment.\n- Some ablations and deeper analysis on the relative importance of PGLA components and SFF would improve clarity.", "was_revised": true, "final_justification": "Thanks for the response. The rebuttal addressed my concerns！", "rebuttal": "We sincerely thank your insightful comments and the appreciation of our work. To the concerns expressed in Weaknesses and Questions, our response is as follows:\n\n### **W1&Q1: Light queries**\nThanks for your valuable question!\nWe set the light directions as \"up, down, left, right\" because the lighting information is derived from a 2D background image. In our view, the light sources in a 2D image tend to align with horizontal and vertical axes. Therefore, **these four directions serve as a natural \"basis\" for representing lighting orientations in 2D space**. Since light queries are high-dimensional features and there are multiple queries responsible for each direction (by default, 4 per direction, totaling 16), they are capable of representing diverse light information. \nBesides, the biased perception of each direction changes continuously (for example, the decay map responsible for leftward light gradually changes from 1 at the far left of the image to 0 at the far right). \nDirection-biased masked attention is a soft constraint rather than a hard distinction. \nHigh-dimensional light queries still have the ability to perceive multiple light sources and the foreground also interacts with all basic direction light features under the guidance of position. As a result, **more complex lighting patterns can be captured by organizing signals from these four base directions**.\nTo prove that, we construct images featuring colors in 6 different directions as conditional background images for testing. Due to the restrictions of the official rules for this rebuttal, we are unfortunately unable to provide the generated results for illustration. We observe that the output relighted results can perceive such complex light source.\n\n\nWe hope this explanation offers greater clarity. If you have any remaining questions, we would be glad to elaborate further.\nWe will also refine the corresponding representations in the manuscript to make it clearer.\nAs for extending the PGLA mechanism to support more flexible or adaptive directional queries, we consider this an interesting direction for exploration. It may contribute to more significant and natural light interaction effects in complex scenarios. However, how to enable them to obtain ideal information is quite challenging, which will be one of the directions for our future work. Thanks!\n\n### **W2&Q2: Training set and evaluation of SFF**\nThanks for your great question!\nSFF is designed to preserve details of the foreground subject (such as the identity of small human faces).\nActually, it can also be trained using real relighting data. Theoretically, if real relighting data has very strong subject consistency, SFF trained on it would achieve good performance.\nHowever, we do not use actual relighting data because they may contain samples where foreground details have changed. Specifically, a large amount of relighting training data is generated by models. Although we have utilized VLMs to filter low-quality data, there are still cases where foreground details have altered. Using such data for training may have adverse effects on SFF. Therefore, we directly use a self-supervised manner (color transformations) to construct pseudo pairs with strongly consistent details for training SFF and find it works well.\n\nAll our evaluation of SFF are conducted on real relighting scenarios. On the one hand, the quantitative evaluations in the paper are all performed on real relighted images. The demonstration figures in the paper (both image-based and text-based ones) are also results processed by SFF, which demonstrates that SFF can capture real light. On the other hand, we have also tested the relighting effects of images from real-world daily scenes, and SFF can maintain the details of the subject without compromising reasonable relighting effects.\n\n### **W3&Q3: Computational cost and inference speed.**\nThanks for your valuable advice! \nWe provide the inference cost (FPS and memory utilization) of the proposed modules in the table below.\nAll the value are obtained on a single H20 GPU.\nFor comparison, we have also listed the relevant cost of ICLight.\n|  | Baseline | + PGLA   | + SFF (final model) | IC-Light   |\n|--------|:--------:|:--------:|:--------:|:--------:|\n| FPS |  0.585    | 0.491   | 0.411   | 0.303|\n| Memory (G) |  4.33    | 5.85   | 9.37   | 6.43|\n\nMoreover, to further demonstrate the inference latency of different parts of our model, we have counted the inference time of each module, and the results are as follows. \n\n|  | Matting | VAE  | PGLA | Diffusion   | Fixer (SFF)   |\n|--------|:--------:|:--------:|:--------:|:--------:|:--------:|\n| Time (s) |  0.32   | 0.29   | 0.92   | 0.31| 0.39|\n\nWe will update these results in the manuscripts.\n\n### **W4: Relative importance of PGLA and SFF.**\nPGLA and SFF have entirely different design purposes and functions. Specifically, PGLA is intended to enable the model to generate more reasonable light interaction effects in image-based relighting scenarios (serving to enhance relighting effects). In contrast, the role of SFF is to maintain the consistency of the subject, especially in regions with high information density such as small human faces (serving for ID preservation). \nThey function in different aspects, so there is no comparison of which is more important. For instance, if only PGLA is used without SFF, the output image would have good relighting effects but the human face may be distorted. Conversely, if only SFF is utilized without PGLA, the model may produce results where foreground details are well preserved but the light effects are unreasonable.\n\n### **Q4: Discussion about failure cases.**\nTypically, thanks to the PGLA module, which enables reasonable light interactions between the foreground and background, our model achieves great relighting effects even in scenes with strong lighting and shadows. However, there are indeed some failure cases in extremely complex scenarios. For example, when sunlight shines through multiple gaps in tree canopies, our model may struggle to generate all fully plausible light spots. \nWe consider these scenarios to be highly challenging, and incorporating 3D-related representations might help address such cases. This will be one of the directions for our future improvements.\n\nThanks again for your valuable time and insightful comments. We hope that our response can address your questions, and if you still have any concerns, we would be pleased to discuss them further with you."}, {"review_id": "8yMoRf147u", "reviewer": "Reviewer_6h7r", "summary": "This paper proposes an image relighting method called DreamLight, which supports both image-based and text-based relighting. To generate more natural lighting effects between the foreground and background, the authors introduce a position-guided lighting adapter. Additionally, to avoid detail loss during VAE encoding, a spectral foreground fixer is proposed to preserve high-frequency details of foreground textures. However, some technical details require further explanation and experimental evaluation.", "questions": "My main concerns are summarized in Weaknesses Section. Considering some issues with the spectral foreground fixer and the confusion brought by ablation experiments, my initial rating to this paper is borderline reject. While I will consider my final rating according the feedback of the authors during the rebuttal section.", "limitations": "The limitations are kindly discussed in the paper.", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths:\n1. DreamLight supports both image-based relighting and text-based relighting.\n2. A direction-biased mask attention mechanism is proposed to extract illumination from the four background directions (up, down, left, right), thereby achieving more accurate foreground relighting.\n3. Qualitative and quantitative experimental results demonstrate that the proposed method achieves more natural and continuous lighting effects.\n\nWeaknesses:\n1. I noticed that the image-based relighting exhibits some incorrect shadow effects. Is it because the spectral foreground corrector directly replaces the foreground region with I_out, weakening the shadow interaction between the foreground and background?\n2. The training data for the spectral foreground fixer is obtained using random color transformations, which significantly differ from actual lighting variations. Although ablation experiments demonstrate the module's effectiveness, what is the actual distribution of the predicted alpha and beta? Visual ablation experiments are needed to prove its specific role.\n3. Why does the spectral foreground fixer predict beta parameters and directly add them to the high-frequency components? Its actual function and physical meaning are unclear.\n4. After removing the spectral foreground fixer module, metrics such as PSNR were lower than IC-Light and IH. This is strange because the position-guided light adapter should better extract illumination information from four directions, and the performance metrics should be higher than these baselines. The authors need to further explain the reason.\n5. The inference latency should be experimentally tested.", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank your insightful comments and valuable feedback. To the concerns expressed in Weaknesses and Questions, our response is as follows:\n\n### **W1: The potential effect of SFF.**\nThanks for your kind question. \nSince high-frequency components are primarily related to the detailed textures of the subject, the modifications made by SFF to high-frequency components are expected not to affect effects such as shadows. \nThis is based on the assumption that the high-frequency components of an image correspond to the pixels varying drastically, such as object boundaries and textures, while the low-frequency components correspond to the high-level semantic information such as the color and light.\nCould you kindly describe what the \"incorrect shadow effects\" are? We are a bit confused about where the issue lies.\nTypically, our model can produce desirable light interaction effects in diverse scenarios.\nThis is also confirmed by non-cherry-picked user study and extensive evaluations experiments. \nCertainly, there exists failure relighting cases in some complex scenarios for our model, but it may be unrelated to SFF. It is mainly because the initial relighting results generated by the diffusion model may have errors. \nWe believe constructing more high-quality data in complex scenarios can further enhance the performance of our model.\n\n### **W2: Discussion about SFF.**\nSFF is designed to preserve details of the foreground subject (such as the identity of small human faces). Actually, it can also be trained using real relighting data. Theoretically, if real relighting data has very strong subject consistency, SFF trained on it would achieve good performance. However, we do not use actual relighting data because they may contain samples where foreground details have changed. Specifically, a large amount of relighting training data is generated by models. Although we have utilized VLMs to filter low-quality data, there are still cases where foreground details have altered. Using such data for training may have adverse effects on SFF. Therefore, we directly use a self-supervised manner (color transformations) to construct pseudo relighting pairs with strongly consistent details for training SFF and find it works well.\n\nFollowing your kind suggestion, we print the numerical distributions of $\\alpha$ and $\\beta$ and compare them with the high-frequency part. \nDue to the restrictions of the official rules for this rebuttal, we are unfortunately unable to provide the distribution visualization images of these parameters for demonstration.\nTherefore, we present the statistical data of their distributions in the table below. \n$HQ _ {in}$ and $HQ' _ {in}$ denote the input and the adjusted \nhigh-frequency component, respectively. $HQ' _ {in} = HQ _ {in}*\\alpha + \\beta$. Mean, Min, and Max indicate the distribution of each parameter.  It can be seen that $\\alpha$ and $\\beta$ work together to adjust the distribution of high-frequency components, so as to adaptively enhance the details of the subject without affecting the relighting effects.\nVisual ablations of Figure 8 in the paper also illustrates the effectiveness of our SFF.\n\n|  | Mean | Min   | Max |\n|--------|--------|--------|--------|\n| $\\alpha$ |  0.87  | 0.38  | 1.20 |\n|$\\beta$ | 0.001 | -0.16  | 0.21   |\n|$HQ _ {in}$ | -0.001 | -0.81  | 1.04   |\n|$HQ'_ {in}$ | 0.001 | -0.75  | 1.14   |\n\n\n### **W3: The meaning of $\\beta$.**\nThe design philosophy behind our SFF being engineered to output both $\\alpha$ and $\\beta$ is inspired by neural network layers. Since neural network layers often contain weights and biases, we intuitively adopted two predictive parameters and defined: $HQ' _ {in} = HQ _ {in}*\\alpha + \\beta$. Ideally, $\\alpha$ is utilized to control the influence degree of initial high-frequency components and $\\beta$ contains the adjustment information.\nBesides, the modulator is somewhat like a style transferrer, and the commonly used AdaLN operation also has scaling coefficients (our $\\alpha$) and offset coefficients (our $beta$).\nWe assume that these commonly used operations contain two parameters is because such approach can facilitate model learning.\n\n### **W4: Performance metrics confusion.**\nThere appears to be some misunderstanding here. The values you compared are likely from Table 4 (the ablation study on SFF) and Table 1 (the main results) in the paper. However, their test settings and metric calculation are actually different. As described in the paper from L304 to L307: \"SFF primarily works on critical small regions. While the refinement of these regions is important for visual quality, it has limited impact on global metrics. Thus, to better test the refinement effect, in Table 4 we crop small face regions for metric calculation.\"\n\nActually, when calculating the metrics for the entire image, the performance of the model without SFF (using only PGLA) is numerically close to that of the final version and significantly outperforms previous methods. The corresponding results are shown in the table below.\n\n| Method  | PSNR | SSIM   | LPIPS | CLIP-IS   |\n|--------|--------|--------|--------|-------|\n| INR |  16.79  | 0.694  | 0.231 | 0.872 |\n| PCT | 18.45 | 0.714  | 0.206   | 0.895 |\n| IH |  20.66  | 0.762  | 0.177  | 0.896 |\n| ICLight | 20.27  | 0.771 | 0.181  | 0.889 |\n| Ours w/o SFF | 22.09    | 0.781   | 0.162    | 0.907 |\n| Ours | 22.15  | 0.783  | 0.158  | 0.908 |\n\n### **W5: Inference latency.**\nThanks for your valuable advice! \nWe provide the inference cost (FPS and memory utilization) of the proposed modules in the table below.\nAll the value are obtained on a single H20 GPU.\nFor comparison, we have also listed the relevant cost of ICLight.\n|  | Baseline | + PGLA   | + SFF (final model) | IC-Light   |\n|--------|:--------:|:--------:|:--------:|:--------:|\n| FPS |  0.585    | 0.491   | 0.411   | 0.303|\n| Memory (G) |  4.33    | 5.85   | 9.37   | 6.43|\n\nMoreover, to further demonstrate the inference latency of different parts of our model, we have counted the inference time of each module, and the results are as follows. \n\n|  | Matting | VAE  | PGLA | Diffusion   | Fixer (SFF)   |\n|--------|:--------:|:--------:|:--------:|:--------:|:--------:|\n| Time (s) |  0.32   | 0.29   | 0.92   | 0.31| 0.39|\n\nWe will update these results in the manuscripts.\n\nThanks again for your valuable time and insightful comments. We hope that our response can address your questions, and if you still have any concerns, we would be pleased to discuss them further with you."}]

Author rebuttal:

Meta-review: The paper proposes an approach for relighting objects or portraits given a text prompt or background image. To that end it introduces new modules for extracting directed lighting information from the background image and for refining the foreground of the harmonized image.

The reviewers appreciated the strong experimental results, clear writing, and novelty of the proposed components. They were concerned about missing experiments on existing datasets, on objects, and to further ablate design choices. For the refinement module, they noted missing implementation details, an intuitive explanation on its functioning, and questioned the use of synthetic data for training.
Results and technical details provided in the rebuttal and discussion addressed these concerns sufficiently and swayed 3 of the reviewers towards accepting the paper. The remaining reviewer with a borderline reject did not respond to the rebuttal and discussion, but the AC concludes that the rebuttal sufficiently addresses the raised questions. The AC agrees with the recommendation of the reviewers.
Decision: Accept (poster)