UniLumos: Fast and Unified Image and Video Relighting with Physics-Plausible Feedback
Authors: Pengwei Liu
Hangjie Yuan
Bo Dong
Jiazheng Xing
Jinwang Wang
Rui Zhao
Weihua Chen
Fan Wang
Venue: NeurIPS 2025 poster ()
Keywords: video generation
illumination editing
video editing
Primary area: deep_learning

Abstract: Relighting is a crucial task with both practical demand and artistic value, and recent diffusion models have shown strong potential by enabling rich and controllable lighting effects. However, as they are typically optimized in semantic latent space, where proximity does not guarantee physical correctness in visual space, they often produce unrealistic results—such as overexposed highlights, misaligned shadows, and incorrect occlusions. We address this with **UniLumos**, a unified relighting framework for both images and videos that brings RGB-space geometry feedback into a flow-matching backbone. By supervising the model with depth and normal maps extracted from its outputs, we explicitly align lighting effects with the scene structure, enhancing physical plausibility. Nevertheless, this feedback requires high-quality outputs for supervision in visual space, making standard multi-step denoising computationally expensive. To mitigate this, we employ path consistency learning, allowing supervision to remain effective even under few-step training regimes. To enable fine-grained relighting control and supervision, we design a structured six-dimensional annotation protocol capturing core illumination attributes.  Building upon this, we propose **LumosBench**, a disentangled attribute-level benchmark that evaluates lighting controllability via large vision-language models, enabling automatic and interpretable assessment of relighting precision across individual dimensions. Extensive experiments demonstrate that UniLumos achieves state-of-the-art relighting quality with significantly improved physical consistency, while delivering a 20x speedup for both image and video relighting. Code is available at https://github.com/alibaba-damo-academy/Lumos-Custom.

UniLumos: Fast and Unified Image and Video Relighting with Physics-Plausible Feedback
Pengwei Li,, Hangjie Yuan fuanf2.3.1., f2,3.1,* Bo Dong Jiazheng Xing 1.2g 12244
Jinwang Wang2.311 Rui Zhao*, Weihua Chen f2.3, Fan Wang*
"Zeejiang University. DAMO Academy, Alibaba Group. "Hupan Lab,
*National University of Singapore
Equal contributions. Corresponding author.
(yuanhangjie.yh) kugang. sugang.cwh)@alll
Abstract
Relighting is crucial task with both practical demand and artistic value, and recent diffusion models have shown strong potential by enabling rich and controllable lighting effects. However, as they are typically optimized in semantic latent space, where proximity does not guarantee physical correctness in visual space, they often produce unrealistic results-such as overexposed highlights, misaligned shadows, and incorrect occlusions. We address this with UniLumos, a unified relighting framework for both images and videos that brings RGB-space geometry feedback into flow-matching backbone. By supervising the model with depth and explicitly align lighting effects with
normal maps extracted from its outputs, the scene structure, enhancing physical plausibility. Nevertheless, this feedback requires high-quality outputs for supervision in visual space, making standard multi-step denoising computationally expensive. To mitigate this, we employ path consistency learning, allowing supervision to remain effective training regimes. To enable fine-grained relighting control and supervision, we design structured six-dimensional annotation protocol capturing core illumination attributes. Building upon this, we propose LumosBench, a disentangled attribute- level benchmark that evaluates lighting controllability via large llnguae models. enabling automatic and interpretable assessment of relighting precision across individual dimensions. Extensive experiments demonstrate that UniLumos achieves state-of-the th-t relighting quality with significantly improved physical sical consistency, while delivering 20x speedup for both image and video relighting. Code is available at httpp://githhbb.co com waliibbb-ddmooo damo-aamooaooo daw.-academyllleeoeaeaaaa Siaieemyy/mooo-usstttee
under few-step
Introudction
Relighting, altering illumination in images or videos while preserving intrinsic scene attributes such as geometry, reflectance, and content, is a longstanding problem in computer vision and graphics 33 (55). It underpins wide range of applications in film production, gaming, and augmented reality, where seamless lighting integration critical to visual fidelity. Beyond realism, lighting conveys rich aesthetic and semantic cues-it defines atmosphere, evokes emotion, and reinforces narrative structure. Relighting is thus not only a technical challenge but also a creative tool that shapes how characters, objects, and environments are perceived. However, achieving physically consistent lighting remains a core challenge- requiring the alignment illumination effects with different scene attributes across both space and time. To address this, traditional approaches often rely inverse rendering pipelines [46 [50 that estimate intrinsic scene properties, such as geometry, reflectance, and environmental lighting, from input images. While these methods provide physically grounded
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Inpul
Relight
Input
Input
Rolight
Text Conditonee Ralighting
Light,
Dynamic,
Neutral, Transmizsion
Reference Background Video
Input
Background Conditionne Relighing
Frame
Frame
Frome
Frame
Figure I: UniLumos performs physically plausible image and video relighting, conditioned textual prompts and reference videos.
results, they typically require complex imputs-such such as high dynamic range images or spherical harmonics coefficients and limited to constrained domains. This makes them impractical for real-world applications, where users often provide only single image, short video, or a high-level lighting prompt as input. These limitations underscore the need for paradigm-one that can deliver high-quality, physically plausible relighting while operating under minimal and naturalistic input conditions.
Recent diffusion- relighting methods [49 6, 12 57 have shown promise by leveraging large- scale image and video datasets produce diverse and controllable lighting effeets under various user-defined conditions, such as reference images or text prompts. However, this strength reveals a fundamental weakness: diffusion models typically operate in semantic latent space, where similarity does not guarantee physical correctness in the visual domain. As a result, they often fail to respect scene geometry, especially in complex scenes with dynamic lighting or temporal constraints. For example, IC-Light [49 and SynthLigght (6], which are primarily designed for image relighting, lack both temporal modeling and explicit physical supervision in the visual domain. Instead, they rely on latent-space representations, such as MI M-.--bseed emheddings (IC-Light) or multi-stage training (SynthLight), which often lead to artifacts like misaligned shadows. overexposed highlights, or incorrect lighting directions- -aaioons--rttillarr.l under complex geometry or extreme illumination. Light- A-Video [12] and RelightVid [12] extend these methods to the video domain, aiming to improve temporal coherence while retaining the visual quality of diffusion-based relighting. Light-A- Video traiming-fre free framework that combines IC-Light with pre-trained video diffusion model via iterative alignment. While this improves frame-wise consistency, it incurs high inference costs due to multiple model passes. RelightVid adopts a joint training strategy with video diffusion backbone, which enhances temporal stability compared to training-fres approaches. However, it still operates without explicit physical supervision, resulting in inaccurate light-scene interactions and limited generalization to complex dynamic environments. In short, existing eeffusiin-bbseee methods

---

excel in synthesizing plausible appearances but fall short in enforeing the physical plausibility that is essential for high-quality relighting.
To bridge the gap between generative flexibility and physical correctness, we propose UniLuums, a unified relighting framework for both images and videos that brings RGB-sppace geometry feedback approaches that operate purely in
into a flowmmatching backbone. Unlike existing diffusion-basee latent space, UniLumos introduces a physics-plausible feedback mechanism that supervises generation with dense geometric signals-speccficall) depth and surface normals-estimated from its outputs. These lighting- invariant cues serve as ideal supervision signals, enabling the model explicitly align illumination with the scene structure, significantly improving shadow alignment, shading consistency, and spatial coherence. Nevertheless, this feedback requires high-quality outputs for supervision in visual space, making standard multi-step denoising computationally expensive. To mitigate this, we employ path consistency learning 13 allowing supervision to remain effective even under few-step training regimes.
lightinginvariant
Beyond model-level improvements, existing relighting methods lack structured illumination descrip- tions and dedicated evaluation metrics. Generic generation scores (e.g., FID, LPIPS) fail to capture lighting-specific errors such as shadow misalignment, intensity mismatch, or incorrect light direction. To address this, we introduce LumosData, a scalable data pipeline that extracts diverse relighting pairs from real-world videos. At its core a structured tix-dimensional annotation protocol covering direction, light source type, intensity. color temperature, temporal dynamics, and optical phenom- enabling both fine-graing fne-grained conditioning during training and physically grounded evaluation at test time. Building upon this, we propose LumosBench, a disentangled attribute-level benchmark that evaluates lighting controllability via large vision- lasion-lagguage models, enabling automatic and interpretable assessment of relighting precision across individual dimensions.
Our contributions are summarized follows:
Unified Relighting with Physics-Plausibbe Feedback: We propose UniLumos, a unified relighting framework for both images and videos that incorporates "GB-space geometry feedback into a flow-matching backbone, explicitly aligning lighting effects with the scene structure to enhance the physical plausibility of relighting.
Structured Illumination Annotation and Evaluation Benchmark: We design structured six-dimensional annotation protocol that captures core illumination attributes, enabling fine-grained control and supervision. Building upon this, we introduce LumosBench, a disentangled attribute- benchmark that leverages large vision-languag, models to automatically and interpretably evaluate relighting controllability across individual lighting dimensions.
Extensive Validation: Extensive experiments demonstrate that Unil Jumos achieves state-of- the-art relighting quality with significantly improved physical consistency, while delivering 20x speedup for both image and video relighting.
2
Related Work
Video Diffusion Models. Recent advances in video diffusion models B 4 7 38
have enabled the generation of temporally coherent video sequences conditioned on various inputs such text or images. In the field of text-to-video (T2V) generation [45) most methods extend existing text-to- image diffusion backbones with additional modules that capture temporal dynamics across frames. In contrast, a few approaches train video diffusion models from scratch to directly learn spatiotemporal priors [38]. For imagc-to-video (I2V) tasks, where static images are animated with plausible motion, several methods propose specialized architectures tailored for image animation [35 40). Other
strategies offer lightweight, plug-and-pla. adapters that can be integrated into pre-trained models. Stable Video Diffusion (2], for example, fine-tunes T2V models for I2V tasks, achieving state-of- the-art performance. Beyond synthesis quality, a growing body of work emphasizes controllability, allowing users to guide generation with fine- mnn-ggai constraints [2 48).
Relighting Methods. Recent advances in deep neural networks have significantly improved lighting control for 2D and 3D visual content, especially in portrait relighting. Methods such as Relightful Harmonization 32. SwitchLight (2I), ConceptSliders [14), Intrinsic Image Diffusion (22), Neural
Gaffer (20), Di-hight 44, SynthLight [6], and IC-Light [9] demonstrate progress in realism and
3

---

controllability. While numerous portrait relighting approaches exist [31 47 42 5] most of them rely heavily on portrait-specific priors. In contrast, UniLumos is designed as general purppos relighting framework that is not constrained to any particular object category. With the rise of enffusiio-bassed generative models, approaches like LumiSculpt [S1] extend lighting control to text-to- -iiieo (T2V) generation. Moreover, RelightVid [2] and Light-A-Viden [57] implemented video relighting based on IC-Light. However, achieving both precise lighting control and high visual quality in video remains challenging due to the trade-off between spatial realism and temporal consistency.
Feedback Learning in Generative Models. Feedback learning has become a powerful tool to improve output alignment in generative models, from language systems trained with human prefer- ences [6 30] to visual diffusion models guided by aesthetic or attribute-basee rewards 43) 25).
e.g., Instruct Video [3 and DRaFT [10]. In visual domains, feedback can also be physical for example, using geometric cues to guide generation toward realism. Recent advances distillation and consistency training 15
[ have accelerated diffusion inference by reducing the number of denoising steps, enabling models to recover high-quality RGB outputs with just few iterations. However, most existing techniques focus on appearance synthesis and overlook geometry-aware feedback, which typically requires high-fidelity outputs and is incompatible with few-step inference. UniLumos bridges this gap by combining physically grounded supervision with path-consistency learning, enabling efficient and physically plausible relighting under fast sampling regimes.
34 29
39
53
Preliminaries
Given an image or video S E RRTxHxxC with intrinsic scene properties (e.g.. geometry, reflectance, content) under initial illumination L1, the goal of relighting is to modify the illumination within a subject region specified by a binary mask M E {0, 1] 1]7"Xxx XW to match a
Problem Formulation.
target lighting condition C. The condition C may take the form of an image, video, or text description and implicitly defines a desired illumination field L2. The relit output S2 E R H xC' should exhibit lighting consistent with Lg in the masked region M while preserving the intrinsic attributes of Sj. This can formulated as a conditional generation problem:
S = (e(S,, C,M),
where fo is the relighting model parameterized by 0.
Flow Matching. To efficiently model complex illumination transformations in relighting, build upon Wan2.1 [38], a foundation video generation model based on flow matching [28] II. Flow
matching formulates generative modeling learning velocity fields between noise xi and data v1 using linear interpolation:
N)0,1)
dx.
21 = t. 21 + (1 - Fo.
= 21 Fu.
(2)
df
where E [0, 1] sampled from logit-normal distribution. The model learns to predict it from Ttz conditioned on timestep and context (e.g., text embeddings), by minimizing the mean squared error:
Co = Ex,. Elvo t, c) - v. v211..
(3)
Path Consistency Learning. To further accelerate inference, we adopt path consistency learning [3] which encourages consistent velocity predictions under larger integration steps. Given velocity field Ug and step size d >> 0, we recursively define:
Fttd xt + d. vo volx,, (,d),
(4)
Moreover, enforce two-step consistency using:
Cistt Ad (o 2d)
juo t, + volxx4a d, d,d]] d))
(5)
This objective enables the model to learn shorteut-consistent velocity fields without separate teacher- student stages, allowing for fast and high-quality generation with arbitrary step budgets.

---

LumosData
UniLumos
Yau E RRMWA
R RoT
Frea peaaee
Noise
Mask
Was-WAE
Was-WE Encoder
Fred RTTAAA
M af
Ray
Sby
Xa
Step2
Step3
Background
Concal
Add
New Capfion
Timeolup
un15
N DIT Blocks
Concal
Desired-atep
Vav, BI
e TuTuuat
Foedbok
Decader
Frozen Weights
Original Copiton moo pland shot
Stepd Caption Augmentation
Vraat gitesweo
Trainable Weights
light
Botoss
Aol Laght cight up tiget Nocr
Visin loegange (VLM)
2.5VL
Light Source lype
Readiing Eight
Plusicsluuiible Feedback
lggtissentty
"Gler 1-100 hameey Mosttrar 2200 Lonenal (1:00)
predroined
Takks Asslyas the
Desss extination
Tespertace
"Lock, Nentral
(e.g. Lohs)
pos plaea skit
Chorges
besiground.
Anbooriiat Medioas, Nestra seet
Opéicll
Dapth Map + (Row Safler, Moowr Byfori bytorss
Nornal Map gir--ra
(itmmu Juatoray dhe
D D:RR
The overall pipeline of UniLumos. The left is LumosData. proposed data construction pipeline, which consists of four stages for generating diverse relighting pairs from real-world sources. The right shows the architecture of UniLumos, unified framework for image and video relighting, designed to achieve physically plausible illumination control.
Figure 2:
Methodology
We present UniLumos, unified framework for physically plausible image and video relighting, as illustrated in Fig. 2 Built upon Wan 2.1 [38]. a flow-matn now-matching diffusion model for video generation, UniLumos relights images and videos under user-specified lighting conditions cuddttons-iinluuinng inncludin reference images, video clips, or text prompts-whin while preserving scene content and temporal coherence.
To bridge the gap between semantic generation and physical correctness, UniLumos incorporates two key innovations: (1) physics-plausible feedback that supervises the model with geometry signals from RGB space, and (2) structured illumination annotation protocol that enables fine- grained control and evaluation. We jointly train the model with geometry-awan supervision and lighting-conditioned objectives, achieving high-quality and efficient few-step inference.
4.1 Physics-Plauue Plausible Feedback
While most relighting methods rely photometric reconstruction or latent-space consistency, such signals offer limited geometric grounding- often often resulting in misaligned shadows, implausible shad- ing, and incorrect light directions. To address this, UniLumos enforces consistency between generated illumination and underlying scene geometry, promoting more realistic light-scent scene interactions.
As illustrated in Fig. [ (right), we introduce physics-plausible feedback that guides the generation process using geometry-aware supervision. This component complements the flow-matching archi- tecture with explicit structural priors, enhancing physical plausibility without altering the model's inference inputs. We adopt depth and surface normals as our supervisory targets due to their general- ity, accessibility, and strong disentanglement from illumination. Unlike shadow masks material properties, which are often ambiguous, entangled with lighting, or costly to obtain, monocular depth and normal maps capture intrinsic scene structure. They be reliably estimated by pre-trained dense estimator (e.g., Lotus [77).
Specifically, after decoding the predicted latent variable into RGB frames via the Wan-VAE De- x
coder [38] extract estimated depth D e xx and normals Ñ xxxxxW using frozen
5

---

dense estimator. These are compared against pseudo sorroo-groun--oo -aruun maps (D,N) from the reference input to compute the geometry- aware feedback loss:
| D
D-12122
Nll2 IN N l
)
phy = Ezo.z1df.
(6)
where M RTxHxx denotes the foreground subject mask. This feedback encourages the model to align its lighting predictions with consistent structural interpretation while keeping inference lightweight and geometry-free.
However, the proposed Ahysics-plausible feedback requires supervision in the RGB domain, which relies on high-quality predictions that typically only available after full-step denoising has been completed. This poses a major computational bottleneck for standard diffusion models. To mitigate this, adopt path consistency learning [3] which reformulates denoising velocity regression task, thereby supporting practical training under few-step regimes. Enforcing consistency between intermediate outputs and final predictions enables reliable geometric feedback without sacrificing inference efficiency.
4.2 Structured Illumination Annotation and Evaluation Benchmark
In the problem formulation, the relighting task involves conditioning on target illumination descriptor C. However, most prior work treats C as unstructured prompts-such as text, images, or reference frames offering limited control interpretability. Moreover, conventional evaluation metrics, such as FID or LPIPS, focus on perceptual similarity but fail to capture lighting-spectin discrepancies, such shadow misalignment or intensity mismatches.
To address this, we construct LumosData, scalable dataset pipeline that enriches C with structured lighting semantics. As shown in Fig. (left), we extract relighting pairs from real-world videos. Given an imput sequence Vieat E [1,+I,,, we first obtain subject masks M E {0, 1I.
using BiRefNet [54] isolate the foreground. We then apply a pre-trained relighting model such as IC-Light [49 to generate synthetic relit versions V deg under diverse lighting conditions, guided by curated prompt set. To avoid entanglement with background semantics, we inpaint the background
using Gaussian noise, ensuring clean illumination signals without introducing artifacts.
Beyond this relighting pipeline, LumosData introduces structured six-dimensional annotation protocol that covers direction, intensity, color temperature, light source type, temporal dynamics, and optical phenomena.
These attributes are automatically generated using vision-language models (e.g.. Qwen2. 2-LVL (I) with carefully designed prompts, and integrated into C to provide enriched semantic label. This protocol serves dual purposes: (1) Fine- Fine-grained conditioning: During training, the model is guided by explicit lighting attributes embedded in C, promoting more interpretable and controllable generation across scenarios. (2) Attribute-alip aliigne Benchmark:
construct L LumosBench. This automatic benchmark Building upon the same attribute protocol, we vision-language models to assess whether generated outputs accurately reflect intended lighting conditions, enabling interpretable, attribute-level controllability evaluation beyond pixel-based metrics.
Each training tuple structured as (V Vbg. M, C) + Vieal, where the model learns to restore realistic lighting consistent with the semantic cue C and structural context. LumosData introduces rich diversity in content and illumination by leveraging a variety of real-world sources. Built on Panda70M (8), we curate ~000 high-quality video pairs and augment training with 1.2M additional relit images using IC-Light. This combination supports robust learning of physically plausible relighting without relying expensive hardware manual annotations. See more details in Appendix
4.3 Joint Learning Objective and Training Strategy
Model Implementation. The inputs of aligned videos (Vdeg. Vbg. Vreat) are passed through a Wan- VAE Encoder (38 to obtain semantic latent representations (Xiegs Xhg, Xo). During training, we generate the noisy latent input via Eq. 2 and concatenate it with the strong conditional signals Xdeg and Xhg along the channel dimension. This combined tensor is injected into the DIT blocks of the Wan backbone. Additionally, to support path-consistency learning, the diffusion step and

---

the expected denoising step d are appended as temporal condition vectors. All new projection and fusion layers are initialized with zero weights to preserve compatibility with the pre-trained Wan initialization and ensure stable optimization from the outset.
Joint Objective. Our training objective integrates three complementary losses to balance appearance fidelity, geometric consistency, and fast inference. The full loss is defined as:
Co X fastt A2
where Co is the standard flow-matching loss that aligns the predicted velocity field with the ground- truth field, Liast is the path consistency loss that improves model performance under few-step denoising regimes, and phy is physics-guidee guiied loss that supervises the RGB outputs using estimated depth and normal maps. We adopt fixed weights of λ 1.0 and \ X2 0.1 for all experiments. This unified objective encourages the model to generate relit results that are photorealistic, temporally smooth, and physically grounded while supporting efficient inference without sacrificing output quality.
Training Strategy. To balance physical super- vision and training efficiency, we adopt a se- lective optimization strategy inspired by path consistency scheduling [13]. In each training iteration, we divide the batch based on super- vision type, following an 80/20 split to avoid prohibitive costs from full supervision while still maintaining effective learning signals. As shown in Alg. [ 20% of each batch is allocated to compute the path consistency loss Liss. which involves three forward passes and one backward pass to enforce consistency across timesteps. The remaining 80% is used for the standard flow- matching loss Co. with 50% of these samples
Algorithm Loss Sampling Strategy per Itera- tion
Require:
Batch size B, total training samples 1: for each training iteration do
2: Randomly sample 20% of batch + Castt
3: Compute path consistency loss:
4: 3x forward, 1x backward
5: Remaining 80% Lo
6: Among those, 50% + RGB reconstruc-
tion
7: Compute physics-guided loss Lphy
8:
end for
further supervised using RGB-space geometry feedback via Cpby (i.e., depth and normal alignment). This probabilistic scheduling ensures high training throughput while allowing the model to benefit from multi-level supervision. To further enhance illumination diversity during training, we apply randomized lighting augmentations on the degraded subject V deg. which introduces realistic lighting variability without the need for explicitly paired captures.
Experiments
5.1 Experimental Details
Training Details. We adopt the
(38 the base model, and initialize all new trainable parameters with zeros to minimize its influence at the beginning of training. We use the AdamW optimizer with the learning rate of le-5 for training the entire framework. All the models are trained with batch size of 8 for 5,000 iterations on 8 NVIDIA H20 GPUs (with 96GB RAM).
Baselines. We compare UniLumos against range of representative image and video relighting methods. For image-based relighting, we include SwitchLight [21]. DiLightNet (4). IC-Light [49], and SynthLight (6), which leverage various forms of latent modeling or light disentanglement to relight single images. For video relighting, we apply IC-Light via frame-by-frand and include Light- A- Viiddoooo V2 and another using Wan 2.1 12V-1.3B 38) These baselines together represent state-of-the-an performance across both image and video relighting settings.
Dataset. For testing, selected samples from the internal dataset, processed using the method described in Sec. These samples were evenly split: half for image generation at 768x512 resolution, and half for video generation at 480p resolution (832x480), with each video sample containing 49 frames. To further demonstrate the model's generalization to non-human scenes, we conducted additional evaluations two public object-centric relighting benchmarks: StanfordOrb 24 and Navi [9), which include objects and sculptures under a variety of lighting environments and are completely disjoint from our training data, and more results in Appendix C..
Evaluation metries. We evaluate relighting performance across three key dimensions: (1) visual fidelity: We assess image quality using Peak Signal-to-Noise Ratio (PSNR), Structural Similarity
7

---

Table 1: Quantitative comparison. Bold number indicate the best performance.
(a) Quality SSIM
Temporal Consistency
(c) Lamos Consistency
Model
PSNR
LPIPS $
R. R.Mottonn.
Avg. Score
Dense 1.2 Emor !
Image Relighting
Siit
21
20.483
0.901
0.094
0.717
0.388
DiLightNet
IC-Light
49
24.316
0.884
0,108
0.703
0.447
Synthl ight
25.572
0.905
0.102
0.791
0.214
UniL.umos
26.719
0.913
0.089
0.912
0.103
Video Relighting
IC-Light Per Frame
49
20.132
0.851
0.133
2.437
0.672
0.432
Light-A- A-Video
57
Cog Vi
19.851
0.859
0.124
1.784
0.641
0.383
LiglotAA-Vidoo
S
Wan2. 188
20.784
0.876
0,129
1,582
0.682
0.371
Unil.umos
25.031
0.891
0.109
1.436
0.871
0.147
Inpul
IClgh
Light Vidao Light Aiddo
CogVideoX
Wan
Unilumos
[Our)
"A mon with beard, weening daakk shirt and The background is plain, colored wall Frord Light, Artiica! light, Maderate, Neatral, Static Light, None
A black hoodle and sis al... The room walls are black, and the floor i light color The cailng fights are arranged in grid pottem. front Ligtt, Amficial light, Moderats, Neutaa, Static ligtt hone
Caplion
Figure 3: Qualitative comparison of baseline methods. Each method takes a subject video and textual illumination description input, generating the related subject with the corresponding background under the specified lighting condition.
Index (SSIM), and Learned Perceptual Image Patch Similarity (LPIPS). For video relighting, we report the average metric across all frames. (2) temporal consistency: Following VBench [18).
we adopt the R-Motion metric, which measures temporal smoothness using motion priors from pre-trained video frame interpolation model (27]. This captures the coherence of lighting transitions across frames. (3) lumos consistency: (i) Lumos Score, computed by applying the same caption- lighting annotation protocol as in our LatosData construction. Six lighting attributes are predicted and compared with targets, and each is weighted equally to yield an average consistency score. (ii) Dense L2 Error, which quantifies the relative L2 error between predicted and reference depth/normal maps, estimated via pre-trained geometry model (e.g., Lotus [17]. This provides physically grounded measure of illumination-geoment alignment.
5.2 Main Results
Quantitative Evaluation. As shown in Tab. UniLumos delivers consistent improvements across three key dimensions: visual fidelity, temporal consistency, and physically grounded lighting align- ment. (1) Visual Fidelity. UniLumos produces higher-quality relighting results across both images and videas. Benefiting from structured lighting supervision and geometry-guidde feedback, our model generates outputs with clearer shading, sharper details, and more coherent illumination compared to prior works. (2) Temporal Consistency. For video relighting, UniLumos ensures smoother transi- tions and reduced flickering artifacts. Our use of flow-matching architecture and path consistency learning helps maintain stable lighting across frames, addressing key limitation in frame-wise

---

Frama
Frome 24
Froma 46
Frome
Relight
baokground
Figure UniLumos performs physically plausible video relighting conditioned on different reference videos.
or training-free methods. (3) Lumos Consistency. Going beyond appearance-basct evaluation, UniLumos aligns well with intended lighting semantics. Through structured caption conditioning and physics-guide guided training, the model better preserves lighting direction, tone, and erometry-validated by both vision-language alignment and dense geometric error metrics.
Efficiency. To assess inference efficiency, we evaluate the video relighting task under a stan- dardized setting, generating 49-frame videos at a 480p resolution. As shown Fig. UniLu- mos achieves significant speedup compared to prior methods, benefiting from its geometry-free inference and few-step generation. While exist- ing models, such as Light-A- Video or IC-Light, require either iterative frame-by-frane process- ing or complex sampling schedules, UniLumos completes generation over 20 times faster with- out sacrificing visual fidelity or physical plausibility. This efficiency advantage makes well-suited for real-time relighting applications and scalable deployment scenarios.
ICLigh
277
Light-A-Vido CogVideoX
Light.A-Video
Won-1.38
756
Unilumos 1.38
Os
400s
800s
Time
Figure 5: Comparison of inference time costs of different methods under the same settings.
Qualitative Results We present qualitative comparisons in Fig. and Fig. highlighting the advantages of UniLumos in terms of lighting realism, temporal conerence, and controllability. (1) Lighting Quality and Controllability:
Fig. 3 UniLumos produces lighting effects that better match the target description, capturing nuanced directional shadows, color tone, and intensity. Competing methods either fail to reflect the intended lighting change or produce overly uniform results that lack realism. (2) Temporal Consistency: Compared to baseline methods such frame- wise IC IC-Light and Bight-A-Videer UniLumos achieves smoother frame transitions without flickering or structural distortion. This benefit arises from the joint modeling of space and time, which is further reinforced by physics-aware supervision and path consistency training. (3) Foreground Detail Preservation: UniLumos preserves fine subject details such facial structure and clothing rexture-bette than baselines. For instance, Light-A-Vides occasionally introduces deformation or identity drift, while model maintains high fidelity over long sequences.
(4) Relighting with Reference Videos: Fig. 4 showcases UniLumos conditioned on different reference videos. The model successfully adapts both global lighting direction and subtle spatial variations across scenes, demonstrating strong generalization under diverse illumination cues.
LumosBench To evaluate the fine- iin-ggrined controllability of lighting generation, we introduce Lumos- Bench, a structured benchmark that targets six core illumination attributes defined in our annotation protocol. Unlike prior works that treat lighting holistically implicitly, LumosBench provides dis- entangled, attribute-level evaluation, enabling precise diagnosis of model behavior under controllable lighting conditions. See more results in Appendix C
5.3 Ablation Study
As shown in Tab. 2 and Fig. we conduct ablation studies to analyze the effectiveness of different components. Physics-Guided Feedback. Removing both depth and normal feedback (w/o All

---

Table 2: Quantitative comparison. Bold number indicate the best performance.
(a) Quality
(b) Temporal Consistency
Lumos Consistency
Model
PSNR
SSIM
LPIPS
R-Motion
Avg. Score
Dense L2 Error
Ablative Study
Depth l'eedback
23.472
0,883
0.118
1.443
0.870
0.265
w/o Normal Feedback
22.115
0.874
0.123
1,446
0.863
0.173
w/o All Feedback
21.433
0.862
0.139
1.473
0.859
0,297
w/o Path Consistency
25.317
0.902
0.113
1.438
0.875
0.153
Effect of Training Domain
Only Vidco
22.487
0.863
0.119
1.487
0.857
0.173
Only Image
24.471
0.872
0.123
2.429
0.841
0.182
UniLumos
25.031
0.891
0.109
1.436
0.871
0.147
Frome
Matching
Plo
Dunassing Stop
Parh
Feeebook
Darosinng Stop
Unilumos
wavy sings into gold-cofored backgraund
fight gray bullon bookground blumed with fights and dark Sile Light Articcall light Moderate. Nestral Stans Light Noxe"
dark and light looking
Caption
woman's mokeup includes dovk syelinar synshadow Fronf
callae
Artiicial Lett Maderate, Cool Tore, Stutic Light, Nore"
Figure 6: Ablation study. We compare the effects of different components under few-step denoising. For I-step, we show the impact of flow-matching with and without path consistency, For 5-step, visualize results before and after introducing physics-plausible feedback.
Feedback) leads to significant degradation both image quality and physical consistency, confirming the necessity of physics-guided loss. Notably, omitting only normal supervision causes larger drop than removing depth, suggesting that surface orientation plays more critical role than distance in shaping light-shadow interactions. Path Consistency Learning. Excluding this component (w/o Path Consistency) yields only minor drops physical metrics while maintaining competitive SSIM and LPIPS scores. This shows that path consistency incurs little performance cost but offers substantial efficiency benefits in few-step regimes, justifying its inclusion. Training Modality. To evaluate the effectiveness of unified training paradigm, we compare domain-specific variants: training solely on videos leads to poor visual quality, while image-only training sacrifices temporal smoothness. In contrast, our unified approach strikes the best balance-achievinng high-quality and temporally coherent relighting across both input types.
6 Conclusion
We introduce UniLumos, a unified framework for physically plausible image and video relighting. aligns illumination with scene geometry via RGB-spaac depth and normal supervision, improving shadow accuracy and spatial consistency. To enhance controllability and evaluation, we propose structured six-dimensional lighting annotation protocol, enabling fine-gn fine-grained conditioning and physically grounded assessment through VLMs. Experiments show that UniLumos achieves superior relighting quality, physical consistency, and inference efficiency.
10

---

Acknowledgment
This work was supported by Damo Academy through Damo Academy Research Intern Program.
References
[1] Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge. Sibo Song, Kai Dang, Peng Wang. Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jiangiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Hang Zhang, Zhibo Yang. Haiyang Xu, and Junyang Lin. Qwen2.5-vl technical report. arXiv preprint ar arXiv:2502. 13923, 2025.
[2] Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Do- minik Lorenz, Yam Levi, Zion English, Vikram Voleti, Adam Letts, al. Stable video diffusion: Scaling latent video diffusion models to large datasets. arXiv preprint arXiv:2311.155 1,12211,,, 2023.
[3] Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Do- minik Lorenz, Yam Levi, Zion English, Vikram Voleti, Adam Letts, al. Stable video diffusion: Scaling latent video diffusion models to large datasets. arXiv preprint arXiv:2311.15122 2023.
[4] Andreas Blattmann, Robin Rombach, Huan Ling. Tim Dockhorn, Seung Wook Kim, Sanja Fidler, and Karsten Kreis. Align your latents: High-resolution video synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference recognition, pages 22563-22511 2023.
conference on computer vision and pattern
[5] Ziqi Cai, Kaiwen Jiang, Shu-Yu Yu Chen, Yu-Kun Lai, Hongbo Fu, Boxin Shi, and Lin Gao. Real-time 3d-aware portrait video relighting. In Proceedings of the IERE/CVF Conference Computer Vision and Pattern Recognition, pages 6221-6231, 2024.
[6] Sumit Chaturvedi, Mengwei Ren, Yannick Hold-Geoffroy, Jingyuan Liu, Julie Dorsey, and Zhixin Shu. Synthlight: Portrait relighting with diffusion model by learning to re-render synthetic faces. arXiv preprint MtXiv:2501.09756 2025.
[7] Haoxin Chen, Menghan Xia, Yingqing He, Yong Zhang, Xiaodong Cun, Shaoshu Yang, Jinbo Xing, Yaofang Liu, Qifeng Chen, Xintao Wang, et al. Videocrafterl: Open diffusion models for high-quality video generation. arXiv preprint arXiv:2310.19512. 2023.
[8] Tsai-Shien Chen, Aliaksandr Siarohin, Willi Menapace, Ekaterina Deyneka, Hsiang-wei Chao, Byung Eun Jeon, Yuwei Fang, Hsin- Hsin-Ying Lee, Jian Ren, Ming-Hsuan Hsuan Yang, et al. Panda- 70m: Captioning 70m videos with multiple cross- mods-moy teachers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13320-133: 13331, 2024.
[9] Weifeng Chen, Jiacheng Zhang. Jie Wu, Hefeng Wu, Xuefeng Xiao, and Liang Lin. Id-aligner: Enhancing identity text-to-image generation with reward feedback learning, arXiv preprint arXiv:2404. 15449, 2024.
[10] Kevin Clark, Paul Vicol, Kevin Swersky, and David Fleet. Directly fine-tuning diffusion models on differentiable rewards. arXiv preprint arXiv:2309. 17400, 2023.
[11] Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et al. Scaling rectified flow trans- formers for high-resolution image synthesis. Forty-first international conference on machine learning, 2024.
[12] Ye Fang, Zeyi Sun, Shangzhan Zhang, Tong Wu, Yinghao Xu, Pan Zhang, Jiaqi Wang, Gordon Wetzstein, and Dahua Lin. Relightvid: Temporal-consistent diffusion model for video relighting. arXiv preprint wrXiv:2501.16330 2025.
[13] Kevin Frans, Danijar Hafner, Sergey Levine, and Pieter Abbeel. One step diffusion via shortcut models, arXiv preprint arXiv:2410.12557 2024.

---

[14] Rohit Gandikota, Joanna Materzyúíska Tingrui Zhou, Antonio Torralba, and David Bau. Concept sliders: Lora adaptors for precise control in diffusion models. European Conference on Computer Vision, pages 172-188. Springer, 2024.
[15] Jianping Gou, Baosheng Yu, Stephen J Maybank, and Dacheng Tao. Knowledge distillation: survey. International Journal of Computer Vision, (29(6): 189-1819, 2021.
[16] Yoav HaCohen, Nisan Chiprut, Benny Brazowski, Daniel Shalem, Dudu Moshe, Eitan Richard- son, Eran Levin, Guy Shiran, Nir Zabari, Ori Gordon, et al. Ltx-video: Realtime video latent diffusion. arXiv preprint arXiv:2501 000003,,, 2024.
[17] Jing He, Haodong Li, Wei Yin, Yixun Liang, Leheng Li, Kaiqiang Zhou, Hongbo Zhang, Bingbing Liu, and Ying-Cong Chen. Lotus: Diffusion-basn based visual foundation model for high-quality dense prediction. arXiv preprint arXiv:2409.881 2024.
[18] Ziqi Huang, Yinan He, Jiashuo Yu, Fan Zhang, Chenyang Si, Yuming Jiang, Yuanhan Zhang. Tianxing Wu, Qingyang Jin, Nattapol Chanpaisit, et al. Vhench: Comprehensive benchmark suite for video generative models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 21807-21818, 2024.
[19] Varun Jampani, Kevis-Kokitsi Maninis, Andreas Engelhardt, Arjun Karpur, Karen Truong, Kyle Sargent, Stefan Popov, André Araujo. Ricardo Martin Brualla, Kaushal Patel, et al. Navi: integory-agnostee image collections with high-quality 3d shape and pose annotations. Advances in Neural Information Processing Systems, 36:76061-76084 2023.
[20] Haian Jin, Yuan Li, Fujun Luan, Yuanbo Xiangli, Sai Bi, Kai Zhang, Zexiang Xu, Jin Sun, and Noah Snavely. Neural gaffer: Relighting any object via diffusion. Advances in Neural Information Processing Systems, 37:141 129-141152, 2024.
[21] Hoon Kim, Minje Jang, Wonjun Yoon, Jisoo Lee, Donghyun Na, and Sanghyun Woo. Switch- light: Co-design of physics-driven architecture and pre-training framework for human portrait relighting. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 25096-25108 206. 2024.
[22] Peter Kocsis, Vincent Sitzmann, and Matthias Nießner. Intrinsic image diffusion for indoor single-view material estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5198-520 5208, 2024.
[23] Weijie Kong, Qi Tian, Zijian Zhang, Rox Min, Zuozhuo Dai, Jin Zhou, Jiangfeng Xiong, Xin Li, Bo Wu, Jianwei Zhang, et al. Hunyuanvideo: A systematic framework for large video generative models. arXiv preprint arXiv:2412.03603 2024.
[24] Zhengfei Kuang, Yunzhi Zhang, Hong-Xing Yu, Samir Agarwala, Elliott Wu, Jiajun Wu, al. Stanford-orb: a real-world 3d object inverse rendering benchmark. Advances Neural Information Processing Systems, 36:46938-4699 2023.
[25] Kimin Lee, Hao Liu, Moonkyung Ryu, Olivia Watkins, Yuqing Du, Craig Boutilier, Pieter Abbeel, Mohammad Ghavamzadch, and Shixiang Shane Gu. Aligning text-to-image models using human feedback. arXiv preprint arXiv:2302 12192, 2023.
[26] Xiaowen Li, Haolan Xue, Peiran Ren, and Liefeng Bo. Diffueraser: A diffusion model for vidco inpainting. arXiv preprint arXiv:2501. 2025.
[27] Zhen Li, Zuo-Liang Zhu, Lin Han, Qibin Hou, Chun-Le Guo, and Ming-Ming Cheng. Amt: All-pairs multi-fich field transforms for efficient frame interpolation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 9801-9810, 2023.
[28] Yaron Lipman, Ricky TQ Chen, Heli Ben-Hamu, Maximilian Nickel, and Matt Le. Flow matching for generative modeling, arXiv preprint arXiv:2210.027: 2022.
[29] Xingchao Liu, Chengyue Gong, and Qiang Liu. Flow straight and fast: Learning to generate and transfer data with rectified flow. arXiv preprint arXiv:2209.03003, 2022.
12

---

[30] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida. Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730-27744, 2022.
[31] Rohit Pandey, Sergio Srts-Escolano, Chloe Legendre. Christian Haene, Sofien Bouaziz, Christoph Rhemann, Paul E Debevec, and Sean Ryan Fanello. Total relighting: learning to relight portraits for background replacement. ACM Trans. Graph., 40(4))43-1, 2021.
[32] Mengwei Ren, Wei Xiong, Jae Shin Yoon, Zhixin Shu, Jianming Zhang, HyunJoon Jung, Guido Gerig, and He Zhang. Relightful harmonization: Lighting-aware portrait background replacement. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6452-6462, 2024.
[33] Peiran Ren, Yue Dong, Stephen Lin, Xin Tong, and Baining Guo. Image based relighting using neural networks. ACM Transactions on Graphics (ToG), 34(4):1-12, 2015.
[34] Tim Salimans and Jonathan Ho. Progressive distillation for fast sampling of diffusion models. arXiv preprint arXiv:2202.00511, 2022.
[35] Aliaksandr Siarohin, Stéphane Lathuilière, Sergey Tulyakov, Elisa Ricci, and Nicu Sebe. First order motion model for image animation. Advances in neural information processing systems,
32, 2019.
[36] Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul Christiano. Learning summarize with human feedback. Advances in neural information processing systems, 33:3008-300 -0201, 2020.
[37] Yuiga Wada, Kanta Kaneda, Daichi Saito, and Komei Sugiura. Polos: Multimodal metric learn- ing from human feedback for image captioning. In Proceedings of the IEFE/CVF Conference on Computer Vision and Pattern Recognition, pages 135668 2024.
[38] Ang Wang, Baole Ai, Bin Wen, Chaojie Mao, Chen Wei Xie, Di Chen, Feiwu Yu, Haiming Zhao, Jianxiao Yang, Jianyuan Zeng. Jiayu Wang. Jingfeng Zhang, Jingren Zhou, Jinkai Wang, Jixuan Chen. Kai Zhu, Kang Zhao, Keyu Yan, Lianghua Huang, Mengyang Feng. Ningyi Zhang, Pandeng Li, Pingyu Wu, Ruihang Chu, Ruili Feng, Shiwei Zhang, Siyang Sun, Tao Fang, Tianxing Wang, Tianyi Gui, Tingyu Weng, Tong Shen, Wei Lin, Wei Wang, Wei Wang, Wenmeng Zhou, Wente Wang. Wenting Shen, Wenyuan Yu, Xianzhong Shi, Xiaoming Huang, Xin Xu, Yan Kou, Yangyu Lv, Yifei Li, Yijing Liu, Yiming Wang, Yingya Zhang, Yitong Huang, Yong Li, You Wu, Yu Liu, Yulin Pan, Yun Zheng, Yuntao Hong, Yupeng Shi, Yutong Feng, Zeyinzi Jiang, Zhen Han, Zhi-Fan Wu, and Ziyu Liu. Wan: Open and advanced large-scale video generative models. arXiv preprint arXiv: 20314, 2025.
[39] Xiang Wang, Shiwei Zhang, Han Zhang, Yu Liu, Yingya Zhang, Changxin Gao, and Nong Sang. Videolem: Video latent consistency model. arXiv preprint arXiv:2312.091099 2023.
[40] Zhongcong Xu, Jianfeng Zhang, Jun Hao Liew, Hanshu Yan, Jia- fei Liu, Chenxu Zhang, Jiashi Feng, and Mike Zheng Shou. Magicanimate: Temporally consistent human image animation using diffuusion model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1481-1490, 2024.
[41] Zhuoyi Yang. Jiayan Teng, Wendi Zheng, Ming Ding, Shiyu Huang, Jiazheng Xu, Yuanming Yang, Wenyi Hong, Xiaohan Zhang, Guanyu Feng, et al. Cogvideox: Text-to-video diffusion models with expert transformer. arXiv preprint arXiv:2408.0661 06072, 2024.
[42] Yu-Yin Ying Yeh, Koki Nagano, Sameh Khamis, Jan Kautz, Ming- Yu Liu, and Ting- Chun Wang. Learning to relight portrait images via a virtual light stage and symthetic-to-real adaptation. ACM Transactions on Graphics (TOG), 41(6):1-21, 2022.
[43] Hangjie Yuan, Shiwei Zhang, Xiang Wang, Yujie Wei, Tao Feng, Yining Pan, Yingya Zhang, Ziwei Liu, Samuel Albanie, and Dong Ni. Instructvideo: Instructing video diffusion models with human feedback. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 6463-6474, 2024.
13

---

[44] Chong Zeng, Yue Dong, Pieter Peers, Youkang Kong, Hongzhi Wu, and Xin Tong. Dilightnet: Fine-grained lighting control for diffusion based image generation. In ACM SIGGRAPH 2024 Conference Papers, pages 1-12, 2024.
[45] David Junhao Zhang, Jay Zhangjie Wu, Jia-Wei Liu, Rui Zhao, Lingmin Ran, Yuchao Gu, Difei Gao, and Mike Zheng Shou. Show-1: Marrying pixel and latent diffusion models for text-to-video generation. International Journal of Computer Vision, pages 1-15, 2024.
[46] Kai Zhang, Fujun Luan, Qiangian Wang, Kavita Bala, and Noah Snavely. Physg:
Inverse render- ing with spherical gaussians for physics-based material editing and relighting. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5453-5462, 2021.
[47] Longwen Zhang, Qixuan Zhang, Minye Wu, Jingyi Yu, and Lan Xu. Neural video portrait relighting real-time via consistency modeling. In Proceedings of the IEEE/CVF international conference on computer vision, pages 802-812, 2021.
[48] Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. In Proceedings of the IEEE/CVE international conference on computer vision, pages 3836-3847, 2023.
[49] Lvmin Zhang. Anyi Rao, and Maneesh Agrawala. Scaling in-the-wild training for diffusion- based illumination harmonization and editing by imposing consistent light transport. In The Thirteenth International Conference on Learning Representations, 2025.
[50] Yuanqing Zhang, Jiaming Sun, Xingyi He. Huan Fu, Rongfei Jia, and Xiaowei Zhou. Modeling indirect illumination for inverse rendering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18643-18652, 2022.
[51] Yuxin Zhang, Dandan Zheng, Biao Gong, Jingdong Chen, Ming Yang, Weiming Dong, and Changsheng Xu. Lumisculpt: A consistency lighting control network for video generation. arXiv preprint arXiv:2410.22979, 2024.
[52] Shihao Zhao, Dongdong Chen, Yen-Chun Chen, Jianmin Bao, Shaozhe Hao, Lu Yuan, and Kwan- Yee K Wong. Uni-controlnet: All-in-one control to text-to-image diffusion models. Advances in Neural Information Processing Systems, 36:11127-11150, 2023.
[53] Jianbin Zheng, Minghui Hu, Zhongyi Fan, Chaoyue Wang, Changxing Ding, Dacheng Tao, and Tat-Jen Cham. Trajectory consistency distillation. arXiv e-prints, pages arXiv-2402, 2024.
[54] Peng Zheng, Dehong Gao, Deng-Ping Fan, Liu, Jorma Laaksonen, Wanli Ouyang, and Nicu Sebe. Bilateral reference for high-resolttion dichotomous image segmentation. CAAI Artificial Intelligence Research, 3:9150038, 2024.
[55] Hao Zhou, Sunil Hadap, Kalyan Sunkavalli, and David W Jacobs. Deep single-image portrait relighting. In Proceedings of the IEEE/CVF internation conference on computer vision, pages 7194-7202, 2019.
[56] Shangchen Zhou, Chongyi Li, Kelvin CK Chan, and Chen Change Loy. Propainter: Improving propagation and transformer for video inpainting. In Proceedings of the IEEE/CVF international conference on computer vision, pages 10477-10486, 2023.
[57] Yujie Zhou, Jiazi Bu, Pengyang Ling, Pan Zhang, Tong Wu, Qidong Huang, Jinsong Li, Xiaoyi Dong, Yuhang Zang. Yuhang Cao, et al. Light-a-video: Training-free video relighting via progressive light fusion. arXiv preprint arXiv:2502.08590, 2025.
14

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
We have made clear claims about contributions and scopes in the abstract and introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We have the seperated limitation section in the appendix.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [No]
15

---

Justification:
The answer NA means that the paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We describe the details of our model and training strategy in the paper for reproduction.
Guidelines:
The answer NA means that the paper does include experiments. If the paper includes experiments, No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how reproduce that algorithm.
(b) If the contribution primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
16

---

Answer:
[Yes]
Justification:
We offer the implementation code to reproduce our work.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.co cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https guwdes/dde/eeee Cooeeeoodeesssseeeeee 7/nips.com htt/://bbb///giidee///eeeeeeisss swiniioiscyp for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification: We have introduced experiment settings and details, and even add more details in the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification:
We do not report specific error bars.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
17

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We have discussed the information about computer resources.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification:
We follow the NeurlPS Code of Ethics in every respect.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discuss this in the appendix.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
18

---

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [Yes]
Justification: Users should follow usage policies and use models with safety filters and access controls.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification:
We have cited the original paper that produced the code package and dataset.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the Datasets.papersstt paperswi thcode. cow/ datasets package should be provided. For popular datasets,
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
19

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
We provide documentation with our code.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/cooe/mmoel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA].
Justification: Our work does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question:
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: Our work does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
20

---

Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification:
We only use LLM for proof-reading.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs any important, original, non-standard components.
Please refer to our LLM policy fattps://neuriie /ttpa/rrpp..ccooooofeeeee/e///// for what should or should not be described.
21

---

Appendix of UniLumos
In this appendix, we provide additional details to complement the main paper. First, we explain the motivation behind introducing physics-plausible feedback in Sec. A Next, present the detailed pipeline of our proposed LumosData relighting data construction process in Sec. B Then, Sec. [ contains three additional experimental results on the public dataset (Sec. [C.1) and the LumosBench (Sec. C.2). Sec. [ provides additional qualitative results to further illustrate the effectiveness of UniLumos. Finally, Sec. Sec. and Sec. discuss the limitations, broader impacts, and safeguards of our method.
A Physics-Plausible Feedback
To further clarify the motivation and design behind our physics-plausine feedback mechanism, we present breakdown of key questions and considerations addressed during its development.
Q1: What is the motivation for introducing physical constraints in relighting?
A1: The primary goal of relighting is to generate visually plausible illumination under new lighting conditions. However, many diffusion-baseed methods lack explicit physical modeling, leading to artifacts such as overexposed highlights, misaligned shadows, or inconsistent light directions. In- troducing physical constraints serves refinement mechanism that aligns generated light with the scene's 's underlying geometry. This helps enforce realism and spatial consistency in illumination, which is especially critical under complex lighting HDR scenarios.
Q2: Why are depth and normal maps chosen as the targets for physical supervision?
A2: Depth and surface normals are among the most accessible and general-purpose dense scene attributes. By design, these estimations intentionally suppress fine-scale lighting effects focus on intrinsic geometry. This makes them ideal for supervising relighting, where the goal is to decouple geometry from illumination and enforce spatial structure in lighting behavior. UniLumos, we align the predicted lighting with reference geometry by minimizing the 1.2 norm error between generated and reference- aligned depth/normal maps via a pre-trained dense estimation model (e.g., Lotus [17]) with frozen parameters. This provides simple yet effective metric to quantify physical plausibility.
the proposed
Q3: Why are alternative physical signals-such as albedo, shadow, or material- used instead?
A3: While albedo, shadow masks, and material properties can provide rich supervision, they come with significant drawbacks. Albedo and shadow estimation often rely inverse rendering and suffer from domain sensitivity or ambiguity. Material annotations are expensive and dataset-dependent Moreover, many of these properties are entangled with illumination, making them less reliable as supervisory signals. In contrast, depth and normals can be predicted from monocular images with high availability and generalize well across scenes, offering favorable balance between supervision quality and computational cost.
Q4: Why are depth and normal maps used as training- time constraints rather than model inputs?
A4: While it is possible to condition the model directly on estimated depth and normal maps, doing so increases the input dimensionality and model complexity. It would also introduce dependency on external estimators during inference, complicating the pipeline and potentially propagating errors. Instead, we use them as supervision signals during training. This design keeps the inference pipeline repling only on image and lighting condition geaeetry-aware behaviors. The supervision acts as a form inductive bias, guiding the model toward physically plausible outputs without requiring additional input channels at the test phase.
still allowing the model to learn
B Details of Datasets
Step 1: Subject Mask. Given an input video Vieat E R.T,), we first extract per-frame subject masks M E R. using BiRefNet (54). These subject masks allow us isolate the target subject foreground and the target background.
22

---

Table 3: Lighting-relann related textual prompts used in Lumos Augmentation from IC-Light prompt can be combined with different canonical light directions during training.
49
Each
ID
Lighting Prompt
Example Light Direction
sunshine from window
None
neon light, city sunset over sea
Left Light
3
Right Light
golden time
Top Light
5
sci-fi RGB glowing, cyberpunk
Bottom Light
natural lighting
7
warm atmosphere, at home, bedroom
magic
9
evil, gothic, Yharnam
10
light and shadow
11
shadow from window
12
soft studio lighting
13
home atmosphere, cozy bedroom illumination
14
neon, Wong Kar-wai, warm
Step Lumos Augmentation. To simulate diverse lighting degradations for training, we relight each subject sequence under multiple lighting conditions using pre-trained 2D relighting model, such as Light (49. This operation is applied independently each frame of the subject region, resulting
in degenerated video Vdg E R 1.H,,,, To generate rich illumination variations. we refer to the description of light and shadow given by IC-Light [49], as listed in Tab. 3 which serve as the semantic guidance for image-level relighting and the light source directions. For each input video, we randomly sample 5 prompts and directions, forming 5 x 3 = 15 unique prompt- dirett-ii pairs. The relighting is applied only to the subject region, extracted using the subject masks M E R R[*+,,H, from Step Notably, combinations for each subject each iteration. This strategy reduces training cost while exposing the model diverse illumination patterns, thereby improving generalization.
randomly sample one degradation condition from the 15 prompt-direction
Step 3: Gaussian Background. To provide external lighting context during training, we generate a background video Vng E R 7+,,H...W. to accompany the relit subject. Instead of relying on complex inpainting-based synthesis (e.g., ProPainter (56). DiffuEraser (26)), we adopt simple yet effective strategy by filling the background with either pure color or Gaussian noise. This design avoids injecting semantic structural priors, allowing the model to focus solely on illumination learning.
Specifically, for each frame E [1, T and channel E {R, G, B). we first define the background region using the subject mask M, E R kl obtained in Step 1. Let S2 = {i, M,(i,) = 0} denote the set of background pixels. We compute the mean and standard deviation of background pixel intensities as:
1
Σ
Ve(ij,c))
1
(1))
(8)
Σ
(V.(i,j.c) - 12))
(K.))n:
We then fill the background with pixel-wise samples from Gaussian distribution:
Vig (1. ~ N(Hh.. (at)?), V(i,j) E Pi.
(9)
This procedure ensures that the background region maintains similar color distribution to the original video while avoiding structural detail that may bias learning. For comparison, we also test a variant that uses pure-color background, where each background pixel is set to μ² (i.e., of 0). In practice, we observe that such statistically consistent placeholders particularryyy Gaussian- fillled ones-ace acce-aaceee early-stage convergence during training. We attribute this to the reduced visual complexity and improved normalization behavior, which make the model less sensitive to background variation. The resulting Vag serves as clean, distributiio--aigned conditioning signal for the relighting network.
23

---

Goussion
Inpaintin
Pure Inpaining
Gousian Impainfing
Pure Inpainting
Figure 7: Comparison of background inpainting strategies of four representative cases. Here, Gaussian Inpainting fills the background using random noise sampled with the same mean and variance as the subject region, ensuring statistical consistency. Pure Inpainting directly fills the background with a uniform color (i.e., gray), without modeling spatial or color variation. The Gaussian strategy provides more realistic signal distribution and accelerates carly-stage convergence in training.
Step 4: Caption Augmentation. In addition relighting augmentation, we generate lighting-awan captions to provide rich semantic supervision aligned with physical lighting behavior. Specifically, we leverage Qwen2.5-VL M vision-language model with fine-grained visual reasoning capabilities, to analyze each input video and generate structured captions describing its lighting attributes. The imput to Qwen2.5-VL consists of the original video and its corresponding scene-level caption. We then apply a custom sestom-designed prompt (see Listing T to steer the model toward predicting six categories of lighting-relater labels as shown in Tab. including all subcategories and their physical interpretations. The output of this process is structured caption C for each video, formatted the six categories their predicted labels (see example Listing [T.. These structured captions serve as auxiliary supervision and evaluation labels in later stages, helping the model better align with interpretable physical lighting semantics. They also enhance downstream controllability and facilitate bbasdd retrieval or evaluation.
dictionary mapping
Table 4: Classification criteria and definitions for light-related scene attributes.
Primars Cateuuey
Suleciilsorrs
Delultion
Faozz Ligh
The light sounce is pustioona dinadly in feom subjoct illuminting il heal-rr. The light source is postioned at 90 degreo er 4t dagree ingle to the suject conti from side. light souee locaeed dimetly subjoc,
Side Light Lipha
Dieccion Light
Top
Bortom Light
poultioed below srtject, The light source illamintes one side subject while leasing the other side is shadow.
The
upwands
Sight
Amhicat I iigh Withhet Clear Dincaiion
Anhieei light mindiiiiioonl unifbemly iluminating the aminnmen from muliple varrze
Natura Light
Illumiration from nature withost humen intervetion varyang with time day, weother and location
Light Source Type
Artificiall
lisht sounces (eg. LEDa) gemtes
indoriiutioor spaces
setisii
light
Diptolly
light
using techniques
moving
Glaae
Exsemelly beight over 1000 lamar that caase discmmoon absare dsail.
Light Intensity
Mederate
Balanced ligtting (200- 1000 lumene), sumable for most actemee and confoctabe viewing
Diss
Low lighting unde: 200 lunzes, ofiee cosy but may redace sisbiily and dotil recogsition.
Tire
1000-110000;
common daylight oxmeat sernee
Temperatue
belanced light typiaal
mrong
yellow
2000K
lighting.
Static Light
Mumiiiation remaits constare is intersity and disction oer tine.
Light Clanges in Time:
Dynanic Light (latanity Changing)
Light intendty shang gralially owe lime icg, davs dhyligh)
Dynami Light: (Meninng Source)
Direction of light. changes doe movement light sounce io headlggtt stgee lights.
Bufraction/Reeleeme (Class)
passe: thoough trttppuent muterials glass possible scuttring absoeption.
Plesenena
Mimm?
diffuses
like signiiilni optical planommana olecroal
redocing visiility.
Nove
24

---

SYSTEM_ PROMPT """
You are helpful, respectful and honest assistant. Always answer as helpfully as possible, whole being safe. Your answers should not include any harmful, unethical, racist, sezist, tewic, dangerous, or sllegal content. Please ensure that your responses are socially unbiased and posstive in nature. In|nIf question does instead of answering something not correct. If you don't know the answer question, please don 't share false information.
any sense, or not factually coherent, emplain why
PROMPT ~ww
Role: You are an expert in image/video light and shadow analysis, good at analyzing light and shadow from multiple angles.
Tasks: Analyze the imput image/video, provide corresponding classification results for the following multiple categories, and return them in the specified output format.
1. Direction of Light:
Task 1: Analyze the image and classify the light source direction as front light, side light, back light, top light, bottom light, or split light. Identify the angle of the light source relative to the subject, and describe its effect shadow formation.
2. Light Source Type: Task 2: Analyze the emage and classify the light source type as either Natural Light, drtificial Light, Rendering Light.
3. Light Intensity:
Task 3: Analyze the image/video to assess the light intensity present. Classify the light intensity into three categories: Glare, Noderate, and Dim. Special attention should be given to situations where bright light sources may create a glaring effect even in otherwise dim envöronments.
4. Color Temperature:
Task 4: Analyze the smage/video to assess the color temperature present. Classify the color temperature into three categories: Cool Tone, Neutral, and Varm Tone.
5. Light Changes in Time: Task 5: Analyze the video
light changes time. Classify the light changes into two main categories: Static Light and Dynamic Light. For Dynamic Light, further categorize it into two subtypes: Intensity Gradient and Noving Light Source.
6. Optical Phenomena:
Task 6: Analyze the smage/video with a focus on the specific scene assess the optical phenomena present. Pay close attention to scenarios involving glass water surfaces, mirrors, and fog. Classify the phenomena into the following categories: Transmission (Glass), Defnaction/Reflecttee (Vater Surface, Hirror), Refraction/Ref (Nirror), Scattering (Fog Effect), and None.
Guidelines:
1. Accuracy: Assign each tag the most appropriate category and subcategory.
2. Hultople Tags: If an action föts multiple categories, assign all relevant tags.
3. Comprehenssveensss: Capture all detectable dynamic attributes without omissions.
4. JSON Validity: Ensure the output JSON is correctly formatted and adheres to the specified structure.
Example Output:
25

---

"Direction of Light"; "Pront Light", "Light Source Type"; "Artificial Light", "Light Intensity": "Hoderate", "Color Temperature": "Cool Tone", "Light Changes in Time": "Dynamic Light (Intensity Changing Light)", "Optical Phenomena"; "Transmission (Glass)"
Listing 1: Prompt Definition
C Additional Experimental Results
C.1 Additional Main Results
To further demonstrate the model's generalization to non-human scenes, we conducted additional evaluations two public object-centric relighting benchmarks: StanfordOrb (24) and Navi [19]. These datasets include objects and sculptures under a variety of lighting environments and are completely disjoint from training data. StanfordOrb contains canonical 3D scanned objects such as the Stanford bunny and dragon, while Navi includes wide range of everyday objects like containers, toys, and mugs. As shown in Tab. 5 (StanfordOrb) and Tab. (Navi). UniL achieves state-of-ther the-trrrtt results across perceptual (LPIPS), structural (SSIM), and physical (R-Motion) metrics, despite the significant domain gap and without any test-time fine-tuning, outperforming all baselines.
Table 5: Quantitative comparison on the StanfordOrb dataset. Bold number indicate the best performance.
Model
PSNR
SSIM
LPIPS 4
R-Motion +
IC-Light Per Frame
49
24.132
0.914
0.126
1.742
Videoo 557 + Cog VideoX
25.617
0.923
0.108
1.279
Light-A-Video 157
Wan2.1 38
25.784
0.926
0.104
1.241
UniLumos
26.512
0.934
0.097
1.103
Table 6: Quantitative comparison on the Navi dataset. Bold number indicate the best performance.
Model
PSNR
SSIM
LPIPS
R-Motion +
IC-Light Per Frame
[49
22,021
0.883
0.125
1.974
Light-A-Video [57 Cog VideoX
23.912
0.891
0.121
1.378
Light-A-Viden
57
Wan2.1 1 38
23.474
0.903
0.116
1.341
UniLumos
24.977
0.911
0.120
1.203
C.2 LumosBench: An Attribute-level Controllability Benchmark
To evaluate the fine-grained controllability of lighting generation, we introduce I LumosBench, structured benchmark that targets six core illumination attributes defined in our annotation protocol. Unlike prior works that treat lighting holistically implicitly, LumosBench provides disentangled, attribute-level evaluation, enabling precise diagnosis of model behavior under controllable lighting conditions.
Specifically, we construct set 2k test prompts, each consisting of a video and a structured caption designed to isolate one lighting attribute at time, while holding other variables constant. These prompts span six categories- dynamics, and optical phenomena-wiin multiple subtypes per category (e.g.. front/side/back for direction). This design facilitates controlled and interpretable evaluation across lighting axes that are often conflated in prior datasets.
light source type, intensity, color temperature, temporal
26

---

Table Quantitative comparison of the attribute-level controllability, Bold number indicate the best
performance.
APanam
Direction
Liglt Sounce Type
hncaally
Tanperaine
Tauguaal Dynanies
Opticail Pleasmenua
Arg. Scrte
Medils
Video
1:9
0.708
0.457
0.403
0.587
CopXatanX
1
5.68
0.817
0,601
0.550
0.730
(1513
0.469
0,644
Husyannaato
138
0.863
0.74)
0.590
0.502
01655
0.481
0,600
Wan2
33
1.38
0.842
0,083
0.436
0.341
0.504
0.433
0.007
Wanz
148
0.871
0.794
0.614
0,529
0.737
0.305
0,735
Speciialized Model
ighn Por-Frure
0.91
1.99%
0:491
0.244
0:319
D.463
Light
0.327
0:356
0.373
0.16
Light A
57
Wan2. [38]
2.20
0,361
0.522
0.577
0,422
0.553
Unil
eaptiots
1.38
0.529
0.798
0.545
0.457
0,662
Unit mmmis
1:38
0,803
0,847
0A82
0,313
(1952
0,392
0,773
To assess alignment between intended and generated lighting attributes, we use the vision-language model Qwen2.5-V1 [1] to analyze relit outputs and classify whether the target attribute is correctly expressed. Each dimension is scored independently, and the final controllability score is the average across all six dimensions.
and task-specific relighting models. This benchmark allows us not only to assess overall lighting quality but also to dis- sect a model's ability to interpret and respond individual lighting controls. Among general models, Wan 14B shows the highest raw capability, demonstrating the strength of large-scale pretraining for visual generation. Notably, our fine-tuned Wan 1.3B variant achieves substantial gains across all six lighting dimensions, surpassing even much larger models. This highlights the benefit of relighting- specific supervision: fine-tuning on LunosData with structured lighting annotations significantly enhances the model's ability to reason about illumination in controllable and disentangled manner.
General vs. Specialized Models. Tab. 7 presentt results for both
By contrast, specialized relighting models consistently underperform despite being designed for lighting manipulation. This is primarily due to the limitations of their base prchitectures-typiice smaller and trained from scratch or on narrow lighting attributes. While they may encode some prior knowledge of lighting physics (e.g., through latent constraints), their restricted modeling capacity hinders semantic alignment with user-intended lighting conditions. These findings underscore the importance of starting from strong pre-trained backbone and introducing structured, high-level lighting supervision achieve controllable and physically plausible relighting.
weaker generalization diverse
Effectiveness of Structured Captions. Within the specialized group, we conduct ablation assess the importance of our proposed lumos captions. w/o lumos captions uses only vanilla scene-level captions during training, omitting structured lighting tags. The performance prop-particularlyy in controllable dimensions like intensity and optical
confirms that our semantic annotations play key role in teaching the model fine-graine mne-grained illumination control. Compared to strong baselines. UniLumos achieves superior scores across nearly all dimensions, demonstrating the impact of LumosBench in pushing model understanding and control of illumination.
D Additional Result Visualization
We present additional image relighting results in Fig.
We present additional background-conditione video relighting results in Fig. and Fig.
10
E Limitation and Future Work
UniLumos is still limited by a broader challenge-achieving physically precise and controllable relighting. UniLumos enforces geometry-aware consistency (e.g., shadows aligned with depth and normals) but does not yet produce physically quantifiable lighting outputs such as radiance or illuminance. Future work may explore finer control over lighting, including editable key lights, intensity ramps, and environmental reflections.
27

---

Input
Relight
Input
Relight
Bottom
Left
Bottom
Right
None
Bottom
Left
Right
Left
Top
Right
Left
Figure 8: UniLumos performs physically plausible image relighting, conditioned on textual prompts.
F
Broader Impact
UniLumos can support creative and technical applications such as virtual cinematography, augmented reality, and digital design. However, generative relighting also raises concerns about visual authentic- ity, content manipulation, and social perception. Our model does not perform identity synthesis or create visual entities, but it can alter illumination in ways that affect narrative tone realism. When used in storytelling or media production, such changes may influence audience interpretation. We encourage transparent and responsible use, including disclosure when relit content appears in downstream applications. Responsible innovation in this area requires anticipating potential misuse, supporting verification tools, promoting media literacy, and engaging with stakeholders define fair and ethical use. Our goal is to enable creative expression while maintaining viewer trust and media integrity.
G Safeguards
To promote responsible use and reduce risks of generative misuse, we include several safeguards the development and release UniLumos. All training data come from public datasets with appropriate licenses and contain personally identifiable information. We also recommend that downstream users apply protective measures such as watermarking, provenance tracking, and clear usage disclosures to support transparency. These practices aim to ensure safe deployment and maintain public trust in generative relighting systems.
28

---

Frame
Frame 24
Frame 48
Input
Relight
Reference Background Video
Figure UniLumos performs physically plausible video relighting, conditioned reference vidcos.
Frame
Frame 24
Frame 48
Input
Relight
Reference Background Video
Figure 10: UniLumos performs physically plausible video relighting, conditioned on reference videos.
29


Reviews:
[{"review_id": "4Dh1Usigst", "reviewer": "Reviewer_hicd", "summary": "UniLumos is a framework building on the Wan 2.1 video diffusion model to enhance the speed and physical plausibility of image and video relighting. The work introduces several techniques for accelerated inference to the task of video and image relighting. Specifically, a few-step flow-matching backbone is extended with path consistency learning. Additionally, monocular depth and normal data are incorporated into an additional loss to improve physical consistency by comparing the extracted depth and normals for the generation to the pseudo ground truth. For training a data generation pipeline, LumosData is introduced, which generates labeled data using a VLM and prior relighting work IC Light for annotation and augmentation. The evaluation shows state-of-the-art results at much lower inference times compared to similar methods.", "questions": "- How does the model perform on real-world relighting data like e.g. the Navi or StanfordOrb dataset for still images and SDSD (dvlab) or LightAtlas (RelightVid) for video? Rendered, synthetic ground truth would also work for the evaluation here as long as it is path traced using a physical renderer.\n\n- Related works: Controllable diffusion models could also include the strand of works around ConceptSliders [1] which offers parametric control in the context of diffusion models. Why are the following relighting methods not included in the related works section and (potentially) evaluation? Intrinsic Image Diffusion [2], Neural Gaffer [3], DI-Light [4].\nHave they been evaluated and why has IC Light been chosen for augmentation as it might not be the best method for the task? Is the limited consistency of IC Light actually helping the robustness of the training? This could be analyzed in some additional ablation.\n\n- p.1, l.5: Why are overexposed highlights physically implausible? In tone-mapped images this is a natural occurrence that often makes it even look more realistic than compressed highlights.\n\n- Table 1 and Figure 4: What data has been used here? It would be helpful to add the evaluation data sources to the captions. I don't think \"relited\" exists as a word (Fig. 4 caption, for example). Probably, relit would be fitting here.\n\n\n1: Gandikota et al. Concept Sliders: LoRA Adaptors for Precise Control in Diffusion Models. ECCV 2024.\n2: Kocsis et al. Intrinsic image diffusion for indoor single-view material estimation. CVPR 2024.\n3: Haian et al. Neural Gaffer: Relighting Any Object via Diffusion, NeurIPS 2024\n4: Chong et al. Dilightnet: Fine-grained lighting control for diffusion-based image generation. SIGGRAPH 2024.", "limitations": "Limitations and broader impact are discussed in the appendix but it would be better to add a condensed version to the main paper.", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n- The method shows an impressive improvement on inference times.\n- Qualitative results look promising.\n- The data generation pipeline presents a helpful approach to semi-self-supervised training of relighting tasks by generating augmentations and annotations using existing (potentially weaker) models.\n\nWeaknesses:\n- By using existing generative models for data generation, there is a distribution gap between the training data and real-world videos. The model is only evaluated quantitatively using the data generated with this pipeline.\n- Using existing modules that have also been used for the given task. The normal and depth constraints have also been employed for supervision of e.g. optimizations of NeRFs (e.g. [1]) and 3D Gaussian Splatting (e.g. [2]). Path consistency is novel to the relighting task but is also an established idea by now.\n- Dynamic light does not seem to work well (based on the example clips in the supplements). This might be an artifact of the augmentation scheme (with a temporally inconsistent model). In case 2 of the supplementary material the guitar keeps some specular reflections through all illumination settings, which looks like an artifact.\n- Sometimes the paper has somewhat redundant text passages (e.g. at the beginning of the ablation study).\n\nOverall, the paper presents some nice engineering effort and promising results but offers limited insights into algorithmic advancements to the relighting task as it mostly (re-)combines existing ideas. The evaluation is lacking comparisons outside the training data distribution.\n\n1: Niemeyer et al. RegNeRF: Regularizing Neural Radiance Fields for View Synthesis from Sparse Inputs. CVPR 2022\n2: Turkulainen et al. DN-Splatter: Depth and Normal Priors for Gaussian Splatting and Meshing. WACV 2025", "was_revised": true, "final_justification": "I appreciate the authors' detailed response and the additional evaluations on the Navi and Stanford Orb relighting datasets, which provide promising evidence of the model’s effectiveness. The inclusion of more diverse qualitative results further strengthens the submission. With the manuscript being revised into a more polished form (related works section, specification of test set), I am inclined to raise my final score. That said, the overall impact of the work could be further enhanced by releasing the dataset or the dataset generation pipeline to the community.", "rebuttal": "Thanks for your comments, and we address them below.\n> **Q1** By using existing generative models for data generation, there is a distribution gap between the training data and real-world videos. The model is only evaluated quantitatively using the data generated with this pipeline.\n\n**A1** All evaluations in our paper are conducted on real, held-out videos from our internal VCG dataset—not on synthetic or IC-Light-generated inputs. During testing, the model receives real foreground and background videos along with a lighting caption, and generates relit outputs accordingly. The strong performance shown in Table 1 demonstrates that our model generalizes well to real-world video, despite being trained on augmented data. This ability is largely due to our physics-guided feedback mechanism, which helps the model generalize beyond the artifacts of synthetic training data.\n\n> **Q2** Using existing modules that have also been used for the given task. The normal and depth constraints have also been employed for supervision of e.g. optimizations of NeRFs (e.g. [1]) and 3D Gaussian Splatting (e.g. [2]). Path consistency is novel to the relighting task but is also an established idea by now.\n\n**A2** While we adopt existing components, our contribution lies in designing a unified framework for fast and physically-plausible video relighting—rather than proposing individual modules. The core of our system is the non-trivial integration of RGB-space geometric feedback with few-step path consistency, applied in a low-supervision setting where conventional geometric signals are weak. Beyond the model, we also introduce a structured annotation pipeline (LumosData) and a benchmark (LumosBench) that enable fine-grained, attribute-level evaluation. Together, these form a complete relighting system tailored to controllability, generalization, and efficiency.\n\n> **Q3** Dynamic light does not seem to work well (based on the example clips in the supplements). This might be an artifact of the augmentation scheme (with a temporally inconsistent model). In case 2 of the supplementary material the guitar keeps some specular reflections through all illumination settings, which looks like an artifact.\n\n**A3** Our current framework does not explicitly model dynamic lighting, which can cause artifacts under complex conditions. Future work will explore incorporating dynamic cues, such as time-varying lighting descriptors or motion-aware supervision, to better handle such cases.\n\n> **Q4** Sometimes the paper has somewhat redundant text passages (e.g. at the beginning of the ablation study).\n\n**A4** We have revised the manuscript for clarity and conciseness, with particular focus on removing redundancy in the introduction and ablation study sections.\n\n> **Q5** How does the model perform on real-world relighting data like e.g. the Navi or StanfordOrb dataset for still images and SDSD (dvlab) or LightAtlas (RelightVid) for video? Rendered, synthetic ground truth would also work for the evaluation here as long as it is path traced using a physical renderer.\n\n**A5** Our initial focus was on a simple, user-friendly pipeline using RGB inputs, which made datasets requiring HDR or complex scene data (e.g., LightAtlas) less suitable. However, we recognize the value of established benchmarks and have added evaluations on Navi and StanfordOrb (RelightVid is not publicly available). Despite being trained on Panda70M, our model performs well on these datasets, demonstrating good generalization. Visualizing results are included in the updated paper. \n\nIn the revised manuscript, we report results (see below) on three datasets: Navi, StanfordOrb, and our LumosData. Across all benchmarks, UniLumos consistently outperforms prior methods in PSNR, SSIM, and temporal consistency, while maintaining strong perceptual quality (LPIPS). These results confirm that our model generalizes well to diverse lighting domains, including real-world and synthetic, and validates its robustness beyond the training distribution.\n\nTable: Quantitative comparison. **Bold** numbers indicate the best performance.\n\n**Navi**\n\n| **Model**| PSNR ↑ | SSIM ↑ | LPIPS ↓ | R-Motion ↓ |\n|---|---|---|---|---|\n| IC-Light Per Frame       | 22.021 | 0.883  | 0.125    | 1.974 |\n| Light-A-Video + CogVideoX| 23.912 | 0.891  | 0.121    | 1.378 |\n| Light-A-Video + Wan2.1   | 23.474 | 0.903  | **0.116**| 1.341 | \n| **UniLumos**             |**24.977**| **0.911** | 0.120 | **1.203** |\n\n**StanfordOrb** \n\n| **Model**| PSNR ↑ | SSIM ↑ | LPIPS ↓ | R-Motion ↓ | \n|---|---|---|---|---|\n| IC-Light Per Frame       | 24.132 | 0.914 | 0.126 | 1.742 | \n| Light-A-Video + CogVideoX| 25.617 | 0.923 | 0.108 | 1.279 | \n| Light-A-Video + Wan2.1   | 25.784 | 0.926 | 0.104 | 1.241 | \n| **UniLumos** | **26.512** | **0.934** | **0.097** | **1.103** |\n\n**LumosData (our)**\n\n|**Model**|PSNR↑|SSIM↑|LPIPS↓|R-Motion↓| \n|---|---|---|---|---|\n| IC-Light Per Frame | 20.132 | 0.851  | 0.133 | 2.437 |\n| Light-A-Video + CogVideoX | 19.851 | 0.859  | 0.124 | 1.784 | \n| Light-A-Video + Wan2.1    | 20.784 | 0.876  | 0.129 | 1.582 | \n| **UniLumos** | **25.031** | **0.891** | **0.109** | **1.436**|\n\n> **Q6** Related works: Controllable diffusion models could also include the strand of works around ConceptSliders [1] which offers parametric control in the context of diffusion models. Why are the following relighting methods not included in the related works section and (potentially) evaluation? Intrinsic Image Diffusion [2], Neural Gaffer [3], DI-Light [4]. Have they been evaluated and why has IC Light been chosen for augmentation as it might not be the best method for the task? Is the limited consistency of IC Light actually helping the robustness of the training? This could be analyzed in some additional ablation.\n\n**A6** We have added relevant controllable diffusion models to the related work section. Some prior relighting methods were not included due to differences in task formulation—many require detailed scene geometry, HDR inputs, or full-scene relighting, which do not align with our goal of a lightweight pipeline based on foreground, background, and caption inputs. Our focus is on controllable relighting under minimal, user-friendly input assumptions.\n\nIC-Light was selected for data augmentation due to its speed, availability, and ability to generate diverse lighting effects. In future work, we plan to incorporate multiple complementary augmentation models into the data pipeline to improve temporal consistency and enrich lighting diversity.\n\nWe explored incorporating IC-Light–style consistency during UniLumos training, but observed no clear performance gain (see below). This is likely due to the highly compressed nature of the latent spatiotemporal space, where consistency signals are weak—even with learnable MLP-based transitions. As a result, the added supervision provided little benefit in practice.\n\n> **Q7** p.1, l.5: Why are overexposed highlights physically implausible? In tone-mapped images this is a natural occurrence that often makes it even look more realistic than compressed highlights.\n\n**A7** Our concern was with physically inconsistent highlights, such as those misaligned with surface geometry or lighting direction. We’ve revised the text to clarify this point, replacing “physically implausible” with more precise terms like “misaligned highlights” or “geometrically inconsistent lighting.”\n\n> **Q8** Table 1 and Figure 4: What data has been used here? It would be helpful to add the evaluation data sources to the captions. I don't think \"relited\" exists as a word (Fig. 4 caption, for example). Probably, relit would be fitting here.\n\n**A8** All results use data from the VCG test set. We will update the captions accordingly and have corrected the typo “relited” to “relit” in the revised manuscript."}, {"review_id": "p84lyoehXB", "reviewer": "Reviewer_gRR4", "summary": "This paper proposes an image and video relighting method. The method is built upon a video diffusion Wan 2.1 and extends it to handle lighting. It constructs a new dataset for training such a model. The method introduces a shortcut-based consistency loss that enables faster inference with variable step sizes. Additionally, it leverages  single-step diffusion models for depth and normal prediction to provide geometry-aware supervision before and after relighting. It achieves temporally consistent, physically plausible, and efficient relighting in videos.", "questions": "1. In Fig.4 (left) and Fig.3 (Appendix), the input video clips appear to depict the same scene under different lighting conditions. However, there seems to be a slight difference in scale, particularly noticeable in the bottom-left corner where the three circles in Fig.4. Could the authors clarify if any cropping or resizing was applied, and whether this affects evaluation?\n2. The dataset includes optical attributes such as “Transmission (Glass)”, yet the paper does not show examples involving such materials. Given that the visual results primarily focus on portrait relighting, could the authors comment on the performance of the method when applied to scenes involving transparent or refractive materials?\n3. Have the authors evaluated how the method performs across a diverse range of subjects, including individuals with varying skin tones and non-human objects? This would help clarify the model’s generalization ability beyond the portrait-centric examples shown.", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. Quality: The relighting results are visually compelling, with strong temporal consistency across frames and realistic lighting effects. The proposed method demonstrates solid performance, particularly in portrait scenarios.\n2. Clarity: The technical exposition is generally sound, and the paper provides sufficient implementation details for reproducibility. The pipeline is well-structured and logically motivated.\n3. Significance: The paper tackles a relatively unexplored task of lighting control in video diffusion models. The direction, ideas, and results are promising and the problem is quite interesting.\n4. Originality: The paper uses single-step diffusion models for additional depth/normal supervision and a shortcut-based consistency loss for efficient inference. These elements contribute to faster and geometry-aware relighting.\n\nWeaknesses:\n1. Quality: One major concern with the paper is that it misses a lot of related work. Despite the paper’s focus on scene relighting, the method largely targets portrait relighting. It is better to cite and discuss other important methods. Below are major ones but not limited to:\n  - a. Total Relighting: Learning to Relight Portraits for Background Replacement, SIGGRAPH 21.\n  - b. Lumos: Learning to Relight Portrait Images via a Virtual Light Stage and Synthetic-to-Real Adaptation, SIGGRAPHa 22.\n  - c. Neural Video Portrait Relighting in Real-time via Consistency Modeling, ICCV 21.\n  - d. Real-time 3D-aware Portrait Video Relighting, CVPR 24.\n\nFurthermore, while the geometric supervision via normals and depth is helpful, it can compromise appearance fidelity in some cases. For instance, in Figure 4 (right), the appearance of the cup deviates noticeably from the input and other methods, indicating a drop in visual consistency.\n\n2. Clarity: The manuscript would benefit from thorough proofreading. The abstract contains grammatical issues (e.g., the phrase “…where controlling illumination under ensures physical consistency…” is unclear). Other minor issues include:\n  - a. Section title \"1. Introudction\" should be \"1. Introduction\"\n  - b. References [2] and [3] are duplicated\n3. Significance: The results are shown primarily on portrait videos with limited motion, which constrains the method’s applicability to more diverse or dynamic real-world video scenarios. \n4. Originality: The use of degraded data as input closely resembles prior work such as RelightVid. Similarly, treating single images as one-frame (T=1) videos is a straightforward extension within the video diffusion framework. The major contributions lie in path consistency loss for faster inference and using single-step pretrained diffusion models for extra supervision. As a result, the overall novelty of the approach is somewhat limited.", "was_revised": true, "final_justification": "The rebuttal and discussion have addressed most of my concerns; therefore, I would like to raise my rating, under the authors' promise that the revised version should reflect the response during this discussion period.", "rebuttal": "Thanks for your comments, and we address them below.\n> **Q1** One major concern with the paper is that it misses a lot of related work. Despite the paper’s focus on scene relighting, the method largely targets portrait relighting. It is better to cite and discuss other important methods. Below are major ones but not limited to:\n\n**A1** We have revised the Related Work to include and discuss key portrait relighting methods such as Total Relighting (SIGGRAPH '21), Lumos (SIGGRAPHa '22), Neural Video Portrait Relighting (ICCV '21), and Real-time 3D-aware Portrait Video Relighting (CVPR '24). While these works often rely on portrait-specific priors like 3DMMs, our method is designed as a general-purpose relighting framework applicable to both images and videos without being limited to a specific object category.\n\n> **Q2** Furthermore, while the geometric supervision via normals and depth is helpful, it can compromise appearance fidelity in some cases. For instance, in Figure 4 (right), the appearance of the cup deviates noticeably from the input and other methods, indicating a drop in visual consistency.\n\n**A2** There is an inherent trade-off between geometric consistency and appearance fidelity. In Fig. 4 (right), the model prioritizes physically plausible lighting, which can lead to deviations in secondary objects like the cup near subject boundaries. This results from the multi-objective loss balancing fidelity L_0 and physical realism L_{phy}. Removing geometric feedback nearly doubles the geometric error (0.147->0.297), confirming its impact. \n\nTo reduce appearance shifts, we adopt a two-stage training strategy: the model is first trained with L_0 only, then fine-tuned with the full loss. This improves appearance consistency while preserving physical plausibility, which can reduce visual artifacts by 19% compared to joint training from scratch.\n\n> **Q3** The manuscript would benefit from thorough proofreading. The abstract contains grammatical issues (e.g., the phrase “…where controlling illumination under ensures physical consistency…” is unclear). Other minor issues include:\n\n**A3** We have rewritten the abstract for clarity, corrected the typo in the “Introduction” heading, and removed the duplicate reference. A full proofread of the manuscript will be completed before submission.\n\n> **Q4** The results are shown primarily on portrait videos with limited motion, which constrains the method’s applicability to more diverse or dynamic real-world video scenarios.\n\n**A4** While some examples focus on portraits, our method is designed as a general-purpose relighting framework. The model is trained on diverse videos from Panda70M, including objects, animals, and dynamic scenes—not limited to human subjects. Figures in the main paper and appendix (e.g., beach and parking lot scenes) already reflect this variety. We will add additional qualitative results on non-human, non-portrait scenarios in the appendix to further demonstrate applicability to general video relighting beyond portrait settings.\n\n> **Q5** Originality: The use of degraded data as input closely resembles prior work such as RelightVid. Similarly, treating single images as one-frame (T=1) videos is a straightforward extension within the video diffusion framework. The major contributions lie in path consistency loss for faster inference and using single-step pretrained diffusion models for extra supervision. As a result, the overall novelty of the approach is somewhat limited.\n\n**A5** While we adopt a degraded-to-real training setup similar to RelightVid, our approach differs in both supervision design and evaluation framework. Rather than relying on latent-space consistency, we introduce explicit RGB-space physical feedback via a differentiable loss based on estimated normals and depth, enforcing alignment between lighting effects and scene geometry. This physically grounded signal is absent in prior work. In addition, our LumosData pipeline and LumosBench benchmark enable structured, attribute-level control and evaluation across a six-dimensional lighting space. Together, these contributions target controllability and physical realism in a unified framework, going beyond prior single-object or latent-only approaches.\n\n> **Q6** In Fig.4 (left) and Fig.3 (Appendix), the input video clips appear to depict the same scene under different lighting conditions. However, there seems to be a slight difference in scale, particularly noticeable in the bottom-left corner where the three circles in Fig.4. Could the authors clarify if any cropping or resizing was applied, and whether this affects evaluation?\n\n**A6** The two clips are from the same source video, shown under different lighting to illustrate relighting behavior. The slight scale mismatch is a rendering artifact from figure layout and does not affect evaluation—models were trained and tested on standardized-resolution videos (e.g., 832x480) without cropping. To avoid confusion, we will update the final version to use different source videos in these figures.\n\n> **Q7** The dataset includes optical attributes such as “Transmission (Glass)”, yet the paper does not show examples involving such materials. Given that the visual results primarily focus on portrait relighting, could the authors comment on the performance of the method when applied to scenes involving transparent or refractive materials?\n\n**A7** Although “Transmission (Glass)” is part of our annotation protocol, such scenes are rare in the dataset, and our current model is not designed to explicitly handle transparent or refractive materials. As an initial reference, we will add qualitative examples involving glass-like materials in the appendix, along with a brief discussion of current limitations and directions for future work.\n\n> **Q8** Have the authors evaluated how the method performs across a diverse range of subjects, including individuals with varying skin tones and non-human objects? This would help clarify the model’s generalization ability beyond the portrait-centric examples shown.\n\n**A8** Yes, as noted in our response to Q4, we will include additional results on non-human objects in the appendix to demonstrate generalization beyond portraits. Regarding skin tone diversity, our model is trained on Panda70M, which includes a broad range of human appearances. Qualitatively, we have not observed performance degradation across skin tones. However, we have not performed a formal fairness analysis with disaggregated metrics."}, {"review_id": "PaiILmbq8I", "reviewer": "Reviewer_4Tza", "summary": "This submission develops a framework, from data collection to modelling, for unified image and video relighting. The authors collect a large video and image relighting dataset, LumosData, using an automated pipeline. Based on the data, the authors propose a text-conditioned video relighting model that works better than prior training free approaches or image relighting approaches.", "questions": "- As mentioned in the weakness section, please justify the task of text-based relighting. \n- For Table 1 in the supplementary, what is the “UniLumos w/o Path Consistency” baseline? Is it w/o the consistency loss but using the same sampling iterations? Or is it w/o the loss but sampling more iterations? It looks better than w/ the consistency loss so it’s necessary to clarify what it is.", "limitations": "Yes.", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "- **Strengths**:\n\t- According to the results in Fig. 4 and Tab. 1, the proposed framework clearly improve over the previous training-frame approach (Light-A-Video) and trained image-relighting baseline (IC-Light). This shows a very promising signal along the presented technical direction.\n\t- The LumosData, if the author can release it, can be a valuable contribution to the community. My final decision will consider the releasing plan of this data.\n     - The path consistency objective and the normal / depth consistency objective, while neither is brand new, seem quite effective in improving the relighting results, according to Tab. 1 in the supplementary. \n\n- **Weakness**:\n\t\n   - First, I’d like to challenge the task itself of text-based relighting, i.e. relighting using text to describe the new lighting condition and generate an image / video for it. The biggest concern is that it is fairly challenging to describe a lighting condition. Imagine how hard it is to “reconstruct” an exact lighting condition just from the text description of it. Also imagine how many possible lighting conditions can fit into the description of “Front Light, Artificial Light, Moderate, Neutral, Static Light”. Practically, it’s not how the film industry or game industry describes lighting either, considering these might be the use cases of the proposed method. I hope the authors could justify this task in the rebuttal. This is important to justify the value of the proposed data and model because both are dependent on the assumption the lighting is described in text.\n   - While the quantitative results showed promising improvements, the qualitative results still reveal clear limitations. For instance, in case 2 and case 3 of the videos, for the parking garage scene, it is obvious that the specularity on the face caused by the lamps on top are not lit correctly because they don’t appear to move although the background video is clearly moving forward.\n   - The technical novelty is limited. The proposed framework is largely built on slightly modified existing methods of architecture. This should not be a big problem for acceptance if other concerns are addressed. However, this is the reason for higher acceptance ratings.\n   - While it is generally challenging to evaluate the effect of the LumosData, it can be interesting to understand it via fine-tuning IC-Light on the image only portion and see how it improves after using LumosData.", "was_revised": true, "final_justification": "I thank the authors' efforts in preparing a polished rebuttal. My questions in the initial review were address. Particularly, the argument of the task setup as attribute-based textual description makes sense to me now. So i'll keep my original acceptance rating. As I mentioned, the reason for not giving higher ratings is the scale of technical novelty. The authors response didn't add new dimensions of contributions that I already acknowledged. Additionally, the LumosData release is not promised either. So my final rating is still borderline accept.", "rebuttal": "Thanks for your comments, and we address them below.\n\n> **Q1** First, I’d like to challenge the task itself of text-based relighting, i.e. relighting using text to describe the new lighting condition and generate an image / video for it. The biggest concern is that it is fairly challenging to describe a lighting condition. Imagine how hard it is to “reconstruct” an exact lighting condition just from the text description of it. Also imagine how many possible lighting conditions can fit into the description of “Front Light, Artificial Light, Moderate, Neutral, Static Light”. Practically, it’s not how the film industry or game industry describes lighting either, considering these might be the use cases of the proposed method. I hope the authors could justify this task in the rebuttal. This is important to justify the value of the proposed data and model because both are dependent on the assumption the lighting is described in text.\n \n**A1** We agree that describing lighting through free-form text alone is ambiguous and insufficient for precise control. That’s why our method does not rely on unconstrained natural language. Instead, we define lighting via a structured 6D attribute representation—direction, intensity, source type, color temperature, temporal dynamics, and optical effects. Text is merely a user-facing interface for presenting these attributes.\n\nThis design has two key benefits: (1) it makes the task more learnable for the model, since each attribute corresponds to interpretable and disentangled supervision; and (2) it improves accessibility for users, compared to HDR-based or physically-based lighting control, which often requires professional expertise or complex scene representations. In contrast, our system supports controllable, prompt-based lighting manipulation using only RGB inputs, which is more practical for many real-world applications.\n\nTo ensure this structured control is actually meaningful, we introduced LumosBench, which evaluates how well the model responds to each attribute individually. High scores across categories (e.g., Direction: 0.893, Color Temp: 0.813) demonstrate the effectiveness and learnability of our attribute-driven design.\n\n> **Q2** While the quantitative results showed promising improvements, the qualitative results still reveal clear limitations. For instance, in case 2 and case 3 of the videos, for the parking garage scene, it is obvious that the specularity on the face caused by the lamps on top are not lit correctly because they don’t appear to move although the background video is clearly moving forward.\n\n**A2** The static specular highlight results from the model not explicitly modeling full 3D scene geometry or light source positions. Although we use depth and normals, the lack of full 3D reconstruction limits view-dependent effects. Addressing this requires more explicit 3D reasoning, which we note as future work.\n\n> **Q3** The technical novelty is limited. The proposed framework is largely built on slightly modified existing methods of architecture. This should not be a big problem for acceptance if other concerns are addressed. However, this is the reason for higher acceptance ratings.\n\n**A3** Our contributions lie in combining them to address a new task with new constraints: (1) integrating RGB-space feedback with few-step path consistency to balance physical accuracy and efficiency, and (2) building a full pipeline (LumosData + LumosBench) for structured, controllable relighting. We appreciate that novelty is not a major blocker and hope our responses clarify the value of this framework.\n\n> **Q4** While it is generally challenging to evaluate the effect of the LumosData, it can be interesting to understand it via fine-tuning IC-Light on the image only portion and see how it improves after using LumosData.\n\n**A4** Fine-tuning IC-Light is not applicable in our setting, as it is an image-based method while our pipeline and benchmark are designed for videos. Instead, we directly assess the value of LumosData through targeted ablations. In Table 3, removing the structured captions drops controllability from 0.773 to 0.662, showing their importance for fine-grained control.\n\nTo systematically evaluate controllability, we introduce LumosBench, a benchmark that tests each of the six lighting attributes (e.g., direction, intensity, color temperature) in isolation. It contains 2k structured prompts, each targeting one attribute while holding others fixed. We use a vision-language model (Qwen2.5-VL) to assess whether the intended lighting change is correctly expressed. This disentangled, attribute-level evaluation provides a precise and interpretable way to measure how well models respond to structured lighting control (see below).\n\nTable: Quantitative comparison of attribute-level controllability. **Bold** numbers indicate the best performance.\n\n| **Model** | #Params | Direction | Light Source Type | Intensity | Color Temperature | Temporal Dynamics | Optical Phenomena | Avg. Score |\n|-----------|---------|-----------|-------------------|-----------|-------------------|-------------------|--------------------|-------------|\n| **General Models** |||||||||\n| LTX-Video          | 1.9B  | 0.794 | 0.644 | 0.487 | 0.708 | 0.487 | 0.403 | 0.587 |\n| CogVideoX          | 5.6B  | 0.837 | 0.692 | 0.552 | 0.739 | 0.532 | 0.449 | 0.634 |\n| HunyuanVideo       | 13B   | 0.863 | 0.741 | 0.599 | 0.802 | 0.655 | 0.481 | 0.690 |\n| Wan2.1             | 1.3B  | 0.842 | 0.685 | 0.436 | 0.741 | 0.504 | 0.433 | 0.607 |\n| Wan2.1             | 14B   | 0.871 | 0.794 | 0.674 | **0.829** | **0.737** | 0.505 | 0.735 |\n| **Specialized Models** |||||||||\n| IC-Light Per-Frame | 0.9B  | 0.793 | 0.547 | 0.349 | 0.493 | 0.284 | 0.339 | 0.468 |\n| Light-A-Video + CogVideoX | 2.9B | 0.787 | 0.581 | 0.327 | 0.536 | 0.493 | 0.373 | 0.516 |\n| Light-A-Video + Wan2.1    | 2.2B | 0.801 | 0.603 | 0.361 | 0.582 | 0.557 | 0.412 | 0.553 |\n| **UniLumos w/o lumos captions**   | 1.3B | 0.868 | 0.774 | 0.529 | 0.798 | 0.543 | 0.457 | 0.662 |\n| **UniLumos**                      | 1.3B | **0.893** | **0.847** | **0.832** | 0.813 | 0.662 | **0.592** | **0.773** |\n\nWithin the specialized group, we conduct an ablation to assess the importance of our proposed lumos captions. \n\n**w/o lumos captions** uses only vanilla scene-level captions during training, omitting structured lighting tags. The performance drop—particularly in controllable dimensions like intensity and optical phenomena—confirms that our semantic annotations play a key role in teaching the model fine-grained illumination control. Compared to strong baselines, UniLumos achieves superior scores across nearly all dimensions, demonstrating the impact of LumosBench in pushing model understanding and control of illumination.\n\n> **Q5** As mentioned in the weakness section, please justify the task of text-based relighting.\n\n**A5** We clarify that the task of “text-based relighting” in our work refers specifically to structured, attribute-driven control—not free-form language description. We formulate lighting as a 6D vector (e.g., direction, intensity, source type, etc.), which makes the task well-defined, learnable, and interpretable. While text serves as the interface, the core formulation is structured control, enabling practical and generalizable relighting without relying on complex HDR inputs or scene geometry.\n\n> **Q6** For Table 1 in the supplementary, what is the “UniLumos w/o Path Consistency” baseline? Is it w/o the consistency loss but using the same sampling iterations? Or is it w/o the loss but sampling more iterations? It looks better than w/ the consistency loss so it’s necessary to clarify what it is.\n\n**A6** “UniLumos w/o Path Consistency” refers to a model trained without the consistency loss (L_fast), evaluated with the same few-step inference schedule as the full model. While it scores slightly higher in PSNR/SSIM, it performs worse on perceptual (LPIPS: 0.113 vs. 0.109) and temporal (R-Motion: 1.438 vs. 1.436) metrics. The baseline lacks few-step consistency training, so pixel-level gains may be incidental. We will clarify this in the revised."}, {"review_id": "W5Qh89O2jB", "reviewer": "Reviewer_hxVQ", "summary": "This paper proposes a method to change the lighting of a pre-specified foreground subject, within a still image or a video.  There are two steps:  generating a dataset of re-lit videos, which they call LumosData. They constructed that dataset based on Panda70M, which led them to 110K relit video pairs.  It's not clear (please clarify) whether that dataset will be made public--line 674, \"We do not release any data or model that has high risks for misuse\".    An image relighting dataset certainly could have a risk of misuse, so does that mean they don't release that dataset, or they don't feel it has a risk of misuse?\nThe second step: they run a latent diffusion process that has an innovation:  a \"physics-plausible\" feedback mechanism, to encourage physically realistic illumination.", "questions": "See questions within Strengths and weaknesses", "limitations": "Authors didn't address possible concerns about faking videos or placing someone in a place where they never were.", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "strength:  The results look nice, and the comparison of inference times versus related algorithms is impressive.\n\n\nweaknesses:  We see just a few examples, even including the appendix and supplemental materials.\nIt would be nice to see visually the effect of ablations regarding the claimed essential contribution of the paper, the physics-plausible feedback mechanism.  What do the re-rendered images look like when it is turned off?\n\nSome parts of the paper make it seem that the paper wasn't proofread:  eg, the two successive sentences in lines 55-59.\n\nis the physics-aware feedback module indeed aware of the physics?  Do you have justification for calling it physics aware?\nI feel like I'm missing various details from the paper. What is the precise form of the Feedback listed in Figure 3?\nWill you release the LumosData?  Will you release the code?           \n\nI have concerns about the reproducibility and clarity of the paper.\n\nPlease expand on lines 217 - 222.", "was_revised": true, "final_justification": "Authors addressed concerns about the paper writing and the physics-aware module.", "rebuttal": "Thanks for your comments, and we address them below.\n\n> **Q1** We see just a few examples, even including the appendix and supplemental materials. It would be nice to see visually the effect of ablations regarding the claimed essential contribution of the paper, the physics-plausible feedback mechanism. What do the re-rendered images look like when it is turned off?\n\n**A1** We have added Fig. 5 to the revised manuscript to visually illustrate the effect of removing the physics-plausible feedback. Compared to the full UniLumos model, the variant without feedback produces flatter lighting with weaker shading and less realistic highlights. With feedback, the results show more coherent lighting aligned with subject geometry, including softer shadows and improved depth cues. This visual comparison highlights the impact of the feedback mechanism and supports its effectiveness as a core component of our method.\n\n> **Q2** Some parts of the paper make it seem that the paper wasn't proofread: eg, the two successive sentences in lines 55-59.\n\n**A2** We have removed the redundant sentences and thoroughly proofread the manuscript to improve clarity throughout.\n\n> **Q3**: Is the physics-aware feedback module indeed aware of the physics? Do you have justification for calling it physics aware?\n\n**A3** Yes, we use the term “physics-plausible” to indicate that the model is explicitly guided by scene geometry—rather than simulating full physical light transport. The feedback loss (L\\_phy) enforces consistency between depth and normal maps extracted from the output and a reference image. This encourages lighting effects that respect 3D structure, helping prevent artifacts like misaligned shadows.\n\n> **Q4** What is the precise form of the Feedback listed in Figure 3? \n\n**A4** The “Feedback” in Fig. 3 refers to the physics-plausible loss (L_phy). It compares depth and normal maps extracted from the model output (via a frozen estimator) to those from the reference, using normalized L2 loss. \n\n> **Q5** Will you release the code?\n\n**A5** Yes, we will release the full source code, and the LumosBench benchmark.\n\n> **Q6** What is LumosBench?\n\n**A6** Existing relighting benchmarks often evaluate lighting holistically, lacking the granularity to assess controllability over specific illumination attributes. This makes it difficult to diagnose model behavior or understand where control fails.\n\nTo address this, we propose LumosBench, a structured benchmark targeting six core lighting attributes from our annotation protocol: direction, source type, intensity, color temperature, temporal dynamics, and optical effects. Each of the 2,000 test prompts isolates a single attribute (e.g., front vs. back light), while keeping others fixed.\n\nWe use the vision-language model Qwen2.5-VL to assess whether the intended attribute is correctly expressed in the output. This disentangled evaluation allows precise measurement of lighting controllability, with scores reported per dimension and averaged for overall performance (**see the table below**).\n \nTable: Quantitative comparison of attribute-level controllability. **Bold** numbers indicate the best performance.\n\n| **Model** | #Params | Direction | Light Source Type | Intensity | Color Temperature | Temporal Dynamics | Optical Phenomena | Avg. Score |\n|-----------|---------|-----------|-------------------|-----------|-------------------|-------------------|--------------------|-------------|\n| **General Models** |||||||||\n| LTX-Video          | 1.9B  | 0.794 | 0.644 | 0.487 | 0.708 | 0.487 | 0.403 | 0.587 |\n| CogVideoX          | 5.6B  | 0.837 | 0.692 | 0.552 | 0.739 | 0.532 | 0.449 | 0.634 |\n| HunyuanVideo       | 13B   | 0.863 | 0.741 | 0.599 | 0.802 | 0.655 | 0.481 | 0.690 |\n| Wan2.1             | 1.3B  | 0.842 | 0.685 | 0.436 | 0.741 | 0.504 | 0.433 | 0.607 |\n| Wan2.1             | 14B   | 0.871 | 0.794 | 0.674 | **0.829** | **0.737** | 0.505 | 0.735 |\n| **Specialized Models** |||||||||\n| IC-Light Per-Frame | 0.9B  | 0.793 | 0.547 | 0.349 | 0.493 | 0.284 | 0.339 | 0.468 |\n| Light-A-Video + CogVideoX | 2.9B | 0.787 | 0.581 | 0.327 | 0.536 | 0.493 | 0.373 | 0.516 |\n| Light-A-Video + Wan2.1    | 2.2B | 0.801 | 0.603 | 0.361 | 0.582 | 0.557 | 0.412 | 0.553 |\n| **UniLumos w/o lumos captions**   | 1.3B | 0.868 | 0.774 | 0.529 | 0.798 | 0.543 | 0.457 | 0.662 |\n| **UniLumos**                      | 1.3B | **0.893** | **0.847** | **0.832** | 0.813 | 0.662 | **0.592** | **0.773** |\n\n> **Q7** Please expand on lines 217 - 222.\n\n**A7** To balance physical supervision and training efficiency, we adopt a selective optimization strategy inspired by path consistency scheduling [1]. In each training iteration, we divide the batch based on supervision type, following an 80/20 split to avoid prohibitive costs from full supervision while still maintaining effective learning signals. 20\\% of each batch is allocated to compute the path consistency loss L_{fast}, which involves three forward passes and one backward pass to enforce consistency across timesteps. The remaining 80\\% is used for the standard flow-matching loss L_0, with 50\\% of these samples further supervised using RGB-space geometry feedback via L_phy (i.e., depth and normal alignment). This probabilistic scheduling ensures high training throughput while allowing the model to benefit from multi-level supervision. To further enhance illumination diversity during training, we apply randomized lighting augmentations on the degraded subject V_deg, which introduces realistic lighting variability without the need for explicitly paired captures.\n\n[1] One step diffusion via shortcut models, arXiv:2410.12557."}]

Author rebuttal:

Meta-review: After a strong rebuttal, all reviewers leaned toward the positive side (with 4 borderline accepts). The AC agrees with the reviewers that video relighting is a relatively underexplored direction and finds the direction, ideas, and results promising.
The AC recommends accepting the paper, but asks the authors to carefully address the concerns raised by the reviewers in the camera-ready version, including releasing the LumosData, providing the preprocessing pipeline, giving clearer explanations of the evaluation, and including additional comparison results, among others.
Decision: Accept (poster)