Title: UniLumos: Fast and Unified Image and Video Relighting with Physics-Plausible Feedback
Venue: NeurIPS 2025
Accepted: yes
Keywords: video generation

illumination editing

video editing
Abstract: Relighting is a crucial task with both practical demand and artistic value, and recent diffusion models have shown strong potential by enabling rich and controllable lighting effects. However, as they are typically optimized in semantic latent space, where proximity does not guarantee physical correctness in visual space, they often produce unrealistic results—such as overexposed highlights, misaligned shadows, and incorrect occlusions. We address this with **UniLumos**, a unified relighting framework for both images and videos that brings RGB-space geometry feedback into a flow-matching backbone. By supervising the model with depth and normal maps extracted from its outputs, we explicitly align lighting effects with the scene structure, enhancing physical plausibility. Nevertheless, this feedback requires high-quality outputs for supervision in visual space, making standard multi-step denoising computationally expensive. To mitigate this, we employ path consistency learning, allowing supervision to remain effective even under few-step training regimes. To enable fine-grained relighting control and supervision, we design a structured six-dimensional annotation protocol capturing core illumination attributes.  Building upon this, we propose **LumosBench**, a disentangled attribute-level benchmark that evaluates lighting controllability via large vision-language models, enabling automatic and interpretable assessment of relighting precision across individual dimensions. Extensive experiments demonstrate that UniLumos achieves state-of-the-art relighting quality with significantly improved physical consistency, while delivering a 20x speedup for both image and video relighting. Code is available at https://github.com/alibaba-damo-academy/Lumos-Custom.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a method to change the lighting of a pre-specified foreground subject, within a still image or a video.  There are two steps:  generating a dataset of re-lit videos, which they call LumosData. They constructed that dataset based on Panda70M, which led them to 110K relit video pairs.  It's not clear (please clarify) whether that dataset will be made public--line 674, \"We do not release any data or model that has high risks for misuse\".    An image relighting dataset certainly could have a risk of misuse, so does that mean they don't release that dataset, or they don't feel it has a risk of misuse?\nThe second step: they run a latent diffusion process that has an innovation:  a \"physics-plausible\" feedback mechanism, to encourage physically realistic illumination.\n\nSTRENGTHS AND WEAKNESSES:\nstrength:  The results look nice, and the comparison of inference times versus related algorithms is impressive.\n\n\nweaknesses:  We see just a few examples, even including the appendix and supplemental materials.\nIt would be nice to see visually the effect of ablations regarding the claimed essential contribution of the paper, the physics-plausible feedback mechanism.  What do the re-rendered images look like when it is turned off?\n\nSome parts of the paper make it seem that the paper wasn't proofread:  eg, the two successive sentences in lines 55-59.\n\nis the physics-aware feedback module indeed aware of the physics?  Do you have justification for calling it physics aware?\nI feel like I'm missing various details from the paper. What is the precise form of the Feedback listed in Figure 3?\nWill you release the LumosData?  Will you release the code?           \n\nI have concerns about the reproducibility and clarity of the paper.\n\nPlease expand on lines 217 - 222.\n\nQUESTIONS:\nSee questions within Strengths and weaknesses\n\nLIMITATIONS:\nAuthors didn't address possible concerns about faking videos or placing someone in a place where they never were.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 2, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis submission develops a framework, from data collection to modelling, for unified image and video relighting. The authors collect a large video and image relighting dataset, LumosData, using an automated pipeline. Based on the data, the authors propose a text-conditioned video relighting model that works better than prior training free approaches or image relighting approaches.\n\nSTRENGTHS AND WEAKNESSES:\n- **Strengths**:\n\t- According to the results in Fig. 4 and Tab. 1, the proposed framework clearly improve over the previous training-frame approach (Light-A-Video) and trained image-relighting baseline (IC-Light). This shows a very promising signal along the presented technical direction.\n\t- The LumosData, if the author can release it, can be a valuable contribution to the community. My final decision will consider the releasing plan of this data.\n     - The path consistency objective and the normal / depth consistency objective, while neither is brand new, seem quite effective in improving the relighting results, according to Tab. 1 in the supplementary. \n\n- **Weakness**:\n\t\n   - First, I’d like to challenge the task itself of text-based relighting, i.e. relighting using text to describe the new lighting condition and generate an image / video for it. The biggest concern is that it is fairly challenging to describe a lighting condition. Imagine how hard it is to “reconstruct” an exact lighting condition just from the text description of it. Also imagine how many possible lighting conditions can fit into the description of “Front Light, Artificial Light, Moderate, Neutral, Static Light”. Practically, it’s not how the film industry or game industry describes lighting either, considering these might be the use cases of the proposed method. I hope the authors could justify this task in the rebuttal. This is important to justify the value of the proposed data and model because both are dependent on the assumption the lighting is described in text.\n   - While the quantitative results showed promising improvements, the qualitative results still reveal clear limitations. For instance, in case 2 and case 3 of the videos, for the parking garage scene, it is obvious that the specularity on the face caused by the lamps on top are not lit correctly because they don’t appear to move although the background video is clearly moving forward.\n   - The technical novelty is limited. The proposed framework is largely built on slightly modified existing methods of architecture. This should not be a big problem for acceptance if other concerns are addressed. However, this is the reason for higher acceptance ratings.\n   - While it is generally challenging to evaluate the effect of the LumosData, it can be interesting to understand it via fine-tuning IC-Light on the image only portion and see how it improves after using LumosData.\n\nQUESTIONS:\n- As mentioned in the weakness section, please justify the task of text-based relighting. \n- For Table 1 in the supplementary, what is the “UniLumos w/o Path Consistency” baseline? Is it w/o the consistency loss but using the same sampling iterations? Or is it w/o the loss but sampling more iterations? It looks better than w/ the consistency loss so it’s necessary to clarify what it is.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes an image and video relighting method. The method is built upon a video diffusion Wan 2.1 and extends it to handle lighting. It constructs a new dataset for training such a model. The method introduces a shortcut-based consistency loss that enables faster inference with variable step sizes. Additionally, it leverages  single-step diffusion models for depth and normal prediction to provide geometry-aware supervision before and after relighting. It achieves temporally consistent, physically plausible, and efficient relighting in videos.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. Quality: The relighting results are visually compelling, with strong temporal consistency across frames and realistic lighting effects. The proposed method demonstrates solid performance, particularly in portrait scenarios.\n2. Clarity: The technical exposition is generally sound, and the paper provides sufficient implementation details for reproducibility. The pipeline is well-structured and logically motivated.\n3. Significance: The paper tackles a relatively unexplored task of lighting control in video diffusion models. The direction, ideas, and results are promising and the problem is quite interesting.\n4. Originality: The paper uses single-step diffusion models for additional depth/normal supervision and a shortcut-based consistency loss for efficient inference. These elements contribute to faster and geometry-aware relighting.\n\nWeaknesses:\n1. Quality: One major concern with the paper is that it misses a lot of related work. Despite the paper’s focus on scene relighting, the method largely targets portrait relighting. It is better to cite and discuss other important methods. Below are major ones but not limited to:\n  - a. Total Relighting: Learning to Relight Portraits for Background Replacement, SIGGRAPH 21.\n  - b. Lumos: Learning to Relight Portrait Images via a Virtual Light Stage and Synthetic-to-Real Adaptation, SIGGRAPHa 22.\n  - c. Neural Video Portrait Relighting in Real-time via Consistency Modeling, ICCV 21.\n  - d. Real-time 3D-aware Portrait Video Relighting, CVPR 24.\n\nFurthermore, while the geometric supervision via normals and depth is helpful, it can compromise appearance fidelity in some cases. For instance, in Figure 4 (right), the appearance of the cup deviates noticeably from the input and other methods, indicating a drop in visual consistency.\n\n2. Clarity: The manuscript would benefit from thorough proofreading. The abstract contains grammatical issues (e.g., the phrase “…where controlling illumination under ensures physical consistency…” is unclear). Other minor issues include:\n  - a. Section title \"1. Introudction\" should be \"1. Introduction\"\n  - b. References [2] and [3] are duplicated\n3. Significance: The results are shown primarily on portrait videos with limited motion, which constrains the method’s applicability to more diverse or dynamic real-world video scenarios. \n4. Originality: The use of degraded data as input closely resembles prior work such as RelightVid. Similarly, treating single images as one-frame (T=1) videos is a straightforward extension within the video diffusion framework. The major contributions lie in path consistency loss for faster inference and using single-step pretrained diffusion models for extra supervision. As a result, the overall novelty of the approach is somewhat limited.\n\nQUESTIONS:\n1. In Fig.4 (left) and Fig.3 (Appendix), the input video clips appear to depict the same scene under different lighting conditions. However, there seems to be a slight difference in scale, particularly noticeable in the bottom-left corner where the three circles in Fig.4. Could the authors clarify if any cropping or resizing was applied, and whether this affects evaluation?\n2. The dataset includes optical attributes such as “Transmission (Glass)”, yet the paper does not show examples involving such materials. Given that the visual results primarily focus on portrait relighting, could the authors comment on the performance of the method when applied to scenes involving transparent or refractive materials?\n3. Have the authors evaluated how the method performs across a diverse range of subjects, including individuals with varying skin tones and non-human objects? This would help clarify the model’s generalization ability beyond the portrait-centric examples shown.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nUniLumos is a framework building on the Wan 2.1 video diffusion model to enhance the speed and physical plausibility of image and video relighting. The work introduces several techniques for accelerated inference to the task of video and image relighting. Specifically, a few-step flow-matching backbone is extended with path consistency learning. Additionally, monocular depth and normal data are incorporated into an additional loss to improve physical consistency by comparing the extracted depth and normals for the generation to the pseudo ground truth. For training a data generation pipeline, LumosData is introduced, which generates labeled data using a VLM and prior relighting work IC Light for annotation and augmentation. The evaluation shows state-of-the-art results at much lower inference times compared to similar methods.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- The method shows an impressive improvement on inference times.\n- Qualitative results look promising.\n- The data generation pipeline presents a helpful approach to semi-self-supervised training of relighting tasks by generating augmentations and annotations using existing (potentially weaker) models.\n\nWeaknesses:\n- By using existing generative models for data generation, there is a distribution gap between the training data and real-world videos. The model is only evaluated quantitatively using the data generated with this pipeline.\n- Using existing modules that have also been used for the given task. The normal and depth constraints have also been employed for supervision of e.g. optimizations of NeRFs (e.g. [1]) and 3D Gaussian Splatting (e.g. [2]). Path consistency is novel to the relighting task but is also an established idea by now.\n- Dynamic light does not seem to work well (based on the example clips in the supplements). This might be an artifact of the augmentation scheme (with a temporally inconsistent model). In case 2 of the supplementary material the guitar keeps some specular reflections through all illumination settings, which looks like an artifact.\n- Sometimes the paper has somewhat redundant text passages (e.g. at the beginning of the ablation study).\n\nOverall, the paper presents some nice engineering effort and promising results but offers limited insights into algorithmic advancements to the relighting task as it mostly (re-)combines existing ideas. The evaluation is lacking comparisons outside the training data distribution.\n\n1: Niemeyer et al. RegNeRF: Regularizing Neural Radiance Fields for View Synthesis from Sparse Inputs. CVPR 2022\n2: Turkulainen et al. DN-Splatter: Depth and Normal Priors for Gaussian Splatting and Meshing. WACV 2025\n\nQUESTIONS:\n- How does the model perform on real-world relighting data like e.g. the Navi or StanfordOrb dataset for still images and SDSD (dvlab) or LightAtlas (RelightVid) for video? Rendered, synthetic ground truth would also work for the evaluation here as long as it is path traced using a physical renderer.\n\n- Related works: Controllable diffusion models could also include the strand of works around ConceptSliders [1] which offers parametric control in the context of diffusion models. Why are the following relighting methods not included in the related works section and (potentially) evaluation? Intrinsic Image Diffusion [2], Neural Gaffer [3], DI-Light [4].\nHave they been evaluated and why has IC Light been chosen for augmentation as it might not be the best method for the task? Is the limited consistency of IC Light actually helping the robustness of the training? This could be analyzed in some additional ablation.\n\n- p.1, l.5: Why are overexposed highlights physically implausible? In tone-mapped images this is a natural occurrence that often makes it even look more realistic than compressed highlights.\n\n- Table 1 and Figure 4: What data has been used here? It would be helpful to add the evaluation data sources to the captions. I don't think \"relited\" exists as a word (Fig. 4 caption, for example). Probably, relit would be fitting here.\n\n\n1: Gandikota et al. Concept Sliders: LoRA Adaptors for Precise Control in Diffusion Models. ECCV 2024.\n2: Kocsis et al. Intrinsic image diffusion for indoor single-view material estimation. CVPR 2024.\n3: Haian et al. Neural Gaffer: Relighting Any Object via Diffusion, NeurIPS 2024\n4: Chong et al. Dilightnet: Fine-grained lighting control for diffusion-based image generation. SIGGRAPH 2024.\n\nLIMITATIONS:\nLimitations and broader impact are discussed in the appendix but it would be better to add a condensed version to the main paper.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-02"}