LumiSculpt: A Consistency Lighting Control Network for Video Generation

Authors: Yuxin Zhang, DanDan Zheng, Biao Gong, Jingdong Chen, Ming Yang, Weiming Dong, Changsheng Xu
Primary area: generative models
Keywords: Light Control, Video generation, Text-to-video generation, diffusion models

Abstract:
Lighting is essential for the naturalness of video generation, which significantly impacts the overall aesthetic quality of the generated video. However, due to the deep coupling between lighting and the temporal features of videos, it is challenging for modeling independent and coherent lighting attributes, resulting in a lack of approaches for controlling lighting in videos. Therefore, inspired by the established controllable T2I models, we propose LumiSculpt, achieving precise and consistency lighting control in video generation models for the first time. LumiSculpt equips the video generation with strong interactive capabilities, allowing for the input of custom lighting reference image sequences. Furthermore, the core learnable plug-and-play module of LumiSculpt enables us to achieve remarkable performance on controlling light intensity, position, trajectory in latent video diffusion models based on the advanced DiT backbone. Additionally, to effectively fine-tune LumiSculpt and address the issue of insufficient lighting data, we construct LumiHuman, a new lightweight and flexible dataset for portrait lighting of images and videos. Experiments demonstrate that LumiSculpt achieves precise and high-quality lighting control in video generation. The code, model, and dataset will be released to facilitate further research. Video results are shown in the supplementary material.

Decision: reject
Reviewer scores: 5, 6, 5, 3

Reviews and discussion:
<thread>
creation date:
1730340614617

thread_type:
official_review

<thread_object>
creator: reviewer_luen

<content>
summary:
This paper introduces LumiSculpt, a novel model that provides precise and consistent lighting control in text-to-video (T2V) generation. LumiSculpt. This allowing users to input custom lighting reference image sequences. The model includes a plug-and-play module that enables adjustable control over lighting intensity, position, and movement in latent video diffusion models, leveraging the advanced DiT (Denoising Transformer) backbone.

To train LumiSculpt effectively and address the lack of sufficient lighting data, the authors developed LumiHuman, a new, lightweight dataset specifically for portrait lighting in images and videos. Experiments demonstrate LumiSculpt’s ability to achieve precise, high-quality lighting control in video generation, marking a significant advancement in T2V lighting manipulation.

strengths:
I felt the biggest pros of this paper is that authors explore how to propsoe a  LumiHuman dataset that could benefits current research community. But authors didn't mention whether they will release the dataset.

weaknesses:
1. My point is that authors biggest contributions are the relighting dataset for video. However, the discussion on these dataset is so limited and there are a lot of details missing on the dataset. 
For example: 1.1 How diverse the MetaHuman dataset is since it is only contains 65 individuals.
1.2 how accurate your caption could describe the lighting since  lighting caption is a very unique task that current LLM model is not doing well. 
1.3 From the results, I didn't see any caption related to lighting. 


2. Since the model is trained on synthetic rendered images, the results are far from photo realism and most of the results from the teaser images are 'fake' portrait with unrealistic facial texture.  

3. It is not clear how authors control the lighting intensity. 


4. IC-light has much better photo-realistic results compared with your methods. And what's the advantage of authors method ? 


5.  It seems that authors only show white/black lighting but not color lighting which ICnet could do. 


6. Regarding model, I don't see any difference between yours and controlnet besides it is a video version.

questions:
See weakness.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730635018616

thread_type:
official_review

<thread_object>
creator: reviewer_wtpx

<content>
summary:
This paper proposes a control module for Text-2-Video models to enable lighting control. This is an extremely challenging task due to the lack of large-scale lighting datasets and the ambiguity between lighting and material properties. The key idea of the paper is to train a control network using a synthetic dataset, LumiHuman, consisting of 65 identities under various frontal illuminations defined over a uniform grid. The proposed dataset allows to combine the frames into various lighting trajectories. During training, a flat shading map is used as conditioning signal. The method shows impressive lighting control for static human portraits and is able to introduce cast shadows as well, which is really impressive.

strengths:
* The proposed LumiHuman dataset can be valuable to the community. It is a great idea to place point lights close to each other to make it possible to compose frames into videos.
* The results look great, especially the cast shadows. 
* The paper can provide interesting insights about the internal lighting prior of recent diffusion models.

weaknesses:
I believe that the paper has multiple crucial flaws, which would require a major revision:
* **W1 -** The proposed dataset, which is one of the main contributions, seems to be a bit limited. The synthetic renderings could follow the usual light stage setup with full coverage, not just frontal lighting. Furthermore, it is not clear whether 65 identities can provide enough diversity. I believe that the main advantage of using a synthetic dataset is that it can be scaled. To verify this benefit, it would also be crucial to show that available real-world light-stage datasets cannot provide enough supervision to achieve such quality for lighting control. 
* **W2 -** The novelty and effectiveness of the proposed pipeline, which is the second contribution, is not clear. It would be important to highlight the key difference to ControlNet. Now, it seems that the key difference is the dual-branch predictions, although the effectiveness of this idea is questionable based on the ablation. Furthermore, the proposed disentanglement loss is not well-motivated. The key assumption is that the latents reflect the appearance. However, the latents contain geometric, material, and also lighting features, thus not being disentangled. 
* **W3 -** It would be great to show the diversity of the generated samples - more samples with the same conditioning.
* **W4 -** Additional baseline comparisons would be important. Although the method uses the T2V models for light editing, the resulting videos are static, making it fair to compare against T2I models. Such comparisons could also give interesting insights about the lighting priors of T2I and T2V models. 
* **W5 -** The key contribution is not clear. Based on the title and abstract it is LumiSculpt, based on the intro (L.087 - Additionally...) it is the dataset LumiHuman. 

Additional weaknesses
* **W6 -** Recent T2I lighting control methods, such as [LightIt](https://peter-kocsis.github.io/LightIt/) could be discussed. 
* **W7 -** The notation on the pipeline figure is not clear. I would recommend naming the specific parameters instead of using "?". However, this might be due to some LaTex errors. 
* **W8 -** Typo in L.204: "using a unreal engine" -> "using Unreal Engine [XYZ]"
* **W9 -** Typo in L.290: "an naive" -> "a naive"
* **W10 -** The formulation could be more precise. E.g., the lighting representation seems to be a flat shading map, but it is not clear. 
* **W11 -** It might be better to narrow the title, reflecting that the domain is human portraits. 
* **W12 -** The lighting is not entirely consistent with the results, e.g., in the first row of Fig. 1. the lighting seems to start from the bottom and move to the top right instead of moving only upwards as depicted in the trajectory.

questions:
* **Q1 -** What is the reason that the generated samples have a very similar geometry and appearance as IC Light, but highly different to Open-Sora, although the proposed method uses Open-Sora. 
* **Q2 -** Could you please give a bit more details, how exactly are the augmented captions used? If I understand it correctly, the goal with those is to give additional noise to the model to avoid overfitting. 
* **Q3 -** The results look oversaturated, what can be the reason for that?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730654277830

thread_type:
official_review

<thread_object>
creator: reviewer_my7d

<content>
summary:
The paper propose a approach to precise lighting control in text-to-video (T2V) generation. The authors design a model named LumiSculpt, which introduces plug-and-play lighting control to enhance video generation with adjustable lighting intensity, direction, and trajectory, all controlled through textual inputs. LumiSculpt uses a dual-branch framework with a frozen branch to ensure diversity in appearances while maintaining lighting control.  Besides, the authors introduce a specialized dataset, LumiHuman, containing diverse portrait lighting data generated via Unreal Engine, which supports various lighting trajectories.  The model’s performance is evaluated against state-of-the-art methods, Open-Sora and IC-Light, demonstrating its superiority in lighting accuracy, inter-frame consistency, and semantic alignment with text descriptions.

strengths:
+ The model design is novel. The introduction of an additional lighting embedding in the diffusion process is reasonable and easy for model to control the lighting. The loss design is cool. It is based on the concept of style transfer but in the latent space. This loss controls the lighting intensity and direction while preserving the appearance.
+ This paper prepares a dataset to enhance lighting control in video generation. Although the sample size is limited, the pipeline design can provide valuable insights for future work.

weaknesses:
- This algorithm seems more suitable for image generation, as I did not observe any specific design tailored for video tasks. Video generation is merely an extension of the algorithm’s application.
- In the comparisons, the authors use images generated by the network as the foreground. Does this imply that, limited by the synthetic data used during training, the algorithm may not generalize well to real-world scenes? I also noticed unnatural foreground (human) generation results in the video demo.

questions:
In addition to the weaknesses mentioned, I have the following concerns:
1. Can this dataset be open-sourced to ensure reproducibility for future work?
2. I find the caption augmentation section somewhat unclear. Is it simply replacing captions, or does it involve corresponding changes in the image background as well?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730696704460

thread_type:
official_review

<thread_object>
creator: reviewer_rlxs

<content>
summary:
This paper proposes LumiSculpt to deal with the lighting control for video generation. Controlling lighting is a fundamental need for image and video synthesis tasks. This work can control the light intensity, position, and trajectory. Besides the proposed model, this work also constructs a dataset composed of portraits with different lighting conditions.

strengths:
1. This work achieves precise control of the lighting of the portrait in the video generation tasks. 
2. The plug-and-play design of the module makes this work flexible to be combined with other foundation models. 
3. The LumiHuman will benefit the community if it can be made public available.

weaknesses:
1. From what I understand, the LumiHuman is synthetic, which may limit the model's performance in real-world cases. I wonder if there can be a thorough evaluation of real-world cases. Meanwhile, there are only 65 individuals in the dataset, which may limit the model to generalize to new portraits. 
2. The generated videos are not informative enough. The motion dynamics are not enough. I wonder if there are results where the portrait and background can move more vividly.

questions:
More cases on real-world cases.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732431057342

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
We would like to thank all the reviewers for their careful reading and insightful comments.

comment:
We would like to thank all the reviewers for their careful reading and insightful comments on the manuscript. We sincerely appreciate reviewers for acknowledging:

- The proposed dataset will benefit the community(r1,r2,r3,r4).

- Our work achieves precise lighting control(r1,r3).

- The proposed module and loss design are novel and flexible(r1,r2).

We have re-uploaded our supplementary materials, which include the complete **response letter** (at .zip/LumiSculpt_response_letter.pdf) along with the relevant figures and tables. The response letter is also contained in the main paper, after page 12. Below, we have addressed each question in detail and hope to clarify any concerns.
We sincerely invite the reviewers to refer to these materials for a better reading experience. We hope that our response satisfactorily addresses your concerns.
 
We have addressed the common comments:

---

**[Common Concern 1] Statistics and Diversity of LumiHuman Dataset**

We introduce the LumiHuman dataset, a continuous lighting video dataset comprising over **195K** different videos (*i.e.*, **2.3 million** images). The resolution of each video is 1024×1024.
LumiHuman includes 65 diverse human subjects, 30K lighting positions, and over 3K lighting trajectories for each people.

LumiHuman of 65 human identities is sufficient for training LumiSculpt, which is supported by extensive qualitative and quantitative experiments. The scalability and diversity of synthetic data lies in the ability to construct diverse light trajectories, leveraging varied lighting data to facilitate the model's learning of illumination.

As shown in the table below, compared to
other lighting datasets Openillumination[1] and Deep Portrait Relighting (DPR) dataset[2] (generated from face image dataset Celeb-A[3]), LumiHuman outperforms in
light positions, light movements and number of images.

| Dataset             | Synthesis | Light Positions | Light Movement | Number of Images | Subject        | Resolutions      |
|---------------------|-----------|----------------|---------------|-----------------|---------------|----------------|
| DPR                | 2D        | 7             | None          | 138K            | -             | 1024×1024      |
| Openillumination   | Light Stage | 142         | None          | 108K            | 64 objects     | 3000×4096      |
| LumiHuman           | 3D        | **35,937**     | **>3K**       | **2.3M**        | **65 individuals** | 1024×1024 |

[1] OpenIllumination: A Multi-Illumination Dataset for Inverse Rendering Evaluation on Real Objects. NeurIPS 2024

[2] Deep Single Portrait Image Relighting. ICCV 2019

[3] Deep Learning Face Attributes in the Wild. ICCV 2015
</content>
</thread_object>

</thread>
<thread>
creation date:
1732431168141

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
[Common Concern 2]

comment:
**[Common Concern 2] Similar to ControlNet**

LumiSculpt is distinct from ControlNet in terms of its task, motivation, module design, training objective, training data, backbone, and generated results. A detailed explanation for each point is provided below:

- **Task:** LumiSculpt is a specialized lighting control method designed for DiT based T2V models.
  ControlNet is a control method that focuses on image geometry (pose, depth map, canny, etc.) for U-Net based T2I models.
- **Motivation:** 
  LumiSculpt's motivation focuses on elements in videos that affect realism and aesthetics, i.e., lighting, and proposes a method to achieve coherent video generation with controllable lighting. 
  ControlNet's motivation stems from the randomness in T2I diffusion models, hence it introduces a method for generating images with controllable geometry.
- **Module Design:** 
  LumiSculpt employs self-attention mechanisms as the lighting encoder and uses linear layers and latent weighting as condition injection mechanisms. ControlNet uses the U-Net Encoder to extract features and injects conditions by adding latents. 
  These atomic components are commonly used and necessary for feature extraction and condition injection, which are not limited to a specific method.
- **Training Objective:** 
  LumiSculpt tackles the core challenge of the entanglement of lighting and appearance. LumiSculpt additionally employs a dual-branch structure and introduces an appearance-lighting disentanglement loss. 
  ControlNet is trained with the diffusion noise prediction loss.
- **Training Data:** 
  LumiSculpt utilizes video data with coherent inter-frame lighting changes, whereas ControlNet is based on independent images.
- **Backbone:** 
  LumiSculpt is build upon DiT-based Open-Sora-Plan, and ControlNet is designed for U-Net structured Stable Diffusion.
- **Generated Results:** 
  LumiSculpt generates coherent videos while ControlNet generates images.

We implement ControlNet to video lighting control by training with paired frames in LumiHuman and generating image sequence as video. The comparison results are shown in the table and in the response letter.

| Method      | Consistency (CLIP↑) | Lighting Accuracy (LPIPS↓) | Lighting Accuracy (Direction↓) | Lighting Accuracy (Brightness↑) | Quality (CLIP↑) |
|-------------|--------------------|--------------------------|-----------------------------|------------------------------|-----------------|
| Open-Sora   | 0.9845             | 1.3503                  | 0.4542                      | 0.8229                        | 0.3182          |
| IC-Light    | 0.9703             | 2.5329                  | 0.5264                      | 0.8632                        | 0.3145          |
| ControlNet  | 0.8081             | 5.9324                  | 0.5500                      | 0.8032                        | 0.3440          |
| **Ours**    | **0.9951**          | **1.1312**               | **0.3500**                   | **0.8779**                    | **0.3597**      |

The best results are marked as **bold**.
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper receives mostly negative scores from reviewers. While reviewer appreciate the simple yet effective design, they also found the result quality limited, the extension from images to videos unclear, and questioned the generalizability to real-world cases. The authors are encouraged to address these comments in the revised version.