Comprehensive Relighting: Generalizable and Consistent Monocular Human
  Relighting and Harmonization
Authors: name: Junying Wang, name: Jingyuan Liu, name: Xin Sun, name: Krishna Kumar Singh, name: Zhixin Shu, name: He Zhang, name: Jimei Yang, name: Nanxuan Zhao, name: Tuanfeng Y. Wang, name: Simon S. Chen, name: Ulrich Neumann, name: Jae Shin Yoon

This paper introduces Comprehensive Relighting, the first all-in-one approach
that can both control and harmonize the lighting from an image or video of
humans with arbitrary body parts from any scene. Building such a generalizable
model is extremely challenging due to the lack of dataset, restricting existing
image-based relighting models to a specific scenario (e.g., face or static
human). To address this challenge, we repurpose a pre-trained diffusion model
as a general image prior and jointly model the human relighting and background
harmonization in the coarse-to-fine framework. To further enhance the temporal
coherence of the relighting, we introduce an unsupervised temporal lighting
model that learns the lighting cycle consistency from many real-world videos
without any ground truth. In inference time, our temporal lighting module is
combined with the diffusion models through the spatio-temporal feature blending
algorithms without extra training; and we apply a new guided refinement as a
post-processing to preserve the high-frequency details from the input image. In
the experiments, Comprehensive Relighting shows a strong generalizability and
lighting temporal coherence, outperforming existing image-based human
relighting and harmonization methods.