SynthLight: Portrait Relighting with Diffusion Model by Learning to
  Re-render Synthetic Faces
Authors: name: Sumit Chaturvedi, name: Mengwei Ren, name: Yannick Hold-Geoffroy, name: Jingyuan Liu, name: Julie Dorsey, name: Zhixin Shu

We introduce SynthLight, a diffusion model for portrait relighting. Our
approach frames image relighting as a re-rendering problem, where pixels are
transformed in response to changes in environmental lighting conditions. Using
a physically-based rendering engine, we synthesize a dataset to simulate this
lighting-conditioned transformation with 3D head assets under varying lighting.
We propose two training and inference strategies to bridge the gap between the
synthetic and real image domains: (1) multi-task training that takes advantage
of real human portraits without lighting labels; (2) an inference time
diffusion sampling procedure based on classifier-free guidance that leverages
the input portrait to better preserve details. Our method generalizes to
diverse real photographs and produces realistic illumination effects, including
specular highlights and cast shadows, while preserving the subject's identity.
Our quantitative experiments on Light Stage data demonstrate results comparable
to state-of-the-art relighting methods. Our qualitative results on in-the-wild
images showcase rich and unprecedented illumination effects. Project Page:
https://vrroom.github.io/synthlight/