SpotLight: Shadow-Guided Object Relighting via Diffusion
Authors: name: Frédéric Fortier-Chouinard; name: Zitian Zhang; name: Louis-Etienne Messier; name: Mathieu Garon; name: Anand Bhattad; name: Jean-François Lalonde
Published: 2024-11-27 16:06:08

Abstract: Recent work has shown that diffusion models can be used as powerful neural
rendering engines that can be leveraged for inserting virtual objects into
images. Unlike typical physics-based renderers, however, neural rendering
engines are limited by the lack of manual control over the lighting setup,
which is often essential for improving or personalizing the desired image
outcome. In this paper, we show that precise lighting control can be achieved
for object relighting simply by specifying the desired shadows of the object.
Rather surprisingly, we show that injecting only the shadow of the object into
a pre-trained diffusion-based neural renderer enables it to accurately shade
the object according to the desired light position, while properly harmonizing
the object (and its shadow) within the target background image. Our method,
SpotLight, leverages existing neural rendering approaches and achieves
controllable relighting results with no additional training. Specifically, we
demonstrate its use with two neural renderers from the recent literature. We
show that SpotLight achieves superior object compositing results, both
quantitatively and perceptually, as confirmed by a user study, outperforming
existing diffusion-based models specifically designed for relighting.