Generative Models for Computational Relighting
Creators: name: Kubiak, Nikolina
Publisher: name: University of Surrey
Year: 2025
Subjects: Relighting
Shadow removal
Shadow detection
Generative models
GANs
Diffusion models

description: Relighting is a long-standing computer vision problem. The task has applications ranging
description: from simple image beautification to domain adaptation, diversification of autonomous
description: vehicle training data, or enhancements in the TV and film production world. In this
description: thesis, we focus on the final area. Motivated by broadcast applications, we explore the
description: idea of capturing visual data under sub-optimal lighting conditions and then adjusting
description: it automatically in post-production. Doing so could meaningfully simplify the process
description: of shooting footage, allow for cheaper coverage of smaller or more resource-constrained
description: events and, thus, wider broadcasting. Driven by the recent advances in deep learning
description: and computer vision, we decide to tackle this problem using powerful generative models.
description: Capturing high quality training data at scale and with meaningful complexity is a challenge.
description: Consequently, we strive to create a system with relaxed supervision requirements.
description: With this in mind, we propose a self-supervised and domain-independent relighting
description: model. Instead of relying on ground truth, our GAN-based solution exploits the rich
description: information contained in the input data and learns desired illumination style from a
description: collection of unsorted examples, not a directly aligned reference. This flexible approach
description: allows us to use a loose definition of the lighting style and, potentially, to adapt to styles
description: existing in already captured materials, even those coming from other environments.
description: Our preliminary solution performs accurate colour and brightness adjustments yet
description: exhibits subpar shadow manipulation performance. Therefore, we decide to address
description: this aspect more explicitly and in our next two technical chapters we investigate the
description: problems of shadow removal and detection. To perform de-shadowing, we adjust our
description: self-supervised system to the new task and modify its losses to account for inconsistencies
description: existing in the benchmark datasets. The resulting system is capable of removing the
description: shadowed areas without significant boundary residue, providing superior visual results.
description: We explore the idea of shadows further in the next technical chapter and consider
description: the common misclassification of dark areas which are often confused for shadows. To
description: alleviate this, we create 2 datasets featuring diverse objects and backgrounds as well as
description: cast and self-cast shadows. We then use this data to create a 3D-aided shadow-caster
description: verification system, identifying sources of real shadows and discouraging the detection
description: of ‘fake’ shadows, i.e. dark or patterned image regions.
description: In our final technical chapter, we take a step back from the fragmented approach and
description: instead consider the most recent foundation models and the wealth of information and
description: world understanding contained within them. We pair the successful design choices from
description: the earlier stages of the PhD with lighting-conditioned diffusion models, and explore
description: their applicability and adaptability to the task of relighting.