Learned representations of artistic style for image retrieval, description, and stylization
Creators: name: Ruta, Dan Sebastian
Publisher: name: University of Surrey
Year: 2023

description: This thesis establishes a comprehensive framework by bridging various areas of machine
description: learning research through a shared representation of artistic style.<br>
description: <br>
description: Our first contribution explores ways to learn a representation of visual artistic style, in
description: our ALADIN model. We experiment with various degrees of supervision, using our novel
description: BAM-FG dataset. We pursue a fine-grained representation to model a highly expressive
description: metric embedding space able to discriminate between small nuances in artistic style.
description: We demonstrate the strengths of weak supervision for this task, using the fine-grained
description: style groupings in BAM-FG.<br>
description: <br>
description: We pursue multiple downstream research directions with this embedding, such as
description: style-based image retrieval, where visual search can focus on purely the artistic style, for
description: the first time. We further extend our representation research into cross-modal learning,
description: bridging a connection between vision and language modalities in StyleBabel. By applying
description: our ALADIN representation, we can adapt machine learning techniques to perform
description: automatic tagging, natural language captioning, and tag-based visual search - again for
description: the first time purely in the artistic style domain.<br>
description: <br>
description: We also extend our research to generative applications, exploring ways of integrating
description: a shared style modality representation into the process. In our NeAT project, we
description: explore using ALADIN as a pre-processing step in curating the first ever large scale
description: high resolution and diverse Neural Style Transfer (NST) dataset. This was crucial in
description: extending NST to achieve state-of-the-art quality and generality. We also show how
description: ALADIN can be used as a conditioning factor in driving stylization in generative models,
description: by exploring the use of HyperNetworks in our HyperNST project. This novel approach
description: induces metric style control capabilities over existing StyleGAN models, enabling novel
description: ways of controlling stylization.<br>
description: <br>
description: Finally, we again demonstrate the merits of a style embedding for style conditioning in
description: diffusion-based generative models. We build DIFF-NST to leverage the Stable Diffusion
description: model to not only guide stylization using ALADIN, but to also achieve a wider gamut of
description: style changes. For the first time, we introduce a general method to achieve style-based
description: form deformation in NST, extending our field’s stylization capabilities to a broader set
description: of style factors, pushing past previous limitations in definitions of style in NST.<br>
description: <br>
description: Through our contributions in this thesis, we address the challenge of representation of
description: artistic style, unifying downstream tasks for style through shared representation. We
description: leverage this representation to extend state-of-the-art for multiple downstream tasks.
description: We document our findings, and propose future directions for the field.<br>