Towards Practicality of Sketch-Based Visual Understanding
Creators: name: BHUNIA, AYAN KUMAR
Publisher: name: University of Surrey
Year: 2022

description: Sketches have been used to conceptualise and depict visual objects from
description: pre-historic times. Sketch research has flourished in the past decade, particularly with the proliferation of touchscreen devices. Much of the utilisation
description: of sketch has been anchored around the fact that it can be used to delineate
description: visual concepts universally irrespective of age, race, language, or demography. The fine-grained interactive nature of sketches facilitates the application of sketches to various visual understanding tasks, like image retrieval,
description: image-generation or editing, segmentation, 3D-shape modelling etc. However, sketches are highly abstract and subjective based on the perception of
description: individuals. Although most agree that sketches provide fine-grained control
description: to the user to depict a visual object, many consider sketching a tedious process due to their limited sketching skills compared to other query/support
description: modalities like text/tags. Furthermore, collecting fine-grained sketch-photo
description: association is a significant bottleneck to commercialising sketch applications.
description: Therefore, this thesis aims to progress sketch-based visual understanding towards more practicality.<br>
description: <br>
description: Being able to model fine-grained details of a visual concept easily, sketch
description: understandably holds immense potential as a medium of query – even better than texts that are at times insufficient to pin down fine-grained visual
description: details. Out of all sketch-related applications therefore, fine-grained sketchbased image retrieval (FG-SBIR) has received the most attention, due to
description: its significant commercial potential in the retail industry. FG-SBIR aims
description: at retrieving a particular photo instance given a user’s query sketch, out
description: of a gallery of photos having a particular category. Given the dominant
description: prevalence of touchscreen devices now, the world is already primed for using sketch as a practical query modality for fine-grained retrieval. However,
description: talking of applicability at an industrial scale, practically using sketch as a
description: query-medium for retrieval is yet to gain traction due to a few significant
description: barriers. Breaking these barriers, this thesis addresses the practicality of
description: FG-SBIR via two themes putting forth five major contributions. The first
description: theme comprises three contributions which focus particularly on the practical deployment of FG-SBIR which is one of the major forefront of sketch
description: research. The second theme caters to the widespread applicability of sketches
description: for real-world applications, consisting of two more contributions.<br>
description: <br>
description: In the first theme, our first chapter starts by figuring out that the
description: widespread applicability of FG-SBIR is hindered as drawing a sketch takes
description: time, and most people struggle to draw a complete and faithful sketch. We
description: thus reformulate the conventional FG-SBIR framework to tackle these challenges, with the ultimate goal of retrieving the target photo with the least
description: number of strokes possible. We further propose an on-the-fly design that
description: starts retrieving as soon as the user starts drawing. Accordingly, we devise a
description: reinforcement learning-based cross-modal retrieval framework that optimises
description: the ground-truth photo’s rank over a complete sketch drawing episode.<br>
description: <br>
description: The second chapter discovers that the lack of sketch-photo pairs largely
description: bottlenecks FG-SBIR performance. We therefore introduce a novel semisupervised framework for instance-level cross-modal retrieval that can leverage large-scale unlabelled photos to account for data scarcity. The core of
description: our semi-supervision design is a sequential photo-to-sketch generation model
description: that aims to generate paired sketches for unlabelled photos. We further introduce a discriminator guided mechanism to guide against unfaithful generation, together with a distillation loss-based regulariser to provide tolerance
description: against noisy training samples.<br>
description: <br>
description: Thirdly, we notice that the fear-to-sketch problem (i.e., “I can’t sketch”)
description: has proven to be fatal for the widespread adoption of fine-grained SBIR. A
description: pilot study revealed that the secret lies in the existence of noisy strokes, but
description: not so much of the “I can’t sketch”. We thus design a stroke subset selector
description: that detects noisy strokes, leaving only those that positively contribute to a
description: successful retrieval. Our Reinforcement Learning based formulation quantifies the importance of each stroke present in a given subset, based on the
description: extent to which that stroke contributes to retrieval.<br>
description: <br>
description: Moving on to our second theme, in the fourth chapter we focus on learning powerful representations via self-supervised learning from unlabelled
description: data, thus increasing the scope of sketches. Towards this, we advocate for
description: the dual-modality nature of sketches – rasterized images and vector coordinate sequences, which is pivotal in designing a self-supervised pre-text task
description: for our goal. We address this dual representation by proposing two novel
description: cross-modal translation pre-text tasks for self-supervised feature learning:
description: Vectorization and Rasterization. Vectorization learns to map image space
description: to vector coordinates, and rasterization the opposite. We show that our
description: learned encoder modules benefit both raster-based and vector-based downstream tasks to analysing hand-drawn data.
description: The final chapter further contributes to the second theme by opening
description: up a new avenue of sketch research with a novel sketch-based visual understanding task. Here we dictate the potential of sketch as a support modality
description: for few-shot class incremental learning (FSCIL) – a setup that explicitly
description: highlights the relevance of sketch as an information-modality. Pushing FSCIL further, we address two key questions that bottleneck its ubiquitous
description: application (i) can the model learn from diverse modalities other than just
description: photo (as humans do), and (ii) what if photos are not readily accessible (due
description: to ethical and privacy constraints). The product is a “Doodle It Yourself”
description: (DIY) FSCIL framework where the users can freely sketch a few examples
description: of a novel class for the model to learn recognising photos of that class.<br>