Visual learning in-the-wild with limited supervision: the role of disentangled factors, auxiliary domains, and active supervision
Creators: name: Wang, Yuting
Publisher: name: No Publisher Supplied
Year: 2023

description: Despite impressive advancements over the last decade, visual recognition in-the-wild remains a fundamental challenge, due to the difficulty of collecting a large amount of fully-annotated training examples. State-of-the-art deep learning-based visual recognition systems heavily rely on strong forms of human supervision, and tend to perform poorly if they lack sufficient supervision. By contrast, humans can remarkably recognize visual concepts, even with limited supervision or from very few examples. This dissertation aims to endow visual recognition systems with the ability to reliably learn from data with minimal human supervision. Our key insight is that the visual world is well-structured and there exist rich sources of intrinsic and extrinsic knowledge, ranging from disentangled factors of variability in data, to relevant signals from auxiliary domains, and to sparse yet highly-informative examples. Such structures and regularities enable the systems to learn with different levels of supervision, such as encoding expressive feature representations in a self-supervised or unsupervised manner, performing weakly-supervised object detection, and further improving the recognition performance by jointly leveraging active learning and semi-supervised learning. Doing so leads to a substantially improved trade-off between model performance and required human labor, not only on general-purpose benchmarks like PASCAL VOC and MS-COCO but also on demanding application-driven datasets such as RealPizza10. Our exploration starts with unsupervised learning from visual data, such as images or videos, which are composed of various complex factors including texture, geometry, and lighting. Identifying these factors and understanding their interactions are critical for not only gaining insights into the sources of data variability but also creating effective representations that can be applied to various visual recognition tasks. To achieve this, we introduce a variational auto-encoder (VAE)-based model that can identify disentangled factors of variability in the data. We extend VAE to a hierarchical Bayesian model by incorporating hyper-priors on the variances of Gaussian latent priors. Our algorithm learns a latent space of data that simultaneously discovers disentangled factors of variability and distinguishes between relevant and irrelevant factors. However, purely unsupervised learning approaches struggle to identify task-aware representations for downstream visual recognition problems. In practice, some forms of weak supervision can be relatively easily or sometimes even automatically obtained, such as hashtags associated with images uploaded to the Internet. Motivated by this, we exploit weakly-supervised learning to optimize task-specific representations with minimal human effort. On the other hand, for structural tasks like object detection and segmentation, the absence of precise localization information is another significant obstacle to achieving the performance on par with fully-supervised models. To tackle this issue, we introduce a fully-labeled auxiliary domain to supplement the target domain with weak annotations. Specifically, for object detection, we propose a simple yet effective object proposal generation strategy that employs ''non-photographic'' synthetic images with accurate object localization labels to improve object localization in real-world data. We now treat weakly-supervised object detection as a domain adaptation problem. By transferring the learning of a fully-supervised object detector from the synthetic domain to the real-world domain and individually adapting foreground and background in a decoupled manner, we progressively enhance detection performance. Our approach successfully identifies most objects and produces more precise bounding boxes. When constructing a satisfactory auxiliary domain becomes difficult, or the domain gap between the auxiliary and target domains cannot be fully reduced via domain transfer techniques, a more effective strategy is to directly obtain a small set of fully-labeled data on the target domain. Correspondingly, we move on to learning from large weakly-labeled target data and small fully-labeled target data. In the context of object detection, we present a novel framework that combines active learning with weakly- and semi-supervised learning paradigms. To minimize human effort, we introduce a new auxiliary image generation technique that uses small labeled sets and a large weakly-labeled set of images to initiate the learning process. Additionally, we propose a new active learning acquisition function that effectively selects the most informative images to annotate by leveraging model disagreement and image uncertainty. To further reduce human labor, we integrate a new labeling strategy with our object detection architecture that refines the detection results generated by our framework, instead of directly annotating the original images.