Self-supervised object-centric representations learning of computer vision and natural language understanding models
Creators: name: Janghorbani, Sepehr
Publisher: name: No Publisher Supplied
Year: 2023

description: Despite their significant advancements, conventional supervised learning models canbe resource-intensive and expensive due to their reliance on large amounts of annotations. Self-supervised learning methods however emerge as a promising alternative, allowing models to learn from unlabeled data, typically by incorporating plausible inductive biases into the models themselves. One particularly effective inductive bias is the concept of “objects” as fundamental building blocks of the physical world, which helps explain the structure and semantics of observed data. This dissertation introduces a self-supervised object-centric learning paradigm and proposes frameworks based on this paradigm for applications in computer vision and natural language processing. The object-centric learning paradigm enables deep learning models to learn rich and meaningful representations without manual annotations. The dissertation introduces several models that utilize this self-supervised object-centric paradigm and apply it in the computer vision and natural language domain. We first discuss SCALOR (SCALable Object-oriented Representation), a self-supervised video understanding model capable of learning object-centric representations from videos. SCALOR can perform unsupervised object detection, object tracking, object segmentation, dynamic background separation, and future video prediction using a novel probabilistic generative model. SCALOR is also highly scalable and efficient, capable of handling crowded scenes with up to 100 moving objects, outperforming its counterparts by several orders of magnitude. The dissertation further presents experimental frameworks, including ablation studies and generalization studies, to evaluate SCALOR’s performance from various experimental aspects.Furthermore, the dissertation explores the application of object-centric modeling paradigms in natural language understanding. In this regard, a self-supervised object-centric natural language understanding model is proposed, which combines rule-based semantic parsing techniques with deep learning to learn a neuro-symbolic representation of natural language text. The model’s efficacy is demonstrated in the context of learning a neuro-symbolic object-centric representation of free-form language and is particularly applied to the domain of intelligent virtual assistant design. The proposed model understands the natural language description of the intended behavior of a virtual agent, infers an object-centric representation of this description and finally automatically generates the code needed for its functionality based on this representation. The proposed model can be directly used for streamlining and automating the design process of interactive intelligent virtual assistants, significantly reducing manual engineering efforts.In addition to introducing new vision and language models, the dissertation also explores the issue of evaluating model bias in such self-supervised vision-language models. In Chapter 6, we introduce a framework for bias evaluation of self-supervised vision language models. Our framework includes a new dataset that extends the current benchmarks to 14 new unexplored protected groups, new experimental frameworks and evaluation metrics and a novel bias mitigation algorithm. Our experiments demonstrate that several prominent self-supervised vision language models can exhibit meaningful bias towards certain groups, emphasizing the need to debias such models. As such, we also introduce a bias mitigation algorithm specifically designed for such large vision-language models which works as a post-processing stage to reduce bias in such models while preserving model accuracy.