Title: Maximally Useful and Minimally Redundant: The Key to Self Supervised Learning for Imbalanced Data
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords: Self Supervised Learning, Contrastive Learning, Imbalanced Data, Multi-view Learning

Abstract: The robustness of contrastive self-supervised learning (CSSL) for imbalanced datasets is largely unexplored. CSSL usually makes use of $\textit{multi-view}$ assumptions to learn discriminatory features via similar and dissimilar data samples. CSSL works well on balanced datasets, but does not generalize well for imbalanced datasets. In a very recent paper, as part of future work, Yann LeCun pointed out that the self-supervised multiview framework can be extended to cases involving $\textit{more than two views}$ (MTTV). Taking a cue from this insight we propose a theoretical justification based on the concept of $\textit{mutual information}$ to support the MTTV objective and apply it to the problem of dataset imbalance in self-supervised learning. The proposed method helps in extracting the representative characteristics of the tail classes by segregating between $\textit{intra}$ and $\textit{inter}$ discriminatory features. We introduce a loss function that helps to learn better representations by filtering out extreme features. Experimental evaluation on a variety of self-supervised frameworks (both contrastive and non-contrastive) also prove that the MTTV objective works well for imbalanced datasets. We achieve a new state-of-the-art accuracy in self-supervised imbalanced dataset classification (2% improvement in Cifar10-LT using Resnet-18, 5% improvement in Cifar100-LT using Resnet-18 and 3% improvement in Imagenet-LT (1k) using Resnet-50).

TL;DR: Self Supervised Learning for Imbalanced Dataset