Exploring Balanced Feature Spaces for Representation Learning

Existing self-supervised learning ( SSL ) methods are mostly applied for training representation models from artificially balanced datasets ( e.g . ImageNet ) . It is unclear how well they will perform in the practical scenarios where datasets are often imbalanced w.r.t . the classes . Motivated by this question , we conduct a series of studies on the performance of self-supervised contrastive learning and supervised learning methods over multiple datasets where training instance distributions vary from a balanced one to a long-tailed one . Our findings are quite intriguing . Different from supervised methods with large performance drop , the self-supervised contrastive learning methods perform stably well even when the datasets are heavily imbalanced . This motivates us to explore the balanced feature spaces learned by contrastive learning , where the feature representations present similar linear separability w.r.t . all the classes . Our further experiments reveal that a representation model generating a balanced feature space can generalize better than that yielding an imbalanced one across multiple settings . Inspired by these insights , we develop a novel representation learning method , called k-positive contrastive learning . It effectively combines strengths of the supervised method and the contrastive learning method to learn representations that are both discriminative and balanced . Extensive experiments demonstrate its superiority on multiple recognition tasks , including both long-tailed ones and normal balanced ones . Code is available at https : //github.com/bingykang/BalFeat . 1 INTRODUCTION . Self-supervised learning ( SSL ) has been popularly explored as it can learn data representations without requiring manual annotations and offer attractive potential of leveraging the vast amount of unlabeled data in the wild to obtain strong representation models ( Gidaris et al. , 2018 ; Noroozi & Favaro , 2016 ; He et al. , 2020 ; Chen et al. , 2020a ; Wu et al. , 2018 ) . For instance , some recent SSL methods ( Hénaff et al. , 2019 ; Oord et al. , 2018 ; Hjelm et al. , 2018 ; He et al. , 2020 ) use the unsupervised contrastive loss ( Hadsell et al. , 2006 ) to train the representation models by maximizing the instance discriminativeness , which are shown to generalize well across various downstream tasks , and even surpass the supervised learning counterparts in some cases ( He et al. , 2020 ; Chen et al. , 2020a ) . Despite the great success , existing SSL methods focus on learning data representations from the artificially balanced datasets ( e.g . ImageNet ( Deng et al. , 2009 ) ) where all the classes have similar numbers of training instances . However in reality , since the classes in natural images follow the Zipfian distribution , the datasets are usually imbalanced and show a long-tailed distribution ( Zipf , 1999 ; Spain & Perona , 2007 ) , i.e. , some classes involving significantly fewer training instances than others . Such imbalanced datasets are very challenging for supervised learning methods to model , leading to noticeable performance drop ( Wang et al. , 2017 ; Mahajan et al. , 2018 ; Zhong et al. , 2019 ) . Thus several interesting questions arise : How well will SSL methods perform on imbalanced datasets ? Will the quality of their learned representations deteriorate as the supervised learning methods ? Or can they perform stably well ? Answering these questions is important for understanding the behavior of SSL in practice . But these questions remain open as no research investigations have been conducted along this direction so far . discriminative feature space . The shadow area ( ) indicates the decision boundary of each class . Our work is motivated by the above questions to study the properties of data representations learned with supervised/self-supervised methods in a practical scenario . We start with two representative losses used by these methods , i.e. , the supervised cross-entropy and the unsupervised contrastive losses ( Hadsell et al. , 2006 ; Oord et al. , 2018 ) , and investigate the classification performance of their trained representation models from multiple training datasets where the instance distribution gradually varies from a balanced one to a long-tailed one . We surprisingly observe that , different from the ones learned from supervised cross-entropy loss where performance drops quickly , the representation models learned from the unsupervised contrastive loss perform stably well , no matter how much the training instance distribution is skewed to be imbalanced . Such a stark difference between the two representation learning methods drives us to explore why SSL performs so stably . We find that using the contrastive loss can obtain representation models generating a balanced feature space that has similar separability ( and classification performance ) for all the classes , as illustrated in Figure 1 . Such a balanced property of the feature spaces from SSL is intriguing and provides a new perspective to understand the behavior of SSL methods . We dig deeper into its benefits via a systematic study . In particular , since a pre-trained representation model is often used as initialization for downstream tasks ( He et al. , 2020 ; Newell & Deng , 2020 ; Hénaff et al. , 2019 ) , we evaluate and compare the generalization ability of the models that produce feature spaces of different balanced levels ( or ‘ balancedness ’ ) . We find that a more balanced model tends to generalize better across a variety of settings , including the out-of-distribution recognition as well as the cross-domain and cross-task applications . These studies imply that feature space balancedness is an important but often neglected factor for learning high-quality representations . Inspired by the above insights , we propose a new representation learning method , the k-positive constrastive learning , which inherits the strength of constrastive learning in learning balanced feature spaces and meanwhile improves the feature spaces ’ discriminative capability . Specifically , different from the contrastive learning methods lacking semantic discriminativeness , the proposed k-positive constrastive method leverages the available instance semantic labels by taking k instances of the same label with the anchor instance to embed semantics into the contrastive loss . As such , it can learn representations with desirable balancedness and discriminativeness ( Figure 1 ) . Extensive experiments and analyses clearly demonstrate its superiority over the supervised learning and latest contrastive learning methods ( He et al. , 2020 ) for various recognition tasks , including visual recognition in both long-tailed setting ( e.g. , ImageNet-LT , iNaturalist ) and balanced setting . This work makes the following important observations and contributions . ( 1 ) We present the first systematic studies on the performance of self-supervised contrastive learning on imbalanced datasets which are helpful to understanding the merits and limitations of SSL in practice . ( 2 ) Our studies reveal an intriguing property of the model trained by contrastive learning—the model can robustly learn balanced feature spaces—that has never been discussed before . ( 3 ) Our empirical analysis demonstrates that learning balanced feature spaces benefits the generalization of representation models and offer a new perspective for understanding deep model generalizability . ( 4 ) We develop a new method to explicitly pursue balanced feature spaces for representation learning and it outperforms the popular cross-entropy and contrastive losses based methods . We believe our findings and the novel k-positive contrastive method are inspiring for future research on representation learning . 2 RELATED WORKS . Self-supervised learning is a form of unsupervised learning . Recently there has been a surge of self-supervised data representation learning methods developed to alleviate the demand for manual annotations by mining free supervision information through specifically designed loss functions and pretext tasks . The contrastive loss measures the similarities of sample pairs in a feature space and is at the core of several recent SSL methods ( Chen et al. , 2020a ; b ; He et al. , 2020 ; Chen et al. , 2020c ) . Adversarial losses that measure the distribution difference are also exploited for self-supervised representation learning ( Donahue et al. , 2016 ; Doersch & Zisserman , 2017 ) . A wide range of pretext tasks have been developed including image inpainting ( Jenni & Favaro , 2018 ; Pathak et al. , 2016 ) , image colorization ( Larsson et al. , 2016 ; 2017 ) , context prediction ( Doersch et al. , 2015 ) , jigsaw puzzles ( Carlucci et al. , 2019 ; Noroozi & Favaro , 2016 ; Wei et al. , 2019 ) , rotation prediction ( Gidaris et al. , 2018 ) . Though very successful , the behavior of SSL largely remains a mystery . Recently Wang & Isola ( 2020 ) analyze contrastive learning from the perspective of uniformity and alignment of learned representations . However investigations on the behavior of contrastive learning on imbalanced datasets are still absent . We present the first study on this problem and our investigation methodology is also applicable to other SSL methods . In practice , the visual data usually follow a long-tailed distribution ( Zipf , 1999 ; Spain & Perona , 2007 ) , challenging supervised learning methods . Due to the imbalance in the number of training instances for different classes , conventional methods tend to perform much more poorly on instancerare classes than on instance-rich ones . To alleviate this performance bias , existing approaches either re-balance the data distribution through sampling ( Chawla et al. , 2002 ; Han et al. , 2005 ; Shen et al. , 2016 ; Mahajan et al. , 2018 ) or the loss for each class ( Cui et al. , 2019 ; Khan et al. , 2017 ; Cao et al. , 2019 ; Khan et al. , 2019 ) by reweighting . Kang et al . ( 2020 ) first propose to decouple representation learning from classifier learning to boost performance , and demonstrate that learning good feature spaces is crucial for long-tailed recognition . Along this direction , SSP ( Yang & Xu , 2020 ) is among the first methods that introduce SSL pretraining into learning the long-tailed recognition models . More specifically , instead of directly training a randomly initialized model from scratch as conventional supervised learning methods , SSP uses a model pretrained with SSL on the same dataset for initialization , which is observed to be able to to alleviate the label bias issue in imbalanced datasets and boost long-tailed recognition performance . In contrast , we conduct a series of systematic studies to directly compare SSL with supervised learning on representation learning . We show that SSL can learn stably well feature spaces robust to the underlying distribution of a dataset . Moreover , inspired by our findings on the benefits of a balanced feature space for generalization , we introduce the k-positive contrastive learning method to explicitly pursue balancedness and discriminativeness for representation learning , which has been shown through experiments to benefit not only long-tailed recognition but also normal recognition tasks . 3 BALANCED FEATURE SPACES FROM CONTRASTIVE LEARNING . In this section , we systematically study the performance of representation models trained by SSL from a collection of training datasets with varying instance number distributions , in contrast with the models learned by supervised learning methods , to explore how SSL performs when the training datasets are not artificially balanced . Furthermore , we investigate the generalization performance of these learned representation models under multiple settings , in order to explore the relationship between the representation model ’ s generalizability and the property of its learned feature space . Notations We define the notations used in this paper . Representation learning aims to obtain a representation model fθ that maps a sample xi into a feature space V such that its corresponding representation vi ∈ V encapsulates desired features for target applications . Let Drep-train = { xi , yi } , i = 1 , . . . , N be the dataset for training the representation model , where yi is the class label for sample xi . Let C denote the number of total classes and nj denote the number of instances within class j . We use { q1 , . . . , qC } with qj = nj/N to denote the discrete instance distribution over the C classes . An imbalanced dataset has significant difference in the class instance numbers , e.g. , q1 qC . We use a multi-layer convolutional neural network fθ ( · ) : xi 7→ vi to implement the representation model . The final classification prediction ŷ is given by a linear classifier ŷ = argmax [ W > v + b ] , where W denotes the classifier weight matrix and b denotes the bias term .

Summary: **Overview:** The paper presents experiments showing that the contrastive learning losses produce better embeddings or feature spaces than those produced by using binary cross-entropy losses. The experiments show that embeddings learned using contrastive learning losses seem to favor long-tailed learning tasks, out-of-distribution tasks, and object detection. The paper also presents an extension of the contrastive loss to improve the embeddings. The experiments in the paper  use common and recent long-tail datasets as well as datasets for object detection and out-of-distribution tasks.