Decoupling Representation and Classifier for Long-Tailed Recognition

1 INTRODUCTION . Visual recognition research has made rapid advances during the past years , driven primarily by the use of deep convolutional neural networks ( CNNs ) and large image datasets , most importantly the ImageNet Challenge ( Russakovsky et al. , 2015 ) . Such datasets are usually artificially balanced with respect to the number of instances for each object/class in the training set . Visual phenomena , however , follow a long-tailed distribution that many standard approaches fail to properly model , leading to a significant drop in accuracy . Motivated by this , a number of works have recently emerged that try to study long-tailed recognition , i.e. , recognition in a setting where the number of instances in each class highly varies and follows a long-tailed distribution . When learning with long-tailed data , a common challenge is that instance-rich ( or head ) classes dominate the training procedure . The learned classification model tends to perform better on these classes , while performance is significantly worse for instance-scarce ( or tail ) classes . To address this issue and to improve performance across all classes , one can re-sample the data or design specific loss functions that better facilitate learning with imbalanced data ( Chawla et al. , 2002 ; Cui et al. , 2019 ; Cao et al. , 2019 ) . Another direction is to enhance recognition performance of the tail classes by transferring knowledge from the head classes ( Wang et al. , 2017 ; 2018 ; Zhong et al. , 2019 ; Liu et al. , 2019 ) . Nevertheless , the common belief behind existing approaches is that designing proper sampling strategies , losses , or even more complex models , is useful for learning high-quality representations for long-tailed recognition . Most aforementioned approaches thus learn the classifiers used for recognition jointly with the data representations . However , such a joint learning scheme makes it unclear how the long-tailed recognition ability is achieved—is it from learning a better representation or by handling the data imbalance better via shifting classifier decision boundaries ? To answer this question , we take one step back and decouple long-tail recognition into representation learning and classification . For learning rep- resentations , the model is exposed to the training instances and trained through different sampling strategies or losses . For classification , upon the learned representations , the model recognizes the long-tailed classes through various classifiers . We evaluate the performance of various sampling and classifier training strategies for long-tailed recognition under both joint and decoupled learning schemes . Specifically , we first train models to learn representations with different sampling strategies , including the standard instance-based sampling , class-balanced sampling and a mixture of them . Next , we study three different basic approaches to obtain a classifier with balanced decision boundaries , on top of the learned representations . They are 1 ) re-training the parametric linear classifier in a class-balancing manner ( i.e. , re-sampling ) ; 2 ) non-parametric nearest class mean classifier , which classifies the data based on their closest class-specific mean representations from the training set ; and 3 ) normalizing the classifier weights , which adjusts the weight magnitude directly to be more balanced , adding a temperature to modulate the normalization procedure . We conduct extensive experiments to compare the aforementioned instantiations of the decoupled learning scheme with the conventional scheme that jointly trains the classifier and the representations . We also compare to recent , carefully designed and more complex models , including approaches using memory ( e.g. , OLTR ( Liu et al. , 2019 ) ) as well as more sophisticated losses ( Cui et al. , 2019 ) . From our extensive study across three long-tail datasets , ImageNet-LT , Places-LT and iNaturalist , we make the following intriguing observations : • We find that decoupling representation learning and classification has surprising results that challenge common beliefs for long-tailed recognition : instance-balanced sampling learns the best and most generalizable representations . • It is advantageous in long-tailed recognition to re-adjust the decision boundaries specified by the jointly learned classifier during representation learning : Our experiments show that this can either be achieved by retraining the classifier with class-balanced sampling or by a simple , yet effective , classifier weight normalization which has only a single hyperparameter controlling the “ temperature ” and which does not require additional training . • By applying the decoupled learning scheme to standard networks ( e.g. , ResNeXt ) , we achieve significantly higher accuracy than well established state-of-the-art methods ( different sampling strategies , new loss designs and other complex modules ) on multiple longtailed recognition benchmark datasets , including ImageNet-LT , Places-LT , and iNaturalist . 2 RELATED WORK . Long-tailed recognition has attracted increasing attention due to the prevalence of imbalanced data in real-world applications ( Wang et al. , 2017 ; Zhou et al. , 2017 ; Mahajan et al. , 2018 ; Zhong et al. , 2019 ; Gupta et al. , 2019 ) . Recent studies have mainly pursued the following three directions : Data distribution re-balancing . Along this direction , researchers have proposed to re-sample the dataset to achieve a more balanced data distribution . These methods include over-sampling ( Chawla et al. , 2002 ; Han et al. , 2005 ) for the minority classes ( by adding copies of data ) , undersampling ( Drummond et al. , 2003 ) for the majority classes ( by removing data ) , and class-balanced sampling ( Shen et al. , 2016 ; Mahajan et al. , 2018 ) based on the number of samples for each class . Class-balanced Losses . Various methods are proposed to assign different losses to different training samples for each class . The loss can vary at class-level for matching a given data distribution and improving the generalization of tail classes ( Cui et al. , 2019 ; Khan et al. , 2017 ; Cao et al. , 2019 ; Khan et al. , 2019 ; Huang et al. , 2019 ) . A more fine-grained control of the loss can also be achieved at sample level , e.g . with Focal loss ( Lin et al. , 2017 ) , Meta-Weight-Net ( Shu et al. , 2019 ) , re-weighted training ( Ren et al. , 2018 ) , or based on Bayesian uncertainty ( Khan et al. , 2019 ) . Recently , Hayat et al . ( 2019 ) proposed to balance the classification regions of head and tail classes using an affinity measure to enforce cluster centers of classes to be uniformly spaced and equidistant . Transfer learning from head- to tail classes . Transfer-learning based methods address the issue of imbalanced training data by transferring features learned from head classes with abundant training instances to under-represented tail classes . Recent work includes transferring the intra-class variance ( Yin et al. , 2019 ) and transferring semantic deep features ( Liu et al. , 2019 ) . However it is usually a non-trivial task to design specific modules ( e.g . external memory ) for feature transfer . A benchmark for low-shot recognition was proposed by Hariharan & Girshick ( 2017 ) and consists of a representation learning phase without access to the low-shot classes and a subsequent low-shot learning phase . In contrast , the setup for long-tail recognition assumes access to both head and tail classes and a more continuous decrease in in class labels . Recently , Liu et al . ( 2019 ) and Cao et al . ( 2019 ) adopt re-balancing schedules that learn representation and classifier jointly within a two-stage training scheme . OLTR ( Liu et al. , 2019 ) uses instance-balanced sampling to first learn representations that are fine-tuned in a second stage with class-balanced sampling together with a memory module . LDAM ( Cao et al. , 2019 ) introduces a label-distribution-aware margin loss that expands the decision boundaries of few-shot classes . In Section 5 we exhaustively compare to OLTR and LDAM , since they report state-of-the-art results for the ImageNet-LT , Places-LT and iNaturalist datasets . In our work , we argue for decoupling representation and classification . We demonstrate that in a long-tailed scenario , this separation allows straightforward approaches to achieve high recognition performance , without the need for designing sampling strategies , balance-aware losses or adding memory modules . 3 LEARNING REPRESENTATIONS FOR LONG-TAILED RECOGNITION . For long-tailed recognition , the training set follows a long-tailed distribution over the classes . As we have less data about infrequent classes during training , the models trained using imbalanced datasets tend to exhibit under-fitting on the few-shot classes . But in practice we are interested in obtaining the model capable of recognizing all classes well . Various re-sampling strategies ( Chawla et al. , 2002 ; Shen et al. , 2016 ; Cao et al. , 2019 ) , loss reweighting and margin regularization over few-shot classes are thus proposed . However , it remains unclear how they achieve performance improvement , if any , for long-tailed recognition . Here we systematically investigate their effectiveness by disentangling representation learning from classifier learning , in order to identify what indeed matters for longtailed recognition . Notation . We define the notation used through the paper . Let X = { xi , yi } , i ∈ { 1 , . . . , n } be a training set , where yi is the label for data point xi . Let nj denote the number of training sample for class j , and let n = ∑C j=1 nj be the total number of training samples . Without loss of generality , we assume that the classes are sorted by cardinality in decreasing order , i.e. , if i < j , then ni ≥ nj . Additionally , since we are in a long-tail setting , n1 nC . Finally , we denote with f ( x ; θ ) = z the representation for x , where f ( x ; θ ) is implemented by a deep CNN model with parameter θ . The final class prediction ỹ is given by a classifier function g , such that ỹ = arg max g ( z ) . For the common case , g is a linear classifier , i.e. , g ( z ) = W > z + b , where W denotes the classifier weight matrix , and b is the bias . We present other instantiations of g in Section 4 . Sampling strategies . In this section we present a number of sampling strategies that aim at rebalancing the data distribution for representation and classifier learning . For most sampling strategies presented below , the probability pj of sampling a data point from class j is given by : pj = nqj∑C i=1 n q i , ( 1 ) where q ∈ [ 0 , 1 ] and C is the number of training classes . Different sampling strategies arise for different values of q and below we present strategies that correspond to q = 1 , q = 0 , and q = 1/2 . Instance-balanced sampling . This is the most common way of sampling data , where each training example has equal probability of being selected . For instance-balanced sampling , the probability pIBj is given by Equation 1 with q = 1 , i.e. , a data point from class j will be sampled proportionally to the cardinality nj of the class in the training set . Class-balanced sampling . For imbalanced datasets , instance-balanced sampling has been shown to be sub-optimal ( Huang et al. , 2016 ; Wang et al. , 2017 ) as the model under-fits for few-shot classes leading to lower accuracy , especially for balanced test sets . Class-balanced sampling has been used to alleviate this discrepancy , as , in this case , each class has an equal probability of being selected . The probability pCBj is given by Eq . ( 1 ) with q = 0 , i.e. , p CB j = 1/C . One can see this as a two- stage sampling strategy , where first a class is selected uniformly from the set of classes , and then an instance from that class is subsequently uniformly sampled . Square-root sampling . A number of variants of the previous sampling strategies have been explored . A commonly used variant is square-root sampling ( Mikolov et al. , 2013 ; Mahajan et al. , 2018 ) , where q is set to 1/2 in Eq . ( 1 ) above . Progressively-balanced sampling . Recent approaches ( Cui et al. , 2018 ; Cao et al. , 2019 ) utilized mixed ways of sampling , i.e. , combinations of the sampling strategies presented above . In practice this involves first using instance-balanced sampling for a number of epochs , and then class-balanced sampling for the last epochs . These mixed sampling approaches require setting the number of epochs before switching the sampling strategy as an explicit hyper-parameter . Here , we experiment with a softer version , progressively-balanced sampling , that progressively “ interpolates ” between instancebalanced and class-balanced sampling as learning progresses . Its sampling probability/weight pj for class j is now a function of the epoch t , pPBj ( t ) = ( 1− t T ) pIBj + t T pCBj , ( 2 ) where T is the total number of epochs . Figure 3 in appendix depicts the sampling probabilities . Loss re-weighting strategies . Loss re-weighting functions for imbalanced data have been extensively studied , and it is beyond the scope of this paper to examine all related approaches . What is more , we found that some of the most recent approaches reporting high performance were hard to train and reproduce and in many cases require extensive , dataset-specific hyper-parameter tuning . In Section A of the Appendix we summarize the latest , best performing methods from this area . In Section 5 we show that , without bells and whistles , baseline methods equipped with a properly balanced classifier can perform equally well , if not better , than the latest loss re-weighting approaches .

Summary: The paper tries to handle the class imbalance problem by decoupling the learning process into representation learning and classification, in contrast to the current methods that jointly learn both of them. They comprehensively study several sampling methods for representation learning and different strategies for classification. They find that instance-balanced sampling gives the best representation, and simply adjusting the classifier will equip the model with long-tailed recognition ability. They achieve start of art on long-tailed data (ImageNet-LT, Places-LT and iNaturalist).