Decoupling Representation and Classifier for Long-Tailed Recognition

1 INTRODUCTION . Visual recognition research has made rapid advances during the past years , driven primarily by the use of deep convolutional neural networks ( CNNs ) and large image datasets , most importantly the ImageNet Challenge ( Russakovsky et al. , 2015 ) . Such datasets are usually artificially balanced with respect to the number of instances for each object/class in the training set . Visual phenomena , however , follow a long-tailed distribution that many standard approaches fail to properly model , leading to a significant drop in accuracy . Motivated by this , a number of works have recently emerged that try to study long-tailed recognition , i.e. , recognition in a setting where the number of instances in each class highly varies and follows a long-tailed distribution . When learning with long-tailed data , a common challenge is that instance-rich ( or head ) classes dominate the training procedure . The learned classification model tends to perform better on these classes , while performance is significantly worse for instance-scarce ( or tail ) classes . To address this issue and to improve performance across all classes , one can re-sample the data or design specific loss functions that better facilitate learning with imbalanced data ( Chawla et al. , 2002 ; Cui et al. , 2019 ; Cao et al. , 2019 ) . Another direction is to enhance recognition performance of the tail classes by transferring knowledge from the head classes ( Wang et al. , 2017 ; 2018 ; Zhong et al. , 2019 ; Liu et al. , 2019 ) . Nevertheless , the common belief behind existing approaches is that designing proper sampling strategies , losses , or even more complex models , is useful for learning high-quality representations for long-tailed recognition . Most aforementioned approaches thus learn the classifiers used for recognition jointly with the data representations . However , such a joint learning scheme makes it unclear how the long-tailed recognition ability is achieved—is it from learning a better representation or by handling the data imbalance better via shifting classifier decision boundaries ? To answer this question , we take one step back and decouple long-tail recognition into representation learning and classification . For learning rep- resentations , the model is exposed to the training instances and trained through different sampling strategies or losses . For classification , upon the learned representations , the model recognizes the long-tailed classes through various classifiers . We evaluate the performance of various sampling and classifier training strategies for long-tailed recognition under both joint and decoupled learning schemes . Specifically , we first train models to learn representations with different sampling strategies , including the standard instance-based sampling , class-balanced sampling and a mixture of them . Next , we study three different basic approaches to obtain a classifier with balanced decision boundaries , on top of the learned representations . They are 1 ) re-training the parametric linear classifier in a class-balancing manner ( i.e. , re-sampling ) ; 2 ) non-parametric nearest class mean classifier , which classifies the data based on their closest class-specific mean representations from the training set ; and 3 ) normalizing the classifier weights , which adjusts the weight magnitude directly to be more balanced , adding a temperature to modulate the normalization procedure . We conduct extensive experiments to compare the aforementioned instantiations of the decoupled learning scheme with the conventional scheme that jointly trains the classifier and the representations . We also compare to recent , carefully designed and more complex models , including approaches using memory ( e.g. , OLTR ( Liu et al. , 2019 ) ) as well as more sophisticated losses ( Cui et al. , 2019 ) . From our extensive study across three long-tail datasets , ImageNet-LT , Places-LT and iNaturalist , we make the following intriguing observations : • We find that decoupling representation learning and classification has surprising results that challenge common beliefs for long-tailed recognition : instance-balanced sampling learns the best and most generalizable representations . • It is advantageous in long-tailed recognition to re-adjust the decision boundaries specified by the jointly learned classifier during representation learning : Our experiments show that this can either be achieved by retraining the classifier with class-balanced sampling or by a simple , yet effective , classifier weight normalization which has only a single hyperparameter controlling the “ temperature ” and which does not require additional training . • By applying the decoupled learning scheme to standard networks ( e.g. , ResNeXt ) , we achieve significantly higher accuracy than well established state-of-the-art methods ( different sampling strategies , new loss designs and other complex modules ) on multiple longtailed recognition benchmark datasets , including ImageNet-LT , Places-LT , and iNaturalist . 2 RELATED WORK . Long-tailed recognition has attracted increasing attention due to the prevalence of imbalanced data in real-world applications ( Wang et al. , 2017 ; Zhou et al. , 2017 ; Mahajan et al. , 2018 ; Zhong et al. , 2019 ; Gupta et al. , 2019 ) . Recent studies have mainly pursued the following three directions : Data distribution re-balancing . Along this direction , researchers have proposed to re-sample the dataset to achieve a more balanced data distribution . These methods include over-sampling ( Chawla et al. , 2002 ; Han et al. , 2005 ) for the minority classes ( by adding copies of data ) , undersampling ( Drummond et al. , 2003 ) for the majority classes ( by removing data ) , and class-balanced sampling ( Shen et al. , 2016 ; Mahajan et al. , 2018 ) based on the number of samples for each class . Class-balanced Losses . Various methods are proposed to assign different losses to different training samples for each class . The loss can vary at class-level for matching a given data distribution and improving the generalization of tail classes ( Cui et al. , 2019 ; Khan et al. , 2017 ; Cao et al. , 2019 ; Khan et al. , 2019 ; Huang et al. , 2019 ) . A more fine-grained control of the loss can also be achieved at sample level , e.g . with Focal loss ( Lin et al. , 2017 ) , Meta-Weight-Net ( Shu et al. , 2019 ) , re-weighted training ( Ren et al. , 2018 ) , or based on Bayesian uncertainty ( Khan et al. , 2019 ) . Recently , Hayat et al . ( 2019 ) proposed to balance the classification regions of head and tail classes using an affinity measure to enforce cluster centers of classes to be uniformly spaced and equidistant . Transfer learning from head- to tail classes . Transfer-learning based methods address the issue of imbalanced training data by transferring features learned from head classes with abundant training instances to under-represented tail classes . Recent work includes transferring the intra-class variance ( Yin et al. , 2019 ) and transferring semantic deep features ( Liu et al. , 2019 ) . However it is usually a non-trivial task to design specific modules ( e.g . external memory ) for feature transfer . A benchmark for low-shot recognition was proposed by Hariharan & Girshick ( 2017 ) and consists of a representation learning phase without access to the low-shot classes and a subsequent low-shot learning phase . In contrast , the setup for long-tail recognition assumes access to both head and tail classes and a more continuous decrease in in class labels . Recently , Liu et al . ( 2019 ) and Cao et al . ( 2019 ) adopt re-balancing schedules that learn representation and classifier jointly within a two-stage training scheme . OLTR ( Liu et al. , 2019 ) uses instance-balanced sampling to first learn representations that are fine-tuned in a second stage with class-balanced sampling together with a memory module . LDAM ( Cao et al. , 2019 ) introduces a label-distribution-aware margin loss that expands the decision boundaries of few-shot classes . In Section 5 we exhaustively compare to OLTR and LDAM , since they report state-of-the-art results for the ImageNet-LT , Places-LT and iNaturalist datasets . In our work , we argue for decoupling representation and classification . We demonstrate that in a long-tailed scenario , this separation allows straightforward approaches to achieve high recognition performance , without the need for designing sampling strategies , balance-aware losses or adding memory modules . 3 LEARNING REPRESENTATIONS FOR LONG-TAILED RECOGNITION . For long-tailed recognition , the training set follows a long-tailed distribution over the classes . As we have less data about infrequent classes during training , the models trained using imbalanced datasets tend to exhibit under-fitting on the few-shot classes . But in practice we are interested in obtaining the model capable of recognizing all classes well . Various re-sampling strategies ( Chawla et al. , 2002 ; Shen et al. , 2016 ; Cao et al. , 2019 ) , loss reweighting and margin regularization over few-shot classes are thus proposed . However , it remains unclear how they achieve performance improvement , if any , for long-tailed recognition . Here we systematically investigate their effectiveness by disentangling representation learning from classifier learning , in order to identify what indeed matters for longtailed recognition . Notation . We define the notation used through the paper . Let X = { xi , yi } , i ∈ { 1 , . . . , n } be a training set , where yi is the label for data point xi . Let nj denote the number of training sample for class j , and let n = ∑C j=1 nj be the total number of training samples . Without loss of generality , we assume that the classes are sorted by cardinality in decreasing order , i.e. , if i < j , then ni ≥ nj . Additionally , since we are in a long-tail setting , n1 nC . Finally , we denote with f ( x ; θ ) = z the representation for x , where f ( x ; θ ) is implemented by a deep CNN model with parameter θ . The final class prediction ỹ is given by a classifier function g , such that ỹ = arg max g ( z ) . For the common case , g is a linear classifier , i.e. , g ( z ) = W > z + b , where W denotes the classifier weight matrix , and b is the bias . We present other instantiations of g in Section 4 . Sampling strategies . In this section we present a number of sampling strategies that aim at rebalancing the data distribution for representation and classifier learning . For most sampling strategies presented below , the probability pj of sampling a data point from class j is given by : pj = nqj∑C i=1 n q i , ( 1 ) where q ∈ [ 0 , 1 ] and C is the number of training classes . Different sampling strategies arise for different values of q and below we present strategies that correspond to q = 1 , q = 0 , and q = 1/2 . Instance-balanced sampling . This is the most common way of sampling data , where each training example has equal probability of being selected . For instance-balanced sampling , the probability pIBj is given by Equation 1 with q = 1 , i.e. , a data point from class j will be sampled proportionally to the cardinality nj of the class in the training set . Class-balanced sampling . For imbalanced datasets , instance-balanced sampling has been shown to be sub-optimal ( Huang et al. , 2016 ; Wang et al. , 2017 ) as the model under-fits for few-shot classes leading to lower accuracy , especially for balanced test sets . Class-balanced sampling has been used to alleviate this discrepancy , as , in this case , each class has an equal probability of being selected . The probability pCBj is given by Eq . ( 1 ) with q = 0 , i.e. , p CB j = 1/C . One can see this as a two- stage sampling strategy , where first a class is selected uniformly from the set of classes , and then an instance from that class is subsequently uniformly sampled . Square-root sampling . A number of variants of the previous sampling strategies have been explored . A commonly used variant is square-root sampling ( Mikolov et al. , 2013 ; Mahajan et al. , 2018 ) , where q is set to 1/2 in Eq . ( 1 ) above . Progressively-balanced sampling . Recent approaches ( Cui et al. , 2018 ; Cao et al. , 2019 ) utilized mixed ways of sampling , i.e. , combinations of the sampling strategies presented above . In practice this involves first using instance-balanced sampling for a number of epochs , and then class-balanced sampling for the last epochs . These mixed sampling approaches require setting the number of epochs before switching the sampling strategy as an explicit hyper-parameter . Here , we experiment with a softer version , progressively-balanced sampling , that progressively “ interpolates ” between instancebalanced and class-balanced sampling as learning progresses . Its sampling probability/weight pj for class j is now a function of the epoch t , pPBj ( t ) = ( 1− t T ) pIBj + t T pCBj , ( 2 ) where T is the total number of epochs . Figure 3 in appendix depicts the sampling probabilities . Loss re-weighting strategies . Loss re-weighting functions for imbalanced data have been extensively studied , and it is beyond the scope of this paper to examine all related approaches . What is more , we found that some of the most recent approaches reporting high performance were hard to train and reproduce and in many cases require extensive , dataset-specific hyper-parameter tuning . In Section A of the Appendix we summarize the latest , best performing methods from this area . In Section 5 we show that , without bells and whistles , baseline methods equipped with a properly balanced classifier can perform equally well , if not better , than the latest loss re-weighting approaches .

Summary: The paper considers the problem of long-tailed image classification, where the class frequencies during (supervised) training of an image classifier are heavily skewed, so that the classifier underfits on under-represented classes. Different known and novel sampling schemes during training as well as post-training procedures to restore the class balance after training are studied. The overall best strategy turns out to be naive training on the skewed training set, and post-hoc rebalancing only of the classification stage. The paper presents various ablation studies and comparisons with related methods on the ImageNet-LT, Places-LT, and iNaturalist data sets, achieving state-of-the-art performance.