Where is the bottleneck in long-tailed classification?

1 INTRODUCTION . Long-tailed distributions are those in which samples from a small number of head ( majority ) classes vastly outnumber the samples for a large number of tail ( minority ) classes . Such distributions can occur naturally , such as rare diseases in medical contexts or minority ethnic groups in face recognition . Learning from long-tailed data is challenging because it combines three problems : imbalanced learning across the head/body/tail , few-shot learning on the tail classes , and a label distribution shift at test time ( long-tailed training set , balanced test set ) . In practice , most research focuses on the data imbalance or label distribution shift problem , which reflects a communal belief that the bottleneck in long-tailed classification lies in the classifier rather than the representation — the representation is though to be ” good enough ” . We question this belief and provide evidence against it through a series of experiments . First , we show that the representation is not good enough ( §1.1 ) and may be a larger bottleneck than the classifier . We show that the long-tailed representations and ” normal ” representations have substantial differences in their second moments ( §2.1 ) , and that long-tailed representations have significantly worse inter-class separation and intra-class compactness ( §2.2 ) , and poorly localize tail classes in feature space ( §8 ) . Finally , we explain the reason why data augmentation boosts performance in long-tailed learning ( Zhong et al . ; Zhang et al. , 2021b ) ( §1.2 ) ) despite having no effect on the imbalance factor , showing that it significantly improves localization of classes ( §8 ) , and confers robustness to distribution shift by improving inter-class separation and inter-class compactness ( §2.2 . 1.1 IS THE REPRESENTATION OR CLASSIFIER THE BOTTLENECK ? . There is a tension between representation learning and classifier learning in long-tailed classification , due to their opposite locations in the bias-variance trade-off spectrum . Representation learning is thought to suffer more from data scarcity rather than data imbalance ( Yang & Xu , 2020 ) , while classifier learning is though to suffer more from data imbalance . Decoupled training ( Zhong et al . ; Kang et al. , 2020 ) methods attempt to address this tension by training the classifier and representation separately ( left side of Fig 1 ) . It is a common belief that the representations are ” good enough ” and the bottleneck is the classifier ( Zhang et al. , 2021a ; Yang & Xu , 2020 ; Kang et al. , 2020 ) , thus many works focus on interventions in the classifier learning phase ( Menon et al. , 2020 ) . We test this belief by carrying out an altered form of decoupled training ( right side of Fig 1 ) . The three most commonly used long-tailed datasets — CIFAR-10/100 LT and ImageNet-LT ( Liu et al. , 2019 ) and ImageNet-LT — all have larger corresponding balanced datasets , namely CIFAR-10/100 ( Krizhevsky ) and ImageNet-1k ( Deng et al. , 2009 ) . Let DLT denote the long-tailed version of a dataset , andD ? denote the normal , balanced version . To assess the relative impact of the representation and classifier , we apply decoupled training without resampling . Instead of resampling , we swap between the DLT and D ? versions of the dataset between Stage 1 and Stage 2 . If the classifier is truly the bottleneck , we expect to see that a representation trained on D ? with a classifier retrained on DLT should perform worse on the balanced test set than a representation trained on DLT with a classifier retrained on D ? . We find that this is not true ( Table 1 ) . As expected , the D ? → D ? model ( with a representation and classifier trained on the true , balanced dataset ) , performs the best . However , across all three datasets , the D ? → DLT model outperforms the DLT → D ? model by a substantial margin . This indicates that contrary to popular belief , the bias in the representation dominates the bias in the classifier , and not the other way around . 1.2 WHY DOES DATA AUGMENTATION HELP ? . At a first glance , it is unclear why class-agnostic input data augmentation should improve long-tailed classification . Data augmentation is incapable of addressing data imbalance — it leaves the imbal- ance factor of a dataset unchanged . Nevertheless , input data augmentation such as MixUp ( Zhang et al. , 2018 ) , Manifold Mixup ( Verma et al. , 2019 ) and AutoAug ( Cubuk et al. , 2019 ) have proven to be highly effective for reducing bias in long-tailed classification ( Zhong et al . ; Zhang et al. , 2021b ; Tan et al. , 2020 ) . When combined with decoupled training ( Kang et al. , 2020 ) , data augmentation results in even larger increases ( Fig 3 ) . However , the mechanism by which data augmentation improves performance and reduces bias is not known . It can not address data imbalance , so data augmentation must be affecting another quality of the representation . 1.3 WHY ISN ’ T REBALANCING ENOUGH ? . Resampling and reweighting strategies have been an essential part of long-tailed classification . However , as first observed by Zhang et al . ( 2021a ) , rebalancing does not seem to be enough . Following the methodology of Zhang et al . ( 2021a ) , we construct an empirical classifier bound by first training a representation on the long-tailed DLT dataset , in this case CIFAR-10 with an imbalance factor of 100 . Next , we follow the standard decoupled training startegy for cRT ( Kang et al. , 2020 ) and retrain the classifier on version ofDLT that has been balanced by class-aware resampling Kang et al . ( 2020 ) after freezing the representation . To obtain the empirical classifier performance bound , we then take the frozen representations from the first stage and train a new classifier on the true dataset D ? . The performance of the classifier trained on D ? can be viewed as an empirical upper bound on the performance obtainable through any resampling strategy , and is the ideal performance to aim for . There is a consistent gap between the class-aware resampling strategy and the upper bound ( Fig 3 ) , suggesting that the resampling strategy alone leaves performance on the table . Moreover , applying data augmentation again in the second stage boosts the performance even further , which is suprising in light of the fact that the classifier is primarily thought to suffer from imbalance — so why should applying data augmentation in the classifier learning phase have such a big impact on performance ? 2 EXPERIMENTS . In the previous section , we showed results that provoke three questions . 1 . Why are the representations learned from the long-tailed data so much poorer than the representations learned from the true dataset ? 2 . Why does data augmentation help in the representation learning phase of decoupledtraining for long-tailed classification ? We provide answers to these questions by analyzing the differences between the representations learned from the long-tailed data , the representations learned from the true dataset , and representations learned from the long-tailed data with augmentation . 2.1 REPRESENTATIONS LEARNED FROM LONG-TAILED DATA HAVE DIFFERENT 2ND MOMENTS . We train a ResNet-32 on a true , balanced dataset D ? and a ResNet-32 on a long-tailed dataset DLT . We then extract features from both models for the entire dataset ( D ? ) , to reflect the true data distribution . We use CIFAR10/100-LT with an imbalance factor of 100 . The representations learned from long-tailed data differ from the representations learned from the true dataset in two significant ways . First , there are notable differences in the variance and covariance of the classes that belonged to the head or tail in the long-tailed dataset within the DLT representation , but not in the D ? representations ( Fig 4 ) . Specifically , the variance of the features belonging to the tail and body classes ( the L1-norm of the covariance matrix diagonal ) diverges substantially from the variance of the features belonging to the head classes . This is not only true of the variance , but of the covariance . The Frobenius norm of the sample covariance matrix computed from the features of a class is also different between the D ? and DLT learners . For the learner trained on D ? , there are no significant differences in variance or covariance between the head , tail , and body , and differences are dominated by noise among the classes . For the learner trained on DLT , the variance of the head is systemically lower than the variance of the head , and this is also true of the covariance . Next , we examine the proportion of variance explained by the first principal component of the data matrix for each class . Concretely , we take all feature vectors belonging to the class from the complete dataset D ? , and compute the first principal component of the data matrix of features for the class . We then plot the proportion of variance explained by the first principal component of each class ( Fig . 5 ) , thus obtaining an estimate of the relative complexity of the subspace each class resides in . Across all three datasets , we see the same result : the first principal component explains more of the variance in the representations learned from DLT than the representations learned from D ? . Intuitively , these results suggest that the representations learned from long-tailed data are substantially different than the representations learned from the true , balanced dataset . The differences are most marked for tail and body classes , and least for the head classes . Geometrically , the long-tailed representations appear to occupy a less complex subspace than the D ? representations , yet exhibit greater variance , suggesting that the feature space is biased — fewer dimensions in the long-tailed feature space are relevant to all classes , and thus the variance concentrates in fewer dimensions of the feature space . 2.2 REPRESENTATIONS LEARNED FROM LONG-TAILED DATA ARE MORE DIFFUSE AND LESS SEPARABLE . We now examine the representations from the angular perspective . Recall that the score for the class k logit in a standard linear classifier for a feature vector x wk · x = ||wk|| × ||x|| × cos θ ( 1 ) Differences in intra-class compactness between representations is a function of the angle θ between x and wk when ||wk|| = ||x|| In practice , the norm ||wk|| of the logits of a class k are strongly correlated with the cardinality of the class ( Menon et al. , 2020 ; Kang et al. , 2020 ; Zhong et al . ) — tail classes have smaller norms and vice versa , reflecting the class priors . However , decoupled training roughly equalizes the logit norms Kang et al . ( 2020 ) , allowing us to ignore the ||wk|| × ||x|| term and focus only on the angle θ . In this setting , the classifier reduces to an angular classifier , and a sample x will be assigned to the logit wk to which it has the smallest angular distance . Thus , we proceed to an examination of the angular distribution created by the features of the long-tailed representation learner and the D ? representation learner . The representations of the tail classes learned by the long-tailed representation learner are significantly less compact than the representations learned by theD ? representation learner with respect to the true data distribution ( Fig 6 ) . While the long-tailed per-class representations are compact w.r.t to the long-tailed distribution , when the unseen samples making up the true data distribution of D ? are added , the angular spread significantly increases for the tail classes . This means that any classifier boundaries learned using only the samples available in the DLT distribution will fail to generalize , as the majority of unseen samples for a tail will escape the classification boundaries . Interestingly , augmentation appears to significantly reduce the compactness gap between the long-tailed and true data distribution for the tail classes . This may partly explain the success of data augmentation in long-tailed classification . However , data augmentation ( MixUp ) also significantly increases the angular spread of all classes , in contrast to the representation trained on D ? , which addresses the distribution shift without compromising the angular spread of the classes . The representations of the tail classes learned by the long-tailed representation learner also display worse inter-class separation with respect to the true data distribution ( Fig 7 ) . The angular distance to the nearest incorrect class center significantly shrinks when the unseen samples from the true data distribution are considered . Again , augmentation makes a significant difference : the angular distribution shift of representations trained on the long-tailed dataset with augmentation is far smaller than the angular distribution shift of representations trained on the long-tailed dataset without augmentation . Thus , two major differences between the representations learned from long-tailed data and representations learned from the D ? data is the effect of distribution shift on intra-class compactness and inter-class separation . When unseen samples from the true data distribution are added , the angular distributions of the tail classes balloon for the representation trained on long-tailed data . The poor intra-class compactness also leads to poor inter-class separation w.r.t to the true data distribution , as the expansion of angular distributions results in each class eating into the margin between it ’ s nearest neighbor . This means that classifier boundaries drawn on the long-tailed distribution will likely be optimistic and fail to generalize due to unseen samples from the true distribution mostly escaping the convex polytope of the long-tailed classifier boundaries . However , data augmentation seems to help , because the representations learned from long-tailed data with data augmentation exhibit much better inter-class separation and intra-class compactness under distribution shift . Differences in inter-class separation between representations

Summary: This paper seeks to study what is the bottleneck in long-tailed learning. Based on extensive experiments, the authors propose that representation learning is the bottleneck in long-tailed classification. Also, this paper analyzes representation learning from the perspectives of intra-class compactness and inter-class separation, as well as the influence of data mixup on long-tailed representation learning.