Where is the bottleneck in long-tailed classification?

1 INTRODUCTION . Long-tailed distributions are those in which samples from a small number of head ( majority ) classes vastly outnumber the samples for a large number of tail ( minority ) classes . Such distributions can occur naturally , such as rare diseases in medical contexts or minority ethnic groups in face recognition . Learning from long-tailed data is challenging because it combines three problems : imbalanced learning across the head/body/tail , few-shot learning on the tail classes , and a label distribution shift at test time ( long-tailed training set , balanced test set ) . In practice , most research focuses on the data imbalance or label distribution shift problem , which reflects a communal belief that the bottleneck in long-tailed classification lies in the classifier rather than the representation — the representation is though to be ” good enough ” . We question this belief and provide evidence against it through a series of experiments . First , we show that the representation is not good enough ( §1.1 ) and may be a larger bottleneck than the classifier . We show that the long-tailed representations and ” normal ” representations have substantial differences in their second moments ( §2.1 ) , and that long-tailed representations have significantly worse inter-class separation and intra-class compactness ( §2.2 ) , and poorly localize tail classes in feature space ( §8 ) . Finally , we explain the reason why data augmentation boosts performance in long-tailed learning ( Zhong et al . ; Zhang et al. , 2021b ) ( §1.2 ) ) despite having no effect on the imbalance factor , showing that it significantly improves localization of classes ( §8 ) , and confers robustness to distribution shift by improving inter-class separation and inter-class compactness ( §2.2 . 1.1 IS THE REPRESENTATION OR CLASSIFIER THE BOTTLENECK ? . There is a tension between representation learning and classifier learning in long-tailed classification , due to their opposite locations in the bias-variance trade-off spectrum . Representation learning is thought to suffer more from data scarcity rather than data imbalance ( Yang & Xu , 2020 ) , while classifier learning is though to suffer more from data imbalance . Decoupled training ( Zhong et al . ; Kang et al. , 2020 ) methods attempt to address this tension by training the classifier and representation separately ( left side of Fig 1 ) . It is a common belief that the representations are ” good enough ” and the bottleneck is the classifier ( Zhang et al. , 2021a ; Yang & Xu , 2020 ; Kang et al. , 2020 ) , thus many works focus on interventions in the classifier learning phase ( Menon et al. , 2020 ) . We test this belief by carrying out an altered form of decoupled training ( right side of Fig 1 ) . The three most commonly used long-tailed datasets — CIFAR-10/100 LT and ImageNet-LT ( Liu et al. , 2019 ) and ImageNet-LT — all have larger corresponding balanced datasets , namely CIFAR-10/100 ( Krizhevsky ) and ImageNet-1k ( Deng et al. , 2009 ) . Let DLT denote the long-tailed version of a dataset , andD ? denote the normal , balanced version . To assess the relative impact of the representation and classifier , we apply decoupled training without resampling . Instead of resampling , we swap between the DLT and D ? versions of the dataset between Stage 1 and Stage 2 . If the classifier is truly the bottleneck , we expect to see that a representation trained on D ? with a classifier retrained on DLT should perform worse on the balanced test set than a representation trained on DLT with a classifier retrained on D ? . We find that this is not true ( Table 1 ) . As expected , the D ? → D ? model ( with a representation and classifier trained on the true , balanced dataset ) , performs the best . However , across all three datasets , the D ? → DLT model outperforms the DLT → D ? model by a substantial margin . This indicates that contrary to popular belief , the bias in the representation dominates the bias in the classifier , and not the other way around . 1.2 WHY DOES DATA AUGMENTATION HELP ? . At a first glance , it is unclear why class-agnostic input data augmentation should improve long-tailed classification . Data augmentation is incapable of addressing data imbalance — it leaves the imbal- ance factor of a dataset unchanged . Nevertheless , input data augmentation such as MixUp ( Zhang et al. , 2018 ) , Manifold Mixup ( Verma et al. , 2019 ) and AutoAug ( Cubuk et al. , 2019 ) have proven to be highly effective for reducing bias in long-tailed classification ( Zhong et al . ; Zhang et al. , 2021b ; Tan et al. , 2020 ) . When combined with decoupled training ( Kang et al. , 2020 ) , data augmentation results in even larger increases ( Fig 3 ) . However , the mechanism by which data augmentation improves performance and reduces bias is not known . It can not address data imbalance , so data augmentation must be affecting another quality of the representation . 1.3 WHY ISN ’ T REBALANCING ENOUGH ? . Resampling and reweighting strategies have been an essential part of long-tailed classification . However , as first observed by Zhang et al . ( 2021a ) , rebalancing does not seem to be enough . Following the methodology of Zhang et al . ( 2021a ) , we construct an empirical classifier bound by first training a representation on the long-tailed DLT dataset , in this case CIFAR-10 with an imbalance factor of 100 . Next , we follow the standard decoupled training startegy for cRT ( Kang et al. , 2020 ) and retrain the classifier on version ofDLT that has been balanced by class-aware resampling Kang et al . ( 2020 ) after freezing the representation . To obtain the empirical classifier performance bound , we then take the frozen representations from the first stage and train a new classifier on the true dataset D ? . The performance of the classifier trained on D ? can be viewed as an empirical upper bound on the performance obtainable through any resampling strategy , and is the ideal performance to aim for . There is a consistent gap between the class-aware resampling strategy and the upper bound ( Fig 3 ) , suggesting that the resampling strategy alone leaves performance on the table . Moreover , applying data augmentation again in the second stage boosts the performance even further , which is suprising in light of the fact that the classifier is primarily thought to suffer from imbalance — so why should applying data augmentation in the classifier learning phase have such a big impact on performance ? 2 EXPERIMENTS . In the previous section , we showed results that provoke three questions . 1 . Why are the representations learned from the long-tailed data so much poorer than the representations learned from the true dataset ? 2 . Why does data augmentation help in the representation learning phase of decoupledtraining for long-tailed classification ? We provide answers to these questions by analyzing the differences between the representations learned from the long-tailed data , the representations learned from the true dataset , and representations learned from the long-tailed data with augmentation . 2.1 REPRESENTATIONS LEARNED FROM LONG-TAILED DATA HAVE DIFFERENT 2ND MOMENTS . We train a ResNet-32 on a true , balanced dataset D ? and a ResNet-32 on a long-tailed dataset DLT . We then extract features from both models for the entire dataset ( D ? ) , to reflect the true data distribution . We use CIFAR10/100-LT with an imbalance factor of 100 . The representations learned from long-tailed data differ from the representations learned from the true dataset in two significant ways . First , there are notable differences in the variance and covariance of the classes that belonged to the head or tail in the long-tailed dataset within the DLT representation , but not in the D ? representations ( Fig 4 ) . Specifically , the variance of the features belonging to the tail and body classes ( the L1-norm of the covariance matrix diagonal ) diverges substantially from the variance of the features belonging to the head classes . This is not only true of the variance , but of the covariance . The Frobenius norm of the sample covariance matrix computed from the features of a class is also different between the D ? and DLT learners . For the learner trained on D ? , there are no significant differences in variance or covariance between the head , tail , and body , and differences are dominated by noise among the classes . For the learner trained on DLT , the variance of the head is systemically lower than the variance of the head , and this is also true of the covariance . Next , we examine the proportion of variance explained by the first principal component of the data matrix for each class . Concretely , we take all feature vectors belonging to the class from the complete dataset D ? , and compute the first principal component of the data matrix of features for the class . We then plot the proportion of variance explained by the first principal component of each class ( Fig . 5 ) , thus obtaining an estimate of the relative complexity of the subspace each class resides in . Across all three datasets , we see the same result : the first principal component explains more of the variance in the representations learned from DLT than the representations learned from D ? . Intuitively , these results suggest that the representations learned from long-tailed data are substantially different than the representations learned from the true , balanced dataset . The differences are most marked for tail and body classes , and least for the head classes . Geometrically , the long-tailed representations appear to occupy a less complex subspace than the D ? representations , yet exhibit greater variance , suggesting that the feature space is biased — fewer dimensions in the long-tailed feature space are relevant to all classes , and thus the variance concentrates in fewer dimensions of the feature space . 2.2 REPRESENTATIONS LEARNED FROM LONG-TAILED DATA ARE MORE DIFFUSE AND LESS SEPARABLE . We now examine the representations from the angular perspective . Recall that the score for the class k logit in a standard linear classifier for a feature vector x wk · x = ||wk|| × ||x|| × cos θ ( 1 ) Differences in intra-class compactness between representations is a function of the angle θ between x and wk when ||wk|| = ||x|| In practice , the norm ||wk|| of the logits of a class k are strongly correlated with the cardinality of the class ( Menon et al. , 2020 ; Kang et al. , 2020 ; Zhong et al . ) — tail classes have smaller norms and vice versa , reflecting the class priors . However , decoupled training roughly equalizes the logit norms Kang et al . ( 2020 ) , allowing us to ignore the ||wk|| × ||x|| term and focus only on the angle θ . In this setting , the classifier reduces to an angular classifier , and a sample x will be assigned to the logit wk to which it has the smallest angular distance . Thus , we proceed to an examination of the angular distribution created by the features of the long-tailed representation learner and the D ? representation learner . The representations of the tail classes learned by the long-tailed representation learner are significantly less compact than the representations learned by theD ? representation learner with respect to the true data distribution ( Fig 6 ) . While the long-tailed per-class representations are compact w.r.t to the long-tailed distribution , when the unseen samples making up the true data distribution of D ? are added , the angular spread significantly increases for the tail classes . This means that any classifier boundaries learned using only the samples available in the DLT distribution will fail to generalize , as the majority of unseen samples for a tail will escape the classification boundaries . Interestingly , augmentation appears to significantly reduce the compactness gap between the long-tailed and true data distribution for the tail classes . This may partly explain the success of data augmentation in long-tailed classification . However , data augmentation ( MixUp ) also significantly increases the angular spread of all classes , in contrast to the representation trained on D ? , which addresses the distribution shift without compromising the angular spread of the classes . The representations of the tail classes learned by the long-tailed representation learner also display worse inter-class separation with respect to the true data distribution ( Fig 7 ) . The angular distance to the nearest incorrect class center significantly shrinks when the unseen samples from the true data distribution are considered . Again , augmentation makes a significant difference : the angular distribution shift of representations trained on the long-tailed dataset with augmentation is far smaller than the angular distribution shift of representations trained on the long-tailed dataset without augmentation . Thus , two major differences between the representations learned from long-tailed data and representations learned from the D ? data is the effect of distribution shift on intra-class compactness and inter-class separation . When unseen samples from the true data distribution are added , the angular distributions of the tail classes balloon for the representation trained on long-tailed data . The poor intra-class compactness also leads to poor inter-class separation w.r.t to the true data distribution , as the expansion of angular distributions results in each class eating into the margin between it ’ s nearest neighbor . This means that classifier boundaries drawn on the long-tailed distribution will likely be optimistic and fail to generalize due to unseen samples from the true distribution mostly escaping the convex polytope of the long-tailed classifier boundaries . However , data augmentation seems to help , because the representations learned from long-tailed data with data augmentation exhibit much better inter-class separation and intra-class compactness under distribution shift . Differences in inter-class separation between representations

Summary: This paper tries to prove that there is a bottleneck in feature learning for long-tailed classification and data augmentation can help relieve the issues in long-tail feature space. Three major experiments were done to prove that feature space 1) is more biased than balanced feature space, 2)  is more disused and less compact than balanced feature space, and 3) less localized in terms of feature centroids. And data augmentation can help alleviate all three issues.