Semi-supervised Long-tailed Recognition using Alternate Sampling

1 INTRODUCTION . Large-scale datasets , which contain sufficient data in each class , has been a major factor to the success of modern deep learning models for computer vision tasks , such as object recognition . These datasets are usually carefully curated and balanced to have an uniform data distribution over all classes . This balanced data distribution favors model training but could be impractical in many real world applications , where the frequency of samples from different classes can be imbalanced , leading to a long-tailed data distribution . As shown in Figure 1 ( b ) , several highly populated classes take up most of the labeled samples , and some of the classes only have very few samples during training . The long-tailed recognition problem has been widely studied in the literature . One major challenge in this setting ( Liu et al . ( 2019 ) ; Kang et al . ( 2020 ) ; Zhou et al . ( 2020 ) ) to deep learning model training is the tendency of under-fitting in less-populated classes . The root causes of this underfitting are the imbalanced training data distribution as well as the scarcity of data samples in the tail classes . More specifically , with an imbalanced training data distribution , when several head classes take up most of the training samples , tail classes contribute little in the training loss . The model is such that biased towards head classes . Prior works ( Lin et al . ( 2017 ) ; Cao et al . ( 2019 ) ; Kang et al . ( 2020 ) ; Zhou et al . ( 2020 ) ) tried to mitigate the issue by re-sampling the training data to be a balanced distribution or calibrating the sample weights in calculating the loss . However , still the scarcity of tail class data samples limits the intra-class variations and overall recognition accuracy . Methods focusing on few-shot learning have been introduced to address this problem through data augmentation and data synthesis ( Wang et al . ( 2018 ) ; Hariharan & Girshick ( 2017 ) ; Liu et al . ( 2020 ) ) . In this work , we resort to a different path to leverage massive unlabeled real data in training to help improve the long-tailed recognition accuracy . Since data collection is much cheaper and accessible comparing to data annotation , additional unlabeled real data could readily be available in many realworld scenarios . This semi-supervised learning setting has been intensively studied in the literature ( Laine & Aila ( 2016 ) ; Rasmus et al . ( 2015 ) ; Tarvainen & Valpola ( 2017 ) ; Berthelot et al . ( 2019 ) ; Sohn et al . ( 2020 ) ) . However , as shown in Figure 1 ( a ) , when we carefully look at the data distribution of the widely used benchmarks , we observe well-balanced labeled subset and unlabeled subset . As discussed above , the manually curated balanced distribution , can lead to a gap to real-world scenarios . This is especially true in unlabeled data . Without labels , people have no way to balance the data among classes . In this paper , we propose a more realistic and challenging setting , namely semi-supervised longtailed recognition . As shown in Figure 1 ( c ) , we assume a long-tailed data distribution of the overall dataset and both the labeled and unlabeled subsets of training data follow the same underlying longtailed data distribution . This setting generally resembles a realistic data collection and annotation workflow . After collecting the raw data , one has no knowledge of its class distribution before annotation . As it is expensive to annotate the full corpus , a common practice is to randomly sample a subset for annotation under a given labeling budget . When the raw data follows a long-tailed class distribution , we should expect the same in the labeled subset . While this new recognition paradigm shares the challenges in both semi-supervised learning and long-tailed recognition , there is no readily naive solution to it . Methods in long-tailed recognition rely on class labels to achieve balanced training , which are not available in the unlabeled portion in the semi-supervised long-tailed recognition . Prior semi-supervised methods without considering the long-tailed distribution could fail as well . Taking one of the competitive baseline methods for example , ( Yang & Xu ( 2020 ) ) proposed to firstly train a recognition model with the labeled subset to generate pseudo labels for the unlabeled subset , then the model is fine-tuned with the full training dataset . However , when the labeled subset follows a long-tailed distribution , the pseudo labels are much less accurate for tail classes than head classes . As a result , the overall pseudo labels quality could be too bad to leverage ( See Section 4.4 for results in CIFAR-10-SSLT and ImageNet-SSLT ) . To address the semi-supervised long-tailed recognition problem , we present a method designed specifically for this setting . We bring the successful class-balanced sampling strategy and combined it with model decoupling in an alternate learning framework to overcome the difficulty of balancing unlabeled training data . Inspired by ( Kang et al . ( 2020 ) ) , we decouple the recognition model into a feature embedding and a classifier , and train them with random sampling and class-balanced sampling respectively . As we are targeting at a semi-supervised setting , the classifier is only trained on labeled data to get around the difficulty of applying correctly class-balanced sampling on unlabeled data , aligning with the intuition that the classifier needs more robust supervision than the feature embedding . After that , with the proposed alternative learning framework , we improve model by updating the feature embedding and the classifier iteratively . We assign pseudo labels with the up-to-date classifier and observed gradually improved accuracy of pseudo labels over iterations . The pseudo labels are then incorporated in fine-tuning the feature embedding with a regularization term to limit its potential negative impacts . Similar iterative design has been proposed in semi-supervised learning literature ( Laine & Aila ( 2016 ) ; Tarvainen & Valpola ( 2017 ) ) but important implementation details differ . To summarize , in this paper , 1 ) we resort to semi-supervised learning to help improve long-tailed recognition accuracy and identify practical gap of current semi-supervised recognition datasets due to their well-balanced unlabeled subset ; 2 ) we propose a new recognition paradigm named semisupervised long-tailed recognition better resembling real-world data collection and annotation workflow ; 3 ) we propose a new alternative sampling method to address the semi-supervised long-tailed recognition and demonstrate significant improvements on several benchmarks . 2 RELATED WORK . Long-tailed recognition has been recently studied a lot ( Wang et al . ( 2017 ) ; Oh Song et al . ( 2016 ) ; Lin et al . ( 2017 ) ; Zhang et al . ( 2017 ) ; Liu et al . ( 2019 ) ; Wang & Hebert ( 2016 ) ) . Several approaches have been proposed , including metric learning ( Oh Song et al . ( 2016 ) ; Zhang et al . ( 2017 ) ) , loss weighting ( Lin et al . ( 2017 ) ) , and meta-learning ( Wang & Hebert ( 2016 ) ) . Some methods design dedicated loss functions to mitigate the data imbalanced problem . For example , lift loss ( Oh Song et al . ( 2016 ) ) introduces margins between many training samples . Range loss ( Zhang et al . ( 2017 ) ) encourages data from the same class to be close and different classes to be far away in the embedding space . The focal loss ( Lin et al . ( 2017 ) ) dynamically balances weights of positive , hard negative , and easy negative samples . As reported by ( Liu et al . ( 2019 ) ) , when applied to long-tailed recognition , many of these methods improved accuracy of the few-shot group , but at the cost of lower accuracy over the many-shot classes . Other methods , e.g . LDAM-DRW ( Cao et al . ( 2019 ) ) replace cross-entropy loss with LDAM loss . This adds a calibration factor to the original cross-entropy loss . When combined with loss reweighting , it improves the accuracy in all splits in long-tailed recognition . However , it can not be easily generalized to semi-supervised learning . Because both the calibration factor and the loss weight are calculated based on the number of samples of each class . In face recognition and person re-identification , the datasets are mostly with long-tailed distribution . LEAP ( Liu et al . ( 2020 ) ) augmented data samples from tail ( few-shot ) classes by transferring intra-class variations from head ( many-shot ) classes . Instead of data augmentation , we introduce unsupervised data to improve the performance of long-tailed recognition . A recent work ( Yang & Xu ( 2020 ) ) rethinks the value of labels in imbalance learning . As part of the discussion , semi-supervised learning is included . However , only the basic pseudo label solution and simple datasets , such as CIFAR and SVHN , are discussed . More recent works ( Kang et al . ( 2020 ) ; Zhou et al . ( 2020 ) ) with improved long-tailed recognition share the observation that feature embedding and the classifier should be trained with different sampling strategies . In this work , we adopt our method on this observation to learn the feature embedding model with random sampling and train the classifier with class-balanced sampling . This design is further closely compatible with semi-supervised learning under alternate learning . Semi-supervised learning has been extensively discussed in recognition discipline ( Laine & Aila ( 2016 ) ; Rasmus et al . ( 2015 ) ; Tarvainen & Valpola ( 2017 ) ) . One common observation is to optimize the traditional cross-entropy loss together with a regularization term that regulates the perturbation consistency of unlabelled data . Ladder net ( Rasmus et al . ( 2015 ) ) is introduced to minimise the reconstruction loss between the network outputs from a given sample and its perturbation . It is then simplified in ( Laine & Aila ( 2016 ) ) as two temporal modules : Π-Model and Temporal Ensembling . The Temporal Ensembling f g' Random Sampling f g Class-Balanced Sampling ( a ) Initialization procedure . A recognition model is first trained with random sampling . After that the feature embedding is used to train a new classifier with class-balanced sampling . In the diagram , CNN components that are updated during training are highlighted in red . f gU U ’ f g ’ D∪U ’ 𝐿 # $ % & f gD 𝐿 # ' ( Stage 1 Stage 2 Stage 3 ( b ) Diagram of alternate learning . CNN modules in green line is only used in forwarding . Those in red are fine-tuned with the corresponding loss . In Stage 1 , samples from U are forwarded through f and g. U ′ consists of samples from U , and pseudo labels acquired from g. In Stage 2 , f and g′ are trained on the combination of D and U ′ . In Stage 3 , only the classifier g is trained . f is fixed and only used in forwarding . encourages the output of the network from unlabeled data to be similar to its counterpart from previous training epoch . More recently , Mean Teacher ( Tarvainen & Valpola ( 2017 ) ) extends it by assembling along training . Instead of storing previous predictions , they assemble a Teacher model by calculating the moving average of the training network , i.e . the Student . The Teacher is then used to provide the consistency of predictions to the Student . In addition to that , MA-DNN ( Chen et al . ( 2018 ) ) introduces a memory module to maintain the category prototypes and provide regularization for learning with unlabeled data . Label propagation ( Li et al . ( 2018 ) ) is also considered with the help of label graph . More recently , Mixmatch ( Berthelot et al . ( 2019 ) ) and Fixmatch ( Sohn et al . ( 2020 ) ) improve the performance by introducing powerful data augmentations and perturbation consistencies . All the semi-supervised methods above do not separate labeled data during semi-supervised training . In fact , it is beneficial to combine labeled data and unlabeled data in a certain proportion ( Laine & Aila ( 2016 ) ; Tarvainen & Valpola ( 2017 ) ) . However , without further knowledge , we have no insight how to deal with this combination when long-tailed distribution is included . Furthermore , long-tailed learning methods require calibration or re-sampling based on the class distribution . This combination of labeled and unlabeled data makes the distribution unstable . In result , this is not suitable for long-tailed recognition . Recently , Salsa ( Rebuffi et al . ( 2020 ) ) proposes to decouple the supervised learning from semisupervised training . Our method follows the alternate training scheme from it , because it is surprisingly compatible with long-tailed learning . In practice , our method differs from Salsa in the following aspects . First , we adopt class-balanced sampling in supervised learning to deal with the long-tailed distribution . Second , we use supervised learning instead of self-supervised learning as initialization . We find that self-supervised learning results in inferior performance in long-tailed scenario . Third , the re-initialization is not needed . Because our initialization is already from supervised learning , there is not a specific starting point to re-initialize the model . In fact , this enhances the soft constraint between the two stages in ( Rebuffi et al . ( 2020 ) ) . With the models continuously optimized along alternate learning , our method achieves superior performance while maintains the same amount of training epochs as fine-tuning on pseudo labels .

Summary: This paper proposes a long-tailed semi-supervised learning setting where both labeled and unlabeled data from a long-tailed distribution exist. To address this setting, they use an iterative decoupling training method, which is based on Kang et al 2020. The only difference is after initialization using the original decoupling method, pseudo labels are obtained from unlabeled data and the feature extractor is updated with pseudo labels, then the classifier is updated using the new feature representations. Two new benchmarks are provided, and the methods had improved performance over selected baselines.