Semi-supervised Long-tailed Recognition using Alternate Sampling

1 INTRODUCTION . Large-scale datasets , which contain sufficient data in each class , has been a major factor to the success of modern deep learning models for computer vision tasks , such as object recognition . These datasets are usually carefully curated and balanced to have an uniform data distribution over all classes . This balanced data distribution favors model training but could be impractical in many real world applications , where the frequency of samples from different classes can be imbalanced , leading to a long-tailed data distribution . As shown in Figure 1 ( b ) , several highly populated classes take up most of the labeled samples , and some of the classes only have very few samples during training . The long-tailed recognition problem has been widely studied in the literature . One major challenge in this setting ( Liu et al . ( 2019 ) ; Kang et al . ( 2020 ) ; Zhou et al . ( 2020 ) ) to deep learning model training is the tendency of under-fitting in less-populated classes . The root causes of this underfitting are the imbalanced training data distribution as well as the scarcity of data samples in the tail classes . More specifically , with an imbalanced training data distribution , when several head classes take up most of the training samples , tail classes contribute little in the training loss . The model is such that biased towards head classes . Prior works ( Lin et al . ( 2017 ) ; Cao et al . ( 2019 ) ; Kang et al . ( 2020 ) ; Zhou et al . ( 2020 ) ) tried to mitigate the issue by re-sampling the training data to be a balanced distribution or calibrating the sample weights in calculating the loss . However , still the scarcity of tail class data samples limits the intra-class variations and overall recognition accuracy . Methods focusing on few-shot learning have been introduced to address this problem through data augmentation and data synthesis ( Wang et al . ( 2018 ) ; Hariharan & Girshick ( 2017 ) ; Liu et al . ( 2020 ) ) . In this work , we resort to a different path to leverage massive unlabeled real data in training to help improve the long-tailed recognition accuracy . Since data collection is much cheaper and accessible comparing to data annotation , additional unlabeled real data could readily be available in many realworld scenarios . This semi-supervised learning setting has been intensively studied in the literature ( Laine & Aila ( 2016 ) ; Rasmus et al . ( 2015 ) ; Tarvainen & Valpola ( 2017 ) ; Berthelot et al . ( 2019 ) ; Sohn et al . ( 2020 ) ) . However , as shown in Figure 1 ( a ) , when we carefully look at the data distribution of the widely used benchmarks , we observe well-balanced labeled subset and unlabeled subset . As discussed above , the manually curated balanced distribution , can lead to a gap to real-world scenarios . This is especially true in unlabeled data . Without labels , people have no way to balance the data among classes . In this paper , we propose a more realistic and challenging setting , namely semi-supervised longtailed recognition . As shown in Figure 1 ( c ) , we assume a long-tailed data distribution of the overall dataset and both the labeled and unlabeled subsets of training data follow the same underlying longtailed data distribution . This setting generally resembles a realistic data collection and annotation workflow . After collecting the raw data , one has no knowledge of its class distribution before annotation . As it is expensive to annotate the full corpus , a common practice is to randomly sample a subset for annotation under a given labeling budget . When the raw data follows a long-tailed class distribution , we should expect the same in the labeled subset . While this new recognition paradigm shares the challenges in both semi-supervised learning and long-tailed recognition , there is no readily naive solution to it . Methods in long-tailed recognition rely on class labels to achieve balanced training , which are not available in the unlabeled portion in the semi-supervised long-tailed recognition . Prior semi-supervised methods without considering the long-tailed distribution could fail as well . Taking one of the competitive baseline methods for example , ( Yang & Xu ( 2020 ) ) proposed to firstly train a recognition model with the labeled subset to generate pseudo labels for the unlabeled subset , then the model is fine-tuned with the full training dataset . However , when the labeled subset follows a long-tailed distribution , the pseudo labels are much less accurate for tail classes than head classes . As a result , the overall pseudo labels quality could be too bad to leverage ( See Section 4.4 for results in CIFAR-10-SSLT and ImageNet-SSLT ) . To address the semi-supervised long-tailed recognition problem , we present a method designed specifically for this setting . We bring the successful class-balanced sampling strategy and combined it with model decoupling in an alternate learning framework to overcome the difficulty of balancing unlabeled training data . Inspired by ( Kang et al . ( 2020 ) ) , we decouple the recognition model into a feature embedding and a classifier , and train them with random sampling and class-balanced sampling respectively . As we are targeting at a semi-supervised setting , the classifier is only trained on labeled data to get around the difficulty of applying correctly class-balanced sampling on unlabeled data , aligning with the intuition that the classifier needs more robust supervision than the feature embedding . After that , with the proposed alternative learning framework , we improve model by updating the feature embedding and the classifier iteratively . We assign pseudo labels with the up-to-date classifier and observed gradually improved accuracy of pseudo labels over iterations . The pseudo labels are then incorporated in fine-tuning the feature embedding with a regularization term to limit its potential negative impacts . Similar iterative design has been proposed in semi-supervised learning literature ( Laine & Aila ( 2016 ) ; Tarvainen & Valpola ( 2017 ) ) but important implementation details differ . To summarize , in this paper , 1 ) we resort to semi-supervised learning to help improve long-tailed recognition accuracy and identify practical gap of current semi-supervised recognition datasets due to their well-balanced unlabeled subset ; 2 ) we propose a new recognition paradigm named semisupervised long-tailed recognition better resembling real-world data collection and annotation workflow ; 3 ) we propose a new alternative sampling method to address the semi-supervised long-tailed recognition and demonstrate significant improvements on several benchmarks . 2 RELATED WORK . Long-tailed recognition has been recently studied a lot ( Wang et al . ( 2017 ) ; Oh Song et al . ( 2016 ) ; Lin et al . ( 2017 ) ; Zhang et al . ( 2017 ) ; Liu et al . ( 2019 ) ; Wang & Hebert ( 2016 ) ) . Several approaches have been proposed , including metric learning ( Oh Song et al . ( 2016 ) ; Zhang et al . ( 2017 ) ) , loss weighting ( Lin et al . ( 2017 ) ) , and meta-learning ( Wang & Hebert ( 2016 ) ) . Some methods design dedicated loss functions to mitigate the data imbalanced problem . For example , lift loss ( Oh Song et al . ( 2016 ) ) introduces margins between many training samples . Range loss ( Zhang et al . ( 2017 ) ) encourages data from the same class to be close and different classes to be far away in the embedding space . The focal loss ( Lin et al . ( 2017 ) ) dynamically balances weights of positive , hard negative , and easy negative samples . As reported by ( Liu et al . ( 2019 ) ) , when applied to long-tailed recognition , many of these methods improved accuracy of the few-shot group , but at the cost of lower accuracy over the many-shot classes . Other methods , e.g . LDAM-DRW ( Cao et al . ( 2019 ) ) replace cross-entropy loss with LDAM loss . This adds a calibration factor to the original cross-entropy loss . When combined with loss reweighting , it improves the accuracy in all splits in long-tailed recognition . However , it can not be easily generalized to semi-supervised learning . Because both the calibration factor and the loss weight are calculated based on the number of samples of each class . In face recognition and person re-identification , the datasets are mostly with long-tailed distribution . LEAP ( Liu et al . ( 2020 ) ) augmented data samples from tail ( few-shot ) classes by transferring intra-class variations from head ( many-shot ) classes . Instead of data augmentation , we introduce unsupervised data to improve the performance of long-tailed recognition . A recent work ( Yang & Xu ( 2020 ) ) rethinks the value of labels in imbalance learning . As part of the discussion , semi-supervised learning is included . However , only the basic pseudo label solution and simple datasets , such as CIFAR and SVHN , are discussed . More recent works ( Kang et al . ( 2020 ) ; Zhou et al . ( 2020 ) ) with improved long-tailed recognition share the observation that feature embedding and the classifier should be trained with different sampling strategies . In this work , we adopt our method on this observation to learn the feature embedding model with random sampling and train the classifier with class-balanced sampling . This design is further closely compatible with semi-supervised learning under alternate learning . Semi-supervised learning has been extensively discussed in recognition discipline ( Laine & Aila ( 2016 ) ; Rasmus et al . ( 2015 ) ; Tarvainen & Valpola ( 2017 ) ) . One common observation is to optimize the traditional cross-entropy loss together with a regularization term that regulates the perturbation consistency of unlabelled data . Ladder net ( Rasmus et al . ( 2015 ) ) is introduced to minimise the reconstruction loss between the network outputs from a given sample and its perturbation . It is then simplified in ( Laine & Aila ( 2016 ) ) as two temporal modules : Π-Model and Temporal Ensembling . The Temporal Ensembling f g' Random Sampling f g Class-Balanced Sampling ( a ) Initialization procedure . A recognition model is first trained with random sampling . After that the feature embedding is used to train a new classifier with class-balanced sampling . In the diagram , CNN components that are updated during training are highlighted in red . f gU U ’ f g ’ D∪U ’ 𝐿 # $ % & f gD 𝐿 # ' ( Stage 1 Stage 2 Stage 3 ( b ) Diagram of alternate learning . CNN modules in green line is only used in forwarding . Those in red are fine-tuned with the corresponding loss . In Stage 1 , samples from U are forwarded through f and g. U ′ consists of samples from U , and pseudo labels acquired from g. In Stage 2 , f and g′ are trained on the combination of D and U ′ . In Stage 3 , only the classifier g is trained . f is fixed and only used in forwarding . encourages the output of the network from unlabeled data to be similar to its counterpart from previous training epoch . More recently , Mean Teacher ( Tarvainen & Valpola ( 2017 ) ) extends it by assembling along training . Instead of storing previous predictions , they assemble a Teacher model by calculating the moving average of the training network , i.e . the Student . The Teacher is then used to provide the consistency of predictions to the Student . In addition to that , MA-DNN ( Chen et al . ( 2018 ) ) introduces a memory module to maintain the category prototypes and provide regularization for learning with unlabeled data . Label propagation ( Li et al . ( 2018 ) ) is also considered with the help of label graph . More recently , Mixmatch ( Berthelot et al . ( 2019 ) ) and Fixmatch ( Sohn et al . ( 2020 ) ) improve the performance by introducing powerful data augmentations and perturbation consistencies . All the semi-supervised methods above do not separate labeled data during semi-supervised training . In fact , it is beneficial to combine labeled data and unlabeled data in a certain proportion ( Laine & Aila ( 2016 ) ; Tarvainen & Valpola ( 2017 ) ) . However , without further knowledge , we have no insight how to deal with this combination when long-tailed distribution is included . Furthermore , long-tailed learning methods require calibration or re-sampling based on the class distribution . This combination of labeled and unlabeled data makes the distribution unstable . In result , this is not suitable for long-tailed recognition . Recently , Salsa ( Rebuffi et al . ( 2020 ) ) proposes to decouple the supervised learning from semisupervised training . Our method follows the alternate training scheme from it , because it is surprisingly compatible with long-tailed learning . In practice , our method differs from Salsa in the following aspects . First , we adopt class-balanced sampling in supervised learning to deal with the long-tailed distribution . Second , we use supervised learning instead of self-supervised learning as initialization . We find that self-supervised learning results in inferior performance in long-tailed scenario . Third , the re-initialization is not needed . Because our initialization is already from supervised learning , there is not a specific starting point to re-initialize the model . In fact , this enhances the soft constraint between the two stages in ( Rebuffi et al . ( 2020 ) ) . With the models continuously optimized along alternate learning , our method achieves superior performance while maintains the same amount of training epochs as fine-tuning on pseudo labels .

Summary: This paper proposes a new setting--semi-supervised long-tailed recognition. To harness the imbalanced unlabeled data, the authors combined the decoupling in long-tailed recognition and pseudo-labeling in semi-supervised learning, which formulates a three-stage method. Stage 1 generates pseudo-labels with a classifier trained with class-aware resampling. Stage 2 fine-tunes the feature extractor on with pseudo-labeling. Stage 3 trains the classifier with class-aware resampling on top of the refined feature extractor. Experiments indicate that the proposed method outperforms baselines.