Asynchronous Modeling: A Dual-phase Perspective for Long-Tailed Recognition

1 INTRODUCTION . Past years have witnessed huge progress in visual recognition with the successful application of deep convolutional neural networks ( CNNs ) on large-scale datasets , e.g. , ImageNet ILSVRC 2012 ( Russakovsky et al. , 2015 ) , Places ( Zhou et al. , 2017 ) . Such datasets are usually artificially collected and exhibit approximately uniform distribution concerning the number of samples in each class . Real-world datasets , however , are always long-tailed that only a few classes occupy the majority of instances in the dataset ( data-rich ) and most classes have rarely few samples ( data-poor ) ( Reed , 2001 ; Van Horn & Perona , 2017 ) . When modeling such datasets , many standard methods suffer from severe degradation of overall performance . More specifically , the recognition ability on classes with rarely few instances are significantly impaired ( Liu et al. , 2019 ) . One prominent direction is to apply class re-sampling or loss re-weighting to balance the influence of different classes ( Byrd & Lipton , 2019 ; Shu et al. , 2019 ) and another alternative is to conduct transferring ( Wang et al. , 2017 ; Liu et al. , 2019 ) by the assumption that knowledge obtained on the data-rich classes should benefit the recognition of data-poor classes . Recently , more sophisticated models are designed to train the model either base on some new findings ( Zhou et al. , 2020 ; Kang et al. , 2020 ) or combine all available techniques ( Zhu & Yang , 2020 ) . However , the property of longtailed setting makes it remain to be difficult to achieve large gains compared to balanced datasets . In contrast to the aforementioned strategies , we approach the long-tailed recognition problem by analyzing gradient distortion in long-tailed data , attributing to the interaction between gradients generated by data-rich and data-poor classes , i.e. , the direction of overall gradient is shifted to be closer to the gradient on data-rich classes and its norm variance is increased due to the dramatic variation in the gradient generated by data-poor classes . The degenerated performance when comparing with balanced datasets indicates the gradient distortion is negative during model training . Motivated by this , we hypothesize that the combined analysis for gradients generated by data-rich and data-poor classes could be improper in long-tailed data and attempt to disentangle these two gradients . We thus propose the conception of asynchronous modeling and split the original network to promote a dual-phase learning , along with the partition of the given dataset . In phase I , data-rich classes keeps the bulk of the original dataset . It facilitates better local representation learning and more precise classifier boundary determination by eliminating the negative gradient interaction produced by datapoor classes . Based on the model learned in phase I , we involve the rest data to do new boundary exploration in the second phase . While transiting from the first phase to the second , it is hoped to reserve the knowledge learned in the first phase . Specifically , we design an exemplar memory bank and introduce a memory-retentive loss . The memory bank reserves a few most prominent examples from classes in the first phase and collaborates with data in the second phase for classification . Also , the collaborated data , together with the new memory-retentive loss , tries to preserve old knowledge when the model adapts to new classes in the second phase . In the experiments , we evaluate the proposed asynchronous modeling strategy by comparing to typical strategies , which include the re-balancing based methods ( Cao et al. , 2019 ) and transferring based methods ( Liu et al. , 2019 ) . Furthermore , we also consider the latest , more sophisticated works , like BBN ( Zhou et al. , 2020 ) , IEM ( Zhu & Yang , 2020 ) . The comprehensive study and comparison across four commonly used long-tailed benchmarks , including CIFAR100-LT , Places-LT , ImageNetLT and iNaturalist 2018 validate the efficacy of our method . 2 RELATED WORK . Class re-sampling . Most works along with this line can be categorized as over-sampling of tail classes ( Chawla et al. , 2002 ; Han et al. , 2005 ; Byrd & Lipton , 2019 ) or under-sampling over head classes ( Drummond et al. , 2003 ) . While the idea of re-sampling makes the overall distribution more balanced , it may encounter the problem of over-fitting on rare data and the missing of critical information on dominant classes ( Chawla et al. , 2002 ; Cui et al. , 2019 ) , thus hurting the overall generalization . Beyond that , Ouyang et al . ( 2016 ) ; Liu et al . ( 2019 ) also involve a more refined idea of fine-tuning after representation extraction to adjust the final decision boundary . Loss re-weighting . Methods based on loss re-weighting generally allocate larger weights for tail classes to increase their importance ( Lin et al. , 2017 ; Ren et al. , 2018 ; Shu et al. , 2019 ; Cui et al. , 2019 ; Khan et al. , 2017 ; 2019 ; Huang et al. , 2019 ) . However , direct re-weighting method is difficult to be optimized when tackling a large-scale dataset ( Mikolov et al. , 2013 ) . Recently , Cao et al . ( 2019 ) considers the margins of the training set and introduces a label-distribution-aware loss to enlarge the margins of tail classes . Hayat et al . ( 2019 ) proposes the first hybrid loss function to jointly cluster and classify feature vectors in the Euclidean space and to ensure uniformly spaced and equidistant class prototypes . Knowledge transfer . Along this line , methods based on knowledge transfer handle the challenge of imbalanced dataset by transferring the information learned on head classes to assist tail classes . While Wang et al . ( 2017 ) proposes to transfer meta-knowledge from the head in a progressive manner , recent strategies take consideration of intra-class variance ( Yin et al. , 2019 ) , semantic feature ( Liu et al. , 2019 ; Chu et al. , 2020 ) or domain adaptation ( Jamal et al. , 2020 ) . Recently , BBN ( Zhou et al. , 2020 ) and LWS ( Kang et al. , 2020 ) boost the landscape of long-tailed problem based on some insightful findings . The former asserts that prominent class re-balancing methods can impair the representation learning and the latter claims that data imbalance might not be an issue in learning high-quality representations . IEM ( Zhu & Yang , 2020 ) designs a more complex model that tries to concern available techniques , like feature transferring and attention . In this paper , we are motivated by gradient distortion in long-tailed data , which is caused by the gradient interaction between data-rich classes and data-poor classes . We thus propose to split the learning stage into two phases . We demonstrate that this separation allows straightforward approaches to achieve high recognition performance , without introducing extra parameters . 3 OUR METHOD . Let X = { xi , yi } , i ∈ { 1 , ... , n } be the training set , where xi is the training data and yi is its corresponding label . The number of instances in class j is denoted as nj and the total number of training samples is denoted as n = ∑C j=1 nj , where C is the number of classes . Without loss of generality , we assume that the classes are sorted in decreasing order , that is , if i > j , ni ≤ nj . We define the whole network as f ( x ; [ Wr ; Wc ] ) , where f is the implemented deep learning model with parameters Wr for representation learning and parameters Wc for classification , and x is the input . 3.1 GRADIENT DISTORTION IN LONG TAIL . Given a long-tailed dataset , our goal is to achieve better overall performance across all classes . In contrast to previous common heuristics ( e.g. , resampling , reweighting and feature transfer ) , we revisit the problem of long-tailed classification from the perspective of gradient distortion . The overall gradient for updating is modulated by the gradients generated by data-rich classes in the head and data-poor classes in the tail . To state the details , we visualize the associated metrics in the training process of vanilla CIFAR100 and long-tailed CIFAR100 ( CIFAR100-LT ) in Fig . 1 . Specifically , the cosine similarity between the gradients is visualized in Fig . 1 ( a ) ( CIFAR100-LT ) and Fig . 1 ( b ) ( vanilla CIFAR100 ) . Similarly , the norm of each gradient is recorded in Fig . 1 ( c ) ( CIFAR100-LT ) and Fig . 1 ( d ) ( vanilla CIFAR100 ) . The higher similarity between the overall gradient and the datarich gradient indicates that the overall gradient is shifted to the direction of the data-rich gradient . Meanwhile , the norm variance of overall gradient is enlarged due to more dramatic fluctuation of the gradient on data-poor classes . Motivated by the degenerated performance in long-tailed dataset , it is hypothesized that synchronous application of two distinctive gradients could impair the overall performance . 3.2 ASYNCHRONOUS MODELING . Rather than directly regulating the overall gradient as previous methods , we begin with the disentanglement of two gradients and propose a dual-phase asynchronous modeling strategy . The data from data-rich classes is first considered in model training and then the rest classes are involved . Such asynchronous operation not only reduces the potential disturbance between two gradients , but also ensures the benefits of each gradient to be exploited . Mathematically , the original dataset is X with C classes . Suppose C1 classes are considered in phase I , we then write X1 as the set of data from C1 classes . The data in rest C2 classes is denoted as X2 , where C2 = C − C1 . Accordingly , the parameters Wc for C classes in f ( x ; [ Wr , Wc ] ) are truncated as W 1c for C1 classes in the first phase . 3.2.1 LEARNING IN THE FIRST PHASE . In model learning from data X1 , the consideration of gradient on data-poor categories is avoided , which keeps the truncated model f ( x ; [ Wr ; W 1c ] ) to be more concentrated . In optimization , the cross-entropy loss over the classes in X1 is minimized with respect to parameters Wr and W 1c . L1 = − ∑ ( x , y ) ∈X1 y log f ( x ; [ Wr ; W 1 c ] ) . ( 1 ) For further improvement in the training , some balanced sampling strategies could be incorporated in this phase . For example , the progressively-balanced strategy in ( Kang et al. , 2020 ) combines instance-balanced sampling and class-balanced sampling , that is , pj ( t ) = ( 1 − tT ) nj∑C i=1 ni + tT 1 C , where pj ( t ) denotes the sampling probability for class j in training epoch t. It is computed with linear combination for instances-based probability nj∑C i=1 ni and class-based probability 1C . T is the total epoch number . 3.2.2 JOINT PREDICTION IN THE SECOND PHASE . We wish to involve the data in X2 to obtain a complete model across all C classes for overall evaluation . To do so , on the basis of parameters Wr obtained in phase I , we introduce the classifier parameters W 2c for the recognition of new classes in X2 . Similar to phase I , the standard crossentropy loss across all data in X2 is considered . However , considering solely on data X2 tends to forget the knowledge learned in the first phase . To tackle with the obstacle , we thus design a memory bank and memory-retentive loss to realize the seamless connection between two data splits . First , representative samples in X1 are retained in an augmented memory module to enable the joint prediction over all classes . Second , the examples reserved in the memory are combined with X2 , which are collaboratively trained with a unified memory-retentive loss . Exemplar memory bank . In maintaining the knowledge obtained in the first phase , we design an exemplar memory bank that selects only a few most representative samples from classes in X1 . For simplicity , the number of selected samples from each class is set to be equal . We denote the reserved data in the memory bank as M . Ideally , the most representative examples are samples that are closest to the center of each class . However , a precise class center is not always accessible . Thus in practice , the center is progressively estimated by accessing over the entries generated in previous steps to infer new entry in the memory bank . Without loss of generalization , we consider class j in dataset X1 to demonstrate the detailed operation . We first compute the average feature from all examples in class j in original training set X1 to serve as a class prototype cj , which is thus the initial estimation of class center . We return the instance which is closest to cj in X1 and set it as the first selected sample for the memory bank , m1 = arg max xi∈X1 { s ( cj , X1 ) } , ( 2 ) where s is a vector space similarity metric , like cosine similarity . m1 is used to denote the returned sample xi . Before selecting the rest instances from X1 , we need to update the estimated center cj . Without loss of generality , suppose we have selected k samples from X1 and denote the feature map of data in memory bank as Mj = [ m1 , m2 , .. , mk ] ∈ Rk×d , where d is the dimension of each feature map . Each sample in Mj serves as a guided hypothesis and its correlation with cj can then be computed for the new state zk+1 , that is , pi = exp ( s ( cj , mi ) ) ∑ i exp ( s ( cj , mi ) ) , ( 3 ) zk+1 = k∑ i=1 pimi = pMj , ( 4 ) where s is the same similarity metric as above . pi is computed by the distances between the selected data and the center prototype and it serves as weights to update state zk+1 . zk+1 is the weighted average of all feature maps in Mj . New samples can then be returned for k + 1 step by performing mk+1 = arg max xi∈X1 { s ( cj + ∆ , X1 ) } , ( 5 ) where ∆ is the residual between cj and zk+1 , i.e. , ∆ = cj − zk+1 . mk+1 is used to denote the returned sample xi . Memory-retentive loss . Based on the memory bank , we obtain a combined data setD by extending X2 with examples in the memory bank M , i.e. , D = M ⋃ X2 . Similarly , the joint prediction with a cross-entropy loss is first considered . When the model is adapted to fit data X2 , the knowledge learned in X1 tends to be forgotten . We thus introduce a new memory-retentive loss LGdis based on graph matching , which provides a strong constraint in memorizing previous knowledge . Specifically , the feature map of each data in the training set D is a node in a graph . Based on the model learned in the first stage and the new model to be trained in the second phase , two graphs Gold and Gnew can thus be constructed . That is , we not only consider feature similarity of a single example on the old model and the new model , but also compute the global matching similarity on the whole training set D. Suppose the feature map of one node in Gold is zi and in Gold is ẑi , thus the similarity between old graph Gold and new graph Gnew is measured by computing the change between any node zi in Gold and any node zj in Gnew , that is , aji = exp ( s ( zi , ẑj ) ) ∑ j′ exp ( s ( zi , ẑj′ ) ) , zi ∈ Gold , ẑj , ẑj′ ∈ Gnew , ( 6 ) µi = ∑ j aji‖zi − ẑj‖ , zi ∈ Gold , ẑj ∈ Gnew , ( 7 ) LGdis = ∑ i µi , zi ∈ Gold , ( 8 ) where s is the vector similarity metric . aji represents the distance between node i in graph Gold and node j in graph Gnew , µi thus ntuitively measures the difference between zj and its closest neighbor in graph Gnew . Consider all nodes in graph Gold together , we obtain the memory-retentive loss LGdis , which describe graph similarity between two graphs . Overall loss . Combined the above analysis together , the overall loss in phase II is thus as below : L = 1 |D| ∑ x∈D ( Lcls ( x ) + Lintra ( x ) ) + λL G dis , ( 9 ) where the first term is for classification and the second is the designed loss which constrains the knowledge in old model through graph matching , λ is a hyperparameter to balance the two terms . Notice that , apart from the standard cross-entropy loss Lcls ( x ) for input x in the first term , we also consider an intra-classification loss Lintra ( x ) to avoid memory data in M being dominated by new classes in X2 . When we consider cosine linear classifier , one of the instantiations could be Lintra ( x ) = ∑K k=1 max ( 0 , m− 〈w̄ , z̄ ( x ) 〉+ 〈w̄k , z̄ ( x ) 〉 , in which , w̄ is the ground-truth class embedding and w̄k denotes the other class embedding , z̄ ( x ) is the normalized feature map of x , m is a margin value . 〈w̄ , z̄ ( x ) 〉 denotes a positive score between w̄ and z̄ ( x ) , while 〈w̄k , z̄ ( x ) 〉 denotes the negative score between w̄k and z̄ ( x ) . Lintra optimizes the network to maintain a margin of m between the positive score and the highest negative score . Finally , for a comprehensive overview of the asynchronous modeling structure , we can find it in Algorithm 1 in Appendix B .

Summary: This paper works on long-tailed classification. The authors conducted an analysis and claimed that the difference of gradients computed on the head and tail classes plays an important role in the performance drop. The authors then proposed a two-stage approach to first train on the head classes and then train on the tail classes in an incremental learning fashion. The proposed algorithm achieved better performance than existing methods on benchmark datasets.