Asynchronous Modeling: A Dual-phase Perspective for Long-Tailed Recognition

1 INTRODUCTION . Past years have witnessed huge progress in visual recognition with the successful application of deep convolutional neural networks ( CNNs ) on large-scale datasets , e.g. , ImageNet ILSVRC 2012 ( Russakovsky et al. , 2015 ) , Places ( Zhou et al. , 2017 ) . Such datasets are usually artificially collected and exhibit approximately uniform distribution concerning the number of samples in each class . Real-world datasets , however , are always long-tailed that only a few classes occupy the majority of instances in the dataset ( data-rich ) and most classes have rarely few samples ( data-poor ) ( Reed , 2001 ; Van Horn & Perona , 2017 ) . When modeling such datasets , many standard methods suffer from severe degradation of overall performance . More specifically , the recognition ability on classes with rarely few instances are significantly impaired ( Liu et al. , 2019 ) . One prominent direction is to apply class re-sampling or loss re-weighting to balance the influence of different classes ( Byrd & Lipton , 2019 ; Shu et al. , 2019 ) and another alternative is to conduct transferring ( Wang et al. , 2017 ; Liu et al. , 2019 ) by the assumption that knowledge obtained on the data-rich classes should benefit the recognition of data-poor classes . Recently , more sophisticated models are designed to train the model either base on some new findings ( Zhou et al. , 2020 ; Kang et al. , 2020 ) or combine all available techniques ( Zhu & Yang , 2020 ) . However , the property of longtailed setting makes it remain to be difficult to achieve large gains compared to balanced datasets . In contrast to the aforementioned strategies , we approach the long-tailed recognition problem by analyzing gradient distortion in long-tailed data , attributing to the interaction between gradients generated by data-rich and data-poor classes , i.e. , the direction of overall gradient is shifted to be closer to the gradient on data-rich classes and its norm variance is increased due to the dramatic variation in the gradient generated by data-poor classes . The degenerated performance when comparing with balanced datasets indicates the gradient distortion is negative during model training . Motivated by this , we hypothesize that the combined analysis for gradients generated by data-rich and data-poor classes could be improper in long-tailed data and attempt to disentangle these two gradients . We thus propose the conception of asynchronous modeling and split the original network to promote a dual-phase learning , along with the partition of the given dataset . In phase I , data-rich classes keeps the bulk of the original dataset . It facilitates better local representation learning and more precise classifier boundary determination by eliminating the negative gradient interaction produced by datapoor classes . Based on the model learned in phase I , we involve the rest data to do new boundary exploration in the second phase . While transiting from the first phase to the second , it is hoped to reserve the knowledge learned in the first phase . Specifically , we design an exemplar memory bank and introduce a memory-retentive loss . The memory bank reserves a few most prominent examples from classes in the first phase and collaborates with data in the second phase for classification . Also , the collaborated data , together with the new memory-retentive loss , tries to preserve old knowledge when the model adapts to new classes in the second phase . In the experiments , we evaluate the proposed asynchronous modeling strategy by comparing to typical strategies , which include the re-balancing based methods ( Cao et al. , 2019 ) and transferring based methods ( Liu et al. , 2019 ) . Furthermore , we also consider the latest , more sophisticated works , like BBN ( Zhou et al. , 2020 ) , IEM ( Zhu & Yang , 2020 ) . The comprehensive study and comparison across four commonly used long-tailed benchmarks , including CIFAR100-LT , Places-LT , ImageNetLT and iNaturalist 2018 validate the efficacy of our method . 2 RELATED WORK . Class re-sampling . Most works along with this line can be categorized as over-sampling of tail classes ( Chawla et al. , 2002 ; Han et al. , 2005 ; Byrd & Lipton , 2019 ) or under-sampling over head classes ( Drummond et al. , 2003 ) . While the idea of re-sampling makes the overall distribution more balanced , it may encounter the problem of over-fitting on rare data and the missing of critical information on dominant classes ( Chawla et al. , 2002 ; Cui et al. , 2019 ) , thus hurting the overall generalization . Beyond that , Ouyang et al . ( 2016 ) ; Liu et al . ( 2019 ) also involve a more refined idea of fine-tuning after representation extraction to adjust the final decision boundary . Loss re-weighting . Methods based on loss re-weighting generally allocate larger weights for tail classes to increase their importance ( Lin et al. , 2017 ; Ren et al. , 2018 ; Shu et al. , 2019 ; Cui et al. , 2019 ; Khan et al. , 2017 ; 2019 ; Huang et al. , 2019 ) . However , direct re-weighting method is difficult to be optimized when tackling a large-scale dataset ( Mikolov et al. , 2013 ) . Recently , Cao et al . ( 2019 ) considers the margins of the training set and introduces a label-distribution-aware loss to enlarge the margins of tail classes . Hayat et al . ( 2019 ) proposes the first hybrid loss function to jointly cluster and classify feature vectors in the Euclidean space and to ensure uniformly spaced and equidistant class prototypes . Knowledge transfer . Along this line , methods based on knowledge transfer handle the challenge of imbalanced dataset by transferring the information learned on head classes to assist tail classes . While Wang et al . ( 2017 ) proposes to transfer meta-knowledge from the head in a progressive manner , recent strategies take consideration of intra-class variance ( Yin et al. , 2019 ) , semantic feature ( Liu et al. , 2019 ; Chu et al. , 2020 ) or domain adaptation ( Jamal et al. , 2020 ) . Recently , BBN ( Zhou et al. , 2020 ) and LWS ( Kang et al. , 2020 ) boost the landscape of long-tailed problem based on some insightful findings . The former asserts that prominent class re-balancing methods can impair the representation learning and the latter claims that data imbalance might not be an issue in learning high-quality representations . IEM ( Zhu & Yang , 2020 ) designs a more complex model that tries to concern available techniques , like feature transferring and attention . In this paper , we are motivated by gradient distortion in long-tailed data , which is caused by the gradient interaction between data-rich classes and data-poor classes . We thus propose to split the learning stage into two phases . We demonstrate that this separation allows straightforward approaches to achieve high recognition performance , without introducing extra parameters . 3 OUR METHOD . Let X = { xi , yi } , i ∈ { 1 , ... , n } be the training set , where xi is the training data and yi is its corresponding label . The number of instances in class j is denoted as nj and the total number of training samples is denoted as n = ∑C j=1 nj , where C is the number of classes . Without loss of generality , we assume that the classes are sorted in decreasing order , that is , if i > j , ni ≤ nj . We define the whole network as f ( x ; [ Wr ; Wc ] ) , where f is the implemented deep learning model with parameters Wr for representation learning and parameters Wc for classification , and x is the input . 3.1 GRADIENT DISTORTION IN LONG TAIL . Given a long-tailed dataset , our goal is to achieve better overall performance across all classes . In contrast to previous common heuristics ( e.g. , resampling , reweighting and feature transfer ) , we revisit the problem of long-tailed classification from the perspective of gradient distortion . The overall gradient for updating is modulated by the gradients generated by data-rich classes in the head and data-poor classes in the tail . To state the details , we visualize the associated metrics in the training process of vanilla CIFAR100 and long-tailed CIFAR100 ( CIFAR100-LT ) in Fig . 1 . Specifically , the cosine similarity between the gradients is visualized in Fig . 1 ( a ) ( CIFAR100-LT ) and Fig . 1 ( b ) ( vanilla CIFAR100 ) . Similarly , the norm of each gradient is recorded in Fig . 1 ( c ) ( CIFAR100-LT ) and Fig . 1 ( d ) ( vanilla CIFAR100 ) . The higher similarity between the overall gradient and the datarich gradient indicates that the overall gradient is shifted to the direction of the data-rich gradient . Meanwhile , the norm variance of overall gradient is enlarged due to more dramatic fluctuation of the gradient on data-poor classes . Motivated by the degenerated performance in long-tailed dataset , it is hypothesized that synchronous application of two distinctive gradients could impair the overall performance . 3.2 ASYNCHRONOUS MODELING . Rather than directly regulating the overall gradient as previous methods , we begin with the disentanglement of two gradients and propose a dual-phase asynchronous modeling strategy . The data from data-rich classes is first considered in model training and then the rest classes are involved . Such asynchronous operation not only reduces the potential disturbance between two gradients , but also ensures the benefits of each gradient to be exploited . Mathematically , the original dataset is X with C classes . Suppose C1 classes are considered in phase I , we then write X1 as the set of data from C1 classes . The data in rest C2 classes is denoted as X2 , where C2 = C − C1 . Accordingly , the parameters Wc for C classes in f ( x ; [ Wr , Wc ] ) are truncated as W 1c for C1 classes in the first phase . 3.2.1 LEARNING IN THE FIRST PHASE . In model learning from data X1 , the consideration of gradient on data-poor categories is avoided , which keeps the truncated model f ( x ; [ Wr ; W 1c ] ) to be more concentrated . In optimization , the cross-entropy loss over the classes in X1 is minimized with respect to parameters Wr and W 1c . L1 = − ∑ ( x , y ) ∈X1 y log f ( x ; [ Wr ; W 1 c ] ) . ( 1 ) For further improvement in the training , some balanced sampling strategies could be incorporated in this phase . For example , the progressively-balanced strategy in ( Kang et al. , 2020 ) combines instance-balanced sampling and class-balanced sampling , that is , pj ( t ) = ( 1 − tT ) nj∑C i=1 ni + tT 1 C , where pj ( t ) denotes the sampling probability for class j in training epoch t. It is computed with linear combination for instances-based probability nj∑C i=1 ni and class-based probability 1C . T is the total epoch number . 3.2.2 JOINT PREDICTION IN THE SECOND PHASE . We wish to involve the data in X2 to obtain a complete model across all C classes for overall evaluation . To do so , on the basis of parameters Wr obtained in phase I , we introduce the classifier parameters W 2c for the recognition of new classes in X2 . Similar to phase I , the standard crossentropy loss across all data in X2 is considered . However , considering solely on data X2 tends to forget the knowledge learned in the first phase . To tackle with the obstacle , we thus design a memory bank and memory-retentive loss to realize the seamless connection between two data splits . First , representative samples in X1 are retained in an augmented memory module to enable the joint prediction over all classes . Second , the examples reserved in the memory are combined with X2 , which are collaboratively trained with a unified memory-retentive loss . Exemplar memory bank . In maintaining the knowledge obtained in the first phase , we design an exemplar memory bank that selects only a few most representative samples from classes in X1 . For simplicity , the number of selected samples from each class is set to be equal . We denote the reserved data in the memory bank as M . Ideally , the most representative examples are samples that are closest to the center of each class . However , a precise class center is not always accessible . Thus in practice , the center is progressively estimated by accessing over the entries generated in previous steps to infer new entry in the memory bank . Without loss of generalization , we consider class j in dataset X1 to demonstrate the detailed operation . We first compute the average feature from all examples in class j in original training set X1 to serve as a class prototype cj , which is thus the initial estimation of class center . We return the instance which is closest to cj in X1 and set it as the first selected sample for the memory bank , m1 = arg max xi∈X1 { s ( cj , X1 ) } , ( 2 ) where s is a vector space similarity metric , like cosine similarity . m1 is used to denote the returned sample xi . Before selecting the rest instances from X1 , we need to update the estimated center cj . Without loss of generality , suppose we have selected k samples from X1 and denote the feature map of data in memory bank as Mj = [ m1 , m2 , .. , mk ] ∈ Rk×d , where d is the dimension of each feature map . Each sample in Mj serves as a guided hypothesis and its correlation with cj can then be computed for the new state zk+1 , that is , pi = exp ( s ( cj , mi ) ) ∑ i exp ( s ( cj , mi ) ) , ( 3 ) zk+1 = k∑ i=1 pimi = pMj , ( 4 ) where s is the same similarity metric as above . pi is computed by the distances between the selected data and the center prototype and it serves as weights to update state zk+1 . zk+1 is the weighted average of all feature maps in Mj . New samples can then be returned for k + 1 step by performing mk+1 = arg max xi∈X1 { s ( cj + ∆ , X1 ) } , ( 5 ) where ∆ is the residual between cj and zk+1 , i.e. , ∆ = cj − zk+1 . mk+1 is used to denote the returned sample xi . Memory-retentive loss . Based on the memory bank , we obtain a combined data setD by extending X2 with examples in the memory bank M , i.e. , D = M ⋃ X2 . Similarly , the joint prediction with a cross-entropy loss is first considered . When the model is adapted to fit data X2 , the knowledge learned in X1 tends to be forgotten . We thus introduce a new memory-retentive loss LGdis based on graph matching , which provides a strong constraint in memorizing previous knowledge . Specifically , the feature map of each data in the training set D is a node in a graph . Based on the model learned in the first stage and the new model to be trained in the second phase , two graphs Gold and Gnew can thus be constructed . That is , we not only consider feature similarity of a single example on the old model and the new model , but also compute the global matching similarity on the whole training set D. Suppose the feature map of one node in Gold is zi and in Gold is ẑi , thus the similarity between old graph Gold and new graph Gnew is measured by computing the change between any node zi in Gold and any node zj in Gnew , that is , aji = exp ( s ( zi , ẑj ) ) ∑ j′ exp ( s ( zi , ẑj′ ) ) , zi ∈ Gold , ẑj , ẑj′ ∈ Gnew , ( 6 ) µi = ∑ j aji‖zi − ẑj‖ , zi ∈ Gold , ẑj ∈ Gnew , ( 7 ) LGdis = ∑ i µi , zi ∈ Gold , ( 8 ) where s is the vector similarity metric . aji represents the distance between node i in graph Gold and node j in graph Gnew , µi thus ntuitively measures the difference between zj and its closest neighbor in graph Gnew . Consider all nodes in graph Gold together , we obtain the memory-retentive loss LGdis , which describe graph similarity between two graphs . Overall loss . Combined the above analysis together , the overall loss in phase II is thus as below : L = 1 |D| ∑ x∈D ( Lcls ( x ) + Lintra ( x ) ) + λL G dis , ( 9 ) where the first term is for classification and the second is the designed loss which constrains the knowledge in old model through graph matching , λ is a hyperparameter to balance the two terms . Notice that , apart from the standard cross-entropy loss Lcls ( x ) for input x in the first term , we also consider an intra-classification loss Lintra ( x ) to avoid memory data in M being dominated by new classes in X2 . When we consider cosine linear classifier , one of the instantiations could be Lintra ( x ) = ∑K k=1 max ( 0 , m− 〈w̄ , z̄ ( x ) 〉+ 〈w̄k , z̄ ( x ) 〉 , in which , w̄ is the ground-truth class embedding and w̄k denotes the other class embedding , z̄ ( x ) is the normalized feature map of x , m is a margin value . 〈w̄ , z̄ ( x ) 〉 denotes a positive score between w̄ and z̄ ( x ) , while 〈w̄k , z̄ ( x ) 〉 denotes the negative score between w̄k and z̄ ( x ) . Lintra optimizes the network to maintain a margin of m between the positive score and the highest negative score . Finally , for a comprehensive overview of the asynchronous modeling structure , we can find it in Algorithm 1 in Appendix B .

Summary: This paper proposes an interesting view to analyze the long-tailed problem. It states that the gradients are dominated by the head classes so that the tail classes perform poorly. From this observation, the authors propose a dual-phase approach that first train $W_r, W_c^1$ with only head-class data, and extend to train $W_r, W_c$ with tail-class data and the constructed exemplar memory bank for head classes with a newly proposed memory retentive loss.