Year: 2020

Abstract:
This work explores deep learning based classification model on real-world datasets with a long-tailed distribution. Most of previous works deal with the long-tailed classification problem by re-balancing the overall distribution within the whole dataset or directly transferring knowledge from data-rich classes to data-poor ones. In this work, we consider the gradient distortion in long-tailed classification when the gradient on data-rich classes and data-poor ones are incorporated simultaneously, i.e., shifted gradient direction towards data-rich classes as well as the enlarged variance by the gradient fluctuation on data-poor classes. Motivated by such phenomenon, we propose to disentangle the distinctive effects of data-rich and data-poor gradient and asynchronously train a model via a dualphase learning process. The first phase only concerns the data-rich classes. In the second phase, besides the standard classification upon data-poor classes, we propose an exemplar memory bank to reserve representative examples and a memoryretentive loss via graph matching to retain the relation between two phases. The extensive experimental results on four commonly used long-tailed benchmarks including CIFAR100-LT, Places-LT, ImageNet-LT and iNaturalist 2018 highlight the excellent performance of our proposed method.

heading: 1 INTRODUCTION
text: Past years have witnessed huge progress in visual recognition with the successful application of deep convolutional neural networks (CNNs) on large-scale datasets, e.g., ImageNet ILSVRC 2012 (Russakovsky et al., 2015), Places (Zhou et al., 2017). Such datasets are usually artificially collected and exhibit approximately uniform distribution concerning the number of samples in each class. Real-world datasets, however, are always long-tailed that only a few classes occupy the majority of instances in the dataset (data-rich) and most classes have rarely few samples (data-poor) (Reed, 2001; Van Horn & Perona, 2017). When modeling such datasets, many standard methods suffer from severe degradation of overall performance. More specifically, the recognition ability on classes with rarely few instances are significantly impaired (Liu et al., 2019).
One prominent direction is to apply class re-sampling or loss re-weighting to balance the influence of different classes (Byrd & Lipton, 2019; Shu et al., 2019) and another alternative is to conduct transferring (Wang et al., 2017; Liu et al., 2019) by the assumption that knowledge obtained on the data-rich classes should benefit the recognition of data-poor classes. Recently, more sophisticated models are designed to train the model either base on some new findings (Zhou et al., 2020; Kang et al., 2020) or combine all available techniques (Zhu & Yang, 2020). However, the property of longtailed setting makes it remain to be difficult to achieve large gains compared to balanced datasets.
In contrast to the aforementioned strategies, we approach the long-tailed recognition problem by analyzing gradient distortion in long-tailed data, attributing to the interaction between gradients generated by data-rich and data-poor classes, i.e., the direction of overall gradient is shifted to be closer to the gradient on data-rich classes and its norm variance is increased due to the dramatic variation in the gradient generated by data-poor classes. The degenerated performance when comparing with balanced datasets indicates the gradient distortion is negative during model training. Motivated by this, we hypothesize that the combined analysis for gradients generated by data-rich and data-poor classes could be improper in long-tailed data and attempt to disentangle these two gradients. We thus propose the conception of asynchronous modeling and split the original network to promote a
dual-phase learning, along with the partition of the given dataset. In phase I, data-rich classes keeps the bulk of the original dataset. It facilitates better local representation learning and more precise classifier boundary determination by eliminating the negative gradient interaction produced by datapoor classes. Based on the model learned in phase I, we involve the rest data to do new boundary exploration in the second phase.
While transiting from the first phase to the second, it is hoped to reserve the knowledge learned in the first phase. Specifically, we design an exemplar memory bank and introduce a memory-retentive loss. The memory bank reserves a few most prominent examples from classes in the first phase and collaborates with data in the second phase for classification. Also, the collaborated data, together with the new memory-retentive loss, tries to preserve old knowledge when the model adapts to new classes in the second phase.
In the experiments, we evaluate the proposed asynchronous modeling strategy by comparing to typical strategies, which include the re-balancing based methods (Cao et al., 2019) and transferring based methods (Liu et al., 2019). Furthermore, we also consider the latest, more sophisticated works, like BBN (Zhou et al., 2020), IEM (Zhu & Yang, 2020). The comprehensive study and comparison across four commonly used long-tailed benchmarks, including CIFAR100-LT, Places-LT, ImageNetLT and iNaturalist 2018 validate the efficacy of our method.

heading: 2 RELATED WORK
text: Class re-sampling. Most works along with this line can be categorized as over-sampling of tail classes (Chawla et al., 2002; Han et al., 2005; Byrd & Lipton, 2019) or under-sampling over head classes (Drummond et al., 2003). While the idea of re-sampling makes the overall distribution more balanced, it may encounter the problem of over-fitting on rare data and the missing of critical information on dominant classes (Chawla et al., 2002; Cui et al., 2019), thus hurting the overall generalization. Beyond that, Ouyang et al. (2016); Liu et al. (2019) also involve a more refined idea of fine-tuning after representation extraction to adjust the final decision boundary.
Loss re-weighting. Methods based on loss re-weighting generally allocate larger weights for tail classes to increase their importance (Lin et al., 2017; Ren et al., 2018; Shu et al., 2019; Cui et al., 2019; Khan et al., 2017; 2019; Huang et al., 2019). However, direct re-weighting method is difficult to be optimized when tackling a large-scale dataset (Mikolov et al., 2013). Recently, Cao et al. (2019) considers the margins of the training set and introduces a label-distribution-aware loss to enlarge the margins of tail classes. Hayat et al. (2019) proposes the first hybrid loss function to jointly cluster and classify feature vectors in the Euclidean space and to ensure uniformly spaced and equidistant class prototypes.
Knowledge transfer. Along this line, methods based on knowledge transfer handle the challenge of imbalanced dataset by transferring the information learned on head classes to assist tail classes. While Wang et al. (2017) proposes to transfer meta-knowledge from the head in a progressive manner, recent strategies take consideration of intra-class variance (Yin et al., 2019), semantic feature (Liu et al., 2019; Chu et al., 2020) or domain adaptation (Jamal et al., 2020).
Recently, BBN (Zhou et al., 2020) and LWS (Kang et al., 2020) boost the landscape of long-tailed problem based on some insightful findings. The former asserts that prominent class re-balancing methods can impair the representation learning and the latter claims that data imbalance might not be an issue in learning high-quality representations. IEM (Zhu & Yang, 2020) designs a more complex model that tries to concern available techniques, like feature transferring and attention. In this paper, we are motivated by gradient distortion in long-tailed data, which is caused by the gradient interaction between data-rich classes and data-poor classes. We thus propose to split the learning stage into two phases. We demonstrate that this separation allows straightforward approaches to achieve high recognition performance, without introducing extra parameters.

heading: 3 OUR METHOD
text: Let X = {xi, yi}, i ∈ {1, ..., n} be the training set, where xi is the training data and yi is its corresponding label. The number of instances in class j is denoted as nj and the total number of
training samples is denoted as n = ∑C
j=1 nj , where C is the number of classes. Without loss of generality, we assume that the classes are sorted in decreasing order, that is, if i > j, ni ≤ nj . We define the whole network as f(x; [Wr;Wc]), where f is the implemented deep learning model with parameters Wr for representation learning and parameters Wc for classification, and x is the input.

heading: 3.1 GRADIENT DISTORTION IN LONG TAIL
text: Given a long-tailed dataset, our goal is to achieve better overall performance across all classes. In contrast to previous common heuristics (e.g., resampling, reweighting and feature transfer), we revisit the problem of long-tailed classification from the perspective of gradient distortion. The overall gradient for updating is modulated by the gradients generated by data-rich classes in the head and data-poor classes in the tail. To state the details, we visualize the associated metrics in the training process of vanilla CIFAR100 and long-tailed CIFAR100 (CIFAR100-LT) in Fig. 1. Specifically, the cosine similarity between the gradients is visualized in Fig. 1(a) (CIFAR100-LT) and Fig. 1(b) (vanilla CIFAR100). Similarly, the norm of each gradient is recorded in Fig. 1(c) (CIFAR100-LT) and Fig. 1(d) (vanilla CIFAR100). The higher similarity between the overall gradient and the datarich gradient indicates that the overall gradient is shifted to the direction of the data-rich gradient. Meanwhile, the norm variance of overall gradient is enlarged due to more dramatic fluctuation of the gradient on data-poor classes. Motivated by the degenerated performance in long-tailed dataset, it is hypothesized that synchronous application of two distinctive gradients could impair the overall performance.

heading: 3.2 ASYNCHRONOUS MODELING
text: Rather than directly regulating the overall gradient as previous methods, we begin with the disentanglement of two gradients and propose a dual-phase asynchronous modeling strategy. The data from data-rich classes is first considered in model training and then the rest classes are involved. Such asynchronous operation not only reduces the potential disturbance between two gradients, but also ensures the benefits of each gradient to be exploited. Mathematically, the original dataset is X with C classes. Suppose C1 classes are considered in phase I, we then write X1 as the set of data from C1 classes. The data in rest C2 classes is denoted as X2, where C2 = C − C1. Accordingly, the parameters Wc for C classes in f(x; [Wr,Wc]) are truncated as W 1c for C1 classes in the first phase.

heading: 3.2.1 LEARNING IN THE FIRST PHASE
text: In model learning from data X1, the consideration of gradient on data-poor categories is avoided, which keeps the truncated model f(x; [Wr;W 1c ]) to be more concentrated. In optimization, the cross-entropy loss over the classes in X1 is minimized with respect to parameters Wr and W 1c .
L1 = − ∑
(x,y)∈X1
y log f(x; [Wr;W 1 c ]). (1)
For further improvement in the training, some balanced sampling strategies could be incorporated in this phase. For example, the progressively-balanced strategy in (Kang et al., 2020) combines instance-balanced sampling and class-balanced sampling, that is, pj(t) = (1 − tT ) nj∑C i=1 ni + tT 1 C , where pj(t) denotes the sampling probability for class j in training epoch t. It is computed with linear combination for instances-based probability nj∑C
i=1 ni and class-based probability 1C . T is the
total epoch number.

heading: 3.2.2 JOINT PREDICTION IN THE SECOND PHASE
text: We wish to involve the data in X2 to obtain a complete model across all C classes for overall evaluation. To do so, on the basis of parameters Wr obtained in phase I, we introduce the classifier parameters W 2c for the recognition of new classes in X2. Similar to phase I, the standard crossentropy loss across all data in X2 is considered. However, considering solely on data X2 tends to forget the knowledge learned in the first phase. To tackle with the obstacle, we thus design a memory bank and memory-retentive loss to realize the seamless connection between two data splits. First, representative samples in X1 are retained in an augmented memory module to enable the joint prediction over all classes. Second, the examples reserved in the memory are combined with X2, which are collaboratively trained with a unified memory-retentive loss.
Exemplar memory bank. In maintaining the knowledge obtained in the first phase, we design an exemplar memory bank that selects only a few most representative samples from classes in X1. For simplicity, the number of selected samples from each class is set to be equal. We denote the reserved data in the memory bank as M . Ideally, the most representative examples are samples that are closest to the center of each class. However, a precise class center is not always accessible. Thus in practice, the center is progressively estimated by accessing over the entries generated in previous steps to infer new entry in the memory bank.
Without loss of generalization, we consider class j in dataset X1 to demonstrate the detailed operation. We first compute the average feature from all examples in class j in original training set X1 to serve as a class prototype cj , which is thus the initial estimation of class center. We return the instance which is closest to cj in X1 and set it as the first selected sample for the memory bank,
m1 = arg max xi∈X1
{s(cj , X1)}, (2)
where s is a vector space similarity metric, like cosine similarity. m1 is used to denote the returned sample xi. Before selecting the rest instances from X1, we need to update the estimated center cj . Without loss of generality, suppose we have selected k samples from X1 and denote the feature map of data in memory bank as Mj = [m1,m2, ..,mk] ∈ Rk×d, where d is the dimension of each feature map. Each sample in Mj serves as a guided hypothesis and its correlation with cj can then be computed for the new state zk+1, that is,
pi = exp(s(cj ,mi))∑ i exp(s(cj ,mi)) , (3)
zk+1 = k∑ i=1 pimi = pMj , (4)
where s is the same similarity metric as above. pi is computed by the distances between the selected data and the center prototype and it serves as weights to update state zk+1. zk+1 is the weighted average of all feature maps in Mj . New samples can then be returned for k + 1 step by performing
mk+1 = arg max xi∈X1
{s(cj + ∆, X1)}, (5)
where ∆ is the residual between cj and zk+1, i.e., ∆ = cj − zk+1. mk+1 is used to denote the returned sample xi.
Memory-retentive loss. Based on the memory bank, we obtain a combined data setD by extending X2 with examples in the memory bank M , i.e., D = M ⋃ X2. Similarly, the joint prediction with a cross-entropy loss is first considered. When the model is adapted to fit data X2, the knowledge learned in X1 tends to be forgotten. We thus introduce a new memory-retentive loss LGdis based on
graph matching, which provides a strong constraint in memorizing previous knowledge. Specifically, the feature map of each data in the training set D is a node in a graph. Based on the model learned in the first stage and the new model to be trained in the second phase, two graphs Gold and Gnew can thus be constructed. That is, we not only consider feature similarity of a single example on the old model and the new model, but also compute the global matching similarity on the whole training set D. Suppose the feature map of one node in Gold is zi and in Gold is ẑi, thus the similarity between old graph Gold and new graph Gnew is measured by computing the change between any node zi in Gold and any node zj in Gnew, that is,
aji = exp(s(zi, ẑj))∑ j′ exp(s(zi, ẑj′)) , zi ∈ Gold, ẑj , ẑj′ ∈ Gnew, (6)
µi = ∑ j aji‖zi − ẑj‖, zi ∈ Gold, ẑj ∈ Gnew, (7)
LGdis = ∑ i µi, zi ∈ Gold, (8)
where s is the vector similarity metric. aji represents the distance between node i in graph Gold and node j in graph Gnew, µi thus ntuitively measures the difference between zj and its closest neighbor in graph Gnew. Consider all nodes in graph Gold together, we obtain the memory-retentive loss LGdis, which describe graph similarity between two graphs.
Overall loss. Combined the above analysis together, the overall loss in phase II is thus as below:
L = 1 |D| ∑ x∈D (Lcls(x) + Lintra(x)) + λL G dis, (9)
where the first term is for classification and the second is the designed loss which constrains the knowledge in old model through graph matching, λ is a hyperparameter to balance the two terms. Notice that, apart from the standard cross-entropy loss Lcls(x) for input x in the first term, we also consider an intra-classification loss Lintra(x) to avoid memory data in M being dominated by new classes in X2. When we consider cosine linear classifier, one of the instantiations could be Lintra(x) = ∑K k=1 max(0,m− 〈w̄, z̄(x)〉+ 〈w̄k, z̄(x)〉, in which, w̄ is the ground-truth class embedding and w̄k denotes the other class embedding, z̄(x) is the normalized feature map of x,m is a margin value. 〈w̄, z̄(x)〉 denotes a positive score between w̄ and z̄(x), while 〈w̄k, z̄(x)〉 denotes the negative score between w̄k and z̄(x). Lintra optimizes the network to maintain a margin of m between the positive score and the highest negative score. Finally, for a comprehensive overview of the asynchronous modeling structure, we can find it in Algorithm 1 in Appendix B.

heading: 4 EXPERIMENTS
text: Datasets. We perform extensive experiments on four long-tailed datasets, including CIFAR100LT (Cao et al., 2019), Places-LT (Liu et al., 2019), ImageNet-LT (Liu et al., 2019), and iNaturalist 2018 (iNatrualist, 2018). CIFAR100-LT is created with three different imbalance factors 50, 100, 200. For different versions of CIFAR100-LT, they are created from the original CIFAR100 that the samples in class y are truncated to nyµ y c−1 , where c is the number of all classes, y is the index of class and ny is the original number of training examples in class y. By varying µ to be 0.02, 0.01, 0.005, we obtain three groups of CIFAR100-LT with imbalance factor 50, 100, 200. More dataset details can be found in Appendix A.
Evaluation Metrics. We evaluate the models on the corresponding balanced test/validation datasets and report the overall top-1 accuracy over all classes, denoted as Overall. Furthermore, to better describe the internal diversity across classes with different training samples, we follow Liu et al. (2019) to split the given dataset into three disjoint sets: Many-shot (classes with more than 100 images), Medium-shot (20∼100 images) and Few-shot (fewer than 20 images) and report the corresponding accuracy for comparison.

heading: 4.1 COMPARISON WITH STATE-OF-THE-ART
text: In this section, we compare our method with a wild range of previous works in addressing long-tailed classification from different directions.
Places-LT. We initialize the ResNet-152 backbone with ImageNet pre-trained parameters following Kang et al. (2020). In Table 1, we report the result of our baseline without asynchronous modeling and denote it as Ours (plain), that is, considering the dataset together without distinguishing the head and tail. The result based on asynchronous modeling is denoted as Ours. In order to compare with baselines like Zhu & Yang (2020), in which more parameters are introduced, we also consider the upgraded version Ours† with extended parameters. By comparing our asynchronous modeling with the plain baseline, we notice that the introduction of asynchronous modeling improves the overall result notably. We also outperform the state-of-the-art methods, including OLTR (Liu et al., 2019), LWS (Kang et al., 2020), etc. In comparison with IEM (Zhu & Yang, 2020), we see that comparable result is achieved without introducing any extra parameters. With more parameters considered, much higher accuracy is achieved in our setting.
ImageNet-LT. For ImageNet-LT, the most commonly adopted architecture is ResNet-10. We also evaluate with different backbones for a thorough comparison to previous works. Table 2 shows the overall results on three different backbones, i.e., ResNet-10, ResNet-50 and ResNet-152. We find that our asynchronously obtained model achieves the top performance with impressive improvements over decoupled methods cRT, NCM and LWS in Kang et al. (2020) across all backbones. Also, when comparing with OLTR (Liu et al., 2019) which also applies the memory mechanism, the memory bank in our strategy is obviously more efficient and useful. What is more, our method also outperforms IEM (Zhu & Yang, 2020) when more parameters are considered. More detailed results, i.e., the performance on three splits can be found in Appendix C.
CIFAR100-LT. We follow Cao et al. (2019) and consider three different long–tailed versions with imbalance factors 50, 100, 200. The results in Table 3 demonstrate that in comparison with state-of-the-arts including CB-Focal (Cui et al., 2019), LDAM (Cao et al., 2019) and BBN (Zhou et al., 2020), our method consistently achieves the best performance across all three versions. Especially for CIFAR100-LT with imbalance factor 100, the incorporation of asynchronous modeling introduces more than 2% gains over our plain baseline.
iNaturalist 2018. We further evaluate our methods on iNaturalist 2018. iNaturalist 2018 is a real-world long-tailed dataset, consisting of over 8K categories. We follow Kang et al. (2020) to train the network for 200 epochs and show the results of two backbones, i.e., ResNet-50 and ResNet-152. From Table 4 we see the results are consistent with the previous datasets: training with asynchronous modeling strategy performs best across different backbones. It not only achieves better results than loss re-weighting or transferring based methods (Cao et al., 2019; Chu et al., 2020) but also outperforms decoupled cRT, NCM, LWS (Kang et al., 2020).

heading: 5 ABLATION STUDY
text: We now perform ablation study to investigate the effect of specific modules. We use ResNet-152 as the backbone and conduct related experiments on Places-LT to study the size of exemplar memory bank and the ratio between the classification loss and the memory-retentive loss. We consider the result under separated {Many, Medium, Few}-shots and the overall result. Similarly in Fig. 2 and Fig. 3, the axis for describing different shots is in the left. The change of overall result is depicted in the right of the figure, which is an independent axis.
Size of memory bank. We first explore the effect of memory bank with different sizes. In the experiment, the size of memory bank depends on the selected number of samples from each class. Particularly, we consider five cases and set the reserved number of samples from each class in X1 as 2, 6, 10, 14, 18, respectively. For each cases, other operations are kept as the same. From Fig. 2(a), we see that with the increment of memory size, the performance on Few-shot is decreasing, which is opposite to the result on Many-shot. Generally speaking, the best overall result is achieved when memory size equals to 10. We notice that the overall result is changed under different memory sizes, but it is rather stable, varying from 39.4 to 39.8.
The ratio between the classification loss and the memory-retentive loss. Similarly, we also study how the ratio between the classification loss and the memory-retentive loss affects the final results. In practice, such balance is controlled by parameters λ in Eq. 9. Based on the initial option λ = √ C/C1, in whichC is the number of all class and C1 is the class used in phase I, the initial λ is scaled properly to obtain other four values. As shown in Fig. 2(b), we conclude that the best overall result of Places-LT is achieved when λ equals to 2.03. More importantly, the overall performance retains good for a wild range of λ, i.e., λ ≤ 2.03.
Through the above analysis of memory bank size and the λ, we notice that the changes of different modules do affect the overall performance. However, the mild variation indicates that our method is robust and stable.
Influence of different partitions. In this part, we investigate the influence of disentanglement point on the final performance. The disentanglement point also corresponds to the index of class since we arrange the order of classes by the number of instance in the paper. We conduct experiments on three datasets, including CIFAR100 with imbalance factor 100, Places-LT and ImageNet-LT and explore five disentanglement points for each dataset. The final results are shown in Fig. 3. To better show the variation of overall performance (the red line), we depict it using a separated vertical axis (the right one in each figure). We also show the change of different shots in each dataset: Many-shot in orange, Medium-shot in blue and Few-shot in purple. From the comparison on three datasets, we conclude the best disentanglement point for each dataset.

heading: 6 CONCLUSION
text: In this paper, we begin with the visual phenomenon of gradient distortion in long tail and propose an asynchronous modeling strategy that learns a unified recognition model through two phases to better exploit the gradients generated by data-rich classes and data-poor classes. In unifying the training process, we introduce a memory bank and a memory-retentive loss to retain the knowledge learned in the first phase while exploring new boundaries in the second phase. The extensive results on four long tailed benchmark datasets which significantly outperform previous works validate the superior efficacy of our method.

heading: A APPENDIX
text: Dataset Details. Places-LT and ImageNet-LT are artificially truncated to follow a long-tailed distribution from Places-2 (Zhou et al., 2017) and ImageNet-2012 (Deng et al., 2009), respectively. Places-LT contains 62.5K images from 365 categories and the number of images per class varies from 4980 to 5. ImageNet-LT has 115.8 samples from 1000 classes and the number of images per class is decreased from 1280 to 5 images. iNaturalist 2018 is a real-world visual recognition dataset, that naturally exhibits long-tailed distribution. It consists 435,713 samples from 8,142 species.
Implementation Details. We use the platform of PyTorch (Paszke et al., 2019) for all experiments. For CIFAR100-LT, we adopt ResNet-32 as the backbone. The batch size is 64 and the learning rate is initialized with 0.1. The number of epoch for training is 200 and we decay the learning rate at the 160th and 180th epochs by 0.01. For Places-LT, we choose ResNet-152 as the backbone with pretrained parameters from ImageNet 2012. The learning rate for representation learning is initialized with 5e-4 and that for classifier is 0.05. We train the model for 60 epochs and all the learning rate is decayed at 20th and 40th epochs by 0.01. On ImageNet-LT, we report results with ResNet-10, 50,101,152 (He et al., 2016). Similarly, ResNet-50, 152 are also used for iNaturalist 2018. For ImageNet and iNaturalist 2018, the learning rate is initialized with 0.05 and cosine learning rate scheduler (Loshchilov & Hutter, 2016) is applied to gradually decay learning rate from 0.05 to 0. For all experiments, if not specified, we use SGD optimizer with momentum 0.9, weight decay 5e-4. The image resolution for CIFAR100-LT is 32×32 and the rest is 224×224. The λ is empirically set based on √ num old num new , where “num old” indicates the number of classes in the first stage and “num new” is the number of new classes in the second stage. The threshold to split the dataset is set as the sum of classes in Many- and Medium-shot. For CIFAR100-LT, the threshold is 70, which means that we first learn the 70 classes in the head and then involve the rest. For ImageNet-LT, the threshold is 864 and Places-LT, the threshold is 294.

heading: B APPENDIX
text: Algorithm 1 Asynchronous Modeling for Long-Tailed Recognition Input: Dataset X = {xi, yi}, learning rate η, training epoch T ; 1: Divide dataset X into two parts according to the number of instances in each class. The one
covered data-rich classes is X1 and the rest is X2. 2: Model parameters W1 = [Wr;W 1c ] in phase I; 3: for i = 1, 2, · · · , T do 4: Sample mini-batch B from training set X1; 5: Compute cross-entropy loss L1 on B; 6: Update overall parameters W1 ←W1 − η∇W1L1; 7: end for 8: Construct memory bank with a few samples from classes in X1 and denote the set as M ; 9: Update training set D = X2 ⋃ M , extend model parameters as W = [Wr;Wc];
10: for i = 1, 2, · · · , T do 11: Sample mini-batch B from training set D; 12: Compute classification loss 1|B| ∑ x∈B(Lcls(x) + Lintra(x)); 13: Compute memory-retentive loss LGdis on B; 14: Compute overall loss L = 1|B| ∑ x∈B(Lcls(x) + Lintra(x)) + λL G dis; 15: Update W ←W − η∇WL; 16: end for Output: Model with parameters W .

heading: C APPENDIX
text: In Table 5, the detailed results of {Many, Medium, Few}-shots on ImageNet-LT are described. Besides from ResNet-{50, 152}, ResNet-101 is also considered here. Compared to the baseline without asynchronous learning (Ours (plain)), our method sacrifices little in Many-shot but improves a lot in Medium- and Few-shot. More importantly, we see that our asynchronous strategy boosts the overall performance across all backbones.

Review summaries:
This paper works on long-tailed classification. The authors conducted an analysis and claimed that the difference of gradients computed on the head and tail classes plays an important role in the performance drop. The authors then proposed a two-stage approach to first train on the head classes and then train on the tail classes in an incremental learning fashion. The proposed algorithm achieved better performance than existing methods on benchmark datasets.

This paper proposes an interesting view to analyze the long-tailed problem. It states that the gradients are dominated by the head classes so that the tail classes perform poorly. From this observation, the authors propose a dual-phase approach that first train $W_r, W_c^1$ with only head-class data, and extend to train $W_r, W_c$ with tail-class data and the constructed exemplar memory bank for head classes with a newly proposed memory retentive loss.