Asynchronous Modeling: A Dual-phase Perspective for Long-Tailed Recognition
Authors: Hu Zhang
Linchao Zhu
Yi Yang
Venue:  (2021)
Keywords: long-tailed classification
gradient distortion
asynchronous modeling

Abstract: This work explores deep learning based classification model on real-world datasets with a long-tailed distribution. Most of previous works deal with the long-tailed classification problem by re-balancing the overall distribution within the whole dataset or directly transferring knowledge from data-rich classes to data-poor ones. In this work, we consider the gradient distortion in long-tailed classification when the gradient on data-rich classes and data-poor ones are incorporated simultaneously, i.e., shifted gradient direction towards data-rich classes as well as the enlarged variance by the gradient fluctuation on data-poor classes. Motivated by such phenomenon, we propose to disentangle the distinctive effects of data-rich and data-poor gradient and asynchronously train a model via a dual-phase learning process. The first phase only concerns the data-rich classes. In the second phase, besides the standard classification upon data-poor classes, we propose an exemplar memory bank to reserve representative examples and a memory-retentive loss via graph matching to retain the relation between two phases. The extensive experimental results on four commonly used long-tailed benchmarks including CIFAR100-LT, Places-LT, ImageNet-LT and iNaturalist 2018 highlight the excellent performance of our proposed method.

Under review as a conference paper at ICLR 2021
ASYNCHRONOUS MODELING: A DUAL-PHASE PER- SPECTIVE FOR LONG-TAILA RECOGNITION
Anonymous authors Paper under double-blind review
ABSTRACT
This work explores deep learning based classification model on real-world datasets with a long-tailed distribution. Most of previous works deal with the long-tailed classification problem by re-balancing the overall distribution within the whole dataset or directly transferring knowledge from data-rich classes to data-poor ones. In this work, we consider the gradient distortion in long-tailed classification when the gradient on data-rich classes and data-poor ones are incor- porated simultaneously, i.e., shifted gradient direction towards data-rich classes as well as the enlarged variance by the gradient fluctuation on data-poor classes. Motivated by such phenomenon, we propose to disentangle the distinctive effects of data-rich and data-poor gradient and asynchronously train a model via a dual- phase learning process. The first phase only concerns the data-rich classes. In the second phase, besides the standard classification upon data-poor classes, we pro- pose an exemplar memory bank reserve representative examples and memory- retentive loss via graph matching to retain the relation between two phases. The extensive experimental results on four commonly used long-tailed benchmarks including CIFAR Places-LT, ImageNet-LT and iNaturalist 2018 highlight the excellent performance of our proposed method.
INTRODUCTION
Past years have witnessed huge progress visual recognition with the successful application of deep convolutional neural networks (CNNs) on large-scale datasets, e.g., ImageNet ILSVRC 2012 (Rus-) sakovsky al. 2015), Places (Zhou et al. 2017). Such datasets are usually artificially collected and exhibit approximately uniform distribution concerning the number of samples in each class. Real-world datasets, however, are always long-tailed that only a few classes occupy the majority of instances in the dataset (data-rich) and most classes have rarely few samples (data-poor) (Reed) 2001 Van Horn & Perona 2017). When modeling such datasets, many standard methods suffer from severe degradation of overall performance. More specifically, the recognition ability classes with rarely few instances are significantly impaired (Liu et al. 2019).
One prominent direction is apply class re-sampling loss re-weighting to balance the influence of different classes (Byrd & Lipton 2019; Shuu et al.) (019)) and another alternative to conduct transferring (Wang et al. 2017 Liu al. 2019) by the assumption that knowledge obtained on the data-rich classes should benefit the recognition of data-poor classes. Recently, more sophisticated
models are designed to train the model either base some new findings (Zhou et al. 2020 Kang
et al. 2020) or combine all available techniques (Zhu & Yang 2020).. However, the property of long- tailed setting makes remain be difficult achieve large gains compared balanced datasets.
In contrast to the aforementioned strategies, we approach the long-tailed recognition problem by analyzing gradient distortion in long-tailed data, attributing to the interaction between gradients gen- erated by data-rich and data-poor classes, i.e., the direction of overall gradient is shifted to be closer to the gradient on data-rich classes and its norm variance is increased due to the dramatic variation in the gradient generated by data-poor classes. The degenerated performance when comparing with balanced datasets indicates the gradient distortion is negative during model training. Motivated by this, hypothesize that the combined analysis for gradients generated by data-rich and data-poor classes could be improper in long-taine tailed data and attempt to disentangle these two gradients. We thus propose the conception of asynchronous modeling and split the original network to promote

---

Under review as a conference paper at ICLR 2021
dual-phase learning, along with the partition of the given dataset. In phase I, data-rich classes keeps the bulk of the original dataset. It facilitates better local representation learning and more precise classifier boundary determination by eliminating the negative gradient interaction produced by data- poor classes. Based on the model learned in phase I, we involve the rest data to do new boundary exploration in the second phase.
While transiting from the first phase to the second, is hoped to reserve the knowledge learned in the first phase. Specifically, design an exemplar memory bank and introduce a memory-retentive loss. The memory bank reserves a few most prominent examples from classes in the first phase and collaborates with data in the second phase for classification. Also, the collaborated data, together with the new metervereeentive loss, tries to preserve old knowledge when the model adapts to new classes in the second phase.
In the experiments, we evaluate the proposed asynchronous modeling strategy by comparing to typical strategies, which include the re-balancing based methods (Cao et al. 2019) and transferring based methods (Liu et al. (2019). Furthermore, we also consider the latest, more sophisticated works, like BBN (Zhou et al. (020), IEM (Zhu & Yang 2020). The comprehensive study and comparison across four commonly used long-tailed benchmarks, including CIFARI CIFARI00-LT, Places-LT, ImageNet- LT and iNaturalist 2018 validate the efficacy of our method.
2 RELATED WORK
Class re-sampling. Most works along with this line can be categorized over-sampling of tail classes (Chawla et al., 2002 Han et al. 2005 [yrd & Lipton 2019) or under-sampling over head classes (Drummond et al. 2003). While the idea of re-sampling makes the overall distribution more balanced, it may encounter the problem of over-fitting on rare data and the missing of critical information on dominant classes (Chawla et al. 2002 Cui al. 2019), thus hurting the overall generalization. Beyond that, Ouyang et al. (2016); Liu et al. (2019) also involve more refined idea of fine-tuning after representation extraction adjust the final decision boundary.
Loss re-weighting. Methods based on loss re- re-weighting generally allocate larger weights for tail classes to increase their importance (Lin et al.] 2017 Ren et al.) 2018); 8. [Shu et al.] 2019 Cui et al. 2019 Khan et al.] 2017 2019: Huang et al. 2019). However, direct re- re-weiggting method is difficult to be optimized when tackling a large-scale dataset (Mikolov et al.] 2013). Recently, [Ca et al. (019)) considers the margins of the training set and introduces a aabelldistribution-aaar loss to enlarge the margins of tail classes. HHaat et al. (2019) proposes the first hybrid loss function to jointly cluster and classify feature vectors in the Euclidean space and to ensure uniformly spaced and equidistant class prototypes.
Knowledge transfer. Along this line, methods based on knowledge transfer handle the challenge of imbalanced dataset by transferring the information learned on head classes to assist tail classes. While Wang al. (2017) proposes transfer meta-knowledge from the head in progressive manner, recent strategies take consideration of intra-class variance (Yin et al. 2019), semantic fea- ture (Liu et al. 2019 Chu et al. 2020) or domain adaptation (Jamal al. (020).
Recently, BBN Zhou et al. 2020) and LWS (Kang et al., 2020) boost the landscape of long-tailed problem based on some insightful findings. The former asserts that prominent class re-balancing methods can impair the representation learning and the latter claims that data imbalance might not be an issue in learning high-quality representations. IEM (Zhu & Yang 2020) designs a more complex model that tries to concern available techniques, like feature transferring and attention. In this paper, we are motivated by gradient distortion long-tailed data, which caused by the gradient interaction between data-rich classes and data-poor classes. We thus propose to split the learning stage into two phases. We demonstrate that this separation allows straightforward approaches to achieve high recognition performance, without introducing extra parameters.
OUR METHOD
Let X = {x,, y E {l.....n} be the training set, where x; is the training data and yr is its corresponding label. The number of instances in class j is denoted as ny and the total number of
2

---

Under review as a conference paper at ICLR 2021
Vare oi gros
hare ansst
Cosine similenty
a..
C200le gioa grad
033
0.50
637
L8o
135
o26 0.5
055
+75
135
Iteration
Iteratio
Iteration
Iteration
(a) CIFARI00-LT (b) vanilla CIFARI00
(c) CIFARI00-LT
(d) vanilla CIFARIO0
Figure 1: g'gral'(' gradient generated by data-rich (data-poor) classes in CIFARIO0-LT
or gradient generated by the same classes vanilla CIFAR-100) 'grad': the overall gradient in both datasets. (a) and (b): Cosine similarity between grad and grad, grad2 and grad; (c) and (d): Norm of gradl, grad2, and grad; (a) and (b) indicate the overall gradient is shifted to be closer to the direction of gradient generated by data-rich class. (c) and (d) show that the variance of overall gradient is enlarged by the fluctuation of gradient on data-poor classes.
training samples is denoted generality, we assume that the classes are sorted in decreasing order, that is, if j. S ng. We define the whole network as fa: [W,; Wel), where f is the implemented deep learning model with parameters W, for representation learning and parameters W. for classification, and is the input.
E5=1 Tg. where C is the number of classes. Without loss of
3.1 GRADIENT DISTORTION IN LONG TAII.
Given a long-tailed dataset, our goal is to achieve better overall performance across all classes. In contrast to previous common heuristics (c.g., resampling, reweighting and feature transfer), we visit the problem of long-tailed classification from the perspective gradient distortion. The overall gradient for updating is modulated by the gradients generated by data-rich classes in the head and data-poor classes in the tail. To state the details, we visualize the associated metrics in the training Specifically,
process of vanilla CIFARIO0 and long-tailed CIFARI00 (CIFAR100-LT) in Fig. [ the cosine similarity between the gradients is visualized in Fig. I(a) CIFAR100-LT and Fig. ((b) (vanilla CIFARI00). Similarly, the norm of each gradient recorded in Fig. [[e] (CIFARI00-LT) and Fig. I(d) (vanilla CIFAR100). The higher similarity between the overall gradient and the data- rich gradient indicates that the overall gradient shifted the direction of the data-rich gradient. Meanwhile, the norm variance of overall gradient is enlarged due to more dramatic fluctuation of the gradient on data-poor classes. Motivated by the degenerated performance in long-tailed dataset, it is hypothesized that synchronous application of two distinctive gradients could impair the overall performance.
3.2 ASYNCHRONOUS MODELING
Rather than directly regulating the overall gradient as previous methods, we begin with the disen- tanglement of two gradients and propose a dual-phase asynchronous modeling strategy. The data from data-rich classes is first considered in model training and then the rest classes involved. Such asynchronous operation not only reduces the potential disturbance between two gradients, but also ensures the benefits of each gradient be exploited. Mathematically, the original dataset is X with C classes. Suppose C₁ classes are considered in phase I, we then write X1 as the set of data from C₁ classes. The data rest C2 classes is denoted as X2. where C2 = C C₁. Accordingly, the parameters W. for C' classes in (a:; [W,, W.]) are truncated as W! for C1 classes in the first phase.
3.2.1 LEARNING IN THE FIRST PHASE
In model learning from data X1, the consideration of gradient data-poor categories is avoided, which keeps the truncated model f(x; [Wr; W:]) to be more concentrated. In optimization, the cross-entropy loss over the classes in X, is minimized with respect to parameters W, and W!
L1 = Σ
y ylog (a: (::; W!).
(1)
(x.
3

---

Under review as a conference paper at ICLR 2021
For further improvement in the training, some balanced sampling strategies could be incorporated in this phase. For example, the progressively-bulane balanced strategy in (Kang et al. 2020) combines instance-balanced sampling and class-balanced sampling, that is, P5(t)
L + t d
where (4) denotes the sampling probability for class j in training epoch t. It is computed with n4 and class-based probability 1 T is the C.
linear combination for instances-based probability Σ total epoch number.
3.2.2 JOINT PREDICTION IN THE SECOND PHASE
We wish to involve the data in X₂ to obtain a complete model across all C' classes for overall evaluation. To do so, the basis of parameters W, obtained in phase I, introduce the classifier parameters W for the recognition of new classes in X2. Similar to phase I, the standard cross- entropy loss across all data in X2 is considered. However, considering solely on data X2 tends to forget the knowledge learned in the first phase, To tackle with the obstacle, we thus design a memory bank and memory- retentive loss to realize the seamless connection between two data splits. First, representative samples in X1 are retained in an augmented memory module to enable the joint prediction over all classes. Second, the examples reserved in the memory are combined with X2. which are collaboratively trained with a unified memorv-retentive loss.
Exemplar memory bank. In maintaining the knowledge obtained in the first phase, we design an exemplar memory bank that selects only few most representative samples from classes in Xi. For simplicity, the number of selected samples from each class set to equal. We denote the reserved data in the memory bank as M. Ideally, the most representative examples are samples that are closest to the center of cach class. However, a precise class center is not always accessible. Thus in practice, the center is progressively estimated by accessing over the entries generated in previous steps to infer new entry in the memory bank.
Without loss of generalization, we consider class in dataset X1 to demonstrate the detailed oper- ation. We first compute the average feature from all examples in class j in original training set X1 serve as class prototype Gj. which is thus the initial estimation of class center. We return the instance which is closest to Ej in X, and set it as the first selected sample for the memory bank,
mi = arg ((cx X1)},
s,EX,
where s is a vector space similarity metric, like cosine similarity. mi is used to denote the returned sample zę- Before selecting the rest instances from X1, we need update the estimated center cy- Without loss of generality, suppose we have selected k samples from X and denote the feature map of data in memory bank as Mj- = [mi, m2, .., my E R where d is the dimension of each feature map. Each sample in M, serves as a guided hypothesis and its correlation with ej can then be computed for the new state 21+ that is,
exp(s(c,,m.)
Pi Σ, (xp(s(cc.mm))
(3)
Sk+1 Σ p'm;
(4)
where is the same similarity metric above. p; is computed by the distances between the selected data and the center prototype and it serves as weights to update state 24+1- 2k+1 is the weighted average of all feature maps in My. New samples can then be returned for k + 1 step by performing
m = arg max (s(g) + A, X1)),
(5)
2.EX,
where A is the residual between ej and 2141, i.e. A Cj 2k+1- m is used denote the returned sample xz.
Memory-retentive loss. Based on the memory bank, we obtain a combined data set D by extending X₂ with examples in the memory bank M, i.e., D = M X2. Similarly, the joint prediction with cross-entropy loss is first considered. When the model is adapted to fit data X2, the knowledge learned in X₁ tends to be forgotten. We thus introduce new memorv-retentive loss dis based on
4

---

Under review as a conference paper at ICLR 2021
graph matching, which provides strong constraint in memorizing previous knowledge. Specifi- cally, the feature map of each data in the training set D is a node in a graph. Based on the model learned in the first stage and the new model to be trained in the second phase. two graphs Gold and Gnew can thus be constructed. That is, we not only consider feature similarity of a single example on the old model and the new model, but also compute the global matching similarity on the whole training set D. Suppose the feature map of one node in Gold is 2, and in Gold is &, thus the similar- ity between old graph Gold and new graph Gnow is measured by computing the change between any node z; in Gold and any node 2j in Inew, that is,
exp(s (2,, 2,)) az Σy exp(p(zi) (20, 22y)) ))² 2, gold: 2z 2y E Inews
(6)
Σ Cajillzi - -2ill, zi E Golds 2j E Gnews
(7)
dis Σ Mr; E Gold:
(8)
where s the vector similarity metric. ajs represents the distance between node in graph Gold and node j in graph Inew: H4 thus ntuitively measures the difference between 2j and its closest neighbor in graph &nww- Consider all nodes in graph Gold together, we obtain the memory-retentive loss I. which describe graph similarity between two graphs
Overall loss. Combined the above analysis together, the overall loss in phase II is thus as below:
L
[Leto(()) + Lintra(x)) + AL dist D
RED
where the first term is for classification and the second is the designed loss which constrains the knowledge in old model through graph matching, À is hyperparameter to balance the two terms. Notice that, apart from the standard cross-entropy loss (xx) for input æ in the first term, we also consider an intra-classffication loss Lintra (x) avoid memory data in M being dominated by new classes in X2. When we consider cosine linear classifier, one of the instantiations could be Lintra (x) embedding and nh denotes the other class embedding, z(x) is the normalized feature map of x, is a margin value. (10,((z)) ((x) denotes a positive score between w and E(x), while (wo (xx) denotes the negative score between wh and E(x). Lintra optimizes the network maintain margin of m between the positive score and the highest negative score. Finally, for comprehensive overview of the asynchronous modeling structure, can find it in Algorithm [ in Appendix B
カー max(0, m (w, (x)) + (w² (xx)). in which, t is the ground-truth class
4 EXPERIMENTS
Datasets. We perform extensive experiments on four long-tailed datasets, including CIFAR100- LT (Cao al. 2019). Places-LT (Liu et al.) 2019), ImageNet-LT (Liu et al.) 2019), and iNaturalist 2018 (iNatrualist) 2018). CIFAR is created with three different imbalance factors 50, 100, 200. For different versions of CIFAR
they are created from the original CIFARIOM 100 that the samples in class y are truncated to ny where e is the number of all classes, y is the index of class and ny is the original number of training examples in class By varying to be 0.02, 0.01, 0.005, we obtain three groups of CIFAR100-LT with imbalance factor 50, 100, 200. More dataset details can be found in Appendix A
Evaluation Metrics. We evaluate the models on the corresponding balanced test/validation datasets and report the overall t accuracy over all classes, denoted Overall. Furthermore, to better de- scribe the internal diversity across classes with different training samples, we follow Liu et al. (2019) to split the given dataset into three disjoint sets: Many-shot (classes with more than 100 images), Medium-shot 120~100 images) and Few-shot (fewer than 20 images) and report the corresponding accuracy for comparison.
4.1 COMPARISON WITH STATE-OF-THE-ART
In this section, we compare our method with a wild range of previous works in addressing long-tailed tailed classification from different directions.
5

---

Under review as a conference paper at ICLR 2021
Table I: Evaluation results on Places-LT. i denotes our result with extended parameters.
Method
Many
Medium
Few
Overall
Joint (Kang ct al.
2020)
45.7
27.3
8.2
30.2
Lifted Loss (Oh Song al.
2016
41.1
35.4
24.0
35.2
Focal loss (L
2017
41.1
34.8
22.4
34.6
Range Loss (Zhang el al. 2017b
41.1
35.4
35.1
FSLwF Giiaars & Komodakis 2018
43.9
29.9
29.5
34.9
OLTR [L.u et al.
2019)
44.7
37.0
25.3
35.9
NCM Kang ct 20200
40.4
37.1
27.3
36.4
cRT
(Kang et al.
20200
42.0
37.6
36.7
LWS
KKan ct al.
2020
40.6
39.1
28.6
37.6
IEM
7hu & Yang
2020)
46.8
39.2
28.0
39.7
Ours
(plam
46.2
36.3
24.5
37.6
Ours
44.2
40.2
30.9
39.8
Ours
44.5
40.9
31.8
40.4
Table 2: Evaluation on ImageNet-LT with different backbones. parameters.
denotes our result with extended
Method
ResNet-10
ResNet-50
ResNet-152
FSL.F GGidaris & Komodakis
2018)
28.4
Focal Loss (Lnn ct al.
2017
30.5
Range Loss (Zhang al.
201 7b)
30.7
Lifted Loss (Oh Song
2016
30.8
FSA (Chu et al. 20200
35.2
IEM Zhuu & Yang 2020
43.2
OLTR (Luu al. 2019
35.6
36.7
43.2
Joint Kang al. 20%0
34.8
41.6
44,9
NCM KKng ct al.
2020
35.5
44.3
47.8
cRT
Kang el al.
2020
41.8
47.3
50.1
LWS (Kang al.
2020
41.4
47.7
50.5
Ours (plain)
41.3
49.7
$1.7
Ours
42,0
51.0
53.0
Ours
43.8
52.2
53.8
Places-LT. We initialize the ResNet-152 backbone with ImageNet pre-trained parameters following Kang et al. 2020). In Table [ we report the result of our baseline without asynchronous modeling and denote it as Ours (plain), that is, considering the dataset together without distinguishing the head and tail. The result based on asynchronous modeling is denoted as Ours. In order to compare with baselines like Zhu Yang (2020), which more parameters are introduced, we also consider upgraded version Ours! with extended parameters. By comparing our asynchronous modeling with the plain baseline, we notice that the introduction of asynchronous modeling improves the overall result notably. We also outperform the state-of-the-at methods, including OLTR (Liu et al. 2019), LWS (Kang et al. 2020), etc. In comparison with IEM (Zhu & Yang 2020), we see that comparable result is achieved without introducing any extra parameters. With more parameters considered, much higher accuracy is achieved in our setting.
ImageNet-LT. For ImageNet-LT,, the most commonly adopted architecture is ResNet-10 10. We also evaluate with different backbones for a thorough comparison to previous works. Table shows the overall results three different backbones. i.e., ResNet-10. ResNet-50 and ResNet- 152. We find that our asynchronously obtained model achieves the top performance with impressive improve- ments over decoupled methods cRT, NCM and LWS Kang al.] (2020) across all backbones. Also, when comparing with OLTR (Liu et al. 2019) which also applies the memory mechanism, the memory bank in our strategy obviously more efficient and useful. What more, our method also outperforms IEM (Zhu & Yang (020) when more parameters are considered, More detailed results, i... the performance on three splits can found Appendix C

---

Under review as a conference paper at ICLR 2021
CIFARIOO-LT. We follow Cao al. 2019) and consider three different Long-tailes versions with imbalance fac- tors 50, 100, 200. The results Table g demonstrate that in comparison with state-of-the-arts including CB-Focal (Cui al. 2019). LDAM (Cao al., 2019) and BBN (Zhou et al. 2020), our method consistently achieves the best performance across all three versions. Especially for CIFAR100-L1 with imbalance factor 100, the incorporation of asynchronous modeling introduces more than 2% gains over our plain baseline.
Table The overall results CIFARI00-LI under
different balance factors (200, 100, 50).
Imbalance factor
200
100
50
Cross Entropy
35.63
38.32
43.85
Focal loss (Lin al.
2017
34.69
38.41
44.32
Mixup (Zhang ct al.
2017a
36.20
39.54
44.99
CB-Focal (C UI et al 2019
36.23
39.60
45.32
LDAM (Cao
2019
38.06
42.04
46.62
BBN (Zhou et 20200
37.93
42.56
47.02
Ours (plam)
39.35
42.48
47.61
Ours
40.53
44.79
49.32
iNaturalist 2018. We further evaluate our methods on iNaturalist 2018. iNaturalist 2018 a real-world long-tailed dataset, consisting of over 8K categories. We fol- low Kang et al. (2020) to train the net- work for 200 epochs and show the re- sults of two backbones, i.... ResNet-50 and ResNet-152. From Table 4 see the results are consistent with the previous datasets: training with asynchronous mod- eling strategy performs best across differ- ent backbones. It not only achieves better results than loss re-weighting
Table 4: Evaluation results on iNaturalist 2018.
Method
ResNet-50
ResNet-152
CB-Focal Cao et al.
2019
61.1
LDAM Cao
2019
68.0
BBN
Zhou
2020
69.6
NCM
KKng et al.
2020
63.1
67.3
Joint
Kang el al.
2020
65.8
69.0
FSA
Chu al.
2020)
65.9
69.1
cRT
Kang
2020
67.6
71.2
transfer-
LWS Kang
69.5
72.1
ring based methods (Cao et al.) 2019 Chu al., 2020 but also outperforms decou- pled cRT, NCM, LWS (Kang et al. 2020)
Ours
69.8
72.5
ABLATION STUDY
We now perform ablation study to investigate the effect of specific modules. We ResNet-152 as the backbone and conduct related experiments on Places-LT to study the size of exemplar memory bank and the ratio between the classification loss and the memory-retentive loss. We consider the result under separated { [Many, Medium, F-wo-shots and the overall result. Similarly in Fig. 2 and Fig. 3 the axis for describing different shots is in the left. The change of overall result depicted in the right of the figure, which is an independent axis.
Size of memory bank. We first explore the effect of memory bank with different sizes. In the experiment, the size of memory bank depends on the selected number of samples from each class. Particularly, we consider five cases and set the reserved number of samples from each class in X1 as 2, 6, 10, 14. 18, respectively. For each cases, other operations are kept as the same. From Fig. 2(a) we see that with the increment of memory size, the performance on Few-shot is decreasing, which is opposite to the result on Many-shot. Generally speaking, the best overall result is achieved when memory size equals to 10. We notice that the overall result is changed under different memory sizes, but it is rather stable, varying from 39.4 to 39.8.
The ratio between the classification loss and the memory-retentive loss. Similarly, we also study how the ratio between the classification loss and the memory-retentive loss affects the final results. In practice, such balance controlled by parameters in Eq. [ Based on the initial option = /C/C, in which C' is the number all class and C₁ is the class used in phase I, the initial is scaled properly to obtain other four values. As shown in Fig. 2(b] we conclude that the best overall result of Places-LT LT is achieved when equals to 2.03. More importantly, the overall performance retains good for wild range of X, i.e., X < 2.03.

---

Under review as a conference paper at ICLR 2021
Places-LT
Places-LT
14
20.5
20.
37.5
22
39.1 38
35.5
14 18
2.33055-2
0.03
2.03
2033
2 20040'2
memory
lambda
-G-Onaal
(a)
(b)
Figure 2: The classification results on Places-LT with backbone ResNet- The right y-axis is to depict the overall result and the left one is for [Many, Medium, Few) Fshots in each figure. (a): The change of classification results under different memory bank size, (b): The change classification results under different A, which balances the classification loss and memory retentive loss.
HT
CIFAR100-LT
ImageNet-LT
40
Mary
Meduu
39.2
12
$0.5
50
46.5
234
274
294
314
334
50 70
90
714
764
814
664
914
Figure 3: The classification results on three datasets with different disentanglement points. The right y-axis is for the overall performance and the left y-axis for results on [Many, Medium, Few jew]-shots. With the movement of the disentanglement point, the overall result first increases then decreases.
Through the above analysis of memory bank size and the X, we notice that the changes of different modules do affect the overall performance. However, the mild variation indicates that our method is robust and stable.
Influence of different partitions. In this part, we investigate the influence of disentanglement point on the final performance. The disentanglement point also corresponds to the index of class since we arrange the order of classes by the number of instance the paper. We conduct experiments on three datasets, including CIFAR1ON 100 with imbalance factor 100, Places-ITT and ImageNet-LLT and explore five disentanglement points for each dataset. The final results are shown Fig. 3 To better show the variation of overall performance (the red line), depict it using separated vertical axis (the right one each figure). We also show the change of different shots in each dataset: Many-shor in orange, Medium-shor shot in blue and Few-shot in purple. From the comparison three datasets, conclude the best disentanglement point for each dataset.
CONCLUSION
In this paper, we begin with the visual phenomenon of gradient distortion in long tail and propose an asynchronous modeling strategy that learns a unified recognition model through two phases to better exploit the gradients generated by data-rich classes and data-poor classes. In unifying the training process, we introduce a memory bank and memory-retentive loss to retain the knowledge learned in the first phase while exploring new boundaries in the second phase. The extensive results four long tailed benchmark datasets which significantly outperform previous works validate the superior efficacy of our method.
8

---

Under review as a conference paper at ICLR 2021
REFERENCES
Jonathon Byrd and Zachary Lipton. What is the effect of importance weighting in deep learning?
In International Conference on Machine Learning, pp. 872 872-881, 2019.
Kaidi Cao, Colin Wei, Adrien Gaidon, Nikos Arechiga, and Tengyu Ma. Learning imbalanced datasets with label-distibution--wwaee margin loss. In Advances in Neural Information Processing Systems, pp. 1567-1578, 2019.
Nitesh V Chawla, Kevin W Bowyer, Lawrence O Hall, and W Philip Kegelmeyer. Smote: synthetic minority over-sampling technique. Journal of artificial intelligence research, 16:321-357, 2002.
Peng Chu, Xiao Bian, Shaopeng Liu, and Haibin Ling. Feature space augmentation for long-tailed data. arXiv preprint arXiv:2008. 20t.::000000..33,, 2020.
Yin Cui, Menglin Jia, Tsung- Yi Lin, Yang Song, and Serge Belongie. Class-balanced loss based on effective number of samples. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 9268 9277, 2019.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- scale hi- erarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248-255. Ieee, 2009.
Chris Drummond, Robert C Holte, et al. C4. 5, class imbalance, and cost sensitivity: why under- sampling beats over-samp In Workshop on learning from imbalanced datasets II, volume 11, pp. 1-8. Citeseer, 2003.
Spyros Gidaris and Nikos Komodakis. Dynamic few-shot visual learning without forgetting, In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4367- 4375, 2018.
Hui Han, Wen- Yuan Wang, and Bing- Huan Mao. Borderline-smone smote: a new over- sampling method in imbalanced data sets learning. In International conference on intelligent computing, pp. 887. Springer, 2005.
Munawar Hayat, Salman Khan, Waqas Zamir, Jianbing Shen, and Ling Shao. Max-margin class imbalanced learning with gaussian affinity. arXiv preprint arXiv:1901.077 00711, 2019.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Chen Huang, Yining Li, Change Loy Chen, and Xiaoou Tang. Deep imbalanced learning for face recognition and attribute prediction. IEEE transactions on pattern analysis and machine intelli- gence, 2019.
iNatrualist. The inaturalist 2018 competition dataset.
hutp:://gilubb.co///s/ppppp/
comp//ree/musster20018. 2018
Muhammad Abdullah Jamal, Matthew Brown, Ming-Hsuan Yang. Liqiang Wang, and Boqing Gong. Rethinking class- methods for long-tailed visual recognition from domain adaptation perspective. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recog- nition, pp. 7610-7619, 2020.
Bingyi Kang, Saining Xie, Mareus Rohrbach, Zhicheng Yan, Albert Gordo, Jiashi Feng, and Yan- nis Kalantidis. Decoupling representation and classifier for long-tailed recognition. In Eighth International Conference on Learning Representations (ICLR), 2020.
Salman Khan, Munawar Hayat, Syed Waqas Zamir, Jianbing Shen, and Ling Shao. Striking the right balance with uncertainty. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 103-112. 2019.
Salman H Khan, Munawar Hayat, Mohammed Bennamoun, Ferdous A Sohel, and Roberto Togneri. Cost-sensitive learning of deep feature representations from imbalanced data. IEEE transactions neural networks and learning systems, 29(8):3573-3587, 2017.

---

Under review as a conference paper at ICLR 2021
Tsung- Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. Proceedings of the IEEE international conference on computer vision, pp- 2980-2988, 2017.
Ziwei Liu, Zhongqi Miao, Xiaohang Zhan, Jiayun Wang, Boqing Gong, and Stella X Yu. Large-scale long-tailed recognition an open world. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2537-2546, 2019.
Ilya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. arXiv preprint 1600:1668.03383, 2016.
Tomas Mikolov, Ilya Sutskever, Kai Chen. Greg S Corrado, and Jeff Dean. Distributed represen- tations of words and phrases and their compositionality. In Advances in neural information pro- cessing systems, pp- 311--3119, 19, 2013.
Hyun Oh Song. Yu Xiang, Stefanie Jegelka, and Silvio Savarese. Deep metric learning via lifted structured feature embedding. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4004-4012, 2016.
Wanli Ouyang, Xiaogang Wang, Cong Zhang, and Xiaokang Yang. Factors in finetuning deep model for object detection with long-tail distribution. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 864 873, 2016.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high- performance deep learning library. Advances neural information processing systems, pp- 2019.
William J Reed. The pareto, zipf and other power laws. Economics letters, 74(1):15-19, 2001.
Mengye Ren, Wenyuan Zeng, Bin Yang, and Raquel Urtasun. Learning to reweight examples for robust deep learning. arXiv preprint vrXiv:/803.09000 2018.
Olga Russakovsky, Jia Deng. Hao Su, Jonathan Krause, Sanjeev Satheesh, Scan Ma, Zhiheng Huang. Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International journal of computer vision, 1 115(3)211-252 2015.
Jun Shu, Qi Xie, Lixuan Yi, Qian Zhao, Sanping Zhou, Zongben Xu, and Deyu Meng. Meta-weight- net: Learning an explicit mapping for sample weighting. In Advances Neural Information Processing Systems, pp. 1919-19229 1930, 2019.
Grant Van Hom and Pietro Perona. The devil is in the tails: Fine-grained classification in the wild. arXiv preprint 1rXiv:1709.0144 2017.
Yu-Xiong Wang, Deva Ramanan, and Martial Hebert. Learning to model the tail. In Advances in Neural Information Processing Systems, pp. 7029-7039, 2017.
Xi Yin, Xiang Yu, Kihyuk Sohn, Xiaoming Liu, and Manmohan Chandraker. Feature transfer learn- ing for face recognition with under-represenned data. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5704-5713 2019.
Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. arXiv preprint arXiv: 1710:0017000.111111. 2017a
Xiao Zhang, Zhiyuan Fang. Yandong Wen, Zhifeng Li, and Yu Qiao. Range loss for deep face recognition with long-tailed training data. In Proceedings of the IEEE International Conference on Computer Vision, pp. 5409-5418, 2017b.
Bolei Zhou, Agata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba. Places: A 10 million image database for scene recognition. IEEE transactions on pattern analysis and machine intelligence, 40(6): 1452-1464 1464, 2017.
Boyan Zhou, Quan Cui, Xiu-Shen Wei, and Zhao-Min Chen. Bbn: Bilateral- Solateral-branch network with cumulative learning for long-tailes visual recognition. In Proceedings of the IEEE/CVF Confer- on Computer Vision and Pattern Recognition, pp. 9719-9728, 2020.
10

---

Under review as a conference paper at ICLR 2021
Linchao Zhu and Yi Yang. Inflated episodic memory with region self-attention for long-tailes vi- sual recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3344-4353, 2020.

---

Under review as a conference paper at ICLR 2021
A APPENDIX
Dataset Details. Places-LT and ImageNet-LLT are artificially truncated to follow a long-tailed dis- tribution from Places-2 (Zhou et al., (017) and ImageNet-2012 (Deng al. 2009), respectively. Places-LT contains 62.5K images from 365 categories and the number of images per class varies from 4980 to ImageNet-LT has 115.8 samples from 1000 classes and the number of images per class is decreased from 1280 to 5 images. iNaturalist 2018 is real-world visual recognition dataset, that naturally exhibits long-tailed distribution. consists 435,713 samples from 8. 122 species.
Implementation Details. We use the platform of PyTorch (Pazzke et al. 2019) for all experiments. For CIFARIOO-LT, we adopt ResNet-32 as the backbone. The batch size is 64 and the learning rate initialized with 0.1. The number of epoch for training is 200 and we decay the learning rate the 160th and 180th epochs by 0.01. For Places-LT, we choose ResNet- 152 as the backbone with pre- trained parameters from ImageNet 2012. The learning rate for representation learning initialized with 5e-4 and that for classifier is 0.05. We train the model for 60 epochs and all the learning rate decayed 20th and 40th epochs by 0.01. On ImageNet-LT, we report results with ResNet-10, 50,101,15, (He et al.] 2016). Similarly, ResNet-50, 152 are also used for iNaturalist 2018. For ImageNet and iNaturalist 2018, the learning rate is initialized with 0.05 and cosine learning rate scheduler (Loshchilov & Hutter, (016) is applied to gradually decay learning rate from 0.05 to O. For all experiments, not specified, we use SGD optimizer with momentum 0.9, weight decay 5e-4. The image resolution for CIFAR 100-LT is 32x32 and the rest is 224x224 The À is empirically set old.
based V num where "num old" indicates the number of classes in the first stage and "num new" is the number of new classes the second stage. The threshold to split the dataset is set as the sum of classes in Many- and Medium-shot. For CIFARI00-LA the threshold is 70, which means that we first learn the classes in the head and then involve the rest. For ImageNet-LT, the threshold 864 and Places-LTe LT, the threshold is 294.
B APPENDIX
Algorithm Asynchronous Modeling for Long- Tailed Recognition
Input:
Dataset X { y/ learning rate 7), training epoch T;
1: Divide dataset X into two parts according to the number of instances each class. The one covered data-rich classes is Xi and the rest is X2.
2: Model parameters Wi- [Wr; W in phase I;
3: for = 1,2, T do
4: Sample mini-batch B from training set X1;
5: Compute cross-entropy loss L1 on B:
6: Update overall parameters W1 < W v/ w; C1:
7: end for
8: Construct memory bank with a few samples from classes in X, and denote the set M;
9: Update training set = X2 U M, extend model parameters as W = [Wr; W.];
10: for i 1,2, T do
11: Sample batch B from training set D;
12:
Compute classification loss [8 ExeD(Tess Trintra (a:));
13: Compute memorv-retentive loss dis B;
14: Compute overall loss L B (Eds (i)tra(x)) NLO
Update W W C:
16:
end for
Output: Model with parameters W.
12

---

Under review as a conference paper at ICLR 2021
C APPENDIX
In Table 5 the detailed results of {Many, Medium, Few)-shots on ImageNet-LT are described. Be- sides from ResNet-(50, 152], ResNet-101 is also considered here. Compared to the baseline without asynchronous learning (Ours (plain)), our method sacrifices little in Many-shot but improves a lot in Medium- and Few-shot. More importantly, we see that our asynchronous strategy boosts the overall performance across all backbones.
Table 5: Comprehensive results on ImageNet-LT with different backbones.
Backbone
Method
Many
Medium
Few
Overall
Joint
Kang et
2020
33.8
5.8
41.6
NCM
[Kang et al.
2020
53.1
42.3
26.5
44.3
cRT
Kang ct at.
2020
58.8
44.0
26.1
47.3
ResNet-50
LWS
Kang al.
2020
57.1
45.2
29.3
47.7
Ours (plan
66.3
45.0
19.2
49.7
Ours
48.3
22.1
51.0
Joint Kang
2020
66.6
36.8
7.1
44.2
NCM (Kang ct a 2020
45.1
28.8
47.4
cRT KKng el al.
2020
61.6
46.5
28.0
49.8
ResNet-101
LWS Kang al.
2020
60.1
47.6
31.2
50.2
Ours (plam)
67.8
46.5
20.6
$1.2
Ours
65.8
49.1
23.5
52.1
Joint Kang al.
2020
66.9
27.7
7.7
44.9
NCM [Kang el a
2020
56.9
45.6
29.9
47.8
cRT Kang et al.
2020
61.8
46.8
28.4
50.1
ResNet-152
LWS (Kang al.
2020
60.6
47.8
31.4
50.5
Ours (plan)
67.9
47.3
21.5
51.7
Ours
66.5
50.6
23.6
53.0
13


Reviews:
[{"review_id": "kkH2FVkvqPk", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes an interesting view to analyze the long-tailed problem. It states that the gradients are dominated by the head classes so that the tail classes perform poorly. From this observation, the authors propose a dual-phase approach that first train $W_r, W_c^1$ with only head-class data, and extend to train $W_r, W_c$ with tail-class data and the constructed exemplar memory bank for head classes with a newly proposed memory retentive loss.\n\n#### Pros:\n1. This paper is well organized and written.\n2. Analyzing through the gradient seems to be an interesting view, which could bring insights into the long-tailed problem.\n\n#### Some questions and concerns:\n1. In Fig.1 (c)(d), despite the norm variance, I don't understand why the \"norm of grad\" is larger than \"the norm of grad1\" and \"the norm of grad2\". They are all norms, right? So why the grad (a combination of grad1 and grad2 in my opinion) becomes larger?\n2. This dual-phase approach somehow seems similar to [this paper](https://openaccess.thecvf.com/content_CVPR_2020/papers/Li_Overcoming_Classifier_Imbalance_for_Long-Tail_Object_Detection_With_Balanced_Group_CVPR_2020_paper.pdf), which actually groups categories that own a similar number of training instances and train the classifier separately. And this paper divides all categories into 2 groups (head and tail), but with more complicated operations.\n3. The authors conduct experiments on almost all common long-tailed classification datasets, which is great. But the improvements seem to be very limited on Places-LT, ImageNet-LT, iNaturalist18 comparing with recent works. It seems that it only works well on small datasets like CIFAR, which is somehow weak.\n4. About the training schedule, in the Appendix.Implementation Details, it states that the number of training epochs is 200. It means that we train both 200 epochs for both phase1 and phase2 right? Though $X_1$ and $X_2$ are not overlapped, the exemplar bank will be trained twice. How much will it cost? How is the cost comparing with re-sampling approaches?", "extra_scores": {}, "extra_text": {}}, {"review_id": "SYbxLlfihF0", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "# Summary\nThis paper works on long-tailed classification. The authors conducted an analysis and claimed that the difference of gradients computed on the head and tail classes plays an important role in the performance drop. The authors then proposed a two-stage approach to first train on the head classes and then train on the tail classes in an incremental learning fashion. The proposed algorithm achieved better performance than existing methods on benchmark datasets.\n\n# Strengths\n\n- The analysis of the gradients between the head and tail classes seems to be novel.\n\n- The proposed methods achieve good performance on benchmark datasets.  \n\n\n# Weaknesses\n\n1. The paper lacks a detailed description of how the gradient analysis is conducted. For example, is the gradient computed at some layers or all the layers? How is the variance computed? Is it possible that the relatively large variance is due to a smaller sample size of tail classes? Why does the gradient computed on both types of classes have a larger norm? I would suggest that the authors provided some equations. Moreover, given just Figure 1 and the difference to balanced training, it is still unclear if such a gradient difference really leads to poor long-tailed performance. An analysis of the relationship between gradients and the classification performance will make the paper stronger. \n\n2. The idea of learning the classifier with two-phase has been proposed (Kaidi Cao et al., 2019; Kang et al., 2020). The most similar ones to the paper are [A, B, C], in which the first stage only considers data-reach classes. The authors, however, failed to identify and discuss them.\n\n[A] Wang et al., Frustratingly Simple Few-Shot Object Detection, ICML 2020\n\n[B] Zhang et al., A Study on Action Detection in the Wild, arxiv 2019\n\n[C] Cui et al., Large Scale Fine-Grained Categorization and Domain-Specific Transfer Learning, CVPR 2018\n\nMemories are also used in (Liu et al., 2019) but the authors did not discuss it.\n\n\n3. Details and motivations of the proposed methods are not clear, making it very hard to understand the proposed algorithm.\n\n- It is unclear why the memory bank is needed. Many existing works, including [A, B, C], have proposed to simply train the second stage with all classes, with the subsampled class-balanced data, and shows promising results. Some of them even \"freeze\" the features but only train the classifier in the second stage.\n\n- The way the authors tackle the second-stage training is similar to incremental learning, but the authors failed to discuss those works. One example is [D].\n\n[D] Rebuffi et al., iCaRL: Incremental Classifier and Representation Learning, CVPR 2017.\n\n- The design of Eq. (2), (3), (4) are not well-described and justified. Why do we need a new state? Why do we need Delta? The computation of Delta seems wrong: c_j + c_j -z_{k+1} = 2*c_j – z_{k+1}? Moreover, what does the state mean here?\n\n- It is unclear why we need graphs. It is unclear what “z” stands for in Eq. (5), (6), (7). It is unclear how a_{ij} is computed.\n\n- The motivations and description of the intra-class loss in Eq. (8) is unclear.\n\n- I would suggest that the authors provide a figure for their algorithm architecture and pipelines.\n\n\n4. Experiments and analysis:\n\n- Back to my comment in 1., there is no analysis to further justify that gradient distortion is really the cause of the poor longtailed performance, and there is no analysis if the proposed algorithm resolves it. There is no analysis if graphs and memory banks are really needed. \n\n- It is unclear what Ours(Plain) refers to. What is the model without asynchronous modeling?\n\n# Minor\nI would suggest that the authors replace \"asynchronous\" with other terms. The proposed algorithm is just a two-stage algorithm. There is no component of distributed learning and communications that require synchronization or not.\n\n# Justification\nWhile the proposed algorithm achieves promising results, the algorithm is not written, well-described, and motivated. It is also unclear if the gradient distortion is really the cause of poor long-tailed performance. I thus give a score of 3.\n\n-------------------------- Post-rebuttal ----------------------------\n\nI read the authors' rebuttal and I appreciate their efforts. I would suggest that the authors incorporate those clarifications into their manuscript. I would also suggest that the authors re-motivate their paper and modify their approach section.\n\nIn terms of the discussions to related work, I do think [A, B, C] is about long-tailed recognition/detection, not few-shot learning. For instance, [A] works on LVIS, a long-tailed object detection dataset; [C] works on iNaturalist, which is clearly long-tailed. [B]'s Fig 1 clearly shows that the problem is long-tailed. I'm surprised that the authors simply said that [A, B, C,] works on different problems but did not intend to discuss the similarity in methodologies.\n\nThere are some very important questions not addressed yet, specifically, my comments 3 and 4: There is no analysis if the proposed algorithm resolves gradient distortion. There is no analysis if graphs and memory banks are really needed. \n\nI also read other reviewers' comments and I agree with R4 that the current version still lacks critical insights and some justifications are questionable.\n\nGiven these, I would keep my initial score unchanged.", "extra_scores": {}, "extra_text": {}}, {"review_id": "G3U4WXeDBM1", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\n\n- This paper proposes a two-stage learning algorithm to address imbalanced datasets. Only data-rich classes is used in the first representation learning stage. In the second stage,  an exemplar memory bank with graph matching is used together with standard classification. Experimental results on benchmarks highlight the effectiveness of the proposed method.\n \nPros:\n- The total experiments conducted are thorough and satisfactory.\n\nCons:\n- The presentation of this paper can be improved. For example, the notation in Figure 1 is very hard to parse. The writing could be improved as well. There's a chance that I didn't understand the paper so I just list all my concerns in questions.\n- In the implementation detail section, there's only implementation details shared with the baseline. There's no detail about the proposed method, e.g., what's the choice of $\\lambda$, s, how does the authors actually do the two-stage training. what is fixed and what is learned in the second stage? \n\nAdditional Questions:\n- For Figure 1 (c) and (d), why is the L2 norm of the overall gradient larger than both grad1 and grad2?\n- What's the rationale behind the design of memory bank? To be more specific, what's the rationale behind the design of Equation (3), (4)?\n- \"In contrast to the aforementioned strategies, we approach the long-tailed recognition problem by\nanalyzing gradient distortion in long-tailed data\"  How does the proposed method get connected with this statement and differ from other two-stage training algorithms?\n- For equation (6), what's the intuition to use $a_{ji}$ to reweight each norm?\n- For $L_{intra}$, why do the authors choose hinge loss rather than cross entropy? To the best of my knowledge, Hinge loss does not work well in deep learning, esperically with large amoung of classes as the gradient can be vary sparse.\n\n----\npost-rebuttal update\n\nI appreciate the authors for the responses. Some of my concerns have been addressed, so I increased my score. However, I this the current version still lacks critical insights and some justifications are questionable.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "A9lAyKvzMlt", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "**Overview:**\n\nThe paper considers the issue of gradient distortion in long-tailed recognition including shifted gradient direction towards data-rich classes and the enlarged variance introduced by data-poor classes. It proposes to disentangle the data-rich and data-poor classes and train a model via a dual-phase learning process. The experimental results proved the effectiveness of the method.\n \n**Strengths:**\n\nThe observation is interesting and the method is reasonable. The memory retentive loss via graph matching in the second phase makes sense, and the method is well evaluated in four commonly used datasets.\n \n \n**Weaknesses, questions, and suggestions:**\n\n1. One of my main concerns is that the separation of the data-rich and data-poor classes is unnatural because the long-tailed distribution is continuous. The main idea of the method is kind of similar to that of Gidaris et al.[1] as the authors also cited, but in the few-shot setting in their paper, the separation of base and novel classes is more natural and reasonable.\nThe issue of gradient distortion is addressed and is the motivation of the method, however, the variety in gradient does not necessarily lead to worse performance, and even when the data-rich classes are extracted for training in Phase I, the imbalance and shifting to the relatively rich classes still exists while being less informative because of fewer classes and less data. The trade-off is not easy to balance and could be sensitive to the disentanglement points as in Figure 3, where 294 and 864 are carefully selected for Places-LT and ImageNer-LT This introduces limitation to the generality of the method.\n \n2. Are the formations of equation (2) and (5) presented correctly? Maybe the authors intend to present \\sum_i{exp{...}} as the denominator?\n \n3. The presentation and clarity need to be improved. For example,\n\n- The caption for Figure 1 is too long and could be more concise, e.g., to use 'titles' to distinguish CIFAR100 and CIFAR100-LT; to use grad_rich / grad_poor or something as the legend and save any extra explanation. Some of the details can be embedded in the main paper, and I’m not sure how are the gradient statistics calculated, layer-by-layer, or directly use the average of the whole network. It would be better to see it clarified in the paper or appendix, and sorry if I missed it.\n\n- There are double y-axis in Figure 2 and 3, but there is no legend or caption showing each plot is assigned to which axis.\n\n- What does 'extended parameters' mean in the experiment section?\n \n4. The performance improvement seems not significant compared with previous works, e.g. ImageNet-LT on ResNet-10 and iNaturalist.\n \n5. I’m not sure I’ve correctly comprehended the construction of the exemplar memory bank and the reason why use s(c_j + \\Delta, X_1) to search for the new entry. More explanation for Equation (4) and a clearer presentation of the algorithm (e.g. use an Algorithm module) is preferred.\n\n[1] Gidaris et al., Dynamic few-shot visual learning without forgetting, in CVPR 2018\n\n**Post-Rebuttal**\n\nAfter reading the rebuttal and other reviewers' comments, I am actually on the fence for this submission. On the one hand, it provides several interesting observations about the phase transition in long-tailed recognition, which would be valuable to the community. On the other hand, its experimental evaluation needs to be strengthened. The authors are encouraged to include more many-shot/medium-shot/few-shot analysis across the dual phases.\n\nTherefore, I upgrade my score to 6 (marginally above acceptance threshold).\n\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper received mixed reviews. Reviewers were concerned about the clarity of the presentation, including both the analyses of gradients and the approach. Some reviewers also suggested improvements to the experiments. The two reviewers who gave a rating of 6 liked the gradient perspective to the long-tailed recognition, but their concerns seemed to overshadow their excitement. AC suggested authors improving the paper, especially its clarity and empirical part, following Reviewers' comments and submitting the paper elsewhere. 
Decision: Reject