Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 3: Clear rejection
review_content: # Summary
This paper works on long-tailed classification. The authors conducted an analysis and claimed that the difference of gradients computed on the head and tail classes plays an important role in the performance drop. The authors then proposed a two-stage approach to first train on the head classes and then train on the tail classes in an incremental learning fashion. The proposed algorithm achieved better performance than existing methods on benchmark datasets.

# Strengths

- The analysis of the gradients between the head and tail classes seems to be novel.

- The proposed methods achieve good performance on benchmark datasets.  


# Weaknesses

1. The paper lacks a detailed description of how the gradient analysis is conducted. For example, is the gradient computed at some layers or all the layers? How is the variance computed? Is it possible that the relatively large variance is due to a smaller sample size of tail classes? Why does the gradient computed on both types of classes have a larger norm? I would suggest that the authors provided some equations. Moreover, given just Figure 1 and the difference to balanced training, it is still unclear if such a gradient difference really leads to poor long-tailed performance. An analysis of the relationship between gradients and the classification performance will make the paper stronger. 

2. The idea of learning the classifier with two-phase has been proposed (Kaidi Cao et al., 2019; Kang et al., 2020). The most similar ones to the paper are [A, B, C], in which the first stage only considers data-reach classes. The authors, however, failed to identify and discuss them.

[A] Wang et al., Frustratingly Simple Few-Shot Object Detection, ICML 2020

[B] Zhang et al., A Study on Action Detection in the Wild, arxiv 2019

[C] Cui et al., Large Scale Fine-Grained Categorization and Domain-Specific Transfer Learning, CVPR 2018

Memories are also used in (Liu et al., 2019) but the authors did not discuss it.


3. Details and motivations of the proposed methods are not clear, making it very hard to understand the proposed algorithm.

- It is unclear why the memory bank is needed. Many existing works, including [A, B, C], have proposed to simply train the second stage with all classes, with the subsampled class-balanced data, and shows promising results. Some of them even "freeze" the features but only train the classifier in the second stage.

- The way the authors tackle the second-stage training is similar to incremental learning, but the authors failed to discuss those works. One example is [D].

[D] Rebuffi et al., iCaRL: Incremental Classifier and Representation Learning, CVPR 2017.

- The design of Eq. (2), (3), (4) are not well-described and justified. Why do we need a new state? Why do we need Delta? The computation of Delta seems wrong: c_j + c_j -z_{k+1} = 2*c_j – z_{k+1}? Moreover, what does the state mean here?

- It is unclear why we need graphs. It is unclear what “z” stands for in Eq. (5), (6), (7). It is unclear how a_{ij} is computed.

- The motivations and description of the intra-class loss in Eq. (8) is unclear.

- I would suggest that the authors provide a figure for their algorithm architecture and pipelines.


4. Experiments and analysis:

- Back to my comment in 1., there is no analysis to further justify that gradient distortion is really the cause of the poor longtailed performance, and there is no analysis if the proposed algorithm resolves it. There is no analysis if graphs and memory banks are really needed. 

- It is unclear what Ours(Plain) refers to. What is the model without asynchronous modeling?

# Minor
I would suggest that the authors replace "asynchronous" with other terms. The proposed algorithm is just a two-stage algorithm. There is no component of distributed learning and communications that require synchronization or not.

# Justification
While the proposed algorithm achieves promising results, the algorithm is not written, well-described, and motivated. It is also unclear if the gradient distortion is really the cause of poor long-tailed performance. I thus give a score of 3.

-------------------------- Post-rebuttal ----------------------------

I read the authors' rebuttal and I appreciate their efforts. I would suggest that the authors incorporate those clarifications into their manuscript. I would also suggest that the authors re-motivate their paper and modify their approach section.

In terms of the discussions to related work, I do think [A, B, C] is about long-tailed recognition/detection, not few-shot learning. For instance, [A] works on LVIS, a long-tailed object detection dataset; [C] works on iNaturalist, which is clearly long-tailed. [B]'s Fig 1 clearly shows that the problem is long-tailed. I'm surprised that the authors simply said that [A, B, C,] works on different problems but did not intend to discuss the similarity in methodologies.

There are some very important questions not addressed yet, specifically, my comments 3 and 4: There is no analysis if the proposed algorithm resolves gradient distortion. There is no analysis if graphs and memory banks are really needed. 

I also read other reviewers' comments and I agree with R4 that the current version still lacks critical insights and some justifications are questionable.

Given these, I would keep my initial score unchanged.
review_title: This paper works on long-tailed classification. The authors conducted an analysis and claimed that the difference of gradients computed on the head and tail classes plays an important role in the performance drop. The authors then proposed a two-stage approach to first train on the head classes and then train on the tail classes in an incremental learning fashion. The proposed methods, however, are not well-motivated and described. More details and clarifications are needed.

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper proposes an interesting view to analyze the long-tailed problem. It states that the gradients are dominated by the head classes so that the tail classes perform poorly. From this observation, the authors propose a dual-phase approach that first train $W_r, W_c^1$ with only head-class data, and extend to train $W_r, W_c$ with tail-class data and the constructed exemplar memory bank for head classes with a newly proposed memory retentive loss.

#### Pros:
1. This paper is well organized and written.
2. Analyzing through the gradient seems to be an interesting view, which could bring insights into the long-tailed problem.

#### Some questions and concerns:
1. In Fig.1 (c)(d), despite the norm variance, I don't understand why the "norm of grad" is larger than "the norm of grad1" and "the norm of grad2". They are all norms, right? So why the grad (a combination of grad1 and grad2 in my opinion) becomes larger?
2. This dual-phase approach somehow seems similar to [this paper](https://openaccess.thecvf.com/content_CVPR_2020/papers/Li_Overcoming_Classifier_Imbalance_for_Long-Tail_Object_Detection_With_Balanced_Group_CVPR_2020_paper.pdf), which actually groups categories that own a similar number of training instances and train the classifier separately. And this paper divides all categories into 2 groups (head and tail), but with more complicated operations.
3. The authors conduct experiments on almost all common long-tailed classification datasets, which is great. But the improvements seem to be very limited on Places-LT, ImageNet-LT, iNaturalist18 comparing with recent works. It seems that it only works well on small datasets like CIFAR, which is somehow weak.
4. About the training schedule, in the Appendix.Implementation Details, it states that the number of training epochs is 200. It means that we train both 200 epochs for both phase1 and phase2 right? Though $X_1$ and $X_2$ are not overlapped, the exemplar bank will be trained twice. How much will it cost? How is the cost comparing with re-sampling approaches?
review_title: Good view, but limited improvement.

final_score: rating: 5: Marginally below acceptance threshold
review_content: Summary:

- This paper proposes a two-stage learning algorithm to address imbalanced datasets. Only data-rich classes is used in the first representation learning stage. In the second stage,  an exemplar memory bank with graph matching is used together with standard classification. Experimental results on benchmarks highlight the effectiveness of the proposed method.
 
Pros:
- The total experiments conducted are thorough and satisfactory.

Cons:
- The presentation of this paper can be improved. For example, the notation in Figure 1 is very hard to parse. The writing could be improved as well. There's a chance that I didn't understand the paper so I just list all my concerns in questions.
- In the implementation detail section, there's only implementation details shared with the baseline. There's no detail about the proposed method, e.g., what's the choice of $\lambda$, s, how does the authors actually do the two-stage training. what is fixed and what is learned in the second stage? 

Additional Questions:
- For Figure 1 (c) and (d), why is the L2 norm of the overall gradient larger than both grad1 and grad2?
- What's the rationale behind the design of memory bank? To be more specific, what's the rationale behind the design of Equation (3), (4)?
- "In contrast to the aforementioned strategies, we approach the long-tailed recognition problem by
analyzing gradient distortion in long-tailed data"  How does the proposed method get connected with this statement and differ from other two-stage training algorithms?
- For equation (6), what's the intuition to use $a_{ji}$ to reweight each norm?
- For $L_{intra}$, why do the authors choose hinge loss rather than cross entropy? To the best of my knowledge, Hinge loss does not work well in deep learning, esperically with large amoung of classes as the gradient can be vary sparse.

----
post-rebuttal update

I appreciate the authors for the responses. Some of my concerns have been addressed, so I increased my score. However, I this the current version still lacks critical insights and some justifications are questionable.

review_title: Unsatisfactory writing with important missing details

final_score: rating: 6: Marginally above acceptance threshold
review_content: **Overview:**

The paper considers the issue of gradient distortion in long-tailed recognition including shifted gradient direction towards data-rich classes and the enlarged variance introduced by data-poor classes. It proposes to disentangle the data-rich and data-poor classes and train a model via a dual-phase learning process. The experimental results proved the effectiveness of the method.
 
**Strengths:**

The observation is interesting and the method is reasonable. The memory retentive loss via graph matching in the second phase makes sense, and the method is well evaluated in four commonly used datasets.
 
 
**Weaknesses, questions, and suggestions:**

1. One of my main concerns is that the separation of the data-rich and data-poor classes is unnatural because the long-tailed distribution is continuous. The main idea of the method is kind of similar to that of Gidaris et al.[1] as the authors also cited, but in the few-shot setting in their paper, the separation of base and novel classes is more natural and reasonable.
The issue of gradient distortion is addressed and is the motivation of the method, however, the variety in gradient does not necessarily lead to worse performance, and even when the data-rich classes are extracted for training in Phase I, the imbalance and shifting to the relatively rich classes still exists while being less informative because of fewer classes and less data. The trade-off is not easy to balance and could be sensitive to the disentanglement points as in Figure 3, where 294 and 864 are carefully selected for Places-LT and ImageNer-LT This introduces limitation to the generality of the method.
 
2. Are the formations of equation (2) and (5) presented correctly? Maybe the authors intend to present \sum_i{exp{...}} as the denominator?
 
3. The presentation and clarity need to be improved. For example,

- The caption for Figure 1 is too long and could be more concise, e.g., to use 'titles' to distinguish CIFAR100 and CIFAR100-LT; to use grad_rich / grad_poor or something as the legend and save any extra explanation. Some of the details can be embedded in the main paper, and I’m not sure how are the gradient statistics calculated, layer-by-layer, or directly use the average of the whole network. It would be better to see it clarified in the paper or appendix, and sorry if I missed it.

- There are double y-axis in Figure 2 and 3, but there is no legend or caption showing each plot is assigned to which axis.

- What does 'extended parameters' mean in the experiment section?
 
4. The performance improvement seems not significant compared with previous works, e.g. ImageNet-LT on ResNet-10 and iNaturalist.
 
5. I’m not sure I’ve correctly comprehended the construction of the exemplar memory bank and the reason why use s(c_j + \Delta, X_1) to search for the new entry. More explanation for Equation (4) and a clearer presentation of the algorithm (e.g. use an Algorithm module) is preferred.

[1] Gidaris et al., Dynamic few-shot visual learning without forgetting, in CVPR 2018

**Post-Rebuttal**

After reading the rebuttal and other reviewers' comments, I am actually on the fence for this submission. On the one hand, it provides several interesting observations about the phase transition in long-tailed recognition, which would be valuable to the community. On the other hand, its experimental evaluation needs to be strengthened. The authors are encouraged to include more many-shot/medium-shot/few-shot analysis across the dual phases.

Therefore, I upgrade my score to 6 (marginally above acceptance threshold).


review_title: Official Blind Review #1

Meta-review: The paper received mixed reviews. Reviewers were concerned about the clarity of the presentation, including both the analyses of gradients and the approach. Some reviewers also suggested improvements to the experiments. The two reviewers who gave a rating of 6 liked the gradient perspective to the long-tailed recognition, but their concerns seemed to overshadow their excitement. AC suggested authors improving the paper, especially its clarity and empirical part, following Reviewers' comments and submitting the paper elsewhere. 
Decision: Reject