Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: This paper proposes a method termed RoutIng Diverse Experts (RIDE) for reducing both the bias and the variance of a long-tailed classifier. Specifically, RIDE consists of three crucial components: 1) a shared architecture for multiple experts; 2) a distribution-aware diversity loss that encourages more diverse decisions for classes with fewer training instances; 3) an expert routing module that dynamically assigns more ambiguous instances to additional experts. Experiments are conducted on three long-tailed benchmark datasets, i.e., CIFAR100-LT, ImageNet-LT, and iNaturalist. Satisfactory classification results of long-tailed visual recognition are observed.

Paper strengths:
- The problem, i.e., long-tailed visual recognition, is practical, important and challenging in computer vision and deserves further studies.
- The proposed method has good motivations and sounds reasonable.
- The experimental results of the proposed RIDE are significantly better than the results of previous work, which shows the effectiveness of RIDE.
- The paper is well written and easy to follow.
- Analyses and ablation studies are sufficient and could bring new insights of long-tailed recognition.

Paper weaknesses:
- There are typos and grammar mistakes in this paper. For example, in the caption of Fig. 2, "ImaeNet-LT and BBN" should be "ImageNet-LT and BBN". The authors should carefully proofread the final version.
- Some references have inconsistent reference formats, e.g, "In Proceedings of the IEEE conference on computer vision
and pattern recognition" of [Xie et al., 2017] vs. "In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition" of [Zhou et al., 2020] vs. "CVPR09" of [Deng et al., 2009]. In addition, some other references lack detailed information, e.g., [He and Garcia, 2009] lacks page, volume information. Some references also have capital issues, e.g., "The inaturalist species classification" -> "The iNaturalist species classification" of [Van Horn et al., 2018], and "Bbn" -> "BBN" of [Zhou et al., 2020].
review_title: Official Blind Review #1

final_score: rating: 7: Good paper, accept
review_content: This paper proposed a simple but effective method which significantly outperforms the state-of-the-art method by 5% to 7%. The experiments are adequate and rigorous. Besides these , the writing is clear and easy to understand. Totally, this paper is a good work
Pros:
1. The writing is clear.
2. The view of bias and variance is novel and interesting.
3. The experiments are adequate and rigorous.
4. The performance of proposed method is very strong.
5. The proposed method is simple, effective and general.
Cons:
1. The procedure of test is not clear enough. For example, how to turn on/off an expert? By a threshold?
2. The description of the part about shared, indepent, and recuded extractor is not clear enough.
review_title: This paper proposed a simple but effective method to solve long-tailed recognition. It finds out that current methods suffer high bias and variance. To relieve this problem, it propose to ensemble several experts to make predictions. It proposes a diversiy loss to guarantee the diversity among experts and learn an expert assignment module to turn on/off an expert when predicting. The proposed method of this paper is general and significantly outperforms the state-of-the-art method by 5% to 7%.

final_score: rating: 7: Good paper, accept
review_content: The majority of feature extraction backbone is shared among different agents and the classifiers of experts are trained with both classification loss and proposed distribution-aware diversity loss. For the second stage, an expert assignment module is trained to re-weight the expert decisions. The whole paper is generally well-organized. However, there are some technical issues authors should further address:
1) In the introduction, the authors measure the mean accuracy, model bias and model variance by randomly sampling the Cifar100 a few times, and reported different methods in Fig1. The details of how such bias and variance is computed are not given. For example, it is stated the bias also measures the accuracy.
2) For the motivation of distribution-aware diversity loss, if the same input batch is observed by all the experts, why is it 'distribution aware'? Also, why requires the diversity among experts if setting the lower temperature to tail classes is to encourage the divergent prediction, how both diversity and divergent exists simultaneously?
3) Is the input batch randomly sampled from the entire training set?
4) For experiments on ImageNet-LT and iNaturalist, is the RIDE combined with the regular CE loss or other methods?
5) Can authors demonstrate some details regarding the performance of expert assignment module during the test, as this is the core module in the proposed framework? For example, for each split (many, low) how the different expert behaves?

[Post Rebuttal Comments] Authors have done a good job for addressing my concerns, especially the additional ablation studies regarding the performance of the expert assignment module. I'm updating my score accordingly for recommending acceptance.

review_title: In this paper, authors propose a stage-wise multi-expert system for long-tailed recognition problem.

final_score: rating: 7: Good paper, accept
review_content: ##########################################################################

Summary:

This paper proposes a Routing Diverse Experts (RIDE) framework to solve the long-tailed classification problem. It has 1) a shared low-level feature extractor and multiple expert classifiers, 2) a distribution-aware diversity loss to encourage experts learning different classification strategies, 3) an expert routing module that dynamically selects a subset of experts for each test instance to make a joint decision. This paper firstly increases the performances on all three splits (many-/med-/few-shot), while most of the existing methods have to sacrifice the head for tail improvements.

##########################################################################

Pros:

+ The proposed Bias & Variance metrics provide a better understanding of why previous work almost always improves the tail accuracy but decreases head performance(trade-off between bias and var).
+ To the best of my knowledge, this paper firstly increases the performances on both head and tail.
+ The proposed RIDE framework can be applied to a variety of long-tailed recognition methods.
+ Although RIDE includes multiple experts, each expert only has 1/4 channels, which allows overall computation overhead even decreases when using 2 experts.

##########################################################################

Concerns:

I'm overall satisfied with this paper, but I do have a few concerns:
- The proposed RIDE does look like an ensemble method to me. Although you provide a comparison with the ensemble method in the appendix, I noticed that 2 ensembles will achieve 45.0 accuracies on CIFAR100-LT, which is exactly the same as 2 experts + L_diversity in Table 4. Although RIDE can achieve better performance when the number of ensemble models/experts is increased to 3 or 4, does it means that diversity loss only works on a larger number of experts?

- The expert routing/assignment module dynamically routes cascaded experts, and all the passed experts before the stop will jointly make a decision. How to make sure the early experts are the more robust ones? Since experts have already been frozen when the routing module is trained, and the diversity loss won't force the early experts to be more robust. There is no guarantee that the involved early experts are more useful than the skipped ones.

##########################################################################

Questions during the rebuttal period: 

Please address and clarify the concerns above. Besides, I think the experts in RIDE may learn to be biased toward different categories due to the diversity loss, and the routing module works as an outlier detector which only stops when it finds a proper expert biased toward the underline ground-truth labels. Is it another way to understand RIDE? or do I misunderstand some parts of our paper?

##########################################################################

Reasons for score: 

Overall, I vote for accepting. I think this paper shows a new direction for long-tailed classification that we don't need to sacrifice the head performance for a balanced classifier anymore. 
review_title: A good paper that shows a new direction for long-tailed classification.

Meta-review: The paper addresses the important problem of classification with unbalanced semantic classes. The key idea is a two-stage process that first learns a representation under various distributions (experts), then assign a cascade of experts to hard samples, whose predictions are combined. The approach can be added on top of various backbone networks. Experiments are systematic and extensive, showing improvement on three standard benchmarks for this task.

All four reviewers recommended accept.  The paper extends a recent research direction of learning "balanced" representations, followed by distribution-aware experts. This general approach could have wider impact on architectures designed for out-of-distribution and low-shot learning.  The authors should update the final paper based on their answers and on reviewer feedback. We also encourage authors to make their code available. 
Decision: Accept (Spotlight)