Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The aim of this paper is to develop a mixture-of-expert (MOE) model to solve the test-agnostic long-tailed recognition problem, where the test class distribution may follow a uniform, forward or backward long-tailed distribution. The method is developed on the basis of RIDE with three experts and consists of two strategies. At training time, SADE utilizes skill-diverse expert learning strategies that require each expert to handle a different class distribution in order to solve distribution-agnostic long-tailed recognition problems. At test time, SADE utilizes a test-time expert aggregation strategy, which is based on a self-supervised learning approach, to determine expert aggregation methods that handle unknown class distributions. Experiments were conducted on various test-time training strategies dealing with class distribution transfer. SADE achieves state-of-the-art performance on multiple long-tailed datasets, including CIFAR100-LT, ImageNet-LT, Places-LT and iNaturalist 2018.
strengths_and_weaknesses: Strengths:
- Evaluating on Forward-LT and Uniform test-class distributions can help us better understanding the performance of various long-tailed algorithms on different testing scenarios. It is shown that SADE can achieve SOTA performance on all these testing distributions.
- This paper is well-written and easy to follow. 
- Authors conducted thorough experiments on various benchmarks and testing scenarios and achieved consistent performance improvements on all these benchmarks. 

For weaknesses, my main concerns are twofold:
- The biggest improvement comes from test data with Backward-LT distributions, however, I find it hard to believe that backward class distributions are common in real-world applications. The many-shot (few-shot) classes in the training data are often the many-shot (few-shot) classes in the testing data as well. Thus, Forward-LT and Uniform test class distributions make more sense, however, SADE achieves marginal improvements in these two testing cases. 
- The test-time self-supervised aggregation strategy requires the model to see all test data (unlabeled) before deployment, however, in real-world applications we more commonly see only 1 test image. This part is similar to my first concern, which is whether this setup is a practical setup that can be used in real-world applications.
questions: I have a few questions on the fairness of the result comparisons and the setting of the test-time adaptation:
- When comparing with the current SOTA method RIDE [46], the results of RIDE (https://github.com/frank-xwang/RIDE-LongTailRecognition/blob/main/MODEL_ZOO.md) is actually achieved by training the model for 100 epochs. However, the results reported in the paper is achieved by training the model for 200 epochs. Therefore, I am concerned about the fairness of the comparisons. 
- For test-time adaptation, have you tried using MC-Dropout [1] as a metric for expert aggregation? You can get the uncertainty of each expert and decide the weight of each expert based on the uncertainty. Does it save you from using all test data for expert aggregation? I think it might help if MC-Dropout is used with a strong data augmentation to produce different inputs.

[1] Gal, Yarin, and Zoubin Ghahramani. "Dropout as a bayesian approximation: Representing model uncertainty in deep learning." international conference on machine learning. PMLR, 2016.
limitations: Yes, the authors have adequately addressed the limitations and potential negative societal impact of their work. The long-tail recognition algorithm aims to alleviate the problem of ignoring underrepresented minorities, which is necessary to obtain an unbiased model and facilitate the development of CNN models for social justice.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper extends the conventional long-tailed learning to the "test-agnostic one", in which the model trained on a long-tailed class distribution should generalize to arbitrary testing distribution not necessarily being uniform. To handle such a problem, this paper proposes a novel method consisting of two modules: (1) diverse experts with different class expertise and (2) a self-supervised test-time weighting strategy that adaptively aggregates the experts to tackle unknown testing distribution. Extensive experiments have shown the efficacy of the proposed method on handling arbitrary class distributions in testing.

strengths_and_weaknesses: ## Strengths
- The proposed test-agnostic setting is challenging and of great practical significance.
- The paper is well-written and easy to follow.
- The proposed test-time aggregation strategy is interesting and has proven to be useful.
- The experiments and ablation studies are comprehensive and convincing.

## Weaknesses
- The weakness is mainly focused on the computation complexity. As mentioned in the paper, the three experts are independent in ResNet blocks (later stages) and fully-connected layers. Though it seems tolerable since it is a quite challenging problem after all, have the authors explored the trade-off between accuracy and complexity? For instance, is it a near-linear relationship between higher accuracy and experts with fewer shared modules? I would like to see how far can it go at the two extreme points: (1) when nothing is shared between experts and (2) everything is shared except the fully-connected layers.

questions: Line 762: how exactly is the expertise-guided loss functions changed to suit different types of distributions?

limitations: The limitations are carefully discussed in the paper, which mainly encompass the extensibility to different tasks and the model complexity of the proposed method.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper studies an interesting problem in long-tailed recognition, i.e.,  the training class distribution is long-tailed while the test class distribution is agnostic rather than a uniform distribution as assumed in previous works. To deal with the problem, this paper proposes a new approach that outperforms existing methods in both vanilla and test-agnostic long-tailed recognition settings.
strengths_and_weaknesses: ### Strength

1. The studies problem is interesting and under-explored.
2. Extensive experiments are conducted to justify the effectiveness of the proposed method.
3. The writing is clear and easy to understand.

## Weakness

1. The technical significance is not enough. Specifically, there are two aspects. First, **the** **skill-diverse expert learning** does not make new contribution to the field because multiple experts have been used in many existing literature, e.g., [1-3]. Moreover, the idea of aggregating multiple diverse models was also explored in [3] though the studied problem is different. Second, **the** **Test-time Self-supervised Aggregation** simply a re-weighting of three models. The key contribution might be the prediction stability maximization, but optimizing this objective does not ensure to obtain the optimal weights. 
2. This paper only considers the transductive setting where the entire test data are accessible at once. However, in many applications, the assumption is not satisfied.
3. This paper incurs more computational cost than previous methods. The Test-time Self-supervised Aggregation has to be performed at each test time.
4. nitpick: some bold numbers in Table 9 are not the best results.

[1] Learning from multiple experts: Self-paced knowledge distillation for long-tailed classification

[2] Long-tailed recognition by routing diverse distribution-aware experts

[3] Cross-Domain Empirical Risk Minimization for Unbiased Long-tailed Classification
questions: 1. Why does this paper choose Balanced softmax and a variant logit adjustment loss?
2. How should we decide the number of experts?
3. Is the classifier re-weighting strategy optimal? And can the Test-time Self-supervised Aggregation learn optimal weights?
4. Why should we use the proposed method instead of existing long-tailed methods such as RIDE? The performance improvement is not significant and the proposed method incurs additional computational costs.
limitations: Yes, the authors have adequately addressed the limitations and potential negative societal impact of their work.
review_title: null

Meta-review: metareview: The reviewers agreed the paper provides some nice insights into tackling the difficult and under-explored problem of test-agnostic long-tailed recognition. The reviewers appreciated the thorough experiments and ablations provided. The author response sufficiently addressed the key concerns the reviewers had.
confidence: Certain
Decision: Accept