Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: In this paper, the authors study the problem of long-tailed visual classification. To be specific, they propose obtaining a stochastic model for representations through stochastic weight averaging (SWA) and retraining a robust classifier by distilling these representations.
strength_and_weaknesses: trengths:
+ A novel approach to obtaining better representations / decision boundaries for long-tailed visual recognition.
+ Positive results over the baselines.

Weaknesses:

1. The paper identifies the main source of imbalanced learning as the decision boundary. However, the recent view identifies the culprit as the representation learning stage and suggests that features of the under-represented class are somehow sub-optimal, which leads to worse decision boundaries for the under-represented class. I suggest the authors to tone down "the main bottleneck is the decision boundary" claim.

2. Fig 2: It is not clear why the dispersion of models along the training trajectory should be correlated with NLL. 

3. The proposed approach is computationally too expensive. One can easily obtain a set of alternative representations through dropout &  perform distillation. Would this not be cheaper? A discussion on complexity is necessary. 

4. The experimental evaluation needs improvement.

4.1. The SOTA comparison in Table 4 seems to have unfair aspects: In some cases, less training epochs are used with the compared baseline (e.g. the comparison against KCL) and more are used in some others (Liu et al. (2021)). 

4.2. The experimental evaluation is missing important comparisons with more recent approaches from 2022. No approach is compared against from 2022.

4.3. The experimental evaluation is missing comparisons on CIFAR10LT & CIFAR100LT. This is important to get a better feeling about how the method performs in a small-scale problem (CIFAR10LT) and a middle/large-scale problem such as CIFAR100LT. In fact, CIFAR100LT is surprisingly challenging for some problems and you can obtain unexpected results.


Minor comments:
- "Confirming that SWA can benefit to long-tailed classification" => "Confirming that SWA can provide benefit to long-tailed classification".
- Eq 2: "x" is forgotten after the dot.
- "parameter θ" => "parameters θ".
- "parameter ϕ" => "parameters ϕ".
- Eq 4: A minor detail but this should include a step size.
- Eq 6: Θ′ => not introduced.
- Below Eq 9: "predictive uncertainty of x," => Predictive or epistemic? They are not the same. Two lines above Eq 9, you said epistemic.
- Figure 2: "asdfsadf." => Agreed :)
- Fig 3: ERM not introduced. 
clarity,_quality,_novelty_and_reproducibility: Generally clear. Good quality. Novel approach. Reproducible.
summary_of_the_review: Novel approach but there are concerns about the evaluation.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper focuses on improving features to be more generalizable to achieve better performance in Long-Tailed Recognition (LTR). In particular, the paper adopts Stochastic Weight Averaging (SWA) for improving the generalization of deep neural networks. Moreover, the paper proposes a classifier re-training algorithm based on the learned representation. The algorithm uses a Gaussian perturbed SWA, and a self-distillation strategy that can harness the diverse stochastic representations based on uncertainty estimates to build more robust classifiers. Experiments on two benchmarks show that the proposed method improves upon previous methods in terms of both prediction accuracy and uncertainty estimation.
strength_and_weaknesses: Strength
- Studying how generalized features improves LTR is interesting.
- Studying uncertainty estimation in the context of LTR is interesting.


Below are some weaknesses.

- While "the success of decoupling naturally motivates obtaining more informative representations from which the classifier re-training can benefit", using SWA seems like less well-motivated. Instead, this motivation should suggest learn more generalized features using more data sources (e.g., external data) and more general loss functions (e.g., contrastive learning). I would like the authors to discuss this further, otherwise the paper just seems incremental especially when it states "to the best of our knowledge, it has never been explored for long-tailed classification problems.", and in the conclusion "to the best of our knowledge, this is the first attempt to introduce SWA into long-tailed learning."

- While the paper is motivated to learn "generalizable" features, a recent work [R1] shows that learning a backbone which has "balanced-norm" filters significantly boosts LTR performance. Moreover, [R1] simply tunes weight decay to regularize backbone's norms and achieves the state-of-the-art. This is an open questions -- whether to learn "balanced backbone" or more "generalized backbone" to achieve better LTR performance? Can authors discuss and show insights?

[R1] Alshammari, et al., "Long-Tailed Recognition via Weight Balancing", CVPR 2022

- While it is interesting to study uncertainty estimation in the context of LTR, the paper does not sufficiently analyze the results. For example, it is unclear how good uncertainty estimation for tail classes vs. head classes. Authors are encouraged to dive into this part and provide insightful discussions.

typos
- Figure 2 caption: "asdfsadf".
clarity,_quality,_novelty_and_reproducibility: The clarity, quality, novelty and reproducibility are okay. The novelty is incremental as the paper seems to apply Stochastic Weight Averaging to improve the generalizability of learned features to improve long-tailed recognition performance. However, the performance improvement is marginal especially given the large computation cost. Code is provided but it is unclear whether running it reproduces the reported results. I'd suggest attaching something like jupyter notebook for better demonstration; authors should not expect reviewers to run their code smoothly.
summary_of_the_review: The paper focuses on using Stochastic Weight Averaging (SWA) to improve features used for long-tailed recognition (LTR). The motivation is to learn more generalizable features for LTR. SWA is an existing method that learns more general features and the paper claims that "to the best of our knowledge, this is the first attempt to introduce SWA into long-tailed learning". However, SWA seems weak as it achieves only marginal improvement given the large added computation burden. Therefore, the rating of the paper is "5: marginally below the acceptance threshold"
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper aims to solve the fundamental problem --- long-tailed recognition. It analyzes SWA/SWAG with long-tailed data. 
           
（1) Classifier re-training can make better use of SWA for long-tailed data.    
  (2) The positive correlation between NLL and dispersion is observed. Based on this phenomenon, a self-distillation strategy is developed to reduce predictive variance.   
  (3) Experiments on ImageNet-LT and iNaturalist 2018 show some improvements.    

strength_and_weaknesses: Strength    

(1) The paper writes clear and is easy to follow.    

(2) The method should be general and is expected to work well with previous methods.   

Weakness  

(1) This paper use SWA to enhance the representation ability of deep models and a distillation method is proposed to reduce predictive variance. However, comparisons with most related work are missed.     
      I. The state-of-the-art methods, like PaCo [1].     
      II. distillation-based methods, like CBD [2].     


[1] Parametric Contrastive Learning. ICCV 2021.     
[2] Class-Balanced Distillation for Long-Tailed Visual Recognition. BMVC 2021.     

(2) About the distillation objective.  Why do the teacher prediction probabilities obey the Dirichlet distribution?    

(3) As shown in Figure2, as the number of stochastic representations increases, the changes of ECE and NLL are limited.  
clarity,_quality,_novelty_and_reproducibility: The code is provided in the supplementary file.
summary_of_the_review: The paper investigates SWA/SWAG for long-tailed data. However, some important comparisons are missed.    
If the concerns are addressed, I'm very glad to raise my score.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper studies the effect of stochastic representations in long-tail recognition scenarios and designs a self-distillation method to adapt stochastic representations to long-tail learning. 
strength_and_weaknesses: I think this paper is very well-written and well-organized. The preliminary experiments on the effect of stochastic representation (Stochastic Weight Average, SWA) on decoupled long-tail learning methods are detailed and intuitive. They start with testing directly applying stochastic representations to single stage and two stage training and acquired results that promote further development of SWA on two-stage/decouple methods. The idea of using Gussian SWA (SWAG) as a measurement of prediction certainty, although not new, but directly connected to the intuition that tail class predictions are usually less certain. A robust certainty measurement is always a challenge in long-tail recognition, and I think using SWAG is an interesting proposition. The discussion and explanation of the limitations of direct usage of SWAG are clear, because it needs multiple runs to get a stochastic distribution. And this leads to the proposition of the proposed self-distillation method, which has multiple teachers during training for a SWAG distribution, and has a student, that learns from the stochastic teachers. Again, this multi-teacher single student framework is not new (e.g., mean teachers in semi-supervised learning). But it aligns well with the challenges mentioned in direct usage of SWAG. The experiments are extensive. Even the state-of-the-art multi-expert methods like  RIDE are compared. However, the improvements, especially the improvements over multi-expert methods, are limited (<1% on average). The biggest drawback of this method is that it is relatively too complicated (multiple teach networks need to be trained for SWAG to work), and, thus, not compatible with existing multi-expert models, which leads to limited improvements. The proposed multi-teacher single student framework itself is already a multi-expertish model. 
clarity,_quality,_novelty_and_reproducibility: This paper is clearly written, code is not yet available. 
summary_of_the_review: This paper overall is a good contribution to the community for its detailed experiments on the effect of stochastic representations on long-tailed learning, and the idea of using stochastic representations as a measurement of confidence. However, the proposed method is relatively complicated and yields limited improvements over state-of-the-art methods.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: 
The submission proposes an approach for long-tailed classification. Briefly, the proposed approach constructs a moving average of parameters from a periodic sample of parameter trajectory during training (SWA, Izmailov et al., 2018). The averaged parameters are used to construct the representation (backbone network). This is then followed by a classifier re-training step based on stochastic representation. The proposal is evaluated on standard long-tailed classification benchmark data (i.e., CIFAR10-LT, CIFAR100-LT, ImageNet-LT, iNaturalist-LT).

All reviewers agree that the submission is well written and has sufficient details for reproducibility. 
Main comments from the four reviewers are

[+] Positive results (in terms of accuracy) over baselines (1FXP). 

[+] Bringing in uncertainty estimation to the context of long-tailed learning is interesting (fqWB).

[-] While the authors added new results to Appendix B.4 in response to this comment, the added results did not sufficiently address the concern (DCKB). The concern was that improvements obtained from the proposed method are limited.

[-] Overall gains shown in experiments appear to be minor, especially when compared to the added computational burden (fqWB, DCKB, eyhi). While the authors have commented that there is no additional training cost in terms of compute, there is a concern on the increase in memory cost.

I tend to agree with clarification from the authors that the proposed approach only requires storing two additional copies of model parameters (for the first and the second moments), and thus the increase in memory cost should not be significant. Regarding minor improvements, firstly the reviewers mainly focused on accuracy gains, which were positive (albeit, small). However, more importantly, the proposed method shows good gains on ECE (expected calibration error), a point that seems to be missing in the discussion among reviewers. To my knowledge calibration is not a common aspect to consider under long-tailed learning settings because of the difficulty of the problem. Often, only balanced accuracy is of interest. Proposing a method that touches upon calibration may open up a new direction. The improvement on ECE also aligns with the proposed method which brings uncertainty estimation into the formulation. With these merits, I think the community will benefit from this work.

Recommendation: accept.

justification_for_why_not_higher_score: Concerns on minor accuracy improvement were the main blocker.
justification_for_why_not_lower_score: I think the proposal **does** offer improvements in terms of calibration. Also, the idea to incorporate uncertainty estimation in the formulation for long-tailed learning is interesting and unusual (in a positive way).
Decision: Accept: poster