Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: This paper addressed the problem of long tail recognition using a novel mixup technique. The proposed mixup technique (Unimix) considered the class prior and helps to create a more balanced virtual dataset. The author provide theoretical analysis on why Unimix is a better mixup technique. Extensive experiment also demonstrates the performance gain of using Unimix.
main_review: Strength 
a.	The paper is well written and all the figures are well illustrated.
b.	The theoretical analysis is interesting and provide strong support to the proposed method.
c.	The experiment is thorough and cover different aspects of the ablation studies.
d.	The performance is promising

Minor weakness
a.	Although the author somehow shows the performance on head/medium/tail classes using the confusion matrix, it will be nice to provide the number and table in the final version. 

limitations_and_societal_impact: The authors have adequately addressed the limitations and potential negative societal impact of their work.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The authors proposed a new method based on two approaches, UniMix and Bayias, to sample input-output pairs from unbalanced datasets and use them in Mixup. The authors report experiments on CIFAR-10-LT, CIFAR-100-LT, ImageNet-LT, and iNaturalist 2018, showing superior performance in terms of accuracy and calibration scores over a variety of baselines.

While I could get the main idea behind the proposed method, which seems appealing, I had problems understanding all the details, mainly because of a substantial lack of clarity. Sentences are often disconnected which makes it difficult to understand what the authors mean.  My score is borderline but I am open to increase it if the authors can provide a solid rebuttal and substantially improve the paper in terms of clarity.
main_review: Strengths
---------

- The method seems to follow a different rebalancing approach compared to other work (e.g. Remix [1]).
- The empirical results show a substantial improvement in terms of accuracy over other baselines.
- Comparison in terms of ECE scores confirm that the method improves calibration.

Weaknesses
----------

1) The paper lacks in terms of clarity and it is often difficult to understand what the authors mean. The paper needs to be substantially improved from this point of view.

2) One of the main claims of the authors is that the proposed method improves the model calibration w.r.t. Mixup on unbalanced datasets. The plots reported by the authors seem to suggest that this is true, but those plots show the accuracy VS confidence tradeoff, that can be misleading. In Appendix D.2 the authors report quantitative measurements in terms of ECE and MCE. It has been showed that ECE scores suffer of several shortcomings [3]. I think that it would be useful to provide a quantitative comparison also in terms of other metrics like ACE and TACE scores [2], which are more robust than ECE.

3) It is unclear why best performances and calibration are obtained by joining UniMix+Bayias. The authors provided an ablation study in Table 3 but there is not a proper discussion to explain these results.

4) The authros claim (lines 42-48) that one of the main contributions is the introduction of the $\\xi\\text{-Aug}$ notation for Mixup (Section 2). I am struggling to understand where is the novelty, since it seems to me that this is a fairly common notation used to describe Mixup. Could the authors expand on this point?

5) All the experimental results do not report the standard deviation over accuracy. Are the reported results just on a single seed? Please provide additional details regarding this point.

6) There is not a proper discussion of the shortcomings of this method. It seems to me that joining UniMix+Bayias has a higher computational overhead compared to other approaches (e.g. Remix [1]). I would like to see a discussion about this point.


References
----------

[1] Chou, H. P., Chang, S. C., Pan, J. Y., Wei, W., & Juan, D. C. (2020, August). Remix: Rebalanced mixup. In European Conference on Computer Vision (pp. 95-110). Springer, Cham.

[2] Nixon, J., Dusenberry, M. W., Zhang, L., Jerfel, G., & Tran, D. (2019, June). Measuring Calibration in Deep Learning. In CVPR Workshops (Vol. 2, No. 7).

[3] Ashukha, A., Lyzhov, A., Molchanov, D., & Vetrov, D. (2020). Pitfalls of in-domain uncertainty estimation and ensembling in deep learning. arXiv preprint arXiv:2002.06470.
limitations_and_societal_impact: Not reported. This work has minimal societal impact.
review_title: null

final_score: rating: 4: Ok but not good enough - rejection
review_content: summary: The authors propose two models to improve calibration for long-tailed recognition. 1) a Uniform Mixup is proposed, which adopts an advanced mixing factor and sampler in favor of the minority. 2) the authors propose a Bayias loss to compensate for the prior bias. The proposed method achieves good results on CIFAR-LT, ImageNet-LT, and iNaturalist 2018.
main_review: The novelty of this paper is limited:

This work is very similar to previous papers [1] and [2]. Concretely, the proposed Bayias loss (Eq.15) is the same as logit adjustment [1] (Eq. 11 in [1]) for long-tailed recognition. [2] analyze long-tailed recognition from the calibration view. Algorithm 1 in this paper is also the same as [2], i.e., adding mixup in Stage-1 and using re-balanced loss in Stage-2. What's more, the whole paper emphasizes calibration. However, the authors just give visualization results (Fig. 1, Fig. 3, and Fig. 5) to illustrate.  Why not show quantitative results (Expected Calibration Error, Brier Score, and Probabilistic sharpness) in experiment part? A lot of theoretical analysis seems meaningless.


The experiment results are not significant:

Balanced Softmax [3] (experiment results please refer to [4]) can get **45.1%** top-1 accuracy on CIAFR-100-LT, $\rho=100$. While the proposed Bayias loss just gets **43.52%** (**both without mixup**).  Balanced Sofemax [3] (experiment results please refer to [4]) **without mixup** can get **69.8%** top-1 accuracy on iNaturalist 2018. While the proposed Bayias loss **with mixup** just gets **69.15%**. 

For calibration, the proposed method is **overall lower** than MiSLAS [2] in terms of **both Expected Calibration Error and top-1 recognition accuracy** on all long-tailed benchmarks. It is a reasonable and related fair comparison because both two models add mixup operation.


Other questions:

1) Based on Algorithm 1, I don't think the proposed method is E2E. Since it also contains two stages, which is similar to LDAM+DRW (Table 1: LDAM+DRW E2E  [x])

2) Both UniMix Factor and UniMix Sampler generate samples, which is mainly a $\xi$-Aug sample of the tail composite with $x_i$ from the head. I don't think more balanced $\xi$-Aug samples will lead to better model recognition performance.

3) What is the definition of $f$ in Eq. (6)?


Ref:

[1] Long-tail Learning via Logit Adjustment, ICLR 2021.

[2] Improving Calibration for Long-Tailed Recognition, CVPR 2021.

[3] Balanced Meta-Softmax for Long-Tailed Visual Recognition, NeuIPS 2020.

[4] Disentangling Label Distribution for Long-tailed Visual Recognition, CVPR 2021.
limitations_and_societal_impact: The paper aims to address long-tailed recognition. There seems no obvious negative societal impact.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper studies a practical problem, namely long-tailed recognition. To address this problem, this paper proposes two methods: UniMix and compensated cross-entropy loss. To be specific, UniMix adopts a balanced data sampler to improve the number of head-tail pairs for mixup. Experimental results show the effectiveness of the proposed method.


main_review: Pros:
1. This paper reveals an interesting problem in existing mixup-based long-tailed recognition (LT) methods, i.e., they will generate more head-head pairs since there exist more samples from the head classes. Such a problem may lead to more imbalanced learning.
2. To address the above problem in existing mixup-based LT methods, this paper proposes an intuitive UniMix strategy.
3. The proposed methods are theoretically analyzed.
4. Extensive results show the effectiveness of the proposed method.

Cons:
1. When this paper analyzes the bias issue of mixup, it assumes the instance number is exponential with $\lambda$. I wonder whether the result is the same when the instance number is not exponential with $\lambda$, since real-world datasets may not follow such a way.
2. For the proposed compensated cross-entropy loss, it is very similar to the balance softmax [1], which is also obtained from the Bayesian inference's perspective. Moreover, similar ideas of logit adjustment and rebalanced mixup have been deeply explored in the work [2,3]. Although this is not the drawback of this paper, I do not think this loss can be regarded as an important contribution.

    [1] Balanced Meta-Softmax for Long-Tailed Visual Recognition, NeurIPS 2020.

    [2] Long-tail Learning Via Logit Adjustment, ICLR 2021.

    [3] Remix: Rebalanced Mixup. arXiv, 2020.

3. In the final algorithm, there are actually two phases, i.e., first mixup training for T1 steps, and then non-mixup training for T2-T1 steps. I understand this is a trick to improve performance [4], but I wonder how effective the first phase is, since the main contribution of this paper is at phase 1, i.e., UniMix.

    [4] Bag of Tricks for Long-Tailed Visual Recognition with Deep Convolutional Neural Networks, AAAI 2021.

4. As proposed in this paper, mixup leads to bias distribution (since generating more head-head pairs) and has a negligible contribution to model calibration. Regarding this, why do existing mixup-based methods also obtain very promising performance?  Does this mean that model calibration may not be an essential metric for the long-tailed recognition problem? 

5. Following question 4, according to Tables 3 and D1, LDAM is harmful to model calibration. However, LDAM is also effective and helps handle the imbalance issue. Does this mean that a method with worse model calibration does not essentially mean a bad long-tailed method, and a method with better model calibration does also not mean a better long-tailed method? These two concerns are important since this paper is based on model calibration.

6. Why not compare to MiSLAS [5]. This paper analyzes mixup for model calibration and thus is very related to the submission. One interesting thing is that from the results reported in two papers, although MiSLAS only uses Mixup and ignores the head-head pair bias, it obtain even better accuracy and lower ECE than the method in this submission. I understand there may exist some differences in methodology (1-stage method v.s. 2-stage method), but it would be better for authors to also discuss MiSLAS and compared with it.
In addition, one interesting thing is that MiSLAS finds that mixup is beneficial to representation learning and bad for classifier learning. From this perspective, the head-head bias seems to mainly influence classifier learning. Could you discuss this？
 
    [5] Improving Calibration for Long-Tailed Recognition, CVPR 2021.

***

****Post Rebuttal****

The most important contribution of this paper is that it theoretically and empirically reveals an interesting problem in existing mixup-based long-tailed recognition (LT) methods, i.e., they will generate more head-head pairs that may lead to more imbalanced learning.

To address the argued problem, this paper proposes a new method, consisting of (1) a Unimix strategy for generating more tail pairs and (2) a Bayias loss for rebalancing the optimization. Nevertheless, the performance improvement of Unimix over mixup is limited, while most performance improvement is derived from the Bayias loss (c.f. Table 3 in the submission and the new results in the response). Moreover, the novelty of the Bayias loss is limited, considering the logits adjustment loss [1] and balanced softmax loss [2]. For the balanced test set, these losses perform similarly. Although the authors argue that the extension of Bayias loss with $\log(C)$ to $\log(\pi')$ can help to handle imbalanced test class distribution when the test class distribution $\pi'$ is known [3], I think that it is very difficult to obtain the test class distribution in real-world applications [4], so the practicability and contribution of the extension are limited.

Considering the above advantages and limitations, the paper seems really a borderline paper and is difficult to decide. Therefore, I return to consider the value and contribution of this paper to the LT community. In my view, although the performance and novelty of the proposed method are not that promising, this paper theoretically reveals an important head-head bias problem in the existing mixup-based LT method, which is insightful to the LT community and even inspires more augmentation-based LT methods. **Considering that performance-only judgment should not be encouraged in the current AI community and novelty can be varied, I believe the contribution of this paper is good enough and decide to increase my score to 7**. Even so, I encourage the authors to further improve the proposed method for better performance, keep exploring augmentation-based LT methods, and try to solve more practical test-agnostic LT task in the future. Thanks again for your effort.

[1] Long-tail Learning Via Logit Adjustment, ICLR 2021. 

[2] Balanced Meta-Softmax for Long-Tailed Visual Recognition, NeurIPS 2020.

[3] Disentangling Label Distribution for Long-tailed Visual Recognition. CVPR, 2021.

[4] Test-Agnostic Long-Tailed Recognition by Test-Time Aggregating Diverse Experts with Self-Supervision. ArXiv, 2021.
limitations_and_societal_impact: Please refer to the main review for the concerns of this paper. I would like to see the response from the authors.
review_title: null

Meta-review: This paper addresses the problem of long-tailed visual recognition. They proposed two complementary methods to address the imbalance problem: (i) a data augmentation scheme based on the Mixup and (ii) an approach to compensate for bias in class prior. The final proposed pipeline is 1 stage, as opposed to 2-stage which has featured prominently in recent work. Results are presented across several different image classification datasets, showing superior performance compared to existing work in most cases. 

During the review process there was a lot of discussion regarding the relative performance compared to the recent CVPR’21 paper, “Improving Calibration for Long-Tailed Recognition” AKA MiSLAS. This paper was published after the NeurIPS submission deadline, and thus the relationship between the two works did not factor into the final decision. However, the authors are strongly encouraged to cite it, and include the new 2-stage comparison to MiSLAS from the response for completeness. In addition, they should more clearly clarify the relationship to the logit adjustment work from ICLR’21. 

The reviewers noted that while the technical contributions were not overly significant, the main insight of the paper (that naive mixup introduces “head to head” class bias, i.e. a bias towards head majority pseudo data) is well characterized and will be of interest to the long-tail community. There were also concerns about the clarity of the exposition and some missing discussion. Many of these issues were cleared up in the discussion, but the authors are strongly encouraged to address these issues with the writing for the final version. This is very important as it will hopefully significantly improve the clarity of the paper. The paper should also be proof-read for grammar (e.g. Line 129). 

In the end, three of the reviewers were broadly supportive of the paper, but one recommended rejection (based mostly on the relationship to MiSLAS - see above). This AC, agrees with the consensus of the reviewers and supports the paper being accepted.  

Decision: Accept (Poster)