Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The authors propose better quanitization techniques for class-imbalanced datasets. 

Specifically, propose 
1. ClimbQ - scale the class distributions such that rare classes have higher variance, followed by bucketization of CDF
2. ClimbQ + homovar - a weighted loss function inspired from Levene’s hypothesis testing.

The authors compare the proposal against related techniques on 3 datasets, mostly by simulating different levels of imbalance.

strengths_and_weaknesses: Strengths.
---------

- The proposal is well-motivated, simple and shows strong experimental results.
- The paper is also well-written.

Weakness
--------
- The generation of simulated data through γ is a bit unconvincing since it was not clear how exactly was the imbalance simulated.
  Having said that, I do think results in Appendix#3 sound convincing.
questions: - Have the authors considered plugging in eq(4) to other competing approaches?
limitations: N/A
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper investigates the issue of quantization on class-imbalanced data. The key observation in this work is that quantizing imbalanced data inclines to obtain a large error due to the differences between separate class distributions, which leads to a significant accuracy loss. In order to tackle this issue, this work proposes ClimbQ, a new framework that focuses on diminishing the inter-class heterogeneity for quantization error reduction. ClimbQ first scales the variance of each class distribution and then projects data through the new distributions to the same space for quantization. To guarantee the homogeneity of class variances after the ClimbQ process, we examine the quantized features and derive that the homogeneity satisfies when data size for each class is restricted (bounded). Accordingly, ClimbQ embeds a new Homogeneous Variance Loss (HomoVar Loss), which reweights the data losses of each class based on the bounded data sizes to satisfy the homogeneity of class variances. Extensive experiments on class-imbalanced and benchmark balanced datasets reveal that ClimbQ outperforms the state-of-the-art quantization techniques, especially on highly imbalanced data
strengths_and_weaknesses: Strengths:
- Important problems and the insights are valuable
- An end-to-end solution to address the class-imbalanced issues within quantization
- Evaluation results are sufficient to justify the effectiveness of the proposed approach
- Source codes available

Weaknesses:
- Limited novelty (the solution donot add new insights on class-imbalanced quantization)
- The proposed solution seems restricted within quantization-aware training
questions: - What are the benefits (and/or downsides) of ClimbQ over other mixed-quantization schemes?
- Can you comment on potential applicability of your approach on post-training quantization?
- Can you comment on potential impacts of your approach on the inference performance?
limitations: - I assume the work requires the knowledge of the distribution on the full dataset (which may limit the practicality of this approach since it may require multi-party sharing).
- I am concerned about the potential implications in terms of the inference performance.
- I would suggest to make the scope of your approach clear upfront.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The authors investigate quantization of (deep neural network) model parameters in class-imbalance settings. This is achieved by scaling the variance of of each class separately, which is then mapped to a common distribution for quantization. Further, a loss function with class reweighing is used to satisfy the homogeneity of class variances. Experiments on four datasets (Synddigit-LT, CIFAR-10-LT, CIFAR-100-LT, ImageNet-ILSVRC) demonstrate state-of-the-art results for quantized models, specially, on highly imbalanced data.
strengths_and_weaknesses: Strengths:
Though the variance scaling and quantization are relatively straightforward, the weighting approach based on hypothesis testing offers a different and interesting angle to class weighting schemes. The experiments are extensive, consider multiple datasets, imbalance ratios and importantly, also consider existing class weighting approaches, which not originally intended for quantization, can be used within the ClimbQ framework.

Weaknesses:
Though the paper is relatively well organized, it is difficult to follow, for instance, N(.,.) is used before being defined, x is introduced as data to then being extended to also the features from a deep learning model, there is abuse of notation (e.g., F(X) a distribution is equated to a random variable U), x' is not formally defined, etc.

Something that is never explained or discussed is why if the data (including features) is being scaled during training, how is the scaling being applied during testing and if not necessary, why is it the case?

Post-rebuttal: increased rating after detailed clarification from the authors.
questions: Though the assumption of variance differences between classes of different sizes is reasonable and can be demonstrated in practice in multiple ways, using t-SNE is perhaps not the best way considering that due to the minimum distance specification in t-SNE, a larger dataset will occupy a larger space regardless of variance. Moreover, being a local linear embedding technique, distributional variance are not necessarily well represented. Can the authors justify or use a different visualization technique?

In Section 3.1.1 the authors claim that differences in class distributions cause massive quantization error, however, it is not justified or demonstrated.

There seems to be a problem in Figure 2, by which N of N(\mu_k,sigma_k) is not shown.

The authors argue that data follows a normal distribution, which is demonstrably not true in general practical settings (without data transformation). Though it is possible that the authors are referring to model parameters and features, the beginning of Section 3.1.2 clearly states that data x is assumed to be Gaussian, though later it is stated that x are features. Please clarify by stating the notation earlier.

What happens in practice when beta -> 1 (0.999 < beta < 1)?

Did the authors consider ClimbQ with a simple inverse probability weighting (1/p_k, where p_k is the proportion of samples in class k)?
limitations: The authors address the impact and limitations of the proposed approach in Section 6.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The authors propose ClimbQ quantitzation for efficient inference in
the context of class-imbalanced problems. The majority (minority)
classes have larger (smaller) variance and they reduces (increase) the
variance.  To project onto a uniform distribution, they use a
cumulative distribution function, which is then quantized.  Based on
Levine's hypothesis testing on homogeneity in variances, they show a
lower and upper bound of class data size to reject/accept the null
hypothesis (homogeneous variances).  Using the bounds, they divide the
classes into: minority, moderated, and majority classes.  They then
design the HomoVar loss, which weight minority classes more and
majority classes less.

Empirical evaluation on 3 imbalanced datasets indicates the proposed
method generally outperform 6 existing methods.

strengths_and_weaknesses: The problem of quantization for class-imbalanced problem is
interesting.

The proposed projection onto an uniform space and HomoVar loss are
interesting. They derived the upper and lower bounds of class size
with respect to Levine's hypothesis testing on homogeneity in
variances.

Empirical evaluation indicates the proposed approach generally
outperform existing methods.

The paper is generally well written.

questions: lines 186-188 vs lines 194-195: the null hypothesis is rejected when
the class size is NOT too small/large (lines 186-188), but it is
rejected when the class size is too small/large (lines 194-195).

Eq 4: more motivation on how w_k is designed would be beneficial,
particularly the intent for the numerator and denominator.

line 170:  nominator -> numerator ?

limitations: Limitations and negative societal impact were discussed.
review_title: null

Meta-review: metareview: After rebuttal, the reviewers unanimously agree that the submission should be accepted for publication at NeurIPS. 
confidence: Certain
Decision: Accept