Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The authors propose a technique for heteroschedastic classification, HET-XL, which does a simple trick to maintain scalability to settings with large amounts of classes.

The key idea is deceptively simple: the authors elect to model the covariance over noise not over class-space in the logits, but over the empedding space pre-logits, leading to a covariance matrix of much smaller dimension int he large class scenario which maintains tractability at ease.

Experiments show this approach also confers some regularization benefits and remains indeed tractable.
strength_and_weaknesses: Strengths:
- this is a simple technique to model heteroschedastic label noise and makes it tractable to large dimensions. 
- the technique is enormously simple to utilize and plug in to existing classifiers.

Weakness:
- in theory, the expressivity of this parametrization should be lower than in the "full rank" scenario. However, the authors show graceful performance in practice. I feel it is noteworthy to keep this in mind in practice.
clarity,_quality,_novelty_and_reproducibility: The paper is clear and benefits form the simplicity of the core idea.
The core idea appears simple and not particularly novel, but it is useful and interesting and appears to work surprisingly well.
summary_of_the_review: The authors propose a simple trick to make heteroschedastic classification scalable to large class-scenarios.
Their key idea is simple, easily applicable, and although it appears to lose out on some modeling oompf theoretically performs quite well in practice.

I think this could be useful for practitioners going after raw performance, though I would caution not to overinterpret the learned covariance matrix semantics as equivalent to one learned in class-space.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: As indicated by the title, this paper considers the problem of scaling heteroscedastic classifiers to problems where the number of classes is large.  The main contribution is a simple trick which eliminates the dependence on the number of classes on the total number of parameters.  The authors also show that the temperature parameter commonly used for these networks does not need to be treated as a hyperparameter; rather, it can be optimized as a trainable parameter.  The authors discuss how these insights can be applied to contrastive learning, and then they provide thorough experiments showing that their method scales more efficiently than existing heteroschedastic classifies and that it outperforms a deterministic baseline.
strength_and_weaknesses: ### Strengths

**Writing quality.**  This is the best-written paper in my batch of ICLR papers this year.  The arguments are clearly articulated and it's easy to understand the main ideas.

**Simplicity.**  I found the simplicity of the main ideas to be appealing.  The trick used to change the layer at which the noise is added is simple, and it's encouraging (as shown in an appendix) that trick comes at no loss of performance.  The main ideas seem like they would easy to implement for those who wanted to reproduce this work, and authors also seem committed to making their code available, which is a positive and encouraging sign.

**Experiments.**  The experiments are thorough and they show that the method seems to offer a performance improvement over both deterministic classifiers (henceforth DET) and heteroscedastic classifiers (henceforth DET).  Furthermore, relative to HET, the proposed architecture HET-XL has a smaller number of parameters; indeed, the authors show their architecture's parameter count does not scale with the number of classes $K$.  These two results -- both highlighted in Table 3 -- are exactly what one would hope to see.

Overall, the rest of the experiments also seem to be quite thorough.  In the main text, we get a couple of ablations concerning the latency and memory usage of the relevant algorithms.  There are also quite a few extra results in the appendix, including ablation studies on learning $\tau$, model scaling, data scaling, and the number of floating point operations.  In eacfh case, it seems that HET-XL compares favorably to the baselines.

## Weaknesses

**Minor points of confusion.**  Here are a couple of things that confused me while I was reading the paper:

* I don't understand why this model requires *marginalizing* over the noise distribution.  Granted, I'm not an expert on heteroscedastic classifiers, so perhaps I have missed something obvious here.  However, from my understanding, the noise is modelled by a Gaussian, which we assume has zero mean.  Furthermore, the covariance matrix of this Gaussian is learned from data.  One one has access to this covariance matrix, why does one need to run MCMC to sample from this noise distribution?  Can one not directly sample, given that the distribution is normal?  Perhaps it has something to do with needed to differentiate through the covariance estimation?  If the authors could elaborate on this point, I think it would make the paper clearer.

* What is the so-called "hashing trick?"  It's relatively difficult to understand the innovation in Section 6.1 without knowing this, and I think that it's possible that many readers of this paper may not have heard of it before.  In my opinion, it would be worth adding a few lines to the paper to describe the main idea behind this trick, and why it's useful here.

**Contrastive learning.**  Having read the paper, I'm not sure what the contribution of the contrastive learning section is.  Indeed, it seems clear that contrastive learning can be formulated as a massive classification problem.  However, the question is: Does this perspective result in better algorithms or architectures for contrastive learning.  And based on the results in Table 4, it's not clear that it does.  The table indicates that on ImageNet, the heteroscadastic classification view of contrastive learning yields a marginal (albeit, statistically significant) improvement.  However, on the other two datasets, HET-XL performs worse than the baseline deterministic classifier.  Therefore, it's somewhat unclear whether it's worth using a classifier with potentially many more parameters relative to DET in the setting of contrastive learning.  For these reasons, I do not see the contrastive results as an impactful contribution.  If I have missed something, I would welcome the thoughts of the authors on this point.

**Efficiency.**  After reading the paper, it seemed unintuitive that when taking many MC steps, HET-XL underperformed relative to HET in terms of latency.  Could the authors shed more light on this?  Why is sampling less efficient for HET-XL?  And if this is the case, in terms of time-complexity, would one then tend to prefer HET over HET-XL.  (Of course, in space-complexity, it does seem that HET-XL is to be preferred in these large-scale settings.)
clarity,_quality,_novelty_and_reproducibility: **Clarity.**  The paper was very clearly written.  This is certainly above average in terms of writing quality, grammar, etc.  

**Quality.**  The quality is also high, in the sense that the experiments seemed to have been conducted in a principled and thorough way.  The description of the architecture was also crisp.

**Novelty.**  There are aspects of this paper that are novel.  The empirical results in Table 3 show that HET-XL surpasses the state-of-the-art.  The method involves some nice tricks, which are novel in this problem setting.  However, overall this paper more or less uses standard or existing tools/tricks, so from the methodological side, it's not particularly novel in my opinion.  

**Reproducibility.**  Clearly this would take a massive amount of compute to reproduce.  But controlling for that, it seems that the authors will release their code, and the descriptions given in the paper seem sufficient to be able to reproduce this work.
summary_of_the_review: Overall, I thought this was a solid paper.  The empirical results are impressive for the heteroscedastic setting.  The method is simple and clearly presented.  And the experiments are thorough.  On the negative side, aside from a few minor points of confusion, I would argue that the contrastive results are not particularly impactful and that the time-complexity of HET-XL is concerning.  I think that this paper has some empirical novelty, but it does not introduce tools that may be more broadly applicable to the learning community.  All of this being said, I think that this paper does exactly what it sets out to do, which is to improve the SOTA in heteroscedastic classification.  Therefore, I recommend that this paper be accepted.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This manuscript proposes HET-XL, which is a simple yet effective solution for handling heteroscedastic classifiers for extreme classification. Specifically, the proposed method has advanced the scaling issues in the existing work. The extensive experiments and ablation studies showed the effectiveness of the proposed method on large-scale image classification benchmarks such as JFT-300M and ImageNet-21k.
strength_and_weaknesses: Strengths
- The manuscript is well-written and easy to understand.
- The proposed idea is simple yet effective.
- The proposed method has advanced training efficiency and scalability of learning heteroscedastic classifiers for extreme classification tasks using a simple idea.
- Extensive experimental results demonstrate the effectiveness of the proposed method.

Weaknesses
- In my understanding, the proposed method takes benefits from breaking the full rank assumption of the covariance matrix (from $K$ to $D$). The authors provide empirical evidence for this, but it is still unclear why it works. Could you provide any theoretical analysis for this? or How big does $D$ work well? Specifically, does the proposed method effectively work on ViT models having small dimensionality (e.g., ViT-small and ViT-tiny)?
- I think applying the proposed method to contrastive learning is an interesting idea. However, the reported improvements are marginal. However, under the perspective of instance discrimination view, there exist very strong class-correlations; for example, CIFAR-100 has actual 100 categories but 50,000 classes in this case. I am curious about whether we can expect heteroscedastic classifiers to work well under such a strong class correlation. Moreover, in contrastive learning, I think adding noise before and after $l_2$ normalization would have different behaviors.

clarity,_quality,_novelty_and_reproducibility: The writing is clear and easy to understand. It clearly identifies a problem and proposes a solution. The solution is validated properly.
summary_of_the_review: Overall, I would recommend the acceptance, as it has solid contributions and is worth sharing. Although the proposed solution is straightforward, it clearly advanced the previous works and has shown significant improvements on large-scale benchmarks.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The authors propose a method for performing Heteroscedastic classification that can scale independently of the number of classes.  The method simply defines noise not over the logits of a network as is commonly done, but over the layer previous to the logins.  In this way the learned instance-specific covariance of the noise distribution is a DxD matrix (where D is the dimensionality of the pre-logit representation) instead of KxK (where K is the number of classes).  The authors also provide some empirical justification for treating the commonly validated temperature Hyperparameter as a free parameter in the training procedure.  In addition to those two contributions, authors define a contrastive learning model that uses their proposed noise model.  These three contributions are aimed at scaling heteroscedastic classification to cases where there are a considerable (> 10,000) classes.  Empirically, their method out-performs both heteroscedastic and deterministic baselines in terms of negative log likelihood and precision while producing models with fewer parameters than other heteroscedastic baselines.
strength_and_weaknesses: Strengths
1. Overall the method is presented in a clear manner.  It is both easy to see why the method would result in models with fewer parameters and likely easy to implement.
2. The empirical results seem to justify the changes to standard heteroscedastic learning as proposed in the paper.

Weaknesses 
1. The biggest weakness is that the the work lacks novelty.  Each of the three contributions is simply applying well known techniques with slight variations to reduce practical run time of training and inference.  The paper provides no principled or theoretical contribution, which makes the paper seem like more incremental than substantial.
2. The empirical results are not very compelling.  First, the number of epochs that each model is trained for does not seem representative of complete training procedures.  The JFT-4B (which are from a proprietary data set, and thus is impossible to replicate) results are from training from a single epoch, for instance.  A more compelling result would be to plot the NLL and/or precision of the model as a function of wall run time, as this would be illustrate the claimed main benefit of HET-XL: With fewer parameters, HET-XL is able to scale better than HET methods.  Further, the ablation results need to be in the main paper and more clearly presented in the context of the results using the individual components.  The memory and train timing figures are a good idea in principle, but it is unclear how the number of MC samples practically effects training.
clarity,_quality,_novelty_and_reproducibility: Clarity - While I think the paper could try to more closely tie the individual contributions together, overall the paper is clearly written and understandable.

Quality - There are no obvious issues in quality.

Novelty - As stated above, the novelty of this work is low due to the 

Reproducibility - While the authors seem to intend to release their code on GitHub, some of their results are on proprietary data, making reproducibility impossible. 
summary_of_the_review: While I feel the contributions of this paper likely improve the practice of learning heteroscedastic classifiers, I feel they are incremental and not significant enough to warrant acceptance.

(See response to rebuttal for change of final score)
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper investigates the use of heteroscedastic classifiers for large scale classification problems. Heteroscedastic classifiers consider uncertainty in the decision function by learning a multivariate gaussian distribution over classifier logits. This paper proposes two simple tricks to scale such approach forclassification problems with large number of classes:
 1) add gaussian noise at the pre-logit level of the networks, to remove the parameter dependencies on the number of classes
 2) learn the softmax temperature instead of more involve strategies to stabilize the MC  sampling processing.

Extensive experimentation on large scale datasets (ImageNet21K, JFT-300M/4B) shows the good scalability of their approaches and good performances over deterministic baseline and a strong hashing-based baseline. 

Authors also apply their approach to a contrastive learning setting and investigate the 0-shot classification setup.

strength_and_weaknesses: Strength:
-	Simplicity of the approach.
-	Extensive empirical study on large scale datasets.
-	Strong results that show both scalability of the approach and good performance.

Weaknesses:
- Lack of evaluation on downstream transfer tasks: Usually, representations learned from large-scale supervised datasets are then evaluated in a transfer setting through finetuning or linear probing. Would you expect HET-XL to show a gain in that setting, beyond the zero-shot case.
- HET extra cost due to Monte-Carlo Sampling: HET requires to perform a MC sampling to compute the network output, which increase the computational cost of the forward prop compared to a deterministic model. How would a deterministic model with the same computational budget for the forward prop than HEC perform?

clarity,_quality,_novelty_and_reproducibility: The paper is easy to follow, extensive evaluation demonstrates the value of the contribution. Reproducibility of some experiments is limited due to use of proprietary datasets. However, authors also report result on publicly available data.
summary_of_the_review: The paper proposes a simple approach with good scalability property and extensive evaluation to demonstrate the value of their proposal. I therefore support acceptance.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper studies heteroscedastic models and extends the previous studies that require an explicit modelling of the noise structure into a simple modelling by reusing the logits from the mean (non-noise part) term. And then the idea is further extended to contrastive learning setting. 
strength_and_weaknesses: - strength
    - the idea is straightfoward and a natural extension of the existing methods
    - the discussion and summary to previous methods are nice
- weakness
    - it seems unclear to me that where the strength of the methods come from 
    - the empirical strength of the methods does not seem clear to me
clarity,_quality,_novelty_and_reproducibility: - clarity: good
- quality: 
    - while the idea of reusing the mean term is very straightforward and intuitive, it seems unintuitive to me that the idea can work very well
        - mathematically speaking or intuitive speaking, reusing the mean term should introduce identification issues in the estimation of the model, and the same effects (from feature to label) can be modelled either by the mean term or the noise term, it is unclear to me that how the authors implementation or design of the method can avoid this issue
        - probably some more careful study/experiments will be needed for clarify these issues. 
        - while it is well expected, by doing this, the model can potentially learn well with much reduced resources cost, it seems to me the estimation problem will be less stable (more sensitive to hyperparameter choice), some relevant discussions on this regard will also be useful. 
    - for empirical results at Table 4, the authors suggest a p-value<0.01, does this mean each p-value is calculated for these three datasets? It's probably better to list all these three p-values, especially the one for CIFAR100 experiment. 
- novelty
    - the idea of reusing the mean term for noise term has been widely studied in statistics community, but might be the first time for this particular problem setting, so probably fine. In those studies, the identification issue is the biggest challenge, which leads to the question above. 
summary_of_the_review: reasonably good work, but some clarification on why the idea can bypass the identification issue might need to be deeply discussed.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper considers the problem of scaling models designed to function in settings when the number of classes K is very large, and one wants to explicitly model noise in the prediction layer. The obvious way to do this would be to have a K x K covariance matrix for some noise to be added after the softmax layer --- which would be prohibitive for large K. Restricting the rank of this matrix to be small would drop this cost to ~R x K, however this seems to come with performance drops in practice. 

The paper introduces two very simple tricks that seem to come at no performance loss, and much improved scaling with K: simply add Gaussian noise on the penultimate layer in the network (of dimension D) prior to applying the softmax. This matrix requires only D^2 terms to parametrize, which can be much smaller for D small (which is the case for many popular architectures). Another simple trick the authors provide is to treat the temperature in the softmax as just another trainable parameter (rather than a hyperparameter). 

Overall, the reviewers liked that the algorithmic change is so small, yet has good performance across the settings considered. During the discussion phase, the reviewers also asked for several extra experiments (e.g. slight variations in settings, p values for some of the experiments, some heuristic explanations for when the method might work or not) --- which the authors provided, and this improved the paper as well. 
justification_for_why_not_higher_score: Due to the simple nature of the algorithmic improvement, a broader consideration of tasks and benchmarks (as well as possibly a theoretical analysis) could be performed to ascertain when this parametrization doesn't cause a drop in performance.
justification_for_why_not_lower_score: The improved performance across the benchmarks considered, and with such simple (thus, easy to implement) algorithmic changes is a strong selling point for the paper.
Decision: Accept: poster