Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper introduces variational classification (VC) by treating the inputs to the softmax layer of a classification model as latent variables with a mixture of Gaussians prior. To achieve this probabilistic interpretation of the softmax classifier, they derived an objective to be minimized, which is similar to the ELBO used in VAEs. The resulting VC model generalizes the softmax classifier (similar to VAE vs deterministic autoencoder) and enables incorporating class-conditional priors. A derivation of the training objective is provided with some practical design choices to optimize it. The evaluations are performed on several datasets (CIFAR-10, CIFAR-100, SVHN, CelebA, MNIST) in terms of accuracy, calibration, OOD generalization, adversarial robustness, and performance on low data regime. 
strength_and_weaknesses: Strengths:

- The generalization of the softmax classifier with a Bayesian interpretation is interesting and it can potentially have impacts on the several desired properties for classification models such as calibration and OOD generalization. 

- The authors provide a derivation for the VC objective with nice connections to VAE. I also liked that they provided practical implementation points of the VC using the commonly employed tools like the reparameterization trick. However, I believe it could be useful to provide an algorithm table or pseudo-code for better clarity. Could the authors also clarify if there are any instability issues faced during the adversarial optimization?

- The experiments were selected to validate different aspects of the proposed VC. More on that below.

Weaknesses: My main concerns are about the experimental results.

- From the results, it is hard to say VC is doing notably better than GM or CE methods, except model calibration (even for that, it is sometimes worse than CE, see Fig. 4. left). Could this be due to optimization difficulties, e.g. due to adversarial objective, or are there other reasons?

- For calibration, adding temperature scaling as another baseline could be helpful to gauge the improvements.

- Can this method be applicable to more complex tasks like ImageNet classification and more strong adversarial attacks like PGD? Also for low data regime, are there any benefits for datasets other than MNIST where the performance is more or less saturated? What happens to calibration under low data regime?

- Demonstration of the experimental results are somewhat inconsistent. For example, why is there no standard deviation in Table 1 from multiple runs while they are provided for Table 4? What do the bars and whiskers represent in Fig. 4?

- Is there a substantial increase in the training time compared to standard training? An analysis on this could be useful for practitioners.

- Outlining the experimental protocol, e.g. used augmentations and hyperparameters, could increase the reproducibility of the work.

- I found Sec. 3.3 to be somewhat confusing. Did the authors observe whether the model learned any disentangled and semantically meaningful representations? 

Minor: There are several typos throughout the paper. Also, the font size in figures are too small.

clarity,_quality,_novelty_and_reproducibility: The paper is clear. For reproducibility, I think more details should be provided.
summary_of_the_review: While the paper makes some interesting points, I think it is still a borderline one as the performance is somewhat mixed and gains are mostly incremental. I would be happy to revise my score if the above concerns are addressed sufficiently.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a variational classifier (VC). Typical machine learning classifiers use sigmoid or softmax to deterministically map last layer feature vector to class label predictions. This paper revisits the MLE, MAP and Bayesian under a variational framework. This work designs a novel objective based on ELBO and adversarial/contrastive technique. Experiments show that VC has some desired properties in interpolation, prediction confidence, out-of-sample detection, etc.
strength_and_weaknesses: Pros:
1. Compared to deterministic methods, probabilistic ones like variational model can also smooth the predictions and alleviate overfitting issues. That's why this method can potentially help with calibration, interpolation, OOD, etc.
2. The paper explains the details very thoroughly, and carefully compare the difference and similarity between existing and proposed methods.
3. The experiments explored multiple domains/tasks.

Cons:
1. From a machine learning perspective, I think another view of understanding this paper is the embedding learning. Both input feature x and label y are mapped to a latent embedding z. The label learnt probabilistic space is the prior space and the input learnt probabilistic space is the posterior space. You kinda try to match the two spaces. While probabilistic method often brings smoothness, the performance loss is also concerned. In general, I feel this work still doesn't jump out of this framework and maybe suffer from the same issues?
2. Though this type of variational learning is relatively novel, the use of VAE in classification has been quite ubiquitous. The latent variable models also learn and align latent subspaces [1, 2]. I think you could discuss these works.
3. While this paper has experiments on multiple setups/scenarios, they look a bit thin to me. First, the major datasets are cifar-10, cifar-100, etc. In the computer vision domain, these are just too small. It's less convincing to only use these datasets. Also, you only tested WideResNet-28-10 and ResNet-50. More and larger models would give readers a more comprehensive view on the performance.
4. You can give some derivation steps for Eq. 5 and 6, at least in Appendix. 
5. Figure 2: Variation Classier -> Variational Classifier

[1] Bai, J., Kong, S. and Gomes, C.P., 2022, June. Gaussian mixture variational autoencoder with contrastive learning for multi-label classification. In ICML (pp. 1383-1398). PMLR.

[2] Bai, J., Kong, S. and Gomes, C., 2021, January. Disentangled variational autoencoder based multi-label classification with covariance-aware multivariate probit model. In IJCAI (pp. 4313-4321).
clarity,_quality,_novelty_and_reproducibility: This paper introduces a lot of concepts, but in a relatively clear way. The quality of the paper can be improved by providing more experimental results. The paper does bring some novelty in addressing classification from a variational direction. The code is not provided and it seems non-trivial to implement the whole algorithm.
summary_of_the_review: This paper is a more theory-oriented paper with empirical experiments support. The idea is interesting, though resembling several prior works. This paper should better shape its relations with literature and elaborate on the model details and experiments.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper presents a Variational Classification (VC), which generalizes the softmax classifier, and the authors claim that it mirrors the relationship between the variational auto-encoder and the deterministic auto-encoder. The main contribution is to include the latent variable to softmax classifier and then VC objective analogous to the ELBO is designed. The experimental results show that the VC classifier outperforms the standard softmax in several ways, in particular in terms of calibration, adversarial attacks and when data is scarce. 

strength_and_weaknesses: Quality/Clarity: the paper is well written and the techniques presented are ok to follow. Its motivation is clear, which is to generalize softmax and design a better model: (1) modeling p(y|x) (2) measure of
similarity between data samples. On a technical level, I do not see how VC objective in Eq. 6 related to ELBO. The authors need to give more details derivation from Eq.5 to Eq. 6.

Originality/significance: the idea is interesting, which introduces latent variable and adds more constraints to generalize the softmax function. However, it gives readers an impression that the VC objective in Eq. 6 is a heuristic result, not from ELBO. If the authors can provide the detailed derivation to bridge the gap as well as show more experimental results, it should be an accepted paper. 
clarity,_quality,_novelty_and_reproducibility: The idea to introduce the latent variable to classification task is interesting, but I think it will be better to add more theoretical analysis and experiments to support the claims. 
summary_of_the_review: The paper adds a latent variable to classifier and impose constraints (class conditional priors) in the objective function.
Overall it is a good paper, but not ready for publication.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper is proposing a replacement of final linear classification of a deep network with a probabilistic classifier using variational learning. Since the classifier has a clearer probabilistic meaning in the proposed setup, it is expected to perform better when the probabilistic meaning is critical. Problems like OOD detection and handling adversarial inputs are all invalidating iid. assumption and requiring reliance of some form of a prior which this method can provide and/or utilise. 

Strengths:
- A clear and theoretically sound idea. The overall promise of the variational final layer is sensible and follows from first principles.
- A flexible approach. The proposed approach is flexible to include other type of priors and domain knowledge which makes it a good candidate to handle OOD family of problems.
- Promising initial experiments. Authors provide some promising results.

Weaknesses:
- The paper was not clear enough for the reviewers. All reviewers shared the concern that some crucial knowledge is missing. After my personal reading, I agree that the paper is not self contained as some derivations are pushed out. Although authors tried to fix this during their rebuttal, they missed the deadline due to partly a technical issue and partly because they failed to upload the new version in a timely manner. Authors added them as a comment but I believe they should be part of the main text not appendix, at least partially. It is not possible to review the updated paper without seeing it since I am not sure how would it fit to the page limit and what kind of changes authors would make. This is a major issue.
- The experiments are promising but all comparisons are self baselines. Authors need to show that their approach is useful for some existing applications compared with some existing baselines. Current results and experiments are clearly enough to validate the idea and its soundness. However, it is not enough to validate its applicability and usefulness. This issue also persists through the fact that all experiments are on small datasets and some specific model families. To show the usefulness, I strongly recommend authors to add their method on top of an existing method which solves an existing benchmark and show its value in such a realistic setup.
justification_for_why_not_higher_score: The idea and the theory of the paper is valid. However, it is not at the state to be published. It needs a significant improvement on the empirical evaluation side. Moreover, it is also not possible to see the state of the writing of the paper since authors missed the deadline to upload their revision.
Decision: Reject