Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: The paper proposes a sparse classifier via  discriminative GMM. This model is trained based on sparse Bayesian learning. The sparsity constraint removes redundant Gaussian components which results in  reducing the number of parameters and improving the generalization. This framework can potentially be embedded into the deep models and trained in an end-to-end fashion. The main motivation is that the proposed model (i.e., SDGM,)  can consider multimodal data while conventional softmax classifiers only assume unimodality for each class. Experimental results show the superiority of the SDGM over existing softmax-based discriminative models.

The paper is well-written and easy to follow. And the paper precisely places the proposed method among the related work.  However, there are some concerns:

1- Sparsity constraint is supposed to improve the performance and generalization, but experimental results do not support this. What is the motivation of employing sparsity learning in this framework? 

2- Conventional softmax classifiers in deep architecture have already provided promising results on real world datasets such as ImageNet which contains classes with multimodal data. However, in this paper, experiments are performed on the small datasets, so was the proposed method evaluated on ImageNet as well? 


3- There are many versions of softmax-classifiers such as “large-margin softmax”, “angular softmax”, and “additive margin softmax” , and  [1 ,2 ] that address conventional softmax-classifiers’ issues. Have you compared your models with them?

[1] Liu W, Wen Y, Yu Z, Yang M. Large-margin softmax loss for convolutional neural networks. InICML 2016 Jun 19 (Vol. 2, No. 3, p. 7).

[2] Weiyang Liu, Yandong Wen, Zhiding Yu, Ming Li, Bhiksha Raj, and Le Song. SphereFace: Deep hypersphere embedding for face recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 212–220, 2017

[3] Feng Wang, Jian Cheng, Weiyang Liu, and Haijun Liu. Additive margin softmax for face verification. IEEE Signal Processing Letters, 25(7):926–930, 2018.

[4] Kaidi Cao, Colin Wei, Adrien Gaidon, Nikos Arechiga, and Tengyu Ma. Learning imbalanced datasets with label-distribution-aware margin loss. In Advances in Neural Information Processing Systems, pp. 1567–1578, 2019.

[5] Bin Liu, Yue Cao, Yutong Lin, Qi Li, Zheng Zhang, Mingsheng Long, and Han Hu. Negative margin matters: Understanding margin in few-shot classification. arXiv preprint arXiv:2003.12060, 2020.

4-  Finally, I could not find any guarantee for the convergence of the learning algorithm in the paper? What is the time-complexity?  I think training with softmax would be much faster, easier and also provides promising classification results in practice.

review_title: A Discriminative Gaussian Mixture Model with Sparsity

final_score: rating: 7: Good paper, accept
review_content: The paper proposes an interesting extension to both discriminative GMMs and CNNs with a softmax outuput. The key innovation is the introduction of the sparsity in discriminative, multimodal settings. This novelty and the clear experiments merit this work to be published at ICLR 21. However, the testing and evaluation could be significantly improved.

There are a few areas where the paper could be improved.
1. The language could use another review. Often, the writing slips into the use of informal language.
2. Appendices were missing in my version; the paper makes an explicit reference to at least appendix A.
3. The connection to Bayesian methods is week.
4. The table describing Algorithm 1 appears abruptly and references formulae that have not been introduced. It is best positioned at the end of Section 3 to improve readability. 
5. Most of the Bayesian approaches used in the comparison are dated.
6. Section 4.2 references high computational costs, but it is unclear which steps of the algorithm make this approach computationally prohibitive.
7. It is unclear from Table 1 that SDGM is significantly better than RVM. A deep dive is warranted to better understand their relative performances. 
8. It would be better if Section 4.2.1 also include large datasets or more challenging domains. The relative error reduction described in Table 2 on a couple of datasets are marginal.
review_title: Review of "A Discriminative Gaussian Mixture Model with Sparsity"; borderline accept recommendation.

final_score: rating: 5: Marginally below acceptance threshold
review_content: * quality
The paper presents an interesting idea that uses sparse Gaussian mixtures, but it lacks theoretical guarantees. Although the method is Bayesian, can we also give frequentist non-asymptotic bounds?
* clarity
The paper is well written.
* originality
The paper's ideas seem original, but they're very straightforward, making its contributions marginally incremental.
* significance
If the paper had more theoretical guarantees, its results would be more significant. The current version is a bit weak. 
review_title: interesting but needs more theory

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: The innovative part of the work consists in the trick (8), which allows for a kernel-based generalisation of Gaussian mixture models viewed under a discriminative perspective.  

The model is trained via maximum a posteriori estimation; due to the analytically intractable form of the posterior expectation of the log-likelihood, the authors resort to a second-order Taylor approximation around the mode (Laplace approximation). That's an old and not so fine approximation, but easy and widely-known. 

The experiments are performed on standard benchmarks, and comparisons are provided against some natural competitors of the method. In essence, the authors replace the penultimate softmax layer of a deep net classifier with their model and train end-to-end.

The experiments yield a marginal improvement over s.o.t.a., as so many papers do nowadays. An aspect that is missing is how they initialised the models. Initialisation may  play a crucial role in such approaches, especially when it comes to ensuring reproducibility of the results. 

In summary, a slightly novel paper with some interesting insights and some pretty standard nowadays, yet marginally impressive experimental results. 
review_title: A Gaussian mixture model for a better penultimate layer in deep net classifiers 

Meta-review: This paper proposes replacing the softmax of deep NNs with a kernel-based Gaussian mixture model, to allow for per-class multi-modality.  Results show that the method is competitive with other output modifications such as the large-margin softmax.  

The  two primary concerns of the reviewers were the lack of large-scale image classification results and theoretical guarantees.  The authors have added CIFAR-100 results.  Moreover, the authors agree that theoretical results would be nice to have, but such results are non-trivial and likely require a PAC-Bayes treatment.

I find the method to be well-motivated and that the paper demonstrates sufficient experimental rigor.  Given the popularity of the softmax throughout deep learning, this paper will likely be of interest---or at least, be of potential use---to a large part of the ICLR community.  I encourage the authors to add the ImageNet results to the final version.

Decision: Accept (Poster)