Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 1: Reject
review_content: The paper proposes a discriminative Gaussian mixture model with a sparsity prior over the decoding weight. They can automatically learn the number of components with the sparsity prior and learn Gaussian-structured feature space. 

1. I think the model is just ARD prior over discriminative GMM which is not that novel. DGMM models have been for a while [1,2]. Adding ARD sparsity prior over the decoding weight is also a classic routine. It's also well known that ARD can do feature selection and removal. 

[1] Discriminative gaussian mixture models for speaker verification
[2] Discriminative Gaussian mixture models: A comparison with kernel classifiers

2. I don't think differentiating between discriminative GMM and generative GMM would make such a big deal. DGMM is basically Gaussian mixtures existing for each class. Any skill applied to GMM can be applied to DGMM. There are many works for component number selection for GMM with non-parametric Bayesian methods. For example, Dirichlet Process Mixture Model can automatically learn the number of components without predefining. 

3. Only comparing SDGM with LR, SVM and RVM is quite weak, not mentioning that the performance is not that dominatingly better. SDGM is GMM+LR. So SDGM should be better than LR if the data has structures. What SVM you compare with? Do you use nonlinear kernels which can learn better nonlinear feature space?

Overall, I think the contribution of the paper is a bit incremental. I vote for a rejection. 


review_title: Official Blind Review #2

final_score: rating: 3: Weak Reject
review_content: This paper proposes a classifier, called SDGM, based on discriminative Gaussian mixture and its sparse parameter estimation. 
The method aims to have a flexible decision boundary due to the mixture model as well as to generalize well owing to the sparse learning strategy. 
While the method incorporates distinct ideas in the literature such as increased model complexity and use of a sparse prior, the paper needs more clearer explanation of the method design and careful empirical investigation. 

My major concerns are as follows. 
1) Explanation of the learning algorithm of Section 2.2 is unsatisfactory. 
Specifically, the objective function and the principle of algorithm are unclear. 
My guess is:
* c, m and T, z are equivalent. (T, z are one-hot representation of c, m.) 
* The objective is to maximize the conditional probability of equation (1) given labeled data (x, t). 
* This maximization is carried out in a similar way to the EM algorithm where m (or z) is regarded as a latent variable. This gives equation (9). 
* A MAP estimate of parameter w is calculated using the prior (8). 
Assuming the points above, I am still unsure about the following points: 
1-a) How is the sparsity induced? 
The prior of w seems to have the l2 regularizer, but how are \pi and r pulled toward zero?
1-b) How is the update of \alpha (17) derived?
Does this strengthen the sparsity by increasing \alpha given small \hat{w}?
What is the "orthogonal component" of \Lambda?
1-c) Is structure in w ignored?
Equation (7) indicates that parameter w has a certain structure such as nonnegative s_{cmii} or the determinant |\Sigma_{cm}| interacting with s_{cmij}. 
In other words, the degree of freedom in w assuming the Gaussian likelihood of x is smaller than H. 
These structure would be violated if the gradient descent or Newton's method is applied. 
Do you mean by *discriminative* that we can freely set the parameter w?
Then, this point should be emphasized. 

2) Which parameter is learned when combined into NN?
Parameters w and \pi? 
How are these parameters made sparse in the end-to-end learning?

3) Ablation test to investigate which aspect impacts the performance. 
Section 4 describes SDGM incorporates disciminative model, mixture model, and sparse Bayesian parameter estimation. 
It is more informative to provide empirical results to see the impact of each property by comparing SDGM with, for example, RVM, discriminative GMM and sparse GMM. 

Some other commets follow.
* \sqrt{\alpha_{cmh}} may be missing from the numerator of (8). 
* For what distribution the expectation with regard to z is taken in e.g. (9, 11, 12, 13, 16)?
* What is D for CIFAR-10 experiment? DenseNet seems to use D=1000 units for the fully connected layer. Did the authors adopt this value?


review_title: Official Blind Review #3

final_score: rating: 3: Weak Reject
review_content: The paper presents an alternative to densely connected shallow classifiers or the conventional penultimate layers (softmax) of conventional deep network classifiers. This is formulated as a Gaussian mixture model trained via gradient descent arguments. 

The paper is interesting and constitutes a useful contribution. The Gaussian mixture model formulation allows for inducing sparsity, thus (potentially) considerably reducing the trainable model (layer) parameters. The model is computationally scalable, a property which renders it amenable to real-world applications.

However, it is unfortunate that the paper does not take into account recent related advances in the field, e.g.

https://icml.cc/Conferences/2019/ScheduleMultitrack?event=4566

The paper should make this sort of related work review, discuss the differences from it, and perform extensive experimental comparisons. 
review_title: Official Blind Review #1

Meta-review: This paper presents a method for merging a discriminative GMM with an ARD sparsity-promoting prior.  This is accomplished by nesting the ARD prior update within a larger EM-based routine for handling the GMM, allowing the model to automatically remove redundant components and improve generalization.  The resulting algorithm was deployed on standard benchmark data sets and compared against existing baselines such as logistic regression, RVMs, and SVMs.

Overall, one potential weakness of this paper, which is admittedly somewhat subjective, is that the exhibited novelty of the proposed approach is modest.  Indeed ARD approaches are now widely used in various capacities, and even if some hurdles must be overcome to implement the specific marriage with a discriminative GMM as reported here, at least one reviewer did not feel that this was sufficient to warrant publication.  Other concerns related to the experiments and comparison with existing work.  For example, one reviewer mentioned comparisons with Panousis et al., "Nonparametric Bayesian Deep Networks with Local Competition," ICML 2019 and requested a discussion of differences.  However, the rebuttal merely deferred this consideration to future work and provided no feedback regarding similarities or differences.  In the end, all reviewers recommended rejecting this paper and I did not find any sufficient reason to overrule this consensus.
Decision: Reject