A Discriminative Gaussian Mixture Model with Sparsity
Authors: Hideaki Hayashi
Seiichi Uchida
Venue: ICLR 2021 Poster (2021)
Keywords: classification
sparse Bayesian learning
Gaussian mixture model

Abstract: In probabilistic classification, a discriminative model based on the softmax function has a potential limitation in that it assumes unimodality for each class in the feature space. The mixture model can address this issue, although it leads to an increase in the number of parameters. We propose a sparse classifier based on a discriminative GMM, referred to as a sparse discriminative Gaussian mixture (SDGM). In the SDGM, a GMM-based discriminative model is trained via sparse Bayesian learning. Using this sparse learning framework, we can simultaneously remove redundant Gaussian components and reduce the number of parameters used in the remaining components during learning; this learning method reduces the model complexity, thereby improving the generalization capability. Furthermore, the SDGM can be embedded into neural networks (NNs), such as convolutional NNs, and can be trained in an end-to-end manner. Experimental results demonstrated that the proposed method outperformed the existing softmax-based discriminative models.

Published as a conference paper at ICLR 2021
A DISCRIMINATIVE GAUSSIAN MIXTURE MODEL WITH SPARSITY
Hideaki Hayashi & Seiichi Uchida
Department of Advanced Information Technology Kyushu University
744, Motooka, Nishi-ku, Fukuoka, 819-0395 JAPAN {hayashi, uchida kyyuuuuueiikkkuuuuuuu kyukku ac. ip
ABSTRACT
In probabilistic classification, a discriminative model based on the softmax func- tion has a potential limitation in that it assumes unimodality for each class in the feature space. The mixture model can address this issue, although it leads to an increase in the number of parameters. We propose sparse classifier based on discriminative GMM, referred to as sparse discriminative Gaussian mixture (SDGM). In the SDGM, a GMM- based discriminative model trained via sparse Bayesian learning. Using this sparse learning framework, remove redundant Gaussian components and reduce the number of parameters used in the remaining components during learning; this learning method reduces the model complexity, thereby improving the generalization capability. Further- more, the SDGM can be embedded into neural networks (NNs), such convo- lutional NNs, and can be trained in an end-to-end manner. Experimental results demonstrated that the proposed method outperformed the existing discriminative models.
simultancously
INTRODUCTION
In probabilistic classification, discriminative model is approach that assigns class label to an input sample æ by estimating the posterior probability P(e æ). The posterior probability P(c æ) should correctly be modeled because it is not only related classification accuracy, but also to the confidence decision making in real-world applications such as medical diagnosis support. In general, the model calculates the class posterior probability using the softmax function after linear feature extraction. Classically, a combination of the kernel method and the softmax function has been used. The recent mainstream method is use deep neural network for representation learning and softmax for the calculation of the posterior probability.
Such general procedure for developing discriminative model potentially contains a limitation due to unimodality. The fetimax-basedd model, such as fully connected (FC) layer with softmax function that often used in deep neural networks (NNs), assumes a unimodal Gaussian distribution for each class (details are shown in Appendix A). Therefore, even if the feature space is transformed into discriminative space via the feature extraction part, P(c x)) cannot correctly be modeled if the multimodality remains, which leads to decrease in accuracy.
Mixture models can address this issue. Mixture models are widely used for generative models, with a Gaussian mixture model (GMM) as typical example. Mixture models are also effective in discriminative models; for example, discriminative GMMs have been applied successfully in various fields, e.g., speech recognition (Tiske et al. 2015 Wang 2007). However, the number of parameters increases the number of mixture components increases, which may lead to over-fitting and an increase in memory usage; this is useful if we can reduce the number of redundant parameters while maintaining multimodality.
In this paper, we propose discriminative model with two important properties; multimodality and sparsity. The proposed model is referred to as the sparse discriminative Gaussian mixture (SDGM). In the SDGM, GMM-base discrininative model is formulated and trained via sparse Bayesian

---

Published as a conference paper at ICLR 2021
16
16.
14-
31
42" d4
3x "44"
d2" 04
-3
-3 d2 0A
14
-15
(A) PC
Goussian process
(c)
Discriminative GMM
(e) SDGM (ouns)
Figure/ 1: Comparison of decision boundaries. The black and green circles represent training samples from classes 1 and 2, respectively. The dashed black line indicates the decision boundary between classes I and 2 and thus satisfies P(c = x) = (c (c=2 2 æ)) = 0.5. The dashed blue and red lines represent the boundaries between the posterior probabilities of the mixture components.
learning. This learning algorithm reduces memory usage without losing generalization capability by obtaining sparse weights while maintaining the multimodality of the mixture model.
The technical highlight of this study is twofold: One is that the SDGM finds the multimodal structure in the feature space and the other is that redundant Gaussian components are removed owing to sparse learning. Figure [ shows comparison of the decision boundaries with other diseriminative models. The two-class data are from Ripley's synthetic data (Ripley 2006), where two Gaussian components are used to generate data for each class. The FC layer with the softmax function, which is often used in the last layer of deep NNs, assumes a unimodal Gaussian for each class, resulting in an inappropriate decision boundary. Kernel Bayesian methods, such as the Gaussian process (GP) classifier (Wenzel et al. (219) and relevance vector machine (RVM) (Tipping 2001), estimate nonlinear decision boundaries using nonlinear kernels, whereas these methods cannot find multimodal structures. Although the discriminative GMM finds multimodal structure, this model retains redundant Gaussian components. However, the proposed SDGM finds a multimodal structure of data while removing redundant components, which leads an accurate decision boundary.
Furthermore, the SDGM can be embedded into NNs, such as convolutional NNs (CNNs), and trained in end-to-end manner with an NN. The proposed SDGM is also considered linear, and sparse expansion of the logistic regression, and thus the SDGM can be used as the last layer of an NN for classification by replacing it with the fully connected (FC) layer with softmax activation function.
mixture, non-
The contributions of this study are as follows:
We propose novel sparse classifier based on a discriminative GMM. The proposed SDGM has both multimodality and sparsity, thereby flexibly estimating the posterior distribution of classes while removing redundant parameters. Moreover, the SDGM automatically deter- mines the number of components by simultaneously removing the redundant components during learning.
From the perspective of the Bayesian kernel methods, the SDGM is considered as the expansion of the GP and RVM. The SDGM estimate the posterior probabilities more flexibly than the GP and RVM owing to multimodality. The experimental comparison using benchmark data demonstrated superior performance the existing Bayesian kernel methods.
This study connects both fields of probabilistic models and NNs. From the equivalence of discriminative model based on Gaussian distribution to an FC layer, demonstrate that the SDGM can be used as a module of deep NN. We also demonstrate that the SDGM exhibits superior performance the FC layer with softmax function via end-to- end learning with an NN on the image recognition task.
2 RELATED WORK AND POSITION OF THIS STUDY
The position of the proposed SDGM among the related methods is summarized Figure 2 In- terestingly, by summarizing the relationships, fields, generative models, discriminative models, and kernel Bayesian methods, are related to each other. Starting from the Gaussian distribution, all the models shown in Figure 2 are connected via
confirm that the three separately developed
2

---

Published as a conference paper at ICLR 2021
four types of arrows. There is undeveloped area in the upper right part, and the development of the area is the contribution of this study.
A (unimodal) Gaussian distribution is used as the most naive generative model in machine learning and is the foundation this relationship dia-
Kond Rayestau mostels
Relevance sestoe machine (RVM)
SDGM
gram. A GMM is the mixture ex-
Gauoian (CIP)
Mistre of GP
Kernel
pansion of the Gaussian distributions. Since the GMM can express (al- most) arbitrary continuous distribu- tions using multiple Gaussian com- ponents, it has been utilized for long time. Since Gaussian fitting requires numerous parameters, the sparsified versions of Gaussian (Hsich et al. 2011 and GMM (Gaiflas & Michel 2014 have been proposed.
Diserminatvv mestals
Sparse logistic reppesim
Lngistic segrooimm
Disciminative GMM
Discriminative
FC layer sullma
Geserative moddla
Sparse Gasssiat
Spunse GMM
Sparee
Unimanhal Chrseaan
Gasasias misiure
model (GMM)
The discriminative models and the generative models are mutually re- lated (Lasserre al.] 2006] Minka
Figure 2: Position of the SDGM among the related methods.
2005). According to Lasserre et al. (2006), the only difference between these models is their sta- tistical parameter constraints. Therefore, given a generative model, can derive corresponding discriminative model. For example, discriminative models corresponding to the Gaussian mixture model have been proposed (Axelrod et al. 2006 Bahl al. 1996 Klautau al. 2003 Tsai & Chang 2002 Tsuji et al. 1999 Tüske et al. 2015 Wang (007). They indicate more flexible fitting capa- bility for classífication problems than the generative GMM because the discriminative models have lower statistical bias than the generative models. Furthermore, as shown by Tüsske et al. (2015); Variani et al. (2155), these models can be used as the last layer of the NN because these models output the class posterior probability.
From the perspective of the kernel Bayesian methods, the GP classifier (Wenzel et al. 2019) and the mixture of GPs (MGP) (Luo & Sun 2017) are the Bayesian kernelized version of the logistic regres- sion and the discriminative GMM, respectively. The SDGM with kernelization is also regarded as kernel Bayesian method because the posterior distribution of weights is estimated during learning instead of directly estimating the weights as points, as with the GP and MGP. The RVM Tipping 2001) is the sparse version of the GP classifier and is the most important related study. The learning algorithm of the SDGM is based on that of the RVM; however, it is extended for the mixture model.
If use kernelization, the SDGM becomes one of the kernel Bayesian methods and considered as the mixture expansion of the RVM or sparse expansion of the MGP. Therefore, the classification capability and sparsity are compared with kernel Bayesian methods Section 4.1 Otherwise, the SDGM is considered as one of the discriminative models and can be embedded in an NN. The comparison with other discriminative models is conducted in Section 4.2 via image classification by combining CNN.
SPARSE DISCRIMINATIVE GAUSSIAN MIXTURE (SDGM)
The SDGM takes continuous variable as its input and outputs the posterior probability of each class, acquiring a sparse structure by removing redundant components via sparse Bayesian learning. Figure how the SDGM is trained by removing unnecessary components while maintaining discriminability. In this training, we set the initial number of components to three for each class. As the training progresses, one of the components for each class gradually becomes small and is removed.
3.1 NOTATION
Let x E RR' D be continuous input variable and t, (c E (1...., C), C' is the number of classes) be a discrete target variable that is coded in one-of-C form, where I, = 1 if a belongs to class c, l,
3

---

Published as a conference paper at ICLR 2021
1.6
161
16-
1.4
1.4
14
14
1.2
100
0.
0.8
08
08
0.0
06
06
06
0-
42
-02
40
10
05
03 (b) 100 opochs
-.0
-15
(a) epoch
(c) 200 epochs
(d) 250 epochs
Figure 3: Snapshots of the training process of SDGM. The meanings of lines and circles the same as Figure There are three components for each class in the initial stage of learning. As the training progresses, one of the components for each class becomes small gradually and is finally removed.
otherwise. Also, let 7cm be discrete latent variable, and 7em = when from class belongs the m-th component (m E .I...., Mc). Mr. the number components for class c), Zem otherwise. For simplicity, in this paper, the probabilities for classes and components are described using only and m;; e.g., we use Pcc, m x) instead P(te = 1, žcm x).
3.2
MODEL FORMULATION
The posterior probabilities of each class given a are calculated as follows:
Tem exp w
#)c| = Σ P(e. x), P(e.m w)
(1)
Te'm exp u
1, " .1.,,1132..... FIED, 22: 1243
(2)
where Tom is the mixture weight that is equivalent to prior each component P(c, m). should be noted that we use Wem E HR which is the weight vector representing the m--th Gaussian component of class e. The dimension of wems i.e. H, the same as that of Φ; namely, H 1 D(D + 3)/2.
Derivation. Utilizing Gaussian distribution conditional distribution of given e and m, P(æ: e, m), the posterior probability of given æ, P(e æ), calculated follows:
Me
Plc, P(x | m)
P(e (c]
(3)
=l Dm Píc, P( c.
P(x c.
(2m) 0
exp
(x
pm " pmm
(4)
where Mem E RDD and Eem E DxD are the mean vector and the covariance matrix for component m in class Since the calculation inside exponential function in (4) is quadratic form, the conditional distributions be transformed follows:
P( m) = exppw Φ).
(5)
where
D 2m lu Deml
Semsy
Semsl
T
Beni D Delmms
Aemil, Bem12, 8cml 8em22,
8cm DD
(6)
Here, Semij is the (i. element of E-1
3.3 DUAL FORM VIA KERNELIZATION
Since Φ is second-order polynomial form, derive the dual form of the SDGM using polyno- mial kernels. By kernelization, we can treat the SDGM as the kernel Bayesian method as described in Section
4

---

Published as a conference paper at ICLR 2021
Let Pem E R.W be novel weight vector for the c, m-th component. Using Hen and the training
dataset {In In= the weight of the original form Wem is represented as
Wem. (d()))... ((ax)]yb..
where ((x,,) the transformation Iy of using
2 Then, reformulated as follows:
P(æ e. m)) = exp ( : +b
exp explv (x)))) ((x). ((xx)"((()))
(8) =
where K(X, x) is an N-dimensional vector that contains kernel functions defined as (x x) ((xx) ((xx) (a.;, x 1)2 for its elements and X data matrix that has x. in the n-th row. Whereas the computational complexity of the original form in Section 3.2 increases in the order of the square of the input dimension D, the dimensionality of this dual form is proportional to When we use this dual form, we use N and k(Ens)) instead of H and Φ(-), respectively.
3.4 LEARNING ALGORITHM
set of training data and target value f Inc] (n = 1,..... is given. We also define æ and as vectors that comprise Tem and Znem as their elements, respectively. As the prior distribution of the weight Wemk, we employ Gaussian distribu- tion with mean of zero. Using different precision parameter (inverse of the variance) Oremh. for each weight Wemb. the joint probability of all the weights
[ ((mms))
p mme (ema))
Mewch o0
a
Womh
Wonsh
represented follows:
Figure 4: Prior for each weight Wembh By maximizing the evidence term of the poste- rior of w, the precision of the prior aemh achieves infinity if the corresponding weight Wemb is redundant.
P(wl | ())
Mc IIIIIII
Clemh
exp
Wemb Ckemt
(9)
2
where and vectors with Wemh and aanh their elements, respectively. During learning, we update not only w but also a. If Oamh >> oc, the prior (9) is O; hence sparse solution is obtained by optimizing as shown in Figure 4
Using these variables, the expectation of the log- liglliillhood function over z, J, defined as follows:
J J= [n X, W. a)) =
Anc In P(c, m an),
where T is a matrix with tne its element. The variable Tnom in the right-hand side corresponds to P(m c, a.)) and can be calculated as Tnom = P(c, m æx)/ /(c æn).
The posterior probability of the weight vector w is described as follows:
P(w T. T, P(T, X, a(w
(10)
P(T (
An optimal w is obtained the point where 10) is maximized. The denominator of the right-hand side in (10) is called the evidence term, and we maximize with respect to a. However, this maxi- mization problem cannot be solved analytically; therefore we introduce the Laplace approximation
described as the following procedure.
With a fixed, we obtain the mode of the posterior distribution of w. The solution is given by the point where the following equation maximized:
Ex [ln P(w T,2,X, ax) E. [T,P(T, X. w. mx)] + InP(w (x) + const.
= T Aw + coust.,
(11)
where A = diag Orenh- We obtain the mode of II via Newton's method. The gradient and Hessian required for this estimation can be calculated as follows:
VR, [In P(w
T, z, T,), VJ Aw.
(12)
5

---

Published as a conference paper at ICLR 2021
Algorithm 1: Learning algorithm of the SDGM
Input:
Training data set and teacher vector 'T
Output: Trained weight w obtained by maximizing I1
Initialize the weights w. hyperparameters a, mixture coefficients म,, and posterior probabilities r;; while a have not converged do
Calculate J using (10):
while have not converged do
while u have not converged do
Calculate gradients using 12
Calculate Hessian 13):
Maximize 11] w.r.t
Calculate [(e m (x.) and P(e wn);
end
Twom P(a. m w.)//((c] a..):
end
Calculate A using 16
Update using
Update using
end
VVE. [h P(w P T, X,,,,, VVJ A.
(13)
Each element of VJ and V is calculated as follows:
ØJ
(Fnemtne
P(c, m zn))dn
(14)
@Wenc
8
Plé,m' m'| (P((( æ.) dent Jonon,
(15)
Ww. OW
where Sec' mmn' is a variable that takes if both = and m = m', 0 otherwise. Hence, the posterior distribution of w can be approximated by Gaussian distribution with a mean of to and covariance matrix of A, where
A (VVE P(t) xT,xX,
(16)
Since the evidence term can be represented using the normalization term of this Gaussian distribu- tion, obtain the following updating rule by calculating its derivative with respect to Ocamb-
dlemh
Olernch
2 cmk
(17)
where Aemh is the diagonal component of A. The mixture weight Tem can be estimated using Tnom as follows:
Tem
Nc
Σ
Fncm:
(18)
where N₂ is the number of training samples belonging class As described above, obtain a sparse solution by alternately repeating the update of pyper-parameters, as in (17) and (18). and the posterior distribution estimation of w using the Laplace approximation. As result of the op- timization, some of aembh approach to infinite values, and Wenk corresponding to aemh have prior distributions with mean and variance both zero shown in (4); hence such Wems are removed be- cause their posterior distributions are also with mean and variance both zero. During the procedure, the {c, m)-th component is eliminated if Tem becomes all the weights Wemh corresponding to the component become O.
The learning algorithm the SDGM summarized Algorithm | In this algorithm, the optimal weight is obtained as maximum a posterior solution. We can obtain sparse solution by optimizing the prior distribution set to each weight simultancously with weight optimization.
EXPERIMENTS
4.1 COMPARATIVE STUDY USING BENCHMARK DATA
To evaluate the capability of the SDGM quantitatively, we conducted a classification experiment using benchmark datasets. The datasets used in this experiment were Ripley's synthetic data (Ripley

---

Published as a conference paper at ICLR 2021
Table 1: Recognition error rate (%) and number of nonzero weights
Accuracy (error rate (%)
Sparsity (number of nonzero weights)
Baselines
Baselines
Dataset
SDGM
GP
MGP
RVM
SDGM
GP
MGP
RVM
Ripley
9.1
9.3
9.3
250
1250
Waveform
10.1
10.4
9.6
10.9
11.0
400
2000
14,6
Banana
10.6
10.5
10.7
10.8
11.1
400
2000
11.4
Titanic
22.6
22.6
22.8
23.0
74.5
150
750
65.3
Breast Cancer
29.4
30.4
30.7
29.9
15.7
200
1000
6.3
Normalized mean
1.00
1.01
1.01
1.03
1.00
25.76
128.79
0.86
2006 (Ripley hereinafter) and four datasets cited from (Rätsch al. 2001): Banana, Waveform, Titanic, and Breast Cancer. Ripley is a synthetic dataset that is generated from a two-dimensional
(D 2) Gaussian mixture model, and 250 and 1,000 samples are provided for training and test, respectively. The number of classes is two (C = 2), and each class comprises two components. The remaining four datasets are all two-class (C 2) datasets, which comprise different data sizes and dimensionality. Since they contain 100 training/test splits, we repeated experiments 100 times and then calculated average statistics.
For comparison, we used three kemee Bayesian methods: GP classifier, an MPG classifier (Tresp 2001 Luo & Sun 2017). and an RVM (Tipping 2001). which are closely related the SDGM from the perspective of sparsity, multimodality. and Bayesian learning, as described in Section 2 In the evaluation, we compared the recognition error rates for discriminability and the number of nonzero weights for sparsity the test data. The results of RVM were cited from (Tipping 2001). By way of summary, the statistics were normalized by those of the SDGM, and the overall mean was shown.
Table D shows the recognition error rates and the number of nonzero weights for each method. The results in Table [ demonstrated that the SDGM achieved better accuracy on average compared to the other kernel Bayesian methods. The SDGM is developed based Gaussian mixture model and is particularly effective for data where a Gaussian distribution can assumed, such as the Ripley dataset. Since the SDGM explicitly models multimodality, it could more accurately represent the sharp changes in decision boundaries near the border of components compared to the RVM, as shown in Figure [ Although the SDGM did not necessarily outperform the other methods in all datasets, it achieved the best accuracy on average. In terms of sparsity, the number of initial weights for the SDGM is the same as MGP, and the SDGM reduced 90.0-99.5% of weights from the initial state due to the sparse Bayesian learning, which leads to drastically efficient use of memory compared non-sparse classifiers (GP and MGP). The results above indicated that the SDGM demonstrated generalization capability and sparsity simultaneously.
4.2 IMAGE CLASSIFICATION
In this experiment, the SDGM is embedded into a deep neural network. Since the SDGM is differ- entiable with respect the weights, SDGM can be embedded into deep NN module and is trained in an end-to-end manner. In particular, the SDGM plays the same role as the softmax func- tion since the SDGM calculates the posterior probability of each class given an input vector. We can show that fully connected layer with the softmax is equivalent to the discriminative model based on single Gaussian distribution for each class by applying a simple transformation (see Appendix A), whereas the SDGM is based on the Gaussian mixture model.
To verify the difference between them, we conducted image classification experiments. Using CNN with softmax function baseline, we evaluated the capability of SDGM by replacing soft- max with the SDGM. We also used a CNN with softmax function trained with T,, regularization, CNN with large margin softmax (Liu et al. 2016), and CNN with the discriminative GMM as other baselines.
In this experiment, used the original form of the SDGM. To achieve sparse optimization during end-to-end training, we employed an approximated sparse Bayesian learning based on Gaussian dropout proposed by Molchanov et al.) (2017). This is because it is difficult to execute the learning algorithm in Section 3.4 with backpropagation due to large computational costs for inverse matrix calculation of the Hessian in (16), which takes O(N3).

---

Published as a conference paper at ICLR 2021
Table 2: Recognition error rates (%) on image classification
MNIST
MNIST
Fashion MNIST
Normalized
(D = 2)
(D) = 10)
CIFAR-10
CIFAR-100
ImtgeNet
mean
Softmax
3.19
1.01
8.78
11.07
22.99
33,45
1.20
Softmax + Lg regularization
3.70
1.58
9.20
10.30
21.56
32.43
1.29
Large margin softmax
2.52
0.80
8.51
11.58
21,00
33.19
1,08
Discriminative GMM
2.43
0.72
8.30
10.05
21.93
34.75
1.05
SDGM
200
Training data
Test data
Training data
25
Test dasa
150
150
20
100
15
10
50
100
200
50
1.0
200
(n) Softmax
(b) Ours
Figure 5: Visualization of CNN features on MNIST (D = 2) after end-to-end learning. In this visualization, five convolutional layers with four max pooling layers between them and a fully con- nected layer with two-dimensional output are used. (a) results when fully connected layer with the softmax function is used as the last layer. (b) when SDGM is used as the last layer instead. The colors red, blue, yellow, pink, green, tomato, saddlebrown, lightgreen, cyan, and black represent classes from 0 to 9, respectively. Note that the ranges of the axis are different between (a) and (b).
4.2.1 DATASETS AND EXPERIMENTAL SETUPS
We used the following datasets and experimental settings in this experiment.
This dataset includes 10 classes of handwritten binary digit images of size 28 x 28 (LeCun ct al. 1998). We used 60,000 images as training data and 10,000 images as testing data. As a feature extractor, we used a simple CNN that consists of five convolutional layers with four max pooling layers between them and a fully connected layer. To visualize the learned CNN features, we first set the output dimension of the fully connected layer of the baseline CNN two (D 2). Furthermore, we tested by increasing the output dimension of the fully connected layer from two to ten (D = 10).
MNIST:
Fashion-MNISTI Fashion-MNIST (Xiao et al. 2017) includes 10 classes of binary fashion images with a size of 28 x 28. includes 60,000 images for training data and 10,000 images for testing data. We used the same CNN in MNIST with 10 the output dimension.
CIFAR-10 and CIFAR-100: CIFAR-10 and CIFAR- 100 (Krizhevsky & Hinton 2009) consist of 60,000 32 32 color images in 10 classes and 100 classes, respectively. There 50,000 training images and 10,000 test images for both datasets. For these datasets, we trained DenseNet (Huang al. 2017) with a depth of 40 and growth rate of as a baseline CNN.
ImageNet: ImageNet classification dataset (Russakovsky et al. 2015) includes 1,000 classes of generic object images with size of 224 x 224. consists of 1,281,167 training images, 50,000 validation images, and 100,000 test images. For this dataset, we used MobileNet (Howard et al. 2017) as a baseline CNN.
It should be noted that we did not employ additional techniques to increase classification accuracy such as hyperparameter tuning and pre-trained models; therefore, the accuracy of the baseline model did not reach the state-of-the-ane This is because we considered that it is not essential to confirm the effectiveness of the proposed method.
4.2.2 RESULTS
Figure shows the two-dimensional feature embeddings on the MNIST dataset. Different feature embeddings were acquired for each method. When softmax was used, the features spread in fan shape and some parts the distribution overlapped around the origin. However, when the SDGM was used, the distribution for each class exhibited ellipse shape and margins appeared between the
8

---

Published as a conference paper at ICLR 2021
class distributions. This is because the SDGM is based on a Gaussian mixture model and functions to push the features into a Gaussian shape.
Table 2 shows the recognition error rates each dataset. SDGM achieved better performance than soltmax. Although sparse learning was ineffective in two out of six comparisons according to the comparison with the discriminative GMM, replacing softmax with SDGM was effective in all the comparisons. As shown in Figure 5 SDGM can create margins between classes by pushing the features into Gaussian shape. This phenomenon positively affected classification capability. Although large-margin softmax, which has the effect of increasing the margin, and the discriminative GMM, which can represent multimodality, also achieved relatively high accuracy, the SDGM can achieve the same level of accuracy with sparse weights.
CONCLUSION
In this paper, we proposed a sparse classifier based on a Gaussian mixture model (GMM), which named sparse discriminative Gaussian mixture (SDGM). In the SDGM, GMM-based discrim- inative model was trained by sparse Bayesian learning. This learning algorithm improved the gen- eralization capability by obtaining sparse solution and automatically determined the number of components by removing redundant components. The SDGM can be embedded into neural net- works (NNs) such as convolutional NNs and could be trained in an end-to-end manner.
In the experiments, we demonstrated that the SDGM could reduce the number of weights via sparse Bayesian learning, thereby improving its generalization capability. The comparison using bench- mark datasets suggested that SDGM outperforms the conventional kernel Bayesian classifiers. We also demonstrated that SDGM outperformed the fully connected layer with the softmax function when it was used as the last layer of deep NN.
One of the limitations of this study is that the proposed sparse learning reduces redundant Gaussian components but cannot obtain the optimal number of components, which should be improved in future work. Since the learning of the EM algorithm into the sparse Bayesian learning, we will tackle theoretical analysis by utilizing the proofs for the EM algorithm (Wu 1983) and the sparse Bayesian learning (Faul & Tipping 2001). Furthermore, the PAC-Bayesian theorem. We will also develop a sparse learning algorithm for whole deep NN structure including the feature extraction part. This will improve the ability of the CNN for larger data classification. Further applications using the probabilistic property of the proposed model such as semi-supervised learning, uncertainty estimation, and confidence calibration will be considered.
the proposed method can be interpreted as the incorporation
would like to tackle the theoretical analysis of error bounds using
ACKNOWLEDGMENTTS
This work was supported in part by JSPS KAKENHI Grant Number JP17K12752 and JST ACT-I Grant Number SMMJPRI8UO.
REFERENCES
Scott Axelrod, Vaibhava Goel, Ramesh Gopinath, Peder Olsen, and Karthik Visweswariah. Dis- criminative estimation of subspace constrained Gaussian mixture models for speech recognition. IEEE Transactions on Audio, Speech, and Language Processing. 15(1):172-189 2006.
Lalit R Bahl, Mukund Padmanabhan, David Nahamoo, and PS Gopalakrishnan. Discriminative training of Gaussian mixture models for large vocabulary speech recognition systems. In Pro- the International Conference on Acoustics, Speech, and Signal Processing Conference Proceedings (ICASSP), volume 2, pp. 613-616, 1996.
ceedings the
Anita Faul and Michael Tipping. Analysis of sparse Bayesian learning. Proceedings of the Advances in Neural Information Processing Systems (NIPS), (4:383-389, 2001.
Stephane Gaiffas and Bertrand Michel. Sparse bayesian unsupervised learning. arXiv preprint arXiv: arXiv:1401.800 /401.8017, 2014.

---

Published as a conference paper at ICLR 2021
Andrew G Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andrectto, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv: 2017.
Cho-Jui Hsieh, Inderjit S Dhillon, Pradeep K Ravikumar, and Mátýás A Sustik. Sparse inverse covariance matrix estimation using quadratic approximation. In Proceedings of the Advances in Neural Information Processing Systems (NIPS), pp. 2330-2338, 2011.
Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Weinberger. Densely connected convolutional networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
Aldebaro Klautau, Nikola Jevtic, and Alon Orlitsky. Discriminative Gaussian mixture models: A comparison with kernel classifiers. In Proceedings of the International Conference Machine Learning (ICML), pp. 353-360, 2003.
Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Tech- nical report, University of Toronto, 2009.
Julia A. Lasserre, Christopher M. Bishop, and Thomas P. Minka. Principled hybrids of genera- tive and discriminative models. In Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR), pp. 87 87-94, 2006.
Yann LeCun, Léon Bottou, Yoshua Bengio, Patrick Haffner, et al. Gradient- learning applied to document recognition. Proceedings of the IEEE, 86(1 1):2278-2223 22244, 1998.
Weiyang Liu, Yandong Wen, Zhiding Yu, and Meng Yang. Large- margin softmax loss for convo- lutional neural networks. In Proceedings of the International Conference on Machine Learning (ICML), volume 48, pp. 507-516, 2016.
Chen Luo and Shiliang Sun. Variational mixtures of gaussian processes for classification. In Pro- ceedings of the International Joint Conferences on Artificial Intelligence (IJCAI), pp- 4603-4600 2017.
Tom Minka. Discriminative models, not discriminative training. Technical report, Technical Report MSR-TR-2005-11 144,,,00-11,, Microsoft Research, 2005.
Dmitry Molchanov, Arseni Ashukha, and Dmitry Vetrov. Variational dropout sparsifies deep neural networks. In Proceedings of the International Conference on Machine Learning (ICML), pp- 2017.
Gunnar Rätsch, Takashi Onoda, and K-R Müller. Soft margins for adaboost. Machine learning. 42 (3):287-320, 2001.
Brian D Ripley. Pattern Recognition and Neural Networks. Cambridge University Press, 2006.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma. Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (IJV),, 115(3)211-255 2015. doi: 10.1007/s1 1263-015-0816-y.
Michael E Tipping. Sparse Bayesian learning and the relevance vector machine. Journal of Machine Learning research, 1(Jun):211 1-244, 2001.
Volker Tresp. Mixtures of gaussian processes. In Proceedings of the Advances Neural Information Processing Systems (NIPS), pp. 654-660, 2001.
Wuei-He Tsai and Wen-Whei Chang. Discriminative training of Gaussian mixture bigram mod-
els with application to Chinese dialect identification. Speech Communication, 36(3-4):11-326, 2002.
Toshio Tsuji, Osamu Fukuda, Hiroyuki Ichinobe, and Makoto Kaneko. A linearizd Gaussian mixture network and its application to EEG pattern classification. IEEE Transactions on Systems, Man, and Cybernetics, Part C: Applications and Reviews, 29(1 29(00-0-7 1999.
10

---

Published as a conference paper at ICLR 2021
Zoltán Tüske, Muhammad Ali Tahir, Ralf Schlüter, and Hermann Ney. Integrating Gaussian mix- tures into deep neural networks: Softmax layer with hidden variables. In Proceedings tf IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), pp. 4285-4289, 2015.
Ehsan Variani, Erik McDermott, and Georg Heigold. A Gaussian mixture model layer jointly opti- mized with discriminative features within a deep neural network architecture. In Proceedings of the International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 4270-
4274, 2015.
Jue Wang. Discriminative Gaussian mixtures for interactive image segmentation. In Proceedings of the IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), volume 1, pp. 601-604, 2007.
Florian Wenzel, Théo Galy-Fajou, Christan Donner, Marius Kloft, and Manfred Opper. Efficient gaussian process classification using polya-gamma data augmentation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pp. 5417-5424, 2019.
CF Jeff Wu. On the convergence properties of the EM algorithm. The Annals of statistics, pp- 95-103, 1983.
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-MNIST: a novel image dataset for bench- marking machine learning algorithms. arXiv preprint arXiv: 1rXiv:1008.07747, 2017.

---

Published as a conference paper at ICLR 2021
SUPPLEMENTARY MATERIALS
A RELATIONSHIP BETWEEN THE DISCRIMINATIVE GAUSSIAN AND LOGISTIC REGRESSION
We show that a fully connected layer with the softmax function, or logistic regression, can be re- garded as a discriminative model based on a Gaussian distribution by utilizing transformation of the equations. Let us consider a case which the class-conditional probability P(æ|c) is a Gaussian distribution. In this case, we can omit m from the equations
If all classes share the same covariance matrix and the mixture weight Tem. the terms in "f, T2 ff;; ,, 222 T2.3, p........ z in (2). and $8111.... 2DDD in can be
canceled; hence the calculation of the posterior probability P(c(x) is also simplified as
exp(w." T ())
P(c(z)) = Σ exp(w. T (b)
where
w. =log P(c)
D
Σ
D
1 SeifHeilles + log 2n + log Ecl,
D ScilMat 1=1
D
SciDMal
1=1
= 1, 2
This is equivalent to fully connected layer with the softmax function, or linear logistic regression.
B EVALUATION OF CHARACTERISTICS USING SYNTHETIC DATA
To evaluate the characteristics of the SDGM, we conducted classification experiments using syn- thetic data. The dataset comprises two classes. The data were sampled from a Gaussian mixture model with eight components for each class. The numbers of training data and test data were 320 and 1,600, respectively. The scatter plot of this dataset shown Figure/6
In the evaluation, we calculated the error rates for the training data and the test data, the number of components after training, the number of nonzero weights after training, and the weight reduction ratio (the ratio of the number of the nonzero weights the number of initial weights), by varying the number of initial components as 2, 2, ........ ...,..., 20. 20. We repeated evaluation five times while regen- erating the training and test data and calculated the average value for each evaluation criterion. We used the dual form of the SDGM in this experiment.
Figure 6 displays the changes in the learned class boundaries according the number of initial components. When the number of components is small, such as that shown in Figure G(a). the decision boundary is simple; therefore, the classification performance is insufficient. However, according to the increase in the number of components. the decision boundary fits the actual class boundaries. It is noteworthy that the SDGM learns the GMM as a discriminative model instead of generative model; an appropriate decision boundary was obtained even if the number of components for the model less than the actual number (.g..())))
Figure 7 shows the evaluation results of the characteristics. Figures Ta), (b), (c), and (d) show the recognition error rate, number of components after training, number of nonzero weights after training, and weight reduction ratio, respectively. The horizontal axis shows the number of initial components in all the graphs.
In Figure a)) the recognition error rates for the training data and test data are almost the same with the few numbers of components and decrease according to the increase in the number of initial components while it is 2 to 6. This implied that the representation capability was insufficient when the number of components was small, and that the network could not accurately separate the classes. Mcanwhile, changes in the training and test enror rates were both flat when the number of initial components exceeded eight, even though the test enror rates were slightly higher than the training emrrr rate. In general, the training error decreases and the test error increases when the complexity of
12

---

Published as a conference paper at ICLR 2021
-10
-10
T0
components
components
(c) components
(d) 16 components
Figure 6: Changes learned class boundaries according to the number of initial components. The blue and green markers represent the samples from class and class 2, respectively. Samples in red circles represent relevant vectors. The black lines are class boundaries where P(c æ)) 0.5.
Triining cime
Tou erner
8
19.0
12 of mnttalll cosppacctt
4
16 18.
8 10 of
30
componus Number of componeets
componens
dofintaa componens
(A) Eror cate
zero weights
(8) Weight reductioon
Figure Evaluation results using synthetic data. (a) recognition error rate, (b) the number of components after training, (c) the number of nonzero weights after training, and (d) weight reduction ratio. The error bars indicate the standard deviation for five trials.
the classifier is increased. However, the SDGM suppresses the increase in complexity using sparse Bayesian learning, thereby preventing overfitting.
In Figure bb) the number of components after training corresponds to the number of initial com- ponents until the number of initial components eight. When the number of initial components exceeds ten, the number of components after training tends to be reduced. In particular, eight com- ponents are reduced when the number of initial components is 20. The results above indicate the SDGM reduce unnecessary components.
From the results in Figure Mc., we confirm that the number of nonzero weights after training in- creases according to the increase in the number of initial components. This implies that the com- plexity of the trained model depends on the number of initial components, and that the minimum number of components is not always obtained.
Meanwhile, in Figure dd) the weight reduction ratio increases according to the increase in the number initial components. This result suggests that the larger the number of initial weights, the more weights were reduced. Moreover, the weight reduction ratio is greater than 99 % in any case. The results above indicate that the SDGM can prevent overfitting by obtaining high sparsity and can reduce unnecessary components.
C DETAILS OF INITIALIZATION
In the experiments during this study, each trainable parameters for the m-th component of the c-th class were initialized as follows (H = 1 + D(D 3)/2, where D is the input dimension, for the original form and = N, where N is the number of the training data, for the kernelized form):
Wem (for the original form): zero vector "!
De (for the kernelized form): A zero vector 0 R
Oeam An all-ones vector E "
Tem: A scalar Σe_ M, where is the number of classes and M, is the number com- ponents for the e-th class.
13

---

Published as a conference paper at ICLR 2021
Tnem: Initialized based on the results of k-means clustering applied to the training data; Tnem = 1 if the n-th sample belongs to class c and assigned to the m-th component by k-means clustering, nnm = 0 otherwise.
14


Reviews:
[{"review_id": "m-TR3nUUftG", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "* quality\nThe paper presents an interesting idea that uses sparse Gaussian mixtures, but it lacks theoretical guarantees. Although the method is Bayesian, can we also give frequentist non-asymptotic bounds?\n* clarity\nThe paper is well written.\n* originality\nThe paper's ideas seem original, but they're very straightforward, making its contributions marginally incremental.\n* significance\nIf the paper had more theoretical guarantees, its results would be more significant. The current version is a bit weak. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "fR0c44FfuR", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes an interesting extension to both discriminative GMMs and CNNs with a softmax outuput. The key innovation is the introduction of the sparsity in discriminative, multimodal settings. This novelty and the clear experiments merit this work to be published at ICLR 21. However, the testing and evaluation could be significantly improved.\n\nThere are a few areas where the paper could be improved.\n1. The language could use another review. Often, the writing slips into the use of informal language.\n2. Appendices were missing in my version; the paper makes an explicit reference to at least appendix A.\n3. The connection to Bayesian methods is week.\n4. The table describing Algorithm 1 appears abruptly and references formulae that have not been introduced. It is best positioned at the end of Section 3 to improve readability. \n5. Most of the Bayesian approaches used in the comparison are dated.\n6. Section 4.2 references high computational costs, but it is unclear which steps of the algorithm make this approach computationally prohibitive.\n7. It is unclear from Table 1 that SDGM is significantly better than RVM. A deep dive is warranted to better understand their relative performances. \n8. It would be better if Section 4.2.1 also include large datasets or more challenging domains. The relative error reduction described in Table 2 on a couple of datasets are marginal.", "extra_scores": {}, "extra_text": {}}, {"review_id": "YsCHMbteZrk", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The innovative part of the work consists in the trick (8), which allows for a kernel-based generalisation of Gaussian mixture models viewed under a discriminative perspective.  \n\nThe model is trained via maximum a posteriori estimation; due to the analytically intractable form of the posterior expectation of the log-likelihood, the authors resort to a second-order Taylor approximation around the mode (Laplace approximation). That's an old and not so fine approximation, but easy and widely-known. \n\nThe experiments are performed on standard benchmarks, and comparisons are provided against some natural competitors of the method. In essence, the authors replace the penultimate softmax layer of a deep net classifier with their model and train end-to-end.\n\nThe experiments yield a marginal improvement over s.o.t.a., as so many papers do nowadays. An aspect that is missing is how they initialised the models. Initialisation may  play a crucial role in such approaches, especially when it comes to ensuring reproducibility of the results. \n\nIn summary, a slightly novel paper with some interesting insights and some pretty standard nowadays, yet marginally impressive experimental results. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "CQI-GN1AEhz", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes a sparse classifier via  discriminative GMM. This model is trained based on sparse Bayesian learning. The sparsity constraint removes redundant Gaussian components which results in  reducing the number of parameters and improving the generalization. This framework can potentially be embedded into the deep models and trained in an end-to-end fashion. The main motivation is that the proposed model (i.e., SDGM,)  can consider multimodal data while conventional softmax classifiers only assume unimodality for each class. Experimental results show the superiority of the SDGM over existing softmax-based discriminative models.\n\nThe paper is well-written and easy to follow. And the paper precisely places the proposed method among the related work.  However, there are some concerns:\n\n1- Sparsity constraint is supposed to improve the performance and generalization, but experimental results do not support this. What is the motivation of employing sparsity learning in this framework? \n\n2- Conventional softmax classifiers in deep architecture have already provided promising results on real world datasets such as ImageNet which contains classes with multimodal data. However, in this paper, experiments are performed on the small datasets, so was the proposed method evaluated on ImageNet as well? \n\n\n3- There are many versions of softmax-classifiers such as “large-margin softmax”, “angular softmax”, and “additive margin softmax” , and  [1 ,2 ] that address conventional softmax-classifiers’ issues. Have you compared your models with them?\n\n[1] Liu W, Wen Y, Yu Z, Yang M. Large-margin softmax loss for convolutional neural networks. InICML 2016 Jun 19 (Vol. 2, No. 3, p. 7).\n\n[2] Weiyang Liu, Yandong Wen, Zhiding Yu, Ming Li, Bhiksha Raj, and Le Song. SphereFace: Deep hypersphere embedding for face recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 212–220, 2017\n\n[3] Feng Wang, Jian Cheng, Weiyang Liu, and Haijun Liu. Additive margin softmax for face verification. IEEE Signal Processing Letters, 25(7):926–930, 2018.\n\n[4] Kaidi Cao, Colin Wei, Adrien Gaidon, Nikos Arechiga, and Tengyu Ma. Learning imbalanced datasets with label-distribution-aware margin loss. In Advances in Neural Information Processing Systems, pp. 1567–1578, 2019.\n\n[5] Bin Liu, Yue Cao, Yutong Lin, Qi Li, Zheng Zhang, Mingsheng Long, and Han Hu. Negative margin matters: Understanding margin in few-shot classification. arXiv preprint arXiv:2003.12060, 2020.\n\n4-  Finally, I could not find any guarantee for the convergence of the learning algorithm in the paper? What is the time-complexity?  I think training with softmax would be much faster, easier and also provides promising classification results in practice.\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes replacing the softmax of deep NNs with a kernel-based Gaussian mixture model, to allow for per-class multi-modality.  Results show that the method is competitive with other output modifications such as the large-margin softmax.  

The  two primary concerns of the reviewers were the lack of large-scale image classification results and theoretical guarantees.  The authors have added CIFAR-100 results.  Moreover, the authors agree that theoretical results would be nice to have, but such results are non-trivial and likely require a PAC-Bayes treatment.

I find the method to be well-motivated and that the paper demonstrates sufficient experimental rigor.  Given the popularity of the softmax throughout deep learning, this paper will likely be of interest---or at least, be of potential use---to a large part of the ICLR community.  I encourage the authors to add the ImageNet results to the final version.

Decision: Accept (Poster)