SDGM: Sparse Bayesian Classifier Based on a Discriminative Gaussian Mixture Model
Authors: Hideaki Hayashi
Seiichi Uchida
Venue:  (2020)
Keywords: classification
sparse Bayesian learning
Gaussian mixture model

Abstract: In probabilistic classification, a discriminative model based on Gaussian mixture exhibits flexible fitting capability. Nevertheless, it is difficult to determine the number of components. We propose a sparse classifier based on a discriminative Gaussian mixture model (GMM), which is named sparse discriminative Gaussian mixture (SDGM). In the SDGM, a GMM-based discriminative model is trained by sparse Bayesian learning. This learning algorithm improves the generalization capability by obtaining a sparse solution and automatically determines the number of components by removing redundant components. The SDGM can be embedded into neural networks (NNs) such as convolutional NNs and can be trained in an end-to-end manner. Experimental results indicated that the proposed method prevented overfitting by obtaining sparsity. Furthermore, we demonstrated that the proposed method outperformed a fully connected layer with the softmax function in certain cases when it was used as the last layer of a deep NN.

Under review as a conference paper at ICLR 2020
SDGM: SPARSE BAYESIAN CLASSIFIER BASED ON A DISCRIMINATIVE GAUSSIAN MIXTURE MODEL
Anonymous authors Paper under double-blind review
ABSTRACT
In prohabilistic classification, a discriminative model based on Gaussian mixture exhibits flexible fitring capability. Nevertheless, it is difficult to determine the number of components. We propose a sparse classifier based on a discriminative Gaussian mixture model (GMM), which is named sparse discriminative Gaussian mixture (SDGM). In the SDGM, a GMM-based discriminative model is trained by sparse Bayesian learning. This learning algorithm improves the generalization capability by obtaining a sparse solution and automatically determines the number of components by removing redundant components. The SDGM can be embed- ded into neural networks (NNs) such as convolutional NNs and can be trained an end-to-end manner. Experimental results indicated that the proposed method prevented overfitting by obtaining sparsity. Furthermore, we demonstrated that the proposed method outperformed a fully connected layer with the softmax function in certain cases when it was used as the last layer of deep NN.
INTRODUCTION
In supervised classification, probabilistic classification is an approach that assigns class label c to input sample æ: by estimating the posterior probability P(c[æ). This approach is primarily categorized into two types of models: discriminative model and generative model. The former optimizes the posterior distribution P(c[æ: directly training set, whereas the latter finds the class conditional distribution P(x|cc and class prior P(c) and subsequently derives the posterior distribution P(c|æ) using Bayes' rule.
The discriminative model and generative model are mutually related (Lasserre et al. 2006 Minka 2005). According to Lasserre al. (006), the only difference between these models is their sta- tistical parameter constraints. Therefore, given a certain generative model, we can derive a corre- sponding discriminative model. For example, the discriminative model corresponding to a unimodal Gaussian distribution is logistic regression (see Appendix A for derivation). Several discriminative models corresponding to the Gaussian mixture model (GMM) have been proposed (Axelrod al.
2006
Bahl et al. 1996 Klautau et al. 2003] Tsai & Chang 2002 Tsuji et al., [999] Tüske et al.,
2015 Wang 2007). They indicate more flexible fitting capability than the generative GMM and
have been applied successfully in fields such speech recognition (Axelrod et al., 2006
Tüske
ct al.
2015 Wang 2007].
The problem to address in mixture models such as the GMM is the determination of the number of components M. Classically, Akaike's information criterion and the Bayesian information criterion have been used; nevertheless, they require considerable computational cost because likelihood must be calculated for every candidate component number. In the generative GMM, methods that optimize M during learning exist (Crouse et al. 2011 Stepánová & Vavrečka (2088). However, in discriminative GMM, a method to optimize M simultaneously during learning has not been clearly formulated.
In this paper, propose novel GMM having two important properties: sparsity and discriminabil- ity, which is named sparse discriminative Gaussian mixture (SDGM). In the SDGM, a GMM-based discriminative model is trained by sparse Bayesian learning. This learning algorithm improves the generalization capability by obtaining a sparse solution and determines the number of components automatically by removing redundant components. Furthermore, the SDGM can be embedded into

---

Under review as a conference paper at ICLR 2020
1.6
1.1
16-
1.4
1.4
14
14
1.2
100
0.
0.8
08
08
0.0
06
06
06
84
02
0-
42
-02
40
-10
05
43
4.0
-15
(a) epoch
(b) 100 opochs
(c) 200 epochs
(d) 250 epochs
Figure 1: Snapshots of the training process of SDGM. The black and green circles are training samples from classes and respectively. The dashed black line the decision boundary between classes and 2 and thus satisfies P(c 1|x) (c = 2/x) 0.5. The dashed blue and red lines are the boundaries between the posterior probabilities of components where p(c,m/=) 0.5.
neural networks (NNs) such as convolutional NNs and trained in an end-to-end manner with an NN. To the authors best knowledge, there no GMM that has both of sparsity and discriminability.
The contributions of this study are as follows:
We propose novel sparse classifier based on a discriminative GMM. The proposed SDGM has both sparsity and discriminability, and determines the number of components automat- ically. The SDGM can be considered as the theoretical extension of the discriminative GMM and the relevance vector machine (RVM) (Tipping) 2001).
This study attempts connect both fields of probabilistic models and NNs. From the equivalence of discriminative model based on Gaussian distribution to fully connected layer, we demonstrate that the SDGM can be used as a module of deep NN. We also show that the SDGM can show superior performance than the fully connected layer with softmax function via an end-to-end learning with an NN on the image recognition task.
SPARSE DISCRIMINATIVE GAUSSIAN MIXTURE (SDGM)
An SDGM takes continuous variable æ E RDD as its input and outputs its posterior probability P(clæ)) for each class e E (I....,C) C}. An SDGM acquires a sparse structure by removing redundant components via sparse Bayesian learning.
Figure | shows how the SDGM trained by removing unnecessary components while keeping discriminability. The two-class training data are from Ripley's synthetic data (Ripley 2006). where a Gaussian mixture model with two components is used for generating data of each class. In this training. we the initial number of components to three for each class. As the training progresses, one of the components for each class becomes small gradually and is removed.
2.1 MODEL. FORMULATION
The posterior probabilities for each class is calculated as follows:
P(c(æ)
Σ P(cmmlx)
(I)
m==
Tem explw
mlc,
Σe EM. Tem exp wm ④
Φ [1, 22. II 22 F2 "]]
(3)
where Me is the number of components for class e and Tem is the mixture weight that is equivalent to the prior of each component P(c, m). It should be noted that we use Wem. R which is the weight vector representing the m-th Gaussian component of class c. The dimension of w woms ie., H. is the same that of Φ; namely, H 1 D(D + 3)/2.
2

---

Under review as a conference paper at ICLR 2020
Algorithm 1:
Weight updating
Input:
Training data set X and teacher vector T.
Output: Trained weight ur obtained by maximizing
11
Initialize the weights w, hyperparameters a, mixture coefficients æ, and posterior probabilities r:
while have not converged do
Calculate using (
while have not converged do
while w have not converged do
Calculate gradients using 122
Calculate Hessian (13);
Maximize 11 w.r.t. wo;
Calculate Plc. m (.)) and P(cla,);
end
Fnom P(c, ((.e)/
end
Calculate using
16
Update using
Update π using
end
Derivation.
Utilizing Gaussian distribution conditional distribution of æ given and m, P(alle, m), the posterior probability of e given æ. P(c(x), is calculated as follows:
Plc.m)
P(c[)) =
(4)
Ee Eme
Plc m) P(rle,
P(ælc, m)
(x
Mmm) " E (x M
(2m) EEm|
where Me E R.D and Eem E DDD the vector and the covariance matrix for component m in class c. Since the calculation inside an exponential function in (5) is quadratic form, the conditional distributions can be transformed follows:
P(ælc, m) explwxm
(6)
where
D
1 In Σ Daml
1
D
Wo
In 2T
Semis SemiyMemsMemp
Semil Cemil
2
2 i=
1=
D
1
DemiDMemi:
- Semil:
-em12 -8m
Sem D.
Sem22.
2
Sem DD
(7)
Here, Semij is the j-th element of E-1
2.2 LEARNING ALGORITHM
Algorithm shows the training of the SDGM. In this algorithm, the optimal weight is obtained as maximum a posteriori solution. We can obtain sparse solution by optimizing the prior distribution set to each weight simultancously with weight optimization.
set training data and target value (In, ( (n = 1,.... .N) is given. The target the is coded in one-of-K form, where tnc if the n-th sample belongs to class c, tne 0 otherwise. A binary random variable 2nem is introduced. The variable 2nem = when the n-th sample from class belongs the m-th component. Otherwise, &nem O. This variable is required for the optimization of the mixture weight 7.m-- We also define π and as vectors that comprise 7em and Fuem their elements, respectively. As the prior distribution of the weight Wambs employ

---

Under review as a conference paper at ICLR 2020
Gaussian distribution with mean of zero. Using different precision parameter (inverse of the variance) Ctem for each weight Wemhs the joint probability of all the weights is represented as follows:
C M II
(c,m) Oh
P(w
exp
[ 2 Wemk 2
aemh
(2T)
c= m= h=1
where and a are vectors with Wemh and aemh as their elements, respectively. During learning. we update not only w but also a. If Qemh + oc, the prior (8) is O; hence a sparse solution is obtained by optimizing a.
Using these variables, the expectation of the log- le-likkeiihodd function over z, J, is defined as follows:
N ΣΣ
== P
Tnom tne In P(c, m (..)
where is a matrix with tne as its element. The training data matrix X contains क; in the n-th row. The variable Tnom in the right-hand side corresponds P(mlc, w..) and can be calculated as Tnem = P(e, m) ml (c)
The posterior probability of the weight vector is described as follows:
P(w][T,,.X,TTa') P(T, x|X w, n,, ax) P(w)a)
(10)
P|T,,|X..c
An optimal obtained the point where (T) is maximized. The denominator of the right-hand side in (IO) is called the evidence term, and we maximize it with respect to a. However, this maxi- mization problem cannot be solved analytically; therefore we introduce the Laplace approximation described as the following procedure.
With a fixed, we obtain the mode of the posterior distribution of w.. The solution is given by the point where the following equation is maximized:
Eg In P( T, 2, X, n,, o:] = Eg ITP 2 w, a.) + In Pn + const.
w "'AAw + const.,
(11)
where A diag Oeehh- We obtain the mode of [I via Newton's method. The gradient and Hessian required for this estimation can be calculated follows:
VE, [In P(w T, X. n. ae] = VJ - Aw,
(12)
VVE, [In P( x, x, a)] VVJ A. Each element of VJ and V.J / calculated follows:
(13)
ÖJ
(Tnomtne - P(c, m/zm)))
(14)
Dwanh
JJJ
P(c', mn)) ml Sec )ondw'
(15)
@w Owe'''n' where See' mmm is variable that takes if both = e' and m = m', 0 otherwise. Hence, the posterior distribution of w can be approximated by a Gaussian distribution with mean of io and a covariance matrix of A, where
A (VVEg [In P(t|T, 2, X, π. a)))
(16)
Because the evidence term can be represented using the normalization term of this Gaussian distri- bution, we obtain the following updating rule by calculating its derivative with respect to aamh-
1 Oeamh
aemh
(17)
where Aemh is the diagonal component of A. The mixture weight mem. can be estimated using Tnom as follows:
No
cm
Tnems
(18)
==1
where N₂ is the number of training samples belonging to class As described above, we obtain
a sparse solution by alternately repeating the update of hyper-parameters, as described in (17 and
(8) and the posterior distribution estimation of w using the Laplace approximation. During the
procedure, the {c, m -th component is eliminated if Tem becomes or all the weights Wenk corre- sponding to the component become O.
4

---

Under review as a conference paper at ICLR 2020
-10
-10
T0
components
components
(c)
10 components
(d) 16 components
Figure 2: Changes in learned class boundaries according to number of initial components.
Triining Tou
3. 1: 10
Foo 200 weighs
[N
09
3
infiniial compainants
Emor
15
infini compentes Namber of components
of intaa compoeene
10
indintaa componana
16
(c) Namber of mommerr wrights
(6) Weight reduction
Figure& 3: Evaluation results using synthetic data. (a) recognition error rate, (b) number of compo-
nents after training, (c) number of nonzero weights after training, and (d) weight reduction ratio.
3 EXPERIMENTS
3.1 EVALUATION OF CIARRACTERISTICS USING SYNTIEEIIC DATA
To evaluate the characteristics of the SDGM, we conducted classification experiments using syn- thetic data. The dataset comprises two classes. The data were sampled from a Gaussian mixture model with eight components for each class. The numbers of training data and test data were 320 and 1,600, respectively. The scatter plot of this dataset shown Figure
In the evaluation, we calculated the error rates for the training data and the test data, the number of components after training, the number of nonzero weights after training, and the weight reduction ratio (the ratio of the number of the nonzero weights the number of initial weights), by varying the number of initial components as 2, ,,8....,20 20.
Figure 2 displays the changes in the learned class boundaries according the number of initial components. When the number components is small, such as that shown in Figure ((a), the decision boundary is simple; therefore, the classification performance is insufficient. However, according to the increase in the number of components, the decision boundary fits the actual class boundaries. It is noteworthy that the SDGM learns the GMM discrininative model instead of generative model; an appropriate decision boundary was obtained even if the number of components for the model less than the actual number (c.g.. c)))
Figure 3 shows the evaluation results of the characteristics. Figures 3(a,, (b), (c), and (d) show the recognition eгrог rate, number of components after training, number of nonzero weights after training, and weight reduction ratio, respectively. The horizontal axis shows the number of initial components in all the graphs.
In Figure B a). the recognition error rates for the training data and test data are almost the same with the few number of components, and decrease according to the increase in the number of initial components while it is 2 to 6. This implied that the representation capability was insufficient when the number of components was small, and that the network could not accurately separate the classes. Meanwhile, changes in the training and test error rates were both flat when the number of initial components exceeded eight, even though the test error rates were slightly higher than the training error rate. In general, the training error decreases and the test error increases when the complexity of the classifier is increased. However, the SDGM suppresses the increase in complexity using sparse Bayesian learning, thereby preventing overfitting.
5

---

Under review as a conference paper at ICLR 2020
Table 1: Recognition error (%) and number of nonzero weights
Error rate (%)
Number nonzero weights
SDGM
Baselines
SDGM
Baselines
Dutaset
w/ sparse
w/o spurse
LR
SVM
RVM
w/ sparse
w/o spurse
LR
SVM
RVM
Ripley
9.1
9.9
11.4
10.6
9.3
6
1255
2
38
4
Banuna
10.6
10.8
47.0
10.9
10.8
11.1
2005
2
135.2
11.4
Waveform
10.1
13.5
10.3
11.0
2005
20.73
146.4
Titanic
22.7
23.3
22.7
22.1
23.0
74.5
755
2,98
93.7
65.3
Breast Cancer
29.4
35.1
27.5
26.9
29.9
15.73
1005
8.88
116.7
6.3
Normalized mean
1.00
1.05
1.79
1.02
1.03
1.00
129.35
0.60
8.11
0.86
In Figure 3 b), the number of components after training corresponds to the number of initial com- ponents until the number of initial components eight. When the number of initial components exceeds ten, the number of components after training tends to be reduced. In particular, eight com- ponents are reduced when the number of initial components is 20. The results above indicate the SDGM can reduce unnecessary components.
From the results in Figure c),, we confirm that the number of nonzero weights after training in- creases according the increase in the number of initial components. This implies that the com- plexity of the trained model depends on the number of initial components, and that the minimum number of components is not always obtained.
Meanwhile, in Figure d), the weight reduction ratio increases according to the increase in the number of initial components. This result suggests that the larger the number of initial weights, the more weights were reduced. Moreover, the weight reduction ratio is greater than 99 % in any case. The results above indicate that the SDGM can prevent overfitting by obtaining high sparsity and can reduce unnecessary components.
3.2
COMPARATIVE STUDY USING BENCHMARK DATA
To evaluate the capability of the SDGM quantitatively, we conducted a classification experiment using benchmark datasets. The datasets used in this experiment
Ripley's synthetic data (Ripley) 2006) (Ripley hereinafter) and four datasets cited from Rätsch et al. (200I); Banana, Waveform, Titanic, and Breast Cancer. Ripley is synthetic dataset that generated from a two-dimensional
(D = 2) Gaussian mixture model, and 250 and 1,000 samples are provided for training and test, respectively. The number of classes is two (C 2). and each class comprises two components. The remaining four datasets are all two-class (C = 2) datasets, which comprise different data size and dimensionality. Since they contain 100 training/test splits, we repeated experiments for 100 times and then calculated average statistics.
For comparison, we used three classifiers that can obtain sparse solution: linear logistic re- gression (LR) with 4 constraint, support vector machine (SVM) (Cortes & Vapnik 1995) and relevance vector machine (RVM) (Tipping 200Tp. In the evaluation, we compared the recognition error rates for discriminability and number of nonzero weights for sparsity on the test data. The results of SVM and RVM were cited from Tipping (2001). For ablation study, we also tested our SDGM without sparse learning by omitting the update of By way of summary, the statistics were normalized by those of the SDGM and the overall mean was shown.
Table shows the recognition results in Table show that the SDGM achieved an equivalent or greater accuracy compared with the SVM and RVM average. The SDGM is developed based Gaussian mixture model and is particularly effective for data where a Gaussian distribution can be assumed, such as the Ripley dataset. On the number of nonzero weights, understandably, the LR showed the smallest number since it is a linear model. Among the remaining nonlinear classifiers, the SDGM achieved relatively small number of nonzero weights thanks to its sparse Bayesian learning. The results above indicated that the SDGM demonstrated generalization capability and a sparsity simultaneously.
rates and number of nonzero weights for each method. The
3.3 IMAGE CLASSIFICATION
In this experiment, the SDGM is embedded into a deep neural network. Since the SDGM is differ- entiable with respect to the weights, SDGM can be embedded into deep NN module and is

---

Under review as a conference paper at ICLR 2020
Table 2: Recognition error rates (%) on image classification
MNIST (D = 2)
MNIST (D = 10)
Fashion-MNIST
CIFAR-10
Softmax
3.19
1.01
8.78
11.07
SDGM
2.43
0.72
8.30
10.05
trained an end-to-end manner. In particular, the SDGM plays the same role as the softmax func- tion since the SDGM calculates the posterior probability of each class given input vector. We can show that fully connected layer with the softmax is equivalent to the discriminative model based on single Gaussian distribution for each class by applying a simple transformation (see Appendix A), whereas the SDGM is based on the Gaussian mixture model.
To verify the difference between them, we conducted image classification experiments. Using CNN with a softmax function as a baseline, we evaluated the capability of SDGM by replacing softmax with the SDGM.
3.3.1 DATASETS AND EXPERIMENTAL SETUPS
We used the following datasets and experimental settings in this experiment.
This dataset includes 10 classes of handwritten binary digit images of size 28 x 28 (LeCun ct al. 1998). We used 60,000 images training data and 10,000 images testing data. As a feature extractor, we used a simple CNN that consists of five convolutional layers with four max pooling layers between them and fully connected layer. To visualize the learned CNN features, we first set the output dimension of the fully connected layer of the baseline CNN as two (D = 2). Furthermore, we tested by increasing the output dimension of the fully connected layer from two to ten (D = 10).
MNIST:
Tashion-MNIST: MASHiT (Xiao et al. 2017) includes 10 classes of binary fashion images with a size of 28 x 28. includes 60,000 images for training data and 10,000 images for testing data. We used the same CNN in MNIST with 10 the output dimension.
CIFAR-10: CIFAR- 10 [Krizhevsky & Hinton 2009) is the labeled subsets of an 80 million tiny image dataset. This dataset consists of 60,000 32x32 color images in 10 classes, with 6,000 images per class. There are 50,000 training images and 10,000 test images. For CIFAR-10, we trained DenseNet (Huang al. 2017) with depth of 40 and a growth rate of 12.
For each dataset, the network was trained with a batch size of 64 for 100 epochs with a learning rate of 0.01 We used a weight decay of 1.0 x 10-5 and the Nesterov optimization algorithm (Sutskever et al. 2013) with a momentum of 0.9. The network weights were initialized using the Glorot uniform (Glorot & Bengio 2010).
3.3.2 RESULTS
Figure shows the two-dimensional feature embeddings on the MNIST dataset. Different feature embeddings were acquired for each method. When softmax was used, the features spread in fan shape and some part of the distribution overlapped around the origin. However, when the SDGM was used, the distribution for each class exhibited ellipse shape and margins appeared between the class distributions. This is because the SDGM is based on a Gaussian mixture model and functions to push the samples into Gaussian shape.
Table 2 shows the recognition error rates on each dataset. SDGM achieved better performance than softmax. As shown in Figure 4 SDGM can create margins between classes by pushing the features into a Gaussian shape. This phenomenon positively affected the classification capability.
4 RELATED WORK AND POSITION OF THIS STUDY
Figure illustrates the relationship of our study with other studies. This study is primarily consists of three factors: discriminative model, Gaussian mixture model, and Sparse Bayesian learning. This study is the first that combines these three factors and expands the body of knowledge in these fields.
From the perspective of the sparse Bayesian classifier, the RVM [Tipping] 2001) is the most impor- tant related study. An RVM is combines logistic regression and sparse Bayesian learning. Since
7

---

Under review as a conference paper at ICLR 2020
Training data
Test data
Training data
25
Test data
200
25
20
150
20
100
15
10
50
0-
50
150
10
20
20
(8) Softmax
(b) Ouns
Figure 4: Visualization of CNN features MNIST after end-to-end learning. In this visualization, five convolutional layers with four max pooling layers between them and a fully connected layer with two-dimensional output are used. (a) results when fully connected layer with the softmax function is used as the last layer. (b) when SDGM is used as the last layer instead. The colors red, blue, yellow, pink, green, tomato, saddlebrown, lightgreen, cyan, and black represent classes from to 9, respectively. Note that the ranges of the axis are different between (a) and (b).
Relevance vector machine (RVM)
Ours
Diszri minative
Bayes
Lagistic regression FC layer softmax
Discriminative (Mauzau, 2003)
Sparse (Gaiffes, 2014)
Sparse Gaussian
Mixture model
Unimodal Gaussian
Goussian midtrre model
Figure 5: Relationship of our study with other studies.
the logistic regression is equivalent to the discriminative model of unimodal Gaussian model, the SDGM can be considered as an extended RVM using GMM. furthermore, from the perspective of the probabilistic model, the SDGM is considered as the an extended discriminative GMM (Klautau al.] 2003) using sparse Bayesian learning, and an extended sparse GMM (Gaiffas & Michel 2014) using the discriminative model.
Sparse methods have often been used in machine learning. Three primary merits of using sparse learning are as follows: improvements in generalization capability, memory reduction, and inter- pretability. Several attempts have been conducted to adapt sparse learning to deep NNs. Grabam (2014) proposed a spatially-spare convolutional neural network. Liu et al, (2015) proposed a sparse convolution neural network. Additionally. sparse Bayesian learning has been applied in many fields. For example, an application to EEG classification has been reported (Zhang et al. 2017).
5 CONCLUSION
In this paper, we proposed a sparse classifier based on a GMM, which is named SDGM. In the SDGM, a GMM-based discriminative model was trained by sparse Bayesian learning. This learning algorithm improved the generalization capability by obtaining a sparse solution and automatically determined the number of components by removing redundant components. The SDGM could be embedded into NNs such as convolutional NNs and could be trained an end-to-end manner.
In the experiments, we demonstrated that the SDGM could reduce the amount of weights via sparse Bayesian learning, thereby improving its generalization capability. The comparison using benchmark datasets suggested that SDGM outperforms the conventional sparse classifiers. We also demonstrated that SDGM outperformed the fully connected layer with the softmax function when was used as the last layer of deep NN.
One of the limitations this study is that sparse Bayesian learning was applied only when the SDGM was trained stand-alone. In future work. we will develop sparse learning algorithm for whole deep NN structure including the feature extraction part. This will improve the ability of the CNN for larger data classification.
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
Scott Axelrod, Vaibhava Goel, Ramesh Gopinath, Peder Olsen, and Karthik Visweswariah. Dis- criminative estimation of subspace constrained Gaussian mixture models for speech recognition. IEEE Transactions on Audio, Speech, and Language Processing, 15(1):172-18, 2006.
Lalit R Bahl, Mukund Padmanabhan, David Nahamoo, and PS Gopalakrishnan. Discriminative training of Gaussian mixture models for large vocabulary speech recognition systems. In Pro- ceedings of the IEEE International Conference on Acoustics. Speech, and Signal Processing Con- ference Proceedings (ICASSP), volume 2, pp. 613-616, 1996.
Corinna Cortes and Vladimir Vapnik. Support-vector networks. Machine learning, 20(3):273-277, 1995.
David F Crouse, Peter Willett, Krishna Pattipati, and Lennart Svensson. A look at Gaussian mixture reduction algorithms. In Proceedings of the 14th International Conference on Information Fusion, pp. 8, 2011.
Stephane Gaiffas and Bertrand Michel. Sparse bayesian unsupervised learning. arXiv preprint arXiv:140..8017, 2014.
Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the 13th International Conference on Artificial Intelligence and Statistics (AISTATS). pp. 249-256, 2010.
Benjamin Graham. opatially-sparse convolutional neural networks. arXiv preprint 1rXiv:140966070. 2014.
Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), volume 1, pp- 3, 2017.
Aldebaro Klautau, Nikola Jevtic, and Alon Orlitsky. Discriminative Gaussian mixture models: A comparison with kernel classifiers. In Proceedings of the 20th International Conference on Ma- chine Learning (ICML), pp. 553-360, 2003.
Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Tech- nical report, University of Toronto, 2009.
Julia A. Lasserre, Christopher M. Bishop, and Thomas P. Minka. Principled hybrids of genera- tive and discriminative models. In Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR), pp. 87-94, 2006.
Yann LeCun, Léon Bottou, Yoshua Bengio, Patrick Haffner, al. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(1 1):2278 1):2278-2324, 1998.
Baoyuan Liu, Min Wang, Hassan Foroosh, Marshall Tappen, and Marianna Pensky. Sparse convolu- tional neural networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 806-814, 2015.
Tom Minka. Discriminative models, not discriminative training. Technical report, Technical Report MSR-TR-2005- 144, Microsoft Research. 2005.
Gunnar Rätsch, Takashi Onoda, and K-R Müller. Soft margins for adaboost. Machine learning, 42 (3):287-320) 2001.
Brian D Ripley. Pattern Recognition and Neural Networks. Cambridge University Press, 2006.
Karla Štepánová and Michal Vavrečka. Estimating number of components in Gaussian mixture model using combination of greedy and merging algorithm. Pattern Analysis and Applications,
21(1):181-122, 2018.
Ilya Sutskever, James Martens, George Dahl, and Geoffrey Hinton. On the importance of initial- ization and momentum in deep learning. In Proceedings of International Conference on Mchine Learning (ICMI), volume 28, pp. 1139-1147, 2013.

---

Under review as a conference paper at ICLR 2020
Michael E Tipping. Sparse Bayesian learning and the relevance vector machine. Journal of Machine Learning research, 1(Jun):211-22 244, 2001.
Wuei-He Tsai and Wen-Whei Chang. Discriminative training of Gaussian mixture bigram mod- els with application Chinese dialect identification. Speech Communication, 36(3-4):31-326, 2002.
Toshio Tsuji, Osamu Fukuda, Hiroyuki Ichinobe, and Makoto Kaneko. A log- ligelinerrized Gaussian mixture network and its application to EEG pattern classification. IEEE Transactions on Systems, Man, and Cybernetics, Part C: Applications and Reviews, 29(1):60-7,, 1999.
Zoltán Tüske, Muhammad Ali Tahir, Ralf Schlüter, and Hermann Ney. Integrating Gaussian mix- tures into deep neural networks: Softmax layer with hidden variables. In Proceedings of IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), pp. 4285 4289, 2015.
Jue Wang. Discriminative Gaussian mixtures for interactive image segmentation. In Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), volume 1, pp. 1-601, 2007.
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-MNIST novel image dataset for bench- marking machine learning algorithms. arXiv preprint 170..07777,7777 2017.
Yu Zhang, Yu Wang, Jing Jin, and Xingyu Wang. Sparse Bayesian learning for obtaining sparsity of EEG frequency bands based feature vectors in motor imagery classification. International Journal of Neural Systems, 1652033 2017.
A APPENDIX
We explain that a fully connected layer with the softmax function, or logistic regression, can be regarded as a discriminative model based on a Gaussian distribution by utilizing transformation of the equations. Let us consider a case in which the class-conditional probability P(x]cc a Gaussian distribution. In this case, we can omit m from the equations
all classes share the same covariance matrix and the mixture weight Tem. the terms Tam (2)
of T1 ... TD x22 22 P2 ... anb in (3), and $511111 $ in (7) can be
canceled; hence the calculation of the posterior probability P(c|x) is also simplified as
P(cxx) = exp(wr. TΦ)
Ee exp(ur, TT)
where
1 D D
We = flog P(c)
Σ Σ
SeijHeilMe + 2
log 2 + log EEcl
1
2 i=1 j=1
Σ SeiPai
Σ
SaDMei) T
Φ
[
This is equivalent to fully connected layer with the softmax function.
10


Reviews:
[{"review_id": "SygQvluBFS", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a classifier, called SDGM, based on discriminative Gaussian mixture and its sparse parameter estimation. \nThe method aims to have a flexible decision boundary due to the mixture model as well as to generalize well owing to the sparse learning strategy. \nWhile the method incorporates distinct ideas in the literature such as increased model complexity and use of a sparse prior, the paper needs more clearer explanation of the method design and careful empirical investigation. \n\nMy major concerns are as follows. \n1) Explanation of the learning algorithm of Section 2.2 is unsatisfactory. \nSpecifically, the objective function and the principle of algorithm are unclear. \nMy guess is:\n* c, m and T, z are equivalent. (T, z are one-hot representation of c, m.) \n* The objective is to maximize the conditional probability of equation (1) given labeled data (x, t). \n* This maximization is carried out in a similar way to the EM algorithm where m (or z) is regarded as a latent variable. This gives equation (9). \n* A MAP estimate of parameter w is calculated using the prior (8). \nAssuming the points above, I am still unsure about the following points: \n1-a) How is the sparsity induced? \nThe prior of w seems to have the l2 regularizer, but how are \\pi and r pulled toward zero?\n1-b) How is the update of \\alpha (17) derived?\nDoes this strengthen the sparsity by increasing \\alpha given small \\hat{w}?\nWhat is the \"orthogonal component\" of \\Lambda?\n1-c) Is structure in w ignored?\nEquation (7) indicates that parameter w has a certain structure such as nonnegative s_{cmii} or the determinant |\\Sigma_{cm}| interacting with s_{cmij}. \nIn other words, the degree of freedom in w assuming the Gaussian likelihood of x is smaller than H. \nThese structure would be violated if the gradient descent or Newton's method is applied. \nDo you mean by *discriminative* that we can freely set the parameter w?\nThen, this point should be emphasized. \n\n2) Which parameter is learned when combined into NN?\nParameters w and \\pi? \nHow are these parameters made sparse in the end-to-end learning?\n\n3) Ablation test to investigate which aspect impacts the performance. \nSection 4 describes SDGM incorporates disciminative model, mixture model, and sparse Bayesian parameter estimation. \nIt is more informative to provide empirical results to see the impact of each property by comparing SDGM with, for example, RVM, discriminative GMM and sparse GMM. \n\nSome other commets follow.\n* \\sqrt{\\alpha_{cmh}} may be missing from the numerator of (8). \n* For what distribution the expectation with regard to z is taken in e.g. (9, 11, 12, 13, 16)?\n* What is D for CIFAR-10 experiment? DenseNet seems to use D=1000 units for the fully connected layer. Did the authors adopt this value?\n\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "HyxLpy9RKS", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes a discriminative Gaussian mixture model with a sparsity prior over the decoding weight. They can automatically learn the number of components with the sparsity prior and learn Gaussian-structured feature space. \n\n1. I think the model is just ARD prior over discriminative GMM which is not that novel. DGMM models have been for a while [1,2]. Adding ARD sparsity prior over the decoding weight is also a classic routine. It's also well known that ARD can do feature selection and removal. \n\n[1] Discriminative gaussian mixture models for speaker verification\n[2] Discriminative Gaussian mixture models: A comparison with kernel classifiers\n\n2. I don't think differentiating between discriminative GMM and generative GMM would make such a big deal. DGMM is basically Gaussian mixtures existing for each class. Any skill applied to GMM can be applied to DGMM. There are many works for component number selection for GMM with non-parametric Bayesian methods. For example, Dirichlet Process Mixture Model can automatically learn the number of components without predefining. \n\n3. Only comparing SDGM with LR, SVM and RVM is quite weak, not mentioning that the performance is not that dominatingly better. SDGM is GMM+LR. So SDGM should be better than LR if the data has structures. What SVM you compare with? Do you use nonlinear kernels which can learn better nonlinear feature space?\n\nOverall, I think the contribution of the paper is a bit incremental. I vote for a rejection. \n\n", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "HJepvMo__r", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper presents an alternative to densely connected shallow classifiers or the conventional penultimate layers (softmax) of conventional deep network classifiers. This is formulated as a Gaussian mixture model trained via gradient descent arguments. \n\nThe paper is interesting and constitutes a useful contribution. The Gaussian mixture model formulation allows for inducing sparsity, thus (potentially) considerably reducing the trainable model (layer) parameters. The model is computationally scalable, a property which renders it amenable to real-world applications.\n\nHowever, it is unfortunate that the paper does not take into account recent related advances in the field, e.g.\n\nhttps://icml.cc/Conferences/2019/ScheduleMultitrack?event=4566\n\nThe paper should make this sort of related work review, discuss the differences from it, and perform extensive experimental comparisons. ", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents a method for merging a discriminative GMM with an ARD sparsity-promoting prior.  This is accomplished by nesting the ARD prior update within a larger EM-based routine for handling the GMM, allowing the model to automatically remove redundant components and improve generalization.  The resulting algorithm was deployed on standard benchmark data sets and compared against existing baselines such as logistic regression, RVMs, and SVMs.

Overall, one potential weakness of this paper, which is admittedly somewhat subjective, is that the exhibited novelty of the proposed approach is modest.  Indeed ARD approaches are now widely used in various capacities, and even if some hurdles must be overcome to implement the specific marriage with a discriminative GMM as reported here, at least one reviewer did not feel that this was sufficient to warrant publication.  Other concerns related to the experiments and comparison with existing work.  For example, one reviewer mentioned comparisons with Panousis et al., "Nonparametric Bayesian Deep Networks with Local Competition," ICML 2019 and requested a discussion of differences.  However, the rebuttal merely deferred this consideration to future work and provided no feedback regarding similarities or differences.  In the end, all reviewers recommended rejecting this paper and I did not find any sufficient reason to overrule this consensus.
Decision: Reject