A Discriminative Gaussian Mixture Model with Sparsity

1 INTRODUCTION . In probabilistic classification , a discriminative model is an approach that assigns a class label c to an input sample x by estimating the posterior probability P ( c | x ) . The posterior probability P ( c | x ) should correctly be modeled because it is not only related to classification accuracy , but also to the confidence of decision making in real-world applications such as medical diagnosis support . In general , the model calculates the class posterior probability using the softmax function after nonlinear feature extraction . Classically , a combination of the kernel method and the softmax function has been used . The recent mainstream method is to use a deep neural network for representation learning and softmax for the calculation of the posterior probability . Such a general procedure for developing a discriminative model potentially contains a limitation due to unimodality . The softmax-based model , such as a fully connected ( FC ) layer with a softmax function that is often used in deep neural networks ( NNs ) , assumes a unimodal Gaussian distribution for each class ( details are shown in Appendix A ) . Therefore , even if the feature space is transformed into discriminative space via the feature extraction part , P ( c | x ) can not correctly be modeled if the multimodality remains , which leads to a decrease in accuracy . Mixture models can address this issue . Mixture models are widely used for generative models , with a Gaussian mixture model ( GMM ) as a typical example . Mixture models are also effective in discriminative models ; for example , discriminative GMMs have been applied successfully in various fields , e.g. , speech recognition ( Tüske et al . 2015 ; Wang 2007 ) . However , the number of parameters increases if the number of mixture components increases , which may lead to over-fitting and an increase in memory usage ; this is useful if we can reduce the number of redundant parameters while maintaining multimodality . In this paper , we propose a discriminative model with two important properties ; multimodality and sparsity . The proposed model is referred to as the sparse discriminative Gaussian mixture ( SDGM ) . In the SDGM , a GMM-based discriminative model is formulated and trained via sparse Bayesian learning . This learning algorithm reduces memory usage without losing generalization capability by obtaining sparse weights while maintaining the multimodality of the mixture model . The technical highlight of this study is twofold : One is that the SDGM finds the multimodal structure in the feature space and the other is that redundant Gaussian components are removed owing to sparse learning . Figure 1 shows a comparison of the decision boundaries with other discriminative models . The two-class data are from Ripley ’ s synthetic data ( Ripley 2006 ) , where two Gaussian components are used to generate data for each class . The FC layer with the softmax function , which is often used in the last layer of deep NNs , assumes a unimodal Gaussian for each class , resulting in an inappropriate decision boundary . Kernel Bayesian methods , such as the Gaussian process ( GP ) classifier ( Wenzel et al . 2019 ) and relevance vector machine ( RVM ) ( Tipping 2001 ) , estimate nonlinear decision boundaries using nonlinear kernels , whereas these methods can not find multimodal structures . Although the discriminative GMM finds multimodal structure , this model retains redundant Gaussian components . However , the proposed SDGM finds a multimodal structure of data while removing redundant components , which leads to an accurate decision boundary . Furthermore , the SDGM can be embedded into NNs , such as convolutional NNs ( CNNs ) , and trained in an end-to-end manner with an NN . The proposed SDGM is also considered as a mixture , nonlinear , and sparse expansion of the logistic regression , and thus the SDGM can be used as the last layer of an NN for classification by replacing it with the fully connected ( FC ) layer with a softmax activation function . The contributions of this study are as follows : • We propose a novel sparse classifier based on a discriminative GMM . The proposed SDGM has both multimodality and sparsity , thereby flexibly estimating the posterior distribution of classes while removing redundant parameters . Moreover , the SDGM automatically determines the number of components by simultaneously removing the redundant components during learning . • From the perspective of the Bayesian kernel methods , the SDGM is considered as the expansion of the GP and RVM . The SDGM can estimate the posterior probabilities more flexibly than the GP and RVM owing to multimodality . The experimental comparison using benchmark data demonstrated superior performance to the existing Bayesian kernel methods . • This study connects both fields of probabilistic models and NNs . From the equivalence of a discriminative model based on a Gaussian distribution to an FC layer , we demonstrate that the SDGM can be used as a module of a deep NN . We also demonstrate that the SDGM exhibits superior performance to the FC layer with a softmax function via end-toend learning with an NN on the image recognition task . 2 RELATED WORK AND POSITION OF THIS STUDY . The position of the proposed SDGM among the related methods is summarized in Figure 2 . Interestingly , by summarizing the relationships , we can confirm that the three separately developed fields , generative models , discriminative models , and kernel Bayesian methods , are related to each other . Starting from the Gaussian distribution , all the models shown in Figure 2 are connected via four types of arrows . There is an undeveloped area in the upper right part , and the development of the area is the contribution of this study . A ( unimodal ) Gaussian distribution is used as the most naive generative model in machine learning and is the foundation of this relationship diagram . A GMM is the mixture expansion of the Gaussian distributions . Since the GMM can express ( almost ) arbitrary continuous distributions using multiple Gaussian components , it has been utilized for a long time . Since Gaussian fitting requires numerous parameters , the sparsified versions of Gaussian ( Hsieh et al . 2011 ) and GMM ( Gaiffas & Michel 2014 ) have been proposed . The discriminative models and the generative models are mutually related ( Lasserre et al . 2006 ; Minka 2005 ) . According to Lasserre et al . ( 2006 ) , the only difference between these models is their statistical parameter constraints . Therefore , given a generative model , we can derive a corresponding discriminative model . For example , discriminative models corresponding to the Gaussian mixture model have been proposed ( Axelrod et al . 2006 ; Bahl et al . 1996 ; Klautau et al . 2003 ; Tsai & Chang 2002 ; Tsuji et al . 1999 ; Tüske et al . 2015 ; Wang 2007 ) . They indicate more flexible fitting capability for classification problems than the generative GMM because the discriminative models have a lower statistical bias than the generative models . Furthermore , as shown by Tüske et al . ( 2015 ) ; Variani et al . ( 2015 ) , these models can be used as the last layer of the NN because these models output the class posterior probability . From the perspective of the kernel Bayesian methods , the GP classifier ( Wenzel et al . 2019 ) and the mixture of GPs ( MGP ) ( Luo & Sun 2017 ) are the Bayesian kernelized version of the logistic regression and the discriminative GMM , respectively . The SDGM with kernelization is also regarded as a kernel Bayesian method because the posterior distribution of weights is estimated during learning instead of directly estimating the weights as points , as with the GP and MGP . The RVM ( Tipping 2001 ) is the sparse version of the GP classifier and is the most important related study . The learning algorithm of the SDGM is based on that of the RVM ; however , it is extended for the mixture model . If we use kernelization , the SDGM becomes one of the kernel Bayesian methods and is considered as the mixture expansion of the RVM or sparse expansion of the MGP . Therefore , the classification capability and sparsity are compared with kernel Bayesian methods in Section 4.1 . Otherwise , the SDGM is considered as one of the discriminative models and can be embedded in an NN . The comparison with other discriminative models is conducted in Section 4.2 via image classification by combining a CNN . 3 SPARSE DISCRIMINATIVE GAUSSIAN MIXTURE ( SDGM ) . The SDGM takes a continuous variable as its input and outputs the posterior probability of each class , acquiring a sparse structure by removing redundant components via sparse Bayesian learning . Figure 3 shows how the SDGM is trained by removing unnecessary components while maintaining discriminability . In this training , we set the initial number of components to three for each class . As the training progresses , one of the components for each class gradually becomes small and is removed . 3.1 NOTATION . Let x ∈ RD be a continuous input variable and tc ( c ∈ { 1 , . . . , C } , C is the number of classes ) be a discrete target variable that is coded in a one-of-C form , where tc = 1 if x belongs to class c , tc = 0 otherwise . Also , let zcm be a discrete latent variable , and zcm = 1 when x from class c belongs to the m-th component ( m ∈ { 1 , . . . , Mc } , Mc is the number of components for class c ) , zcm = 0 otherwise . For simplicity , in this paper , the probabilities for classes and components are described using only c and m ; e.g. , we use P ( c , m | x ) instead of P ( tc = 1 , zcm = 1 | x ) . 3.2 MODEL FORMULATION . The posterior probabilities of each class c given x are calculated as follows : P ( c | x ) = Mc∑ m=1 P ( c , m | x ) , P ( c , m | x ) = πcm exp [ w T cmφ ] ∑C c′=1 ∑Mc′ m′=1 πc′m′ exp [ w T c′m′φ ] , ( 1 ) φ = [ 1 , xT , x21 , x1x2 , . . . , x1xD , x 2 2 , x2x3 , . . . , x 2 D ] T , ( 2 ) where πcm is the mixture weight that is equivalent to the prior of each component P ( c , m ) . It should be noted that we usewcm ∈ RH , which is the weight vector representing the m-th Gaussian component of class c. The dimension of wcm , i.e. , H , is the same as that of φ ; namely , H = 1 +D ( D + 3 ) /2 . Derivation . Utilizing a Gaussian distribution as a conditional distribution of x given c and m , P ( x | c , m ) , the posterior probability of c given x , P ( c | x ) , is calculated as follows : P ( c | x ) = Mc∑ m=1 P ( c , m ) P ( x | c , m ) ∑C c=1 ∑Mc m=1P ( c , m ) P ( x | c , m ) , ( 3 ) P ( x | c , m ) = 1 ( 2π ) D 2 |Σcm| 1 2 exp [ −1 2 ( x− µcm ) TΣ−1cm ( x− µcm ) ] , ( 4 ) where µcm ∈ RD and Σcm ∈ RD×D are the mean vector and the covariance matrix for component m in class c. Since the calculation inside an exponential function in ( 4 ) is quadratic form , the conditional distributions can be transformed as follows : P ( x | c , m ) = exp [ wTcmφ ] , ( 5 ) where wcm = [ − D 2 ln 2π − 1 2 ln |Σcm| − 1 2 D∑ i=1 D∑ j=1 scmijµcmiµcmj , D∑ i=1 scmi1µcmi , . . . , D∑ i=1 scmiDµcmi , − 1 2 scm11 , −scm12 , . . . , −scm1D , − 1 2 scm22 , . . . , − 1 2 scmDD ] T . ( 6 ) Here , scmij is the ( i , j ) -th element of Σ−1cm .

Summary: The paper proposes a sparse classifier via  discriminative GMM. This model is trained based on sparse Bayesian learning. The sparsity constraint removes redundant Gaussian components which results in  reducing the number of parameters and improving the generalization. This framework can potentially be embedded into the deep models and trained in an end-to-end fashion. The main motivation is that the proposed model (i.e., SDGM,)  can consider multimodal data while conventional softmax classifiers only assume unimodality for each class. Experimental results show the superiority of the SDGM over existing softmax-based discriminative models.