A Variational Autoencoder for Probabilistic Non-Negative Matrix Factorisation

1 INTRODUCTION . 1.1 NON-NEGATIVE MATRIX FACTORIZATION . There has been a considerable increase in interest in NMF since the publication of a seminal work by Lee & Seung ( 1999 ) ( although earlier Paatero & Tapper ( 1994 ) had studied this field ) in part because NMF tends to produce a sparse and parts based representation of the data . This sparse and parts based representation is in contrast to other dimensionality reduction techniques such as principal component analysis which tends to produce a holistic representation . The parts should represent features of the data , therefore NMF can produce a representation of the data by the addition of extracted features . This representation may be considerably more interpretable than more holistic approaches . Consider a data matrix V ∈ Rm×n with m dimensions and n data points which has only nonnegative elements . If we define two matrices , also with only non-negative elements : W ∈ Rm×r and H ∈ Rr×n , then non-negative matrix factorisation ( NMF ) can reduce the dimensionality of V through the approximation V ≈WH where , generally , r < min ( m , n ) . The columns of W make up the new basis directions of the dimensions we are projecting onto . Each column of H represents the coefficients of each data point in this new subspace . There are a range of algorithms to conduct NMF , most of them involving minimising an objective function such as min W , H ||V−WH||2F subject to Wi , j ≥ 0 , Hi , j ≥ 0 . 1.2 USING AUTOENCODERS FOR NMF . Several authors have studied the addition of extra constaints to an autoencoder to perform NMF ( Lem me et al. , 2012 ; Ayinde et al. , 2016 ; Hosseini-Asl et al. , 2016 ; Smaragdis & Venkataramani , 2017 ) . These methods show some potential advantages over standard NMF including the implicit creation of the H matrix and straightforward adaptation to online techniques . In Figure 1 we show a representation of a one-hidden layer autoencoder for performing NMF . We feed in a data-point v ∈ Rm×1 , the latent representation is produced by h = f ( W1v ) where f is an element-wise ( linear or non-linear ) function with non-negative ouputs , h ∈ Rr×1 is the representation in the latent space and W1 ∈ Rr×m contains the weights of the first layer . It is also possible to add additional layers to make a deeper network with multiple hidden layers before arriving at the constriction layer which produces the h output . The final set of weights must be kept non-negative with an identity activation function such that x = Wfh where Wf ∈ Rm×r . The final weights of the autoencoder then can be interpreted as the dimensions of the new subspace with the elements of h as the coefficients in that subspace . The network is then trained so that x ≈ v . 1.3 VARIATIONAL AUTOENCODERS . NMF and autoencoders both produce a lower dimensional representation of some input data . However , neither produce a probability model , just a deterministic mapping . It is also not obvious how to generate new data from these lower dimensional spaces . Generative models solve both of these problems , enabling a probability distribution to be found linking the input and latent spaces whilst enabling new data to be created . One of the most popular recent generative model is the variational autoencoder ( VAE ) ( Kingma & Welling , 2013 ; Rezende et al. , 2014 ) . A VAE is a probabilistic model which utilises the autoencoder framework of a neural network to find the probabilistic mappings from the input to the latent layers and on to the output layer . Unlike a standard autoencoder the VAE finds a distribution between the latent and seen variables , which also enables the production of new data by sampling from the latent distributions . 1.4 PROBABILISTIC NON-NEGATIVE MATRIX FACTORISATION . Several authors have presented versions of NMF with a probabilistic element ( Paisley et al. , 2014 ; Cegmil , 2008 ; Schmidt et al. , 2009 ) which involve the use of various sampling techniques to estimate posteriors . Other work has been done using hidden Markov models to produce a probabilistic NMF ( Mohammadiha et al. , 2013 ) and utilising probabilistic NMF to perform topic modelling ( Luo et al. , 2017 ) . However , to our knowledge no one has utilised the ideas behind the VAE to perform NMF . Although probabilistic methods for NMF have been developed , even a full Bayesian framework still faces the problem that , for the vast majority of problems where NMF is used , we have little idea about what is the appropriate prior . We would therefore be forced to do model selection or introduce hyperparameters and perform inference ( maximum likelihood or Bayesian ) based on the evidence . However , as the posterior in such cases is unlikely to be analytic this is likely to involve highly time consuming Monte Carlo . In doing so we would expect to get results close to those we obtain using PAE-NMF . However , for machine learning algorithms to be of value they must be practical . Our approach , following a minimum description length methodology , provides a principled method for achieving automatic regularisation . Because it fits within the framework of deep learning it is relatively straightforward and quick to implement ( using software such as Keras or PyTorch with built-in automatic differentiation , fast gradient descent algorithms , and GPU support ) . In addition , our approach provides a considerable degree of flexibility ( e.g . in continuous updating or including exogenous data ) , which we believe might be much more complicated to achieve in a fully probabilistic approach . The next section lays out the key alterations needed to a VAE to allow it to perform NMF and is our main contribution in this paper . 2 PAE-NMF . The model proposed in this paper provides advantages both to VAEs and to NMF . For VAEs , by forcing a non-negative latent space we inherit many of the beneficial properties of NMF ; namely we find representations that tend to be sparse and often capture a parts based representation of the objects being represented . For NMF we introduce a probabilisitic representation of the vectors h which models the uncertainty in the parameters of the model due to the limited data . 2.1 IDEAS BEHIND PAE-NMF . Kingma & Welling ( 2013 ) proposed the VAE , their aim is to perform inference where the latent variables have intractable posteriors and the data-sets are too large to easily manage . Two of their contributions are showing that the “ reparameterization trick ” allows for use of standard stochastic gradient descent methods through the autoencoder and that the intractable posterior can be estimated . In a standard autoencoder we take some data point v ∈ Rm and run it through a neural network to produce a latent variable h = f ( v ) ∈ Rr where r < m and f is some non-linear element-wise function produced by the neural network . This is the encoding part of the network . We then run h through another neural network to produce an output v̂ = g ( h ) ∈ Rm , which is the decoding part of the network . The hope is that due to r < m the latent variables will contain real structure from the data . A standard VAE differs in that instead of encoding a deterministic variable h we find a mean , µ , and variance , σ2 of a Gaussian distribution . If we want to generate new data we can then sample from this distribution to get h and then run h through the decoding part of the network to produce our new generated data . PAE-NMF utilises the same objective function as a standard VAE , so for each data-point , we are minimising : obj =Eqφ ( h|v ) ( − log ( pθ ( v|h ) ) ) + DKL ( qφ ( h|v ) ||p ( h ) ) ( 1 ) ≈ 1 2σ2 ||v− v̂||2 + DKL ( qφ ( h|v ) ||p ( h ) ) ( 2 ) where DKL is the KL divergence , φ and θ represent the parameters of the encoder and decoder , respectively , v is an input vector with v̂ the reconstructed vector , created by sampling from p ( h|v ) and running the h produced through the decoding part of the network . The first term represents the reconstruction error between the original and recreated data-point , which we assume to be approximately Gaussian . The second term is the KL divergence between our prior expectation , p ( h ) , of the distribution of h and the representation created by the encoding part of our neural network , qφ ( h|x ) . We can interpret this as a regularisation term , it will prevent much of the probability density being located far from the origin , assuming we select a sensible prior . Another way of looking at it is that it will prevent the distributions for each datapoint being very different from one another as they are forced to remain close to the prior distribution . This objective function can also be interpreted as the amount of information needed to communicate the data ( Kingma et al. , 2014 ) . We can think of vectors in latent space as code words . Thus to communicate our data , we can send a code word and an error term ( as the errors fall in a more concentrated distribution than the original message they can be communicated more accurately ) . The log-probability term can be interpreted as the message length for the errors . By associating a probability with the code words we reduce the length of the message needed to communicate the latent variables ( intuitively we can think of sending the latent variables with a smaller number of significant figures ) . The KL divergence ( or relative entropy ) measures the length of code to communicate a latent variable with probability distribution qφ ( h|v ) . Thus by minimising the objective function we learn a model that extracts all the useful information from the data ( i.e . information that allows the data to be compressed ) , but will not over-fit the data . 2.2 STRUCTURE OF THE PAE-NMF . The structure of our PAE-NMF is given in Figure 2 . The input is fed through an encoding neural network which produces two vectors k and λ , which are the parameters of the qφ ( h|x ) distribution . The latent vector for that data-point , h , is then created by sampling from that distribution . However , this causes a problem when training the network because backpropagation requires the ability to differentiate through the entire network . In other words , during backpropagation we need to be able to find ∂hi∂ki and ∂hi ∂λi , where the i refers to a dimension . If we sample from the distribution we can not perform these derivatives . In variational autoencoders this problem is removed by the “ reparameterization trick ” ( Kingma & Welling , 2013 ) which pushes the stochasticity to an input node , which does not need to be differentiated through , rather than in the middle of the network . A standard variational autoencoder uses Gaussian distributions . For a univariate Gaussian the trick is to turn the latent variable , h ∼ q ( h|x ) = N ( µ , σ2 ) , which can not be differentiated through , into z = µ + σ where ∼ N ( 0 , 1 ) . The parameters µ and σ are both deterministic and can be differentiated through and the stochasticity is added from outside . We can not use the Gaussian distribution because we need our hi terms to be non-negative to fulfill the requirement of NMF . There are several choices for probability distributions which produce only non-negative samples , we use the Weibull distribution for reasons detailed later in this section . We can sample from the Weibull distribution using its inverse cumulative distribution and the input of a uniform distribution at an input node . In Figure 2 we display this method of imposing stochasticity from an input node through . Similarly to using a standard autoencoder for performing NMF the same restrictions , such as Wf being forced to stay non-negative , apply to the PAE-NMF .

Summary: This paper replaces the Gaussian latent variables in standard VAE with the Weibull distribution and therefore presents a VAE solution to nonnegative matrix factorization, under squared Euclidean distance loss function. The literature review summarizes four related work very well. The adopted Weibull distribution provides a tractable inverse CDF function and analytical form of the KL divergence, facilitating VAE inference. In particular, the effects of the entropy term are discussed in detail.  Experiments illustrate the generated data from the model,  the learned part-based dictionaries, and the distributions of latent variables from similar data points.