# Collaborative Filtering Based on a Variational Gaussian Mixture Model
## 3 Method
### Variational Autoencoder Based on the GMM for CF  
In this part, we modified the prior distribution of the latent factors of the standard VAE into a Gaussian mixture model, and we propose a GVAE-CF algorithm for the CF problem. We show the difference between GVAE-CF and the most advanced algorithms, and then we define the model structure and a novel loss function. First of all, we briefly introduce the Gaussian mixture model. The mathematical form of Gaussian mixture is as follows:  
\[f(z)=\sum_{i=1}^{k}w_{i}g_{i}(z|\mu_{i},\Sigma_{i}), \tag{5}\]  
where \(g_{i}(\cdot)\) is the standard Gaussian model with mean \(\mu_{i}\) and covariance matrix \(\Sigma_{i}\), \(w_{i}\) is the weight coefficient of the i-th standard Gaussian model, and \(w_{i}>0\), \(\sum_{i=1}^{k}w_{i}=1\), \(k\) is the number of single Gaussian models. For each user \(u\), we need to learn approximately the computable posterior distribution \(p(z_{u}|x_{u})\). Variational inference simply uses a variational distribution \(q(z_{u}|x_{u})\) as the approximate uncalculated posterior. Here, we set \(q(z_{u}|x_{u})\) to be k completely diagonal Gaussian distributions. Therefore, the covariance matrix \(\Sigma_{i}\) can be represented by the vector \(\sigma_{i}\). Similar to the standard variational autoencoder, GVAE-CF is also represented as a latent factor model. The most critical difference lies in the structural changes brought about by the distribution of the latent factors. As shown in Figure 1, a vector \(x\in\mathbb{R}^{D}\) is used as the input, and it is mapped to a hidden layer \(z\in\mathbb{R}^{M}\) through the following functions:  
\[ \begin{array}{l}\mu=\mu_{\phi}(x_{u})\\ \sigma=\sigma_{\phi}(x_{u})\\ w=w_{\phi}(x_{u}).\end{array}  \tag{6}\]  
Included are the mean vector \(\mu=\sum_{i=1}^{K}w_{i}\mu_{i}\), the variance vector \(\sigma=\sum_{i=1}^{K}w_{i}\sigma_{i}\), and the coefficient vector \(w\in\mathbb{R}^{K}\), among which \(\sigma\) refers to the diagonal matrices, which can be represented by vectors. The latent variable \(z\) adopts the reparameterization trick:  
\[z_{u}=\mu+\sigma^{\frac{1}{2}}\odot\epsilon. \tag{7}\]  
The sampling process of \(\epsilon\) is: \(\epsilon\sim\mathcal{N}(0,I_{M})\). Here, \(\epsilon\) is a vector, and \(\odot\) means the dot product between two vectors. Learning the GVAE-CF latent factor algorithm is similar to the standard variational inference [25]. The reconstruction error term in Equation (2) is not a required change. As we needed to modify the corresponding KL divergence loss. We tried:  
Figure 1: Variational autoencoder based on a Gaussian mixture model (GMM). \[ \begin{split}\text{KL}(q_{\phi}(z|x)||p_{\theta}(z|x)))& =min\{p(z_{u}|x_{u})||q(z_{u}|x_{u})\}\\ &\leq\frac{1}{2}\sum_{i=1}^{K}w_{i}\Big{\{}tr(\Sigma_{i}+\mu_{i}^ {T}\mu_{i})\Big{\}}+\sum_{i=1}^{K}w_{i}log(sum_{i=1}^{K}z_{ij})\\ &\leq\frac{1}{2}\sum_{i=1}^{K}w_{i}\Big{\{}Mlog(2\pi\epsilon)-Mtr (\Sigma_{i}+\mu_{i}^{T}\mu_{i})\Big{\}}\\ &\qquad+\sum_{i=1}^{K}w_{i}log(Csum_{i=1}^{K}z_{ij})-\sum_{i=1}^ {K}w_{i}log(C),\end{split}  \tag{8}\]  
where \(M\) is the dimension of the latent space, \(e\) is Euler number, and \(C\) is a larger constant set to prevent the gradient from disappearing. When calculating \(z_{ij}=\mathcal{N}(\mu_{i}|\mu_{j},\sigma_{i}+\sigma_{j})\), in order to simplify the model and to facilitate the calculation, we can consider the spherical distribution we use, that is, the diagonal matrix of the covariance matrix, so \(z_{ij}\) can be simplified as:  
\[z_{ij}=\frac{1}{(2\pi)^{\frac{N}{2}}|\sigma_{i}+\sigma_{j}|^{2}}e^{-\frac{1}{ 2}((\mu_{i}-\mu_{j})^{2})^{T}(\sigma_{i}-\sigma_{j})^{-1}}. \tag{9}\]  
As per the standard VAE, both estimate the parameters \(\theta\) and \(\phi\). We can learn the unbiased estimation of ELBO by sampling \(z_{u}\sim q_{\phi}\), and can update and optimize the parameters through the stochastic gradient descent algorithm. However, the process of sampling is non-differentiable. Here, we used the reparameterization trick to avoid this problem [26; 27].