# Consensus Big Data Clustering for Bayesian Mixture Models
## 3 Methodology  
### Finite Mixture of Normals  
Suppose we have a set of samples \(X_{1},X_{2},\ldots,X_{n}\) that can be modelled as  
\[p(X_{i},|\mu_{1:k},\tau_{1:k},q_{1:k})=\sum_{j}^{k}q_{j}\text{N}(\mu_{j},\tau_ {j}^{-1}), \tag{3}\]  
where N(.) denotes the Normal distribution and \(q_{j}\) represents the weight of the \(j\)-th component, with \(q_{j}>0\) and \(\sum_{j=1}^{k}q_{j}=1\). In addition, let us introduce the latent variable \(Z_{1:n}\) to induce the mixture. Thus, we have that \(X_{i}|Z_{i}=j\sim\text{N}(\mu_{j},\tau_{j})\). Given the introduction of the latent variable, we can rewrite the likelihood function as  
\[p(X_{1:n}|Z_{1:n},\mu_{1:k},\tau_{1:k},Z_{1:n})=\prod_{j=1}^{k}\prod_{i:l_{(Z_{ i}=j)}}^{n}\text{N}(\mu_{j},\tau_{j}), \tag{4}\]  
where \(I_{(Z_{i}=j)}=1\) if \(Z_{i}=j\), and \(I_{(Z_{i}=j)}=0\) otherwise. In addition, the latent variables \(Z_{1:n}\sim\text{Categorical}(1,q)\). That is,  
\[p(Z_{1:n}|q_{1:k})=\prod_{i=1}^{k}\prod_{j=1}^{k}q_{j}^{I_{(Z_{i}-i)}}=\prod_{ j=1}^{k}q_{j}^{n_{j}}, \tag{5}\]  
where \(n_{j}=\sum_{i}^{n}I_{(Z_{i}=j)}\) represents the number of observations falling into component \(j\). With that, we can define the joint distribution of \(X_{1:n}\) and \(Z_{1:n}\) as  
\[p(X_{1:n},Z_{1:n}|\mu_{1:k},\tau_{1:k},q_{1:k})=p(X_{1:n}|Z_{1:n },\mu_{1:k},\tau_{1:k})p(Z_{1:n}|q_{1:k}), \tag{6}\] \[=\prod_{j=1}^{k}\left[\prod_{i:l_{(Z_{i}=j)}}^{n}\text{N}(\mu_{j},\tau_{j})\right]q_{j}^{n_{j}}. \tag{7}\]  
For notational convenience, let us denote \(\theta_{k}=\{\mu_{1:k},\tau_{1:k},q_{1:k}\}\), and \(\omega_{ij}=p(Z_{i}=j|\theta_{k},X_{1:n})/\sum_{j}p(Z_{i}=j|\theta_{k},X_{1:n})\). Given the expressions above, we have that \(Z_{1:n}\) conditioned on \(X_{1:n}\) are independent with a probability of classification given by  
\[p(Z_{i}=j|\theta_{k},X_{1:n}) \propto p(X_{i}|Z_{i},\mu_{j},\tau_{j},Z_{i})p(Z_{i}), \tag{8}\] \[\propto\text{N}(\mu_{j},\tau_{j})q_{j}. \tag{9}\]  
In the end, we have that  
\[p(Z_{i}|\theta_{k},X_{1:n})\sim\text{Categorical}(1,\omega_{ij}). \tag{10}\]To estimate the components of the finite mixture of Normals under the Bayesian paradigm, we consider the following priors:  
\[\mu_{j}|\tau \sim\mathrm{N}(m_{j},v_{j}/\tau_{j}), \tag{11}\] \[\tau_{j} \sim\mathrm{G}(a_{j},b_{j}),\] (12) \[q_{1:k} \sim\mathrm{Dirichlet}(\tau_{1},r_{2},...,r_{k}). \tag{13}\]  
To construct the MCMC structure, we need the full conditionals for \(\mu_{j}\), \(\tau_{j}\), and \(q_{j}\), which are given below.  
\[p(\mu_{j}|-)\propto p(X_{1:n}|\theta_{k},Z_{1:n})p(\mu_{j}),\mu_{j}|-\sim \mathrm{N}(M_{j},V_{j}), \tag{14}\]  
where  
\[M_{j}=(n_{j}+1/v_{j})^{-1}\left(\sum_{i:Z_{i}=j}x_{i}+m_{j}/v_{j}\right) \tag{15}\]  
and  
\[V_{j}=\frac{v_{j}}{(n_{j}v_{j}+1)\tau_{j}},\tau|-\sim\mathrm{G}(A_{j},B_{j}), \tag{16}\]  
where  
\[A_{j}=\frac{n_{j}}{2}+a_{j},B_{j}=b_{j}+\frac{m_{j}^{2}}{2v_{j}}+\frac{\sum_{ i:Z_{i}=j}X_{i}^{2}}{2}-\frac{1}{2}\big{(}n_{j}+1/v_{j}\big{)}M_{j}^{2}. \tag{17}\]  
From the above, we have:  
\[p(q_{1:k}|-) \propto p(Z_{1:n}|q_{1:k})p(q_{1:k}),\] \[\propto\prod_{j=1}^{k}q_{j}^{n_{j}}p(q_{1:k}),\] \[\propto\mathrm{Multinomial}(n,q_{1:k})\times\mathrm{Dirichlet}( \tau_{1:k}).\]  
\[q_{1:k}|-\sim\mathrm{Dirichlet}(\tau_{1:k}+n_{1:k}) \tag{18}\]  
where \(n=\sum_{j}n_{j}\).