# Integrated Model Selection and Scalability in Functional Data Analysis Through Bayesian Learning
## 3 Methods
### Variational Bayesian Inference  
Variational inference optimizes \(\mathcal{Q}\) by maximizing the lower bound \(\mathcal{L}\) (minimizing the KL divergence between the actual and surrogate distributions):  
\[\mathbb{E}_{\mathcal{Q}}\bigg{[}\ln\frac{\Pr[\Theta,Y]}{\mathcal{Q}(\Theta)} \bigg{]}=-\mathrm{KL}(\mathcal{Q}(\Theta)||\Pr[\Theta|Y])+\ln\Pr[Y]\propto- \mathrm{KL}(\mathcal{Q}(\Theta)||\Pr[\Theta|Y]). \tag{5}\]  
The mean field variational family is used for \(\mathcal{Q}\). It simplifies the optimization by assuming that the surrogate posterior distributions are independent, allowing each variable in the posterior to be optimized independently: \(\mathcal{Q}_{\Theta}=\prod_{i}\mathcal{Q}_{\Theta_{i}}\). The posterior for each variable is chosen to be conjugate, further simplifying the optimization. Thus, the posteriors of the component scores \(Z\), the weighting matrix \(W\), and the mean weights \(Z\) are normal distributions. Here, \(W\) is vectorized via \(\mathrm{vec}\left(W\right)\) without altering its normality assumption. Meanwhile, the posteriors of the precision variables of noise \(\sigma^{-2}\), components \(\alpha\), basis functions \(\beta\), and mean weights \(\eta\) are Gamma distributions:  
Figure 1: Probabilistic graphical model for the full model. \[\mathcal{Q}_{Z}(Z) =\prod_{i}\mathcal{Q}_{Z_{i}}(Z_{i})=\prod_{i}\mathcal{N}(Z_{i}|\mu_{ Z_{i}^{\prime}},\Sigma_{Z_{i}}) \tag{6}\] \[\mathcal{Q}_{W}(W) =\mathcal{N}(\text{vec}(W)|\mu_{\text{vec}(W)},\Sigma_{\text{vec }(W)})\] (7) \[\mathcal{Q}_{Z}(Z) =\mathcal{N}(Z|\mu_{Z},\Sigma_{Z})\] (8) \[\mathcal{Q}_{c}(\sigma) =\Gamma(\sigma^{-2}|a_{\sigma},b_{\sigma})\] (9) \[\mathcal{Q}_{\alpha}(\alpha) =\prod_{j}\mathcal{Q}_{\alpha_{j}}=\prod_{j}\Gamma(\alpha_{j}|a_ {\alpha_{j}},b_{\alpha_{j}})\] (10) \[\mathcal{Q}_{\beta}(\beta) =\prod_{k}\mathcal{Q}_{\beta_{k}}=\prod_{k}\Gamma(\beta_{k}|a_{ \beta_{k}},b_{\beta_{k}})\] (11) \[\mathcal{Q}_{\eta}(\eta) =\Gamma(\eta,|a_{\eta},b_{\eta}) \tag{12}\]  
Update Steps  
In mean field approximation using the surrogate posterior \(\mathcal{Q}_{\Theta}=\prod_{i}\mathcal{Q}_{\Theta_{i}}\) conditioned on observations \(Y\), the lower bound is maximized with respect to each unknown \(\Theta_{i}\). With the conjugate prior, the optimal updates (denoted with \"\(\leftarrow\)\") make the moments of \(\mathcal{Q}_{\Theta_{i}}\) equal to the moments conditioned on the remaining parts of \(\mathcal{Q}_{\Theta}\)[38]:  
\[\mathcal{Q}_{\Theta_{i}}\leftarrow\frac{\exp\left(\mathbb{E}_{\mathcal{Q}_{ \Theta_{i}}}[\ln(\Pr[Y,\Theta])]\right)}{\int\exp\left(\mathbb{E}_{\mathcal{Q} _{\Theta_{i}}}[\ln(\Pr[Y,\Theta])]\right)d\Theta_{i}} \tag{13}\]  
From Equation (13), detailed update rules for each variable are presented subsequently, and the derivations of these formulas are given in the Appendix part. **Updates for the parameters of the posterior for the precision of components \(\mathcal{Q}_{\alpha_{j}},\forall j=1:J\)**:  
Figure 1:0 Figure 1:1  
where Equation (14) calculates the corrected degrees of freedom and Equation (15) calculates the corrected sum of squares. As \(a_{0}\) and \(b_{0}\) approach 0, the expectation of precision \(\alpha_{j}\), which is \(\mathbb{E}_{\mathcal{Q}_{\alpha_{j}}}[\alpha_{j}]=\frac{a_{\alpha_{j}}}{b_{ \alpha_{j}}}\), is exactly the inverse of the empirical or sample variance. **Updates for the parameters of the posterior of the precision of the mean weights \(\mathcal{Q}_{\eta}\)**:  
Figure 1:2 Figure 1:3  
**Updates for the parameters of the posterior of the precision of basis functions \(\mathcal{Q}_{\beta_{k}},\forall k=1:K\)**:  
Figure 1:4 Figure 1:5 Figure 1:6  
**Updates for the parameters of the posterior of the mean weights \(\mathcal{Q}_{Z}\)**:Figure 1:7 Figure 1:8  
where \(\operatorname{diag}(\beta)\) denotes the diagonal matrix with diagonal entries given by \(\beta\).