## 2 The Fundamental Theories
### Sparse Bayesian Extreme Learning Machine (SBELM)  
SBELM learns the output weights by using Bayesian inference. For a two-class classification problem, the probability of each training sample is expressed as \(\mathrm{P}(t\,|\,x)\). The likelihood function can be represented as:  
\[\mathrm{P}(t|\beta,h)=\prod_{i=1}^{N}\sigma[y(h_{i},\beta)]^{t_{i}}(1-\sigma[ y(h_{i},\beta)])^{1-t_{i}} \tag{5}\]  
where \(y(h_{i},\beta)=h\beta\), \(t=(t_{1},\ldots,t_{N})^{T}\) is the class label of \(N\) samples in which \(t_{i}\ \in\ \{0,1\}\) and \(\sigma(\cdot)\) represents sigmoid function as follows:  
\[\sigma[y(h_{i},\beta)]=1/\left(1+e^{-y(h_{i},\beta)}\right) \tag{6}\]  
The output weight \(\beta_{i}\) obeys Gaussian conditional probability distribution with a mean of 0 and a variance of \(a_{i}{}^{-1}\):  
\[\mathrm{P}(\beta_{i}|\alpha_{i})=\mathcal{N}\Big{(}\beta_{i}\big{|}0,{a_{i}}^ {-1}\Big{)},\ \alpha\ =[\alpha_{1},\ldots,\alpha_{L}]^{T} \tag{7}\]\[\mathrm{P}(\beta|\alpha)=\prod\
olimits_{i=1}^{L}\frac{\alpha_{i}}{\sqrt{2\pi}} \exp\left(-\frac{\alpha_{i}\beta_{L}^{2}}{2}\right) \tag{8}\]  
where \(\alpha\) is the hyper-parameter to determine the prior distribution of \(\beta\). The key task is to focus on the marginal likelihood distribution \(\mathrm{P}(t|\alpha,H)\) of output t on condition of \(\alpha\) and \(H\). The parameters \(\alpha\) can be obtained by maximizing the marginal likelihood using the Laplace approximation approach.  
Generally, the iterative reweighted least squares algorithm (IRLS) is used to calculate the mean value of the Gauss distribution \(\beta_{MP}\). To update \(\beta_{MP}\), gradient \(\
abla\mathrm{E}\) and Hessian matrix \(\Phi\) are calculated firstly:  
\[\
abla\mathrm{E}\ =\
abla_{\beta}\log\{P(t|\beta,H)P(\beta|\alpha)\}=H^{\mathrm{ T}}(t-y)-A\beta \tag{9}\]  
\[\Phi\ =\
abla_{\beta}\
abla_{\beta}\log\{P(t|\beta,H)P(\beta|\alpha)\}=- \left(H^{\mathrm{T}}BH+A\right) \tag{10}\]  
\[A=\mathrm{diag}(a) \tag{11}\]  
where \(\mathrm{y}\ =(y_{1},\ldots,y_{N})^{T}\) represents the predicted output of N samples, B represents the diagonal matrix and \(b_{i}=y_{i}(1-y_{i})\). According to IRLS, \(\beta_{MP}\) is updated as follows:  
\[\beta_{MP}=\beta_{MP}-\ \Phi^{-1}\
abla\mathrm{E}\ =\left(H^{\mathrm{T}}BH+A \right)^{-1}\!H^{\mathrm{T}}B\hat{t} \tag{12}\]  
where \(f\ =\ H\beta+B^{-1}(t-y)\). \(\beta_{MP}\) and \(\sum\) of the weight parameter Gauss distribution which is approximated using the Laplace approximation method can be expressed as:  
\[\sum=\left(H^{\mathrm{T}}\mathrm{BH}\ +\ \mathrm{A}\right)^{-1} \tag{13}\]  
\[\beta_{MP}=\sum H^{\mathrm{T}}B\hat{t} \tag{14}\]  
After obtaining the Gauss distribution of weight parameter \(\beta\), the logarithm of the marginal likelihood function is expressed as:  
\[\mathcal{L}(a)=\log P(t|a,H) \tag{15}\]  
Calculate the partial derivative of \(\mathcal{L}(a)\) to the hyper-parameter \(a\) and set the value to 0:  
\[\frac{\partial\mathcal{L}(a)}{\partial a_{i}}=\frac{1}{2a_{i}}-\frac{1}{2} \Sigma_{ii}-\frac{1}{2}\beta_{MP_{i}}^{2}=>a_{i}^{new}=\frac{1-a_{i}\Sigma_{ ii}}{\beta_{MP_{i}}^{2}} \tag{16}\]  
Using Formula (16), most of \(a_{i}\) tends to infinity, and its corresponding weight \(\beta_{i}\) tends to 0. The hidden layer nodes with a weight of 0 are deleted to achieve the sparseness of hidden layer nodes. With the sparse hidden layer weight vector \(\beta^{\prime}\), predict the probability of unknown sample \(x_{new}\) using \(\mathrm{y}(h,\beta^{\prime})=h\beta^{\prime}\):  
\[\sigma\big{[}\mathrm{y}\big{(}h,\beta^{\prime}\big{)}\big{]}=1/\left(1+e^{- \mathrm{y}(h,\beta^{\prime})}\right) \tag{17}\]