# BELMKN: Bayesian Extreme Learning Machines Kohonen Network
## 2 Methodology
### Feature Learning Using Extreme Learning Machine (ELM)  
In most of the applications, it is not just the best clustering algorithm that matters but also the choice of the right feature learning method [22]. In this study, the first phase of the BELMKN uses ELM-based feature learning approach. ELM-based feature learning is useful in projecting the input space onto a \(\beta\) dimension, which is efficient for predicting the number of clusters, as well as for clustering the dataset efficiently [28]. ELM is chosen over other gradient-based methods for feature learning due to its non-iterative nature, and it arrives at a closed form solution by converging to global minima by minimizing the error at a faster rate.  
Consider the input data \(X_{i}\), in which \(i=1,2,\ldots,d\) with dimension \(R^{d}\). ELM consists of three layers: the input layer, a hidden layer, and the output layer. The weights between the input and the hidden layer are randomly initialized from a uniform distribution, given by \(WeR^{d\times a}\), in which \(d\) is the number of input neurons and \(\alpha\) is the number of hidden neurons with a bias. The weight (\(\delta\)) between the hidden layer and the output layer needs to be computed. A feed-forward pass is performed between the input and hidden layer. The hidden layer activations (\(H\)) are calculated using,  
\[H=s(XW+b) \tag{1}\]  
in which \(HeR^{n_{i}\times a}\), \(n_{i}\) is the number of input neurons, \(s(.)\) is the sigmoidal activation function; in general, given as \(s(v)=\frac{1}{1+e^{-v}}\), in which \(v=XW+b\), and \(b\) is the input layer bias. The hidden to output layer weights are calculated using the objective function [28],  
\[\min_{\delta\alpha\times\beta}\lVert\delta\rVert^{2}+\lambda Tr(\delta^{T}H^{T }L\delta H) \tag{2}\]  
in which \(Tr(.)\) denotes the trace of the matrix, \(\lambda\) is the tradeoff parameter, and \(L\) is the graph Laplacian given by,  
\[L=D-S\]  
Figure 1: BELMKN architecture.  
in which \(S\) is the similarity matrix computed by the Gaussian function and \(D\) is the diagonal matrix given by,  
\[D_{ii}=\sum_{j=1}^{n}S_{ij} \tag{4}\]  
The above equation is a closed-form solution, i.e., it attains minima at \(\delta=0\). In general, the weights between the hidden and output layers should avoid converging to zero. Hence, we impose the constraints on Equation (2) as \((H\delta)^{T}H\delta=I_{\beta}\), in which \(I_{\beta}\) is the identity matrix.  
The optimal solution to Equation (2) is computed by choosing \(\beta\) smallest eigenvalues with corresponding normalized eigenvectors using [28],  
\[(I_{\alpha}+\lambda H^{T}LH)v=\gamma H^{T}Hv \tag{5}\]  
The eigenvectors capture useful representations of the data, and after sorting the eigenvalue in ascending order, the eigenvector corresponding to the first eigenvalue is discarded, as it is not useful for data representation. Let \(\gamma_{j}\), in which \(j=1\leq 2\leq\ldots\leq\beta+1\) is the smallest \(\beta+1\) eigenvalues of Equation (3) with \(v_{j}\) is the corresponding eigenvectors.  
The matrix \(\delta=[\overset{\sim}{\gamma}_{2},\overset{\sim}{\gamma}_{3},\ldots, \overset{\sim}{\gamma}_{\beta+1}]\), in which \(\overset{\sim}{\gamma}_{j}=\frac{v_{j}}{\|Hv_{j}\|},j=2,3,\ldots,\beta+1\) are the normalized eigenvectors. After obtaining the \(\delta\) matrix, the output matrix \(EeR^{\alpha\times\beta}\) is calculated using [28],  
\[E=H\delta \tag{6}\]