# Pattern Classification Based on RBF Networks with Self-Constructing Clustering and Hybrid Learning
## 4 Network Setup Phase
### Initialization of Weights and Biases  
Next, we derive initial values for the weights \(w_{f,j}\), \(1\leq j\leq J\), \(1\leq f\leq m\), between the hidden layer and the output layer, and for the biases \(b_{f}\), \(1\leq f\leq m\), in the output layer. Consider the sum of squared error of the whole training set \(X\):  
\[\sum_{i=1}^{N}\parallel\mathbf{y}^{(i)}-\mathbf{\overset{\alpha}{y}}^{(i)} \parallel^{2}=\sum_{i=1}^{N}\sum_{f=1}^{m}\left[y_{f}^{(i)}-\hat{y}_{f}^{(i)} \right]^{2} \tag{10}\]  
By Equations (4) and (5), we have  
\[\hat{y}_{f}^{(i)}=\text{sign}\left(s_{f}^{(i)}\right), \tag{11}\]  
\[s_{f}^{(i)}=w_{1,f}h_{1}^{(i)}+w_{2,f}h_{2}^{(i)}+\cdots+w_{I,f}h_{J}^{(i)}+b_ {f} \tag{12}\]  
where \(h_{J}^{(i)}=B_{j}\left(\mathbf{x}^{(i)}\right)\), for \(1\leq f\leq m\) and \(1\leq i\leq N\). For pattern \(i\), we have  
\[ \begin{split}& s_{1}^{(i)}=w_{1,1}h_{1}^{(i)}+w_{1,2}h_{2}^{(i)}+ \cdots+w_{1,J}h_{J}^{(i)}+b_{1}\\ & s_{2}^{(i)}=w_{2,1}h_{1}^{(i)}+w_{2,2}h_{2}^{(i)}+\cdots+w_{2,J}h _{J}^{(i)}+b_{2}\\ &\cdots=\cdots,\\ & s_{m}^{(i)}=w_{m,1}h_{1}^{(i)}+w_{m,2}h_{2}^{(i)}+\cdots+w_{m,J}h _{J}^{(i)}+b_{m}\end{split}  \tag{13}\]  
In total, there are \(N\times m\) such equations. Although only the sign of \(s_{f}^{(i)}\) matters in Equation (11), to be robust we'd like \(s_{f}^{(i)}\) as close to \(y_{f}^{(i)}\), which is either \(+1\) or \(-1\), as possible. Therefore, we demand\[y_{1}^{(i)} =w_{1,1}h_{1}^{(i)}+w_{1,2}h_{2}^{(i)}+\cdots+w_{1,J}h_{J}^{(i)}+b_{1}\] \[y_{2}^{(i)} =w_{2,1}h_{1}^{(i)}+w_{2,2}h_{2}^{(i)}+\cdots+w_{2,J}h_{J}^{(i)}+b_{2}\] \[\cdots =\cdots\, \tag{14}\] \[y_{m}^{(i)} =w_{m,1}h_{1}^{(i)}+w_{m,2}h_{2}^{(i)}+\cdots+w_{m,J}h_{J}^{(i)}+b _{m}\]  
for \(1\leq i\leq N\). Equation (14) can be rewritten as  
\[\left\|\mathbf{A}\mathbf{W}-\mathbf{Y}\right\|^{2} \tag{15}\]  
where  
\[\mathbf{A}=\left[ \begin{array}{cccc}h_{1}^{(1)}&h_{1}^{(2)}&\cdots&h_{1}^{( N)}\\ h_{2}^{(1)}&h_{2}^{(2)}&\cdots&h_{2}^{(N)}\\ \vdots&\vdots&\ddots&\vdots\\ h_{I}^{(1)}&h_{I}^{(2)}&\cdots&h_{I}^{(N)}\\ 1&1&\cdots&1\end{array} \right]^{T}, \tag{16}\]  
\[\mathbf{W}=\left[ \begin{array}{cccc}w_{1,1}&w_{1,2}&&\cdots&w_{1,J}&b_{1}\\ w_{2,1}&w_{2,2}&&\cdots&w_{2,J}&b_{2}\\ \vdots&\vdots&\ddots&\vdots&\vdots\\ w_{m,1}&w_{m,2}&&\cdots&w_{m,J}&b_{m}\end{array} \right]^{T}, \tag{17}\]  
\[\mathbf{Y}=\left[ \begin{array}{cccc}y_{1}^{(1)}&y_{1}^{(2)}&\cdots&y_{1}^{( N)}\\ y_{2}^{(1)}&y_{2}^{(2)}&\cdots&y_{2}^{(N)}\\ \vdots&\vdots&\ddots&\vdots\\ y_{m}^{(1)}&y_{m}^{(2)}&\cdots&y_{m}^{(N)}\end{array} \right]^{T} \tag{18}\]  
with size of \(N\times(J+1)\), \((J+1)\times m\), and \(N\times m\), respectively. To improve the robustness of the system [54], instead of minimizing Equation (15), we would like to minimize the penalized residual sum of squares:  
\[\left\|\mathbf{A}\mathbf{W}-\mathbf{Y}\right\|^{2}+\lambda\left\|\mathbf{W} \right\|^{2} \tag{19}\]  
where \(\lambda\) is a pre-defined non-negative coefficient to balance the robustness against collinearity and the capability of fitting the desired outputs well. To find the solution to the minimization problem, we first rewrite Equation (19) as  
\[\left\|\mathbf{A}\mathbf{W}-\mathbf{Y}\right\|^{2}+\lambda\left\|\mathbf{W} \right\|^{2}=\left\|\mathbf{A}\mathbf{W}-\mathbf{Y}\right\|^{2}+\left\|\sqrt{ \lambda}\mathbf{I}\mathbf{W}-\mathbf{0}\right\|^{2}=\left\|\left[ \begin{array} []{c}\mathbf{A}\\ \sqrt{\lambda}\mathbf{I}\end{array} \right]\mathbf{W}-\left[ \begin{array}{c} \mathbf{Y}\\ \mathbf{0}\end{array} \right]\right\|^{2}\]  
where \(\mathbf{I}\) is the \((J+1)\times(J+1)\) identity matrix and \(\mathbf{0}\) is the \((J+1)\times m\) zero matrix. Let \(\mathbf{A}^{\prime}=\left[ \begin{array}{c}\mathbf{A}\\ \sqrt{\lambda}\mathbf{I}\end{array} \right]\) and \(\mathbf{Y}^{\prime}=\left[ \begin{array}{c}\mathbf{Y}\\ 0\end{array} \right]\). Then Equation (19) becomes  
\[\left\|\mathbf{A}^{\prime}\mathbf{W}-\mathbf{Y}^{\prime}\right\|^{2} \tag{20}\]By the least squares method [55], the solution minimizing Equation (19) is  
\[