# Correntropy-Based Constructive One Hidden Layer Neural Network
## 4 Proposed Method
### Learning from Data Samples
#### 4.4.2 Output Side Optimization  
When the input parameters of the new node are obtained from the previous step, the new node is named \(\mathbf{G}_{L}^{\text{sim}(\epsilon)}=\left\{\mathcal{C}_{L1}^{\text{sim}( \epsilon)},\ldots,\mathcal{C}_{LN}^{\text{sim}(\epsilon)}\right\}\), where \(\mathcal{G}_{Li}^{\text{sim}\left(\epsilon\right)}=g_{L}^{\text{sim}(\epsilon )}\left(x_{i}\right)\) and the output parameter is adjusted using the following optimization problem:  
\[V_{L}^{\beta}=\underset{\beta_{L}}{\text{max}}\left(E\left(\frac{1}{\sqrt{2 \pi\sigma}}\exp\left(-\frac{\left\|\varepsilon_{L-1}-\beta_{L}\mathcal{C}_{Li }^{\text{sim}(\epsilon)}\right\|^{2}}{2\sigma^{2}}\right)\right)\right).\]  
The expectation can be approximated on training samples:  
\[\widehat{V}_{L}^{\beta}=\underset{\beta_{L}}{\text{max}}\left(\frac{1}{N\sqrt {2\pi\sigma}}\sum_{i=1}^{N}\left(\exp\left(-\frac{\left\|E_{(L-1)i}-\beta_{L} \mathcal{C}_{Li}^{\text{sim}(\epsilon)}\right\|^{2}}{2\sigma^{2}}\right) \right)\right).\]  
The constant term \(\frac{1}{N\sqrt{2\pi\sigma}}\) can be removed and the following problem can be solved instead \(\widehat{V}_{L}^{\beta}\):  
\[\widehat{U}_{L}^{\beta}=\underset{\beta_{L}}{\text{max}}\left(\sum_{i=1}^{N} \left(\exp\left(-\frac{\left\|E_{(L-1)i}-\beta_{L}\mathcal{C}_{Li}^{\text{sim }(\epsilon)}\right\|^{2}}{2\sigma^{2}}\right)\right)\right).\]  
Similar to the previous step, the half-quadratic method is employed to adjust the output parameter. Based on Proposition 1, we obtain  
\[u_{L}^{\beta}(\gamma,\beta_{L})=\underset{\beta_{L}\gamma}{\text{max}}\left( \sum_{i=1}^{N}\left(\gamma_{i}\frac{\left\|E_{(L-1)i}-\beta_{L}\mathcal{C}_{ Li}^{\text{sim}(\epsilon)}\right\|^{2}}{2\sigma^{2}}-\Phi(\gamma_{i}) \right)\right).\]The local solution of the above optimization problem is adjusted using the following iterative process:  
\[\left\{ \begin{array}{c}\gamma_{i}^{t+1}=-k_{\sigma}\Big{(}E_{Li}-\beta_{L}^{t}C_ {Li}^{\text{sim}(e)}\Big{)}\\ \beta_{L}^{t+1}=\arg_{\beta_{L}}\max\left(\sum_{i=1}^{N}\left(\gamma_{i}^{t+1} \frac{\|E_{Li}-\beta_{L}^{t}C_{Li}^{\text{sim}}(e)}{2\gamma^{2}}\right)\right) \end{array} \right..\]  
i.e., the following optimization problem is required to be solved in each iteration:  
\[\mathcal{V}_{L}^{\beta}(\gamma,\beta_{L})=\max_{\beta_{L}}\left( \sum_{i=1}^{N}\left(\gamma_{i}^{t+1}\frac{\left\|E_{Li}-\beta_{L}C_{Li}^{\text {sim}(e)}\right\|^{2}}{2\sigma^{2}}\right)\right),\] \[\
u_{L}^{\beta}(\gamma,\beta_{L})=\max_{\beta_{L}}\left(\frac{1}{ 2\sigma^{2}}\bigg{(}\Big{(}E_{L}-\beta_{L}\mathbf{G}_{L}^{\text{sim}(e)}\Big{)} \Theta\Big{(}E_{L}-\beta_{L}\mathbf{G}_{L}^{\text{sim}(e)}\Big{)}^{T}\bigg{)} \right).\]  
where \(\Theta\) is a diagonal matrix with \(\Theta_{ii}=\gamma_{i},i=1,\ldots,N\). \[\mathcal{V}_{L}^{\beta}(\gamma,\beta_{L})=\max\biggl{(}\frac{1}{2\sigma^{2}} \Big{(}E_{L}\Theta E_{L}^{\text{T}}+\beta_{L}^{2}\mathbf{G}_{L}^{\text{sim}(e)} \Theta\mathbf{G}_{L}^{\text{sim}(e)^{T}}-\beta_{L}\mathbf{E}_{L}\Theta\mathbf{G}_{L}^{ \text{sim}(e)^{T}}-\beta_{L}\mathbf{G}_{L}^{\text{sim}(e)}\Theta E_{L}^{T}\Big{)} \biggr{)}.\]  
The optimum output weight is adjusted by differentiating \(\mathcal{V}_{L}^{\beta}(\gamma,\beta_{L})\) with respect to \(\beta_{L}\) as  
\[\left(2\beta_{L}\mathbf{G}_{L}^{\text{sim}(e)}\Theta\mathbf{G}_{L}^{\text{sim}(e)^{T} }-E_{L}\Theta\mathbf{G}_{L}^{\text{sim}(e)^{T}}-\mathbf{G}_{L}^{\text{sim}(e)}\Theta E _{L}^{T}\right)=0,\]  
\[\beta_{L}=\frac{\mathbf{E}_{L}\Theta\mathbf{G}_{L}^{\text{sim}(e)^{T}}}{\mathbf{G}_{L}^{ \text{sim}(e)}\Theta\mathbf{G}_{L}^{\text{sim}(e)^{T}}}.\]  
Finally, the output weight is adjusted by the following iterative process:  
\[\left\{ \begin{array}{c}\gamma_{i}^{t+1}=-k_{\sigma}\Big{(}E_{Li}-\beta_{L}^ {t}C_{Li}^{\text{sim}(e)}\Big{)}\\ \beta_{L}^{t+1}=\frac{\mathbf{E}_{L}\mathbf{G}_{L}^{\text{sim}(e)^{T}}}{\mathbf{G}_{L}^{ \text{sim}(e)}\Theta\mathbf{G}_{L}^{\text{sim}(e)^{T}}}.\end{array} \right..\]  
In these two phases, the parameters of the new node ( \(L\)-th added node where \(L\in\mathbb{N}\)) are tuned and then fixed.