# Correntropy-Based Constructive One Hidden Layer Neural Network
## 4 Proposed Method
### Learning from Data Samples
#### 4.4.1 Input Side Optimization  
The optimization problem to adjust the input parameters is as follows:  
\[V_{L}^{S}=\max_{G_{L}}\Bigg{(}E\Bigg{(}\frac{1}{\sqrt{2\pi}\sigma}\exp\Bigg{(} -\frac{\left\|E_{L-1}-k_{L}G_{L}\right\  
In this paper, the tanh function is selected as the activation function, which is bipolar and invertible. Therefore:  
\[g^{-1}\bigg{(}\frac{E_{(L-1)i}}{k_{L}}\bigg{)}=X_{i}W_{(d+1)*1}\quad i=1,\ldots,N,\]  
where \(X_{i}=\left[ \begin{array}{cc}x_{i}&1\end{array} \right],W_{(d+1)*1}=\left[ \begin{array}{c}W_{L}\\ b_{L}\end{array} \right]\),  
\[X=\left[X_{1}^{T},\ldots,X_{N}^{T}\right]^{T}\]  
in which \(abs(. )\) is the absolute function. The range of \(g\) (domain of \(g^{-1}\)) is \([-1,1]\). Thus, it is necessary to rescale the error signal to be in the range. To do so, \(k_{L}\) is assigned as follows:  
\[k_{L}=\frac{\max(abs(E_{L}))}{\lambda^{\prime}},\]  
where \(\lambda^{\prime}\in(-1,1)-\{0\}\). Let \(H_{Li}=g^{-1}\bigg{(}\frac{E_{(L-1)i}}{k_{L}}\bigg{)}\), and therefore, the term  
\[\left\|E_{(L-1)i}-k_{L}G_{Li}\right\|^{2}\]  
can be replaced by  
\[\|H_{Li}-X_{i}W_{L}\|^{2},\]  
and thus the following problem is presented to adjust the input parameters  
\[\widehat{D}_{L}^{g}=\max_{W_{L}}\bigg{(}\sum_{i=1}^{N}\bigg{(}\exp\bigg{(}- \frac{\|H_{Li}-X_{i}W_{L}\|^{2}}{2\sigma^{2}}\bigg{)}\bigg{)}\bigg{)}.\]  
To achieve better generalization performance, the norm of the weights needs to be kept minimized too; thus, the problem above is reformulated as  
\[\widehat{D}_{L}^{g}=\max_{W_{L}}\bigg{(}\sum_{i=1}^{N}\bigg{(}\exp\bigg{(}- \frac{\|H_{Li}-X_{i}W_{L}\|^{2}}{2\sigma^{2}}\bigg{)}\bigg{)}-\frac{C}{2}\|W_ {L}\|^{2}\bigg{)}.\]  
It should be considered that if \(C=0\), both problems from above are equivalent. Since the necessary condition for convergence is that in each step and by adding each node amount of correntropy of the error should be increased, in the experiment section, \(C\approx 0\) and other amounts for \(C\) are checked and the best result is selected. This guarantees convergence of the method according to Theorem 6. The half-quadratic method is employed to adjust the input parameters. Based on Proposition 1, we have  
\[\cap_{L}^{g}(\alpha,W_{L})=\max_{W_{L}\alpha}\bigg{(}\sum_{i=1}^{N}\bigg{(} \alpha_{i}\frac{\|H_{Li}-X_{i}W_{L}\|^{2}}{2\sigma^{2}}-\Phi(\alpha_{i})\bigg{)} -\frac{C}{2}\|W_{L}\|^{2}\bigg{)}.\]  
The local solution of the above optimization problem is adjusted using the following iterative process:  
\[\left\{ \begin{array}{c}\alpha_{i}^{t+1}=-G(H_{Li}-X_{i}W_{L})\\ W_{L}^{t+1}=\arg_{W_{L}}\max\left(\sum_{i=1}^{N}\bigg{(}\alpha_{i}^{t+1}\frac{ \|H_{Li}-X_{i}W_{L}\|^{2}}{2\sigma^{2}}\bigg{)}-\frac{C}{2}\|W_{L}\|^{2}\right) \end{array} \right.,\]  
i.e., the following optimization problem needs to be solved in each iterate:  
\[\mathcal{V}_{L}^{g}(\alpha,W_{L})=\max_{W_{L}}\bigg{(}\sum_{i=1}^{N}\bigg{(} \alpha_{i}^{t+1}\frac{\|H_{Li}-X_{i}W_{L}\|^{2}}{2\sigma^{2}}\bigg{)}-\frac{C} {2}\|W_{L}\|^{2}\bigg{)}.\]Since \(\sigma^{2}\) is a constant term, it can be removed from the optimization problem. Then, the optimization problem can be multiplied by \(\frac{1}{c}\). We set \(C^{\prime}=\frac{1}{c}\). Thus, the following constraint optimization problem is obtained:  
\[\left\{ \begin{array}{l}\max\sum_{i=1}^{N}\frac{C^{\prime}}{2}\left(\alpha_{i} ^{t+1}\xi_{i}^{2}\right)-\frac{1}{2}\|W_{L}\|^{2}\\ \text{s.t. }X_{i}W_{L}=H_{Li}-\xi_{i}\quad i=1,\ldots,N\end{array} \right.\]  
The Lagrangian is constituted as  
\[L(\xi_{i},\eta_{i},W_{L})=\sum_{i=1}^{N}\frac{C^{\prime}}{2}\left(\alpha_{i}^{ t+1}\xi_{i}^{2}\right)-\|W_{L}\|^{2}-\sum_{i=1}^{N}\eta_{i}(X_{i}W_{L}-H_{Li}+ \xi_{i}).\]  
The derivations of the Lagrangian function with respect to its variables are the following  
\[\frac{\partial L}{\partial W_{L}}=0\to W_{L}=-\sum_{i=1}^{N}\eta_{i}X_{i}^{T} =-X^{T}\eta,\]  
where \(\eta=[\eta_{1},\ldots,\eta_{N}]^{T}\).