## 2 The Hessian for Neural Network Tuning
### Design of the Hessian  
In this article, we utilize a neural network with only one hidden layer. It could be expanded to a multilayer neural network, but in this article, we utilize a small neural network. This neural network utilizes sigmoid maps in the hidden layer and a linear map in the output layer. We express the neural network as  
\[q_{l}=\sum_{j}\varphi_{lj}g(\sum_{i}\theta_{ji}b_{i}) \tag{1}\]  
where \(\theta_{ji}\) is the scale parameter of the hidden layer, \(\varphi_{lj}\) is the scale parameter of the output layer, \(g\) is the activation map, \(b_{i}\) are the inputs, and \(q_{l}\) are the outputs. We take into account the neural network in Figure 1 where \(n\) is the input layer, \(l\) is the hidden layer, and \(m\) is the output layer. We express the scale parameters from the input layer to the hidden layer as \(\theta_{ji}\) and the scale parameters from the hidden layer to the output layer as \(\theta_{lm}\). We express the cost map as  
\[E=\frac{1}{2}\sum_{l=1}^{L_{T}}(q_{l}-t_{l})^{2} \tag{2}\]  
where \(q_{l}\) is the output of the neural network and \(t_{l}\) is the target, \(L_{T}\) is the total output number. We express the forward propagation as  
\[ \begin{array}{l}z_{j}=\sum\limits_{i}\theta_{ji}b_{i}\\ o_{j}=g(z_{j})\\ x_{l}=\sum\limits_{j}\varphi_{lj}o_{j}\\ q_{l}=f(x_{l})=x_{l}\end{array}  \tag{3}\]where \(b_{i}\) is the input, \(q_{l}\) is the output of the neural network, \(\theta_{ji}\) are hidden layer scale parameters, and \(\varphi_{lj}\) are output layer scale parameters. We take into account the activation map in the hidden layer as  
\[g(z_{j})=\frac{1}{1+e^{-z_{j}}}. \tag{4}\]  
The first and second derivatives of the sigmoid map (4) are  
\[g^{\prime}(z_{j})=g(z_{j})\Big{(}1-g(z_{j})\Big{)}, \tag{5}\] \[g^{\prime\prime}(z_{j})=g(z_{j})\Big{(}1-g(z_{j})\Big{)}\Big{(}1-2 g(z_{j})\Big{)}.\]  
We take into account the activation map of the output layer as the linear form  
\[f(x_{l})=x_{l}. \tag{6}\]  
The first and second derivatives of the linear map (6) are  
\[ \begin{array}{l}f^{\prime}(x_{l})=1,\\ f^{\prime\prime}(x_{l})=0.\end{array}  \tag{7}\]  
The first and second derivatives of the cost map (2) are  
\[ \begin{array}{l}\frac{\partial E}{\partial q_{l}}=(q_{l}-t_{l}),\\ \frac{\partial^{2}E}{\partial q_{l}^{2}}=1.\end{array}  \tag{8}\]  
Using the cost map (2) and \(f(x_{l})=x_{l}\) (6), we express the propagation of the output layer as  
\[ \begin{array}{l}\frac{\partial E}{\partial\varphi_{lj}}=\frac{\partial E}{ \partial q_{l}}\frac{\partial q_{l}}{\partial x_{l}}\frac{\partial x_{l}}{ \partial\varphi_{lj}}\\ =(q_{l}-t_{l})\frac{\partial f(x_{l})}{\partial x_{l}}o_{j}\\ =(q_{l}-t_{l})\frac{\partial f(x_{l})}{\partial x_{l}}o_{j}\\ =(q_{l}-t_{l})(1)g(z_{j}),\\ \Rightarrow\frac{\partial E}{\partial\varphi_{lj}}=(q_{l}-t_{l})g(z_{j}).\end{array}  \tag{9}\]  
Figure 1: The neural network. Using the cost map (2) and \(g^{\prime}(z_{j})=\frac{\partial\omega_{j}}{\partial z_{j}}=\frac{\partial g(z_{j}) }{\partial z_{j}}=g(z_{j})\Big{(}1-g(z_{j})\Big{)}\) (5), we express the propagation of the hidden layer as  
\[ \begin{array}{c}\frac{\partial E}{\partial\theta_{j}}=\frac{\partial E}{ \partial\theta_{il}}\frac{\partial q_{l}}{\partial x_{l}}\frac{\partial x_{l} }{\partial\omega_{j}}\frac{\partial o_{l}}{\partial z_{j}}\frac{\partial z_{j} }{\partial\theta_{\bar{j}}}\frac{\partial z_{j}}{\partial\theta_{\bar{j}}}\\ =(q_{l}-t_{l})(1)\varphi_{lj}g^{{}^{\prime}}(z_{j})b_{i},\\ \Rightarrow\frac{\partial E}{\partial\theta_{\bar{j}}}=(q_{l}-t_{l})\varphi_ {lj}g(z_{j})\Big{(}1-g(z_{j})\Big{)}b_{i}.\end{array}  \tag{10}\]  
We express the second derivative of \(E\) as the Hessian \(H\):  
\[H=\
abla\
abla E=\frac{\partial^{2}E}{\partial\theta^{2}}=\left[ \begin{array} []{cc}\frac{\partial^{2}E}{\partial\theta_{\bar{j}}^{2}}&\frac{\partial^{2}E }{\partial\theta_{j}\varphi\varphi_{lj}}\\ \frac{\partial^{2}E}{\partial\theta_{j}\varphi\varphi_{lj}}&\frac{\partial^{2 }E}{\partial\theta_{lj}^{2}}\end{array}