# Distribution System State Estimation Using Model-Optimized Neural Networks
## 3 Neural Network-Based State Estimation
### Neural Network Theory  
On the surface, the output of a neural network can be treated as a very long linear equation as shown by Equation (5) for an \(N_{L}\)-layer, \(N_{x}\)-input, \(N_{y}\)-output fully connected neural network as shown in [33; 34; 35], where the \(j\)th neuron in the \(r\)th layer is the sum of the \(i\) weighted and connected neurons in the \((r-1)\) layer and the \(r\)th layer's bias vector--all of which are subject to the activation function, \(h\). An important thing to note with the activation function, \(h\), is that it squeezes the numbers to digestible ranges with different behaviors depending on the choice. Usual choices of activation functions include sigmoid, hyperbolic tangent, and the rectified linear unit. In the case of dimensions, the 2 nd layer's neurons take from the \(N_{x}\) weighted input vector, \(\mathbf{x}\), and the \(N_{L}\)th layer gives the \(N_{y}\) output vector, \(\mathbf{y}\). Equation (6) shows this relationship in matrix form with \(n_{r}\) and \(n_{(r-1)}\) representing the number of neurons of the \(r\)th and \((r-1)\)th layer and \(H\) representing the activation function applied to each row. This is the equation generally followed during testing and actual implementation, called the feed-forward path. \[a_{j}^{(r)}=h\left(\sum_{i}W_{j,i}^{(r-r-1)}\cdot a_{i}^{(r-1)}+b_{j}^{(r)}\right) \tag{5}\]  
\[\left[ \begin{array}{c}a_{1}\\ a_{2}\\ \vdots\\ a_{n}^{(r)}\end{array} \right]^{(r)}=H\left(\left[ \begin{array}{cccc}W_{1,1}&W _{1,2}&\ldots&W_{1,n^{(r-1)}}\\ W_{2,1}&W_{2,2}&\ldots&W_{2,n^{(r-1)}}\\ \vdots&\vdots&\ddots&\vdots\\ W_{n^{(r)},1}&W_{n^{(r)},2}&\ldots&W_{n^{(r)},n^{(r-1)}}\end{array} \right]^{( r-r-1)}\left[ \begin{array}{c}a_{1}\\ a_{2}\\ \vdots\\ a_{n}^{(r-1)}\end{array} \right]^{(r-1)}+\left[ \begin{array}{c}b_{1}\\ b_{2}\\ \vdots\\ b_{n}^{(r)}\end{array} \right]^{(r)}\right) \tag{6}\]  
During the process of training, however, there is an additional step to reliably tune the weights to its proper values, called the back-propagation. Intuitively, this step takes the gradient between the error and the individual weights as shown in [33; 34; 35] and uses it to update the weight values in order to minimize the error. The loss function, which is the \"modeled difference\" between the predicted output, \(\hat{y}_{i}\), and the ground truth, \(y_{i}\), is at the forefront of this step. One of the most common loss functions is the mean square error (MSE), which is shown in Equation (7). \[C=L(y_{i},\hat{y}_{i})\to C=\frac{1}{N}\sum_{i}^{N}(\hat{y}_{i}-y_{i})^{2} \tag{7}\]  
The challenge is finding the gradient of the cost function with respect to the weights. Shown in the succeeding equations below is the derivation to achieve the desired form. To obtain \(\frac{\partial C}{\partial W}\), remember from Equation (7) that \(C\) is dependent on \(\hat{y}_{i}\) and \(y_{i}\), and by chain, \(\hat{y}_{i}\) is dependent on the sum of the weighted neuron values, seen in Equation (5) (when \(r\) is the last/output layer, \(a_{i}^{(r)}=\hat{y}_{i}\)). Performing the simplified differentiation and applying chain rule, we obtain Equation (8). Figure 2: A neural network with two sets of two-unit fully connected hidden layers. The input and output layers are \([x_{1},x_{2}]\) and \([y_{1},y_{2}]\). \[\frac{\partial C}{\partial W_{i,j}^{(r)}}=\frac{\partial C}{\partial a_{i}^{(r)}} \cdot\frac{a_{i}^{(r)}}{W_{i,j}^{(r)}} \tag{8}\]  
\[a_{i}^{(r)}=h\left(\sum_{j}W_{i,j}^{(r)}\cdot a_{j}^{(r-1)}+b_{j}^{(r)}\right) \rightarrow\frac{\partial a_{i}^{(r)}}{\partial W_{i,j}^{(r)}}=a_{j}^{(r-1)} \tag{9}\]  
Now, Equation (9) recognizes that the weighted neuron values, \(a_{i}^{(r)}\), are dependent also on their own sum of weighted neuron values. Therefore, it will propagate back up to the first layer, as implied by Equation (10).