# Distribution System State Estimation Using Model-Optimized Neural Networks
## 3 Neural Network-Based State Estimation
### Neural Network Model Optimization  
The key proposal of this paper is the use of model optimization techniques to reduce the model's size. Proposed as early as 1990 by [33], the concept of neural network pruning is removing weights that are not responsive to back-propagation as [33] mentions. As discussed in Section 3, back-propagation is in charge of updating the weight values given  
Figure 4: The structure of the LSTM-based state estimators. The AC subgrid neural network takes in 52 measurements for 5 time steps and returns 68 values, 2 for each bus corresponding to bus voltage magnitude and angle. The DC subgrid neural network takes in 12 measurements for the same time steps and returns 9 values, one for each bus voltage magnitude. Figure 3: The visual representation of the LSTM-based state estimator. The historical sequence input of measurement values (circles) to the LSTM from \(t-4\) to \(t\) will be considered in estimating the unknown (X-marks) value at the current time step, \(t\). the gradient of the error with the weights. If the responsiveness of the weights to back-propagation--or in this paper's terms, the gradient--is negligible, then it is plausible to conclude that the weight (neural connection) is not needed. These days, not only the gradient is evaluated but the value itself is also checked, that is, if the weight is zero (or plays around zero), that connection is also removed. The work by [37] has only recently proposed a standard way to benchmark gains received from pruning. Popular machine learning frameworks such as TensorFlow and pyTorch have also only recently released their pruning libraries at nearly the same time as [37] was published. There are a few general trends between pruning methods as [37] points out: Structure, Scoring, Scheduling, and Fine-tuning. Each of these differences can exploit advantages but have their own intrinsic disadvantages too. An example would be networks whose structure has been pruned, which is largely removing weight connections as shown in Figure 5. Since the parameters are exactly zero, deploying these models in small-scale devices such as micro-controllers and remote terminal units is largely plausible due to the advantage they obtain when their model files are being compressed. Compression algorithms work better when there are true zeros in the data, more so if the file is sparse in general. However, speed up features such as those in CUDA-enabled hardware may or may not be available to these models as the APIs themselves may or may not be tailored to handle random sparsity. Scheduling, on the other hand, prunes the network bit by bit at every step of the training process or immediately to target sparsity. An example of a pruning schedule is shown in Figure 6. It is also possible to be creative with the training process and start the network with all the weights active to help the network reach near optimum point before pruning. Once there, pruning can proceed as scheduled to save space and to focus on updating the responsive weights, which leads to this paper's main proposal: the use of scheduled weight pruning to optimize the neural network-based power system state estimator. The pruning schedule used is Polynomial Decay that starts from 0% sparsity at training step 0 and ends with the target ( 25%, 50%, 75%) sparsity. The rate at which the weights are pruned follows the default power setting, which is a cubic ramp-up function as stated in the TensorFlow-Model Optimization Toolkit documentation. It is important to remember here that the TensorFlow way of weight pruning does not immediately cull the weights, rather, a masking layer is created alongside the neural network. The mask layer is Boolean, 1 for an active weight and 0 for an inactive (or pruned) weight. Thus, it is important at the end of the process to strip the mask layer using the strip pruning function such that when the model is saved, the active and inactive weights are committed. Figure 5: Visual representation of neural network weight pruning.