# MADDPG-Based Security Situational Awareness for Smart Grid with Intelligent Edge
## 6 Performance Evaluation
### Performance Evaluation  
According to the parameter settings given in Section 6.1, we have compared the total rewards values obtained by the MADDPG-based and the single-agent DDPG-based algorithms as follows. The single-agent DDPG-based here means that only one edge agent gathers all the power terminals' situational elements. Then, DDPG is performed for awareness training. Similar to MADDPG, the rewards of DDPG are also set to the penalty for awareness failure coupled with the minimum processing cost. Further, we set the situational elements' detection error rate all to 0.01 and then iterate the proposed MADDPG-based and DDPG-based algorithm for 50,000 episodes, respectively. Figure 4 shows the total episode rewards obtained by the 3-agent MADDPG algorithm and the single-agent DDPG algorithm over 50,000 episodes of the same initial situational awareness environment, respectively. In each episode, the total episode rewards obtained by both algorithms are different. In the early training phase, the total episode rewards of both algorithms fluctuate drastically due to the agent's exploration of action strategies. As the training episode grows, the reward values gradually stabilize. After reaching the 5000 episode, both achieve convergence. While the total episode rewards obtained by the MADDPG-based algorithm are clustered around \(-100\) to \(0\), the single-agent-based one is around \(-200\) to \(0\). It illustrates that the total episode rewards of the MADDPG-based algorithm proposed in this paper are preferred over the single-agent DDPG algorithm after training. On the other hand, for the training process, we take the mean rewards for every 1000 episodes of the training. Figure 5 shows the mean rewards received by a single edge agent with MADDPG-based and DDPG-based algorithm, respectively. Note that the MADDPG-based algorithm shows the mean rewards for one edge agent, and the DDPG-based algorithm shows the mean rewards for the edge agent executing the training mission divided by the value of the number of edge agents 3. It is observed that the rewards under the proposed MADDPG-based algorithm for a single edge agent are always greater than those of the DDPG-based algorithm. This indicates that the mean processing cost of the proposed multi-agent SSA model is lower than that of the single-agent DDPG-based. Further, the comparison of the mean rewards after convergence also verified this view. Moreover, we verify the effect of the detection error rate of the situational elements on the algorithm's performance. The proposed MADDPG-based algorithm's rewards are the minimum processing cost, which is obtained under the condition that the detection error rate is less than expected. Under the WASA model, we evaluate the algorithm's performance by controlling the detection error rate of external environment, vulnerable internal nodes, and the possible attacks. Figure 6 shows the impact of four threat detection error rates on the proposed MADDPG-based algorithm's rewards. Note that, the rewards value here is the system's total rewards after convergence of the training model under different error rate requirements, including the sum of the rewards of all edge agents. It can be seen that with a comparatively high error rate, the edge agents frequently make \begin{table}
\begin{tabular}{c c} \hline \hline
**Parameter** & **Value** \\ \hline Layers of Critic network & 5 \\ Layer type of Critic & Fully connected \\ Neurons of hidden layers for critic networks & [1024,512,256] \\ Learning rate of Critic & 0.0001 \\ Layers of Actor network & 4 \\ Layer Type of Actor & Fully connected \\ Neurons of hidden layers for Actor networks & [512,128] \\ Learning rate of Actor & 0.0001 \\ Activation Function & Relu \\ Mini-batch & 256 \\ Buffer Size & 20,000 \\ Discount factor & 0.95 \\ \hline \hline \end{tabular}
\end{table} 
Table 2: The neural network and training parameters settings. wrong action strategies to be aware of the situational elements, and the total rewards of the proposed algorithm are relatively low.