# MADDPG-Based Security Situational Awareness for Smart Grid with Intelligent Edge
## 5 MADDPG-Based Security Situational Awareness
### Algorithm Design  
Algorithm 1 is the pseudo-code of our given MADDPG algorithm for the SSASGEC problem. The details of the proposed MADDPG algorithm are described as follows.  
```
1:the actor's evaluation and critic's target networks for each edge agent
2:for episode = 1 to \(M\)do
3: Initialize a random process \(\mathcal{G}\) for exploration of action
4: Receive initial state \(\mathbf{s}\)
5:for\(t=1\) to \(\mathbb{N}\)do
6: for each edge agent \(k\), select action \(a_{k}(t)=\mu_{\theta_{k}}(o_{k})+\mathcal{G}_{t}\) from the action space, w.r.t. the current policy and exploration
7: Execute actions \(a(t)=(a_{1}(t),a_{2}(t),\ldots,a_{K}(t))\), then observe rewards \(r\) and the new state \(\mathbf{s}^{\prime}\)
8: Store \((\mathbf{s},a,r,\mathbf{s}^{\prime})\) in replay buffer \(\mathcal{D}\)
9:\(\mathbf{s}\leftarrow\mathbf{s}^{\prime}\)
10:for agent \(k=1\) to \(K\)do
11: Sample a random minibatch of \(\mathcal{X}\) samples \((\mathbf{s}^{i},a^{i},r^{i},\mathbf{s}^{i})\) from \(\mathcal{D}\)
12: Set \(y^{i}=r_{k}^{i}+\gamma\mathcal{Q}_{k}^{i^{\prime}}(\mathbf{s}^{\prime i},a_{1 }^{i},\cdots,a_{K}^{i})\big{|}_{a_{i}=r_{k}^{i}(o_{k})}\)
13: Update the parameter of critic's evaluation network by minimizing the loss \(L(\theta_{k})=\frac{1}{\lambda}\sum\limits_{i}\big{(}y^{i}-Q_{k}^{i^{\prime}} (\mathbf{s}^{i},a_{1}^{i},\cdots,a_{K}^{i})\big{)}^{2}\)
14: Update the parameter of actor's evaluation network by maximizing the policy gradient \(\
abla_{\theta_{k}}\mathcal{J}\approx\frac{1}{\lambda}\sum\limits_{i}\
abla_{ \theta_{k}}\mu_{k}(o_{j}^{i})\
abla_{a_{k}}\mathcal{Q}_{k}^{i}(\mathbf{s}^{i}, a_{1}^{i},\cdots,a_{K},\cdots,a_{K}^{i})\big{|}_{a_{k}=\mu_{k}(o_{k}^{i})}\)
15:endfor
16: Update target network parameters for each agent \(k\): \(\theta_{k}^{\prime}\leftarrow\tau\theta_{k}+(1-\tau\theta_{k}^{\prime})\)
17:endfor
```  
**Algorithm 1**Multi-agent deep deterministic policy gradient algorithm for the SSASGEC problem.  
First, the actor and critic networks are initialized for each edge agent. At the beginning of each episode, the detection noise \(\mathcal{G}\) is initialized, and the initial state \(\mathbf{s}\) is acquired (lines 1-3). Then the iterations are carried out, and each edge agent chooses the action to perform based on its policy and the noise (lines 4-5). After each edge agent performs an action, the total rewards \(r\) is observed and a new state \(\mathbf{s}^{{}^{\prime}}\) is generated (line 6). The current state \(\mathbf{s}\), the performed action \(a\), the rewards \(r\), and the new state \(\mathbf{s}^{{}^{\prime}}\) are stored in the experience replay buffer \(\mathcal{D}\). In addition, the new state \(\mathbf{s}^{{}^{\prime}}\) is used as the state at the beginning of the next iteration (lines 7-8).  
After executing a full episode, each edge agent \(k\) randomly selects a small sample from the experience replay buffer \(\mathcal{D}\) among \(\mathcal{X}\). Here, the samples are approximations of the agents other than agent \(k\) (line 10). To set the target value of Q-function into \(y^{i}\) (line 11). Then, the parameters \(\theta\) of the critic-network are updated by minimizing the distance between \(y^{i}\) and \(Q_{k}^{y^{i}}(\mathbf{s}^{i},a_{1}^{i},\cdots,a_{K}^{i})\) for the selected samples among \(\mathcal{X}\). In the same way, the update of the policy parameter \(\theta\) of the actor-network is derived by maximizing the gradient ascent (lines 12-13). After each edge server agent updates the actor and critic networks, the target-network parameter \(\theta_{k}^{i}\) is updated (line 15). Such an update method is designed to achieve learning stability by limiting the target values' update rate.