# MADDPG-Based Security Situational Awareness for Smart Grid with Intelligent Edge
## 5 MADDPG-Based Security Situational Awareness
### Transformation of the Problem  
By interacting with the environment many times, each edge agent accumulates a certain amount of experience in awareness of the situation elements. As a Markov process, we denote it as \((s_{k}(t),a_{k}(t),r_{k}(t),s_{k+1}(t))\). Here, \(s_{k}(t)\in\mathcal{S}\), denotes the state in which the edge agent \(k\) at time slot \(t\) is in, i.e., the observed environmental situation elements. \(a_{k}(t)\in\mathcal{A}\) indicates the action taken by the edge agent \(k\) in the current state \(s_{k}(t)\) at time slot \(t\). \(r_{k}(t)\in\mathcal{R}\) denotes the rewards received by the edge agent \(k\) at time slot \(t\) after taking action \(a_{k}(t)\in\mathcal{A}\) in the current state \(s_{k}(t)\). For each given state \(s\in\mathcal{S}\), the edge agent takes policy \(\pi:\mathcal{S}\mapsto\mathcal{A}\) to select an action from its action space to act on the current state \(s\). In the following, we provide the state space, action space, and rewards that are required by the algorithm.  
1. State space: The state space of each edge agent composes of three dimensions of situational elements. Specifically, it includes information about the external environment, the internal node vulnerability, and the possible attacks. These situational elements constitute the smart grids' environment information at each given time slot \(t\), which is taken as a state here. For a cluster of edge agents at time slot \(t\), the state space is represented as \[s(t)=(s_{1}(t),s_{2}(t),\ldots,s_{K}(t)).\] (11) For a given agent \(k\), \(k\in\{1,2,\ldots,K\}\), the state at time slot \(t\) is represented as \[s_{k}(t)=\Big{\{}O^{exPol}_{kp(t)}(t),O^{exLog}_{kk(t)}(t),O^{in}_{kn(t)}(t),O ^{At}_{kk(t)}(t),\mathcal{E}^{exPol}_{k},\mathcal{E}^{exLog}_{k},\mathcal{E}^ {in}_{k},\mathcal{E}^{At}_{k}\Big{\}}.\] (12)
2. Action space: Each edge agent chooses the action to execute from its state space in the current state by using policy \(\pi:\mathcal{S}\mapsto\mathcal{A}\). The action here is the edge agent's security-aware behavior based on the situational elements in the current environment. We define it as the edge agent's awareness of four situational elements in three dimensions. For a cluster of edge agents, the action space in time slot \(t\) is represented as \[a(t)=(a_{1}(t),a_{2}(t),\ldots,a_{K}(t)).\] (13)For each agent \(k,k\in\{1,2,\ldots,K\}\), its act of detecting anomalies at time slot \(t\) is taken as an action, denoted as \[a_{k}(t)=\Big{\{}\mathcal{A}_{\texttt{lk}}^{p(t)},\mathcal{A}_{\texttt{lk}}^{e(t) },\mathcal{A}_{\texttt{lk}}^{n(t)},\mathcal{A}_{\texttt{lk}}^{a(t)}\Big{\}}.\] (14)
3. Rewards: After each edge agent performs an action in the current state, we define \(r_{k}(t)\) as the immediate rewards given to edge agent \(k\) in the present time slot \(t\). Hence, \(r_{k}(t)\) is a state and action function that will guide edge agent \(k\) to the optimal state-action policy. Considering that SSA is to obtain the minimum processing cost with the minimum detection error rate, we design the immediate rewards according to the initially formulated problem goal. Thus, we set the immediate rewards of edge agent \(k,k\in\{1,2,\ldots,K\}\) at time slot \(t\) as \[r_{k}(t)=-\mathcal{C}(t)-\Gamma(t).\] (15) Note that \(\Gamma(t)\) is the perceived failure punishment for each edge agent at time slot \(t\), considering the loss to the smart grids from actions taken that fail to detect threats. The value of \(\Gamma(t)\) will increase if a normal situational element is detected as a threat or if an attack occurs but not be detected.