## 3 Multi-Layer Security Approach Based on Vulnerability Indices
### Reinforcement Learning-Based Private Defense  
The private defense as the second security layer is designed and embedded for the significant areas assessed by the vulnerability indices of the power grid. With the descriptions mentioned before, this section is dedicated to providing the details of the private defense scheme organized based on the features of the learning machine method. In recent years, sundry methods considering the learning machine theory have been proposed to detect cyber-attacks, that is, supervised learning, RL, and unsupervised learning. The recent investigations have shown that reinforcement-based detection methods are more effective and reliable schemes than the other methods due to interaction with the environment [39]. The fundamental framework of the RL method is represented in Figure 4. As can be seen in this figure, such an approach comprises of two main parts: (1) environment (2) agent. For more clarification about the detection process, note that The RL method is generally executed based on two main phases: (1) learning phase (2) detection phase. In the first phase, the agent tries to recognize the environment with the use of opting for sundry actions related to the environment observation. By doing so, the agent would receive different rewards depending on the action selection. This trend is stopped when the agent takes the maximum reward, which results in the best action. It is essential to say that the detection method function needs to be defined based on the Partially Observable Markov Decision Processes (POMDP) problem as described in [45]. In this regard, the objective function of this method is defined to get into the attack alarm concerning the minimum detection delay. To this end, the POMDP concept-based relevant scheme to detect attack is defined in accordance with Figure 5. Suppose that the attacker trends to attack the system considering the unknown method at time \(\kappa\). Regarding the unknown attack strategy, the relevant states are defined by the \"pre-attack\", \"post-attack\" and \"final\" states. As it can be seen in Figure 5, to transfer the agent from the \"pre-attack\", \"post-attack\" states to \"final\" states, there are two permissible actions including \"continue\", and \"stop\". Hence, the attack alarm occurs when the agent takes a \"final\" state for the sake of opting for the \"stop\" action. If so, the agent would keep the \"pre-attack\" state using the selection of the \"continue\" action. As mentioned before, the agent will take the  
Figure 4: The reinforcement learning approach. Figure 3: A block construction in blockchain. rewards 1 and 0 regarding the relevant action. On the other hand, the reward b is defined as the \"continue\" action selected by the agent in the \"post-attack\" state. Keeping this in mind, the fitting function of the agent is developed to decline the summation of the penalty coefficients arising from action choice as shown in Equation (38). \[\text{min}^{pen}=E^{x}\Bigg{[}(re_{t}|t_{s}<T_{t})+\sum_{t=T}^{\infty}b|t_{s}>T_ {t}\Bigg{]} \tag{38}\]  
Wherein the objection function is dependent on the stopping time (\(t_{s}\)) and the expected value of the penalty coefficient (\(P^{pen}\)). According to the attack time and the received reward, the objective function is defined in terms of \(t_{s}<T\) and \(t_{s}>T\). For more clarification, two underlying phases related to this approach can be seen in Algorithms 1 and 2. Algorithms 1 describes the trend of obtaining random action-observation pairs in a pseudo-code manner. Based on the learning phase in the random action-observation pair section, there is a need to first describe the arbitrary action-observation pair (\(P\left(o,a\right)\)) named the action value. Based on the learning phase, there is a need to first describe the arbitrary action-observation pair (\(P\left(o,a\right)\)) named the action value. As a next stage, the arbitrary action and observation are necessary to define in the \"_pre-attack_\" state (\(U\)) at time 1.