# A Comprehensive Analysis of Smart Grid Systems against Cyber-Physical Attacks
## 7 Resource Allocation in the Smart Grid
### Simulation Analysis  
Figure 13a-d represents the simulation results of RL algorithms that show the moving average reward against the number of episodes. We calculate the moving average reward for 300 episodes: (1) over 100,000 episodes for Q-learning for a value of \(\varepsilon\) = 0.2 in Figure 13a, \(\varepsilon\) = 0.6 in Figure 13b and \(\varepsilon\) = 0.8 in Figure 13c and (2) 300,000 episodes for SARSA learning in Figure 13d. Figure 12: Each node represents a function and its status (V: Vulnerable, H: Hacked, U: Unknown, P: Patched). (a) system admin chooses action scan-S2; (b) system admin chooses action patch-S2. S1-Smart Meter Collector, S2-Smart Meter sync head, S3-Billing Engine, S4-Outage Management System, S5-Vendor Specific Head, S6-Meter Data Management, S7-Electricity Control Center. Figure 13: Simulation results. (a) Q-learning: plot of the moving average of the 300 average rewards per episode for 100,000 trials with \(\varepsilon\)-greedy policy for exploration and exploitation \(\varepsilon\) = 0.2, learning rate \(\alpha\) = 0.2 and discount factor \(\gamma\) = 0.2; (b) Q-learning: plot of the moving average of the 300 average rewards per episode for 100,000 trials with \(\varepsilon\)-greedy policy for exploration and exploitation \(\varepsilon\) = 0.6, learning rate \(\alpha\) = 0.2 and discount factor \(\gamma\) = 0.2; (c) Q-learning: plot of the moving average of the 300 average rewards per episode for 100,000 trials with \(\varepsilon\)-greedy policy for exploration and exploitation \(\varepsilon\) = 0.8, learning rate \(\alpha\) = 0.2 and discount factor \(\gamma\) = 0.2; (d) State-Action-Reward-State-Action (SARSA)-learning: plot of the moving average of the 300 average rewards per episode for 300,000 trials, learning rate \(\alpha\) = 0.2, lambda \(\lambda\) = 0.4 and discount factor \(\gamma\) = 0.2. The Q-learning algorithm will converge if we set the value to any other constant between [0,1]. The exploration (with probability \(\varepsilon\)) diminishes over time, and the policy becomes greedy and, thus, optimal. In Figure 13a, the agent takes the optimal action with probability 0.8 (and random actions with probability \(\varepsilon\) = 0.2). Therefore, we see stable average rewards after some episodes are completed. In contrast to Figure 13a, in Figure 13b,c, the agent takes a random action with high probability (\(\varepsilon\) = 0.6 for Figure 13b and \(\varepsilon\) = 0.8 for Figure 13c) when the optimal policy has already been computed, and that is why we see variation in rewards even when optimal policy is already learned. The simulation results show the difference between the Q-learning (off-policy) and SARSA learning (on-policy) approach. The average reward per episode in SARSA learning (see Figure 13d) is more than Q-learning, which represents that SARSA learning is useful when we want to optimize the reward for the agent that is exploring the state space of the system (not exploiting the best possible action). SARSA follows the current exploration policy, which may or may not be greedy, due to which SARSA learning converges and reaches the terminal state quickly and may have a different optimal policy as compared to Q-learning. It might be possible that Q-learning takes a long time to find an optimal policy as compared to SARSA learning but it always computes the optimal policy at a particular state [41]. Once the Q-values converge, we calculate the optimal policy using Equation (5). The actions will be stored within the tool after computation, so that SA should be able to access which action to take in a particular state. Consider state: s: [S1: Unknown, S2: Unknown, S3: Patched, S4: Vulnerable, S5: Patched, S6: Unknown, S7: Vulnerable]. Using Q-learning results (Figure 13c), the best action is patch-S7 (electricity control center) with a Q-value for this state of 282.2781. Since S7 is vulnerable, it is beneficial to protect S7 first, otherwise the admin will receive a \(-500\) reward if it gets compromised.