## 3 Proposed RL-Based Real-Time Control Framework
### Overall Flowchart of Training RL Agents for Power System Operation  
The main flowchart for training RL agents for real-time volt/var control and line flow control is shown in Figure 2 and consists of several key major steps:  
**Step A**: Large-scale historical operating data are collected from the SCADA system in a power grid. Today's SCADA system can typically capture full-topology information from wide-area deployed measurement sensors before sending it to the EMS to provide the best guess of the current system operating conditions in a quasi-steady state. Next, the converged state estimator snapshots are saved in full-topology model format. The sampling rate is set to 5 min in this work to ensure comprehensive operational scenarios of bulk power systems are collected. **Step B**: Control objectives and the corresponding settings are specified for training SAC agents. The control objectives can be volt/var control only, transmission loss reduction, or a combination of these two objectives. The thresholds of each control objective and every constraint also need to be specified by the power engineers or system operators for real-time application, including voltage security limit, line flow limit, etc. **Step C**: The volt/var control RL agent is trained using historical EMS snapshots and the high-fidelity simulation software. The RL agent weights are initialized and then the offline training process is entered. For each historical snapshot, i, the EMS case is loaded and AC power flow analysis is conducted. If divergence occurs, this episode is abandoned. Voltage magnitude violations and transmission losses are then calculated as reward values of the agents to reflect the health condition of the current EMS snapshot. A control loop is started by forming state space from the power flow information, obtaining the best action from the RL agent, verifying control performance using an AC power flow solver, and \begin{table}
\begin{tabular}{c c c} \hline \hline  & **Voltage Control** & **Line Flow Control** \\ \hline \multicolumn{3}{c}{**Control Objectives**} \\ \hline \multirow{3}{*}{Corrective Actions} & minimum reactive power control & minimum active power control \\  & actions: & actions: \\  & \(\min\sum\
olimits_{k}^{\textbf{k}max}c(k),k\in\mathbb{C}\) & \(\min\sum\
olimits_{k}^{\textbf{k}max}p(i)\), \(i\in\Gamma\) \\ \hline \multirow{3}{*}{Loss Minimization} & as an objective function: \\  & \(\min\sum_{i,j}P_{loss}(i,j)\), \((i,j)\in\Omega_{L}\cup\Omega_{T}\) \\ \hline \multicolumn{3}{c}{**Constraints Modeled**} \\ \hline \hline \(P_{i}^{g}-P_{i}^{d}-g_{i}V_{i}^{2}=\sum\limits_{j\in B_{i}}P_{ij}(y),i\in B\) \\  & \(Q_{i}^{g}-Q_{i}^{d}-b_{i}V_{i}^{2}=\sum\limits_{j\in B_{i}}Q_{ij}(y),i\in B\) \\  & **where**\(\textbf{y}=[\Theta\textbf{ V}]^{T}\) \\  & \(P_{i}^{g}=\sum\limits_{n\in G}P_{n}^{d},i\in B\) \\  & \(Q_{i}^{g}=\sum\limits_{n\in Gi}Q_{n}^{g},i\in B\) \\  & \(P_{i}^{d}=\sum\limits_{m\in Gi}P_{m}^{d},i\in B\) \\  & \(Q_{i}^{d}=\sum\limits_{m\in Gi}Q_{m}^{d},i\in B\) \\  & \(P_{ij}\) and \(Q_{ij}\) are active power and reactive power on branches, respectively \\ \hline \multirow{3}{*}{Generation Limits} & \(P_{m}^{min}\leq P_{n}\leq P_{m}^{max},n\in G\) \\  & \(Q_{m}^{min}\leq Q_{n}\leq Q_{m}^{max},n\in G\) \\ \hline \multicolumn{3}{c}{Voltage Limits} & \(V_{i}^{min}\leq V_{i}\leq V_{i}^{max},i\in B\) \\ \hline \multicolumn{3}{c}{Transmission Line Limits} & \(\sqrt{P_{ij}^{2}+Q_{ij}^{2}}\leq S_{ij}^{max},(i,j)\in\Omega_{L}\cup\Omega_{T}\) \\ \hline \end{tabular}
\end{table} 
Table 1: Control objectives and constraints of the proposed framework. computing the reward value. This control process iterates until a \"Done\" condition is met. This happens when the following occur:  
1. The RL agent reaches the maximum control iteration;
2. Power flow diverges;
3. The RL agent's action successfully meets the desired control performance goal. After training the current episode, the RL agent's weights in the neural networks are updated. This process continues until it exhausts all the available EMS episodes.