## 3 Proposed Intelligent Weighting Factors Design Approach
### Proposed Weighting Factor Tuning Scheme Based on DRL
#### 3.2.5 Learning Scheme  
The proposed approach to train the RL agent with the TD3 algorithm is summarized in Algorithm 1. The algorithm's inputs are three vectors of real parameters, \(\chi_{1}\), \(\chi_{2}\), and \(\phi\), used to parameterize the two critics \(Q_{\chi_{1}}\) and \(Q_{\chi_{2}}\) and the actor \(\pi_{\phi}\), the discount factor \(\gamma\), and the structure of Figure 3 using the parameters from Table 3 for the environment's simulation. We use two critic networks as presented in Section 3.2.2 to take advantage of the clipped double-Q learning trick [32], allowing us to reduce the overestimation bias by using the minimum of the two Q-values when updating the policy. From these initial parameters, we initialize two vectors \(\bar{\chi}_{1}\) and \(\bar{\chi}_{2}\) and use them to parameterize two target critic networks \(Q_{\chi_{1}}\) and \(Q_{\chi_{2}}\). To stabilize the learning process, these target networks are updated slowly as a weighted average of the critic and target critic parameters \(\chi_{i}\) and \(\bar{\chi}_{i}\), with \(i\in\{1,2\}\), through a hyperparameter \(\zeta\in(0,0.5)\) as shown on line 20 of Algorithm 1. We finally initialize a set \(\mathcal{D}\), the replay buffer, that is used to store past experiences. After training, the algorithm outputs are \(\chi_{1}^{*}\), \(\chi_{2}^{*}\), and \(\phi^{*}\), the optimized values of the parameter vectors \(\chi_{1}\), \(\chi_{2}\), and \(\phi\). Figure 6: Schematic representation of the interactions between the RL agent and its environment. ```
Input :\(\chi_{1},\chi_{2},\phi,\gamma\), the FCS-MPCC controlled SPMSM of Figure 3 (the environment) Output :\(\chi_{1}^{*},\chi_{2}^{*},\phi^{*}\)
1 Initialization:
2\(\tilde{\chi}_{1}\leftarrow\chi_{1},\tilde{\chi}_{2}\leftarrow\chi_{2}\)// Target networks' parameters' initialization \(\mathcal{D}\leftarrow\emptyset\)// Replay buffer initialization
3 end for
4 for\(n\gets 1\)to\(N_{\text{episode}}\)do
5 Initialize the environment with parameters from Table 3 and initial conditions for\(k\gets 0\)to\(N_{\text{steps}}\)do
6 Select the action to apply to the environment \(a(k)=\pi_{\phi}(s(k))\)
7 Apply \(a(k)\) to the FCS-MPCC
8 Find the gating pulses \(S_{1}(k)\), \(S_{3}(k)\), and \(S_{5}(k)\) to apply to the VSI with the FCS-MPCC procedure described in Section 2.3
9 Apply the gating pulses to the VSI to make the simulated system evolve one step and retrieve \(s(k+1)\)
10 Compute \(\rho(s(k),a(k))\) with (22) \(\mathcal{D}\leftarrow\mathcal{D}\cup\{(s(k),a(k),\rho(s(k),a(k)),s(k+1))\}\)if\(\text{Card}(\mathcal{D})\geq\text{Card}(\mathcal{B})\)then
11 Sample mini-batch \(\mathcal{B}\) of 4-tuples \((\zeta,\eta,\varrho,\varsigma^{+})\) from replay buffer \(\mathcal{D}\) \(\text{Y}(k)=\mathbb{E}_{(\zeta,\eta,\zeta^{+})\sim\mathcal{B}}\bigg{[}\rho( \zeta,\eta)+\gamma\min_{i\in\{1,2\}}Q_{\mathcal{R}_{i}}(\varsigma^{+},\pi_{ \phi}(\varsigma^{+}))\bigg{]}\)for\(i\gets 1\)to\(2\)do
12\(J_{\mathcal{Q}}(\chi_{i})=\mathbb{E}_{\zeta\sim\mathcal{B}}\Big{[}\big{(}Q_{ \mathcal{R}_{i}}(\varsigma)-\text{Y}(k)\big{)}^{2}\Big{]}\)// Compute critic loss \(\chi_{i}\leftarrow\chi_{i}-\alpha_{\text{tr}}\
abla_{\chi_{i}}J_{\mathcal{Q} }(\chi_{i})\)// Update critic weights \(\tilde{\chi}_{i}\leftarrow\zeta\tilde{\chi}_{i}+(1-\zeta)\tilde{\chi}_{i}\)// Soft-update target networks
13 end for
14\(J_{\pi}(\phi)=\mathbb{E}_{\varsigma\sim\mathcal{B}}\bigg{[}\min_{i\in\{1,2\}} Q_{\mathcal{R}_{i}}(\varsigma,\pi_{\phi}(\varsigma))\bigg{]}\)// Compute actor loss \(\phi\leftarrow\phi-\alpha_{\text{tr}}\
abla_{\phi}J_{\pi}(\phi)\)// Update actor weights
15 end for
16
17 end for
```  
**Algorithm 1**TD3 training algorithm. The learning scheme runs in an episodic fashion for a given number \(N_{\text{episode}}\) of episodes of \(N_{\text{steps}}\) time steps each.