## 3 Proposed Intelligent Weighting Factors Design Approach
### Proposed Weighting Factor Tuning Scheme Based on DRL
#### 3.2.4 Problem Formulation  
Based on what precedes, we propose a TD3-based method to train an RL agent to learn the best policy for tuning the weighting factors of the decision function (11). After training, the resulting actor network is then used in real-time to provide the weighting factors \(\lambda_{i}\) and \(\lambda_{f}\) in the control strategy of Figure 3.  
To be able to apply the TD3 algorithm, some elements presented in Section 3.2.1 have to be reformulated in terms of the control objective of the paper. We mentioned that the goal of the RL agent's policy is to provide the weighting factors to the predictive controller of Figure 3. It follows naturally that the action space is then \(\mathcal{A}=[\underline{\lambda}_{i},\overline{\lambda}_{i}]\times[ \underline{\lambda}_{f},\overline{\lambda}_{f}]\) where \(\underline{\lambda}_{i}\), \(\overline{\lambda}_{i}\), \(\underline{\lambda}_{f}\), and \(\overline{\lambda}_{f}\) are as in Section 3.1, with the actions being \(\lambda_{i}\) and \(\lambda_{f}\). The environment on which the RL agent acts is the structure Figure 3, except for the weighting factors, as they are defined to be the action of the RL agent. Finally, to describe the environment, a set of states has to be chosen. Numerous possibilities arise from the structure of Figure 3. However, as for the inputs of the MOGA-based neural network of Figure 4, the environment's states were chosen by trial and error, resulting in  
\[s(k)= \begin{bmatrix}\tau_{a}(k)&\Delta i_{a}(k)&\int\Delta i_{a}(k)&f_{\mathrm{ sw}}(k)&f_{\mathrm{sw}}^{*}(k)&\Delta f_{\mathrm{sw}(k)}\end{bmatrix} ^{\top}\] \begin{table}
\begin{tabular}{l c c c c} \hline \hline
**Algorithm** & **Stability** & **Sample Efficiency** & **Computational Cost** & **Control Performance** \\ \hline DDPG & Low & Moderate & Low & Moderate \\ SAC & High & High & High & High \\ TD3 & High & Moderate & Moderate & High \\ \hline \hline \end{tabular}
\end{table} 
Table 4: Qualitative comparison of RL algorithms for weighting factor tuning in MPC [37; 38]. where \(\tau_{a}(k)\) is the instantaneous total harmonic distortion of the SPMSM's phase \(a\) current \(i_{a}\), \(\Delta i_{a}(k)=i_{a}^{*}(k)-i_{a}(k)\), with \(i_{a}^{*}(k)\) the phase \(a\) current reference obtained from \(i_{a}^{*}(k)\) and \(i_{q}^{*}(k)\) using the transforms (2) and (3), and \(\Delta f_{\text{sw}}(k)=f_{\text{sw}}^{*}(k)-f_{\text{sw}}(k)\), with \(f_{\text{sw}}^{*}(k)\) the switching frequency reference for the VSI at time \(k\) and \(f_{\text{sw}}(k)\) the switching frequency of the VSI at time \(k\). We see in the state vector the appearance of the switching frequency \(f_{\text{sw}}\), its reference \(f_{\text{sw}}^{*}\), and its error \(\Delta f_{\text{sw}}\). This information might seem redundant as the error is directly obtained from the values of the switching frequency and its reference. However, the fact that the agent performs better by using the three input features at the same time might indicate that it is unable to infer their relationship by itself. In addition, we see that the set of states, that is, the set of inputs to the neural network approximating the agent's policy, is not the same as the one considered in Section 3.1. This is due to the trial and error performed to select the set of inputs for both methods and the learning strategies (supervised learning in Section 3.1 and DRL here), which are relatively sensitive to the information they receive in their input layer. However, we can see that they are related:  
* The number of commutations of the VSI's legs is directly linked to the switching frequency;
* The information contained in the \(q\)-axis current and its error is contained, at least partly, in the SPMSM's phase \(a\) current, its error, and its harmonic distortion. It therefore makes sense to compare the method of Section 3.1 and the proposed one, as the chosen set of inputs is the one giving the best performance for each strategy.