## 3 Proposed TD3-Based MMC-VSG
### TD3 Algorithm  
The TD3 algorithm combines reinforcement learning's capacity for decision making with deep learning's perception [25]. In order to successfully address various flaws in the DDPG algorithm, such as Q-value overestimation, overfitting, and poor updates, TD3-based DRL is applied to create an adaptive control technique.  
The TD3 algorithm also builds under the actor-critic framework as does DDPG. TD3 algorithm adopts the method in double Q-learning, which is different from DDPG, which uses two independent neural networks \(Q_{\theta 1}\) and \(Q_{\theta 2}\) with the network parameters \(\theta_{1}\) and \(\theta_{2}\) to estimate the critic (value). TD3 algorithmindependently emulates two Q-functions, and constructs the target value of critic learning with the smaller Q-value to slow down the overestimation of the critic.  
Additionally, each critic network and actor network are accompanied by a corresponding target network \(Q^{{}^{\prime}}\). The equivalent target networks \(Q^{{}^{\prime}}_{\theta 1}\), \(Q^{{}^{\prime}}_{\theta 2}\), and \(n^{{}^{\prime}}_{\phi}\) correspond to the actor neural network \(\pi_{\phi}\) and the two critic neural networks \(Q_{\theta 1}\) and \(Q_{\theta 2}\). The objective of the actor network is to pick up the actions depending on the input variables, while the  
Figure 8: The framework of DRL.  
critic network is trained to rate the gradient message and equivalent actions given by the actor network, which helps the actor network find a better score.  
To implement the TD3 algorithm, first, the target actor network is used to compute action \(a^{{}^{\prime}}\) under the state \(S^{{}^{\prime}}\).  
\[a^{\prime}=\pi_{\phi^{\prime}}\big{(}s^{\prime}\big{)} \tag{29}\]  
Then, adding noise to the target action \(a^{{}^{\prime}}\) is based on the target policy smoothing regularization  
\[\tilde{a}=\pi^{\prime}_{\phi}\big{(}s^{\prime}\big{)}+\varepsilon=a^{\prime}+\varepsilon \tag{30}\]  
where \(\varepsilon\sim clip(N(0,\widetilde{\sigma}),-c,c)\) is a clipped noise. The loss function is minimized by Equation (31) with the renovation of two critic neural networks  
\[L\big{(}\theta^{Q_{\theta i}}\big{)}=\mathbb{E}\Big{[}\Big{(}y-Q_{\theta i}(s,a)^{2}\Big{)}\Big{]} \tag{31}\]  
where the iterative target value \(y\) can be derived from the target neural networks to comply with the Bellman equations  
\[y=r(s,a)+\gamma\min_{i=1,2}Q_{\theta^{\prime}_{i}}\big{(}s^{\prime},\tilde{a }\big{)} \tag{32}\]  
Using the deterministic policy gradient (DPG) algorithm [26], action network parameters can be updated as follows  
\[\
abla_{\phi}J(\phi)=\,\mathrm{N}^{-1}\sum\
abla_{a}Q_{\theta_{1}}(s,a)\Big{|} _{a=\pi_{\phi}(s)}\
abla_{\phi}\pi_{\phi}(s) \tag{33}\]  
The target network and delayed policy update methods are adopted to ensure the training process' validity and stability. The policy network update is delayed due to the actor-critic method's slow parameter update. First, postponing updates might cut down on needless updates that are repeated. However, it is also possible to lessen the faults that gather after numerous upgrades. A soft updating mechanism should also be employed while decreasing the update frequency, which can be expressed as  
\[\begin{split}\theta^{\prime}_{i}&\leftarrow \tau\theta_{i}+(1-\tau)\theta^{\prime}_{i}\\ \phi^{\prime}&\leftarrow\tau\phi+(1-\tau)\phi^{ \prime}\end{split} \tag{34}\]  
where \(\theta^{\prime}_{i}\) is the parameter associated with the target critic network, \(\phi^{{}^{\prime}}\) is for the target actor network, and (\(\gamma\in[0,1]\) ) is the rate of update.