## 3 Proposed Algorithm
The first one is Clipped Double Q-learning. TD3 alleviates the overestimation problem of Q function by estimating the value of state-action pairs using two Q functions \(Q_{\theta_{1}}\), \(Q_{\theta_{2}}\). First, to update the neural network, randomly sample a mini-batch of B transactions from the replay buffer. The main critic network is updated using the sampled mini-batch. The main critic network updates by calculating the mean square of the TD error of the current Q value (\(Q_{\theta_{i}}(s,a)\)) and the target Q value (\(y\)). This can be found through Equation (10). The target Q value can be calculated using Equation (11). \[\theta_{i}\gets argmin_{\theta_{i}}B^{-1}\sum(y-Q_{\theta_{i}}(s,a))^{2}, \tag{10}\]  
\[y\gets r+\gamma\text{min}_{i=1,2}Q_{\theta^{\prime}_{i}}(s^{\prime}, \widetilde{a}), \tag{11}\]  
Here, \(r\) represents the reward of the current state, \(\gamma\) represents the discount factor, \(s^{\prime}\) represents the next state, and \(\widetilde{a}\) is the action of \(s^{\prime}\) obtained through the target actor network. This is performed by obtaining the Q values for the next state (\(s^{\prime}\)) and the next action (\(\widetilde{a}\)) through two target critic networks, and then using a small Q value to avoid overestimation. The second is Delayed policy updates. TD3 delays policy updates so that the policy network learns based on a stable Q-function estimation. This delayed update reduces the instability of learning and maintains a balance between policy exploration and exploitation even in dynamic environments such as HSR environments. \[\
abla_{\phi}J(\phi)=B^{-1}\sum\
abla_{a}Q_{\theta_{1}}(s,a)\left|{}_{a=\pi_{ \phi}(s)}\
abla_{\phi}\pi_{\phi}(s),\right. \tag{12}\]  
Equation (12) shows that the parameter of the main actor network is updated by using the deterministic policy gradient. The actor network updates using one of the two critic networks, \(Q_{\theta_{1}}(s,a)\) being the first main critic network. \(\pi_{\phi}\) represents the main actor network. \[\left\{ \begin{array}{l}\theta^{\prime}_{i=1,2}\leftarrow\tau\theta_{i}+(1- \tau)\theta^{\prime}_{i}\\ \phi^{\prime}\leftarrow\tau\phi+(1-\tau)\phi^{\prime},\end{array} \right. \tag{13}\]  
Here, \(\phi\) is the parameter of the actor network, and \(\theta\) is the parameter of the critic network. \(\phi^{\prime}\) is the parameter of the target actor network, and \(\theta^{\prime}\) is the parameter of the target critic network. After updating the actor network through Equation (12), the target network parameters (\(\phi^{\prime}\), \(\theta^{\prime}\)) are updated through Equation (13). In the process of updating, TD3 proposed a method of updating the main actor network and the target network slower than the main critic network for stable update in the process of updating. The third is Target policy smoothing regularization. It induces changes in the action space to prevent the actor from being overly dependent on a specific action and plays a role in increasing the stability of the Target Q-value. As shown in Equation (14), TD3 adds Gaussian noise (\(\epsilon\)) to the actions of the target policy and clips it. This prevents the Q function from learning excessively rapid changes. This increases the stability of the Q-function learning and supports smooth and generalized policy learning. \[\widetilde{a}\leftarrow\pi_{\phi^{\prime}}(s^{\prime})+\epsilon,\epsilon\, \sim clip(\mathcal{N}(0,\sigma),-c,c), \tag{14}\]  
The proposed algorithm is designed based on the TD3 algorithm. The proposed algorithm can be seen in Algorithm 1. It basically follows the structure of TD3, but some structures are changed as follows to solve the requirements of the task partitioning ratio decision problem in the HSR environment with MECSs.