## 3 System Model
### DRL Module
Accordingly, the target \(Q\)-value \(y_{t}\) can be expressed as follows:  
\[y_{t}=r_{t}+\gamma Q^{\prime}(s_{t+1},\mu^{\prime}(s_{t+1})). \tag{4}\]  
The \(Q\)-network is subsequently trained using the mean squared error (MSE) loss to minimize the discrepancy between the current estimate \(Q(s_{t},a_{t})\) and the previously defined target value \(y_{t}\), as expressed by the following objective:  
\[L(\phi)=\mathbb{E}\Big{[}(Q(s_{t},a_{t};\phi)-y_{t})^{2}\Big{]}. \tag{5}\]  
Finally, for policy optimization, the gradient of the deterministic policy \(\mu\) is updated indirectly via the \(Q\)-network by applying the chain rule, as formulated below:  
\[\
abla_{\theta}\!\!\!/(\theta)=\mathbb{E}\Big{[}\
abla_{a}Q(s,a;\phi)\big{|}_ {a=\mu(s)}\
abla_{\theta}\!\!\!/(s;\theta)\Big{]}. \tag{6}\]  
In contrast, TD3 improves upon DDPG by introducing two critic networks and employing a minimum \(Q\)-value selection strategy for the target critics to mitigate overestimation within the critic module. \begin{table}
\begin{tabular}{l c l} \hline \hline
**DRL Algorithm** & **Experience Reusability** & **Key Characteristics** \\ \hline DDPG & Off-Policy & Deterministic Actor–Critic Architecture \\ TD3 & Off-Policy & Twin \(Q\)-Networks, Delayed Policy Updates \\ SAC & Off-Policy & Maximum Entropy Framework with Soft \\  & & \(Q\)-Function \\ TQC & Off-Policy & Distributional RL, Quantile-Based \(Q\)-Value \\ CrossQ & Off-Policy & Estimation \\ PPO & On-Policy & Policy Update Clipping for Stability \\ \hline \hline \end{tabular}
\end{table} 
Table 2: Characteristics of DRL algorithms. Then, the target \(Q\)-function \(y_{t}\), which selects the minimum value between the two \(Q\)-networks \(Q_{1}\) and \(Q_{2}\), can be expressed as follows:  
\[y_{t}=r_{t}+\gamma\min_{i=1,2}Q_{i}^{\prime}(s_{t+1},\mu^{\prime}(s_{t+1})+ \epsilon),\quad\epsilon\sim\text{clip}(\mathcal{N}(0,\sigma),-c,c), \tag{7}\]  
where the exploration noise \(\epsilon\) is introduced to prevent overfitting and is defined using a value sampled from \(\mathcal{N}(0,\sigma)\) and clipped within the range \([-c,c]\). The loss function for the \(Q\)-update can be defined as follows:  
\[L(\phi_{i})=\mathbb{E}\Big{[}(Q_{i}(s_{t},a_{i};\phi_{i})-y_{t})^{2}\Big{]} \tag{8}\]  
where \(\phi_{1}\) and \(\phi_{2}\) denote the neural network weights of the two \(Q\)-networks, \(Q_{1}\) and \(Q_{2}\), respectively. Each policy is updated periodically, and for the sake of stability, the update is performed based solely on \(Q_{1}\):  
\[\
abla_{\theta}J(\theta)=\mathbb{E}\Big{[}\
abla_{a}Q_{1}(s,a;\phi_{1})\big{|} _{a=\mu(s)}\
abla_{\theta}\mu(s;\theta)\Big{]} \tag{9}\]  
SAC utilizes a stochastic policy and performs critic updates based on the smaller of the two \(Q\)-values from the critic and target critic, while also incorporating an entropy bonus into both policy and critic updates to encourage exploration. Therefore, the target \(Q\)-network in SAC can be formulated as follows by incorporating the entropy bonus:  
\[y_{t}=r_{t}+\gamma\mathbb{E}_{a^{\prime}\sim\pi}\big{[}Q(s_{t+1},a^{\prime})- \alpha\log\pi(a^{\prime}|s_{t+1})\big{]} \tag{10}\]  
Figure 4: A comparative overview of the DRL algorithms used in the experiment and their corresponding module architectures. where \(\alpha\) denotes the entropy temperature coefficient, which regulates the contribution of the entropy term. Similarly, to ensure that the predicted \(Q\)-value for the current state-action pair closely approximates the target \(y_{t}\), the critic network is optimized using the loss function \(L(\phi_{i})\), which is formulated as follows:  
\[L(\phi_{i})=\mathbb{E}\Big{[}(Q_{i}(s_{t},a_{t};\phi_{i})-y_{t})^{2}\Big{]} \tag{11}\]  
Through this formulation, the policy is updated to favor actions with higher estimated returns--based on the minimum of the two \(Q\)-values--while simultaneously promoting diversity by incorporating an entropy term.