## 3 Proposed Intelligent Weighting Factors Design Approach
### Proposed Weighting Factor Tuning Scheme Based on DRL
#### 3.2.2 Twin Delayed Deep Deterministic Policy Gradient  
As presented above, the overarching goal of RL is to find the policy \(\pi^{*}\) maximizing the objective (12). Such a policy often has an intractable mathematical expression. For this reason, we often replace it by a neural network parameterized by a vector of real parameters \(\phi\) that we denote by \(\pi_{\phi}\). One of the purposes of the RL research field is then to define algorithms allowing us to find the optimal vector of parameters \(\phi^{*}\) such that \(\pi_{\phi^{*}}\approx\pi^{*}\).  
One such algorithm is the TD3 [32]. It is an actor-critic algorithm for continuous action spaces aiming to find a deterministic policy represented by the neural network \(\pi_{\phi}\), also called the _actor_. Since the resulting policy is deterministic, we can adopt the notation of Remark 5 such that the action \(a(k)\in\mathcal{A}\) at time \(k\) is \(a(k)=\pi(s(k))\), with \(s(k)\in\mathcal{S}\) the state of the environment at time \(k\). The _critic_ is another neural network parameterized by a vector of real parameters \(\chi\) denoted by \(Q_{\chi}\), approximating the action-value function of the agent, that is the expected return when taking action \(a\in\mathcal{A}\) in state \(s\in\mathcal{S}\). Like the policy, the true action-value function \(Q^{\pi}\): \(\mathcal{S}\times\mathcal{A}\to\mathbb{R}\) is often intractable for large MDPs. To maximize the objective (12) with the TD3 algorithm, a policy iteration, which is a method of alternating between policy evaluation and policy improvement within the actor-critic framework, is used.  
In the policy evaluation step, the critic parameter vector \(\chi\) is obtained by minimizing the Bellman error  
\[J_{Q}(\chi)=\mathbb{E}_{(s,a,s^{+})\sim\mathcal{D}}\Big{[}\big{(}Q_{\chi}(s,a )-\mathrm{Y}(s,a,s^{+})\big{)}^{2}\Big{]} \tag{13}\]  
where \(\mathcal{D}\) is a replay buffer of past experiences--triplets containing a state \(s\in\mathcal{S}\) of the environment that the agent observed during its training--associated with the action \(a\in\mathcal{A}\) it performed following its policy, and the subsequent environment's state \(s^{+}\in\mathcal{S}\) - and \(\mathrm{Y}\) is the Bellman regression target defined, for any \(s,s^{+}\in\mathcal{S}\) and \(a\in\mathcal{A}\), as  
\[\mathrm{Y}(s,a,s^{+})=\rho(s,a)+\gamma Q_{\chi}(s^{+},\pi_{\phi}(s^{+})). \tag{14}\]  
In TD3, two critic networks \(Q_{\chi_{1}}\) and \(Q_{\chi_{2}}\), with \(\chi_{1}\) and \(\chi_{2}\) their respective vectors of parameters, are trained simultaneously using (13) by replacing \(\chi\) by either \(\chi_{1}\) or \(\chi_{2}\). To reduce overestimation bias, TD3 uses the minimum of the two Q-values to compute the Bellman regression target as \[\mathrm{Y}(s,a,s^{+})=\rho(s,a)+\gamma\min_{i=1,2}Q_{X_{i}}(s^{+},\pi_{\phi}(s^{+})) \tag{15}\]  
for both critics \(Q_{\chi_{1}}\) and \(Q_{\chi_{2}}\).  
In the policy improvement step, the actor parameter vector \(\phi\) is updated to maximize the expected return according to the current critic estimates, that is, to maximize  
\[J_{\pi}(\phi)=\mathbb{E}_{s\sim\mathcal{D}}[Q_{\chi_{1}}(s,\pi_{\phi}(s))]. \tag{16}\]