## 3 Resource Allocation Based on a Multi-Agent Reinforcement Learning Algorithm
### The Proposed Cascaded Multi-Agent Deep Deterministic Policy Gradient  
Despite the fact that the abovementioned framework produces fair outcomes, the overestimation bias problem persists as a consequence of function approximation and the twin critic's minimization issues. The policy gradient algorithms generate overestimation bias, whereas TD3 algorithms generate underestimation bias. Subsequently, a novel Cascaded Multi-Agent Deep Deterministic Policy Gradient (CMADDPG) is proposed that utilizes the two contrasting biases to produce further precision in the estimation. The CMADDPG architecture depicted in Figure 3 is described in Algorithm 1. While a single-critic DDPG delivers overestimation and twin critics deliver underestimation, tri critics are employed in a cascaded mode to accomplish reliable predictions. Therefore, the architecture of the Cascaded Multi-Agent Deep Deterministic Policy Gradient (CMADDPG) algorithm was designed on the pinnacle of the DDPG and TD3 framework, as shown in Figure 3. In the proposed CMADDFG algorithm, the target value is updated by determining the minimum Q-value between the two critics followed by a Weighting operation over the obtained minimum value, and the Q-value of the third critic. Consequently, the existing \(Critic_{Global}\) is substituted by Cascaded Tri critics \(c=\{1,2,3\}\), and the policy gradient outcome is given by  
\[\
abla_{\theta_{i}}\text{Z}_{i}=\underbrace{\mathbb{E}_{s,a\sim\mathcal{D}}[ \
abla_{\theta_{i}}\tau_{i}(a_{i}|s_{i})\
abla_{a_{i}}\mathcal{G}_{\forall c}^ {c-gl_{c}}(s,\mathbf{a})]}_{Critic_{Global}}+\underbrace{\mathbb{E}_{s_{i},a_{i }\sim\mathcal{D}}[\
abla_{\theta_{i}}\tau_{i}(a_{i}|s_{i})\
abla_{a_{i}} \mathcal{G}_{\forall c}^{l}(s_{i},a_{i})]}_{Critic_{Local}} \tag{21}\]  
Figure 3: The CMADDPG architecture. where \((\mathsf{s},\mathsf{a})\) are the total state and action spaces. \(\mathcal{Q}_{\psi_{c}}^{c-glo_{c}}\) and \(\mathcal{Q}_{\psi_{c}}^{i}\) are the centralized action-value functions that estimates the Q-value for platoon \(i\) using the agents' states and actions as input. The primary term in (21) corresponds to the cascaded global critic, which assesses the team reward based on the states and actions of the agents. The secondary term in (21) refers to the local critic of each agent, which, in contrast to the cascaded global critic, solely considers the local state and activity of each agent to rate the performance of each agent individually. The \(Critic_{Cascaded\ Global}\) is updated as  
\[\mathfrak{AG}(\psi_{c})=\mathbb{E}_{\mathsf{s},\mathsf{a},\mathsf{r},\mathsf{ s}^{\prime}}\Big{[}(\mathcal{Q}_{\psi_{c}}^{c-glo_{c}}(\mathsf{s},\mathsf{a})-y_{c- glo})^{2}\Big{]} \tag{22}\]  
and the target value \(y_{c-glo}\) is given as  
\[y_{c-glo}=r_{c-glo}+\gamma\{\beta\min_{c=1,2}\mathcal{Q}_{\psi_{c}}^{c-glo_{c} }(\mathsf{s}^{\prime},\mathsf{a}^{\prime})+(1-\beta)\mathcal{Q}_{\psi_{3}}^{ c-glo_{3}}(\mathsf{s}^{\prime},\mathsf{a}^{\prime})\}|_{a^{\prime}_{i}=\pi^{ \prime}_{i}(s^{\prime}_{i})} \tag{23}\]  
in which the twin critics' weight is denoted by \(\beta\in\{0,1\}\) and the update of local critics is analogous to the global critics. In (23), \(\pi^{\prime}_{i}\) refers to the target policies. In the course of implementation, a \(\mathcal{Q}_{1}\) optimized single actor is utilized, and the identical target values are fixed for \(\mathcal{Q}_{1}\), \(\mathcal{Q}_{2}\), and \(\mathcal{Q}_{3}\). In every update, \(\mathcal{Q}_{c}\) where \(c=1,2\) is considered as an unbiased estimation. Similarly, \(\mathcal{Q}_{3}\) is considered to be overestimated with the DDPG bias by assuming the fact that DDPG encourages overestimation in all the scenarios.