## 3 Resource Allocation Based on a Multi-Agent Reinforcement Learning Algorithm
### Conventional Multi-Agent Deep Deterministic Policy Gradient  
The key concepts of MADDPG is illustrated in Figure 2, Based on the MADDPG principle, two networks are present in every agent: the actor network \(\pi\) and the critic network \(Q\). The actor network is responsible for computing the executed actions depending on the states obtained by the agent, whereas the critic network is responsible for estimating the action computed by the actor network in an effort to enhance the actor network's efficiency. To eliminate the redundancies in the training data and improve the training stability, a particular portion of experiences obtained from training are stored in a buffer called the Experience Replay buffer, denoted by \(\mathfrak{D}\), from which the critic network randomly selects while updating the network. During the training phase, the actor network solely learns from its own observations; concurrently, the critic network learns from data, which include the actions and observations of additional agents. During the execution phase, the agent mandates the participation of the actor network alone, hence signifying the decentralized execution. Let us consider \(\Theta_{\pi}=\left(\omega_{\pi}^{(1)}\dots\omega_{\pi}^{(L_{\pi})}\right)\) and \(\Phi_{Q}=\left(\omega_{Q}^{(1)}\dots\omega_{Q}^{(L_{Q})}\right)\) as the agent's space parameters of the actor network and critic network. Let the global critic's space parameter be \(\Psi_{glo}=\left(\omega_{glo}^{(1)}\dots\omega_{glo}^{(L_{glo})}\right)\), in which the quantity of hidden layers in the agent's actor, critic, and global critic is given as \(L_{\pi},L_{Q},L_{glo}\), respectively, with \(\omega\) being the weights of the neural network. The agent's policies for the Vehicular platooning scenario comprising \(P\) platoons as agents are given as \(\pi=\pi_{1},\dots,\pi_{P}\). The Q-functions, global critic's Q-function, and the agent's policies are given as \(\mathcal{Q}_{\hat{\varphi}}^{i},\mathcal{Q}_{\hat{\varphi}}^{glo},\pi_{i}\), and characterized as \(\phi_{i},\psi,\theta_{i}\), respectively, in which \(\phi_{i}\in\Phi_{Q},\psi\in\Psi_{glo}\), and \(\theta_{i}\in\Theta_{\pi}\). The MADDPG for the platoon \(i\) is defined as  
\[\
abla_{\theta_{i}}\mathrm{Z}_{i}=\mathbb{E}[\
abla_{\theta_{i}}\pi_{i}(a_{i }|s_{i})\
abla_{a_{i}}\mathcal{Q}_{\hat{\varphi}}^{\pi}(s,a)|_{a_{i}=\pi_{i}(s _{i})}], \tag{15}\]  
Figure 2: The MADDPG framework. where the absolute state and the absolute action spaces are given by \(\mathsf{s}=(s_{1},s_{2}\ldots,s_{P})\) and \(\mathsf{a}=(a_{1},a_{2},\ldots,a_{P})\), respectively, and the centralized action-value function is given by \(\mathcal{Q}_{i}^{n}(\mathsf{s},\mathsf{a})\). The Q-value for platoon \(i\) is determined by feeding the states and actions of the agent to the centralized function. Formulated from the architecture illustrated in Figure 3, the general Multi-Agent Deep Deterministic Policy Gradient for an individual agent with twin critics is given as  
\[\
abla_{\theta_{i}}\mathsf{Z}_{i}=\underbrace{\mathbb{E}_{\mathsf{s},a \sim\mathcal{D}}[\
abla_{\theta_{i}}\pi_{i}(a_{i}|s_{i})\
abla_{a_{i}} \mathcal{Q}_{\psi}^{\text{$\beta$lo}}(\mathsf{s},\mathsf{a})]}_{Critic_{Global}}+ \underbrace{\mathbb{E}_{\mathsf{s},a\sim\mathcal{D}}[\
abla_{\theta_{i}}\pi_{ i}(a_{i}|s_{i})\
abla_{a_{i}}\mathcal{Q}_{\phi_{i}}^{i}(s_{i},a_{i})]}_{Critic_{Local}} \tag{16}\]  
in which the well-fitting action picked by the action \(i\) supervening its policy \(\pi_{i}\) is defined as \(a_{i}^{t}=\pi_{i}(s_{i}^{t})\). The term \(Critic_{Global}\) in Equation (16) evaluates the team reward based on \(i\)'s state and action; however, the term \(Critic_{Local}\) evaluates the performance of each agent by exclusively considering the local state and action of an individual agent.