## 3 Resource Allocation Based on a Multi-Agent Reinforcement Learning Algorithm
### Formulation of Rewards
\]  
```
Input: Observation state \(\mathcal{S}\) Output: Action \(\mathcal{A}\)  Start generating platoons from environment simulator  Initialize main global critics \(\mathcal{Q}_{\psi_{1}}^{c-\beta 1o_{1}}\), \(\mathcal{Q}_{\psi_{2}}^{c-\beta o_{2}}\) and \(\mathcal{Q}_{\psi_{3}}^{c-\beta o_{3}}\)  Initialize target global critics \(\mathcal{Q}_{\psi_{1}}^{c-\beta o_{1}}\), \(\mathcal{Q}_{\psi_{2}}^{c-\beta o_{2}}\) and \(\mathcal{Q}_{\psi_{3}}^{c-\beta o_{3}}\)  Initialize policy networks and critic networks defined by task of every agent  Initialize Experience Replay Buffer \(\mathfrak{D}\) with \(K\) previous networks \([\mathcal{Q}_{\psi_{3}^{\prime}}]^{1},\dots,[\mathcal{Q}_{\psi_{3}^{\prime}}]^ {K}\) for each episodedo  Receive initial observation state \(s_{1}\)  Update location and channel gains of each platoon  Reset the CAM message size \(\aleph\) and time limit \(T\) determined as 100 ms for each time slot \(t\)do for each agent \(b\)do  Observe \(s_{b}^{t}\) and choose action \(a_{b}^{t}=\pi_{\theta_{b}}(s_{b}^{t})+\epsilon\) based on the current policy  and exploration noise \(\epsilon\in\mathcal{N}(0,\sigma)\) endfor \(s^{t}=[s_{1}^{t},s_{2}^{t},\dots,s_{p}^{t}]\) and \(\mathrm{a}^{t}=[a_{1}^{t},a_{2}^{t},\dots,a_{p}^{t}]\)  Receive global and local rewards, \(r_{\beta o}^{t}\) and \(r_{loc}^{t}\)  Store transition \((s^{t},\mathrm{a}^{t},r_{\beta o}^{t},r_{loc}^{t},s^{t+1})\) in Experience Replay Buffer \(\mathfrak{D}\) endfor  Sample random mini-batch of \(\mathrm{S}\) transitions \((s^{i},\mathrm{a}^{i},r_{\beta o}^{i},r_{loc}^{i},s^{i})\) from \(\mathfrak{D}\)  Set \(a_{b}^{t}=\pi_{\theta_{b}}(s_{b}^{t})+\epsilon,\epsilon\sim\text{clip }( \mathcal{N}(0,\sigma),-c,c)\)  Set \(\mathcal{Q}_{1}^{\prime}=min_{c=1,2}\mathcal{Q}_{\psi_{c}^{c}}^{c-\beta o}(s^ {\prime},\mathrm{a}^{t})\)  Set \(\mathcal{Q}_{2}^{\prime}=\frac{1}{K}\sum_{k}[\mathcal{Q}_{\psi_{3}}^{c-\beta o }(s^{\prime},\mathrm{a}^{t})]^{k}\)  Set \(y_{c-\beta o}=r_{c-\beta o}+\lambda[\beta\mathcal{Q}_{1}^{\prime}+(1-\beta) \mathcal{Q}_{2}^{\prime}]\)  Update global critics by minimizing the loss: \(\mathfrak{DS}(\psi_{c})=\frac{1}{S}\sum_{i}{(\mathcal{Q}_{\psi_{c}}^{c-\beta o _{c}}(s^{i},\mathrm{a}^{i})-y_{c-\beta o}^{i})}^{2}\) ifepisode mod \(d\)then  Train local actor network and critic network for each agent \(j\)do  for each task \(b\)do  Set \(y_{loc,b}^{j}=r_{loc,b}^{j}+\gamma\mathcal{Q}_{\psi_{i}^{\prime},b}^{j,b}(s_{ i}^{\prime},a_{i}^{t})\)  Update local critics by minimizing the loss: \(\mathfrak{DS}(\phi_{i},b)=\frac{1}{S}\sum_{i}{(\mathcal{Q}_{\phi,b}^{i,b}(s_{ i},a_{i})-y_{i\alpha,b}^{i})}^{2}\) endfor  Update local actor network using deterministic policy gradient: \(\
abla_{\theta_{i}}Z_{i}=\frac{1}{S}\sum_{i}\
abla_{\theta_{i}}\pi_{i}(a_{i} |s_{i})\
abla_{a_{i}}\mathcal{Q}_{\psi_{c}}^{c-\beta o_{c}}(s,\mathrm{a})+ \sum_{b=1}^{M}\
abla_{\theta_{i}}\pi_{i}(a_{i}|s_{i})\
abla_{a_{i}}\mathcal{Q} _{\psi_{b},b}^{i,b}(s_{i},a_{i})\)  Update target network parameters: for each task \(b\)do \(\phi_{i,b}^{\prime}\leftarrow\tau\phi_{i,b}+(1-\tau)\phi_{i,b}^{\prime}\) endfor \(\theta_{i}^{\prime}\leftarrow\tau\theta_{i}+(1-\tau)\theta_{i}^{\prime}\)  Update \(\mathfrak{D}\) by broadcasting the earliest and populating the current target critic network endfor endif endfor
```  
**Algorithm 1** CMADDPG  
Update local critic by minimizing the loss: \(\mathfrak{DS}(\phi_{i},b)=\frac{1}{S}\sum_{i}{(\mathcal{Q}_{\psi_{c}}^{i,b}(s_{ i},a_{i})-y_{i\alpha,b}^{i})}^{2}\)  endfor  Update local actor network using deterministic policy gradient: \(\
abla_{\theta_{i}}Z_{i}=\frac{1}{S}\sum_{i}\
abla_{\theta_{i}}\pi_{i}(a_{i}|s_ {i})\
abla_{a_{i}}\mathcal{Q}_{\psi_{c}}^{c-\beta o_{c}}(s,\mathrm{a})+\sum_{b= 1}^{M}\
abla_{\theta_{i}}\pi_{i}(a_{i}|s_{i})\
abla_{a_{i}}\mathcal{Q}_{\psi_{b },b}^{i,b}(s_{i},a_{i})\)  Update target network