## 3 Beam Allocation Algorithm Based on MAPPO and Stackelberg Game
### Beam Allocation Algorithm Based on MAPPO
To facilitate the training of the GRU in the network, at the end of each episode, the trajectory of this episode is saved to the experience pool. Each time step in the trajectory is saved as a tuple \((\Theta,\mathcal{S},\mathcal{M},\mathcal{O}^{\prime},\mathcal{S}^{\prime}, \mathcal{R})\), which is the set of local states of all agents at the current time slot \(n\); \(\mathcal{S}\) is the global state at the current time slot \(n\); \(\mathcal{A}=\{a_{1,n},\cdots,a_{K,n}\}\) is the set of actions of all agents at the current time slot \(n\); \(\mathcal{O}^{\prime}=\{o_{1,n+1},\cdots,o_{K,n+1}\}\) is the set of local states of all agents at the next time slot \(n+1\); \(\mathcal{S}^{\prime}\) is the global state at the next time slot \(n+1\); \(\mathcal{R}\) is the set of rewards of all agents at the current time slot \(n\), where each element is equal to \(r_{n}\). Since MAPPO is an on-policy algorithm, when the number of episodes in the experience pool equals the batch size \(samp\), training is performed, and the experience pool is cleared after training. The loss function for the actor network is calculated as follows:  
\[L(w_{act})=-L^{clip}(w_{act})-S[\pi_{w_{act}}], \tag{15}\]  
where \(w_{act}\) represents the network parameters of the actor network; \(L^{clip}(w_{act})\) denotes the proximal policy optimization loss, which stabilizes training results by clipping the update magnitude through the clip strategy [36]. The proximal policy optimization loss is defined as follows:  
\[L^{clip}(w_{act})=\mathbb{E}_{n}\big{[}\min\big{(}rto_{n}(w_{act}) \hat{A}_{n}, \tag{16}\] \[clip(rto_{n}(w_{act}),1-\epsilon,1+\epsilon)\hat{A}_{n})\big{]},\]  
where \(rto_{n}(w_{act})=\frac{\pi_{w_{act}}(a_{n}|a_{n})}{\pi_{w_{act}}^{old}(a_{n}|a _{n})}\) represents the ratio of the probability of taking action \(a_{n}\) under the current policy \(\pi_{w_{act}}\) in local state \(o_{n}\) to the probability of taking the same action under the old policy \(\pi_{w_{act}}^{old}\), used to measure the update magnitude of the policy; \(\hat{A}_{n}=V_{w_{val}}(s_{n})-V_{avg}(s_{n})\) represents the advantage function, used to measure the value of the current global state \(V_{w_{val}}(s_{n})\) relative to the average level \(V_{avg}(s_{n})\); \(w_{val}\) is the network parameter of the critic network. The value of the global state can be understood as the expected future return starting from the current global state. The purpose of clipping is to limit the update magnitude of the policy to avoid the instability caused by large updates. \(s[\pi_{w_{act}}]\) represents the entropy of the policy, where a higher entropy indicates a more uniform probability distribution of actions and stronger exploration ability. The loss function for the critic network is calculated as follows:  
\[L(w_{val})=\mathbb{E}\Big{[}\big{(}V_{w_{val}}(s_{n})-V_{tar}\big{)}^{2}\Big{]}, \tag{17}\]  
where \(V_{w_{val}}(s_{n})\) represents the value function of the global state \(s_{n}\); \(V_{tar}=\sum\limits_{i=n}^{N-1}\gamma^{i-n}r_{i}+\gamma^{N-n}V_{w_{val}}(s_{N})\) represents the target value function, which is the expected future return, also known as the expected discounted reward, with \(\gamma\) as the discount factor. The second term in the function is calculated only when \(N\) is the episode end step. As shown in Figure 2, the joint power allocation and target vehicle switching algorithms need to be integrated into the MAPPO framework. The specific implementation details are provided in Algorithms 1 and 2. In summary, the process flow of the beam allocation algorithm based on MAPPO is summarized in Algorithm 3.