## 3 Beam Allocation Algorithm Based on MAPPO and Stackelberg Game
### Beam Allocation Algorithm Based on MAPPO  
V2V reuses the orthogonal frequency bands of I2V, leading to inter-vehicle interference between V2V and I2V. Additionally, V2V supports multi-beam multi-target communication, resulting in inter-beam interference. To mitigate the impact of interference, the beam direction and frequency band allocation for V2V are crucial. These factors not only affect the sensing-communication performance of the current V2V link but also significantly impact the sensing-communication performance of other V2V and I2V links. If V2V vehicles are treated as single agents for decision-making, the reward for the current agent under the same local state-action pair may vary due to the actions of other agents. This environmental instability prevents the agent from proper training. To address the issue of environmental instability, the multi-agent reinforcement learning framework of centralized training and distributed execution has emerged. In this framework, each agent only needs to input its locally observed state into the decision network to make local decisions, saving communication costs between agents. During training, the global state is input into a centralized network for evaluation, guiding the update of the decision network. Considering that the objective function of problem (11) is to maximize overall sensing-communication performance, we adopt the MAPPO algorithm to promote cooperation among agents, achieving better beam direction and frequency band allocation. In the MAPPO algorithm, the actor network serves as the decision network, while the critic network serves as the centralized network. The critic network evaluates the value of the global state to guide the update of the decision network. For ease of expression, in this chapter, V2V vehicles are also referred to as agents, and their local state can be expressed as  
\[ \begin{split} o_{k,n}=\left[\mathbf{I}_{k\to m},\mathbf{I}_{c}, \mathbf{I}_{r},c_{k,n-1},P_{k,n-1},v_{k,n-1},\\ \left[SI_{1,n-1},\cdots,SI_{K^{\prime},n-1}\right]\right]^{\prime} \end{split}  \tag{12}\]  
where the local state at time slot \(n\) is composed of the values from time slot \(n-1\). \(P_{k,n-1},v_{k,n-1}\) represent the position and velocity of vehicle \(k\), respectively; \(\mathbf{SI}_{k,n-1}=\left[SI_{1,n-1},\cdots,SI_{K^{\prime},n-1}\right]\) represents the sensing information of each beam's target vehicle, including position, velocity, and direction angle; \(c_{k,n-1}\) denotes the frequency band index selected by vehicle \(k\); \(\mathbf{I}_{k\to m}\) represents the interference power of vehicle \(k\) on each I2V vehicle \(m\); \(\mathbf{I}_{c}\) represents the interference power received by vehicle \(k\) for each beam's downlink communication; \(\mathbf{I}_{r}\) represents the interference power received by vehicle \(k\) for each beam's echo signal. Since the sensing information includes noise, this decision process is modeled as a POMDP. To improve decision-making, we modify the actor and critic networks to the DRQN network structure, utilizing temporal features from the historical state sequence to compensate for the current state. The global state \(s_{n}\), which includes the local states of all agents, can be expressed as  
\[s_{n}=[o_{1,n},\cdots,o_{K,n}]. \tag{13}\]  
The action of agent \(k\) is denoted as \(a_{k,n}=[\mathbf{I}_{k,n},c_{k,n}]\). \(\mathbf{I}_{k,n}\) represents the relative change in each beam direction, and \(c_{k,n}\) represents the frequency band index selected in the current time slot. Regarding the reward, to promote cooperation, all agents receive the same reward, representing the overall reward of the environment. Therefore, the reward function should be the objective function in problem (11):  
\[r_{n}=\sum_{k=1}^{K}\left(w_{c}E_{k,n}+w_{r}R_{k,n}^{r}\right)+w_{R}R_{n}^{R}, \tag{14}\]where \(w_{e},w_{r},w_{R}\) represent the weights of each part of the reward. By adjusting these weights, resources can be biased towards a specific target.