# Service-Driven Dynamic Beam Hopping with Resource Allocation for LEO Satellites
## 3 A Cooperative Multi-Agent RL for Single-Satellite Beam Hopping
### MDP Model  
The MDP can be described as a five-tuple \((S,A,P,R,\gamma)\), where \(S\) represents the state space, which means the set of environmental states. \(A\) indicates the action space, signifying available actions for the agent. \(P=P(S^{\prime}|S,A)\) stands for the state transition probability, implying the probability of transitioning to state \(S^{\prime}\) from state \(S\) after executing action \(A\). \(R=R(S,A,S^{\prime})\) denotes the reward function, meaning the immediate reward obtained when transitioning to state \(S^{\prime}\) from state \(S\) under action \(A\). \(\gamma\in[0,1]\) refers to the discount factor. In a single-agent architecture, the entire satellite is regarded as an independent agent. The agent's actions consist of two parts--beam power and beam hopping pattern--which raises the issue of the size of the action space. Suppose that in the current LEO beam hopping satellite system, there are \(M\) beam positions on the ground and a total of \(K\) service beams working simultaneously. Then, the number of beam hopping patterns is \(C_{M}^{K}=\frac{M!}{K!(M-K)! }\) in time slot \(t\). Combining with \(N\) beam powers, the overall size of the satellite's action space is \(C_{M}^{K}\times N\), which is an excessive burden for the onboard storage capacity of LEO satellites. Considering the characteristics of small on-board memory in satellites, we apply a cooperative MARL-VDN algorithm framework to reduce the action space of the satellite. In this multi-agent architecture, each service beam of the satellite is regarded as an independent agent. Since all agents are on the same satellite, they can share states and rewards. In the VDN architecture shown in Figure 4, the Q-network of each agent is independent, but gradient updates are performed through joint Q-values during training. The multi-agent model can achieve dynamic beam hopping on the satellite after training. In this paper, the objective optimization problem in Equation (14) can be described as a discrete-time MDP. We perform MDP modeling and design the state space, action space, and reward as follows. State space: In this paper, the traffic requests are represented by the data volume of each beam position. In time slot \(t\), the traffic requests of all beam positions are observed as the global state, and each agent can share this global state. It can be expressed as  
\[S_{t}=s_{t}^{k}=D_{t}=\sum_{m\in\
abla}D_{t}^{m} \tag{16}\]  
\[D_{t}=[D_{t}^{1},D_{t}^{2},\cdots,D_{t}^{M}]^{T}= \begin{bmatrix}d_{t}^{1}&d_{t- 1}^{1}&\cdots&d_{t-T_{th}+1}^{1}\\ d_{t}^{2}&d_{t-1}^{2}&\cdots&d_{t-T_{th}+1}^{2}\\ \vdots&\vdots&&\vdots\\ d_{t}^{M}&d_{t-1}^{M}&\cdots&d_{t-T_{th}+1}^{M}.\end{bmatrix}  \tag{17}\]  
where \(S_{t}\) is the global state of the beam hopping satellite in time slot \(t\). \(s_{t}^{k}\) represents the state information of the \(k\)-th agent in time slot \(t\). \(D_{t}\) is the traffic request of each beam position in time slot \(t\). \(d_{t}^{m}\) is defined above, representing the traffic demand of beam position \(m\) in time slot \(t\). It can be seen from the above formula that the state space matrix is composed of two dimensions: time and beam position. The total data packet delay in the packet buffer queue is obtained by  
\[delay_{t}=\sum_{m\in\mathcal{V}}\sum_{e=1}^{T_{th}}(t^{m}-t_{e}^{m}) \tag{18}\]  
where \(t_{e}^{m}\) represents the time slot \(t_{e}\) when the data packet arrives at beam position \(m\) after being generated. Action space: In the multi-agent architecture of the LEO beam hopping satellite communication system, each service beam dynamically selects a beam position for illumination in each time slot. The above decision-making process involves designing a beam hopping pattern and allocating the corresponding beam power. For the optimization problem in this paper, a total of \(K\) agents need to cooperatively make decisions.