# Service-Driven Dynamic Beam Hopping with Resource Allocation for LEO Satellites
## 3 A Cooperative Multi-Agent RL for Single-Satellite Beam Hopping
### Cooperative MARL-VDN Algorithm Framework  
The parameters of the MARL-VDN network are shown in Table 2. The MARL-VDN algorithm decomposes the joint action-value function \(Q_{total}(S_{t},A_{t})\) of the multi-agent architecture into the sum of individual Q-functions of each agent (\(Q_{total}=\sum_{k=1}^{K}Q(s_{t}^{k},a_{t}^{k})\)) through linear addition and decomposition. The decomposition uses global rewards during the training phase to promote the cooperation of multi-agents, centrally optimizes the Q-networks of all agents, and avoids policy conflicts among agents. During execution, each agent only relies on its Q-function \(Q(s_{t}^{k},a_{t}^{k})\) to select actions, achieving CTDE. The Q-network parameters of each agent are defined as \(\omega=\{\omega^{k}|k=1,2,\cdots,K\}\), then the Q-network of the \(k\)-th agent in time slot t is expressed as \(Q(s_{t}^{k},a_{t}^{k},\omega^{k})\). This paper adopts a variant of the recurrent neural network as a function approximator. In time slot \(t\), agents select and execute the joint action \(A_{t}\) according to the environmental observation \(S_{t}\) and the \(\varepsilon\)-greedy strategy. After taking the action, agents obtain the global reward \(R_{t}\), and the environmental state is updated from \(S_{t}\) to the next state \(S_{t+1}\). In a multi-agent environment, synchronous updates of all agents' strategies will make the environmental dynamics appear non-stationary to individual agents, and individual Q-networks may fall into local optima due to biases in local consecutive samples. To address the above problem, we adopt an experience replay buffer and independent target Q-networks to counter non-stationarity. The experience information \((S_{t},A_{t},R_{t},S_{t+1})\) is stored in the experience buffer for training the network. To balance gradient variance and computational efficiency, agents randomly select small-batch sample data from the experience buffer to calculate the target value and loss function and update the gradients of the Q-network. In the MARL-VDN architecture, the target Q-network parameters of agents are defined as \(\omega^{-}=\{\omega^{-k}|k=1,2,\cdots,K\}\). Since its network structure is the same as that of the Q-network, the target Q-network of the \(k\)-th agent in time slot \(t\) is expressed as \(Q^{-}(s_{t}^{k},a_{t}^{k},\omega^{-k})\). When training the network, after sampling m samples from the experience buffer, the target value of the temporal difference (TD) error can be calculated as  
\[y_{t}=R_{t}+\gamma\left[\sum_{k=1}^{K}Q^{-}(s_{t+1}^{k},a_{t+1}^{k};\omega^{- k})\right] \tag{24}\] \begin{table}
\begin{tabular}{c c c c} \hline \hline
**Layer** & **Input Size** & **Output Size** & **Activation** \\ \hline Input Layer (fc1) & 76 & 64 & Relu \\ Hidden Layer 1 (GRU Cell) & 64 & 64 & GRU-internal \\ Output Layer (fc2) & 64 & \(M\times N\) & Linear \\ \hline \hline \end{tabular}
\end{table} 
Table 2: Parameters of the MARL-VDN network. \[a_{t+1}^{k}=\underset{a_{t+1}^{k}}{\arg\max}Q^{-}(s_{t+1}^{k},a_{t+1}^{k};\omega^{- k}) \tag{25}\]  
where \(R_{t}\) represents the immediate reward obtained in time slot \(t\). \(\gamma\in[0,1]\) is the discount factor, which is used to measure the importance of future rewards. \(s_{t+1}^{k}\) and \(a_{t+1}^{k}\) are the state and action of the \(k\)-th agent in time slot \(t+1\), respectively. The minimum mean square error loss can be described as  
\[L(\omega^{k})=\mathbb{E}\Bigg{[}\left(\sum_{k=1}^{K}Q(s_{t}^{k},a_{t}^{k}; \omega^{k})-y_{t}\right)^{2}\Bigg{]} \tag{26}\]  
The Adam optimizer is used to simultaneously optimize and update the Q-network parameters \(\omega\) of all agents. Finally, through a delayed parameter synchronization update, the parameters \(\omega\) of the Q-network are copied to the target Q-network \(\omega^{-}\) every G steps. The detailed description of the cooperative MARL-VDN algorithm for dynamic beam hopping is shown in Algorithm 1. ```
0: Number of agents \(K\). Exploration rate \(\epsilon\). Size of the experience replay buffer \(|D|\).