# A Multi-Branch DQN-Based Transponder Resource Allocation Approach for Satellite Communications
## 3 Proposed Method
### Structure of MBDQN  
This section describes the network structure of the proposed MBDQN, including its RP-Branch, TL-Branch, and Value-Branch, as well as the detailed structure (as shown in Figure 4). RP-Branch is a convolutional neural network (CNN) composed of convolutional modules, each consisting of convolutional layers, batch normalization (BN) layers [20], and ReLU activation. The resource pool state matrix \(s_{rp}\) calculated from Equation (2) has \begin{table}
\begin{tabular}{l l} \hline \hline
1. & **Training Process:** \\
2. & **Initialize** params of MBDQN and target network \(\omega=\omega^{-}\) \\
3. & **Initialize** hyperparams for training \\
4. & **For** step in episode **do:** \\
5. & Observe states \(s_{rp}\) and \(s_{tl}\) from environment at time \(t\) \\
6. & Format \(s_{rp}\) and \(s_{tl}\), and obtain state \(s_{t}\) \\
7. & Evaluate MBDQN and obtain \(Q(s_{t},a_{t};\omega)\) \\
8. & Select action \(a_{t}\) by Equation (4) \\
9. & Execute action \(a_{t}\) and observe state \(s_{t+1}\) \\
10. & Calculate the reward \(r_{t}\) by Equation (5) \\
11. & Save transition \(\langle s_{t},a_{t},s_{t+1},r_{t}\rangle\) in replay memory \\
12. & Sample a minibatch of \(\langle s_{t},a_{t},s_{t+1},r_{t}\rangle\) from replay memory \\
13. & Calculate TD target \(y_{t}\) by Equation (6) \\
14. & Calculate loss function \(L(\omega)\) by Equation (7) \\
15. & Update params \(\omega\) of MBDQN through SGD optimizer \\
16. & For every \(T\) step, update params \(\omega^{-}\) of target network \\
17. & **End For** \\
18. & **Save** and **export** params \(\omega\) \\ After training \\
1. & **Testing Process:** \\
2. & **Load** and **initialize** the trained params \(\omega\) of MBDQN \\
3. & **For**\(m\) in number \(M\) **do:** \\
4. & Observe states \(s_{rp}\) and \(s_{tl}\) from environment at time \(t\) \\
5. & Format \(s_{rp}\) and \(s_{tl}\), and obtain state \(s_{t}\) \\
6. & Evaluate MBDQN and obtain \(Q(s_{t},a_{t};\omega)\) \\
7. & Select action \(a_{t}=\underset{a}{argmax}Q(s_{t},a_{t};\omega)\) \\
8. & Execute action \(a_{t}\) and observe state \(s_{t+1}\) \\
9. & **End For** \\ \hline \hline \end{tabular}
\end{table} 
Table 2: Overall operations of the MBDQN model. Figure 4: The structure of the proposed MBDQN resource allocation model. an input size of \(N*N\) that resembles a 2D image. Therefore, CNNs are more conducive to extracting discriminative features \(f_{rp}\) of the satellite resource pool state. The role of RP-Branch is to extract features from the input \(s_{rp}\) and leverage them as the prerequisite for deriving the action-value function. TL-Branch is a fully connected (FC) neural network. The satellite task list input sequence \(s_{tl}\) with an input size of 3 * \(M\) is calculated by Equation (1). Input data \(s_{tl}\) are serialized for FC layers to extract features from them. The objective of TL-Branch is to extract the features from the input \(s_{tl}\), which are used to estimate the action-value function together with \(f_{rp}\). RP-Branch and TL-Branch are in parallel, and their outputs are concatenated via a Concat layer. Then, \(f_{rp}\) and \(f_{tl}\) are concatenated to yield the global feature \(f\) for action-value function estimation, which is subsequently used as the input of Value-Branch to calculate \(Q(s_{t},a_{t};\omega)\). Similar to TL-Branch, Value-Branch is also a neural network based on FC layers. TL-Branch, RP-Branch, and Value-Branch jointly constitute MBDQN's overall structure, as depicted in Table 3. Compared with general reinforcement learning models, MBDQN combines the characteristics of multiple satellite resource types, is more suitable for different types of state inputs, and is more attentive to the resource state of the satellite communication system. In particular, a deeper and more complex network could be leveraged as the backbone of the MBDQN model, depending on the complexity of the satellite allocation problem to be solved (e.g., number of tasks, size of the resource pool block division).