# A Multi-Branch DQN-Based Transponder Resource Allocation Approach for Satellite Communications
## 3 Proposed Method
### Modeling of MBDQN
For task allocation with the same patch size, the closer the resource occupancy rate is to the upper limit, the larger the reward value should be. Based on the above principles, the reward is designed as Equation (5):  
\[r=-\log\left(1-\frac{\sum_{m=1}^{M}o_{m}*\phi_{1}(t_{m})*\phi_{2}(f_{m})}{\Delta t *\Delta f}+\varepsilon\right) \tag{5}\]  
where the non-negative multiplier \(\varepsilon\) is used to avoid infinite values. (4) Replay memory  
For a satellite task allocation system, there exists a salient correlation between the order of task allocation and the final resource occupancy ratio. However, deep neural networks require that the input samples are uncorrelated. Hence, in the training phase, we adopt the replay memory trick to eliminate the correlation of the sample data generated by the interaction with the environment in the satellite task allocation system. Moreover, replay memory ensures that a sample can participate in training multiple times, thus improving learning efficiency. The transition \(e_{t}=\langle s_{t},a_{t},s_{t+1},r_{t}\rangle\) generated in each training step is saved in a replay memory pool. (5) Target network  
The outcome of the allocated action in a satellite task allocation problem is usually a positive reward, even though the action is not necessarily optimal, and there will be an overestimation problem similar to DQN. Meanwhile, every time the binary \(\langle s_{t},a_{t}\rangle\) is used to update MBDQN parameters \(\omega\), MBDQN tends to overestimate the \(Q\) value of \(\langle s_{t},a_{t}\rangle\). Therefore, in the training phase, we use the target network to calculate the TD target, and SGD only updates the parameters of the DQN without updating those of the target network. TD target is calculated as  
\[y_{t}=r_{t}+\gamma*max_{a}Q^{\prime}\big{(}s_{t+1},a;\omega^{-}\big{)} \tag{6}\]  
where \(\gamma\) is the discount factor reward. The parameter of the target network \(\omega^{-}\) is updated at regular intervals, with the target update interval empirically set to 100 in MBDQN. Parameters of the target network are updated by calculating the weighted average of \(\omega\) and \(\omega^{-}\) before assigning the value to \(\omega^{-}\). (6) Loss function  
Considering the self-consistency of the \(Q\) function, similar to DQN, we update the action-value function as per the principle of minimizing the estimation error of \(Q(s_{t},a_{t};\omega)\) and \(y_{t}\). A minibatch of data \(\langle s_{t},a_{t},s_{t+1},r_{t}\rangle\) is obtained by model inference in an iteration, that is, after obtaining the reward value \(r_{t}\) between states \(s_{t}\) and \(s_{t+1}\). The loss function \(L(\omega)\) is calculated by the following equation  
\[L(\omega)=E\Big{[}(y_{t}-Q(s_{t},a_{t};\omega))^{2}\Big{]} \tag{7}\]  
where \(y_{t}\) is calculated by Equation (6). MBDQN's overall operations are outlined in Table 2. During the training phase, the MBDQN model is initialized by Kaiming initialization [20], and parameters of the trained model are exported and saved at the end of a full episode. During testing, these saved parameters are loaded to initialize the MBDQN model, and the action-value function \(Q(s_{t},a_{t};\omega)\) is calculated according to the current input state. The action \(a_{t}=\underset{a}{argmax}Q(s_{t},a_{t};\omega)\) is selected in a deterministic way, and this loop is executed \(M\) times.