# A Multi-Branch DQN-Based Transponder Resource Allocation Approach for Satellite Communications
## 3 Proposed Method
### Problem Formulation  
A multi-branch DQN based resource allocation approach for satellite communications can be defined as follows. Given the satellite communication system conditions, the satellite resource and task demand generation of the satellite communication system is defined as the environment of reinforcement learning. In this environment, there is a state \(s_{t}\), including the resource pool state \(s_{rp}\) and task list state \(s_{tl}\), as well as a series of valid action sets \(A(s)\) for decision-making. The agent controlled by MBDQN will select an effective action \(a_{t}\in A(s)\) in the action set \(A(s)\), and execute the action \(a_{t}\) in the environment to obtain reward \(r_{t}\) at time \(t\), after which the state of the environment is transferred to \(s_{t+1}\). In this method, an iteration will be executed until the environment reaches the termination condition, that is, when all allocations are completed and the resource pool has no further resources to allocate.  
The MBDQN model essentially finds a better sort sequence than those by heuristic algorithms. Its ultimate goal is to minimize the resource occupation rate of the allocation result and shorten the total satellite task allocation time. An off-policy training method based on the DQN model [16], MBDQN is a typical value-optimized reinforcement learning algorithm.  
The objective of DQN is to learn an action-value function \(Q(s,a;\omega)\), where \(\omega\) represents all parameters of the current action-value function, and \(Q(s,a;\omega)\) the estimated value of action \(a\) under the parameters \(\omega\) with state \(s\). It can also be understood as the expected sum of all reward values to be obtained from the environment, still based on the parameters \(\omega\) interacting with the environment. Finally, in accordance with the principle of maximizing the value of the action-value function, the action selected by the DQN algorithm is \(a(s)=\underset{a}{\text{argmax}}Q(s,a;\omega)\). When the agent interacts with the environment and collects a certain amount of empirical data, the action-value function can be updated according to the principle of minimizing the estimation TD error.  
\begin{table}
\begin{tabular}{c c} \hline \hline
**Symbol** & **Definition** \\ \hline \(N\) & Number of patches \\ \(M\) & Number of task lists \\ \(s_{t}\) & State of environment at time \(t\) \\ \(s_{rp}\) & State of resource pool \\ \(s_{tl}\) & State of task lists \\ \(a_{t}\) & Action selected at time \(t\) \\ \(a_{p}\) & Priority action selected \\ \(a_{c}\) & Choose action selected \\ \(A(s)\) & Action space of MBDQN \\ \(\Delta t\) & Effective time resource range \\ \(\Delta f\) & Effective frequency resource range \\ \(p_{(i,j)}\) & Patch \((i,j)\) of resource pool \\ \(r_{t}\) & Reward at time \(t\) \\ \(\omega\) & Parameters of MBDQN \\ \(\omega^{-}\) & Parameters of target network \\ \(Q(.)\) & Action-value function of MBDQN \\ \(Q^{\prime}(.)\) & Action-value function of target network \\ \(\phi_{1}(t)\) & State reformulation on time \\ \(\phi_{2}(f)\) & State reformulation on frequency \\ \(\gamma\) & Discount factor reward \\ \(y_{t}\) & TD target at time \(t\) \\ \(L\) & Loss function of the model \\ \(f_{rp}\) & Feature embedding of RP-Branch \\ \(f_{tl}\) & Feature embedding of TL-Branch \\ \(o_{m}\) & Status of the \(m\)-th task \\ \hline \hline \end{tabular}
\end{table}
Table 1: Notation definitions.