# A Multi-Branch DQN-Based Transponder Resource Allocation Approach for Satellite Communications
## 3 Proposed Method
### Modeling of MBDQN  
The primary goal of MBDQN is to model the satellite operation control program as the agent and model the task requirements and satellite communication resources as the environment. Furthermore, the main elements of reinforcement learning defined in the MBDQN model for the satellite resource allocation problem are discussed, including state space, action space, and reward. Meanwhile, the definitions of replay memory, target network, and loss function in the model training stage are also introduced in detail. (1) State space  
The state \(s_{t}=\left(s_{rp},s_{tl}\right)_{t}\) of the task requirements and satellite communication resources at time t is the observation based on both the state of resource pool \(s_{rp}\) and that of task list \(s_{tl}\). Whenever a new resource lease requirement is generated, we format the task and update the state of task list \(s_{tl}\), which can be expressed as  
\[s_{tl}=\left\{\ \left[o_{1},\phi_{1}(t_{1}),\phi_{2}(f_{1})\right],\ldots, \left[o_{m},\phi_{1}(t_{m}),\phi_{2}(f_{m})\right]\ \right\} \tag{1}\]  
where \(o_{m}\) represents the allocation status of the \(m\)-th task in the resource pool; \(t_{m}\) and \(f_{m}\) denote the satellite time and frequency resources occupied by the task, respectively. \(\phi_{1}(t_{m})=N*t_{m}/\Delta t\) and \(\phi_{2}(f_{m})=N*f_{m}/\Delta f\) are the state reformulation on \(t_{m}\) and \(f_{m}\) to make them conform to the tensor size of the model input. Both \(t_{m}\) and \(f_{m}\) are reformulated in range \([1,N]\). \(\Delta f\) and \(\Delta t\) represent the practical time resource and the frequency resource range in the satellite resource pool, respectively. By dividing frequency resources and time resources N-1 times in each dimension, the resource pool can be divided into \(N*N\) patch. The state matrix \(s_{rp}\) is used to represent the occupancy of each resource block in the resource pool. The representation of \(s_{rp}\) is as follows:  
\[s_{rp}= \begin{bmatrix}p_{(1,1)}&\cdots&p_{(1,n)}\\ \vdots&\ddots&\vdots\\ p_{(n,1)}&\cdots&p_{(n,n)}\end{bmatrix} _{N*N} \tag{2}\]  
where \(p_{(n,n)}\) is the occupancy indication of the resource pool in the \((n,n)\) patch. (2) Action space  
In the satellite resource allocation problem, the available action space \(A(s)\) is the decision space of the satellite control system, and the action \(a_{t}\) is selected from \(A(s)\) according to the current state \(s_{t}\) at time \(t\). The available action space depends on the type of allocated resources and the limitation of resources. It includes the priority action space \(A_{p}\) and the chosen action space \(A_{c}\). Formally, \(A(s)\) can be defined as:  
\[A(s)=\left\{\left(A_{c}(i),\ A_{p}(j)\right)\middle|1\leq i\leq m,\ j=0,1\right\} \tag{3}\]  
where \(a_{c}=A_{c}(i)\) is the action of task selection, which indicates the selected task number in the list in this turn; \(a_{p}=A_{p}(j)\) is the action of resource search priority, which represents the priority of frequency search or time search during the allocation. The exploration mechanism of action selection directly affects the sampling performance. As such, in the training phase, action selection adopts a \(\epsilon\)-greedy strategy. MBDQN samples from \(A(s)\) randomly with probability \(\epsilon\), which can be expressed as  
\[a_{t}=\left(a_{c},a_{p}\right)_{t}=\left\{ \begin{array}{ll}\text{sample from }A(s),&\text{with probability }\epsilon\\ argmaxQ(s_{t},a_{t};\omega),&\text{others}\end{array} \right. \tag{4}\]  
In the testing phase, MBDQN will select the \(a_{t}\) with the highest probability in the probability distribution \(Q^{*}(s_{t},a_{t})\). (3) Reward definition  
In this paper, the resource occupancy rate is used as the optimization objective of the algorithm, that is, the reward value \(r_{t}\) of the network that represents the evaluation of the selected action \(a_{t}\) in state \(s_{t}\). After the task is allocated, the higher the resource occupancyrate, the higher the reward value.