# A Multi-Branch DQN-Based Transponder Resource Allocation Approach for Satellite Communications
## 1 Introduction  
Resource leasing is widely adopted in satellite communication systems based on transparent transponders, especially for frequency resources [1]. As shown in Figure 1, the transponder resource is shared among multiple satellite-terrestrial integrated networks. In general, the organizer of the satellite communication network and the lesser of the resource calculate the network frequency demand in advance and submit it together with the lease time of the frequency to the operation control center of the communication satellite who is the owner of the frequency. The operation control center maintains a table of allocated resources for each transponder of the satellite, and the lease time of each resource segment in the transponder is recorded in the table. Subsequently, the operators will find a free resource segment that satisfies the lessor's needs in the table, update the corresponding lease time period, and inform the lesser of the allocation result. Finally, the leaser uses the allocated satellite transponder resources to establish the satellite communication network. The above process of transparent satellite transponder resource allocation mainly relies on manual labor, and is therefore only feasible when the maintenance leasing tasks are not laborious. However, with the development of the maintenance process, the intensity of the leasing business continues to increase, and the fragmentation of satellite transponder frequency caused by the leasing process is becoming increasingly serious, posing considerable challenges to the maintenance of resource leasing and the search for idle resource segments. These challenges have increased the task complexity and technical requirements for operators as well as the operating cost of the satellite system [2]. In addition, manual search struggles to guarantee optimal utilization of resources due to its empirical nature. Moreover, when the requirements of high-priority tasks cannot besatisfied immediately, and the existing resources of low-priority tasks need to be preempted, how to select preemption objects to ensure the least impact is also a problem to be considered by the operators. The problem of resource allocation under multiple constraints has been proven to be NP-hard [3]. In order to solve the above problem, some traditional operations research algorithms [3; 4; 5], greedy algorithms [6] and heuristic algorithms [7; 8; 9; 10; 11] have been applied in the automation of transparent satellite transponders and intelligent resource allocation. In actual resource allocation, however, there exists a plethora of interdependent nodes, which result in a set of variables and constraints to be solved in operations research models and heuristic algorithms. Therefore, the high computational complexity renders it inapplicable to problems with strict timeliness requirements, especially in satellite resource allocation. With the development of the satellite communication field and the research of reinforcement learning, more and more empirical data are stored, and deep reinforcement learning (DRL) can exploit these data due to its characteristics to discover regular features and learn the policies. Hence, resource allocation methods based on reinforcement learning have started to receive increasing attention [12]. In this paper, we propose the Multi-Branch Deep Q-Network (MBDQN) for satellite communication resource allocation. By introducing TL-Branch and RP-Branch, this DRL-based model can simultaneously extract features for the satellite resource pool (RP) state and task list (TL) state. The feature embeddings are concatenated and fed into Value-Branch to calculate the action-value function. The task selection action and the priority action are calculated according to the optimal action-value function.