## 1 Introduction
[12] solved the multi-UAV deployment problem by a circular deployment scheme and jointly optimized the offloading and scheduling strategies to minimize the weighted sum of transmission and lower energy consumption. [PERSON] et al. [13] studied multi-UAV-assisted data collection scenarios with circular trajectories and considered the deployment of UAVs and device association to minimize the total energy consumption of IoT devices. [PERSON] et al. [14] jointly optimized UAV trajectories, ground equipment scheduling, and power allocation by using the technologies of block coordinate descent and successive convex approximation to maximize the overall throughput of ground devices. However, the above work only considered the uplink or downlink transmission scenarios. [PERSON] et al. [15] studied the simultaneous uplink and downlink transmission systems of multiple UAVs, in which one UAV is responsible for downlink transmission and the other UAV is responsible for uplink transmission. Furthermore, [PERSON] et al. [16] studied that each UAV can dynamically change the transmission mode in each time slot, but the premise of this work is to know the UAVs' trajectories. It is significant that UAVs can change the mission mode according to the transmission requirements. For example, a UAV adopts the mode of offloading after completing the collection mission. When the UAV fails during the collection mission, all the data collected by the UAV cannot be returned to the OBS, resulting in data loss. The high coupling of UAV trajectories, association relationships, and mission mode is a challenge for us, which can usually be described as a mixed-integer non-convex problem. The above works effectively solved this kind of problem through traditional convex optimization techniques and iterative methods, but they have high computational complexity and the curse of dimensionalitywith the increase of the action and state dimension of the system due to the high dynamics of UAVs. With the development of machine learning technology, deep reinforcement learning (DRL) algorithms have received extensive attention with the advantage of dealing with highly dynamic environments [17; 18; 19]. In reinforcement learning, the agent maximizes the cumulative reward by constantly exploring the environment. DRL combines the advantages of the deep neural network (DNN) and can handle more complex actions and states. At present, much research has applied DRL algorithms to multi-UAV trajectory optimization. References [20; 21; 22] studied the multi-UAV trajectory optimization scheme based on the multi-agent deep deterministic policy gradient (MADDPG) algorithm and regarded the UAVs as the agents. [PERSON] et al. [20] aimed to minimize the average age-of-information (AoI) and proposed an MADDPG algorithm to jointly optimize the UAV sensing location and transmission location. However, the MADDPG algorithm can only handle a continuous action space. Hence, [PERSON] et al. [21] combined a low-complexity mission decision-making method with the MADDPG algorithm. [PERSON] et al. [22] used a potential game approach to determine the service allocation between users and multiple UAVs, which effectively solved the mixed-integer non-linear problem. For using DRL algorithms to process the hybrid discrete and continuous action space, it undoubtedly reduces the flight accuracy of UAVs if a continuous action is discretized, and this is inconsistent with the actual situation of UAV flight actions; relaxing the discrete action space into a continuous action space increases the complexity of the action space, such as in [PERSON] et al. [23]. [PERSON] et al. [24] proposed the parameterized deep Q-network learning (PDQN) algorithm based on the parameterized action space. [PERSON] et al. [25] proposed a multi-agent parametrized deep Q-network (MAPDQN) algorithm to maximize both the overall throughput and the fairness throughput. However, the action decision-making method of the PDQN algorithm may reduce the maximum Q-value [26]. Then, [PERSON] et al.