For instance, rotary-wing UAVs are highly suitable for tasks requiring hovering, such as data collection at specific points. In contrast, fixed-wing UAVs are ideal for surveillance and fast mission execution due to their higher speed and endurance. Additionally, multiple UAVs can be deployed as a swarm in either homogeneous or heterogeneous systems for efficient task allocation and execution [23]. Thus, UAVSN behavior can be treated as multi-agent systems [24; 25]. The action space with neighboring agents becomes highly coupled; more specifically, actions taken by the neighboring agents have a higher impact on the network utility of a particular agent. As a result, existing fully centralized model-based iterative algorithms are not suitable for this unknown and highly dynamic environment. Here, collecting the centralized information on the dynamic topology is cumbersome, and this outdated centralized information-based network optimization can trigger performance degradation. Furthermore, the computational complexity and control overhead increase with the increasing number of UAVs and GUs, which limits scalability. To combat the above-mentioned challenges, model-free data-driven multi-agent deep reinforcement learning (MA-DRL) can be efficiently utilized [26]. By leveraging the partially observable Markov decision process (POMDP), each agent can iteratively interact with the dynamic UAVSN to optimize the long-term cumulative reward based on their current observation. MA-DRL approaches can deal with multiple dynamic constraints by imposing penalty terms in the reward for choosing unfeasible actions by the agent without requiring convexity. Unlike the single-agent DRL algorithm, MA-DRL can avoid the environmental nonstationary problem by sharing observation-action via centralized training and distributed execution. Existing value-based MA-DRL approaches such as double deep Q-learning (DDQN) [27; 28] and Q-learning-based mixing network (QMIX) [29] are suitable only for small-scale discrete action spaces. Hence, these algorithms are not suitable for large hybrid action spaces and extracting features from the dynamic UAVSNs. In contrast, policy gradient-based actor-critic learning can efficiently tackle large observation spaces and hybrid action spaces [30]. Here, the actor network is responsible for generating continuous and discrete actions. Consequently, the critic network is responsible for evaluating the action by estimating the value function, which is further utilized to train the actor network and critic network [31]. The discrete action spaces such as binary task off-loading ratio, discrete frequency band selection, discrete next hop selection to relay data packet, and binary UAV-GUs association can be mapped into continuous probability distribution using softmax activation function at the output layer of the actor-network. Then, during exploitation, the possible action can be sampled according to the maximum probability [32]. According to the existing literature, policy-based multi-agent actor-critic algorithms can be classified as off policy-based deep deterministic policy gradient (DDPG) [8], on policy-based trust region policy optimizations (TRPO) [33], and on policy-based proximal policy optimization (PPO) [31]. Among them, DDPG can handle only continuous action spaces such as trajectory control of UAVs or resource allocation in continuous domains. TRPO and PPO can tackle both continuous and discrete action spaces. Although these policy-based MA-DRL approaches have superior performance compared with the DDQN or QMIX, they encounter instability during policy updates due to the variability of the step-size parameter in policy optimization [4]. Small step sizes trigger slow learning, while large ones can degrade performance. To address this, TRPO defines a trusted region for updating policy using a complex second-order method. PPO simplifies it by employing a first-order method, which bounds policy updates within a range instead of TRPO's hard constraint. PPO has two key variants: PPO-penalty and PPO-clip.