Variations in Multi-Agent Actor-Critic Frameworks for Joint Optimizations in UAV Swarm Networks: Recent Evolution, Challenges, and Directions  
[PERSON]  
1 Department of Electrical and Electronic Engineering, American International University-Banglades, Dhaka 1229, Bangladesh; [EMAIL_ADDRESS] (M.M.A. ); [EMAIL_ADDRESS] (S.A.T. ); [EMAIL_ADDRESS] (T.H. ); [EMAIL_ADDRESS] (S.M. ); [EMAIL_ADDRESS] (M.S.A.) [PERSON]  
1 Department of Electrical and Electronic Engineering, American International University-Banglades, Dhaka 1229, Bangladesh; [EMAIL_ADDRESS] (M.M.A. ); [EMAIL_ADDRESS] (S.A.T. ); [EMAIL_ADDRESS] (T.H. ); [EMAIL_ADDRESS] (S.M. ); [EMAIL_ADDRESS] (M.S.A.) [PERSON]  
1 Department of Electrical and Electronic Engineering, American International University-Banglades, Dhaka 1229, Bangladesh; [EMAIL_ADDRESS] (M.M.A. ); [EMAIL_ADDRESS] (S.A.T. ); [EMAIL_ADDRESS] (T.H. ); [EMAIL_ADDRESS] (S.M. ); [EMAIL_ADDRESS] (M.S.A.) [PERSON]  
1 Department of Electrical and Electronic Engineering, American International University-Banglades, Dhaka 1229, Bangladesh; [EMAIL_ADDRESS] (M.M.A. ); [EMAIL_ADDRESS] (S.A.T. ); [EMAIL_ADDRESS] (T.H. ); [EMAIL_ADDRESS] (S.M. ); [EMAIL_ADDRESS] (M.S.A.) [PERSON]  
1 Department of Electrical and Electronic Engineering, American International University-Banglades, Dhaka 1229, Bangladesh; [EMAIL_ADDRESS] (M.M.A. ); [EMAIL_ADDRESS] (S.A.T. ); [EMAIL_ADDRESS] (T.H. ); [EMAIL_ADDRESS] (S.M. ); [EMAIL_ADDRESS] (M.S.A.) [PERSON]  
2T. W. A. A.  
UAVSNs requires jointly optimizing several continuous and discrete decision variables. It may include controlling the continuous trajectory of UAVs [13], binary UAV-GUs association, determining binary or partial task-offloading, next-hop selection [14], and continuous resource allocation (i.e., transmit power, bandwidth/timeslot, caching, and computing). In these optimization processes, several mixed integer nonlinear dynamic constraints are involved related to UAV's mobility, resource limitations (i.e., bandwidth, transmit power, onboard energy, and maximum queue size), UAV-GUs association, signal-to-interference-plus-noise ratio (SINR) threshold, and minimum delay or AoI threshold. Furthermore, owing to the mobility of UAVs and GUs, the network topology and channel state becomes very dynamic. Thus, the network state and decision space become very large. Additionally, the environment becomes both cooperative and competitive in terms of collaborative trajectory planning and resource sharing in multiple layers along with multiple decision-makers (i.e., UAVs, IoT devices/GUs, high altitude platforms, and satellites), as illustrated in Figure 1.  
For UAVSNs communication links are classified as several bidirectional links such as GUs to UAVs (GU2U) or UAV to GUs (U2 GU), UAV to UAV (U2U), UAV to high-altitude platform (HAP) (U2H), HAP to HAP (H2H), HAP to satellites (H2S), and satellites-to-satellites (S2S) links, as shown in Figure 1. Particularly, GU2U or U2 GU links encounter obstacles, but the probability of obtaining line-of-sight paths is high owing to the mobility adjustment of UAVs. In contrast, U2U links are highly dynamic due to the mobility of UAVs. Both GU2U and U2U are very resource-constrained in terms of transmit power, computing, and bandwidth resources [15; 16]. On the other hand, U2H and H2S links have higher transmission delays, as these links have very long-distance free-space propagation  
Figure 1: An illustration of a UAV/SN-assisted mission. paths [17]. Additionally, owing to the mobility of LEO satellites, H2S links are dynamic in nature but predictable [18]. U2H, H2S, and S2S links are backhaul links for UAVSNs to route their sensed data to the cloud server and ensure global access [19]. In [13; 20], UAVs are classified into fixed-wing, rotary-wing, and tethered UAVs. Rotary-wing UAVs can support vertical take-off and landing, whereas fixed-wing UAVs can move very fast with more payload capacity and endurance [21; 22]. The type of UAVs should be selected based on the mission and application requirements.