### Key Contributions  
The major contributions of this paper are summarized as follows:  
* The evolution of recently proposed state-of-the-art actor-critic frameworks is extensively reviewed based on their internal neural network architecture, optimization objectives, control variables with constraints, POMDP formulation, training methods, policy gradient, modifications in the loss function, key contributions, advantages, and limitations in addressing joint optimization problems in the UAVSNs environment. * A novel taxonomy for recently proposed multi-agent actor-critic frameworks is provided based on their internal neural network structure to address joint optimization problems in UAVSNs while considering reasonable key constraints. \begin{table}
\begin{tabular}{c c c} \hline \hline
**Ref.** & **Year** & \multicolumn{2}{c}{**Key Contributions**} \\ \hline
[35] & 2023 & Existing distributed machine learning algorithms are reviewed and classified as federated learning and multi-agent reinforcement learning, to optimize trajectory and resource allocation in UAVSNs. \\ \hline
[13] & 2022 & Existing trajectory and topology control algorithms in UAVSNs are reviewed and classified based on their underlying mathematical model. \\ \hline
[36] & 2024 & Existing edge computing algorithms for metaverse environments are reviewed based on optimization techniques. \\ \hline
[37] & 2022 & Existing edge computing algorithms for UAVSNs are reviewed to support various artificial intelligence-based applications in computer vision, navigation, power control, security, and communication. \\ \hline
[38] & 2024 & The application of transformer-based encoder–decoder networks in UAV-assisted wireless networks is investigated to perform compression, schematic communication, and channel estimation. \\ \hline
[39] & 2024 & Existing task offloading algorithms in UAVSNs are reviewed by classifying them based on value-based and policy-based methods. \\ \hline
[40] & 2024 & Existing trajectory-aware tasking offloading algorithms are reviewed by classifying them into general iterative optimization and reinforcement learning-based optimization methods. \\ \hline
[41] & 2023 & Existing data aggregation techniques are reviewed for UAVSN-assisted IoT devices. \\ \hline
[11] & \(\bullet\) & An overview of UAVSN environments is provided, focusing on mission-oriented optimization objectives, control variables, available links, and the potential to adopt actor-critic-based MA-DRL frameworks. \\ \(\bullet\) & The evolution of existing actor–critic frameworks used to solve joint optimization problems in UAVSNs is extensively studied and classified based on modifications to the internal actor–critic neural networks. \\ \(\bullet\) & Additionally, a comprehensive comparative study is presented, considering optimization objectives, control variables with constraints, POMDP formulation, training methods, modifications in policy gradient, loss function, state embedding, advantages, and limitations in addressing joint optimization problems in UAVSNs. \\ \(\bullet\) & Key open research issues are discussed, along with potential solutions for UAVSNs. \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Comparison of key contributions in existing survey articles related to this survey. It will facilitate researchers to choose and design appropriate actor-critic frameworks with internal neural network cell and activation function, policy gradient, loss, and reward with penalty terms for obtaining optimal solutions in complex UAVSNs environment with reasonable training in a high-fidelity digital twin environment. Additionally, it will assist researchers in understanding how to extract key features from large dynamic time series state parameters in UAVSNs, facilitating improved decision-making in multi-agent cooperative-competitive UAVSN environments. * According to the comparative study, key open issues and research challenges are identified.