## 4 Research Challenges and Directions
### Joint Model-Based and Model-Free MA-DRL Algorithms Development for UAVSNs  
UAVSN-assisted missions, such as joint tasks or data collection, beam forming, task offloading, resource allocation (e.g., computing, caching, power, bandwidth, or timeslot), and trajectory optimization, involve large and dynamic state-action spaces with stringent short- and long-term constraints related to QoS and dynamic resource availability [77]. To address such joint optimization problems efficiently, formulating sub-problems is necessary to reduce computational complexity and enable real-time decision-making. A combination of model-based optimization and MA-DRL algorithms is highly suitable for these situations. Specifically, long-term constraints and queue stability under energy constraints can be simplified using the Lyapunov drift-plus-penalty method [78; 79; 80; 7]. Alternatively, problems that allow for convex relaxation can be solved iteratively through model-based Lagrange relaxation, leveraging Karush-Kuhn-Tucker conditions and an adaptive step-size-based gradient descent method [81]. Simultaneously, control parameters requiring adaptive decision-making in dynamic environments with decentralized knowledge can be addressed using a meta-learning-inspired adaptive actor-critic MA-DRL framework [65].  
During recursive state estimation for UAV trajectory prediction, particularly for tasks such as routing data packets [82] or task offloading, beamforming with neighboring UAVs [83] or between base stations and UAVs [84] can be enhanced by integrating model-free MA-DRL with model-based Kalman filters (e.g., extended Kalman filter or unscented Kalman filter) [85]. This hybrid approach can accelerate DRL convergence, improve optimal decision-making, and make it highly suitable for online execution. Kalman filter variations are well-suited for predicting nonlinear state dynamics, utilizing both motion models and noisy sensor data (e.g., inertial measurement units, gyroscopes, accelerometers, and light detection and ranging sensors) to enhance estimation accuracy [86]. Consequently, hybrid algorithms that combine model-based and model-free MA-DRL methods hold significant potential but require further investigations.  
Joint Energy-Harvesting and Resource Allocation for Task Computation or Data Collection Using UAVSNs  
UAVSNs can be leveraged to execute missions such as AoI-aware data collection or task computation for highly intensive, low-latency applications of low-power IoT devices. To extend the battery life of IoT devices, especially those located in remote areas,one effective solution is energy harvesting at IoT devices using low-altitude UAVs with proper beamforming. Since most IoT devices are equipped with a single antenna, timeslot optimization is required along with optimal resource allocation for energy harvesting from UAVs and data/task uploading to UAVs, subject to AoI and resource constraints. Existing algorithms primarily use convex approximations and iterative methods to solve such dynamic optimization problems [87; 88; 89; 90; 91]. These approaches often rely on centralized knowledge and involve numerically computing the gradient of the objective function with respect to the control variables, resulting in higher time complexity, limited scalability, and reduced adaptability to environmental changes. Moreover, most existing studies assume that energy harvesting is linear in nature, whereas, in reality, it is nonlinear and depends on channel state conditions. To address the joint energy harvesting and resource allocation problem for AoI-aware task/data collection or computation, further research is required to develop adaptive MA-DRL-based frameworks [92]. These frameworks should account for the nonlinear nature of energy harvesting and provide scalable, real-time solutions that adapt to dynamic environments.