## 7 Research Gaps and Future Directions  
The IoT, LLMs, and FL have seen rapid progress, establishing a notable current state of development and research. However, despite these advancements, substantial challenges persist. This section aims to provide a structured overview by first briefly acknowledging key aspects of the current landscape within specific domains of this integration. Building on this, we then identify critical research gaps, supported by detailed evidence and insights from recent literature. Finally, based on these identified gaps, we delineate promising future directions for advancing the synergistic application of these technologies. Efficiency for Extreme Edge: LLMs are notoriously resource-intensive, but edge IoT devices often operate on milliwatts of power with kilobytes of RAM. Techniques like QLoRA [62] reduce fine-tuning memory use by combining 4-bit quantization and low-rank adaptation, making LLMs tractable for edge execution. Similarly, SparseGPT achieves one-shot pruning with negligible accuracy drop on billion-parameter models [67]. SmoothQuant  
Figure 6: Illustration of the privacy–utility trade-off under differential privacy for Federated Learning models. The plot compares the model accuracy (%) of two different DP-based optimization methods, DPSGD and DP-Adam, across a range of privacy budgets (\(\epsilon\)) on a logarithmic scale. As the privacy guarantee strengthens (smaller \(\epsilon\)), model performance consistently degrades, highlighting the inherent trade-off between privacy protection and predictive utility. Key privacy levels (\(\epsilon\) = 0.01, 0.1, 1, and 10) are annotated to demonstrate performance sensitivity. enhances post-training quantization by aligning activations and weights to improve stability under int8 quantization [68]. Backpropagation-free training is emerging as a potential direction to eliminate memory-heavy gradient calculations; the survey in [87] reviews biologically inspired and forward-forward alternatives relevant to constrained hardware. These are particularly promising when combined with hardware-aware co-design, as advocated in [3], for FL in 6G IoT networks. Robustness to Heterogeneity and Fairness: Extreme client heterogeneity in IoT-FL, both in data and hardware, poses serious convergence and fairness challenges. [PERSON] et al. [24] analyze system-level disparities and advocate for client-specific adaptation layers. [PERSON] et al. [37] further highlight how adversarial alignment in neural networks can propagate biases, underscoring the need for fairness constraints in model design. Multi-prototype FL, as discussed in the Wevolver report [12], enables clients to specialize on subsets of prototypes that better represent their local distributions. [PERSON] et al. [73] propose a hierarchical knowledge transfer scheme that separates global, cluster, and local models, reducing the negative transfer from outlier clients. Formal fairness-aware FL protocols, however, are still lacking. Practical Privacy Guarantees: Applying PETs to LLM-based FL is non-trivial. While traditional DP mechanisms such as those in [45; 48] remain foundational, [PERSON] et al. [49] show that when applied to LLMs in FL, DP introduces substantial performance degradation unless combined with hybrid masking and adaptive clipping strategies. [PERSON] et al. [70] propose DP-LoRA, which selectively adds noise only to low-rank adaptation matrices, achieving a trade-off between utility and formal privacy. Yet, computational cost remains high. HE and SMPC offer stronger privacy but with significant communication and computational overheads unsuitable for IoT [53; 55]. Efficient and scalable PET integration into low-power FL deployments remains an open issue. Advanced Security and Trust: Foundation models open new attack surfaces in FL. [PERSON] et al. [118] demonstrate that compromised foundation models can inject imperceptible backdoors into global models during federated fine-tuning. [PERSON] et al.