## 2 Foundational Concepts
### Privacy-Preserving Techniques  
FL's privacy benefits can be further enhanced using PETs, with significant advantages and disadvantages, particularly relevant in the resource-constrained IoT context:  
Differential Privacy (DP): DP provides a formal, mathematical definition of privacy guarantees [45; 46]. A randomized mechanism \(\mathcal{M}\) satisfies \((\epsilon,\delta)\)-DP if, for any two adjacent datasets \(D_{1}\) and \(D_{2}\) (differing by at most one element), and for any possible subset of outputs \(S\), the following inequality holds:  
\[\mathbb{P}[\mathcal{M}(D_{1})\in S]\leq e^{\epsilon}\mathbb{P}[\mathcal{M}(D_{ 2})\in S]+\delta, \tag{2}\]  
where \(\epsilon\) is the privacy budget, and \(\delta\) represents the probability that the strict \(\epsilon\)-DP guarantee might be violated. The privacy budget (\(\epsilon\)) is a fundamental concept in differential privacy that quantifies the maximum amount of information leakage or privacy loss permitted in a privacy-preserving mechanism. A smaller privacy budget (a smaller \(e\) value) indicates stronger privacy by limiting the influence of any single data point on the output, thereby making it harder to infer individual information. However, this typically results in lower utility of the data or model, as more noise is often required to achieve stronger privacy. In the context of Federated Learning with LLMs, the privacy budget must be carefully managed across multiple rounds of training and participating clients to balance privacy protection with model performance, especially in sensitive IoT applications like healthcare or smart homes. For \(\delta\), it is typically set to a very small value (e.g., less than the inverse of the dataset size \(|D|\)), representing a small probability that the pure \(\epsilon\)-DP guarantee is broken. This definition ensures that the output distribution of the mechanism is statistically similar regardless of the presence or absence of any single individual's data [47]. DP guarantees are commonly achieved by adding carefully calibrated noise (e.g., following a Gaussian or Laplace distribution) to function outputs, gradients, or model updates, as implemented in algorithms like DP-SGD [48]. DP offers strong, mathematically rigorous privacy guarantees against inference attacks. Its computational overhead is generally lower compared to cryptographic methods like HE or SMPC. However, a key challenge of DP is the inherent trade-off between privacy and utility, where increasing noise (reducing \(\epsilon\)) to enhance privacy typically degrades model accuracy [49], as conceptually illustrated in Figure 3. This figure compares the relative computational and communication overheads of various privacy-preserving techniques in FL. It highlights that while DP introduces additional costs, its overhead remains modest compared to more complex methods like SMPC and HE. Notably, homomorphic encryption incurs the highest total overhead, underscoring the practicality of DP in resource-constrained edge scenarios. Managing privacy budgets effectively across rounds and clients is complex [50; 51; 52], and DP noise can disproportionately affect fairness for underrepresented groups [3]. Homomorphic Encryption (HE): HE allows specific computations (e.g., addition for averaging updates) on encrypted data [53]. The server aggregates ciphertexts without decrypting them. The advantage of HE lies in the fact that it provides strong confidentiality against the server (server learns nothing about individual updates), hence no impact on model accuracy (utility) compared to non-private aggregation. However, HE can have extremely high computational overhead for encryption/decryption and homomorphic operations, significantly expanding the communication data size (ciphertext size). Thus, HE is currently impractical for direct implementation on most resource-constrained IoT devices [7]. Figure 3: Conceptual illustration of the privacy–utility trade-off in DP.