## 7 Defenses against Privacy Attacks
### Knowledge Distillation  
Knowledge distillation (KD) involves transferring knowledge from a large model (the \"teacher\") to a smaller model (the \"student\") by training the student to mimic the teacher's  
Figure 15: Visual representation of homomorphic encryption.  
predictions [145]. This technique helps to create efficient yet accurate models suitable for various applications [146]. A visual diagram of the approach is shown in Figure 16.  
In FL, KD plays a crucial role in enhancing the model performance by enabling the creation of efficient student models directly on decentralized devices [147]. A powerful teacher model, trained either centrally or collaboratively, distills its knowledge to student models located on individual devices [148; 149]. This approach alleviates the computational burden on edge devices while preserving high accuracy and generalization, thereby optimizing the overall efficiency of FL systems [150].  
FedKD (Federated Knowledge Distillation) [151] further advances privacy preservation by focusing on the exchange of updates from smaller \"mentee\" models instead of transmitting full, large mentor models. This approach significantly reduces the volume of information shared, thereby minimizing potential privacy risks. Moreover, FedKD employs gradient encryption before uploading the local gradients to the server, ensuring that the sensitive patterns within the gradients are obscured and protected. Gradient compression using techniques like Singular Value Decomposition (SVD) also contributes to minimizing sensitive information leakage. FedKD dynamically adjusts compression precision, prioritizing higher precision during convergence when gradients contain less private information. This holistic approach in FedKD ensures robust privacy preservation while maintaining effective model performance.  
Additionally, the FEDGEN framework [152] emphasizes privacy by focusing on learning a generator model solely from the prediction layers of the local user models, thereby avoiding direct access to private input data. By sharing only prediction layers rather than entire models, FEDGEN reduces privacy risks as the prediction layers contain less information about the underlying private training data. The generator model in FEDGEN operates within a compact latent space, significantly less information-dense than high-dimensional input spaces, further minimizing the information shared and lowering the privacy breach risks. An extended version of FEDGEN retains the feature extraction layers locally while sharing only prediction layers, reducing the communication overhead and enhancing the privacy protection by keeping sensitive data securely on local devices.  
Similarly, the FedFTG framework [153] innovatively enhances privacy through data-free knowledge distillation. FedFTG uses pseudo-data generated by models instead of real client data, ensuring that the actual data are never transmitted, thus preserving user privacy. The pseudo-data capture only high-level patterns and cannot reconstruct the original training samples, safeguarding the individual client data attributes. This method effectively keeps real data local and private, mitigating privacy breach risks.