###### Abstract  
Federated learning (FL) offers a privacy-preserving framework for distributed machine learning, enabling collaborative model training across diverse clients without centralizing sensitive data. However, statistical heterogeneity, characterized by non-independent and identically distributed (non-IID) client data, poses significant challenges, leading to model drift and poor generalization. This paper proposes a novel algorithm, pFedKD-WCL (Personalized Federated Knowledge Distillation with Weighted Combination Loss), which integrates knowledge distillation with bi-level optimization to address non-IID challenges. pFedKD-WCL leverages the current global model as a teacher to guide local models, optimizing both global convergence and local personalization efficiently. We evaluate pFedKD-WCL on the MNIST dataset and a synthetic dataset with non-IID partitioning, using multinomial logistic regression (MLR) and multilayer perceptron models (MLP). Experimental results demonstrate that pFedKD-WCL outperforms state-of-the-art algorithms, including FedAvg, FedProx, PerFedAvg, pFedMe, and FedGKD in terms of accuracy and convergence speed. For example, on MNIST data with an extreme non-IID setting, pFedKD-WCL achieves accuracy improvements of 3.1%, 3.2%, 3.9%, 3.3%, and 0.3% for an MLP model with 50 clients compared to FedAvg, FedProx, PerFedAvg, pFedMe, and FedGKD, respectively, while gains reach 24.1%, 22.6%, 2.8%, 3.4%, and 25.3% for an MLR model with 50 clients. p +
Footnote †: journal: _Algorithms_  
e1e1e2e1e3e1e4e  
data is similarly distributed, FedAvg struggles with non-IID data, as local optima diverge from the global target, leading to poor performance on individual clients [8]. Conversely, relying solely on individual learning without FL collaboration leads to poor generalization as well, as clients lack sufficient data to train robust models independently [9]. To overcome this, personalized FL has gained attention, aiming to tailor models to each client's unique data while still benefiting from collaborative learning [10]. Personalized FL seeks to strike a balance between global knowledge sharing and local adaptation, a task complicated by the need to maintain privacy and efficiency in heterogeneous settings [11]. Numerous strategies have been developed to enable personalization in FL. For example, local customization techniques, such as FedProx [4], modify FedAvg by introducing regularization to align local updates with the global model. FedPer [12] splits neural networks into shared base layers trained server-side and personalized layers fine-tuned by clients. Bi-level optimization offers another avenue, exemplified by pFedMe [13], which separates global model aggregation from local personalization, using the global model as a reference point to guide client-specific solutions, an approach that proves effective, but less so with limited data. Meta-learning strategies, like PerFedAvg [14], draw from Model-Agnostic Meta-Learning to craft an initial shared model that clients can quickly adapt, though computational challenges arise from Hessian approximations [15]. Multi-task learning contributes frameworks like MOCHA [16], which tackles both statistical and system-level diversity by treating clients as distinct tasks. Meanwhile, methods like APFL [17] blend local and global models adaptively, balancing personalization with collaboration. Clustering-based approaches, such as those proposed by [PERSON] et al. [18], group clients with similar data distributions to improve personalization efficiency. Despite these advances, many PFL approaches grapple with overfitting when client datasets are small, underscoring the need for robust solutions tailored to data scarcity [19]. Knowledge Distillation (KD), introduced by [PERSON] et al. [20], provides another powerful tool to enhance FL under heterogeneity.