# Personalized Federated Learning with Progressive Local Training Strategy and Lightweight Classifier
## 3 Methodology
### Local Training with Lightweight Classifier  
The model's ability to generalize is a key factor that directly influences its performance and can be effectively enhanced by reducing the upper bound on its generalization. To quantify the upper bound on generalization, we apply the theory of Rademacher complexity as outlined in [44]. Let \(\mathcal{G}\) be a set of functions mapping \(\mathcal{Z}\rightarrow[a,b]\), and let \(\text{S}=(Z_{1},\ldots,Z_{n})\) be i.i.d. random variables on \(\mathcal{Z}\), drawn from some distribution, \(P\). The empirical Rademacher complexity of \(\mathcal{G}\) with respect to the sample \((Z_{1},\ldots,Z_{n})\) is defined as  
\[\widehat{\mathfrak{R}_{S}}(\mathcal{G}):=\mathbb{E}_{\sigma}\left[\underset{g \in\mathcal{G}}{sup}\frac{1}{n}\sum_{i=1}^{n}\sigma_{i}g(x_{i})\right], \tag{5}\]  
where \(\sigma=(\sigma_{1},...,\sigma_{n})^{T}\) and \(\sigma_{i}\backsim unif\{-1,1\}\), which are known as Rademacher random variables. Moreover, with a probability of at least \(1-\delta\), we have, with respect to that drawn from \(S\),  
\[\forall g\in\mathcal{G},\mathbb{E}[g(\mathcal{Z})]\leq\frac{1}{n}\sum_{i=1}^{ n}g(x_{i})+2\widehat{\mathfrak{R}_{S}}(\mathcal{G})+3(b-a)\sqrt{\frac{log(2/ \delta)}{2n}}, \tag{6}\]  
where the right-hand side of the equation is the upper bound on the generalization of the model. As evident from Equation (6), the upper bound on the model's generalization capability is determined by three key factors: the empirical risk, the Rademacher risk, and the combined effect of the sample size and the number of categories. For neural networks, the Rademacher risk has a bound with an explicit dependence on the dimension, which is primarily influenced by the model complexity [45]. Inspired by the above theory, reducing the model's complexity can lower the Rademacher risk and improve the model's generalization ability. The classifier head, located at the end of a deep learning model, is responsible for mapping extracted features to class predictions. While it can fit local data distributions and features well, it is prone to overfitting local data [46]. In the pFL framework, the classifier is locally trained and updated, causing its parameters to become overly complex and leading to a significant increase in the model complexity. This increase in complexity tends to raise the Rademacher risk and exacerbate the overfitting phenomenon. To address this issue, we propose to introduce lightweight classifiers to limit the complexity of the model and thus reduce the Rademacher risk and then improve the generalization performance of the model. Regularization techniques, commonly used in machine learning, help reduce overfitting and improve model generalization. Therefore, we propose adding a classifier regularization term during the local training process in pFL to alleviate the degree of overfitting of the classifier and improve the model performance. In this section, we propose utilizing L2 regularization to enhance the training process of local models. By applying the L2 norm to the classifier, we can control its parameter density, making it relatively sparse. This approach helps achieve the goal of creating lightweight classifiers. Building on the original loss function, we add an L2 regularization term for the lightweight classifier, controlled by the hyperparameter \(\lambda\). Thus, the local loss consists of two parts: the empirical loss, which we use the cross-entropy loss for, and the regularization loss. For the total loss, we define it as follows:  
\[\mathcal{L}_{local}=\mathcal{L}_{CE}+\lambda\mathcal{L}_{REG}, \tag{7}\]  
In the above equation, \(\mathcal{L}_{CE}\) represents the cross-entropy loss, and \(\mathcal{L}_{REG}\) represents the regularization loss. Here, we define the mapping function from the features to the classifier as \(h\). Thus, the classification output for data x can be represented as \(h_{\phi}(f_{\theta}(x))\).