# Personalized Federated Learning with Progressive Local Training Strategy and Lightweight Classifier
## 3 Methodology
### The Convergence of PFPS-LWC  
To prove the convergence of PFPS-LWC, we begin by introducing the following assumptions.  
**Assumption 1**.: _Lipschitz Smoothness. The gradients of client \(i\)'s local complete heterogeneous model \(w_{i}\) are \(L1-\text{Lipschitzsmooth}\):_  
\[\begin{split}\left\|\
abla\mathcal{L}_{i}^{t_{1}}\Big{(}w_{i}^{ t_{1}};x,y\Big{)}-\
abla\mathcal{L}_{i}^{t_{2}}\Big{(}w_{i}^{t_{2}};x,y\Big{)} \right\|\leq L_{1}\|w_{i}^{t_{1}}-w_{i}^{t_{2}}\|,\\ \forall t_{1},t_{2}>0,i\in\{0,1,...,N-1\}_{\sigma}(x,y)\in D_{i} \end{split} \tag{11}\]  
_The above formulation can be further derived as_  
\[\mathcal{L}_{i}^{t_{1}}-\mathcal{L}_{i}^{t_{2}}\leqslant\left\langle\
abla \mathcal{L}_{i}^{t_{2}},\Big{(}w_{i}^{t_{1}}-w_{i}^{t_{2}}\Big{)}\right\rangle +\frac{L_{1}}{2}\|w_{i}^{t_{1}}-w_{i}^{t_{1}}\|_{2}^{2} \tag{12}\]  
The smoothness assumption guarantees that the gradient updates are controlled and do not cause erratic or unstable model updates, which could hinder convergence. It provides a theoretical foundation for maintaining stability during federated model aggregation and facilitates the model's smooth convergence.  
**Assumption 2**.: _Unbiased Gradient and Bounded Variance. Client \(i\)'s random gradient \(g_{i}^{\text{tr},i}=\
abla\mathcal{L}_{i}^{t}\big{(}w_{i}^{t},\mathcal{B}_{i} ^{t}\big{)}(\mathcal{B}\) is a batch of local data) is unbiased:_  
\[\mathbb{E}_{\mathcal{B}_{i}^{t}\subseteq D_{i}}\Big{[}g_{i}^{\text{tr},i} \Big{]}=\
abla\mathcal{L}_{i}^{t}\big{(}w_{i}^{t}\big{)} \tag{13}\]  
_and the variance of the random gradient \(g_{i}^{\text{tr},i}\) is bounded by_  
\[\mathbb{E}_{\mathcal{B}_{i}^{t}\subseteq D_{i}}\Big{[}\left\|\
abla\mathcal{ L}_{i}^{t}(w_{i}^{t};\mathcal{B}_{i}^{t})-\
abla\mathcal{L}_{i}^{t}(w_{i}^{t}) \right\|_{2}^{2}\Big{]}\leq\sigma^{2} \tag{14}\]  
This assumption ensures that the model's optimization process is based on correct and stable gradient updates. It also prevents the introduction of large variance during training, which would otherwise affect the stability of the global model aggregation and hinder convergence.  
**Assumption 3**.: _Bounded Parameter Variation. The parameter variations of the homogeneous feature extractor \(\theta_{i}^{t}\) and \(\theta^{t}\) before and after aggregation are bounded as_\[\|\theta^{t}-\theta^{t}_{i}\|\leq\delta^{2} \tag{15}\]  
This assumption ensures that despite local data heterogeneity, the model updates remain manageable and do not cause large disparities between local and global models. It helps in maintaining the stability of the federated learning system by ensuring that parameter updates are consistently within a reasonable range, facilitating convergence and preventing instability during model aggregation.  
Using Assumptions 1 and 2, we can establish the following lemma.  
There is an upper bound on the loss range of any client's local model, \(w\), in the \(t\) local training round.  
\[\mathbb{E}[\mathcal{L}_{t+1}^{E+0}]\leq\mathcal{L}_{t}^{E+0}+(\frac{L_{1}\eta ^{2}}{2}-\eta)\sum_{\epsilon=1}^{E}\left\|\
abla\mathcal{L}_{t}^{E+\epsilon} \right\|_{2}^{2}+\frac{L_{1}E\eta^{2}\sigma^{2}}{2} \tag{16}\]  
Leveraging Assumption 3 and Lemma 1, we derive Lemma 2.  
For the \((t+1)\) local training round, the loss of any client before and after aggregating local homogeneous small feature extractors on the server is bounded by  
\[\mathbb{E}[\mathcal{L}_{t+1}^{E+0}]\leq\mathbb{E}[\mathcal{L}_{t}^{E+1}]+\eta \delta^{2} \tag{17}\]  
By synthesizing Lemma 1 and 2, we ultimately arrive at the following conclusion:  
\[\eta<\frac{2(\epsilon-\delta^{2})}{L_{1}(\epsilon+E\sigma^{2})} \tag{18}\]  
Given that \(\epsilon\), \(L1\), \(\delta^{2}\), \(\sigma^{2}\), and \(E\) are all positive constants, it follows that \(\eta\) has well-defined solutions. Consequently, when the learning rate \(\eta\) satisfies the aforementioned condition, convergence is assured for any client's local complete heterogeneous model. The detailed certification process is presented in A.