# FedSeq: Personalized Federated Learning via Sequential Layer Expansion in Representation Learning
## 2 Related Work
### Personalized Federated Learning  
Personalized federated learning (PFL) has been proposed to address the heterogeneity of client data and can be categorized into several approaches such as clustering, meta-learning, and representation [9]. This section compares four related personalization methods, including representation learning, which are considered similar. Table 1 provides a detailed summary of these personalization methods, highlighting their techniques, descriptions, and limitations.  
\begin{table}
\begin{tabular}{p{108.4 pt} p{108.4 pt} p{108.4 pt} p{108.4 pt}} \hline \hline
**Approach** & **Technique** & **Description** & **Limitations/Drawbacks** \\ \hline \multirow{4}{*}{Meta Learning} & MAML [10] & Adapts to new tasks with minimal updates for personalization. & High computational cost and inefficient scaling reduce efficiency in complex environments. \\ \cline{2-4}  & FOMAML [11] & First-order approximation of MAML to reduce complexity. & Less accurate than second-order differential-based meta-learning performance. \\ \cline{2-4}  & MetaVers [12] & Enhances robustness across clients with distance-based learning. & High complexity and computational overhead. \\ \hline \multirow{4}{*}{Transfer Learning} & FedMD [13] & Model distillation for heterogeneous model collaboration. & The model performance is affected by the quality of the public dataset used in the model distillation process. \\ \cline{2-4}  & FedHealth [14] & Uses pre-trained models for wearable healthcare. & The heterogeneity and noise in wearable data make the model performance unstable. \\ \hline \multirow{4}{*}{Multi-Task Learning} & Federated MTL [15] & Personalizes models while sharing a global structure. & High communication and computational costs, and the risk of overfitting in some clients can pose challenges. \\ \hline \multirow{4}{*}{Representation Learning} & FedPer [16] & Adds client-specific layers for personalized learning. & Balancing personalization and model size is challenging. \\ \cline{2-4}  & LG-FedAvg [17] & Each client learns features tailored to local data while sharing information with the global model to enhance overall performance. & Balancing local and global representations is challenging, with data heterogeneity and high computational and communication costs. \\ \cline{2-4}  & FedRep [18] & Uses multiple local updates for client-specific classifiers. & Additional personalization processes can increase computational costs, making them inefficient for resource-constrained devices. \\ \hline \hline \end{tabular}
\end{table}
Table 1: Summary of Personalization Methods in Federated Learning.  
\begin{table}
\begin{tabular}{c c c c} \hline
**Approach** & **Technique** & **Description** & **Limitations/Drawbacks** \\ \hline \multirow{6}{*}{Representation Learning} & FedROD [19] & Improve the global model using Balanced Risk Minimization (BRM) loss function to address class distribution imbalances & High computational and communication costs make it inefficient in resource-constrained environments. \\ \cline{2-4}  & FedBABU [20] & By updating only the base (feature extractor) and not the head (classifier), it reduces computational costs and achieves efficient personalization. & Enforces a uniform base representation for all clients, which may not be suitable for highly heterogeneous data. \\ \cline{2-4}  & FedCP [21] & Separates global and personalized features using a Conditional Policy Network, it is trained and prevents the collision of feature information. \\ \cline{2-4}  & FedCRL [22] & Uses contrastive learning for non-IID data personalization to address label heterogeneity. & High communication and computational overhead, along with sensitivity to incorrect label configurations, can affect the efficiency of contrastive learning-based mechanisms. \\ \hline \end{tabular}
\end{table}
Table 1: Cont.