# FedSeq: Personalized Federated Learning via Sequential Layer Expansion in Representation Learning
## 3 Proposed Algorithm  
In this section, this work presents our proposed scheduling algorithms in detail, namely _Vanilla_ Scheduling and _Anti_ Scheduling. An overview of the proposed algorithm is shown in Figure 1. The proposed algorithm executes the following steps: (1) model distribution; (2) local training via sequential layer expansion (Vanilla or Anti Scheduling); (3) update the server with the refined base parameter \(\theta_{i,b}\); (4) aggregate global model \(\theta_{G}\). Then, steps (1)-(4) are repeated up to the predefined condition. Finally, the server fine tunes with base parameter \(\theta_{i,b}\) and head parameter \(\theta_{i,h}\). As shown in Figure 2, Vanilla Scheduling starts by unfreezing the shallowest layers (closest to the input) and progressively moves towards deeper layers. This method allows the model to capture low-level features early in the training process, which is crucial for building a robust foundational understanding before more high-level features are introduced. Conversely, Anti Scheduling begins with the deepest layers (closest to the output) and progressively unfreezes toward the input. This approach prioritizes the learning of high-level features from the outset, which can be beneficial for complex pattern recognition tasks that depend heavily on such features. As shown in Figure 2, the structured progression from shallow to deep layers in Vanilla Scheduling, and the reverse in Anti Scheduling, structurally differ from traditional representation learning. In traditional representation learning methods, the base and head of the model are typically trained simultaneously without specific layer prioritization. In our approach, however, each layer's training is specifically timed and prioritized. This strategic scheduling improves the model's ability to adapt to the varying complexities of features throughout the training process. \begin{table}
\begin{tabular}{l l l l l} \hline \hline
**Technique** & **Computation** & **Head Layer** & **Scalability** & **Flexibility of** \\  & **Efficiency** & **Utilization** & & **Personalization** \\ \hline FedPer [16] & ✓ & Base + Head & Moderate & High \\ \hline LG-FedAvg [17] & \(\times\) & Base + Head & Low & Moderate \\ \hline FedRep [18] & \(\times\) & Base + Head & Moderate & High \\ \hline FedROD [19] & \(\times\) & Base + Head & Low & Moderate \\ \hline FedBABU [20] & ✓ & Base Only & High & Moderate \\ \hline FedCP [21] & \(\times\) & Base + Head & Moderate & High \\ \hline FedCRL [22] & \(\times\) & Base + Head & Low & Moderate \\ \hline FedSeq (Proposed) & ✓ & Base Only & High & High \\ \hline \hline \end{tabular}
\end{table} 
Table 2: Comparison of Representation Learning Methods. Furthermore, as detailed in Section 5.2, Vanilla Scheduling can achieve comparable accuracy with significantly lower computational costs in environments characterized by high data and class heterogeneity. In contrast, Anti Scheduling excels in achieving the highest accuracy in scenarios where both data and class heterogeneity are pronounced. They both follow the basic federated learning setup and the representation learning as explained below. The list of symbols used in the paper are shown in Table 3.  
Figure 1: Overview of FedSeq. Figure 2: Illustration of the proposed Vanilla and Anti Scheduling algorithms of FedSeq. The **upper** shows Vanilla Scheduling, starting with the shallowest layer and advancing to deeper ones. The **lower** shows Anti Scheduling, which begins with the deepest layer and progresses inversely. As a basic federated learning setup, this work assumes that each client \(C_{i}\) possesses data \(D_{i}=(x_{i},y_{i})\in R^{d}\), where \(i\in\{1,2,\ldots,N\}\) represents \(i\)-th client out of a total of \(N\) clients and \(d\) represents the input dimension.