Visual Prompt Based Personalized Federated Learning
Authors: Guanghao Li
Wansen Wu
Yan Sun
Li Shen
Baoyuan Wu
Dacheng Tao
Venue: Accepted by TMLR ()

Abstract: As a popular paradigm of distributed learning, personalized federated learning (PFL) allows personalized models to improve generalization ability and robustness by utilizing knowledge from all distributed clients. Most existing PFL algorithms tackle personalization in a model-centric way, such as personalized layer partition, model regularization, and model interpolation, which all fail to take into account the data characteristics of distributed clients. In this paper, we propose a novel PFL framework for image classification tasks, dubbed pFedPT, that leverages personalized visual prompts to implicitly represent local data distribution information of clients and provides that information to the aggregation model to help with classification tasks. Specifically, in each round of pFedPT training, each client generates a local personalized prompt related to local data distribution. Then, the local model is trained on the input composed of raw data and a visual prompt to learn the distribution information contained in the prompt. During model testing, the aggregated model obtains client-specific knowledge of the data distributions based on the prompts, which can be seen as an adaptive fine-tuning of the aggregation model to improve model performances on different clients. Furthermore, the visual prompt can be added as an orthogonal method to implement personalization on the client for existing FL methods to boost their performance. Experiments on the CIFAR10 and CIFAR100 datasets show that pFedPT outperforms several state-of-the-art (SOTA) PFL algorithms by a large margin in various settings. The code is available at: https://github.com/hkgdifyu/pFedPT.

Published in Transactions on Machine Learning Research (01/2024)
Visual Prompt Based Personalized Federated Learning
Guanghao Li National University of Defense Technology
lglt Breud edu.cn
Wansen Wu
wwwanssen @rudt edu.cn
National University of Defense Technology
Yan Sun The University of Sydney
wwodenchilll955oolloooocco
Shen JD Explore Academy
mathshendi@gmail.com
Baoyuan Wu
wwweeoyuun119870@maai..com
The Chinese University of Hong Kong, Shenzhen
Dacheng Tao
dacheng. too com
The University of Sydney & JD Explore Academy
Reviewed on OpenReview: https: // openreview. net/ forum? id= dUVejidX07
Abstract
As popular paradigm of distributed learning, personalized federated learning (PFT.) allows personalized models to improve generalization ability and robustness by utilizing knowledge from all distributed clients. Most existing PFL algorithms tackle personalization in a model- centric way, such as personalized layer partition, model regularization, and model interpola- tion, which all fail to take into account the data characteristics of distributed clients. In this paper, we propose a novel PFL framework for image classification tasks, dubbed pFedPT, that leverages personalized visual prompts to implicitly represent local data distribution information of clients and provides that information to the aggregation model to help with classification tasks. Specifically, in each round of pFedPT training, cach client generates a local personalized prompt related to local data distribution. Then, the local model is trained on the input composed of raw data and a visual prompt to learn the distribution information contained in the prompt. During model testing, the aggregated model obtains client-specifie knowledge of the data distributions based the prompts, which can be an adaptive fine-tuning of the aggregation model improve model performances dif- ferent clients. Furthermore, the visual prompt be added as orthogonal method to implement personalization on the client for existing FL methods to boost their performance. Experiments on the CIFAR10 and CIFAR100 datasets show that pFedPT outperforms sev- eral state-of-thes (SOTA) PFL algorithms by a large margin in various settings. The code is available at: https://gi com wwwm:///ggifff///eelelllol
1 Introduction
Personalized federated learning (PFL.) (Deng et al. 2020 Huang et al. 2022 Sattler et al. 2019 Shamsian
et al. 2021 is a novel paradigm proposed to overcome the impacts heterogeneity across isolated clients.
Instead of training single aggregated model like in Federated learning (FL) (Gao al.
2022 Li et al.
*Co-first authors
*Corresponding author

---

Published in Transactions on Machine Learning Research (01/2024)
Class
Class 2 Feature
Class 1 Feature Prompts
Class 2 Feature Prompts
Global Classiffer Proxy
Local Classifier Presy
Broadcasting and Local U pdate Server
Aggregation
Server
20-
30-
Intra-Var
Client
Client
Client
Clien: 2
(a) FedAvg
(c) FedAvg Feature
Broadcasting Local pdate
Aggregation
later-Var. 12.52
Server
cbrr
-10-
-20
Intra-Var (4): Intr-- 6.56
Client
Client
Client
Client 2
-20
(b) PFedPT
(d) PFedPT Feature
Figure 1: Differences in local update and aggregation phases between FedAvg and pFedPT. In the figure, the lines represent the decision boundaries defined by the backbone. Assume that each client has two classes represented by diffcrent shapes. (a) In FedAvg, due to the hetcrogencity of data in each client, the significant difference local updates affects the final model aggregation. The aggregation model doesn't 't fit well with the data on each client. However, (b) the pFedPT adds personalized visual prompts to the client data, which change the original data characteristics and improve the fit of the backbone on the client. The aggregation model can perform well on each client's data with prompt. (c) and (d) are the t-SNE visualization results of the final hidden layer trained by FedAvg and pFedPT on the client with only classes 1 and 4. pFedPT increases the inter-class variation (Inter-Vat and decreases the intra-class variation (Intra- Var).
McMahan et al., 2017; Tan et al. 2022, PFL generates personalized local model each client that is more in line with the local data distribution by jointly considering the aggregated model and the personalized data. There are two main challenges lying in PFL. One is how to extract useful global features from modols trained on each local heterogencous datasct. The othor is how to incorporate the extracted global features with the personalized features, yielding a better client-specific model.
2022
Several works have been proposed to address the above challenges from a model perspective. PFL algorithms with decoupling model (Arivazhagan et al. 2019 Collins al.) 2021 Oh al. 2021] split the local model into shared part to be aggregated with those from other clients, and private part of maintaining locality, The shared part is used to transfer public knowledge among clients, and the private part is used to adapt to local data distribution. Clustered FL (Dinh et al.) 2021) groups clients according to the similarity of the local parameters and trains an aggregated model for each group of clients. Clustered FL extracts common knowledge from similar clients within a group to generate unified model for the group. These methods, however, still fall short in two aspects. First, these approaches rely on the effectivencss of aggregating or clustering the shared parts and may fail with highly heterogeneous data. Second, these methods simply extract the common knowledge and implement the personalization at the model level, while ignoring the potential at the data level, which may further strengthen the personalized adaptation between the aggregated model and local dataset.
In the community of computer vision (CV), both visual prompts (Liu et al.) 2021 and adversarial repro- gramming (Elsayod et al.) 2018) employ set of learnable parameters as a continuous task-spocific vector, which can be tuned based on training data from the downstream task. Visual prompts can effectively help large-scale pre-trained model achieve fast task transfer by simply training task-related prompts without
2

---

Published in Transactions on Machine Learning Research (01/2024)
changing any pre-trained model parameters. The prompt parameters are like the attention guidance to implicitly hint at the lask-related information for improving model performance on the new task (Liu et al.; 2021). This motivates us to regard the different clients as different tasks and adopt client-specific prompts to fine-tune the aggregated model on each local client, which helps to incorporate the extracted global features with the personalized ones.
Based on this insight, we propose a novel PFL framework named pFedPT. Our approach addresses the shortcomings mentioned above by using visual prompt to implicitly provide a hint of the data distribution client for the aggregated model locally, Specifically, each client model integrates set of learnable Prompt parameters with a backbone participating in aggregation for classification. The prompt parameters can generate personalized visual prompts for its affiliated clients based on their local data distribution. During local training of PFedPT, the generated personalized visual prompts are added to the images. Fig. [ (a) and (b) show the difference in the training process between FodAvg and pFedPT. For different classes of data, Fig. | (c) and (d) show that the generated prompts increases the inter-class variation (Inter- Var) while decreasing the intra-class variation (Intra-Var)). Different, class data with visual prompt is easily distinguished by an aggregated backbone, thereby improving the local performance of the local clients. Then, the backbone is trained on the input composed of raw data and visual prompts to learn the distribution information contained in the prompt. Upon achieving convergence of the two models through alternate training, the backbone implements the extraction of common knowledge from clients and can recognize the visual prompts of different clients. The generated visual prompt reflects the client's characteristics as a cindi-non vector and implements fine-tuning of the backbone in the local client. As a result, the backbone can capture implicit knowledge about the client's 's data distribution based the visual prompt and therefore obtain better-personalize interest and added as plugin for other FL algorithms. It can fine-tune the model received by clients, which can implement the personalized improvement of the model trained by FL algorithms in different clients or further boost the performance of PFL algorithms.
model. On the other hand, the visual prompt can be of independent
We validate pFedPP on two image classification datasels, including CIFAR10 (Kizhevsky el al. 2009) and CIFAR100 (Krizhevsky et al., 2009). Empirical results show that pFedPT beats other SOTA methods of PFL with 1%-3% improvement in test accuracy. In summary, our main contributions are four-fold:
We propose a novel PFL framework, dubbed pFedPT, for federated image classification tasks that use the visual prompts from each client to fine-tune the aggregated model and imbue the aggregated local model with information about the local data distribution.
PFedPT distinguishes itself with a user-oriented design that necessitates only the addition of prompt parameters. pFedPT stands out for its user-centric design. We show that pFedPT can integrate with several existing FL and decoupled PFL methods to boost their performance, which may be of independent interest.
We conduct extensive experiments to evaluate the effectiveness of pFedPT, which significantly out- performs several SOTA baselines on CIFAR10 and CIFAR100 datasets. Besides, the experimental results illustrate that the prompt can indeed learn personalized knowledge related to the client.
2 Related Work
Personalized Federated Learning (PFL). PFL has drawn significant research interests (Cho et al.
2021 Dai et al.] 2022: Fallah et al. 2020 Hanzely et al. 2021 Li et al. 2023 Shi et al.] 2023] Dinh
2023 The main difficulty of PFL is to characterize the data distributions of clients and integrate them into the federated learning training process, followed by providing personalized local model for each client. Currently, the core idca of PFL is to decouple the model into shared layers for feature extraction and personalized layers for classification (Arivazhagan el al. 2201 Collins let al. 2021: Oh et al.) 2021). Each client's s parameters of the shared layer are generally updated globally using the FodAvg (McMahan et al.) 2017) algorithm. In contrast, the personalized layers are trained locally and will not be shared with others. Those works focus on training a general feature extractor and a personalized classifier head for personalization.
et al. 2020 Tan et al. 2022 Wang et al.
3

---

Published in Transactions on Machine Learning Research (01/2024)
Local Training Stage
3
Sending Model Clesk
Cliaar
Cleen
Prompt
Genee
Server
Frozen
Tuned
Madel Aggregation
Local Training Stage
Adapting Model
Piompt
The pipeline of the pFedPT. 0 stands for the predicted logits of all classes. The dashed lines in steps 1 and 2 represent the loss backward for the model update. Each client contains a set of personalized learnable parameters preserved locally, and a Backbone, which the server will aggregate with those other clients. The raw image input will be added to visual prompt (colored pixels padded around the image) and then passed into the backbone for prediction.
Figure 2:
Other work aims to combine other related machine-learning techniques with PFL. Briggs et al. (2020) and Mansour al. (2020) use the clustering technique to divide similar clients into groups and learn a separate model for each group without inter-group federation. Dinh et al. (2021) use multitasking learning to take advantage of shared representations between clients to improve the generalization performance of each model. Yang et al. (2020) and Chen et al. (2020) use transfer learning to enhance local models by transferring knowledge between relevant clients. ' Dinh et al. (2020) add regularizers to the aggregated model to prevent. customers' models from simply overfitting their own data sels. Chen el al. (2018) and Fallah et al. (2000) attempt to develop a well-initialized shared aggregated model using a model-agnostic meta-learning (MAML) approach (Finn et al. 2017). In addition, fine-tuning using the aggregated model Icarned by the FedAvg algorithm can also improve the performance of personalized local models (Huang al. 2023 Jiang
al. 2.19).
The previous works enable the model to recognize the characteristies of clients and implement
personalization for them. There are also some other works (Chen et al. 2022 Goetz & Tewari 2020 Hao
ct al. 2021) that focus on the data level of the client and improve the training effect of FL global model
by improving the data quality on the client. However, adding additional information at the data level to
achiove better performance for PFL. has always been ignored. Our pFedPT framework uses visual prompts to implicitly represent the data distribution clients, which achieves personalization by incorporating the characteristics of clients into the training process the data level.
Prompt Learning. Prompt learning Liu et al. 2021 as novel application paradigm for large-scale pre-trained models, was first proposed in Natural Language Processing (NLP), and refers to prepending a language instruction to the original text input Li & Liang 2021b. In this way, pre-trained models can given hints about what tasks are currently being performed, thereby achieving strong generalization downstream transfer learning tasks without fine-tuning the whole model (Floridi & Chiriatti 2020 Compared to hard prompts, soft prompts avoid the trouble of manual design, and are more expressive. Lester et al. (021]) task-specific continuous vectors soft prompts and can be optimized by training. In the CV area, Radford et al. (2021) propose the CLIP model using language prompts to solve the vision-languas tasks, which is similar to following works (Tsimpoukelli et al. 2021 Yao et al. 2021). ). In (Bahng et al. 2022 the visual prompts are designed as an inpul-agnostic perturbation. which is padded around the input images. The certurbation-generaaingg function includes a small number trainable parameters, which helps the pre- trained vision models perform downstrcam tasks without finc-tuning any parameters. Visual Prompt Tuming (VPT) (Jia et al.) 22022) is introduced as a parameter-efficien alternative to full fine-tuning for pre-trained model (Dosvvisskiy et al. 2021

---

Published in Transactions on Machine Learning Research (01/2024)
Two contemporary studies, PROMPTFL Guo et al 2022 and FedPrompt (Zhao et al. 2022 also integrate prompt learning into Federated Learning (FI.. However, our approach, pFedPT, exhibits several key differ- ences from these works. A succinct comparison elucidates these distinctions: (i) FL Category: PFedPT is specifically tailored for personalized FL, focusing on individual client nceds. Conversely, PROMPTEL and FedPrompl are designed for generic FL., aiming for broader applicability without client-specific customiza- tion. (ii) Prompt Deployment: In PFodPT, prompts are retained locally at cach client, cultivating a unique, personalized prompt for every participant. This contrasts with PROMPTFL and FedPrompt, where prompt parameters shared among all clients, culminating in universally applicable FL model. (iii) Training Dynamics: PFedPT diverges significantly in its training methodology- Unlike PROMPTFI, and FedPrompt, which freeze the pre-trained model and only update prompt parameters, pFedPT dynamically trains both backbone and prompt parameters. This alternating training regimen enhances model person- alization and adaptability to specific client data. These distinctions underscore pFedPT's commitment to personalization FL, setting apart from PROMPTFI, and FedPrompt's more generalized approach.
3 Methodology
In this section, we introduce the proposed visual prompt based personalized federated learning (pFedPT) framework. Below, we first provide several preliminaries on PFL.
3.1 Problem setup
Suppose that there are N clients, denoted as CI,..., Cn. respectively. Client C, has a local dataset D with |D'| samples. The goal of traditional FL (McMahan et al.) 2017 is collaboratively learn machine learning model wo over the dataset D A Use[ D' with the help of a central server, while the raw data are not exchanged. The objective of FL is defined below:
D D
arg min L(w) w
Σ
L.(w),
(1)
where L,w = (a,))) (ee (w: (z, y))l is the empirical loss of C, However, rather than aiming at a single aggregated model w in FL, PFL is supposed to train personalized models wy for different clients (Tan et al. 2022), which is defined as the following optimization problem:
N
D
arg min L(W) = Σ (ww))
(2)
D
W
where W = {[w,..., wNn is the personalizod models set for all clients.
3.2 Workflow of pFedPT
We introduce novel visual prompt based PFL framework for solving the PFL task, dubbed pFedPT. The central insight of the pFedPT is to train learnable continuou visual prompts about data distribution for each client and use them to fine-tune backbones locally on those clients. Prompts on each client is client- specific knowledge aiding the backbone to complete the training task. The pFedPT currently focuses on visual-related tasks, wherein each client maintains set of prompt parameters and backbone, shown in Fig. [2] When performing image classification tasks, pFedPT first adds prompts to each image, which is then passed into the backbone for classification prediction. Generally, complete pFedPT training process mainly includes four steps, as shown in Fig. 2
Step 1.
To begin with, the parameters of the prompt each client are updated with local data while the whole backbone is frozen.
After training several epochs, the prompt parameters will be frozen, and the backbone will begin to update for fixed number of epochs.
Step 2.
5

---

Published in Transactions on Machine Learning Research (01/2024)
When the training process of all clients is finished, they send the trained backbone to the server, followed by the aggregation operation conducted by the server.
Step 3.
The aggregated backbone will be broadcast to every client to replace the old backbone stored locally.
Step 4.
Repeat the Step 1-Step 4 until the training process of the prompt parameters and backbone converges. At this point, the prompt parameters for each client are based on local data distribution and can be seen as a guide to fine-tuning the prediction results of the backbone for the input images. Since the prompt is client-specific, the same backbone can generate different fine-tuning effects when used by different clients achieve personalization.
Below, specify the key components of pFedPT, i.e,, prompt parameters, which are parameterized with the parameter å for the prompt. The prompt is added to the input image to form a prompted image X,+ During the local evaluation, the optimized prompt is added to all test images,
X = {ar + o,,...,,," x + +z
(3)
There are several ways to design visual prompt in terms of template and size. Following the settings of [Bahng et al.) 2022), we explore three visual templates: pixel patch at a random location, pixel patch at a fixed location, and padding. We explore various prompt sizes p. where the actual number of parameters is Cp² for patches and 2Cp(H+WW- respectively. In order explore the effect of different prompts on the results, we conducted an experiment CIFAR10 dataset with Dirichlet (0.3) partition. Fig. g shows that padding prompts with p 4 size achieve the best performance over other design choices. We use this as the default for all our experiments.
for padding, where C, H, W are the image channels, height, and width,
3.3 Modeling for PFedPT
Our goal is learn personalized prompt d for each client and a backbone w. The prompt d, is also trained by the local data. Our objective is to solve the following:
D
arg min L(w, 0:) 10.6,
Σ
Le 6:),
(4)
where L,Ww, 0,)
(aa)) (ww; (x 8, w))] is the empirical loss of C. To achieve the goal in Eq. 2 existing PFL algorithms usually add a regularizer to the model to perform information exchange between 2021b), partition the layers as shared and personalized parts clients (Ariazhhaaan al. 2019 Li et al.
by exchanging the shared layers (Tan et al. 2022 or interpolate the aggregated model with local models (Li et al. 22221b Mansour et al. 2020). However, pFedPT still uses the aggregated model w to deliver public knowledge between clients, and personalized knowledge is incorporated by adding 5, to the data. Specifically, the shared backbone is responsible for the extraction the common knowledge of each client and identifying the information carriod by the visual prompt of the individual clients. The elient-speeific prompt is responsible for increasing the guidance of the backbone to achieve finze-tuning to adapt to the client's data distribution. We implement personalized prediction of the backbone the client data level.
3.4 Optimization for PFedPT
To achieve the optimization goal of Eq. 4 we alternately update the prompt parameters and the backbone on each client using gradient descent. PFedPT first trains the prompt parameters with the aggregation model fixed, and the model maximizes the likelihood of the correct label y, which is equivalent to solving:
min mim Li 5,) B( (&: (x 8). 6,
After updating the prompt parameters locally, we freeze the parameters of the prompt parameters, and then train the backbone for several epochs. The backbone has the following objective function in the client

---

Published in Transactions on Machine Learning Research (01/2024)
Algorithm 1: pFedPT framework
Input: number communication rounds T, the set of clients (Ci,.., Cn], number of local epochs Es for backbone, number of local epochs Eg for the prompt parameters, learning rate Th for backbone, learning rate ? the prompt parameters, initialization parameters w² for backbone, initialization parameters 8 for the prompt parameters in client i.
Output: The final model =²
Server exccutes: initialize w 8 : choose random set of devices from C for t 0, I,.., for C; E S in parallel do
do
send aggregated model w' to C, wi + LocalTraiinngg( w')
w c+1
|8| DD [D] w
return T
9 LocalTraining(i, w'): w t w'
for epoch i = 2, ...,, Eg do for each batch = (I,8} of D' do Training for prompt parameters: 6, + 6 Ve.(( (x 6 6; v)))
11
12
for epoch 1,2, do for each batch b (z, y) of D' do
14
15
local backbone training: w +- wt m (x
16 return wi server
during the training process:
arg arg min Li(w,, 6,) (x,,)) [C<(w;; (x 6, y))]
A locally trained backbone can learn the client data distribution corresponding to prompt on the client and prompt knowledge is passed between clients via model aggregation the server. The backbone the server aggregales according to the following formula:
D D
w [+1
w
(7)
where / represents the number of training rounds. We summarize the detailed procedures of pFedPT in Algorithm
In the end, we give several comments on the differences between our PFedPT and decoupled FedRep (Collins et al.) 2021). Fig describes their training process. Note that pFedPT also has a private part and public part, but the private part is the prompt parameters that we added the client data level additionally. The personalized visual prompt generated adds the client's personalized knowledge the training process by fine-tuning backbone's input without changing backbone's inference process. FedRep is to separate the private part in the inference model, and different clients have different inference processes. The objective functions of FedRep and PFedPT are also different. Furthermore, the visual prompt is orthogonal to FedRop type hods which can be integrated together to further boost their performance.
4 Experiments
this section, we evaluate the effectiveness of pFedPT and compare with several advanced methods various datasets and settings. We also conduct number of exploratory experiments to find out how PFodPT works and verify the effectiveness of PFedPT in terms of client data distribution. The aim of our experiments is to address the following research questions:

---

Published in Transactions on Machine Learning Research (01/2024)
F[Cv, X)
F[(V,u) (CC. (X,+ P.)
14
Back Propagation
Shared
V
Personliedd
Input X
Input X
Prompt
(a) FedRep
(b) pledPT
Figure 3: Differences between PTedPT and decoupled personalized FL algorithm (FedRep)
RQ1 Can the proposed pFedPT achieve better performance than other methods?
RQ2 Is pFedPT robust and performs well in different situations?
RQ3 Whether pFedPT can be combined with other algorithms to improve performance?
RQ4 Whether the prompt that is added to the image noticed by the backbone model?
RQ5 Is the information contained in the prompt different from client to client?
RQ6 Does prompt contain information about client data distribution?
RQ7 Whether the parameters of prompts can converge?
RQ8 What are the effects of different types of prompts?
4.1 Experimental Setup
Comparison methods. We compare pFedPT with several advanced FL methods. FedAvg (McMahan
et al. 2017) is proposed as the basic framework in federated learning. FedProx (Li et al. 2020) adds a proximal term to the objective function of the local model and allows for the emergence of incomplete training of the local model. MOON (Li et al. 2021a is to utilize the similarity between model representations to correct the local training of individual parties, conducting contrastive learning the model level. Fino-tuning of the global model has been retonstrated be highly effective in PFL (Chen & Chao 2021 Yu et. al. 2020), therefore, apply fine-tuning' to the global models learned using FedAvg, FedProx, and MOON. Per- FedAvg (Fallah et al. 2020) proposes personalized variant of FL in heterogencous setting and introduces the PerFedAvg algorithm as decentralized implementation to solve the problem. FedPer (Arivazhagan et al. 2019h and FedRep (Collins et al. 2021) are base + personalization layer approaches for federated training deep fordafrrward neural networks, which can combat the ill-effects statistical heterogeneity, FedMTL (Smith et al.) (2017) uses a multi-task learning (MTL) framework learn separate models for each client. FedBABU (Oh et al. 2021) achieves good personalization performance by freezing the last discriminative layer of the network and finc-tuning it after training. We also compare bascline named Local, where each client trains model with its local data without federated learning.
Datascts. We conduct experiments two benchmark datasets: CIFAR10 Krizhevsky et al. 2009 and
CIFAR100 (Krizhevsky el al.) 2009). The CIFARIO dataset contains 50,000 training data and 10,000 test
data in 10 classes. Each data sample is a 32 x2 color image. CIFAR100 (Krizhevsky et al., 2009) includes 50,000 training data and 10,000 test data. 100 classes as 500 training samples per class. CIFAR100 is a more difficult dataset for classification tasks than CIFAR10. For CIFAR10 and CIFAR100, we normalize the pixel value within specific mean and std value in code, which are [0.5, 0.5, 0.5) for the mean and
8

---

Published in Transactions on Machine Learning Research (01/2024)
10.5, 0.5, 0.5] for the std. We consider two different scenarios for simulating non-identical data distributions (Non-ITD) across federated clients. Dirichlet Partition follows works (Hsu et. al. 2019), where we partition the training data according to a Dirichlet distribution Dir(a) for each client and generate the corresponding test data for each client following the same distribution. We specify a equal 0.3 for each datasct. In addition, we evaluate with the pathological partition setup similar to (Zhang et al. 2020), which each client is only assigned a limited number of classes at random from the total number of classes. We specify that each client possesses classes for CIFARIO and 50 classes for CIFARI00.
Evaluation Metrics. We are distributing the test data set to each client in the same way as the training set and the final accuracy of each method reported in our results the average accuracy of the local model for each client on its own test set after training.
Implementation Details, We verify the experimental results based on CNN and ViT architectures. The CNN model consists of convolutional layers with 64 5x5 filters followed by fully connected layers with 394 and 192 neurons and a softmax layer. We use tiny ViT architecture consisting of 8 blocks with 8 self-attention layers in each block. The corresponding attention head number is the patch size is 4, and the embedding dimension is 128. We set the number of clients to 50, and then each client has 20% chance of participating in each communication round. We utilize the SGD algorithm (Cherry et al. 1998) as the local optimizer for all methods. We use padding as our prompt method. We set batch size as 16 in the local training phase, the local training epochs for the prompt paramcters and backbone as 5 in each round, the learning rate for the backbone as 0.005, the learning rate for the prompt parameters as 1, and the padding prompt size as 4. The number of communication rounds is set to 150 for CIFAR10, 300 for CIFAR100, where all FL approaches have very limited or accuracy gain with more communications.
Settings. We fix the learning rate for local training as 0.005 and for the prompt parameters training as 1.0. We fix the training batch size as 16 and fix the epoch for local training as For the specific parameters in FedProx, the proximal rate is sel as 0.0001. For the specifie parameters in MOON, the j is set as 1.0. For the specific parameters in FedRep, the personalized learning rate is set 0.01. For the specific parameters in FedMTL, the iterations for solving quadratic sub-problems are set 4000. For the specific parameters in FedBABU, the fine tuning step is set as 1. For fine-tuning, we train the global model learned in FedAvg, FedProx, and MOON for epochs.
Table 1: The results of PT and baseline methods on the image datasets with different non-lID settings. denotes the finc-tuning results.
CIFARIO
CIFARI00
#setting ViT
IID
Dirichlet
Pathological
IID
Dirichlet
Pathological
#client
CNN
ViT
CNN
VIT
CNN
ViT
CNN
VIT
CNN
ViT
CNN
FodAvg
60.50
67.13
53.01
61.92
54.98
63.62
29.60
26.42
25.93
26.50
27.71
30.28
Avd
60.47
67.12
72.43
77.52
67.27
74.95
29.97
25.75
35.88
32.51
34.79
31.84
FedProx
57.04
66.94
53.14
61.95
55.02
63.29
27.71
26.29
26.00
26.48
27.84
30.52
FedProx
57.05
66.82
73.94
77.43
68.54
74.90
27.43
25.51
36.54
32.17
34.22
35.77
MOON
60.99
66.88
61.12
62.53
65.98
63.52
29.32
26.43
24.95
26.93
27. 67
29.00
M
60.06
66.31
73.46
78.69
70.12
75.10
29.41
25.31
36.20
30.96
33.74
36.02
Per-FedAvg
57.48
61.35
68.63
76.81
74.56
72.32
26. 26.41
26.12
31.67
30.17
34.65
32.38
FodPer
61.57
51.46
73.16
77.98
75.20
79.97
29.74
10.82
36.78
27.79
35.36
31.13
FedRep
48.38
49.70
74.11
77.65
74.48
78.39
17.84
9.13
35.06
27.39
36.13
32.41
FNIMTL
45.65
45.65
68.48
73.95
65.39
70.94
17.91
7.34
26.08
25.85
25.46
26.32
FedBABU
50.41
61.17
74.21
80.11
74.30
80.69
20.61
22.55
36.17
31.66
35.74
35.45
Local
45.37
39.04
68.40
73.98
64.83
70.76
18.01
7.33
26.23
25.15
24.65
25.34
PT (ours)
60.01
66.09
74.92
80.83
75.42
81.16
31.66
26.41
36.80
32.47
36.88
37.98
4.2
Main Results
We run vast experiments to determine the superiority of pFedPT on the model performance in different datasets. Our results highlight the benefit of PTedPT compared to the existing PFL optimization approaches.

---

Published in Transactions on Machine Learning Research (01/2024)
Comparisons with SOTA methods. Tab. | presents comparative analysis of pFedPT's best accuracy against. baseline methods evaluation datasets under various settings, addressing RQ1. On CIFARIO and CIFAR100 datasets, especially in Non-lID settings, pFedPT consistently registers the highest test accuracy. For example, with a CNN trained on CIFAR10 data following a Dirichlet distribution, pFedPT achieves a test accuracy of 80.83%, surpassing FedAvg's 61.92%, FedPer's 77.98%, and even FedAvg with fine-luning at 77.52%. This demonstrates that, while fine-tuning can enhance model accuracy, pFedPT excels beyond these methods. The superior performance of pFedPT can be attributed to the effective utilization of prompts in each client, significantly boosting the backbone model's performance. In the CIFAR100 dataset as well, PFedPT outshines most baseline methods across different settings and delivers competitive results in the Dirichlet setting. More experimental results on Tiny-ImageNe can be found in the Appendix A.2
Table 2: Architecture Agnostic analysis. We insert the proposed framework (denote as PT) to existing pFL methods.
CIFAR10
CIFAR100
#sctting
IID
Dirichlet
Pathological
IID
Dirichlet
Pathological
FedProx
66.94
61.95
63.29
26.29
26.48
30.52
Ped
67.60
80.47
81.48
26.87
31.95
37.88
MOON
66.88
62.53
63.52
26.43
26.93
29.00
MOON+PT
66.92
77.84
76.00
26.58
28.67
34.60
FedPer
51.76
77.98
79.97
10.82
27.79
31.13
PPd
52.02
78.40
80.59
11.37
28.83
31.14
FedRep
49.70
77.65
78.39
9.13
27.39
32.41
PTT
49.88
77.65
79.11
9.87
29.19
32.75
Robustness of pFedPT. Our pFedPT demonstrates significant success in both VIT and CNN models. It exhibits enhanced performance as the FL. tasks become increasingly challenging with a greater Non- ID data distribution, effectively addressing RQ2. In the IID setting, however, we observe a performance degradation in all personalized solutions when compared to the FedAvg approach. Our interpretation of this phenomenon is that, in the IID setting, data across different clients closely mirror the overall data distribution. Consequently, the effectiveness of many PFL methods is diminished, with FedAvg frequently outperforming most PFL approaches in such contexts. pFedPPT leverages client-specific data distribution information through visual prompts. In scenarios where data is IID, the output across various clients tends to converge, resulting in performance akin to that of FedAvg.
Improvements of prompt for other algorithms. To answer RQ3, we combine PfedPT with other methods and find visual prompts can improve the performance of backbones clients by fine-tuning the backbone with prompts about the distribution of the client's is data. We explore the usefulness of visual prompts as client-specific knowledge for other FL algorithms, and Tab. 2 presents these results. In the Dirichlet setting of CIFAR10, the final test accuracy of FedProx increases from 61 61.95% to 80.47% after adding prompts, and the test accuracy of MOON increases from 62.53% to 77.84%. We find that a visual prompt enables fine- tne-tuming of the backbone of the client, which helps FL algorithms that pursue high precision fuse client information for personalization. Similarly, PFL algorithms with model decoupling, like FedRep and FedPer, also yield performance boost by integrating pFodPT. Therefore, prompt be used additional component to improve the personalization performance of some existing FLL algorithms.
Compared with other baselines, PFedPT takes full advantage of the data improvement space. Additional prompts are addod to the data entered into the model to improve the performance of cach client.
4.3 Exploratory Study
To provide more explanation for pFedPT, additionally conduct several exploratory studies pFedPT.
10

---

Published in Transactions on Machine Learning Research (01/2024)
Raw Images
FodAvg-ViT
PedAvg-CNN
Images with prompts
VVT
PFedPT-CNN
Figure 4: Visualization results generated by FedAvg and pFodPT with different backbones.
Visualization of attention maps. To illustrate the effectiveness of visual prompts and answer RQ4, we conducted some validation experiments. We train ten clients using FedAvg and pf with VIT and CNN backbones under the Dirichlet setting of the CIFAR10 dataset, respectively. As shown in Fig, 4 make visualization of the attention map of the last layer in the ViT and CNN by Grad-CAM (Selvaraju et al. 2020). The first three rows in the figure show that FedAvg focuses on some salient classification features of the raw image. The fourth row contains the input images with the padding visual prompts, which are added by the prompt parameters of pFedPT according to Eq. (3). Both PFedPT+VIT and PFedPT+CNN shift some attention to the added prompts, which can help obtain the client-specific knowledge for the model inference process, thus improving the performance of the model.
The guidance information contained in the prompts. visual prompts and answer RQ5, we generated 100 different pure color images with the shape of x2 32). Using the pure color picture, pFedPT can exclude the disturbance of image contents and pay more attention to visual prompts. We feed those color pictures into PFedPT models in different clients with different prompts and visualize the output embeddings of their last MLP layer. We project them into a two-dimensional plane using the t-SNE algorithm (Van der Maaten & Hinton, 2008). Fig. 5 illustrates that with the incorporation of visual prompts, the representation space of a given client bocomes distinct from that of other clients. Although this difference may require the backbone to exert more effort to align on the server, enables the model outputs of various clients be easily distinguished. This indicates that the prompts encapsulate client-specific knowledge within the client model, thereby significantly facilitating the classification task.
order further explore the influence of
The connection between Prompt and the client data distribution. single, pure-color image as input investigate the relationship between the local model output and the data distribution of each client. Tdeally, the output distribution classes of each PFedPT client should
To answer answer RQ6 we use
11

---

Published in Transactions on Machine Learning Research (01/2024)
Clamt Clent
Clint Cleck
Clant
Clant
Figure 5: t-SNE visualization of embedding for pure color images with learned prompts in different clients.
Output Logits of cach client model
The data distribution of each client
08
08
4
02
0.0
-044
-04
The data distribution of each client
The data distribution of each client
(a)
(b)
Figure 6: Similarity comparison between the distribution of the predicted classes and the local data.
align with the local data distribution. Fig. G reveals that after adding the visual prompts, the outputs of the PFedPT will be similar to the distribution of the client itself. The difference between the visual prompts generated by clients with similar data distribution is also smaller, which means that the visual prompts indeed contain the data distribution information the clients. Therefore, the visual prompts provide the model with certain client-specific knowledge when classifying a specific client and assist in the classification task.
Empirical analysis of the learned prompts. Fig. 7 records the average difference of the prompts generated between the two rounds before and after ten clients during the pFedPT training process. The overall experimental results are divided into two stages: first ascending and then descending, In our settings, the initial prompt parameters of each client are the same, and the rising stage is the mapping process between each client and the prompts based on its own data distribution. The descending stage is when the aggregated model tends to converge, and the mapping between the prompt and the client data distribution on each client is complete. Eventually, the change in prompt embedding approaches 0, and each client establishes stable prompts that conform to its own data distribution which answer RQ7.
Impact of different types of visual prompts. We analyze different choices on how and where to insert prompts in the input images and how they would affect the final performance to answer RQ8. We perform an ablation study on different prompt sizes in = 62,4,6..... 16) in CIFARI0 with Dirichlet distribution. As shown in Fig. padding prompts reach the highest performance with a size of 4. The test accuracy of fixed location and random location prompts grows gradually with the increase in prompt size, but it is
12

---

Published in Transactions on Machine Learning Research (01/2024)
0.8
30
70
0.6
Test Accuracy
Test Accuracy
65
Difference
20
test Accuracy prompt difference
0.4
55
10-
0.2
Fixed location
Random location
Padding
0.0
45
60
100
12
Communication Rounds
The size of visual prompts
Figure 7: The difference of prompt between two con- secutive rounds.
Figure 8: Effect of different types of prompts
still slightly lower than the padding prompt. In contrast, the accuracy of padding prompts decreases as the prompt size increases. A possible explanation that the padding method covers more pixels of the original images than the other two methods when using the same length of prompts. As result, the key information for classification could be blocked by the prompts and harm the performance of the model. Overall, the padding prompts with size 4 achieve the best performance. Note that other visual tasks may require significantly different kinds of prompts.
5 Conclusion
In this work, we propose a novel framework named pFedPT, a personalized federated learning method based on visual prompts. We make the first attempt introduce visual prompts to personalized foderatod learning, using set of prompl parameters to distill information from local data into the visual prompts and fine- tune the backbone. In the process of pFedPT training, the backbone could use the guidance information from visual prompts to perform the personalized downstream tasks. Since the prompt parameters is trained locally on the client, it does not reveal data distribution information about the client to others or the server. PFedPF can also serve a strong plugin to boost the performance of existing FT, methods, which could be of independent interest. We provide extensive experiments to illustrate how the pFedPT works and demonstrate offectiveness in experiments with heterogencous settings and several types of dataset partition.
Acknowledgment
This work supported by STI 2030 Major Projects (No. 20212D0201105). would like express my sincere gratitude to the contributors of the GitHub repository PFL1ib /https://githen com/TsingZo/ PFL1ib). Their work laid the foundation for my rescarch and significantly contributed to the success of this work.
References
Manoj Ghuhan Arivazhagan, Vinay Aggarwal, Aaditya Kumar Singh, and Sunav Choudhary. Federated 2019.
learning with personalization layers. arXiv preprint arXio:
Hyojin Bahng, Ali Jahanian, Swami Sankaranarayanan, and Phillip Isola. Exploring visual prompts for adapting large-scale models. arXie preprint orXiv:2208.1727, 1(3):4, 2022.
Christopher Briggs, Zhong Fan, and Peter Andras. Federated learning with hicrarchical clustering of local updates improve training on non-IID data. In 2020 International Joint Conference on Newral Networks (IJCNN) pp. 1 1-9. IEFF, 2020.
13

---

Published in Transactions on Machine Learning Research (01/2024)
Fei Chen, Mi Luo, Zhenhua Dong, Zhenguo Li, and Xiuqiang He. Federated meta-learning with fast conver- gence and efficient communication. arXiv preprint orXiv: 11808800777766 2018.
Haokun Chen, Ahmed Frikha, Denis Krompass, and Volker Tresp. FRAug: Tackling Federated Learning with Non-IID Features via Representation Augmentation. arXiv preprint arXiv:2205.1440 14900, 2022.
Hong-You You Chen and Wei-Lun Chao. On bridging generic and personalized federated learning for image classification. In International Conference on Learning Representations, 2021.
Yiqiang Chen, Xin Qin, Jindong Wang, Chaohui Yu, and Wen Gao. Fedhealth: A federated transfer learning framework for wearable hcalthcare. IEEE Intelligent Systems, 35(4):83-93, 2020.
J Michael Cherry, Caroline Adler, Catherine Ball, Stephen A Chervitz, Selina S Dwight, Erich T Hester, Yankai Jia, Gail Juvik, TaiYun Roe, Mark Schroeder, et al. SGD: Saccharomyces genome database. Nucleic acids research, 1998.
Yae Jee Cho, Jianyu Wang, Tarun Chiruvolu, and Gauri Joshi. Personalized federated learning for hetero- geneous clients with clustered knowledge transfer. arXiv preprint orXiv:2109.081 2021.
Liam Collins, Hamed Hassani, Aryan Mokhtari, and Sanjay Shakkottai. Exploiting shared representations for personalized federated learning. In International Conference on Machine Learning, pages=2089-209
PMLR, 2021.
Rong Dai, Li Shen, Fengxiang Hc, Xinmei Tian, and Dacheng Tao. Dispfl: Towards communication-cffiinn personalized federated learning via decentralized sparse training. In International Conference on Machine Learning, pp. 4587-4604. PMLR, 2022.
Yuyang Deng, Mohammad Mahdi Kamani, and Mehrdad Mahdavi. Adaptive personalized federated learning. arXiv preprint ar Xrv:20003 13461, 2020.
Canh T Dinh, Tung Vu, Nguyen H Tran, Minh N Dao, and Hongyu Zhang. Fedu: A unified framework for federated multi-task learning with laplacian regularization. arXiv preprint arXiv:2102.00 07440448 2021.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Un- terthiner, Mostafa Dohghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale. In International Conference on Learning Representations, 2021.
Gamaleldin F Elsayed, Ian Goodfellow, and Jascha Sohl-Dickstein. Adversarial Reprogramming of Neural Networks. In International Conference on Learning Representations, 2018.
Alireza Fallah, Aryan Mokhtari, and Asuman Ozdaglar. Personalized federated learning: A meta-learning approach. arXiv preprint arXix:2002.0794 2020.
Chelsca Finn, Picter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In International conference on machine learning, pages= 1126 1135, 2017.
Luciano Floridi and Massimo Chiriatti. GPT-3: Its nature, scope, limits, and consequences. Minds and Machines, 30(4):681-69) -644, 2020.
Liang Gao, Huazhu Fu, Li Li, Yingwen Chen, Ming Xu, and Sheng-Zhong Xu. FedDC: Federated Learn- ing with NID- Data via Local Drift Decoupling and Correction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10112-10121, 2022.
Jack Goetz and Ambuj Tewari. Federated learning via synthetic data. arXie preprint 2xXx:2008.044880 2020.
Tao Guo, Song Guo, Junxiao Wang. and Wenchao Xu. PromptFL: Let Federated Participants Coopera- Learning in Age of Foundation Model. arXiv preprint
tively Learn Prompts Instead of arXiv:2208.1100 2022.
14

---

Published in Transactions on Machine Learning Research (01/2024)
Filip Hanzely, Boxin Zhao, and Mladen Kolar. Personalized federated learning: A unified framework and universal optimization techniques. arXiv preprint arXim:2102.097/4 2021.
Weituo Hao, Mostafa El-Khamy, Jungwon Lee, Jianyi Zhang, Kevin J Liang, Changyou Chen, and Lawrence Carin Duke. Towards fair federated learning with zero-shot data augmentation. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3310-3319, 2021.
Tzu-Ming Harry Hsu, Hang Qi, and Matthew Brown. Measuring the effects of non-identicn data distribution for federated visual classification. arXiv preprint arXi:: 1ee11999900800 2019.
Tiansheng Huang, Shiwei Tiu, Li Shen, Fengxiang He, Weiwei Lin, and Dacheng Tao. Achieving personalized federated learning with sparse local models. Xiv preprint orXiv:2201. 113800 2022.
Tiansheng Huang, Li Shen, Yan Sun, Weiwei Lin, and Dacheng Tao. Fusion of Global and Local Knowledge for Personalized Federated Learning. arv preprint arXiv:2302.11055 2023.
Menglin Jia, Luming Tang, Bor- Cor-Chun Chen, Claire Cardie, Serge Belongie, Bharath Hariharan, and Ser-Nam Lim. Visual prompt tuning. arXiv preprint arXiv:2203.12) 12111 2022.
Yihan Jiang, Jakub Konečný, Keith Rush, and Sreeram Kannan. Improving federated learning personaliza- tion via model agnostic meta learning. arXie preprint (1f::1999.1248,, 2019.
Alex Krizhevsky, Geoffrey Hinton, al. Learning multiple layers of features from tiny images. 2009.
Brian Lester, Rami Al-Rfou, and Noah Constant. The Power of Scale for Parameter-Efficien Prompt Tuning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 3045-3059, 2021.
Guanghao Li, Yue Hu, Miao Zhang, Ji Liu, Quanjun Yin, Yong Peng, and Dejing Dou. FedHiSyn: A hierarchical synchronous federated learning framework for resource and data heterogeneity. In Proceedings of the 51st International Conference on Parallel Processing, pp. 1-11, 2022.
Hongxia Li, Zhongyi Cai, Jingya Wang, Jiangnan Tang, Weiping Ding, Chin-Teine Lin, and Ye Shi. FedTP: Federated Learning by Transformer Personalization. JEEE transactions on neural nelworks and learning systems, pp. 1 15, 2023.
Qinbin Li, Bingsheng He, and Dawn Song. Model-contrastive federated learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10713-10722, 2021a.
Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, and Virginia Smith. Federated optimization in heterogeneous networks. Proceedings of Machine learning and systems, 2:429-450, 2020.
Tian Li, Shengyuan Hu, Ahmad Beirami, and Virginia Smith. Ditto: Fair and robust federated learning through personalization. In International Conference on Machine Learning, pages=6357 6368. PMLR, 2021b.
Xiang Lisa Li and Percy Liang. Prefix-Tuning Optimizing Continuous Prompts for Generation. In Pro- ceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th In- ternational Joint Conference on Notural Language Processing (Volume 1: Long Popers), pp. 4582 4597, 2021.
Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing. arXiv preprint 2xXic:2107.1566 2021.
Yishay Mansour, Mchryar Mohri, Jac Ro, and Ananda Theertha Suresh. Three approaches for personaliza- tion with applications to federated learning. arXio preprint 2ii::2002.10614 2020.
Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. Communictiion-fffiinn learning of deep networks from decentralized data. In Artificial intelligence and slatistics, pages=1277 1273-121111111 1282, 2017.
15

---

Published in Transactions on Machine Learning Research (01/2024)
Jachoon Oh, Sangmook Kim, and Se- Young Yun. Fedbabu: Towards enhanced representation for federated image classification. Xiv preprint or arXim:2106.0600 2021.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint arXi::807.0037 2018.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, pages=8748-877 8763. PMLR, 2021.
Felix Sattler, Simon Wiedemann, Klaus-Robert Müller, and Wojeiech Samek. Robust and communication- efficient federated learning from non-iid data. IEEE transactions on neurul nelworks and learning systems, 31(90)34000 3413, 2019.
Ramprasaath R. Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient- based localization. Int. J. Comput. Vis., 12:33) 359, 2020.
Aviv Shamsian, Aviv Navon, Ethan Fetaya, and Gal Chechik. Personalized federated learning using hyper- networks. In International Conference on Machine Learning, pages-9489 9502. PMLR, 2021.
Yifan Shi, Yingqi Liu, Yan Sun, Zihao Lin, Li Shen, Xueqian Wang, and Dacheng Tao. Towards more suitable personalization in federated learning via decentralized partial model training. arXiv preprint arXiv:2305.1511 2023.
Virginia Smith, Chao-Kat Kai Chiang, Maziar Sanjabi, and Ameet S Talwalkar. Federated multi-tast learning. Advances in neural information processing systems, 30, 2017.
Canh T Dinh, Nguyen Tran, and Josh Nguyen. Personalized federated learning with moreau envelopes. Advances in Neural Information Processing Systems, 33:21394 21405, 2020.
Alysa Ziying Tan, Han Yu, Lizhen Cui, and Qiang Yang. Towards personalized federated learning. IEEE Transactions on Neural Networks and Learning Systems, 2022.
Maria Tsimpoukelli, Jacob Menick, Serkan Cabi, SM Eslami, Oriol Vinyals, and Felix Hill. Multimodal few-shot learning with frozen language models. Advances in Newral Information Processing Systems, 34: 200-213 212, 2021.
Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-SNE. Journal of machine learning research, 9(11,, 2008
Dui Wang, Li Shen, Yong Luo, Han Hu, Kehua Su, Yonggang Wen, and Dacheng Tao. FedABC: Targeting Fair Competition in Personalized Federated Learning. arXiv preprint arXix:2302.07450, 2023.
Hongwei Yang, Hui He, Weizhe Zhang, and Xiaochun Cao. FedSteg: A federated transfer learning framework for secure image steganalysis. IEEE Transactions Network Science and Engineering. 8(2):1084-1094 2020.
Yuan Yao, Ao Zhang, Zhengyan Zhang, Zhiyuan Liu, Tat-Seng Chua, and Maosong Sun. Cpt: Colorful prompt tuning for pre-trained rision-language models. orXiv preprint orXtv:2109.11799 2021.
Tao Yu, Eugene Bagdasaryan, and Vitaly Shmatikov. Salvaging federated learning by local adaptation.
arXiv preprint arXiv:2002.0477 04788, 2020.
Michael Zhang, Karan Sapra, Sanja Fidler, Serena Yeung, and Jose M Alvarez. Personalized federated learning with first order model optimization. arXiv preprint 2xxx:2012208563 2020.
Haodong Zhao, Wei Du, Fangqi Li, Peixuan Li, and Gongshen Liu. Reduce Communication Costs and Preserve Privacy: Prompt Tuning Method in Federated Learning. aiv preprint Xtx::22088.12 12268, 2022.
16

---

Published in Transactions on Machine Learning Research (01/2024)
A Appendix: More Experiment Results
We run experiments on the real-world datasots for image classification tasks, including CIFARIO, CIFARI00, and Tiny ImageNet. We conduct comprehensive investigations for the impact on client heterogeneity by designing IID and non-IID data scenarios. For comparison, we utilize the FedAvg, FedProx, and FedPer algorithms as baselines. The experiment settings are described in detail below.
A.1 Setups
Dataset. We adopt real-world datasets for the image classification task, inchuding CIFARIO, CIFARI00, and Tiny ImageNet. The CIFAR10 dataset contains 50,000 training data and 10,000 test data in 10 classes. Fach data sample is a x 32 x 32 color image. CTFAR100 (Krizhevsky et al., 2009 includes 50,000 training data and 10,000 test data in 100 classes as 500 training samples per class. Tiny FinylmageNet (Oord et al. (0188) involves 100,000 training images and 10,000 test images in 200 classes for 3 64 x 64 color images, as shown
in Table 3. For CIFAR10/10 and Tiny Imngee,eeet, we normalize the pixel value within a specific mean and std value in our code, which are [0.5, 0.5, 0.5) for the mcan and [0.5, 0.5, 0.5] for the std.
Table 3:
The similarity between predicted and real data distribution
Datasets
Training Data
Test Data
Class
Size
CIFAR-10
50,000
10,000
10
3 x 32 x 32
CIFAR-100
50,000
10,000
100
3 x 32 x 32
Tiny ImageNt
100,000
10,000
200
3 x 64 x 64
Backbone. We adopt two backbones including ViT and CNN for experiments. Given an image I E R3xhxw, the ViT reshapes it to a sequence of flattened 2D patches Ip E TRax(""-e), where e is the number of channels. (h, w) is the height and width of the original image, while (p.p) is the size of each image patch. A trainable linear projection flattens the patches into a latent D-dimensional embedding space, which is then embedded with a positional embedding. The transformer encoder consists of N layers of Multi-Head Self-Attention (MSA) and Multi-Layer Perceptron (MLP) blocks. For MSA, the queries, keys and values are generated via linear transformations on the inputs for K times with one individual learned weight for each head. Then in parallel, the attention function is applied to all queries, keys, and values. The sequence of image patches Ip is passed into the MSA, followed by MLP for N times to get the final outputs.
The VIT used by us consists of blocks with self-attention layers in each block. The corresponding attention head number is 8, the patch size 4 and the embedding dimension 128. The CNN consists of the basic modules of CNN, including two conventional layers with 64 of 5 x 5 convolution kernels, each conventional layer followed by down-pooling larger, after that are two fully connected layers with 394 and 192 neurons and a softmax layer for prediction.
Three types of generating Visual Prompts. Following the settings of (Bahng et al. 2022 we explore three visual templates: pixel patch at a random location, pixel patch at a fixed location, and padding. We describe in Fig. g the difference between different visual prompts and how these visual prompts templates are added to the input picture. The comparison results are shown in Fig. g
Dataset Partitions. To fairly compare with the other baselines, we introduce heterogeneity by splitting the total dataset and sampling the label ratios from the Dirichlet distribution and Pathological distribution. An additional parameter is used to control the level of heterogeneity of the entire data partition. In order to visualize the distribution of heterogencous data, we make heat maps of the label distribution in different datasets, as shown in Fig. 10 It could be seen that for heterogeneity weight equal to 0.3 in the Dirichlet distribution, about 10% to 20% of the categories dominate on each client, which is the blue block in Fig. I0] For heterogencity weight equal to 5 in Pathological distribution, 50% of the categorics dominate on each client, which is the black block in Fig. 10 The IID dataset is totally averaged for each client, which is the green block in Fig. 10
17

---

Published in Transactions on Machine Learning Research (01/2024)
1 Fixed Location
Random Location
Prompt Generator
Input Images
Padding
Figure 9: Different visual prompt templates
Dirichlet Distribution
Pathologica Distribution
125
Classes
230
052
65S.A 829 666 Clinats
Clinats
Clinets
(a) IID
(b) Dirichlet
(c) Pathological
Figure 10: Heat maps for each client with CIFAR10 dataset under different data partitions. The color bar denotes the number of data samples. Each rectangle represents the number of data samples of specific class in a party.
A.2 Other Experimental Results
Table 4: The results of the PFedPT and the benchmark methods on the TinyImageNet dataset with different non-llD settings and backbones." denotes the fine-tuning results.
Imagenteeeet
#setting
I
Dirichlet
Pathological
#client
VIT
CNN
VIT
CNN
VIT
CNN
FedAvg
15.47
16.22
12.06
11.01
9.37
10.53
FedAvg"
16.82
16.98
21.32
17.83
19.25
18.02
FedProx
15.53
15.81
13.82
10.98
12.84
11.32
FedProx
15.62
16.21
21.89
18.85
20.48
18.32
MOON
15.20
16.78
15.51
11.05
13.67
10.24
MOON
15.84
17.08
22.33
18.96
19.68
17.85
FedPer
14.85
13.18
23.95
20.21
19.23
17.99
FedRep
12.91
11.56
25.24
21.32
23.43
20.42
FedMTL.
9.75
11.02
21.14
17.96
18.30
17.39
FedBABU
13.61
12.04
24.34
18.62
22.46
18.41
Local
9.82
10.72
20.86
17.43
18.19
17.12
PFedPT (ours)
18.72
16.53
21.21
21.42
25.95
20.66
18

---

Published in Transactions on Machine Learning Research (01/2024)
Experimental results for TinyImageNet. We compare the performance of pFedPT and other baselines on the TinyInageNet dataset. In our experiments on the Tiny-ImageNet dataset, we also configured 50 clients with a 20% participation rate, The Dirichlet distribution parameter a is set to 0.3. In the Pathological partition, each client retained 80 categories, while the rest of the settings remained consistent with those for CIFAR10 and CIFAR100. Hyperparameters are determined through a grid search. The results are as Tab. which shows that the experimental results TinyImageNet are still consistent with the interpretation in Sec. 4.2 For Pathological distribution with ViT, the test accuracy of the pFedPT 25.95%, the accuracy of FedAvg is 9.37% and the accuracy of the FedPer is 19.23%. However, the test accuracy of the pFedPT is not the highest in Dirichlet distribution, our explanation is that it is hard to generate stable prompts in 500 rounds due to the increased task difficulty.
The results of using pre-trained VIT. Due to the pre-training on large-scale datasets, pre-trained VIT has demonstrated strong performance in most downstream tasks. However, the purpose this study is to explore the design of an effective and generic prompt method specifically for the PFL task, rather than solely aiming to enhance model performance. Nonetheless, weights, with only the output dimension of the last classification layer modified. We set the Dirichlet distribution parameter is set to 0.3, and in the Pathological partition, each client retained 5 categories. The results are shown in the Tab. 5 It can be observed that the performance of the pre-trained ViT has indeod shown a significant improvement and PFedPT can also enhance the accuracy of FedAvg in non-lII settings. However, it is important to note that the model size has increased by 85 times (#parameters 1,076,826 vs 3446. Therefore, the pre-trained ViT model can be used when computational efficiency is concern and when the sole objective to pursue model accuracy.
also conducted validation using pre-trained
vit-base-patch1-
Table 5: The comparison results of using pre-trained VIT weights.
Models CifarI0-IID Cifarl0-Dirichlet Cifar Pattholoaaholliccaall
FedAvg 98.74
96.96
97.15
PFedPT 98.43
97.10
97.99
The results with soft visual prompts. We incorporate soft visual prompts from Visual Prompt Tuning (VPT) (Jia ct al. 2022) into PFedPT to replace the original prompts. The main difference lics in the fact that the original prompts need be added to the original images and input logether into the backbone. contrast, the soft prompts are added to the front of the sequence of features obtained after the patch embed- ding layer. It should be noted that soft visual prompts can only be applied backbones with a Transformer architecture. The results in Table D correspond to the Non-IID scenarios, where the Dirichlet distribution parameter a is set to 0.3, and each client. possesses 5 categories within the Pathological partition. The table reveals that, although the accuracy of soft visual prompts may not exceed that of the original pFedPT, it still surpassos FedAvg's performance. This suggests that other types of prompts are also applicable to PFedPT, serving as client-specific knowledge to assist in completing classification tasks.
Table 6: The results of pFedPT with soft visual prompts. Models
Cifar 10-Dihhhhlett Cifar 10- Pathoologola
FedAvg
53.01
54.98
PFedPT w/ soft visual prompt
62.04
57.99
PFedPT
74.92
75.42
The results under more challenging non-IID scenarios. We have conducted experiments on the CIFARIO and CIFAR100 datasets with a backbone structure of CNN, where a is set to 0.1 and the class possession is 20%. Tab. 7 demonstrates that pFedPT continues to outperform other algorithms, even when faced with more severe data heterogeneity.
The necessity of a four-step implementation. To verify the effectiveness of our proposed four-step training pipeline, we conduct. experiments the CIFAR10 and CIFAR100 dataset. with Dirichlet. set
19

---

Published in Transactions on Machine Learning Research (01/2024)
Table 7: The results under more challenging non-IID settings. denotes the fine-tuning results.
#Setting Cifar10-DirO,, Cifar10-Path
Cifar 100-Dirr.
Cifar100- 1afaal00-Path(20%)
FedAvg 53.28
53.44
26.46
27.60
FedAvg" 83.44
89.86
44.96
41.27
FedProx
53.82
53.56
26.31
26.49
FedProx"
86.20
90.58
45.46
43.47
Moon
56.81
55.37
26.61
27.11
Moon*
86.38
87.93
47.04
44.39
FedPer
86.38
89.97
45.42
45.73
FedRep
86.84
89.91
47.04
47.71
FedMTL.
85.29
87.69
44.90
40.60
FedBABU
87.16
89.84
47.36
48.26
Local
85.46
87.66
43.88
39.46
pFedPT 87.22
90.84
47.70
49.61
to 0.3 and the class possession 50%, where both prompts and the backbone are jointly updated in local training using CNN backbone structure. The results are shown in Tab. which indicates that the four-step implementation achieves higher accuracy, emphasizing its necessity.
Table The comparison of joint training and our proposed four-step implementation. Cifar10-IID Cifarl0-DD Cifar10-Path
#Setting
Cifar100-IID
Cifar100-DD
Cifar100-Path
PFePPT-joint
65.94
68.84
75.84
25.62
27.71
30.52
PFedPT 66.09
80.83
81.16
26.41
32.47
37.89
Comparison FedAvg pFedPT
1.0
FedAvg Avemge
Avemge
FedAvg
pFodPT
dinibutiin
0.6
0.2
disent
clien12
chim3
client
cliemS Clietts
cliums
cliveu77
cliem&
cliez19
diivnt
Figure 11: The results of data leakage prevention.
The effectiveness of data leakage prevention. Client-specific prompt information is not uploaded to the server in our method, thus preventing any additional data leakage. Furthermore, the model aggregated by PFedPT requires the loading of cach client's prompt onto the image for completing the classification task. In the absence of visual prompts, the risk of inferring client-side data distribution from models uploaded to the client is significantly reduced. When testing on a balanced dataset with client-uploaded models, it bocomes possible infer the client's data distribution. Fig- [] illustrates the similarity of 10 clients participating in the training and their inferred data distribution based on the model uploaded to the server. PT reduces the predicted similarity by average of 30% compared to FedAvg.
20

---

Published in Transactions on Machine Learning Research (01/2024)
Cise 1
Cave 2
Cise
Case
Case 5
Label
Figure 12: The attention visualization results of different samples from the same category.
More visualization results. We observed a phenomenon during the visualization attention on two specific clients while processing data from the same category, as shown in Fig. 12 Specifically, we found that client 1, when processing data labeled 0, primarily focuses on the left-middle region of the bounding box. Similarly, client 2, when processing data labeled bounding box. The similarity of attention regions for the same category suggests that prompt effectively guides local models by providing insights into the data distribution, thus facilitating accurate classification.
primarily focuses the right-middle region of the
21


Reviews:
[{"review_id": "t7QE1pTJk9", "reviewer": "Reviewer_9Pof", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strength**\n1. The proposed method is novel and reasonable. \n2. The proposed method should be easily reproducible.\n3. The paper is well-written and fairly easy to follow.\n\n**Weaknesses**\n1. The comparison to related work is not clear and informative. Specifically, the authors cited two papers using prompts in FL, PROMPTFL (Guo et al., 2022) and FedPrompt (Zhao et al., 2022). The authors listed two key differences between pFedPT and these two related works. However, the first difference --- training from scratch vs. from a pre-trained backbone --- is more like an experimental setup, not a fundamental technical difference. Could the authors provide a more direct comparison, like whether PROMPTFL and FedPrompt aim for generic FL or personalized FL?\n\n2. The main weaknesses of the papers are in the experiments.\n- The authors should not treat FedProx and MOON as PFL baselines and compare them in Table 1. These methods are not designed for PFL.\n- The authors need to include one critical baseline: fine-tuning. For example, fine-tuning the global model learned in FedAvg, FedProx, and MOON. Fine-tuning has been shown to be very strong in PFL [a, b].\n- The authors miss some strong PFL baselines, such as [c, d].\n- The datasets are too small and limited. Please consider datasets such as (Tiny-)ImageNet.\n- The backbones (both CNN and ViT) seem to differ from those used in existing works (e.g., in terms of layers). Would you be able to comment on this?\n- The ViT results are a bit disappointing. In Table 1, ViT performs worse than CNN in most results. I suggest that the authors consider a pre-train ViT in the experiments.\n- While the authors argued that the proposed method could be more beneficial in challenging non-IID situations, $\\alpha=0.3$ and possessing $50$% classes in CIFAR are not that challenging. I suggest the authors report $\\alpha=0.1$ and $20$% class possession. \n- The explanation on page 10 *\"... in the IID setting, we show that all the personalized solutions exhibit some extent of performance degradation, which becomes more significant as the dataset becomes more challenging. Our interpretation of this phenomenon is that when the data are distributed under the IID setting, the PFL approach does not effectively take advantage of the personalization characteristics among clients, resulting in performance degradation.\"* is a bit hard to grasp. First, what becomes more significant? In Table 1, all the PFL solutions improve the PFL accuracy when the datasets become more challenging (i.e., more non-IID). Second, it does not make sense to me why, under the IID setting, the PFL approach still needs to take advantage of the personalization characteristics among clients. Please note that, under the IID setting, FedAvg outperforms most of the PFL approaches.\n- While Table 2 shows the compatibility, most of the results are worse than pFedPT (i.e., FedAvg + PT), which seems to suggest that pFedPT and other methods are not complementary to each other to boost the performance further. Would you be able to comment on this? Could you provide IID results in Table 2?\n- The results in Fig 4 and the corresponding text are not convincing. First, for the FedAvg baseline, the authors should consider fine-tuning the whole model, not just the last layer. Second, the results are too low. The FedAvg results are merely 0.3, even lower than Local in Table 1 (both are tested on the $\\alpha=30$% clients). Please note that in Table 1, even the FedAvg global model can achieve a PFL accuracy >0.5 on CIFAR-10.\n\n**References**\n\n[a] Chen et al., On Bridging Generic and Personalized Federated Learning for Image Classification, ICLR 2022.\n\n[b] Yu et al., Salvaging federated learning by local adaptation. arXiv preprint arXiv:2002.04758, 2020.\n\n[c] Shamsian et al., Personalized federated learning using hypernetworks, ICML 2021.\n\n[d] Fallah et al., Personalized federated learning: A meta-learning approach, NeurIPS 2020.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper studies personalized federated learning (PFL). It proposes a novel approach to learning and leveraging \"personalized prompts\" to capture clients' characteristics to achieve promising personalized accuracy.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "**Major**\n1. Please see the above weaknesses.\n\n2. For the ViT backbone, I was curious why the authors did not consider adding visual prompt tokens as in Visual\nPrompt Tuning (VPT) (Jia et al., 2022). In the VPT papers, adding visual prompt tokens seems to perform better than adding padding patches.\n\n3. Can the authors compare their four-step implementation in section 3.2 with a baseline approach that jointly updates both the prompts and the backbone in local training? This will be important to justify the necessity of a four-step implementation.\n\n**Minor**\n1. Some claims are a bit too strong. Some word usage is a bit inaccurate. For example, \n- *\" ... which all fail to take into account the data characteristics of distributed clients.\"* In my humble opinion, most of the existing PFL approaches are proposed to capture the non-IID data characteristics of clients so as to perform well for each client.\n- *\"... prior knowledge ...\"* Prompts are learned from clients' data. Can the authors provide more justification as to why they can be seen as prior?\n\n2. The reference style could be improved. For example, *\" ... both visual prompts Liu et al. (2021) and adversarial reprogramming\nElsayed et al. (2018) ...\"* should be *\"... both visual prompts (Liu et al., 2021) and adversarial reprogramming\n(Elsayed et al., 2018) ...\"* by using \\citep rather than \\cite.", "broader_impact_concerns": "No concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "rtB4PMUx39", "reviewer": "Reviewer_y9m3", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: \n1.\tThe proposed framework pFedPT is novel, which first introduces the pixel patch-style visual prompt into Personalized Federated Learning frameworks. \n2.\tAccording to the experimental results and corresponding setting, the proposed method outperforms several state-of-the-art methods.\n3.\tThe proposed method is flexible to combine with other methods.\n4.\tSeveral research questions are discussed in the experimentation, which can be used to guide further researches in the community.\n\nWeaknesses: \n1.\tThe writing requires improvement. Especially, citations have to be fixed throughout the manuscript. \n2.\tThe first contribution of this paper is overclaiming. Prompting learning or prompt tuning has already been introduced in Personalized Federated Learning tasks, e.g., \na.\tYang, F. E., Wang, C. Y., & Wang, Y. C. F. (2023). Efficient model personalization in federated learning via client-specific prompt generation. In Proceedings of the IEEE/CVF International Conference on Computer Vision (pp. 19159-19168).\nb.\tGuo, T., Guo, S., & Wang, J. (2023, April). pFedPrompt: Learning Personalized Prompt for Vision-Language Models in Federated Learning. In Proceedings of the ACM Web Conference 2023 (pp. 1364-1374).\nIn order to emphasize the contribution of this work, the authors are expected to highlight the advantage of the prompting method over others. \n3.\tThe experimental results presented in Table 1 are different from the original paper, e.g. MOON and FedBABU, especially on CIFAR-100, they were performing much better. It would be much more convincing to refer to the original result of these works and compare with them according to their settings.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes a personalized federated learning framework, pFedPT, which introduces learnable visual prompts at each client site for fine-tuning the aggregated model. The visual prompts are trained to encode the local data distribution of each client to achieve better local adaptivity. The resulting pFedPT can be integrated with existing FL and decoupled PFL methods to improve performance.\nSeveral experiments are conducted alongside with discussing several research questions, which show that pFedPT can outperform several state-of-the-art methods.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Besides the weaknesses above, the following questions should be considered: \n1.\tCan the authors compare with other state-of-the-art methods in line with their original performance, which will make the experiment results more convincing. \n2.\tThe research question 2 requires more evidence since the performance on CIFAR-100 is still relatively unsatisfactory.\n3.\tCan the work provide experimental comparison and discussion with other types of prompts, such as comparing with reference a listed in the previous question?\n4.\tIt would be beneficial to discuss whether the learnable visual prompt will cause data leakage.\n5.\tIt would be better to add one more dataset, such as Office-Caltech10.", "broader_impact_concerns": "None.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "TW7ndjbsBf", "reviewer": "Reviewer_dv1f", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\n1. The paper is well-written and very clear.\n2. The concurrent works involve training prompts using a fixed pre-trained backbone. In contrast, the focus of this paper is on personalized prompt federated learning in scenarios where there is no pre-trained backbone.\n3. The proposed method can be applied as an orthogonal approach to existing federated learning methods, demonstrating performance improvements across multiple datasets.\n### Weaknesses\n1. There is no relevant introduction to whether the use of prompts can accelerate convergence or reduce communication overhead in the paper.\n2. In the described scenario, adding prompts to the input appears to primarily enhance the local data's generalization ability, and the assistance for model aggregation seems less pronounced. While the use of prompts can increase the inter-class variance of representations, there may be missing categories, as indicated in your experimental setting. For example, FedPAC[1] discussed alignment in representation spaces. However, in this paper, the addition of prompts may make it more challenging to align the representation spaces of each client.\n3. RQ7 and Figure 7 appear as if adding a prompt acts as a form of noise to the client, disturbing the client's original distribution. However, this does not necessarily imply that the prompt contains information about the client's data distribution.\n\n[1] Personalized Federated Learning with Feature Alignment and Classifier Collaboration. ICLR 2023", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This work introduced an innovative Personalized Federated Learning framework, referred to as pFedPT, for image classification tasks. pFedPT utilizes individualized visual cues to implicitly capture the local data distribution characteristics of clients, furnishing the aggregation model with additional information to enhance its classification capabilities. Comprehensive experiments demonstrate that this approach surpasses several state-of-the-art Personalized Federated Learning algorithms across diverse scenarios.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Already listed in the weaknesses, but I have a few additional suggestions regarding the role of prompts in federated learning:\n1. In Figure 5, attention maps for images of different categories are visualized, demonstrating that different images focus on different regions within the prompt. This observation is quite intriguing, and it is suggested to display the overall attention map for each category within the prompt. It might provide stronger evidence if different categories correspond to different regions.\n2. Figure 6 displays t-SNE visualization of embedding for pure color images with prompts. I would like to understand how prompts affect the t-SNE visualization of the original data in each client. For example, what differences can be observed in the t-SNE results when prompts are removed from or added to the images?\n3. The work should be added some discussion and experiments on representation space alignment in federated learning.", "broader_impact_concerns": "No broader impact concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper focuses on personalized federated learning, and addresses an issue of not considering the data distributions of distributed clients, which is overlooked by existing methods. In the scenario of image classification, the paper proposes a framework that leverages personalized visual prompts to implicitly represent local data distributions of clients; this prior knowledge of local distributions from visual prompts can also be used during testing. Experiments on CIFAR datasets show the efficacy of the proposed framework.

While reviewers generally acknowledge the novelty of introducing visual prompts into the PFL problem, they have concerns on the writings (e.g., overclaiming of contributions, reference formats, etc), on the settings of experiments to compare with existing methods, on the less thorough designs of the experiments, and on discussions of the downside effects by using visual prompts into PFL. To make fair claims of the contributions, the authors are expected to emphasize the specific, technical advantages proposed in this submission; the authors are also highly encouraged to compare different methods under one or several same settings - although in the responses the authors explained that this could be challenging, however, if a fair and consistent protocol of comparisons cannot be established for PFL topic, the topic itself could vanish eventually. The authors would further revise the paper by inclusion of fine tuning as a critical baseline, and by inclusion of other stronger PFL baselines. Some of these revisions have already been done, while others need to be conducted and included in the revised paper. In addition, in the revised version, the authors are expected to provide experiments and discussions with other types of prompts.
Decision: Accept (poster)