Federated Learning from Pre-Trained Models: A Contrastive Learning Approach
Authors: Yue Tan
Guodong Long
Jie Ma
Lu Liu
Tianyi Zhou
Jing Jiang
Venue: NeurIPS 2022 Accept (2022)
Keywords: Federated Learning
Contrastive Learning
Pre-trained Models

Abstract: Federated Learning (FL) is a machine learning paradigm that allows decentralized clients to learn collaboratively without sharing their private data. However, excessive computation and communication demands pose challenges to current FL frameworks, especially when training large-scale models. To prevent these issues from hindering the deployment of FL systems, we propose a lightweight framework where clients jointly learn to fuse the representations generated by multiple fixed pre-trained models rather than training a large-scale model from scratch. This leads us to a more practical FL problem by considering how to capture more client-specific and class-relevant information from the pre-trained models and jointly improve each client's ability to exploit those off-the-shelf models. Here, we design a Federated Prototype-wise Contrastive Learning (FedPCL) approach which shares knowledge across clients through their class prototypes and builds client-specific representations in a prototype-wise contrastive manner. Sharing prototypes rather than learnable model parameters allows each client to fuse the representations in a personalized way while keeping the shared knowledge in a compact form for efficient communication. We perform a thorough evaluation of the proposed FedPCL in the lightweight framework, measuring and visualizing its ability to fuse various pre-trained models on popular FL datasets.

Federated Learning from A Contrastive Learning Approach
Models:
Yue Tan', Guodong Long', Jie Ma', Lu Liu?, Tianyi Zhou Jing Jiang' Australian Artificial Intelligence Institute, FEIT, University of Technology Sydney
Gooole Research, UUnivessity of Washington, University of Maryland yue. Sraansstudent uts edu. au, guodong. long@uts edu au jie wat.maa-555ttt 505t uus.uuus.u. edu. au, lu.liu. celiiuucs@iie aseiccoud. com tianyizheum.c edu, jing. jiangeuts. edu. au
Abstract
Federated Learning (FL) is machine learning paradigm that allows decentral- ized clients to learn collaboratively without sharing their private data. However, excessive computation and communication demands pose challenges current FI. frameworks, especially when training large-scale models. To prevent these issues from hindering the deployment of FL systems, propose lightweight framework where clients jointly learn to fuse the representations generated by multiple fixed pre-trained models rather than training large-scale model from scratch. This leads us to more practical FL problem by considering how to capture more client-specific and class-relevant information from the pre-trained models and jointly improve each client's 's ability to exploit those uff-the-shelf models. In this work, we design a Federated Prototype-we approach which shares knowledge across clients through their class prototypes and builds client-specific representations in a prototype-wist contrastive manner. Sharing prototypes rather than learnable model parameters allows each client to fuse the representations in personalized way while keeping the shared knowledge in a compact form for efficient communication. We perform a thorough evaluation the proposed FedPCL the lightweight framework, measuring and visualizing its ability to fuse various pre-trained models on popular FL datasets.
Contrastive Learning (FedPCL)
1 Introduction
Federated learning (FL) is promising field of machine learning that allows multiple clients to train together without sharing their private data [1] Vanilla FL aims to train single global model over all participating clients by periodically synchronizing their model parameters. However, the learned model usually does not perform well on all clients due the statistical heterogeneity among local datasets 2 3 4 Personalized federated learning (PFL) is proposed to solve this problem by training personalized model for each client. Recent studies on PFL leverage various techniques to enable more common underlying information shared across different clients [5 6 So far, FL and PFL have been widely used computer vision [8) natural language processing 9 graph data mining [0 III,
and some practical applications, e.g., healthcare [12] finance 13] mobile Internet [14
[5] etc.
However, the models real-world applicatioon are usually large-scale neural networks which incur high computation costy and require high communication handwidth when trained from scratch. This can make it infeasible to train such models in some practical FL. scenarios, e.g., low-resource device- based federated learning. To alleviate the above issues, we propose lightweight FL framework that uses multiple fixed pre-trained backhones as the encoder, followed by learnable layers to fuse the rep- resentations generated by the backbones for each client. The proposed framework is capable of fusing the representations generated by pre-traines models with various architectures or obtained from vari-
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

ous source data, expanding the scope of federated learning by integrating off-the-shelf foundation mod- els. Also, it makes it possible to utilize large-scale pre-trained models, c.g., VisionTransformer [T6] and Swin Transformer [1]] in a computation essorcce-cnnsrriing performance. Using the pre-trained foundation models as the fixed encoder can efficiently reduce costs because neither complicated backward propagation computation nor large-scale neural network transmission between the server and clients is needed during the training stage, As shown in Table] [] compared to training ResNet18 [18] from scratch using FedAvg M. a much smaller number of pa- rameters are communicated per round when learning to fuse the representations generated by fixed pre- trained models. Besides, higher performance can be achieved when using pre-trained models after the same communication rounds. Compared with using a single pre- trained model, using multiple models pre-trained on diverse datasets can provide a more comprehensive view for an input sample [9].
case to enhance the overall
To enable better personalized represen- tation ability for each client under this lightweight FI. framework, we need to select an appropriate information carrier to share common underlying knowledge across clients. Motivated by [22 23 (2]]
Table 1:
Compared with training single ResNet18 from scratch, less training time per round and fewer learnable parame- ters are needed when using multiple fixed ResNet18 pre-trained on different datasets (20). Experiments are implemented with FedAvg on Digit-5 dataset (1 under feature shift non-IlI) setting 3
The number of communication rounds is 50.
class-wise wise prototypes, defined as "a repre- sentative embedding" for specific class, can be an effective information carrier for communication between the server and clients. Sharing prototypes allows for better knowledge sharing across vari- ous learning domains, which has been proved in transfer learning (25 and multi-task learning [26]
Mode
Param.
Timel
Acct
Training from scratch
11M
0.95s
31.75(3.07)
Using pre-trained model
0.13M
0.31s
38.65(2.65)
Using 3 pre-trained models
0.40M
0.64s
40.47(3.05)
scenarios. To efficiently extract the useful shared information learned from the pre-trained mod- els via prototypes, we design an algorithm called Federated Prototype-wise Contrastive Learning (FedPCL) where both local prototypes and global prototypes are used for knowledge sharing in supervised contrastive manner. By maximizing the agreement between the fused representation and its corresponding prototypes with contrastive learning, class-relevant information and semantically meaningful knowledge are captured by each client. Concretely, global prototypes force the fused representation to be closer the global class center, while local prototypes force clients to share more higher-level feature information in a pairwise way. Using prototypes to realize inter-client com- munication allows for knowledge sharing a latent space and brings additional benefits. Compared with directly transmitting the representation of sample, prototypes can eliminate bias from single sample and protect clients' privacy. Compared with model parameters, prototypes are much smaller in size, which significantly reduces communication costs.
We quantitatively evaluate the performance of FedPCL and satte-of-theart FL algorithms under the lightweight framework based on benchmark foundation models and datasets. We also perform extensive experiments validate the effectiveness of FedPCL in fusing representations output by different backbones and its capability to integrate knowledge from backbones with different model architectures. Our main contributions are summarized as follows:
We take the first step towards integrating pre-trained models into federated learning to form a lightweight FL. framework, which substantially reduces computation and communication costs of current FL frameworks.
We further propose FedPCL., a novel algorithm that uses class prototypes as the information carrier and conducts contrastive learning during local updates, which allows clients to share more class-relevant knowledge from both local and global prototypes.
Experiments are conducted on variety of benchmark foundation models and datasets to measure FedPCL's 's ability to fuse various pre-trained models for each client. The results indicate that FedPCL. outperforms baselines with a better personalization and knowledge integration ability.
2 Related Work
Personalized Federated I. Learning. Personalized Federated Learning (PFL) aims to train a personal- ized model for each client in the FL framework so that the model can achieve better performance on the local dataset. Existing approaches for PFL are based various techniques. [27 28 29 add

---

an additional term to the original loss function of each client to produce better personalized models according to the private data. B 7 share part of the model and keep personalized layers private to achieve personalization. 30 proposes to use central hypernetwork to generate personalized models for clients. [31] enables a more flexible personalization by adaptively weighted aggregation. 5 32 study PFL from a meta-learning perspective where a meta-model is learned to generate the initialized local model for each client.
Contrastive Learning. Contrastive learning has been widely applied to self-supervised learning scenarios in recent years, achieving tftte-ofthhe-ee art performance in the unsupervised training of deep image models [33] (4] and graph models [35] 36, 37]. A number of works are focused on learning an encoder where the embeddings of the same sample are pulled closer and those of different samples are pushed apart [38, 39 40 41. 42 extends contrastive learning from self-supervise settings to fully supervised settings, enabling us to better exploit label information with contrastive learning. There are also some works incorporate contrastive learning into federated learning to assist local 45
training to achieve higher model performance
Prototype Learning. Prototypes have been widely used in variety of tasks in transfer learning 25
multi-task learning [26], and few-shot learning 22
47 It usually defined as the mean feature vectors of samples within the same class 48 49/. The authors in (30] represent task-agnostic
46
information by prototypes for distributed machine learning systems and propose a multi-task model fusion method that integrates prototypes for new task. Since the prototype has the ability to generalize semantic knowledge from similar samples, it is used to assist the local training in federated 43
learning by several studies [5] 24
Pre-Trained Foundation Model. Due to the huge number of parameters and the broad data available for training, pre-trained foundation models can better capture knowledge for downstream tasks and lead to green AI [52 53 Recently, pre-trained models (e.g., ViT (16), DETR (54), BERT (55) have been widely investigated in both vision and natural language processing (NLP) tasks (56). Extracting task-specific knowledge from the pre-trained models has the potential to achieve the state-of-the-art
performance due to their generality and adaptability to different tasks [57 [58] [9]. However, how to integrate the pre-trained models into a federated learning paradigm and keep the whole framework lightweight still remains an open problem.
3 Problem Formulation
In this section, we formulate our proposed lightweight FL. framework which integrates off-the-shelff pre-trained models as fixed backbones and learns to fuse them adaptively. We start from the general framework of FL., and then explicitly define the problem and explain the proposed global objective.
General FL Framework. Formally, the global objective of general FL across m clients is
1 m
[Dal N
min
L, (wi; D:)
(ary Wa m. m.
where I. and wi the local loss function and model parameters for client respectively. Dj is the private dataset of the i-th client. N is the total number of samples among all clients.
The objective of general FL, such as vanilla FedAvg and its variants, is to learn an optimal global model w across m clients, where w = w1 = W2 = = Wm (]. This can be achieved by periodically synchronizing the model parameters of all clients at the server. However, parameter synchronization sometimes deteriorates the performance on local datasets in the presence of data heterogeneity 77). Some recent studies [6] 60) also explore personalized FL by applying various constraints and regularization terms, which allows clients to keep different models w,, i E [1, m] to achieve higher performance their local datasets.
The Proposed Lightweight FL Framework. Similar to most FL frameworks, there are m clients and a central server involved in the multiple pre-trained backbone-basse framework. Each client i E [L. m] owns K shared and fixed backbones and a private dataset D; that cannot be shared with each other. Each learning model can be seen as combination of at least two parts: (i) Feature Encoder r(:; 2*)= ddd + xd. xd. comprising K fixed pre-trained backbones, each of which maps the raw sample x of size d to representation vector of size de. K representation vectors are concatenated together as the output r(x; Ø²), denoted as Tx for short. (ii) Projection Network (:; 0.): Kx, >> Rds, which fuses the K representation vectors and maps Tx from one latent space to another for further representation learning. The formal definition is provided as follows.
3

---

avg
Client
x(x)
Lp
m
Backbones frr)
Projection Network
k(:00,)
g
Client 712
z(x)
C
avg
Eq. (6)
Lp
Backbones (rt)
Projection Network
8
h(: fm))
Lg
C
Local prototype
Global prototype
Figure An overview proposed lightweight federated learning framework. This example assumes that for each client, there are three pre-trained backbones, with the block in different colors illustrating their backbone-spedee representation.
Definition 3.1.
Let 0f where & E K] denote the optimal parameter of the k-th backbone pre- trained on specific dataset, 7% be the embedding function of the k-th backbone, and x denote an instance sampled from local dataset. We define the concatenated representation output by the feature encoder
rx: Φ*) concat (r1(x;
(x)
(2)
For client i, the projection network h, parameterized by 0; aims to fuse the representations output by multiple backbones into another abstract space. The output of the projection network computed as
=xx) h (rx; 0:).
Based on the above definition, we formulate the global objective of the lightweight FL framework as
m
D
min (01 (01.02..... Pm
E E (0,; (x),2))
s.t. =(x) = h (r(x; T 0.) where क' {Φi .., xx)
The target of the framework to learn the personalized projection network {0, ] for each client. Given an input sample x, the representations output by pre-trained backbones are concatenated together as r(x; )²). Then, the projection network h(0,) optimized for each client converts the concatenated representation to 2(x).
The proposed multi-backbone setting friendly most FL methods. The learnable parameter 0 in the projection network can be fully or partly shared across clients for different purposes. For example, FedAvg || and its variants [6] 62 63 64] can synchronize 0 every round, and personalized FL
methods can also adapt to this framework without much modification (27 7 65
4 Federated Prototype-wise Contrastive Learning (FedPCL)
In this section, we elaborate our proposed algorithm Federated Prototype-wise Contrastive Learning (FedPCL., which is illustrated Figure In each client, =(x) is generated by the projection network which fuses the representations from multiple backbones. Then, to share the common underlying knowledge across clients, we employ a prototype-based communication scheme to transmit and aggregate prototype sets between the server and clients. With the prototypes returned from the server, we perform local optimization via well-crafted prototype-wis wise contrastive loss function, which extracts class-relevant information while sharing more inter-client knowledge in the latent space. We provide detailed illustration for each procedure in the rest this section.
Prototype as the Information Carrier. To capture more class-relevan information and semantically meaningful knowledge, we propose to transmit prototypes between server and clients. Compared with transmitting the learnable model parameters, there are several advantages brought by transmitting class-wise prototypes. Firstly, prototype is more compact in form, which significantly decreases communication costs required during the training process. Secondly, parammettticc communication

---

allows each client to learn a more customized local model without synchronizing parameters with others. Thirdly, prototypes are high-level statistic information rather than raw features, which raise no additional privacy concerns to the system and are robust to gradient-based attacks [66 67
To extract useful class-relevant information from the local side, we construct local prototype as the information carrier for knowledge uploading. Specifically, it is defined in the latent space of projection network's output by the mean of the fused representations within the same class j.
clu)
1
Σ (x):
Da.j
(x.y)EDs,
where Di, refers to the subset of D; composed of all instances belonging to class j. and C2 denotes the local prototype set of the i-th client. After the above computation, the local prototype set of each client is sent to the central server for knowledge aggregation, which shares the local class-relevant information extracted on each specific client based on its local dataset.
Server Aggregation. After receiving local prototype sets {C;]"" from all participating clients, the server first computes the global prototype as
1 LNGI
|Dial N₂
c(u))
(6)
VEN,
where Nj denotes the set of clients that own instances of class j. and N, denotes the number of instances belonging to class j over all clients. The global prototype set is denoted as C {Ca), (12) ..... ). With such aggregation mechanism, the global prototype summarizes coarse- grained class-relevant knowledge shared by all clients, which provides a high-level perspective for representation learning.
After aggregation, the server sends both the global prototype set and the full local prototype sets collected from all clients back to every clients. For the situation where only few classes in each client some non-IID cases, we introduce a prototype padding procedure in the server to ensure that each local prototype set contains prototypes corresponding to all classes:
-P-[( ( i E (6))
c =
i &
Prototype- based communication and aggregation allow each client own unique projection network that is able to fuse the general representations in a customized way. The returned local prototype sets can encourage a mutual learning from a client-relevant perspective while the global prototype set, where each element indicates a class center in the overall data, provides a chance to learn from a highly summarized client-inrelevant perspective.
Local Training. After receiving the prototype sets from the server, the main target of local training is to efficiently extract useful knowledge from the local and the global prototypes, respectively, so as to maximally benefit local representation learning. To achieve that, we propose prototype-wise supervised contrastive loss that consists of two terms, i.e., global term and local term.
To force the fused representation 2 2xx) generated by the local projection network to be closer to its corresponding global class center so as to extract more class-relevan but client-irrelevat information, we define the global prototype-based loss term as
exp (2x C(u) /T]
Lg
Σ log Σ Ege)A(e) exp (Ex Clua)/T) (x.)))
(8)
where xx represents z(x) for short, A(y) :- (va E [1, |CI] Yo + y} is the set of labels distinct from y. is the temperature that can adjust the tolerance for feature difference [68 41 42 For a specific instance x sampled from D,, we use an inner dot product to measure the similarity between the fused representation 2x and prototypes.
Apart from the global term, to align =(x) with each client's local prototypes by alternate client-wise contrastive learning in the latent space and enable more inter-client knowledge sharing, we define the local prototype- based loss term as
exp
[xx
C
/TT
I,p = Σ (x
log
(9)
m
B. E(() exp 3x ((2u)) (T) Σ
p=1
5

---

Given the ground-trute truth label of x, prototype set can be divided as one positive prototype C(v) and set of negative prototypes ((3a), Both Lg and Lp force the representation of a sample to be closer to those positive prototypes and apart from those negative prototypes. C(v) in Lg and (()) in Ip summarize abstract class-relevant information different granularity, which provides guidance for optimizing the local projection network from different perspectives. For the i-th client, the local objective function Eq. defined as a combination of Ly and Lp the following form,
L (1: (x). (Cp)pm)) Lg (0.s (x))=(x))))))))))))))))))))))( (x). C) (p 2 (x), (Cp)p=))
(10)
At the end of the local training in each round, clients upload their local prototype set C; to the server. We present the Fed- PCL algorithm in detail in Algorithm [
Algorithm 1 FedPCI.
Input: D,, 8,, = 1, m, and K pre-trained back- bones with parameters of 02 . respectively. Server executes:
Prototype-based Inference. After ade- quate optimization, in each client, the local prototypes generated by projection network not only contain compact local class-wise
1: Initialize prototype sets {C ppl p=1-
2: for each round 7 = 1,2, do
3: for each client in parallel do (C }p))))))
C: 5 LocalllUpaeee C, (C 5: end for Update global prototype by Eq. 7: end for LocalUpdate(i, C, (C pm 1: for each local epoch do
information but also absorb general knowl- edge from all participating clients. Consid- cring their powerful representation capabil- ity, we leverage the local prototypes to pre- dict unknown labels in the inference stage. Concretely, for a test sample, we first calcu- late the similarity scores between the out- put of projection network and every local prototypes. Then, the predicted result is the class with the maximum similarity score.
2: for each batch in D; do
3: Compute Lg by Eq. 8
with global prototypes.
4: Compute T.p by Eq. with local prototypes.
5: Update 0 by Eq.
end for
Generalization Bound. We provide in- sights into the performance analysis of Fed- PCL. A detailed description and derivations can be found in Appendix C
7: end for
8: Compute local prototypes by Eq. 5
9: return C;
Theorem 4.1. (Generalization Bound of FedPCL.) Consider an FL system with m clients. Let D1, D2..... ... Dy the true data distribution and D1, Dz...., Dm be the empirical data distribution. Denote the projection network h the hypothesis from H and d be the VC-dimension of H. The total number of samples over all clients N. Then, with probability at least
V D.
LoD, (Di: (Cp
m
D. N
L... (0:;C, (Cpppm))
maX (21.0g.....mm)
N
(1I)
N
(m
eN N d
d
log
log
2
5
Theorem 4.1 indicates that compared with the model trained on an ideal data distribution, the performance of the empirically trained model is associated with the VC-dimension of H and the size of the prototype set C. An expected performance gap can be achieved by using appropriate projection network and number of classes. The generalization bound is also important to ensure a satisfying performance of FedPCL, especially when applying it to some safety-critical scenarios (60 69 70
Experiments
5.1
Experimental Setup
Datasets and Non-IID Settings. We evaluate our proposed framework on the following three benchmark datasets:
Digit-5 (2I), Office-10 10 [71]. and DomainNet dataset (72]. Digit-5 (21] is benchmark dataset for digit recognition, including five benchmark datasets, namely SVHN, USPS, SynthDigits, MNIST-M, and MNIST. Office- 10 [1 is standard benchmark dataset consisting of four datasets, namely Amazon, Caltech, DSLR and WebCam. DomainNet (73 is large-scale dataset consisting of six datasets, namely Clipart, Info, Painting, Quickdraw, Real, and Sketch.

---

Table 2: Test accuracy under the feature shift non-III setting. In the column labeled BB (short for backhone), is for single pre-trained backhone and m for multiple pre-trained backhones. # of Comm Params refers to the average number of parameters sent from a client to the server per round.
# of Comm Params
BB
Method
MNIST
SVHN
USPS
Synth
MNIST-M
Avg
FodAvg
70.65(1.15)
17.10(0.20)
70.24(1.62)
32.90(0.75)
29.33(1.18)
44.04(0.98)
133,632
pl'edMe
71.13(3633
13.18(1.78)
69.20(0.30)
36.25(3.35)
25.25(2.25)
43.00(2.26)
133,632
PerFedAvg
52.68(7.03)
16.28(1.23)
53.66(6.58)
29.05(3.45)
24.38(2.38)
35.21(4.13)
133,632
FedRep
64.00(2.20)
17.88(1088)
70.44(1.27)
36.50(1.55)
31.90(0.05)
44.14(2.03)
131,072
FedProto
80.40(2.75)
17.03(0.38)
88.47(0.91)
40.90(1.10)
32.85(0.75)
51.93(1.18)
2,560
Solo
60.40(2.25)
15.60(0.20)
75.28(4.48)
34.65(0.05)
28.48(0.53)
42.88(1.50)
Ours
82.75(0.40)
18.12(0.42)
18.82(0.15)
41.40(0.60)
33.05(0.955)
$2.83(0.21)
2,560
FedAvg
71.68(2.93)
18.45(0.45)
(2.95(0.86)
37.35(1.35)
33.70(2.55)
46.83(1.63)
395,776
PFedMe
67.45(2.70)
(5.43(0.38)
65.66(7.20)
33.55(4.60)
31.80(0200)
42.78(3.01)
395,776
PerFodAvg
56.03(2.73)
17.03(0.63)
57.55(0.27)
34.90(288)
30.98(1.53)
39.30(1.59)
395,776
m
FedRep
77.25(1.75)
(6.40(0.550
80.25(0.32)
37.63(.18)
36.53(0.28)
49.61(1.05)
393,216
FedPPooto
83.78(0.83)
17,90(0. 10)
91.74(0.00)
43.70(2.5))
$4.71(0.99) 36.4((1.88)
2,560
Solo
70.43(4.63)
84.90(0.24) 15.00(0.40)
37.18(2.73)
34,35(2.20)
48.37(2.04)
Ours
84.65(0.15)
19.38(0.63)
90.74(0.53)
44.73(0.37)
37.25(0.28)
55.34(0.34)
2,560
To mimic non-III scenarios in a more general way, we investigate three different non-IID settings: (i) Feature shift non-IID: The datasets owned by clients have the same label distribution but different feature distributions. (ii) Label shift non-IID: The datasets owned by clients have the same feature distribution but different label distributions which is simulated by Dirichlet distribution with parameter [69]. (iii) Feature & Label shift non-III: The datasets owned by clients are different in both label distribution and feature distribution, which is more common but challenging in real-world scenarios. Details about the non- loD settings can be found in Appendix A.1
Baselines and Implementation. We compare our proposed method with popular FL algorithms including FedAvg [1.. pFedMe (27). PerFedAvg [5], FedRep 77 FedProto 124, and Solo, i.e., training independently within each client. In feature shift and label shift non-IID setting, the number of clients is 5. In feature & label shift setting, the number of clients is 5, 4, 6 for Digit-5, Office-10 10, and DomainNet, respectively.
Similar to 19 20 we use the ResNer18 18 pre-trained on Quick Draw, Aircraft, and CU-Birds 20 the backbones. For all baseline methods, the backhone module is followed by two fully connected layers, corresponding to projection network and classifier, respectively, whereas for our method, the backbone module followed by only one fully connected layer as the projection network. The output dimension of each backbone and the projection network are 512 and 256, respectively, Note that for a fair comparison, all baselines use the same network architecture on top of the frozen backbones as FedPCL except their essential classifier.
We use batch size of 32, and an Adam 74 optimizer with weight decay le-4 and learning rate 0.001. The default setting for local update epochs is E = and the temperature T is 0.07. We implement all the methods using Py PyTorch and conduct all experiments GPU. Details about the model and each dataset can be found in Appendix A.
NVIDIA Tesla V100
5.2 Performance Comparison
Table 2 and Table 3 report the results of our method and baselines in mean (std) format over clients with three independent runs. Table 2 shows the results of two cases: (1) s: a single backbone available; (2) m: multiple backbones are available.
Table Test accuracy under the feature & label shift non-III setting for Office- 10, under the label shift non IID setting for DomainNet.
Method
Office- 10
Domainnet
FedAvg PFedMc
33.84(4.59)
28.09(2.91)
The results suggest that: (1) compared with single backbone cases, multiple pre-trained backbones lead to higher test accuracy in most cases and about a 1% 4% test accuracy improvement for our method; (2) apart from locally training each client (Solo), our method achieves relatively smaller deviation across different runs compared with most baselines. which demon- strates that FedPCI. is able to fuse the representations in more stable way; (3) the number of communicated parameters prototype-based method is much lower than that of the model parameter- based methods.
30.00(1.41)
32.65(0.72)
Per-FedAvg
26.04(1.46)
34.64(0.54)
FedRep
37.24(1.54)
48.82(0.55)
FedProto
34.54(2.65)
(4.48(0.58)
Solo
36.38(0.54)
46.70(0.75)
Ours
41.40(1.19)
52.92(3.47)
7

---

Robustness to varying levels of hetero- geneity. In the label shift non-lID set- ting, we use Dirichlet distribution with parameter a 0 to simulate the hetero- geneity level over 10 clients. As a be- comes smaller, the label distribution be- comes more heterogeneous. We evaluate our proposed FedPCL with FedRep and Solo by varying E {0.5, 2,5, 10) on 10 clients. Figure ((a) suggests that Fed- PCL achieves the best performance for all values of (.. Moreover, FedPCL. has the smallest deviation across several runs, which implies the stable convergence and robustness of FedPCL.
FedRep
FecAvi FeePCL
Avg test acc
FedPCL
Avg test scc
Non-lid level
Num of clients
(a)
(b)
Figure 2: Test accuracy on Digit-5 under varying non-lII levels (leff) and varying numbers of clients (right). Both are conducted in the label shift non-III setting. In (a), non-IID level is controlled by a, ranging from 0.5 to 10. In (h), the number of clients (m) ranges from to 80.
Effeet of varying numbers participating clients. We also compare FedPCL with the vanilla FedAvg by varying the number of clients (m) from 5 to 80 as shown in Figure 2(b) The non-IID level remains the same when more clients participate in the training procedure. The results show that the average test accuracy of FedPCL is higher than that of FedAvg by about 15%. Furthermore, the deviation of FedPCL across three runs much lower than that FedAvg, especially when m becomes larger. This indicates the scalability of the lightweight framework and the ability of FedPCL to adapt to large-scale FL system.
5.3 Integrating Backbones with Various Architectures
The above experiments use backbones with the same architecture but pre-trained on different source data, showing the capability of FedPCL on integrating knowledge from different source domains. In Table 4 we also show that our proposed lightweight framework capable of (i) using pre-trained models with different archi- tectures, i.e., a two-layer CNN, AlexNet (75), and VGGNet [76); (ii) integrating large-scale pre-trained models, i.c., ViT [16] (77). to en-
Table 4: Integrating backbones with various architec- tures into proposed framework. Experiments are implemented with FedPCL. under feature & label shift non-IID setting.
Backbone- -omonn
Digit-5
Office- 10
[ResNet18 QuickDraw, ResNet18 AAirerrft, [
42.87(1.47) 41.40(1.19)
[MLP IntgeenageNet, AlexNet-ImageNet, VGG1 [GGll-ImageNer)
hance the performance without the need of huge computation resources to train or fine-tune them. Pre-trained models with different representation abilities should be adaptively leveraged for dif- ferent tasks to achieve expected performance. They can be selected based on small set of data before large-scale training.
55.80(2.09) 70.11(1.78)
[tiny-ViT ViT-ImageNNet, small-ViT- -i--------aaeeet,
40.96(2.87) 84.63(2.57)
base-ViT-Image VYT-Imageeeet]
5.4 Ablation Study
In this section, we present the results for various ablation experiments test the effect global/local prototypes and the contrastive loss. More results can be found in Appendix A.2.3
Effect of the contrastive loss. We test the perfor- mance of the local loss function different forms: (1) Cross Entropy loss: (2) Cross Entropy loss + Pro- toDist term, which takes the distance between proto- type and fused representation as an additional term to regularize the cross entropy loss; (3) Supervised Contrastive loss that only uses local embedding for contrastive learning; (4) Our prototype-wise con- trastive loss that uses both global and local proto- types for local contrastive learning. As shown in Table () our prototype- supervised contrastive loss outperforms others.
Table 5: Comparison between the cases when dif-
ferent local losses are used for local training. Ex- periments
Digit-5 dataset under the feature label shift non-IID setting where the Dirichlet parameter is 1, the number of clients is 5, and the number of pre-trained backbones is 3.
conducted
Loss Type
Aec
Cross Entropy
32.98(3.44)
Cross Entropy ProtoDist 24
43.94(3.02)
Supervised Contrastive 42
42.18(3.25)
Ours
45.22(3.65)

---

Effect of prototypes. To verify the effectiveness of different kinds of prototypes used for local training our proposed FedPCL, we compare the following three cases: (i) Only global prototypes computed at the server are used for local training; (ii) Only local prototypes aggregated at the server are used for local training; (iii) Both global and local prototypes are used for local training. All these three cases are implemented under three non-lll) settings. As shown in Table O
without global or local prototypes being used for local supervised contrastive learning, the performance drops 0.3%-2%, indicating that the knowledge conveyed by global prototypes and local prototypes can benefit the local learning framework from different perspectives.
Table 6: Comparison between the cases when only global prototypes are used for local training, only local prototypes from all clients are used for local training, and both of them are used for local training. Experiments are conducted on Digit-5 dataset under three non-IID settings where the Dirichlet parameter a is 1, the number of clients is and the number of pre-trained backbones is 3.
Prototypes
Feature shift non IID Label shift non-llD
Feature & Label shift non- IID
Global only
34.69(0.15
44.75(1.77) 44.52(1.73)
43. (3.099 43,08(3.87)
Local only
55,01
Global and local
55.34(0.34)
45.35(1.58)
45.22(3.65)
5.5 Visualizing the Fusing Results of FedPCI.
To better understand how FedPCL fuses the representation output by backbones, visualize the normalized similarity scores under the feature shift non-IID setting. The heatmap in Figure 3 sum- marizes the values of the normalized similarity scores, computed by the in- ner product between local prototypes in the single backbone case and the multi- backbone case. For example, in Fig- ure 3[a] the element 0.5 on row and column represents the similarity score between the local prototype computed when only one backbone pre-trained in Quickdraw is available and the local prototype computed when three backbones pre-trained in Quickdraw, Aircraft, and Birds are available.
Qukckdraw-..4 0.4
0.5 0.5 0.6 0.5
0.6
Quickdraw- 0.5
0.6
-04
Aircraft-0.4
Aircraft-O8
0.5 0.4 0.4 0.3
0.2
Birds-B.2
0.1
0.1
0.1
Birds-
0.1
MNNST
USP
Synth
MRI M
0.0
Amazon
Calfech
DSLR
Webccam
(a) Digit-5
(b) Office-10 10
Figure Similarity scores generated two datasets: (a) Digit-S and (b) Office-10. Rows correspond to the pre-trained backhones (x(x) and columns correspond to different local datasets with feature shift.
We find that the backbone ore-trained on Quickdraw contributes more to the fused prototype, while the backbone pre- pre-trained on Birds contributes the least. Clients who a specific dataset show different levels of utilization when fusing the representations from various backbones.
5.6 Incorporating with Privacy-Preeervvine Techniques
To further eliminate concerns about privacy leakage, we also incorporate FedPCI. with privacy- preserving techniques to observe the variation in performance. Concretely, we add various random noise to the prototypes to be communicated and the original images, respectively. The results show that the performance of FedPCL remains high after noise injection. Details are given in Appendix A.2.4
6 Conclusion
To address the problems excessive computation and communication demands in current federated learning frameworks, we propose a lightweight framework that leverages multiple neural networks as fixed pre-trained backbones replace the learnable feature extractor. To customize the general rep- resentations generated by these backbones for each client, -lass-wise prototypes are shared across the clients and the server. To efficiently extract the shared knowledge from the prototypes, we develop Fed- PCL algorithm that uses contrastive learning at the client-side during the local update. Extensive exper- iments are conducted to show the superiority of FedPCL under the proposed lightweight framework.
This research has great potential to bring existing FL frameworks to a new paradigm and provide more options on local neural network architectures. However, we mainly focus on the vision task in this work. Methods for language tasks can be further explored in future studies. Considering the evident motivation and effectiveness of the lightweight framework and FedPCL, we believe this work is intriguing for future studies.

---

References
[1] H Brendan McMahan, Eider Moore, Daniel Ramage, al. Communication-efficci learning of deep networks from decentralized data. AISTATS, 2017.
Peter Kairouz, H Brendan McMahan, et al. Advances and open problems federated learning. Foundations and Trends in Machine Learning, 14(1-2)1 22:1-210,, 2021.
[3] Xiaoxiao Li, Meirui Jiang, Xiaofei Zhang, Michael Kamp, and Qi Dou. Fedbn: Federated learning on non-III features via local batch normalization. In ICLR, 2020.
[4] Chen Chen, Yuchen Liu, Xingjun Ma, and Lingjuan Lyu. CalFAT: Calibrated federated adversarial training with label skewness. In NeurlPS, 2022.
[5] Alireza Fallah, Aryan Mokhtari, and Asuman Ozdaglar. Personalized federated learning: A meta-learning approach. NeurlPS, 2020.
[6] Canh T Dinh, Nguyen Tran, and Tuan Dung Nguyen. Personalized federated learning with moreau envelopes. In NeurlPS, 2020.
[7] Liam Collins, Hamed Hassani, Aryan Mokhtari, and Sanjay Shakkottai. Exploiting shared representations for personalized federated learning. In ICML, 2021.
[8] Sangjoo Park, Gwanghyun Kim, Jeongsol Kim, Boah Kim, and Jong Chul Ye. Federated split task- agnostic vision transformer for COVID-19 CXR diagnosis. In Newr/PS, 2021.
[9] Bill Yuchen Lin, Chaoyang He, Zihang Zeng, Hulin Wang. Yufen Huang, Mahdi Soltanolkotabi, Xiang Ren, and Salman Avestimehr. FedNL.P: IP: A research platform for federated learning in natural language processing. arXiv preprint arXiv:2 104.0885, 2021.
[10] Han Xie, Jing Ma, Li Xiong, and Carl Yang. Federated graph classification over non-lID graphs. In NewrIPS, 2021.
[11] Fengwen Chen, Guodong Longr, Zonghan Wu, Tianyi Zhou, and Jing Jiang. Personalized federated learning with structure. In IJCAI, 2022.
[12] Guodong Long, Tao Shen, Yue Tan, Leah Gerrard, Allison Clarke, and Jing Jiang. Federated learning for privacy-preserving open innovation future digital health. In Humanity Driven Al, pages 113-133. Springer, 2022.
[13] Guodong Long, Yue Tan, Jing Jiang, and Chengqi Zhang. Federated learning for open hanking. In Federated learning, pages 240-254. Springer, 2020.
[14] Yuzheng Ren, Renchao Xie, F. Richard Yu, Tao Huang, and Yunjie Liu. Quantum collective learning and many-to-many matching game in the metaverse for connected and autonomous vehicles. /EEE TVT, pages 1-12,202
[15] Jing Jiang, Shaoxiong Ji, and Guodong Long. Decentralized knowledge acquisition for mobile internet applications. World Wide Web, 23(5):2633 2669, 2020.
[16] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. In ICLR, 2020.
[17] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In ICCV, 2021.
[18] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016.
[19] Liu, William L Hamilton, Guodong Long, Jing Jiang, and Hugo Larochelle. universal representation transformer layer for few-shot image classification. In ICLR, 2020.
[20] Eleni Triantafiliou, Tyler Zhu, Vincent Dumoulin, Pascal Lamblin, Utku Evei, Kelvin Xu, ct al. Meta- dataset: A dataset of datasets for learning to leam from few examples. In ICLR, 2020.
[21] Kaiyang Zhou, Yongxin Yang. Timothy Hospedales, and Tao Xiang. Learning to generate novel domains for domain generalization. In ECCV, 2020.
10

---

[22] Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning. In NewrIPS, 2017.
[23] Mi Luo, Fei Chen, Dapeng Hu, Yifan Zhang, Jian Liang. and Jiashi Feng. No fear of heterogeneity; Classifier calibration for federated learning with Non-IID data, In NewrIPS, 2021.
[24] Yue Tan, Guodong Long, Lu Liu, Tianyi Zhou, Qinghua Lu, Jing Jiang, and Chengqi Zhang. FedProto: Federated prototype learning over heterogeneous devices. In AAAI, 2022.
[25] Ariadna Quattoni, Michael Collins, and Trevor Darrell. Transfer leamning for image classification with sparse prototype representations. In CVPR, 2008.
[26] Zhuoliang Kang, Kristen Grauman, and Fei Sha. Learning with whom to share in multi-task fcature learning. In ICML, 2011.
[27] Canh T Dinh, Nguyen H Tran, and Tuan Dung Nguyen. Personalized federated learning with moreau envelopes. In NeurlPS, 2020.
[28] Filip Hanzely, Slavomír Hanzely, Samuel Horváth, and Peter Richtárik. Lower bounds and optimal algorithms for personalized federated learning. In NeurIPS, 2020.
[29] Tian Li, Shengyuan Hu, Ahmad Beirami, and Virginia Smith. Ditto: Fair and robust federated learning through personalization. In ICML, 2021.
[30] Aviv Shamsian, Aviv Navon, Ethan Fetaya, and Gal Chechik. Personalized federated learning using hypernetworks. In ICML, 2021.
[31] Michael Zhang, Karan Sapra, Sanja Fidler, Serena Yeung, and Jose M Alvarez. Personalized federated learning with first order model optimization. In ICLR, 2020.
[32] Yihan Jiang. Jakub Konečnš, Keith Rush, and Sreeram Kannan. Improving federated learning personaliza- tion via model agnostic meta learning. arXiv preprint arXiv:1909.12490 2019.
[33] Xiao Liu, Fanjin Zhang, Zhenyu Hou, Li Mian, Zhaoyu Wang, Jing Zhang, and Jie Tang. Self-supervieed learning: Generative or contrastive. IEEE TKDE, 2021.
[34] Enze Xie, Jian Ding, Wenhai Wang, Xiaohang Zhan, Hang Xu, Peize Sun, Zhenguo Li, and Ping Luo. Detco: Unsupervised contrastive learning for object detection. In ICCV, 2021.
[35] Yixin Liu, Yu Zheng, Daokun Zhang, Hongxu Chen, Hao Peng, and Shirui Pan. Towards unsupervised deep graph structure learning. In The Web Conference, 2022.
[36] Yixin Liu, Zhao Li, Shirui Pan, Chen Gong, Chuan Zhou, and George Karypis. Anomaly detection on attributed networks via contrastive sele-supervised leamning. IEFE TNNLS, 3366):2378-2392, 2021.
[37] Yizhen Zheng, Shirui Pan, Vincent Cs Lee, Yu Zheng, and Philip S Yu. Rethinking and scaling up graph contrastive learning: An extremely efficient approach with group discrimination. NeurlPS, 2022.
[38] Yizhen Zheng, Ming Jin, Shirui Pan, Yuan-Fang Li, Hao Peng, Ming Li, and Zhao Li. Towards graph 2021.
self-supervised learning with contrastive adjusted zooming. arXiv preprint arXiv:2111.
[39] R Devon Hjelm, Alex Fedorov, Samuel MariiiMarr Karan Grewal, Phil Bachman, Adam Trischler, and Yoshua Bengio. Learning deep representations by mutual information estimation and maximization. In ICLR, 2018
[40] Mang Ye, Xu Zhang, Pong C Yuen, and Shih-Fu Chang. Unsupervised embedding learning via invariant and spreading instance feature. In CVPR, 2019.
[41] Ting Chen, Simon Korbllihh, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In ICML, 2020.
[42] Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. Supervised contrastive learning. In NewrIPS, 2020.
[43] Qinbin Li, Bingsheng Hc, and Dawn Song. Model-contrastive federated learning. In CVPR, 2021.
[44] Xutong Mu, Yulong Shen, Ke Cheng, Xueli Geng, Jiaxuan Fu, Tao Zhang, and Zhiwei Zhang. Fedproc: Prototypical contrastive federated learning on non-III data. arXiv preprint arXiv:2109.1m 12273, 2021.
11

---

[45] Fengda Zhang, Kun Kuang, Zhaoyang You, Tao Shen, Jun Xiao, Yin Zhang, Chao Wu, Yueting Zhuang. and Xiaolin Li. Federated unsupervised representation learning. arXiv preprint arXiv:2010.08888 2020.
[46] Lu Liu, Tianyi Zhou, Guodong Long, Jing Jiang, and Chengqi Zhang. Learning to propagate for graph meta-learning. In NeurIPS, 2019.
[47] Bingcong Li, Bo Han, Zhuowei Wang, Jing Jiang, and Guodong Long. Confusable learning for large-class few-shot classification, In ECML PKDD, 2020.
[48] John Wieting, Mohit Bansal, Kevin Gimpel, and Karen Livescu. Towards universal paraphrastic sentence embeddings. In ICLR, 2015.
[49] Artem Babenko and Victor Lempitsky, Aggregating local deep features for image retrieval. In ICCV, 2015.
[50] Nghia Hoang. Thanh Lam, Bryan Kian Hsiang Low, and Patrick Jaillet. Learning task-agnostic embedding of multiple black-box experts for multi-task model fusion. In ICML, 2020.
[51] Umberto Michieli and Mete Ozay. Prototype guided federated learning of visual feature representations. arXiv preprint arXiv:2 089808 2021.
[52] Xu Han, Zhengyan Zhang. Ning Ding, Yuxian Gu, Xiao Liu, Yuqi Huo, Jiezhong Qiu, Liang Zhang, Wentao Han, Minlie Huang, et al. Pre-trainee models: Past, present and future. Al Open, 2021.
[53] Yuzheng Ren, Renchao Xie, Fei Richard Yu, Tao Huang, and Yunjie Liu, Green intelligence networking for connected and autonomous vehicles in smart cities. IEEE Transactions on Green Communications and Networking, 6(3):1591-1033 2022.
[54] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In ECCV, 2020.
[55] Jacob Devlin, Ming-Wa Wei Chang. Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In NAACL, 2019.
[56] Rishi Bommasani, Drew Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, al. On the opportunities and risks of foundation models. arXiv preprint arXiv:2108.072533 2021.
[57] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. arXiv preprint Xiv: 00t:222030000 203000000, 2021.
[58] Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In ECCV, 2014.
[59] Xinlei Chen, Hao Fang, Tsung-Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco captions: Data collection and evaluation server. arXiv preprint 1504v:1504.00335.. 2015.
[60] Jie Zhang, Song Guo, Xiaosong Ma, Haozhao Wang, Wenchao Xu, and Feijie Wu, Parameterized knowledge transfer for personalized federated learning. In NeurlPS, 2021.
[61] Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Amect Talwalkar, and Virginia Smith, Federated optimization in heterogencous networks. In MLSys, 2020.
[62] Guodong Long, Ming Xie, Tao Shen, Tianyi Zhou, Xianzhi Wang, and Jing Jiang. Multi-center federated learning: clients clustering for better personalization. World Wide Web, 2022.
[63] Jie Zhang, Chen Chen, Bo Li, Lingjuan Lyu, Shuang Wa, Jianghe Xu, Shouhong Ding, and Chao Wu. A practical data-free approach to one-shot federated learning with heterogeneity. arXiv preprint /2. 2021.
[64] Avishek Ghosh, Jichan Chung. Dong Yin, and Kannan Ramchandran. An efficient framework for clustered federated learning. In NewrIPS, 2020.
[65] Jie Ma. Guodong Long, Tianyi Zhou, Jing Jiang, and Chengqi Zhang. On the convergence of clustered federated learning, arXiv preprint 2202 2022.
[66] Ligeng Zhu, Zhijian Liu, and Song Han. Deep leakage from gradients. In NeurIPS, 2019.
12

---

[67] Chen Chen. Lingjuan Lyu, Han Yu, and Gang Chen. Practical attribute reconstruction attack against federated learning. IEEE Transactions on Big Data, 2022.
[68] Oliver Zhang, Mike Wu, Jasmine Bayrooti, and Noah Goodman. Temperature as uncertainty in contrastive learning. arXiv preprint arXiv:2110.044033 2021.
[69] Tao Lin, Lingjing Kong, Sebastian U Stich, and Martin Jaggi. Ensemble distillation for robust model fusion in federated learning. In NewrlPS, 2020.
[70] Yishay Mansour, Mehryar Mohri, Jac Ro, and Ananda Theertha Suresh. Three approaches for personaliza- tion with applications to federated learning. urXiv:2002.10119 2020.
[71] Boqing Gong, Yuan Shi, Fei Sha, and Kristen Grauman. Geodesic flow kernel for unsupervised domain adaptation. In CVPR, 2012.
[72] Xingchao Peng and Kate Saenko. Synthetic real adaptation with generative correlation alignment networks. In WACV, 2018.
[73] Xingchao Peng. Qinxun Bai, Xide Xia, Zijun Huang, Kate Saenko, and Bo Wang. Moment matching for multi-source domain adaptation. In 2019,
[74] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint 1rXiv:1412.9980 2014.
[75] Alex Krizhevsky, Iya Sutskever, and Geoffrey Hinton. Imagenet classification with deep convolutional neural networks. In NeurlPS, 2012.
[76] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recogni- tion. arXiv preprint arXiv: 44444095566 2014.
177] Liangqiong Qu, Yuyin Zhou, Paul Pu Liang, Yingda Xia, Feifei Wang. Li Fei-Fei, Ehsan Adeli, and Daniel Rubin. Rethinking architecture design for tackling data heterogeneity in federated learning. In CVPR, 2022.
[78] Hongyi Wang, Mikhail Yurochkin, Yuekai Sun, Dimitris Papailiopoulos, and Yasaman Khazaeni. Federated learning with matched averaging. In ICLR, 2020.
[79] Jonas Jongejan, Henry Rowley, Takashi Kawashima, Jongmin Kim, and Nick Fox- Fox-Gieg. The quick, draw!-Al experiment. Mount View, CA, accessed Feb, 17(2018)4, 2016.
[80] Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi. Fine- fine-grained visual classification of aircraft. arXiv preprint 1306:v:3300.1111 2013.
[81] Tian Li, Maziar Sanjabi, Ahmad Beirami, and Virginia Smith. Fair resource allocation in federated learning. In ICLR, 2019.
[82] Bo Han, Quamming Yao, Xingrui Yu, Gang Niu, Miao Xu, Weihua Hu, Ivor Tsang, and Masashi Sugiyama. Co-teaching: Robust training of deep neural networks with extremely noisy labels. In NewrIPS, 2018.
[83] Zhuowei Wang. Jing Jiang, Bo Han, Lei Feng. Bo An, Gang Niu, and Guodong Long. SLmiNLL: framework of noisy-label learning by semi-supervised learning. Transactions on Machine Learning Research, 2022.
[84] Zhuowei Wang. Tianyi Zhou, Guodong Long, Bo Han, and Jing Jiang. FedNoiL: A simple two-level sampling methad for federated learning with noisy labels. arXiv preprint arXiv:2205. 101 2022.
[85] Hanting Chen, Yunhe Wang. Tianyu Guo, Chang Xu, Yiping Deng, Zhenhua Liu, Siwei Ma, Chunjing Xu, Chao Xu, and Wen Gao. Pre-trained image processing transformer. In CVPR, 2021.
[86] Kaichao You, Yong Liu, Jianmin Wang, and Mingsheng Long. Logme: Practical assessment of pre-trained models for transfer learning. In ICMI, 2021.
[87] Gary Cheng, Karan Chadha, and John Duchi Federated asymptotics: a model to compare federated learning algorithms. arXiv preprint arXiv:2108.077 2021.
[88] Lin Zhang, Li Shen, Liang Ding, Dacheng Tao, and Ling- Yu Duan. Fine-tuning global model via data-frree knowledge distillation for non-lID foderated learning. In CVPR, 2022.
13


Reviews:
[{"review_id": "s1W9GunBrhA", "reviewer": "Reviewer_R1m9", "summary": "This paper proposes a lightweight federated learning framework where clients jointly learn to fuse the representations generated by multiple fixed pre-trained models rather than training a large-scale model.  The paper is generally well-written and in a good shape. Extensive experiments were conducted to show the effectiveness of the proposed FedPCL. \n\n", "questions": "1. The generalization bound of FedPCL given in Theorem 4.1 is unclear without sufficient analysis. The authors claimed that ”the bound is associated with the complexity of training models and the size of prototype size”. The reviewer believe that this is a quite straightforward observation even without Theorem 4.1. Then, what is the purpose of Theorem 4.1. The reviewer are encouraged to analyze the convergence of FedPCL with the fixed pre-trained models. \n\n2. In addition, the derivations of Theorem 4.1 in the appendix is also not clear. For example, the derivation of (13) by McDiarmid’s inequality and the derivation of (14) by Rademacher complexity are not clear.  \n", "limitations": "This work adequately addressed the limitations. The authors developed a lightweight federated learning framework to reduce the computation and communication costs and integrated pre-trained models to extract prototypes for federated aggregation. The is a new try for federated learning. ", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: Using multiple pre-trained models to capture client-specific and class-relevant information is new in federated learning. FedPCL enjoys light computation and communication costs, compared to most other federated learning.  \n\nWeaknesses:  \n1.\tApart from the multiple pre-trained models, FedPCL is built on the idea of prototypical learning and contrastive learning, which are not new in federated learning. \n2.\tThe performance of FedPCL heavily relies on the selection of different pre-trained models, limiting its applications to more wide areas. As shown in Table 4, the model accuracy is quite sensitive to the pre-trained models. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "hoJanweX8JV", "reviewer": "Reviewer_9cCn", "summary": "This paper proposes a lightweight framework that enables federated learning from multiple pre-trained models, which significantly reduces communication and computation costs and makes it possible to integrate large-scale models into federated learning framework. To efficiently learn and share learned knowledge across clients under this framework, a prototype-based contrastive learning algorithm is developed to fully leverage the representation ability of pre-trained models in the latent space. A comprehensive empirical study is conducted and shows promising results achieved by the proposed algorithm. \n", "questions": "1. Existing SOTA personalized FL methods that achieve high performance in traditional non-IID settings, e.g., pFedMe, FedRep, do not work well under this lightweight framework. What is their limitation when using pre-trained models? Can this be fixed by introducing some simple but efficient techniques?\n\n2. Is it possible to allow clients to have different pre-trained models and keep sharing knowledge in this manner? This would be more common in real-world cases.\n", "limitations": "The authors have well addressed the limitations of this work in terms of specific target tasks and illustrated the potential directions of their future studies.", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Pros:\n1. The problem studied in this paper is interesting and important. This work first proposes to integrate pre-trained models into federated learning to form a lightweight FL framework, which expands the scope of current FL frameworks and makes an important contribution to the FL community.\n\n2. This paper has its originality and technically sound. The main claims regarding the proposed setting are well supported in the methodology and experimental parts. By transmitting prototype sets and optimizing the well-designed prototype-wise contrastive loss, class-specific knowledge can be fully extracted and shared during local training and server aggregation procedures, respectively.\n\n3. Extensive and comprehensive experiments are conducted. Experimental results on three benchmark datasets under different non-IID settings show superior or comparable performance achieved by FedPCL compared to existing FL/PFL methods. Sufficient ablation studies are provided to demonstrate the advantages of the loss and prototypes in the approach. Detailed data and model settings are provided in the appendix for reproduction. \n\n\nCons:\n1. It is better to provide more analysis on the performance improvement achieved by FedPCL in Section 5.2 and 5.4 to let the readers well understand how contrastive loss and prototypes make their contribution to the improvement. \n\n2. Although there is a relatively thorough literature review in the related work part, I prefer to see a discussion on the relation of this work and the fine-tuning techniques in FL since both of them have the potential to integrate pre-trained foundation models into the framework.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "LE68BouOkhj", "reviewer": "Reviewer_qfrZ", "summary": "This paper presents Federated Prototype-wise Contrastive Learning (FedPCL) which aim to exploit pre-trained models in FL environment for reducing communication (COM) costs.\n\n\nThe authors claim the following contributions:\n-  First to integrate pre-trained models into federated learning to reduce COM cost.\n- Proposing FedPCL.\n- Experiments showcasing FedPCL advantages over existing P/FL works.", "questions": "See above.", "limitations": "See above.", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "***Pros:***\n- The paper is well written and easy to follow.\n- The motivation of this paper is well explained - reducing COM liabilities is a major challenge in P/FL.\n- The authors provided generalization bound for FedPCL\n\n\n***Cons:***\n - The proposed method increases the computational demand from the clients (which in most cases are edge devices e.g. wearables, smartphones, etc.).\n- From eq (6,7) I understand that the server will be able to \"know\" which classes each client has. This raises some privacy issues. For example in medical applications we don't want the server to know which client has/not illness. \n- Missing citations of important and strong PFL papers (e.g. [1,2]).\n- Problematic results in the experimental part - in lots of cases Solo (local training) outperforms most PFL methods.\n- It is not clear if and how the authors performed hyperparameter tuning for FedPCL / baseline approaches. In addition, some experimental details are missing such train/val/test split, etc.\n- Experiments were conducted on limited numbers of clients (6 max). Rea-life P/FL systems obtain thousands / millions of clients. \n\n\n[1] Personalized federated learning using hypernetworks.\n\n[2] Personalized Federated Learning with First Order Model Optimization.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["Privacy and Security (e.g., consent)"]}, "extra_text": {}}, {"review_id": "5RED7_epHbt", "reviewer": "Reviewer_TqKW", "summary": "This paper studies a specific federated learning problem, which considers to leverage the large-scale pre-trained models. It builds a federated prototype-wise contrastive learning framework to capture more client-speciﬁc and class-relevant information. A generalization analysis has been provided to discuss the effect of the built prototypes. The authors conduct a range of experiments to demonstrate the performance of the proposed method over state-of-the-art performance.", "questions": "Please explain the questions in the weakness part listed above.", "limitations": "NONE", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n(1) By constructing a FedPCL mechanism, the proposed framework trains the model in a lightweight manner and simultaneously shares the knowledge across the clients. Besides, the projection network provides the personalized feature selection for each client.\n\n(2) A theoretical analysis points out the how the proposed prototype-based learning method affects the generalization bound, which helps how this type of methods differ from previous non-prototype-based works.\n\n(3) A range of experiments have been conducted to show the promise of the proposed method over the state-of-the-art methods, and FedPCL can be further boosted by integrating different pretrained models.\n\nWeakness\n(1) The relationship between two loss functions in Eq.(8) and Eq.(9) have not sufficiently explained, since the latter provide the whole supervision information about the global C. In this perspective, why we need Eq.(8).\n\n(2) Another question is about the pre-trained backbones. It seems that the proposed framework actually can be applied to the wider scenarios where there are not pre-trained models as the feature extractors. Limiting the model to this scenario raises the concerns that whether the prototype-based federated learning methods easily overfits the local data without the auxiliary feature extractors. The authors could discuss more about this point.\n\n(3) It lacks of a baseline like [1] and the authors should discuss how your method is different from this work and other prototype-based methods like FedProto and why yours is better.\n\n[1] FedProc: Prototypical Contrastive Federated Learning on Non-IID data", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}]

Author rebuttal:

Decision: Accept