FedProf: Selective Federated Learning with Representation Profiling
Authors: Wentai Wu
Ligang He
Weiwei Lin
carsten maple
Rui Mao
Venue: ICLR 2022 Submitted (2022)
Keywords: federated learning
neural network
representation learning
distributed computing

Abstract: Federated Learning (FL) has shown great potential as a privacy-preserving solution to learning from decentralized data that are only accessible to end devices (i.e., clients). In many scenarios however, a large proportion of the clients are probably in possession of low-quality data that are biased, noisy or even irrelevant. As a result, they could significantly slow down the convergence of the global model we aim to build and also compromise its quality. In light of this, we propose FedProf, a novel algorithm for optimizing FL under such circumstances without breaching data privacy. The key of our approach is a data representation profiling and matching scheme that uses the global model to dynamically profile data representations and allows for low-cost, lightweight representation matching. Based on the scheme we adaptively score each client and adjust its participation probability so as to mitigate the impact of low-value clients on the training process. We have conducted extensive experiments on public datasets using various FL settings. The results show that FedProf effectively reduces the number of communication rounds and overall time (up to 4.5x speedup) for the global model to converge and provides accuracy gain.

Under review as a conference paper at ICLR 2022
FEDPROF: SELECTIVE FEDERATED LEARNING WITH REPRESENTATION PROFILING
Anonymous authors Paper under double-blind review
ABSTRACT
Federated Learning (FL) has shown great potential as privacy-preserving solu- tion to learning from decentralized data that only accessible to end devices (i.e., clients). In many scenarios however, a large proportion of the clients are probably in possession of low-quality data that are biased, noisy even irrel- evant. As result, they could significantly slow down the convergence of the global model we aim build and also compromise its quality. In light of this, we propose FEDPROF, a novel algorithm for optimizing FL under such circumstances without breaching data privacy. The key of our approach is data representation profiling and matching scheme that uses the global model to dynamically profile data representations and allows for low-cost, lightweight representation matching. Based the scheme adaptively score each client and adjust its participation probability so as to mitigate the impact of low-value clients on training pro- cess. We have conducted extensive experiments FL settings. The results show that FEDPROF effectively reduces the number communication rounds and overall time (up to 4.5x speedup) for the global model to converge and provides accuracy gain.
public datasets using various
INTRODUCTION
With the advances in Artificial Intelligence (Al), we are seeing a rapid growth in the number of AI-driven applications as well as the volume of data required train them. However, a large pro- portion of data used for machine learning are often generated outside the data centers by distributed resources such as mobile phones and loT (Internet Things) devices. generated by IoT devices will account for 75% of the total in 2025 (Meulen) 2018).. Under this circumstance, it will be very costly to gather all the data for centralized training. More importantly, moving the data out of their local devices (e.g., mobile phones) is now restricted by law in many countries, such as the General Data Protection Regulation (GDPR enforced in EU.
predicted that the data
We face three main difficulties to learn from decentralized data: i) massive scale of end devices; ii) limited communication bandwidth at the network edge; and iii) uncertain data distribution and data quality. As an promising solution, Federated Learning (FL. (MeMahan et al.) 2017) is framework for efficient distributed machine learning with privacy protection (i.e., no data exchange). A typical process of FL organized rounds where the devices (clients) download the global model from the server, perform local training on their data and then upload their updated local models to the server for aggregation. Compared traditional distributed learning methods, FL naturally more communication-effice efficient at scale (Konečný et al.) 2016; Wang et al. 2019). Nonetheless, several issues stand out.
MOTIVATION
1) FL is susceptible to biased and low-quality local data. Only fraction of clients selected for a round of FL (involving too many clients leads to diminishing gains (Li et al. 2019)). The stan- dard FL algorithm (McMahan et al. 2017) selects clients randomly, which implies that every client (and its local data) is considered equally important. This makes the training process susceptible local data with strong heterogeneity and of low quality (e.g., user-generated texts (Hard et al.

---

Under review as a conference paper at ICLR 2022
Accuracy global model
0.8
0.6
0.4
unbiased and noiseless
biased local data
0.2
noisy local data
biased and noisy
0
2
8 10
12
14
16
18
20
Communication rounds
Figure I: (Preliminary experiment) The global model's 's convergence under different data conditions. We ran the FL. process with 100 clients to learn CNN model on the MNIST dataset, which is partitioned and allocated to clients in four different ways where the data 1) original (black line): noiseless and evenly distributed across the clients, 2) biased (magenta linc); locally class- imbalanced, 3) noisy (blue line): blended with noise, 4) biased and noisy (red line). The noise (if applied) covers 65% of the clients; the dominant class accounts for >50% of the samples for biased local data. The fraction of selected clients is 0.3 for each round.
2018) and noisy photos). In some scenarios, local data may contain irrelevant or even adversarial samples (Bhagoji et al., 2019 Bagdasaryan et al.) 2020) from malicious clients (Fang et al.) [2020] Bagdasaryan et al. 2020 Tolpegin et al., 2020). Traditional solutions such as data augmentation YYo et al. 2020) and ramsampling (Lin et al., 2017) prove useful for centralised training but ap- plying them to local datasets may introduce extra noise (Cui et al.) 2019) and increase the risk of information leakage (Yu et al. (02T). Another naive solution is directly exclude those low-value clients with low-quality data, which, however, is often impractical because i) the quality of the data depends on the learning task and is difficult to gauge; ii) some noisy or biased data could be useful to the training at early stages (Feelders 1996); and iii) sometimes low-quality data are very common across the clients.
In Fig. I we demonstrate the impact of involving "low-value" clients running FL over 100 clients to learn a CNN model on MNIST using the standard FEDAVG algorithm. From the traces we can that training over clients with problematic or strongly biased data compromise the efficiency and efficacy of FL, resulting in an inferior global model that takes more rounds to converge.
2) Learned representations can reflect data distribution and quality. Representation learning is vital to the performance of deep models because learned representations can capture the intrinsic structure of data and provide useful information for the downstream machine learning tasks (Bengio et al. 2013). In ML research, The value of representations lies in the fact that they characterize the domain and learning task and provide task specific knowledge (Morcos al. 2018 Kornblith al, 2019). In the context of FL, the similarity of representations are used for refining the model update rules (Li et al. 2021 Feng Yu 2200)), but the distributional difference between representations of heterogeneous data is not yet explored.
Our study also motivated by a key observation that representations from neural networks tend to have Gaussian patterns. As a demonstration trained two different models (LeNet-5 and ResNet- 18) on two different datasets (MNIST and CIFAR-IO0 100) separately. Fig. Showsows the neuron-wing wise distribution of representations extracted from the first dense layer (FC-1) of LeNet-5. Fig. 2b shows the distribution of fused representations (in a channel-wise manner) extracted from a plain convolution layer and residual block of ResNet-18.
These observations motivate us to study the distributional property of data representations and use it means to differentiate clients" value.
2

---

Under review as a conference paper at ICLR 2022
1000
Convt
Ross Mook
(a) Representations from FC-1 of a LeNet-5 (b) Fused representations from a standard convolu- model after being trained for 1, 6 and 10 epochs tion layer (Ist row) and residual block (2nd row) of ResNet-18 model trained for 100 epochs CIFAR-100.
MNIST.
Figure 2: (Preliminary experiment) Demonstration of learned representations from distributional perspective. The representations are generated by forward propagation in model evaluation. Each box corresponds to randomly sampled element in the representation vector.
1.2 CONTRIBUTIONS
Our contributions are summarized as the following:
We first provide theoretical proof for the observation that data representations from neural networks tend to follow Gaussian distribution, based on which propose representation profiling and matching scheme for fast, low-cost comparison between different representa- tion profiles.
We present novel FL algorithm FEDPROF that adaptively adjusts clients' participation probability based representation profile dissimilarity.
Results of extensive experiments show that FEDPROF reduces the number of communica- tion rounds by up to 77%, shortens the overall training time (up to 4.5x speedup) while increasing the accuracy of the global model by up to 2.5%.
2 RELATED WORK
Different from traditional distributed training methods (e.g., Alistarh al.
(2017), Wu et al. (2018);
Zheng al. (2017)), Federated Learning assumes strict constraints of data locality and limited com-
munication capacity (Konečný al., 2016). Much effort has been made in optimizing FL and covers
a variety of perspectives including communication (Konečný et al.] [0166] Niknam et al.] 20.0 Cui
2021a Leroy al.] 2019 Luping al. 2019 flexible aggregation (Wang et al. 2019 Wu et al. 2021b) and personalization (Fallah et al.
al. 2021). , update nules (Li et al.
2020; Wu et al.
2020
Tan al. 2021 Deng et al. 2020).
The control of device participation is imperative in cross-device FL scenarios (Kairouz et al.
2019
Yang al. 2020) where the quality of local data uncontrollable and the clients show varied value for the training task (Tuor et al. 2020). To this end, the selection of clients is pivotal to the
convergence of FL over heterogeneous data and devices (Nishio & Yonetani 2019 Wang et al., 2020b; Chai et al. 2019; Acar et al.) 2020). Non-uniform client selection is widely adopted in existing studies (Li et 2020, Goetz al. 2019 Cho et al., 2020 Li al.) 2019 Chen et al. 2020b Wang et al.) (2200a) and has been theoretically proven with convergence guarantees (Chen ct al. 2020b Li et al. 2019). Many approaches sample clients based on their performance (Nishio & Yonetani, 2019 Chai et al. 2020) or aim to jointly optimize the model accuracy and training time (Shi et al.; 2020 Chen et al. 2020a; 2021). ). popular strategy to use loss as the information to guide client selection (Goetz et al. 2019 Lai al. 2021 Sarkar et al. 2020). For example, AFL (Goetz et al. 2019) prioritizes the clients with high loss feedback on local data, but it is potentially susceptible to noisy and unexpected data that yield illusive loss values. Data representations

---

Under review as a conference paper at ICLR 2022
useful in the context of FL for information exchange (Feng & Yul 2020) objective adaptation. For example, Li et al. 2021 introduces representation similarities into local objectives. This contrastive learning approach guides local training to avoid model divergence. Nonetheless, the study the distribution data representations still lacking whilst its connection to clients' training value is hardly explored either.
DATA REPRESENTATION PROFILING AND MATCHING
In this paper, consider typical cross-device FL setting (Kairouz et al. 2019) in which multiple end devices collaboratively perform local training on their own datasets D;, = 1,2,..., n. The server owns a validation dataset D* for model evaluation. Every dataset only accessible to its owner.
Considering the distributional pattern of data representations (Fig. and the role of the global model in FL. we propose to profile the representations of local data using the global model. In this section, we first provide theoretical proof to support our observation that representations from neural network models tend to follow Gaussian distribution. Then we present novel scheme to profile data representations and define profile dissimilarity for fast and secure representation comparison.
3.1 GAUSSIAN DISTRIBUTION OF REPRESENTATIONS
We first make the following definition to facilitate analysis. Definition 1 (The Lyapunov's 's condition). A set of random variables {Z1. Zz...., Zu} satisfy the Lyapunov's 's condition if there exists 6 such that
lim
E
[1
- pm 2+5] O,
(1)
224
k=1
where μi = E[Z,], of E[(Z ) and E
The Lyapumov's 's condition can be intuitively explained as limit the overall variation (with Z Hs uu|2t8 [2+8 being the (2 + -)h moment of Z of a set of random variables.
Now present Proposition | and Proposition 2 The Propositions provide theoretical support for our representation profiling and matching method to be introduced in Section 3.2
Proposition The representations from linear operators (e.g.. a pre-activation dense layer or plain convolutional layer) in neural network tend to follow the Gaussian distribution if the layer's 's weighted inputs satisfy the Lyapunov's 's condition.
Proposition 2.
The fused representation from non-linear operators (e.g., hidden layer of LSTM or a residual block of ResNet) in neural network tend to follow the Gaussian distribution if the layer's output elements satisfy the Lyapunov's condition.
The proofs of Propositions I
and and are provided in Appendices A. and A.2 respectively.
We base our proof on the Lyapunov's CLT which assumes independence between the variables. The assumption theoretically holds by using the Bayesian network concepts: let X denote the layer's input and Hę denote the k-th component in its output. The inference through the layer produces dependencies X + Ha for all k. According to Local Markov Property, we have H; independent of any H, (j & i) given X. Also, the Lyapunov's 's condition typically met when the model is properly initialized and batch normalization applied. Next, we discuss the proposed representation profiling and matching scheme.
3.2 DISTRIBUTIONAL PROFILING AND MATCHING
Based on the Gaussian pattern of representations, we compress the data representations statistically into a compact form called representation profiles. The profile produced by the global model w on
Fused representations refer to the sum of elements in the original representations produced by single layer (channel-wise for residual block).
4

---

Under review as a conference paper at ICLR 2022
dataset D, denoted by RP(w, D), has the following format:
RP(w, D) (N(u),
where y is the profile length determined by the dimensionality of the representations. For example, is equal to the number of kernels for channel-wist wise fused representations from convolutional layer. The tuple of) contains the mean and the variance of the i-th representation element.
Local representation profiles are generated by clients and sent to the server for comparison (the cost of transmission is negligible considering each profile is only x 8 bytes). Let RPs denote the local profile from client k, and RP" denote the baseline profile (generated model evaluation) on the server. The dissimilarity between RP& and RP* denoted by div(RP, RP*). ) is defined as:
where KL(.) denotes the Bacllback-Leibles (KL) divergence. An advantage of our profiling scheme that much simplified KL divergence formula can be adopted because the Gaussian distribution property (see Roberts & Penny (2002) Appendix B) for details), which yields:
(0; (k)
+ (44 (K) -2))²
KT(N?" (W?)) = log of [R]
+
(4)
2(0;)²
Eq. computes the KL. divergence without calculating any integral, which is computationally cost-efficient. Besides, the computation of profile dissimilarity be performed under the Homo- morphic Encryption for minimum knowledge disclosure (see Appendix for details).
4 THE TRAINING ALGORITHM FEDPROF
Our research aims to optimize the global model over a large group of clients (datasets) of disparate training value. Given the client set U((V' = N). let De denote the local dataset client A: and D the validation set on the server, We formulate the optimization problem in (5) where the coefficient p differentiates the importance of the local objective functions F& (w) and depends on the data in
D₁. Our global objective is in a sense similar to the agnostic learning scenario (Mohri et al.) 2019) where a non-unifora mixture of local data distributions is implied.
N
arg min F(w) PeFFu(w), t0
(5)
km1
where is the parameter set of the global model. The coefficients {px J add up to 1. Fl(w) is client k's local objective function training based on the loss function E(-):
1
Fe(w)=
Σ (x..D)ED
((x, M65 w),
[Dal
Involving the "right" clients facilitates the convergence. With this motivation we score each client with A each round based on the representation profile dissimilarity:
A exp( div (Vk),, RP" (v:)),
where Uy is the version of RPt (i.c., the version of the global model that client k receives) and preference factor deciding how biased the selection strategy needs to towards the clients with small profile dissimilarity (i.e., higher training value). With = O. our strategy is equivalent to random selection. The scores connect the representation profiling and matching scheme to the design of the selective client participation strategy adopted in our FL training algorithm FEDPROF, (see Appendix B for a detailed version with both client and server
which is outlined in Algorithm | processes). The key steps of our algorithm are local representation profiling (line 5), baseline repre- sentation profiling (line 9) and client scoring (line 3). Fig. 3 iillusstrtees the workflow of the proposed algorithm from representation profiling, matching scheduling.
5

---

Under review as a conference paper at ICLR 2022
benchmark data
baseline footprints
RP
D
0.75
the global model
AP2
0.49
match & rate
RPn
0.15
local data
representations local profiles
client credits
Credit-based
Model distribution
Update footprints
Local training & model uploading
Model aggregation
client selection
and credits
Round +1
Figure 3: The workflow of the proposed FEDPROF algorithm.
Algorithm 1: the FEDPROF algorithm
Input: maximum number of rounds Toa.. iterations per round E, fraction C
Initialize global model and generate baseline profile R.P
Collect initial representation profiles [ReP from all clients for round + T man do
Update client scores {Ax heE and compute A EKEU A N C' clients by probability distribution A kEE
+ Choose for client k in S in parallel do w, T 1)
R&& + uuddateProiilet
Wi < localTraining(k ks w, F))
end
Collect local profiles from the clients in S
Update via model aggregation
Evaluate w and update RP*
end
10 return w
The convergence rate of FL algorithms with opportunistic client selection (sampling) has been ex- tensively studied in the literature (Li et al. 2019 Chen al., 2020b Wang et al. 2019). Inspired by these studies. we present Theorem to guarantee the global model's 's convergence for our algorithm. Similar Stich al. 2018);; Zhang et (2033); Li al. (2019), we assume that {Fk k= are L-smooth and j--trronly convex and that in expectation, the variance of local stochastic gradients are bounded by e² and their squared norms are bounded by G2. For each local update step & we define S(t) as the set of selected clients in the associated round.
Theorem 1. Using partial aggregation and a strategy N that selects clients by the probability distri- bution {ge N km the global model w(t) converges in expectation by having Ok = Pk. an aggregation interval E 2 1. and a decreasing step size (learning rate) 1): pu(++)))
2(B + +C)
+ 1
E[F(w[t))] - F* < (7+t)
(8)
112
2
where E E (nF = 1,2,... max 8L a
F.)
B, -NN
PE -
C $ F. r p* -N Px A (1) w 112, K
N.C.
The proof of Theorem 1 is provided in Appendix

---

Under review as a conference paper at ICLR 2022
5 EXPERIMENTS
We conducted extensive experiments to evaluate FEDPROF under various FL settings. Apart from FEDAVG (McMahan et al., 2017). we also reproduced several state-of-the-art FL algorithms for comparison. For fair companson, the algorithms are grouped by the aggregation method (i.e., full aggregation and partial aggregation) and configured following the papers (if any). Table [ summariizs these FL algorithms. Note that our algorithm adapt to both aggregation methods.
settings in their
Table 1: The implemented FL algorithms for comparison
Algorithm FEDAVG (McMahan et al.
Aggregation method
Rule of selection
2017)
full aggregation
random selection submission order
CFCFM (wu al. 202Tb)
full aggregation
FEDAVG-RP (Li ci al. 2019)
partial (Scheme II) partial aggregation partial with momentum partial with momentum full/partial aggregation
random selection weighted random by data ratio
FEDPROX [Li et al. 2020
FEDADAM (eeroy et al. 2019
random selection
AFL (Geetz et al. 2019
local loss valuation weighted random by score
FEDPROF (ours)
5.1 EXPERIMENT SETUP
We built simulated FL system and implemented the algorithms based the Pytorch framework (Build 1.7.0). We first set up a Small-scale Task (S-Task) to learn a multi-layer seed-forward network model from decentralized sensor data for predicting carbon monoxide (CO) and nitrogen oxides (NOx) emissions using the GasTurbing datarrstttttsteet...... Next, we set up a Large-scale Task (L-Task) to train a CNN over large population of user devices for image classification (EMNIST In both tasks, data sharing is not allowed between any parties. The data are non- -ID across the end devices We introduce diversity of noise into the local datasets on end devices to simulate the discrepancy in data quality. The S-Task is performed over 50 sensor clients and 50% of them produce noisy data (including 10% invalid). The population for L-Task contains 1000 end devices across which the data spread with strong class imbalance roughly 60% of the samples each device fall into the same class. 15% of the local datasets in the L-Task are irrelevant images whereas another 45% of them low-quality images (blurred or affected by calt-and-pepper noise). Considering the population of the clients, the setting of the selection fraction C' is based on the scale of the training participants suggested by [Kiirouz et al. (2019). For both tasks, the clients are heterogeneous in terms of both performance and communication bandwidth. More experimental settings are listed Table in Appendix E where the environment setup is also given details.
5.2 EVALUATION RESULTS
We evaluate the performance our FEDPROF algorithm terms of the efficacy (best accuracy achieved) and efficiency (costs for convergence) in establishing global model for the two tasks. Tables and the average results of multiple runs with standard deviations for our algorithm. Figs. 4 and the accuracy traces from the round-wise evaluations of the global model.
I) Convergence in different aggregation modes: Our results show great difference in convergence rate under different aggregation modes. From Figs. 4 and 5 we observe that partial aggregation facilitates faster convergence of the global model than full aggregation, which is consistent with the observations made by Li et al. (2019). The advantage is especially obvious in the L-Task where partial aggregation requires much fewer communication rounds to reach the 90% accuracy. Our FEDPROF algorithm yields the fastest convergence both groups of comparison for both tasks because selective participation benefits both aggregation methods.
We use the digits subset of EMNIST. 'In the S-Task, local datasets are of different sizes that follow Gaussian distribution. In the T.-Task, local data largely imbalanced where around 60% of the samples each client have the same class label.
"hups://wwwnist, http://www..isstoiiillnpdddsss www.gooiii///ouue----d----

---

Under review as a conference paper at ICLR 2022
The results of running the S-Task. The best accuracy is achieved by running for long enough. Other metrics are recorded upon reaching the target accuracy (80% for the S- s-Task).
Table 2:
Full aggregation
C=0.2
C=0.3
For accuracy @0.8
For accuracy @0.88
Best acc
Best acc
Rounds
Time(s)
E(Wh)
Rounds
Time(s)
E(Wh)
FEDAVG CFCFM
0.805
56
2869.66
2.87
0.8(1
52
3160.01
4.12
0.806
39
1230.81
1.61
0.802
42
1495.34
2.91
Ours
0.824
16
803.74
0.80
0.827
12
701.18
0.94
(std.)
6. 6.13E-03
2.06
139.76
0.14
8.01E-03
2.94
122.63
0.20
Partial aggregation
C=0.2
C=0.3
For accuracy @0.88
For accuracy @00.8
Best acc
Best acc
Rounds
Time(s)
E(Wh)
Rounds
Time(s)
E(Wh)
FEDAVG-RP
0.819
13
735.48
0.71
0.817
466.90
0.64
FEDPROX
0.821
16
899.93
0.79
0,810
16
841.65
1.08
FEDADAM
0.818
8
438.20
0.42
0.819
12
667.00
0,94
AFL
0.816
6
313.81
0.30
0.813
298.81
0.42
Ours
0.844
5
283.76
0.27
0.841
235.22
0.35
(std.)
1.70E-03
0.47
30.69
0.03
7.07E-03
1.70
90.12
0.14
Table 3: The results of running the L-Task. The best accuracy is achieved by running for long enough. Other metrics are recorded upon reaching the target accuracy (90% for the L.-Task).
Full aggregation
C=0.05
C=0.1
For accuracy @0.9
@009
For accuracy @0.9
Best acc
Best acc
Rounds
Time(s)
E(Wh)
Rounds
Time(s)
E(Wh)
FEDAVG
0.906
213
10407.30
60.01
0.929
76
3894.26
43.16
CFCFM
0.923
251
8645.17
61.37
0.932
75
2675.56
37.62
Ours
0.926
98
4846.15
28.22
0.945
45
2295.03
25.98
(std.)
8.16E-05
1.89
96.57
0.43
4.71E-04
0.82
29.46
0.01
Partial aggregation
C=0.05
C=0.1
For accuracy@@l) @..9
For accuracy@0.. @0.9
Best acc
Best acc
Rounds
Time(s)
E(Wh)
Rounds
Time(s)
E(Wh)
FEDAVG-RP
0.937
12
572.90
3.29
0.938
12
603.94
6.57
FEDPROX
0.936
13
640.01
3.58
0.942
11
559.16
5.91
FEDADAM
0.940
12
599.96
3.47
0.939
12
608.85
6.76
AFL
0.952
10
479.73
2.73
0.944
9
476.62
5.26
Ours
0.962
8
383.94
2.26
0.962
8
413.16
4.64
(std.)
9.43E-04
0.471
17.67
0.15
8.16E-04
0.47
11.64
0.03
2) Best accuracy of the global model: Through the training process of FL, the global model is evaluated each round on the server. The best global model obtained thus far is stored on the server. As shown in the 2nd column of Tables 2 and 3 our FEDPROF algorithm improves the accuracy achieved by 2.5% when compared against the baselines (FEDAVG and FEDAVG-RP). The AFL algorithm uses a loss-oriented client selection strategy, which shows the closest performance to our algorithm in the L-Task but the worst accuracy in the S-Task.
3) Total communication rounds for convergence: The number of communication rounds required for reaching convergence is key indicator to the efficiency of FL. In the S-Task, our algorithm takes less than half the communication rounds required by other algorithms in most cases. In the L-Task with C=0.05, algorithm reaches 90% accuracy within 100 rounds whilst FEDAVG and CFCFM need more than 200. In this case, FEDPROF also achieves approximately 7% higher accuracy in 50 rounds. Partial aggregation turns out to be much more communication-effcieentt FEDAVG-RP
8

---

Under review as a conference paper at ICLR 2022
Full aggregation, Cu0.2
Full aggregation, Co0.3
Partial aggregation,
Partial aggregation,
0.86
0.82
0.78
0.3
0.78
0.8
0.82
Appurass
0.0
0.74
0.72
Faddfn 0FFFPM
0.72
0.74
0.70
Feddfo CFOFM
0.78
Fooldam
FooFrox
Feckuug-pp
0.25
0.78
FedPhix
Fastkgg-FF
0.7
20'5
0.74
APL
0.72
80
40
60
80
100
80
40 Communication rounds
Communicatin rounds
Communicatin rounce
Communication rounds
(b)
Figure 4: The traces of evaluation accuracy of the global model through 100 rounds in the S-Task using (a) the -ggreegttntnnnn method, and (b) the aggregatin method.
Full aggregation, C:0.05
Full aggregation, C:0.10
Partial aggregation, C.0.05
Partial aggregation, C=0.10
0.9
0.8
0.9
0.9
0.8
0.8
2 0.6
0.7
0.7
0.7
0.7
0.6
0.5
0.5
FedAug-Fit RiP
Fwdtug-RPP
0.4
0.3
AdAug
0.4
0.5
0.4
FedFrox
Feohdam
0.4
FedPhox
Fedecam
02
CFCFM
0.2
CFCFM
AFL
0.2
0urs
OUPS
0
50
100
150
200
250
300
56
100
150
200
250
300
20
30
Communication rounds
Communication rounds
Communication rounds
Communicatin rounds
(b)
Figure 5: The traces of evaluation accuracy of the global model in the L-Task using (a) the full- aggregation method (through 300 rounds), and (b) the partial aggregation method (through 50 rounds).
needs rounds to reach the accuracy target for the S-Task and 12 rounds for the L-Task, whilst our algorithm reduces the numbers to 5 and 8, respectively.
4) Total time needed for convergence: The overall time consumption closely related to total com- munication rounds needed for convergence and the time cost for each round. Algorithms requir- ing more rounds to converge typically take longer to reach the accuracy target except the case of CFCFM, which priorities the clients that work faster. Using FEDAVG as the baseline, CFCFM accelerates the training process by 2.1x whilst our algorithm provides a 4.5x speedup for achieving the same target accuracy (S-Task, C=0.3). FEDPROF also has clear advantage RP, FEDPROX and FEDADAM in the partial aggregation group where it shows 2.6x speedup over FEDAVG-RP in the S-Task with C=0.2.
FEDAVG-
5) Energy consumption of end devices: A main concern for the end devices, as the participants of FL, is their power usage (Watt hours). Full aggregation methods experience slower convergence and thus endure higher energy cost on the devices. For example, with a small selection fraction C=0.05 in the L-Task, FEDAVG and CFCFM consume over 60Wh reach the target accuracy. this case, our algorithm reduces the cost by more than half (28.22Wh). With the partial aggregation mode, the reduction by our algorithm is up to 62% (S- Task, achieves the target accuracy with the least energy cost, providing an reduction of 29% ~ 53%.
0.2). Considering all the cases, FEDPROF
6) Differentiated participation with FEDPROF: Fig. reflects the preference of our selection strat- egy. In the S-Task we can observe that the clients with useless samples or noisy data get significantly less involved (<10 on average) in training. In the L-Task our algorithm also effectively limits (ba- sically excludes) the clients who owns the image data of poor quality (i.e., irrelevant severely blurred), whereas the clients with moderately noisy images are selected with reduced frequency as compared those with normal data. A potential issue of having the preference towards some of the devices is about faimess. Nonetheless, one can apply our algorithm together with an incentive mechanism (e.g.. Yu et al. (2000) address it.

---

Under review as a conference paper at ICLR 2022
Full participation, FedProt
Full participation, FedProl
pante partepaiion poton court
dients with melevant nony
cliems
SartSpation COUrE
with
diients
noley data
clients with normal data
Imolvvant
15
25
30
40
50
100
200
300
400
5.00
1002
700
8CD
900
1003
C
participation, data
participation.
partcpakin coint
dients with inelevent notey data
paricpation count
with inelevant
dients
clients with romea deta
noiey data
1
clentid
(a)
(b)
Figure 6: Total counts by client of participation (i.c., being selected) (a) the S-Task and (b) the
L-Task. For clarity, clients indexed according their local data quality.
6 CONCLUSION
Federated learning provides preercccppreeiiigggg approach to decentralized training but is vulnerable to the heterogeneity and uncertain quality of on-device data. In this paper, we use novel approach to address the issue without violating the data locality restriction. We first provide key insights for the distribution of data representations and then develop dynamic data representation profiling and matching scheme. Based on the scheme we propose a selective FL training algorithm FEDPROF that adaptively adjusts clients' participation chance based on their profile dissimilarity. We have conducted extensive experiments on public datasets under various environment settings. Evaluation results show that our algorithm significantly improves the efficiency of FL and reduces the time and energy costs for the global model to converge.
REFERENCES
Durmus Alp Emre Acar, Yuc Zhao, Ramon Matas, Matthew Mattina, Paul Whatmough, and Venkatesh Saligrama. Federated learning based on dynamic regularization. In International Con- ference on Learning Representations, 2020.
Dan Alistarh, Demjan Grubic, Jerry Li, Ryota Tomioka, and Milan Vojnovic. Qsgd: Communication-efficeett sgd via gradient quantization and encoding. Advances in Neural In- formation Processing Systems, 30:1 10:1709-1720, 2017.
Eugene Bagdasaryan, Andreas Veit, Yiqing Hua, Deborah Estrin, and Vitaly Shmatikov. How to backdoor federated learning. In International Conference on Artificial Intelligence and Statistics, pp. 2938-2948. PMLR, 2020.
Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation learning: A review and new perspectives. IEEE transactions on paltern analysis and machine intelligence, 3588):1798-1822 2013.
Arjun Nitin Bhagoji. Supriyo Chakraborty, Prateek Mittal, and Seraphin Calo. Analyzing federated learning through an adversarial lens. In International Conference on Machine Learning, pp. 634 643. PMLR, 2019.
Patrick Billingsley. Probability and measure. John Wiley Sons, 2008
Aaron Carroll, Gemmot Heiser, et al. An analysis of power consumption in a smartphone. In USENIX annual technical conference, volume 14, pp. 21-21. Boston, MA, 2010.
10

---

Under review as a conference paper at ICLR 2022
Zheng Chai, Hannan Fayyaz, Zeshan Fayyaz, Ali Anwar, Yi Zhou, Nathalie Baracaldo, Heiko Lud- wig. and Yue Cheng. Towards taming the resource and data heterogeneity in federated learning. In 2019 {USENIX) Conference Operational Machine Learning (OpML 19), pp. 19-21, 2019.
Zheng Chai, Ahsan Ali, Syed Zawad, Stacey Truex, Ali Anwar, Nathalie Baracaldo, Yi Zhou, Heiko Ludwig, Feng Yan, and Yue Cheng. Tifl: A tier-based federated learning system. In Proceedings of the 29th International Symposium on pp. 136, 2020.
Parallel and Distributed Computing,
Mingzhe Chen, H Vincent Poor, Walid Saad, and Shuguang Cui. Convergence time optimization for federated learning over wireless networks. IEEE Transactions on Wireless Communications,, 20 (4):2457-2471, 2020a.
Mingzhe Chen, Nir Shlezinger, H Vincent Poor, Yonina C Eldar, and Shuguang Cui. Communication-eficient federated learning. Proceedings of the National Academy of Sciences, 118(17), 2021.
Wenlin Chen, Samuel Horvath, and Peter Richtarik. Optimal client sampling for federated learning. arXiv preprint arXiv:2010.13725. 2020b.
Yae Jee Cho, Jianyu Wang, and Gauri Joshi. Client selection in federated learning: Convergence analysis and power-of-choice selection strategies. arXiv preprint arXiv:2010.01243, 2020.
Laizhong Cui, Xiaoxin Su, Yipeng Zhou, and Yi Pan. Slashing communication traffic in federated learning by transmitting clustered model updates. IEEE Journal on Selected Areas in Communi- cations, 2021.
Yin Cui, Menglin Jia, Tsung-Y. Yi Lin, Yang Song, and Serge Belongie. Class-balanced loss based effective number of samples. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 9268-9277, 2019.
Yuyang Deng, Mohammad Mahdi Kamani, and Mehrdad Mahdavi. Adaptive personalized federated learning. arXiv preprint 2rXhv:2003.13466 2020.
Alireza Fallah, Aryan Mokhtari, and Asuman Ozdaglar. Personalized federated learning with the- oretical guarantees: A model-agnostic meta-learning approach. Advances in Neural Information Processing Systems, 33:3557-3568, 2020.
Minghong Fang, Xiaoyu Cao, Jinyuan Jia, and Neil Gong. Local model poisoning attacks to byzantine-robust federated learning. In 29th (USENIX) Security Symposium (!USENIX) Se- curity 20), pp. 1605-1622, 2020.
AJ Feelders. Learning from biased data using mixture models. In KDD, pp. 102-107, 1996.
Siwei Feng and Han Yu. Multi multi-participant multi-clan vertical federated learning. arXiv preprint arXiv:2001. 11.22001.1115,, 2020.
Craig Gentry, Fully homomorphic encryption using ideal lattices. In Proceedings of the forty-first annual ACM symposium Theory of computing, pp. 169-178, 2009.
Jack Goetz, Kshitiz Malik, Due Bui, Seungwhan Moon, Honglei Liu, and Anuj Kumar. Active federated learning. arXiv preprint arXiv: /2641, 2019.
Andrew Hard, Kanishka Rao, Rajiv Mathews, Swaroop Ramaswamy, Françoise Beaufays, Sean Augenstein, Hubert Eichner, Chloé Kiddon, and Daniel Ramage. Federated learning for mobile 2018.
keyboard prediction. arXiv preprint arXiv:
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEFE conference on computer vision and pattern recognition, pp- 770-778, 2016.
Peter Kairouz, H Brendan McMahan, Brendan Avent, Aurélien Bellet, Mehdi Bennis, Arjun Nitin Bhagoji, Kallista Bonawitz, Zachary Charles, Graham Cormode, Rachel Cummings, et al. Ad- vances and open problems in federated learning. arXiv preprint 191iv: 2019.
11

---

Under review as a conference paper at ICLR 2022
Jakub Konečný, H Brendan McMahan, Felix X Yu, Peter Richtárik, Ananda Theertha Suresh, and Dave Bacon. Federated learning: Strategies for improving communication efficiency. arXiv preprint arXiv:1610.05492 2016.
Simon Kornblith, Mohammad Norouzi, Honglak Lee, and Geoffrey Hinton. Similarity of neural network representations revisited. In International Conference on Machine Learning, pp. 3519- 3529. PMLR, 2019.
Fan Lai, Xiangfeng Zhu, Harsha V Madhyastha, and Mosharaf Chowdhury. Oort: Efficient federated learning via guided participant selection. In 1 15th {USENIX) Symposium on Operating Systems Design and Implementation ( ({OSDI) 21). pp. 19-35, 2021.
Don Lemons. An Introduction Stochastic Processes in Physics. Johns Hopkins University Press, 2003.
David Leroy, Alice Coucke, Thibaut Lavril, Thibault Gisselbrecht, and Joseph Dureau. Federated learning for keyword spotting. In ICASSP 2019-2019 IEEE International Conference on Acous- tics, Speech and Signal Processing (ICASSP), pp. 6341-6345 IEEE, 2019.
Qinbin Li, Bingsheng He, and Dawn Song. Model-contrastive federated learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10713-10722, 2021.
Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, and Virginia Smith. Federated optimization in heterogeneous networks. In The 3rd MLSys Conference, 2020.
Xiang Li. Kaixuan Huang, Wenhao Yang, Shusen Wang, and Zhihua Zhang. On the convergence of
fedavg on non-iid data. In International Conference on Learning Representations, 2019.
Wei-Chao Lin, Chih-Fong Tsai, Ya-Han Hu, and Jing- Jhang. Clustering- undersampling in class-imbalanced data. Information Sciences, 409:17-26, 2017.
WANG Luping, WANG Wei, and LI Bo. Cmfl: Mitigating communication overhead for feder- ated learning. In 2019 IEEE 39th International Conference Distributed Computing Systems (ICDCS), pp. 954-964. IEEE, 2019.
Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. Communicaton--fficcentt learning of deep networks from decentralized data. In Artificial Intelli- gence and Statistics (AISTATS), pp. 1273-1282. PMLR, 2017.
Robert van der Meulen. What edge computing means for infrastructure and opera- wtt.:///wwmmmteeiiiiaattttt/aaaa 2018. Accessed: 2021 100--10.
tions leaders.
means-for-
Mehryar Mohri, Gary Sivek, and Ananda Theertha Suresh. Agnostic federated learning. In Interna- tional Conference on Machine Learning, pp. 4615 4625. PMLR, 2019.
Ari Morcos, Maithra Raghu, and Samy Bengio. Insights on representational similarity in neural networks with canonical correlation. In Proceedings of the 32nd International Conference Neural Information Processing Systems, pp. 5732-5741, 2018.
Solmaz Niknam, Harpreet S Dhillon, and Jeffrey H Reed. Federated learning for wireless commu- nications: Motivation, opportunitics, and challenges. IEEE Communications Magazine, 58(6): 46-51, 2020.
Takayuki Nishio and Ryo Yonetani. Client selection for federated learning with heterogeneous resources in mobile edge. In (CCC20/9200 IEEE International Conference on Communications (ICC), pp. 1-7. IEEE, 2019.
Stephen J Roberts and Will D Penny. Variational bayes for generalized autoregressive models. IEEE Transactions on Signal Processing. 50(9):2245-2257, 2002.
Dipankar Sarkar, Ankur Narang, and Sumit Rai. Fed-focal loss for imbalanced data classification in federated learning, arXiv preprint arXiv:2011.06288 2020.
12

---

Under review as a conference paper at ICLR 2022
Wenqi Shi, Sheng Zhou, and Zhisheng Niu. Device scheduling with fast convergence for wireless federated learning. In 2020-2022 IEEE International Conference on Communications (ICC), pp. 1-6. IEEE, 2020.
Jie Song, Tiantian Li, Zhi Wang, and Zhiliang Zhu. Study on energy- consupioonn regularities of cloud computing systems by a novel evaluation model. Compuling, 95(4):269-25) 2013.
Sebastian U Stich, Jean- Baptiste Cordonnier, and Martin Jaggi. Sparsified sgd with memory. Ad- vances in Neural Information Processing Systems, 31:4447-4458, 2018.
Alysa Ziying Tan, Han Yu, Lizhen Cui, and Qiang Yang. Towards personalized federated learning. arXiv preprint MXXiv:2003.007000 2021.
Vale Tolpegin, Stacey Truex, Mehmet Emre Gursoy, and Ling Liu. Data poisoning attacks against federated learning systems. In European Symposium on Research in Computer Security, pp. 480- 501. Springer, 2020.
Nguyen H Tran, Wei Bao, Albert Zomaya, Minh NH Nguyen, and Choong Seon Hong. Federated learning over wireless networks: Optimization model design and analysis. In IEEE INFOCOM 2019-IEER Conference on Computer Communications, pp. 1387-1395. IEEE, 2019.
Tiffany Tuor, Shiqiang Wang, Bong Jun Ko, Changchang Liu, and Kin Leung. Overcoming noisy and irrelevant data in federated learning. arXiv preprint arXiv:2001.08300 2020.
Hao Wang, Zakhary Kaplan, Di Niu, and Baochun Li. Optimizing federated learning non-iid data with reinforcement learning. In IEEE INFOCOM 2020-IEEE Conference on Computer Commu- nications, pp. 1698-1707. IEEE, 2020a.
Jianyu Wang, Qinghua Liu, Hao Liang, Gauri Joshi, and H Vincent Poor. Tackling the objective in- consistency problem in heterogeneous federated optimization. arXiv preprint arXiv:2007.07481, 2020b.
Shiqiang Wang, Tiffany Tuor, Theodoros Salonidis, Kin K Leung, Christian Makaya, Ting He, and Kevin Chan. Adaptive federated learning resource constrained edge computing systems. IEEE Journal on Selected Areas in Communications, 37(6): 205-1221, 2019.
Jiaxiang Wu, Weidong Huang, Junzhou Huang, and Tong Zhang. Error compensated quantized sgd and its applications to large-scale distributed optimization. In International Conference on Machine Learning, pp. 3325-5333 PMLR, 2018.
Wentai WU, Ligang He, Weiwei Lin, and Rui Mao. Accelerating federated learning over reliability- agnostic clients in mobile edge computing systems. IEEE Transactions Parallel and Dis- tributed Systems, 32(7):1539-1555 1551, 2021a.
Wentai Wu, Ligang He, Weiwei Lin, Rui Mao, Carsten Maple, and Stephen A Jarvis. Safa: semi-asynchronous protocol for fast federated learning with low overhead. IEEE Transactions on Computers, 70(5):655-66) 668, 2021b.
Haibo Yang, Minghong Fang, and Jia Liu. Achieving linear speedup with partial worker partici- pation in non-iid federated learning. In International Conference on Learning Representations, 2020.
Jaejun Yoo, Namhyuk Ahn, and Kyung-Ah Sohn. Rethinking data augmentation for image super- resolution: A comprehensive analysis and new strategy. In Proceedings of the IEEF/CVF Con- ference on Computer Vision and Pattern Recognition, pp. 3375-8384, 2020.
Da Yu, Huishuai Zhang, Wei Chen, Jian Yin, and Yan Liu. How does data augmentation affect privacy in machine learning? In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp. 177446-1753,, 2021.
Han Yu. Zelei Liu, Yang Liu, Tianjian Chen, Mingshu Cong. Xi Weng, Dusit Niyato, and Qiang Yang. A fairness-aw aware incentive scheme for federated learning. Proceedings of the AAAI/ACM Conference AI, Ethics, and Society, pp. 393-399, 2020.
13

---

Under review as a conference paper at ICLR 2022
Yuchen Zhang, John C Duchi, and Martin J Wainwright. Communication-efficieen algorithms for statistical optimization. The Journal of Machine Learning Research, (6():33311-366, 2013.
Shuxin Zheng, Qi Meng, Taifeng Wang, Wei Chen, Nenghai Yu, Zhi-Ming Ma, and Tie- Yan Liu. Asynchronous stochastic gradient descent with delay compensation. In International Conference on Machine Learning, pp. 4120-4129. PMLR, 2017.
14

---

Under review as a conference paper at ICLR 2022
A PROOF OF PROPOSITIONS
A.1 PROOF OF PROPOSITION 1
Without loss of generality, provide the proof of Proposition T for the pre-activation represen- lations from dense (fully-connected) layers and standard convolutional layers, respectively. The results can be easily extended other linear neural operators.
Dense layers
Proof. Let {newy, neug, ....,........................... neu, J denote dense layer (with q neurons) of any neural network model and H3 denote the pre-activation output of new in Q. We first provide the theoretical proof to support the observation that H3 tends to follow the Gaussian distribution.
Let = R" denote the input feature space (with v features) and assume the feature X, (which is a random variable) follows certain distribution (i of) (not necessarily Gaussian) with finite mean
[4, = F and variance of = F /4]. For each neuron neu;, let wg = [Wk,1 Wk.2 Wk.o)
denote the neuron's 's weight vector, by denote the bias, and Zy X denote the i-th weighted
input. Let H₁ denote the output of neug. During the forward propagation, we have:
Hs Xwy + bb
Xywkki + bk
Σ Zk, + bk-
(9)
Apparently Ze.; is random variable because Zx XyWk, (where the weights wk, are constants during forward pass), thus H₃ also a random variable according to Eq. (9).
In an ideal situation, the inputs variables X2.... X. distribution, in which case Proposition [ automatically holds due the property of multivari- ate normal distribution that every linear combination of the components of the random vector follows Gaussian distribution (Lemons] 2003). In other words, H3 + WK, by is normally distributed variable since WK. and by (k = 1.,,..., v) are constants in the forward propagation. A special case for this condition that X1, .5,,, Xx independent each other and X, follows Gaussian distribution N N(H, of) for -1,2.... In this case, by the definition of Zk,á, we have:
may follow multivariate Gaussian
(X1, X2,,X,,... xX,,..,,.))) X|Wk,1 + X2Wa,2
ZR Xewk,t N (Wk,,)4i, (w,
(10)
where Z₁, Ez,..., Z, are independent on each other. Combining Eqs. and 110). we have:
H NO
Wk,sh
bk Σ (w),
(11)
For more general cases where X1, X2,... X, are not necessarily normally distributed, we assume the weighted inputs Z of the dense layer satisfy the Lyapunov's condition (see definition 1 As a result, we have the following according to the Central Limit Theorem (CLT) (Billingseyy 2008 considering that X, follows ((44, of):
ZZ, We,sH.) 4 10,
(12)
Sk
where 8t =VEi (w ) and \00 1) denotes the standard normal distribution. Equivalently, for every neug we have:
Σ Zas 4 NO Wk,sk4i 22)
(13)
15

---

Under review as a conference paper at ICLR 2022
Combining Eqs.
( and 13
can derive that:
H
wkis/hs + bk, sk))
(14)
i=1
which means that Hk = 1, 2.2...., v) tend to follow Gaussian distribution and proves our Proposi- tion for fully-connected layers.
Convolutional layers
Proof. Standard convolution in CNNs is also linear transformation of the input feature space and its main difference from dense layers rests on the restricted size of receptive field. Without loss of generality, we analyze the representation (output) of single kernel. To facilitate our analysis for convolutional layers, let C' denote the number of input channels and K denote the kemel size. For ease of presentation, (k for channel index, i and j for indices on the same channel) of elements in the feature map (i.e., representations) to the input features. For the k-th kernel, let W₂ denote its weight tensor (with Wk.c being the weight matrix for channel c) and by its bias.
define receptive field mapping function i,, that maps the positions
Given the corresponding input patch Xe(k,s.j)> The element HaRs the representations from convolutional layer can be formulated as:
K HK.i. ΣΣ c11'=11==1
(Xe(ka.)) o Wk.)) 2. + bk.
(15)
where denotes Hadamard product. The three summations reduce the results of element-wise product between the input patch and the k-th kernel to the correspond representation element Ha.i in the feature map. For ease of presentation, here we use the notation 2(k)
e. to replace (Xe(ki)) Wk,e),.y and let ((Ma,) a². 3') be the distribution that (kk) "e,""," follows. Note that can be any distribution since we do not make any distributional assumption z(k) c
J"
With the notations, Eq. 15 be rewritten in similar form Eq. 9
C
K
K
HK..
(K) + bk-
(16)
e,3'
We use the condition that the random variables Z(0,) satisfy the Lyapunov's condition, i.c., there exists d such that
C
K
K
lim Cx
E
(k) "' - Me, (2t8] = 0,
(17)
246
=1=11/=
where
/ EC
EK
K Ei=1
K
Then according to the Lyapunov CLT, the following holds:
HKSs 4 N
Σ Pe,'' + bk.
Σ
Lang)):
(18)
c,5' (.E((kB()))))))
(.
which proves our Proposition for standard convolution layers.
A.2 PROOF OF PROPOSITION 2
Without loss of generality, prove Proposition for the fused representations from the LSTM layer and the residual block of ResNet models, respectively. The results can be easily extended to other non -onear neural operators.
LSTM
16

---

Under review as a conference paper at ICLR 2022
Proof. Long Short-Term Memory (LSTM) models popular for extracting useful representations from sequence data for tasks such as speech recognition and language modeling. Each LSTM layer contains multiple neural units. For the k-th unit, it takes input the current input feature vector X, = (X,,1, .x.,,...), hidden state vector Ht-1 and its cell state Ct-1kk- The outputs of the unit are its new hidden state R and cell state Ct,k- In this paper, we study the distribution of h, Multiple gates are adopted in an LSTM unit: by &t,k, ft,ks A and Ot we denote the input gate, forget gate, cell gate and output gate of the LSTM unit time step t. The update rules of these gates and the
cell state are:
f.A sigmoid (1))
b(a))),
ft, (ggmoid(((px((4)11111 + b(nk),
0t.k tanh(W (0)a ( 1, X + b())))
OL,k = sigmoid (o)) + b(o)k);
Ct S1, -t-1k + it,k 9t.kr
(19)
where the Wiy W Wi)h and W(o)s are the weight parameters and b(z)k b(j)k b()) and b(oj are the bias parameters for the gates.
The output of the LSTM unit he is calculated as the following:
h4 = OL. tanh(ct,x).
(20)
Using the final hidden states hT. (with T being the length of the sequence) as the elements of the layer-wise representation, we apply the following layer-wise fusion to further produce H over all the hy.k in a single LSTM layer:
H
hr,k:
(21)
where d is the dimension of the LSTM layer. Again, by C(us, of) we denote the distribution of f (where the notation T dropped here since is typically a fixed parameter). With (hr,, 1, 1,2, ..2,...,u d} satisfying the Lyapunov's condition and by Central Limit Theorem H,, tends to follow the Gaussian distribution:
4,
H
pkk
(22)
which proves the Proposition for waser-wise fused representations from LSTM.
Residual blocks
Proof. Residual blocks are the basic units in the Residual neural network (ResNet) architecture (He et al. 2016). A typical residual block contains two convolutional layers with batch normalization (BN) and uses the ReLU activation function. The input of the whole block is added to the output of the second convolution (after BN) through skip connection before the final activation. Since the convolution operators are the same as we formulate the second part of Section [. here we use the notation (XX denote the sequential operations of convolution on X followed by BN, i... (X) & (N(CCnn(X))) Again, we reuse the receptive field mapping @k, as defined in Section A. to position the inputs of the residual block corresponding to the element Zk.i, in the output representation of the whole residual block.
Let X denote the input of the residual block and Zk,i.j denote an element in the output tensor of the whole residual block. Then we have:
ZE.i.z (Xuis + BN (N ((NCm(((((((((mmm(XXxx))))]]]
=f(Xnss + U (F(((XX())))))
(23)
where f is the activation function (ReLU).
17

---

Under review as a conference paper at ICLR 2022
We perform channel-wist fusion the representation from the residual block to produce H3 for the k-tth channel:
di dw
H Zk.i.j:
(24)
where dæ and dw are the dimensions of the feature map and l is the channel index.
Let ((Ha,,g R denote the distribution that Zu.i.j follows. Then we apply the Lyapunov's con- dition to the representation elements layer-wise, i.e..
dæ dw
lim dw xdR
E. [|
[2t = 0,
(25)
5k
1=1 y=1
V
where 8/
With the above condition satisfied, by CLT Hş (the fused representation on channel k) tends to follow the Gaussian distribution:
dg dw
dy dw Pkii
H₂
(26)
;=
which proves the Proposition for channel-wise fused representations from any residual block.
B PSEUDO-CODE OF FEDPROF (DETAILED VERSION)
See Algorithm 2 for a complete version of the proposed training algorithm.
C CONVERGENCE ANALYSIS
In this section we provide the proof of the proposed Theorem The analysis is mainly based on the results provided by Li et al. (2019). We first introduce several notations to facilitate the analysis.
C.I NOTATIONS
Let U (UU) = N) denote the full set of clients and S(t) ((S(t) = K) denote the set clients selected for participating. By wx (t) we denote the local model client i: at time step We define an auxiliary sequence (f) for each client to represent the immediate local model after local SGD update. Note that (t) is updated from (t 1) with learning rate 1
un (1) = wn (1 1) 7/- F. (wr (t - (r.e-)) where V (w (t. from DR with regard wie( 1).
(27)
E1, the stochastic gradient computed over a batch of data Ex,e-1 drawn
We also define two virtual sequences (tt Em pk and ((t = Aggregate (vy (t)) for every time step (Note that the actual global model w()) is only updated the aggregation steps TA = { 2,, 3E,.. .)). Given an aggregation interval E' > 1, we provide the analysis for the partial aggregation rule that yields (t) as:
((t)
Σ vx(t). KES(1)
(28)
where S(t) ((((t) K) is the selected set of clients for the round [+] that contains step f. At the aggregation steps TA, wi()) is equal to @tt), i.e., w(t) (t) if TA.
To facilitate the analysis, we assume each client always performs model update (and synchroniza- tion) produce (t) and E(t) (but obviously it does not affect the resulting w and for k & S(t)).
wk (V)((), if $ TA (()),, ift TA
(29)
18

---

Under review as a conference paper at ICLR 2022
Algorithm 2: the FEDPROF protocol (detailed version)
Input maximum number of rounds T maax, local iterations per round E, client set U, client fraction C, validation set D);
Output: the global model w
// Server process: running on the server Initialize global model to using a seed 0 // version of the latest global model Broadcast the seed to all clients for identical model initialization
Collect initial profiles (RPx keE from all the clients
2kk <- O, vk E U
Generate initial baseline profile RP*(0) on D
K + -11
for round + max do
Calculate div (U)). R.P)()))PPP for each client As Update client scores {eu and compute Σ Av Au
10
S & Choose K clients by probability distribution (x KEU
11
Distribute w to the clients S for client 4: in S in parallel do
12
UK <- U. vk E S
13
(.P((u)) update w, Wk f- localTraining(k, E)
14
end
Collect local profiles from the clients in S Update w via model aggregation
16
17
u+ T
18
Evaluate w and generate RP* (v)
end
19
return
Client process: running on client A: UpdateProfile(k; w, u):
29
Generate RP; on Dr with the global w received
21
Label profile RP2 with version number v Return RPR
return
Jocallraining(k; W. E): Wik f
23
for step e E
24
Update wA using gradient-based method
end
25 Return Wk
return
For ease of presentation, we also define two virtual gradient sequences: g(() = N A- Pk V (www(()) and g(t) EN Ph (wx, Ex,). Thus we have E[g(t)] g(t) and
(t - 1) Tht 1g -1).
C.2 ASSUMPTIONS
We formally make four assumptions support our analysis of convergence. Assumptions and 2 are standard in the literature (Li et al. 2019: Wang et al. 2019 Cho et al. 2020) defining the convexity and smoothness properties of the objective functions. Assumptions and bound the
variance of the local stochastic gradients and their squared norms in expectation, respectively. These two assumptions are also made by Li al. (2019).
Assumption 1. Fi, F2,..., FN are L-smooth, i.e., for any k U. and yr Fely) Fx(x) + +(v
2) Fx(x)) lly x =1
19

---

Under review as a conference paper at ICLR 2022
It is obvious that the global objective F also L-smooth linear combination of F1, F2..... FN with Pl, P2..... PN being the weights.
Assumption 2. F2. FN p-strongly convex, i.e., for all & e U and any y Fo(y) 2 Fx(x)) + (y x) (F(()) + lll x1B
Assumption 3. The variance of local stochastic gradients each device is bounded: For all ls E U, E| (t),
Fx (w)((t))) =
Assumption 4. The squared norm of local stochastic gradients on each device is bounded: For all
k: E U, (V) v G
C.3 KEY LEMMAS
To facilitate the proof of our main theorem, we first present several key lemmas.
Lemma 1 (Result of one SGD step). Under Assumptions / and 2 and with me < 41. for any t it holds true that
1
El (1+ (111 w0 -n Elgg Ds 2 +
Pa pullwa (1) (())
(30)
where r = P F- Σ P
Lemma (Gradient variance bound). Under Assumption
one can derive that
E Egg
Dell
P.
(31)
R=1
Lemma 3 (Bounded divergence of ws (t)) Assume Assumption 4 holds and a con-increasing step size Th s.t. TJ. < 2 for all t 1, 1, 2....... it follows that
pu ww - w())p <
(32)
Lemmas and hold for both full and partial participation and are independent of the client selection strategy. We refer the readers Li al. (2019) for their proofs and focus our analysis on opportunistic selection.
The next two lemmas give important properties of the aggregated model w as a result of partial participation and non-uniform client selection/sampling
Lemma 4 (Unbiased aggregation). For any aggregation step E TA and with Qs Ph in the selection of S(t), it follows that
Esuy(w(()) = E(t).
(33)
Proof. First, present key observation given by ei al. 20199 as an important trick handle the randomness caused by client selection with probability distribution {9x I. expectation over S(t), it follows that
By taking the
N
Essoo Σ X KES K X Σ
(34)
kES(1)
k=1
20

---

Under review as a conference paper at ICLR 2022
Let 9: Pk. take the expectation of (t) over S(t) and notice that B(t) ExEU Pb)l)(t):
Es(e) [wo((] Es(e)
Σ vx((]]
K
kES(1)
Escel
[vx(1]
K
kES())
1
KE [txc)]
K
E queu (t) KEU vtt).
Lemma (Bounded variance of i(t)). For any aggregation step E TA and with non-increasing step size T7 s.t. T7: I 2-1+ follows that
Es(nllu(t) - b(t)ll2 K 1i11EE...
(35)
Proof. First, one can prove that (1) an unbiased estimate of ö(t) for any k:
(t)] = Σ
(t)x(t) = E(t).
(36)
KEC
Then by the aggregation rule w(t) = 1 EkEs(O) (t), we have:
K
1
Es()) (t) o(t)]1²
K2 ES(()) (tt(t) Ko(()
K
Es ||
Σ vx(1)) Σ 5()))
K2
kES(t)
k=1
Es(a) ||
Σ (ux(t) o(t)) 1|?
K2
kES(t)
Es(t))
Σ
|| (t) 0())
K
kES(1)
+ Es(e)
(o:(t)
(37)
(.JES()).
where the second term on the RHS of 37 equals zero because (ue(t))keu are independent and
unbiased (see Eq. 36 Further, by noticing & E E TA (because t E TA) which implies that
wk (t- E) w( E) since the last communication, we have:
Es(o (t) 0(t]l2
K2 Es(t)
[lxx(t) b(t)l12
kES())
K2 KE (((())) 8(1)11
F (vx(() wil F?)) (T(1) w(t F2)
K
S K
Escollva (1) - w(1 - E)
(38)
21

---

Under review as a conference paper at ICLR 2022
where the last inequality results from (lex(t) (t E)] -(t)- E) and that -
E||X|| 112. Further, we have:
Escoll((t u(t)ll = Es(c)llu(()) :i E)1
K
(xEs() (1x - (t E)12
K
q Es(e)
||
Σ ( R
(u)))
(39)
K
Let fm = arg max; Vi (wx El. It-E,t- 1]. By using the Cauchy- Schwarz incquality, Assumption 4 and choosing non-inereasing Th s.t. tp. E-1+E-1, have:
Z1 Escol
Σ 7i (()
Σ
Σ
Estyy (n FF (1,) V.
i E jj=t=E E
Es()) (1. (w)
Σ
- E
Σ
Σ
Mh Esco FF (wx 1l?
< E²
(40)
Plug Z1 back into 39 and notice that E qhe we have:
1 N
[ste)ll((t)
a(t)ll2
yh E E²
4
Mt E2
K
C.4 PROOF OF THEOREM 1
Proof. Taking expectation of ((t(t) w. w1, we have:
Et] - w0112 = F(s()) (1) - F(() + a(t) w/ 11
(t) - 22 El(((t) w' 112 + ((w(t) - B(t), o(t)
w²)
(41)
Ag
Aa
where Ag vanishes because (() is an unbiased estimate of E(t) by first taking expectation over S(t)
(Lemma 4
22

---

Under review as a conference paper at ICLR 2022
To bound Ag for t TA. apply Lemma
A2 Et) w' 112 Σ (1 7t-1/1 wit - 1) w= n ge-
11.
6L, r + E
Ps aelluss 1) wi(t 1)11?
(42)
B2
Then we use Lemmas 2 and 3 to bound B₁ and B, respectively, which yields:
A2 = El((() w= 1| s (1 11-1 (t 1) w w011 + vi B
(43)
where B Σ 2 779 + -
To bound A1, one can first take expectation over S(t) and apply Lemma where the upper bound actually eliminates both sources of randomness. Thus, it follows that
4 A E() =(0)
12
-
(44)
K
Let C = E² and plug A₁ and A2 back into 41
E (l) - w: 1|? 2 (1 - 111 - 1) w" =12 2 1 C)..
(45)
Equivalently, let A, Bl( w" 11?, then have the following recurrence relation for any
t t 1:
A, S (1 - 1/- 1 A. + 1 + C).
(46)
(B+C) (7 1)41] } using an 3p-1
Next we prove by induction that A. S
V where v max
aggregation interval E 2 1 and a diminishing step size 7) :+y for some B > P that 7/1 S m : z and 1). S 24t+E.
1 and > 0 such
First, for f = the conclusion holds that A S 74I given the conditions. Then by assuming it holds for some f, one can derive from (46) that
v
A++1 v (1 T111) A, + (B +C)
$ (1 3p
+ ( (tt)
2
(B CC)
7t
1+7-1 +
[2 + +C)
B1 - (t + 7)2
(++
(t+
20
++4-1
(++
1+7-1 1/
(++))=-1
(47)
++
which proves the conclusion A; < # for any 2 1.
Then by the smoothness of the objective function F, follows that
B[F(w(())
- F
< El w()) - w1
2
T L
A <
(48)
2
2 27 +t
23

---

Under review as a conference paper at ICLR 2022
Specifically, by choosing 2 (i.e., 1it
- w ) max + E. 1, we have
P
p (B)+ +(x 1)
max
(B CC)
Bp +
((B+C)
+ (
(49)
1²2
By definition, we have w(t) = (t) at the aggregation steps.
Therefore, for & E Ta:
E[F(((t))] F - 2
L
(2(B +CB+C)
77+1 41)
+
( + 122
2
D PROFILE DISSIMILARITY UNDER HOMOMORPHIC ENCRYPTION
The proposed representation profiling scheme encodes the representations data into a list of dis- tribution parameters, namely RP(w, D) {14, of)li 1.2,.... 2} where q is the length of the profile. Theoretically, the information leakage (in terms of the data in D) by exposing RP(w, D) very limited and it is basically impossible reconstruct the samples in D given RP(w: w. D)- Nonetheless, Homomorphic Encryption (HE) can be applied to the profiles (both locally and on the server) as guarantee zero knowledge disclosure while still allowing profile matching under the encryption. In the following we give details on how to encrypt a representation profile and compute profile dissimilarity under Homomorphie Encryption (HE).
To calculate (3 and (4) under encryption, client needs to encrypt (denoted as [[-I]) every single μ4 and of in profile RP(w,, (ww,, D) locally before upload whereas the server does the same for its RP* D*). Therefore, according to Eq. (4) we have:
[KL(A])) [A/5)) = log log[l(a;)311 - logil(o (2) 11:11
((((,;+)))] + ((Im{*)) - [[x]])
(50)
2[(
where the first two terms on the right-hand side require logarithm operation on the ciphertext. How- ever, this may not be very practical because most HE schemes are designed for basic arithmetic operations on the ciphertext. Thus we also consider the situation where HE scheme at hand only provides additive and multiplicative homomorphisms (Gentry 2009). In this case, avoid the log- arithm operation, the client needs to keep every of in RP""w, D:) as plaintext and only encrypts Ns. likewise for the server. As result, the KL divergence can be computed under encryption as:
(k) 1 2
[| log( of )a (k)
2
2 2 (l1 (Tx11))
(51)
where the first term on the right-hand side is encrypted after calculation with plaintext values (a and (a*)2 whereas the second term requires multiple operations on the ciphertext values [[x{] and [[
Now, in either case, we can compute profile dissimilarity under encryption by summing up all the KL divergence values ciphertext:
g
[ddo RP")]]
(52)
24

---

Under review as a conference paper at ICLR 2022
E DETAILS OF EXPERIMENTAL SETUP
E.I ENVIRONMENT SETUP
Table 4: Experimental setup.
Setting
Symbol
S-Task
L-Task
Model
W
FFN
CNN
Dataset
D
GasTurbine
EMNIST digits
Total data size
DI
36.7k
280k
Validation (benchmark) set size
D
11.0k
40k
Client population
N
50
1000
Data distribution
(514, 154")
non-IID, dominant<:60%
Noise applied
fake, gaussian noise (00.5 0.1²)
fake, blur, s&p
Client specification (GHz)
8R
V(1.0,0.13)
Comm. bandwidth (MHz)
bwr
(.0.5.0.13)
(11.0,0.13)
Signal-noise ratio
SNR
lc2
le2
Bits per sample
BPS
11*8*4
28*28*1*8
Cycles per bit
CPB
300
400
local epochs
M
2
Loss function
L
MSE Loss
NLL Loss
Learning rate learning rate decay
"
le-2
le-2
0.99
0.99
Detailed experimental settings are listed in Table 4 In the S-Task, the total population is 50 and the data collected by a proportion of the sensors (i.c., end devices of this task) are of low-quality: 10% of the sensors have no valid data and 40% of them produce noisy data. In the L-Task, we up relatively large population (1000 end devices) and spread the data (from EMNIST digits) across the devices with strong class imbalance roughly 60% of the samples on each device fall into the same class. Besides, many local datasets are of low-quality: the images on 15% of the clients are irrelevant (valueless for the training this task), 20% are (Gaussian) blurred, and 25% are affected by the salt-and-pepper -pppper noise (random black and white dots the image, density=0.3). For the S- Task, the maximum number of rounds mma is set to 100 for both aggregation modes, whilst it is set to 300 and 50 for the full aggregation and partial aggregation, respectively, for the L-Task. The preference factor a for our protocol set to 10. Considering the population of the clients, the setting of the selection fraction C' is based the scale of the training participants suggested by Kairouz et al. (2019).
To simulate realistic FL system that consists of disparate end devices, the clients are heterogeneous in terms of both performance and communication bandwidth (see Table 4). A validation set is kept by the server (as the benchmark data) and used for model evaluation.
E.2 DETAILS OF COST FORMULATION
In each FL round, the server selects a fraction (i.e., C) of clients, distributes the global model to these clients and waits for them finish the local training and upload the models. Given selected set of clients S, the time cost and energy cost of a communication round can be formulated as:
T Tround max (T (Tcomm + Tetrain T T.PP KES
(53)
Eg F.comm Eraiin
F
(54)
where Teomm and Tiroin are the communication time and local training time, respectively. The device-side energy consumption E, mainly comes from model transmission (through wireless chan- nels) and local processing (training), corresponding to Ecoum and Etrain respectively. TRRP and E estimate the time and energy costs for generating and uploading local profiles and only apply to FEDPROF.
25

---

Under review as a conference paper at ICLR 2022
Eq. 133 formulates the length of one communication round of FL, where Te can be modeled by Eq. 55 according to Tran et al. (209). where bw' is the downlink bandwidth device & (in MHz); SNR the Signal-to-Noiee Ratio of the communication channel, which is set to be constant as in general the end devices are coordinated by the hase stations for balanced SNR with the fairness- based policies; msize is the size of the model; the model upload time is twice much that for model download since the uplink bandwidth is set as 50% of the downlink bandwidth.
Tkmm T k
Tpppload -.dounlond
2 Idlonntoad + T Troowloood
msize
3 x bock log(1 SNR)
(55)
Ttrai in Eq. 53 can be modeled by Eq. (56). where sæ is the device performance (in GHz) and the numerator computes the total number of processor cycles required for processing M epochs of local training on Dę-
M M.
M. BPS CPB
T
Terain
(56)
TRP consists of two parts: T& Re for local model evaluation (to generate the profiles of D₂) and TuRPup for uploading the profile. TRP can be modeled
T T
RPgen
RPup T
1
RPsize
Tprrain
(57)
M
2 bwy
log(1 + SNR)
where TE TrRP estimated as the time cost of one epoch of local training: T RPup is computed in similar way to the calculation of Tromm in Eq. (55) (where the uplink bandwidth is set as one half of the total bur)); RPsize the size of a profile, which is equal to 4 q q (four bytes for each floating point number) according to our definition of profile in 2
Using Eq. (44p we model the energy cost of each end device by mainly considering the energy consumption of the transmitters for communication (Eq. (88) and on-device computation for local training (Eq. (59). For FEDPROF, there is an extra energy cost for generating and uploading profiles (Eq. 60).
Ex Ecoom
Ptrans
Troom
(58)
E ptrain
Prs
Terrin
(59)
RPP Perans
Th RPup + Pyst T RPgen
(60)
where Pys is simplified computation power consumption model (Song et al. 2013 and Pr is the
power of baseline processor. Ptrans is the transmitter's 's power. We set P Ptrans and P to 0.5 and
0.7 Watts respectively based on the benchmarking data provided by Carroll et al. 2010
26


Reviews:
[{"review_id": "ZR3SkH29hpi", "reviewer": "Reviewer_pcRb", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strenth:\n\nI personally like the style of this work, which attempts to formulate and present results on FL in a more theoretical way.\n\nWeakness:\n\nOverall the presentation on theoretical results can be improved, and the connection of some statements to the main results is not clear.\n\n1. For example in Proposition 1 and 2, all non-linear components in the neural network seem to be bypassed and the statements are simply equivalent ways of stating CLT. If it is the case, this needs to be clearly mentioned before those propositions. Besides, it is also not clear to me that how this over-simplified model is related to the rest of the works.\n\n2. The main Theorem needs to be better stated and explained. For example, the most relevant question to the readers is how the construction of profiles affects the performance of the algorithm.   \n\n3. To better understand the main Theorem, it would help if the stated learning bound can be compared to some benchmarks, e.g., the performances of using fixed \\lambda.\n\n4. Minor: the last equations in all Proposition proofs are stated in terms of convergence in distribution, which may not hold.\n\n ", "summary_of_the_paper": "This paper proposes a federated learning algorithm called FedProf, by using a training rule that updates the model based on divergences between data representation. ", "summary_of_the_review": "Overall, the results in this paper can be better presented, and the connections between certain statements are not clear. Therefore, I recommend a rejection.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "JYYGvt_wyWw", "reviewer": "Reviewer_5Q6Y", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths: The proposed method is intuitive and simple to implement. It also comes with theoretical guarantees (though the assumptions are highly idealized). \n\nWeaknesses and some questions:\n- If I understand correctly (pls correct me if I'm wrong), Proposition 1 and 2 are consequences of Lyapunov CLT, thus one needs each coordinate of the representation vector to be independent. It is not clear to me if the independence assumption would hold in practice.\n- Following my previous point, I am curious about what will happen if we replace the product Gaussian (Eq. 2) by a Gaussian with some non-identity covariance matrix.\n- Is the choice of KL divergence (Eq. 4) essential for the performance? What will happen if we choose other metrics?\n- How sensitive is the performance with respect to the choice of $\\alpha$?", "summary_of_the_paper": "This paper proposes FedProf, a new client sampling scheme that speeds up the convergence of FedAvg type algorithms. The author proves convergence of FedProf under a set of simplifying conditions and they demonstrate the utility of FedProf via empirical studies.", "summary_of_the_review": "I appreciate the simplicity of the idea and the performance boost it brings about. From what I understand, it can be applied to any FL algorithms that involve client sampling. Therefore, I think this paper deserves 6: marginally above the acceptance threshold.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "A1Wqqyvqn8u", "reviewer": "Reviewer_1UFK", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper is well-written and easy to understand. The motivation is clear and intuitive. The proposed method is technically sound and the results are strong.\n\nI have a few comments, particularly for the evaluation part.\n\n\n- More Datasets: it seems to work well on the reported datasets. I wonder if the tendency is still consistent with more benchmark datasets. I would like to suggest common datasets such as CIFAR-10 and CIFAR-100. I also would like to see the results when the number of classes is increased.\n\n- Standard Deviation: Can you provide statistical information in Table 2 and 3? I would like to see the variance of the proposed model. Few performances seem marginal compared to the base models. \n\n\n- The Current Baseline Models: I found that the baseline models used in the paper are slightly outdated. I suggest [1], the state-of-the-art method tackling similar scenarios, via performing model-level contrastive learning.\n\n\n- Overview: Please consider adding the concept illustration for FedProf, which can be helpful for common readers to capture high-level ideas. \n\n[1] Model-Contrastive Federated Learning, CVPR'21\n\n\n\n", "summary_of_the_paper": "The authors tackle the federated learning scenarios where local data is biased, noisy, or even irreverent, which could significantly degenerate the global performance, and the authors propose FedProf, which utilizes data representation profiling and matching to mitigate the impact of low-quality clients during training.\n", "summary_of_the_review": "Overall, I enjoyed reading the paper, and I'll raise my score if my concerns are properly addressed.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "8zxBazBhBpc", "reviewer": "Reviewer_kUXa", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The key idea of the paper is to select “high-quality” data under the hypothesis noisy and low-quality data has different distributions of representations. This hypothesis needs further justification. First, FL is known for dealing with non-iid heterogeneous datasets. Consider a scenario where clients have high-quality but heterogeneous datasets. How general does this hypothesis hold? In some sense, the idea here is opposite to another idea that has been used for client selection where clients with high loss are more likely to be selected. \n\nAlso, I wonder if the algorithm could get stuck in a local suboptimal condition. Specifically, let’s say there are two clusters of clients with different focuses. By chance, initial training selected users from one cluster, then the global model will further select users from the same cluster. Could this happen?\n\nThe current evaluation is limited to two simple and homogeneous tasks with added noise, which provides an ideal situation where the proposed algorithm would shine. While these evaluations illustrate the effectiveness of the proposed idea under ideal conditions, more thorough evaluations under general heterogeneous conditions are needed to better understand the pros and cons of the proposed algorithm. \n\nAlso, most of the comparison algorithms were proposed in 2019 except one that focuses on communications. Given the fast development of FL, I wonder if there are more recent algorithms that should be compared with. \n\nThe authors made an observation that representation layers follow a Gaussian distribution. I understand that this makes it more efficient to calculate distributional difference. However, in principle, this can be applied to cases where Gaussian distribution does not hold, right? Can you elaborate more on this, both in terms of using Gaussian distribution to approximate a non-Gaussian one and using the true non-Gaussian distribution?  \n\nIn the proofs of Prop. 1 and Prop. 2, central limit theorem is applied and thus requires the Lyapunov’s condition (Def. 1). Could you comment on how restrictive this condition is? For example, in your experiments, does this assumption hold?\n\nOne clarification question, in the experiments, how did you select the representation layer? \n\n\n", "summary_of_the_paper": "In this paper, the authors propose a user selection algorithm for federated learning (FL). The key motivation is to select high quality clients for update and thus to reduce the impact of low quality data on FL training. A hidden hypothesis is that high quality data has similar representations while noisy and low-quality data has different distributions of representations. Based on this hypothesis, the key idea is to select users based on their representation layer distribution difference from the global model, the higher the difference, the lower the chance of the client being selected.  Furthermore, the authors observed and proved that representation layers follow a Gaussian distribution, which makes the representation difference learning more efficient. The authors evaluated the proposed algorithm in comparison to existing algorithms using a small-scale sensor dataset and a large-scale EMNIST dataset. The proposed algorithm performs well in the evaluation.", "summary_of_the_review": "In summary, the authors propose to more favorably select FL clients whose representation distribution is more similar to that of the global model. While the current evaluation results are encouraging, data heterogeneity in FL and model robustness need to be carefully considered. \n\n\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a federated learning method called FedProf that adaptively selects different subsets of the clients' data in training the global model. There were several concerns brought up in the reviews and discussion. The multivariate Gaussian (with identity covariance) assumption on the neural network representation is limited. The paper also claimed to provide privacy preservation, but there is no formal statement of the actual privacy guarantees. (The fact that it's running federated learning does not guarantee privacy protection.) The presentation could use improvement. The reviewers had issues trying to understand the main theorem. Overall, there is not sufficient support for acceptance.
Decision: Reject