FedProf: Selective Federated Learning with Representation Profiling

1 INTRODUCTION . With the advances in Artificial Intelligence ( AI ) , we are seeing a rapid growth in the number of AI-driven applications as well as the volume of data required to train them . However , a large proportion of data used for machine learning are often generated outside the data centers by distributed resources such as mobile phones and IoT ( Internet of Things ) devices . It is predicted that the data generated by IoT devices will account for 75 % of the total in 2025 ( Meulen , 2018 ) . Under this circumstance , it will be very costly to gather all the data for centralized training . More importantly , moving the data out of their local devices ( e.g. , mobile phones ) is now restricted by law in many countries , such as the General Data Protection Regulation ( GDPR ) 1 enforced in EU . We face three main difficulties to learn from decentralized data : i ) massive scale of end devices ; ii ) limited communication bandwidth at the network edge ; and iii ) uncertain data distribution and data quality . As an promising solution , Federated Learning ( FL ) ( McMahan et al. , 2017 ) is a framework for efficient distributed machine learning with privacy protection ( i.e. , no data exchange ) . A typical process of FL is organized in rounds where the devices ( clients ) download the global model from the server , perform local training on their data and then upload their updated local models to the server for aggregation . Compared to traditional distributed learning methods , FL is naturally more communication-efficient at scale ( Konečnỳ et al. , 2016 ; Wang et al. , 2019 ) . Nonetheless , several issues stand out . 1.1 MOTIVATION . 1 ) FL is susceptible to biased and low-quality local data . Only a fraction of clients are selected for a round of FL ( involving too many clients leads to diminishing gains ( Li et al. , 2019 ) ) . The standard FL algorithm ( McMahan et al. , 2017 ) selects clients randomly , which implies that every client ( and its local data ) is considered equally important . This makes the training process susceptible to local data with strong heterogeneity and of low quality ( e.g. , user-generated texts ( Hard et al. , 1https : //gdpr.eu/what-is-gdpr/ 2018 ) and noisy photos ) . In some scenarios , local data may contain irrelevant or even adversarial samples ( Bhagoji et al. , 2019 ; Bagdasaryan et al. , 2020 ) from malicious clients ( Fang et al. , 2020 ; Bagdasaryan et al. , 2020 ; Tolpegin et al. , 2020 ) . Traditional solutions such as data augmentation ( Yoo et al. , 2020 ) and re-sampling ( Lin et al. , 2017 ) prove useful for centralised training but applying them to local datasets may introduce extra noise ( Cui et al. , 2019 ) and increase the risk of information leakage ( Yu et al. , 2021 ) . Another naive solution is to directly exclude those low-value clients with low-quality data , which , however , is often impractical because i ) the quality of the data depends on the learning task and is difficult to gauge ; ii ) some noisy or biased data could be useful to the training at early stages ( Feelders , 1996 ) ; and iii ) sometimes low-quality data are very common across the clients . In Fig . 1 we demonstrate the impact of involving ” low-value ” clients by running FL over 100 clients to learn a CNN model on MNIST using the standard FEDAVG algorithm . From the traces we can see that training over clients with problematic or strongly biased data can compromise the efficiency and efficacy of FL , resulting in an inferior global model that takes more rounds to converge . 2 ) Learned representations can reflect data distribution and quality . Representation learning is vital to the performance of deep models because learned representations can capture the intrinsic structure of data and provide useful information for the downstream machine learning tasks ( Bengio et al. , 2013 ) . In ML research , The value of representations lies in the fact that they characterize the domain and learning task and provide task-specific knowledge ( Morcos et al. , 2018 ; Kornblith et al. , 2019 ) . In the context of FL , the similarity of representations are used for refining the model update rules ( Li et al. , 2021 ; Feng & Yu , 2020 ) , but the distributional difference between representations of heterogeneous data is not yet explored . Our study is also motivated by a key observation that representations from neural networks tend to have Gaussian patterns . As a demonstration we trained two different models ( LeNet-5 and ResNet18 ) on two different datasets ( MNIST and CIFAR-100 ) separately . Fig . 2a shows the neuron-wise distribution of representations extracted from the first dense layer ( FC-1 ) of LeNet-5 . Fig . 2b shows the distribution of fused representations ( in a channel-wise manner ) extracted from a plain convolution layer and a residual block of ResNet-18 . These observations motivate us to study the distributional property of data representations and use it as a means to differentiate clients ’ value . 1.2 CONTRIBUTIONS . Our contributions are summarized as the following : • We first provide theoretical proof for the observation that data representations from neural networks tend to follow Gaussian distribution , based on which we propose a representation profiling and matching scheme for fast , low-cost comparison between different representation profiles . • We present a novel FL algorithm FEDPROF that adaptively adjusts clients ’ participation probability based on representation profile dissimilarity . • Results of extensive experiments show that FEDPROF reduces the number of communication rounds by up to 77 % , shortens the overall training time ( up to 4.5x speedup ) while increasing the accuracy of the global model by up to 2.5 % . 2 RELATED WORK . Different from traditional distributed training methods ( e.g. , Alistarh et al . ( 2017 ) ; Wu et al . ( 2018 ) ; Zheng et al . ( 2017 ) ) , Federated Learning assumes strict constraints of data locality and limited communication capacity ( Konečnỳ et al. , 2016 ) . Much effort has been made in optimizing FL and covers a variety of perspectives including communication ( Konečnỳ et al. , 2016 ; Niknam et al. , 2020 ; Cui et al. , 2021 ) , update rules ( Li et al. , 2020 ; Wu et al. , 2021a ; Leroy et al. , 2019 ; Luping et al. , 2019 ) , flexible aggregation ( Wang et al. , 2019 ; Wu et al. , 2021b ) and personalization ( Fallah et al. , 2020 ; Tan et al. , 2021 ; Deng et al. , 2020 ) . The control of device participation is imperative in cross-device FL scenarios ( Kairouz et al. , 2019 ; Yang et al. , 2020 ) where the quality of local data is uncontrollable and the clients show varied value for the training task ( Tuor et al. , 2020 ) . To this end , the selection of clients is pivotal to the convergence of FL over heterogeneous data and devices ( Nishio & Yonetani , 2019 ; Wang et al. , 2020b ; Chai et al. , 2019 ; Acar et al. , 2020 ) . Non-uniform client selection is widely adopted in existing studies ( Li et al. , 2020 ; Goetz et al. , 2019 ; Cho et al. , 2020 ; Li et al. , 2019 ; Chen et al. , 2020b ; Wang et al. , 2020a ) and has been theoretically proven with convergence guarantees ( Chen et al. , 2020b ; Li et al. , 2019 ) . Many approaches sample clients based on their performance ( Nishio & Yonetani , 2019 ; Chai et al. , 2020 ) or aim to jointly optimize the model accuracy and training time ( Shi et al. , 2020 ; Chen et al. , 2020a ; 2021 ) . A popular strategy is to use loss as the information to guide client selection ( Goetz et al. , 2019 ; Lai et al. , 2021 ; Sarkar et al. , 2020 ) . For example , AFL ( Goetz et al. , 2019 ) prioritizes the clients with high loss feedback on local data , but it is potentially susceptible to noisy and unexpected data that yield illusive loss values . Data representations are useful in the context of FL for information exchange ( Feng & Yu , 2020 ) or objective adaptation . For example , Li et al . ( 2021 ) introduces representation similarities into local objectives . This contrastive learning approach guides local training to avoid model divergence . Nonetheless , the study on the distribution of data representations is still lacking whilst its connection to clients ’ training value is hardly explored either . 3 DATA REPRESENTATION PROFILING AND MATCHING . In this paper , we consider a typical cross-device FL setting ( Kairouz et al. , 2019 ) , in which multiple end devices collaboratively perform local training on their own datasets Di , i = 1 , 2 , ... , n. The server owns a validation dataset D∗ for model evaluation . Every dataset is only accessible to its owner . Considering the distributional pattern of data representations ( Fig . 2 ) and the role of the global model in FL , we propose to profile the representations of local data using the global model . In this section , we first provide theoretical proof to support our observation that representations from neural network models tend to follow Gaussian distribution . Then we present a novel scheme to profile data representations and define profile dissimilarity for fast and secure representation comparison . 3.1 GAUSSIAN DISTRIBUTION OF REPRESENTATIONS . We first make the following definition to facilitate our analysis . Definition 1 ( The Lyapunov ’ s condition ) . A set of random variables { Z1 , Z2 , . . . , Zv } satisfy the Lyapunov ’ s condition if there exists a δ such that lim v→∞ 1 s2+δ v∑ k=1 E [ |Zk − µk|2+δ ] = 0 , ( 1 ) where µk = E [ Zk ] , σ2k = E [ ( Zk − µk ) 2 ] and s = √∑v k=1 σ 2 k. The Lyapunov ’ s condition can be intuitively explained as a limit on the overall variation ( with |Zk− µk|2+δ being the ( 2 + δ ) -th moment of Zk ) of a set of random variables . Now we present Proposition 1 and Proposition 2 . The Propositions provide theoretical support for our representation profiling and matching method to be introduced in Section 3.2 . Proposition 1 . The representations from linear operators ( e.g. , a pre-activation dense layer or a plain convolutional layer ) in a neural network tend to follow the Gaussian distribution if the layer ’ s weighted inputs satisfy the Lyapunov ’ s condition . Proposition 2 . The fused representations2 from non-linear operators ( e.g. , a hidden layer of LSTM or a residual block of ResNet ) in a neural network tend to follow the Gaussian distribution if the layer ’ s output elements satisfy the Lyapunov ’ s condition . The proofs of Propositions 1 and 2 are provided in Appendices A.1 and A.2 , respectively . We base our proof on the Lyapunov ’ s CLT which assumes independence between the variables . The assumption theoretically holds by using the Bayesian network concepts : let X denote the layer ’ s input and Hk denote the k-th component in its output . The inference through the layer produces dependencies X → Hk for all k. According to Local Markov Property , we have Hi independent of any Hj ( j ̸= i ) given X . Also , the Lyapunov ’ s condition is typically met when the model is properly initialized and batch normalization is applied . Next , we discuss the proposed representation profiling and matching scheme .

Summary: In this paper, the authors propose a user selection algorithm for federated learning (FL). The key motivation is to select high quality clients for update and thus to reduce the impact of low quality data on FL training. A hidden hypothesis is that high quality data has similar representations while noisy and low-quality data has different distributions of representations. Based on this hypothesis, the key idea is to select users based on their representation layer distribution difference from the global model, the higher the difference, the lower the chance of the client being selected.  Furthermore, the authors observed and proved that representation layers follow a Gaussian distribution, which makes the representation difference learning more efficient. The authors evaluated the proposed algorithm in comparison to existing algorithms using a small-scale sensor dataset and a large-scale EMNIST dataset. The proposed algorithm performs well in the evaluation.