PERSONALIZED FEDERATED LEARNING VIA HETEROGENEOUS MODULAR NETWORKS

A computer-implemented method for personalizing heterogeneous clients is provided. The method includes initializing a federated modular network including a plurality of clients communicating with a server, maintaining, within the server, a heterogenous module pool having sub-blocks and a routing hypernetwork, partitioning the plurality of clients by modeling a joint distribution of each client into clusters, enabling each client to make a decision in each update to assemble a personalized model by selecting a combination of sub-blocks from the heterogenous module pool, and generating, by the routing hypernetwork, the decision for each client.

BACKGROUND

Technical Field

The present invention relates to personalized federated learning, and, more particularly, to personalized federated learning via heterogeneous modular networks.

Description of the Related Art

Personalized Federated Learning (PFL) which collaboratively trains a federated model while considering local clients under privacy constraints has attracted much attention. Despite its popularity, it has been observed that existing PFL approaches result in sub-optimal solutions when the joint distribution among local clients diverges.

SUMMARY

A method for personalizing heterogeneous clients is presented. The method includes initializing a federated modular network including a plurality of clients communicating with a server, maintaining, within the server, a heterogenous module pool having sub-blocks and a routing hypernetwork, partitioning the plurality of clients by modeling a joint distribution of each client into clusters, enabling each client to make a decision in each update to assemble a personalized model by selecting a combination of sub-blocks from the heterogenous module pool, and generating, by the routing hypernetwork, the decision for each client.

A non-transitory computer-readable storage medium comprising a computer-readable program for personalizing heterogeneous clients is presented. The computer-readable program when executed on a computer causes the computer to perform the steps of initializing a federated modular network including a plurality of clients communicating with a server, maintaining, within the server, a heterogenous module pool having sub-blocks and a routing hypernetwork, partitioning the plurality of clients by modeling a joint distribution of each client into clusters, enabling each client to make a decision in each update to assemble a personalized model by selecting a combination of sub-blocks from the heterogenous module pool, and generating, by the routing hypernetwork, the decision for each client.

A system for personalizing heterogeneous clients is presented. The system includes a processor and a memory that stores a computer program, which, when executed by the processor, causes the processor to initialize a federated modular network including a plurality of clients communicating with a server, maintain, within the server, a heterogenous module pool having sub-blocks and a routing hypernetwork, partition the plurality of clients by modeling a joint distribution of each client into clusters, enable each client to make a decision in each update to assemble a personalized model by selecting a combination of sub-blocks from the heterogenous module pool, and generate, by the routing hypernetwork, the decision for each client.

DETAILED DESCRIPTION OF PREFERRED EMBODIMENTS

The huge quantity of data available nowadays is usually stored in the form of isolated islands. The barriers between data sources are usually difficult to break. In this context, Federated Learning (FL) emerges as a prospective solution that facilitates distributed collaborative learning without disclosing original training data whilst naturally complying with government regulations. FL works by collaboratively training a model under the orchestration of a central server (e.g., a service provider) while keeping the training data decentralized. Instead of aggregating the raw data to a centralized data center for training, FL leaves the raw data distributed on the client devices and trains a shared model on the server by aggregating locally computed updates, thus mitigating systemic privacy risks and costs resulting from conventional centralized machine learning approaches. Consequently, different clients share the same model structure and global model parameters.

In real applications, local data stored across devices are usually heterogeneous. The data may be distributed in a non-independently and identically distributed (e.g., non-IID) manner across multiple devices. In addition, some users may probably produce significantly more or less data than others. Moreover, the number of edge device owners may be significantly larger than the average number of training samples on each device. The problem of data heterogeneity deteriorates the performance of the global FL model on individual clients due to the lack of solution personalization. The global model shared across clients will not generalize well on a local distribution that is very different from the global distribution. To tackle this issue, researchers focus on Personalized Federated Learning (PFL), which aims to make the global model fit the distributions on most of the devices.

The conventional PFL approaches first learn a global model and then locally adapt it to each client by fine-tuning the global parameters. In this case, the trained global model can be regarded as a meta-model ready for further personalization of each local client. In order to build a better meta-model, many efforts have been made to bridge the FL and the Model Agnostic Meta Learning (MAML). However, the global generalization error usually does not decrease much for these approaches. Thus, the performance cannot be significantly improved. Another line of research focuses on jointly training a global model and a local model for each client to achieve personalization. This strategy does not perform well on the clients whose local distributions are far from their average. Cluster-based PFL approaches address this issue by grouping the clients into several clusters. The clients in a cluster share the same model while those belonging to different clusters have different models. Unfortunately, the model trained in one cluster will not benefit from the knowledge of the clients in other clusters, which limits the capability to share knowledge, and, therefore, results in a sub-optimal solution.

An alternative strategy is to adopt the Multi-Task Learning (MTL) framework to train a PFL model. However, some efforts are restricted to solve a convex objective due to the multi-task penalty. They are usually transformed into a dual problem to get a closed-form solution during the updating. Other MTL-based approaches are flexible to modern deep models and can be personalized to each client.

However, most existing efforts do not consider the difference in conditional distribution between clients, which is an important problem when building a federated model. For example, labels sometimes reflect sentiment. Some users may label a laptop as cheap while others may label the laptop as expensive. This conditional distribution heterogeneity problem will cause model inaccuracies on some clients where the p(y|x) is far from the average. To address the problem, recent works have assumed the data distribution of each client is a mixture of M underlying distributions and a flexible framework was proposed in which each client learns a combination of M shared components with different weights. It optimizes the varying conditional distribution pi(y|x) under the assumption that the marginal distribution pi(x)=p(x) is the same for all clients. This assumption, however, is problematic. For instance, in handwriting recognition, users who write the same words might still have different stroke widths, slants, etc. In this cases, pi(x)≠pj(x) for client i and j.

Other recent works either assume the marginal distribution pi(X) or the conditional distribution pi(y|x) the same across clients. In reality, data on each client may be deviated from being identically distributed, say, Pi≠Pjfor client i and j. That is, the joint distribution Pi(x,y) (can be rewritten as Pi(y|x)Pi(x) or Pi(x|y)Pi(y)) may be different across clients. This is referred to as the “joint distribution heterogeneity” problem. Existing approaches fail to completely model the difference of joint distribution between clients because they assume one term to be the same while varying the other one. Moreover, to accommodate different data distributions, the homogeneous model would be too large so that the given prediction power can be satisfied. Thus, the communication costs between the server and the clients would be huge. In this case, communication would be a key bottleneck to consider when developing FL methods. To this end, it is desirable to design an effective PFL model to accommodate heterogeneous clients in an efficient manner.

To solve the aforementioned issues, a Federated Modular Networks (FedMN) approach is presented, which personalizes heterogeneous clients efficiently. The main idea is that the exemplary methods implicitly partition the clients by modeling their joint distribution into clusters and the clients in the same cluster have the same architecture (FIG.1). Specifically, a shared module pool115with layers of module blocks (e.g., MLPs or ConvNets) is maintained in the server120. Each client130decides in each update to assemble a personalized model by selecting a combination of the blocks from the module pool115. A light-weighted routing hypernetwork110with differentiable routers is adopted to generate the decision of module block selection for each client130. The routing hypernetwork110considers the joint distribution pi(x,y) for client i by taking the joint distribution of the data set as the input. A decision parameterized by the routing hypernetwork110is a vector of discrete variables following the Bernoulli distribution. It selects a subset of the blocks from the module pool115to form an architecture for each client130. Clients with similar decisions will be implicitly assigned to the same cluster in each communication round. The proposed FedMN100enables a client130to upload only a subset of model parameters to the server120, which decreases the communication burden compared to traditional FL algorithms.

To sum up, the contributions are as follows, that is, the problem of joint distribution heterogeneity in the personalized FL is addressed and a FedMN approach is presented to alleviate this issue. An efficient mechanism is devloped to selectively upload model parameters which decreases the communication cost between the clients130and the server120.

As shown inFIG.3, FedMN100adopts modular networks310,320which include a group of encoders305,315in the first layer and multiple modular blocks307,317in the following or subsequent layers. The connection decisions between blocks in the modular networks310,320are made by a routing hypernetwork330.

The modular networks310,320first encode the data feature into low-dimensional embeddings by a group of encoders305,315. Then, personalized feature embeddings are obtained by discovering and assembling a set of modular blocks307,317in different ways for different clients. The modular networks310,320have L layers and the l-th layer has nlblocks of sub-networks. The encoders305,315in the 1st layer are n1independent blocks which learn feature embeddings for each client.

Formally, let xibe the i-th sample, and the feature embedding zi(j)is obtained after the j-th encoder is applied:

The choices of encoder networks are flexible. For example, convolutional neural networks (CNNs) can be adopted as encoders305,315for image data and transformers for text data. The set of feature embeddings {zi(1), . . . ,zi(n1)} of data point X, resulting from the encoders305,315in the 1st layer is the input of the following modular sub-networks constructed by a subset of the modular blocks307,317. There are L−1 layers of blocks in the sub-networks and each one is independent of the others. Each modular block j in layer l receives a list of tensors of feature embeddings from the modular sub-networks in the layer l−1.

MLPs are used as the modular blocks and each pair of them in successive layers may be connected or not. At most, there are E possible connection paths between modular blocks that can be calculated as follows:

To determine which path would be connected, the exemplary methods need to learn a decision VmϵZ2Efor client m. Each element vi(m)ϵVmis a binary variable with values chosen from {0,1}. vi(m)=1 indicates that the path between two blocks is connected, and 0 otherwise. Since some blocks may not have connected paths, Vmalso determines which subset of blocks will be selected from the modular pool115for each client130(FIG.1). Therefore, after obtaining Vm, the architecture for a client130is determined.

With the defined modular networks, the exemplary methods can formally define the learning objective. Specifically, in a generic FL with M clients where each client has a local dataset Dm={(xi,yi)}i=1|Dm|, the learning objective can be formulated by:

Here, W is the model parameter, D=∪mDmis the aggregated data set from all clients, and Lm(w) is an empirical risk computed from client m's data. The objective in (3) is optimized by iterating between local training and global aggregation for multiple communication rounds. For generic FL, the exemplary methods perform ŷi=ƒw(xi) to make a prediction in the local updating.

In the FedMN framework, after getting Vm, the architecture of the modular network for client m is fixed at an epoch during local updating. The model ƒ can be parameterized by θ which includes parameters in modular networks310,320and the routing hypernetwork330.

When making a prediction, the exemplary methods have ŷi=ƒθ(xi; Vm). Then, it is easy to extend the generic FL to get the empirical risk of FedMN100as:

However, the direct optimization of the objective in (4) is intractable as there are 2Ecandidates for each Vm. Thus, a relaxation is considered by assuming that the decision of each connection path in vi(m)ϵVmis conditionally independent to each other. Formally, it is given as:

A straightforward instantiation of P(vi(m)) is the Bernoulli distribution vi(m):Bern (πi(m)). P(vi(m)=1)=πi(m)is the probability that the i-th path exists in Vm. With this relaxation, the objective in (4) can be rewritten as:

where q(Πm) is the distribution of the decision variable parameterized by π(m)'s.

Due to the binary nature of Vm, it is impractical to optimize (6) with gradient-based back prorogation. To enable efficient computation, the exemplary methods further approximate the binary vector VmϵZ2Ewith a continuous real-valued vector in [0,1]E. In practice, the exemplary methods approximate each Bernoulli distribution vi(m):Bern (πi(m)) with a binary concrete distribution.

Formally, letting σ(·) as the Sigmoid function, it is given as:

The hyper-parameter τ is a temperature variable to trade-off between approximation and binary output.

For justification, when the temperature τ approaches to 0, the binary concrete distribution of vi(m)in (7) converge to the Bernoulli distribution vi(m):Bern(πi(m)). Specifically,

Since ϵ and πi(m)both lies in (0,1), and function

is monotonically increasing in this region.

Therefore, with reparameterization, combining (6) and (7) the learning objective is given as:

When temperature τ>0, the objective function in (8) has a well-defined gradient that enables efficient optimization with backpropagation.

The routing hypernetwork330that automatically learns Πmfrom the joint distribution is presented.

Suppose M clients are provided and such clients own local datasets D1, . . . , DM, where Dm={(x1, y1), . . . , (xnm, ynm)} is a set of size nmon client m. It is intended to obtain the joint n distribution embedding for each client. The kernel embeddings of joint distributions can be extended from that of the marginal distributions. Without loss of generality, a joint distribution P of variables X1, . . . , Xpcan be embedded into a p-th order tensor product feature space ⊗η=1pHηas:

where X1:pis a set of p variables {X1, . . . , Xp} defined on xη=1pΩηΩ1x . . . xΩp, ϕηis the feature map of variable Xηendowed with kernel kηin RKHS Hη, ⊗η=1pϕη(xη)ϕ1(x1)⊗ . . . ⊗ϕp(xp) is the feature map in the tensor product Hilbert space, where the inner product satisfies⊗η=1pϕη(xη), ⊗η=1pϕη(x′η)=Πη=1pkη(xη,x′η). The joint embedding is an uncentered cross-covariance operator CX1:pby the standard equivalence between tensor and linear map. In other words, the covariance of a set of functions ƒ1, . . . , ƒpcan be obtained by: EX1:p[Πη=2pƒη(Xη)]=⊗η=1pƒη, CX1:p.

To estimate the embeddings of distribution P (X1, . . . ,Xp), finite samples can be used. For a sample set DX1:p={x11:p, . . . , xn1:p} of size n which is drawn i.i.d. from P(X1, . . . ,Xp), the joint embedding can be estimated empirically by:

which converges to its population counterpart in RKHS norm. For instantiation, since the joint distribution on feature domain X is considered and domain Y is labeled for client m, m∈[M], the joint embedding is given as:

The mappings ϕx(X) and ϕy(y) are flexible. The tensor product ϕx(X)⊗ϕx(X) or higher order ones can be used, such as ϕx(X)⊗ϕx(X)⊗ϕx(X). θhis denoted as the parameters used in the routing hypernetwork, which is a part of the model parameters θ. The exemplary methods parameterize the feature mappings by employing neural networks, and thus the joint embedding estimator in (11) results in:

Then, two fixed-size vector representations of a dataset are provided by the averaged output of the two neural networks: ϕθhx:xRdxand ϕθhy:yRdy. By the Universal Approximation Theorem, the exemplary methods concat ϕθhx(xi) and ϕθhy(yi) and adopt a single-layer perceptron hθh:Rdx+dyRE, where E is the total possible number of paths between successive modulars as in (2), and, thus, the product operator in (12) can be approximated by:

which results in a vector of joint embedding of the local dataset at client m.

Since Vmdetermines the connection paths between blocks, some blocks may not have connections with other ones. To clarify the message passing between blocks, the connection paths between blocks in the layer (l−1) and the layer l are denoted as

with the element Cjk(m)∈{0,1} in its j-th row and k-th column.

Letting uj(l)be the input tensor for the j-th block in the layer l, and ũj(l)be its output:

To decrease the number of model parameters transmitted between clients and the server, a block-wise strategy is devloped for clients to upload the local models to the server and copy them from the server. In detail, when the decision Vmis obtained, it is known that the inputs for some blocks are 0's from (15). Therefore, some blocks are still active whose input is not all 0's. In total, it is denoted that there are B blocks in the modular network, where B=n2+ . . . +nl. Let am∈Z2Bto denote which blocks are active for the local model at client m, with the element ai(m)=1 if the input for the i-th block is not 0 while ai(m)=0 otherwise. When uploading the model to the server, the client only uploads the active blocks whose ai(m)=1. When copying the model from the server, the client only copies the parameters of active blocks from the global model. This strategy significantly reduces unnecessary communication costs between clients and the server.

When all the clients upload their local models to the server, the server averages the model to get the global modules. In the proposed FedMN100, the aggregation for the routing hypernetwork110is similar to FedAvg. For the modular networks310,320ofFIG.3, the aggregation, however, is in a block-wise manner. Specifically, let θi(m)be the model parameters of the i-th modular block for client m, and the server performs the aggregation to obtain the global parameter of the i-th modular block θiby:

The federated learning process of FedMN100is provided in Algorithm 1 below. The computation complexity in each round at each client in FedMN100is the same as that in FedAvg. The FedMN algorithm is a personalized FL method whose convergence is guaranteed.

Regarding the Federated Modular Networks Algorithm:

Input: Number of clients M; local dataset {Dm}m=1M, where Dm={(xi, ydi)}i=1|Dm|; number of layers of modular network L; number of modular blocks in each layer {nl}l=1L; number of communication rounds T; number of local epochs K; learning rate η.

The exemplary methods address the problem of joint distribution heterogeneity in the personalized FL. To tackle this issue, the exemplary methods propose a novel FedMN approach that adaptively assembles architectures for each client by selecting a subset of module blocks from a module pool in the global model. The proposed FedMN100adopts a light-weighted routing hypernetwork to model the joint distributions for each client and produce the module selection decisions. Advised by the decision, each client selects its personalized architecture. When federated updating, each client uploads, and downloads only part of the module parameters, which reduces the communication burden between the server and the clients.

FIG.1is a block/flow diagram of an exemplary architecture100for personalized federated learning, in accordance with embodiments of the present invention.

The routing hypernetwork110produces decisions for each of the clients130. The clients130with similar decisions are grouped into the same cluster which copies the same subset of blocks as the local model from the module pool115in the server120. After the local updating on each client130, the clients130send their model parameters back to the server120. The server120aggregates the model parameters block wisely, which results in a global model pool115.

FIG.2is a block/flow diagram of exemplary applications of the architecture for personalized federated learning, in accordance with embodiments of the present invention.

There are various applications230for the proposed architecture100for personalized federated learning. For all general supervised or unsupervised learning tasks that includes edge devices210such as smartphones, sensors, radars, and so forth, the proposed architecture100can provide personalized prediction for edge devices, and at the same time, the prediction model can have the knowledge shared by other edge devices. The whole framework is privacy protected. The communication costs between edges are low. The diverse artificial intelligence (AI) services220provided can includes services222, such as, anomaly detection, label prediction, sales prediction, finance prediction, medical prediction, natural language processing (NLP), etc.

FIG.3is a block/flow diagram of an exemplary architecture for personalized federated learning with heterogenous modular networks, in accordance with embodiments of the present invention.

The modular networks310,320include a group of encoders305,315in the first layer and modular blocks307,317in the following layers. The connection paths between blocks are determined by a decision from the routing hypernetwork330. The input of modular networks310,320is in sample-wise while the input of routing hypernetwork330is the full dataset for each client.

FIG.4is a block/flow diagram of an exemplary workflow of the personalized federated learning architecture, in accordance with embodiments of the present invention.

At block410, input data in edge services.

At block420, edge devices are locally trained by using local data.

At block430, local selected modules' parameters and local hyper network parameters are sent to the server.

At block440, the server aggregates block-wise module parameters and the hyper network parameters.

At block450, the aggregated global parameters are sent back to local clients.

At block460, a prediction is made.

FIG.5is a block/flow diagram of an exemplary workflow of the edge device components and local selected modules' components, in accordance with embodiments of the present invention.

At block420, edge devices are locally trained by using local data.

At block522, adaptively select a subset of modules from a module pool to assemble heterogeneous architectures for different clients.

At block524, use a light-weighted routing hyper network to model the joint data distribution of the local client.

At block526, edge devices use the local routing hyper network.

At block430, local selected modules' parameters and local hyper network parameters are sent to the server.

At block532, only selected modules' parameters and hyper networks are sent to the server.

At block534, it is noted that this will significantly decrease communication costs.

FIG.6is a block/flow diagram of an exemplary workflow of the block-wise module parameters and the aggregated global parameters, in accordance with embodiments of the present invention.

At block440, the server aggregates block-wise module parameters and the hyper network parameters.

At block642, the aggregation is to weighted average for the parameters of blocks, that is only average a block if this block is chosen by k number of clients (k>0); hyper network parameters are averaged over clients.

At block644, blocks not chosen by any clients are not aggregated.

At block450, the aggregated global parameters are sent back to local clients.

At block652, only those blocks that are updated need to be sent back to the clients.

FIG.7is an exemplary processing system for personalizing heterogeneous clients, in accordance with embodiments of the present invention.

The processing system includes at least one processor (CPU)904operatively coupled to other components via a system bus902. A GPU905, a cache906, a Read Only Memory (ROM)908, a Random Access Memory (RAM)910, an input/output (I/O) adapter920, a network adapter930, a user interface adapter940, and a display adapter950, are operatively coupled to the system bus902. Additionally, the Federated Modular Network (FedMN)100is presented, a novel PFL approach that adaptively selects sub-modules from a module pool to assemble heterogeneous neural architectures for different clients. FedMN100adopts a light-weighted routing hypernetwork110to model the joint distribution on each client130and produce the personalized selection of the module blocks for each client130. To reduce the communication burden in existing FL, an efficient way to interact between the clients130and the server120is developed.

A storage device922is operatively coupled to system bus902by the I/O adapter920. The storage device922can be any of a disk storage device (e.g., a magnetic or optical disk storage device), a solid-state magnetic device, and so forth.

A transceiver932is operatively coupled to system bus902by network adapter930.

User input devices942are operatively coupled to system bus902by user interface adapter940. The user input devices942can be any of a keyboard, a mouse, a keypad, an image capture device, a motion sensing device, a microphone, a device incorporating the functionality of at least two of the preceding devices, and so forth. Of course, other types of input devices can also be used, while maintaining the spirit of the present invention. The user input devices942can be the same type of user input device or different types of user input devices. The user input devices942are used to input and output information to and from the processing system.

A display device952is operatively coupled to system bus902by display adapter950.

FIG.8is a block/flow diagram of an exemplary method for personalizing heterogeneous clients, in accordance with embodiments of the present invention.

At block1001, initializing a federated modular network including a plurality of clients communicating with a server.

At block1003, maintaining, within the server, a heterogenous module pool having sub-blocks and a routing hypernetwork.

At block1005, partitioning the plurality of clients by modeling a joint distribution of each client into clusters.

At block1007, enabling each client to make a decision in each update to assemble a personalized model by selecting a combination of sub-blocks from the heterogenous module pool.

At block1009, generating, by the routing hypernetwork, the decision for each client.

As used herein, the terms “data,” “content,” “information” and similar terms can be used interchangeably to refer to data capable of being captured, transmitted, received, displayed and/or stored in accordance with various example embodiments. Thus, use of any such terms should not be taken to limit the spirit and scope of the disclosure. Further, where a computing device is described herein to receive data from another computing device, the data can be received directly from the another computing device or can be received indirectly via one or more intermediary computing devices, such as, for example, one or more servers, relays, routers, network access points, base stations, and/or the like. Similarly, where a computing device is described herein to send data to another computing device, the data can be sent directly to the another computing device or can be sent indirectly via one or more intermediary computing devices, such as, for example, one or more servers, relays, routers, network access points, base stations, and/or the like.