KNOWLEDGE TRANSFER IN COLLABORATIVE LEARNING

Examples of ensemble knowledge transfer in collaborative learning include: receiving, at a primary node, from a plurality of remote nodes, a plurality of trained proxy machine learning (ML) models, wherein each proxy ML model is received from a different one of the plurality of remote nodes, and wherein each of the plurality of remote nodes is remote across a network from the primary node; training a primary ML model using the plurality of proxy ML models, wherein training the primary ML model comprises: for each of a plurality of training cases of a primary training dataset, weighting results from each of the proxy ML models based on at least a confidence of the respective proxy ML model regarding the training case.

BACKGROUND

Federated learning (FL) is a privacy-friendly solution for training machine learning (ML) models, in which a plurality of edge-device nodes (clients), such as cellular user devices or internet of things (IoT) devices, participate in collaborative learning - without disclosing their data. The datasets used for training on each edge-device node, which may contain user data such as speech samples or imagery, remains on the node, and is used to locally train an ML model. The locally-trained ML model is then sent to an aggregating server for transferring the learned knowledge, for example using matched averaging or aggregation with locally-trained ML models from other nodes. FL may be used when using a labeled training dataset for traditional ML training is undesirable, due to cost and/or privacy concerns.

Unfortunately, these approaches require that the ML models have identical architectures. Resource limitations (e.g., memory and processing power) on the clients preclude the training of large ML models on clients, thereby rendering large ML model architectures and some ML architectures unsuitable for FL, even on the aggregating server (which may not have the same resource limitations).

SUMMARY

The disclosed examples are described in detail below with reference to the accompanying drawing figures listed below. The following summary is provided to illustrate some examples disclosed herein. It is not meant, however, to limit all examples to any particular configuration or sequence of operations.

Examples of ensemble knowledge transfer in collaborative learning include: receiving, at a primary node, from a plurality of remote nodes, a plurality of trained proxy machine learning (ML) models (an ensemble of proxy ML models), wherein each proxy ML model is received from a different one of the plurality of remote nodes, and wherein each of the plurality of remote nodes is remote across a network from the primary node; training a primary ML model using the plurality of proxy ML models, wherein training the primary ML model comprises: for each of a plurality of training cases of a primary training dataset, weighting results from each of the proxy ML models based on at least a confidence of the respective proxy ML model regarding the training case.

DETAILED DESCRIPTION

The various examples will be described in detail with reference to the accompanying drawings. Wherever possible, the same reference numbers will be used throughout the drawings to refer to the same or like parts. References made throughout this disclosure relating to specific examples and implementations are provided solely for illustrative purposes but, unless indicated to the contrary, are not meant to limit all examples.

Examples of ensemble knowledge transfer in collaborative learning include: receiving, at a primary node, from a plurality of remote nodes, a plurality of trained proxy machine learning (ML) models (an ensemble of proxy ML models), wherein each proxy ML model is received from a different one of the plurality of remote nodes, and wherein each of the plurality of remote nodes is remote across a network from the primary node; training a primary ML model using the plurality of proxy ML models, wherein training the primary ML model comprises: for each of a plurality of training cases of a primary training dataset, weighting results from each of the proxy ML models based on at least a confidence of the respective proxy ML model regarding the training case.

Aspects of the disclosure improve the operations of computing devices by enabling unsupervised training of a primary ML model with an unlabeled dataset, by distilling knowledge from proxy ML models that may be of different architectures - and which is able to leverage datasets on remote nodes without exposing those datasets to the primary ML model or exposing the primary ML model to any of the remote nodes. This capability is able to significantly drive down the cost of training large primary ML models, preserve the privacy of datasets at the remote nodes (e.g., user devices), while providing robust training.

Aspects of the disclosure leverage weighted consensus to achieve the above-identified advantages by, for each of a plurality of training cases of a primary training dataset, weighting results from each of the proxy ML models based on at least a confidence of the respective proxy ML model regarding the training case. Aspects of the disclosure further leverage diversity regularization, in which proxy ML models that do not follow a consensus of the plurality of proxy ML models also transfer representations to the primary ML model. This further improves the performance of the primary ML model. Aspects of the disclosure further permit the use of a larger and different architecture for the primary ML model than is used for the proxy ML models, which enables smaller ML models to execute on the remote nodes while providing enhanced performance for the primary ML model. This advantageous operation, along with the advantageous use of (less expensive, more readily-available) unlabeled training data is facilitated by the use of a weighted consensus-based distillation scheme.

FIG.1illustrates an example arrangement100that advantageously provides ensemble knowledge transfer in collaborative learning. In some examples, arrangement100is implemented using examples of computing device500ofFIG.5. For example, each of a primary node102, a remote node132a, a remote node132b, a remote node132c, an operational node140, and a network530may comprise one or more computing devices500. Although only three remote nodes132a-132cand three proxy ML models130a0130care illustrated, it should be understood that the number of remote nodes and proxy ML models may be significantly larger, for example numbering in the thousands.

A primary ML model110is to be trained to perform an ML task on ML task data142, when deployed on operational node140. The ML task may be image classification, object detection or recognition, speech recognition, language classification, or another task. ML task data142may be image, video, or audio data, or another type of data. Training of primary ML model110is performed on primary node102by a primary training manager114, using a primary training dataset116. The training may be unsupervised. Primary training dataset116may include or be entirely unlabeled training cases, and may be considered to be a public dataset. Primary training manager114may also provide initial training of proxy ML models130a-130c.

After initial training, proxy ML models130a-130care trained on a respective one of remote nodes132a-132c. For example, proxy ML model130amay be trained on remote node132aby a local training manager134a, using local training dataset136a; proxy ML model130bmay be trained on remote node132bby a local training manager134b, using local training dataset136b; and proxy ML model130cmay be trained on remote node132cby a local training manager134c, using local training dataset136c. Client-side training (e.g. training at remote nodes134a-134c) may be supervised or unsupervised. Training datasets136a-136cmay be labeled or unlabeled.

In some examples, remote nodes132a-132cmay be user devices, and training datasets136a-136cmay contain data for which privacy is a concern. Arrangement100does not require disclosure of any of training datasets136a-136coutside of their respective remote nodes132a-132c, in order for primary ML model110to benefit from the content of training datasets136a-136c. This enhances the privacy of users of remote nodes132a-132c.

A coordinator120selects a remote node for each of proxy ML models130a-130c, deploys proxy ML models130a-130cacross network530on the selected remote node for training, and later retrieves proxy ML models130a-130cfor training primary ML model110. Deployment, retrieval, and training may iterate to improve the performance of primary ML model110. Proxy ML models130a-130cand primary ML model110rotate among student and teacher roles when distilling knowledge. In a student-teacher training technique, the teacher is trained first, and then is used to train the student.

Here, proxy ML models130a-130care trained remotely, possibly using labeled training data, and then retrieved to be teachers to primary ML model110in the role of the student. Training cases from primary training dataset116are provided to the ensemble of proxy ML models130a-130c, and the output results are used to teach primary ML model110. Because the output of pre-trained proxy ML models130a-130cis used as what primary ML model110should output, there is no need for the training cases from primary training dataset116to be labeled.

Coordinator120may select the remote nodes randomly, or use selection criteria122that includes model architecture information, training history (to diversify training of each proxy ML model among differing remote nodes), remote node dataset type, remote node dataset size, and other criteria. Different ML architectures, which is a possibility with arrangement100, and/or different data distributions result in different behaviors.

When distilling knowledge from proxy ML models130a-130cto primary Ml model110, proxy ML models130a-130ceach report a confidence level, in addition to a result on a training case. This confidence may be based on a distribution of logits or an indep algorithm. For a proxy ML model, if logits have a sharp peak, then confidence is high, whereas if logits are spread, confidence will be low. Consensus among plurality of proxy ML models130a-130cis weighted by at least this confidence.

In some examples, an arbiter124scores each of proxy ML models130a-130cand assigns an external weight to the output of a proxy ML model (in addition to a weight based on each proxy ML model’s self-reported confidence). Arbiter124uses weighting criteria126to determine weights for the various ones of proxy ML models130a-130c, which may include training history information that indicates which of proxy ML models130a-130chave received more training or training on a larger number of different ones of remote nodes132a-132c.

After primary ML model110has been trained by proxy ML models130a-130c, primary ML model110will have the benefit of the aggregate knowledge of proxy ML models130a-130c, whereas (after only the initial round), each of proxy ML models130a-130cwill have only the knowledge it received individually, from its training history. Thus, primary ML model110is likely to have the benefit of more comprehensive training than the individual ones of proxy ML models130a-130c. At this point the student-teacher training roles swap, and training cases from primary training dataset116are provided to primary ML model110. The output of primary ML model110is now used to further train each of proxy ML models130a-130c. In some examples, both training primary ML model110by proxy ML models130a-130cand also training proxy ML models130a-130cby primary ML model110comprise transfer learning.

Transfer learning is an ML process that focuses on storing knowledge gained while solving one problem and applying it to a different, but related problem. For example, proxy ML models130a-130cgain knowledge from solving problems in training datasets136a-136con remote nodes132a-132c, under the direction of local training managers134a. They then solve related problems in primary training dataset116on primary node102, along with primary ML model110, under the direction of primary training manager114. Primary training manager114uses the results from proxy ML models130a-130cas what primary ML model110needs to learn.

FIG.2illustrates the movement of proxy ML models and knowledge in arrangement100. On primary node102, training manager114provides initial training to proxy ML models130a-130c. Coordinator120assigns each of proxy ML models130a-130cto a respective one of remote nodes132a-132c, and deploys them, where they are each trained using one of local training datasets136a-136c. Coordinator120then retrieves each of proxy ML models130a-130cback to primary node102.

Training manager114distills knowledge from proxy ML models130a-130cinto primary ML model110, as described below. Primary ML model110has now been trained in a least a first round of training. However, additional rounds of training may be leveraged. In preparation for redeploying proxy ML models130a-130cfor further training, training manager114distills the combined knowledge of primary ML model110into proxy ML models130a-130c. Proxy ML models130a-130cnow likely have superior training and performance than at the time of their initial deployment.

Coordinator120assigns each of proxy ML models130a-130cto a respective one of remote nodes132a-132c, and redeploys them, where they are each trained using one of local training datasets136a-136c. In some examples, the assignment of a proxy ML model to a remote node is at least partially random. In some example, coordinator120attempts to diversify the training, such that each of proxy ML models130a-130cis likely to be deployed to a different one of remote nodes132a-132con second and subsequent deployments. As illustrated, in this second deployment, proxy ML model130ais deployed to remote node132c, proxy ML model130bis deployed to remote node132a, and proxy ML model130cis deployed to remote node132b. Coordinator120then retrieves each of proxy ML models130a-130cback to primary node102for the next round of training primary ML model110. This loop may iterate as needed.

Arrangement100provides an ensemble knowledge transfer framework that trains primary ML model110with smaller and heterogeneous models (in some examples) that are trained on clients (remote nodes132a-132c), using primary training dataset116(an unlabeled public dataset, in some examples). Three consecutive steps are used, as shown inFIG.2, and may be iterated: (1) clients local training and representation transfer, (2) weighted consensus distillation with diversity regularization, and (3) server representation transfer.

A cross-device setup, which may involve Federated learning (FL) in some examples, with an N-class classification task where K clients are connected to a server (e.g., primary node102), is used. Each client k∈[K] has its local training dataset Bkand each data sample ξ is a pair (x, y) with input x∈ ℝdand label yE[1,N]. Each client has its local objective

with f(w,ξ) being the composite loss function. Having a large w with identical architecture across all resource-constrained clients, as done in the standard FL framework, may be infeasible. Moreover, the local minimums

k∈[1,K] minimizing Fk(w) can be different from each other due to data-heterogeneity. These obstacles are overcome by training a large server model with a data-aware ensemble transfer from the smaller models trained on clients.

For U small and heterogeneous models at the server with M = {1 :w1, ..., U : w̅U}, where M is the hashmap with the keys 1, ..., U as model identifiers, the values M [i] =wi∈ ℝnias the models, and nias the number of parameters for i∈[U]. The heterogeneous models are not necessarily od the same architecture as each other or the primary model being trained at the server. All of the small models in M have a representation layer hi∈ℝU,i∈[U], which includes the classification layer, connected to the end of their different model architectures u<<ni; i∈[U]. Each client is designated by its model to use from M depending on its resource capability. With slight customization of notation, the model identifier chosen by client k is denoted as M(k)∈[1;U], and the local model for that client k∈[K] as wk=wM(k)= M[M(k)], which has its respective representation layer defined as hk.

The server has its global model defined asw∈ ℝnalso with its representation layer defined ash∈ ℝu. The server model is assumed to be much larger than the models in M, (i.e., n>>ni; i∈[U]). As shown below, the representation layersh, hk, k∈[K] are shared bidirectionally between clients and server to transfer the representations learned from their respective training. Only the server has access to an unlabeled public dataset denoted as P. The local models wk, k∈[K], and server modelwoutput soft-decisions (logits) over the predefined number of classes N, which is a probability vector over the N classes. The soft-decision of model wkover any input data x in either the private or public dataset is S(wk, x). ℝnM(k)× (Bk∪ P) → ΔN, where ΔNstands for the probability simplex over N.

Step1: Client Local Training & Representation Transfer. For each communication round t, the server gets the set of m<K clients, denoted as S(t,0), by selecting them in proportion to their dataset size. The upper-subscript (t,r) denotes the tthcommunication round and rthlocal iteration. Note that S(t,0)is independent of the local iteration index. For each client kES(t,0), the most recent version of its designated model

M[M(k)] is sent from the server to the client. The clients perform local mini-batch stochastic-gradient descent (SGD) steps on their local model

with their private dataset Bk, k∈[K]. Accordingly, the clients k∈S(t,0)perform local updates so that for every communication round their local models are updated as:

where ηtis the learning rate and

is the stochastic gradient over mini-batch

of size b randomly sampled from Bk. After the clients

finish their local updates, the models

k∈S(t,0)are sent to the server. Each client has different representation layers

in their respective models

k∈S(t,0). The server receives these models from the clients and updates its representation layer with the ensemble models as

This pre-conditions the server model with the clients’ representations for Step2where the server model is trained with the ensemble loss.

Step2: Ensemble Loss by Weighted Consensus with Diversity Regularization. Next, the server model is trained via a weighted consensus-based knowledge distillation scheme from the small models received from the clients. A key characteristic of this ensemble is that each model may be trained on data samples from different data distributions. Therefore, some clients may be more confident than others on each of the public data samples. However, all clients may still have useful representations to transfer to the server, even when they are not very confident about that particular data sample. Thus, a weighted consensus distillation scheme with diversity regularization is employed, where the server model is trained on the consensus knowledge from the ensemble of models while regularized by the clients that do not follow the consensus.

Weighted Consensus: A reliable consensus is derived over the ensemble of models by evaluating the variance within the logit vectors

for each client

This variance is denoted as

which is the variance taken over the N total probability values for the N-multi-class classification task. Higher

indicates higher confidence, i.e., a more confident client k about how well it models data sample x, and the reverse. Thus, the logits from the clients with high

are weighted more heavily than low-variance logit clients. A confidence based weighted average over the logits is set for each data sample x∈P denoted as:

where the weights are defined as:

The resulting weighted consensus logit

efficiently derives the consensus out of the ensemble of models trained on heterogeneous datasets, due to filtering out the following two main adversaries: (1) the non-experts with low intravariance within each logit, and (2) overly-confident but erroneous outliers by utilizing the power of ensemble where multiple experts contribute to the consensus.

For each data sample x, the most probable label from

The pair

is the consensus-derived data sample from the unlabeled public dataset P, which is then used to train the server model with the cross-entropy loss

The cross-entropy loss term used in the final ensemble loss for training the server model is:

Diversity Regularization: While the confidence based weighted consensus can derive a more reliable consensus from the ensemble, the diversity across the participating models is less represented. Meaningful representation information of what clients learned from their private data may be included, in some examples, even when certain clients have low confidence and may have different logits from the consensus. Encouraging diversity across models can improve the generalization performance of ensemble learning. Thus, the logits are gathered from the clients that do not coincide with the consensus:

and formulate a regularization term:

where the weights are

Accordingly, the diversity regularization term for the final ensemble loss is where KL(·,·) is the KL-divergence loss between two logits:

Final Ensemble Loss: Finally, combining the weighted consensus based cross-entropy loss in Eq. (5) with the diversity regularization in Eq. (9), the server model is updated, in every communication round t, by minimizing the following objective function:

To minimize the ensemble loss in Eq. (10), rather than going through the entire dataset P, the server model takes τsminibatch SGD steps by sampling a mini-batch

of bsdata samples from P uniformly at random, without replacement. Then, for every communication round t the server performs:

Step3: Model Aggregation and Server’s Representation Transfer: Finally, the server aggregates the received clients’ models based on their architecture and updates the models in M. With

the models in M are aggregated as:

Since the server has been transferred the knowledge from the ensemble of models in a data-aware manner, the server has now a better representation than the clients’ models. The updatedh(t,rs)from the updated server modelw(t,rs)is therefore transferred to the models in M.

Example algorithm: The above description demonstrates component that provide federated ensemble transfer with heterogeneous models. An additional description of an algorithm is provided below:

Designated Model Ids for each client k ∈ [K]: M(k) ∈ [1, U]; Selected set of m<K clients S(0,0).

04: Clients k∈S(t,0)in parallel do:05: Receivewkt,0=w¯Mkt,0=Mkfrom server06: Getwkt,rfrom update rule (1)07: Send updated local modelwkt,rto the server08: Global server do:09: Receive all updated local modelwkt,r,k∈S(t,0)10: Transfer clients’ representations to the server byh¯t+1,0=1st,0∑k∈St,0hkt,r11: Get w̅(t,rs)fromfrom update rule (11)12: Transfer server’s representation to update models in M by update rule (12)13: Select m clients for S(t+1,0)uniformly at random, without replacement in proportion to the dataset sizeEND.

FIG.3shows a flowchart300illustrating exemplary operations associated with arrangement100. In some examples, operations described for flowchart300are performed by computing device500ofFIG.5. Flowchart300commences with operation302, which initializes plurality of proxy ML models130a-130cwith initial training. In some examples, at least two of proxy ML models130a-130chave different architectures from each other (e.g., plurality of proxy ML models130a-130cmay be heterogeneous). In some examples, at least one of proxy ML models130a-130chas a different architecture than primary ML model110.

Operation304deploys plurality of proxy ML models130a-130cto plurality of remote nodes132a-132c. In some examples, at least one node of plurality of remote nodes132a-132ccomprises a user device. In some examples, each node of plurality of remote nodes132a-132ccomprises a user device. Operation306trains each of plurality of proxy ML models130a-130con its respective remote node.

Operation308includes receiving, at primary node102, from plurality of remote nodes132a-132c, plurality of trained proxy ML models130a-130c. Each proxy ML model is received from a different one of plurality of remote nodes132a-132c, and each of plurality of remote nodes132a-132cis remote across network530from primary node102.

Operation310trains primary ML model110using plurality of proxy ML models130a-130c, with primary ML model110in the role of student and plurality of proxy ML models130a-130c(the ensemble) in the role of teachers. The training of primary ML model110is performed with a weighted consensus-based distillation scheme, as described above. Training primary ML model110comprises, for each of the plurality of training cases of the training dataset, determining a weighted consensus of plurality of proxy ML models130a-130c. In some examples, primary training dataset116comprises unlabeled training cases (e.g., the primary training dataset is unlabeled). In some cases one or more, or even each of the plurality of training cases is unlabeled. In some examples, training primary ML model110comprises unsupervised training. Operation310is performed using one or more of operations310-316.

Operation312weights results from each of proxy ML models130a-130cfor each of a plurality of training cases of primary training dataset116, based on at least a confidence of the respective proxy ML model regarding the training case. Proxy ML models having higher confidence are weighted more heavily than proxy ML models having less confidence. In some examples, weighting results from each of proxy ML models130a-130ccomprises further weighting the results from each of proxy ML models130a-130cbased on at least a score assigned to each of proxy ML models130a-130cby arbiter124in operation314. In some examples, training primary ML model110comprises diversity regularization in operation316. In diversity regularization, proxy ML models that do not follow a consensus of plurality of proxy ML models130a-130calso transfer representations to primary ML model110, but with lesser weight.

Decision operation318determines whether to continue the training of primary ML model110with additional stages, or deploy primary ML model110for operation. If additional training is selected, operation320trains each of proxy ML models130a-130cwith (now trained) primary ML model110. The student and teacher roles are swapped, with primary ML model110in the role of teacher and plurality of proxy ML models130a-130c(the ensemble) in the role of students.

Operation322selects from among plurality of remote nodes132a-132cfor further training of plurality of proxy ML models130a-130c. In some examples, the selection is random. In some examples, selecting a remote node for further training is based on at least a training history of the proxy ML model. In some examples, selecting a remote node for further training is based on at least an architecture of the proxy ML model. In some examples, selecting a remote node for further training is based on at least a dataset type at the remote node. In some examples, selecting a remote node for further training is based on at least a dataset size at the remote node.

Operation324deploys plurality of trained proxy ML models130a-130cto plurality of remote nodes132a-132cfor further training. Each proxy ML model is deployed to a different one of plurality of remote nodes132a-132c. In some examples, deploying plurality of trained proxy ML models130a-130cfor further training comprises deploying plurality of trained proxy ML models130a-130cto the selected remote nodes. In some examples, deploying plurality of trained proxy ML models130a-130cfor further training comprises deploying plurality of trained proxy ML models130a-130cto a random one of remote nodes132a-132c.

Operation326includes receiving, at primary node102, from plurality of remote nodes132a-132c, (further-trained) plurality of proxy ML models132a-132c. Operation328further trains primary ML model110using (further-trained) plurality of proxy ML models130a-130c. Flowchart300then returns to operation310.

When training is complete, operation330deploys trained primary ML model110to operational node140. In operation332trained primary ML model110performs an ML task. In some examples, the ML task comprises image classification, object detection, object recognition, speech recognition, or language classification. In some examples, primary ML model110performs ML task on primary node102, contemporaneously with ongoing training.

FIG.4shows a flowchart400illustrating exemplary operations associated with arrangement100. In some examples, operations described for flowchart400are performed by computing device500ofFIG.5. Flowchart400commences with operation402, which includes receiving, at a primary node, from a plurality of remote nodes, a plurality of trained proxy machine learning (ML) models, wherein each proxy ML model is received from a different one of the plurality of remote nodes, and wherein each of the plurality of remote nodes is remote across a network from the primary node.

Operation404includes training a primary ML model using the plurality of proxy ML models, and is performed with operation406. Operation406includes, for each of a plurality of training cases of a primary training dataset, weighting results from each of the proxy ML models based on at least a confidence of the respective proxy ML model regarding the training case.

ADDITIONAL EXAMPLES

An example system comprises: a processor; and a computer-readable medium storing instructions that are operative upon execution by the processor to: receive, at a primary node, from a plurality of remote nodes, a plurality of trained proxy ML models, wherein each proxy ML model is received from a different one of the plurality of remote nodes, and wherein each of the plurality of remote nodes is remote across a network from the primary node; and train a primary ML model using the plurality of proxy ML models, wherein training the primary ML model comprises: for each of a plurality of training cases of a primary training dataset, weighting results from each of the proxy ML models based on at least a confidence of the respective proxy ML model regarding the plurality of training cases.

An example computerized method comprises: receiving, at a primary node, from a plurality of remote nodes, a plurality of trained proxy ML models, wherein each proxy ML model is received from a different one of the plurality of remote nodes, and wherein each of the plurality of remote nodes is remote across a network from the primary node; and training a primary ML model using the plurality of proxy ML models, wherein training the primary ML model comprises: for each of a plurality of training cases of a primary training dataset, weighting results from each of the proxy ML models based on at least a confidence of the respective proxy ML model regarding the plurality of training cases.

One or more computer storage devices having computer-executable instructions stored thereon, which, upon execution by a computer, cause the computer to perform operations comprising: receiving, at a primary node, from a plurality of remote nodes, a plurality of trained proxy ML models, wherein each proxy ML model is received from a different one of the plurality of remote nodes, and wherein each of the plurality of remote nodes is remote across a network from the primary node; and training a primary ML model using the plurality of proxy ML models, wherein training the primary ML model comprises: for each of a plurality of training cases of a primary training dataset, weighting results from each of the proxy ML models based on at least a confidence of the respective proxy ML model regarding the plurality of training cases.

Alternatively, or in addition to the other examples described herein, examples include any combination of the following:initializing the plurality of proxy ML models with initial training;at least two of the proxy ML models have different architectures from each other;at least one of the proxy ML models has a different architecture than the primary ML model;deploying the plurality of proxy ML models to the plurality of remote nodes;at least one node of the plurality of remote nodes comprises a user device;each node of the plurality of remote nodes comprises a user device;training each of the plurality of proxy ML models on its respective remote node;training the primary ML model with a weighted consensus-based distillation scheme;training the primary ML model comprises, for each of the plurality of training cases of the training dataset, determining a weighted consensus of the plurality of proxy ML models;proxy ML models having higher confidence are weighted more heavily than proxy ML models having less confidence;weighting results from each of the proxy ML models comprises further weighting the results from each of the proxy ML models based on at least a score assigned to each of the proxy ML models;training the primary ML model comprises unsupervised training;the primary training dataset comprises unlabeled training cases;the primary training dataset is unlabeled;each of the plurality of training cases is unlabeled;training the primary ML model comprises diversity regularization;proxy ML models that do not follow a consensus of the plurality of proxy ML models also transfer representations to the primary ML model;training each of the proxy ML models with the trained primary ML model;for each proxy ML model, selecting a remote node for further training, based on at least a training history of the proxy ML model;for each proxy ML model, selecting a remote node for further training, based on at least an architecture of the proxy ML model;for each proxy ML model, selecting a remote node for further training, based on at least a dataset type at the remote node;for each proxy ML model, selecting a remote node for further training, based on at least a dataset size at the remote node;deploying the plurality of trained proxy ML models to the plurality of remote nodes for further training, wherein each proxy ML model is deployed to a different one of the plurality of remote nodes;deploying the plurality of trained proxy ML models for further training comprises deploying the plurality of trained proxy ML models to the selected remote nodes;deploying the plurality of trained proxy ML models for further training comprises deploying the plurality of trained proxy ML models to a random one of the remote nodes;receiving, at the primary node, from the plurality of remote nodes, the further-trained plurality of proxy ML models;further training the primary ML model using the plurality of proxy ML models;deploying the trained primary ML model to an operational node;performing an ML task with the trained primary ML model;the ML task comprises image classification;the ML task comprises object recognition;the ML task comprises speech recognition; andthe ML task comprises language classification.

Example Operating Environment

FIG.5is a block diagram of an example computing device500for implementing aspects disclosed herein, and is designated generally as computing device500. In some examples, one or more computing devices500are provided for an on-premises computing solution. In some examples, one or more computing devices500are provided as a cloud computing solution. In some examples, a combination of on-premises and cloud computing solutions are used. Computing device500is but one example of a suitable computing environment and is not intended to suggest any limitation as to the scope of use or functionality of the examples disclosed herein, whether used singly or as part of a larger set.

Neither should computing device500be interpreted as having any dependency or requirement relating to any one or combination of components/modules illustrated. The examples disclosed herein may be described in the general context of computer code or machine-useable instructions, including computer-executable instructions such as program components, being executed by a computer or other machine, such as a personal data assistant or other handheld device. Generally, program components including routines, programs, objects, components, data structures, and the like, refer to code that performs particular tasks, or implement particular abstract data types. The disclosed examples may be practiced in a variety of system configurations, including personal computers, laptops, smart phones, mobile tablets, hand-held devices, consumer electronics, specialty computing devices, etc. The disclosed examples may also be practiced in distributed computing environments when tasks are performed by remote-processing devices that are linked through a communications network.

Computing device500includes a bus510that directly or indirectly couples the following devices: computer storage memory512, one or more processors514, one or more presentation components516, input/output (I/O) ports518, I/O components520, a power supply522, and a network component524. While computing device500is depicted as a seemingly single device, multiple computing devices500may work together and share the depicted device resources. For example, memory512may be distributed across multiple devices, and processor(s)514may be housed with different devices.

Bus510represents what may be one or more busses (such as an address bus, data bus, or a combination thereof). Although the various blocks ofFIG.5are shown with lines for the sake of clarity, delineating various components may be accomplished with alternative representations. For example, a presentation component such as a display device is an I/O component in some examples, and some examples of processors have their own memory. Distinction is not made between such categories as “workstation,” “server,” “laptop,” “hand-held device,” etc., as all are contemplated within the scope ofFIG.5and the references herein to a “computing device.” Memory512may take the form of the computer storage media referenced below and operatively provide storage of computer-readable instructions, data structures, program modules and other data for the computing device500. In some examples, memory512stores one or more of an operating system, a universal application platform, or other program modules and program data. Memory512is thus able to store and access data512aand instructions512bthat are executable by processor514and configured to carry out the various operations disclosed herein.

In some examples, memory512includes computer storage media. Memory512may include any quantity of memory associated with or accessible by the computing device500. Memory512may be internal to the computing device500(as shown inFIG.5), external to the computing device500(not shown), or both (not shown). Additionally, or alternatively, the memory512may be distributed across multiple computing devices500, for example, in a virtualized environment in which instruction processing is carried out on multiple computing devices500. For the purposes of this disclosure, “computer storage media,” “computer-storage memory,” “memory,” and “memory devices” are synonymous terms for the computer-storage memory512, and none of these terms include carrier waves or propagating signaling.

Processor(s)514may include any quantity of processing units that read data from various entities, such as memory512or I/O components520. Specifically, processor(s)514are programmed to execute computer-executable instructions for implementing aspects of the disclosure. The instructions may be performed by the processor, by multiple processors within the computing device500, or by a processor external to the client computing device500. In some examples, the processor(s)514are programmed to execute instructions such as those illustrated in the flow charts discussed below and depicted in the accompanying drawings. Moreover, in some examples, the processor(s)514represent an implementation of analog techniques to perform the operations described herein. For example, the operations may be performed by an analog client computing device500and/or a digital client computing device500. Presentation component(s)516present data indications to a user or other device. Exemplary presentation components include a display device, speaker, printing component, vibrating component, etc. One skilled in the art will understand and appreciate that computer data may be presented in a number of ways, such as visually in a graphical user interface (GUI), audibly through speakers, wirelessly between computing devices500, across a wired connection, or in other ways. I/O ports518allow computing device500to be logically coupled to other devices including I/O components520, some of which may be built in. Example I/O components520include, for example but without limitation, a microphone, joystick, game pad, satellite dish, scanner, printer, wireless device, etc.

The computing device500may operate in a networked environment via the network component524using logical connections to one or more remote computers. In some examples, the network component524includes a network interface card and/or computer-executable instructions (e.g., a driver) for operating the network interface card. Communication between the computing device500and other devices may occur using any protocol or mechanism over any wired or wireless connection. In some examples, network component524is operable to communicate data over public, private, or hybrid (public and private) using a transfer protocol, between devices wirelessly using short range communication technologies (e.g., near-field communication (NFC), Bluetooth™ branded communications, or the like), or a combination thereof. Network component524communicates over wireless communication link526and/or a wired communication link526ato a remote resource528(e.g., a cloud resource) across network530. Various different examples of communication links526and526ainclude a wireless connection, a wired connection, and/or a dedicated link, and in some examples, at least a portion is routed through the internet.

The order of execution or performance of the operations in examples of the disclosure illustrated and described herein is not essential, and may be performed in different sequential manners in various examples. For example, it is contemplated that executing or performing a particular operation before, contemporaneously with, or after another operation is within the scope of aspects of the disclosure. When introducing elements of aspects of the disclosure or the examples thereof, the articles “a,” “an,” “the,” and “said” are intended to mean that there are one or more of the elements. The terms “comprising,” “including,” and “having” are intended to be inclusive and mean that there may be additional elements other than the listed elements. The term “exemplary” is intended to mean “an example of.” The phrase “one or more of the following: A, B, and C” means “at least one of A and/or at least one of B and/or at least one of C.”