Patent Description:
This present application relates generally to machine learning and more particularly to systems and methods for distributed training of deep learning models.

Deep learning is a branch of machine learning that provides state-of-the-art performance in many applications, including computer vision, speech processing, natural language processing, and audio recognition. Deep learning uses a computational model that resembles the architecture of neurons in brains. In particular, the computational model of deep learning, hereinafter referred to as the deep learning model, uses layers of "artificial neurons" to model the desired functionality of the computational model. Each of the artificial neurons are associated with one or more weights, which can be adjusted (e.g., trained) to provide a desired functionality when the artificial neurons are operated in the aggregate.

Weights in a deep learning model can be trained using training data. The training data can include an input data and a label associated with the input data. The weights in the deep learning model can be trained (or determined) in such a way that, when the deep learning model receives an input data, the deep learning model outputs a label corresponding to the input data. <CIT> describes that machine learning may be personalized to individual users of computing devices, and can be used to increase machine learning prediction accuracy and speed, and/or reduce memory footprint. Personalizing machine learning can include hosting, by a computing device, a consensus machine learning model and collecting information, locally by the computing device, associated with an application executed by the client device. Personalizing machine learning can also include modifying the consensus machine learning model accessible by the application based, at least in part on the information collected locally by the client device. Modifying the consensus machine learning model can generate a personalized machine learning model.

In the following description, numerous specific details are set forth regarding the systems and methods of the disclosed subject matter and the environment in which such systems and methods may operate, etc., in order to provide a thorough understanding of the disclosed subject matter. It will be apparent to one skilled in the art, however, that the disclosed subject matter may be practiced without such specific details, and that certain features, which are well known in the art, are not described in detail in order to avoid complication of the disclosed subject matter. In addition, it will be understood that the embodiments provided below are examples, and that it is contemplated that there are other systems and methods that are within the scope of the disclosed subject matter.

One of the challenges in deep learning is training weights in a deep learning model. Weights in a deep learning model are generally trained using a centralized deep learning training platform. A centralized deep learning training platform includes a host server and a plurality of local devices. Each of the local devices can gather input data and transmit the input data to the host server. The host server can aggregate the input data received from each of the local devices, create a training data set from the aggregated input data by associating desired labels with the aggregated input data, and train a deep learning model (e.g., the weights for the deep learning model) using the training data set.

Unfortunately, a centralized deep learning training platform is computationally expensive, especially when the training data set is large, because it is difficult to determine and associate correct labels for input data in the training data set. Such a massive labelling effort is realistic only for entities with a large amount of resources. Also, a centralized deep learning training platform requires a large amount of data communication bandwidth because the host server has to receive a large amount of input data from a large number of local devices. In addition, a centralized deep learning training platform is prone to privacy compromises because local devices have to share the raw input data, which may include private information, with the host server.

To address these issues with centralized training, the systems and methods disclosed herein provide a distributed training platform for training a deep learning model. An example distributed training platform includes a host server and a plurality of local devices. Each of the plurality of local devices is configured to receive input data. In contrast to the centralized approach, each of the example local devices in the distributed system is configured to locally label the input data to create a local training data set, and use the local training data set to train a local deep learning model. Because the training data set is created locally at the local devices, the host server does not need to determine and associate labels to a large amount of input data. Therefore, the host server is relieved from the computationally expensive process of creating a large training data set.

Once the local deep learning model is trained (e.g., weights of the local deep learning model are determined), each of the local devices can transmit the weights of its local deep learning model to a host server. This is in contrast to the centralized approach in which the local device sent the raw input data to the host server. Because the input data itself is not sent to the host server, the local device is not forced to share private information in the input data, thereby reducing the possibility of privacy compromises.

When the host server receives the weights from the local devices, the host server can aggregate the weights to determine the aggregated weights. The aggregated weights approximate the weights of a deep learning model that would have been obtained if the host server had trained the deep learning model using all training data sets used by the local devices. By leveraging the distributed computation of weights by the local devices, the host server can determine the approximated weights of a deep learning model, without actually training the deep learning model by itself. In some sense, the host server is crowd-sourcing the training of a deep learning model from the local devices.

In some examples, the disclosed distributed training platform is beneficial because (<NUM>) it allows the host server to learn a deep learning model that takes into account all available information (e.g., training data) and provide the globally-learned deep learning model to the local devices, and (<NUM>) it allows the local devices to adopt the globally-learned deep learning model and also adjust it to take into account any local variations. Therefore, the disclosed distributed training platform provides a deep learning paradigm for "training globally, adapting locally.

In some embodiments, the distributed deep learning training platform can be used to train deep learning models for auto white balance or other image processing systems for processing primary raw input images. In other embodiments, the distributed deep learning training platform can be used to train deep learning models for detecting a status of a movement (e.g., at rest, walking, or running) using an accelerometer input signal. In other embodiments, the distributed deep learning training platform can be used to train deep learning models for detecting audio commands or events. The disclosed deep learning training platform may, alternatively, be used for any task in which machine learning is desired.

<FIG> is a block diagram of an example distributed deep learning training platform <NUM> in accordance with some embodiments. The example platform <NUM> includes a plurality of example local devices <NUM> communicatively coupled with an example host server <NUM> via an example network <NUM>. According to the illustrated example, the host server <NUM> is coupled with an example operator terminal <NUM> to allow an operator to control operation of the host server <NUM>. For clarity, throughout this disclosure, reference is made to a single local device <NUM>, which may be representative of one or more of the plurality of local devices <NUM>.

The example local device <NUM> is a computing device that receives an input data, trains a local deep learning model, and transmits the local deep learning model (or characteristics thereof) to the host server. According to the illustrated example, the input data is received directly at the first local device without passing through the example host server <NUM>. As used herein, stating that input data is directly received at the local device <NUM> is defined to mean receiving data directly or indirectly from a data source, wherein the data does not pass through a host server (e.g., the host server <NUM>) that is performing deep learning training. In some examples, the input data may be received from a sensor (e.g., a measurement device, a data input, a data collector, a user interface that accepts user input, etc.) of the local device <NUM>, may be received from such a sensor communicatively coupled (e.g., coupled directly to the local device, coupled to the local device <NUM> via one or more intermediate devices (e.g., an intermediate device other than the host server <NUM>), etc.) to the local device <NUM>, etc..

The example device <NUM> is described in further detail in conjunction with <FIG>.

The example host server <NUM> aggregates the local deep learning model data received from the plurality of local devices <NUM> and distributes the aggregated results back to the plurality of local devices <NUM>. The example host server <NUM> includes an example weight aggregator <NUM>, an example weight distributor <NUM>, and an example global deep learning trainer <NUM>.

The example weight aggregator <NUM> receives the weights of local deep learning models from the plurality of local devices <NUM> and aggregates the weights to determine aggregated weights. The example weight aggregator <NUM> aggregates the weights by averaging corresponding weights received from the local devices <NUM>. For example, when the weight aggregator <NUM> receives a first set of weights (a_1, b_1, c_1) from a first local device <NUM> and a second set of weights (a_2, b_2, c_2) from a second local device <NUM>, the example weight aggregator <NUM> averages the corresponding weights from the local devices <NUM> to determine the aggregated weights: ((a_1 + a_2)/<NUM>, (b_1 + b_2)/<NUM>, (c_1 + c_2)/<NUM>). In some examples, the weight aggregator <NUM> can simplify the weight aggregation process by aggregating only the weights associated with deeper layers in the deep learning model.

Once the weight aggregator <NUM> determines the aggregated weights, the example weight aggregator <NUM> provides the aggregated weights to the example weight distributor <NUM>.

The example weight distributor <NUM> provides the aggregated weights to the local devices <NUM> so that the local devices <NUM> can update their own local deep learning models using the aggregated weights. The example weight distributor <NUM> includes a network interface (e.g., a wired network interface and/or a wireless network interface) for transmitting the aggregated weights to the plurality of local devices <NUM> via the example network <NUM>. Alternatively, the weight distributor <NUM> may communicate the aggregated weights to the local devices via a direct connection, via a removable storage device, etc..

The example host server <NUM> includes the example global deep learning trainer <NUM> to train a global deep learning model using training data. The global deep learning trainer <NUM> can be configured to train the global deep learning model using a variety of training techniques, including, for example, back propagation, contrastive divergence, alternative direction method of multipliers (ADMM), and/or tensor factorization. For example, the host server <NUM> may include the global deep learning trainer <NUM> when the local devices <NUM> will provide input data to the example host server <NUM>. Alternatively, in some examples, the host server <NUM> may not include the global deep learning trainer <NUM>.

In some examples, the weight aggregator <NUM> provides the aggregated weights to the global deep learning trainer <NUM> in addition to or as alternative to providing the aggregated weights to the weight distributor <NUM>. For example, the global deep learning trainer <NUM> may update the aggregated weights with any training data available at the host server <NUM>, and provide the updated aggregated weights to the weight distributor <NUM> for distribution to the local devices <NUM>.

In some examples, the host server <NUM> and the plurality of local devices <NUM> collaborate with one another to create a global deep learning model that takes into account all training data sets available to all local devices <NUM> and/or the host server <NUM>.

The example network <NUM> is a wide area network that communicatively couples the local devices <NUM> to the host server <NUM>. For example, the network <NUM> may be the internet. Alternatively, any other type of network may be utilized such as, for example, a local area network, a wireless network, a wired network, or any combination of network(s).

The example operator terminal <NUM> is a computing device providing a user interface in which a human operator can interaction with and control operation of the host server <NUM>. For example, the human operator may review the weight aggregation process, view an operation status, etc. An example user interface for the operator terminal <NUM> is described in conjunction with <FIG>.

In an example operation of the platform <NUM>, the local devices <NUM> receive input data (e.g., from sensors or other inputs coupled to the local devices <NUM>). To ensure privacy, limit bandwidth usage, etc. the local devices <NUM> do not transmit the input data to the host server <NUM> according to this example. The example local devices <NUM> train respective local deep learning models using the input data. The example local devices <NUM> transmit the weights and/or other details of the respective local deep learning models to the example host server <NUM>. The example weight aggregator <NUM> of the example host server <NUM> aggregates the weights to develop a global set of weights. The example weight distributor <NUM> distributes the aggregated weights back to the local devices <NUM> to update the respective local deep learning models with the globally aggregated weights. For example, the local devices <NUM> may then utilize the globally updated respective local deep learning models to classify test data (e.g., data that has not been classified or for which classification is desired).

<FIG> is a block diagram of an example implementation of the local device <NUM> of <FIG>. The example local device <NUM> includes an example data receiver <NUM>; example input samplers <NUM>, an example sample controller <NUM>, an example reference generator <NUM>; an example deep learner <NUM> which includes an example deep learning model <NUM>, example output samplers <NUM>, an example trainer <NUM>, and an example updater <NUM>.

The example data receiver <NUM> receives input data to be processed by the example local device <NUM>. For example, the data receiver <NUM> may be a sensor, a measurement device, a network interface, a user input device, a connection for a removable storage device, etc. For example, the input data may be received from an image sensor, an audio sensor, a data communication channel, a user interface, and/or any source that is capable of providing data to the local device <NUM>.

According to the illustrated example, the data receiver <NUM> provides the received input data to the example reference generator <NUM> and the example deep learner <NUM> via the example input samplers <NUM> that are controlled by the example sample controller <NUM>. For example, the input data can be sampled by the input samplers <NUM> to reduce the size of the input data and to simplify the training process.

The example sample controller <NUM> determines how the input data should be sampled. In some examples, the sample controller <NUM> is configured to select one or more random segments of the input data having a predetermined size and provide the random segment(s) to the example reference generator <NUM> and the example deep learner <NUM>. For example, the sample controller <NUM> may be implemented using a linear-feedback shift register (LFSR) that is configured to select a pseudo-random portion of the input data. The pseudo-random selection of the input data allows the example trainer <NUM> of the example deep learner <NUM> to use the appropriate distribution of samples for training the local deep learning model <NUM>. In some implementations, the LFSR can be implemented in hardware, such as a programmable hardware. In other implementations, the LFSR can be implemented as a software module including a set of computer instructions stored in a memory device.

For example, if the input data is an image(s), the sample controller <NUM> may randomly crop one or more portions of the input image and provide the cropped portion(s) to the reference generator <NUM> and the deep learner <NUM>. In other instances, when the input data is an image, the sample controller <NUM> may down-sample the input image and provide the down-sampled input image to the reference generator <NUM> and the deep learner <NUM>.

The example reference generator <NUM> processes the input data to determine a label associated with the input data (or the sampled input data). For example, the input data to be used for training may include an indication of a label, classification, result, etc. In some examples, the reference generator <NUM> may receive user input that identifies a label for input data (e.g., input data may be presented via a user interface and a user may select an appropriate label for the data). The reference generator <NUM> outputs the labelled data for comparison with the result of applying the input data to the example deep learning model <NUM>. The output of the reference generator <NUM> may be sampled by the example output sampler <NUM>.

The example local deep learning model <NUM> receives the input data (or sampled input data) and processes the input data to determine an output. For example, the local deep learning model <NUM> may operate using the same set of labels utilized by the reference generator <NUM>. The output of the deep learning model <NUM> may be sampled by the example output sampler <NUM>. The local deep learning model <NUM> may include, for example, an implementation of deep neural networks, convolutional deep neural networks, deep belief networks, recurrent neural networks, etc..

According to the illustrated example, before the trainer <NUM> analyses the (<NUM>) the label determined by the reference generator <NUM> and (<NUM>) the output of the deep learning model <NUM>, the label and the output of the deep learning model <NUM> are sampled by the output samplers <NUM>. For example, the output samplers <NUM> may be utilized when the amount of training to be performed is otherwise too onerous for, for example, an embedded platform in terms of computational intensity, power dissipation or both. For example, image and video input data may present computational complexity that may be reduced by sampling the outputs.

The example trainer <NUM> determines a difference between (<NUM>) the label determined by the reference generator <NUM> and (<NUM>) the output of the example deep learning model <NUM>. The example trainer uses the difference to train/adjust the example local deep learning model <NUM>. For example. The trainer <NUM> may train the local deep learning model <NUM> using a variety of training techniques, including, for example, back propagation, contrastive divergence, alternative direction method of multipliers (ADMM), and/or tensor factorization.

According to the illustrated example, the trainer <NUM> transmits the weights associated with the local deep learning model <NUM> to the host server <NUM>. Alternatively, the example trainer <NUM> may transmit the local deep learning model <NUM> and/or the input data to the example host server <NUM>. In some implementations, the trainer <NUM> transmits the weights when the local device <NUM> is requested to send the weights to the host server <NUM>. Alternatively, the trainer <NUM> may transmit the weights when the deep learner <NUM> has completed the training of the local deep learning model <NUM>.

In some examples, when the trainer <NUM> transmits the weights to the host server <NUM>, the trainer <NUM> can also send (<NUM>) the number of training intervals (e.g., iterations of training) performed by the deep learner <NUM> and/or (<NUM>) time-series data describing error convergence over time. In some cases, if there was any input data that was difficult to train on the local deep learning model <NUM>, the trainer <NUM> also transmits that input data, or one or more labels that were output by the local deep learning model <NUM>. For example, the trainer <NUM> may determine that an input data was challenging to train on the local deep learning model <NUM> when the local deep learning model <NUM> outputs two or more labels with similar confidence levels for the input data.

The example updater <NUM> receives the aggregated weights from the host server <NUM> and updates the weights of the local deep learning model <NUM> with the aggregated weights. For example, the updater <NUM> can replace the weights of the local deep learning model <NUM> with the aggregated weights. As another example, the updater <NUM> can replace the weights of the local deep learning model <NUM> with a weighted average of (<NUM>) the weights of the local deep learning model 214and (<NUM>) the aggregated weights received from the host server <NUM>.

In some examples, the reference generator <NUM> and the deep learner <NUM> process new input data as it becomes available. When the trainer <NUM> determines that training of the local deep learning model <NUM> is completed, the deep learner <NUM> can be configured to stop additional training. For example, the trainer <NUM> may stop additional training when determining that the accuracy of the deep learning model <NUM> has reached a threshold level, when the accuracy has substantially stopped increasing, etc. Alternatively, the trainer <NUM> may continue training as long as additional input data and labels from the reference generator <NUM> are presented.

While an example manner of implementing the local device <NUM> of <FIG> is illustrated in <FIG>, one or more of the elements, processes and/or devices illustrated in <FIG> may be combined, divided, re-arranged, omitted, eliminated and/or implemented in any other way. Further, the example data receiver <NUM>, the example input samplers <NUM>, the example sample controller <NUM>, the example reference generator <NUM>, the example trainer <NUM>, the example updater <NUM> (and/or, more generally, the example deep learner <NUM>), the example output samplers <NUM>, and/or, more generally, the example local device <NUM> of <FIG> may be implemented by hardware, software, firmware and/or any combination of hardware, software and/or firmware. Thus, for example, any of the example data receiver <NUM>, the example input samplers <NUM>, the example sample controller <NUM>, the example reference generator <NUM>, the example trainer <NUM>, the example updater <NUM> (and/or, more generally, the example deep learner <NUM>), the example output samplers <NUM>, and/or, more generally, the example local device <NUM> of <FIG> could be implemented by one or more analog or digital circuit(s), logic circuits, programmable processor(s), application specific integrated circuit(s) (ASIC(s)), programmable logic device(s) (PLD(s)) and/or field programmable logic device(s) (FPLD(s)). When reading any of the apparatus or system claims of this patent to cover a purely software and/or firmware implementation, at least one of the example data receiver <NUM>, the example input samplers <NUM>, the example sample controller <NUM>, the example reference generator <NUM>, the example trainer <NUM>, the example updater <NUM> (and/or, more generally, the example deep learner <NUM>), and/or the example output samplers <NUM> is/are hereby expressly defined to include a non-transitory computer readable storage device or storage disk such as a memory, a digital versatile disk (DVD), a compact disk (CD), a Blu-ray disk, etc. including the software and/or firmware. Further still, the example local device <NUM> of <FIG> may include one or more elements, processes and/or devices in addition to, or instead of, those illustrated in <FIG>, and/or may include more than one of any or all of the illustrated elements, processes and devices.

A flowchart representative of example machine readable instructions for implementing the local device <NUM> of <FIG> and/or <NUM> and/or the host server <NUM> of <FIG> is shown in <FIG>. In this example, the machine readable instructions comprise a program for execution by a processor such as the processor <NUM> and/or the processor <NUM> shown in the example processor platform <NUM> and/or the example processor platform <NUM> discussed below in connection with <FIG> and <FIG>. The program may be embodied in software stored on a non-transitory computer readable storage medium such as a CD-ROM, a floppy disk, a hard drive, a digital versatile disk (DVD), a Blu-ray disk, or a memory associated with the processor <NUM>, <NUM>, but the entire program and/or parts thereof could alternatively be executed by a device other than the processor <NUM>,<NUM> and/or embodied in firmware or dedicated hardware. Further, although the example program is described with reference to the flowchart illustrated in <FIG>, many other methods of implementing the example local device <NUM> and/or the host server <NUM> may alternatively be used. For example, the order of execution of the blocks may be changed, and/or some of the blocks described may be changed, eliminated, or combined. Additionally or alternatively, any or all of the blocks may be implemented by one or more hardware circuits (e.g., discrete and/or integrated analog and/or digital circuitry, a Field Programmable Gate Array (FPGA), an Application Specific Integrated circuit (ASIC), a comparator, an operational-amplifier (op-amp), a logic circuit, etc.) structured to perform the corresponding operation without executing software or firmware.

As mentioned above, the example processes of <FIG> may be implemented using coded instructions (e.g., computer and/or machine readable instructions) stored on a non-transitory computer and/or machine readable medium such as a hard disk drive, a flash memory, a read-only memory, a compact disk, a digital versatile disk, a cache, a random-access memory and/or any other storage device or storage disk in which information is stored for any duration (e.g., for extended time periods, permanently, for brief instances, for temporarily buffering, and/or for caching of the information). As used herein, the term non-transitory computer readable medium is expressly defined to include any type of computer readable storage device and/or storage disk and to exclude propagating signals and to exclude transmission media. Thus, whenever a claim lists anything following any form of "include" or "comprise" (e.g., comprises, includes, comprising, including, etc.), it is to be understood that additional elements, terms, etc. may be present without falling outside the scope of the corresponding claim. As used herein, when the phrase "at least" is used as the transition term in a preamble of a claim, it is open-ended in the same manner as the term "comprising" and "including" are open ended.

The program of <FIG> begins at block <NUM> when the example data receiver <NUM> of the example local device <NUM> receives input data. The input data can be received from a variety of data sources including, for example, an image sensor, an audio sensor, a data communication channel, a user interface, and/or any source that is capable of providing data to the first local device <NUM>.

The example reference generator <NUM> determines a label associated with the input data (block <NUM>). For example, the reference generator <NUM> may determines labels for the input data after sampling by the example input sampler <NUM> and the example sample controller <NUM>. In some examples, the reference generator <NUM> may determine labels by causing the input data to be operated on a reference system. The reference system can be a target system to be modelled by the deep learning model. In other words, the reference system can refer to the input-output relationship (e.g., a transfer function) to be learned by the global DLM. In some embodiments, the reference system can be implemented in hardware as an integrated chip; in other embodiments, the reference system can be implemented as a software module that includes a set of computer instruction executable by a processor.

The example trainer <NUM> trains the example local deep learning model <NUM> (block <NUM>). For example, according to the illustrated example of <FIG>, the trainer <NUM> trains the deep learning model <NUM> based on a difference between the label indicated by the example reference generator <NUM> and an output of the input data applied to the example deep learning model <NUM>.

The example trainer <NUM> of the local device <NUM> transmits the weights associated with the local deep learning model <NUM> to the host server <NUM> (block <NUM>). For example, trainer <NUM> may transfer the weights in response to a request from the host server <NUM>, may transfer the weights once the trainer <NUM> determines that the training of the local deep learning model <NUM> is completed, etc..

According to the illustrated example, the process of blocks <NUM>-<NUM> is carried out by multiple local devices <NUM> in parallel.

When the example weight aggregator <NUM> of the example host server <NUM> receives the weights from the local devices <NUM>, the example weight aggregator <NUM> aggregates the received weights from the local devices <NUM> (block <NUM>). [<NUM>]According to the illustrated example, the weight aggregator <NUM> computes an average of the weights to determine the aggregated weights. In other examples, the weight aggregator <NUM> aggregates the weights by creating standard deviations for individual weights and filtering out outliers.

The example weight distributor <NUM> transmits the aggregated weights to the local devices <NUM> (block <NUM>).

In some examples, in the event that there are multiple distributions for individual weights, outliers from these individual distributions can be filtered out by the weight aggregator <NUM> and separate derivative networks with differing weights, one for each of the filtered distributions, can be generated and the weight distributor <NUM> may transmit the respective weights back to the relevant sub-groups of the local devices <NUM>.

When the updater <NUM> of the local device <NUM> receives the aggregated weights from the host server <NUM>, the example updater <NUM> updates the weights of the local deep learning model <NUM> using the aggregated weights to take into account all of the local training data sets created by the plurality of local devices <NUM> (block <NUM>). According to the illustrated example, updater <NUM> replaces the weights of the local deep learning model <NUM> with the aggregated weights so that the local devices <NUM> have access to the global deep learning model. In other examples, the updater <NUM> updates the weights of the local deep learning model <NUM> with a weighted average of (<NUM>) the weights of the local deep learning model <NUM> and (<NUM>) the aggregated weights received from the example host server <NUM>.

The process of <FIG> then terminates. Alternatively, the process of <FIG> may restart at block <NUM>, may restart at block <NUM> when new input data is received, etc..

<FIG> is a block diagram of another implementation of the local device <NUM>. The implementation of the local device <NUM> of <FIG> is similar to the implementation of the local device <NUM> of <FIG>, except that the local device <NUM> of <FIG> also includes a global deep learning model <NUM>. The global deep learning model <NUM> is a deep learning model trained at the host server <NUM>. When the updater <NUM> of the local device <NUM> of <FIG> receives the aggregated weights from the host server <NUM>, the example updater <NUM> replaces the weights in the global deep learning model <NUM> using the received aggregated weights.

Because the global deep learning model <NUM> of <FIG> is trained using all information available to all local devices, the global deep learning model <NUM> may not be tailored to address the characteristics of local input data available to a particular local device <NUM>. To address this issue, the trainer <NUM> trains and maintains the local deep learning model <NUM> that is configured to augment the global deep learning model <NUM>. In particular, the local deep learning model <NUM> can be configured to capture the characteristics of local input data available to a particular local device <NUM> so that the global deep learning model <NUM> and the local deep learning model <NUM> can together capture both the global characteristics and the local variations of the training data.

The operation of the system in <FIG> is substantially similar to the operation of the system in <FIG> except input data is also provided to the global deep learning model <NUM> (e.g., via an input sampler <NUM>). The example trainer <NUM> of the illustrated example of <FIG> determines a difference between (<NUM>) the label determined by the reference generator <NUM> and (<NUM>) a summation of the output of the local deep learning model <NUM> and an output of the global deep learning model <NUM>. The example trainer <NUM> uses the difference to train the local deep learning model <NUM>.

<FIG> is a block diagram of another implementation of the local device <NUM>. The implementation of the local device <NUM> of <FIG> is similar to the implementation of the local device <NUM> of <FIG>, except that the local device <NUM> of <FIG> also applies different weights in training the local deep learning model <NUM>. For example, the outputs of the local deep learning model <NUM> and the global deep learning model <NUM> may be given different weights than the output of the reference generator <NUM> to control the influence of the models and the reference data on the training. Additionally or alternatively, different weights may be applied to each of the output of the local deep learning model <NUM> and the output of the global deep learning model <NUM>.

<FIG> illustrates an example user interface <NUM> that may be presented by the example operator terminal <NUM>. The example user interface <NUM> presents a visualization of deep learning models down to the level of individual weights. The example user interface <NUM> may present a visualization of particular local devices <NUM> and clusters of local devices <NUM> and the active deep learning models operating there via a secure connection with the local devices <NUM>. The secure connection may only allow bi-directional transmission of weights without other information (e.g., input data) via a simple Application Programming Interface (API) with example commands <NUM> listed in <FIG>. The secure connection can enable secure authenticated communication with one of the local devices <NUM> or cluster of the local devices <NUM>, and can transmit or receive the state of the local device <NUM>, the local deep learning models <NUM> to be uploaded from the local device <NUM>, and the global deep learning model(s). The secure connection can be closed down using one or more commands. The user interface <NUM> allows states of the local devices <NUM> as well as the precise geographical position for an authorized human operator to be visualized.

The user interface <NUM> and/or the operator terminal <NUM> allow a human operator of the host server <NUM> to select a weight aggregation strategy to be employed by the host server <NUM>. For example, the user interface <NUM> may include a menu system (as shown in <FIG>), external file, command-line option, and/or any other mechanisms. An operator can also specify the generation of derivative networks to be used to specialize across distributed local deep learning models <NUM> in the case where weights from the local devices <NUM> are found to have multiple distributions or to force aggregation of weights by averaging or other mathematical means where the operator judges that this is a reasonable trade off.

In some examples, the user interface <NUM> presents visualization of labelled data in the event it is shared with the host server <NUM> by local devices <NUM>. For example, if the local deep learning model <NUM> of produces classification accuracies below a user-defined threshold in the case of the top N most likely classifications produced by the network, or for instance if the difference or standard deviation in the top N classifications are below a user-defined threshold, if an operator has opted-in for the sharing of labelled data, the local device <NUM> can upload labelled data to the host server <NUM> for visualization with the user interface <NUM>.

<FIG> is a block diagram of an example processor platform <NUM> capable of executing the instructions of <FIG> to implement the host server <NUM> of <FIG>. The processor platform <NUM> can be, for example, a server, a personal computer, a mobile device (e.g., a cell phone, a smart phone, a tablet such as an iPad™), a personal digital assistant (PDA), an Internet appliance, a DVD player, a CD player, a digital video recorder, a Blu-ray player, a gaming console, a personal video recorder, a set top box, or any other type of computing device.

The hardware processor may be a semiconductor based (e.g., silicon based) device. In this example, the processor <NUM> implements the example weight aggregator <NUM>, the example weight distributor <NUM>, and the example global trainer <NUM>.

The output devices <NUM> can be implemented, for example, by display devices (e.g., a light emitting diode (LED), an organic light emitting diode (OLED), a liquid crystal display, a cathode ray tube display (CRT), a touchscreen, a tactile output device, a printer and/or speakers).

The coded instructions <NUM> of <FIG> may be stored in the mass storage device <NUM>, in the volatile memory <NUM>, in the non-volatile memory <NUM>, and/or on a removable tangible computer readable storage medium such as a CD or DVD.

<FIG> is a block diagram of an example processor platform <NUM> capable of executing the instructions of <FIG> to implement the local device <NUM> of <FIG>, <FIG>, <FIG>, and/or <NUM>. The processor platform <NUM> can be, for example, a server, a personal computer, a mobile device (e.g., a cell phone, a smart phone, a tablet such as an iPad™), a personal digital assistant (PDA), an Internet appliance, a DVD player, a CD player, a digital video recorder, a Blu-ray player, a gaming console, a personal video recorder, a set top box, or any other type of computing device.

The hardware processor may be a semiconductor based (e.g., silicon based) device. In this example, the processor <NUM> implements the example data receiver <NUM>, the example input samplers <NUM>, the example sample controller <NUM>, the example reference generator <NUM>, the example output samplers <NUM>, the example trainer <NUM>, and the example updater <NUM>.

From the foregoing, it will be appreciated that example methods, apparatus and articles of manufacture have been disclosed that enable the training of a deep learning model by a plurality of local devices. Utilizing multiple local devices facilitates the distributed processing among a plurality of devices. In addition, input data received at each local device may processed at the respective local device to avoid the bandwidth cost of transferring the input data to a central server for processing. In addition, privacy of the locally received input data may be maintained to processing at the local devices instead of transferring to a central server.

It is noted that this patent claims priority from <CIT>.

Example methods, apparatus, systems and articles of manufacture to detect anomalies in electronic data are disclosed herein.

Claim 1:
A method to train deep learning models, the method comprising:
labelling (<NUM>), by a reference generator (<NUM>), by executing an instruction with at least one processor at a local device (<NUM>), input data received at the local device (<NUM>) to generate training data;
training (<NUM>), by executing an instruction with the at least one processor, a local deep learning model; characterized in
transmitting (<NUM>) the local deep learning model to a server (<NUM>), the server (<NUM>) to receive a plurality of local deep learning models from a plurality of local devices (<NUM>), the server (<NUM>) to determine a set of weights for a global deep learning model by averaging corresponding weights received from the plurality of local devices, wherein the set of weights are aggregated weights based on the plurality of local deep learning models from the plurality of local devices (<NUM>); and
updating (<NUM>), by an updater (<NUM>), by executing an instruction with the at least one processor at the local device (<NUM>), the local deep learning model by determining a weighted average of the averaged set of weights received from the server (<NUM>) and weights of the local deep learning model;
wherein the local device comprises a copy of the global deep learning model trained by the server;
wherein training the local deep learning model includes:
determining a first output of the copy of the global deep learning model for the input data;
determining a second output of the local deep learning model for the input data;
determining a first difference between the first output and the second output;
determining a second difference between the second output and the label determined by the labelling for the input data; and
training, by a trainer (<NUM>), the local deep learning model based on a third difference between the first difference and the second difference.