Patent ID: 12190247

DETAILED DESCRIPTION

In the following description, numerous specific details are set forth regarding the systems and methods of the disclosed subject matter and the environment in which such systems and methods may operate, etc., in order to provide a thorough understanding of the disclosed subject matter. It will be apparent to one skilled in the art, however, that the disclosed subject matter may be practiced without such specific details, and that certain features, which are well known in the art, are not described in detail in order to avoid complication of the disclosed subject matter. In addition, it will be understood that the embodiments provided below are examples, and that it is contemplated that there are other systems and methods that are within the scope of the disclosed subject matter.

One of the challenges in deep learning is training weights in a deep learning model. Weights in a deep learning model are generally trained using a centralized deep learning training platform. A centralized deep learning training platform includes a host server and a plurality of local devices. Each of the local devices can gather input data and transmit the input data to the host server. The host server can aggregate the input data received from each of the local devices, create a training data set from the aggregated input data by associating desired labels with the aggregated input data, and train a deep learning model (e.g., the weights for the deep learning model) using the training data set.

Unfortunately, a centralized deep learning training platform is computationally expensive, especially when the training data set is large, because it is difficult to determine and associate correct labels for input data in the training data set. Such a massive labelling effort is realistic only for entities with a large amount of resources. Also, a centralized deep learning training platform requires a large amount of data communication bandwidth because the host server has to receive a large amount of input data from a large number of local devices. In addition, a centralized deep learning training platform is prone to privacy compromises because local devices have to share the raw input data, which may include private information, with the host server.

To address these issues with centralized training, the systems and methods disclosed herein provide a distributed training platform for training a deep learning model. An example distributed training platform includes a host server and a plurality of local devices. Each of the plurality of local devices is configured to receive input data. In contrast to the centralized approach, each of the example local devices in the distributed system is configured to locally label the input data to create a local training data set, and use the local training data set to train a local deep learning model. Because the training data set is created locally at the local devices, the host server does not need to determine and associate labels to a large amount of input data. Therefore, the host server is relieved from the computationally expensive process of creating a large training data set.

Once the local deep learning model is trained (e.g., weights of the local deep learning model are determined), each of the local devices can transmit the weights of its local deep learning model to a host server. This is in contrast to the centralized approach in which the local device sent the raw input data to the host server. Because the input data itself is not sent to the host server, the local device is not forced to share private information in the input data, thereby reducing the possibility of privacy compromises.

When the host server receives the weights from the local devices, the host server can aggregate the weights to determine the aggregated weights. The aggregated weights approximate the weights of a deep learning model that would have been obtained if the host server had trained the deep learning model using all training data sets used by the local devices. By leveraging the distributed computation of weights by the local devices, the host server can determine the approximated weights of a deep learning model, without actually training the deep learning model by itself. In some sense, the host server is crowd-sourcing the training of a deep learning model from the local devices.

In some examples, the disclosed distributed training platform is beneficial because (1) it allows the host server to learn a deep learning model that takes into account all available information (e.g., training data) and provide the globally-learned deep learning model to the local devices, and (2) it allows the local devices to adopt the globally-learned deep learning model and also adjust it to take into account any local variations. Therefore, the disclosed distributed training platform provides a deep learning paradigm for “training globally, adapting locally.”

In some embodiments, the distributed deep learning training platform can be used to train deep learning models for auto white balance or other image processing systems for processing primary raw input images. In other embodiments, the distributed deep learning training platform can be used to train deep learning models for detecting a status of a movement (e.g., at rest, walking, or running) using an accelerometer input signal. In other embodiments, the distributed deep learning training platform can be used to train deep learning models for detecting audio commands or events. The disclosed deep learning training platform may, alternatively, be used for any task in which machine learning is desired.

FIG.1is a block diagram of an example distributed deep learning training platform100in accordance with some embodiments. The example platform100includes a plurality of example local devices102communicatively coupled with an example host server104via an example network106. According to the illustrated example, the host server104is coupled with an example operator terminal108to allow an operator to control operation of the host server104. For clarity, throughout this disclosure, reference is made to a single local device102, which may be representative of one or more of the plurality of local devices102.

The example local device102is a computing device that receives an input data, trains a local deep learning model, and transmits the local deep learning model (or characteristics thereof) to the host server. According to the illustrated example, the input data is received directly at the first local device without passing through the example host server104. As used herein, stating that input data is directly received at the local device102is defined to mean receiving data directly or indirectly from a data source, wherein the data does not pass through a host server (e.g., the host server104) that is performing deep learning training. In some examples, the input data may be received from a sensor (e.g., a measurement device, a data input, a data collector, a user interface that accepts user input, etc.) of the local device102, may be received from such a sensor communicatively coupled (e.g., coupled directly to the local device, coupled to the local device102via one or more intermediate devices (e.g., an intermediate device other than the host server104), etc.) to the local device102, etc.

The example device102is described in further detail in conjunction withFIG.2.

The example host server104aggregates the local deep learning model data received from the plurality of local devices102and distributes the aggregated results back to the plurality of local devices102. The example host server104includes an example weight aggregator110, an example weight distributor112, and an example global deep learning trainer114.

The example weight aggregator110receives the weights of local deep learning models from the plurality of local devices102and aggregates the weights to determine aggregated weights. The example weight aggregator110aggregates the weights by averaging corresponding weights received from the local devices102. For example, when the weight aggregator110receives a first set of weights (a_1, b_1, c_1) from a first local device102and a second set of weights (a_2, b_2, c_2) from a second local device102, the example weight aggregator110averages the corresponding weights from the local devices102to determine the aggregated weights: ((a_1+a_2)/2, (b_1+b_2)/2, (c_1+c_2)/2). In some examples, the weight aggregator110can simplify the weight aggregation process by aggregating only the weights associated with deeper layers in the deep learning model.

Once the weight aggregator110determines the aggregated weights, the example weight aggregator110provides the aggregated weights to the example weight distributor112.

The example weight distributor112provides the aggregated weights to the local devices102so that the local devices102can update their own local deep learning models using the aggregated weights. The example weight distributor112includes a network interface (e.g., a wired network interface and/or a wireless network interface) for transmitting the aggregated weights to the plurality of local devices102via the example network106. Alternatively, the weight distributor112may communicate the aggregated weights to the local devices via a direct connection, via a removable storage device, etc.

The example host server104includes the example global deep learning trainer114to train a global deep learning model using training data. The global deep learning trainer114can be configured to train the global deep learning model using a variety of training techniques, including, for example, back propagation, contrastive divergence, alternative direction method of multipliers (ADMM), and/or tensor factorization. For example, the host server104may include the global deep learning trainer114when the local devices102will provide input data to the example host server104. Alternatively, in some examples, the host server104may not include the global deep learning trainer114.

In some examples, the weight aggregator110provides the aggregated weights to the global deep learning trainer114in addition to or as alternative to providing the aggregated weights to the weight distributor112. For example, the global deep learning trainer114may update the aggregated weights with any training data available at the host server104, and provide the updated aggregated weights to the weight distributor112for distribution to the local devices102.

In some examples, the host server104and the plurality of local devices102collaborate with one another to create a global deep learning model that takes into account all training data sets available to all local devices102and/or the host server104.

The example network106is a wide area network that communicatively couples the local devices102to the host server104. For example, the network106may be the internet. Alternatively, any other type of network may be utilized such as, for example, a local area network, a wireless network, a wired network, or any combination of network(s).

The example operator terminal108is a computing device providing a user interface in which a human operator can interaction with and control operation of the host server104. For example, the human operator may review the weight aggregation process, view an operation status, etc. An example user interface for the operator terminal108is described in conjunction withFIG.6.

In an example operation of the platform100, the local devices102receive input data (e.g., from sensors or other inputs coupled to the local devices102). To ensure privacy, limit bandwidth usage, etc. the local devices102do not transmit the input data to the host server104according to this example. The example local devices102train respective local deep learning models using the input data. The example local devices102transmit the weights and/or other details of the respective local deep learning models to the example host server104. The example weight aggregator110of the example host server104aggregates the weights to develop a global set of weights. The example weight distributor112distributes the aggregated weights back to the local devices102to update the respective local deep learning models with the globally aggregated weights. For example, the local devices102may then utilize the globally updated respective local deep learning models to classify test data (e.g., data that has not been classified or for which classification is desired).

FIG.2is a block diagram of an example implementation of the local device102ofFIG.1. The example local device102includes an example data receiver202; example input samplers204, an example sample controller206, an example reference generator210; an example deep learner212which includes an example deep learning model214, example output samplers215, an example trainer216, and an example updater218.

The example data receiver202receives input data to be processed by the example local device102. For example, the data receiver202may be a sensor, a measurement device, a network interface, a user input device, a connection for a removable storage device, etc. For example, the input data may be received from an image sensor, an audio sensor, a data communication channel, a user interface, and/or any source that is capable of providing data to the local device102.

According to the illustrated example, the data receiver202provides the received input data to the example reference generator210and the example deep learner212via the example input samplers204that are controlled by the example sample controller206. For example, the input data can be sampled by the input samplers204to reduce the size of the input data and to simplify the training process.

The example sample controller206determines how the input data should be sampled. In some examples, the sample controller206is configured to select one or more random segments of the input data having a predetermined size and provide the random segment(s) to the example reference generator210and the example deep learner212. For example, the sample controller206may be implemented using a linear-feedback shift register (LFSR) that is configured to select a pseudo-random portion of the input data. The pseudo-random selection of the input data allows the example trainer216of the example deep learner212to use the appropriate distribution of samples for training the local deep learning model214. In some implementations, the LFSR can be implemented in hardware, such as a programmable hardware. In other implementations, the LFSR can be implemented as a software module including a set of computer instructions stored in a memory device.

For example, if the input data is an image(s), the sample controller206may randomly crop one or more portions of the input image and provide the cropped portion(s) to the reference generator210and the deep learner212. In other instances, when the input data is an image, the sample controller206may down-sample the input image and provide the down-sampled input image to the reference generator210and the deep learner212.

The example reference generator210processes the input data to determine a label associated with the input data (or the sampled input data). For example, the input data to be used for training may include an indication of a label, classification, result, etc. In some examples, the reference generator210may receive user input that identifies a label for input data (e.g., input data may be presented via a user interface and a user may select an appropriate label for the data). The reference generator210outputs the labelled data for comparison with the result of applying the input data to the example deep learning model214. The output of the reference generator210may be sampled by the example output sampler215.

The example local deep learning model214receives the input data (or sampled input data) and processes the input data to determine an output. For example, the local deep learning model214may operate using the same set of labels utilized by the reference generator210. The output of the deep learning model214may be sampled by the example output sampler215. The local deep learning model214may include, for example, an implementation of deep neural networks, convolutional deep neural networks, deep belief networks, recurrent neural networks, etc.

According to the illustrated example, before the trainer216analyses the (1) the label determined by the reference generator210and (2) the output of the deep learning model214, the label and the output of the deep learning model214are sampled by the output samplers215. For example, the output samplers215may be utilized when the amount of training to be performed is otherwise too onerous for, for example, an embedded platform in terms of computational intensity, power dissipation or both. For example, image and video input data may present computational complexity that may be reduced by sampling the outputs.

The example trainer216determines a difference between (1) the label determined by the reference generator210and (2) the output of the example deep learning model214. The example trainer uses the difference to train/adjust the example local deep learning model214. For example. The trainer216may train the local deep learning model214using a variety of training techniques, including, for example, back propagation, contrastive divergence, alternative direction method of multipliers (ADMM), and/or tensor factorization.

According to the illustrated example, the trainer216transmits the weights associated with the local deep learning model214to the host server104. Alternatively, the example trainer216may transmit the local deep learning model214and/or the input data to the example host server104. In some implementations, the trainer216transmits the weights when the local device102is requested to send the weights to the host server104. Alternatively, the trainer216may transmit the weights when the deep learner212has completed the training of the local deep learning model214.

In some examples, when the trainer216transmits the weights to the host server104, the trainer216can also send (1) the number of training intervals (e.g., iterations of training) performed by the deep learner212and/or (2) time-series data describing error convergence over time. In some cases, if there was any input data that was difficult to train on the local deep learning model214, the trainer216also transmits that input data, or one or more labels that were output by the local deep learning model214. For example, the trainer216may determine that an input data was challenging to train on the local deep learning model214when the local deep learning model214outputs two or more labels with similar confidence levels for the input data.

The example updater218receives the aggregated weights from the host server104and updates the weights of the local deep learning model214with the aggregated weights. For example, the updater218can replace the weights of the local deep learning model214with the aggregated weights. As another example, the updater218can replace the weights of the local deep learning model214with a weighted average of (1) the weights of the local deep learning model214and (2) the aggregated weights received from the host server104.

In some examples, the reference generator210and the deep learner212process new input data as it becomes available. When the trainer216determines that training of the local deep learning model214is completed, the deep learner212can be configured to stop additional training. For example, the trainer216may stop additional training when determining that the accuracy of the deep learning model214has reached a threshold level, when the accuracy has substantially stopped increasing, etc. Alternatively, the trainer216may continue training as long as additional input data and labels from the reference generator210are presented.

While an example manner of implementing the local device102ofFIG.1is illustrated inFIG.2, one or more of the elements, processes and/or devices illustrated inFIG.4may be combined, divided, re-arranged, omitted, eliminated and/or implemented in any other way. Further, the example data receiver202, the example input samplers204, the example sample controller206, the example reference generator210, the example trainer216, the example updater218(and/or, more generally, the example deep learner212), the example output samplers215, and/or, more generally, the example local device102ofFIG.1may be implemented by hardware, software, firmware and/or any combination of hardware, software and/or firmware. Thus, for example, any of the example data receiver202, the example input samplers204, the example sample controller206, the example reference generator210, the example trainer216, the example updater218(and/or, more generally, the example deep learner212), the example output samplers215, and/or, more generally, the example local device102ofFIG.1could be implemented by one or more analog or digital circuit(s), logic circuits, programmable processor(s), application specific integrated circuit(s) (ASIC(s)), programmable logic device(s) (PLD(s)) and/or field programmable logic device(s) (FPLD(s)). When reading any of the apparatus or system claims of this patent to cover a purely software and/or firmware implementation, at least one of the example data receiver202, the example input samplers204, the example sample controller206, the example reference generator210, the example trainer216, the example updater218(and/or, more generally, the example deep learner212), and/or the example output samplers215is/are hereby expressly defined to include a non-transitory computer readable storage device or storage disk such as a memory, a digital versatile disk (DVD), a compact disk (CD), a Blu-ray disk, etc. including the software and/or firmware. Further still, the example local device102ofFIG.1may include one or more elements, processes and/or devices in addition to, or instead of, those illustrated inFIG.2, and/or may include more than one of any or all of the illustrated elements, processes and devices.

A flowchart representative of example machine readable instructions for implementing the local device102ofFIGS.1and/or2and/or the host server104ofFIG.1is shown inFIG.3. In this example, the machine readable instructions comprise a program for execution by a processor such as the processor812and/or the processor912shown in the example processor platform800and/or the example processor platform900discussed below in connection withFIGS.8and9. The program may be embodied in software stored on a non-transitory computer readable storage medium such as a CD-ROM, a floppy disk, a hard drive, a digital versatile disk (DVD), a Blu-ray disk, or a memory associated with the processor812,912, but the entire program and/or parts thereof could alternatively be executed by a device other than the processor812,912and/or embodied in firmware or dedicated hardware. Further, although the example program is described with reference to the flowchart illustrated inFIG.3, many other methods of implementing the example local device102and/or the host server104may alternatively be used. For example, the order of execution of the blocks may be changed, and/or some of the blocks described may be changed, eliminated, or combined. Additionally or alternatively, any or all of the blocks may be implemented by one or more hardware circuits (e.g., discrete and/or integrated analog and/or digital circuitry, a Field Programmable Gate Array (FPGA), an Application Specific Integrated circuit (ASIC), a comparator, an operational-amplifier (op-amp), a logic circuit, etc.) structured to perform the corresponding operation without executing software or firmware.

As mentioned above, the example processes ofFIG.3may be implemented using coded instructions (e.g., computer and/or machine readable instructions) stored on a non-transitory computer and/or machine readable medium such as a hard disk drive, a flash memory, a read-only memory, a compact disk, a digital versatile disk, a cache, a random-access memory and/or any other storage device or storage disk in which information is stored for any duration (e.g., for extended time periods, permanently, for brief instances, for temporarily buffering, and/or for caching of the information). As used herein, the term non-transitory computer readable medium is expressly defined to include any type of computer readable storage device and/or storage disk and to exclude propagating signals and to exclude transmission media. “Including” and “comprising” (and all forms and tenses thereof) are used herein to be open ended terms. Thus, whenever a claim lists anything following any form of “include” or “comprise” (e.g., comprises, includes, comprising, including, etc.), it is to be understood that additional elements, terms, etc. may be present without falling outside the scope of the corresponding claim. As used herein, when the phrase “at least” is used as the transition term in a preamble of a claim, it is open-ended in the same manner as the term “comprising” and “including” are open ended.

The program ofFIG.3begins at block302when the example data receiver202of the example local device102receives input data. The input data can be received from a variety of data sources including, for example, an image sensor, an audio sensor, a data communication channel, a user interface, and/or any source that is capable of providing data to the first local device102.

The example reference generator210determines a label associated with the input data (block304). For example, the reference generator210may determines labels for the input data after sampling by the example input sampler304and the example sample controller206. In some examples, the reference generator210may determine labels by causing the input data to be operated on a reference system. The reference system can be a target system to be modelled by the deep learning model. In other words, the reference system can refer to the input-output relationship (e.g., a transfer function) to be learned by the global DLM. In some embodiments, the reference system can be implemented in hardware as an integrated chip; in other embodiments, the reference system can be implemented as a software module that includes a set of computer instruction executable by a processor.

The example trainer216trains the example local deep learning model214(block306). For example, according to the illustrated example ofFIG.2, the trainer216trains the deep learning model214based on a difference between the label indicated by the example reference generator210and an output of the input data applied to the example deep learning model214.

The example trainer216of the local device102transmits the weights associated with the local deep learning model214to the host server104(block308). For example, trainer216may transfer the weights in response to a request from the host server104, may transfer the weights once the trainer216determines that the training of the local deep learning model214is completed, etc.

According to the illustrated example, the process of blocks302-308is carried out by multiple local devices102in parallel.

When the example weight aggregator110of the example host server104receives the weights from the local devices102, the example weight aggregator110aggregates the received weights from the local devices102(block310). According to the illustrated example, the weight aggregator110computes an average of the weights to determine the aggregated weights. In other examples, the weight aggregator110aggregates the weights by creating standard deviations for individual weights and filtering out outliers.

The example weight distributor112transmits the aggregated weights to the local devices102(block312).

In some examples, in the event that there are multiple distributions for individual weights, outliers from these individual distributions can be filtered out by the weight aggregator110and separate derivative networks with differing weights, one for each of the filtered distributions, can be generated and the weight distributor112may transmit the respective weights back to the relevant sub-groups of the local devices102.

When the updater218of the local device102receives the aggregated weights from the host server104, the example updater218updates the weights of the local deep learning model214using the aggregated weights to take into account all of the local training data sets created by the plurality of local devices102(block314). According to the illustrated example, updater218replaces the weights of the local deep learning model214with the aggregated weights so that the local devices102have access to the global deep learning model. In other examples, the updater218updates the weights of the local deep learning model214with a weighted average of (1) the weights of the local deep learning model214and (2) the aggregated weights received from the example host server104.

The process ofFIG.3then terminates. Alternatively, the process ofFIG.3may restart at block302, may restart at block302when new input data is received, etc.

FIG.4is a block diagram of another implementation of the local device102. The implementation of the local device102ofFIG.4is similar to the implementation of the local device102ofFIG.2, except that the local device102ofFIG.4also includes a global deep learning model402. The global deep learning model402is a deep learning model trained at the host server104. When the updater218of the local device102ofFIG.4receives the aggregated weights from the host server104, the example updater218replaces the weights in the global deep learning model402using the received aggregated weights.

Because the global deep learning model402ofFIG.4is trained using all information available to all local devices, the global deep learning model402may not be tailored to address the characteristics of local input data available to a particular local device102. To address this issue, the trainer216trains and maintains the local deep learning model214that is configured to augment the global deep learning model402. In particular, the local deep learning model214can be configured to capture the characteristics of local input data available to a particular local device102so that the global deep learning model402and the local deep learning model214can together capture both the global characteristics and the local variations of the training data.

The operation of the system inFIG.4is substantially similar to the operation of the system inFIG.2except input data is also provided to the global deep learning model402(e.g., via an input sampler204). The example trainer216of the illustrated example ofFIG.4determines a difference between (1) the label determined by the reference generator210and (2) a summation of the output of the local deep learning model214and an output of the global deep learning model402. The example trainer216uses the difference to train the local deep learning model214.

FIG.5is a block diagram of another implementation of the local device102. The implementation of the local device102ofFIG.5is similar to the implementation of the local device102ofFIG.3, except that the local device102ofFIG.5also applies different weights in training the local deep learning model214. For example, the outputs of the local deep learning model214and the global deep learning model402may be given different weights than the output of the reference generator210to control the influence of the models and the reference data on the training. Additionally or alternatively, different weights may be applied to each of the output of the local deep learning model214and the output of the global deep learning model402.

FIG.6illustrates an example user interface600that may be presented by the example operator terminal108. The example user interface600presents a visualization of deep learning models down to the level of individual weights. The example user interface600may present a visualization of particular local devices102and clusters of local devices102and the active deep learning models operating there via a secure connection with the local devices102. The secure connection may only allow bi-directional transmission of weights without other information (e.g., input data) via a simple Application Programming Interface (API) with example commands700listed inFIG.7. The secure connection can enable secure authenticated communication with one of the local devices102or cluster of the local devices102, and can transmit or receive the state of the local device102, the local deep learning models214to be uploaded from the local device102, and the global deep learning model(s). The secure connection can be closed down using one or more commands. The user interface600allows states of the local devices102as well as the precise geographical position for an authorized human operator to be visualized.

The user interface600and/or the operator terminal108allow a human operator of the host server104to select a weight aggregation strategy to be employed by the host server104. For example, the user interface600may include a menu system (as shown inFIG.6), external file, command-line option, and/or any other mechanisms. An operator can also specify the generation of derivative networks to be used to specialize across distributed local deep learning models214in the case where weights from the local devices102are found to have multiple distributions or to force aggregation of weights by averaging or other mathematical means where the operator judges that this is a reasonable trade off.

In some examples, the user interface600presents visualization of labelled data in the event it is shared with the host server104by local devices102. For example, if the local deep learning model214of produces classification accuracies below a user-defined threshold in the case of the top N most likely classifications produced by the network, or for instance if the difference or standard deviation in the top N classifications are below a user-defined threshold, if an operator has opted-in for the sharing of labelled data, the local device102can upload labelled data to the host server104for visualization with the user interface600.

FIG.8is a block diagram of an example processor platform800capable of executing the instructions ofFIG.3to implement the host server104ofFIG.1. The processor platform800can be, for example, a server, a personal computer, a mobile device (e.g., a cell phone, a smart phone, a tablet such as an iPad™), a personal digital assistant (PDA), an Internet appliance, a DVD player, a CD player, a digital video recorder, a Blu-ray player, a gaming console, a personal video recorder, a set top box, or any other type of computing device.

The processor platform800of the illustrated example includes a processor812. The processor812of the illustrated example is hardware. For example, the processor812can be implemented by one or more integrated circuits, logic circuits, microprocessors or controllers from any desired family or manufacturer. The hardware processor may be a semiconductor based (e.g., silicon based) device. In this example, the processor812implements the example weight aggregator110, the example weight distributor112, and the example global trainer114.

The processor812of the illustrated example includes a local memory813(e.g., a cache). The processor812of the illustrated example is in communication with a main memory including a volatile memory814and a non-volatile memory816via a bus818. The volatile memory814may be implemented by Synchronous Dynamic Random Access Memory (SDRAM), Dynamic Random Access Memory (DRAM), RAMBUS Dynamic Random Access Memory (RDRAM) and/or any other type of random access memory device. The non-volatile memory816may be implemented by flash memory and/or any other desired type of memory device. Access to the main memory814,816is controlled by a memory controller.

The processor platform800of the illustrated example also includes an interface circuit820. The interface circuit820may be implemented by any type of interface standard, such as an Ethernet interface, a universal serial bus (USB), and/or a PCI express interface.

In the illustrated example, one or more input devices822are connected to the interface circuit820. The input device(s)822permit(s) a user to enter data and/or commands into the processor812. The input device(s) can be implemented by, for example, an audio sensor, a microphone, a camera (still or video), a keyboard, a button, a mouse, a touchscreen, a track-pad, a trackball, isopoint and/or a voice recognition system.

One or more output devices824are also connected to the interface circuit820of the illustrated example. The output devices824can be implemented, for example, by display devices (e.g., a light emitting diode (LED), an organic light emitting diode (OLED), a liquid crystal display, a cathode ray tube display (CRT), a touchscreen, a tactile output device, a printer and/or speakers). The interface circuit820of the illustrated example, thus, typically includes a graphics driver card, a graphics driver chip and/or a graphics driver processor.

The interface circuit820of the illustrated example also includes a communication device such as a transmitter, a receiver, a transceiver, a modem and/or network interface card to facilitate exchange of data with external machines (e.g., computing devices of any kind) via a network826(e.g., an Ethernet connection, a digital subscriber line (DSL), a telephone line, coaxial cable, a cellular telephone system, etc.).

The processor platform800of the illustrated example also includes one or more mass storage devices828for storing software and/or data. Examples of such mass storage devices828include floppy disk drives, hard drive disks, compact disk drives, Blu-ray disk drives, RAID systems, and digital versatile disk (DVD) drives.

The coded instructions832ofFIG.3may be stored in the mass storage device828, in the volatile memory814, in the non-volatile memory816, and/or on a removable tangible computer readable storage medium such as a CD or DVD.

FIG.9is a block diagram of an example processor platform900capable of executing the instructions ofFIG.3to implement the local device102ofFIGS.1,2,4, and/or5. The processor platform900can be, for example, a server, a personal computer, a mobile device (e.g., a cell phone, a smart phone, a tablet such as an iPad™), a personal digital assistant (PDA), an Internet appliance, a DVD player, a CD player, a digital video recorder, a Blu-ray player, a gaming console, a personal video recorder, a set top box, or any other type of computing device.

The processor platform900of the illustrated example includes a processor912. The processor912of the illustrated example is hardware. For example, the processor912can be implemented by one or more integrated circuits, logic circuits, microprocessors or controllers from any desired family or manufacturer. The hardware processor may be a semiconductor based (e.g., silicon based) device. In this example, the processor912implements the example data receiver202, the example input samplers204, the example sample controller206, the example reference generator210, the example output samplers215, the example trainer216, and the example updater218.

The processor912of the illustrated example includes a local memory913(e.g., a cache). The processor912of the illustrated example is in communication with a main memory including a volatile memory914and a non-volatile memory916via a bus918. The volatile memory914may be implemented by Synchronous Dynamic Random Access Memory (SDRAM), Dynamic Random Access Memory (DRAM), RAMBUS Dynamic Random Access Memory (RDRAM) and/or any other type of random access memory device. The non-volatile memory916may be implemented by flash memory and/or any other desired type of memory device. Access to the main memory914,916is controlled by a memory controller.

The processor platform900of the illustrated example also includes an interface circuit920. The interface circuit920may be implemented by any type of interface standard, such as an Ethernet interface, a universal serial bus (USB), and/or a PCI express interface.

In the illustrated example, one or more input devices922are connected to the interface circuit920. The input device(s)922permit(s) a user to enter data and/or commands into the processor912. The input device(s) can be implemented by, for example, an audio sensor, a microphone, a camera (still or video), a keyboard, a button, a mouse, a touchscreen, a track-pad, a trackball, isopoint and/or a voice recognition system.

One or more output devices924are also connected to the interface circuit920of the illustrated example. The output devices924can be implemented, for example, by display devices (e.g., a light emitting diode (LED), an organic light emitting diode (OLED), a liquid crystal display, a cathode ray tube display (CRT), a touchscreen, a tactile output device, a printer and/or speakers). The interface circuit920of the illustrated example, thus, typically includes a graphics driver card, a graphics driver chip and/or a graphics driver processor.

The interface circuit920of the illustrated example also includes a communication device such as a transmitter, a receiver, a transceiver, a modem and/or network interface card to facilitate exchange of data with external machines (e.g., computing devices of any kind) via a network926(e.g., an Ethernet connection, a digital subscriber line (DSL), a telephone line, coaxial cable, a cellular telephone system, etc.).

The processor platform900of the illustrated example also includes one or more mass storage devices928for storing software and/or data. Examples of such mass storage devices928include floppy disk drives, hard drive disks, compact disk drives, Blu-ray disk drives, RAID systems, and digital versatile disk (DVD) drives.

The coded instructions932ofFIG.3may be stored in the mass storage device928, in the volatile memory914, in the non-volatile memory916, and/or on a removable tangible computer readable storage medium such as a CD or DVD.

From the foregoing, it will be appreciated that example methods, apparatus and articles of manufacture have been disclosed that enable the training of a deep learning model by a plurality of local devices. Utilizing multiple local devices facilitates the distributed processing among a plurality of devices. In addition, input data received at each local device may processed at the respective local device to avoid the bandwidth cost of transferring the input data to a central server for processing. In addition, privacy of the locally received input data may be maintained to processing at the local devices instead of transferring to a central server.

It is noted that this patent claims priority from U.S. patent application Ser. No. 16/326,361, which was filed on Feb. 18, 2019, and is hereby incorporated by reference in its entirety.

Example methods, apparatus, systems and articles of manufacture to detect anomalies in electronic data are disclosed herein. Further examples and combinations thereof include the following:

Example 1 is a local device to train deep learning models, the local device comprising: a reference generator to label input data received at the local device to generate training data, a trainer to train a local deep learning model and to transmit the local deep learning model to a server that is to receive a plurality of local deep learning models from a plurality of local devices, the server to determine a set of weights for a global deep learning model, and an updater to update the local deep learning model based on the set of weights received from the server.

Example 2 includes the local device as defined in claim1, further including a data receiver to receive the input data directly at the local device.

Example 3 includes the local device as defined in claim1, wherein the local device does not transmit the input data to the server.

Example 4 includes the local device as defined in claim1, wherein the set of weights are aggregated weights based on the plurality of local deep learning models from the plurality of local devices.

Example 5 includes the local device as defined in one of examples 1-4, further including a sample controller to sample the input data.

Example 6 includes the local device as defined in claim5, wherein the sample controller is to sample the input data by selecting a pseudo-random portion of the input data.

Example 7 includes the local device as defined in claim5, wherein the sample controller is to sample the input data by down sampling the input data to reduce a data size of the input data.

Example 8 includes the local device as defined in one of examples 1-4, wherein the trainer is further to determine a difference between a label determined by the labelling and an output of the local deep learning model.

Example 9 includes the local device as defined in example 8, further including a sample controller to sample the output of the local deep learning model prior to the trainer determining the difference.

Example 10 includes the local device as defined in example 8, wherein the trainer is further to adjust the local deep learning model based on the difference.

Example 11 is a non-transitory computer readable medium comprising instructions that, when executed, cause a local device to at least: label input data received at the local device to generate training data, train a local deep learning model, transmit the local deep learning model to a server, the server to receive a plurality of local deep learning models from a plurality of local devices, the server to determine a set of weights for a global deep learning model, and update the local deep learning model based on the set of weights received from the server.

Example 12 includes the non-transitory computer readable medium as defined in example 11, wherein the input data is received directly at the local device.

Example 13 includes the non-transitory computer readable medium as defined in example 11, wherein the input data is not transmitted to the server.

Example 14 includes the non-transitory computer readable medium as defined in example 11, wherein the set of weights are aggregated weights based on the plurality of local deep learning models from the plurality of local devices.

Example 15 includes the non-transitory computer readable medium as defined in one of examples 11-14, wherein the instructions, when executed, cause the local device to sample the input data.

Example 16 includes the non-transitory computer readable medium as defined in example 15, wherein the instructions, when executed, cause the local device to sample the input data by selecting a pseudo-random portion of the input data.

Example 17 includes the non-transitory computer readable medium as defined in example 15, wherein the instructions, when executed, cause the local device to sample the input data by down sampling the input data to reduce a data size of the input data.

Example 18 includes the non-transitory computer readable medium as defined in one of examples 11-14, wherein the instructions, when executed, cause the local device to determine a difference between a label determined by the labelling and an output of the local deep learning model.

Example 19 includes the non-transitory computer readable medium as defined in example 18, wherein the instructions, when executed, cause the local device to sample the output of the local deep learning model prior to the local device determining the difference.

Example 20 includes the non-transitory computer readable medium as defined in example 18, wherein the instructions, when executed, cause the local device to adjust the local deep learning model based on the difference.

Example 21 is a method to train deep learning models, the method comprising: labelling, by executing an instruction with at least one processor at a local device, input data received at the local device to generate training data, training, by executing an instruction with the at least one processor, a local deep learning model, transmitting the local deep learning model to a server, the server to receive a plurality of local deep learning models from a plurality of local devices, the server to determine a set of weights for a global deep learning model, and updating, by executing an instruction with the at least one processor at the local device, the local deep learning model based on the set of weights received from the server.

Example 22 includes the method as defined in example 21, wherein the input data is received directly at the local device.

Example 23 includes the method as defined in example 21, wherein the input data is not transmitted to the server.

Example 24 includes the method as defined in example 21, wherein the set of weights are aggregated weights based on the plurality of local deep learning models from the plurality of local devices.

Example 25 includes the method as defined in one of examples 21-24, further including sampling the input data.

Example 26 includes the method as defined in example 25, wherein the sampling of the input data includes selecting a pseudo-random portion of the input data.

Example 27 includes the method as defined in example 25, wherein the sampling of the input data includes down sampling the input data to reduce a data size of the input data.

Example 28 includes the method as defined in one of examples 21-24, further including determining a difference between a label determined by the labelling and an output of the deep learning model.

Example 29 includes the method as defined in example 28, further including sampling the output of the deep learning model prior to the determining of the difference.

Example 30 includes the method as defined in example 28, further including adjusting the deep learning model based on the difference.

Example 31 is a server comprising: a weight aggregator to aggregate weights of a plurality of local deep learning models received from a plurality of local devices, and a weight distributor to distribute the aggregated weights to the plurality of local devices.

Example 32 includes the server as defined in example 31, further including a global deep learning trainer to train a global deep learning model based on the aggregated weights.

Example 33 includes the server as defined in example 31 or example 32, wherein the server does not receive the input data utilized by the plurality of local devices to generate the plurality of local deep learning models.

Example 34 includes the server as defined in example 31 or example 32, wherein the weight aggregator is to aggregate the set of weights by averaging weights of the plurality of local deep learning models.

Example 35 is a non-transitory computer readable medium comprising instructions that, when executed, cause a server to at least: aggregate weights of a plurality of local deep learning models received from a plurality of local devices, and transmit the aggregated weights to the plurality of local devices.

Example 36 includes the non-transitory computer readable medium as defined in example 35, wherein the instructions, when executed, cause the server to train a global deep learning model based on the aggregated weights.

Example 37 includes the non-transitory computer readable medium as defined in example 35 or example 36, wherein the server does not receive the input data utilized by the plurality of local devices to generate the plurality of local deep learning models.

Example 38 includes the non-transitory computer readable medium as defined in example 35 or example 36, wherein the instructions, when executed, cause the server to aggregate the set of weights by averaging weights of the plurality of local deep learning models.

Example 40 is a method to train deep learning models, the method comprising: aggregating, by executing an instruction with at least one processor of a server, weights of a plurality of local deep learning models received from a plurality of local devices, and transmitting the aggregated weights to the plurality of local devices.

Example 41 includes the method as defined in example 40, further including training a global deep learning model based on the aggregated weights.

Example 42 includes the method as defined in example 40 or example 41, wherein the server does not receive the input data utilized by the plurality of local devices to generate the plurality of local deep learning models.

Example 43 includes the method as defined in example 40 or example 41, wherein the aggregating includes averaging weights of the plurality of local deep learning models.

Example 44 is a system to train deep learning models, the system comprising: local devices to, respectively: label input data received at the corresponding local device to generate training data, train a local deep learning model, transmit the local deep learning model over a network, and a server to: aggregate weights of local deep learning models received from the local devices, and transmit the aggregated weights to the local devices, the local devices to update the local deep learning model based on the set of weights received from the server.

Example 45 includes the system as defined in example 44, wherein the input data is received directly at the local devices.

Example 46 includes the system as defined in example 44, wherein the input data is not transmitted to the server.

Example 47 includes the system as defined in one of examples 44-46, wherein the local devices are further to sample the input data.

Example 48 includes the system as defined in example 47, wherein the local devices are to sample the respective input data by selecting a pseudo-random portion of the input data.

Example 49 includes the system as defined in example 47, wherein the local devices are to sample of the respective input data by down sampling the input data to reduce a data size of the input data.

Example 50 includes the system as defined in one of examples 44-46, wherein the local devices are further to determine a difference between a label determined by the labelling at the respective local device and an output of the local deep learning model at the respective local device.

Example 51 includes the system as defined in example 50, wherein the local devices are further to sample the output of the respective local deep learning model at the respective local device prior to the local devices determining the difference.

Example 52 includes the system as defined in example 50, wherein the local devices are further to adjust the local deep learning model based on the difference.

Example 53 is an apparatus to train deep learning models, the apparatus comprising: means for labelling input data received at the local device to generate training data, means for training a first local deep learning model, means for transmitting the local deep learning model to a server, the server to receive a plurality of local deep learning models from a plurality of local devices, the plurality of local deep learning models including the first deep learning model, the server to determine a set of weights for a global deep learning model, and means for updating the first local deep learning model based on the set of weights received from the server.

Example 54 includes the apparatus as defined in example 53, further including means for receiving the input data directly at the local device.

Example 55 includes the local device of Example 1, the non-transitory computer readable medium of Example 11, or the example method of Example 21, wherein the input data does not pass through the server.

Example 56 includes the local device of Example 1, the non-transitory computer readable medium of Example 11, or the example method of Example 21, wherein the local device includes a sensor to collect the input data.

Example 57 includes the local device of Example 1, the non-transitory computer readable medium of Example 11, or the example method of Example 21, wherein the local device and/or the data receiver is communicatively coupled to a sensor that collects the input data and transmits the input data to the local device.

Although certain example methods, apparatus and articles of manufacture have been disclosed herein, the scope of coverage of this patent is not limited thereto. On the contrary, this patent covers all methods, apparatus and articles of manufacture fairly falling within the scope of the claims of this patent.