Patent Description:
Machine learning models can help in solving a variety of tasks that have traditionally been difficult for a computing system. However, the machine learning models are often large and require a considerable amount of storage capacity and transfer bandwidth to be delivered to a computing system. As such, to make machine learning models more appealing for application in bandwidth-limited networks, for example in mobile applications and the like, ways are needed to reduce the cost of transfer and storage of machine learning models.

<NPL>) describes incremental neural network quantization.

Aspects and advantages of embodiments of the present disclosure will be set forth in part in the following description, or may be learned from the description, or may be learned through practice of the embodiments.

The invention is described by the independent claims. Preferred embodiments are described by the dependent claims.

Other example aspects of the present disclosure are directed to systems, apparatus, tangible, non-transitory computer-readable media, user interfaces, memory devices, and electronic devices.

These and other features, aspects, and advantages of various embodiments will become better understood with reference to the following description and appended claims. The accompanying drawings illustrate embodiments of the present disclosure and, together with the description, serve to explain the related principles.

Generally, the present disclosure is directed to systems and methods to compress and/or distribute machine-learning models. In particular, the systems and methods of the present disclosure can leverage compression methods to reduce the size (e.g., the data storage and transfer requirements) of a machine-learned model. By reducing the size of the model, the systems and methods of the present disclosure can reduce the network and/or computational expense associated with transfer, storage, and/or use of the model. In particular, enabling reduced network expenses associated with transfer or other forms of distribution of the machine-learned models can make the machine-learned models more appealing or useful in limited-bandwidth networks. Likewise, enabling reduced storage expense and/or computational expense associated with storage and/or use of the machine-learned models can make the machine-learned models more appealing or useful in resource-limited devices or environments such as, for example, mobile applications, devices, and/or environments.

Thus, the present disclosure provides compression techniques that can be used to reduce the size of machine-learned models. In particular, the systems and methods of the present disclosure can perform various types of quantization of weights of a model with loss management. As one example, the systems and methods of the present disclosure can provide model compression such that the distortion introduced by compression has a bounded increase in loss. As another example, in some implementations, when model weights are quantized, the quantization errors can be compensated by later quantization errors. For example, quantization error can be distributed among one model layer to the previous or next layer, among nodes within a layer, within connections of a single node, based on node correlation, and/or according to other schemes. As another example, an iterative training process can be performed in which a certain number of the best matching (e.g., lowest quantization error producing) nodes are quantized and then frozen while the model is subjected to additional training. By implementing quantization with loss management, as described herein, machine-learned models can be compressed more densely while reducing the fitness degradation of the machine-learned model.

In addition to compression techniques, the present disclosure provides additional techniques for improved distribution, storage, and/or use of machine-learned models. For example, in some implementations, the systems and methods of the present disclosure can perform or enable one or more of: patching for model updates, distributed learning, training for quantization, training for patching, and/or transfer learning. As an example, in some implementations, machine-learned model distribution can include patching where a smaller part (e.g., only a portion) of the machine-learned model is delivered to update a deployed version of the machine-learned model, such as on a mobile device, rather than updating the entire machine-learned model. In another example, in some implementations, distributed learning can be performed in which machine-learned model weights are changed at a client device and are then gathered through patches from the client device. The use of patches for uploading of updated weights from the client device reduces the bandwidth used by the client device.

In yet another example, in some implementations, the systems and methods of the present disclosure can perform or enable training for quantization, in which certain weights of a model are selected and frozen during training to improve error, and training for patching, in which whole layers of a model are selected and frozen during training to improve compression. In another example, in some implementations, machine-learned model distribution can include transfer learning in which patching can be used to repurpose a model already existing on a client device, thereby eliminating the need to transmit an entirely new model.

More particularly, according to one aspect of the present disclosure, machine-learned model distribution can include compressing machine-learned models by implementing quantization with loss management. Using quantization with loss management, a machine-learned model can be compressed for distribution by quantizing model weights where quantization errors can be compensated by other quantization errors. In some implementations, for example, quantization error can be compensated for by dithering to distribute the quantization error among other associated weights. When a neural weight is quantized, some error is introduced which can then be distributed among weights that have yet to be quantized to compensate for the error For example, when a weight is quantized, a determination can be made of which other weights (that have yet to be quantized) are associated with the quantized weight. The quantized weight's quantization error can then be fractionally distributed to those associated weights, for example, by using a predetermined multiplier. The next associated weight then takes its original value plus the transferred quantization error, and that new weight value is then quantized. The quantization error from the quantization of this next associated weight can then be distributed among the remaining associated weights that have not yet been quantized.

In some implementations, as the quantization of the weights continues, the quantization error can accumulate until reaching or exceeding a threshold which can then cause an opposite error to be made and thereby reduce the error in the remaining weight values. In some implementations, quantization decisions are not made one at a time, but instead consider the impact on other quantization decisions.

Accordingly, quantization with loss management can include various strategies to reduce the fitness degradation of the model. In some implementations, for example, quantization of weights can be performed based on a best-matching coefficient. For example, the quantization begins by determining the best matching coefficient (e.g., the weight that would result in the least amount of quantization error), and quantizing this weight. The quantization error from this weight can then be transferred to the other weights. By quantizing the best matching weight first, the transfer of the quantization error causes the worse matching weights to move and, no matter the direction of the move, these weights then have a higher probability of becoming better matching weights than if they were quantized in another order.

In some implementations, one or more of the best matching coefficients are quantized and then additional training iterations are completed while keeping the quantized coefficients locked. This can allow the first quantization errors to propagate to coefficients that are more difficult to quantize, allowing less overall error to be done in the quantization process. In some implementations, sets of best matching weights can be done simultaneously for faster computation, for example, the best matching <NUM>% of weights or the like.

In some implementations, for example, quantization error can be dithered from the previous layer. For example, a distance measure can be used between nodes within a layer, and signals coming from the nodes of a previous cell can be quantized with consideration (e.g., partial accumulation) of the quantization error from similar nodes. As one example, node K1 and node K2 of layer K may have a similar connection to a previous layer J. As such, if the J1-to-K1 connection is quantized down, the J1-to-K2 connection can be enforced to be more likely to be quantized up.

In some implementations, the quantization error can be dithered within the node. For example, if a J1-to-K1 connection is quantized down, a J2-to-K1 connection can be enforced to be more likely to be quantized up.

In some implementations, the quantization error can be dithered within the node considering the correlation of activity, where the correlation of activity between two weights is descriptive of an activation relationship between the two weights. One goal for such correlation-based distribution scheme is that when the quantization error is dithered to a highly correlated node, the loss is decreased. By way of example, if a J1-to-K1 connection is quantized down, a J2-to-K1 connection should be more likely to be quantized as a function of the correlation between J1 and J2. In some implementations, if there is an inverse correlation, the dithering may be done in the opposite direction, potentially leading to a smaller error.

According to another aspect of the present disclosure, in some implementations, machine-learned model distribution can include patching where a smaller part of the machine-learned model is updated and delivered for changing a deployed version of the machine-learned model. For example, patching can provide for a part of a machine-learned model to be delivered (for example, to a mobile device or the like) without updating other parts of the machine-learned model. For example, patching can be used to deliver changes which are critical that they be delivered quickly to a large group of recipients. In some implementations, patching provides for updating smaller parts of a machine-learned model that is deployed, for example, in mobile devices, and how those smaller parts of the model go through the learning process of neural network computation. Patching can allow for updates that only concern a small subset of the neural graph so that they can be updated in such a way compress the size of the update. For example, to develop a smaller patch, the part of the neural network structure where training is allowed could be limited such that training only happens with a small subset of the weights. By way of example, in some implementations, training could be done with new weights being found. It can then be determined where the weights have changed the most, and another round of training could be done where only a subset of the weights, for example, the top five percentile of changed weights, are allowed to change. This would then result in a smaller number of weights changing, thereby creating a smaller patch. In another example, to develop a smaller patch, the neural network can be retrained and then a set of nodes whose values changed the most, for example the top five percentile of the changed weights (e.g., nodes with top five percent of ∥retrained value - original value||), can be identified and included in the patch.

In some implementations, patching can include patches for a rule-based system verifying the results of a deep learning model. For example, when some situation calls for an immediate change to a model but a high-bandwidth update cannot be delivered to many users, patches could be used to check final results of text-based models or zero out parts of a machine-learned model. In some implementations, patching can include using previous weights to give a statistical prior for new weights, allowing for tighter compression of the new weights.

According to another aspect of the present disclosure, in some implementations, machine-learned model distribution can include distributed learning in which machine-learned model weights may be changed at a client device and are then gathered through patches from the client device to reduce the device bandwidth use. For example, in some implementations, machine-learned model weights may be changed at a client device to further optimize the model loss function with real data. In some implementations, the changed weights could then be gathered in "reverse" patches to minimize the client device bandwidth use. In some implementations, distributed learning using client data can be applied in such a way that a smaller set of node weights is chosen, for example, partly randomly (e.g., to improve privacy and coverage) and partly by what a central controller has seen as nodes worth updating (e.g., to improve efficiency).

According to another aspect of the present disclosure, in some implementations, machine-learned model distribution can include training for quantization and training for patching. For example, training for quantization can include determining weights to freeze during training to improve error. In some implementations, for example, an annealing process can be applied to weights by freezing a set of weights that introduce the smallest error. Training for patching can include, for example, choosing which layers to freeze during training. For example, in some models, the lower layers may typically remain unchanged whereas higher layers exhibit more change. In some implementations, this can be made explicit during a training phase, such as by adding rules into learning to freeze layers (e.g., either stochastically or by another measure of fitness impact), to produce smaller patch sizes.

According to another aspect of the present disclosure, in some implementations, machine-learned model distribution can include transfer learning whereby patching can benefit from an already learned model on a client device. For example, it can sometimes be beneficial to start training of a model from an already learned model. In some implementations, a patch approach can be applied for developing such models, thereby benefitting from an already deployed learned model such that an entire model does not need to be transferred, for example, providing for transfer of only small deltas.

The systems and methods described herein provide a number of technical effects and benefits. For instance, the use of learned-model distribution strategies such as quantization with loss management leads to reductions in bandwidth use which is a significant cost to mobile computing. Thus the systems and methods described herein allow for increased use of machine learned-models in bandwidth-limited networks, such as mobile computing, where the machine learned-models can provide benefits to a variety of user applications. Additionally, the systems and methods described herein may also provide a technical effect and benefit of reducing update size and improving performance due to reducing cache and memory bandwidth requirements.

The systems and methods described herein also provide resulting improvements to computing technology tasked with the distribution and use of machine-learned models. For example, through the use of advanced compression techniques for machine-learned model distribution as described herein, computing systems may optimize bandwidth use and reduce transfer costs and more efficiently provide machine-learned models for use in various applications, such as mobile applications. Further, the systems and methods described herein may provide reductions in storage requirements and system resources, thus making machine-learned models more appealing in limited-bandwidth networks.

<FIG> depicts a block diagram of an example computing system <NUM> that can perform compression and distribution of machine learning models. The system <NUM> includes a user computing device <NUM>, a server computing device <NUM>, and a training computing system <NUM> that are communicatively coupled over a network <NUM>.

The user computing device <NUM> can be any type of computing device, such as, for example, a personal computing device (e.g., laptop or desktop), a mobile computing device (e.g., smartphone or tablet), a gaming console or controller, a wearable computing device, an embedded computing device, a personal assistant computing device, or any other type of computing device.

The user computing device <NUM> includes one or more processors <NUM> and a memory <NUM>. The one or more processors <NUM> can be any suitable processing device (e.g., a processor core, a microprocessor, an ASIC, a FPGA, a controller, a microcontroller, etc.) and can be one processor or a plurality of processors that are operatively connected. The memory <NUM> can include one or more non-transitory computer-readable storage mediums, such as RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks, etc., and combinations thereof. The memory <NUM> can store data <NUM> and instructions <NUM> which are executed by the processor <NUM> to cause the user computing device <NUM> to perform operations.

Furthermore, according to an aspect of the present disclosure, the user computing device <NUM> can store or include one or more machine-learned models <NUM>. The machine-learned models <NUM> can be or can otherwise include one or more neural networks (e.g., deep neural networks); Markov models (e.g., hidden Markov models); classifiers; regression models; support vector machines; Bayesian networks; multi-layer non-linear models; or other types of machine-learned models. Neural networks (e.g., deep neural networks) can be feed-forward neural networks, convolutional neural networks, autoencoders, recurrent neural networks (e.g., long short-term memory neural network, gated recurrent units, etc.) and/or various other types of neural networks.

In some implementations, the one or more machine-learned models <NUM> can be received from the server computing device <NUM> over network <NUM>, stored in the user computing device memory <NUM>, and then used or otherwise implemented by the one or more processors <NUM>. In some implementations, the user computing device <NUM> can implement multiple parallel instances of a single machine-learned model <NUM>.

More particularly, machine-learned model(s) <NUM> can be implemented to provide assistance in various situations and/or applications. As an example, the machine-learned model(s) <NUM> can be employed within the context of a mobile application of the user computing device <NUM>, providing benefits and advantages during the execution of such mobile applications. Thus, one or more models <NUM> can be stored and implemented at the user computing device <NUM>.

The user computing device <NUM> can also include model trainer(s) <NUM>. The model trainer <NUM> can train or re-train machine-learned models <NUM> stored at user computing device <NUM> using various training or learning techniques, such as, for example, backwards propagation of errors (e.g., truncated backpropagation through time). In particular, the model trainer <NUM> can train or re-train one or more of the machine-learned models <NUM> using locally stored data as training data. The model trainer <NUM> can perform a number of generalization techniques to improve the generalization capability of the models being trained. In some implementations, some information about the trained model's parameters can be delivered by the user computing device <NUM> back to the server computing device <NUM>.

The user computing device <NUM> can also include one or more input/output interface(s) <NUM>. One or more input/output interface(s) <NUM> can include, for example, devices for receiving information from or providing information to a user, such as a display device, touch screen, touch pad, mouse, data entry keys, an audio output device such as one or more speakers, a microphone, haptic feedback device, etc. An input/output interface(s) <NUM> can be used, for example, by a user to control operation of the user computing device <NUM>.

The user computing device <NUM> can also include one or more communication/network interface(s) <NUM> used to communicate with one or more systems or devices, including systems or devices that are remotely located from the user computing device <NUM>. The communication/network interface(s) <NUM> can include any circuits, components, software, etc. for communicating with one or more networks (e.g., network <NUM>). In some implementations, the communication/network interface(s) <NUM> can include, for example, one or more of a communications controller, receiver, transceiver, transmitter, port, conductors, software, and/or hardware for communicating data.

The server computing device <NUM> includes one or more processors <NUM> and a memory <NUM>. The one or more processors <NUM> can be any suitable processing device (e.g., a processor core, a microprocessor, an ASIC, a FPGA, a controller, a microcontroller, etc.) and can be one processor or a plurality of processors that are operatively connected. The memory <NUM> can include one or more non-transitory computer-readable storage mediums, such as RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks, etc., and combinations thereof. The memory <NUM> can store data <NUM> and instructions <NUM> which are executed by the processor <NUM> to cause the server computing device <NUM> to perform operations.

In some implementations, the server computing device <NUM> includes or is otherwise implemented by one or more server computing devices. In instances in which the server computing device <NUM> includes plural server computing devices, such server computing devices can operate according to sequential computing architectures, parallel computing architectures, or some combination thereof.

As described above, the server computing device <NUM> can store or otherwise include one or more machine-learned models <NUM>. The machine-learned models <NUM> can be or can otherwise include one or more neural networks (e.g., deep neural networks); Markov models (e.g., hidden Markov models); classifiers; regression models; support vector machines; Bayesian networks; multi-layer non-linear models; or other types of machine-learned models. Neural networks (e.g., deep neural networks) can be feed-forward neural networks, convolutional neural networks, autoencoders, recurrent neural networks (e.g., long short-term memory neural network, gated recurrent units, etc.) and/or various other types of neural networks.

The server computing device <NUM> can also include a model compressor <NUM> that can perform compression of one or more machine learning models <NUM> to reduce the size (e.g., the data storage and transfer requirements) of the machine-learned model(s). In particular, in some implementations, the model compressor <NUM> can perform quantization of one or more weights of the machine learning models <NUM> where the quantization error introduced by the quantization can be compensated by later quantization errors.

Additionally, in some implementations, model compressor <NUM> can also provide for additional techniques for improved distribution, storage, and/or use of machine-learned models. For example, in some implementations, the model compressor <NUM> can perform or enable one or more of: patching for model updates, distributed learning, training for quantization, training for patching, and/or transfer learning.

The server computing device <NUM> can also include one or more input/output interface(s) <NUM>. The one or more input/output interface(s) <NUM> can include, for example, devices for receiving information from or providing information to a user, such as a display device, touch screen, touch pad, mouse, data entry keys, an audio output device such as one or more speakers, a microphone, haptic feedback device, etc. An input/output interface(s) <NUM> can be used, for example, by a user to control operation of the server computing device <NUM>.

The server computing device <NUM> can also include one or more communication/network interface(s) <NUM> used to communicate with one or more systems or devices, including systems or devices that are remotely located from the server computing device <NUM>. The communication/network interface(s) <NUM> can include any circuits, components, software, etc. for communicating with one or more networks (e.g., network <NUM>). In some implementations, the communication/network interface(s) <NUM> can include, for example, one or more of a communications controller, receiver, transceiver, transmitter, port, conductors, software, and/or hardware for communicating data.

The server computing device <NUM> can train the machine-learned models <NUM> via interaction with the training computing system <NUM> that is communicatively coupled over the network <NUM>. The training computing system <NUM> can be separate from the server computing device <NUM> or can be a portion of the server computing device <NUM>.

The training computing system <NUM> includes one or more processors <NUM> and a memory <NUM>. The one or more processors <NUM> can be any suitable processing device (e.g., a processor core, a microprocessor, an ASIC, a FPGA, a controller, a microcontroller, etc.) and can be one processor or a plurality of processors that are operatively connected. The memory <NUM> can include one or more non-transitory computer-readable storage mediums, such as RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks, etc., and combinations thereof. The memory <NUM> can store data <NUM> and instructions <NUM> which are executed by the processor <NUM> to cause the training computing system <NUM> to perform operations. In some implementations, the training computing system <NUM> includes or is otherwise implemented by one or more server computing devices.

The training computing system <NUM> can include one or more model trainer (s) <NUM> that trains the machine-learned models <NUM> stored at the server computing device <NUM> using various training or learning techniques, such as, for example, backwards propagation of errors (e.g., truncated backpropagation through time). The model trainer <NUM> can perform a number of generalization techniques (e.g., weight decays, dropouts, etc.) to improve the generalization capability of the models being trained. In particular, the model trainer <NUM> can train a machine-learned model <NUM> based on a set of training data <NUM>. The training data <NUM> can include centrally collected data or remotely obtained data.

The training computing system <NUM> can also include one or more input/output interface(s) <NUM>. The one or more input/output interface(s) <NUM> can include, for example, devices for receiving information from or providing information to a user, such as a display device, touch screen, touch pad, mouse, data entry keys, an audio output device such as one or more speakers, a microphone, haptic feedback device, etc. An input/output interface(s) <NUM> can be used, for example, by a user to control operation of the training computing system <NUM>.

The training computing system <NUM> can also include one or more communication/network interface(s) <NUM> used to communicate with one or more systems or devices, including systems or devices that are remotely located from the training computing system <NUM>. The communication/network interface(s) <NUM> can include any circuits, components, software, etc. for communicating with one or more networks (e.g., network <NUM>). In some implementations, the communication/network interface(s) <NUM> can include, for example, one or more of a communications controller, receiver, transceiver, transmitter, port, conductors, software, and/or hardware for communicating data.

Each of model trainer <NUM> and model trainer <NUM> can include computer logic utilized to provide desired functionality. Each of model trainer <NUM> and model trainer <NUM> can be implemented in hardware, firmware, and/or software controlling a general purpose processor. For example, in some implementations, each of model trainer <NUM> and model trainer <NUM> includes program files stored on a storage device, loaded into a memory and executed by one or more processors. In other implementations, each of model trainer <NUM> and model trainer <NUM> includes one or more sets of computer-executable instructions that are stored in a tangible computer-readable storage medium such as RAM hard disk or optical or magnetic media.

<FIG> illustrates one example computing system that can be used to implement the present disclosure. Other computing systems can be used as well.

<FIG> illustrate example methods of the present disclosure. Although <FIG> respectively depict steps in a particular order for purposes of illustration and discussion, the methods of the present disclosure are not limited to the particularly illustrated order or arrangement. The various steps of the methods of <FIG> can be omitted, rearranged, combined, and/or adapted in various ways without deviating from the scope of the present disclosure.

<FIG> depicts a flowchart diagram of an example method <NUM> of compressing machine-learning models via model quantization with loss management.

At <NUM>, a computing device, such as server computing device <NUM> of <FIG>, can obtain a machine learning model, for example, a machine learning model that is to be compressed for distribution to other computing systems, such as in bandwidth-limited networks.

At <NUM>, the computing device can select one or more weights to be quantized as part of reducing the size of the machine learned model. In particular, in some implementations, the computing device can select the one or more weights based on various determinations. For example, in some implementations, the computing device can select the one or more weights to quantize first based on a best-matching coefficient (e.g., the weight that would result in the least amount of quantization error).

At <NUM>, the computing device can quantize the selected weight(s). In some implementations, sets of best matching weights can be done simultaneously for faster computation, for example, the best matching <NUM>% of weights or the like.

At <NUM>, a quantization error for the weight being quantized can be obtained. In particular, when a neural weight is quantized some amount of quantization error is introduced. This quantization error can be compensated for, at least partially, in the following manner.

At <NUM>, the computing device can determine if there are the remaining weights of the machine learning model that have yet to be quantized. If there are remaining non-quantized weights, the method <NUM> can continue to <NUM>. If there are no remaining weights to be quantized, the method <NUM> can continue the <NUM> and provide the quantized machine learning model for distribution.

At <NUM>, the computing device can determine one or more of the non-quantized weights that are associated with the quantized weight (e.g., the weight being quantized and introducing quantization error). For example, the associated non-quantized weights can be determined from nodes within a same layer, nodes from a previous layer or following layer, within connections of a single node, based on node correlation, and/or according to other schemes.

At <NUM>, the quantization error can be propagated or distributed among one or more of the associated non-quantized weights. In particular, in some implementations, for example, the quantization error can be compensated for by dithering to distribute the quantization error among the other associated weights. In one example, the quantization error can be fractionally distributed to the associated weight(s), for example, by using a predetermined multiplier. The next associated weight(s) then takes its original value plus the transferred quantization error, and that new weight value can then be quantized. In some implementations, as the quantization of the weights continues, the quantization error can accumulate until reaching or exceeding a threshold which can then cause an opposite error to be made and thereby reduce the error in the remaining weight values. According to various implementations, quantization error can be distributed from one model layer to the next layer, among nodes within a layer, within connections of a single node, based on node correlation, and/or according to other schemes, for example.

At <NUM>, the computing device can then select one or more next weights to quantize from among the non-quantized weights, and then return to <NUM> to iteratively continue the quantization with loss management.

At <NUM>, after completing the quantization iterations of the model weights (e.g., when there are no weights remaining to be quantized at <NUM>), the computing system can provide the quantized machine learning model, for example, to be distributed to other computing devices.

At <NUM>, the computing device can determine one or more of the non-quantized weights that are associated with the quantized weight (e.g., the weight being quantized and introducing quantization error). For example, the associated non-quantized weights can be determined from nodes within a same layer, nodes from a previous layer or a following layer, within connections of a single node, based on node correlation, and/or according to other schemes.

At <NUM>, the quantization error can be propagated or distributed among one or more of the associated non-quantized weights. In particular, in some implementations, example, the quantization error can be compensated for by dithering to distribute the quantization error among the other associated weights. In one example, the quantization error can be fractionally distributed to the associated weight(s), for example, by using a predetermined multiplier. The next associated weight(s) then takes its original value plus the transferred quantization error, and that new weight value can then be quantized.

At <NUM>, the computing device can perform one or more additional training iterations for the machine learning model while keeping the quantized weights locked. For example, this locking of weights can allow the first quantization errors to propagate to coefficients that are more difficult to quantize, allowing less overall error to be introduced in the quantization process.

<FIG> depicts a flowchart diagram of an example method <NUM> of quantizing machine-learning models.

At <NUM>, the computing device can estimate the importance of one or more coefficients of the machine learning model. For example, the importance of a coefficient can be based at least in part on its magnitude.

At <NUM>, the computing device can estimate the change in size and change in loss for each of the coefficients introduced by one or more quantization strategies. In particular, in some implementations, the quantization strategies can rely on coefficient importance estimation data.

At <NUM>, the computing device can select one or more quantization strategies to be applied to the machine learning model, based at least in part on the size change and loss change estimations.

At <NUM>, the computing device can apply the one or more selected quantization strategies to the machine learning model.

At <NUM>, the computing device can then output the quantized machine learning model, for example, to be distributed to one or more computing systems (e.g., bandwidth-limited systems such as mobile applications, devices, and/or networks).

At <NUM>, the computing device can propagate the quantization error from the quantized weight(s) to one or more non-quantized weights. For example, in some implementations, the quantization error can be compensated for by dithering to distribute the quantization error among other non-quantized weights.

At <NUM>, the computing device can quantize the non-quantized weight(s).

At <NUM>, the computing system can provide the quantized machine learning model, for example, to be distributed to other computing devices.

Claim 1:
A computer-implemented method (<NUM>) to compress neural networks, the method comprising:
obtaining (<NUM>), by one or more computing devices, a neural network, wherein the neural network comprises a plurality of layers that each include one or more nodes, and a plurality of weights that each connect nodes of different layers;
selecting (<NUM>), by the one or more computing devices, a weight of the neural network;
quantizing (<NUM>), by the one or more computing devices, the weight;
determining (<NUM>), by the one or more computing devices, a quantization error associated with the weight, wherein the quantization error associated with the weight is a change in the weight due to quantization;
distributing (<NUM>), by the one or more computing devices, at least a part of the quantization error to one or more non-quantized weights of the neural network;
quantizing, by the one or more computing devices, one or more of the non-quantized weights; and
providing (<NUM>), by the one or more computing devices, a quantized neural network,
wherein the step of distributing the quantization error comprises:
determining (<NUM>), by the one or more computing devices, one or more non-quantized weights associated with the weight, wherein the associated non-quantized weights are determined from nodes within a same layer of a node connected by the weight, nodes from a previous layer or following layer, and/or within connections of a single node connected by the weight; and
transferring, by the one or more computing devices, a fraction of the quantization error associated with the weight to each of the associated non-quantized weights by summing an associated non-quantized weight with said fraction of the quantization error associated with the weight, and based on a correlation of activity between said associated non-quantized weight and the weight, wherein the correlation of activity is descriptive of an activation relationship between the weight and an associated non-quantized weight.