Patent Description:
Neural networks (NN) may be utilized for many different applications in many different types of devices, such as mobile phones. Examples of technologies include image and video analysis and processing, social media data analysis, device usage data analysis, or the like. A neural network is a computation graph comprising several layers of computation. A layer may comprise one or more units, for example nodes. Each unit may be configured to perform an elementary computation. A unit may be connected to one or more other units and this connection may be associated with a weight. The weight may be used for scaling the signal passing through the associated connection. Weights are an example of learnable parameters, which may be updated during a training process to train the neural network for a particular task. Running neural networks at a device or delivering neural networks between devices may require significant memory or bandwidth resources, which may not always be available. Document "<NPL>) presents an approach for fine-tuning a decoder-side neural network in the context of image compression, such that the weight-updates are better compressible. Document <CIT> relates to requesting a neural network model for processing a cognitive computing task and receiving a second neural network model obtained by trimming the first neural network model.

Example embodiments provide a system, where compressed neural networks may be efficiently delivered between devices, from a server to a client. Further implementation forms are provided in the dependent claims, the description, and the drawings.

According to a first aspect, a server device comprises at least one processor; and at least one memory including computer program code; the at least one memory and the computer program code configured to, with the at least one processor, cause the server device at least to: receive a request for a neural network from a client device, wherein the request comprises at least one requirement for the neural network; obtain a first compressed neural network based on the at least one requirement; transmit the first compressed neural network to the client device; obtain a residual between at least a subset of the first compressed neural network and at least a subset of the second compressed neural network in response to receiving, from the client device or a third-party device, an indication that the first compressed neural network does not satisfy a performance requirement, wherein the first compressed neural network and the second compressed neural network are configured to perform a same task, wherein the second compressed neural network is obtainable based on adding the residual to the first compressed neural network, and wherein a size and performance of the second compressed neural network are higher than a size and performance of the first compressed neural network; and transmit the residual to the client device.

According to a second aspect, a method comprises receiving, by a server device, a request for a neural network from a client device, wherein the request comprises at least one requirement for the neural network; obtaining a first compressed neural network based on the at least one requirement; transmitting, by the server device, the first compressed neural network to the client device; obtaining, by the server device, a residual between at least a subset of the first compressed neural network and at least a subset of the second compressed neural network in response to receiving, from the client or a third-party device, an indication that the first compressed neural network does not satisfy a performance requirement, wherein the first compressed neural network and the second compressed neural network are configured to perform a same task, wherein the second compressed neural network is obtainable based on adding the residual to the first compressed neural network, and wherein a size and performance of the second compressed neural network is higher than a size and performance of the first compressed neural network; and transmitting, by the apparatus, the residual to the client device.

According to a third aspect, a computer program is configured, when executed by a served device, to cause the server device at least to: receive a request for a neural network from a client device, wherein the request comprises at least one requirement for the neural network; obtain a first compressed neural network based on the at least one requirement; transmit the first compressed neural network to the client device; obtain a residual between at least a subset of the first compressed neural network and at least a subset of the second compressed neural network in response to receiving, from the client or a third-party device, an indication that the first compressed neural network does not satisfy a performance requirement, wherein the second compressed neural network is obtainable based on adding the residual to the first compressed neural network, and wherein a size and perfomance of the second compressed neural network is higher than a size and performance of the first compressed neural network; and transmit the residual to the client device.

According to a fourth aspect, an apparatus comprises means for receiving a request for a neural network from a client device, wherein the request comprises at least one requirement for the neural network; means for obtaining a first compressed neural network based on the at least one requirement; means for transmitting the first compressed neural network to the client device; means for obtaining a residual between at least a subset of the first compressed neural network and at least a subset of the second compressed neural network in response to receiving, from the client or a third-party device, an indication that the first compressed neural network does not satisfy a performance requirement, wherein the second compressed neural network is obtainable based on adding the residual to the first compressed neural network, and wherein a size and perfomance of the second compressed neural network is higher than a size and performance of the first compressed neural network; and means for transmitting the residual to the client device.

According to a fifth aspect, a client device comprises at least one processor; and at least one memory including computer program code; the at least one memory and the computer program code configured to, with the at least one processor, cause the client device at least to: transmit a request for a neural network to a server device, wherein the request comprises at least one requirement for the neural network; receive a first compressed neural network; provide the first compressed neural network for evaluation, wherein the evaluation comprises determining whether the first compressed neural network satisfies a performance requirement; and receive a residual between the first compressed neural network and a second compressed neural network and reconstruct the second compressed neural network based on adding the residual to the first compressed neural network, wherein the first compressed neural network and the second compressed neural network are configured to perform a same task, and wherein a size and performance of the second compressed neural network is higher than a size and performance of the first compressed neural network.

According to a sixth aspect, a method comprises transmitting, by a client device, a request for a neural network to a server device, wherein the request comprises at least one requirement for the neural network; receiving, by the client device, a first compressed neural network; providing, by the client device, the first compressed neural network for evaluation, wherein the evaluation comprises determining whether the first compressed neural network satisfies a performance requirement; and receiving, by the client device, a residual between the first compressed neural network and a second compressed neural network and reconstructing, by the client device, the second compressed neural network based on adding the residual to the first compressed neural network, wherein the first compressed neural network and the second compressed neural network are configured to perform a same task, and wherein a size and performance of the second compressed neural network is higher than a size and performance of the first compressed neural network.

According to a seventh aspect, a computer program is configured, when executed by a client device, to cause the client device to: transmit a request for a neural network to a server device, wherein the request comprises at least one requirement for the neural network; receive a first compressed neural network; provide the first compressed neural network for evaluation, wherein the evaluation comprises determining whether the first compressed neural network satisfies a performance requirement; and receive a residual between the first compressed neural network and a second compressed neural network and reconstruct the second compressed neural network based on adding the residual to the first compressed neural network, wherein the first compressed neural network and the second compressed neural network are configured to perform a same task, and wherein a size and performance of the second compressed neural network is higher than a size and performance of the first compressed neural network.

According to an eighth aspect, a client device comprises means for transmitting a request for a neural network to a server device, wherein the request comprises at least one requirement for the neural network; means for receiving a first compressed neural network; means for providing the first compressed neural network for evaluation, wherein the evaluation comprises determining whether the first compressed neural network satisfies a performance requirement; and means for receiving a residual between the first compressed neural network and a second compressed neural network and reconstructing the second compressed neural network based on adding the residual to the first compressed neural network, wherein the first compressed neural network and the second compressed neural network are configured to perform a same task, and wherein a size and performance of the second compressed neural network is higher than a size and performance of the first compressed neural network.

Many of the attendant features will be more readily appreciated as they become better understood by reference to the following detailed description considered in connection with the accompanying drawings.

The accompanying drawings, which are included to provide a further understanding of the example embodiments and constitute a part of this specification, illustrate example embodiments and together with the description help to explain the example embodiments. In the drawings:.

Reference will now be made in detail to example embodiments, examples of which are illustrated in the accompanying drawings. The detailed description provided below in connection with the appended drawings is intended as a description of the present examples and is not intended to represent the only forms in which the present example may be constructed or utilized. The description sets forth the functions of the example and the sequence of steps for constructing and operating the example.

According to an example embodiment a client requests a neural network from a server. The request includes requirements for the requested network, for example regarding size and performance of the network. After receiving the request from the client, the server obtains a compressed neural network in accordance with the size indicated in the request. The server may further use its own local dataset to determine whether the performance requirement is satisfied. However, since the server may not have or be able to access the same dataset as the client, the performance of the compressed neural network may not be reliably assessed. Therefore, after receiving the compressed neural network from the server, the client determines whether the compressed neural network satisfies the performance requirement with client's dataset. If not, the client requests an updated compressed network, which is provided as a residual with respect to the first compressed neural network. The process may be iterated until a satisfactory size vs. performance trade-off has been achieved.

<FIG> illustrates an example of a system <NUM> for delivering compressed neural networks between devices, according to an embodiment. System <NUM> comprises a server <NUM>, which may be communicatively coupled to a client <NUM> over a communication network <NUM>. Server <NUM> comprises a neural network library <NUM> comprising one or more neural networks configured to perform various tasks. Each neural network of the library <NUM> may be associated with an identifier such that a corresponding neural network may be retrieved from the library upon request. Server <NUM> may be further capable of training new neural networks as well as fine-tuning any existing neural network for a particular task, for example based on one or more training datasets <NUM>. Neural network library <NUM> and/or the training datasets <NUM> may be located at a memory of server <NUM>, or at a remote memory accessible to server <NUM>.

The server <NUM> may further be capable of compressing any of the neural networks by various means. Neural network compression may refer to the compression of neural network's parameters, for example weights, which may form the biggest part in terms of bits required to represent the neural network. The other parts, for example, the architecture definition, may not require as many bits to be represented. This may be the case for example for big neural networks comprising a high number of layers and parameters. The input to the neural network compression system may be an original trained network, which may have been trained for example by using at least one task loss. A task loss may refer to a loss function that is minimized during training to achieve a desired output from the network, for example to train the neural network for a particular task.

System <NUM> may further comprise a third-party device <NUM>, which may be for example configured to perform evaluations of compressed neural networks based on information received from client <NUM> and to provide evaluation results to client <NUM> and/or server <NUM>. In some example embodiments, the third-party device <NUM> may be operated by the same service provider as client <NUM>.

Server <NUM>, client <NUM>, and third-party device <NUM> may be implemented by various means. For example, server <NUM> may comprise a server device, server apparatus, server software, server cloud, or the like. Client <NUM> may comprise a client device, client apparatus, client software, or a client application downloadable to a client device. Third-party device <NUM> may comprise a third-party apparatus, third-party software, or a third-party service.

Compressing neural networks may be desired for different reasons, such as for example for reducing bitrate required to transmit the network over a communication channel, for reducing storage requirements, for reducing memory consumption at runtime, or for reducing computational complexity at runtime, etc..

Performance of neural network compression may be assessed for example based on the reduction of the number of bits required to represent the neural network and on the degradation of task performance. A compression algorithm is expected to reduce the number of bits, or bitrate, as much as possible, while minimizing degradation of task performance. Task performance may refer to the performance on the task for which the network was trained, such as for example classification accuracy for a classifier or MSE for a network performing regression.

Several approaches may be used to compress neural networks. For example, some approaches may be based on quantization of weights, some approaches may be based on pruning (removing) small values, some approaches may be based on low-rank decompositions of weight matrices, and some approaches may include a training or retraining step. Regarding the latter, the retraining may be performed for each neural network that is subject to compression. This may include retraining the neural network to be compressed with a different loss with respect to the task loss that the network was originally trained with, such as with a combination of at least a task loss and a compression loss. The compression loss may be computed on the weights, for example to enforce pruning or sparsification, for example to enforce many weights to have low values, or to enforce easier quantizability, for example to enforce that weights values to be near quantization values. Such retraining may take a long time, for example up to one week even with most powerful hardware acceleration currently available.

The compression process may comprise a pruning process followed by a quantization process. The neural network subject to compression may be obtained in any way. One example includes retraining or fine-tuning a neural network such that it becomes more compressible. For example, filter pruning may be performed according to a learnt importance of each layer, for example an automatically learnt scale parameter as a part of a batch normalization layer for each filter. Pruning may be performed for example based on thresholding this scale parameter and removing an entire filter that has a scale value under the threshold. The quantization process may be performed for example by a k-means method. It is noted that the pruning of the filters may be more relevant for the amount of memory for storing activations, quantization may be more relevant for the bandwidth limitation. However, if the quantization process ensures weights with fixed-point precision and if the software at client side has a fixed-point implementation of neural networks, quantization to fixed point values may also help to reduce the amount of memory needed to store activations as well as memory needed to store weights of the neural network.

Compression of a neural network provides a neural network that is capable of performing the same task as the original network, but not necessarily with the same quality. The compressed neural network may be provided in a data format that enables reconstruction of the neural network from the compressed representation, for example, by inserting zeros to replace pruned weights. This may be done in accordance with signaling information provided within or along with the compressed representation of the neural network. If the compressed neural network is postprocessed, for example by entropy coding, corresponding inverse operation may be performed before applying the compressed neural network.

A client <NUM> may be configured with at least one application <NUM>. Application <NUM> may use at least one neural network received from server <NUM> over network <NUM>. Capabilities of client <NUM> may be limited with respect to available memory, computational power, battery power, or in general any resource needed to perform inference with the neural network. Client <NUM> may be also limited with respect to receiving or storing the neural network. For example, client <NUM> may have memory limitations such that it may not be able to run the entire neural network in one go. Instead, the client <NUM> may be configured to run the neural network in subsets, for example layer by layer. Given this constraint, the neural network to be used in inference by client <NUM> may be beneficially configured to satisfy memory constraints regarding weights of at least one neural network layer and the intermediate activations output from this layer.

In another example, the client <NUM> may have limited communication capabilities even though communication channel <NUM> itself would not be bandwidth-limited, for example because of other simultaneous communication sessions running at client <NUM>. In general, the example embodiments enable to satisfy both memory and performance requirements for a client device.

Client <NUM> may comprise any functions to collect data subject to inference by a neural network. For example, client <NUM> may be equipped with one or more cameras configured to capture video or image content, one or more microphones configured to capture audio content, or one or more sensors to capture sensor data, for example information associated with acceleration, temperature, or the like. Alternatively, or additionally, client <NUM> may be configured to receive the data subject to inference from another device, for example from an accessory device such as a smartwatch, from an external camera, microphone or sensor, or an Internet-of-Things (IoT) device.

<FIG> illustrates an example of an apparatus <NUM> according to an embodiment, for example server <NUM> or client <NUM>. Apparatus <NUM> comprises at least one processor <NUM>.

The at least one processor may comprise, for example, one or more of various processing devices, such as for example a coprocessor, a microprocessor, a controller, a digital signal processor (DSP), a processing circuitry with or without an accompanying DSP, or various other processing devices including integrated circuits such as, for example, an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), a microcontroller unit (MCU), a hardware accelerator, a special-purpose computer chip, or the like.

The apparatus further comprises at least one memory <NUM>. The memory may be configured to store, for example, computer program code or the like, for example operating system software and application software. The memory may comprise one or more volatile memory devices, one or more non-volatile memory devices, and/or a combination thereof. For example, the memory may be embodied as magnetic storage devices (such as hard disk drives, floppy disks, magnetic tapes, etc.), optical magnetic storage devices, or semiconductor memories (such as mask ROM, PROM (programmable ROM), EPROM (erasable PROM), flash ROM, RAM (random access memory), etc.).

Apparatus <NUM> may further comprise communication interface <NUM> configured to enable apparatus <NUM> to transmit and/or receive information, for example messages to/from other devices. In one example, apparatus <NUM> may use communication interface <NUM> to transmit or receive compressed neural networks and/or control messages associated with provision of the compressed neural networks. The communication interface may be configured to provide at least one wireless radio connection, such as for example a 3GPP mobile broadband connection (e.g. <NUM>, <NUM>, <NUM>); a wireless local area network (WLAN) connection such as for example standardized by IEEE <NUM> series or Wi-Fi alliance; a short range wireless network connection such as for example a Bluetooth, NFC (near-field communication), or RFID connection; a local wired connection such as for example a local area network (LAN) connection or a universal serial bus (USB) connection, or the like; or a wired Internet connection.

Apparatus <NUM> may further comprise a user interface <NUM> comprising an input device and/or an output device. The input device may take various forms such a keyboard, a touch screen, or one or more embedded control buttons. The output device may for example comprise a display, a speaker, a vibration motor, or the like.

When the apparatus is configured to implement some functionality, some component and/or components of the apparatus, such as for example the at least one processor and/or the memory, may be configured to implement this functionality. Furthermore, when the at least one processor is configured to implement some functionality, this functionality may be implemented using program code <NUM> comprised, for example, in the memory <NUM>.

The functionality described herein may be performed, at least in part, by one or more computer program product components such as software components. According to an embodiment, the apparatus comprises a processor or processor circuitry, such as for example a microcontroller, configured by the program code when executed to execute the embodiments of the operations and functionality described. For example, and without limitation, illustrative types of hardware logic components that can be used include Field-programmable Gate Arrays (FPGAs), application-specific Integrated Circuits (ASICs), application-specific Standard Products (ASSPs), System-on-a-chip systems (SOCs), Complex Programmable Logic Devices (CPLDs), Graphics Processing Units (GPUs).

The apparatus comprises means for performing at least one method described herein. In one example, the means comprises the at least one processor, the at least one memory including program code configured to, when executed by the at least one processor, cause the apparatus to perform the method.

Apparatus <NUM> may comprise for example a computing device such as for example a server, mobile phone, a tablet computer, a laptop, an internet of things (IoT) device, or the like. Examples of IoT devices include, but are not limited to, consumer electronics, wearables, sensors, and smart home appliances. In one example, apparatus <NUM> may comprise a vehicle such as for example a car. Although apparatus <NUM> is illustrated as a single device it is appreciated that, wherever applicable, functions of apparatus <NUM> may be distributed to a plurality of devices, for example to implement example embodiments as a cloud computing service.

<FIG> illustrates an example of a neural network <NUM>, according to an embodiment. Neural network <NUM> may comprise an input layer, one or more hidden layers, and an output layer. Nodes of the input layer, i<NUM> to in, may be connected to one or more of the m nodes of the first hidden layer, n<NUM> to n<NUM>. Nodes of the first hidden layer may be connected to one or more of the k nodes of the second hidden layer, n<NUM> to n<NUM>. It is appreciated that even though the example neural network of <FIG> illustrates two hidden layers, a neural network may apply any number and any type of hidden layers. Neural network <NUM> may further comprise an output layer. Nodes of the last hidden layer, in the example of <FIG> the nodes of second hidden layer, may be connected to one or more nodes of the output layer, o<NUM> to oj. It is noted that the number of nodes may be different for each layer of the network. A node may be also referred to as a neuron, a unit, a computation unit, or an elementary computation unit. Terms network, neural network, neural net, or model may be used interchangeably. Weights of the neural network may be referred to as learnable parameters or simply as parameters. In the example of <FIG>, one or more of the layers may be fully connected layers, for example layers where each node is connected to every node of a previous layer. A fully connected layer may be represented for example as a matrix, where the number of rows is the dimensionality of the input activations, and the number of columns is the number of output activations.

Two example architectures of neural networks include feed-forward and recurrent architectures. Feed-forward neural networks are such that there is no feedback loop. Each layer takes input from one or more previous layers and provides its output as the input for one or more of the subsequent layers. Also, units inside certain layers may take input from units in one or more of preceding layers and provide output to one or more of following layers.

Initial layers, for example layers close to the input data, may extract semantically low-level features. In an example of image data, the low-level features may correspond to edges and textures in images. Intermediate and final layers may extract more high-level features. After the feature extraction layers there may be one or more layers performing a certain task, such as classification, semantic segmentation, object detection, denoising, style transfer, super-resolution, or the like.

In recurrent neural networks there is a feedback loop from one or more nodes of one or more subsequent layers. This causes the network to become stateful. For example, the network may be able to memorize information or a state.

<FIG> an example of a node <NUM>. A node may be configured to receive one or more inputs, a<NUM> to an, from one or more nodes of one or more previous layers and compute an output based on the input values received. A node may also receive feedback from one or more nodes of one or more subsequent layers. Inputs may be associated with parameters to adjust the influence of a particular input to the output. For example weights w<NUM> to wn associated with the inputs a<NUM> to an may be used to multiply the input values a<NUM> to an. The node may be further configured combine the inputs to an output, or an activation. For example, the node may be configured to sum the modified input values. A bias or offset b may be also applied to add a constant to the combination of modified inputs. Weights and biases may be learnable parameters. For example, when the neural network is trained for a particular task, the values of the weights and biases associated with different inputs and different nodes may be updated such that an error associated with performing the task is reduced to an acceptable level.

Furthermore, an activation function f() may be applied to control when and how the node <NUM> provides its output. Activation function may be for example a non-linear function that is substantially linear in the region of zero but limits the output of the node when the input increases or decreases. Examples of activation functions include, but are not limited to, a step function, a sigmoid function, a tanh function, a ReLu (rectified linear unit) function. The output may be provided to nodes of one or more following layers of the network, and/or to one or more nodes of one or more previous layers of the network.

A forward propagation or a forward pass may refer to feeding a set of input data through the layers of the neural network <NUM> and producing an output. During this process the weights and biases of the neural network <NUM> affect the activations of individual nodes and finally the output provided by the output layer.

One property of neural networks and other machine learning tools is that they are able to learn properties from input data, for example in supervised way or in unsupervised way. Learning may be based on teaching the network by a training algorithm or based on a meta-level neural network providing a training signal.

In general, a training algorithm may include changing some properties of the neural network such that its output becomes as close as possible to a desired output. For example, in the case of classification of objects in images, the output of the neural network may be used to derive a class or category index, which indicates the class or category that the object in the input image belongs to. Training may happen by minimizing or decreasing the output's error, also referred to as the loss.

During training the generated or predicted output may be compared to a desired output, for example ground-truth data provided for training purposes, to compute an error value. The error may be calculated based on a loss function. Updating the neural network may be then based on calculating a derivative with respect to learnable parameters of the network. This may be done for example using a backpropagation algorithm that determines gradients for each layer starting from the final layer of the network until gradients for the learnable parameters have been obtained. Parameters of each layer are updated accordingly such that the loss is iteratively decreased. Examples of losses include mean squared error, cross-entropy, or the like. In deep learning, training comprises an iterative process, where at each iteration the algorithm modifies parameters of the neural network to make a gradual improvement of the network's output, that is, to gradually decrease the loss.

Training phase of the neural network may be ended after reaching an acceptable error level. Inference phase may refer to applying the trained neural network for a particular task. For example, to provide a classification of an unseen image to one of a plurality of classes based on content of an input image.

Training a neural network may be seen as an optimization process, but the final goal may be different from a typical goal of optimization. In optimization, the goal may be to minimize a functional. In machine learning, a goal of the optimization or training process is to make the model learn the properties of the data distribution from a limited training dataset. In other words, the goal is to learn to use a limited training dataset in order to learn to generalize to previously unseen data, that is, data which was not used for training the model. This may be referred to as generalization.

In some examples, data may be split into at least two sets, a training data set and a validation data set. The training data set may be used for training the network, for example to modify its learnable parameters in order to minimize the loss. The validation data set may be used for checking performance of the network on data which was not used to minimize the loss as an indication of the final performance of the model. In particular, the errors on the training set and on the validation data set may monitored during the training process to understand the following issues: <NUM>) if the network is learning at all - in this case, the training data set error should decrease, otherwise the model is in the regime of underfitting; <NUM>) if the network is learning to generalize - in this case, also the validation set error should decrease and not be much higher than the training data set error. If the training data set error is low, but the validation data set error is much higher than the training data set error, or it does not decrease, or it even increases, the model is in the regime of overfitting. This means that the model has just memorized properties of the training data set and performs well on that set, but performs poorly on a set not used for tuning its parameters.

<FIG> illustrates an example of a convolutional neural network <NUM>. A convolutional neural network <NUM> may comprise at least one convolutional layer. A convolutional layer may perform convolution type of operations to extract information from input data, for example image <NUM>, to form a plurality of feature maps <NUM>. A feature map may be generated for example based on applying a filter or a kernel to a subset of input data, for example block <NUM> in image <NUM>, and sliding the filter through the input data to obtain a value for each element of the feature map. The filter may comprise a matrix or a tensor, which may be for example multiplied with the input data to extract features corresponding to that filter. A plurality of feature maps may be generated based on applying a plurality of filters. A further convolutional layer may take as input the feature maps from a previous layer and apply similar filtering process on the feature maps <NUM> to generate another set of feature maps <NUM>. Weights of the filters may be learnable parameters and they may be updated during a training phase, similar to parameters of neural network <NUM>. Similar to node <NUM>, an activation function may be applied to the output of the filter(s). The convolutional neural network may further comprise one or more other type of layers such as for example fully connected layers <NUM> after and/or between the convolutional layers. An output may be provided by an output layer <NUM>.

As an example of compressing neural networks, a convolutional layer with <NUM> filters may output H × W × <NUM> feature maps, where H and W are the height and width of the feature map output by each filter. Such convolutional layer may be compressed, for example by pruning methods, to a convolutional layer with <NUM> filters, which would output H × W × <NUM> feature maps, thus requiring much less memory space.

An architecture of a neural network may refer to the type, size, and relationships of the neural network layers. An example of a neural network having a particular architecture is ResNet50. This network may be trained for many different tasks. During training the learnable parameters, such as for example weights, filter values, biases, or the like, may be updated. However, during training the architecture of the network may be preserved. Neural network architectures may be enumerated. For example, server <NUM> and client <NUM> may exchange information, for example a look-up table, to assign identifiers for different architectures.

Example embodiments may be applied in at least two scenarios. In a first scenario the memory available on client <NUM> may be very limited. For example, only one layer and either the input activations or output activations or both can be stored. The client <NUM> may be alternatively able to store a subset of the layers, for example two or three layers, and corresponding input, output and intermediate activations. In a second scenario the memory on client <NUM> may be limited such that the whole network can be stored, but only either the input activations or output activations or both from a single layer can be stored.

In both scenarios, the memory for storing the input activations, the output activations, or both may not be enough for storing the activations from an original, uncompressed, layer. Therefore the original layers may be made smaller (compressed) such that the output activations occupy less memory.

<FIG> illustrates an example of delivering neural network(s) from a server <NUM> to a client <NUM>, with an optional performance evaluation step at a third-party device <NUM>, according to an embodiment. Client <NUM> may initially determine that it needs to perform a task. This may be for example during an execution of an application <NUM> at the client <NUM>, upon obtaining data subject to inference by a neural network, or upon a request from a user.

At <NUM>, client <NUM> determines to transmit a request for a neural network. The request may for example comprise a request message (REG MSG) or a request signal. The request may comprise one or more request messages or signals and the information included in the request may be distributed among the one or more request messages or signals.

According to an example embodiment, the request may comprise a size associated with the requested neural network. The size may for example comprise a memory limit for the neural network. The size may be determined by the client <NUM> based on memory available at the client <NUM>. The available memory may refer to a total amount of memory available at the client <NUM> or an amount of memory allocated for executing the neural network, for example an amount of memory allocated to an application <NUM>. Size of the neural network may be provided in various formats, for example as number of bits or bytes, number of nodes, number and/or precision of activations, or the like.

According to an example embodiment, the request may comprise a size associated with a subset of the neural network, for example at least one layer of the neural network. The size may comprise for example a size, for example a maximum size, of the layer of neural network. Alternatively, or additionally, the size may comprise a size, for example a maximum size, of activations output from the layer.

According to an example embodiment, the request may comprise an identifier of a task to be performed by the neural network. For example, a particular identifier, for example an integer value, may be associated with a particular task, such as for example image classification. Alternatively, name of the task may be included in the request as a string of characters.

According to an example embodiment the request may comprise an identifier of an architecture of the neural network. The architecture may be provided for example as a number of layers, type of layers, numbers of nodes at the layers, parameters of filters, or the like. Alternatively, an architecture may be identified based on a reference to a predefined architecture, such as for example ResNet50. The reference may be provided as an identifier, for example an integer number, or as a string of characters.

According to an example embodiment the request may comprise at least one performance requirement for the neural network. The performance requirement may be associated with the task indicated by the identifier of the task included in the request. The performance requirement may be provided in any suitable format. For example, for image classification the performance requirement may comprise an accuracy in the image classification task. This may be provided for example as a percentage of correct classifications, such as for example <NUM> out of <NUM>. In another example, for an image compression task, the performance requirement may be expressed as mean-square-error (MSE) between the output image and a ground-truth image.

At <NUM>, server <NUM> receives the request for the neural network from the client <NUM>. At <NUM>, the server <NUM> obtains a first compressed neural network. Obtaining the first compressed network is based on the at least one requirement included in the request. According to an example embodiment, obtaining the first compressed neural network may comprise retrieving a compressed neural network that complies with the request from a memory of server <NUM>, or requesting such network form an external device.

According to an example embodiment, the first compressed neural network may be obtained by compressing a neural network retrieved from memory of server <NUM> or requested form an external device. Target of the compression may be to obtain a smallest reasonable network (SRN), which satisfies the performance requirement on an annotated dataset possessed by or accessible to the server <NUM>.

The neural network subject to compression may be referred to as an original neural network, although it is appreciated that the original neural network may have been subject to one or more previous compression steps or other processing steps. According to an example embodiment, the server <NUM> may select the original network based on the identifier of the neural network or the identifier of the architecture of the neural network. For example, if the request indicates ResNet50, server <NUM> may retrieve ResNet50 from its neural network library <NUM>. According to an example embodiment, the server <NUM> may select the original neural network based on the identifier of the task. For example, the server <NUM> may be configured with a mapping between a plurality of tasks and a plurality of networks. For example, if the request indicates that the task is image classification, the server <NUM> may be configured to select ResNet50 as the original neural network.

According to an example embodiment, the request sent by client <NUM> may include identifier of a task and an identifier of a neural network architecture. For example, the request may comprise a TaskID indicating ImageNetClassification and an ArchitectureID indicating ResNet50. In another example, the request may comprise a TaskID indicating CIFAR100Classification and an ArchitectureID indicating ResNet50. The request may further comprise additional identifiers, for example for indicating training hyperparameters. For example, the request may comprise a TaskID indicating CIFAR100Classification, an ArchitectureID indicating ResNet50 as well as a TrainingID, for example UsingBatchNormLayers.

Server <NUM> may determine parameters for compressing the original neural network based on the size and/or the performance requirement indicated in the request. In some embodiments, the performance, for example accuracy, may not be considered by the server <NUM>. Instead, the memory or bandwidth requirement may be considered without considering the performance. Memory and/or bandwidth limitations may be considered with respect to a subset of the compressed neural network, for example with respect to any layer of the compressed neural network. In some embodiments, the performance requirement on the data available locally at the server side may be predefined, for example a default performance requirement, whereas in other embodiments the performance requirement may be received from the client <NUM>, or from another device such as for example third-party device <NUM>. Alternatively, or additionally, a bandwidth requirement associated with a communication channel of network <NUM> may be considered when determining the compression parameters.

Based on the at least one requirement, server <NUM> may compress the original network to obtain a suitable compressed neural network. For example, server <NUM> may select a pruning threshold and number of clusters, k, in a k-means based compression method such that when pruned and quantized the compressed network will satisfy the size or bandwidth, and/or performance requirement(s). For example, a default performance requirement or received performance requirement may be <NUM> percent of the performance of the original model. The compressed network may be for example represented by using a binary tensor indicating positions of zeros (pruned weights), another tensor containing the non-pruned and quantized weights in the form of cluster labels, and a mapping between cluster labels and centroid values, for example a dictionary or a look-up table. However, any other suitable representation of a compressed neural network may be used.

Operation <NUM> may further comprise determining whether the compressed neural network satisfies the at least one requirement included in the request received from the client <NUM>. For example, server <NUM> may compare size of the compressed network to the size indicated in the request, and determine to send the compressed neural network to client <NUM> if the size is less than or equal to the size indicated in the request. According to an example embodiment, server <NUM> may determine whether the compressed neural network satisfies the at least one performance requirement included in the request. This may be based on performing inference on a server dataset, for example a local dataset available to server <NUM>.

It is noted that even if performance, when evaluated with the local dataset of the server <NUM>, satisfies the performance requirement, this does not necessarily imply satisfying the performance requirement at the client <NUM>, when inference is performed at the client <NUM> with a different dataset.

At <NUM>, server <NUM> transmits the compressed neural network to client <NUM>. Transmitting the compressed neural network may be in response to determining that the compressed neural network satisfies, at least substantially, the at least one performance requirement, size, and/or other requirements.

At <NUM>, the client <NUM> receives the compressed neural network from server <NUM>. The client <NUM> may then reconstruct the compressed neural network based on the received representation. For example, the client <NUM> may insert zeros or other predetermined values to certain locations, for example based on an indication included in signaling information received within or along the representation of the compressed neural network. The indicated locations may correspond to pruned weights. Non-pruned weights may be replaced for example by cluster centroids depending on the label vector that is sent for each weight.

At <NUM>, the client <NUM> may execute, run, or infer the compressed neural network. This may comprise providing input data to the neural network and obtaining output data from the compressed neural network.

At <NUM>, the client <NUM> may transmit the output of the compressed neural network a third-party device <NUM>. Transmitting the output may further comprise transmitting instructions to evaluate performance of the compressed neural network based on the output data provided by client <NUM>.

At <NUM>, the third-party device may receive the output data from client <NUM>. The third-party device <NUM> may possess annotations or labels possessed by client <NUM>. Based on the annotation or labels and the output data, the third-party device <NUM> may evaluate performance of the compressed neural network at <NUM>. At <NUM>, the third-party device <NUM> may transmit the evaluation result to client <NUM>.

As part of the evaluation at <NUM>, the third-party device <NUM> may determine whether the compressed neural network satisfies the performance requirement and/or other requirements. In response to determining that the compressed neural network satisfies the requirement(s), the third-party device <NUM> may proceed to operation <NUM> to acknowledge delivery of an acceptable neural network to client <NUM>, for example by sending an "OK" message to server <NUM>. The acknowledgement message may comprise an identifier of client <NUM> and/or an identifier of the compressed neural network.

In response to determining that the compressed neural network does not satisfy the requirement(s), the third-party device <NUM> may transmit an indication that the compressed neural network does not satisfy the requirement(s) to the server <NUM>, for example a "NOT OK" message at operation <NUM>. The indication may comprise an identifier of client <NUM> and/or an identifier of the compressed neural network. Transmitting the "OK" or "NOT OK" messages directly to server <NUM> may reduce delay in providing feedback to server <NUM> regarding the compression of the neural network. Alternatively, or additionally, at <NUM> the third-party device <NUM> may send the evaluation result to client <NUM>.

At <NUM>, the client <NUM> may evaluate the compressed neural network to determine whether it satisfies the performance requirement and/or other requirements. Evaluating the performance may be done based on a client dataset, which may comprise a local dataset available to client <NUM>. As discussed above, the client dataset may be different from the server dataset. The server dataset and client datasets may be annotated datasets.

In response to determining that the compressed neural network satisfies the requirement(s), the client <NUM> may directly proceed to operation <NUM> to acknowledge reception of an acceptable neural network, for example by sending an "OK" message to server <NUM>.

In response to determining that the compressed neural network does not satisfy the requirement(s), the client <NUM> transmits an indication that the compressed neural network does not satisfy at least one performance requirement, for example a "NOT OK" message at operation <NUM>. Determining whether the compressed neural network satisfies the requirement(s) may be based on the evaluation at <NUM> or the evaluation result received from the third-party device <NUM> at <NUM>.

Alternatively, or additionally, at <NUM> client <NUM> may send a second request, for example a second request message or a second request signal, to server <NUM>. The second request may comprise an indication of a required performance improvement. The indication of the required performance improvement may be provided with respect to the compressed neural network received from server <NUM> at <NUM>. The indication of the required performance improvement may comprise at least one of: a performance of the first compressed neural network, when evaluated based on a client dataset; or a difference between the performance requirement and the performance of the first compressed neural network, when evaluated by the client <NUM> on the client dataset. For example, the client <NUM> may indicate that the performance with the client dataset is <NUM>/<NUM>, which is lower than the required performance <NUM>/<NUM>. Alternatively, the client <NUM> may indicate that the performance with the client dataset is <NUM>/<NUM> lower than the required performance. Although not illustrated in <FIG>, this indication may be also sent by third-party device <NUM>.

At <NUM> or <NUM>, the server <NUM> receives the indication that the compressed neural network does not satisfy at least one performance requirement, either from the client <NUM> or the third-party device <NUM>. Alternatively, or additionally, at <NUM> the server <NUM> may receive the second request from client <NUM>, or third-party device <NUM>. Based on the indication received at <NUM> or <NUM> and/or the second request at <NUM>, the server <NUM> determines to obtain an updated neural network, for example a second compressed neural network, in order to satisfy the at least one performance requirement also with the client dataset. If server <NUM> does not receive the second request, it may use a default performance improvement when obtaining the updated compressed neural network.

At <NUM>, the server <NUM> updates the compressed neural network. Starting from a previous neural network (PNN) that was sent to the client <NUM>, for example the first compressed neural network, the server <NUM> obtains a residual, for example a residual weight vector, to be added to the PNN. The resulting network should still satisfy the bandwidth and/or size requirement. In general, a PNN may refer to a network that is obtainable from an initial network, for example the smallest reasonable network (SRN), by adding all previous residuals sent to the client <NUM>. The neural network obtained based on adding the residual to the PNN, for example the second compressed neural network, should increase performance on the server dataset.

Depending on the method that the server uses and depending on the availability of the information, the server <NUM> may use the PNN's performance and the received requirement of performance improvement to update the network. For example, server <NUM> may determine a second pruning threshold (thr) and/or second quantization hyperparameter(s), for example a number of clusters (k) to improve performance of the first compressed neural network. Determining the second pruning threshold and/or the second quantization hyperparameter(s) may be based on the required performance improvement included in the second request. A weight update may be obtained based on the pruning and the quantization process that is applied in this stage compared to the previous stage. For example, if in the previous stage thr=<NUM> and k=<NUM> were applied and during this stage thr=<NUM> and k=<NUM> were applied there may be two options: A) the new weights that are not pruned in this stage but were pruned in the previous stage can be sent to the device with the labels for these weights calculated according to the previous cluster centroids. B) the server <NUM> may decide to recalculate the quantization cluster centroids and labels for every non-pruned weight may be recalculated and sent.

At <NUM>, the server <NUM> transmits a residual between the first compressed neural network and the second compressed neural network to the client <NUM>. According to an example embodiment, the residual may comprise non-pruned weights which were not transmitted to the client <NUM> as part of the first compressed neural network. According to an example embodiment, the residual may comprise a difference between non-pruned weights of the first compressed neural network and the non-pruned weights of the second compressed neural network. The latter may be applied for example if different quantization cluster centroids are used when generating the second compressed neural network.

The size of the second compressed neural network is higher than the size of the first compressed neural network. Furthermore, performance of the second compressed neural network is higher than performance of the first compressed neural network, when evaluated either based on the server dataset or the client dataset.

At <NUM>, the client <NUM> receives the residual between the first compressed neural network and the second compressed neural network.

At <NUM>, the client <NUM> may update the previous neural network. The client <NUM> reconstructs a second compressed neural network based on the previously received first compressed neural network and the residual. This may be done for example based on reconstructing the received residual, for example based on adding predetermined values at signaled locations, and adding the reconstructed residual to the first compressed neural network.

At <NUM>, the client <NUM> may execute, run, or infer the updated neural network, for example the second compressed neural network, similar to operation <NUM>.

At <NUM>, the client <NUM> may transmit the output of the updated neural network, for example the second compressed neural network to a third-party device <NUM>, similar to operation <NUM>.

At <NUM>, the third-party device <NUM> may receive the output of the updated neural network. At <NUM>, the updated neural network may be evaluated similar to operation <NUM>. An evaluation result may be provided to client at <NUM>, similar to operation <NUM>, and an acknowledgement of delivery of an acceptable neural network to client <NUM> may be provided at <NUM>, for example by sending an "OK" message.

At <NUM>, the client <NUM> may evaluate performance of the updated neural network, for example the second compressed neural network, on the client dataset, similar to operation <NUM>.

In response to determining that the updated neural network still does not satisfy the requirement(s), the client <NUM> may proceed to operation <NUM> and/or operation <NUM> to provide an indication of the updated neural network not satisfying the performance requirement and/or the required performance improvement. Alternatively, client <NUM> may proceed to operation <NUM> and/or operation <NUM>, in response to receiving the evaluation result from the third-party device <NUM> at <NUM>, for example if the evaluation result indicates that the updated neural network still does not satisfy the requirement(s).

In response to determining that the updated neural network satisfies the requirement (s), at <NUM> the client <NUM> may acknowledge reception of an acceptable neural network, for example by transmitting an "OK" message to server <NUM>. This may be based on the evaluation at <NUM> or the evaluation result received from the third-party device <NUM> at <NUM>. According to an example embodiment, sending an "OK" message may be optional. In this case the client <NUM> may start using the updated compressed neural network without acknowledging it to server <NUM>. If not receiving a further request from the client <NUM>, the server <NUM> may determine that the updated compressed neural network was satisfactory.

<FIG> illustrates an example of layer wise delivery of neural network(s) from a server <NUM> to a client <NUM>, with an optional performance evaluation step performed at a third-party device <NUM>, according to an embodiment. It is noted that even though the embodiments of <FIG> are described using layers as an example of a subset, the embodiments may be applied to any other subsets of neural networks. For example, convolutional layers of the neural network could be delivered filter-by-filter. Transmitting or updating at least a subset may refer to transmitting or updating a subset of the neural network or the whole neural network, for example all layers of the neural network.

The example embodiment of <FIG> may comprise operations <NUM> to <NUM>, similar to operations <NUM> to <NUM>, respectively. In operation <NUM> the server may select compression parameters such that the memory and/or bandwidth limitations are not exceeded for any single layer of the resulting compressed neural network.

At <NUM>, server <NUM> may transmit a first layer of the first compressed neural network to client <NUM>.

At <NUM>, the client <NUM> may receive the first layer of the compressed neural network from the server <NUM>.

At <NUM>, the client <NUM> may execute, run, or infer the first layer of the compressed neural network. Execution of the first layer may comprise providing input data to the first layer and storing intermediate activations output by the first layer. Operation <NUM> may further comprise transmitting an acknowledgement message to server <NUM>. This may be done to indicate that the first layer was successfully executed and that the client <NUM> is ready to receive the next layer for execution.

The server <NUM> may be configured to iterate transmitting the layers at <NUM> until all layers of the network have been delivered to client <NUM>. A next layer may be transmitted in response to receiving the acknowledgement message from the client <NUM>. Alternatively, the server <NUM> may be configured to transmit the layers periodically, for example after a predetermined waiting time has elapsed from transmitting a previous layer.

The client <NUM> may iterate receiving <NUM> and executing <NUM> the layers until all layers have been executed. Receiving a next layer may comprise overwriting at least part of a previous layer in the memory of the client <NUM>. Execution of a next layer may be based on the intermediate activations output by the previous layer and stored in the memory of the client <NUM>. Output of the next layer may be stored as intermediate activations, for example by overwriting at least part of the previous intermediate activations.

At <NUM>, the output from the final layer may be provided for evaluation at a third-party device <NUM>, similar to operation <NUM>.

The example embodiment of <FIG> may further comprise operations <NUM> to <NUM>, similar to operations <NUM> to <NUM>, respectively.

At <NUM>, server <NUM> may obtain a residual between a first layer of the previous neural network, and a first layer of an updated neural network. The residual may be referred to as a residual layer. The previous neural network may comprise a first compressed neural network and the updated neural network may comprise a second compressed neural network.

Obtaining the residual layer may comprise fine-tuning a previous neural network with a compression loss and computing a weight-wise difference between the new and the previous neural network layers. Alternatively, obtaining the residual may comprise fine-tuning the previous neural network with the compression loss and keeping track of (e.g. by accumulating) the weight updates computed at each iteration. The final weight update may be obtained as the sum of weight updates. The final weight update may comprise an array or tensor of values, which may be sliced (partitioned) to get values corresponding to each layer. In yet another example, the compression step may be re-run with a less severe compression rate target. The obtained weights may be compared to the weights of layers of the previous neural network to obtain the residual as difference between the weights.

At <NUM>, the server may add the residual layer to the first layer of the previous neural network, for example the first compressed neural network. Adding the residual may be advantageously done at the server <NUM> in order to enable client <NUM> to release memory resources after execution of each layer at <NUM>.

At <NUM>, the server <NUM> may transmit the updated first layer of the updated neural network to client <NUM>. The server <NUM> may iterate obtaining the residual layers <NUM>, adding the residual layers <NUM> to layers of the previous neural network, and transmitting <NUM> the updated layers to the client <NUM>. It is however noted that the server <NUM> may also obtain residuals for a plurality layers, update the layers, and after that transmit the updated layers to client <NUM> layer by layer, similar to operation <NUM>.

At <NUM>, the client <NUM> may receive the first residual layer from the server <NUM>.

At <NUM>, the client <NUM> may execute, run, or infer the first layer of the updated neural network, for example the second compressed neural network, similar to operation <NUM>.

The server <NUM> may be configured to iterate transmitting the update layers at <NUM> until all residual layers of the updated neural network have been delivered to client <NUM>, similar to operation <NUM>.

The client <NUM> may iterate receiving the updated layers <NUM> and executing the updated layers <NUM> until all layers of the updated neural network have been executed, similar to operation <NUM>.

At <NUM>, the client <NUM> may transmit the output of the final layer of the updated neural network, for example the second compressed neural network, to the third-party device <NUM>, similar to operation <NUM>.

The example embodiments of <FIG> or <FIG> may be generalized to a dynamic use case, where available bandwidth and/or memory changes over time. The server <NUM> and/or client <NUM> may be therefore configured monitor bandwidth and/or memory limitations over time. According to an example embodiment, the procedure of delivering an updated compressed neural network may be triggered based on a change in available bandwidth and/or memory. For example, in response to reallocation of memory at client <NUM>, for example as a result of launching another application <NUM>, the client <NUM> may transmit a request message to obtain a more compressed neural network from server <NUM>.

Other requirements that may be considered when providing compressed neural networks include computational complexity, which may determine speed of inference. For example, client <NUM> may require a neural network that is able to perform an inference step on its hardware within a period, for example within <NUM>. Therefore, a request for a neural network may comprise a required time period for performing inference with the neural network. The time period may be expressed for example with respect to a reference hardware, for which an execution speed is known. According to an example embodiment, the request may indicate a computational complexity, for example a maximum complexity that is not to be exceeded. The computational complexity may be expressed for example as number of operations, for example multiplication or additions, needed to perform inference.

Example embodiments improve delivery of compressed neural networks between devices. Example embodiments are beneficial for example when the devices do not share the same dataset for evaluating performance of the compressed neural network or when the client device is not able to perform the evaluation. Delivering a residual of the updated neural network with respect to a previous neural network reduces required transmission resources. Layer wise delivery of the compressed layers enables the compressed neural networks to be executed with lower memory requirements. Applying the residual may be useful when optimizing the size of the compressed neural network iteratively. Example embodiments further enable dynamically adapting resources needed for performing inference on a neural network.

<FIG> illustrates an example of a method <NUM> for transmitting neural network(s) to a client, according to an example embodiment.

At <NUM>, the method comprises receiving a request for a neural network from a client, wherein the request comprises at least one requirement for the neural network.

At <NUM>, the method comprises obtaining a first compressed neural network based on the at least one requirement.

At <NUM>, the method comprises transmitting the first compressed neural network to the client device.

At <NUM>, the method comprises obtaining a residual between at least a subset of the first compressed neural network and at least a subset of the second compressed neural network in response to receiving, from the client or a third-party device, an indication that the first compressed neural network does not satisfy a performance requirement.

At <NUM>, the method comprises transmitting the residual to the client; or updating the subset of the first compressed neural network based on the residual to obtain a subset of the second compressed neural network and transmitting the subset of the second compressed neural network to the client.

<FIG> illustrates an example of a method <NUM> for requesting and receiving neural network(s) form a server, according to an example embodiment.

At <NUM>, the method comprises transmitting a request for a neural network, wherein the request comprises at least one requirement for the neural network.

At <NUM>, the method comprises receiving a first compressed neural network.

At <NUM>, the method comprises providing the first compressed neural network for evaluation, wherein the evaluation comprises determining whether the first compressed neural network satisfies a performance requirement.

At <NUM>, the method comprises receiving a residual between the first compressed neural network and a second compressed neural network and reconstructing the second compressed neural network based on the first compressed neural network and the residual; or receiving a subset of the second compressed neural network.

Further features of the methods directly result from the functionalities and parameters of the server <NUM> and/or client <NUM> as described in the appended claims and throughout the specification and are therefore not repeated here.

An apparatus, for example server <NUM> or client <NUM> may be configured to perform or cause performance of any aspect of the method(s) described herein. Further, a computer program may comprise instructions for causing, when executed, an apparatus to perform any aspect of the method(s) described herein. Further, an apparatus may comprise means for performing any aspect of the method(s) described herein. According to an example embodiment, the means comprises at least one processor, and memory including program code, the at least one processor, and program code configured to, when executed by the at least one processor, cause performance of any aspect of the method(s).

Rather, the specific features and acts described above are disclosed as examples of implementing the claims.

The steps or operations of the methods described herein may be carried out in any suitable order, or simultaneously where appropriate. Aspects of any of the embodiments described above may be combined with aspects of any of the other embodiments described to form further embodiments without losing the effect sought.

The term 'comprising' is used herein to mean including the method, blocks, or elements identified, but that such blocks or elements do not comprise an exclusive list and a method or apparatus may contain additional blocks or elements.

Although subjects may be referred to as 'first' or 'second' subjects, this does not necessarily indicate any order or importance of the subjects. Instead, such attributes may be used solely for the purpose of making a difference between subjects.

As used in this application, the term 'circuitry' may refer to one or more or all of the following: (a) hardware-only circuit implementations (such as implementations in only analog and/or digital circuitry) and (b) combinations of hardware circuits and software, such as (as applicable): (i) a combination of analog and/or digital hardware circuit(s) with software/firmware and (ii) any portions of hardware processor(s) with software (including digital signal processor(s)), software, and memory(ies) that work together to cause an apparatus, such as a mobile phone or server, to perform various functions) and (c) hardware circuit(s) and or processor (s), such as a microprocessor (s) or a portion of a microprocessor(s), that requires software (e.g., firmware) for operation, but the software may not be present when it is not needed for operation.

Claim 1:
A server device, comprising:
at least one processor; and
at least one memory including computer program code;
the at least one memory and the computer program code configured to, with the at least one processor, cause the server device at least to:
receive a request for a neural network from a client device, wherein the request comprises at least one requirement for the neural network;
obtain a first compressed neural network based on the at least one requirement;
transmit the first compressed neural network to the client device;
obtain a residual between at least a subset of the first compressed neural network and at least a subset of a second compressed neural network in response to receiving, from the client device or a third-party device, an indication that the first compressed neural network does not satisfy a performance requirement, wherein the first compressed neural network and the second compressed neural network are configured to perform a same task, wherein the second compressed neural network is obtainable based on adding the residual to the first compressed neural network, and wherein a size and performance of the second compressed neural network are higher than a size and performance of the first compressed neural network; and
transmit the residual to the client device.