Patent Description:
Many practical applications rely on the availability of semantic information about the content of the media, such as images, videos, audio etc. Semantic information may be represented by metadata which may express the type of scene, the occurrence of a specific action/activity, the presence of a specific object, etc. Such semantic information can be obtained by analyzing the media.

Recently, the development of various neural network techniques has enabled learning to recognize content directly from the raw data. Neural networks are computerized models that comprise an input layer and an output layer. In addition, the neural networks comprise layered units that form input data into data that is usable by the output layer.

A publication <CIT> discloses a solution receiving a compressed convolutional neural network (CNN). A media content item to be processed can be acquired. The compressed CNN is utilized to apply a media processing technique to the media content item to produced information about the media content item. It is determined, based on at least some of the information about the media content item, whether to transmit at least a portion of the media content item to one or more remote servers for additional media processing.

Now there has been invented an improved method and technical equipment implementing the method, for neural networks. Various aspects of the invention include a method, and an apparatus, which are characterized by what is stated in the independent claims. Various embodiments of the invention are disclosed in the dependent claims.

In the following, various embodiments will be described in more detail with reference to the appended drawings, in which.

In the following, several embodiments of the invention will be described in the context of neural networks. Neural networks have recently prompted an explosion of intelligent applications for loT (Internet of Things) devices, such as mobile phones, smart watches and smart home appliances. Albeit transferring data to a centralized computation server for processing is appealing, considering the high computational complexity and battery consumption, concerns over data privacy and latency of large volume data transmission have been promoting distributed computation scenarios. To this end, it is beneficial to develop and standardize common communication and representation formats for neural networks in order to enable the efficient, error resilient and safe transmission and reception among device or service vendors.

<FIG> shows an apparatus representing a client device for the purposes of the present embodiments. The functional blocks of the apparatus can also be found in a server according to the present embodiment. The generalized structure of the apparatus will be explained in accordance with the functional blocks of the system. Several functionalities can be carried out with a single physical device, e.g. all calculation procedures can be performed in a single processor if desired. A data processing system of an apparatus according to an example of <FIG> comprises a main processing unit <NUM>, a memory <NUM>, a storage device <NUM>, an input device <NUM>, an output device <NUM>, and a graphics subsystem <NUM>, which are all connected to each other via a data bus <NUM>. A client may be understood as a client device or a software client running on an apparatus.

The main processing unit <NUM> is a processing unit arranged to process data within the data processing system. The main processing unit <NUM> may comprise or be implemented as one or more processors or processor circuitry. The memory <NUM>, the storage device <NUM>, the input device <NUM>, and the output device <NUM> may include other components as recognized by those skilled in the art. The memory <NUM> and storage device <NUM> store data in the data processing system <NUM>. Computer program code resides in the memory <NUM> for implementing, for example, machine learning process. The input device <NUM> inputs data into the system while the output device <NUM> receives data from the data processing system and forwards the data, for example to a display. While data bus <NUM> is shown as a single line it may be any combination of the following: a processor bus, a PCI bus, a graphical bus, an ISA bus. Accordingly, a skilled person readily recognizes that the apparatus may be any data processing device, such as a computer device, a personal computer, a server computer, a mobile phone, a smart phone or an Internet access device, for example Internet tablet computer.

Deep learning is a solution for analyzing data and is a sub-field of machine learning which has emerged in the recent years. Deep learning is a field, which studies artificial neural networks (ANN), also referred to as neural network (NN). A neural network is a computation graph representation, usually made of several layers of successive computation. Each layer is made of units or neurons computing an elemental/basic computation. Deep learning may involve learning of multiple layers of nonlinear processing units, either in supervised or in unsupervised manner, or in semi-supervised manner. Each learned layer extracts feature representations from the input data. Features from lower layers represent low-level semantics (i.e. less abstract concepts, such as edges and texture), whereas higher layers represent higher-level semantics (i.e., more abstract concepts, like scene class). Unsupervised learning applications typically include pattern analysis and representation (i.e., feature) learning, whereas supervised learning applications may include classification of image objects (in the case of visual data).

Deep learning techniques may be used e.g. for recognizing and detecting objects in images or videos with great accuracy, outperforming previous methods. In addition, deep learning techniques are utilized in an ever-creasing number of applications for any type of device, such as an apparatus of <FIG>. Examples of the applications include various media (video, image, audio) analysis and processing, social media data analysis, device usage data analysis, etc. One difference of deep learning image recognition technique compared to previous methods is learning to recognize image objects directly from the raw data, whereas previous techniques may be based on recognizing the image objects from hand-engineered features (e.g. SIFT features).

Deep learning expects training in order to be able to perform the expected analysis. During the training stage, deep learning techniques build computation layers which extract features of increasingly abstract level. Thus, at least the initial layers of an artificial neural network represent a feature extractor. An example of a feature extractor in deep learning techniques is included in the Convolutional Neural Network (CNN), shown in <FIG>. This example of a CNN comprises one or more convolutional layers, fully connected layers, and a classification layer on top. CNNs are relatively easy to train compared to other deep neural networks and have fewer parameters to be estimated. Therefore, CNNs are highly attractive architecture to use, especially in image and speech applications.

In the example of <FIG>, the input to a CNN is an image, but any other data could be used as well. Each layer of a CNN represents a certain abstraction (or semantic) level, and the CNN extracts multiple feature maps. A feature map may for example comprise a dense matrix of Real numbers representing values of the extracted features. The CNN in <FIG> has only three feature (or abstraction, or semantic) layers C1, C2, C3 for the sake of simplicity, but CNNs may have more than three convolution layers.

The first convolution layer C1 of the CNN may comprise extracting <NUM> feature-maps from the first layer (i.e. from the input image). These maps may represent low-level features found in the input image, such as edges and corners. The second convolution layer C2 of the CNN, which may extract <NUM> feature-maps from the previous layer, increases the semantic level of the extracted features. Similarly, the third convolution layer C3 may represent more abstract concepts found in images, such as combinations of edges and corners, shapes, etc. The last layer of the CNN, referred to as fully connected Multi-Layer Perceptron (MLP) may include one or more fully-connected (i.e., dense) layers and a final classification layer. The MLP uses the feature-maps from the last convolution layer in order to predict (recognize) for example the object class. For example, it may predict that the object in the image is a house.

The goal of a neural network is to transform the input data into a more useful output. One example is classification, where input data is classified into one of N possible classes (e.g., classifying if an image contains a cat or a dog). Another example is regression, where input data is transformed into a Real number (e.g. determining the music beat of a song).

The power of neural networks comes from the internal representation which is built inside the layers. This representation is distributed among many units and is hierarchical, where complex concepts build on top of simple concepts. A neural network has two main modes of operation: training phase and testing phase. The training phase is the development phase, where the neural network learns to perform the final task. Learning may include iteratively updating the parameters of the neural network, for example weights or connections between units. The testing phase is the phase in which the neural network performs the task. Learning can be performed in several ways. The main ones are supervised, unsupervised, and reinforcement learning. In supervised training, the neural network model is provided with input-output pairs, where the output may be a label. In supervised training, the neural network is provided only with input data (and also with output raw data in case of self-supervised training). In reinforcement learning, the supervision is sparser and less precise; instead of input-output pairs, the neural network gets input data and, sometimes, delayed rewards in the form of scores (E. , -<NUM>, <NUM>, or +<NUM>).

The neural network may be trained on a training data set, which is supposed to be representative of the data on which the neural network will be used. During training, the neural network uses the examples in the training dataset to modify its learnable parameters (e.g., its connections' weights) in order to achieve the desired task. Input to the neural network is the data, and the output of the neural network represents the desired task. Examples of desired tasks are classification of objects in images, denoising of images (or other types of data, such as heart-rate signals), semantic segmentation. For such tasks, the output of the neural network may be a probability distribution over object classes for the whole image, a denoised image, a probability distribution over classes for each input pixel, respectively.

Convolutional Neural Networks may be used in various applications. However, they have a huge data storage requirement, whereupon a compressed representation of neural networks (NN) is a preferred feature. The compressed representation of NNs includes core neural network compression features that, for example, represent different artificial neural network types (e.g., feedforward neural networks such as CNN and autoencoder, recurrent neural networks such as LSTM (Long short-term memory), etc.); enable efficient incremental updates of compressed representations of NNs; enable scalability, i.e. NNs of different performance can be obtained from subsets of the complete compressed representation; inference with compressed neural network, enable use under resource limitations (computation, memory, power, bandwidth).

Various compression methods include, for example, a weight quantization; lossless source coding; pruning, where less contributing weights are removed, SVD (Singular Value Decomposition); filter selection; structure design. While compression reduces the size of the neural network, it may adversely affect performance of the neural network. Therefore, there may be a performance deterioration value or level associated with a compression method and/or a compression ratio. Deterioration of the compressed neural network may be determined by comparing performance metrics of the original and the compressed neural network, as disclosed in various embodiments.

Sharing trained neural network models or parts of deep neural networks has been very important practices in the rapid progress of research and development of AI systems. At the same time, it is imperative to share NN models, which are often complex and computational resources demanding, in an efficient and manageable manner.

The present embodiments are targeted to a NN representation that supports intelligent and flexible sharing of compressed neural networks as web-based services in cloud environments. Thus, the present embodiments improve the provision of customized neural networks to clients.

The present embodiments are generally related to a communication between a client and a server, where a client receives a compressed neural network from a server, which compressed neural network has been generated according to client specified requirement(s). In one embodiment, the server may cause generation of the compressed neural network in response to receiving the request from the client. Alternatively, the server may have caused generation of the compressed neural network prior to receiving the request from the client. In this case, the server may select an available compressed neural network that substantially complies with the request.

According to a first example, the client comprises a neural network for a certain task. The client uploads the neural network with specified requirement(s) to the NN server that is configured to perform a compression of the neural network according to the requirement(s). The compressed neural network, or at least an identification for accessing the compressed neural network, is transmitted to the client, whereupon the client is able to operate with the compressed neural network or to share it forward, e.g. within an application, or to cause the neural network to perform a desired task.

According to a second example, the client may not have a neural network, but needs one for a specific task. The client may send a request for the NN server, wherein the request comprises at least parameters for a needed neural network. If there is a neural network (e.g. VGG16, ResNet51, DenseNet, etc.) available in a public database, the client may also include an indication/pointer/URL to such neural network. The server either selects a suitable or identified neural network from the database or trains a new neural network if a suitable neural network (i.e. a NN that mostly matches with the requirements) is not found. Such a selected neural network is then compressed and transmitted in the compressed form to the client. The client is then able to operate with the compressed neural network or share it forward, e.g. within an application, or to cause the neural network to perform a desired task.

According to an embodiment, the client is an AI capable smartphone. AI related neural networks may be deep neural networks, which are very big in size (for example <NUM> GB (gigabytes)). Distribution of such a NN as an application or operating system update to millions of smartphones may not be feasible. Therefore, a compression of a neural network to a reasonable size with reasonable loss of performance may be needed. The "NN Compression as a service" platform according to present embodiments is able to perform the compression of the neural network (which is either received from the client; or selected from a database; or generated as a response to the request) and transmit the compressed NN as a response to the client. The server may have specified the final parameters about a performance of the neural network, etc., in the response. After having received the compressed neural network, the client may distribute it within an application or platform. Since the NN compression may use hardware and storage resources that may need to be scaled, the compression service can be implemented as a cloud service that can be accessed by a web service.

<FIG> illustrates an overview of a use case and a diagram of system components for NN-on-demand services. In the <FIG>, NN receiver <NUM> represents a client device requesting a neural network from a server. The NN receiver <NUM>, e.g. a client, is configured to perform a task e.g. analyzing and processing of various media (video, image, audio), analyzing of social media data or of device usage data, just to mention few as examples, by means of a neural network. However, as discussed in the example <NUM>, the NN receiver <NUM> may not permanently comprise any neural networks of its own, for example for memory saving purposes. Therefore, the NN receiver <NUM> is configured to request for a suitable neural network for a certain task from a server. Alternatively, according to example <NUM>, the NN receiver <NUM> has a neural network, compression of which the NN receiver <NUM> is configured to ask from the server.

The NN receiver <NUM> may include at least one or more requirements for the neural network and/or for the compression of the neural network in the request. In addition, according to an example <NUM>, the NN receiver <NUM> may include the original neural network to the request or an identification of one that should be compressed. The requirement(s) may be defined in a performance requirement matrix <NUM>, and they can comprise for example the required inference performance, compression ratio, or time of delivery, etc. The requirement(s) may also be provided in other data structure or format, for example one or more tables, lists, or files. For simplicity, embodiments have been described as using a performance requirement matrix as an example. The request may be sent to the NN provider through data communication network <NUM>. Based on the request from the NN receiver <NUM>, the NN provider, which may be for example a server or another type of computing apparatus such as for example a user device, a mobile phone, or a tablet computer, is configured to obtain a suitable neural network for compression. The obtaining can be achieved by selecting a suitable neural network from the a set of original neural networks <NUM>; by training a new version of the neural network to be the suitable neural network, or to obtain the neural network from the request of the client. The term original neural network may be understood as a neural network which is subject to compression. It is appreciated that such original network may have been obtained based on prior operations performed on the original neural network. For example, the original neural network may be a fine-tuned version of another neural network, or it may have been compressed and decompressed earlier.

After having obtained a suitable neural network, the NN provider is configured to perform a compression <NUM> of the neural network to have a compressed representation of the neural network, i.e. compressed neural network <NUM>. According to another embodiment, the server may request the compressed version of the neural network from another party, for example from another server over a communication network. The compression <NUM>, which is discussed in more detailed manner with reference to <FIG>, can be implemented according to NNR (Neural Network Representation) standards <NUM>. The compressed neural network <NUM>, or at least some information identifying and/or accessing it, is then provided to the NN receiver <NUM> through a communication network <NUM>. Accessing a neural network may be understood for example as providing instructions for executing the neural network, or, downloading, updating, or modifying the neural network, or parts thereof. For example, a client may access neural network located at a server to cause execution of the neural network by the server. The client may also access the neural network by downloading it for execution at the client. Downloading may be done in streaming mode, for example, the client may initiate execution of some layers of the neural network, while other layers are still being downloaded from the server. In another example, a server may access a neural network located at the server or another server to update or modify the neural network, for example based on training carried out or caused by the server. In response to receiving the compressed neural network or metadata associated with the compressed neural network, the NN receiver <NUM> may be configured to determine whether the received compressed neural network is satisfactory. The determining can be based on the information provided in the performance requirement matrix <NUM>. In addition, the performance requirement matrix can be updated by the compression server with the actual metrics for the compressed neural networks, when the compression has been applied.

If the neural network compression process is unsatisfactory based on the requirements, the client device <NUM> may request for a new neural network from the server by means of a new request with new requirements.

If the received neural network is in accordance with the requirements of the client device <NUM>, the client device <NUM> is configured to perform the expected task by means of it by using the local training datasets <NUM>.

As said, the communication between the NN receiver <NUM> and the NN provider is enabled through Internet to which the NN receiver <NUM> and the NN provider are able to access via a communication network <NUM>. The communication network can be a wireless or wired local area network, a mobile telecommunications network, for example Global System for Mobile Communications (GSM), Genera IPacket Radio Service (GPRS), cdmaOne, CDMA2000 (Code-Division Multiple Access), Evolution-Data Optimized (EV-DO), Enhanced Data Rates for GSM Evolution (EDGE), Universal Mobile Telecommunications System (UMTS), Digital Enhanced Cordless Telecommunications (DECT), Digital AMPS (IS-<NUM>/TDMA), Integrated Digital Enhanced Network (iDEN). Therefore, both the client device (i.e. NN receiver <NUM>) and the server device (which is the NN provider in this example) comprises network-specific means for connecting to the respective data communication network <NUM> and means for transferring data through said data communication network <NUM>, in addition to the functional elements discussed in connection to <FIG>.

<FIG> illustrates an example of a compression, referred to with a reference number <NUM> in <FIG>. The compression begins by obtaining an original neural network(s) <NUM>, which is(are) compressed according to any compression approach, e.g. pruning, SVD, filter selection, structure design. The compression results in a compressed NN model <NUM>, that can be further fine-tuned by fine-tuning the compression method. The compressed model <NUM> may be quantized to result in a quantized NN model <NUM>. Finally, the quantized NN model <NUM> is encoded to obtain a coded model <NUM> of the neural network.

An exchange format for neural network compression according to an embodiment is illustrated in <FIG>. Another embodiment for the exchange format is illustrated in <FIG>.

<FIG> illustrates an embodiment for the example <NUM>, relating to a Rest API based exchange format of compressed NN between client and server. REST API, HTTP and other protocols are provided as examples and may be used in exchange of other similar network based processing and access protocols. <FIG> shows a NN storage server, a client, a NN compression server and a compressed NN storage server. In this embodiment, a client requests a NN compression service to compress a neural network. The request comprises a link to, or another identification of, the desired (original) neural network. The request may also include configuration parameters or descriptions according to which the compression is to be performed. The NN compression service requests the indicated neural network from a NN storage server and receives the NN model data as a response. The client device is configured to request the compressed NN from the NN compression service. If the compressed neural network is not ready, the NN compression service may ask the client to retry the request and may include an estimated finalization time to the response. If the compressed neural network is ready, the NN compression service may include the final configuration parameters, for example an identification of the compressed neural network, to the response, whereupon the client is able to ask the compressed NN from the compressed NN storage server.

<FIG> illustrates another embodiment for the example <NUM>, that relates to a storage format based message exchange. <FIG> shows a client and a server, wherein a client sends a file, for example a media file, containing a NN to be compressed and/or configuration parameters associated with the neural network and/or the desired compression to the server. Then the client sends a request to access the compressed neural network. The server responds with an OK if the compressed neural network is ready, wherein the response may comprise also final configuration parameters of the compressed neural network. If the neural network is not ready, the server response indicates (but not limited to) "Resource not available" or a similar message. It is appreciated that even though content of a request or response message may have been illustrated to be included in a single message, in some embodiments such information may be distributed between more than one request or response, for example at least two consecutive requests or responses. For example, a first response message may comprise an estimated time of delivery of the neural network indicated in a request message. A second response message may comprise and indication of a performance of the neural network. The second response message may be sent after completing generation of the requested neural network.

In a request from a client device <NUM> (which is an example of a NN receiver) to a server (example of a NN provider), the exchange format comprises any one or more of the following fields relating to:.

The fields of the exchange format from client to server are discussed in more detailed manner in below:
Descriptions of required performance metric: Depending on different compression ratios, the performance of original NNs <NUM> may be compromised to various extents. A desired performance metric may therefore include pairs of (compression ratio and performance deterioration, e.g.;.

It is appreciated that the compression rations may be specified by the client device <NUM>, for example according to the available computational resources, e.g. the available memory on GPUs to run requested neural networks. The required NN model size may be expressed e.g. in MB (megabytes). Compression ratio may be calculated according to the sizes of the compressed and original NN models, for example ratio=compressed_size / original_size.

Deterioration level(s) may be specified by the client device <NUM>. According to an embodiment, the desired deterioration level equals to the original performance minus the acceptable performance for corresponding compression ratio, i.e. desired_deterioration_level=original_performance - acceptable performance. Alternatively, the deterioration level can be defined as the ratio of the acceptable performance with respect to the original performance, i.e. deterioration_level=acceptable-performance/original performance.

How to determine the performance depends on the application/task of the neural network. For example, for image classification tasks, the accuracy may be measured by the ratio between correctly_classified_image with respect to all_images. For semantic segmentation tasks, the accuracy may be measured by the Intersection over Union (IoU) ratio, i.e. accuracy =intersection /union, wherein the intersection corresponds to predicted segmentation labels, ground-truth labels, and union corresponds to the predicted segmentation labels, ground-truth labels.

According to yet another embodiment, the deterioration level can be determined by the service provider.

Description of required model size(s): The client device (or other requesting device) may have a limited storage capacity (in byte size) for the NN in the run-time environment. Thus, by specifying the target model size for the NN, together with the deterioration_level, the client device can provide the requirement for the server to apply a compression so that the NN does not exceed the target byte size, as well as meet the deterioration_level target. These two targets may be conflicting, whereupon it may turn into a compression optimization problem on the server side. Multiple NN models with different sizes (below or slightly exceeding the provided model size target) may be returned by the server, letting the client to make the final decision.

Description of required computational complexity(ies): According to an embodiment, a client device <NUM> may also one or more computational complexities that are expected from the compressed neural network(s). For example, a number of floating-point operations (FLOPs) may be defined in the request. The number of FLOPs may for example define a maximum number of operations that are allowed to be assigned to the requested neural networks is specified.

Description(s) of local datasets: According to an embodiment, a client device <NUM> may specify either proprietary or public datasets that can be used to train, fine-tune or prune the original NNs <NUM>. A reference field in the request may include e.g. a unique identifier or a link to such dataset. Optionally, metadata of the datasets may also be specified to facilitate the training/re-training/pruning of the original NNs <NUM>. The metadata may include various types of information about the dataset, for example, one or more data types of the dataset (e.g. image, video), one or more file formats of the datasets of the like.

Information on the local datasets may be sent to the NN service provider, so that the estimation of computational complexities, time and probabilities of success can be calculated accordingly by the NN provider.

Descriptions of required NN tasks and compression methods: This field may specify one or more tasks to be performed by the required NNs. Optionally, this field may also specify one or more compression methods, e.g. pruning, sparsifying and/or channel-coding (e.g. entropy coding, Huffman coding or CABAC coding).

Based on a request received from the client <NUM>, the NN provider may determine one or more neural networks to be provided to the client, unless already included in the client's request. In one embodiment, the server selects a neural network that meets, or at least substantially meets, the request of the client. In one embodiment, the NN provider may train a new neural network, or a new version of a neural network based on client's requirements specified in the request and/or the information in the client's request. In a response from the server (or, in general any NN provider) to client device (or in general any NN receiver), the exchange format may comprise any one or more of the following fields relating to:.

Fields of the exchange format may be also referred to as configuration parameters. Configuration parameters may be related to a requested neural network or a provided neural network.

The fields of the exchange format are discussed in more detailed manner in below:
Descriptions of returned NN handles (or links): This field may specify one or more program handles to the one or more returned neural networks. In general, a handle may comprise a reference for accessing the neural network. In one embodiment, the handle may be used as a function call to invoke execution of the neural network. The neural network may be executed internally at the client or the client may cause remote execution of the neural network, for example at the NN provider server or another server. Such a handle can be for example directly used by the program scripts running on the client side <NUM>. The availability of NNs, however, may be determined by other fields elaborated below.

According to an embodiment, such a handle can be efficiently and uniquely implemented as a <NUM>-bit universally unique identifier (UUID), which can be directly referenced in program scripts of different programming languages, e.g. python or JavaScript. Therefore, in one embodiment the identification of the compressed neural network received in the response message from the server may be included, e.g. copied, in one or more predetermined locations in a program code. This may be done to access the identified neural network, for example to cause execution of it. A neural network handle, a link to a neural network, or an UUID are examples of means for identifying a neural network.

Descriptions of estimated time when returned NNs are active: Since the required NN (e.g. a NN with required performance) may not be available on the server, it may initiate one or more training processes to train or fine-tune the required NN. This field may specify the estimated date and time by which the required NN is expected to be available. For example, this field may adopt the Coordinated Universal Time (CUT) for this purpose. A client may use this information to access the neural network at the estimated time.

In terms of the estimations of this time, for example at least one of the two following categories of methods can be adopted:.

The server may initiate the training of requested NNs with the entire training dataset, for a few epochs, e.g. <NUM>, and measure the required time e.g. <NUM> minutes, the estimated time of the entire training process may then be calculated as the number of epochs * <NUM> minutes_per_epoch.

Optionally, the server may initiate the training of requested NNs with a subset of the training data, e.g. <NUM>%, and measure the required time, the estimated time of the entire training process may then be calculated as the_measured_training_time * <NUM>.

Still, according to yet another embodiment, the server may initiate the training in a cloud environment with e.g. a thousand GPUs to be used for the actual training process, then the estimated time = the measured training/<NUM>.

Optionally, the combination of above two approaches can also be adopted.

Descriptions of estimated success probabilities of returned NNs: The probability that the NN provider is able to provide required NN(s) varies, according to the nature of required tasks, performance metric, or given training datasets. if the requested ratio of compression by pruning is too high, it may be possible that the required NNs end up with NO filters kept at certain layers, thus destructing the entire NNs.

For example, at least one or the two following types of approaches can be used to estimate the probability of success:.

According to an embodiment, the ratio of r pruning filters may be used to determine the probability of success. This dependency can be succinctly represented by the formula: <MAT> in which r is the pruning ratio, and p is the probability of success. Specifically, four points (<NUM>, <NUM>), (r0, p0), (r1, p1), (<NUM>, <NUM>) jointly define a piece-wise linear model with three line segments connecting these four points.

The actual parameters (r0, p0), (r1, p1) are different according to the training dataset, the requested neural network architectures etc., and may be estimated from historical data.

Once the model parameters are estimated, then it can be used to estimate the probability of success for given pruning ratios.

Optionally, the combination of above two approaches can also be adopted. The client may use the probability of success to determine whether to wait for training the new neural network, or to issue an updated request with updated requirements. The estimated time for providing the new neural network may be also used for the same purpose. For example, the client may be configured to wait for the new neural network, if the estimated time of delivery is within a predetermined time, for example a minute or an hour, and/or if the estimated probability of success is at least a predetermined percentage, for example <NUM> %. Otherwise, the client may send the updated request. For example, the client may decide to allow higher compression ratio (less compression), a higher memory, higher complexity, and/or higher deterioration for the new neural network. The client may issue a new request without waiting for provision of the previously requested neural network.

Descriptions of estimated and actual performance matrix: Although clients may specify required performance metric, the actual performance metric of the returned neural network may vary. Also, the estimated performance metric on the server side may be different too. This field may specify the estimated and/or actually measured performance metric, to be returned to the requesting client.

Depending on the actual performance metric, the returned NN model(s) may be suitable for deployment or not. Depending on the performance metric of the returned neural network, the client may decide to pursue different actions, e.g. to accept the returned NN model if it is satisfactory, or to re-issue a request with modified performance metrics if the returned NN model does not fulfill the requirement.

Descriptions of actual model sizes: In the same vein, the actual NN model size may be different from the requested ones. This field may specify the actual model size(s) of the returned neural network.

Depending on the returned NN model size, it may be suitable to be deployed, or in case the returned NN model size is too big, the request has to be re-issued, possibly with compromised performance metric.

The term "returned neural network" may refer to a neural network selected or trained by the NN provider. Subsequently this neural network may or may not be provided to the client. According to an embodiment, the server may send a message to the client to inform the client about the new neural network, for example at least one parameter of the returned neural network, such as performance. The server may provide the returned neural network to the client in response to receiving an acceptance message from the client. According to an embodiment, the server may determine that a neural network that is already available substantially meets the requirement(s) of the client and may provide the neural network or inform the client about this neural network. The term 'substantially meeting the requirements' may be understood as meeting most of the requests, for example meeting most of the requirements but failing to meet a predetermined number of requirements, and/or as almost meeting individual requests, for example failing to meet all requests by a small margin, for example a few per cents in terms of memory size or complexity. The client may therefore select between fast delivery of a suboptimal neural network or waiting for provision of a neural network that fulfils the requirements.

Description(s) of actual NN compression method(s) and parameter(s): For example, if the compression method comprises pruning, this method may be also associated with certain parameter(s), such as for example to which layer it should be applied, a pruning technique, or threshold(s), etc. These may also overlap with the performance matrix to some extent. As another example, if the compression method comprises parameter binarization, the method may include quantization of the neural network model weights. Different quantization levels can be applied to different layers, e.g. one layer may have eight bit precision, while another may have four bit precision. Such configurations can be pre-selected by the client and sent via parameters of selected NN compression method to the NN provider.

<FIG> is a flowchart illustrating a method according to an embodiment. A method comprises generating <NUM> a request for a neural network, wherein the generating comprises including into the request information on at least one requirement for the neural network; sending <NUM> the request to a server; receiving <NUM> from the server a response comprising at least means for identifying a compressed version of the requested neural network, said neural network having been compressed according to the at least one requirement; and causing <NUM> the neural network to perform a task, wherein the neural network is accessed by means of said identification.

An apparatus according to an embodiment is a client device comprising means for generating a request for a neural network, wherein the generating comprises including into the request information on at least one requirement for the neural network; means for sending the request to a server; means for receiving from the server a response comprising at least means for identifying a compressed version of the neural network, said neural network having been compressed according to the at least one requirement; and means for causing the neural network to perform a task, wherein the neural network is accessed by means of said identification. The means comprises a processor, a memory, and a computer program code residing in the memory. The processor may further comprise a processing circuitry.

<FIG> is a flowchart illustrating a method according to an embodiment. A method comprises receiving <NUM> a request for a neural network from a client, wherein the request comprises information on at least one requirement for the neural network; selecting <NUM> a neural network; compressing <NUM> the selected neural network according to the at least one requirement to obtain a compressed version of the selected neural network; generating a response, said response comprising at least means for identifying the compressed version of the neural network; and sending <NUM> the response to the client.

An apparatus according to an embodiment is a server, comprising means for receiving a request for a neural network from a client, wherein the request comprises information on at least one requirement for the neural network and/or for the compression of the neural network; means for selecting a neural network; means for compressing the selected neural network according to the at least one requirement to obtain a compressed version of the selected neural network; means for generating a response, said response comprising at least means for identifying the compressed version of neural network; and means for sending the response to the client. The means comprises a processor, a memory, and a computer program code residing in the memory. The processor may further comprise a processing circuitry.

The various embodiments of the invention can be implemented with the help of computer program code that resides in a memory and causes the relevant apparatuses to carry out the invention. For example, a device may comprise circuitry and electronics for handling, receiving and transmitting data, computer program code in a memory, and a processor that, when running the computer program code, causes the device to carry out the features of an embodiment. Yet further, a network device like a server may comprise circuitry and electronics for handling, receiving and transmitting data, computer program code in a memory, and a processor that, when running the computer program code, causes the network device to carry out the features of an embodiment. The computer program code may reside on a non-transitory computer readable medium.

Claim 1:
A method executed by a client device, comprising:
- generating a request for a compressed neural network, wherein the generating request comprises at least an indication of an original neural network for a certain task, and including into the request the following requirements
a description of a task of the neural network;
a desired performance of the compressed neural network executing the task, wherein the desired performance of the compressed neural network is defined by a compression ratio and a deterioration level, where the compression ratio is specified according to computational resources of the client device, and wherein the used compression method is at least one of the following: a weight quantization, lossless source coding, pruning, singular value decomposition, filter selection, or structure design;
- sending the request to a server;
- receiving from the server a response comprising at least an identification for accessing at least one compressed neural network, said compressed neural network having been generated by the server with respect to an original neural network according to said requirements; and
- obtaining the compressed neural network by means of said identification, and
- running the compressed neural network to perform the task.