Patent Description:
One or more embodiments may be applied to a hardware accelerator engine configured to perform ANN processing, such as neural processing units (NPUs), for instance.

A convolutional (artificial) neural network (briefly, CNN) comprises computer-based tools which exploit deep learning algorithms to perform image processing tasks.

Classifying the image of a big feline as a leopard or a jaguar may be mentioned as an example of such processing.

A CNN comprises a plurality of layers, for instance a plurality of (hidden) processing layers coupled to the input layer and configured to apply data processing to an image tensor received therefrom. Processing layers may comprise convolutional and/or pooling processing, for instance, and the CNN may comprise an output layer coupled to the input layer via the hidden layers.

Convolutional processing layers perform convolution of the input image data with filters, also known as convolution kernels, and apply an activation function to the convolved data, producing a set of features as a result.

Pooling processing layers reduces the dimensions of the set of features by performing subsampling, such as max or average sampling, for instance.

Processing layers may use up to millions of parametric values, also known as weights. Weight values are "learnt", that is pre-set, through a training processing phase which may involve large training datasets.

For instance, a CNN can be trained using a plurality of pictures of animals, and it can be configured to identify leopards or jaguars.

As mentioned, millions of weight values may be used in CNN processing, which may use large memory footprints.

Processing layers may involve large numbers of dot product operations between the weights and feature vectors, which may lead to high computation cost.

Existing solutions (as discussed, for instance, in <NPL>) investigate lossy compression of deep neural networks (DNNs) by weight quantization and lossless source coding for memory-efficient deployment, introducing "universal" DNN compression by universal randomized lattice quantization of DNNs, which randomizes DNN weights by uniform random dithering before lattice quantization and can perform near-optimally on any source without relying on knowledge of its probability distribution. The document cited presents a method of fine-tuning vector quantized DNNs to recover the performance loss after quantization.

Such a solution may exhibit various drawbacks such as:.

Reducing power consumption and costs of CNNs (facilitating Internet-of-Things (IoT) applications which may be based on edge computing, for instance) is thus a goal worth pursuing.

Processing circuits configured to perform ANN processing, such as neural processing units (NPU), for instance, may comprise hardware accelerator engines including a compression/decompression functionality associated to weights, possibly reducing memory storage constraints of CNNs.

Related hardware implementations may suffer from drawbacks such as:.

An object of one or more embodiments is to contribute in overcoming the drawbacks discussed in the foregoing. The invention is set out in the appended set claims.

According to one or more embodiments, such an object can be achieved by means of a method having the features set forth in claim <NUM> that follows.

One or more embodiments may relate to a corresponding computer program product.

To that effect, one or more embodiments may comprise a computer program product loadable in the memory of at least one processing circuit (e.g., a computer) and comprising software code portions for executing the steps of the method when the product is run on at least one processing circuit. As used herein, reference to such a computer program product is understood as being equivalent to reference to computer-readable medium containing instructions for controlling the processing system in order to co-ordinate implementation of the method according to one or more embodiments. Reference to "at least one computer" is intended to highlight the possibility for one or more embodiments to be implemented in modular and/or distributed form.

One or more embodiments may relate to a corresponding system (a HW accelerator system may be exemplary of such a system) having stored therein weights compressed with a method as exemplified herein.

One or more embodiments may relate to a corresponding method of de-compressing such weights and a corresponding computer program product.

The claims are an integral part of the technical teaching provided herein with reference to the embodiments.

One or more embodiments may reduce computational burdens related to performing an inference forward pass in CNN processing.

One or more embodiments may thus facilitate avoiding having large parallel computing data-paths associated to operations executed frequently, such as 3D tensor convolutions which may result in a large number of multiply-add-accumulate operations, e.g. proportional to the number of coefficients (weights) of a certain neural network.

One or more embodiments may provide one or more of the following advantages:.

One or more embodiments will now be described, by way of non-limiting example only, with reference to the annexed Figures, wherein:.

Throughout the figures annexed herein, like parts or elements are indicated with like references/numerals; for brevity a corresponding description will not be repeated for each and every figure.

Also, throughout this description, the wording "neural network (processing)" as used, for instance, in expressions like artificial neural network (ANN) processing or convolutional neural network (CNN) processing, is intended to designate machine-implemented processing of signals performed via hardware (HW) and/or software (SW) tools.

In addition to an input layer, configured to receive an input image tensor I with a certain size, for instance an image tensor I having a size given by the image width L times an image height (e.g., equal to the image width L) times an image depth (e.g., <NUM>) times a number of images in the tensor I (e.g., <NUM>), a convolutional neural network (CNN) <NUM> as illustrated in <FIG> may comprise a plurality of processing layers <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM> comprising:.

As illustrated, the processing layers <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, may be configured to produce respective feature maps F1, F2, F3, F4. Each such feature map may have a size given by a feature map width L1, L2, L3, L4 times a feature map height (which may be equal to the width L1, L2, L3, L4) times feature map channels (e.g., three channels for a RGB image having red, green and blue colors), times a number of maps.

In one or more embodiments, the processing layers <NUM>, <NUM>, <NUM>, <NUM>, <NUM> may have a multi-layer perceptron (briefly, MLP) architecture, comprising a plurality of processing units indicated as perceptrons.

A single i-th perceptron in the plurality of perceptrons may be identified by a tuple of values comprising weight values wi, offset values bi and an activation function ρi.

As exemplified in <FIG>, a convolutional processing layer, such as the one referenced as <NUM> (taken as an example of the various layers), comprises at least one convolution kernel (matrix) wi having a size which may be expressed as: <MAT> where.

The output layer <NUM> may comprise a fully connected layer, that is a type of convolutional layer having connections to all activations in the previous layer.

A convolutional layer such as <NUM> (again taken as a possible example) may be configured to apply an activation function to a sliding dot product.

Such an operation may be expressed as, for instance: <MAT> where.

As exemplified in <FIG>, a neural processing unit (NPU) <NUM> may comprise processing circuitry configured to perform CNN processing.

Document <CIT> discusses an NPU circuit suitable for use in accelerating CNN processing, for instance.

As exemplified in <FIG>, such a NPU circuit <NUM> comprises:.

In particular, weight values may be decoded starting from encoded weight values which may be encoded (or compressed) to reduce the memory footprint of running the CNN on the NPU <NUM>.

For instance, weight values may be encoded by applying quantization processing thereto.

Vector quantization techniques may be found suitable to perform such a quantization processing. For instance, a d-dimensional vector may be mapped to a finite set of vectors C = {ci: i = <NUM>, <NUM>,. , N}, where each vector ci may be indicated as a "codeword". The set of all the codewords may be indicated as "codebook". Each codeword may be associated to a nearest neighbour region indicated as "Voronoi region".

In lattice vector quantization (briefly, LVQ), the codebook may comprise points of a lattice, with centroids of the Voronoi regions used as approximating vectors.

As discussed herein, a lattice L of dimension d is a set of all integer linear combinations of basis vectors b<NUM>,. , bd in the d-dimensional Euclidean space, which may be expressed as: <MAT>.

In comparison other vector quantization techniques, LVQ may exhibit advantages such as:.

For the sake of simplicity, one or more embodiments are discussed in the following with respect to a trained CNN having pre-set weight values, being otherwise understood that such a scenario is purely exemplary and in no way limiting.

As exemplified at <NUM> in <FIG>, a method to compress weight values, may comprise:.

For the sake of simplicity, embodiments are mainly discussed herein with respect to a bidimensional lattice, (e.g., d=<NUM>), being otherwise understood that such a lattice dimension is purely exemplary and in no way limiting.

It is noted that in one or more embodiments it may be advantageous to select a value of the dimensional parameter d which is an integer multiple or divisor of the size of the kernels, as this may facilitate vectorization <NUM> of weight values, as discussed in the following.

As exemplified in <FIG>, a matrix of weight values wi of a convolutional layer, for instance that indicated by <NUM> in <FIG>, may be represented as a multi-dimensional grid wi whose elements are the weight values, the grid wi having a grid height H, a grid width T, a number of channels C and a number of kernels K. Any element of the multi-dimensional grid wi may be identified, for instance, by a set of indexes, one for each dimension, e.g., <MAT>.

For the sake of simplicity, <FIG> refer to an exemplary case where the matrix has a same height, width and number of channels, e.g., H=C=T=<NUM>, being otherwise understood that such exemplary values are purely exemplary and in no way limiting.

<FIG> are diagrams exemplary of performing vectorization of weight values <NUM>, which may comprise:.

Re-arranging weight elements of the matrix/grid of weights wi as exemplified in <FIG> may comprise:.

In the example considered, re-arranging the produced set of vectors u<NUM>, u<NUM>, u<NUM> may produce the matrix U having the first vector u1, second vector u2 and third vector u3 as respective first, second and third columns, which may be expressed as: <MAT>.

A way of collecting values from the grid elements as exemplified in <FIG> may be substantially follow a "boustrophedon" path along a certain dimension, e.g., the channel dimension C, collecting weight values while "plowing" in the grid and rearranging them in vectors whenever a number of "plows" equal to selected dimensional parameter d has been reached.

In one or more embodiments, the matrix U produced as a result of rearranging weight values <NUM> may be used in applying normalization processing <NUM>, which may comprise using stochastic gradient descent (SGD) to solve an optimization problem which may be expressed as: <MAT> where.

Specifically, the regularization term ΩL(u) may be expressed as: <MAT> where.

Such a regularization term ΩL(u) is designed to reach a minimum value if the k-th column uk of the matrix U is also a codeword of the lattice L.

The choice of the aforementioned optimization problem to solve may be based on the following rationale:.

<FIG> is a plot exemplary of the function ΩL(u) in a tridimensional Cartesian space.

<FIG> is a projection of the previous plot, Ωl(u) for different variance values σ<NUM>, σ<NUM>, σ<NUM> , e.g., σ<NUM> = <NUM>, σ<NUM> = <NUM>, σ<NUM> = <NUM>.

Optionally, when computing the regularization term ΩL(u), a further scaling factor λk may be applied to the matrix U, so as to improve matching the weight vectors to the selected lattice L.

<FIG> are diagrams showing possible different distributions of the elements of the matrix U (represented by filled circle points in the diagrams), when different scaling factors λ1, λ2, λ3 are applied thereto with respect to a bidimensional (d=<NUM>) lattice L having basis vectors b1, b2 which may be expressed as: b<NUM> = [<NUM>,<NUM>];b<NUM> = [<NUM>; <NUM>] in Cartesian coordinates (Voronoi region centroids of the lattice L are represented by cross points in the diagrams), e.g., λ1=<NUM>, λ2=<NUM>, λ3=<NUM>.

In one or more embodiments, decreasing the value of the scaling factor λ may increase a density of the distribution of elements of the matrix U with respect to the selected lattice L.

As exemplified herein, solving the normalization problem, with or without scaling, may provide a normalized matrix U' to further processing stages, such as the stage of performing lattice vector quantization, LVQ, <NUM>.

This may involve selecting a lattice L having a set of basis vectors as a function of the selected dimensional parameter d, the lattice L configured to be used as scheme of quantization, where the lattice L may comprise a finite set of points indicated as codebook CB, for instance CB may contain <NUM> lattice points having a lowest norm, where the lattice points are the codewords cw.

As exemplified herein, performing LVQ to normalized weights comprises mapping each column of the normalized matrix to a nearest codeword thereof in the codebook CB.

A method as discussed in document <NPL>, was found advantageous in performing such mapping, as appreciable to those of skill in the art. That document discusses a very fast algorithm for finding, for each of the lattices A_{n}(n geq <NUM>), D_{n}(n geq <NUM>), E_{<NUM>}, E_{<NUM>}, E_{<NUM>} and their duals, the closest lattice point to an arbitrary point, so that if these lattices are used for vector quantizing of uniformly distributed data, the algorithm finds the minimum distortion lattice point and if the lattices are used as codes for a Gaussian channel, the algorithm performs maximum likelihood decoding.

<FIG> is a diagram exemplary of stages of such mapping processing.

As exemplified in <FIG>, performing weight compression of any weight vector u<NUM>,u<NUM>,u<NUM>,uk involves:.

A method discussed in document <NPL>, was found to be advantageous in indexing lattice points <NUM>.

As exemplified herein, indexing lattice points <NUM> may comprise encoding any lattice point x to a tuple of indices ( <MAT>) which may comprise:.

As exemplified herein, indexing <NUM> may be performed in such a way that storing indexed codewords uses a reduced amount of memory with respect to storing the full codebook, resulting beneficial to the overall compression of the network weights.

A memory impact of uncompressed weight values may be estimated to be given by an amount of <NUM> bits per weight values (assuming float representation), this number of bits multiplied by the number of weights.

The memory footprint of using the method <NUM> and in particular of indexing <NUM> may be solely that used to store the LUT with the absolute leaders values and to store the tuple of indexes values, facilitating a reduced memory footprint of CNN processing. The NPU <NUM> may subsequently use compressed weights with a reduced memory to store such weights. Tables I and II in the following provide estimates of a memory impact of the method <NUM> as exemplified herein.

In alternative embodiments, value of leaders may be generated through a generative function in place of being stored in the LUT.

A method as discussed in document <NPL> may be found suitable for this purpose, as appreciable to those of skill in the art.

The following Table III may summarize how much time (in seconds) it may take to computationally generate respectively <NUM> and <NUM> absolute leaders in different dimensions.

As discussed in the foregoing, circuitry <NUM>, <NUM> of the NPU <NUM>, may be configured for:.

As exemplified herein, NPU circuits may be configured to perform a decompression method, walking backwards the steps of the (compression) method <NUM>, for instance. The method of decompression may be configured for co-operating with the inter-related method (of compression) <NUM>, for instance using same compression/decompression parameters, e.g., value of the dimension d of type of lattice L.

Such a method of decompression may have a low complexity from a hardware point of view, facilitating operation of a neural processing units not only in terms of memory footprint reduction associated to memory storage, but also positively impacting NPU performance by:.

As exemplified herein, decompressing weights "on-the-fly" or dynamically may refer to the possibility include the decompression logic performing the task to decode the incoming stream of compressed indexes directly into or attached to the hardware convolutional unit without the need for large intermediate buffers. This can significantly improve the performance of the NPU unit <NUM>, facilitating managing memory bottlenecks dominated critical paths when performing certain kinds of neural network workloads and operators such as Fully Connected (aka Matrix/vector multiply) and recurrent networks (RNN), Long Short Term Memory (LSTM) or Gated Recurrent Units (GRU), for instance.

A computer-implemented method (for instance, <NUM>) as exemplified herein comprises:.

As exemplified herein, said regularization term ΩL(u) is configured to reach a minimum value when said distance from said lattice points of the selected lattice of an item uk of the matrix of weight vectors is negligible.

As exemplified herein, said optimization problem is expressed as: <MAT> where.

As exemplified herein, said regularization term ΩL(u) is expressed as: <MAT> where:.

As exemplified herein, computing said regularization term ΩL(u) comprises applying to the matrix of normalized weight vectors a further scaling factor (for instance, λ<NUM>, λ<NUM>, λ<NUM>) , having a value between <NUM> and <NUM>, for instance.

As exemplified herein, said normalization processing comprises computing said optimization problem using stochastic gradient descent, SGD, processing.

As exemplified herein, performing said vectorization of weight values comprises:.

As exemplified herein, said LVQ processing comprises:.

As exemplified herein, said ANN processing stage is a convolutional neural network, CNN, processing stage.

A computer program product as exemplified herein comprises instructions which, when the program is executed by a computer, cause the computer to carry out the method as exemplified herein.

A computer-readable medium has exemplified herein has stored therein normalized weight values obtained using the method as exemplified herein.

A method of operating a hardware accelerator engine configured to perform artificial neural network, ANN processing as a function of weight values, as exemplified herein, comprises:.

A computer program product as exemplified herein comprises instructions which, when the program is executed by a computer, cause the computer to carry out the method of operating a hardware accelerator engine.

A computer-readable medium as exemplified herein, comprises instructions which, when executed by a computer, cause the computer to carry out the method of operating a hardware accelerator engine.

A hardware accelerator engine circuit (for instance, <NUM>) as exemplified herein, comprises memory circuitry having stored therein:.

It will be otherwise understood that the various individual implementing options exemplified throughout the figures accompanying this description are not necessarily intended to be adopted in the same combinations exemplified in the figures. One or more embodiments may thus adopt these (otherwise non-mandatory) options individually and/or in different combinations with respect to the combination exemplified in the accompanying figures.

Claim 1:
A computer-implemented method (<NUM>), comprising:
providing an artificial neural network, ANN, processing stage (<NUM>) comprising a plurality of processing layers (<NUM>, <NUM>, <NUM>, <NUM>, <NUM>) having respective processing layer parameters (L<NUM>, L<NUM>, L<NUM>, L<NUM>) , said processing layer parameters (L<NUM>, L<NUM>, L<NUM>, L<NUM>) including at least one set of weight parameters (wi), at least one input activation parameter (aj), at least one output activation parameter (bj ) and at least one activation function parameter (ρ),
setting a dimensional parameter (d) of a lattice to an integer value, said lattice having a plurality of lattice points and identified by a set of basis vectors (b<NUM>, b<NUM>),
selecting (<NUM>) a set of weight parameters (wi) of a respective processing layer (<NUM>) of said plurality of processing layers (<NUM>, <NUM>, <NUM>, <NUM>, <NUM>) of said ANN processing stage (<NUM>),
applying vectorization processing (<NUM>) to said selected set of weight parameters (wi) producing a set of weight vectors (u<NUM>, u<NUM>, u<NUM>) as a result, and arranging said set of weight vectors (u<NUM>, u<NUM>, u<NUM>) as items of a matrix of weight vectors (U),
performing normalization processing (<NUM>) of said matrix of weight vectors (U), producing a matrix of normalized weight vectors (U') as a result,
applying lattice vector quantization, LVQ, processing (<NUM>) to said matrix of normalized weight vectors (U'), producing a codebook (CB) of codewords as a result,
applying indexing processing (<NUM>) to said produced codebook (CB), said indexing (<NUM>) comprising encoding codewords of the codebook (CB) as a function of the lattice L, producing respective tuples of indices ( <MAT>) as a result,
providing said produced tuples of indices ( <MAT>) to a user circuit (<NUM>),
wherein performing said normalization processing (<NUM>) of said matrix of weight vectors (U) comprises computing an optimization problem having:
a first term configured to provide normalized weight values which approximate the at least output activation parameter (bj) of the ANN processing stage (<NUM>) as a function of the at least one input activation parameter (aj) of the ANN processing stage (<NUM>), and
a regularization term ΩL (u) configured to amplify normalized weight values that are closer to, having a reduced distance from, said lattice points of the selected lattice.