Patent Description:
A Deep Neural Network (DNN) is a form of artificial neural network comprising a plurality of interconnected layers that can be used for machine learning applications. In particular, a DNN can be used in signal processing applications, including, but not limited to, image processing and computer vision applications. <FIG> illustrates an example DNN <NUM> that comprises a plurality of layers <NUM>, <NUM>, <NUM>. Each layer <NUM>, <NUM>, <NUM> receives input data, and processes the input data in accordance with the layer to produce output data. The output data is either provided to another layer as the input data, or is output as the final output data of the DNN. For example, in the DNN <NUM> of <FIG>, the first layer <NUM> receives the original input data <NUM> to the DNN <NUM>, and processes the input data in accordance with the first layer <NUM> to produce output data <NUM>. The output data <NUM> of the first layer <NUM> becomes the input data to the second layer <NUM>, and the second layer <NUM> processes the input data <NUM> in accordance with the second layer <NUM> to produce output data <NUM>. The output data <NUM> of the second layer <NUM> becomes the input data to the third layer <NUM>, and the third layer <NUM> processes the input data <NUM> in accordance with the third layer <NUM> to produce output data <NUM>. The output data <NUM> of the third layer <NUM> is then output as the final output data of the DNN. Where the DNN is used for classification, the output data <NUM> may be a vector of length A, wherein A is the number of classes and each value in the vector represents the probability of a certain class.

Reference is made to <FIG> which illustrates an example overview of the format of data <NUM> utilised in a DNN. As can be seen in <FIG>, the data <NUM> used in a DNN may be formed of a plurality of planes. For example, the input data may be arranged as C planes of data, where each plane has a dimension H × W. Each plane may be referred to as a channel of the data.

The processing that is performed on the input data to a layer depends on the type of layer. For example, each layer of a DNN may be one of a plurality of different types. Example DNN layer types include, but are not limited to, a convolution layer, an activation layer, a normalisation layer, a pooling layer, and a fully connected layer. It will be evident to a person of skill in the art that these are example DNN layer types and that this is not an exhaustive list and there may be other DNN layer types.

A convolution layer convolves the input data with weights associated with the layer. Specifically, each convolution layer is associated with a plurality of weights k<NUM>. kg, which may also be referred to as filter weights or coefficients. The weights are grouped to form, or define, one or more filters or kernels, and each filter may be associated with an offset bias bias. Each filter may have a dimension M × N × C (i.e. each filter may comprise a set of M × N × C weights k) and may be applied to the input data according to a convolution operation across steps sW and sH in the W and H directions. The step sizes sW and sH may be referred to as the strides of the convolution. The number of filters and/or the number of weights per filter may vary between convolution layers. A convolutional neural network (CNN), which is a specific type of DNN that is effective for image recognition and classification, generally comprises a plurality of convolution layers.

An activation layer, which typically, but not necessarily follows a convolution layer, performs one or more activation functions on the input data to the layer. An activation function receives an input tensor and performs a certain non-linear mathematical operation on each value or element in the input tensor. In other words, the activation function operates on each value or element in the input tensor separately. In some examples, an activation layer may act as rectified linear unit (ReLU) by implementing an ReLU function (i.e. f(x) = max (<NUM>, x)) or a Parametric Rectified Linear Unit (PReLU) by implementing a PreLU function.

A normalisation layer is configured to perform a normalising function, such as a Local Response Normalisation (LRN) Function on the input data. A pooling layer, which is typically, but not necessarily inserted between successive convolution layers, performs a pooling function, such as a max, min or average function, to summarise subsets of the input data. The purpose of a pooling layer is thus to reduce the spatial size of the representation to reduce the number of parameters and computation in the network, and hence to also control overfitting.

A fully connected layer, which typically, but not necessarily follows a plurality of convolution and pooling layers takes a three-dimensional set of input data values and outputs a vector of length A. Where the DNN is used for classification, A may be the number of classes, and each value in the vector may represent the probability of a certain class. The vector of length A is generated through a matrix multiplication of a set of weights, optionally followed by a bias offset. A fully connected layer thus receives a set of weights and a bias.

Accordingly, each layer of a DNN receives input data values and generates output data values; and some layers (such as, but not limited to, convolution layers and fully-connected layers) also receive weights and/or biases.

A DNN can be configured to perform a specific function by learning the weights through a training process. Training is often performed through a technique referred to as back-propagation which is illustrated in <FIG>. Specifically, to train a DNN via back-propagation, a model of the DNN <NUM> is configured to use a particular set of weights, training data is then applied to the model, and the output F of the model in response to the training data is recorded. This is referred to as performing a forward pass of the DNN and is shown generally at <NUM> of <FIG>. A differentiable error or loss metric, E, is then calculated from the recorded output F which quantitatively indicates the performance of the DNN using that particular set of weights. The derivative of the error or loss metric, E, is then back-propagated, via the chain rule and the partial derivative rule, to the weights, K, of each layer the DNN to produce gradients, or derivatives of, the loss or error metric, E, with respect to each weight, <MAT> (see <NUM> of <FIG>). Although only a single convolution layer <NUM> is shown in <FIG> it will be evident to a person of skill in the art that this is used as an example only and that the gradient or derivative of the loss or error metric, E, with respect to each weight of each layer may be determined. The weights are then adjusted based on the gradients so as to reduce the loss or error metric E. This process may be repeated until the loss or error metric E converges.

<NPL> describe an automatic compiler based FPGA accelerator with <NUM>-bit floating point precision for complete CNN training, including Forward Pass (FP), Backward Pass (BP) and Weight Update (WU).

The embodiments described below are provided by way of example only and are not limiting of implementations which solve any or all of the disadvantages of methods and systems for training a DNN.

Preferred aspects of the invention are set out in the dependent claims.

Embodiments are described by way of example only.

As described above, training a DNN via back-propagation involves configuring a model of the DNN <NUM> to use a particular set of weights, applying training data to the model <NUM>, and recording the output F of the model in response to the training data. This is referred to as performing a forward pass of the DNN which is shown generally at <NUM> of <FIG>. A differentiable error or loss metric E is then calculated from the recorded output F which quantitatively indicates the performance of the DNN using that particular set of weights. The derivative of the loss or error metric E is then back-propagated, via the chain rule and the partial derivative rule, to the weights K of each layer of the DNN to produce gradients, or derivatives, of the loss or error metric F with respect to each weight <MAT> (see <NUM> of <FIG>). The weights K of each layer are then adjusted based on the gradients so as to reduce the loss or error metric. Although only a single convolution layer <NUM> is shown in <FIG> it will be evident to a person of skill in the art that this is used as an example only and that the gradient or derivative of the loss or error metric F with respect to each weight of each layer may be determined. This process may be repeated until the loss or error metric E converges.

Training of a DNN via back-propagation is often performed on a general-purpose computer using a software tool, such as, but not limited to, TensorFlow®. Such software tools are often configured to determine the gradient, or derivative, of the error or loss metric with respect to the weights of each convolution layer via matrix multiplication. It may not be possible to implement this method of determining the gradients efficiently in some hardware, particularly hardware, such a DNN accelerator, designed to implement a DNN. The inventors have identified that the derivative of the error or loss metric with respect to the weights of a convolution layer can be determined efficiently in hardware via one or more group convolutions between values of the input data to the convolution layer and the gradients or derivatives of the output error or loss metric with respect to the convolution layer outputs. In particular, such group convolutions can be implemented efficiently in hardware, such as a DNN accelerator, that comprises one or more convolution engines, wherein a convolution engine is dedicated hardware that is configured to perform a convolution operation between input data and a filter.

A convolution layer of a DNN often, but not necessarily, implements a two-dimensional (2D) convolution. Reference is now made to <FIG> which illustrates an example 2D convolution. In the 2D convolution illustrated in <FIG>, the input data X <NUM> to the layer forms a three-dimensional (3D) signal or tensor with a height H, width W, and channels C; the weights K <NUM> form a four-dimensional (4D) signal or tensor comprising a plurality of (e.g. L) 3D filters or kernels <NUM><NUM>, <NUM><NUM>, <NUM><NUM>, <NUM><NUM> with a height M, width N, and channels C; and each filter <NUM><NUM>, <NUM><NUM>, <NUM><NUM>, <NUM><NUM> is convolved with the 3D input data X <NUM>. Each filter <NUM><NUM>, <NUM><NUM>, <NUM><NUM>, <NUM><NUM> produces one channel, or one plane, of the output data Y <NUM>. Therefore, if there are L filters, the output data <NUM> comprises L channels or planes. Accordingly, the number of filters may also be referred to as the number of output channels.

Each neuron or element of the output tensor Y (i,j,l) of a 2D convolution can be generated from the input tensor X and the weight tensor K by equation (<NUM>), wherein X (h, w, c) refers the neuron or element of the input tensor X at the hth row, wth column and cth plane, Y(i, j, l) refers to the neuron or element of the output tensor Y at the ith row, jth column and the lth plane, K(m, n, c, l) refers to the weight at the mth row, nth column, cth plane of the lth kernel or filter, s represents the stride of the convolution, d represents the dilation, and <MAT> and <MAT> represent the padding in the height and width dimensions respectively. <MAT> <MAT> <MAT> <MAT>.

Accordingly, the output tensor Y is generated by sliding each filter of the weight tensor K over the input tensor X in the height dimension H and the width dimension W (in accordance with the stride(s) and dilation(s)), and generating the dot product of the input values and the weights at each position. Therefore, in a 2D convolution there are two traversed dimensions - the height dimension H and width dimension W. As is known to those of skill in the art, the stride and the dilation may be different in the height dimension H and the width dimension W. For example, the stride in the height dimension H may be <NUM>, and the stride in the width dimension W may be <NUM>. However, for simplicity, in the examples described herein it has been assumed that the stride is the same in the height and width dimensions, and the dilation is the same in the height and width dimensions. However, the methods and techniques described herein are not limited to convolution layers where the stride and dilation are the same in the height and width dimensions.

As is known to those of skill in the art, the stride s defines the step size of the filter or kernel. Specifically, it specifies how many elements of the input data the filter shifts each time. For example, as illustrated at <NUM> of <FIG>, for a stride of one, a filter of size M × N × C is first convolved with M × N × C values <NUM> of the input data selected from element (m, n, c) of the input data, then the filter is convolved with M × N × C values <NUM> of the input data selected from element (m, n + <NUM>, c). In other words, with a stride of one, the window of the input data moves one element each time. In contrast, as illustrated at <NUM> of <FIG>, for a stride of two, the filter is convolved with M × N × C values <NUM> of the input data selected from element (m, n, c), then the filter is convolved with M × N × C values <NUM> of the input data starting with element (m, n + <NUM>, c). In other words, with a stride of two, the window of the input data moves two elements each time. The stride can be set to any integer greater than or equal to <NUM>. As described above, the stride can be different in the height and width dimensions. For example, the stride can be <NUM> in the height dimension and <NUM> in the width dimension.

As is known to those of skill in the art, the dilation d, which may also be referred to as the dilation rate, describes the spacing between the positions where the values in a filter or kernel are applied. For example, as shown at <NUM> of <FIG>, a <NUM>×<NUM>×<NUM> filter <NUM> with a dilation of <NUM> will be convolved with the twelve values in a <NUM>×<NUM>×<NUM> block of values <NUM> of the input tensor X. In contrast, as shown at <NUM> of <FIG>, a <NUM>×<NUM>×<NUM> filter <NUM> with a dilation of <NUM> will be convolved with twelve values in a <NUM>×<NUM>×<NUM> block of values <NUM> of the input tensor X. As described above, the dilation can be different in the height and width dimensions.

As is known to those of skill in the art, by default, as shown at <NUM> of <FIG>, a convolution starts with data or element (<NUM>,<NUM>,<NUM>) (i.e. the element in the top left corner) <NUM> of the input tensor X aligned with data or element (<NUM>,<NUM>,<NUM>) of the filter <NUM>, and then the filter <NUM> is stepped across the image according to the stride s. This is referred to as a zero-padding convolution. Padding is the addition of rows and/or columns with null values to the input tensor X. For example, as shown at <NUM> of <FIG>, a row of padding <NUM> is added to the top of a <NUM>×<NUM>×<NUM> input tensor X and a column of padding <NUM> is added to the left of the input tensor X. It can be seen that in this example the convolution starts with the element or neuron at location (<NUM>,<NUM>,<NUM>) (i.e. the element in the top left corner) <NUM> of the input tensor X aligned with the element or neuron at location (<NUM>,<NUM>,<NUM>) of the filter <NUM>. The notations <MAT> and <MAT> are used herein to refer to the number of rows of padding that are added to the top of the input tensor X and the number of columns of padding that are added to the left of the input tensor X, respectively. The notations <MAT> and <MAT> are used herein to refer to the number of rows of padding that are added to the bottom of the input tensor X and the number of columns of padding that are added to the right of the input tensor X, respectively.

If the output error or loss metric of the DNN is denoted F then the derivative of the output error or loss metric E with respect to a particular weight K (m, n, c, l) of a convolution layer can be represented using equation (<NUM>) where <MAT> is the derivative or gradient of the output error or loss metric E with respect to a particular element or neuron of the convolution output tensor Y.

Let the derivative or gradient of the output error or loss metric E with respect to a particular output element or neuron of the output tensor Y (i,j,l) of the convolution layer be represented by G (i,j,l) as shown in equation (<NUM>). Therefore G, which may be referred to herein as the output gradient tensor, is a 3D signal or tensor comprising the derivative or gradient of the output error or loss metric E with respect to each of the neurons or elements of the output tensor Y.

From equation (<NUM>) the derivative or gradient of the output of the convolution layer with respect a particular weight <MAT> can be expressed as shown in equation (<NUM>).

Using equations (<NUM>) and (<NUM>), equation (<NUM>) can then be expressed as shown in equation (<NUM>). Therefore B, which may be referred to herein as the weight gradient tensor, is a 4D signal or tensor comprising the derivative or gradient of the output error or loss metric E with respect to each weight in the weight tensor K. <MAT> <MAT> <MAT>.

The inventors have identified that the results of equation (<NUM>) can be determined efficiently in hardware as a group convolution between the values from the input tensor X and the values from the output gradient tensor G. In other words, the inventors have identified that the values of the 4D signal or tensor B (the weight gradient tensor) can be calculated efficiently in hardware, and specifically, hardware designed to implement a deep neural network (e.g. a DNN accelerator) via a group convolution between a tensor formed of values from the input tensor X and a tensor formed of values from the output gradient tensor G.

As described above with respect to <FIG>, in a 2D convolution, each filter has the same number of channels as the input tensor X and is convolved with all of the input tensor X. In contrast, in a group convolution, the input tensor X is subdivided into multiple, g, subsets or groups on a channel basis and a different set of filters (i.e. a different set of the weights K) is convolved with each subset or group of the input tensor. Each filter has the same number of channels as its associated subset. In other words, each subset of the input tensor comprises <MAT> channels or planes of the input, and each filter comprises <MAT> channels or planes where U is the number of channels in the input tensor X. Like in a 2D convolution, each filter in a group convolution produces a channel or plane of the output. Accordingly, if there are R output channels or planes, each subset of the input produces <MAT> channels or planes of the output tensor Y.

An example group convolution is illustrated in <FIG>. In this example, the input tensor X <NUM> has a height H, a width W, and U planes or channels, and is divided into two groups <NUM><NUM> and <NUM>, (i.e. g=<NUM>). Each input group <NUM><NUM> and <NUM><NUM> comprises <MAT> planes of the input tensor X. Since U = <NUM> in this example, each input group <NUM><NUM> and <NUM>, comprises two planes. The weights K <NUM> are divided into two filter groups <NUM><NUM>, <NUM><NUM>. Each filter <NUM><NUM>, <NUM><NUM>, <NUM><NUM>, <NUM><NUM> has height M, width N and <MAT> planes or channels. Each filter <NUM><NUM>, <NUM><NUM>, <NUM><NUM>, <NUM><NUM> in a filter group <NUM><NUM> and <NUM>, is convolved with the corresponding input group <NUM><NUM> and <NUM>, to generate a channel or layer of the output tensor Y <NUM>. For example, each filter <NUM><NUM>, <NUM><NUM> in the first filter group <NUM><NUM> is convolved with the first input group <NUM><NUM> to generate a first group or subset <NUM><NUM> of the output tensor Y <NUM>, and each filter <NUM><NUM>, <NUM><NUM> in the second filter group <NUM>, is convolved with the second input group <NUM>, to generate a second group or subset <NUM><NUM> of the output tensor Y <NUM>.

Each element or neuron of the output tensor Y of a group convolution can be generated in accordance with equation (<NUM>) wherein, as above, s is the stride, d is the dilation and <MAT> and <MAT> represent the padding in the height and width dimensions, respectively. <MAT> <MAT> <MAT>.

The inventors have identified that the values of the four-dimensional weight gradient tensor B of equation (<NUM>) may be generated by a group convolution between the values from the input tensor X and the values from the weight gradient tensor G. Specifically, the values of the weight gradient tensor B may be generated by a group convolution between a first three-dimensional tensor comprised of values from the input tensor X (i.e. the input values of the convolution layer) and a second four-dimensional tensor comprised of values from the output gradient tensor G (i.e. the gradients of the loss or error metric with respect to the outputs of the convolution layer). Group convolutions can be implemented, or performed, efficiently in hardware, such as a DNN accelerator, that comprises one or more convolution engines, wherein a convolution engine is dedicated hardware configured to perform convolution operations. In some cases, a convolution engine may comprise a plurality of multipliers followed by an adder tree. Each multiplier is configured to multiply a first value with a second value to produce a multiplication output. The adder tree may be configured to produce the sum of the multiplication outputs.

Accordingly, reference is now made to <FIG> which illustrates an example method <NUM> of generating the gradients of the output error or loss metric of a DNN with respect to the weights of a convolution layer using hardware logic. Preferably the hardware logic comprises one or more convolution engines. In some cases, the hardware logic may be a neural network accelerator, such as a DNN accelerator. An example DNN accelerator is described below with respect to <FIG>. The method <NUM> comprises, at block <NUM>, using the hardware logic to perform a group convolution between a first three-dimensional tensor comprised of values from the input tensor X (i.e. the input values of the convolution layer) and a second four-dimensional tensor comprised of values from the output gradient tensor G (i.e. the gradients of the loss or error metric with respect to the output values of the convolution layer).

It can be seen from equation (<NUM>) that the values of the weight gradient tensor B are generated by convolving each channel of the input tensor X with each channel of the output gradient tensor G with a stride of d and a dilation of s, wherein s is the stride for the convolution layer and d is the dilation for the convolution layer. Specifically, if the input tensor X comprises three channels and the output gradient tensor G has four channels, then the values of the weight gradient tensor B are generated by convolving the first channel of the input tensor X with the first channel of the output gradient tensor G, convolving the first channel of the input tensor X with the second channel of the output gradient tensor G, convolving the first channel of the input tensor X with the third channel of the output gradient tensor G, convolving the first channel of the input tensor X with the fourth channel of the output gradient tensor G, convolving the second channel of the input tensor X with the first channel of the output gradient tensor G, convolving the second channel of the input tensor X with the second channel of the output gradient tensor G etc..

Accordingly, the first three-dimensional tensor, the second four-dimensional tensor and the group convolution are configured so as to generate the output of a convolution operation between each channel of the input tensor X and each channel of the output gradient tensor G. Example group convolutions of this form are described below.

Where the hardware logic comprises one or more convolution engines the group convolution is performed by, or implemented by, the one or more convolution engines.

Depending on the specific group convolution performed and how the values of the input tensor X and gradients of the output gradient tensor G are received, the method <NUM> may further comprise at block <NUM>, reformatting the input tensor X to generate the first three-dimensional tensor and/or reformatting the output gradient tensor G to generate the second four-dimensional tensor. Example methods for reformatting the input tensor X and/or the output gradient tensor G to generate the first three-dimensional tensor and the second four-dimensional tensor respectively are described below.

Similarly, in some cases, depending on the particular group convolution that is performed, the method <NUM> may further comprise at block <NUM>, reformatting the output of group convolution to generate the weight gradient tensor B. Example methods of reformatting the output of the group convolution are described below.

Reference is now made to <FIG> which illustrate a first example group convolution between a first three-dimensional tensor comprised of values from the input tensor X (i.e. the input values of the convolution layer) and a second four-dimensional tensor comprised of values from the output gradient tensor G (i.e. the gradients of the error or loss metric E with respect to the outputs Y of the convolution layer) which generates the values of the weight gradient tensor B of equation (<NUM>) (i.e. the gradients of the error or loss metric E with respect to the weights K of the convolution layer). In this first example group convolution there are C groups (i.e. g = C) and there are C * L filters, where C is the number of channels in the input tensor X of the convolution layer, and L is the number of channels in the output tensor Y of the convolution layer (which can alternatively be described as the number of filters in the weight tensor K of the convolution layer).

In this example, the first three-dimensional tensor comprised of values from the input tensor X is equal to the input tensor X. As described above, the values of the weight gradient tensor B of equation (<NUM>) are generated by convolving each channel of the input tensor X with each channel of the output gradient tensor G with a stride of d and a dilation of s, wherein s is the stride for the convolution layer and d is the dilation for the convolution layer. Therefore, the gradients of the weight gradient tensor B can be generated by a group convolution between the input tensor X and a four-dimensional tensor that comprises C groups of filters, and each group of filters comprises L filters, one filter for each of the L channels of the output gradient tensor G.

An example of such a four-dimensional tensor is one that comprises C * L filters wherein the ath filter corresponds to the bth channel of the output gradient tensor G, wherein a is any integer in the set <NUM> to (C * L) - <NUM>, C is the number of channels of the input tensor, and b is equal to a modulo L. In other words, a four-dimensional tensor comprising C * L filters wherein every Lth filter is the same and corresponds to one of the channels of the output gradient tensor G.

For example, <FIG> illustrates an example output gradient tensor C <NUM> with a height I of <NUM>, a width J of <NUM> and <NUM> channels (i.e. L = <NUM>). An example four-dimensional tensor G" <NUM> that can be convolved, via a group convolution, with an input tensor X with <NUM> channels (i.e. C = <NUM>) to generate the values of the weight gradient tensor B comprises <NUM> (C * L = <NUM> * <NUM>) filters, the <NUM> filters can be divided into <NUM> (i.e. C) groups wherein each group comprises a <NUM>×<NUM>×<NUM> (i.e. I × J × <NUM>) filter that is equivalent to a channel of the output gradient tensor G <NUM>. Specifically, in this example, the first filter in each group is equivalent to the first channel of the output gradient tensor G <NUM>, the second filter in each group is equivalent to the second channel of the output gradient tensor G <NUM>, the third filter in each group is equivalent to the third channel of the output gradient tensor G <NUM> and so on.

Since there are C groups of filters in the four-dimensional tensor G" <NUM>, each group of filters is convolved with one channel of the input tensor X <NUM>. Since each group of filters in the four-dimensional tensor G" <NUM> comprises a filter that corresponds to each of the channels of the output gradient tensor G, such a group convolution produces the output of a convolution operation between each channel of the input tensor X <NUM> and each channel of the output gradient tensor G <NUM>.

It will be shown mathematically that such a group convolution will generate the values of the weight gradient tensor B of equation (<NUM>). Specifically, equation (<NUM>) can be re-written with G" <NUM> as shown in equation (<NUM>).

This is not a group convolution as the output is a four-dimensional tensor or signal, rather than a three-dimensional tensor or signal. If the third and fourth dimensions of the weight gradient tensor B are combined, equation (<NUM>) can be re-written to generate a three-dimensional output as shown in equation (<NUM>).

If r = cL + l then equation (<NUM>) can be re-written as equation (<NUM>).

Since <MAT> is an integer because it is substituted for c which is an integer, equation (<NUM>) can be re-written as equation (<NUM>).

If the number of groups g in a group convolution is equal to the number of input channels U, the number of input channels U is equal to C, and the number of filters R in the group convolution is C * L then the group convolution of equation (<NUM>) can be re-written as equation (<NUM>). This can then be simplified to equation (<NUM>), and then to equation (<NUM>). <MAT> <MAT> <MAT>.

It can be seen that equations (<NUM>) and (<NUM>) are equivalent if Y in equation (<NUM>) is replaced with B', and K in equation (<NUM>) is replaced with G". Accordingly, equation (<NUM>) is a group convolution between X and G" that produces the values of the weight gradient tensor B.

In this example, tensor G" <NUM> is the second four-dimensional tensor. Therefore block <NUM> of method <NUM> may comprise generating the second four-dimensional tensor G" <NUM> from the output gradient tensor C <NUM>. As shown in <FIG>, the four-dimensional tensor G" <NUM> may be generated from the output gradient tensor C <NUM> by expanding the three-dimensional output gradient tensor G <NUM> to a four-dimensional tensor G' (I×J×<NUM>×L) <NUM> with L three-dimensional tensors (which may be referred to as filters), wherein each 3D tensor/filter thereof is equal to a channel of the output gradient tensor G (I × J × L); and then tiling the expanded tensor G' (I × J × <NUM> × L) <NUM> C times to generate the tensor G" (I × J × <NUM> × CL) <NUM>. The term expanding a tensor is used herein to mean adding an axis of size <NUM> before the last axis. For example, as shown in <FIG> expanding the output gradient tensor C <NUM> with dimensions (I × J × L) results in a four-dimensional tensor G' <NUM> with dimensions (I × J × <NUM> × L). The term tiling a tensor e times along v means copying the tensor e times along the v axis or dimension. For example, the four-dimensional tensor G' is copied C times along the fourth axis to generate a four-dimensional tensor G" with dimensions (I × J × <NUM> × CL). As can be seen in <FIG>, G" comprises C * L 3D tensors/filters wherein every Lth 3D tensor/filter is the same and corresponds to one of the channels of the output gradient tensor G. It will be evident to a person of skill in the art that this is one example method of generating the four-dimensional tensor G" <NUM> from the output gradient tensor G <NUM>, and that the four-dimensional tensor G" <NUM> may be generated from the output gradient tensor G <NUM> in another manner. For example, the four-dimensional tensor G" may alternatively be generated from the output gradient tensor G <NUM> by first tiling the output gradient tensor G, C times in the channel dimension, and then expanding the tiled tensor.

<FIG> illustrates the group convolution between the four-dimensional tensor G" <NUM> and the input tensor X <NUM> where the number of groups is C. It can be seen that this group convolution results in a three-dimensional tensor B' <NUM> (equation (<NUM>)). While, as shown above, tensor B' <NUM> comprises all of the values of the weight gradient tensor B, tensor B' is three-dimensional, and the weight gradient tensor B is four-dimensional. Accordingly, in this example, block <NUM> of the method <NUM> may comprise reformatting the output of the group convolution (i.e. tensor B') so as to generate the weight gradient tensor B. This can be alternatively described as reversing the steps to generate equation (<NUM>) from equation (<NUM>).

Reference is now made to <FIG> which illustrates an example method of generating the weight gradient tensor B <NUM> from the output of the group convolution of <FIG> (i.e. three-dimensional tensor B' <NUM>). In this example method, the weight gradient tensor B <NUM> is generated from the group convolution output (i.e. three-dimensional tensor B' <NUM>) by reshaping the three-dimensional tensor B'. The term reshaping a tensor comprises converting a tensor of o dimensions into a reshaped tensor of o + <NUM> dimensions by splitting a dimension of the tensor. For example, a tensor of dimensions (M × N × CL) may be reshaped to a tensor of dimensions (M × N × C × L).

It will be evident to a person of skill in the art that the four-dimensional tensor G" <NUM> of <FIG> is just an example of a four-dimensional tensor that can be used to generate the gradients of the weight gradient tensor B via group convolution with the input tensor X wherein the group convolution has C groups, and other four-dimensional tensors generated from the output gradient tensor G can be used to generate the weight gradient tensor B via a group convolution with the input tensor X. Specifically, so long as each group of filters in the four-dimensional tensor comprises a filter that corresponds to each channel of the output gradient tensor G, then the filters within in each group may be in any order within a group. For example, each group will comprise one channel of the output gradient tensor G, however the channels within a group do not have to be in the same order as they are in the output gradient tensor G, and/or each group does not need to have the channels in the same order. For example, a first group may have channel <NUM> of the output gradient tensor G as the first filter, and a second group may have channel <NUM> of the output gradient tensor G as the second filter etc..

Similarly, it will be evident to a person of skill in the art that the input tensor X is simply an example of a three-dimensional tensor that can be used to generate the values of the weight gradient tensor via group convolution with a four-dimensional tensor, such as, but not limited to, tensor G", where there are C groups. For example, the three-dimensional tensor may be any three-dimensional tensor that comprises C channels wherein each channel of the three-dimensional tensor corresponds to a channel of the input tensor X. In other words, the channels of the input tensor X may be in any order (i.e. they do not have to be in the same order that they are in the input tensor X) so long as the channels of the output of the group convolution can be mapped to the channels of the weight gradient tensor B.

Reference is now made to <FIG> which illustrate a second example group convolution between a first three-dimensional tensor comprised of values from the input tensor X (i.e. the input values of the convolution layer) and a second four-dimensional tensor comprised of values from the output gradient tensor G (i.e. the gradients of the error or loss metric E with respect to the outputs Y of the convolution layer) which generates the gradients of the weight gradient tensor B of equation (<NUM>) (i.e. the gradients of the error or loss metric E with respect to the weights K of the convolution layer). In this second example group convolution there are C * L groups (i.e. g = C * L) and there are C * L filters, where C is the number of channels in the input tensor X of the convolution layer, and L is the number of channels in the output tensor Y of the convolution layer (which can alternatively be described as the number of filters in the weight tensor K of the convolution layer).

In this example, the first three-dimensional tensor comprised of values from the input tensor X comprises L copies of each channel of the input tensor X. An example of such a three-dimensional tensor is one which comprises L copies of the input tensor in the channel dimension. For example, <FIG> illustrates an example input tensor X <NUM> with a height H of <NUM>, a width W of <NUM>, and <NUM> channels (i.e. C = <NUM>). An example three-dimensional tensor X' <NUM> that can be convolved with a four-dimensional tensor of output gradients to generate the values of the weight gradient tensor B, via a group convolution with C * L groups, comprises L copies of the input tensor X <NUM> along the channel axis, wherein L = <NUM> in this example. Specifically, in this example, the three-dimensional tensor X' <NUM> comprises C * L channels and the qth channel of the three-dimensional tensor X' <NUM> comprises the rth channel of the input tensor X <NUM>, wherein q is any integer in the set <NUM> to (C * L) - <NUM>, C is a number of channels in the input tensor X <NUM>, L is a number of filters in the convolution layer, and r is q modulo L.

As described above, the values of the weight gradient tensor B of equation (<NUM>) are generated by convolving each channel of the input tensor X with each channel of the output gradient tensor G with a stride of d and a dilation of s, wherein s is the stride for the convolution layer and d is the dilation for the convolution layer. Therefore, the gradients of the weight gradient tensor B can be generated by a group convolution between the three-dimensional tensor X' <NUM> and a four-dimensional tensor that G" that comprises C copies of each channel of the output gradient tensor G, the copies being arranged so that each channel of the output gradient tensor G is convolved with each channel of the input tensor X.

An example of such a four-dimensional tensor G", which can be convolved with the three-dimensional tensor X' of <FIG>, is one that comprises C * L filters and the tth filter comprises the uth channel of the output gradient tensor G, wherein t is any integer in the set <NUM> to (C * L) - <NUM>, C is the number of channels of the input tensor X <NUM>, and u is equal to <MAT>. In other words, a four-dimensional tensor comprising C * L filters wherein each set of C filters in row are the same and correspond to one of the channels of the output gradient tensor G.

For example, <FIG> illustrates an example output gradient tensor G <NUM> with a height I of <NUM>, a width J of <NUM>, and <NUM> channels (i.e. L = <NUM>). An example four-dimensional tensor G" <NUM> that can be convolved, via a group convolution, with the three-dimensional tensor X' <NUM> of <FIG> to generate the gradients of the weight gradient tensor B comprises <NUM> (C * L = <NUM> * <NUM>) filters, wherein each set of C consecutive filters are the same and are equivalent to one channel of the output gradient tensor C <NUM>. Specifically, the first three filters in the four-dimensional tensor G" <NUM> are the same and are equivalent to the first channel of the output gradient tensor C <NUM>, the next three filters in the four-dimensional tensor G" <NUM> are the same and are equivalent to the second channel of the output gradient tensor G <NUM>, the next three filters in the four-dimensional tensor G" <NUM> are the same and are equivalent to the third channel of the output gradient tensor C <NUM> and so on.

Since the filters in the tensor G" <NUM> are divided into C * L groups and the three-dimensional tensor X' <NUM> comprises C * L channels, each filter group is convolved with one channel of the three-dimensional tensor X' <NUM>. It can be seen that the channels of the three-dimensional tensor X' <NUM> and four-dimensional tensor G" <NUM> have been arranged such that a group convolution between the two tensors produces the output of a convolution operation between each channel of the input tensor X <NUM> and each channel of the output gradient tensor G <NUM>.

It will be shown mathematically that such a group convolution will generate the values of the weight gradient tensor B of equation (<NUM>). Specifically, equation (<NUM>) can be re-written with X' <NUM> and G" <NUM> as shown in equation (<NUM>).

If r = c + lC then equation (<NUM>) can be re-written as equation (<NUM>).

If the number of groups g in a group convolution is equal to the number of input channels U, the number of input channels U is equal to C * L, and the number of filters R in the group convolution is C * L then the group convolution of equation (<NUM>) can be re-written as equation (<NUM>). This can then be simplified to equation (<NUM>), and then to equation (<NUM>). <MAT> <MAT> <MAT>.

It can be seen that equations (<NUM>) and (<NUM>) are equivalent if Y in equation (<NUM>) is replaced with B', X in equation (<NUM>) is replaced with X', and K in equation (<NUM>) is replaced with G". Accordingly, equation (<NUM>) is a group convolution between X' and G" that generates the gradients of the weight gradient tensor B.

In this example, tensor X' <NUM> is the first three-dimensional tensor of the group convolution. Therefore, block <NUM> of method <NUM> may comprise generating the first three-dimensional tensor X' from the input tensor X <NUM>. As shown in <FIG>, the three-dimensional tensor X' <NUM> may be generated from the input tensor X <NUM> by tiling the input tensor X L times along the channel, C, dimension. As described above, the term tiling a tensor e times along v means copying the tensor e times along the v axis. For example, as shown in <FIG>, the input tensor X <NUM> is copied L times along the channel axis or dimension to generate a three-dimensional tensor X' <NUM> with dimensions (I × J × CL). As can be seen in <FIG>, X' comprises C * L channels wherein every Lth channel is the same and corresponds to one of the channels of the input tensor X. It will be evident to a person of skill in the art that this is one example method of generating the three-dimensional tensor X' <NUM> from the input tensor X <NUM>, and that the three-dimensional tensor X' <NUM> may be generated from the input tensor X <NUM> in another manner.

In this example, tensor G" <NUM> is the second four-dimensional tensor of the group convolution. Therefore, block <NUM> of method <NUM> may comprise generating the second four-dimensional tensor G" from the output gradient tensor G. As shown in <FIG>, the four-dimensional tensor G" <NUM> may be generated from the output gradient tensor G by expanding the three-dimensional output gradient tensor G <NUM> to a four-dimensional tensor G' (I × J × <NUM> × L) <NUM> with L three-dimensional tensors (which may be referred to as filters), wherein each 3D tensor/filter thereof is equal to a channel of the output gradient tensor G (I × J × L) <NUM>; and then repeating each filter of the expanded tensor G' <NUM> C times in the fourth dimension to generate the tensor G" (I × J × <NUM> × CL) <NUM>. As described above, the term expanding a tensor is used herein to mean adding an axis of size <NUM> before the last axis. For example, as shown in <FIG>, expanding the output gradient tensor C <NUM> with dimensions (I × J × L) results in a four-dimensional tensor G' <NUM> with dimensions (I × J × <NUM> × L). The term repeating a tensor a times along an axis or dimension k means repeating each entry on dimension k, a times. For example, each 3D filter is repeated C times in a row to generate a four-dimensional tensor G" <NUM> with dimensions (I × J × <NUM> × CL). As can be seen in <FIG>, G" comprises C * L 3D tensors/filters wherein each set of C filters in a row are the same and correspond to one of the channels of the output gradient tensor G. It will be evident to a person of skill in the art that this is one example method of generating the four-dimensional tensor G" <NUM> from the output gradient tensor C <NUM>, and that the four-dimensional tensor G" <NUM> may be generated from the output gradient tensor C <NUM> in another manner. For example, the four-dimensional tensor G" may be alternatively generated from the output gradient tensor C by first repeating the channels of G, C times, and then expanding the repeated tensor.

<FIG> illustrates the group convolution between the three-dimensional tensor X' <NUM> and the four-dimensional tensor G" <NUM> where there are C * L groups. It can be seen that this group convolution results in a three-dimensional tensor B' <NUM> (equation (<NUM>)). While, as shown above, tensor B' comprises all of the gradients of the weight gradient tensor B, tensor B' <NUM> is three-dimensional, and the weight -gradient tensor B is four-dimensional. Accordingly, in this example, block <NUM> of the method <NUM> may comprise reformatting the output of the group convolution (i.e. tensor B' <NUM>) so as to generate the weight gradient tensor B.

Reference is now made to <FIG> which illustrates an example method of generating the weight gradient tensor B <NUM> from the output of the group convolution of <FIG> (tensor B' <NUM>). In this example method, the weight gradient tensor B <NUM> is generated from the group convolution output (tensor B') by reshaping tensor B' <NUM>. The term reshaping a tensor comprises converting a tensor of o dimensions into a reshaped tensor of <NUM> + <NUM> dimensions by splitting a dimension of the tensor. For example, a tensor of dimensions (M × N × CL) may be reshaped to a tensor of dimensions (M × N × C × L).

Similarly, it will be evident to a person of skill in the art that the three-dimensional tensor X' <NUM> is simply an example of a three-dimensional tensor that can be used to generate the values of the weight gradient tensor via group convolution with a four-dimensional tensor, such as, but not limited to, tensor G" <NUM>, where there are C * L groups. For example, the three-dimensional tensor may be any three-dimensional tensor that comprises C * L channels which comprises L copies of each channel of the input tensor X <NUM>. In other words, the L copies of a channel of the input tensor X in the three-dimensional tensor do not have to be every Lth channel, the channels of the input tensor X could be in a different order.

Similarly, it will be evident to a person of skill in the art that the tensor G" <NUM> is just an example of a four-dimensional tensor that can be used to generate the values of the weight gradient tensor B via a group convolution with a three-dimensional tensor, such as X' <NUM>, wherein the group convolution has C * L groups, and other four-dimensional tensors can be used. Specifically, any four-dimensional tensor that comprises C copies of each channel of the output gradient tensor C wherein the channels are arranged so that each channel of the input tensor is convolved with each channel of the output gradient may be used. For example, if the first three-dimensional tensor comprises the L copies of a channel of the input tensor in succession, then the second four-dimensional tensor may comprise sets of L filters wherein each set comprises one filter that corresponds to each channel of the output gradient tensor G.

Where the method <NUM> comprises reformatting one or more of the input tensor X , the output gradient tensor G, and the output of the group convolution to generate the first three-dimensional tensor, the second four-dimensional tensor, and the weight gradient tensor respectively, in some cases, one or more of the reformatting steps (e.g. repeating, tiling, expanding and reshaping) may be performed by one or more convolution engines. For example, any repeating and/or tiling may be performed via one or more convolution engines. In other cases, one or more of the reformatting steps (e.g. repeating, tiling, expanding and reshaping) may be performed by general purpose logic. Specifically, tiling a tensor may be performed using the one or more convolution engines by convolving the tensor with a set of one-hot 1x1 filter which are <NUM> where they carry the input channel to the output and <NUM> otherwise. For example, to implement the tiling of G' <NUM> C times as shown in <FIG> there would be L × C filters. Similarly, repetition of a tensor could be performed using the one or more convolution engines by performing a group convolution (e.g. as shown in <FIG>) between the tensor and a unit 1x1 kernel with the number of filters equal to C × L.

In yet other cases, the hardware logic used to perform the method <NUM> may comprises a dedicated hardware unit, which may be referred to as a memory management module, which is capable of performing data manipulation, such as repeating, tiling, reshaping and/or expanding on a received tensor. A memory management module may be capable of receiving a multi-dimensional tensor (e.g. from memory) and re-shaping the manner/order in which those dimensions are presented/stored in memory. An example of a memory management module is described in the Applicant's <CIT>. Using a dedicated hardware unit or module to perform one or more of the reformatting steps may significantly improve the efficiency of the method.

Although the methods of determining the gradients of a loss metric of the DNN with respect to the weights of a convolution layer described above have been described with respect to a convolution layer that implements a 2D convolution, the described methods are not limited to convolution layers that implement a 2D convolution and may be similarly used for convolution layers that implement any Z-D convolution wherein Z is the dimension of the convolution and is an integer greater than or equal to one.

For example, as is known to those of skill in the art, a 1D convolution convolves a two-dimensional input tensor and a three-dimensional weight tensor comprising a set of two-dimensional filters. Each filter is slid or traversed over only one dimension of the input tensor (vs the two traversed dimensions for a 2D convolution). The method <NUM> of <FIG> described above can be used to determine the gradients of a loss metric of the DNN with respect to the weights of a convolution layer that implements a 1D convolution by converting the 1D convolution into a 2D convolution by adding a dummy dimension (i.e. the additional dimension has a value of <NUM>) to the input tensor and the weight tensor (and thus the output tensor). For example, a two-dimensional input tensor (W × C) may be converted to a three-dimensional input tensor (H × W × C) wherein H = <NUM> (which may be referred to as the augmented input tensor), and the three-dimensional weight tensor (N × C × L) may be converted to a four-dimensional weight tensor (M × N × C × L) wherein M = <NUM> (which may be referred to as the augmented weight tensor). The output of the 2D convolution will also then have a dummy dimension. The gradients of a loss metric of the DNN with respect to the weights of a convolution layer that implements a 1D convolution can then be determined in the same manner as described above by performing a group convolution between a first three-dimensional tensor and a second four-dimensional tensor, the first three-dimensional tensor being formed of values from the augmented input tensor, and the second four-dimensional tensor being formed of values from the augmented output gradient tensor (i.e. the output gradient tensor with the additional dummy dimension).

As is known to those of skill in the art, a 3D convolution can be described as convolving a four-dimensional input tensor X<NUM> (H × W × Cin × O) and a five-dimensional weight tensor K<NUM> (M × N × Cin × Cout × KD). The fourth and fifth dimensions of the input tensor X<NUM> and weight tensor K<NUM> respectively (i.e. O, KD) are referred to as the depth dimension. The input tensor X<NUM> can be described as comprising O three-dimensional tensors (H × W × Cin) and each three-dimensional tensor may be referred to as a slice of the input tensor. The weight tensor K<NUM> may be described as comprising KD groups of three-dimensional filters (M × N × Cin) , wherein each filter group comprises Cout three-dimensional filters. Each group of filters may be referred to as a slice of the weight tensor.

The depth dimension of the weight tensor is smaller than or equal to the depth dimension of the input tensor (i.e. KD ≤ <NUM>). So, each filter is slid or traversed over three dimensions of the input tensor (i.e. height, width and depth dimensions). The output of a 3D convolution is a four-dimensional tensor Y<NUM> (I × J × Cout × L). As described in the Applicant's co-pending <CIT>, a 3D convolution can be represented as a set of 2D convolutions and summations.

Each neuron or element of the four-dimensional output tensor Y<NUM> (i,j,cout, l) of a 3D convolution can be generated from the four-dimensional input tensor X<NUM> and the five-dimensional weight tensor K<NUM> in accordance with equation (<NUM>). For simplicity, in this example it is assumed that the stride is one in all dimensions and the dilation is one in all dimensions. However, the methods and techniques described herein are not limited to cases where the stride is one and dilation is one.

Equation (<NUM>) can be written as equation (<NUM>) which shows a 3D convolution can be represented by a plurality of 2D convolutions and summations. Specifically, it can be seen that the inner part of the calculation (inside the square brackets) is made up of a plurality of 2D convolution operations like those described above with respect to <FIG> and equation (<NUM>).

This will be illustrated visually using <FIG>. Specifically, <FIG> shows a four-dimensional input tensor X<NUM> that has a depth of <NUM> (i.e. O = <NUM>) such that there are three slices of the input tensor <NUM><NUM>, <NUM><NUM>, <NUM><NUM>; and a five-dimensional weight tensor K<NUM> wherein there are two slices of the weight tensor <NUM><NUM>, <NUM>, (i.e. KD = <NUM>), and each slice <NUM><NUM>, <NUM>, comprises two <NUM>-dimensional filters <NUM><NUM>, <NUM><NUM> or <NUM><NUM>, <NUM><NUM> (i.e. Cout = <NUM>). <FIG> shows the five-dimensional weight tensor K<NUM> in a first position (first shift) where the first slice of the weight tensor <NUM><NUM> is aligned with the first slice of the input tensor <NUM><NUM>; and the second slice of the weight tensor <NUM>, is aligned with the second slice of the input tensor <NUM><NUM>.

When presented in this way, it can be seen that the calculation of the 3D convolution, for the first shift in the depth dimension, consists of two 2D convolutions (identified by a *) followed by an element-wise summation. The first slice of the input tensor <NUM><NUM> is convolved with the first slice of the weight tensor <NUM><NUM> (i.e. the first two filters <NUM><NUM>, <NUM><NUM>) to produce a first partial result <NUM><NUM>. The second slice of the input tensor <NUM>, is convolved with the second slice of the weight tensor <NUM>, (i.e. the last two filters <NUM><NUM>, <NUM><NUM>) to produce a second partial result <NUM><NUM>. Each of these partial results <NUM><NUM>, <NUM>, is equivalent to the result of a 2D convolution, such as that shown in <FIG>. To produce the first slice of the output <NUM><NUM> for the 3D convolution for this shift of the weight tensor K<NUM>, the two partial results <NUM><NUM> and <NUM><NUM> are summed by summation block <NUM>.

Then in a second shift (not shown) the first slice of the weight tensor <NUM><NUM> is aligned with the second slice of the input tensor <NUM><NUM> and the second slice of the weight tensor <NUM>, is aligned with the third slice of the input tensor <NUM><NUM>. Again, for this shift, there are two 2D convolutions followed by an element-wise summation. Specifically, the second slice of the input tensor <NUM>, is convolved with the first slice of the weight tensor <NUM><NUM> (i.e. filters <NUM><NUM>, <NUM><NUM>) to produce a third partial result. The third slice of the input tensor <NUM><NUM> is convolved with the second slice of the weight tensor <NUM>, (i.e. filters <NUM><NUM>, <NUM><NUM>) to produce a fourth partial result. To produce the second slice of the output for the 3D convolution for this shift of the weight tensor K<NUM>, the two partial results are summed.

The restructuring of a 3D convolution in terms of multiple 2D convolutions can also be represented in the form of a computational graph. Neural networks may be envisaged as computational graphs; the advantage to this representation is that a 3D convolution in a neural network can be replaced with an equivalent computational sub-graph, making deployment on a NNA (neural network accelerator) supporting 2D convolutions and element-wise summation straightforward. An example of this is shown in <FIG>. Here the input tensor is illustrated as comprising five constituent slices <NUM><NUM>, <NUM><NUM>, <NUM><NUM>, <NUM><NUM>, <NUM><NUM> (e.g. O = <NUM>). The weight tensor is represented as three slices (e.g. KD = <NUM>). In this example, the stride in the depth dimensions, <NUM>, is <NUM>. Based on these parameters, the output tensor can be represented as two output slices <NUM><NUM> and <NUM><NUM>. The first output slice <NUM><NUM> can be produced by performing three 2D convolutions <NUM><NUM>, <NUM><NUM>, <NUM><NUM> in which the first three slices of the input tensor <NUM><NUM>, <NUM><NUM>, <NUM><NUM>, are convolved with the first, second and third slices of the weight tensor. The outputs of these 2D convolutions are summed element-wise, at summation block <NUM><NUM>, to produce the first slice of the output tensor <NUM><NUM>.

The second slice of the output tensor <NUM>, can likewise be produced by performing three 2D convolutions <NUM><NUM>, <NUM><NUM>, <NUM><NUM> in which the third, fourth and fifth slices of the input tensor <NUM><NUM>, <NUM><NUM>, <NUM><NUM>, are convolved with the first, second and third slices of the weight tensor, respectively. The outputs of these 2D convolutions are summed element-wise, at summation block <NUM><NUM>, to produce the second slice of the output tensor <NUM><NUM>. Note that the weights in the 2D convolution <NUM><NUM> and the 2D convolution <NUM><NUM> are the same. In both cases these are the weights in the first slice of the weight tensor. Likewise, the weights in the 2D convolution <NUM>, and the 2D convolution <NUM><NUM> are the same (the second slice of the weight tensor); and the weights in the 2D convolution <NUM><NUM> and the 2D convolution <NUM><NUM> are the same (the third slice of the weight tensor).

The gradients of a loss or error metric of a DNN with respect to the weights of a convolution layer that implements a 3D convolution may be determined by representing the 3D convolution as a set of 2D convolutions and summations (e.g. as described above); performing the method of <FIG> for each 2D convolution, where the first three-dimensional tensor is formed of the set of values in the slice of the input tensor used in that 2D convolution, and the second four-dimensional filter is formed of the set of values of the output gradient tensor for the convolution layer that relate to the relevant slice of the output tensor; and combining the group convolution outputs that relate to the same weights. This will be described in more detail with reference to <FIG> and <FIG>.

Reference is now made to <FIG> which illustrates an example method <NUM> of identifying the gradients of a loss or error metric of a DNN with respect to the weights of a convolution layer that implements a 3D convolution using the method <NUM> of <FIG>. The method <NUM> begins at block <NUM> where the 3D convolution is represented (for example, using the method described above) as a set of 2D convolutions and summations. Each 2D convolution is between a group or slice of the input tensor and a group or slice of the weight tensor. For example, as shown in <FIG> and described above, a 3D convolution between an input tensor divided into five input slices or groups <NUM><NUM>, <NUM><NUM>, <NUM><NUM>, <NUM><NUM>, <NUM><NUM> and a weight tensor divided into three slices or groups of sub-filters, and the stride in the <NUM> dimension is two can be represented by the 2D convolutions shown in Table <NUM>.

At block <NUM>, once the 3D convolution is represented as a set of 2D convolutions and summations, the method <NUM> of <FIG> is performed for each 2D convolution wherein the first three-dimensional tensor of the group convolution is formed of the set of values of the input tensor relevant to the 2D convolution (i.e. the input slice relevant to that 2D convolution), and the second four-dimensional tensor of the group convolution is formed of the set of values of the output gradient tensor that correspond to the relevant output slice (i.e. the values of the output gradient tensor that correspond to the output slice that the 2D convolution output is used to generate). For example, the first 2D convolution <NUM><NUM> of <FIG> is performed between the first slice of the input tensor <NUM><NUM> and the first slice of the weight tensor, and the output thereof is used to generate the first slice of the output tensor <NUM><NUM>. Therefore, the group convolution for the first 2D convolution <NUM><NUM> is performed between a first three dimensional tensor-formed of the set of values in the first slice of the input tensor, and a second four-dimensional tensor formed of the set of values of the output gradient tensor G that correspond to the first slice of the output tensor <NUM><NUM>. Table <NUM> provides a complete list of the group convolutions that may be performed for a 3D convolution which can be represented by the 2D convolutions and summations set out in <FIG> and Table <NUM>.

The first three-dimensional tensor and the second four-dimensional tensor used in a group convolution can be generated from the relevant set of values of the input tensor and the relevant set of values of the output gradient tensor in any of the manners described above. Once the method <NUM> of <FIG> has been performed for each 2D convolution, the method <NUM> proceeds to block <NUM>.

At block <NUM>, the outputs of the group convolutions that are related to the same weights are combined to generate gradients of the loss or error metric for the DNN with respect to those weights. For example, in the example shown in <FIG> and Table <NUM> the first and fourth 2D convolutions <NUM><NUM> and <NUM><NUM> use the same weights (the first slice of the weight tensor). Thus, the outputs of the first and fourth group convolutions are combined (e.g. summed element wise) to generate gradients of the loss or error metric for the DNN with respect to those weights. Similarly, the second and fifth 2D convolutions <NUM>, and <NUM><NUM> use the same weights, so the output of the second and fifth group convolutions are combined; and the third and sixth 2D convolutions <NUM><NUM> and <NUM><NUM> use the same weights, so the output of the third and sixth group convolutions are combined. Once the relevant group convolution outputs are combined, the method <NUM> ends.

Although it has been described above how to represent a 3D convolution by a set of 2D convolutions and summations, any convolution of a dimension greater than or equal to <NUM> may be converted into a set of 2D convolutions and summations in a similar manner. Therefor the method of <FIG> may be used to determine the gradients of a loss or error metric of the DNN with respect to the weights of a convolution layer that implements a convolution of three dimensions or higher.

Therefore the method <NUM> of <FIG> for identifying the gradients of a loss or error metric of the DNN with respect to the weights of a convolution layer of a DNN, can be generalised as performing one or more group convolutions, each group convolution between a first y-dimensional tensor and a second z-dimensional tensor wherein z = y + <NUM>, the first y-dimensional tensor being formed of a set of values from the input tensor, and the second z-dimensional tensor being formed of a set of values from an output gradient tensor comprising gradients of the loss metric for the DNN with respect to the output values of the convolution layer. The number of group convolutions that are performed may be based on the dimension of the convolution implemented by the convolution layer and/or one or more other factors. For example, for a 1D or 2D convolution, a single group convolution may be performed as described above, whereas for a 3D convolution (or a convolution with a degree higher than <NUM>) multiple group convolutions may be performed, and the output thereof combined.

Reference is now made to <FIG> which illustrates an example method <NUM> of training a DNN using the method described above for determining the gradients of the loss metric with respect to the weights of a convolution layer of the DNN. The method <NUM> begins at block <NUM> where the output of a model of the DNN in response to input data (e.g. training data) is determined. The training data may be selected so as to configure the DNN to perform a specific function. The method <NUM> then proceeds to block <NUM> where an error or loss metric E is determined based on the output of the DNN. At block <NUM>, gradients of the error or loss metric E with respect to the outputs of a convolution layer of the DNN is determined (e.g. the output gradient tensor C for a convolution layer is determined). There are many known methods for generating the gradients of the error or loss metric E with respect to the outputs of a convolution layer. Then at block <NUM>, the gradients of the error or loss metric E with respect to the weights of the convolution layer may be determined in accordance with the method <NUM> of <FIG> or the method <NUM> of <FIG>. Then at block <NUM>, the weights of the convolution layer may be adjusted based on the gradients of the error or loss metric with respect to the weights.

In some cases, blocks <NUM> to <NUM> may be implemented for each convolution layer of the DNN. The method <NUM> of <FIG> may be repeated multiple times. In some cases, the method <NUM> of <FIG> may be repeated until the error or loss metric E converges. Once a set of adjusted weights have been identified, hardware logic configurable to implement a DNN (e.g. a DNN accelerator) may be configured to implement the DNN using the adjusted or trained weights.

As known to those of skill in the art, generating the gradients of the error or loss metric E with respect to the outputs (i.e. the elements of the output tensor) of a convolution layer in block <NUM> may comprise generating the gradients of the loss metric with respect to the inputs (i.e. the element of the input tensor) of a subsequent convolution layer in the DNN. For example, if a DNN comprises a first convolution layer that is subsequently followed (either directly or indirectly) by a second convolution layer, generating the gradients of the loss or error metric F with respect to the outputs (i.e. the elements of the output tensor) of the first convolution layer may comprise generating the gradients of the loss or error metric F with respect to the inputs (i.e. the elements of the input tensor) to the second convolution layer.

In some cases, the gradients of the loss or error metric F with respect to the inputs (i.e. the element of the input tensor) of a convolution layer <MAT> may be determined by performing a convolution transpose operation between the weights K of the convolution layer and the values of the output gradient tensor G for the convolution layer. Let V, which may be referred to as the input gradient tensor, represent the gradients of the loss or error metric F with respect to the inputs as shown in equation (<NUM>).

As described in the Applicant's published patent application <CIT>, a convolution transpose operation between a filter and an input tensor can be performed efficiently in hardware, such as a DNN accelerator, with one or more convolution engines, by performing a plurality of direct convolutions on the input tensor and interleaving the output of the direct convolutions to generate the output of the convolution transpose. Specifically, the filter is divided into a plurality of sub-filters; a convolution operation is performed between each sub-filter and the input tensor to generate a sub-output tensor; and the elements of the sub-output tensors are interleaved to generate the final output. As described in <CIT>, implementing a convolution transpose operation in this manner allows a convolution transpose operation to be implemented efficiently in hardware, such as a DNN accelerator, with one or more convolution engines, not only because it can take advantage of how efficiently the convolution engines can perform a convolution operation, but there are also no intermediate results that have to be stored.

This method can be described with reference to <FIG> which illustrate an example one dimensional (1D) convolution between a <NUM>-element input tensor P <NUM> and a <NUM>-weight filter D and the corresponding convolution transpose. A convolution operation can typically be represented as a matrix multiplication between an input vector PV and a sparse matrix S as shown in equation (<NUM>) where the non-zero elements of the sparse matrix S are the weights k of the filter. The input vector PV is the elements of the input tensor unrolled from left to right and top to bottom (and front to back if 3D).

In contrast, a convolution transpose layer (which may also be referred to as a deconvolution layer, a transpose convolution layer, or a fractionally strided convolution layer) performs the reverse operations of a convolution. Specifically, in a convolution transpose layer the input tensor P is processed by transposing the sparse matrix S for the corresponding direct convolution to generate a transposed sparse matrix ST and performing a matrix multiplication between the input vector PV and the transposed sparse matrix ST as shown in equation (<NUM>).

<FIG> illustrates a 1D convolution between a <NUM>-element input tensorP = [<NUM> p<NUM> p<NUM> p<NUM> p<NUM> pS p<NUM> p<NUM> p<NUM> p<NUM> p<NUM>] <NUM> with a leading padding zero and a <NUM>-weight filter D = [k<NUM> k<NUM> k<NUM>] with a stride of <NUM> in the width direction that produces a <NUM>-element output tensor Q = [q<NUM> q<NUM> q<NUM> q<NUM> q<NUM>] <NUM>. The output tensor is generated by sliding the filter D over the input tensor P <NUM> at two element increments and generating the dot product of the input elements and the weights at each position. Specifically, the ith output element qi is equal to the dot product of the input values and filter weights when the first filter weight k<NUM> is aligned with the (i * stride) - <NUM>)th input element. For example, the <NUM>st output element k<NUM> of the output tensor is equal to the dot product of the input elements and the filter weights when the <NUM>st filter weight k<NUM> is aligned with the <NUM>th input element; and the <NUM>nd output element q<NUM> is equal to the dot product of the input elements and the filter weights when the <NUM>st filter weight k<NUM> is aligned with the <NUM>nd input element p<NUM>. As described above, such a convolution can be expressed as a matrix multiplication between the corresponding input vector P and a sparse matrix S <NUM>.

<FIG> illustrates the corresponding convolution transpose operation (i.e. a convolution transpose between a <NUM>-element input tensor P = [p<NUM> p<NUM> p<NUM> p<NUM> p<NUM>] <NUM> and the <NUM>-weight filter D = [k<NUM> k<NUM> k<NUM>] of <FIG> that produces a <NUM>-element output tensor Q = [q<NUM> q<NUM> q<NUM> q<NUM> q<NUM> q<NUM> q<NUM> q<NUM> q<NUM> q<NUM>] <NUM>. As described above, a convolution transpose operation can be expressed as a matrix multiplication between an input vector P and the transposed sparse matrix ST <NUM>. As is shown in <FIG>, the convolution transpose operation is equivalent to padding the input tensor P with zeros between each element so that the input elements are spaced apart by the stride in the width and height directions and on one or more edges, and convolving the padded input tensor Pp with a reflected version of the filter DR.

It can be seen in <FIG> and <FIG> that each even output element is equal to the dot product of the ith and (i + <NUM>)th input elements and the third and first filter weights k<NUM> and k<NUM>, wherein i = n/<NUM> and n is the output element number. Accordingly, the even output elements can be generated by performing a direct convolution between the input tensor P and a first sub-filter Ds<NUM> = [k<NUM> k<NUM>]. Similarly, each odd output element is equal to the product of the ith input element pi and the second filter weight k<NUM>, wherein i = (n + <NUM>)/<NUM> and n is the output element number. Accordingly the odd output elements can be generated by performing a direct convolution between the input tensor P and a second sub-filter Ds<NUM> = [<NUM> k<NUM>]. The final output tensor Q of the convolution transpose can then be generated by interleaving the outputs of the first convolution and the second convolution as shown in <FIG>. Specifically, the final output tensor Q of the convolution transpose can be generated by alternating between the outputs of the first and second convolutions.

The method described in <CIT> may be used to efficiently calculate the input gradient tensor V. Specifically, the input gradient tensor V can be generated by dividing each filter of the weight tensor K into a plurality of sub-filters; a convolution operation is performed between each sub-filter and the output gradient tensor G to generate a sub-output tensor; and the elements of the sub-output tensors related to the same filter are interleaved to generate a layer or channel of the input gradient tensor V. For example, let there be a weight tensor K that comprises a first filter which is divided into two sub-filters, and a second filter which is divided into two sub-filters. The two sub-filters of the first filter are convolved with the output gradient tensor to produce two sub-output tensors. The elements of these two sub-output tensors are then interleaved to form a first channel of the input gradient tensor V. Then the two-sub-filters of the second filter are convolved with the output gradient tensor G to produce two sub-output tensors. The elements of these two sub-output tensors are then interleaved to form a second channel of the input gradient tensor V.

It will be evident to a person of skill in the art that the methods and techniques described above for determining the gradients of a loss metric of a DNN with respect to the weights of a convolution layer of the DNN are not limited to use in determining the gradients for a convolution layer that is part of a DNN, but can be used to determine the gradients for any convolution layer that is part of any neural network. Similarly, it will be evident to a person of skill in the art that the methods described above for training a DNN are not limited to use with a DNN, and can be used to train any neural network.

Reference is now made to <FIG> which illustrates example hardware logic which can be configured to implement a DNN and to implement the method <NUM> of <FIG> or the method <NUM> of <FIG> to determine the gradients of an error or loss metric with respect to the weights of a convolution layer of a DNN. Specifically, <FIG> illustrates an example DNN accelerator <NUM>.

The DNN accelerator <NUM> of <FIG> is configured to compute the output of a DNN through a series of hardware passes (which also may be referred to as processing passes) wherein during each pass the DNN accelerator receives at least a portion of the input data for a layer of the DNN and processes the received input data in accordance with that layer (and optionally in accordance with one or more following layers) to produce processed data. The processed data is either output to memory for use as input data for a subsequent hardware pass or output as the output of the DNN. The number of layers that the DNN accelerator can process during a single hardware pass may be based on the size of the data, the configuration of the DNN accelerator and the order of the layers. For example, where the DNN accelerator comprises hardware logic to perform each of the possible layer types a DNN that comprises a first convolution layer, a first activation layer, a second convolution layer, a second activation layer, and a pooling layer may be able to receive the initial DNN input data and process that input data according to the first convolution layer and the first activation layer in the first hardware pass and then output the output of the activation layer into memory, then in a second hardware pass receive that data from memory as the input and process that data according to the second convolution layer, the second activation layer, and the pooling layer to produce the output data for the DNN.

The example DNN accelerator <NUM> of <FIG> comprises input logic <NUM>, a convolution engine <NUM>, an accumulation buffer <NUM>, element-wise operations logic <NUM>, activation logic <NUM>, normalisation logic <NUM>, pooling logic <NUM>, output interleave logic <NUM> and output logic <NUM>. Each logic component or engine implements or processes all or a portion of one or more types of layers. Specifically, together the convolution engine <NUM> and the accumulation buffer <NUM> can implement or process a convolution layer or a fully connected layer. The activation logic <NUM> can process or implement an activation layer. The normalisation logic <NUM> processes or implement a normalisation layer. The pooling logic <NUM> can implement a pooling layer and the output interleave logic <NUM> can perform interleaving of tensors.

The input logic <NUM> is configured to receive the input data to be processed and provides it to a downstream logic component for processing. The input logic <NUM> may comprise a memory interface which is configured to receive the data to be processed by the downstream logic component from memory, and/or an input buffer for storing the input data for a layer. In some cases the input logic <NUM> may also receive the weights of a layer and store the received weights in the coefficient buffer <NUM>.

The convolution engine <NUM> is configurable to perform a convolution operation on the received input data using the weights associated with a particular convolution layer. The weights for each convolution layer of the DNN may be stored in a coefficient buffer <NUM> as shown in <FIG> and the weights for a particular convolution layer may be provided to the convolution engine <NUM> when that particular convolution layer is being processed by the convolution engine <NUM>. Where the DNN accelerator supports variable weight formats then the convolution engine <NUM> may be configured to receive information indicating the format or formats of the weights of the current convolution layer being processed to allow the convolution engine <NUM> to properly interpret and process the received weights.

The convolution engine <NUM> may comprise a plurality of multipliers (e.g. <NUM>) and a plurality of adders which add the results of the multipliers to produce a single sum. Although a single convolution engine <NUM> is shown in <FIG>, in other examples there may be multiple (e.g. <NUM>) convolution engines so that multiple windows can be processed simultaneously. The output of the convolution engine <NUM> is fed to the accumulation buffer <NUM>.

The accumulation buffer <NUM> is configured to receive the output of the convolution engine and add it to the current contents of the accumulation buffer <NUM>. In this manner, the accumulation buffer <NUM> accumulates the results of the convolution engine <NUM> over several hardware passes of the convolution engine <NUM>. Although a single accumulation buffer <NUM> is shown in <FIG>, in other examples there may be multiple (e.g. <NUM>, one per convolution engine) accumulation buffers. The accumulation buffer <NUM> outputs the accumulated result to the element-wise operations logic <NUM> which may or may not operate on the accumulated result depending on whether an element-wise layer is to be processed during the current hardware pass.

The element-wise operations logic <NUM> is configurable to receive either the input data for the current hardware pass (e.g. when a convolution layer is not processed in the current hardware pass) or the accumulated result from the accumulation buffer <NUM> (e.g. when a convolution layer is processed in the current hardware pass). The element-wise operations logic <NUM> may either process the received input data or pass the received input data to other logic (e.g. the activation logic <NUM> and/or or the normalisation logic <NUM>) depending on whether an element-wise layer is processed in the current hardware pass and/or depending on whether an activation layer is to be processed prior to an element-wise layer. When the element-wise operations logic <NUM> is configured to process the received input data the element-wise operations logic <NUM> performs an element-wise operation on the received data (optionally with another data set (which may be obtained from external memory)). The element-wise operations logic <NUM> may be configurable to perform any suitable element-wise operation such as, but not limited to add, multiply, maximum, and minimum. The result of the element-wise operation is then provided to either the activation logic <NUM> or the normalisation logic <NUM> depending on whether an activation layer is to be processed subsequent the element-wise layer or not.

The activation logic <NUM> is configured to receive one of the following as input data: the original input to the hardware pass (via the element-wise operations logic <NUM>) (e.g. when a convolution layer is not processed in the current hardware pass); the accumulated data (via the element-wise operations logic <NUM>) (e.g. when a convolution layer is processed in the current hardware pass and either an element-wise layer is not processed in the current hardware pass or an element-wise layer is processed in the current hardware pass but follows an activation layer). The activation logic <NUM> is configurable to apply an activation function to the input data and provide the output data back to the element-wise operations logic <NUM> where it is forwarded to the normalisation logic <NUM> directly or after the element-wise operations logic <NUM> processes it. In some cases, the activation function that is applied to the data received by the activation logic <NUM> may vary per activation layer. In these cases, information specifying one or more properties of an activation function to be applied for each activation layer may be stored (e.g. in memory) and the relevant information for the activation layer processed in a particular hardware pass may be provided to the activation logic <NUM> during that hardware pass.

In some cases, the activation logic <NUM> may be configured to store, in entries of a lookup table, data representing the activation function. In these cases, the input data may be used to lookup one or more entries in the lookup table and output values representing the output of the activation function. For example, the activation logic <NUM> may be configured to calculate the output value by interpolating between two or more entries read from the lookup table.

In some examples, the activation logic <NUM> may be configured to operate as a Rectified Linear Unit (ReLU) by implementing a ReLU function. In a ReLU function, the output element yi,j,k is calculated by identifying a maximum value as set out in equation (<NUM>) wherein for x values less than <NUM>, y = <NUM>: <MAT>.

In other examples, the activation logic <NUM> may be configured to operate as a Parametric Rectified Linear Unit (PReLU) by implementing a PReLU function. The PReLU function performs a similar operation to the ReLU function. Specifically, where <MAT> are constants, the PReLU is configured to generate an output element yi,j,k as set out in equation (<NUM>): <MAT>.

The normalisation logic <NUM> is configurable to receive one of the following as input data: the original input data for the hardware pass (via the element-wise operations logic <NUM>) (e.g. when a convolution layer is not processed in the current hardware pass and neither an element-wise layer nor an activation layer is processed in the current hardware pass); the accumulation output (via the element-wise operations logic <NUM>) (e.g. when a convolution layer is processed in the current hardware pass and neither an element-wise layer nor an activation layer is processed in the current hardware pass); and the output data of the element-wise operations logic and/or the activation logic. The normalisation logic <NUM> then performs a normalisation function on the received input data to produce normalised data. In some cases, the normalisation logic <NUM> may be configured to perform a Local Response Normalisation (LRN) Function and/or a Local Contrast Normalisation (LCN) Function. However, it will be evident to a person of skill in the art that these are examples only and that the normalisation logic <NUM> may be configured to implement any suitable normalisation function or functions. Different normalisation layers may be configured to apply different normalisation functions.

The pooling logic <NUM> may receive the normalised data from the normalisation logic <NUM> or may receive the input data to the normalisation logic <NUM> via the normalisation logic <NUM>. In some cases, data may be transferred between the normalisation logic <NUM> and the pooling logic <NUM> via a crossbar ("XBar") <NUM>. The term crossbar is used herein to refer to a simple hardware logic that contains routing logic which connects multiple logic components together in a dynamic fashion. In this example, the crossbar may dynamically connect the normalisation logic <NUM>, the pooling logic <NUM> and/or the output interleave logic <NUM> depending on which layers will be processed in the current hardware pass. Accordingly, the crossbar may receive information each pass indicating which logic components <NUM>, <NUM>, <NUM> are to be connected.

The pooling logic <NUM> is configurable to perform a pooling function, such as, but not limited to, a max or mean function, on the received data to produce pooled data. The purpose of a pooling layer is to reduce the spatial size of the representation to reduce the number of parameters and computation in the network, and hence to also control overfitting. In some examples, the pooling operation is performed over a sliding window that is defined per pooling layer.

The output interleave logic <NUM> may receive the normalised data from the normalisation logic <NUM>, the input data to the normalisation function (via the normalisation logic <NUM>), or the pooled data from the pooling logic <NUM>. In some cases, the data may be transferred between the normalisation logic <NUM>, the pooling logic <NUM> and the output interleave logic <NUM> via a crossbar <NUM>. The output interleave logic <NUM> is configured to perform a rearrangement operation to produce data that is in a predetermined order. This may comprise sorting and/or transposing the received data. The data generated by the last of the layers is provided to the output logic <NUM> where it is converted to the desired output format for the current hardware pass.

The normalisation logic <NUM>, the pooling logic <NUM>, and the output interleave logic <NUM> may each have access to a shared buffer <NUM> which can be used by these logic components <NUM>, <NUM> and <NUM> to write data to and retrieve data from. For example, the shared buffer <NUM> may be used by these logic components <NUM>, <NUM>, <NUM> to rearrange the order of the received data or the generated data. For example, one or more of these logic components <NUM>, <NUM>, <NUM> may be configured to write data to the shared buffer <NUM> and read the same data out in a different order. In some cases, although each of the normalisation logic <NUM>, the pooling logic <NUM> and the output interleave logic <NUM> have access to the shared buffer <NUM>, each of the normalisation logic <NUM>, the pooling logic <NUM> and the output interleave logic <NUM> may be allotted a portion of the shared buffer <NUM> which only they can access. In these cases, each of the normalisation logic <NUM>, the pooling logic <NUM> and the output interleave logic <NUM> may only be able to read data out of the shared buffer <NUM> that they have written into the shared buffer <NUM>.

The logic components of the DNN accelerator <NUM> that are used or active during any hardware pass are based on the layers that are processed during that hardware pass. In particular, only the logic components related to the layers processed during the current hardware pass are used or active. As described above, the layers that are processed during a particular hardware pass is determined (typically in advance, by, for example, a software tool) based on the order of the layers in the DNN and optionally one or more other factors (such as the size of the data). For example, in some cases the DNN accelerator may be configured to perform the processing of a single layer per hardware pass unless multiple layers can be processed without writing data to memory between layers. For example, if a first convolution layer is immediately followed by a second convolution layer each of the convolution layers would have to be performed in a separate hardware pass as the output data from the first hardware convolution needs to be written out to memory before it can be used as an input to the second. In each of these hardware passes only the logic components, or engines relevant to a convolution layer, such as the convolution engine <NUM> and the accumulation buffer <NUM>, may be used or active.

Although not shown in <FIG>, the DNN accelerator <NUM> may also comprise a memory management module, as described above, which is configured to read from the input logic <NUM> and write to the output logic <NUM>.

Although the DNN accelerator <NUM> of <FIG> illustrates a particular order in which the logic components, engines etc. are arranged and thus how the processing of data flows through the DNN accelerator, it will be appreciated that this is an example only and that in other examples the logic components, engines may be arranged in a different manner. Furthermore, other hardware logic (e.g. other DNN accelerators) may implement additional or alternative types of DNN layers and thus may comprise different logic components, engines etc..

<FIG> illustrates various components of an exemplary general purpose computing-based device <NUM> which may be implemented as any form of a computing and/or electronic device, and in which embodiments of the methods <NUM>, <NUM>, <NUM> of <FIG>, <FIG> and <FIG> described above may be implemented.

Computing-based device <NUM> comprises one or more processors <NUM> which may be microprocessors, controllers or any other suitable type of processors for processing computer executable instructions to control the operation of the device in order to assess the performance of an integrated circuit defined by a hardware design in completing a task. In some examples, for example where a system on a chip architecture is used, the processors <NUM> may include one or more fixed function blocks (also referred to as accelerators) which implement a part of the method of determining the gradients of an error of loss metric of a DNN with respect to weights of a convolution layer (rather than software or firmware). Platform software comprising an operating system <NUM> or any other suitable platform software may be provided at the computing-based device to enable application software, such as computer executable code <NUM> for implementing one or more of the methods <NUM>, <NUM>, <NUM> of <FIG>, <FIG> and <FIG>, to be executed on the device.

The computer executable instructions may be provided using any computer-readable media that is accessible by computing-based device <NUM>. Computer-readable media may include, for example, computer storage media such as memory <NUM> and communications media. Computer storage media (i.e. non-transitory machine readable media), such as memory <NUM>, includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data. Computer storage media includes, but is not limited to, RAM, ROM, EPROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information for access by a computing device. In contrast, communication media may embody computer readable instructions, data structures, program modules, or other data in a modulated data signal, such as a carrier wave, or other transport mechanism. As defined herein, computer storage media does not include communication media. Although the computer storage media (i.e. non-transitory machine readable media, e.g. memory <NUM>) is shown within the computing-based device <NUM> it will be appreciated that the storage may be distributed or located remotely and accessed via a network or other communication link (e.g. using communication interface <NUM>).

The computing-based device <NUM> also comprises an input/output controller <NUM> arranged to output display information to a display device <NUM> which may be separate from or integral to the computing-based device <NUM>. The display information may provide a graphical user interface. The input/output controller <NUM> is also arranged to receive and process input from one or more devices, such as a user input device <NUM> (e.g. a mouse or a keyboard). In an embodiment the display device <NUM> may also act as the user input device <NUM> if it is a touch sensitive display device. The input/output controller <NUM> may also output data to devices other than the display device, e.g. a locally connected printing device (not shown in <FIG>).

<FIG> shows a computer system in which the hardware logic (e.g. DNN accelerator <NUM>) configurable to implement a DNN described herein may be implemented. The computer system comprises a CPU <NUM>, a GPU <NUM>, a memory <NUM> and other devices <NUM>, such as a display <NUM>, speakers <NUM> and a camera <NUM>. The computer system also comprises hardware logic configurable to implement a DNN <NUM> (e.g. the DNN accelerator <NUM> of <FIG>) which may receive control information from the CPU <NUM> and/or the GPU <NUM>. The components of the computer system can communicate with each other via a communications bus <NUM>. In other examples, the hardware logic configurable to implement a DNN <NUM> may be implemented as part of the CPU or the GPU. In some examples, there may not be a GPU.

The DNN accelerator <NUM> of <FIG> is shown as comprising a number of functional blocks. This is schematic only and is not intended to define a strict division between different logic elements of such entities. Each functional block may be provided in any suitable manner. It is to be understood that intermediate values described herein as being formed by a DNN accelerator or a processing module need not be physically generated by the DNN accelerator or the processing module at any point and may merely represent logical values which conveniently describe the processing performed by the DNN accelerator or the processing module between its input and output.

The hardware logic configurable to implement a DNN (e.g. the DNN accelerator <NUM> of <FIG>) described herein may be embodied in hardware on an integrated circuit. Generally, any of the functions, methods, techniques or components described above can be implemented in software, firmware, hardware (e.g., fixed logic circuitry), or any combination thereof. The terms "module," "functionality," "component", "element", "unit", "block" and "logic" may be used herein to generally represent software, firmware, hardware, or any combination thereof. In the case of a software implementation, the module, functionality, component, element, unit, block or logic represents program code that performs the specified tasks when executed on a processor. The algorithms and methods described herein could be performed by one or more processors executing code that causes the processor(s) to perform the algorithms/methods. Examples of a computer-readable storage medium include a random-access memory (RAM), read-only memory (ROM), an optical disc, flash memory, hard disk memory, and other memory devices that may use magnetic, optical, and other techniques to store instructions or other data and that can be accessed by a machine.

It is also intended to encompass software which defines a configuration of hardware as described herein, such as HDL (hardware description language) software, as is used for designing integrated circuits, or for configuring programmable chips, to carry out desired functions. That is, there may be provided a computer readable storage medium having encoded thereon computer readable program code in the form of an integrated circuit definition dataset that when processed (i.e. run) in an integrated circuit manufacturing system configures the system to manufacture hardware logic configurable to implement a DNN (e.g. DNN accelerator) described herein. An integrated circuit definition dataset may be, for example, an integrated circuit description.

Therefore, there may be provided a method of manufacturing, at an integrated circuit manufacturing system, hardware logic configurable to implement a DNN (e.g. DNN accelerator <NUM> of <FIG>) as described herein. Furthermore, there may be provided an integrated circuit definition dataset that, when processed in an integrated circuit manufacturing system, causes the method of manufacturing hardware logic configurable to implement a DNN (e.g. DNN accelerator <NUM> of <FIG>) to be performed.

An example of processing an integrated circuit definition dataset at an integrated circuit manufacturing system so as to configure the system to manufacture hardware logic configurable to implement a DNN (e.g. DNN accelerator) will now be described with respect to.

<FIG> shows an example of an integrated circuit (IC) manufacturing system <NUM> which is configured to manufacture hardware logic configurable to implement a DNN (e.g. DNN accelerator) as described in any of the examples herein. In particular, the IC manufacturing system <NUM> comprises a layout processing system <NUM> and an integrated circuit generation system <NUM>. The IC manufacturing system <NUM> is configured to receive an IC definition dataset (e.g. defining hardware logic configurable to implement a DNN (e.g. DNN accelerator) as described in any of the examples herein), process the IC definition dataset, and generate an IC according to the IC definition dataset (e.g. which embodies hardware logic configurable to implement a DNN (e.g. DNN accelerator) as described in any of the examples herein). The processing of the IC definition dataset configures the IC manufacturing system <NUM> to manufacture an integrated circuit embodying hardware logic configurable to implement a DNN (e.g. DNN accelerator) as described in any of the examples herein.

In other examples, processing of the integrated circuit definition dataset at an integrated circuit manufacturing system may configure the system to manufacture hardware logic configurable to implement a DNN (e.g. DNN accelerator) without the IC definition dataset being processed so as to determine a circuit layout. For instance, an integrated circuit definition dataset may define the configuration of a reconfigurable processor, such as an FPGA, and the processing of that dataset may configure an IC manufacturing system to generate a reconfigurable processor having that defined configuration (e.g. by loading configuration data to the FPGA).

Claim 1:
A method (<NUM>) of generating, at a computer system comprising a neural network accelerator, gradients of a loss metric for a neural network (NN) with respect to weights of a convolution layer of the NN, the convolution layer of the NN configured to receive an input tensor (<NUM>, <NUM>) of input values and a weight tensor of weights, and generate an output tensor of output values, the method (<NUM>) comprising:
performing (<NUM>), using a plurality of convolution engines of the neural network accelerator, a group convolution between a first y-dimensional tensor (<NUM>, <NUM>) and a second z-dimensional tensor (<NUM>, <NUM>) wherein z = y + <NUM> and the second z-dimensional tensor comprises one or more filters and each filter comprises a single channel, the first y-dimensional tensor (<NUM>, <NUM>) being formed of a set of values from the input tensor, and the second z-dimensional tensor (<NUM>, <NUM>) being formed of a set of values from an output gradient tensor (<NUM>, <NUM>) comprising gradients of the loss metric for the NN with respect to the output values, each convolution engine of the plurality of convolution engines being dedicated hardware configured to perform convolution operations;
wherein the first y-dimensional tensor (<NUM>, <NUM>), the second z-dimensional tensor (<NUM>, <NUM>) and the group convolution are configured to generate an output of a convolution operation between each channel of the set of values of the input tensor and each channel of the set of values of the output gradient tensor;
wherein the group convolution between the first y-dimensional tensor (<NUM>, <NUM>) and the second z-dimensional tensor (<NUM>, <NUM>) is divided into a plurality of distinct sub-convolutions, and performing the group convolution using the plurality of convolution engines of the neural network accelerator comprises processing in parallel multiple sub-convolutions of the plurality of sub-convolutions by separate convolution engines of the plurality of convolution engines.