Patent Description:
Typically, neural networks comprise computationally intensive operations that are distributed in parallel across many processors. Such operations can include matrix multiplication, gradient descent mechanisms, and other like operations. Moreover, such operations are typically performed on large quantities of numerical values, which are expressed in standard formats where a single value is represented by thirty-two bits or more of data. Due to the large quantities of numerical values, the storage of such numerical values may require hundreds of gigabytes, or even terabytes of data storage capacity. Additionally, the processing of such numerical values, such as the matrix multiplication thereof, can consume significant processing capabilities. However, merely reducing the precision of the format in which such numerical values are represented, such as, for example, by representing numerical values utilizing only eight bits of data, instead of thirty-two bits, provides insufficient precision. Thus, while such a reduction in precision can decrease memory consumption and processor utilization, it fails to provide sufficient precision to enable the operations to complete properly. For example, when training a neural network, iterative processing converges upon weight values that are to be subsequently utilized by the neural network to perform an evaluation. Insufficient precision in the format utilized to represent values can result in the weight values failing to converge, and, consequently, the neural network failing to be trained. <NPL>, describes that DNNs are ubiquitous datacenter workloads, requiring orders of magnitude more computing power from servers than traditional workloads. <NPL>, describes a method to train deep neural networks using <NUM>-bit floating point representation for weights, activations, errors, and gradients.

Bounding box quantization can reduce the quantity of bits utilized to express numerical values prior to the multiplication of matrices comprised of such numerical values, thereby reducing both memory consumption and processor utilization. Stochastic rounding can provide sufficient precision to enable the storage of weight values in reduced-precision formats without having to separately store weight values in a full-precision format. Alternatively, other rounding mechanisms, such as round to nearest, can be utilized to exchange weight values in reduced-precision formats, while also storing weight values in full-precision formats for subsequent updating. To facilitate the conversion of the representation of numerical values in full-precision formats to reduced-precision formats, reduced-precision formats such as brain floating-point format can be utilized which can comprise a same quantity of bits utilized to represent an exponential value as a full-precision format, thereby making such conversion as simple as discarding the least significant bits of the significand. Similarly, to the extent that existing processes are utilized that consume numerical values expressed in full-precision formats, upconversion to such full-precision formats, including by bit padding reduced-precision formats, can be performed.

Additional features and advantages will be made apparent from the following detailed description that proceeds with reference to the accompanying drawings.

The following detailed description may be best understood when taken in conjunction with the accompanying drawings, of which:.

The following description relates to the successful training of neural networks utilizing values expressed in reduced-precision format, thereby decreasing the memory and storage utilization and processor consumption during such training. Bounding box quantization can reduce the quantity of bits utilized to express numerical values prior to the multiplication of matrices comprised of such numerical values, thereby reducing both memory consumption and processor utilization. Stochastic rounding can provide sufficient precision to enable the storage of weight values in reduced-precision formats without having to separately store weight values in a full-precision format. Alternatively, other rounding mechanisms, such as round to nearest, can be utilized to exchange weight values in reduced-precision formats, while also storing weight values in full-precision formats for subsequent updating. To facilitate the conversion of the representation of numerical values in full-precision formats to reduced-precision formats, reduced-precision formats such as brain floating-point format can be utilized which can comprise a same quantity of bits utilized to represent an exponential value as a full-precision format, thereby making such conversion as simple as discarding the least significant bits of the significand. Similarly, to the extent that existing processes are utilized that consume numerical values expressed in full-precision formats, upconversion to such full-precision formats, including by bit padding reduced-precision formats, can be performed.

Although not required, the description below will be in the general context of computer-executable instructions, such as program modules, being executed by a computing device. More specifically, the description will reference acts and symbolic representations of operations that are performed by one or more computing devices or peripherals, unless indicated otherwise. As such, it will be understood that such acts and operations, which are at times referred to as being computer-executed, include the manipulation by a processing unit of electrical signals representing data in a structured form. This manipulation transforms the data or maintains it at locations in memory, which reconfigures or otherwise alters the operation of the computing device or peripherals in a manner well understood by those skilled in the art. The data structures where data is maintained are physical locations that have particular properties defined by the format of the data.

Generally, program modules include routines, programs, objects, components, data structures, and the like that perform particular tasks or implement particular abstract data types. Moreover, those skilled in the art will appreciate that the computing devices need not be limited to conventional personal computers, and include other computing configurations, including servers, hand-held devices, multi-processor systems, microprocessor based or programmable consumer electronics, network PCs, minicomputers, mainframe computers, and the like. Similarly, the computing devices need not be limited to stand-alone computing devices, as the mechanisms may also be practiced in distributed computing environments where tasks are performed by remote processing devices that are linked through a communications network. In a distributed computing environment, program modules may be located in both local and remote memory storage devices.

Prior to detailing mechanisms by which neural network training can be accomplished more efficiently, with decreased memory consumption and decreased processor utilization, a general overview of neural network operations and the training thereof is provided to act as context for the descriptions below. Neural networks, including deep neural networks (DNNs) and convolution neural networks (CNNs), can achieve high accuracy on human recognition tasks such as image and speech recognition. Neural networks may include a number of different processing "layers", including linear layers and convolutional layers. Outputs of convolutional layers may be processed with "pooling" operations which subsample the convolved output, and non-linearization steps, such as sigmoid or tanh applied to the output. <FIG> is a simplified diagram depicting a three-dimensional (3D) CNN <NUM> that includes three exemplary 3D volumes, namely the exemplary volumes <NUM>, <NUM>, <NUM>. Each 3D volume <NUM>, <NUM>, <NUM> can represent an input to a layer, and can transformed into a new 3D volume feeding a subsequent layer. In the example of <FIG>, there are two convolutional layers, namely the exemplary convolution layers <NUM> and <NUM>. Volume <NUM>, with <NUM> planes, can be an input to convolutional layer <NUM>, which can generate volume <NUM>, with H planes, which, in tum, can be an input to convolutional layer <NUM>, which can generate volume <NUM>, with K planes.

For example, volume <NUM> can include image data in three planes, such as the well-known "red", "green" and "blue" layers of a color image. Each plane can include a two-dimensional array of data. For example, if the exemplary volume <NUM> was a portion of an image, then the portion could be, for example, one-hundred pixels wide by one-hundred pixels high. In such an instant, the variable "J", shown in <FIG>, can be a value of one-hundred. More or fewer than three planes may be used, and each plane need not include a square array. Additionally, data other than image data may be used.

A 3D input volume, such as the exemplary input volume <NUM>, can be convolved with weight kernels. For example, as shown in <FIG>, the exemplary input volume <NUM> can be of dimensions L x L x D, where D is three in the present example. Such an exemplary input volume can be convolved with kernel weights, such as the exemplary kernel weights <NUM>, which can also have a dimension of L x L x D, with, again, the dimension D being three in the present example. Each kernel weight can shifted in a sliding-window-like fashion across the input volume, such as the exemplary volume <NUM>. A stride value can define an amount of such a shift offset. During each shift, each weight in to the 3D kernel weight is multiplied and added with corresponding pair-wise input elements from the overlapping region of input volume <NUM>.

Such a process is illustrated in greater detail in <FIG>. More specifically, <FIG> shows an exemplary convolution of a volume, namely the exemplary volume <NUM>, with a set of kernel weights, namely the exemplary kernel weights <NUM>, to generate a first plane <NUM> of a second volume <NUM>, that is shown in <FIG>. The exemplary first volume <NUM> can comprise three planes, namely the three exemplary planes <NUM>, <NUM> and <NUM>, with a nine-by-nine array of image data, for example. As indicated above, in the context of image data, the three exemplary planes can comprise a single plane of each of the colors red, green and blue. Each of a first set of kernel weights <NUM> can have an exemplary dimensionality of three-by-three-by-three.

As illustrated in <FIG>, data value x0 of an exemplary first plane <NUM> of the exemplary second volume <NUM> can be determined by multiplying every weight in the first set of kernel weights <NUM> with every pair-wise input element from the overlapping region of a first input volume, such as the overlapping regions <NUM>, <NUM> and <NUM>. According to one aspect, the data value x0 can be expressed as follows: <MAT>.

First set of kernel weights <NUM> can then slide by a quantity of horizontal data values of first volume <NUM> determined by the stride value. <FIG> illustrates a stride value of two. Thus, as illustrated in <FIG>, data value x1 of the exemplary first plane <NUM> of the exemplary second volume <NUM> can determined by multiplying every weight in the first set of kernel weights <NUM> with every pair-wise input element from a slid overlapping region of a second input volume, such as the exemplary slid overlapping region <NUM>, <NUM> and <NUM>. According to one aspect, the data value x1 can be expressed as follows: <MAT>.

Such a process can continue, with the first set of kernel weights <NUM> sliding two horizontal values, in the illustrated example, of the exemplary first volume <NUM>, with each iteration, until the first row of data values (x0, x1, x3, x3) of first plane <NUM> is complete. The first set of kernel weights <NUM> can then slide down, for example, two rows and back to the leftmost column of first volume <NUM> to calculate the second row of data values (x4, x5, x6, x7) of first plane <NUM>. This process can continue until all four rows of data values of the first plane <NUM> are complete.

As illustrated in <FIG>, the exemplary data value y4 of the exemplary second plane <NUM> of second volume <NUM> can be determined by multiplying every weight in the second set of kernel weights <NUM> with every pair-wise input element from the down-slid overlapping region <NUM>, <NUM> and <NUM>. According to one aspect, the data value y4 can be expressed as follows: <MAT>.

Such a process can continue until all data values of second plane <NUM> of second volume <NUM> are complete, and also continues for each of the H weight volumes to generate the H planes in of second volume <NUM>. Referring again to <FIG>, volume <NUM>, determined such as in the manner illustrated by <FIG>, and detailed above, then becomes an input layer to convolutional layer <NUM>, which can include K weight volumes to generate the K planes of volume <NUM>.

According to one aspect, the values of the weights utilized in the convolutions detailed above can be derived as part of the "training" of a neural network. Typically, such training starts with initial weight values proceeds iteratively, where, for each iteration, the weight values are modified in accordance with information, such as gradient information, obtained during the processing of a prior iteration. As such, the training typically entails the performance of so-called "forward" processing, or forward propagation, and "backwards" processing, or backpropagation. More specifically, forward propagation of one or more input activations through the neural network can be utilized to generate output activations, which can be prediction. Then, gradients can be determined for each of the neurons in the neural network via back-propagation of "errors" from the output layer back to the input layer. Such gradients can then be utilized to update the weights at each neuron. Repetition of such processes can continue until the weights converge.

In gradient descent, several choices can be available for selecting a number of inputs to use per iteration. A first method, batch gradient descent, can utilize all available training data, such as, for example, pre-labeled images, in each iteration between weight updates. This method can be very expensive. A second method, stochastic gradient descent, can represent another extreme by selecting one random example from the corpus between weight updates. A third method, mini-batch gradient descent, can use a random subset of the corpus to perform gradient computation, followed by a single weight update.

In practice, mini-batch gradient descent can often be a good balance between training accuracy and training time. Furthermore, mini-batching can facilitate implementation of available parallelism in hardware, by allowing gradients for different inputs to be computed in parallel (without a serial dependence on weight updates). The remainder of this description will assume mini-batch gradient descent for training.

<FIG> illustrates an implementation of forward propagation of a single training input through a single layer. In this example, the input activations are represented by a <NUM>-tuple vector [x<NUM>, x1, x<NUM>, x<NUM>]T in Layer i-<NUM>. Every neuron in Layer i processes the input vector of Layer i-<NUM> using an activation function and generates output activations of Layer i. Typically, the activation function can be a weighted sum of products, taking the input activation of each neuron and scaling it by a tunable weight parameter. The dot product is further transformed by a non-linear differentiable function such as hyperbolic tangent, sigmoid or other non-linear differentiable function.

In the implementation depicted in <FIG>, the output activations y<NUM>, y<NUM>, y<NUM>, y<NUM> of Layer i are expressed as: <MAT> <MAT> <MAT> <MAT>.

Forward propagation can be expressed mathematically as follows: the output activations of Layer i (before non-linearization) can equal a matrix of weights for Layer i multiplied by a vector of input activations from Layer i-<NUM>: <MAT>.

Thus, the bulk of computation is in computing the dot products. In mini-batch gradient descent, multiple input activation vectors can be processed per iteration of forward propagation. In this case, the mathematical expression shown above generalizes to matrix-matrix multiplication followed by the non-linear transformation.

After forward propagation, the result can be a set of output activations in the final output layer. In backpropagation, these output activations can then be utilized to compute output "errors" that can be propagated backwards through the network, to compute the gradients at each neuron. An example implementation of backpropagation is depicted in <FIG>.

As illustrated in <FIG>, errors can propagate backwards in a similar fashion to forward propagation. One difference is that the error function can take, as input, the weighted sum of products and the original input activation used in a derivative term, as follows: <MAT> <MAT> <MAT> <MAT> where h'() is a derivative function.

For example, if the non-linear differentiable function in the forward propagation is the sigmoid function, the errors at Layer i-<NUM> may be expressed as: <MAT> <MAT> <MAT> <MAT>.

Backpropagation can be expressed mathematically as follows: <MAT> That is, a transposed weight matrix (Layer i) multiplied by an input error vector (Layer i) equals an output error vector (Layer i-<NUM>) (before multiplying by the derivative).

Backpropagation can thus be similarly expressed as a matrix-vector multiplication that takes in a transposed weight matrix multiplied against a vector of errors, and scaled by the original activations computed during forward propagation. As indicated previously, the use of mini-batching generalizes this operation to matrix-matrix multiplication.

Once the errors are computed for each neuron, each neuron's weights are updated. In gradient descent, the weight update is given by: <MAT> where µ is a parameter that represents the learning rate of the neural network.

As detailed above, forward propagation and backpropagation can require substantial use of matrix-matrix and matrix-vector multiplication operators. Additionally, a substantial quantity of values can need to be exchanged among multiple different computing devices during the forward propagation and backpropagation stages. Traditionally, neural network training, such as that detailed above, is performed utilizing values expressed in full-precision formats, such as the ubiquitous <NUM>-bit single precision floating-point format. For example, the exemplary system <NUM>, illustrated in <FIG>, illustrates the utilization of full-precision formats to express the values utilized by the neural network training process. In particular, activations from a prior layer, represented as the exemplary activations <NUM> in <FIG>, can be expressed in a full-precision format and can be utilized as input into one or more matrix multiplications, such as the exemplary matrix multiplication <NUM>. Similarly, weights from a current layer, represented as the exemplary weights <NUM>, can also be expressed in the full-precision format and can be provided as the other input into one or more matrix multiplications, such as the exemplary matrix multiplication <NUM>. Output from such matrix multiplications can be activations of a current layer, represented as the exemplary activations <NUM>, which can also be expressed in the full-precision format. The forward propagation <NUM>, comprising the provision of the activations <NUM> and the weights <NUM> to the matrix multiplication <NUM> to generate the activations <NUM>, is meant as a more general illustration of the forward propagation detailed above, such as with reference to <FIG>.

Similarly, activations from the prior layer, represented as the exemplary activations <NUM>, and errors from a current layer, represented as the exemplary errors <NUM>, can be provided as input to one or more matrix multiplications, such as the exemplary matrix multiplication <NUM>. Again, the exemplary activations <NUM> and exemplary errors <NUM> can have their values expressed in the full-precision format, as illustrated in <FIG>. Likewise, the exemplary errors <NUM> can be provided, as input, together with weights from a current layer, such as the exemplary weights <NUM>, to one or more matrix multiplications, such as the exemplary matrix multiplication <NUM>. The exemplary errors <NUM> and the exemplary weights <NUM> can have their values expressed in the full-precision format. Output of the exemplary matrix multiplication <NUM> and <NUM> can also be expressed in the full-precision format. Thus, as illustrated in <FIG>, the output of the exemplary matrix multiplication <NUM> can be errors of a prior layer, such as the exemplary errors <NUM>, and output of the exemplary matrix multiplication <NUM> can be the gradients of a current layer, such as the exemplary gradients <NUM>, both of which can be expressed in the full-precision format. The backpropagation, comprising the provision of the activations <NUM> and the errors <NUM> to the matrix multiplication <NUM> to generate the gradients <NUM>, and the provision of the errors <NUM> and the weights <NUM> to the matrix multiplication <NUM> to generate the errors <NUM>, is meant as a more general illustration of the backpropagation detailed above, such as with reference to <FIG>.

As part of the neural network training, the weights can be updated utilizing the gradients derived by the backpropagation, such as the exemplary backpropagation <NUM>. According to one aspect, such a weight update can include gradient descent operations, such as the exemplary gradient descent <NUM>, as well as other relevant mathematical operations. As illustrated in <FIG>, the exemplary gradient descent <NUM> can receive, as input, the exemplary weights <NUM>, and the exemplary gradients <NUM>, in order to generate a new iteration of the weights. Both the exemplary weights <NUM> and the exemplary gradients <NUM> can be provided to the exemplary gradient descent <NUM> in the full-precision format, as illustrated, and the output of the exemplary gradient descent <NUM> can also be in the full-precision format.

However, as detailed above, neural network training can utilize large quantities of numerical values, which, when expressed in the full-precision format, can consume large quantities of memory and processing capabilities. For example, the exemplary gradient descent <NUM>, illustrated in <FIG>, may not be performed with the same computing devices that perform the exemplary matrix multiplication <NUM>, or the exemplary matrix multiplication <NUM> or <NUM>. Consequently, the values representing the exemplary weights <NUM> or <NUM>, for example, or the exemplary activations <NUM> or <NUM>, or the exemplary errors <NUM>, can need to be transmitted among computing devices. When expressed utilizing full-precision format, the exchange of such values can comprise the exchange of hundreds of megabytes, or even gigabytes of data. Reducing the quantity of bits utilized to represent the values, such as gradient values, activation values, weight values or error values, can reduce the quantity of data exchanged among computing devices by <NUM>% or more, thereby enabling such data exchanges to be performed twice as quickly. Similarly, the storage of such values on computer readable storage media, from which they can be read to be provided as input to the various processes illustrated, can consume large quantities of storage, such as hundreds of megabytes or even gigabytes of storage capacity. Reducing the quantity of bits utilized to represent the values can also reduce the quantity of storage capacity consumed, including reducing such consumption by <NUM>% or more.

In addition, matrix multiplication can consume substantial processing resources. Performing such matrix multiplication utilizing values expressed in full-precision form can require the processing of large quantities of data, which can take a long time, generate large quantities of waste heat, and prevent, or delay, the processing of other functions or instructions. Reducing the quantity of bits utilized to represent values can reduce the time taken to perform the matrix multiplication by <NUM>% or more, enable the performance of more multiplications in parallel utilizing the same processing hardware, which can result in the multiplication being performed in less than half the time, or otherwise reduce the consumption of processing resources and increase the speed and efficiency with which matrix multiplication is performed.

According to one aspect, therefore, the matrix multiplication performed by the forward propagation and backpropagation can be performed utilizing numerical values expressed in reduced-precision formats. Turning to <FIG>, the exemplary system <NUM> shown therein illustrates an exemplary performance of forward propagation and backpropagation utilizing numerical values expressed in reduced-precision formats. More specifically, the exemplary forward propagation <NUM> illustrates the exemplary matrix multiplication <NUM> of the exemplary forward propagation <NUM> shown in <FIG>, except with such matrix multiplication <NUM> now being performed utilizing values expressed in bounding box floating-point format, and expressed utilizing a fewer quantity of bits, than the full-precision format utilized in the exemplary forward propagation <NUM> shown in <FIG>. In particular, activations from a prior layer, represented as the exemplary activations <NUM>, can still be expressed in a full-precision format and can still be provided as input into the exemplary matrix multiplication <NUM>. Similarly, weights from a current layer, represented as the exemplary weights <NUM>, can also still be expressed in the full-precision format and can still be provided as the other input into the exemplary matrix multiplication <NUM>. However, prior to performing the exemplary matrix multiplication <NUM>, a bounding box floating-point quantization, such as the exemplary bounding box floating-point quantization <NUM> can quantize, into a bounding box floating-point reduced-precision format, the values of the exemplary activations <NUM>, and the exemplary weights <NUM>, previously expressed in the full-precision format. The exemplary matrix multiplication <NUM> can then be performed on the values of the exemplary activations and the exemplary weights as expressed utilizing the bounding box reduced-precision format.

Turning to <FIG>, the exemplary system <NUM> shown therein illustrates an exemplary conversion from a higher-precision numerical format to a lower-precision numerical format in accordance with bounding box techniques. For example, an exemplary set of six numerical values <NUM> is illustrated in <FIG>. As illustrated, each of the numerical values of the set <NUM> can be expressed utilizing one bit representing whether the value is positive or negative, three bits representing, in binary, the value of the exponent, and six bits representing, in binary, the magnitude of the value, which is sometimes referred to as the "fraction", the "significand" and/or the "mantissa". An initial step <NUM>, in converting the higher-precision format, in which the values of the set <NUM> are represented, into a lower-precision format in accordance with bounding box techniques, can comprise determining a shared exponent. For example, a shared exponent can be the largest exponent value in the set of values. Thus, in the exemplary set of values <NUM>, a shared exponent can be the exponent <NUM><NUM>, as illustrated in <FIG>. Subsequently, at step <NUM>, the values in the exemplary set of values <NUM> can all be shifted to be represented in the shared exponent. As can be seen, the values of the significand can decrease as the corresponding exponent is increased to be the same as the shared exponent. At step <NUM>, because all of the values now share the same exponent, only a single instance of the exponent can be represented, thereby eliminating redundancy in having each value consume a certain quantity of bits to represent the same, shared exponent. A further reduction can then reduce the quantity of bits utilized to express the significand. Thus, for example, at step <NUM>, a quantity of bits utilized to represent the significand can be reduced from six bits in the set <NUM> to four bits in the set being generated by step <NUM>. Such a reduction can be performed by rounding to the nearest value directly representable by four bits, such as in the manner shown. Other rounding mechanisms can likewise be utilized, such as simply discarding the least significant bits, stochastic rounding, which will be described in further detail below, or other rounding mechanisms.

As can be seen, fewer quantities of bits can be utilized to represent values, at least in part because all of the values in a set of values share a common exponent. Such a common exponent can reduce the quantity of bits utilized to represent the value, and can also decrease the quantity of bits processed by the circuitry of a processing unit in performing mathematical operations on such values. According to one aspect, a common exponent can be inferred, such that each representation of a value utilizes zero bits as part of an explicit reference to the common exponent. According to other aspects, one or more bits, within the representation of the value, can be utilized to reference a common exponent, with savings still being realized to the extent that the quantity of bits utilized to reference the common exponent is less than the quantity of bits that would have been utilized to represent the common exponent itself.

Turning back to <FIG>, as detailed above, the bounding box floating-point quantization <NUM> can convert the activation values <NUM>, in the full-precision format, to activation values <NUM> in a reduced-precision format, such as the bounding box format detailed above, which can utilize fewer quantities of bits to express the activation values. In a similar manner, the bounding box floating-point quantization <NUM> can convert the weight values <NUM>, in the full-precision format, to weight values <NUM> in a reduced-precision format. The matrix multiplication <NUM> can then be performed utilizing the activation values <NUM> and the weight values <NUM>, both in the reduced-precision format. As such, the matrix multiplication <NUM> can process meaningfully fewer bits of data and, consequently, can be completed more quickly, and with less processor utilization.

Output from such matrix multiplications can be activations of a current layer, represented as the exemplary activations <NUM>, which can be expressed in the full-precision format. More specifically, the multiplication of two values, represented with a given quantity of bits, can result in a third value that can have a greater quantity of bits. In such a manner, while the matrix multiplication <NUM> may receive, as input, values expressed utilizing reduced-precision formats, the resulting output of the matrix multiplication <NUM> can be values expressed in utilizing higher-precision formats. Such a higher-precision format can include a standard format, such as the exemplary full-precision format detailed above. In instances where output of the matrix multiplication <NUM> can comprise fewer bits then that called for by the full-precision format, bit padding or other like mechanisms can be utilized to increase a quantity of bits utilized to represent the values being output by the matrix multiplication <NUM>.

Bounding box floating-point quantization can also be utilized, in an equivalent manner to that detailed above, to increase a speed and efficiency with which the matrix multiplications <NUM> and <NUM>, which can be part of the exemplary backpropagation <NUM>, can be performed. More specifically, the exemplary activations <NUM>, and the exemplary errors <NUM>, can be provided as input to the exemplary matrix multiplication <NUM>, with their values expressed in the full-precision format. And the exemplary errors <NUM> can be provided, with the exemplary weights <NUM>, to the exemplary matrix multiplication <NUM> with their values expressed in the full-precision format. As above, the bounding box floating-point quantization <NUM> can convert the activation values <NUM> and the errors <NUM>, in the full-precision format, to activation values <NUM> and errors <NUM> in a reduced-precision format, such as the bounding box format detailed above. Similarly, the bounding box floating-point quantization <NUM> can convert the weight values <NUM> and errors <NUM>, in the full-precision format, to weight values <NUM> and errors <NUM> in a reduced-precision format. The matrix multiplication <NUM> can then be performed utilizing the activation values <NUM> and the error values <NUM>, both in the reduced-precision format and, analogously, the matrix multiplication <NUM> can then be performed utilizing the error values <NUM> and the weight values <NUM>, again, both in the reduced-precision format. As before, such a change can result in the matrix multiplications <NUM> and <NUM> processing meaningfully fewer bits of data and, consequently, enabling such matrix multiplications to be completed more quickly, and with less processor utilization. The output of the matrix multiplications <NUM> and <NUM>, namely the gradients <NUM> and the errors <NUM> can be expressed in a higher-precision format, such as the full-precision format, in accordance with the mechanisms and reasons detailed above.

The exemplary system <NUM>, shown in <FIG>, still entails the exchange of values expressed utilizing high-precision formats among computing devices as the inputs and outputs of the forward propagation, backpropagation and weight update portions of the neural network training process. As detailed above, such can require the transmission of hundreds of megabytes, or even gigabytes, of data, and, likewise, the storage of hundreds of megabytes, or even gigabytes, of data locally on computing devices performing the above detailed processing. According to one aspect, to reduce the transmission time, and memory utilization and storage requirements, while still providing sufficient precision for the training process to complete properly, reduced-precision formats can be utilized for the inputs and outputs of the forward propagation, backpropagation and weight update portions of the neural network training process.

Turning to <FIG>, the exemplary system <NUM> shown therein illustrates an exemplary forward propagation <NUM>, with the exemplary matrix multiplication <NUM> being performed utilizing values expressed in a reduced-precision format, such as the bounding box floating-point format detailed above. In particular, activations from a prior layer, represented as the exemplary activations <NUM> and weights from a current layer, represented as the exemplary weights <NUM>, can be provided as input into the bounding box floating-point quantization <NUM> which can convert such values into the activations <NUM>, in a reduced-precision format, and the weights <NUM>, in the reduced-precision format, which are then utilized by the exemplary matrix multiplication <NUM> to generate the exemplary activations <NUM> in a higher-precision format.

Within the exemplary system <NUM> shown in <FIG>, however, input and output values can be exchanged utilizing reduced-precision formats. Thus, for example, the exemplary activations <NUM> and the exemplary weights <NUM> can be received in a reduced-precision format. Similarly, the exemplary activations <NUM>, generated by the matrix multiplication <NUM>, can be converted to a lower-precision format. For example, the round to nearest process <NUM> can reduce the quantity of bits utilized to represent the activations <NUM>, outputting the activations in a reduced-precision format as the exemplary activations <NUM>. According to one aspect, the format utilized for the exemplary activations <NUM> can be a brain floating-point format, sometimes abbreviated as "bfloat". One advantage to utilizing a brain floating-point format can be that a quantity of bits utilized to represent the exponent value can be the same between, for example, a <NUM>-bit brain floating-point format and the ubiquitous <NUM>-bit single precision floating-point format. Consequently, the round to nearest <NUM> can focus only on the bits representing the significand, rendering such a conversion operation faster and more efficient. Other mechanisms for rounding, such as stochastic rounding, or simply discarding the least significant bits, can be used in place of the round to nearest <NUM>. Although illustrated as converting from <NUM>-bit single precision floating-point format to <NUM>-bit brain floating-point format, the output of the matrix multiplication <NUM> may not comprise thirty-two bits of precision. To the extent that the output of the matrix multiplication <NUM> comprises greater precision than that of the reduced-precision format in which inputs and outputs are to be exchanged, the round to nearest <NUM> can still perform some form of rounding to reduce a quantity of bits and, thereby, reduce the precision of the format in which the values output by the matrix multiplication <NUM>, namely the exemplary activations <NUM>, are provided as output to the exemplary forward propagation <NUM>.

In an analogous manner, the exemplary backpropagation <NUM> can receive values in reduced-precision formats, and, similarly, output values in reduced-precision formats. More specifically, and as illustrated in <FIG>, the exemplary activations <NUM>, errors <NUM> and weights <NUM> that act as input to the exemplary backpropagation <NUM> can be in reduced-precision formats, such as those detailed above. A bounding box floating-point quantization, such as the exemplary bounding box floating-point quantizations <NUM> and <NUM>, can convert the values received in one reduced-precision format, such as the aforementioned brain floating-point format, to a different reduced-precision format, such as the aforementioned bounding box floating-point format. Thus, for example, the bounding box floating-point quantization <NUM> can convert the activation values <NUM> and the errors <NUM>, in one reduced-precision format, such as the brain floating-point format, to activation values <NUM> and errors <NUM> in a different reduced-precision format, such as the bounding box format detailed above. Similarly, the bounding box floating-point quantization <NUM> can convert the weight values <NUM> and errors <NUM> to weight values <NUM> and errors <NUM>. The matrix multiplication <NUM> can then be performed utilizing the activation values <NUM> and the error values <NUM> and the matrix multiplication <NUM> can then be performed utilizing the error values <NUM> and the weight values <NUM>. As with the exemplary forward propagation <NUM>, the output of the matrix multiplication <NUM>, namely the gradients <NUM> and errors <NUM>, respectively, both in a higher-precision format, can be downconverted to a reduced-precision format, such as by the round to nearest <NUM> and <NUM>, respectively. The round to nearest <NUM> can downconvert the gradient values <NUM> to be represented by the same bit quantity brain floating-point format utilized for the input into the exemplary backpropagation <NUM>, thereby outputting the exemplary gradient values <NUM>, and the round to nearest <NUM> can, similarly, downconvert the error values <NUM> to be represented by the same bit quantity brain floating-point format, outputting the exemplary error values <NUM>.

According to one aspect, weight updates, such as the weight updates performed by the exemplary gradient descent <NUM>, can be performed utilizing higher-precision formats, such as the exemplary <NUM>-bit single precision floating-point format. However, the weight update can receive, as input, gradients in a lower-precision format, such as the exemplary gradients <NUM>. Accordingly, an upconversion, such as the exemplary upconversion <NUM>, can increase the quantity of bits utilized to represent the gradient values received as input. Such upconverted gradient values <NUM> can then be input into the gradient descent <NUM>. Similarly, the weight update can receive, as input, prior weights in a lower-precision format.

Depending on the mechanism utilized to generate the lower-precision format versions of the prior weights, the lower-precision format versions of the prior weights may not have sufficient precision to allow repeated iterations of the system <NUM> to converge onto a narrow range of weight values, or singular weight values, and, consequently, fail to properly train the neural network. In such an instance, prior weight values in a higher-precision format can be locally retained and utilized in a subsequent iteration. Thus, for example, the exemplary system <NUM> illustrates that weight values <NUM>, generated by the gradient descent <NUM>, in a higher-precision format, such as a full-precision format, can be stored on the storage media <NUM> for utilization during a subsequent iteration of the gradient descent <NUM>. The weight values <NUM>, in the higher-precision format, can also be converted to a reduced-precision format, such as the reduced-precision format utilized for the input and output values of the forward propagation, backpropagation and weight update in the system <NUM>. Accordingly, the weight values <NUM> can be converted to such a reduced-precision format by the round to the nearest <NUM>, or other like rounding mechanism. Such an output can then be provided as input to a subsequent iteration of the exemplary forward propagation <NUM>.

According to another aspect, rather than retaining a copy of the weight values <NUM> in a higher-precision format, stochastic rounding can be utilized in place of the round to the nearest <NUM>. In such an instance, the added precision provided by stochastic rounding can be sufficient to enable weight values to converge and the training of the neural network to be successful. More specifically, and as illustrated by the exemplary system <NUM> of <FIG>, the weight values <NUM>, generated by the gradient descent <NUM>, in a higher-precision format, such as a full-precision format, can be stochastically rounded, such as by the exemplary stochastic rounding <NUM>, to a reduced-precision format, such as the brain floating-point format detailed above. In such an instance, a subsequent iteration of the gradient descent <NUM> can utilize the previously generated weight values in the reduced-precision format, except upconverted, such as by the exemplary upconversion <NUM>, to a higher-precision format, such as the full-precision format. In such a manner, a separate copy of the weight values in a higher-precision format need not be retained.

Turning to <FIG>, an exemplary stochastic rounding mechanism is illustrated with reference to the system <NUM> shown therein. More specifically, a set of values <NUM> can, prior to a rounding operation, such as the exemplary round to nearest step <NUM>, have a randomness added to the values, as illustrated by step <NUM>. The resulting values <NUM> can then be rounded at step <NUM> to achieve reduced-bit and reduced-precision values <NUM>. As a simple example, a value of ¾ would be rounded to a value of "one" utilizing a round to nearest approach. By contrast, utilizing stochastic rounding, the randomness added to the value of ¾ can include negative values such that the value of ¾ is reduced and, indeed <NUM>% of the time, the value of ¾ would be reduced below ½ such that it would be rounded down to "zero" in those <NUM>% of cases. When considered in aggregate, stochastic rounding can be more accurate. For example, if ten values of ¾ were added together, the total would be "seven and one-half". If the ten values of ¾ were rounded first, utilizing round to nearest, and then added, the total value would be "ten" because each of those ten values would be rounded up to "one". By contrast, utilizing stochastic rounding, the total value would be closer to "eight", since, in approximately <NUM>% of the instances, ¾ would be reduced below ½ by the randomness that is added prior to rounding, and would, thus, be rounded to "zero". As can be seen, when considered in aggregate, stochastic rounding can be more accurate.

Accordingly, in addition to being utilized in the manner detailed above to reduce the precision of the output of the gradient descent <NUM>, stochastic rounding, such as in the manner shown by the exemplary system <NUM> of <FIG>, can be utilized in other precision-reduction operations, including, the bounding box floating point quantizations <NUM>, <NUM> and <NUM> and the round to nearest <NUM>, <NUM> and <NUM> and corresponding round to nearest <NUM>, <NUM> and <NUM>. Tuming first to the bounding box floating point quantizations <NUM>, <NUM> and <NUM>, and with reference to the exemplary system <NUM> of <FIG>, the step <NUM> could reduce the quantity of bits utilized to express the significand by utilizing stochastic rounding, such as in the manner illustrated by the exemplary system <NUM> of <FIG>. Any one or more of the bounding box floating point quantizations <NUM>, <NUM> and <NUM>, including as part of the exemplary system <NUM> of <FIG>, the exemplary system <NUM> of <FIG> and/or the exemplary system <NUM> of <FIG>, can utilize stochastic rounding, such as in the manner detailed herein. Similarly, turning to the round to nearest <NUM>, <NUM> and <NUM> and corresponding round to nearest <NUM>, <NUM> and <NUM>, any one or more of them, including as part of the exemplary system <NUM> of <FIG> and/or the exemplary system <NUM> of <FIG>, can also utilize stochastic rounding, such as in the manner detailed herein.

Turning back to <FIG>, the remainder of the processing illustrated in the exemplary system <NUM> can be analogous to that described in detail above with reference to exemplary system <NUM>, with the exception of the exemplary stochastic rounding <NUM> and upconversion <NUM>. Since the utilization of stochastic rounding can result in different values, different identifying numerals are utilized within the exemplary system <NUM> to identify the values of the weights, activations, errors and gradients. However, numbers identifying values in the exemplary system <NUM> having a form of 8xx correspond to the same values in the exemplary system <NUM> having a form 7xx, where "xx" are the same digits, the values in the exemplary system <NUM> having been described in detail above.

Turning to <FIG>, the exemplary flow diagram <NUM> shown therein illustrates an exemplary series of steps that could be performed by the exemplary system <NUM>, shown in <FIG>, to train a neural network. An exemplary backpropagation is represented by the steps <NUM> through <NUM>, an exemplary weight update is represented by the steps <NUM> through <NUM>, and an exemplary forward propagation is represented by the steps <NUM> through <NUM>. More specifically, at step <NUM>, activation, weight and error values, expressed in a reduced-precision format, can be received as input. Prior to performing matrix multiplication, such as at step <NUM>, bounding box quantization can be performed at step <NUM>. The resulting values, expressed utilizing a reduced quantity of bits, can be utilized to perform the matrix multiplication of step <NUM>, and the resulting output can be converted to a reduced-precision format at step <NUM> for purposes of being output at step <NUM>. The output, at step <NUM>, can include gradient values and error values, represented in reduced-precision format.

The gradient values in reduced-precision format can be part of the input received at step <NUM>, and such gradient values can be upconverted at step <NUM> prior to being utilized to perform gradient descent at step <NUM>. At step <NUM>, the weight values generated by a prior iteration of step <NUM>, and stored by a prior iteration of step <NUM>, can be obtained. Those high-precision weight values can be utilized to perform the gradient descent of step <NUM>, and the resulting output can be stored in a high-precision format at step <NUM>. The output weight values can also be converted to a reduced-precision format at step <NUM> and output in that format at step <NUM>.

The weight values in the reduced-precision format can be part of the input received at step <NUM>, together with activations, also in a reduced-precision format. Prior to performing matrix multiplication, such as at step <NUM>, a bounding box quantization can be performed at step <NUM>. The resulting values, expressed utilizing a reduced quantity of bits, can be utilized to perform the matrix multiplication of step <NUM>, and the resulting output can be converted to a reduced-precision format at step <NUM> for purposes of being output at step <NUM>. The output, at step <NUM>, can include activation values represented in reduced-precision format.

Turning to <FIG>, the exemplary flow diagram <NUM> shown therein illustrates an exemplary series of steps that could be performed by the exemplary system <NUM>, shown in <FIG>, to train a neural network. To the extent that the steps of the exemplary flow diagram <NUM> are the same as those of the exemplary flow diagram <NUM> shown in <FIG> and described in detail above, those steps are nominated with the same identifiers in <FIG> as they are in <FIG>. As can be seen, <FIG> can differ from <FIG> in the weight update represented by steps <NUM> through <NUM>. More specifically, at step <NUM> the input received by the weight update can comprise both gradients and weights in a reduced-precision format, both of which can then be upconverted at step <NUM> to a higher-precision format. The gradient descent performed by step <NUM> can be the same as that of the exemplary flow diagram <NUM> described above. Subsequently, the updated weight values generated by the performance of step <NUM> can be converted to a reduced-precision format utilizing stochastic rounding at step <NUM>. The weight values, expressed utilizing such a reduced-precision format, can then be output at step <NUM>. In such a manner, the exchange of such weight values can occur more quickly than if such a weight values were expressed in higher-precision formats, and the weight values can be stored in less memory. Moreover, a separate copy of higher-precision weight values can be unnecessary, thereby preventing additional storage utilization and saving storage capacity.

Turning to <FIG>, an exemplary computing device <NUM> is illustrated which can perform some or all of the mechanisms and actions described above. The exemplary computing device <NUM> can include, but is not limited to, one or more central processing units (CPUs) <NUM>, one or more hardware accelerator processing units <NUM>, a system memory <NUM>, and a system bus <NUM> that couples various system components including the system memory to the processing units <NUM> and <NUM>. The central processing unit <NUM> can comprise circuitry that provides for the performance of general processing operations. The hardware accelerator processing unit <NUM> can comprise circuitry that provides for the performance of specific processing operations for which such circuitry is designed, and often comprises multiple such circuitry in parallel. The system bus <NUM> may be any of several types of bus structures including a memory bus or memory controller, a peripheral bus, and a local bus using any of a variety of bus architectures. Depending on the specific physical implementation, one or more of the CPUs <NUM>, the system memory <NUM>, the hardware accelerator processing unit <NUM> and other components of the computing device <NUM> can be physically co-located, such as on a single chip. In such a case, some or all of the system bus <NUM> can be nothing more than silicon pathways within a single chip structure and its illustration in <FIG> can be nothing more than notational convenience for the purpose of illustration.

The computing device <NUM> also typically includes computer readable media, which can include any available media that can be accessed by computing device <NUM> and includes both volatile and nonvolatile media and removable and non-removable media. Computer storage media includes media implemented in any method or technology for storage of content such as computer readable instructions, data structures, program modules or other data. Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium which can be used to store the desired content and which can be accessed by the computing device <NUM>. Computer storage media, however, does not include communication media. Communication media typically embodies computer readable instructions, data structures, program modules or other data in a modulated data signal such as a carrier wave or other transport mechanism and includes any content delivery media. By way of example, and not limitation, communication media includes wired media such as a wired network or direct-wired connection, and wireless media such as acoustic, RF, infrared and other wireless media. Combinations of the any of the above should also be included within the scope of computer readable media.

A basic input/output system <NUM> (BIOS), containing the basic routines that help to transfer content between elements within computing device <NUM>, such as during start-up, is typically stored in ROM <NUM>. By way of example, and not limitation, <FIG> illustrates operating system <NUM>, other program modules <NUM>, and program data <NUM>.

The computing device <NUM> may also include other removable/non-removable, volatile/nonvolatile computer storage media. By way of example only, <FIG> illustrates a hard disk drive <NUM> that reads from or writes to non-removable, nonvolatile magnetic media. Other removable/non-removable, volatile/nonvolatile computer storage media that can be used with the exemplary computing device include, but are not limited to, magnetic tape cassettes, flash memory cards, digital versatile disks, digital video tape, solid state RAM, solid state ROM, and other computer storage media as defined and delineated above. The hard disk drive <NUM> is typically connected to the system bus <NUM> through a non-volatile memory interface such as interface <NUM>.

The drives and their associated computer storage media discussed above and illustrated in <FIG>, provide storage of computer readable instructions, data structures, program modules and other data for the computing device <NUM>. In <FIG>, for example, hard disk drive <NUM> is illustrated as storing operating system <NUM>, other program modules <NUM>, and program data <NUM>. Note that these components can either be the same as or different from operating system <NUM>, other program modules <NUM> and program data <NUM>. Operating system <NUM>, other program modules <NUM> and program data <NUM> are given different numbers hereto illustrate that, at a minimum, they are different copies.

The computing device <NUM> may operate in a networked environment using logical connections to one or more remote computers. The computing device <NUM> is illustrated as being connected to the general network connection <NUM> (to the network <NUM>) through a network interface or adapter <NUM>, which is, in turn, connected to the system bus <NUM>. In a networked environment, program modules depicted relative to the computing device <NUM>, or portions or peripherals thereof, may be stored in the memory of one or more other computing devices that are communicatively coupled to the computing device <NUM> through the general network connection <NUM>. It will be appreciated that the network connections shown are exemplary and other means of establishing a communications link between computing devices may be used.

Although described as a single physical device, the exemplary computing device <NUM> can be a virtual computing device, in which case the functionality of the above-described physical components, such as the CPU <NUM>, the system memory <NUM>, the network interface <NUM>, and other like components can be provided by computer-executable instructions. Such computer-executable instructions can execute on a single physical computing device, or can be distributed across multiple physical computing devices, including being distributed across multiple physical computing devices in a dynamic manner such that the specific, physical computing devices hosting such computer-executable instructions can dynamically change over time depending upon need and availability. In the situation where the exemplary computing device <NUM> is a virtualized device, the underlying physical computing devices hosting such a virtualized computing device can, themselves, comprise physical components analogous to those described above, and operating in a like manner. Furthermore, virtual computing devices can be utilized in multiple layers with one virtual computing device executing within the construct of another virtual computing device. The term "computing device", therefore, as utilized herein, means either a physical computing device or a virtualized computing environment, including a virtual computing device, within which computer-executable instructions can be executed in a manner consistent with their execution by a physical computing device. Similarly, terms referring to physical components of the computing device, as utilized herein, mean either those physical components or virtualizations thereof performing the same or equivalent functions.

Claim 1:
A method of increasing processing speed and decreasing digital storage consumption in performing neural network training utilizing one or more computing devices, the method comprising:
performing, with the one or more computing devices, a forward propagation of the neural network training that receives (<NUM>), as input, a first set of weight values in a first reduced-precision format and a first set of activation values in the first reduced-precision format, and that outputs (<NUM>), into storage, a second set of activation values in the first reduced-precision format;
performing, with the one or more computing devices, a backpropagation of the neural network training that receives (<NUM>), as input, a first set of weight values in the first reduced-precision format, a first set of error values in the first reduced-precision format and the first set of activation values in the first reduced-precision format, and that outputs (<NUM>), into storage, a first set of gradient values in the first reduced-precision format and a second set of error values in the first reduced-precision format; and
performing, with the one or more computing devices, a weight update of the neural network training utilizing (<NUM>, <NUM>) a first set of weight values in a full-precision format and a first set of gradient values in the full-precision format, the performing the weight update resulting in a storage (<NUM>, <NUM>) of a second set of weight values in the reduced-precision format;
wherein the forward propagation and the backpropagation comprise performing bounding box quantization on input values (<NUM>, <NUM>), the bounding box quantization converting values expressed utilizing different exponents to values expressed utilizing a common exponent, the values expressed utilizing the common exponent comprising only an identification of the common exponent, instead of the common exponent itself, the identification of the common exponent consuming fewer bits than the common exponent itself.