Activation function computation for neural networks

A computer-implemented method for improving the efficiency of computing an activation function in a neural network system includes initializing, by a controller, weights in a weight vector associated with the neural network system. Further, the method includes receiving, by the controller, an input vector of input values for computing a dot product with the weight vector for the activation function, which determines an output value of a node in the neural network system. The method further includes predicting, by a rectifier linear unit (ReLU), which computes the activation function, that the output value of the node will be negative based on computing an intermediate value for computing the dot product, and based on a magnitude of the intermediate value exceeding a precomputed threshold value. Further, the method includes, in response to the prediction, terminating, by the ReLU, the computation of the dot product, and outputting a 0 as the output value.

BACKGROUND

The present invention relates generally to computing technology, and particularly to improved efficiency of neural network computations by facilitating an efficient dot-product computation using predictive zero-skipping during activation function computations.

Deep learning has led to state-of-the-art improvements in the accuracy of many artificial intelligence tasks such as large-category image classification and recognition; speech recognition, and nature language processing. Neural networks have demonstrated an ability to learn such skills as face recognition, reading, and the detection of simple grammatical structure. More particularly, neural networks can be considered to be models defining a multivariate function or a distribution over a set of discrete classes. In some instances, neural network models can be associated with a particular learning method or learning rule. The deep learning architecture can involve complex and many-layered neural networks (e.g., deep neural networks (DNN)) that can require intense computation for training and/or evaluation.

The ability to train increasingly deep networks has been due, in part, to the development of pre-training algorithms and forms of random initialization, as well as the availability of faster computers.

SUMMARY

According to one or more embodiments of the present invention, a computer-implemented method for improving the efficiency of computing an activation function in a neural network system includes initializing, by a controller, weights in a weight vector associated with the neural network system. Further, the method includes receiving, by the controller, an input vector of input values for computing a dot product with the weight vector for the activation function, which determines an output value of a node in the neural network system. The method further includes predicting, by a rectifier linear unit (ReLU), which computes the activation function, that the output value of the node will be negative based on computing an intermediate value for computing the dot product, and based on a magnitude of the intermediate value exceeding a precomputed threshold value. Further, the method includes, in response to the prediction, terminating, by the ReLU, the computation of the dot product, and outputting a 0 as the output value.

According to one or more embodiments of the present invention, a system for implementing a machine learning function includes at least one rectifier linear unit (ReLU), and at least one controller coupled with the at least one ReLU to perform a method for computing a dot product. The method includes initializing weights in a weight vector. The method further includes receiving an input vector of input values for computing the dot product of the input vector with the weight vector. The method further includes predicting that the output value of the dot product will be negative by computing an intermediate value for computing the dot product, and based on a magnitude of the intermediate value exceeding a precomputed threshold value. The method further includes, in response to the prediction, terminating the computation of the dot product, and outputting a 0 as the result of the dot product.

According to one or more embodiments of the present invention, a rectifier linear unit (ReLU) includes a storage medium, a comparator, several multipliers, and an adder tree. The ReLU performs a method for computing a dot product. The method includes initializing weights in a weight vector. The method further includes receiving an input vector of input values for computing the dot product of the input vector with the weight vector. The method further includes predicting that the output value of the dot product will be negative by computing an intermediate value for computing the dot product, and based on a magnitude of the intermediate value exceeding a precomputed threshold value. The method further includes, in response to the prediction, terminating the computation of the dot product, and outputting a 0 as the result of the dot product.

In one or more embodiments of the present invention, the intermediate value is computed at each computation cycle b as Sb=2B−b−1sB−1+2B−b−2sB−2+ . . . sb, wherein B is number of bits used to represent each x, and sb is the sum of the dot products of the b-th bits of each input value. in one or more embodiments of the present invention, the dot product is computed as part of computing an activation function.

DETAILED DESCRIPTION

The subject disclosure is directed to computer processing systems, computer-implemented methods, apparatus and/or computer program products that facilitate an efficiency within a neural network. A neural network (sometimes referred to as an artificial neural network, or a deep neural network) generally is a computer system that seeks to mimic a brain. A neural network can be utilized in a variety of ways, such as by being trained to identify the presence of human faces in images, or translate spoken speech from a first language to a second language.

A neural network generally contains multiple neurons, and connections between those neurons. A neuron generally is a part of a neural network computer system that determines an output based on one or more inputs (that can be weighted), and the neuron can determine this output based on determining the output of an activation function with the possibly-weighted inputs. Examples of activation functions include a rectifier/rectified linear unit (ReLU) activation function, which produces an output that ranges between 0 and infinity, inclusive; tan h, which produces an output that ranges between −1 and 1, inclusive; and sigmoid, which produces an output that ranges between 0 and 1, inclusive. While several of the non-limiting examples described herein concern a ReLU activation function, it can be appreciated that these techniques can be applied to other activation functions.

FIG.1illustrates an example, non-limiting neural network system for which an efficiency can be facilitated in accordance with one or more embodiments described herein. The neurons of a neural network can be connected, so that the output of one neuron can serve as an input to another neuron. Neurons within a neural network can be organized into layers, as shown inFIG.1. The first layer of a neural network can be called the input layer (124), the last layer of a neural network can be called the output layer (128), and any intervening layers of a neural network can be called a hidden layer (126). Aspects of systems (e.g., system100and the like), apparatuses or processes explained herein can constitute machine-executable component(s) embodied within machine(s), e.g., embodied in one or more computer readable mediums (or media) associated with one or more machines. Such component(s), when executed by the one or more machines, e.g., computer(s), computing device(s), virtual machine(s), etc. can cause the machine(s) to perform the operations described. Repetitive description of like elements employed in respective embodiments is omitted for sake of brevity.

The system100and/or the components of the system100can be employed to use hardware and/or software to solve problems that are highly technical in nature, that are not abstract and that cannot be performed as a set of mental acts by a human. For example, system100and/or the components of the system100can be employed to use hardware and/or software to perform operations including facilitating an efficiency within a neural network. Further, some of the processes performed can be performed by specialized computers for carrying out defined tasks related to facilitating an efficiency within a neural network. System100and/or components of the system100can be employed to solve new problems that arise through advancements in technology, computer networks, the Internet and the like. System100can further provide technical improvements to live and Internet based learning systems by improving processing efficiency among processing components associated with facilitating an efficiency within a neural network.

System100, as depicted inFIG.1, is a neural network that includes five neurons—neuron102, neuron104, neuron106, neuron108, and neuron110. The input layer124of this neural network is comprised of neuron102and neuron104. The hidden layer126of this neural network is comprised of neuron106and neuron108. The output layer128of this neural network is comprised of neuron110. Each of the neurons of input layer124is connected to each of the neurons of hidden layer126. That is, a possibly-weighted output of each neuron of input layer124is used as an input to each neuron of hidden layer126. Then, each of the neurons of hidden layer126is connected to each of the neurons (here, one neuron) of output layer128.

The neural network of system100presents a simplified example so that certain features can be emphasized for clarity. It can be appreciated that the present techniques can be applied to other neural networks, including ones that are significantly more complex than the neural network of system100.

In the context of artificial neural networks, an ReLU provides an activation function that is generally referred to as “rectifier”, which is defined as the positive part of its argument: f(x)=x+=max(0,x), where x is the input to a neuron (102-110).FIG.2depicts a ReLU210according to one or more embodiments of the present invention. Here, consider that the ReLU210receives a vector {right arrow over (x)} as an input and {right arrow over (w)} is a vector of weight assigned to a neuron associated with the ReLU210. The ReLU210computes an output y as a scalar dot product of the input {right arrow over (x)} and the weights {right arrow over (w)}. However, the ReLU210only outputs a positive y; if the product of {right arrow over (x)} and {right arrow over (w)} results in a negative value, the output y is 0 (zero).

A neural network, such as system100, can include large number of such ReLUs210(e.g. thousands) that compute the scalar dot products that are passed from one layer to another until a final result of the neural network100is obtained. The performance of the neural network100can be improved if the efficiency of the dot product computation can be improved. For example, it can be energy-efficient if a negative value result of the dot product can be predicted even before computing the entire {right arrow over (x)}·{right arrow over (w)} so that the ReLU210can provide output=0 without fully computing the dot product of the vectors. Embodiments of the present invention address such technical challenges and facilitate technical improvements to the dot product computations. One or more embodiments of the present invention facilitate predicting the negative inner product output at early stages. Further, one or more embodiments of the present invention facilitate hardware components to support such negative result detection and aborting the dot product computation dynamically and providing the zero output instead.

FIG.3depicts an ReLU for computing a bit-wise dot product. The depiction inFIG.3describes bit-serial computation for computing the dot product where most significant bit (MSB) is computed first and least significant bit (LSB) is computed last. If the number of bits in the input values (x) is B, the scalar dot product is computed over B cycles, each cycle computing a bitwise products (310) with the weights (w). The result of the bitwise products is accumulated using an adder tree320. A clock330causes each cycle to change. As depicted, in a first cycle the LSB, i.e., bit number B−1 from each x in the input vector x is input to the ReLU210. Sequentially, for each cycle until the Bth cycle, the next bit from each x value is input. The Bth cycle uses the MSB, i.e., bit number 0 to complete the computation.

The output (sb) from the adder tree320at any cycle b, provides b-th bit's partial sum. In other words, sbprovides output of adder tree320at given cycle b. The final dot product can be represented as=s0+2s1+ . . . +2B−1sB−1. Alternatively, or in addition, a total accumulated value at the adder tree320at any given cycle b can be represented as Sb=2B−b−1sB−1+2B−b−2sB−2+ . . . sb. The output from the adder tree320is stored in a register340in one or more examples.

FIG.4depicts an ReLU for computing a dot product in a bit-wise manner using a prediction according to one or more embodiments of the present invention. As can be seen an ReLU410according to one or more embodiments of the present invention includes a comparator420and a threshold table in a memory430that facilitate to detect a negative value at as early as possible stage during the dot product computation. After each cycle which computes each bit position of x, as described with respect toFIG.3, the accumulated result Sbis compared with a value from the threshold table. If it is determined thatis going to be negative regardless of the remaining computation, based on the comparison, controller440terminates the computation and sets the output to 0 using a multiplexer450. The ReLU410can be included in a computing device that implements a neural network system100.

FIG.5depicts a flowchart of a method for computing a dot product in a bit-wise manner using the ReLU410according to one or more embodiments of the present invention. The method500includes receiving the input vector {right arrow over (x)} that includes N x values, each x being represented using B-bits, at block502. The method500also includes receiving the initial weight vector {right arrow over (w)} that includes N w values, each w being represented as B-bit number, at block504. The method500further includes inputting the b-th bits of each x value during the b-th cycle of computation, at block506. The ReLU410uses MSB-first computing, which provides an opportunity to filter-out potential negative values at an early stage as described herein.

The method500further includes computing threshold values for each computation cycle of the bit-wise dot product computation, at block508. It should be noted that some of the operations of the method500can be performed in a sequence that is different from what is described herein. For example, the threshold values can be computed earlier, as soon as the weight values are initiated, in one or more examples. Such change in sequence of some of the operations would be obvious to a person skilled in this art, for example, to optimize the operation such as to parallelize some operations. The threshold values are stored in the threshold table in the memory430. In one or more examples, the memory includes B memory registers or other types of storage locations to store the B threshold values, one for each of the B computation cycles. In one or more embodiments of the present invention, the threshold values are computed using following formula: THb=(2−b+2−b+1+ . . . +2−1)Σi∈Gpwi, where Gp=group of positive wi, and b is the computation cycle for which the threshold is being computed.

The method500further includes computing the value Sbfor the b-th computation cycle, at block510.

The method500further includes, during each b-th computation cycle, comparing, using the comparator420, the accumulated Sbvalue with the Thbthreshold value, at block512. If the Sb, total accumulated result, is negative, and the absolute magnitude is too large, further computations cannot turn the result into a positive value. Accordingly, the comparator420checks if Sb<0, and if |Sb|>Thb. If both these conditions are met, the controller440predicts that the result of the dot product is negative, and accordingly, aborts the computation and sets the output to 0, at block514. Accordingly, the ReLU410can save further time and resources that might have been used for the dot product computation and instead, can start another (next) dot product computation. Alternatively, if the conditions are not met, the ReLU410continues the dot product computation for the next bits in the x values, by repeating the above operations. The operations in the method500are repeated until a negative value is predicted or until the dot product is computed (after B computation cycles). The result of the dot product is output as the result of the activation function in this case, at block516.

Consider the example scenario shown inFIG.6. The input vector and the weight vector are shown with example values. Here N=4, B=4. It is understood that the values used in this example are for explanation, and that in one or more embodiments of the present invention, the values can be different, and in most cases much larger. In this example, the Gpset only includes two values {1, 4} from the weight vector as the other weight values are negative. The threshold values for the cycles are shown in the table610. The Sbvalues for first three computation cycles (b=1, b=2, and b=3) are also shown in the table610.

As can be seen, in this case, the ReLU410can predict that the output will be negative value by only calculating one cycle (because |Sb|>Thb). Accordingly, the ReLU410can abort the computation after the first cycle and start a different computation altogether.

By providing such predictions embodiments of the present invention provide power and speed benefits among other advantages. For instance, statistically, roughly 50% of {right arrow over (x)}·{right arrow over (w)} are negative in neural network algorithms according to empirical data. Further, out of B-bits, computation is terminated approximately after 40% of B-bits are computed by assuming uniform distributed values of wi. Therefore, 30% [=0.5*(1−0.4)] power and 30% speed benefits are expected by using embodiments of the present invention. Further yet, in terms of energy-delay-product, about 51% improvement is expected (0.7*0.7=0.49) because of embodiments of the present invention.

It should also be noted that the use of the table430and comparator420occur only once after N elements' addition in the adder tree320. The table430can be registers to store B words (thresholds) and the comparator420can be a subtractor. Considering that typical vector length N is quite large (e.g., >512) compared to B (e.g. 16), the area of the table430and the comparator420is amortized in the ReLU design, occupying negligible portion of entire hardware.

FIG.7depicts a flowchart of another method for computing a dot product in a bit-wise manner using the ReLU410according to one or more embodiments of the present invention. In this method700, compared to the method500described herein, the respective threshold values are computed at runtime (dynamically) during each computation cycle. Accordingly, the threshold values do not have to be precomputed and stored in a threshold table. This in turn facilitates having a single register as part of the memory430to store the threshold value for the ongoing (or about to be started) computation cycle. This further reduces the area requirement of the ReLU410because fewer memory locations are now required.

As shown inFIG.7, the method700includes receiving the input vector {right arrow over (x)}, receiving the initial weight vector, and inputting the b-th bits of each x value during the b-th cycle of computation, at blocks502,504, and506, as described herein. Further, during the b-th computation cycle, the ReLU410computes the threshold value Thbfor the ongoing b-th cycle itself, at block708. In this case, W(=Σi∈Gpwi) is precomputed and stored in the memory430. The threshold value is computed at real time using only shift (2−b) and one subtraction per step using the formula:

The method700proceeds similar to the method500after this, by computing the value Sbfor the b-th computation cycle, at block510. The method700further includes, comparing, using the comparator420, the accumulated Sbvalue with the Thbthreshold value, at block512. The comparator420checks if Sb<0, and if |Sb|>Thb, and if both these conditions are met, the controller440predicts that the result of the dot product is negative. Accordingly, the controller440aborts the computation and sets the output of the activation function to 0, at block514. The output can be set to zero using a multiplexer. Accordingly, the ReLU410can save further time and resources that might have been used for the dot product computation and instead, can start another (next) dot product computation. Alternatively, if the conditions are not met, the ReLU410continues the dot product computation for the next bits in the x values, by repeating the above operations. The operations in the method500are repeated until a negative value is predicted or until the dot product is computed (after B computation cycles). The result of the dot product is output as the result of the activation function in this case, at block516.

In yet other embodiments of the present invention, the method for computing a dot product in a bit-wise manner using the ReLU410can further improves the efficiency of the activation function computation by the ReLU410, particularly in the case where the input values x are being received from another ReLU layer in the neural network100. As described earlier, empirical data indicates that if the previous layer is also ReLU, approximately half of the xi's are zero in average number of cases. Accordingly, the efficiency of the ReLU410can be further improved by disabling the branches with xi=0. To this end, the threshold (Th) is dependent to xi, and is not precomputed. Computing the threshold is performed using only one N-input addition (to calculate Σi∈Gp∩Gnwi) at the beginning, and one subtraction and logical shift per computation cycle. Here, Gnis a set of non-negative xis.

Accordingly, in this case, the memory430is populated with W=Σi∈Gp∩Gnwi. Further, during each computation b-th cycle, the threshold for that particular computation cycle is computed as:

Turning now toFIG.8, a computer system800is generally shown in accordance with an embodiment. The computer system800can be an electronic, computer framework comprising and/or employing any number and combination of computing devices and networks utilizing various communication technologies, as described herein. For example, the computer system800can include one or more of the ReLUs as described herein to implement an artificial neural network system. Alternatively, or in addition, the computer system800controls an array of multiple instances of an ReLU that is described herein, wherein the array is used to implement an artificial neural network system. The computer system800, accordingly, acts as a controller that can input data to the ReLU, instruct the ReLU to perform certain operations (such as computing, aborting etc.), and receive output from the ReLU. Further, the computer system800can cause one ReLU, or a layer (or set) of ReLUs to output data to another ReLU, or another set of ReLUs.

The computer system800can be easily scalable, extensible, and modular, with the ability to change to different services or reconfigure some features independently of others. The computer system800may be, for example, a server, desktop computer, laptop computer, tablet computer, or smartphone. In some examples, computer system800may be a cloud computing node. Computer system800may be described in the general context of computer system executable instructions, such as program modules, being executed by a computer system. Generally, program modules may include routines, programs, objects, components, logic, data structures, and so on that perform particular tasks or implement particular abstract data types. Computer system800may be practiced in distributed cloud computing environments where tasks are performed by remote processing devices that are linked through a communications network. In a distributed cloud computing environment, program modules may be located in both local and remote computer system storage media including memory storage devices.

As shown inFIG.8, the computer system800has one or more central processing units (CPU(s))801a,801b,801c, etc. (collectively or generically referred to as processor(s)801). The processors801can be a single-core processor, multi-core processor, computing cluster, or any number of other configurations. The processors801, also referred to as processing circuits, are coupled via a system bus802to a system memory803and various other components. The system memory803can include a read only memory (ROM)804and a random access memory (RAM)805. The ROM804is coupled to the system bus802and may include a basic input/output system (BIOS), which controls certain basic functions of the computer system800. The RAM is read-write memory coupled to the system bus802for use by the processors801. The system memory803provides temporary memory space for operations of said instructions during operation. The system memory803can include random access memory (RAM), read only memory, flash memory, or any other suitable memory systems.

The computer system800comprises an input/output (I/O) adapter806and a communications adapter807coupled to the system bus802. The I/O adapter806may be a small computer system interface (SCSI) adapter that communicates with a hard disk808and/or any other similar component. The I/O adapter806and the hard disk808are collectively referred to herein as a mass storage810.

Software811for execution on the computer system800may be stored in the mass storage810. The mass storage810is an example of a tangible storage medium readable by the processors801, where the software811is stored as instructions for execution by the processors801to cause the computer system800to operate, such as is described herein below with respect to the various Figures. Examples of computer program product and the execution of such instruction is discussed herein in more detail. The communications adapter807interconnects the system bus802with a network812, which may be an outside network, enabling the computer system800to communicate with other such systems. In one embodiment, a portion of the system memory803and the mass storage810collectively store an operating system, which may be any appropriate operating system, such as the z/OS or AIX operating system from IBM Corporation, to coordinate the functions of the various components shown inFIG.8.

Additional input/output devices are shown as connected to the system bus802via a display adapter815and an interface adapter816and. In one embodiment, the adapters806,807,815, and816may be connected to one or more I/O buses that are connected to the system bus802via an intermediate bus bridge (not shown). A display819(e.g., a screen or a display monitor) is connected to the system bus802by a display adapter815, which may include a graphics controller to improve the performance of graphics intensive applications and a video controller. A keyboard821, a mouse822, a speaker823, etc. can be interconnected to the system bus802via the interface adapter816, which may include, for example, a Super I/O chip integrating multiple device adapters into a single integrated circuit. Suitable I/O buses for connecting peripheral devices such as hard disk controllers, network adapters, and graphics adapters typically include common protocols, such as the Peripheral Component Interconnect (PCI). Thus, as configured inFIG.8, the computer system800includes processing capability in the form of the processors801, and, storage capability including the system memory803and the mass storage810, input means such as the keyboard821and the mouse822, and output capability including the speaker823and the display819.

In some embodiments, the communications adapter807can transmit data using any suitable interface or protocol, such as the internet small computer system interface, among others. The network812may be a cellular network, a radio network, a wide area network (WAN), a local area network (LAN), or the Internet, among others. An external computing device may connect to the computer system800through the network812. In some examples, an external computing device may be an external webserver or a cloud computing node.

It is to be understood that the block diagram ofFIG.8is not intended to indicate that the computer system800is to include all of the components shown inFIG.8. Rather, the computer system800can include any appropriate fewer or additional components not illustrated inFIG.8(e.g., additional memory components, embedded controllers, modules, additional network interfaces, etc.). Further, the embodiments described herein with respect to computer system800may be implemented with any appropriate logic, wherein the logic, as referred to herein, can include any suitable hardware (e.g., a processor, an embedded controller, or an application specific integrated circuit, among others), software (e.g., an application, among others), firmware, or any suitable combination of hardware, software, and firmware, in various embodiments.

Although specific embodiments of the invention have been described, one of ordinary skill in the art will recognize that numerous other modifications and alternative embodiments are within the scope of the invention. For example, any of the functionality and/or processing capabilities described with respect to a particular system, system component, device, or device component may be performed by any other system, device, or component. Further, while various illustrative implementations and architectures have been described in accordance with embodiments of the invention, one of ordinary skill in the art will appreciate that numerous other modifications to the illustrative implementations and architectures described herein are also within the scope of this invention. In addition, it should be appreciated that any operation, element, component, data, or the like described herein as being based on another operation, element, component, data, or the like may be additionally based on one or more other operations, elements, components, data, or the like. Accordingly, the phrase “based on,” or variants thereof, should be interpreted as “based at least in part on.”