Low precision deep neural network enabled by compensation instructions

A compensated deep neural network (compensated-DNN) is provided. A first vector having a set of components and a second vector having a set of corresponding components are received. A component of the first vector includes a first quantized value and a first compensation instruction, and a corresponding component of the second vector includes a second quantized value and a second compensation instruction. The first quantized value is multiplied with the second quantized value to compute a raw product value. The raw product value is compensated for a quantization error according to the first and second compensation instructions to produce a compensated product value. The compensated product value is added into an accumulated value for the dot product. The accumulated value is converted into an output vector of the dot product. The output vector includes an output quantized value and an output compensation instruction.

BACKGROUND

Technical Field

The present disclosure generally relates to computation by neural networks.

Description of the Related Art

The field of Artificial Intelligence (AI) has witnessed a quintessential growth in recent years with the advent of Deep Neural Networks (DNNs) that have achieved state-of-the-art classification accuracies on many recognition tasks involving images, videos, text and natural language. DNNs are multi-layered networks of basic compute units called artificial neurons. Each layer in the network is associated with a set of weights. Each neuron in a layer evaluates a multi-input, single-output function that computes dot-product of its inputs and weights, followed by a non-linear activation function on the weighted sum. DNNs operate in two phases: (i) Training and (ii) Inference. Training is performed based on a labeled dataset, where the weights of the DNN are iteratively refined using the Stochastic Gradient Descent (SGD) algorithm. During inference, inputs hither to unseen are classified using the trained model.

SUMMARY

Some embodiments of the disclosure provide a processing element for an artificial neuron of a deep neural network (DNN). The processing element is configured to produce a dot product based on a first vector having a set of components and a second vector having a set of corresponding components. A component of the first vector includes a first quantized value and a first compensation instruction and a corresponding component of the second vector includes a second quantized value and a second compensation instruction. The processing element includes a computation module configured to multiply the first quantized value with the second quantized value to compute a raw product value. The processing element includes a compensation module configured to compensate the raw product value for a quantization error according to the first and second compensation instructions to produce a compensated product value. The processing element includes an accumulation module configured to add the compensated product value into an accumulated value for the dot product. The processing element includes a conversion module configured to convert the accumulated value into an output vector of the dot-product, the output vector including an output quantized value and an output compensation instruction.

A compensation instruction includes a direction bit and a magnitude bit for compensating a quantization error of the quantized value. In some embodiments, a compensation instruction includes no more than four bits that include a direction bit, a zero compensation bit, and two or less magnitude bits. The zero compensation bits indicates whether the quantization error is less than a threshold, such that the estimated quantization error is zero and the quantized value need not be compensated. In some embodiments, the compensation module is in a low power mode when the zero compensation bits of both the first quantization instruction and the second quantization instruction indicate that the estimated quantization errors for both the first and second quantized values are zero.

The preceding Summary is intended to serve as a brief introduction to some embodiments of the disclosure. It is not meant to be an introduction or overview of all inventive subject matter disclosed in this document. The Detailed Description that follows and the Drawings that are referred to in the Detailed Description will further describe the embodiments described in the Summary as well as other embodiments. Accordingly, to understand all the embodiments described by this document, a Summary, Detailed Description and the Drawings are provided. Moreover, the claimed subject matter is not to be limited by the illustrative details in the Summary, Detailed Description, and the Drawings, but rather is to be defined by the appended claims, because the claimed subject matter can be embodied in other specific forms without departing from the spirit of the subject matter.

DETAILED DESCRIPTION

Deep Neural Networks (DNN) are the state-of-the-art solutions in many recognition problems involving images, video, text, and natural language. However, the computational and storage demands imposed by these large-scale networks have been the primary bottleneck to their ubiquitous adoption, as the amount of data processed by DNNs impose significant computational challenges. A key scenario that exemplifies this extreme computational challenge is low-power inference, where DNN models are executed on deeply-embedded wearable and Internet of Things (IoT) devices that have stringent energy and area constraints. One approach to improve the efficiency of DNNs, specifically in the context of low-power platforms (such as mobile, wearables, and other IoT devices), is to explore low-precision implementations using low-precision fixed point (<16 bits) representation. However, low-precision implementations suffer from quantization errors that are inherent in any fixed-point implementation. The choice of bit-widths is therefore limited if the system is to maintain application-level accuracy. Increasing the network size and/or re-training the DNN has been proposed to minimize loss of accuracy due to quantization, albeit with limited success.

Some embodiments of the disclosure provide compensated-DNN, in which errors introduced by quantization are dynamically compensated during execution. Numbers in compensated-DNN are represented in Fixed Point with Error Compensation (FPEC) format. The bits in FPEC are split between computation bits and compensation bits. The computation bits use conventional floating-point notation (FxP) to represent the number at low-precision. The compensation bits explicitly capture an estimate (direction and magnitude) of the quantization error in the representation. For a given word length, FPEC may use fewer computation bits compared to FxP representation. This enables a near-quadratic improvement in energy in the multiply-and-accumulate (MAC) operations in the DNN. In some embodiments, a low-overhead sparse compensation scheme based on the compensation bits is used to estimate the error accrued during MAC operations, which is then added to the MAC output to minimize the impact of quantization.

It is observed that MAC operations intrinsically exhibit the property of error compensation. Specifically, when MAC operations accumulate multiplication results, the quantization errors of the multiplication operations are also being accrued. If the quantization errors of different multiplication operations take opposite signs, the quantization errors partially cancel each other out, reducing the quantization error in the eventual dot-product output. In practice, the quantization errors of the multiplication operations may be positive or negative. Whether the quantization error is positive or negative is determined based on several factors, including the rounding mode of the inputs, the sign of the inputs, among others. Regardless, the DNN dynamically estimates the accrued quantization error and explicitly offsets the error, thereby minimizing degradation in classification accuracy.

FIG.1illustrates a compensated-DNN100in which the MAC operations are based on numerical values represented in FPEC format to facilitate dynamic compensation of quantization errors, consistent with an exemplary embodiment. As illustrated, the DNN100includes artificial neurons110-119. The artificial neurons110-119are organized into multiple interconnected layers. Each neuron receives a set of inputs and performs dot-product based on its received inputs to produce a single output. The inputs to an artificial neuron may include outputs from other artificial neurons and/or primary inputs of the network. Each neuron may be implemented as a module of software instructions or a module of electrical circuits. The inputs to the artificial neurons may be implemented as data structures in a computer memory or electrical wires in an electronic device.

Each artificial neuron110-119computes a dot-product between a first vector and a second vector. The components of the first vector may be a set of values received from the artificial neuron's interconnections. The components of the second vector may be a set of corresponding weights for the components of the first vector. As illustrated, the artificial neuron116receives inputs x1, x2, and x3 from outputs of neurons110,111, and112. The artificial neuron116applies weights y1, y2, and y3 to the inputs x1, x2, and x3, respectively. The artificial neuron116includes a dot-product processing element120that is configured to compute a weighted sum of the first vector (corresponds to X vector in the figure) with components [x1, x2, x3] using weights in the second vector (corresponds to Y vector in the figure) with components [y1, y2, y3]. An example dot-product processing element120will be described by reference toFIG.4below.

The components of the X vector and the Y vector are in FPEC format. The bit-fields in FPEC are split into 2 groups: computation bits and compensation bits. For a value x (e.g., component x1 in the X vector or component y2 in the Y vector) that is represented in FPEC format, the computation bits provide the quantized value qx. The compensation bits specify an estimate of quantization error Δx, or an estimated error Δx(est).

FIG.2illustrates the FPEC format for representing floating-point numbers, consistent with an exemplary embodiment. As illustrated, the bit-fields in FPEC are split into computation bits and compensation bits. The computation bits include a sign bit (Sign), integer bits (IB), and fractional bits (FB) to capture the range and resolution of numbers. The compensation bits specify the direction and magnitude of the error incurred during quantization. The compensation bits are sub-divided into 3 fields: (i) Error Direction Bit (EDB), (ii) Error Magnitude Bit (EMB) and (iii) Error Zero Bit (EZB). EDB indicates whether the number is rounded up or down, i.e., the direction of the quantization error. EMB denotes the magnitude of the quantization error. The compensation bits of a FPEC number are used to compute an estimated quantization error or a compensation value for compensating the quantized value in the computation bits. The compensation bits of a FPEC are therefore also a compensation instruction.

If the EDB and EMB fields contain δ and θ respectively, then the estimated error is −1δ*2−FB-1-θ. In some embodiments, the error estimates are constrained to powers of 2 to avoid multipliers in the compensation logic. In some embodiments, the EMB field is optional in a FPEC representation. For example, if only the direction of error is specified, θ is assumed to be 0 and the magnitude of the error is half of the resolution i.e., and the estimated error is ±2−FB-1.

The EZB field is a single bit that indicates zero quantization error. EZB enables FPEC to limit compensation to only selected computations. When EZB is 1, EDB and EMB fields are don't cares (e.g., EDB and EMB are not processed by the processing element120).

FIG.3illustrates the mapping between quantization errors and compensation bits of FPEC, consistent with an exemplary embodiment. For a value x represented in FPEC, the figure shows a continuous range300between qx and qx+2−FB, which are two successive quantized values. The range is divided into a number of regions301-306, whose estimated quantization errors are represented in FPEC. Quantization causes values above qx+2−FB-1to be rounded up to qx+2−FBand values below qx+2−FB-1to be rounded down to qx. The EDB indicating the direction of the rounding.

For actual values of x in regions301and306(close to qx or qx+2FB), the quantization error is sufficiently small. The EZB is therefore set to 1 and the estimated error Δx(est) is 0. The size of the region301and306for setting EZB to 1 is controlled by a threshold that can be modulated to control the degree of sparsity in compensation.

For values in regions302-305where EZB=0, the EMB field (optionally) splits the range in powers of 2. The values with larger quantization errors have EMB set to 1 (regions303and304) and the estimated error Δx(est) encoded in the compensation bits is 2−FB-1. The values with small quantization errors have EMB set to 0 (regions302and305) and the estimated error Δx(est) encoded in the compensation bits is 2−FB-2.

Table 1 below illustrates example values that are represented by FPEC format in which the bit-widths of EDB, EMB, and EZB are all 1. For each actual value represented by FPEC, the table shows its corresponding quantized value, quantization error, FPEC estimated error, and FPEC representation. Note that a ‘d’ represents a “don't care” bit. The bit-width of IB is 3 and the bit-width of FB is 0.

The quantization error for the dot product is therefore
Δ(X·Y)=ΔY·QX+ΔX·QY+ΔX·ΔYEq. (2)

The computation bits of the components of the X and Y vectors provide values for quantized values QX and QY. The compensation bits of the components of X and Y provide estimates for quantization errors ΔX and ΔY, or estimated errors ΔX(est) and ΔY(est). The estimated quantization errors ΔX(est) and ΔY(est) can then be used to compute the estimated quantization error for the dot product:
Δ(X·Y)(est)=ΔY(est)·QX+ΔX(est)·QY+ΔX(est)·ΔY(est)  Eq. (3)

In some embodiments, approximations are used during the computation of the estimated quantization error of the dot product. First, since the magnitude of quantization error is typically smaller than the actual value (i.e., QX,QY>>ΔX,ΔY), the higher order term ΔX(est)·ΔY(est) is ignored. Second, input quantization errors are approximated to the nearest power of 2, thereby converting multipliers to shifters when computing ΔY(est)·QX and ΔX(est)·QY. Third, at most 1 or 2 bits are used to capture the magnitude of quantization errors (so only 1 or 2 EMB bits and 3 or 4 compensation bits in total), which makes the shifters less expensive. Fourth, ΔX(est) and ΔY(est) are made sparse by ignoring smaller quantization errors in the input. Therefore, compensation is performed only for certain selected computations, and the compensation logic is active for only a fraction of the overall execution cycles. These approximations significantly reduce the energy expended for estimating the quantization error.

FIG.4illustrates an example design of a processing element400of an artificial neuron. The processing element (PE) performs dot product computation between an X vector and a Y vector. The processing element accepts values in FPEC format. The processing element400is therefore also referred to as an FPEC-PE. The FPEC-PE performs compensation of quantization errors by computing estimated errors based on the FPEC compensation bits. Approximations are used during the computation of the estimated errors.

As illustrated, the FPEC-PE400includes (i) a Computation unit410, (ii) a Compensation unit420, and (iii) a Conversion unit430. In some embodiments, the Computation unit410, the Compensation unit420, and the Conversion unit430are modules of software instructions being executed by one or more processing units (e.g., a processor) of a computing device. In some embodiments, the Computation unit410, the Compensation unit420, and the Conversion unit430are modules of hardware circuits implemented by one or more integrated circuits (ICs) of an electronic apparatus. An example computing device that may implement the FPEC-PE will be described by reference toFIG.6below.

The computation unit410includes a multiply-and-accumulate engine412that evaluates the dot-product using the values in [IB,FB] fields (computation bits) of FPEC. Specifically, the multiply-and-accumulate engine412multiplies a component from the X vector with a corresponding component from the Y vector to produce a raw product value to be added by an adder414to an accumulator register416.

The compensation unit420simultaneously evaluates the quantization error at the dot-product output using the [EDB,EMB,EZB] fields (compensation bits). This involves shifting the X vector component using EMB bits of xi(at shifter421) and the Y vector component using EMB bits of yi(at shifter422) and appropriately adding/subtracting (at adder424) them from the compensation sum (at adder426and ErrComp register428) based on the respective EDB bits. The EZB bits of xiand yicombine (logic AND) to render the compensation unit420inactive or in low power mode (by e.g., clock gating, signal gating, etc.). After all inputs are processed, the compensation sum (i.e., estimated error for the dot product Δ(X·Y)(est)) is added to the accumulator register412in the computation unit410to produce a compensated dot-product output.

The conversion unit430quantizes the output to its desired FPEC format. The conversion unit430includes a shifter to scale the output based on [IB,FB]. The conversion unit430includes a bit-wise logic to infer output compensation bits [EDB,EMB,EZB] based on the scaled bits. The output can be used as a vector to another processing element that uses the quantized values in the output to compute a raw product value and compensates the computed raw product according to the compensation bits in the output.

It is worth noting that the PE-FPEC400is energy efficient because FPEC representation allows quantization error to be expressed by fewer computation bits. This yields a near-quadratic improvement in the computation unit410, as the multiplier, adder, and accumulator bit-widths are correspondingly reduced. On the other hand, the adders and registers in the compensation unit420do incur a linear increase in logic. However, they are used sporadically, such as when the EZBs of xiand yiare asserted (once every 3 to 5 computations in experiments). The overhead in the conversion unit430is negligible, as it is active only once per dot-product, which takes thousands of computation cycles in large-scale DNNs.

By representing values in the FPEC format, an artificial neuron is able to perform dot product operations with minimal quantization errors by using fewer computation bits and very few compensation bits. An architecture based on FPEC such as the processing element400can rely on approximations to compute estimated quantization errors with reduced energy expenditure and computation resources. This improves DNNs in terms of size and power, which is critical for lower power applications.

FIG.5conceptually illustrates a process500for computing a dot product between two vectors at an artificial neuron of a DNN, consistent with an exemplary embodiment. In some embodiments, a processing element (e.g., PE-FPEC400) of an artificial neuron performs the process500. In some embodiments, one or more processing units (e.g., processor) of a computing device implementing the DNN100perform the process500by executing instructions stored in a computer readable medium.

The process500starts when the processing element receives (at510) a first vector having a set of components (e.g., X vector). Each component of the first vector having a first quantized value (e.g., qx) and a first compensation instruction (e.g., estimated quantization error ΔX(est)). The process element also receives (at520) a second vector having a corresponding set of components (e.g., Y vector). Each component of the second vector has a second quantized value (e.g., qy) and a second compensation instruction (e.g., estimated quantization error ΔY(est)).

For some embodiments, the components of the first vector correspond to input connections to the artificial neuron, while the components of the second vector correspond to weights that are to be applied to the input connections. For a vector component that is in FPEC format, its quantized value is represented by the computation bits [IB,FB] and its compensation instruction includes the compensation bits [EDB, EMB, EZB]. The compensation bits also represent an estimated quantization error for the vector component.

The processing element multiplies (at530) a component of the first vector with a corresponding component of the second vector by multiplying the first quantized value with the second quantized value to produce a raw product value as part of the dot product computation.

The processing element determines (at535) whether to compensate for quantization errors based on EZB of the first and second compensation instructions. If the EZB of both the first and second compensation instructions are asserted (indicating both ΔX(est) and ΔY(est) are zero), the process proceeds to560as there is no need to perform compensation. Otherwise the process proceeds to540to perform quantization error compensation.

At540, the processing element computes a compensation value (or estimated quantization error) for the raw product value based on EMB and EDB in the first and second compensation instructions. The process unit compensates (at550) the raw product value by using the computed compensation value to produce a compensated product value. The processing element then adds (at555) the compensated product value to an accumulated value for the dot product. The process then proceeds to580.

At560, the processing element disables the error compensation operation by placing the compensation unit in low power mode, disabling the compensation unit, etc., since the estimated error of both X component and Y component are zeros. The processing element adds (at570) the raw product value to the accumulated value for the dot product without computing the compensation value. The process then proceeds to580.

At580, the processing element determines whether there are more components in the first and second vectors that have yet to be multiplied and accumulated for the dot product. If so, the process returns to530to process the next pair of components from the first and second vectors. If all of the components have been multiplied and accumulated for the dot product, the process proceeds to590.

At590, the processing element converts the accumulated value into an output vector of the dot product that includes an output quantized value and an output compensation instruction (with EZB, EDB, and EMB). This operation corresponds to the conversion unit430. The process500then ends.

Computer readable program instructions described herein can be downloaded to respective computing/processing devices from a computer readable storage medium or to an external computer or external storage device via a network, for example, the Internet, a local area network, a wide area network and/or a wireless network. The network may comprise copper transmission cables, optical transmission fibers, wireless transmission, routers, firewalls, switches, gateway computers and/or edge servers. A network adapter card or network interface in each computing/processing device receives computer readable program instructions from the network and forwards the computer readable program instructions for storage in a computer readable storage medium within the respective computing/processing device. Computer readable program instructions for carrying out operations of the present disclosure may be assembler instructions, instruction-set-architecture (ISA) instructions, machine instructions, machine dependent instructions, microcode, firmware instructions, state-setting data, configuration data for integrated circuitry, or either source code or object code written in any combination of one or more programming languages, including an object oriented programming language such as Smalltalk, C++, or the like, and procedural programming languages, such as the “C” programming language or similar programming languages. The computer readable program instructions may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection may be made to an external computer (for example, through the Internet using an Internet Service Provider). In some embodiments, electronic circuitry including, for example, programmable logic circuitry, field-programmable gate arrays (FPGA), or programmable logic arrays (PLA) may execute the computer readable program instructions by utilizing state information of the computer readable program instructions to personalize the electronic circuitry, in order to perform aspects of the present disclosure.

The computer readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus or other device to produce a computer implemented process, such that the instructions which execute on the computer, other programmable apparatus, or other device implement the functions/acts specified in the flowchart and/or block diagram block or blocks. The flowchart and block diagrams in the Figures (e.g.,FIG.5) illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of instructions, which comprises one or more executable instructions for implementing the specified logical function(s). In some alternative implementations, the functions noted in the blocks may occur out of the order noted in the Figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems that perform the specified functions or acts or carry out combinations of special purpose hardware and computer instructions.

FIG.6shows a block diagram of the components of data processing systems600and650that may be used to implement a DNN, an artificial neuron, or a FPEC processing element in accordance with an illustrative embodiment of the present disclosure. It should be appreciated thatFIG.6provides only an illustration of one implementation and does not imply any limitations with regard to the environments in which different embodiments may be implemented. Many modifications to the depicted environments may be made based on design and implementation requirements.

The data processing systems600and650may include a set of internal components600and a set of external components650illustrated inFIG.6. The set of internal components600includes one or more processors620, one or more computer-readable RAMs622and one or more computer-readable ROMs624on one or more buses626, and one or more operating systems628and one or more computer-readable tangible storage devices630. The one or more operating systems628and programs such as the programs for executing the process500are stored on one or more computer-readable tangible storage devices630for execution by one or more processors620via one or more RAMs622(which typically include cache memory). In the embodiment illustrated inFIG.6, each of the computer-readable tangible storage devices630is a magnetic disk storage device of an internal hard drive. Alternatively, each of the computer-readable tangible storage devices630is a semiconductor storage device such as ROM624, EPROM, flash memory or any other computer-readable tangible storage device that can store a computer program and digital information.

The set of internal components600also includes a R/W drive or interface632to read from and write to one or more portable computer-readable tangible storage devices686such as a CD-ROM, DVD, memory stick, magnetic tape, magnetic disk, optical disk or semiconductor storage device. The instructions for executing the processes500can be stored on one or more of the respective portable computer-readable tangible storage devices686, read via the respective R/W drive or interface632and loaded into the respective hard drive630.

The set of internal components600may also include network adapters (or switch port cards) or interfaces636such as a TCP/IP adapter cards, wireless Wi-Fi interface cards, or 3G or 4G wireless interface cards or other wired or wireless communication links. Instructions of processes or programs described above can be downloaded from an external computer (e.g., server) via a network (for example, the Internet, a local area network or other, wide area network) and respective network adapters or interfaces636. From the network adapters (or switch port adaptors) or interfaces636, the instructions and data of the described programs or processes are loaded into the respective hard drive630. The network may comprise copper wires, optical fibers, wireless transmission, routers, firewalls, switches, gateway computers and/or edge servers.

The set of external components650can include a computer display monitor670, a keyboard680, and a computer mouse684. The set of external components650can also include touch screens, virtual keyboards, touch pads, pointing devices, and other human interface devices. The set of internal components600also includes device drivers640to interface to computer display monitor670, keyboard680and computer mouse684. The device drivers640, R/W drive or interface632and network adapter or interface636comprise hardware and software (stored in storage device630and/or ROM624).