Patent Description:
An artificial neuron is a mathematical function whose output is a nonlinear function of a linear combination of its inputs. Two neurons are connected if the output of one is an input to the other. A weight is a scalar value encoding the strength of the connection between the output of one neuron and the input of another neuron.

A neuron computes its output, called an activation, by applying a nonlinear activation function to a weighted sum of its inputs. A weighted sum is an intermediate result computed by multiplying each input with the corresponding weight and accumulating the products. A partial sum is a weighted sum of a subset of inputs. A weighted sum of all inputs may be computed in stages by accumulating one or more partial sums.

A neural network is a collection of one or more neurons. A neural network is often divided into groups of neurons called layers. A layer is a collection of one or more neurons that all receive input from the same layers and all send output to the same layers, and typically perform a similar function. An input layer is a layer that receives input from a source outside the neural network. An output layer is a layer that sends output to a target outside the neural network. All other layers are intermediate processing layers. A multilayer neural network is a neural network with more than one layer. A deep neural network is a multilayer neural network with many layers.

A tensor is a multidimensional array of numerical values. A tensor block is a contiguous subarray of the elements in a tensor.

Each neural network layer is associated with a parameter tensor V, weight tensor W, input data tensor X, output data tensor Y, and intermediate data tensor Z. The parameter tensor contains all of the parameters that control neuron activation functions σ in the layer. The weight tensor contains all of the weights that connect inputs to the layer. The input data tensor contains all of the data that the layer consumes as input. The output data tensor contains all of the data that the layer computes as output. The intermediate data tensor contains any data that the layer produces as intermediate computations, such as partial sums.

The data tensors (input, output, and intermediate) for a layer may be <NUM>-dimensional, where the first two dimensions may be interpreted as encoding spatial location and the third dimension as encoding different features. For example, when a data tensor represents a color image, the first two dimensions encode vertical and horizontal coordinates within the image, and the third dimension encodes the color at each location. Every element of the input data tensor X can be connected to every neuron by a separate weight, so the weight tensor W generally has <NUM> dimensions, concatenating the <NUM> dimensions of the input data tensor (input row a, input column b, input feature c) with the <NUM> dimensions of the output data tensor (output row i, output column j, output feature k). The intermediate data tensor Z has the same shape as the output data tensor Y. The parameter tensor V concatenates the <NUM> output data tensor dimensions with an additional dimension o that indexes the parameters of the activation function σ.

An element of a layer's output data tensor Y can be computed as in Equation <NUM> where the neuron activation function σ is configured by the vector of activation function parameters V[i, j, k, : ], and the weighted sum Z[i, j, k] can be computed as in Equation <NUM>. <MAT> <MAT> For simplicity of notation, the weighted sum in Equation <NUM> may be referred to as the output, which is equivalent to using a linear activation function Y[i, j, k] = σ(Z[i, j, k]) = Z[i, j, k], with the understanding that the same statements apply without loss of generality when a different activation function is used. However, there is a need for flexible precision of neural inferencing. <CIT>, <NPL>) discloses Massively parallel neural inference computing elements. A plurality of multipliers is arranged in a plurality of equal - sized groups. Each of the plurality of multipliers is adapted to , in parallel , apply a weight to an input activation to generate an output. A plurality of adders is operatively coupled to one of the groups of multipliers. Each of the plurality of adders is adapted to , in parallel , add the outputs of the multipliers within its associated group to generate a partial sum. A plurality of function blocks is operatively coupled to one of the plurality of adders. Each of the plurality of function blocks is adapted to , in parallel , apply a function to the partial sum of its associated adder to generate an output value. Therefore, there is a need in the art to address the aforementioned problem.

According to embodiments of the present disclosure, neural inference chips are provided, comprising a neural core. The neural core comprises a vector-matrix multiplier defined by the subject-matter of claim <NUM>.

According to an embodiment of the present disclosure, a method according to claim <NUM> is provided. A computer program product not encompassed by the wording of the claims for flexible precision neural inference is provided. In various embodiments, a weight matrix having a first precision is received. An activation vector is received having the first precision. A vector-matrix multiplication is computed of the weight matrix and the activation vector, yielding a partial sum vector a second precision. One or more vector functions is performed on the partial sum vector to yield a vector processor output vector having the second precision. An activation function is applied to the vector processor output vector, yielding an output activation vector having a third precision. At least one of the first, second, and third precision is varied at runtime.

The present invention will now be described, by way of example only, with reference to preferred embodiments, as illustrated in the following figures:.

In various embodiments, computation of the output data tensor as described above is decomposed into smaller problems. Each problem may then be solved on one or more neural core, or on one or more core of a conventional multicore system in parallel.

With reference now to <FIG>, a neural core according to embodiments of the present disclosure is depicted. A neural core <NUM> is a tileable computational unit that computes one block of an output tensor. A neural core <NUM> has M inputs and N outputs. In various embodiments, M = N. To compute an output tensor block, a neural core multiplies an M x <NUM> input tensor block <NUM> with an M × N weight tensor block <NUM> and accumulates the products into weighted sums that are stored in a <NUM> × N intermediate tensor block <NUM>. A O × N parameter tensor block (<NUM>) contains the O parameters that specify each of the N neuron activation functions that are applied to the intermediate tensor block <NUM> to produce a <NUM> × N output tensor block <NUM>.

Multiple neural cores may be tiled in a neural core array. In some embodiments, the array is <NUM>-dimensional.

A neural network model is a set of constants that collectively specify the entire computation performed by a neural network, including the graph of connections between neurons as well as the weights and activation function parameters for every neuron. Training is the process of modifying the neural network model to perform a desired function. Inference is the process of applying a neural network to an input to produce an output, without modifying the neural network model.

An inference processing unit is a category of processors that perform neural network inference. A neural inference chip is a specific physical instance of an inference processing unit.

Referring to <FIG>, an exemplary Inference Processing Unit (IPU) is illustrated according to embodiments of the present disclosure. IPU <NUM> includes a memory <NUM> for the neural network model. As described above, the neural network model may include the synapse weights for a neural network to be computed. IPU <NUM> includes an activation memory <NUM>, which may be transient. Activation memory <NUM> may be divided into input and output regions, and stores neuron activations for processing. IPU <NUM> includes a neural computation unit <NUM>, which is loaded with a neural network model from model memory <NUM>. Input activations are provided from activation memory <NUM> in advance of each computation step. Outputs from neural computation unit <NUM> are written back to activation memory <NUM> for processing on the same or another neural computation unit.

In various embodiments a microengine <NUM> is included in IPU <NUM>. In such embodiments, all operations in the IPU are directed by the microengine. As set out below, central and/or distributed microengines may be provided in various embodiments. A global microengine may be referred to as a chip microengine, while a local microengine may be referred to as a core microengine or local controller. In various embodiments a microengine comprises one or more microengines, microcontrollers, state machines, CPUs, or other controllers.

Referring to <FIG>, a multi-core Inference Processing Unit (IPU) is illustrated according to embodiments of the present disclosure. IPU <NUM> includes a memory <NUM> for the neural network model and instructions. In some embodiments, memory <NUM> is divided into weigh portion <NUM> and instruction portion <NUM>. As described above, the neural network model may include the synapse weights for a neural network to be computed. IPU <NUM> includes an activation memory <NUM>, which may be transient. Activation memory <NUM> may be divided into input and output regions, and stores neuron activations for processing. IPU <NUM> includes a plurality of cores <NUM>. Each core <NUM> includes a neural computation unit <NUM>, which is loaded with a neural network model from model memory <NUM>. Each core also include a local activation memory <NUM>. Input activations are provided from local activation memory <NUM> in advance of each computation step. Outputs from neural computation unit <NUM> are written back to activation memory <NUM> for processing on the same or another neural computation unit.

IPU <NUM> includes an array <NUM> of neural cores <NUM>. Each core <NUM> includes a computation unit <NUM>, which is loaded with a neural network model from model memory <NUM> and is operative to perform vector computation. Each core also includes a local activation memory <NUM>. Input activations are provided from local activation memory <NUM> in advance of each computation step. Outputs from computation unit <NUM> are written back to activation memory <NUM> for processing on the same or another computation unit.

IPU <NUM> includes one or more network-on-chip (NoC) <NUM>. In some embodiments, a partial sum NoC <NUM> interconnects the cores <NUM> and transports partial sums among them. In some embodiments, a separate parameter distribution NoC <NUM> connects cores <NUM> to memory <NUM> for distributing weights and instructions to cores <NUM>. It will be appreciated that various configurations of NoC <NUM> and <NUM> are suitable for use according to the present disclosure. For example, broadcast networks, row broadcast networks, tree networks, and switched networks may be used.

In various embodiments a global microengine <NUM> is included in IPU <NUM>. In various embodiments, a local core controller <NUM> is included on each core <NUM>. In such embodiments, the direction of operations is shared between the global microengine (chip microengine) and the local core controller (core microengine). In particular, at <NUM>, compute instructions are loaded from model memory <NUM> to the neural computation unit <NUM> on each core <NUM> by global microengine <NUM>. At <NUM>, parameters (e.g., neural network/synaptic weights) are loaded from model memory <NUM> to the neural computation unit <NUM> on each core <NUM> by global microengine <NUM>. At <NUM>, neural network activation data are loaded from activation local activation memory <NUM> to neural computation unit <NUM> on each core <NUM> by local core controller <NUM>. As noted above, the activations are provided to the axons of the particular neural network defined by the model, and may originate from the same or another neural computation unit, or from outside the system. At <NUM>, neural computation unit <NUM> performs the computation to generate output neuron activations as directed by local core controller <NUM>. In particular, the computation comprises applying the input synaptic weights to the input activations. It will be appreciated that various methods are available for performing such computations, including in silico dendrites, as well as vector multiplication units. At <NUM>, the results from computation are stored in local activation memory <NUM> as directed by local core controller <NUM>. As described above, these stages may be pipelined, in order to provide efficient usage of the neural computation unit on each core. It will also be appreciated that inputs and outputs may be transferred from local activation memory <NUM> to global activation memory <NUM> according to the requirements of a given neural network.

Computation unit <NUM> performs the computation to generate output neuron activations as directed by local core controller <NUM>. In particular, the computation comprises applying the input synaptic weights to the input activations. It will be appreciated that various methods are available for performing such computations, including in silico dendrites, as well as vector multiplication units. The results from computation are stored in local activation memory <NUM> as directed by local core controller <NUM>. These stages may be pipelined, in order to provide efficient usage of the computation unit on each core. It will also be appreciated that inputs and outputs may be transferred from local activation memory <NUM> to global activation memory <NUM> according to the requirements of a given neural network.

Accordingly, the present disclosure provides for runtime control of operations in an Inference Processing Unit (IPU). In some embodiments, the microengine is centralized (single microengine). In some embodiments, the IPU computation is distributed (performed by an array of cores). In some embodiments, runtime control of operations is hierarchical - both a central microengine and distributed microengines participate.

The microengine or microengines direct the execution of all operations in the IPU. Each microengine instruction corresponds to several sub-operations (e.g., address generation, load, compute, store, etc.) In the distributed case, core microcode is run on the core microengines (e.g., <NUM>). The core microcode includes instruction(s) to execute a full, single tensor operation. For example, a convolution between a weight tensor and a data tensor. In the context of a single core, the core microcode includes instruction(s) to execute a single tensor operation on the locally stored subset of the data tensor (and partial sums). Chip microcode is run on the chip microengine (e.g., <NUM>). Microcode includes instructions to execute all of the tensor operations in a neural network.

With reference now to <FIG>, an exemplary neural core and associated networks are illustrated according to embodiments of the present disclosure. Core <NUM>, which may be embodied as described with reference to <FIG> is interconnected with additional cores by networks <NUM>. In this embodiments, network <NUM> is responsible for distributing weights and/or instructions, network <NUM> is responsible for distributing partial sums, and network <NUM> is responsible for distributing activations. However, it will be appreciated that the various embodiments of the present disclosure may combine these networks, or further separate them into multiple additional networks.

Input activations (X) are distributed to core <NUM> from off-core via activation network <NUM> to activation memory <NUM>. Layer instructions are distributed to core <NUM> from off-core via weight/instruction network <NUM> to instruction memory <NUM>. Layer weights (W) and/or parameters are distributed to core <NUM> from off-core via weight/instruction network <NUM> to weight memory <NUM> and/or parameter memory <NUM>.

The weight matrix (W) is read from weight memory <NUM> by Vector Matrix Multiply (VMM) unit <NUM>. The activation vector (V) is read from activation memory <NUM><NUM> by Vector Matrix Multiply (VMM) unit <NUM>. Vector Matrix Multiply (VMM) unit <NUM> then computes vector-matrix multiplication Z = XTW and provides the result to Vector-Vector unit <NUM>. Vector-Vector unit <NUM> reads additional partial sums from partial sum memory <NUM>, and receives additional partial sums from off-core via partial sum network <NUM>. A vector-vector operation is computed by Vector-Vector unit <NUM> from these source partial sums. For example, the various partial sums may in turn be summed. The resulting target partial sums are written to partial sum memory <NUM>, sent off-core via partial sum network <NUM>, and/or fed back for further processing by Vector-Vector unit <NUM>.

The partial sum results from Vector-Vector unit <NUM>, after all computation for a given layer's inputs is complete, are provided to activation unit <NUM> for the computation of output activations. The activation vector (Y) is written to activation memory <NUM>. Layer activations (including the results written to activation memory) are redistributed across cores from activation memory <NUM> via activation network <NUM>. Upon receipt, they are written to local activation memory to each receiving core. Upon completion of processing for a given frame, the output activations are read from activation memory <NUM> and sent off-core via network <NUM>.

Low-precision computation has certain advantages with respect to power, performance, and area. In particular, less energy is required per operation. Higher operation frequency is achievable (due to fewer levels of logic). A smaller circuit implementation area is needed.

However, low-precision computation also has certain disadvantages. There is the potential for loss of accuracy. For example, for networks that are trained for higher precision, there may be losses where inference is performed in low precision.

To addresses these conflicting goals, the present disclosure provides for flexible precision in neural inference, combining the advantages of both high and low precision computation. In various embodiments, high-precision computation is performed when/where needed, and low-precision when/where performance can use it.

In various embodiments, a flexible precision computation unit is provided, including a flexible precision Vector-Matrix Multiplier (VMM), a flexible prevision vector unit, and a flexible precision activation function unit. In various embodiments, flexible precision data transmission is provided, including flexible precision activation, weight, and partial sum buses or networks. In various embodiments, flexible prevision storage is provided, including activation, weight, and partial sum memory. In various embodiments, conversion between precisions is provided, including reformatting of values from compute to memory.

In some exemplary embodiments, the VMM converts from flexible precision (e.g., <NUM> bit/<NUM> bit/<NUM> bit) to high, fixed-precision output (e.g., <NUM> bit). In such embodiments, the Vector Unit is high, fixed precision (e.g., <NUM> bits). The activation function unit converts from high, fixed precision (e.g., <NUM> bit) input to flexible precision (e.g., <NUM> bit/<NUM> bit/<NUM> bit) output. The activation functions are used as squashing/re-ranging functions.

In other exemplary embodiments, the VMM converts from flexible precision (e.g., <NUM> bit/<NUM> bit/<NUM> bit) to a higher, also flexible-precision (e.g., <NUM> bit/<NUM> bit/<NUM> bit) internal representation. In such embodiments, the Vector Unit is high, flexible precision (e.g., <NUM> bit/<NUM> bit/<NUM> bit) and the activation function unit is high, flexible precision (e.g., <NUM> bit/<NUM> bit/<NUM> bit) input to low, flexible precision output (e.g., <NUM> bit/<NUM> bit/<NUM> bit). The activation functions are used as squashing/re-ranging functions.

In an exemplary flexible precision VMM, the VMM Unit performs the operation: Z = XTW, supports input precisions (X, W) of <NUM> bit/<NUM> bit/<NUM> bit, and always outputs precision (Z) of <NUM> bit. Referring to Table <NUM>, the VMM inputs and outputs are shown. Referring to Table <NUM>, the weight sizes for each configuration are provided.

In an exemplary activation function unit, the input precision (Z) is <NUM> bit, and output precisions (Y) of <NUM> bit/<NUM> bit/<NUM> bit are supported. To provides flexible precision, activation function computation is performed in two stages. In stage <NUM>, the activation function is computed using <NUM> bit input precision and <NUM> bit output precision. In stage <NUM>, the <NUM> bit output is rounded to the appropriate precision if required (keeping the most significant <NUM> or <NUM> bits of the <NUM> bit output). Referring to Table <NUM>, the precision conversion in various configurations is illustrated.

The quad precision (4x activation memory, index <NUM> above) selection is used for full precision readout of partial sum values as neuron output. This is used for functions such as linear regression. When quad output precision is selected, the Activation Function computation is ignored. Instead, the full <NUM> bit zj is directly assigned to the quad (<NUM> bit) yj output. The quad precision readout mode requires <NUM> cycles to perform readout, limiting to <NUM> cycles between back-to-back operations.

In various embodiments, the activation functions are designed by construction as saturating functions to avoid overflow and underflow. The saturating functions use the full <NUM> bit zj for floor/ceiling bound checking. They saturate at:.

The output precision conversion functions avoid overflow and underflow by converting after the saturating activation functions. The saturating activation function uses the same <NUM> bit uj range in all cases, and then rounds off the least significant bits for the appropriate output precision.

Exemplary saturating functions include binary, trinary, sigmoid, tanh, bounded ReLU, bound LU, and others known in the art.

Referring to <FIG>, exemplary activation functions are illustrated according to the present disclosure. These activation transfer function shapes perform re-ranging. They are well suited for multiple precisions. <FIG> depicts a Boolean function. <FIG> depicts a trinary function. <FIG> depicts a bounded linear unit. <FIG> depicts a bounded ReLU. <FIG> depicts a bounded shifted ReLU. <FIG> depicts a bounded pReLU. <FIG> depicts a bounded exp ReLU. <FIG> depicts a sigmoid function. <FIG> depicts a tanh function.

In various embodiments, flexible precision communication and storage is provided. The same networks and memories support all precisions (e.g., <NUM> bit/<NUM> bit/<NUM> bit) by changing the format of the data, using the same underlying physical substrate. In various embodiments, the overall network bandwidth (e.g., expressed in the number of wires on a bus) is maintained constant, while the number of elements and precision vary. In particular, the overall network usage is given by the number of elements multiplied by the precision. In this way, ongoing adjustments may be made to precision and element count while using a target amount of bandwidth (e.g., all available wires). Likewise, in various embodiments, overall storage utilization is maintained constant, while the number of elements and precision vary.

In various embodiments, the weight NoC supports precisions of <NUM> bit/<NUM> bit/<NUM> bit. In some embodiments, a <NUM> wire bus is provided, supporting <NUM> elements at <NUM> bit, <NUM> elements at <NUM> bit, and <NUM> elements at <NUM> bit.

In various embodiments, the activation NoC supports precisions of <NUM> bit/<NUM> bit/<NUM> bit. In some embodiments, a <NUM> wire bus is provided, supporting <NUM> elements at <NUM> bit, <NUM> elements at <NUM> bit, and <NUM> elements at <NUM> bit.

In various embodiments, the weight memory is organized as <NUM> read data wires. In such embodiments, the <NUM> wires support <NUM> x <NUM> elements at <NUM> bit, <NUM> x <NUM> elements at <NUM> bit, and <NUM> x <NUM> elements at <NUM> bit.

In various embodiments, the activation memory is organized as <NUM> read data wires. In such embodiments, the <NUM> wires support <NUM> elements at <NUM> bit, <NUM> elements at <NUM> bit, and <NUM> elements at <NUM> bit.

Referring now to <FIG>, an activation memory is illustrated according to embodiments of the present disclosure. Activation memory <NUM> supports <NUM> bit/<NUM> bit/<NUM> bit format. As shown, an activation block <NUM> with <NUM> bit precision, an activation block <NUM> with <NUM> bit precision, and an activation block with <NUM> bit precision are supported.

Referring now to <FIG>, an activation memory is illustrated according to embodiments of the present disclosure, in this example, activation memory <NUM> uses a quad precision (<NUM> bit) format. In this example, the activation block spans multiple bank word addresses.

Referring now to <FIG>, a weight memory is illustrated according to embodiments of the present disclosure. As shown, a <NUM> bit format <NUM>, <NUM> bit format <NUM>, and <NUM> bit format <NUM> are supported.

In various embodiments an activation function unit output is <NUM> elements x <NUM> bit, <NUM> bit, or <NUM> bit. However, the activation word size is <NUM> elements at <NUM> bit, <NUM> elements at <NUM> bit, or <NUM> elements at <NUM> bit. For low-precision computation multiple activation function unit output words are combined as follows: 1x <NUM> elements at <NUM> bit, 2x <NUM> elements at <NUM> bit, or 4x <NUM> elements at <NUM> bit.

To accommodate this reformatting, in various embodiments, sub vectors [<NUM>:<NUM>] at <NUM> bit or [<NUM>:<NUM>] at <NUM> bit are stored in a register prior to writing the full activation vector [<NUM>:<NUM>] back to memory. At <NUM> bit, <NUM> cycle accommodates the full activation vector. At <NUM> bit, <NUM> cycles accommodate the full activation vector, where cycle <NUM> covers bits <NUM>:<NUM> and cycle <NUM> covers bits <NUM>:<NUM>. At <NUM> bit, <NUM> cycles accommodate the full activation vector, where cycle <NUM> covers bits <NUM>:<NUM>, cycle <NUM> covers bits <NUM>:<NUM>, cycle <NUM> covers bits <NUM>:<NUM>, and cycle <NUM> covers bits <NUM>:<NUM>.

In various embodiments, partial full vectors are zero padded, where zeros go into the unspecified sub vector elements. A full vector can be written back to memory even if not all of the sub vectors have been specified. Unused sub vector slots are zeros. This reduces computation time for <NUM> bit <NUM> bit precision networks where the full dimension of the activation vector is not required. For example, in <NUM> bit mode, if the output activation vector is only <NUM> elements, only two cycles of computation are required.

Referring to <FIG>, a method of flexible precision neural processing is illustrated according to embodiments of the present disclosure. At <NUM>, a weight matrix having a first precision is received. At <NUM>, an activation vector is received having the first precision. At <NUM>, a vector-matrix multiplication is computed of the weight matrix and the activation vector, yielding a partial sum vector having a second precision. At <NUM>, one or more vector functions is performed on the partial sum vector to yield a vector processor output vector having the second precision. At <NUM>, an activation function is applied to the vector processor output vector, yielding an output activation vector having a third precision. At least one of the first, second, and third precision is varied at runtime.

Referring now to <FIG>, a schematic of an example of a computing node is shown. Computing node <NUM> is only one example of a suitable computing node and is not intended to suggest any limitation as to the scope of use or functionality of embodiments described herein. Regardless, computing node <NUM> is capable of being implemented and/or performing any of the functionality set forth hereinabove.

In computing node <NUM> there is a computer system/server <NUM>, which is operational with numerous other general purpose or special purpose computing system environments or configurations. Examples of well-known computing systems, environments, and/or configurations that may be suitable for use with computer system/server <NUM> include, but are not limited to, personal computer systems, server computer systems, thin clients, thick clients, handheld or laptop devices, multiprocessor systems, microprocessor-based systems, set top boxes, programmable consumer electronics, network PCs, minicomputer systems, mainframe computer systems, and distributed cloud computing environments that include any of the above systems or devices, and the like.

Computer system/server <NUM> may be described in the general context of computer system-executable instructions, such as program modules, being executed by a computer system. Generally, program modules may include routines, programs, objects, components, logic, data structures, and so on that perform particular tasks or implement particular abstract data types. Computer system/server <NUM> may be practiced in distributed cloud computing environments where tasks are performed by remote processing devices that are linked through a communications network. In a distributed cloud computing environment, program modules may be located in both local and remote computer system storage media including memory storage devices.

As shown in <FIG>, computer system/server <NUM> in computing node <NUM> is shown in the form of a general-purpose computing device. The components of computer system/server <NUM> may include, but are not limited to, one or more processors or processing units <NUM>, a system memory <NUM>, and a bus <NUM> that couples various system components including system memory <NUM> to processor <NUM>.

Bus <NUM> represents one or more of any of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, and a processor or local bus using any of a variety of bus architectures. By way of example, and not limitation, such architectures include Industry Standard Architecture (ISA) bus, Micro Channel Architecture (MCA) bus, Enhanced ISA (EISA) bus, Video Electronics Standards Association (VESA) local bus, Peripheral Component Interconnect (PCI) bus, Peripheral Component Interconnect Express (PCIe), and Advanced Microcontroller Bus Architecture (AMBA).

Computer system/server <NUM> typically includes a variety of computer system readable media. Such media may be any available media that is accessible by computer system/server <NUM>, and it includes both volatile and non-volatile media, removable and non-removable media.

System memory <NUM> can include computer system readable media in the form of volatile memory, such as random access memory (RAM) <NUM> and/or cache memory <NUM>. Computer system/server <NUM> may further include other removable/non-removable, volatile/non-volatile computer system storage media. By way of example only, storage system <NUM> can be provided for reading from and writing to a non-removable, non-volatile magnetic media (not shown and typically called a "hard drive"). Although not shown, a magnetic disk drive for reading from and writing to a removable, non-volatile magnetic disk (e.g., a "floppy disk"), and an optical disk drive for reading from or writing to a removable, non-volatile optical disk such as a CD-ROM, DVD-ROM or other optical media can be provided. In such instances, each can be connected to bus <NUM> by one or more data media interfaces. As will be further depicted and described below, memory <NUM> may include at least one program product having a set (e.g., at least one) of program modules that are configured to carry out the functions of embodiments of the disclosure.

Program/utility <NUM>, having a set (at least one) of program modules <NUM>, may be stored in memory <NUM> by way of example, and not limitation, as well as an operating system, one or more application programs, other program modules, and program data. Each of the operating system, one or more application programs, other program modules, and program data or some combination thereof, may include an implementation of a networking environment. Program modules <NUM> generally carry out the functions and/or methodologies of embodiments as described herein.

Computer system/server <NUM> may also communicate with one or more external devices <NUM> such as a keyboard, a pointing device, a display <NUM>, etc.; one or more devices that enable a user to interact with computer system/server <NUM>; and/or any devices (e.g., network card, modem, etc.) that enable computer system/server <NUM> to communicate with one or more other computing devices. Such communication can occur via Input/Output (I/O) interfaces <NUM>. Still yet, computer system/server <NUM> can communicate with one or more networks such as a local area network (LAN), a general wide area network (WAN), and/or a public network (e.g., the Internet) via network adapter <NUM>. As depicted, network adapter <NUM> communicates with the other components of computer system/server <NUM> via bus <NUM>. It should be understood that although not shown, other hardware and/or software components could be used in conjunction with computer system/server <NUM>. Examples, include, but are not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data archival storage systems, etc..

The present disclosure may be embodied as a system, a method, and/or a computer program product.

Computer readable program instructions for carrying out operations of the present disclosure may be assembler instructions, instruction-set-architecture (ISA) instructions, machine instructions, machine dependent instructions, microcode, firmware instructions, state-setting data, or either source code or object code written in any combination of one or more programming languages, including an object oriented programming language such as Smalltalk, C++ or the like, and conventional procedural programming languages, such as the "C" programming language or similar programming languages. In some embodiments, electronic circuitry including, for example, programmable logic circuitry, field-programmable gate arrays (FPGA), or programmable logic arrays (PLA) may execute the computer readable program instructions by utilizing state information of the computer readable program instructions to personalize the electronic circuitry, in order to perform aspects of the present disclosure.

Claim 1:
A neural inference chip comprising a neural core (<NUM>), the neural core comprising:
a vector-matrix multiplier (<NUM>) adapted to: receive a weight matrix having a weight matrix precision; receive an input activation vector having an input activation vector precision; wherein the vector-matrix multiplier is further adapted to vary the weight matrix precision and dimension and/or the input activation vector precision and dimension while maintaining constant bandwidth; and compute a partial sum vector by multiplying the input activation vector by the weight matrix, the partial sum vector having a partial sum vector precision, the partial sum vector comprising an output of the vector-matrix multiplier;
a vector processor (<NUM>) adapted to: receive one or more partial sum vectors from one or more vector sources, the one or more vector sources including the vector-matrix multiplier; and perform one or more vector functions on the one or more partial sum vectors to yield a vector processor output vector, the vector processor output vector having a precision equal to the partial sum vector precision;
responsive to a required output activation precision, the weight matrix precision and the input activation vector precision, an activation unit (<NUM>) operatively coupled to the vector processor and adapted to apply an activation function to the vector processor output vector, yielding an output activation vector having an output activation precision; and
wherein each of the vector-matrix multiplier, vector processor, and/or activation unit are adapted to operate at variable precisions.