Patent Description:
Typical machine learning inference and other processing functions involve computing an accumulation of products fed into a non-linear activation function. Some computer architectures support vector and/or matrix operations for numeric multiplication and addition as well as MAX and MIN. Some activation functions are simple and can be performed by some vector primitives. Other activation functions are less simple. Improvements to processor architectures that implement these types of functions (e.g., non-linear functions, activation functions, etc.) can improve the overall performance of the processor from a computational efficiency standpoint, a reduced processing time standpoint, and/or a reduced energy consumption standpoint.

<NPL>, relates to convolutional neural network implementation deployment on Virtex-<NUM> while varying the activation function such as ReLU, PReLU, and Tanh Exponentional activation functions.

<CIT> relates to a deep neural network architecture using piecewise linear approximation.

The dependent claims define preferred embodiments.

A processing unit according to claim <NUM> is provided.

The hardware approximation of the activation function may be performed in an absence of performing a memory lookup. The lookup table may be used for data aggregation during the hardware approximation of the activation function. The input vector may be part of a matrix. The activation function may be made available on a vector instruction list. The activation function may comprise a sigmoid function. The activation function may comprise a tanh(x) function. The activation function may comprise a non-linear function. The input vector may comprise N elements, wherein each of the N elements are processed during a single clock cycle.

A system according to claim <NUM> is provided.

The processing unit may comprise a Central Processing Unit. The processing unit may comprise a Graphics Processing Unit. The processing unit may comprise a Data Processing Unit.

The activation function may comprise a non-linear function. The activation function may comprise at least one of a sigmoid function and a tanh(x) function.

A device according to claim <NUM> is provided.

The activation function may comprise at least one of a sigmoid function and a tanh(x) function.

Referring now to <FIG>, additional details of a DPU <NUM> will be described in accordance with at least some embodiments. The DPU <NUM> is shown to provide processing capabilities that include a Network Interface Controller (NIC) subsystem <NUM> and a processor cores subsystem <NUM>. The NIC subsystem <NUM> and processor cores subsystem <NUM> are shown to be connectable through a PCIe switch <NUM>. While the DPU <NUM> is shown to include a NIC subsystem <NUM> and processor cores subsystem <NUM>, it should be appreciated that the DPU <NUM> can provide other processor functions or types including, without limitation, CPU processors, GPU processors, and/or any other suitable type of processing architecture.

The processor cores subsystem <NUM> may be configured to provide general processing capabilities and may include a processing complex <NUM>, one or more acceleration engines <NUM>, and one or more network interfaces <NUM>. The processing complex <NUM> may include one or multiple processing cores (e.g., Advanced RISC Machine (ARM) processing cores, RISCV cores, CPU cores, GPU cores, etc.). As will be discussed in further detail herein, one or more processing cores of the processing complex <NUM> may include programmable cores <NUM> and/or approximation circuitry <NUM> as shown in the NIC subsystem <NUM>; however, such components are not shown for ease of reference and discussion.

The acceleration engine(s) <NUM> may provide hardware acceleration capabilities for the processors in the processing complex <NUM> and/or for external GPU(s) <NUM>. As an example, a processing core in the processing complex <NUM> may use one or more acceleration engines <NUM> to perform a specific function whereas other undefined functions may be performed within the processing core of the processing complex <NUM>. The acceleration engine(s) <NUM> can be appropriately configured to perform specified functions more quickly, with fewer computations, etc. as compared to other components of the processing complex <NUM>.

The network interface(s) <NUM> may provide connectivity between components of the processor cores subsystem <NUM> and other components external to the processor cores subsystem <NUM>. Illustratively, the network interface(s) <NUM> may provide connectivity to the PCIe switch <NUM> and/or one or more other external elements, such as an external network <NUM>, a DDR <NUM>, an SSD <NUM>, and/or a GPU <NUM>.

The network interface(s) <NUM> may include physical, mechanical, optical, and/or electrical components that allow a remote device to communicate with the processing complex <NUM> and/or acceleration engine(s) <NUM> of the processor cores subsystem <NUM>. The network interface(s) <NUM> may enable physical connections to a cable, wire, fiberoptic, etc. Alternatively or additionally, the network interface(s) <NUM> may facilitate wireless communications, in which they may include one or more antennas, drivers, or the like.

The NIC subsystem <NUM> is illustrated as another element of the DPU <NUM>. It should be appreciated that the components of the NIC subsystem <NUM> and components of the processor cores subsystem <NUM> may be in communication with one another via the PCIe switch <NUM> or by some other communication mechanism. The NIC subsystem <NUM> and processor cores subsystem <NUM> may be provided on a common substrate, motherboard, or silicon. Alternatively, the NIC subsystem <NUM> and processor cores subsystem <NUM> may be provided on totally separate substrates, motherboards, or silicon.

As a non-limiting example, the NIC subsystem <NUM> may provide functionality similar to a network adapter or other type of networking device. Illustrated components provided in the NIC subsystem <NUM> include, without limitation, a Data Processing Accelerator (DPA) <NUM> and one or more network interfaces <NUM>. The DPA <NUM> may include one or more programmable cores <NUM>, memory <NUM>, a vector instruction list <NUM>, and approximation circuitry <NUM>. While illustrated as separate components, it should be appreciated that certain components of the DPA <NUM> may be combined with one another. For instance, the vector instruction list <NUM> and/or approximation circuitry <NUM> may be included in the one or more programmable cores <NUM>. Alternatively or additionally, the memory <NUM> may be provided external to the DAP <NUM> or may be integrated as part of the programmable core(s) <NUM>.

The programmable core(s) <NUM> may include one or more hardware and/or software components that are programmable and may support one or more functions of the DPU <NUM>. Examples of a suitable programmable core <NUM> include, without limitation, a programmable logic core (PLC), a programmable logic array (PLA), etc. The programmable core(s) <NUM> may be implemented in hardware and/or software on any type of medium. For instance, the programmable core(s) <NUM> may be provided as a programmable SoC, a programmable ASIC, a programmable digital circuit, combinations thereof, or the like. The programmable core(s) <NUM> may be similar or identical to other cores described herein, such as processing cores that were described as being included in the processing complex <NUM>.

The memory <NUM> may correspond to any suitable type of memory device or collection of memory devices already described herein. Non-limiting examples of devices that may be provided as memory <NUM> include RAM, ROM, flash memory, buffer memory, combinations thereof, and the like. In some embodiments, the memory <NUM> may be cache line aligned.

The vector instruction list <NUM> may include one or more instructions (e.g., vector instructions) that are capable of being performed in the programmable core(s) <NUM>. In some embodiments, the vector instruction list <NUM> may provide a listing of functions that can be performed by the approximation circuitry <NUM> or by other components (e.g., programmable core(s) <NUM>, the GPU(s) <NUM>, etc.). In some embodiments, functions (e.g., vector functions) that may be supported by the DPU <NUM> and, thereby, made available in the vector instruction list <NUM> include, without limitation, non-linear functions, linear functions, a hyperbolic tangent function (tanh(x)) function, a sigmoid function, a Rectified Linear Activation (ReLU) function, a softmax function, a softsign function, and an Exponential Linear Unit (ELU) function. Other suitable functions (whether activation functions or not) may also be listed in the vector instruction list. Non-limiting examples of such functions other than an activation function include a multiply add function, a vector accumulate function, a vector add function, a vector multiply function, a vector load function, and a vector store function. One or more of the instructions provided in the vector instruction list <NUM> may be carried out completely in hardware (e.g., using the approximation circuitry <NUM>) and/or may utilize buffer(s) and/or a lookup table as will be described herein. In some embodiments, the approximation circuitry <NUM> may be configured to compute approximations of functions listed in the vector instruction list <NUM> and such approximations may be computed alone or with the assistance of additional components such as a buffer and/or a lookup table.

The network interface <NUM> may be similar or identical to the network interface <NUM> included in the processor cores subsystem <NUM> and may include hardware and/or software components that enable operations of the NIC subsystem <NUM> at the network layer. The network interface <NUM> may also facilitate connectivity to the PCIe switch <NUM>. Examples of protocols that may be supported by the network interface <NUM> include, without limitation, Ethernet, WiFi, Fibre Channel, Asynchronous Transfer Mode (ATM), Fiber Distributed Data Interface (FDDI), RDMA/TCP/UDP, ASAP<NUM>, InfiniBand, etc..

The PCIe switch <NUM> may include hardware and/or software that includes an expansion bus for a PCIe hierarchy on the DPU <NUM>. In some embodiments, the PCIe switch <NUM> may include switching logic that routes packets between one or more ports of the PCIe switch <NUM>. The PCIe switch <NUM> may include two or more different ports that are included as or that are connected to the network interface(s) <NUM> of the NIC subsystem <NUM> and processor cores subsystem <NUM>.

With reference now to <FIG>, additional details of an illustrative instruction <NUM> format will be described in accordance with at least some embodiments of the present disclosure. An instruction <NUM> that can be processed within the DPU <NUM> (or any other processor described herein) may include a sign bit (S) in a sign bit field <NUM>, one or more exponent bits (E) in an exponent bit field <NUM>, and one or more mantissa bits (M) in a mantissa bit field <NUM>. The instruction <NUM> may be provided in any suitable instruction format. Non-limiting examples of instruction formats that may be used for the instruction <NUM> include float32, float16, bfloat16, etc. As a more specific but non-limiting example, the instruction <NUM> may include one sign bit field <NUM>, eight exponent bit fields <NUM>, and at least seven mantissa bit fields <NUM>. Alternatively or additionally, the instruction may include a greater or lesser number of exponent bit fields <NUM> and/or a greater or lesser number of mantissa bit fields <NUM>.

The DPU <NUM> and, specifically the approximation circuitry <NUM>, may be configured to receive input vectors in the form of or including the instruction <NUM>. Said another way, the DPU <NUM> may be configured to operate on instructions <NUM> and/or vectors having the format depicted in <FIG>. Alternatively or additionally, the DPU <NUM>, the programmable core(s) <NUM>, and/or approximation circuitry <NUM> may be configured to process a portion of an instruction <NUM>. In some embodiments, the instruction <NUM> may be provided as a <NUM>-bit vector (e.g., with a total of <NUM> bit fields).

The instruction <NUM> can be provided as an operation code (Opcode), or any other suitable form or format. Illustratively, the instruction <NUM> or input processed by the programmable core(s) <NUM> and/or approximation circuitry <NUM> may include a machine language instruction or portion of machine language instruction that specifies an operation to be performed by the circuitry. In some embodiments, the operation specified by the instruction includes a multiplication operation, vector activation function, non-linear function, etc..

Referring now to <FIG>, an example of a matrix multiplication approach that may be implemented by the programmable core(s) <NUM> and/or approximation circuitry <NUM> in connection with performing an activation function will be described. As discussed above, machine learning inference may sometimes involve multiplying a matrix with a vector. Sometimes, the matrix holds fixed weights while the vector content is variable. When using vector single instruction, multiple data (SIMD) instructions for matrix multiplication, there are a number of possible approaches that can be taken. One example of a matrix multiplication approach is depicted in <FIG>, but it should be appreciated that other approaches are possible.

In the example of <FIG>, a matrix <NUM> is multiplied with vector <NUM> to obtain a resultant vector <NUM>. The matrix <NUM> is shown to be organized in memory by columns <NUM>, but it should be appreciated that the matrix <NUM> can alternatively be organized by rows without departing from the scope of the present disclosure. The illustrative, but non-limiting, matrix <NUM> of <FIG> is shown to include <NUM> rows and <NUM> columns, forming a <NUM> x <NUM> matrix <NUM>.

In this approach, the matrix <NUM> is organized in memory by columns <NUM>. For each column <NUM> the following process is followed. A scalar value is extracted from an input vector <NUM>. The extracted scalar value is vector multiplied with each of the elements in the corresponding/associated column <NUM>. For example, if the scalar value is extracted from the first element (e.g., the top-most element) of the input vector <NUM>, then the extracted scalar value is multiplied with each of the elements in the first column <NUM> (e.g., the left-most column). If the scalar value is extracted from the second element (e.g., the second-top-most element) of the input vector <NUM>, then the extracted scalar value is multiplied with each of the elements in the second column <NUM> (e.g., the second-left-most column).

The results of the vector multiplication described above (e.g., the multiplication of the extracted scalar value from the input vector <NUM> with each element of the corresponding/associated column <NUM>) are then added together. In some embodiments, the results of the vector multiplication are added element-by-element to a vector accumulator. After each column <NUM> has been multiplied by the elements of the input vector <NUM>, the output vector <NUM> is constructed with each element of the output vector <NUM> having the results of each element-by-element addition.

Referring now to <FIG>, two illustrative configurations of circuitry <NUM> and <NUM> that may be used as part of the approximation circuitry <NUM> will be described. The configuration of elements depicted and described in <FIG> are intended to provide examples and should not be construed as limiting the claims to any particular configuration of circuit elements.

The circuitry <NUM> of <FIG> is illustrated to include a number of adders. In some embodiments, the circuitry <NUM> may facilitate a vector add instruction with a vector size of eight elements. <FIG> illustrates circuitry <NUM> that may facilitate a vector tanh function with a vector size of eight elements. It should be appreciated that either circuitry <NUM>, <NUM> may be adjusted to accommodate larger or smaller vectors. The illustrated types of circuitry <NUM>, <NUM> may be configured to obtain <NUM> operations per clock cycle, meaning that each clock cycle will output <NUM> operations. Circuitry <NUM>, <NUM> may be expanded to support more operations per clock cycle or contracted to support fewer operations per clock cycle. In some embodiments, the circuitry <NUM>, <NUM> may be provided as part of the approximation circuitry <NUM> to support performance of vector activation functions defined in the vector instruction list <NUM>. Because all operations can be performed on the data using the approximation circuitry <NUM>, it becomes possible to perform vector activation functions without requiring access to separate memory <NUM>, thereby increasing the speed of processing.

With reference now to <FIG>, additional details of an illustrative approximation circuitry <NUM> will be described in accordance with at least some embodiments of the present disclosure. Approximation circuitry <NUM> is configured to include one or more input circuits <NUM>, one or more approximation circuits <NUM>, one or more output circuits <NUM>, and a lookup table <NUM>. While the components of the approximation circuitry <NUM> are shown as being separate or distinct components, it should be appreciated that the input circuit(s) <NUM>, approximation circuit(s) <NUM>, and/or output circuit(s) <NUM> may be combined into a single circuit or may be divided into more circuits than depicted.

Even though the lookup table <NUM> is shown as being included in the approximation circuitry <NUM>, it should be appreciated that the lookup table <NUM> may be outside and separate from the approximation circuitry <NUM>. The lookup table <NUM> may provide information that supports vector aggregation. Alternatively or additionally, the lookup table <NUM> may correspond to a hardware-based lookup table. The lookup table <NUM> identifies particular circuits to use within the approximation circuitry <NUM> when processing a particular instruction (e.g., when approximating a specific function). As an example, the lookup table <NUM> may identify first circuitry to use when approximating a first activation function and the lookup table <NUM> may identify second circuitry to use when approximating a second activation function.

In some embodiments, the approximation circuitry <NUM> receives an input vector <NUM> at the input circuit(s) <NUM> and then provide the input vector to the approximation circuit(s) <NUM>. The approximation circuit(s) <NUM> then applies an activation function to the input vector by performing a hardware approximation of the activation function in a vector manner. The output circuit(s) <NUM> then generates an output vector <NUM> based on the activation function.

The lookup table <NUM> is referenced during the hardware approximation of the activation function. Moreover, the lookup table <NUM> may be used to reference associated hardware to use during the approximation of the activation function (e.g., different hardware and/or circuitry may be used to perform the approximation of different activation functions). In some embodiments, the hardware approximation of the activation function may be performed in the absence of performing a memory <NUM> lookup.

Operations of processes described herein can be performed in any suitable order unless otherwise indicated herein or otherwise clearly contradicted by context. In at least one embodiment, a process such as those processes described herein (or variations and/or combinations thereof) is performed under control of one or more computer systems configured with executable instructions and is implemented as code (e.g., executable instructions, one or more computer programs or one or more applications) executing collectively on one or more processors, by hardware or combinations thereof. In at least one embodiment, code is stored on a computer-readable storage medium, for example, in form of a computer program comprising a plurality of instructions executable by one or more processors. In at least one embodiment, a computer-readable storage medium is a non-transitory computer-readable storage medium that excludes transitory signals (e.g., a propagating transient electric or electromagnetic transmission) but includes non-transitory data storage circuitry (e.g., buffers, cache, and queues) within transceivers of transitory signals. In at least one embodiment, code (e.g., executable code or source code) is stored on a set of one or more non-transitory computer-readable storage media having stored thereon executable instructions (or other memory to store executable instructions) that, when executed (i.e., as a result of being executed) by one or more processors of a computer system, cause computer system to perform operations described herein. In at least one embodiment, set of non-transitory computer-readable storage media comprises multiple non-transitory computer-readable storage media and one or more of individual non-transitory storage media of multiple non-transitory computer-readable storage media lack all of code while multiple non-transitory computer-readable storage media collectively store all of code. In at least one embodiment, executable instructions are executed such that different instructions are executed by different processors - for example, a non-transitory computer-readable storage medium store instructions and a main central processing unit ("CPU") executes some of instructions while a graphics processing unit ("GPU") executes other instructions. In at least one embodiment, different components of a computer system have separate processors and different processors execute different subsets of instructions.

Accordingly, in at least one embodiment, computer systems are configured to implement one or more services that singly or collectively perform operations of processes described herein and such computer systems are configured with applicable hardware and/or software that enable performance of operations. Further, a computer system that implements at least one embodiment of present disclosure is a single device and, in another embodiment, is a distributed computer system comprising multiple devices that operate differently such that distributed computer system performs operations described herein and such that a single device does not perform all operations.

In a similar manner, term "processor" or "processing unit" may refer to any device or portion of a device that processes electronic data from registers and/or memory and transform that electronic data into other electronic data that may be stored in registers and/or memory. As non-limiting examples, "processor" may be a CPU or a GPU. A "computing platform" may comprise one or more processors or processing units. As used herein, "software" processes may include, for example, software and/or hardware entities that perform work over time, such as tasks, threads, and intelligent agents. Also, each process may refer to multiple processes, for carrying out instructions in sequence or in parallel, continuously or intermittently. In at least one embodiment, terms "system" and "method" are used herein interchangeably insofar as system may embody one or more methods and methods may be considered a system.

Claim 1:
A processing unit configured to execute instructions, comprising:
circuitry (<NUM>) that, in response to an instruction, receives an input vector (<NUM>), applies an activation function to the input vector by performing a hardware approximation of the activation function, and then generates an output vector (<NUM>) based on the activation function; and
a lookup table (<NUM>) that is referenced during the hardware approximation of the activation function, wherein the lookup table identifies particular circuits of the circuitry to use when processing a particular activation function.