Apparatus and methods for vector operations

Aspects for vector operations in neural network are described herein. The aspects may include a vector caching unit configured to store a vector, wherein the vector includes one or more elements. The aspects may further include a computation module that includes one or more comparers configured to compare the one or more elements to generate an output result that satisfies a predetermined condition included in an instruction.

BACKGROUND

Multilayer neural networks (MNN) are widely applied to the fields such as pattern recognition, image processing, functional approximation and optimal computation. In recent years, due to the higher recognition accuracy and better parallelizability, multilayer artificial neural networks have received increasing attention by academic and industrial communities. More specifically, logical operations for vectors may be performed frequently in deep learning processes in MMNs.

A known method to perform logical operations for vectors in a multilayer artificial neural network is to use a general-purpose processor. However, one of the defects of the method is low performance of a single general-purpose processor which cannot meet performance requirements for usual multilayer neural network operations with respect to a vector with a large number of elements.

Another known method to perform logical operations for vectors of the multilayer artificial neural network is to use a graphics processing unit (GPU). Such a method uses a general-purpose register file and a general-purpose stream processing unit to execute general purpose single-instruction-multiple-data (SIMD) instructions to support the algorithms in MNNs. However, since GPU only contains rather small on-chip caching, then data of the vector elements may be repeatedly moved from the off-chip, and off-chip bandwidth becomes a main performance bottleneck, causing huge power consumption.

SUMMARY

One example aspect of the present disclosure provides an example apparatus for vector operations in a neural network. The example apparatus may include a vector caching unit configured to store a vector, wherein the vector includes one or more elements. The example apparatus may further include a computation module that includes one or more comparers configured to compare the one or more elements to generate an output result that satisfies a predetermined condition included in an instruction.

The example aspect of the present disclosure also provides an example method for vector operations in a neural network. The example method may include storing, by a vector caching unit, a vector, wherein the vector includes one or more elements; and comparing, by one or more comparers of a computation module, the one or more elements to generate an output result that satisfies a predetermined condition included in an instruction.

DETAILED DESCRIPTION

In the present disclosure, the term “comprising” and “including” as well as their derivatives mean to contain rather than limit; the term “or”, which is also inclusive, means and/or.

In this specification, the following various embodiments used to illustrate principles of the present disclosure are only for illustrative purpose, and thus should not be understood as limiting the scope of the present disclosure by any means. The following description taken in conjunction with the accompanying drawings is to facilitate a thorough understanding to the illustrative embodiments of the present disclosure defined by the claims and its equivalent. There are specific details in the following description to facilitate understanding. However, these details are only for illustrative purpose. Therefore, persons skilled in the art should understand that various alternation and modification may be made to the embodiments illustrated in this description without going beyond the scope and spirit of the present disclosure. In addition, for clear and concise purpose, some known functionality and structure are not described. Besides, identical reference numbers refer to identical function and operation throughout the accompanying drawings.

A vector may refer to one or more values formatted in a one-dimensional data structure. The values included in a vector may be referred to as elements. The number of the elements in the vector may be referred to as a length of the vector. Various types of vector operations may be performed in a neural network. For example, the vector operations may include a logical MAX operation to identify a maximum value of the elements and a logical MIN operation to identify a minimum value of the elements.

FIG. 1illustrates a block diagram of an example neural network acceleration processor by which vector operations may be implemented in a neural network. As depicted, the example neural network acceleration processor100may include an instruction caching unit104, a controller unit106, a direct memory access unit102, a computation module110, and a vector caching unit112. Any of the above-mentioned components or devices may be implemented by a hardware circuit (e.g., application specific integrated circuit (ASIC), Coarse-grained reconfigurable architectures (CGRAs), field-programmable gate arrays (FPGAs), analog circuits, memristor, etc.).

In some examples, a vector operation instruction may originate from an instruction storage device134to the controller unit106. An instruction obtaining module132may be configured to obtain a vector operation instruction from the instruction storage device134and transmit the instruction to a decoding module130.

The decoding module130may be configured to decode the instruction. The instruction may include one or more operation fields that indicate parameters for executing the instruction. The parameters may refer to identification numbers of different registers (“register ID” hereinafter) in the instruction register126. Thus, by modifying the parameters in the instruction register126, the neural network acceleration processor100may modify the instruction without receiving new instructions. The decoded instruction may be transmitted by the decoding module130to an instruction queue module128. In some other examples, the one or more operation fields may store immediate values such as addresses in the memory101and a scalar value, rather than the register IDs.

The instruction queue module128may be configured to temporarily store the received instruction and/or one or more previously received instructions. Further, the instruction queue module128may be configured to retrieve information according to the register IDs included in the instruction from the instruction register126.

For example, the instruction queue module128may be configured to retrieve information corresponding to operation fields in the instruction from the instruction register126. Information for the operation fields in a VMAX instruction may include an address of a vector and a length of the vector. As depicted, in some examples, the instruction register126may be implemented by one or more registers external to the controller unit106. Once the relevant values are retrieved, the instruction may be sent to a dependency processing unit124.

The dependency processing unit124may be configured to determine whether the instruction has a dependency relationship with the data of the previous instruction that is being executed. This instruction may be stored in the storage queue module122until it has no dependency relationship on the data with the previous instruction that has not finished executing. If the dependency relationship does not exist, the controller unit106may be configured to decode one of the instructions into micro-instructions for controlling operations of other modules including the direct memory access unit102and the computation module110.

For example, the controller unit106may receive a vector-maximum (VMAX) instruction that includes a starting address of a vector, a length of the vector, and an address for an output result. According to the VMAX instruction, the direct memory access unit102may be configured to retrieve the vector from an external storage device, e.g., a memory101, according to the starting address in the VMAX instruction. The retrieved vector may be transmitted to and stored in the vector caching unit112.

In some other examples, the controller unit106may receive a vector-minimum (VMIN) instruction that includes a starting address of a vector, a length of the vector, and an address for an output result. According to the VMIN instruction, the direct memory access unit102may be configured to retrieve the vector from an external storage device, e.g., a memory101according to the starting address in the VMIN instruction. The retrieved vector may be transmitted to and stored in the vector caching unit112.

The above mentioned instructions may be formatted as follows and may be stored in the instruction caching unit104:

Register 0Register 1Register 2VMAXStarting address of a vectorLength of the vectorOutput resultVMINStarting address of a vectorLength of the vectorOutput result

Hereinafter, a caching unit (e.g., the vector caching unit112etc.) may refer to an on-chip caching unit integrated in the neural network acceleration processor100, rather than other storage devices in memory101or other external devices. In some examples, the on-chip caching unit may be implemented as a register file, an on-chip buffer, an on-chip Static Random Access Memory (SRAM), or other types of on-chip storage devices that may provide higher access speed than the external memory. In some other examples, the instruction register126may be implemented as a scratchpad memory, e.g., Dynamic random-access memory (DRAM), embedded DRAM (eDRAM), memristor, 3D-DRAM, non-volatile memory, etc.

FIG. 2illustrates an example logical operation process that may be performed by the example neural network acceleration processor.

As depicted, the computation module110may be configured to perform logical operation to a vector (“Vector A”). The vector may include one or more elements respectively denoted as A(1), A(2), . . . A(n).

The computation module110may include one or more comparers. In response to a VMAX instruction, the one or more comparers may be configured to compare the elements, e.g., A(1), A(2), . . . A(n), to select a maximum value from the elements. The selected maximum value may be designated as an output result.

In response to a VMIN instruction, the one or more comparers may be configured to compare the elements, e.g., A(1), A(2), . . . A(n), to select a minimum value from the elements. The selected minimum value may be designated as an output result.

FIG. 3illustrates an example computation module110in the example neural network acceleration processor by which vector operations may be implemented in a neural network.

As depicted, the computation module110may include a computation unit302, a data dependency relationship determination unit304, a neuron caching unit306. The computation unit302may further include one or more comparers310.

The data dependency relationship determination unit304may be configured to perform data access operations (e.g., reading or writing operations) on the caching units including the neuron caching unit306during the computation process. The data dependency relationship determination unit304may be configured to prevent conflicts in reading and writing of the data in the caching units. For example, the data dependency relationship determination unit304may be configured to determine whether there is dependency relationship in terms of data between a micro-instruction which to be executed and a micro-instruction being executed. If no dependency relationship exists, the micro-instruction may be allowed to be executed; otherwise, the micro-instruction may not be allowed to be executed until all micro-instructions on which it depends have been executed completely. The dependency relationship may be determined when a target operation range of the micro-instruction to be executed overlaps a target operation range of a micro-instruction being executed. For example, all micro-instructions sent to the data dependency relationship determination unit304may be stored in an instruction queue within the data dependency relationship determination unit304. The instruction queue may indicate the relative priorities of the stored micro-instructions. In the instruction queue, if the target operation range of reading data by a reading instruction conflicts with or overlaps the target operation range of writing data by a writing instruction of higher priority in the front of the instruction queue, then the reading instruction may not be executed until the writing instruction is executed.

The neuron caching unit306may be configured to store the elements in the vector.

The computation unit302may be configured to receive the micro-instructions decoded from the vector operation instruction from the controller unit106. In the example that the computation unit302receives micro-instructions decoded from a VMAX instruction, the one or more comparers310may be configured to compare the elements in the vector, e.g., A(1), A(2), . . . A(n), and to select a maximum value from the elements. In response to a VMIN instruction, the one or more comparers may be configured to compare the elements, e.g., A(1), A(2), . . . A(n), to select a minimum value from the elements. The selected minimum value may be designated as an output result.

FIG. 4illustrates a flow chart of an example method400for performing logical operations between two vectors in a neural network. The method400may be performed by one or more components the apparatus ofFIGS. 1 and 3. Optional or alternative operations may be shown in dash-lined blocks.

At block401, the example method400may include receiving, by a controller unit, a vector operation instruction that includes an address of a vector and a predetermined condition. For example, the controller unit106may be configured to receive a VMAX instruction or a VMIN instruction that includes the address of the vector. The predetermined condition may indicate whether a minimum value or a maximum value of the elements should be selected.

At block402, the example method400may include receiving, by a computation module, the vector based on the address included in the vector operation instruction, wherein the vector includes one or more elements. For example, the computation module110may receive the vector that includes the elements, e.g., A(1), A(2), . . . A(n).

At block404, the example method400may include comparing, by one or more comparers of the computation module, the one or more elements to generate an output result that satisfies the predetermined condition included in the vector operation instruction. For example, the one or more comparers310may be configured to compare the elements and select an element that satisfies a predetermined condition in a received instruction.

At block406, the example method400may optionally include selecting, by the one or more comparers, a maximum element from the one or more elements as the output result. In the example that the computation unit302receives micro-instructions decoded from a VMAX instruction, the predetermined condition in the VMAX instruction may refer to a maximum value among multiple values. The one or more comparers310may be configured to compare the elements in the vector, e.g., A(1), A(2), . . . A(n), and to select a maximum value from the elements as an output result.

At block408, the example method400may optionally include selecting, by the one or more comparers, a minimum element from the one or more elements as the output result. In the example that the computation unit302receives micro-instructions decoded from a VMIN instruction, the predetermined condition in the VMIN instruction may refer to a minimum value among multiple values. The one or more comparers310may be configured to compare the elements in the vector, e.g., A(1), A(2), . . . A(n), and to select a minimum value from the elements as an output result.

The process or method described in the above accompanying figures can be performed by process logic including hardware (for example, circuit, specific logic etc.), firmware, software (for example, a software being externalized in non-transitory computer-readable medium), or the combination of the above two. Although the process or method is described above in a certain order, it should be understood that some operations described may also be performed in different orders. In addition, some operations may be executed concurrently rather than in order.

In the above description, each embodiment of the present disclosure is illustrated with reference to certain illustrative embodiments. Apparently, various modifications may be made to each embodiment without going beyond the wider spirit and scope of the present disclosure presented by the affiliated claims. Correspondingly, the description and accompanying figures should be understood as illustration only rather than limitation. It is understood that the specific order or hierarchy of steps in the processes disclosed is an illustration of exemplary approaches. Based upon design preferences, it is understood that the specific order or hierarchy of steps in the processes may be rearranged. Further, some steps may be combined or omitted. The accompanying method claims present elements of the various steps in a sample order, and are not meant to be limited to the specific order or hierarchy presented.