Patent ID: 12190160

5. DETAILED DESCRIPTION OF THE DRAWINGS

In the following detailed description, numerous specific details are set forth in order to provide a thorough understanding of the invention. However, it will be understood by the person having ordinary skill in the art that the invention may be practised without these specific details. In other instances, well known methods, procedures and/or components have not been described in detail so as not to obscure the invention.

The invention will be more clearly understood from the following description of the embodiments thereof, given by way of example only with reference to the accompanying drawings, which are not drawn to scale.

As backbone AI inference software stack is running on the embedded processor, hardware acceleration based on custom instruction approach allows a seamless integration between the software stack and hardware accelerators. The layer accelerator architecture of the present invention makes use of custom instructions and direct memory access (DMA).

In general, an instruction set architecture (ISA) defines the instructions that are supported by a processor. There are ISAs for certain processor variants that include custom instruction support, where specific instruction opcodes are reserved for custom instruction implementations. This allows developers or users to implement their own customized instruction based on targeted applications. Differing from ASIC chip where the implemented custom instruction(s) are to be fixed at development time, custom instruction implementation using an FPGA101is configurable/programmable by users for different applications using the same FPGA chip.FIG.1illustrates the block diagram of an embedded processor102in said FPGA101with custom instruction support connected to a neural network accelerator103through custom instruction interface, whereby said neural network accelerator103is with Advanced extensible Interface (AXI) master interface for connectivity with at least one DMA and implementation of said DMA.

As shown inFIG.1, an example of custom instruction interface comprises of mainly two groups of signals: input related signals and output related signals. The input related signals are “command_valid” signal and “command_ready” signal that are used to indicate the validity of the “input0” signal, the “input1” signal, and the “function_id” signal. The output related signals are “response_valid” signal and the “response_ready” signal that are used to indicate the validity of the “output” signal. With a single/one custom instruction opcode and M-bit of “function_id” signal, a total of 2Mcustom instructions can be implemented.FIG.1shows an example of an embedded processor102based on the VexRiscv CPU architecture with custom instructions support, whereby funct7, rs2, rs1, funct3 and rd are of the R-type RISC-V base instruction format used for custom instructions. Register file105, arithmetic logic unit (ALU)107, pipeline control109, and custom instruction plugin111are part of this CPU architecture of the embedded processor102. Even through RISC-V is shown as an example of the embedded processor102to be used in the present invention, any other suitable embedded processors102can be used for the purpose of the present invention. On the other hand, a AXI master interface comprises of signals for write channel address and control, write channel data, write channel response, read channel address and control, read channel data and other valid and ready signals.

This invention focuses on the architecture of neural network layer accelerator, which is based on custom instruction interface and AXI master interface for DMA purposes. The proposed neural network architecture is depicted inFIG.2. As shown inFIG.2, the architecture of the neural network accelerator103in an FPGA of the present invention comprises of a command control block301, at least one neural network layer accelerator303, a response control block305and an AXI control block307. As shown inFIG.2, the single or one custom instruction interface and AXI master interface is shared among the layer accelerators303which are available in said neural network accelerator103for scalability, flexibility and efficient resource utilization. With the sharing of a single or one custom instruction interface and AXI master interface, the number of neural network layer accelerators that can be implemented can be configured easily in said FPGA, which is highly flexible and scalable. Users, designers or developers can enable only the specific types of layer accelerator303that are required for a targeted neural network application, instead of enabling all the available layer accelerators303, thereby providing efficient resource utilization. Furthermore, more layer accelerators303for different neural network layers or operations can be easily added to the proposed neural network accelerator103, due to the scalable architecture of said neural network accelerator103.

In addition, with the AXI master interface for DMA, the neural network layer architecture of the present invention can obtain higher memory throughput (compared to using custom instruction interface for data access), thus achieving faster layer accelerator speed-up. Note that, with the neural network accelerator architecture shown inFIG.2, it is also possible for said neural network accelerator103to be configured to have both the layer accelerator only based on custom instructions and layer accelerator based on custom instructions and AXI master interface; based on the needs of the applications. Depending on the needs or requirement of the application or target AI model used, if the AXI interface is not utilized, less logic resources are utilized in said neural network accelerator103, but with less acceleration performance. If the AXI interface is utilized for data access, more logic resources are utilized in said neural network accelerator103, but with more acceleration performance. This flexibility to trade-off between speed and resources utilization is another advantage of the present invention.

The layer accelerator303is implemented based on layer type e.g., convolution layer, depthwise convolution layer, and fully connected layer, which can be reused by a neural network model that comprises of a plurality of layers of the same type. Not all targeted AI models would require all the layer accelerators to be implemented. The present invention allows configuration at compile time for individual layer accelerator enablement for efficient resource utilization. Each layer accelerator has its own set of “command_valid” signal, “command_ready” signal, “response_valid” signal, “output” signals and AXI master signals.

The command control block301is used for sharing the “command_ready” signals from said plurality of layer accelerators303and “command_valid” signals to said plurality of layer accelerators303by using the “function_id” signal for differentiation. The command control block301receives said “function_id” signal from said embedded processor102while become intermediary for transferring of “command_valid” signal from said embedded processor102to said neural network layer accelerator303and transferring of “command_ready” signal from said neural network layer accelerator303to said embedded processor102. The M-bit “function_id” signal can be partitioned to multiple function ID blocks, whereby one function ID block is allocated specifically for one layer accelerator303. One example method of partitioning the “function_id” signal is based on the one of more most significant bit (MSB) bit(s) of the “function_id” signal, depending on how many function ID blocks are desired. For example, if the partition is based on 1-bit MSB, two blocks will be created, while if the partition is based on 2-bit MSB, four blocks will be created. In general, partitioning N-bits of MSB creates 2Nblocks. The command control block301refers to the MSB bit(s) of said “function_id” signal to identify the specific layer accelerator303that is associated with the specific incoming custom instruction command.

The response control block305becomes intermediary by means of multiplexing for transferring of the “response_valid” signal and the “output” signal from each neural network layer accelerator303to one “response_valid” signal and one “output” signal of the custom instruction interface to said embedded processor102. As neural network inference typically executes the model layer-by-layer, only one layer accelerator303would be active at a time. In this case, straightforward multiplexing can be used in the response control block305.

The layer accelerator303receives said “input0” signal, “input1” signal, said “response_ready” signal and said “function_id” signal from said embedded processor102; receives “command_valid” signal from said embedded processor102through said command control block301; transmits “command_ready” signal to said embedded processor102through said command control block301; transmits “response_valid” signal and “output” signal to said embedded processor102through said response control block305.

The AXI control block307in said neural network accelerator103manages the sharing of AXI master interface across said neural network layer accelerators303through multiplexing, AXI interconnect module or any other suitable signal management methodology. The AXI control block307is used for sharing of a AXI master interface across multiple layer accelerators. As neural network inference typically executes the model layer-by-layer, only one layer accelerator would be active at a time. Either AXI interconnect module or straightforward multiplexing can be applied in the AXI control block307. In essence, an arbiter is needed to share the AXI master interface so that said neural network accelerator103can support multiple masters and slaves.

The proposed neural network accelerator103makes use of the AXI master interface for communication between said neural network accelerator103and said DMA in tasks such as input data retrieval, output data storage, etc.; while using the custom instruction interface for passing individual layer accelerator's303parameters, and related controls signals such as those for triggering the computation in the layer accelerator303, reset certain block(s) in the neural network accelerator103, etc. In each individual layer accelerator block, a specific set of custom instructions are created to transfer said layer accelerator's303parameters and control signals, by utilizing the allocated function IDs for said respective layer accelerator303type accordingly. Note that, for the layer accelerator architecture of the present invention, the designer may opt for implementing custom instructions to speed-up only certain compute-intensive computations in a neural network layer or implementing a complete layer operation into the layer accelerator, with consideration of design complexity and achievable speed-up. The data/signal transfer between the neural network accelerator103and said embedded processor102are controlled by said embedded processor's102modified firmware/software, which may be within an AI inference software stack or a standalone AI inference implementation. Note that, for the layer accelerator architecture of the present invention, it is typically more efficient to implement a complete layer operation into the layer accelerator to effectively utilize the achievable DMA throughput.FIG.4is a waveform showing an example of the process of the custom instruction interfaces used in the VexRISC-V CPU architecture.

FIG.3illustrates the block diagram of a general layer accelerator of the present invention. As shownFIG.3, a general layer accelerator303of the present invention comprises of a control unit401, a compute unit405, a data buffer403and a DMA control block407. Note that, the layer accelerator303design may vary based on targeted layer type. The control unit401interprets at least one custom instruction input of said custom instruction interface based on the respective function ID to differentiate whether they are the layer parameters, input data to be stored in data buffer403for subsequent computation, input data to be used directly for computation, or control signals and so on. Layer parameter information are to be retained until the completion of a layer execution to facilitate the related control for data storage and retrieval to/from data buffer403, computations, etc. On the other hand, the compute unit405performs at least one operation, computation or combination thereof, required by at least one targeted layer type of said neural network accelerator103.

For layer accelerator303type that requires more than one set of inputs simultaneously for efficient parallel computations by the compute unit405, the data buffer403can be used to hold the data from said custom instruction input while waiting for the arrival of the other set(s) of input data to start the computations. Also, data buffer403can be used to store data from said custom instruction input that is highly reused in the layer operation computations. The control unit401facilitates transfer of computation output from said compute unit405to said response control block305.

The role of DMA control block407is to facilitate input and output data access for the layer accelerator303computations through AXI master interface. The addresses of input and output data arrays are obtained through the custom instruction, and control unit401generates the corresponding triggers to the DMA control block407based on the targeted layer operation.

While the present invention has been shown and described herein in what are considered to be the preferred embodiments thereof, illustrating the results and advantages over the prior art obtained through the present invention, the invention is not limited to those specific embodiments. Thus, the forms of the invention shown and described herein are to be taken as illustrative only and other embodiments may be selected without departing from the scope of the present invention, as set forth in the claims appended hereto.