Patent Description:
The present subject matter described herein, in general, relates to a reconfigurable convolution engine and more particularly to performing a convolution operation on an image by using the reconfigurable convolution engine.

In recent times, Convolution Neural Network (CNN) technique is finding greater applications in computer vision. The computer vision is used to detect a scene or an object in real time from an image captured in various systems. Example of the various systems include, but not limited to, pedestrian detection, lane detection, autonomous driving, sign board detection, activity detection, and face recognition. In order to detect the object in real time, complex computations need to be performed. However, there is a limit on computation power of any system. This is because the hardware capabilities of any system cannot be extended in real time. In other words, the computation power is based on one or more available on-chip resources of the Field Programmable Gate Arrays (FPGA) and Application Specific Integrated Circuits (ASIC). Thus, the conventional systems and methodologies performs convolution operation only on the available on-chip resources thereby failing to perform convolution operation in real time.

<CIT> describes a method for data management. The method comprises: storing the plurality of items in a contiguous space within the memory, executing an instruction containing an address and a size that together identify the contiguous space to transmit the plurality of items from the main memory to a random-access memory (RAM) on a chip, and the chip includes a computing unit comprising a plurality of multipliers; and instructing the computing unit on the chip to: retrieve multiple of the plurality of items from the RAM; and perform a plurality of parallel operations using the plurality of multipliers with the multiple items to yield output data.

<CIT> describes a reconfigurable stream switch (<NUM>) formed in an integrated circuit. The stream switch includes a plurality of output ports (<NUM>), a plurality of input ports (<NUM>), and a plurality of selection circuits. The output ports (<NUM>) each have an output port architectural composition, and each is arranged to unidirectionally pass output data and output control information. The input ports (<NUM>) each have an input port architectural composition, and each is arranged to unidirectionally receive first input data and first input control information. Each one of the selection circuits is coupled to an associated one of the output ports (<NUM>). Each selection circuit is further coupled to all of the input ports (<NUM>) such that each selection circuit is arranged to reconfigurably couple its associated output port (<NUM>) to no more than one input port (<NUM>) at any given time.

The present invention comprises a method, a reconfigurable convolution engine and a non-transitory computer readable medium as defined in the claims. Examples that do not fall within the scope of the claims are to be interpreted as examples useful for understanding the invention. Before the present systems and methods, are described, it is to be understood that this application is not limited to the particular systems, and methodologies described, as there can be multiple possible embodiments which are not expressly illustrated in the present disclosure. It is also to be understood that the terminology used in the description is for the purpose of describing the particular versions or embodiments only, and is not intended to limit the scope of the present application, which is solely defined by the claims.

This summary is provided to introduce concepts related to systems and methods for performing a convolution operation on an image using a reconfigurable convolution engine and the concepts are further described below in the detailed description. This summary is not intended to identify essential features of the claimed subject matter.

In one implementation, a method for performing a convolution operation on an image using a reconfigurable convolution engine, in accordance with independent claim <NUM>, is disclosed.

In another implementation, a reconfigurable convolution engine for performing a convolution operation on an image, in accordance with independent claim <NUM>, is disclosed.

In yet another implementation, non-transitory computer readable medium embodying a program, in accordance with independent claim <NUM>, is disclosed.

The foregoing detailed description of embodiments is better understood when read in conjunction with the appended drawings. For the purpose of illustrating the disclosure, example constructions of the disclosure are shown in the present document; however, the disclosure is not limited to the specific methods and apparatus disclosed in the document and the drawings.

The detailed description is given with reference to the accompanying figures. The same numbers are used throughout the drawings to refer like features and components.

Some embodiments of this disclosure, illustrating all its features, will now be discussed in detail. The words "receiving," "determining," "generating," "configuring," "executing," and "filtering," and other forms thereof, are intended to be equivalent in meaning and be open ended in that an item or items following any one of these words is not meant to be an exhaustive listing of such item or items, or meant to be limited to only the listed item or items. Although any systems and methods similar or equivalent to those described herein can be used in the practice or testing of embodiments of the present disclosure, the exemplary, systems and methods are now described. The disclosed embodiments are merely exemplary of the disclosure, which may be embodied in various forms.

Various modifications to the embodiment will be readily apparent to those skilled in the art and the generic principles herein may be applied to other embodiments. However, one of ordinary skill in the art will readily recognize that the present disclosure is not intended to be limited to the embodiments illustrated, but is to be accorded the widest scope consistent with the appended claims.

The present invention performs a convolution operation on an image using a reconfigurable convolution engine. It is to be noted that the reconfigurable convolution engine utilizes already available on-chip resources of at least Field Programmable Gate Arrays (FPGA) and Application Specific Integrated Circuits (ASIC). Example of the on-chip resources include, but not limited to, registers, Digital Signal Processing (DSP) chips, memory controllers, multipliers, multiplexers, and adders. The reconfigurable convolution engine comprises a (MxM) multiplier array followed by a pipelined adder structure and a configurable filter function. The reconfigurable convolution engine facilitates reuse of the on-chip resources by performing concurrent computations. It is to be noted that the reconfigurable convolution engine adapts various combinations of kernel sizes and strides. Various versions of kernel size include, but not limited to, 3x3, 5x5, 7x7, 9x9, and nxn.

In order to perform the convolution operation on the image, initially, image data pertaining to the image are received from a host processor. The image data may comprise pixel resolution, number of filters to be applied, and comprise a convolution layer. In an implementation, the convolution layer indicates a layer to be used for convolution operation of the image. In another implementation, one convolution layer may comprise different kernel size than another convolution layer. Upon receipt of the image data, kernel size for performing the convolution operation is determined. In an implementation, kernel size may be determined based on the convolution layer. The convolution layer may be configured by the host processor. Subsequent to determining the kernel size, a plurality of instances are generated. The plurality of instances is configured to operate in parallel mode. Each instance of the plurality of instances is configured to perform convolution operation with the clock speed of the convolution engine. Each instance of the plurality of instances comprises M×M blocks of multiplier.

Upon generating the plurality of instances, an adder engine comprising a plurality of adders is configured. The plurality of adders are configured to operate in a pipelined structure and in parallel with the plurality of instances. After configuration, the convolution operation on each of the plurality of instances and the adder engine is executed. Subsequently, an output of the convolution operation is filtered by using the configurable filter function thereby performing the convolution operation on the image using the reconfigurable convolution engine. While aspects of described system and method for performing the convolution operation on the image using the reconfigurable convolution engine and may be implemented in any number of different computing systems, environments, and/or configurations, the embodiments are described in the context of the following exemplary reconfigurable convolution engine.

Referring now to <FIG>, a hardware implementation <NUM> of a reconfigurable convolution engine <NUM> for performing the convolution operation on an image is disclosed. The reconfigurable convolution engine <NUM> comprises a reconfigurable multiplier engine <NUM>, a reconfigurable adder engine <NUM> (hereinafter may also be referred as adder engine), and a Rectification Liner Unit (ReLU) <NUM>. The reconfigurable convolution engine <NUM> may be implemented in a variety of computing systems, such as a laptop computer, a desktop computer, a notebook, a workstation, and a mainframe computer. The reconfigurable convolution engine <NUM> may be configured to utilize on-chip resources of at least one of Field Programmable Gate Arrays (FPGA) and Application Specific Integrated Circuits (ASIC). The on-chip resources may comprise a host processor <NUM>, a host interface <NUM>, memory controller 114a and 114b (collectively referred as memory controller <NUM>), a kernel register space <NUM>, a threshold register space <NUM>, multiplexers 116a and 116b (collectively referred as multiplexer <NUM>), a system controller <NUM>, an input line buffer <NUM>, a data splitter stride <NUM>, and an output line buffer <NUM>.

The host processor <NUM> may be a Central Processing Unit (CPU) installed in at least one of the variety of computing systems. To perform the convolution operation on the image, the image data is received from the host interface <NUM>. The host interface <NUM> may be a bus interface configured to execute a protocol for data transfer between the host processor <NUM> and the convolution engine <NUM>. It is to be understood that a user may interact with the reconfigurable convolution engine <NUM> via the host interface <NUM>. The host interface <NUM> may include a variety of software and hardware interfaces, for example, a web interface, a graphical user interface, and the like. The host interface <NUM> may allow the reconfigurable convolution engine <NUM> to interact with the user directly or through other client devices. Further, the host interface <NUM> may enable the reconfigurable convolution engine <NUM> to communicate with other computing devices, such as web servers and external data servers (not shown). The host interface <NUM> can facilitate multiple communications within a wide variety of networks and protocol types, including wired networks, for example, LAN, cable, etc., and wireless networks, such as WLAN, cellular, or satellite. The host interface <NUM> may include one or more ports for connecting a number of devices to one another or to another server. In an implementation, the host interface <NUM> may facilitate on-chip communication by implementing on-chip bus protocols including, but not limited to, Advanced Microcontroller Bus Architecture (AMBA) Advanced High-Performance bus (AHB) and Advanced Extensible Interface (AXI), Wishbone Bus, Open Core Protocol (OCP) and Core Connect Bus. In other implementation, the host interface <NUM> may facilitate off-chip communication by implementing off-chip bus protocols including, but not limited to, Universal Serial Bus (USB), and High speed interface.

The kernel register space <NUM> may be configured to hold kernel values related to the convolution layer in operation. The threshold register space <NUM> may be configured hold parameters for a filter function. Example of the filter function include, but not limited to, the ReLU, Sigmoid or Logistic, and Hyperbolic tangent function-Tanh. The multiplexer <NUM> may be configured to pass the kernel values to the convolution engine <NUM> for the convolution operation. The input line buffer <NUM> may be configured to hold the image data for the convolution operation. The output line buffer <NUM> may be configured to receive the output of the convolution engine <NUM> and buffer the output before passing to next stage of processing.

The memory controller <NUM> may include any computer-readable medium or computer program product known in the art including, for example, volatile memory, such as static random access memory (SRAM) and dynamic random access memory (DRAM), and/or non-volatile memory, such as read only memory (ROM), erasable programmable ROM, flash memories, hard disks, optical disks, and magnetic tapes. The memory controller <NUM> is further connected to the input line buffer <NUM> to fetch external image data. Based on instructions provided by the host interface <NUM>, the data splitter stride <NUM> slices the image data into one or more strides based on signal received from the system controller <NUM>.

The system controller <NUM> is connected to the host interface <NUM>, the kernel register space <NUM>, the threshold register space <NUM>, the multiplexer <NUM>, the memory controller <NUM>, and the adder engine <NUM>. The system controller <NUM> may be configured to generate a mode signal based on the image data received from the host interface <NUM>. The mode signal indicates kernel size and layers of convolution to be performed on the image. The system controller <NUM> may also be configured to store programmed instructions to operate the reconfigurable convolution engine <NUM>. In an implementation, the system controller <NUM> is configured to provide the mode signal to the data splitter stride <NUM>, the adder engine <NUM>, the reconfigurable multiplier engine <NUM>, and the ReLU <NUM>.

The data splitter stride <NUM> is connected to the input line buffer <NUM>. In an example, if the reconfigurable multiplier engine <NUM> is of MxM size, then the data splitter stride <NUM> may comprise M+<NUM> input and M+<NUM> output lines. The M+<NUM> input lines may be connected to M+<NUM> input line buffer, and M+<NUM> output lines may be connected to the reconfigurable multiplier engine <NUM>. The data splitter stride <NUM> may be configured to perform two functions. First, the data splitter stride <NUM> is configured to pack the image data received from the input line buffer <NUM> for the reconfigurable multiplier engine <NUM>. The image data may be packed based on the mode signals. Second, the data splitter stride <NUM> may skip the image data from the input line buffer <NUM> based on the stride selected by the host processor <NUM>. The image data may be skipped based on a shift signal transmitted by the system controller <NUM>.

The reconfigurable multiplier engine <NUM> comprises a plurality of multipliers. Each multiplier is referred as an instance. It is to be noted that each multiplier is represented as a unique identification number, for example, MULT_ID. In one implementation, size of the reconfigurable multiplier engine <NUM> is configured dynamically based on the kernel size. The kernel size is configured by the mode signal. Upon receipt of the mode signal, the reconfigurable multiplier engine <NUM> is segmented into the plurality of the instances of size MxM. Each instance of the plurality of instances may comprise a connection table, a data slicer, and a multiplier function. The connection table is configured to route the input line buffer <NUM> data to the data slicer. It is important to note that the connection table is configured to route the data based on the mode signal and the MULT_ID associated to each of the instance. The data slicer is configured to slice off the image data into pixel data and kernel based on the mode signal and the MULT_ID. It is to be noted that the MULT_ID may optimize functionality of the data slicer during compile time and the mode signal may assist the data slicer in run time. The multiplier function may represent a mathematic functional block configured to multiply image data with the kernel. The output of the mathematic functional block is concatenated with the MULT_ID and reserved bits for debugging and further expansion of the multiplier function. In one aspect, when the reconfigurable convolution engine <NUM> is implemented on the FPGA, the multiplier functional may be implemented on the on-chip DSP. In another aspect, when the reconfigurable convolution engine <NUM> is implemented on the ASIC, the multiplier functional block may be used as a separate functional block.

The adder engine <NUM> is configured to perform adder operation on the outputs of the plurality of instances running in parallel. The adder engine comprises a plurality of adders arranged in a pipelined structure. Each adder from the plurality of adders may comprise a connection table and an adder logic. In an example, for the reconfigurable multiplier engine <NUM> comprising N instances, total number of adders to be required is N-<NUM> for performing concurrent convolution operations.

The ReLU <NUM> may be configured to filter the output of the adder engine <NUM> based on a threshold set by the host processor <NUM>. In an implementation, the ReLU <NUM> may comprise a bypass function to transmit the output of the adder engine <NUM> without filtering.

Referring now to <FIG>, the reconfigurable convolution engine <NUM> is illustrated in accordance with an embodiment of the present subject matter. In one embodiment, the reconfigurable convolution engine <NUM> may include at least one processor <NUM>, a host interface <NUM>, and a memory controller <NUM>. The at least one processor <NUM> may be implemented as one or more microprocessors, microcomputers, microcontrollers, digital signal processors, central processing units, state machines, logic circuitries, and/or any devices that manipulate signals based on operational instructions. Among other capabilities, the at least one processor <NUM> is configured to fetch and execute computer-readable instructions stored in the memory controller <NUM>. The memory controller <NUM> may include modules <NUM> and data <NUM>.

The modules <NUM> include routines, programs, objects, components, data structures, etc., which perform particular tasks or implement particular abstract data types. In one implementation, the modules <NUM> may include a data receiving module <NUM>, a determination module <NUM>, a generation module <NUM>, a configuration module <NUM>, an execution module <NUM>, a filter module <NUM>, and other modules <NUM>. The other modules <NUM> may include programs or coded instructions that supplement applications and functions of the reconfigurable convolution engine <NUM>. The modules <NUM> described herein may be implemented as software modules that may be executed in the cloud-based computing environment of the reconfigurable convolution engine <NUM>.

The data <NUM>, amongst other things, serves as a repository for storing data processed, received, and generated by one or more of the modules <NUM>. The data <NUM> may also include a system database <NUM> and other data <NUM>. The other data <NUM> may include data generated as a result of the execution of one or more modules in the other modules <NUM>.

As there are various challenges observed in the existing art, the challenges necessitate the need to build the reconfigurable convolution engine <NUM> for performing a convolution operation on an image. In order to perform the convolution operation on the image, at first, a user may use the host device to access the reconfigurable convolution engine <NUM> via the host interface <NUM>. The user may register them using the host interface <NUM> in order to use the reconfigurable convolution engine <NUM>. In one aspect, the user may access the host interface <NUM> of the reconfigurable convolution engine <NUM>. The reconfigurable convolution engine <NUM> may employ the data receiving module <NUM>, the determination module <NUM>, the generation module <NUM>, the configuration module <NUM>, the execution module <NUM>, and the filter module <NUM>. The detail functioning of the modules is described below with the help of figures.

The present subject matter describes the reconfigurable convolution engine <NUM> for performing a convolution operation on an image. To do so, initially, the data receiving module <NUM> receives image data pertaining to the image. The image data comprises pixel resolution, number of filters to be applied, and a convolution layer. In one implementation, a filter from the number of filters may be considered as a kernel for performing the convolution operation. The kernel may also be referred as a feature detector. Upon receiving the image data, the determination module <NUM> determines a kernel size based on the image data, clock speed associated to the convolution engine and number of available on-chip resources. It is to be noted that the clock speed of the convolution engine is same as the clock speed of a Graphical Processing Unit (GPU) installed in at least one of the variety of computing systems. In an implementation, the kernel size may also be determined based on the mode signal received from the host processor <NUM>. In an implementation, the kernel size may be determined based on the convolution layer received from the host interface <NUM>.

Subsequent to determining the kernel size, the generation module <NUM> generates a plurality of instances based on the kernel size. In one aspect, the plurality of instances is configured to operate in parallel mode. In another aspect, each instance of the plurality of instances is configured to perform convolution operation with the clock speed of the convolution engine. Each instance comprises a connection table, a data slicer, and a multiplier. It is to be noted that each multiplier comprises a multiplier function. In an implementation, the multiplier function may multiply the pixel resolution with the kernel. In another implementation, the generation module <NUM> may generate the plurality of the instances of size MxM based on the mode signal. In an example, when the kernel size is 3x3 and size of the reconfigurable multiplier engine <NUM> is 10x10, <NUM> parallel instances of size 3x3 are created. In an alternative example, when the kernel size is 5x5 and size of the reconfigurable multiplier engine <NUM> is 10x10, <NUM> parallel instances of size 5x5 are created. In yet another example, when the kernel size is 7x7 and size of the reconfigurable multiplier engine <NUM> is 10x10, <NUM> parallel instance of size 7x7 is created. In yet another example, when the kernel size is 9x9 and size of the reconfigurable multiplier engine <NUM> is 10x10, <NUM> parallel instance of size 9x9 is created.

After generating the plurality of instances, the configuration module <NUM> configures an adder engine based on the plurality of instances. In one aspect, the adder engine comprises a plurality of adders configured to operate in a pipelined structure and in parallel with the plurality of instances. Each adder from the plurality of adders comprises a connection table and an adder logic. In an implementation, when the plurality of the instances is of the size MxM, the configuration module <NUM> may configure M-<NUM> adders.

Upon configuration of the plurality of the adders, the execution module <NUM> executes the convolution operation on each of the plurality of instances and the plurality of adders. The execution module <NUM> executes the convolution operation in a concurrent environment based on the mode signal. In an implementation, the mode signal represents number of layers required by the adder engine to execute the convolution operation concurrently.

Once the convolution operation is executed, the filter module <NUM> filters an output of the convolution operation by using a filter function. Example of the filter function include, but not limited to, a Rectified Linear Unit (ReLU), Sigmoid or Logistic, and Hyperbolic tangent function-Tanh. In one implementation, the ReLU <NUM> may be configured to operate on a predefined threshold function provided by the host interface <NUM> thereby performing the convolution operation on the image using the reconfigurable convolution engine <NUM>. In one implementation, the convolution engine <NUM> may be utilized to perform convolution operation for at least one of Convolution Neural Network (CNN) technique, Deep Recurrent Neural Network (Deep RNN) technique, and Artificial Neural Network (ANN) technique.

In order to elucidate further, consider an example where image size = 640x400; stride = <NUM>; kernel size = 3x3; and number of input filter = <NUM>. Assuming the clock speed of the reconfigurable convolution engine <NUM> as <NUM> i.e. 10ns and size of the multiplier engine as 10x10.

Number of convolutions for the image of size 640x400 = <NUM>×<NUM><NUM> convolutions.

Referring now to <FIG>, a method <NUM> for performing a convolution operation on an image using a reconfigurable convolution engine is shown, in accordance with an embodiment of the present subject matter. The method <NUM> may be described in the general context of computer executable instructions. Generally, computer executable instructions can include routines, programs, objects, components, data structures, procedures, modules, functions, etc., that perform particular functions or implement particular abstract data types. The method <NUM> may also be practiced in a distributed computing environment where functions are performed by remote processing devices that are linked through a communications network. In a distributed computing environment, computer executable instructions may be located in both local and remote computer storage media, including memory storage devices.

In the embodiments described below, the method <NUM> is implemented as described in the reconfigurable convolution engine <NUM>.

At block <NUM>, image data is received for performing a convolution operation on an image by using a convolution engine. In one implementation, the image data, for performing a convolution operation on an image by using a convolution engine, may be received by a data receiving module <NUM>.

At block <NUM>, a kernel size is determined based on the image data, clock speed associated to the convolution engine and number of available on-chip resources. In one implementation, the kernel size may be determined by a determination module <NUM>.

At block <NUM>, a plurality of instances are generated based on the kernel size. In one aspect, the plurality of instances is configured to operate in parallel mode. In another aspect, each instance of the plurality of instances is configured to perform convolution operation with the clock speed of the convolution engine. In one implementation, the plurality of instances are generated based on the kernel size by a generation module <NUM>.

At block <NUM>, an adder engine is configured based on the plurality of instances. In one aspect, the adder engine comprises plurality of adders configured to operate in a pipelined structure and in parallel with the plurality of instances. In one implementation, the adder engine may be configured by a configuration module <NUM>.

At block <NUM>, the convolution operation on each of the plurality of instances and the adder engine is executed. In one implementation, the convolution operation on each of the plurality of instances and the adder engine by may be executed by an execution module <NUM>.

At block <NUM>, an output of the convolution operation is filtered by using a filter function configured to operate on a predefined threshold function thereby performing the convolution operation on the image using the reconfigurable convolution engine. In one implementation, output of the convolution operation may be filtered by a filtering module <NUM>.

Exemplary embodiments discussed above may provide certain advantages. Though not required to practice aspects of the disclosure, these advantages may include those provided by the following features.

Some embodiments enable a system and a method to perform convolution operations in concurrent.

Some embodiments enable a system and a method to enhance processing power of the available on chip resources by concurrently performing convolution operations.

Some embodiments enable a system and a method to reuse same resource for one or more convolution layer.

Some embodiments enable a system and a method to reconfigure the convolution engine based on various kernel sizes.

Some embodiments enable a system and a method to increase throughput of the reconfigurable convolution engine by increasing an operating frequency of the reconfigurable convolution engine.

Claim 1:
A method for performing a convolution operation on an image using a reconfigurable convolution engine (<NUM>) comprising a processor (<NUM>) and an adder engine (<NUM>), the method comprising:
receiving, by the processor (<NUM>), image data for performing a convolution operation on an image by using a convolution engine, the image data comprising a convolution layer to be performed on the image;
determining, by the processor (<NUM>), a kernel size based on the image data, clock speed associated to the convolution engine and number of available on-chip resources;
generating, by the processor (<NUM>), a plurality of instances based on the kernel size, wherein each instance comprises a respective multiplier of a reconfigurable multiplier engine (<NUM>) comprised in the reconfigurable convolution engine, wherein the reconfigurable multiplier engine is segmented into the plurality of the instances having size MxM that is equal to the kernel size, wherein the plurality of instances is configured to operate in parallel mode, and wherein each instance of the plurality of instances is configured to perform convolution operation with the clock speed of the convolution engine, and wherein each instance of the plurality of instances is represented as a unique identification number;
configuring, by the processor (<NUM>), the adder engine (<NUM>) based on the plurality of instances, wherein the adder engine (<NUM>) comprises a plurality of adders configured to operate in a pipelined structure and in parallel with the plurality of instances;
executing, by the processor (<NUM>), the convolution operation on each of the plurality of instances and the adder engine (<NUM>); and
filtering, by the processor (<NUM>), an output of the convolution operation by using a filter function configured to operate on a predefined threshold function thereby performing the convolution operation on the image using the reconfigurable convolution engine.