Patent Description:
However, this well-known structure suffers of low efficiency where large amounts of graphical data is involved, due to the large amounts of handling resources required for managing the transfer of the graphical data between the CPU and the MPU, back-and-forth. In some cases, the net time usable for data computations in a CPU-MPU computing structure may be as low as <NUM>% or less. For example, a Nvidia ® Compute Unified Device Architecture (CUDA) parallel computing platform and application programming interface model typical time portions spent for graphical data handling may be <NUM>% for transferring the graphical data from the CPU environment to the GPU environment (e.g. to CUDA memory), <NUM>% for transferring the graphical data from the GPU environment back to the CPU environment (CUDA memory) and no more than <NUM>% for graphical computations. Such very low graphical computation efficiency stems from the common architectures defining the way graphical data is transferred between the processors.

There is a need to enable substantial raise of the MPU efficiency, that is substantial raise of the time portion assigned to graphical calculations.

A circuit for handling unprocessed data is disclosed comprising a data stream divider unit (DSDU) and a graphics processing unit (GPU). The DSDU comprising an array comprising plurality of first-in-first-out (FIFO) registers, configured to receive a stream of data and to divide it into portions of data and to pass each of the portions of data through one of the plurality of FIFO registers and a first advanced extensible interface (AXI) unit configured to receive the data portions. The GPU comprising a second advanced extensible interface (AXI) unit configured to receive data portions from the first AXI unit and a plurality of streaming multiprocessors (SM) configured to receive each data portion from a respective FIFO register, and to process the received data portion.

In some embodiments a specific FIFO register in the DSDU is connected to an assigned SM in the GPU via an assigned first AXI unit in the DSDU and an assigned second AXI unit in the GPU.

In some embodiments each of the FIFO registers in the DSDU is connected to an assigned SM in the GPU via a first common AXI unit in the DSDU and a common AXI unit in the GPU.

A method for efficiently processing large amount of data is disclosed comprising receiving a stream of unprocessed data, dividing the stream to a plurality of data portions, passing each data portion via a specific FIFO register in a data stream divider unit (DSDU), and transferring the data portion from the specific FIFO register to an assigned streaming multiprocessor (SM) in graphics processor unit (GPU) for processing.

In some embodiments the data portions are transferred via a first specific advanced extensible interface (AXI) unit in the DSDU and a second specific advanced extensible interface (AXI) unit in the GPU.

In some embodiments a data portion received from a specific FIFO register is transferred to the assigned SM in the GPU via an assigned first AXI unit in the DSDU and an assigned second AXI unit in the GPU.

In some embodiments each of the data portion received from FIFO registers in the DSDU is transferred to the assigned SM in the GPU via a common first AXI unit in the DSDU and a common second AXI unit in the GPU.

The bottle-neck of CPU-GPU mutual operation in known computing systems lies, mostly, in the data transfer channels used for directing graphical related data by the CPU to the GPU and receiving the processed graphical data back from the GPU. Typically, the CPU and the GPU processors operate and communicate in standard computing environments.

Reference is made to <FIG>, schematically illustrating data flow in computing unit <NUM> using a GPU. Computing unit <NUM> comprise CPU <NUM>, CPU dynamic RAM (DRAM) 111A, and computing unit peripheral controlling unit (such as main board chipset) <NUM>. Unit <NUM> further comprises GPU unit <NUM>, communicating data with the CPU via unit <NUM>.

GPU unit <NUM> typically comprises GPU DRAM unit <NUM>, interfacing data between unit <NUM> and the GPU processors, GPU cache units <NUM> (such as L2 cache units) that is adapted to cache data for the GPU processing units, and GPU processing units <NUM> (such as streaming multiprocessor / SM).

The flow of graphical data that enters processing unit <NUM> and is intended to be processed by GPU <NUM> is described by data flow (DF) arrows. First Data flow - DF1 depicts the flow of data into computing unit <NUM>, where CPU <NUM> directs the flow - DF2 - via peripheral controlling unit (PCU) <NUM>, to DRAM 111A and back from it - DF3 - via PCU <NUM> - DF4 - to GPU <NUM>. At GPU <NUM> the flow of the data passes through DRAM unit <NUM> and through cache units <NUM> to the plurality of streaming multiprocessors (SMs) units <NUM> where graphical processing takes place.

It is a target of methods and structures according to the present invention to eliminate as much data flow bottle-necks as possible.

Reference is made now to <FIG>, which is a schematic block diagram of a typical streaming multiprocessor (SM) <NUM> in a GPU unit. SM <NUM> may comprise processing core unit <NUM> (sometimes called Compute Unified Device Architecture (CUDA) core), register file <NUM> to mediate data between core <NUM> and cache units <NUM> (constant cache), <NUM> (unified cache) and with shared memory <NUM>. Data inbound towards SM <NUM> and outbound from it is exchanged with the GPU cache unit <NUM> (such as cache units <NUM> (L2)) of <FIG>. When graphical processing is carried out in known methods, the GPU unit will await until the entire amount of data to be processed is loaded onto the memory units of its several SM <NUM> units before graphical processing commences.

One way of reducing data transfers time is minimization of redundant data transfers. For example, intermediate results calculated by core <NUM> may be stored in register file <NUM> instead of storing them in the DRAM. Further, shared memory <NUM> may be used for storing data that is frequently used within SM <NUM>, instead of circulating it outbound, as is commonly done. In some embodiments the level of frequency of use is determined by the PCU. Still further, constant memory units and/or cache memory units may be defined in SM <NUM>.

According to further embodiments of the present invention data flow bottle-neck between the CPU computing environment and the GPU computing environment may be reduced or eliminate, by replacing the CPU with a specifically structured computing unit for all handling of graphical-related data.

Reference is made now to <FIG>, which is a schematic block diagram depicting an unprocessed data (UPD) handling unit (UPDHU) <NUM>, structured and operative according to embodiments of the present invention, and to <FIG>, which are schematic block diagrams of two different embodiments, <NUM> and <NUM>, of UPDHU, such UPDHU <NUM> of <FIG>. The term 'unprocessed data' as used herein after relates to large stream of data that is about to be processed, and that typically requires large computation capacity, such as fast stream of graphical data (e.g. received from <NUM> video camera) that needs to be processed in virtually "real time" (i.e. with as small as possible latency). The architecture of UPDHU <NUM> depicted in <FIG> is designed to overcome the inherent bottle neck of data stream flow typical to CPU-GPU known architectures, where incoming stream of data acquisition is first handled by a the CPU, then being temporarily stored in the CPU memory and/or RAM associated with the CPU, then it is transferred, again as data stream (e.g. over Peripheral Component Interconnect Express (PCIe) bus), to the GPU and again being handled by the GPU processor before it is sent to the multiple streaming processors being part of the GPU. The example described herein with regard to <FIG> demonstrates use of field programmable gate array (FPGA) programmed to operate according to the advanced extensible interface (AXI), however it would be apparent to those skilled in the art that the method of operation described herein may be embodied using other computing units that are adapted to interface with a respective GPU and to transfer large amount of graphical - related data in high throughput. According to embodiments of the invention data stream divider unit (DSDU) <NUM> may be embodied by, for example, using FPGA that is programmed to receive large amount of streamed UPD, e.g. video stream from a camera, to distribute it into plurality of smaller streams and to transfer the streams to SMs of a GPU. The FPGA and the GPU may further be programmed to operate so that the GPU begins processing of the graphical data transferred to it as soon as at least one SM of the plurality of the SMs of the GPU is fully loaded. In most cases the fully loaded SMs hold amount of data that is smaller than the full data file, therefore the processing by the GPU will begin, according to this embodiment, much earlier compared to commonly known embodiments where the processing begins after the entire data file was loaded to the GPU.

In an exemplary embodiment UPDHU <NUM> comprises a Multi Streamer unit (MSU) <NUM> that may comprise a DSDU <NUM> comprising plurality of first-in-first-out (FIFO) registers/storage units array 304A (the FIFO units are not shown separately), of which one FIFO unit may be assigned to each of the SMs <NUM> of GPU <NUM>. In some embodiments the received UPD stream that is received by DSDU <NUM> may be partitioned to multiple data units, which may be transferred to GPU <NUM> via FIFO units 304A, broadcasted to the GPU over an interface unit, such as AXI interface, such that data unit in each FIFO 304A is transferred to the associated SM <NUM>, thereby enabling, for example, single action multiple data (SIMD) computing. When each (even a single) SM <NUM> of GPU <NUM> is loaded with the respective portion of the unprocessed data received from the associated FIFO 304A unit over an AXI interface, GPU <NUM> may start processing, not having to wait until the entire UPD file is loaded.

MSU <NUM> may comprise unprocessed data interface unit <NUM>, configured to receive long streams of graphical data. The large amount of unprocessed data received via interface unit <NUM> may be partitioned to smaller size, plurality number of data units, to be transferred each via an assigned FIFO unit in FIFO unit 304A and then, over an AXI channel <NUM>, via GPU AXI interface <NUM> to the assigned SM <NUM> of GPU <NUM>.

Data units that were processed by the respective SM of SMs <NUM> may then be transferred back, over AXI connection, to the MSU. As seen, large overhead that is typical to CPU - GPU architectures is saved in the embodiments described above.

<FIG> depict schematic block diagrams of two optional architectures embodying MSU <NUM> of <FIG>, according to embodiments of the present invention. <FIG> depicts MSU <NUM> comprises FIU <NUM> and GPU <NUM>. FIU <NUM> may comprise plurality of FIFO units (collectively named 356A) - FIFO<NUM>, FIFO<NUM>. each FIFO unit may be in active communication with an assigned FPGA AXI (F-AXI) unit - F-AXI<NUM>, F-AXI<NUM>. F-AXIn (collectively named 356B). Each of the separate F-AXI units may be in direct connection with an assigned GPU AXI (G-AXI) unit- G-AXI<NUM>, G-AXI<NUM>. each of the G-AXI interface units may be in active connection with, and may provide data to an assigned SM - SM<NUM>, SM<NUM>. According to yet another embodiment, as depicted in <FIG>, MSU <NUM> comprises FIU <NUM> and GPU <NUM>. FIU <NUM> may comprise plurality of FIFO units (collectively named 386A) - FIFO<NUM>, FIFO<NUM>. each FIFO unit may be in active communication with a FPGA AXI (F-AXI) unit that may be configured to control administer the data streams from the plurality of FIFO units into a single AXI stream. The AXI stream may be transmitted to an AXI interface unit 388A of GPU <NUM> and may then be divided to the respective SMs units - SM<NUM>, SM<NUM>. The architecture depicted in <FIG> may provide a faster overall performance but may require larger number of pins (for an integrated circuit (IC) embodying the described circuit) and a larger number of wires/conduits. The architecture depicted in <FIG> may provide a relatively slower overall performance but may require smaller number of pins (for an integrated circuit (IC) embodying the described circuit) and a smaller number of wires/conduits.

The above described devices, structures and methods may accelerate the processing of large amount of unprocessed data, compared to known architectures and methods. For example, in known embodiments there is the need to transfer the whole image before the process / algorithm could start on the GPU. If the image size is 1GB, the theoretical throughput of the PCI-E bus transferring data to the GPU is 32GB/s, so latency would be 1GB/(32GB/s)= <NUM>/<NUM> =<NUM> ≈ <NUM>. in contrary, with the FPGA according to embodiment of the invention it is just needed to fully load all SM units. For example, in the Tesla P100 GPU there are <NUM> SM units, and in each SM there are <NUM> cores that support 32bit (in single precision mode) or <NUM> cores that support 64bit (extended precision mode), thus the data size for a fully loaded GPU (same result for single or extended precision modes) is <NUM>*<NUM>*<NUM> = <NUM> bits = <NUM> Mbytes. The FPGA to GPU AXI stream theoretical throughput is 896MB/s (for <NUM> lanes), so latency is <NUM>. 336MB/(896MB/s) = <NUM>/<NUM> = <NUM>, which is substantially half the latency.

Claim 1:
A circuit for handling unprocessed data comprising:
a data stream divider unit (DSDU) (<NUM>), comprising:
an array comprising plurality of first-in-first-out (FIFO) registers, configured to receive a stream of data and to divide it into portions of data and to pass each of the portions of data through one of the plurality of FIFO registers; and
a first advanced extensible interface (AXI) unit configured to receive the data portions; and
a graphics processing unit (GPU) comprising:
a second advanced extensible interface (AXI) unit configured to receive data portions from the first AXI unit; and
a plurality of streaming multiprocessors (SM) configured to receive each data portion from a respective FIFO register, and to process the received data portion.