Patent ID: 12229057

In the figures, like reference numbers may indicate functionally similar elements. The systems and methods illustrated in the figures, and described in the Detailed Description below, may be arranged and designed in a wide variety of different implementations. Neither the figures nor the Detailed Description are intended to limit the scope of the claims. Instead, they merely represent examples of different implementations of the disclosed technology.

DETAILED DESCRIPTION

A host runtime processing system as described herein can dynamically select the most efficient data passing method for a given set of source and target devices in an application defined pipeline (ADP) based on the latency & bandwidth requirement of the operating device stages. The host runtime processor hides the underlying details of methodologies of data passing from the application and picks between one of at least the three different methods, including two memory to memory transfers via a common buffer area of a CPU, one memory to memory transfer without using any buffer(s) on the CPU, and memory extension operation as further described herein. The first two data passing methods involve memory to memory transfer, whereas the memory extension operation involves direct memory access (for directly reading data or directly writing data) without any memory to memory transfer operations.

Such dynamic operation enables pre & post data passing stages of the ADP to read or write directly to or from addressable memory of an operating device as part of the compute-bound operations. Application users do not need to take care of selecting the data passing method or know details on how to program a particular data passing method. The host runtime processor uses the user-defined metadata of data passing, such as the source and destination device types, the memory addresses on both sides of the data-passing, ADP bandwidth and latency requirement of operating device stages, number of configured common or multiple buffers, etc., to determine optimal data passing method for the input or output, and to program the data passing or memory access method to orchestrate and complete data access between source and target devices.

Traditional compilers translate human-readable computer source code into machine code that can be executed on a Von Neumann computer architecture. In this architecture, a processor serially executes instructions in one or more threads of software code. The architecture is static and the compiler does not determine how execution of the instructions is pipelined, or which processor or memory takes care of which thread. Thread execution is asynchronous, and safe exchange of data between parallel threads is not supported.

High-level programs for machine learning (ML) and artificial intelligence (AI) may require massively parallel computations, where many parallel and interdependent threads (meta-pipelines) exchange data. Such programs are ill-suited for execution on Von Neumann computers. They require architectures that are optimized for parallel processing, such as coarse-grained reconfigurable architectures (CGRAs) or graphic processing units (GrPUs). The ascent of ML, AI, and massively parallel architectures places new requirements on compilers, including how computation graphs, and in particular dataflow graphs, are pipelined, which operations are assigned to which compute units, how data is routed between various compute units and memory, and how synchronization is controlled particularly when a dataflow graph includes one or more nested loops, whose execution time varies dependent on the data being processed.

Terminology

As used herein, the phrase one of should be interpreted to mean exactly one of the listed items. For example, the phrase “one of A, B, and C” should be interpreted to mean any of: only A, only B, or only C.

As used herein, the phrases at least one of and one or more of should be interpreted to mean one or more items. For example, the phrase “at least one of A, B, and C” or the phrase “at least one of A, B, or C” should be interpreted to mean any combination of A, B, and/or C. The phrase “at least one of A, B, and C” means at least one of A and at least one of B and at least one of C.

Unless otherwise specified, the use of ordinal adjectives first, second, third, etc., to describe an object, merely refers to different instances or classes of the object and does not imply any ranking or sequence.

The following terms or acronyms used herein are defined at least in part as follows:AGCU—address generator (AG) and coalescing unit (CU).APU—artificial intelligence (AI) and AI processing unit.AIR—arithmetic or algebraic intermediate representation.ALN—array-level network.Buffer—an intermediate storage of data.CGR—coarse-grained reconfigurable. A property of, for example, a system, a processor, anarchitecture (see CGRA), an array, or a unit in an array. This property distinguishes the system, etc., from field-programmable gate arrays (FPGAs), which can implement digital circuits at the gate level and are therefore fine-grained configurable.CGRA—coarse-grained reconfigurable architecture. A data processor architecture that includes one or more arrays (CGR arrays) of CGR units.Compiler—a translator that processes statements written in a programming language to machine language instructions for a computer processor. A compiler may include multiple stages to operate in multiple steps. Each stage may create or update an intermediate representation (IR) of the translated statements. Compiler stages are illustrated with reference toFIG.5.Computation graph—some algorithms can be represented as computation graphs. As used herein, computation graphs are a type of directed graphs comprising nodes that represent mathematical operations/expressions and edges that indicate dependencies between the operations/expressions. For example, with machine learning (ML) algorithms, input layer nodes assign variables, output layer nodes represent algorithm outcomes, and hidden layer nodes perform operations on the variables. Edges represent data (e.g., scalars, vectors, tensors) flowing between operations. In addition to dependencies, the computation graph reveals which operations and/or expressions can be executed concurrently.CGR unit—a circuit that can be configured and reconfigured to locally store data (e.g., a memory unit or a PMU), or to execute a programmable function (e.g., a compute unit or a PCU). A CGR unit includes hardwired functionality that performs a limited number of functions used in computation graphs and dataflow graphs. Further examples of CGR units include a CU and an AG, which may be combined in an AGCU. Some implementations include CGR switches, whereas other implementations may include regular switches.CU—coalescing unit.Data Flow Graph—a computation graph that includes one or more loops that may be nested, and wherein nodes can send messages to nodes in earlier layers to control the dataflow between the layers.Datapath—a collection of functional units that perform data processing operations. The functional units may include memory, multiplexers, ALUs, SIMDs, multipliers, registers, buses, etc.FCMU—fused compute and memory unit—a circuit that includes both a memory unit and a compute unit.Graph—a collection of nodes connected by edges. Nodes may represent various kinds of items or operations, dependent on the type of graph. Edges may represent relationships, directions, dependencies, etc.IC—integrated circuit—a monolithically integrated circuit, i.e., a single semiconductor die which may be delivered as a bare die or as a packaged circuit. For the purposes of this document, the term integrated circuit also includes packaged circuits that include multiple semiconductor dies, stacked dies, or multiple-die substrates. Such constructions are now common in the industry, produced by the same supply chains, and for the average user often indistinguishable from monolithic circuits.A logical CGR array or logical CGR unit—a CGR array or a CGR unit that is physically realizable, but that may not have been assigned to a physical CGR array or to a physical CGR unit on an IC.ML—machine learning.PCU—pattern compute unit—a compute unit that can be configured to perform one or more operations.PEF—processor-executable format—a file format suitable for configuring a configurable data processor.Pipeline—a staggered flow of operations through a chain of pipeline stages. The operations may be executed in parallel and in a time-sliced fashion. Pipelining increases overall instruction throughput. CGR processors may include pipelines at different levels. For example, a compute unit may include a pipeline at the gate level to enable correct timing of gate-level operations in a synchronous logic implementation of the compute unit, and a meta-pipeline at the graph execution level to enable correct timing of node-level operations of the configured graph. Gate-level pipelines are usually hard wired and unchangeable, whereas meta-pipelines are configured at the CGR processor, CGR array level, and/or GCR unit level.Pipeline Stages—a pipeline is divided into stages that are coupled with one another to form a pipe topology.PMU—pattern memory unit—a memory unit that can locally store data.PNR—place and route—the assignment of logical CGR units and associated processing/operations to physical CGR units in an array, and the configuration of communication paths between the physical CGR units.RAIL—reconfigurable dataflow unit (RDU) abstract intermediate language.CGR Array—an array of CGR units, coupled with each other through an array-level network (ALN), and coupled with external elements via a top-level network (TLN). A CGR array can physically implement the nodes and edges of a dataflow graph and is sometimes referred to as a reconfigurable dataflow unit (RDU).SIMD—single-instruction multiple-data—an arithmetic logic unit (ALU) that simultaneously performs a single programmable operation on multiple data elements delivering multiple output results.TLIR—template library intermediate representation.TLN—top-level network.

Implementations

The architecture, configurability and dataflow capabilities of an array of CGR units enable increased compute power that supports both parallel and pipelined computation. A CGR processor, which includes one or more CGR arrays (arrays of CGR units), can be programmed to simultaneously execute multiple independent and interdependent dataflow graphs. To enable simultaneous execution, the dataflow graphs may need to be distilled from a high-level program and translated to a configuration file for the CGR processor. A high-level program is source code written in programming languages like Spatial, Python, C++, and C, and may use computation libraries for scientific computing, ML, AI, and the like. The high-level program and referenced libraries can implement computing structures and algorithms of machine learning models like AlexNet, VGG Net, GoogleNet, ResNet, ResNeXt, RCNN, YOLO, SqueezeNet, SegNet, GAN, BERT, ELMo, USE, Transformer, and Transformer-XL.

Translation of high-level programs to executable bit files is performed by a compiler. While traditional compilers sequentially map operations to processor instructions, typically without regard to pipeline utilization and duration (a task usually handled by the hardware), an array of CGR units requires mapping operations to processor instructions in both space (for parallelism) and time (for synchronization of interdependent computation graphs or dataflow graphs). This requirement implies that a compiler for a CGRA must decide which operation of a computation graph or dataflow graph is assigned to which of the CGR units, and how both data and, related to the support of dataflow graphs, control information flows among CGR units, and to and from external hosts and storage. This process, known as “place and route”, is one of many new challenges posed to compilers for arrays of CGR units.

FIG.1illustrates an example system100including a CGR processor110, a host180, and a memory190. CGR processor110has a coarse-grained reconfigurable architecture (CGRA) and includes an array of CGR units120such as a CGR array. CGR processor110further includes an IO interface138, and a memory interface139. Array of CGR units120is coupled with IO interface138and memory interface139via databus130which may be part of a top-level network (TLN). Host180communicates with IO interface138via system databus185, and memory interface139communicates with memory190via memory bus195. Array of CGR units120may further include compute units and memory units that connected with an array-level network (ALN) to provide the circuitry for execution of a computation graph or a dataflow graph that may have been derived from a high-level program with user algorithms and functions. The high-level program may include a set of procedures, such as learning or inferencing in an AI or ML system. More specifically, the high-level program may include applications, graphs, application graphs, user applications, computation graphs, control flow graphs, dataflow graphs, models, deep learning applications, deep learning neural networks, programs, program images, jobs, tasks and/or any other procedures and functions that may need serial and/or parallel processing. In some implementations, execution of the graph(s) may involve using multiple units of CGR processor110. In some implementations, CGR processor110may include one or more ICs. In other implementations, a single IC may span multiple coarsely reconfigurable data processors. In further implementations, CGR processor110may include one or more units of array of CGR units120.

Host180may be, or include, a computer such as further described with reference toFIG.2. Host180runs runtime processes, as further referenced herein, and may also be used to run computer programs, such as the compiler further described herein with reference toFIG.12. In some implementations, the compiler may run on a computer that is similar to the computer described with reference toFIG.2, but separate from host180.

CGR processor110may accomplish computational tasks by executing a configuration file (for example, a PEF file). For the purposes of this description, a configuration file corresponds to a dataflow graph, or a translation of a dataflow graph, and may further include initialization data. A compiler compiles the high-level program to provide the configuration file. In some implementations described herein, a CGR array is configured by programming one or more configuration stores with all or parts of the configuration file. A single configuration store may be at the level of the CGR processor or the CGR array, or a CGR unit may include an individual configuration store. The configuration file may include configuration data for the CGR array and CGR units in the CGR array, and link the computation graph to the CGR array. Execution of the configuration file by CGR processor110causes the CGR array (s) to implement the user algorithms and functions in the dataflow graph.

CGR processor110can be implemented on a single integrated circuit die or on a multichip module (MCM). An IC can be packaged in a single chip module or a multichip module. An MCM is an electronic package that may comprise multiple IC dies and other devices, assembled into a single module as if it were a single device. The various dies of an MCM may be mounted on a substrate, and the bare dies of the substrate are electrically coupled to the surface or to each other using for some examples, wire bonding, tape bonding or flip-chip bonding.

FIG.2illustrates an example of a computer200, including an input device210, a processor220, a storage device230, and an output device240. Although the example computer200is drawn with a single processor, other implementations may have multiple processors. Input device210may comprise a mouse, a keyboard, a sensor, an input port (for example, a universal serial bus (USB) port), and any other input device known in the art. Output device240may comprise a monitor, printer, and any other output device known in the art. Furthermore, part or all of input device210and output device240may be combined in a network interface, such as a Peripheral Component Interconnect Express (PCIe) interface suitable for communicating with CGR processor110. Input device210is coupled with processor220to provide input data, which an implementation may store in memory226. Processor220is coupled with output device240to provide output data from memory226to output device240. Processor220further includes control logic222, operable to control memory226and arithmetic and logic unit (ALU)224, and to receive program and configuration data from memory226. Control logic222further controls exchange of data between memory226and storage device230. Memory226typically comprises memory with fast access, such as static random-access memory (SRAM), whereas storage device230typically comprises memory with slow access, such as dynamic random-access memory (DRAM), flash memory, magnetic disks, optical disks, and any other memory type known in the art. At least a part of the memory in storage device230includes a non-transitory computer-readable medium (CRM235), such as used for storing computer programs.

FIG.3illustrates example details of a CGR architecture300including a top-level network (TLN330) and two CGR arrays (CGR array310and CGR array320). A CGR array comprises an array of CGR units (e.g., PMUs, PCUs, FCMUs) coupled via an array-level network (ALN), e.g., a bus system. The ALN is coupled with the TLN330through several AGCUs, and consequently with I/O interface338(or any number of interfaces) and memory interface339. Other implementations may use different bus or communication architectures.

Circuits on the TLN in this example include one or more external I/O interfaces, including I/O interface338and memory interface339. The interfaces to external devices include circuits for routing data among circuits coupled with the TLN and external devices, such as high-capacity memory, host processors, other CGR processors, FPGA devices, and so on, that are coupled with the interfaces.

Each depicted CGR array has four AGCUs (e.g., MAGCU1, AGCU12, AGCU13, and AGCU14 in CGR array310). The AGCUs interface the TLN to the ALNs and route data from the TLN to the ALN or vice versa.

One of the AGCUs in each CGR array in this example is configured to be a master AGCU (MAGCU), which includes an array configuration load/unload controller for the CGR array. The MAGCU1 includes a configuration load/unload controller for CGR array310, and MAGCU2 includes a configuration load/unload controller for CGR array320. Some implementations may include more than one array configuration load/unload controller. In other implementations, an array configuration load/unload controller may be implemented by logic distributed among more than one AGCU. In yet other implementations, a configuration load/unload controller can be designed for loading and unloading configuration of more than one CGR array. In further implementations, more than one configuration controller can be designed for configuration of a single CGR array. Also, the configuration load/unload controller can be implemented in other portions of the system, including as a stand-alone circuit on the TLN and the ALN or ALNs.

The TLN is constructed using top-level switches (switch311, switch312, switch313, switch314, switch315, and switch316) coupled with each other as well as with other circuits on the TLN, including the AGCUs, and external I/O interface338. The TLN includes links (e.g., L11, L12, L21, L22) coupling the top-level switches. Data may travel in packets between the top-level switches on the links, and from the switches to the circuits on the network coupled with the switches. For example, switch311and switch312are coupled by link L11, switch314and switch315are coupled by link L12, switch311and switch314are coupled by link L13, and switch312and switch313are coupled by link L21. The links can include one or more buses and supporting control lines, including for example a chunk-wide bus (vector bus). For example, the top-level network can include data, request and response channels operable in coordination for transfer of data in any manner known in the art.

FIG.4illustrates an example CGR array400, including an array of CGR units in an ALN. CGR array400may include several types of CGR unit401, such as FCMUs, PMUs, PCUs, memory units, and/or compute units. For examples of the functions of these types of CGR units, see Prabhakar et al., “Plasticine: A Reconfigurable Architecture for Parallel Patterns”, ISCA2017, Jun. 24-28, 2017, Toronto, ON, Canada. Each of the CGR units may include a configuration store402comprising a set of registers or flip-flops storing configuration data that represents the setup and/or the sequence to run a program, and that can include the number of nested loops, the limits of each loop iterator, the instructions to be executed for each stage, the source of operands, and the network parameters for the input and output interfaces. In some implementations, each CGR unit401comprises an FCMU. In other implementations, the array comprises both PMUs and PCUs, or memory units and compute units, arranged in a checkerboard pattern. In yet other implementations, CGR units may be arranged in different patterns. The ALN includes switch units403(S), and AGCUs (each including two address generators405(AG) and a shared coalescing unit404(CU)). Switch units403are connected among themselves via interconnects421and to a CGR unit401with interconnects422. Switch units403may be coupled with address generators405via interconnects420. In some implementations, communication channels can be configured as end-to-end connections, and switch units403are CGR units. In other implementations, switches route data via the available links based on address information in packet headers, and communication channels establish as and when needed.

A configuration file may include configuration data representing an initial configuration, or starting state, of each of the CGR units that execute a high-level program with user algorithms and functions. Program load is the process of setting up the configuration stores in the CGR array based on the configuration data to allow the CGR units to execute the high-level program. Program load may also require loading memory units and/or PMUs.

The ALN includes one or more kinds of physical data buses, for example a chunk-level vector bus (e.g., 512 bits of data), a word-level scalar bus (e.g., 32 bits of data), and a control bus. For instance, interconnects421between two switches may include a vector bus interconnect with a bus width of 512 bits, and a scalar bus interconnect with a bus width of 32 bits. A control bus can comprise a configurable interconnect that carries multiple control bits on signal routes designated by configuration bits in the CGR array's configuration file. The control bus can comprise physical lines separate from the data buses in some implementations. In other implementations, the control bus can be implemented using the same physical lines with a separate protocol or in a time-sharing procedure.

Physical data buses may differ in the granularity of data being transferred. In one implementation, a vector bus can carry a chunk that includes 16 channels of 32-bit floating-point data or 32 channels of 16-bit floating-point data (i.e., 512 bits) of data as its payload. A scalar bus can have a 32-bit payload and carry scalar operands or control information. The control bus can carry control handshakes such as tokens and other signals. The vector and scalar buses can be packet-switched, including headers that indicate a destination of each packet and other information such as sequence numbers that can be used to reassemble a file when the packets are received out of order. Each packet header can contain a destination identifier that identifies the geographical coordinates of the destination switch unit (e.g., the row and column in the array), and an interface identifier that identifies the interface on the destination switch (e.g., North, South, East, West, etc.) used to reach the destination unit.

A CGR unit401may have four ports (as drawn) to interface with switch units403, or any other number of ports suitable for an ALN. Each port may be suitable for receiving and transmitting data, or a port may be suitable for only receiving or only transmitting data.

A switch unit, as shown in the example ofFIG.4, may have eight interfaces. The North, South, East and West interfaces of a switch unit may be used for links between switch units using interconnects421. The Northeast, Southeast, Northwest and Southwest interfaces of a switch unit may each be used to make a link with an FCMU, PCU or PMU instance using one of the interconnects422. Two switch units in each CGR array quadrant have links to an AGCU using interconnects420. The AGCU coalescing unit arbitrates between the AGs and processes memory requests. Each of the eight interfaces of a switch unit can include a vector interface, a scalar interface, and a control interface to communicate with the vector network, the scalar network, and the control network. In other implementations, a switch unit may have any number of interfaces.

During execution of a graph or subgraph in a CGR array after configuration, data can be sent via one or more switch units and one or more links between the switch units to the CGR units using the vector bus and vector interface(s) of the one or more switch units on the ALN. A CGR array may comprise at least a part of CGR array400, and any number of other CGR arrays coupled with CGR array400.

A data processing operation implemented by CGR array configuration may comprise multiple graphs or subgraphs specifying data processing operations that are distributed among and executed by corresponding CGR units (e.g., FCMUs, PMUs, PCUs, AGs, and CUs).

FIG.5illustrates an example500of a PMU510and a PCU520, which may be combined in an FCMU530. PMU510may be directly coupled to PCU520, or optionally via one or more switches. PMU510includes a scratchpad memory515, which may receive external data, memory addresses, and memory control information (write enable, read enable) via one or more buses included in the ALN. PCU520includes two or more processor stages, such as SIMD521through SIMD526, and configuration store528. The processor stages may include ALUs, or SIMDs, as drawn, or any other reconfigurable stages that can process data.

Each stage in PCU520may also hold one or more registers (not drawn) for short-term storage of parameters. Short-term storage, for example during one to several clock cycles or unit delays, allows for synchronization of data in the PCU pipeline.

FIG.6shows an example of a computation graph600. Computation graphs represent mathematical expressions, and comprise nodes and directed edges. InFIG.6, nodes are drawn as circles and directed edges are drawn as arrows. A node can represent a constant, a variable, for example from an input, an operation, an equation, or an output value. A directed edge can represent a dependency. Node610represents a variable A1, whose present value equals 12. Node611represents a variable A2, whose present value equals 251. Node612represents the constant π. Node613represents a multiplication operation. It receives its input data from node611via directed edge621and from node612via directed edge622. Node614represents an addition operation. Node614receives its input data from node610via directed edge620and from node613via directed edge623. Node614outputs its result in output node615via directed edge624. Computation graph600as a whole represents the equation Output=A1+pi*A2.

The depicted computation graph600is very simple and could be implemented electronically in many ways. For example, it could be hardwired as a circuit of digital gates in an application-specific IC (ASIC), or an FPGA could be configured to emulate the circuit of digital gates, or a CGR processor could be configured to perform the addition and multiplication functions, or a CPU could run a conventional computer program to perform the functions. In all implementations, the timing is important. Node614is not able to calculate a valid output value until all its input values are valid. That means node613must be finished first. Most digital circuits are implemented as pipelines of clocked stages. If the add operation of node614is in a later stage than the multiplication operation of node613, then a fixed-delay buffer may need to be inserted between node610and node614to synchronize the value of variable A1 with the result of the multiplication in node613. The fixed-delay buffer can be added to the graph to make it physically implementable.

Most computation graphs are a-cyclic, i.e., they don't include loops. One class of computation graphs, dataflow graphs, may include loops, and even nested loops. This can make delays of operations performed by nodes variable, dependent on the data flowing through a pipeline of operations. When a high-level program includes multiple pipelines of parallel, interdependent operations, then synchronization can become highly complex. Synchronization can be further complicated when directed edges are implemented as data channels in a network, since the data channels can become congested. A CGR processor, may resolve both problems by using dataflow control information, sent as messages from consuming nodes to producing nodes to indicate that the consuming node is ready to receive the information, and a credit token system that prevents congestion of the data channels between the producing and consuming nodes.

FIG.7shows an example of a dataflow graph700. This example, one head of a multi-head attention module in the Transformer model first published by Vaswani, et al., “Attention Is All You Need,” 31 st Conference on Neural Information Processing Systems, 2017, is well known in the industry. It includes a loop709within a loop710. Loop710includes four general matrix multiplications, GeMM702, GeMM712, GeMM722, and GeMM708. Loop709includes an ingress matrix multiplication GeMM703, mask fill node704, softmax node705, dropout node706, and egress matrix multiplication node707.

To physically implement dataflow graph700, an implementation may insert three types of stage buffers: (1) inter-stage buffers, (2) intra-stage buffers, and (3) interface buffers. The interface buffers are used because the granularity of communication (i.e., the size of tensors or data produced or consumed) varies between loops at different levels. Further, an implementation must add dataflow control information, to synchronize the various stages of asynchronous computation.

FIG.8shows the dataflow graph ofFIG.7with buffers and dataflow control information added. A compiler in the technology presented herein can create graph800from dataflow graph700, assign the nodes to compute units and memory units in a CGR array, and assign edges and dataflow control information to data channels in an array-level network that connects the compute units and memory units.

To get from dataflow graph700to graph800, one compiler implementation divides the dataflow graph in stages (stages 0, 1, and 2 are shown in this example), and where there are nested loops also in substages (substages 1.0 through 1.4 are shown). The implementation inserts buffers between the stages to allow for pipelined processing in one or more parallel meta-pipelines that may interact. The buffers are shown as blocks labeled A . . . L. They are different from buffers at the gate level, which may be single or double inverters used to boost the energy level of digital signals that need to travel through long wires or that need to drive high-capacitance loads, or which may be flipflops operated by a system clock and used to implement synchronous logic. The buffers at the meta-pipeline level may be memories, register files, shift registers, or first-in-first-out (FIFO) memories of fixed or variable length, storing one or more data items (e.g., scalars, vectors, or tensors). They may be clocked by a producer node to store data or by a consumer node to release data. They may further be controlled by dataflow control information coming from, for example, downstream nodes.FIG.8shows the same operation nodes asFIG.7(with like numbering), but the edges (solid arrows), where data flows, are interrupted by the buffers to partition the graph into stages, and dataflow control information is added (shown as dashed arrows for loop810and dash-dot arrows for loop809). In the example shown, data travels downstream (solid arrows from the left to the right) and dataflow control information travels upstream (dashed arrows from the right to the left).

In further preparation for a physical implementation of graph800, an implementation may assign each operation node to one or more logical compute units or memory units, and each buffer to one or more logical memory units. Some implementations may perform further preparations and optimizations. All implementations proceed to place and route, i.e., assign the logical units to physical units in a layout of a coarsely reconfigurable array, and (in some implementations) assign the data connections and the dataflow control information connections to data channels in the ALN in the CGR array.

FIG.9is a block diagram of a compiler stack900implementation suitable for generating a configuration file for a CGR processor. As depicted, compiler stack900includes several stages to convert a high-level program with user algorithms and functions, e.g., algebraic expressions and functions, to configuration data for the CGR units. Compiler stack900may take its input from application platform910, or any other source of high-level program statements suitable for parallel processing, which provides a user interface for general users. It may further receive hardware description915, for example defining the physical units in a reconfigurable data processor or CGRA processor. Application platform910may include libraries such as PyTorch, TensorFlow, ONNX, Caffe, and Keras to provide user-selected and configured algorithms. Application platform910outputs a high-level program to compiler920, which in turn outputs a configuration file to the reconfigurable data processor or CGRA processor where it is executed in runtime processes930. Compiler920may include dataflow graph compiler921, which may handle a dataflow graph, algebraic graph compiler922, template graph compiler923, template library924, and placer and router PNR925. In some implementations, template library924includes RDU abstract intermediate language (RAIL) and/or assembly language interfaces for power users.

Dataflow graph compiler921converts the high-level program with user algorithms and functions from application platform910to one or more dataflow graphs. The high-level program may be suitable for parallel processing, and therefore parts of the nodes of the dataflow graphs may be intrinsically parallel unless an edge in the graph indicates a dependency. Dataflow graph compiler921may provide code optimization steps like false data dependency elimination, dead-code elimination, and constant folding. The dataflow graphs encode the data and control dependencies of the high-level program. Dataflow graph compiler921may support programming a reconfigurable data processor at higher or lower-level programming languages, for example from an application platform910to C++ and assembly language. In some implementations, dataflow graph compiler921allows programmers to provide code that runs directly on the reconfigurable data processor. In other implementations, dataflow graph compiler921provides one or more libraries that include predefined functions like linear algebra operations, element-wise tensor operations, non-linearities, and reductions required for creating, executing, and profiling the dataflow graphs on the reconfigurable processors. Dataflow graph compiler921may provide an application programming interface (API) to enhance functionality available via the application platform910.

Algebraic graph compiler922may include a model analyzer and compiler (MAC) level that makes high-level mapping decisions for (sub-graphs of the) dataflow graph based on hardware constraints. It may support various application frontends such as Samba, JAX, and TensorFlow/HLO. Algebraic graph compiler922may also transform the graphs via autodiff and GradNorm, perform stitching between sub-graphs, interface with template generators for performance and latency estimation, convert dataflow graph operations to AIR operation, perform tiling, sharding (database partitioning) and other operations, and model or estimate the parallelism that can be achieved on the dataflow graphs.

Algebraic graph compiler922may further include an arithmetic or algebraic intermediate representation (AIR) level that translates high-level graph and mapping decisions provided by the MAC level into explicit AIR graphs. Key responsibilities of the AIR level include legalizing the graph and mapping decisions of the MAC, expanding data parallel, tiling, metapipe, region instructions provided by the MAC, inserting stage buffers and skip buffers, eliminating redundant operations, buffers and sections, and optimizing for resource use, latency, and throughput.

Template graph compiler923may translate AIR graphs into TLIR graphs, optimizing for the target hardware architecture and/or into unplaced units suitable for PNR925. Template graph compiler923may add further information (name, inputs, input names and dataflow description) for PNR925and make the graph physically realizable through each performed step. Template graph compiler923may for example provide translation of AIR graphs to specific model operation templates such as for general matrix multiplication (GeMM). An implementation may convert part or all intermediate representation operations to templates, stitch templates into the dataflow and control flow, insert necessary buffers and layout transforms, generate test data and optimize for hardware use, latency, and throughput.

Implementations may use templates for common operations. Templates may be implemented using assembly language, RAIL, or similar. RAIL is comparable to assembly language in that memory units and compute units are separately programmed, but it can provide a higher level of abstraction and compiler intelligence via a concise performance-oriented domain-specific language for CGR array templates. RAIL enables template writers and external power users to control interactions between logical compute units and memory units with high-level expressions without the need to manually program capacity splitting, register allocation, etc. The logical compute units and memory units also enable stage/register allocation, context splitting, transpose slotting, resource virtualization and mapping to multiple physical compute units and memory units (e.g., PCUs and PMUs).

Template library924may include an assembler that provides an architecture-independent low-level programming interface as well as optimization and code generation for the target hardware. Responsibilities of the assembler may include address expression compilation, intra-unit resource allocation and management, making a template graph physically realizable with target-specific rules, low-level architecture-specific transformations and optimizations, and architecture-specific code generation.

PNR925translates and maps logical (i.e., unplaced physically realizable) CGR units to the physical chip level (e.g., a physical array of CGR units), determines physical data channels to allow for communication among the CGR units and between the CGR units and circuits coupled via the TLN, allocates ports on the CGR units and switches, provides configuration data and initialization data for the target hardware, and produces configuration files, e.g., processor-executable format (PEF) files. It may further provide bandwidth calculations, allocate network interfaces such as AGCUs and virtual address generators (VAGs), provide configuration data that allows AGCUs and/or VAGs to perform address translation, and control ALN switches and data routing. PNR925may provide its functionality in multiple steps and may include multiple modules (not shown inFIG.9) to provide the multiple steps, e.g., a placer, a router, a port allocator, and a PEF file generator. PNR925may receive its input data in various ways. For example, it may receive parts of its input data from any of the earlier modules (dataflow graph compiler921, algebraic graph compiler922, template graph compiler923, and/or template library924). In some implementations, an earlier module, such as template graph compiler923, may have the task of preparing all information for PNR925and no other units provide PNR input data directly.

Further implementations of compiler920provide for an iterative process, for example by feeding information from PNR925back to an earlier module, so that the earlier module can execute a new compilation step in which it uses physically realized results rather than estimates of or placeholders for physically realizable circuits. For example, PNR925may feed information regarding the physically realized circuits back to algebraic graph compiler922.

Memory allocations represent the creation of logical memory spaces in on-chip and/or off-chip memories for data required to implement the dataflow graph, and these memory allocations are specified in the configuration file. Memory allocations define the type and the number of hardware circuits (functional units, storage, or connectivity components). Main memory (e.g., DRAM) may be off-chip memory, and scratchpad memory (e.g., SRAM) is on-chip memory inside an RDU. Other memory types for which the memory allocations can be made for various access patterns and layouts include cache, read-only look-up tables (LUTs), serial memories (e.g., FIFOs), and register files.

Compiler920binds memory allocations to unplaced memory units and binds operations specified by operation nodes in the dataflow graph to unplaced compute units, and these bindings may be specified in the configuration data. In some implementations, compiler920partitions parts of a dataflow graph into memory subgraphs and compute subgraphs, and specifies these subgraphs in the PEF file. A memory subgraph may comprise address calculations leading up to a memory access. A compute subgraph may comprise all other operations in the parent graph. In one implementation, a parent graph is broken up into multiple memory subgraphs and exactly one compute subgraph. A single parent graph can produce one or more memory subgraphs, depending on how many memory accesses exist in the original loop body. In cases where the same memory addressing logic is shared across multiple memory accesses, address calculation may be duplicated to create multiple memory subgraphs from the same parent graph.

Compiler920generates the configuration files with configuration data (e.g., a bit stream) for the placed positions and the routed data and control networks. In one implementation, this includes assigning coordinates and communication resources of the physical CGR units by placing and routing unplaced units onto the array of CGR units while maximizing bandwidth and minimizing latency.

A first example of accelerated deep learning is using a deep learning accelerator implemented in a CGRA to train a neural network. A second example of accelerated deep learning is using the deep learning accelerator to operate a trained neural network to perform inferences. A third example of accelerated deep learning is using the deep learning accelerator to train a neural network and subsequently perform inference with any one or more of the trained neural network, information from the trained neural network, and a variant of the same.

Examples of neural networks include fully connected neural networks (FCNNs), recurrent neural networks (RNNs), graph neural networks (GNNs), convolutional neural networks (CNNs), graph convolutional networks (GCNs), long short-term memory (LSTM) networks, autoencoders, deep belief networks, and generative adversarial networks (GANs).

An example of training a neural network is determining one or more weights associated with the neural network, such as by back-propagation in a deep learning accelerator. An example of making an inference is using a trained neural network to compute results by processing input data using the weights associated with the trained neural network. As used herein, the term ‘weight’ is an example of a ‘parameter’ as used in various forms of neural network processing. For example, some neural network learning is directed to determining parameters (e.g., through back-propagation) that are usable for performing neural network inferences.

A neural network processes data according to a dataflow graph comprising layers of neurons. Example layers of neurons include input layers, hidden layers, and output layers. Stimuli (e.g., input data) are received by an input layer of neurons and the computed results of the dataflow graph (e.g., output data) are provided by an output layer of neurons. Example hidden layers include rectified linear unit (ReLU) layers, fully connected layers, recurrent layers, graphical network layers, long short-term memory layers, convolutional layers, kernel layers, dropout layers, and pooling layers. A neural network may be conditionally and/or selectively trained. After being trained, a neural network may be conditionally and/or selectively used for inference.

Examples of ICs, or parts of ICs, that may be used as deep learning accelerators, are processors such as central processing units (CPUs), CGR processor ICs, graphics processing units (GrPUs), FPGAs, ASICs, application-specific instruction-set processor (ASIP), and digital signal processors (DSPs). The disclosed technology implements efficient distributed computing by allowing an array of accelerators (e.g., reconfigurable processors) attached to separate hosts to directly communicate with each other via buffers.

As previously described, different processor types may include a general processing unit (GPU), a reconfigurable dataflow unit (RDU), a host processor such as a central processing unit (CPU), etc. A GPU may be implemented in any suitable manner, such as, for example, an artificial intelligence processing unit (APU). Examples of APUs include a graphics processing unit (GrPU), a data processing unit (DPU), a streaming processing unit (SPU), a neural processing unit (NPU), a tensor processing unit (TPU), etc. Examples of APUs, which may be used to implement a GPU, include a graphics processing unit (GrPU), a data processing unit (DPU), a streaming processing unit (SPU), a neural processing unit (NPU), a tensor processing unit (TPU), etc. Each processor type may be configured to optimally execute different portions of an application. In addition, each processor may include accessible memory associated with that processor.

A host runtime processing system as described herein can dynamically select the most efficient data passing or access method for a given set of source and target devices (RDU, CPU and GPU) in an application defined pipeline (ADP) based on the latency & bandwidth requirement of the operating device stages. The host runtime processor hides the underlying details of methodologies of data passing from the application and picks between one of at least the three different methods, including two memory to memory transfer via a common buffer area of a CPU, one memory to memory transfer without using any buffer(s) on the CPU, and memory extension operation using direct memory access without data transfer as further described herein.

FIG.10is a simplified block diagram of a heterogeneous processing system1000according to one implementation. The heterogeneous processing system1000includes multiple processors (or processing units) coupled together via a switch and bus interface as further described herein. The illustrated processors include a host central processing unit (CPU)1002, a reconfigurable data flow unit (RDU)1004, and a general processing unit (GPU)1006, although alternative types of processors are contemplated. The RDU1004shown in the following description may be implemented according to that shown and described inFIGS.1-9, whereas the CPU1002represents or otherwise may be used to implement the host180previously described. A bus switch (BUS SW1)1008uses local buses1010,1012, and1014to operatively couple the CPU1002, the RDU1004, and the GPU1006together. The bus1010is a local host bus (HB), the bus1012is a local RDU bus (RB), and the bus1014is a local GPU bus (GB) for the respective processors. Examples of local buses1010,1012, and1014include Peripheral Component Interconnect (PCI) or PCI Express (PCIe), Cache Coherent Interconnect for Accelerators (CCIX), Compute Express Link (CXI, and Open Coherent Accelerator Processor Interface (OpenCAPI), although alternative bus or interface configurations that currently exist or that are newly developed may also be used.

The CPU1002, the RDU1004, and the GPU1006each include a local memory, shown as a host memory (HMEM)1016, an RDU memory (RMEM)1018, and a GPU memory (GMEM)1020, respectively. Each of the memories HMEM1016, RMEM1018, and GMEM1020may include storage resources for storing data and information including any combination of random-access memory (RAM), such as DRAM or SRAM or the like, read-only memory (ROM), latches, registers, and other types of memory devices. The heterogeneous processing system1000is shown as a logical configuration in which dashed lines are shown coupling the memories HMEM1016, RMEM1018, and GMEM1020, to the CPU1002, the RDU1004, and the GPU1006, respectively, for purposes of illustration. In an actual physical configuration, the memories may be incorporated within the respective processors, such as is the case for system-on-chip (SoC) configurations.

The CPU1002, the RDU1004, and the GPU1006may each be coupled to or otherwise incorporate a local data transfer resource, such as, for example, a direct memory access (DMA) engine, for transferring data between memory devices via the BUS SW11008. As shown, a DMA engine1022of the CPU1002enables data transfer between the HMEM1016and either the RMEM1018or the GMEM1020, a DMA engine1024of the RDU1004enables data transfer between the RMEM1018and either the HMEM1016or the GMEM1020, and a DMA engine1026of the GPU1006enables data transfer between the GMEM1020and either the HMEM1016or the RMEM1018. Each of the DMA engines1022,1024, and1026are logically shown as external to the corresponding processors using dashed lines, but may actually be incorporated within the respective processors.

In addition, each of the DMA engines1022,1024, and1026may incorporate different DMA engine types or different DMA channels that may be separately programmed and selected by the CPU1002depending upon the data transfer needs at any given time. For example, the different DMA engine types or different DMA channels of each DMA engine may include various bandwidth levels or priority levels or the like as understood by those of ordinary skill in the art.

The CPU1002programs any one or more of the DMA engines1022,1024, and1026with source and destination addresses for performing desired memory to memory data transfers, and then prompts the programmed DMA engine to perform the data transfer. Generally, DMA operations may be performed by each DMA engine independently from the corresponding processor for optimal performance once the DMA engine is programmed and prompted to perform the data transfer.

In various implementations, the CPU1002is a general-purpose processor suitable for performing host functions including execution of software programs. The CPU1002is shown including runtime logic1028for executing the programs and generally controlling operation of the heterogeneous processing system1000including programming the DMA engines1022,1024, and1026. The runtime logic1028may be configured to provide on-demand access to the RDU1004and the GPU1006. In one implementation, the runtime logic1028may include components such as a runtime library, tools, and a daemon service in a user space and a kernel module or driver in a kernel space. The CPU1002is shown including a compiler1030which receives one or more software applications and which generates corresponding configuration files for the applications that are adapted to be executed by a compute engine1032on the RDU1004. It is noted that the runtime logic1028and the compiler1030are conceptual illustrations implemented by the CPU1002while executing corresponding applications or software.

The GPU1006generally represents another processing unit type depending upon system needs in a given configuration, such as, for example, an artificial intelligence (AI) processing unit or any other generic processing type, although not limited to any particular processing type. Different types of AI processing units are contemplated, such as a graphics processing unit (GrPU), a streaming processing unit (SPU), a neural processing unit (NPU), a data processing unit (DPU), a tensor processing unit (TPU), etc.

The CPU1002and the GPU1006are each shown including an application programming interface (API), shown as API1034and API1038, respectively. It is noted that the illustrated APIs are also conceptual implementations performed by the corresponding processing unit when executing corresponding applications or software or the like. The APIs1034and1038may be configured to enable data transfer of between the different processors. Each API may be configured, for example, to perform address mapping for translating local or virtual addresses of a local processor to corresponding physical addresses used for transferring data. In those implementations in which the local buses1010,1012and1014are configured according to PCIe, the APIs may be used to map local virtual addresses to physical PCIe addresses. Although the RDU1004may also be configured with an API, the CPU1002via the compiler1030or the like may program the compute engine1032with appropriate addresses for accessing memories of other processors. Such addresses may be physical addresses used for accessing the HMEM1016and PCIe addresses or the like for accessing the GMEM1020.

FIG.11is a simplified timing diagram depicting overlapping batch operations in time that may be performed by the heterogeneous processing system1000. Each stage of a pipeline, including the computations on different processing units and the data passed between them, can overlap when working on different applications or different batches of data from the same application. As shown, three batch operations BATCH 1, BATCH 2, and BATCH 3 are mapped versus time, in which the batch operations 1-3 may overlap each other in time as shown. A batch operation may include multiple iterations or runs of the same application executed on multiple processors of the heterogeneous processing system1000.

The following notations are described for each of the batch operations 1-3. The notation “C→G” depicts a move of data from the memory HMEM1016of the CPU1002to the memory GMEM1020of the GPU1006. The notation “GC” depicts computation or processing by the GPU1006on the data just moved into the GMEM1020, which may generate output data stored back into the GMEM1020. The notation “G→C” depicts a move of data, such as the output data just generated by the GPU1006, from GMEM1020to HMEM1016. The notation “CC” depicts computation or processing by the CPU1002on the data just moved into the HMEM1016, which may generate output data stored back into the HMEM1016. The notation “C→R” depicts a move of data, such as the output data just generated by the CPU1002, from HMEM1016to the memory RMEM1018of the RDU1004. The notation “RC” depicts computation or processing by the RDU1004on the data just moved into the RMEM1018, which may generate output data stored back into the RMEM1018. The notation “R→C” depicts a move of data, such as the output data just generated by the RDU1004, from RMEM1018to HMEM1016.

The first batch operation BATCH 1 is shown proceeding without delay between each of the data transfer steps and subsequent processing steps. The data transfer C→G of BATCH 2 is initiated after completion of the data transfer C→G of BATCH 1 and is completed during the processing step GC of BATCH 1. The processing step GC of BATCH 2 may begin after completion of the processing step GC of BATCH 1. From that point in time forward, BATCH 2 may proceed without further delay between each of the data transfer steps and subsequent processing steps. The data transfer C→G of BATCH 3 is initiated after completion of the data transfer C→G of BATCH 2 and is completed during the processing step GC of BATCH 2. The processing step GC of BATCH 3 may begin after completion of the processing step GC of BATCH 2. From that point in time forward, BATCH 2 may proceed without further delay between each of the data transfer steps and subsequent processing steps.

FIG.12depicts the heterogeneous processing system1000in simplified form performing a batch operation1201of a corresponding application using two memory to memory transfer operation according to one implementation. Dashed-line arrows indicate data flow and circled numeric designations illustrate memory to memory data transfers during operation of the batch operation1201. In this case, the arrows and the circled numeric designations coincide with each other. The batch operation1201is similar to each of the batch operations BATCH 1-3 ofFIG.11except without the intermediate processing step CC by the CPU1002. Instead, the HMEM1016of the host system is used for transferring data between the other memories GMEM1020and RMEM1018. Preliminarily, the CPU1002may allocate buffer space1202in the HMEM1016, which may include either a single buffer or multiple buffers in different configurations or under different conditions. Such buffer allocation may be according to a preliminary operation or function before execution of the batch operation1201, or may be a dynamic operation in anticipation of or during operation of the batch operation1201.

In one configuration, the buffer space1202can be implemented as a DMA addressable single or common buffer that can be used to minimize data movement cost between the GMEM1020of the GPU1006or the RMEM1018of the RDU1004to the HMEM1016and vice-versa. A “common” buffer means that there are no intermediate buffer to buffer transfers performed within the HMEM1016during memory to memory transfers between the RMEM1018and the GMEM1020. In one configuration, the buffer space1202may be configured as a pinned memory buffer or the like meaning that it is not swappable or is not located within a swappable page of memory so that physical addresses are not changed during operation. Thus, the buffer space1202may be a single buffer located within a non-pageable or non-swappable portion of the HMEM1016. It is noted that the GPU1006and the RDU1004can access a single buffer in the HMEM1016by virtue of it being a common buffer.

Alternatively, the buffer space1202may be configured as multiple buffers. For example, the buffer space1202may include two buffers bufferA and bufferB (not shown) addressable to the CPU1002. The first buffer bufferA may be used to move the data from the GMEM1020memory to the HMEM1016, an additional memory copy may be performed to copy bufferA to bufferB, and the data in bufferB is moved to the RMEM1018. The opposite direction is also contemplated for moving data from the RMEM1018to the GMEM1020via the buffer space1202configured as the two buffers bufferA and bufferB. It is noted that multiple buffers may be used for overlapping batch operations. The CC operation shown inFIG.11may be an example of such conversion functions.

A preliminary data transfer “0” is shown moving initial data from the buffer space1202of the HMEM1016to the GMEM1020depicted in the batch operation1201as C→G. The GPU1006executes a first part of the application, labeled “GC” in the batch operation1201, while writing output data back into the GMEM1020. The CPU1002then prompts the DMA engine1026to perform a first memory to memory data transfer “1” of the output data from the GMEM1020to the buffer space1202in the HMEM1016, depicted in the batch operation1201as G→C.

The data that was transferred to the buffer space1202is intended to be used as input data for the next part of the application to be executed on the RDU1004. In one embodiment, the DMA engine1024(or the DMA engine1022) may be used to perform a second memory to memory data transfer “2” to transfer the data from the buffer space1202to the RMEM1018, which is depicted in the batch operation1201as C→R. It is noted that when the buffer space1202is configured as a common buffer as previously described, there is no need for buffer to buffer transfer between the memory data transfers “1” and “2”. Thus, the data that was transferred into the buffer space1202from the GMEM1020for the memory data transfer “1” remains unmoved until it is transferred to the RMEM1018for the memory transfer “2”. Alternatively, when the buffer space1202is configured as multiple buffers, there may be an intermediate transfer between two buffers within the buffer space1202. The RDU1004executes a second part of the application using the transferred data, labeled “RC” in the batch operation1201, while writing output data back into the RMEM1018. A final data transfer “3” may be performed, depicted in the batch operation1201as R→C, to transfer the output data stored in the RMEM1018back to the buffer space1202of the HMEM1016for evaluation or examination by the CPU1002.

FIG.12illustrates a serial data transfer operation in which the buffer space1202is used for each of the data transfers to and from the HMEM1016between the GMEM1020and the RMEM1018. In this manner, the DMA engine1026of the GPU1006may be programmed with addresses of the buffer space1202and the GMEM1020for performing both the preliminary data transfer “0” and the first data transfer “1”, and the DMA engine1024of the RDU1004may be programmed with addresses of the buffer space1202and the RMEM1018for performing both the second data transfer “2” and the final data transfer “3”. DMA programming by the CPU1002may be performed preliminarily, such as during initialization or other time before execution of an application or batch operations or the like, or may be dynamic, such as when prompted during operation of the application or batch operation.

When the data transfers are part of a pipelined data transfer operation, then additional buffers in the HMEM1016may be defined and used for the first and last data transfers, and the DMA engines1024and1026may be dynamically programmed and reprogrammed during operation. For example, the DMA engine1026may be dynamically programmed with addresses between a first additional buffer (not shown) within the HMEM1016for transferring initial data from the HMEM1016to the GMEM1020for the preliminary data transfer “0”, and then be dynamically reprogrammed with addresses of the buffer space1202for transferring data from the GMEM1020to the buffer space1202for the data transfer “1”. Meanwhile, the CPU1002may fill the first additional buffer with more data for continued pipelined operation, and then the DMA engine1026is dynamically programmed again for the next data transfer. Ideally, this is not limited to the first and last data transfers. For example, if buffer space1202is configured with multiple buffers (e.g., buffers A and B), then data transfer “1” between GMEM1020and HMEM1016on buffer A can be overlapped with data transfer “2” between HMEM1016on buffer B to RMEM1018.

Similarly, the DMA engine1024may be dynamically programmed with addresses between the buffer space1202and the RMEM1018for the data transfer “2”, and then be dynamically reprogrammed with addresses of a second additional buffer (not shown) within the HMEM1016for transferring final data from the RMEM1018to the second additional buffer for the last data transfer “3”. Then the DMA engine1024is dynamically programmed again with addresses of the buffer space1202for the next data transfer. The CPU1002retrieves data from the second additional buffer for evaluation before being filled again by the DMA engine1024. Even so, only the buffer space1202is used for data transfer between the GMEM1020and the RMEM1018.

The DMA engines1024and1026may each have multiple DMA channels in which additional DMA channels may be employed to streamline the pipelined data transfer operation. For example, a first DMA channel of each of the DMA engines1024and1026may be programmed once for performing the preliminary data transfer “0” and the last data transfer “3”, respectively, while additional DMA channels may be programmed to transfer data between the first and second additional buffers. In this case, a first DMA channel of the DMA engine1026is programmed either preliminarily or dynamically during operation to transfer data from the GMEM1020to the buffer space1202, and a first DMA channel of the DMA engine1024is programmed either preliminarily or dynamically during operation to transfer data from the buffer space1202to the RMEM1018. During data transfer operation, additional DMA channels are programmed to transfer data from the first additional buffer to the GMEM1020(data transfer “0”), and after processing by the GPU1006, to transfer data from the GMEM1020to the buffer space1202(data transfer 1). The DMA transfer engine1024then transfers data from the buffer space1202to the RMEM1018(data transfer “2”), and after processing by the RDU1004, the DMA engine1022, after having been reprogrammed, transfers data from the RMEM1018to the second additional buffer (data transfer 3). Operation repeats in this manner as long as necessary.

FIG.13is a flowchart diagram illustrating operation of the heterogeneous processing system1000performing the batch operation1201using two memory to memory transfer according to one implementation. Operation may begin with the CPU1002allocating the buffer space1202in the HMEM1016(1302). As previously described, the buffer space1202may include a common buffer or multiple buffers, and such buffer allocation may be preliminary or dynamic.

A preliminary data transfer may be performed from the HMEM1016to the GMEM1020for providing initial input data for the GPU1006, such as from the buffer space1202to the GMEM1020, denoted C→G in the batch operation1201. This initial data transfer may be performed by the DMA engine of the CPU1002or the GPU1006. For the configuration illustrated, the CPU1002dynamically programs the DMA engine1026with addresses of the GMEM1020and the buffer space1202for performing data transfers in either direction, and then prompts the DMA engine1026(1304) to perform the preliminary data transfer “0”. In addition, the DMA engine1026may be instructed to use addresses of the buffer space1202as source addresses and addresses of the GMEM1020as destination addresses for the immediate operation. The DMA engine1026is then prompted to perform the preliminary data transfer “0” C→G of the batch operation1201for providing initial input data for the GPU1006(1306).

The GPU1006then executes the first part of the application, denoted GC in the batch operation1201, using input data in the GMEM1020and writing output data in the GMEM1020(1308). The CPU1002may then prompt the DMA engine1026to perform the first memory to memory data transfer “1” from the GMEM1020to the buffer space1202, denoted G→C in the batch operation1201(1310). Since the DMA engine1026is already programmed for data transfers in either direction, it is instructed to use addresses of the GMEM1020as source addresses and addresses of the buffer space1202as destination addresses, and then to perform the data transfer G→C in the batch operation1201.

In order to perform the memory to memory data transfer “2” to transfer output data from the buffer space1202to the RMEM1018as input data for the RDU1004, denoted C→R in the batch operation1201, the CPU1002dynamically programs the DMA engine1024with addresses of the RMEM1018and the buffer space1202for performing data transfers in either direction, and then prompts the DMA engine1024(1312). In addition, the DMA engine1024may be instructed to use addresses of the buffer space1202as source addresses and addresses of the RMEM1018as destination addresses. The DMA engine1024then performs the data transfer “2” denoted C→R in the batch operation1201(1314).

The RDU1004then executes the second part of the application, denoted RC in the batch operation1201, using the input data transferred from the GMEM1020via the buffer space1202and stored in the RMEM1018, and writing output data back into the RMEM1018(1316). Finally, if desired, the DMA engine1024is prompted to transfer the output data generated by the RDU1004and stored in the RMEM1018to the buffer space1202in the HMEM1016for evaluation or examination by the CPU1002, which is data transfer “3” denoted R→C in the batch operation1201(1318), and operation is completed. Since already programmed, the DMA engine1024is instructed to use addresses of the RMEM1018as source addresses and addresses of the buffer space1202as destination addresses.

The buffer space1202in the HMEM1016may be implemented as DMA addressable memory that minimizes data movement cost when transferring data from the GMEM1020to the RMEM1018. Additional buffers may be used in the HMEM1016for pipelined data transfer operations as previously described, these additional buffers are used for input and output data for the CPU1002and not for memory to memory data transfer between the GMEM1020and the RMEM1018.

FIG.14depicts the heterogeneous processing system1000in simplified form performing a batch operation1401of a corresponding application using two memory to memory transfer operation according to one implementation. The batch operation1401is substantially similar to the batch operation1201except that data flows in the opposite direction. Again, dashed-line arrows indicate data flow and circled numeric designations illustrate memory to memory data transfers during operation of the batch operation1401, and again, the arrows and the circled numeric designations coincide with each other in this case. Again, the CPU1002may allocate the buffer space1202in the HMEM1016as a single buffer or as multiple buffers.

A preliminary data transfer “0” is shown moving initial data from the buffer space1202to the RMEM1018depicted in the batch operation1401as C→R. The RDU1004executes a first part of the application, labeled “RC” in the batch operation1401, while writing output data back in the RMEM1018. The CPU1002then prompts the DMA engine1024to perform a first memory to memory data transfer “1” of the output data from the RMEM1018to the buffer space1202in the HMEM1016, depicted in batch operation1301as R→C.

The data that was transferred to buffer space1202is intended to be used as input data for the next part of the application to be executed on the GPU1006. In this case, the DMA engine1026is used to perform a second memory to memory data transfer “2” to retrieve the data from the buffer space1202to the GMEM1020, which is depicted in batch operation1401as C→G. Again, it is noted that when the buffer space1202is configured as a common buffer as previously described, there is no need for buffer to buffer transfer between the memory data transfers “1” and “2”. Thus, the data that was transferred into the buffer space1202from the RMEM1018for the memory data transfer “1” remains unmoved until it is transferred to the GMEM1020for the memory transfer “2”. Alternatively, when the buffer space1202is configured as multiple buffers, there may be an intermediate transfer between two buffers within the buffer space1202. The GPU1006executes a second part of the application using the transferred data, labeled “GC” in the batch operation1401, while writing output data back into the GMEM1020. A final data transfer “3” may be performed, depicted in batch operation1401as G→C, to transfer the output data stored in the GMEM1020back to the HMEM1016for evaluation or examination by the CPU1002.

FIG.14illustrates a serial data transfer operation in which the buffer space1202is used for each of the data transfers to and from the HMEM1016between the RMEM1018and the GMEM1020. In this manner, the DMA engine1024of the RDU1004may be programmed with addresses of the buffer space1202and the RMEM1018for performing both the preliminary data transfer “0” and the first data transfer “1”, and the DMA engine1026of the GPU1006may be programmed with addresses of the buffer space1202and the GMEM1020for performing both the second data transfer “2” and the final data transfer “3”. Again, DMA programming may be performed preliminarily, such as during initialization or other time before execution of an application or batch operations or the like, or may be dynamic, such as when prompted during operation of the application or batch operation.

When the data transfers are part of a pipelined data transfer operation, then separate buffers in the HMEM1016may be defined and used for the first and last data transfers, and the DMA engines1024and1026may be dynamically programmed and reprogrammed during operation in a similar manner as previously described for the batch operation1201. In the alternative, the DMA engine1022may be employed to streamline the pipelined data transfer operation, in which both of the DMA engines1024and1026are programmed only once while the DMA engine1022is dynamically programmed during operation between the first and second additional buffers for performing the preliminary data transfer “0” and the last data transfer “3” in a similar manner as previously described for the batch operation1201.

FIG.15is a flowchart diagram illustrating operation of the heterogeneous processing system1000performing the batch operation1401using two memory to memory transfer operation according to one implementation. Initial operations are substantially similar as described for the batch operation1201. Operation may begin with the CPU1002allocating the buffer space1202in the HMEM1016(1502). As previously described, the buffer space1202may include a common buffer or multiple buffers, and such buffer allocation may be preliminary or dynamic.

A preliminary data transfer may be performed from the HMEM1016to the RMEM1018for providing initial input data for the RDU1004, such as from the buffer space1202to the RMEM1018, denoted C→R in the batch operation1401. This initial data transfer may be performed by the DMA engine of the CPU1002or of the RDU1004. For the configuration illustrated, the CPU1002dynamically programs the DMA engine1024with addresses of the RMEM1018and the buffer space1202for performing data transfers in either direction, and then prompts the DMA engine1024(1504) to perform the preliminary data transfer “0”. In addition, the DMA engine1026may be instructed to use addresses of the buffer space1202as source addresses and addresses of the GMEM1020as destination addresses for the immediate operation. The DMA engine1024is then prompted to perform the preliminary data transfer “0” C→R in the batch operation1401for providing initial input data for the RDU1004(1506).

The RDU1004then executes the first part of the application, denoted RC in the batch operation1401, using input data from the RMEM1018and writing output data in the RMEM1018(1508). The CPU1002may then prompt the DMA engine1024to perform the first memory to memory data transfer “1” of the output data just generated and stored in the RMEM1018to the buffer space1202, denoted R→C in the batch operation1401(1510). Since the DMA engine1024is already programmed for data transfers in either direction, it is instructed to use addresses of the RMEM1018as source addresses and addresses of the buffer space1202as destination addresses, and then to perform the data transfer R→C in the batch operation1401.

In order to perform the memory to memory data transfer “2” to transfer output data from the buffer space1202to the GMEM1020as input data for the GPU1006, denoted C→G in the batch operation1401, the CPU1002dynamically programs the DMA engine1026with addresses of the GMEM1020and the buffer space1202for performing data transfers in either direction, and then prompts the DMA engine1026(1512). In addition, the DMA engine1026may be instructed to use addresses of the buffer space1202as source addresses and addresses of the GMEM1020as destination addresses. The DMA engine1026then performs the data transfer “2” denoted C→G in the batch operation1401(1514).

The GPU1006then executes the second part of the application, denoted GC in the batch operation1401, using the input data transferred from the GMEM1020via the buffer space1202and stored in the GMEM1020, and writing output data back into the GMEM1020(1516). Finally, if desired, the DMA engine1026is prompted to transfer the output data generated by the GPU1006and stored in the GMEM1020to the buffer space1202in the HMEM1016for evaluation or examination by the CPU1002(1518), which is data transfer “3” denoted G→C in the batch operation1401, and operation is completed. Since already programmed, the DMA engine1026is instructed to use addresses of the GMEM1020as source addresses and addresses of the buffer space1202as destination addresses.

As with the batch operation1201, the buffer space1202in the HMEM1016reduces the number of memory to memory data for the batch operation1401when transferring data from the RMEM1018to the GMEM1020. Although additional buffers may be used in the HMEM1016for pipelined data transfer operations as previously described, these additional buffers are used for input and output data for the CPU1002and not for memory to memory data transfer between the GMEM1020and the RMEM1018.

FIG.16depicts the heterogeneous processing system1000in simplified form performing a batch operation1601of a corresponding application using one memory to memory transfer operation according to one implementation. Again, dashed-line arrows indicate data flow and circled numeric designations illustrate memory to memory data transfers during operation of the batch operation1601. Also, the arrows and circled numeric designations coincide. The CPU1002or the GPU1006may perform a preliminary data transfer “0” moving initial data from the HMEM1016to the GMEM1020depicted in the batch operation1601as C→G. The GPU1006executes a first part of the application depicted as “GC” in the batch operation1601, generating output data in the GMEM1020. In this case, the virtual addressable memory of the output data in GMEM1020is mapped to physical PCIe addresses.

In order to perform the first memory to memory transfer “1” of the data from the GMEM1020directly to the RMEM1018, depicted in batch operation1601as G→R, the CPU1002dynamically programs the DMA engine1024to directly transfer data between the mapped physical PCIe addresses to corresponding addresses in the RMEM1018. The CPU1002then prompts the DMA engine1024to perform the first memory to memory transfer “1” of the data from the GMEM1020directly to the RMEM1018, depicted in batch operation1601as G→R. The RDU1004executes a second part of the application using the transferred data as input data, labeled “RC” in the batch operation1601, generating output data which is stored in the RMEM1018. A final data transfer “2” may be performed by the RDU1004or the CPU1002, depicted in batch operation1601as R→C, to transfer the output data stored in the RMEM1018back to the HMEM1016for evaluation or examination by the CPU1002.

FIG.17is a flowchart diagram illustrating operation of the heterogeneous processing system1000performing the batch operation1601using one memory to memory transfer operation according to one implementation. Operation may begin by mapping addresses of the GMEM1020to physical PCIe addresses (1702). As previously described, the virtual addressable memory of the output data in GMEM1020is mapped to physical PCIe addresses. In one implementation, only that portion of the GMEM1020for writing output data from the GPU1006is mapped.

The DMA engine of the CPU1002or the GPU1006may first perform a preliminary data transfer “0” to the GMEM1020for providing initial input data for the GPU1006, denoted C→G in the batch operation1601(1704). The GPU1006executes the first part of the application, denoted “GC” in the batch operation1601, using the input data stored in the GMEM1020and generates output data stored in the GMEM1020(1706). In order to perform the first memory to memory data transfer “1” from the GMEM1020to the RMEM1018denoted G→R in the batch operation1601, the CPU1002dynamically programs the DMA engine1024using mapped PCIe addresses and addresses of RMEM1018, and prompts the DMA engine1024to perform the first memory to memory data transfer “1” using the mapped PCIe addresses as source addresses and addresses of the RMEM1018as destination addresses (1708). In response, the DMA engine1024performs the first memory to memory data transfer “1” G→R from the GMEM1020using the mapped PCIe addresses to the RMEM1018(1710).

The RDU1004then executes the second part of the application, denoted “RC” in the batch operation1601, using the input data from the RMEM1018transferred from the GPU1006and stores output data in the RMEM1018(1712). Finally, if desired, output data generated by the RDU1004and stored in the RMEM1018may be transferred back to the HMEM1016for evaluation or examination by the CPU1002(1714), which is memory to memory data transfer “2” denoted R→C in the batch operation1601, and operation is completed.

FIG.18depicts the heterogeneous processing system1000in simplified form performing a batch operation1801of a corresponding application using one memory to memory transfer operation according to one implementation. The batch operation1801is substantially similar to the batch operation1601except that data flows in the opposite direction. Again, dashed-line arrows indicate data flow and circled numeric designations illustrate memory to memory data transfers during operation of the batch operation1801. Also, the arrows and circled numeric designations coincide. The DMA engine of the CPU1002or the RDU1004may perform a preliminary data transfer “0” moving initial input data from the HMEM1016to the RMEM1018depicted in the batch operation1801as C→R. The RDU1004executes a first part of the application depicted as “RC” in the batch operation1801, generating output data in the RMEM1018. Again, the virtual addressable memory of the output data in GMEM1020is mapped to physical PCIe addresses.

In order to perform the first memory to memory transfer “1” of the data from the RMEM1018directly to the GMEM1020, depicted in batch operation1801as R→G, the CPU1002dynamically programs the DMA engine1024to directly transfer data between the RMEM1018and the mapped physical PCIe addresses. The CPU1002prompts the DMA engine1024to perform the first memory to memory transfer “1” of the data from the RMEM1018directly to the GMEM1020, depicted in batch operation1801as R→G. The GPU1006executes a second part of the application using the transferred data as input data, labeled “GC” in the batch operation1801, generating output data which is stored in the GMEM1020. A final data transfer “2” may be performed, depicted in batch operation1801as G→C, to transfer the output data stored in the GMEM1020back to the HMEM1016for evaluation or examination by the CPU1002. The final transfer may be performed by the DMA engine of the CPU1002or of the GPU1006.

FIG.19is a flowchart diagram illustrating operation of the heterogeneous processing system1000performing the batch operation1801using one memory to memory transfer operation according to one implementation of the present disclosure. Operation may begin by mapping addresses of the GMEM1020to physical PCIe addresses (1902). As previously described, the virtual addressable memory of the GMEM1020is mapped to physical PCIe addresses. In one implementation, only that portion of the GMEM1020for writing data is mapped.

The DMA engine of the CPU1002or the RDU1004may first perform a preliminary data transfer “0” to the RMEM1018for providing initial input data for the RDU1004, denoted C→R in the batch operation1801(1904). The RDU1004executes the first part of the application, denoted “RC” in the batch operation1801, using the input data stored in the RMEM1018and generates output data stored in the RMEM1018(1906). In order to perform the first memory to memory data transfer “1” from the RMEM1018to the GMEM1020denoted R→G in the batch operation1801, the CPU1002dynamically programs the DMA engine1024using mapped PCIe addresses and addresses of RMEM1018, and prompts the DMA engine1024to perform the first memory to memory data transfer “1” using the mapped PCIe addresses as source addresses and addresses of the RMEM1018as destination addresses (1908). In response, the DMA engine1024performs the first memory to memory data transfer “1” R→G from the RMEM1018to the GMEM1020using the mapped PCIe addresses (1910).

The GPU1006then executes the second part of the application, denoted “GC” in the batch operation1801, using the input data from the GMEM1020transferred by the RDU1004and stores output data in the GMEM1020(1912). Finally, if desired, output data generated by the GPU1006and stored in the GMEM1020may be transferred back to the HMEM1016for evaluation or examination by the CPU1002(1914), which is memory to memory data transfer “2” denoted G→C in the batch operation1801, and operation is completed.

The memory extension achieved by addressable memory mapping enables the RDU1004to directly access memory of other processors or accelerators or the like, such as, for example, the GMEM1020of the GPU1006. The phrase “directly access” as used herein means to directly read or to directly write depending upon the memory operation being performed. In this manner, the RDU1004can directly read input data from and write output data to addressable memory of the GPU1006. In such a heterogeneous processing system1000, the memory extension of the RDU1004enables users to avoid any explicit host-initiated data transfer operations between the RDU1004and the GPU1006. Memory to memory data transfers using the host memory HMEM1016are avoided.

FIG.20depicts the heterogeneous processing system1000in simplified form performing a batch operation2001of a corresponding application using memory extension operation according to one implementation. Again, dashed-line arrows indicate data flow and circled numeric designations illustrate memory to memory data transfers during operation of the batch operation2001. In this case, however, the arrows and circled numeric designations do not coincide because memory to memory data transfers between GMEM1020and RMEM1018are effectively eliminated. Instead, the memory extension operation enables direct memory access (direct read or direct write). Preliminary and Post data transfers to and from the host memory HMEM106may still be used.

The DMA engine of the CPU1002or the GPU1006may perform a preliminary data transfer “0” moving initial input data from the HMEM1016to the GMEM1020depicted in the batch operation2001as C→G. The GPU1006executes a first part of the application, depicted as “GC” in the batch operation2001, using input data in the GMEM1020and generating output data also stored in the GMEM1020. As with the batch operation1801, the virtual addressable memory of the output data in GMEM1020is mapped to physical PCIe addresses for the batch operation2001. In this case, however, the runtime logic1028of the CPU1002configures the compute engine1032of the RDU1004to use the mapped PCIe physical addresses to directly read the output data stored in the GMEM1020. The RDU1004executes a second part of the application directly reading the output data stored in the GMEM1020as shown by arrow2003from GMEM1020to the RDU1004.

In one implementation, the output data generated by the RDU1004may be written into the RMEM1018in a similar manner as previously described for previous batch operations as indicated by arrow2004. In another implementation, the runtime logic1028of the CPU1002may additionally program the compute engine1032of the RDU1004to directly write the output data into the HMEM1016as indicated by arrow2005. It is noted that the virtual addressable memory of where the data is to be located in the HMEM1016may be mapped to physical PCIe addresses (as part of the initial step). Since the RDU1004directly reads data from the memory (e.g., GMEM1020) of one processor (e.g., the GPU1006) and directly writes the data to the memory (e.g., HMEM1016) of another processor (e.g., the CPU1002), memory to memory data transfers are entirely avoided. The combined operation of execution by the RDU1004and direct storage of output data into the HMEM1016is depicted as “RC: G→R & R→C” in the batch operation2001.

FIG.21is a flowchart diagram illustrating operation of the heterogeneous processing system1000performing the batch operation2001using memory extension operation according to one implementation. Operation may begin by mapping addresses of the GMEM1020to physical PCIe addresses (2102). As previously described, the virtual addressable memory of the output data in GMEM1020is mapped to physical PCIe addresses. In one implementation, only that portion of the GMEM1020for writing output data from the GPU1006is mapped. Also, as part of the initial step, it is noted that the virtual addressable memory of where the data is to be located in the HMEM1016may be mapped to a physical PCIe addresses. The DMA engine of the CPU1002or the GPU1006may first perform a preliminary data transfer “0” to the GMEM1020for providing initial input data for the GPU1006(2104), denoted C→G in the batch operation2001. The GPU1006executes the first part of the application, denoted “GC” in the batch operation2001, using the input data from the GMEM1020and writes output data in the GMEM1020(2106).

The CPU1002dynamically configures the compute engine1032of the RDU1004to read directly from the GMEM1020using the mapped PCIe addresses (2108); in addition, the RDU1004may be programmed to directly write output data into the HMEM1016(2108). Alternatively, the RDU1004may be programmed to write the output data into the RMEM1018as previously described. The programming of the RDU1004by the CPU1002may occur at any time before actual data transfer, including, for example, while the GPU1006is executing the first part of the application (GC). The RDU1004then executes the second part of the application, denoted “RC: G→R & R→C” in the batch operation2001, including directly reading the data from the GMEM1020using the mapped PCIe addresses (2110) and directly writing output data generated by the RDU1004into the HMEM1016using programmed addresses (2110), and operation is completed.

FIG.22depicts the heterogeneous processing system1000in simplified form performing a batch operation2201of a corresponding application using memory extension operation according to one implementation. The batch operation2201is substantially similar to the batch operation2001except in the reverse direction. Again, dashed-line arrows indicate data flow and circled numeric designations illustrate memory to memory data transfers during operation of the batch operation2201, and again, the arrows and circled numeric designations do not coincide because memory to memory data transfers are effectively eliminated (with the possible exception of an initial CPU data transfer).

As with previously described batch operations, the virtual addressable memory of the output data in GMEM1020is mapped to physical PCIe addresses for the batch operation2201including mapping of the HMEM106to physical PCIe addresses. In this case, the runtime logic1028of the CPU1002programs the compute engine1032of the RDU1004to directly read from the HMEM1016as indicated by arrow2203, and to directly write the output data into the GMEM1020using the mapped PCIe physical addresses as indicated by arrow2205, while executing a first part of the application, collectively denoted “RC: C→R & R→G” in the batch operation2201. Alternatively, the RDU1004may be configured to read input data from the RMEM1018as indicated by arrow2204(which may include a preliminary data transfer from HMEM1016into RMEM1018). The GPU1006executes a second part of the application reading input data in the GMEM1020stored as the output data from the RDU1004, and writes output data in the GMEM1020. A final data transfer operation “0” may be performed to write the output data to the HMEM1016for examination by the CPU1002.

FIG.23is a flowchart diagram illustrating operation of the heterogeneous processing system1000performing the batch operation2201using memory extension operation according to one implementation of the present disclosure. Operation may begin by mapping addresses of the GMEM1020to physical PCIe addresses (2302). As previously described, the virtual addressable memory of the output data in GMEM1020is mapped to physical PCIe addresses. In one implementation, only that portion of the GMEM1020used for writing output data from the RDU1004is mapped. Also, as part of the initial step, it is noted that the virtual addressable memory of where the data is to be retrieved from the HMEM1016may be mapped to a physical PCIe addresses. Also, in one embodiment, only that portion of the HMEM1016need be mapped.

The CPU1002dynamically configures the compute engine1032of the RDU1004to read directly from the HMEM1016and to write directly to the GMEM1020using the mapped PCIe addresses (2304). The RDU1004executes the first part of the application reading input data from the HMEM1016and writing output data stored in the GMEM1020(2306), denoted as “RC: C→R & R→G” in the batch operation2201. Alternatively, the input data may be read from the RMEM1018as previously described. The GPU1006then executes the second part of the application, denoted “GC”, using input data stored in the GMEM1020that was transferred by the RDU1004and writing output data in the GMEM1020(2308). Finally, if desired, output data generated by the GPU1006and written into the GMEM1020may be transferred back to the HMEM1016for examination or evaluation by the CPU1002, which is denoted G→C in the batch operation2101as data transfer “0” (2310), and operation is completed.

The memory transfer achieved by addressable memory mapping as illustrated inFIGS.20-23enables the compute engine1032of the RDU1004to directly access (directly read from or directly write to) memory of other processors or accelerators or the like, such as, for example, the GMEM1020of the GPU1006or the HMEM1016of the CPU1002. In this manner, the RDU1004can directly read input data from or directly write output data to addressable memory of the GPU1006, or the RDU1004can directly read input data from or directly write output data to addressable memory of the CPU1002. In such a heterogeneous processing system1000, the memory extension of the RDU1004enables users to avoid any explicit host-initiated data transfer operations between the RDU1004and the GPU1006.

FIG.24is a simplified timing diagram depicting overlapping batch operations in time that may be performed by the heterogeneous processing system1000, in which each batch operation uses a two memory to memory transfer operation method. As shown, three overlapping batch operations BATCH 1, BATCH 2, and BATCH 3 are mapped versus time, each substantially the same as the batch operation1201previously described using the same notation (“C→G” depicts a move of data from HMEM1016to GMEM1020, “GC” depicts processing by the GPU1006, “G→C” depicts a move of data from GMEM1020to HMEM1016, “C→R” depicts a move of data from HMEM1016to RMEM, “RC” depicts computation or processing by the RDU1004, and “R→C” depicts a move of data from RMEM1018to HMEM1016).

The first batch operation BATCH 1 is shown proceeding without delay between each of the data transfer steps and subsequent processing steps. The data transfer C→G of BATCH 2 is initiated after completion of the data transfer C→G of BATCH 1 and is completed during the processing step GC of BATCH 1. The processing step GC of BATCH 2 begins after completion of the processing step GC of BATCH 1, followed by G→C and C→R, and the processing step RC of BATCH 2 begins after completion of the processing step RC of BATCH 1, which is then followed by R→C. The data transfer C→G of BATCH 3 is initiated after completion of the data transfer C→G of BATCH 2. The processing step GC of BATCH 3 begins after completion of the processing step GC of BATCH 2, followed by G→C and C→R, and the processing step RC of BATCH 3 begins after completion of the processing step RC of BATCH 2, which is then followed by R→C. Operation of overlapping batch operations each similar to the batch operation1401is substantially similar but in the opposite direction.

The notations B1_0, B2_0, and B3_0 denote the data transfer operation C→G of the batch operations BATCH 1, 2, and 3, respectively, the notations B1_1, B2_1, and B3_1 denote the data transfer operation G→C of the batch operations BATCH 1, 2, and 3, respectively, the notations B1_2, B2_2, and B3_2 denote the data transfer operation C→R of the batch operations BATCH 1, 2, and 3, respectively, and the notations B1_3, B2_3, and B3_3 denote the data transfer operation R→C of the batch operations BATCH 1, 2, and 3, respectively.

FIG.25is a simplified block diagram illustrating buffer utilization within the memories HMEM1016, RMEM1018, and GMEM1020for performing the batch operations BATCH 1 and BATCH 2 ofFIG.24. Although not shown, BATCH 2 is handled in similar manner. Two separate buffers BUF1 and BUF2 are allocated in each memory for the corresponding batch operations BATCH 1 and BATCH 2, respectively. An arrow labelled B1_0 denotes transfer of data into BUF1 of GMEM1020, such as from a memory location (not shown) of the HMEM1016. The GPU1006performs GC on the data in BUF1, and may store updated data back into BUF1 of GMEM1020. An arrow labelled B2_0 denotes transfer of data into BUF2 of GMEM1020, such as from another memory location (not shown) of the HMEM1016, in which the data transfer B2_0 is performed after B1_0 as shown inFIG.24. The GPU1006performs GC on the data in BUF2 after performing GC on the data in BUF1, and may store updated data back into BUF2 of GMEM1020. An arrow labelled B1_1 denotes transfer of updated data from BUF1 of GMEM1020into BUF1 of HMEM1016, and then another arrow labelled B1_2 denotes transfer of the data in BUF1 of HMEM1016into BUF1 of RMEM1018. In similar manner, an arrow labelled B2_1 denotes transfer of updated data from BUF2 of GMEM1020into BUF2 of HMEM1016, and then another arrow labelled B2_2 denotes transfer of the data in BUF2 of HMEM1016into BUF2 of RMEM1018. The RDU1004performs RC on the data in BUF1 of RMEM1018, and may store updated data back into BUF1 of RMEM1018. Then RDU performs RC on the data in BUF2 of RMEM1018, and may store updated data back into BUF2 of RMEM1018. Arrows labelled B1_3 and B2_3 denotes transfer of updated data from BUF1 and BUF2 of RMEM1018to other memory locations (not shown) of HMEM1016. It is noted that in each case, the input and output regions can be shared if so determined by the compiler or if the sizes are the same, or may be stored in mutually exclusive locations if so determined by the compiler or if the sizes are different.

FIG.26is a simplified timing diagram depicting overlapping batch operations in time that may be performed by the heterogeneous processing system1000, in which each batch operation uses a one memory to memory transfer operation method. As shown, three overlapping batch operations BATCH 1, BATCH 2, and BATCH 3 are mapped versus time, each substantially the same as the batch operation1601previously described using the same notation. Operation is substantially similar to the overlapping batch operations shown inFIG.24, except that the two data transfer operations G→C and C→R are replaced by one data transfer operation G→R in each of the batch operations BATCH 1-3. This is because the intermediate buffer transfer using the HMEM1016is bypassed as shown inFIG.16. Operation of overlapping batch operations each similar to the batch operation1801is substantially similar but in the opposite direction.

FIG.27is a simplified timing diagram depicting overlapping batch operations in time that may be performed by the heterogeneous processing system1000, in which each batch operation is performed using memory extension operation. As shown, three overlapping batch operations BATCH 1, BATCH 2, and BATCH 3 are mapped versus time, each substantially the same as the batch operation2001previously described using the same notation. As previously described, the intermediate data transfer operations between the memories GMEM1020and RMEM1018are avoided since the RDU1004is configured to directly read data from the memory (e.g., GMEM1020) of one processor (e.g., the GPU1006) and may further directly write the data to the memory (e.g., HMEM1016) of another processor (e.g., the CPU1002). GC performed by the GPU1006for each subsequent batch operation is only delayed by GC of the previous batch operation. Also, the combined RC and data access functions performed by the RDU1004for each subsequent batch operation is only delayed by the combined RC and data transfer functions performed by the RDU1004for the prior batch operation. Operation of overlapping batch operations each similar to the batch operation2201is substantially similar but in the opposite direction.

FIG.28is a flowchart diagram illustrating a process performed by the CPU1002during operation of the heterogeneous processing system1000for determining the most efficient data access method according to one implementation. The CPU1002detects an application that is to be executed, or that is currently executing, on both the RDU1004and the GPU1006(2802). Generally, this means that a part of the application is performed on the RDU1004and another part on the GPU1006in either order (e.g., first part of application executed on the RDU1004and second part on the GPU1006, or vice-versa). In addition, this provides an indication that data may need to be accessed or transferred between RDU1004and GPU1006in either direction, or between the corresponding memories GMEM1020and the RMEM1018in either direction, or between a memory of one processor to the other processor and vice-versa. Such indication of data access may be provided by commands or instructions or the like provided by, or from applications executing on, any one of the processing units including the CPU1002, the RDU1004, or the GPU1006, or indications by any one of the data transfer devices such as the DMA engines1022,1024, and1026.

The CPU1002then determines the most efficient data access method to pass data based on current latency and bandwidth requirements and the like (2804). In one configuration, the CPU1002uses user-defined metadata of data passing, such as the source and destination device types, the memory addresses on both sides of the data passing, the bandwidth of the application defined pipeline and latency requirement of operation device stages, the number of configured common or multiple buffers of the buffer space1202, among other operating parameters, to determine the optimal data passing method of the input or output tensor. The available data access methods have been previously described, including the two memory to memory transfer operation, the one memory to memory transfer operation, or the memory extension operation (without data transfer).

The CPU1002then configures the heterogeneous processing system1000according to the most efficient method to orchestrate and complete memory passing between source and target devices. Thus, the CPU1002configures the heterogeneous processing system1000according to the two memory to memory transfer operation when determined as most efficient (2806). Alternatively, the CPU1002configures the heterogeneous processing system1000according to the one memory to memory transfer operation when determined as most efficient (2808). Alternatively, the CPU1002configures the heterogeneous processing system1000according to memory extension operation when determined as most efficient (2810). Once configured, the heterogeneous processing system1000proceeds to pass data when prompted by the application according to the configured data access operation method (2812) and operation is completed.

Further or Additional Considerations

We describe various implementations of a method and apparatus for accessing or transferring data between accessible memories of multiple processors in a heterogeneous processing system.

The technology disclosed can be practiced as a system, method, or article of manufacture. One or more features of an implementation can be combined with the base implementation. Implementations that are not mutually exclusive are taught to be combinable. One or more features of an implementation can be combined with other implementations. This disclosure periodically reminds the user of these options. Omission from some implementations of recitations that repeat these options should not be taken as limiting the combinations taught in the preceding sections—these recitations are hereby incorporated forward by reference into each of the implementations described herein.

Although the description has been described with respect to particular implementations thereof, these particular implementations are merely illustrative, and not restrictive. The description may reference specific structural implementations and methods and does not intend to limit the technology to the specifically disclosed implementations and methods. The technology may be practiced using other features, elements, methods and implementations. Implementations are described to illustrate the present technology, not to limit its scope, which is defined by the claims. Those of ordinary skill in the art recognize a variety of equivalent variations on the description above.

All features disclosed in the specification, including the claims, abstract, and drawings, and all the steps in any method or process disclosed, may be combined in any combination, except combinations where at least some of such features and/or steps are mutually exclusive. Each feature disclosed in the specification, including the claims, abstract, and drawings, can be replaced by alternative features serving the same, equivalent, or similar purpose, unless expressly stated otherwise.

Although the description has been described with respect to particular implementations thereof, these particular implementations are merely illustrative, and not restrictive. For instance, many of the operations can be implemented in a CGRA system, a System-on-Chip (SoC), application-specific integrated circuit (ASIC), programmable processor, in a programmable logic device such as a field-programmable gate array (FPGA) or a graphics processing unit (GrPU), obviating a need for at least part of the dedicated hardware. Implementations may be as a single chip, or as a multi-chip module (MCM) packaging multiple semiconductor dies in a single package. All such variations and modifications are to be considered within the ambit of the present disclosed technology the nature of which is to be determined from the foregoing description.

One or more implementations of the technology or elements thereof can be implemented in the form of a computer product, including a non-transitory computer-readable storage medium with computer usable program code for performing any indicated method steps and/or any configuration file for one or more RDUs to execute a high-level program. Furthermore, one or more implementations of the technology or elements thereof can be implemented in the form of an apparatus including a memory and at least one processor that is coupled to the memory and operative to perform exemplary method steps, and/or an RDU that is operative to execute a high-level program based on a configuration file. Yet further, in another aspect, one or more implementations of the technology or elements thereof can be implemented in the form of means for carrying out one or more of the method steps described herein and/or executing a high-level program described herein. Such means can include (i) hardware module(s); (ii) software module(s) executing on one or more hardware processors; (iii) bit files for configuration of a CGR array; or (iv) a combination of aforementioned items.

Thus, while particular implementations have been described herein, latitudes of modification, various changes, and substitutions are intended in the foregoing disclosures, and it will be appreciated that in some instances some features of particular implementations will be employed without a corresponding use of other features without departing from the scope and spirit as set forth. Therefore, many modifications may be made to adapt a particular situation or material to the essential scope and spirit of the technology disclosed.

Clauses

1. A heterogeneous processing system, comprising:a host processor coupled to a host memory;a first processor coupled to a first memory;a second processor coupled to a second memory;a plurality of data transfer resources; andswitch and bus circuitry that communicatively couples the host processor, the first processor, the second processor, and the plurality of data transfer resources;wherein the host processor is configured to detect an application for execution by both the first processor and the second processor, to select one of a plurality of different data access methods for accessing data between the first and second processors, and to configure the heterogeneous processing system based on the selected data access method.2. The heterogeneous processing system of clause 1, wherein the host processor selects memory extension operation by mapping virtual addresses of the second memory to physical addresses of the switch and bus circuitry and by configuring the first processor to directly access the second memory using the mapped physical addresses according to the memory extension operation.3. The heterogeneous processing system of clause 2, wherein the second processor is programmed to execute a first part of the application to generate and write first data into the second memory, and wherein the first processor is configured to directly access the first data from the second memory while executing a second part of the application using the first data.4. The heterogeneous processing system of clause 3, wherein the first processor is configured to directly access the host memory and to write second data output from executing the second part of the application directly into the host memory.5. The heterogeneous processing system of clause 2, wherein the first processor is configured to execute a first part of the application to generate first data and to directly write the first data into the second memory.6. The heterogeneous processing system of clause 2, wherein the first processor is configured to directly read first data from the host memory while executing a first part of the application using the first data to generate second data and to directly write the second data into the second memory while executing the first part of the application.7. The heterogeneous processing system of clause 1, wherein the host processor is configured to map virtual addresses of the second memory to physical addresses of the switch and bus circuitry and to configure a first data transfer resource to perform a one memory to memory transfer operation between the first memory and the second memory.8. The heterogeneous processing system of clause 7, wherein the host processor is configured to program the first data transfer resource to directly transfer data between the first memory and the second memory using the mapped physical addresses.9. The heterogeneous processing system of clause 7, wherein the second processor is programmed to execute a first part of the application to generate and write first data into the second memory, and wherein the host processor is configured to prompt the first data transfer resource to transfer the first data from the second memory to the first memory.10. The heterogeneous processing system of clause 9, wherein the first processor is configured to execute a second part of the application using the first data and write second data into the first memory, and wherein the host processor is configured to prompt the first data transfer resource to transfer the second data into the host memory.11. The heterogeneous processing system of clause 7, wherein the first processor is configured to execute a first part of the application to generate and write first data into the first memory, and wherein the host processor is configured to prompt the first data transfer resource to transfer the first data from the first memory to the second memory.12. The heterogeneous processing system of clause 11, wherein the second processor is programmed to execute a second part of the application using the first data and write second data into the second memory.13. The heterogeneous processing system of clause 12, wherein the host processor is configured to prompt the second data transfer resource to transfer the second data from the second memory into the host memory.14. The heterogeneous processing system of clause 1, wherein the host processor selects a two memory to memory transfer operation by allocating buffer space within the host memory, wherein the first processor is configured to execute a first part of the application which generates and writes first data into the first memory, and wherein the host processor is further configured to program the plurality of data transfer resources to transfer the first data from the first memory to the buffer space of the host memory and to transfer the first data from the buffer space of the host memory into the second memory.15. The heterogeneous processing system of clause 14, wherein the buffer space comprises a common buffer that is allocated within a non-swappable portion of the host memory.16. The heterogeneous processing system of clause 14, wherein the second processor is configured to execute a second part of the application and to write second data in the second memory, and wherein the host processor is configured to prompt one of the plurality of data transfer resources to transfer the second data from the second memory to the host memory.17. The heterogeneous processing system of clause 1, wherein the host processor is configured to select the most efficient one of the plurality of different data access methods based on current latency and bandwidth requirements.18. The heterogeneous processing system of clause 1, wherein the host processor is configured to select the most efficient one of the plurality of different data access methods based on user-defined metadata of data passing, including source and destination device types, memory addresses, and bandwidth of application-defined pipeline and latency requirement of operation device stages.19. A method of selecting data access method in a heterogeneous system, wherein the heterogeneous system includes a host processor coupled to a host memory, a first processor coupled to a first memory, a second processor coupled to a second memory, a plurality of data transfer resources, and switch and bus circuitry that communicatively couples the host processor, the first processor, the second processor, and the plurality of data transfer resources, and the second data transfer resource, the method comprising:detecting an application for execution by both the first processor and the second processor;selecting one of a plurality of different data access methods for accessing data between the first and second processors; andconfiguring the heterogeneous processing system according to the selected data access method.20. The method of clause 19, further comprising:selecting memory extension operation;mapping, by the host processor, virtual addresses of the second memory to physical addresses of the switch and bus circuitry; andconfiguring, by the host processor, the first processor to directly access the second memory using the mapped physical addresses according to the memory extension operation.21. The method of clause 20, further comprising:programming the second processor to execute a first part of the application to generate and write first data into the second memory; andconfiguring the first processor to directly access the first data from the second memory while executing a second part of the application.22. The method of clause 21, further comprising configuring the first processor to directly access the host memory and to write second data output from executing the second part of the application directly into the host memory.23. The method of clause 20, further comprising configuring the first processor to execute a first part of the application to generate first data and to directly write the first data into the second memory.24 The method of clause 20, further comprising configuring the first processor to directly read first data from the host memory while executing the application using the first data to generate second data and to directly write the second data into the second memory while executing the application.25. The method of clause 19, further comprising:selecting one memory to memory transfer operation;mapping, by the host processor, virtual addresses of the second memory to physical addresses of the switch and bus circuitry; andconfiguring, by the host processor, the first data transfer resource to perform a one memory to memory transfer operation between the first memory and the second memory using mapped addresses.26. The method of clause 25, wherein the configuring comprises configuring the first data transfer resource to directly transfer data between the first memory and the second memory using mapped physical addresses.27. The method of clause 25, further comprising:programming the second processor to execute a first part of the application to generate and write first data into the second memory; andwherein said configuring comprises configuring the first data transfer resource to transfer the first data from the second memory to the first memory.28. The method of clause 27, further comprising configuring the first processor to execute a second part of the application using the first data and write second data into the first memory, and prompting the first data transfer resource to transfer the second data into host memory coupled to the host processor.29. The method of clause 25, further comprising configuring the first processor to execute a first part of the application to generate and write first data into the first memory, and prompting the first data transfer resource to transfer the first data from the first memory to the second memory.30. The method of clause 29, further comprising programming the second processor to execute a second part of the application using the first data and write second data into the second memory.31. The method of clause 30, further comprising programming the second processor to prompt a second data transfer resource to transfer the second data from the second memory into the host memory.32. The method of clause 19, further comprising:selecting a two memory to memory transfer operation;allocating buffer space in the host memory;programming a first data transfer resource with source addresses of the first memory and destination addresses of the buffer space;programming a second data transfer resource with source addresses of the buffer space and destination addresses of the second memory;executing a first part of an application by the first processor which generates and writes first data into the first memory;causing the first data transfer resource to transfer the first data from the first memory to the buffer space of the host memory; andcausing the second data transfer resource to transfer the first data from the buffer space of the host memory to the second memory.33. The method of clause 32, wherein the allocating buffer space comprises allocating a common buffer within a non-swappable portion of the host memory.34. The method of clause 32, wherein at least one of the plurality of data transfer resources comprises a direct memory access (DMA) engine and wherein the allocating buffer space comprises allocating the buffer space as DMA addressable memory.35. The method of clause 32, wherein the first processor comprises a compute engine, further comprising configuring the compute engine by the host processor to execute the first part of the application to generate the first data and to write the first data into the first memory, and causing the first data transfer resource to transfer the first data from the first memory to the buffer space of the host memory.36. The method of clause 32, wherein the second processor comprises a compute engine, further comprising causing the second data transfer resource to transfer the first data from the buffer space of the host memory to the second memory, and configuring, by the host processor, the compute engine to execute a second part of the application using the first data to generate second data and to write the second data into the second memory.37. The method of clause 32, wherein the plurality of data transfer resources comprises a first direct memory access (DMA) engine incorporated within the first processor and a second DMA engine incorporated within the second processor, further comprising programming the first DMA engine to transfer the first data from the first memory to the buffer space of the host memory, prompting the first DMA engine after the first data is written into the first memory to transfer the first data from the first memory to the buffer space, programming the second DMA engine to transfer the first data from the buffer space into the second memory, and prompting the second DMA engine after the first data is written into the buffer space of the host memory to transfer the first data from the buffer space to the second memory.