Patent Description:
A general-purpose processor, such as a central processing unit (CPU), typically performs input/output (I/O) operations for a software application. In a system that includes multiple processors, the CPU might be the only processor that can generate network messages. The system can also include a data parallel processor in addition to the general-purpose processor. The data parallel processor has a highly parallel execution architecture. Data parallel processors can include graphics processing units (GPUs), digital signal processors (DSPs), and so forth. A data parallel processor incurs delays in computations while waiting for the CPU to coordinate network communication on its behalf. In a computing system with a CPU and a GPU, the CPU is often referred to as "the host". In these systems, the GPU is a second class citizen with regards to network operations. This limits the ability of the GPU to initiate network operations, and requires the CPU to be involved in any network operations launched by the GPU. Using the host to generate network messages for the GPU can potentially involve several back-to-back round trips from the GPU to the host and from the host to the network interface, resulting in a reduction in performance.

<CIT> discloses a system in which a queue of communication commands can be pre-generated using a CPU and stored in a network interface controller device memory. If a GPU has data to communicate to a remote GPU, it stores the data in a send buffer, where the location of the buffer is pointed to by a pre-generated command. The GPU then signals to the interface device that the data is ready, triggering execution of the pre-generated command to send the data.

According to the invention there is provided a system according to claim <NUM>, a method according to claim <NUM> and a processor according to claim <NUM>.

In the following description, numerous specific details are set forth to provide a thorough understanding of the methods and mechanisms presented herein. However, one having ordinary skill in the art should recognize that the various implementations may be practiced without these specific details. In some instances, well-known structures, components, signals, computer program instructions, and techniques have not been shown in detail to avoid obscuring the approaches described herein. For example, the dimensions of some of the elements may be exaggerated relative to other elements.

Various systems, apparatuses, and methods for implementing network packet templating are disclosed herein. A first processor (e.g., central processing unit (CPU)) creates a network packet according to a template and populates a first subset of fields of the network packet with static data. Next, the first processor stores the network packet in a memory. A second processor (e.g., graphics processing unit (GPU) initiates execution of a kernel and detects a network communication request prior to the kernel completing execution. Responsive to detecting the network communication request, the second processor populates a second subset of fields of the network packet with runtime data. Then, the second processor generates a notification that the network packet is ready to be processed. A network interface controller (NIC) processes the network packet using data retrieved from the first subset of fields and from the second subset of fields responsive to detecting the notification.

In one implementation, a circular buffer is maintained in memory which is accessible by the first and second processors. The circular buffer stores a plurality of network packets. In one implementation, the first processor periodically checks the status of the circular buffer, and if the number of existing network packets is below a threshold, then the first processor adds one or more network packets to the circular buffer. When adding network packets to the circular buffer, the first processor populates the first subset of fields of each network packet added to the circular buffer. In one implementation, the first subset of fields include includes a network control bits field and a memory access key field. In one implementation, the second subset of fields includes a source offset field, a destination offset field, and an operation type field.

Referring now to <FIG>, a block diagram of one implementation of a computing system <NUM> is shown. In one implementation, computing system <NUM> includes at least processors 105A-N, input/output (I/O) interfaces <NUM>, bus <NUM>, memory controller(s) <NUM>, network interface controller (NIC) <NUM>, and memory device(s) <NUM>. In other implementations, computing system <NUM> includes other components and/or computing system <NUM> is arranged differently. Processors 105A-N are representative of any number of processors which are included in system <NUM>. In one implementation, processor 105A is a general purpose processor, such as a central processing unit (CPU). In one implementation, processor 105N is a data parallel processor with a highly parallel architecture. Data parallel processors include graphics processing units (GPUs), digital signal processors (DSPs), field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), and so forth. In some implementations, processors 105A-N include multiple data parallel processors.

Memory controller(s) <NUM> are representative of any number and type of memory controllers accessible by processors 105A-N and I/O devices (not shown) coupled to I/O interfaces <NUM>. Memory controller(s) <NUM> are coupled to any number and type of memory devices(s) <NUM>. Memory device(s) <NUM> are representative of any number and type of memory devices. For example, the type of memory in memory device(s) <NUM> includes Dynamic Random Access Memory (DRAM), Static Random Access Memory (SRAM), NAND Flash memory, NOR flash memory, Ferroelectric Random Access Memory (FeRAM), or others. In one implementation, a given memory device <NUM> includes a circular buffer <NUM> for storing newly created network packets generated by a first processor 105A according to a network packet template. The first processor 105A populates each newly created network packet with static information. In one implementation, the given memory device <NUM> is local to a second processor 105N. When the second processor is ready to initiate a network transmission, the second processor 105N updates a network packet stored in circular buffer <NUM> with dynamic, runtime information and then the second processor 105N notifies NIC <NUM> that the network packet is ready to be processed. NIC <NUM> processes the network packet and performs the requested network communication on network <NUM>. As used herein, a network packet is a formatted data structure with a plurality of fields, wherein the data structure is created by a first processor 105A and updated by a second processor 105B. In other words, a network packet is a formatted data structure which has a first subset of fields that are written to by a first processor 105A and a second subset of fields which are written to and/or updated by a second processor 105N.

I/O interfaces <NUM> are representative of any number and type of I/O interfaces (e.g., peripheral component interconnect (PCI) bus, PCI-Extended (PCI-X), PCIE (PCI Express) bus, gigabit Ethernet (GBE) bus, universal serial bus (USB)). Various types of peripheral devices (not shown) are coupled to I/O interfaces <NUM>. Such peripheral devices include (but are not limited to) displays, keyboards, mice, printers, scanners, joysticks or other types of game controllers, media recording devices, external storage devices, network interface cards, and so forth. Network interface controller (NIC) <NUM> receives and sends network messages across network <NUM>. In one implementation, a given processor 105N generates requests for NIC <NUM> to process a network packet. In one implementation, NIC <NUM> processes a request generated by processor 105N without involvement of processor 105A. In this implementation, processor 105A is a CPU and processor 105N is a GPU. In other implementations, other types of processors are able to perform these actions.

Network <NUM> is representative of any type of network or combination of networks, including wireless connection, direct local area network (LAN), metropolitan area network (MAN), wide area network (WAN), an Intranet, the Internet, a cable network, a packet-switched network, a fiber-optic network, a router, storage area network, or other type of network. Examples of LANs include Ethernet networks, Fiber Distributed Data Interface (FDDI) networks, and token ring networks. In various implementations, network <NUM> further includes remote direct memory access (RDMA) hardware and/or software, transmission control protocol/internet protocol (TCP/IP) hardware and/or software, router, repeaters, switches, grids, and/or other components.

In various implementations, computing system <NUM> is a computer, laptop, mobile device, game console, server, streaming device, wearable device, or any of various other types of computing systems or devices. It is noted that the number of components of computing system <NUM> varies from implementation to implementation. For example, in other implementations, there are more or fewer of each component than the number shown in <FIG>. It is also noted that in other implementations, computing system <NUM> includes other components not shown in <FIG>. Additionally, in other implementations, computing system <NUM> is structured in other ways than shown in <FIG>.

Turning now to <FIG>, a block diagram of another implementation of a computing system <NUM> is shown. In one implementation, system <NUM> includes GPU <NUM>, system memory <NUM>, and local memory <NUM>. System <NUM> also includes other components which are not shown to avoid obscuring the figure. GPU <NUM> includes at least command processor <NUM>, dispatch unit <NUM>, compute units 255A-N, memory controller <NUM>, global data share <NUM>, level one (L1) cache <NUM>, and level two (L2) cache <NUM>. In other implementations, GPU <NUM> includes other components, omits one or more of the illustrated components, has multiple instances of a component even if only one instance is shown in <FIG>, and/or is organized in other suitable manners.

In various implementations, computing system <NUM> executes any of various types of software applications. In one implementation, as part of executing a given software application, a host CPU (not shown) of computing system <NUM> launches kernels to be performed on GPU <NUM>. As used herein, the term "kernel" or "compute kernel" is defined as a function or task comprising executable instructions that are launched and executed as one or more threads on one or more compute units. Command processor <NUM> receives kernels from the host CPU and issues kernels to dispatch unit <NUM> for dispatch to compute units 255A-N. Threads within kernels executing on compute units 255A-N initiate network communications by updating network packets stored in circular buffer <NUM> with runtime information and then notifying a NIC (not shown). The NIC processes a given network packet responsive to receiving a corresponding notification. Although not shown in <FIG>, in one implementation, compute units 255A-N also include one or more caches and/or local memories within each compute unit 255A-N.

Turning now to <FIG>, a timing diagram <NUM> of one implementation of a network control path being offloaded from a CPU to a GPU is shown. In one implementation, CPU <NUM> launches a kernel <NUM> on GPU <NUM>, and then GPU <NUM> executes the kernel <NUM> independently of the CPU <NUM>. While GPU <NUM> is executing kernel <NUM>, CPU <NUM> is able to perform other tasks. In some implementations, CPU <NUM> is coupled to other GPUs, and CPU <NUM> is able to launch kernels on the other GPUs, with these kernels executing in parallel with kernel <NUM> on GPU <NUM>. CPU <NUM> generates network packets according to a template and populates a portion of the fields of each network packet. Then, CPU <NUM> stores each network packet in a location accessible by GPU <NUM> and NIC <NUM>. CPU <NUM> generates network packets ahead of time and without waiting for an explicit request from GPU <NUM>.

During execution of the kernel <NUM> and without waiting for kernel <NUM> to complete, GPU <NUM> populates the remaining portion of fields of a given network packet, and then GPU <NUM> sends a network packet request to network interface controller (NIC) <NUM> as illustrated with send block <NUM>. It is noted that GPU <NUM> sends this network request to NIC <NUM> without any involvement by CPU <NUM>. NIC <NUM> processes the network packet request and puts the request on the network while kernel <NUM> continues to execute as illustrated with put block <NUM>. Although not shown in <FIG>, it is noted that kernel <NUM> is able to send multiple network packet requests to NIC <NUM> which are processed by NIC <NUM> during execution of kernel <NUM>. At a later point in time, the execution of kernel <NUM> completes, which is represented by done block <NUM>.

Turning now to <FIG>, a diagram of one implementation of a queue <NUM> for storing network packets is shown. In one implementation, queue <NUM> is a circular queue in which operations are performed based on a first in, first out (FIFO) principle and with the last position connected back to the first position to make a circle. It is noted that a "circular queue" is also referred to as a "circular buffer" herein. In other implementations, queue <NUM> is implemented using any of various other types of data structures. In one implementation, queue <NUM> is used by a computing system which includes a CPU, GPU, and NIC.

In one implementation, CPU <NUM> adds network packets to queue <NUM>, wherein each network packet is generated according to a network packet template. As used herein, a "network packet template" is defined as a structure of a communication packet, created by a first processor, but with one or more empty fields to be filled in and/or updated by a second processor. In other words, the "network packet template" defines the structure of a communication packet but at least a portion of the contents of the communication packet will be filled in by the second processor. Typically, CPU <NUM> will fill in a portion of the fields of a network packet with static information (e.g., network control bits, memory access key (Rkey)) before storing the packet at an available slot in queue <NUM>. After the second processor writes the second subset of fields to the network packet, a notification is conveyed to NIC <NUM> to process the network packet. In other words, the network packet becomes ready once the second processor completes filling out all of the fields and generates a notification for NIC <NUM>. In one implementation, the second processor notifies NIC <NUM> that the network packet is ready by writing a notification to a doorbell register. In other implementations, the second processor uses other techniques for conveying a notification to NIC <NUM>.

In one implementation, CPU <NUM> adds newly created network packets to queue <NUM> on a periodic basis. For example, CPU <NUM> periodically checks the status of queue <NUM>. In one implementation, each queue entry includes a valid indicator to indicate if the queue entry stores a valid network packet. It is noted that a "queue entry" is also referred to as a "queue slot" herein. In one implementation, if there are any unused (i.e., available) slots in queue <NUM> as indicated by the valid indicators, then CPU <NUM> adds one or more network packets to these empty entries of queue <NUM>. In another implementation, if the number of network packets currently stored in queue <NUM> is less than a threshold, then CPU <NUM> adds one or more network packets to the available slots of queue <NUM>. In various implementations, the number of <NUM> network packets which CPU <NUM> adds to queue <NUM> is fixed, programmable, based on the number of available slots, or determined in other suitable manners.

In one implementation, NIC <NUM> maintains a pointer which points to the next location in queue <NUM> to be processed. After NIC <NUM> receives a notification and processes a network packet, NIC <NUM> increments the pointer to point to the next location in the queue. Similarly, GPU <NUM> maintains a pointer which points to a location in queue <NUM> which stores a network packet which is ready to be modified and then processed by NIC <NUM>. In one implementation, the pointer maintained by GPU <NUM> is offset by one queue element from the pointer maintained by NIC <NUM>. Similarly, in one implementation, CPU <NUM> maintains a pointer which points to an available location in queue <NUM> for storing a newly created network packet.

In one implementation, CPU <NUM> queries the entries of queue <NUM> to determine if there are any entries available for storing a newly created network packets. In one implementation, if CPU <NUM> detects an available entry in queue <NUM>, CPU <NUM> generates a new network packet and stores the network packet in the available entry. In another implementation, if CPU <NUM> detects an available entry in queue <NUM> and a first condition is met, CPU <NUM> generates a new network packet and stores the network packet in the available entry. In one implementation, the first condition is queue <NUM> having less than a threshold number of network packets that are ready to be used by GPU <NUM>. In other implementations, the first condition is any of various other types of conditions. In various implementations, CPU <NUM> creates network packets ahead of time and out of the critical path. In this way, when GPU <NUM> needs to generate a network packet, GPU <NUM> accesses a network packet stored on queue <NUM> in real-time and sets up the network packet for immediate consumption by NIC <NUM> without any involvement of CPU <NUM>.

Referring now to <FIG>, a diagram of one implementation of a network packet template <NUM> is shown. In various implementations, a network packet template as described herein includes a plurality of indicators and/or fields with various static and runtime information. For example, in one implementation, network packet template <NUM> includes a valid indicator <NUM>, a packet ready indicator <NUM>, and a plurality of fields 520A-N. In other implementations, network packet template <NUM> includes other numbers and/or types of indicators and fields and/or network packet template <NUM> is structured in other suitable manners.

In one implementation, valid indicator <NUM> indicates if the entry stores a valid network packet. A valid network packet is a packet which has been created and initialized by a first processor (e.g., CPU) and is ready to be updated by a second processor (e.g., GPU, processing in memory (PIM) device). In one implementation, packet ready indicator <NUM> specifies when the entry stores a network packet which is ready to be processed by a NIC. In various implementations, fields 520A-N store any of various network communication settings. In one implementation, a first subset of fields 520A-N are programmed by the first processor when a network packet is created according to network packet template <NUM>, and a second subset of fields 520A-N are programmed in real-time when the second processor needs to initiate a network transmission.

Turning now to <FIG>, a diagram of another implementation of a network packet template <NUM> is shown. In one implementation, network packet template <NUM> includes a valid indicator <NUM> and a packet ready indicator <NUM>. Network packet template <NUM> also includes network control bits <NUM> and memory access key <NUM>. In one implementation, a first processor (e.g., CPU) fills in the fields for network control bits <NUM> and memory access key <NUM> when creating a network packet according to template <NUM> and storing the network packet on a queue (e.g., queue <NUM> of <FIG>) in a memory accessible by a second processor (e.g., GPU). Although not shown, in other implementations, network packet template <NUM> also includes one or more other fields which are filled in by the first processor upon the creation and storing of a network packet in the queue.

Network packet template <NUM> also includes various fields which are filled in dynamically by the second processor with runtime information. For example, in one implementation, network packet template <NUM> includes source offset <NUM>, destination offset <NUM>, and operation type <NUM> which are filled in with runtime information by the second processor. Source offset <NUM> specifies where in a source buffer or source memory device to begin the transfer of data. Destination offset <NUM> specifies where in a destination buffer or destination memory device to store the transferred data. Operation type <NUM> specifies the type of operation (e.g., read, write, atomic) that is to be performed. It is noted that in other implementations, network packet template <NUM> includes other numbers and types of fields that are filled in dynamically with runtime information by the second processor. Also, in further implementations, network packet template <NUM> is structured in other suitable manners.

Referring now to <FIG>, one implementation of a method <NUM> for creating and using network packet templates is shown. For purposes of discussion, the steps in this implementation and those of <FIG> are shown in sequential order. However, it is noted that in various implementations of the described methods, one or more of the elements described are performed concurrently, in a different order than shown, or are omitted entirely. Other additional elements are also performed as desired. Any of the various systems or apparatuses described herein are configured to implement method <NUM>.

A first processor creates a network packet according to a template and populates a first subset of fields of the network packet (block <NUM>). In one implementation, the first processor is a CPU. The first processor stores the network packet in a location accessible by a second processor (block <NUM>). In one implementation, the first processor performs steps <NUM> and <NUM> without receiving an explicit request for a network packet template from the second processor. This allows steps <NUM> and <NUM> to be performed ahead of time and out of the critical path. In one implementation, a ready flag is not set for the network packet stored by the first processor to indicate that the network packet is not yet ready to be processed by a NIC. The ready flag not being set also indicates that a second processor still needs to add more data to the network packet before the network packet becomes ready.

Also, the first processor launches a kernel on a second processor (block <NUM>). In one implementation, the second processor is a GPU. It is noted that step <NUM> is sometimes performed in parallel with steps <NUM> and <NUM> or prior to steps <NUM> and <NUM>. Next, during execution of the kernel, the second processor populates a second subset of fields of the network packet responsive to detecting a network communication request by the kernel (block <NUM>). Then, the second processor notifies a NIC that the network packet is ready to be processed (block <NUM>). Next, the NIC processes the network packet using data retrieved from the first subset of fields and from the second subset of fields (block <NUM>). After block <NUM>, method <NUM> ends.

Turning now to <FIG>, one implementation of a method <NUM> for determining when to add network packet templates to a queue is shown. A first processor monitors a status of a queue which stores network packets (block <NUM>). If a first condition is detected for the queue (conditional block <NUM>, "yes" leg), then the first processor creates and adds one or more network packets to the queue (block <NUM>). In one implementation, the first condition is the occupancy level of the queue being below a threshold, wherein the occupancy level is specified as a number of valid network packets which are stored in the queue and ready to be populated by the second processor. In one implementation, the threshold is programmed by a second processor which updates network packets which are stored in the queue. In this implementation, the second processor adjusts the threshold based on a status of the kernel currently being executed by the second processor. For example, in one implementation, if the second processor executes a kernel which is generating a relatively large number of network requests, then the second processor dynamically reduces the threshold value to ensure that the first processor keeps plenty of network packets on the queue. In other implementations, the second processor adjusts the threshold based on one or more other factors.

In another implementation, the first condition is the queue having one or more available slots for storing network packets. In other implementations, the first condition is any of various other types of conditions. If the first condition is not detected for the queue (conditional block <NUM>, "no" leg), then the first processor waits a programmable amount of time before checking the status of the queue again (block <NUM>). In one implementation, the programmable amount of time is determined by the second processor based on one or more factors associated with the kernel currently being executed. After block <NUM>, method <NUM> returns to block <NUM>.

Referring now to <FIG>, another implementation of a method <NUM> for determining when to add network packet templates to a queue is shown. In a system with a first processor (e.g., CPU) and a second processor (e.g., GPU), the second processor monitors a status of a queue which stores network packets (block <NUM>). If a first condition is detected for the queue (conditional block <NUM>, "yes" leg), then the second processor generates an interrupt for the first processor to create and add one or more network packets to the queue (block <NUM>). In response to receiving the interrupt, the first processor creates and adds one or more network packets to the queue (block <NUM>).

In one implementation, the first condition is the occupancy level of the queue being below a threshold, wherein the occupancy level is specified as a number of valid network packets which are stored in the queue and ready to have one or more fields populated by the second processor. In another implementation, the first condition is the queue having one or more available slots for storing network packets. In other implementations, the first condition is any of various other types of conditions. If the first condition is not detected for the queue (conditional block <NUM>, "no" leg), then the second processor waits a programmable amount of time before checking the status of the queue again (block <NUM>). After block <NUM>, method <NUM> returns to block <NUM>.

In various implementations, program instructions of a software application are used to implement the methods and/or mechanisms described herein. For example, program instructions executable by a general or special purpose processor are contemplated. In various implementations, such program instructions are represented by a high level programming language. In other implementations, the program instructions are compiled from a high level programming language to a binary, intermediate, or other form. Alternatively, program instructions are written that describe the behavior or design of hardware. Such program instructions are represented by a high-level programming language, such as C. Alternatively, a hardware design language (HDL) such as Verilog is used. In various implementations, the program instructions are stored on any of a variety of non-transitory computer readable storage mediums. The storage medium is accessible by a computing system during use to provide the program instructions to the computing system for program execution. Generally speaking, such a computing system includes at least one or more memories and one or more processors configured to execute program instructions.

Claim 1:
A system comprising:
a memory comprising a circular buffer (<NUM>, <NUM>);
a network interface controller (<NUM>); and
first and second processors (<NUM>, <NUM>);
the first processor (<NUM>) configured to:
periodically check a status of the circular buffer (<NUM>, <NUM>); and
responsive to detecting an occupancy level of the circular buffer being below a threshold, the occupancy level comprising the number of valid network packets stored in the circular buffer and ready to be populated by the second processor (<NUM>):
create a network packet according to a template and populate a first subset of fields (<NUM>, <NUM>) of the network packet; and
store the network packet in the circular buffer (<NUM>, <NUM>);
the second processor (<NUM>) configured to:
initiate execution of a kernel on the second processor (<NUM>);
responsive to detecting a network communication request during execution of the kernel and prior to the kernel completing execution:
identify a network packet in the circular buffer as being ready to be updated by the second processor (<NUM>);
populate a second subset of fields (520A - 520N) of the network packet; and
generate a notification that the network packet is ready to be processed;
wherein the network interface controller is configured to:
process, prior to the kernel completing execution, the network packet responsive to detecting the notification, wherein the second processor is further configured to adjust the occupancy level threshold based on the status of the kernel currently being executed by the second processor.