A system is described having an input/output (IO) device and a processing unit coupled with the IO device. The processing unit is enabled to elect a thread from among a plurality of threads to atomically update a queue head pointer, uses the queue head pointer to reserve space in a plurality of memory registers for work queue elements belonging to the plurality of threads, and submit the work queue elements to the IO device.

FIELD OF THE DISCLOSURE

The present disclosure is generally directed to systems, methods, and devices for transmitting data between nodes and, in particular, toward improving asynchronous kernel-initiated communications.

BACKGROUND

In modern high-performance computing systems, communication between computing devices is typically facilitated by a network of interconnected nodes. Each computing device, which may contain a central processing unit (CPU), a graphics processing unit (GPU), and/or other hardware peripheral, can be considered a node in the network. Data is transmitted between such nodes in a series of discrete operations, with each node serving as a relay point for the data. This structure enables parallel processing and data sharing, significantly improving overall system performance and enabling complex computational tasks. Communication between nodes is governed by various protocols, which can vary depending on the specific requirements of the system and the type of devices involved.

The concept of queue pairs (QPs) supports efficient operation of these inter-network communications. A QP is composed of a work queue including a send queue and a receive queue, acting as endpoints for data transmission between nodes. The send queue holds instructions for outgoing data, while the receive queue accommodates incoming data instructions. QPs also require completion queues which signal the completion of work requests posted to the work queue. The use of QPs enables network technologies such as InfiniBand to provide high-speed, low-latency communication between nodes. The implementation and management of QPs, however, can be complex, necessitating detailed handling of data transmission protocols and error management.

Latency and memory consumption are key factors in the performance and efficiency of these communication networks. Latency refers to the delay experienced during data transmission between nodes, which can impact the overall performance in real-time or high-speed applications. Memory consumption on the other hand relates to the amount of memory resources utilized for data transmission and processing. High memory consumption can lead to inefficiencies, potentially slowing down other processes and limiting the overall system performance. Optimizing both latency and memory consumption is therefore a continuous challenge in the development and operation of high-performance computing systems. Various strategies and technologies are employed to tackle these issues, aiming to deliver fast, efficient, and reliable communication between devices.

Technical shortcomings of conventional computing system networks relating to memory consumption and latency negatively affect real-world applications involving, for example, artificial intelligence models, mathematical calculations, and other computationally-complex applications.

SUMMARY

In some communication protocols, such as an MLX5 post-send protocol, work queue entry (WQE) submission involves enqueueing a WQE into a ring buffer and updating the head pointer to submit work to a network interface card (NIC) or similar type of Input/Output (IO) device. In particular, a post-send protocol may include: (1) writing the WQE (or WQEs) in a work queue (WQ) buffer; (2) updating the doorbell record (DBR); and ringing the doorbell (DB).

In the GPUDirect Async-Kernel Initiated networking protocol (GDA-KI), WQ and DBR are in GPU memory. The DB is usually provided on the NIC. In CPU-centric libraries such as libibverbs, WQ and DBR are in host memory and the DB is on the NIC.

This is an inherently sequential process as it was designed for use by CPUs. Communication protocols such as GDA-KI, which utilize GPUs instead of CPUs, leverage a GPU streaming multiprocessor (SM) to submit WQEs to the NIC. If traditional WQE submission algorithms are strictly followed, the GPU will need to (1) lock the network QP, which limits the concurrency, or (2) create one QP per thread, which may consume hundreds GB of GPU memory in real applications. In addition, each WQE submission will require issuing a memory barrier, which incurs significant latency for the GPU SMS.

Ideally, multiple threads (from the same or different thread blocks (CTAs)) should be able to create WQEs in parallel and submit them to the same QP without locking. Also, the number of issued memory barriers should be reduceable (e.g., by issuing one memory barrier for submitting multiple WQEs).

Embodiments of the present disclosure aim to improve communication efficiencies by increasing the parallelism of the process described above for threads within the same thread block. Embodiments of the present disclosure further contemplate improving communication efficiencies while working within the framework of existing post-send protocol(s). Aspects of the present disclosure may include two levels: (1) warp-level coalescing of WQE slot reservation and WQE creation and (2) coalescing of memory barriers and doorbell updates.

Embodiments of the present disclosure are contemplated for use in an architecture having a scalable array of multithreaded SMs. Each SM may include a set of execution units, a set of registers, and a chunk of shared memory. In some embodiments, the basic unit of execution for a processing unit (e.g., a CPU or GPU) may be referred to as a warp. A warp may correspond to a collection of threads (e.g., 32 threads may belong to a warp) that are executed simultaneously by an SM. Multiple warps can be executed on an SM at once. When a program on a host CPU invokes a kernel grid, the blocks of the grid may be enumerated and distributed to SMs with available execution capacity. The threads of a thread block execute concurrently on one SM, and multiple thread blocks can execute concurrently on one SM. As thread blocks terminate, new blocks are launched on the vacated SMs. The mapping between warps and thread blocks can impact the performance of the kernel.

While embodiments of the present disclosure will be described in connection with an architecture having a scalable array of multithreaded SMs, it should be appreciated that features depicted and described herein can be utilized in other architectures. Specifically, but without limitation, embodiments of the present disclosure can be deployed in any computing architecture in which threads issue WQE slot reservation and/or WQE creation instructions/requests.

As mentioned above, one aspect of the present disclosure is to reduce the number of WQE submissions within each warp and is not targeted at reducing the overall number of WQEs. This approach elects a thread from those present in the warp to atomically update the QP head pointer to reserve space for WQEs from all present threads. Once the space has been reserved, threads can create WQEs in parallel and a single submitter thread is selected to submit the WQEs to the IO device. Algorithms or approaches depicted and described herein may be used by some or all threads within a warp.

Another aspect of the present disclosure is to coalesce WQE submissions. In some embodiments, WQE submissions may be coalesced from multiple submitted threads. An advantage offered by such coalescing is that the number of memory barriers needed can be reduced. In some approaches, a scenario is handled where: (1) a QP is dedicated to a CTA, and (2) a QP is shared among CTAs differently.

Prior approaches use the sequential algorithm described above and may create many QPs to obtain parallelism. However, creating many QPs consumes significant amounts of memory and may also lead to poor performance.

In contrast, the approach described herein provides at least the following advantages: leverage parallelism of GPU to create multiple WQEs concurrently; opportunistically and automatically reduce the number of memory barriers that need to be issued; reduce the scope of locking to between updating doorbell record (DBR) and writing to a doorbell (DB). This locking can be completely removed if the IO device supports DBR-less QP. Yet another advantage offered by the approaches described herein is that the coalescing causes only a few threads to wait on the lock. Other threads are freed to do other important tasks, rather than also wait.

Approaches described herein are useful when the processing unit (e.g., GPU) is interacting with any IO device that uses a similar command submission mechanism like the NIC mechanism described above. It can be viewed generically as a technique for coalescing entries when multiple threads on the processing unit enqueue entries into a circular buffer in memory.

In view of the above, one or more of the following are contemplated:

One aspect of the present disclosure is to provide a system that includes: an input/output (IO) device; and a processing unit coupled with the IO device, where the processing unit: elects a thread from among a plurality of threads to atomically update a queue head pointer; uses the queue head pointer to reserve space in a plurality of memory registers for work queue elements belonging to the plurality of threads; and submits the work queue elements to the IO device.

In some embodiments, the work queue elements are submitted to the IO device by a single submitter thread.

In some embodiments, the single submitter thread is the same as the thread elected from among the plurality of threads.

In some embodiments, the single submitter thread is different from the thread elected from among the plurality of threads.

In some embodiments, the IO device includes a Network Interface Card (NIC).

In some embodiments, the processing unit includes at least one of a Graphics Processing Unit (GPU), a Central Processing Unit (CPU), and a Data Processing Unit (DPU).

In some embodiments, a first thread is elected to submit the work queue elements, where ownership of submitting the work queue elements is transferred from the first thread to a second thread, and where the first thread is still enabled to submit a pending work request in response to a predetermined amount of time elapsing even after the ownership of submitting the work queue elements has been transferred to the second thread.

In some embodiments, a communication library is used to determine whether a queue pair is dedicated to a particular thread or is shared among the plurality of threads prior to allocating memory for the plurality of threads.

In some embodiments, the processing unit coalesces two or more work queue element submissions from different threads and allocates memory to the two or more work queue element submissions and the threads associated therewith.

In some embodiments, the processing unit checks whether a queue pair associated with a first submission and a queue pair associated with a second submission are shared between a common manager unit from the processing unit.

In some embodiments, the processing unit issues a single work queue element write back to the IO device for both the first submission and the second submission if all threads associated with the first submission and the second submission are shared between the common manager unit.

In some embodiments, the processing unit further enables the plurality of threads to create respective work requests in parallel with one another.

Another aspect of the present disclosure is to provide a method that includes: electing, at a processing unit, a thread from among a plurality of threads to atomically update a queue head pointer; using the queue head pointer to reserve space in a plurality of memory registers for work queue elements belonging to the plurality of threads; and submitting, from the processing unit to an input/output (IO) device, the work queue elements.

In some embodiments, a first thread is elected to submit the work queue elements, where ownership of submitting the work queue elements is transferred from the first thread to a second thread, and where the first thread is still enabled to submit a pending work request in response to a predetermined amount of time elapsing even after the ownership of submitting the work queue elements has been transferred to the second thread.

In some embodiments, the method further includes: determining whether a queue pair is dedicated to a particular thread or is shared among the plurality of threads prior to allocating memory.

In some embodiments, the method further includes: checking whether a queue pair associated with a first submission and a queue pair associated with a second submission are shared between a common manager unit from the processing unit; and issuing a single work queue element write back to the IO device for both the first submission and the second submission if all threads associated with the first submission and the second submission are shared between the common manager unit.

Another aspect of the present disclosure is to provide a device that includes: a processing unit in communication with an input/output (IO) device, where the processing unit supports multiple threads and enables the multiple threads to create work queue elements in parallel and then submit the work queue elements to a common queue pair without requiring a locking of the common queue pair.

In some embodiments, the processing unit atomically updates a queue head pointer using an elected thread from the multiple threads.

In some embodiments, the queue head pointer reserves space from memory registers for the multiple threads.

In some embodiments, a first thread is elected to submit the work queue elements, where ownership of submitting the work queue elements is transferred from the first thread to a second thread, and where the first thread is still enabled to submit a pending work request in response to a predetermined amount of time elapsing even after the ownership of submitting the work queue elements has been transferred to the second thread.

Additional features and advantages are described herein and will be apparent from the following Description and the figures.

DETAILED DESCRIPTION

Numerous additional features and advantages are described herein and will be apparent to those skilled in the art upon consideration of the following Detailed Description and in view of the figures.

The ensuing description provides embodiments only, and is not intended to limit the scope, applicability, or configuration of the claims. Rather, the ensuing description will provide those skilled in the art with an enabling description for implementing the described embodiments. It being understood that various changes may be made in the function and arrangement of elements without departing from the spirit and scope of the appended claims. Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure belongs. It will be further understood that terms, such as those defined in commonly used dictionaries, should be interpreted as having a meaning that is consistent with their meaning in the context of the relevant art and this disclosure.

It will be appreciated from the following description, and for reasons of computational efficiency, that the components of the system can be arranged at any appropriate location within a distributed network of components without impacting the operation of the system.

Further, it should be appreciated that the various links connecting the elements can be wired, traces, or wireless links, or any appropriate combination thereof, or any other appropriate known or later developed element(s) that is capable of supplying and/or communicating data to and from the connected elements. Transmission media used as links, for example, can be any appropriate carrier for electrical signals, including coaxial cables, copper wire and fiber optics, electrical traces on a printed circuit board (PCB), or the like.

The terms “determine,” “calculate,” and “compute,” and variations thereof, as used herein, are used interchangeably, and include any appropriate type of methodology, process, operation, or technique.

Various aspects of the present disclosure will be described herein with reference to drawings that may be schematic illustrations of idealized configurations.

References in the specification to “one embodiment,” “an embodiment,” “an example embodiment,” “some embodiments,” etc., indicate that the embodiment described may include a particular feature, structure, or characteristic, but every embodiment may not necessarily include the particular feature, structure, or characteristic. Moreover, such phrases may not necessarily refer to the same embodiment. Further, when a particular feature, structure, or characteristic is described in conjunction with one embodiment, it is submitted that the description of such feature, structure, or characteristic may apply to any other embodiment unless so stated and/or except as will be readily apparent to one skilled in the art from the description. The present disclosure, in various embodiments, configurations, and aspects, includes components, methods, processes, systems and/or apparatus substantially as depicted and described herein, including various embodiments, sub combinations, and subsets thereof. Those of skill in the art will understand how to make and use the systems and methods disclosed herein after understanding the present disclosure. The present disclosure, in various embodiments, configurations, and aspects, includes providing devices and processes in the absence of items not depicted and/or described herein or in various embodiments, configurations, or aspects hereof, including in the absence of such items as may have been used in previous devices or processes, e.g., for improving performance, achieving case, and/or reducing cost of implementation.

The use of GPUs as a means to offload computationally-intensive tasks from CPUs and the use of networks of computing nodes to implement computationally-intensive tasks, whether executed by CPUs or GPUs, is increasingly important to users such as scientific researchers seeking to execute artificial intelligence (AI) models and other computationally-intensive processes. The growing demand for high-performance computing in various domains, including scientific simulations, machine learning, and image processing, has driven the need for efficient and cost-effective computational resources. The limitations of network communication performance and the increasing importance of parallelism have prompted researchers and other users to explore alternatives to the use of single computing devices for performing data processing. As a result, GPUs have emerged as an approach to offload computationally-intensive tasks from CPUs and networks of computing systems have become useful for executing complex processing applications.

Embodiments of the present disclosure may increase the message rate that GDA-KI can achieve by several folds (e.g., up to 36×). Embodiments of the present disclosure are useful when the GPU is interacting with any IO device that employs a similar command submission mechanism like the NIC mechanism(s) described herein. Aspects of the present disclosure provide a technique for coalescing entries when multiple threads on the GPU enqueue entries into a circular buffer in memory.

The systems and methods described herein may be used by computing systems in which a GPU communicates with GPUs of peers through NVSHMEM or other GPU-accelerated libraries. For example, embodiments of the present disclosure may include a GPU performing data packet preparation, scheduling, sending or transmission. Through such a system, workloads may be shifted from a CPU to computationally viable GPUs.

Reference is now made toFIGS.1and2, in which a computing architecture100and system200are illustrated in accordance with one or more embodiments. The computing architecture100may include one or more peer computing systems108a-dcommunicating with each other via a network104.

Each of the one or more peer computing systems108a-dmay connect to each other as well as to other peer devices112to access shared resources, services, and data, via the network104. The peer computing systems108a-dmay be, for example, client devices such as personal computers, laptops, smartphones, IoT devices, as well as switches or servers, or any type of computing system capable of sending data to and receiving data over a network104.

Each of the peer devices112may comprise network interfaces including, for example, a transceiver. Some or all of the peer devices112may be capable of receiving and transmitting packets in conformance with applicable protocols such as TCP, although other protocols may be used. Peer devices112may also be configured to receive and transmit packets to and from network104.

In some implementations, one or more peer computing systems108a-dand devices112may be switches, proxies, gateways, load balancers, etc. Such systems108a-dand devices112may serve as intermediaries between clients and/or servers, relaying or modifying the communication between the clients and/or servers.

In some implementations, one or more of the peer computing systems108a-dand devices112may be IoT devices, such as sensors, actuators, and/or embedded systems, connected to the networks104. Such IoT devices may act as clients, servers, or both, depending on implementations and the specific IoT applications. For example, a first peer computing system or device may be a smart thermostat acting as a client, while a second peer computing system or device may be a central server for analysis or a smartphone executing an app.

As should be appreciated, in the realm of high-performance computing, a myriad of peer computing systems108a-dcan utilize QPs and WQ buffers for network communication. For example, in server farms, data centers, or clusters used for big data analytics and scientific computing, CPUs and/or GPUs of a peer computing system108a-dmay use QPs and WQ buffers to send and receive data between each other, such as via protocols including InfiniBand or Ethernet.

A system200having one or more CPUs204and one or more GPUs220may correspond to an example of a peer computing system108a-dand/or a peer device112. The advent of general-purpose computing on GPU has led to widespread use of GPUs220for tasks beyond just rendering graphics, especially in fields like machine learning, deep learning, and data mining. GPUs220may be capable of handling thousands of threads simultaneously, making them well-suited for massively parallel tasks.

While the system200may be configured to communicate with other systems200over the network104as described herein, it should be appreciated that the systems200may also communicate with other peer computing systems108a-dand/or peer devices112, which may or may not utilize the network104.

The network104illustrated inFIG.1may rely on various networking hardware and protocols to establish communication between the a peer computing system108a-dand other peer computing systems108a-dand/or peer devices112. Such infrastructure may include one or more routers, switches, and/or access points, as well as wired and/or wireless connections. The network104may be, for example, a local area network (LAN) interconnecting peer computing systems108a-dand peer devices112. A LAN may use Ethernet or Wi-Fi technologies to provide communication between the peer computing systems.

In some implementations, the network104may be, for example, a wide area network (WAN) and may be used to connect peer computing devices112with one or more peer computing systems108a-d. A WAN may comprise, for example, one or more of lines, satellite links, cellular networks. WANs may use various transmission technologies, such as leased lines, satellite links, or cellular networks, to provide long-distance communication. Transmission control protocol (TCP) communication over a WAN may be used, for example, to enable peer computing systems108a-dto communicate reliably across vast distances. In some implementations, network104may comprise the Internet, one or more mobile networks, such as 4G, 5G, LTE, virtual networks, such as a VPN, or some combination thereof.

System200, like the peer computing systems108a-d, may be or include client devices and may encompass a wide range of devices, including desktop computers, laptops, smartphones, IoT devices, etc. Such systems200may execute one or more applications which communicate with other systems200to access resources or services. For example, a first system200may execute a web browser and a second system200may act as a web server. The first computing system200may communicate with the second system200to request and display web content. As another example, a first system200may execute a file-sharing application and a second system200may act as a file server. The first system200may communicate with the second system200to upload or download files. As another example, a first system200may act as an AI server capable of being used by a second system200to offload computationally-intensive processes for execution in parallel by one or more GPUs220of the first system200. Applications running on the systems200may be responsible for initiating communication with other systems200making requests for resources or services, and processing data. The network104may enable the systems200to communicate any number of concurrent communications with any number of peer computing systems108a-dand/or peer devices112simultaneously.

It should also be appreciated that in some embodiments, the systems and methods described herein may be executed without a network104connection. For example, one or more peer computing systems108a-d(or systems200) may be capable of communicating directly with other peer computing systems108a-d(or systems200) without relying on any particular network104.

As illustrated inFIG.2, each a system200may include one or more CPUs204, one or more GPUs220, and one or more NICs224. Each of the CPUs204, GPUs220, and NICs224may communicate via an interface216.

The NIC224may comprise one or more circuits capable of acting as an interface between components of the system200, such as the CPU204and the GPU220. The NIC224may also act as an interface between components of the system200and the network104. The NIC224may enable data transmission and reception such that peer computing systems108a-dmay communicate with the system200. A NIC224may comprise one or more of a peripheral component interconnect express (PCIe) card, a USB adapter, and/or may be integrated into a PCB such as a motherboard. The NIC224may be capable of supporting any number of network protocols such as Ethernet, Wi-Fi, fiber channel, etc.

As described herein, the NIC224may be capable of receiving packets from one or more peer computing systems108a-dvia the network104. The NIC224may process a header of each received packet to determine whether each packet should be handled by the CPU204or the GPU220. In some implementations, the NIC224may be in direct communication with each of the GPU(s)220and the CPU(s)204via the interface216as well as in external communication with the network104via, for example, Ethernet in combination with TCP.

One or more CPUs204of the system200may each comprise one or more circuits capable of executing instructions and performing calculations. The CPUs204may be capable of interpreting and processing data received by the system200via the NIC224. CPUs204of a system200may each comprise one or more arithmetic logic units (ALUs) capable of performing arithmetic and/or logical operations, such as addition, subtraction, and bitwise operations. The CPUs204may also or alternatively comprise one or more control unit (CUs) which may be capable of managing the flow of instructions and data within the CPU204. CUs of the CPU204may be configured to fetch instructions from CPU memory208or system memory212, decode the instructions, and direct appropriate components to execute operations based on the instructions.

A CPU204of the system200may include, for example, a reduced instruction set computing (RISC) processor, a complex instruction set computing (CISC) processor, a digital signal processor (DSP) such as a baseband processor, an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), a radio-frequency integrated circuit (RFIC), another processor (including those discussed herein), or any suitable combination thereof. Similarly, a GPU220as described herein may include a processor228such as a streaming multiprocessor (SM), a RISC processor, a CISC processor, a DSP, a baseband processor, an ASIC, an FPGA, an RFIC, another processor (including those discussed herein), or any suitable combination thereof.

A CPU204and/or a processor228of a GPU220as described herein may incorporate multiple processing cores, allowing the CPU204(and/or the GPU220) to execute multiple instructions simultaneously, and/or may be capable of performing hyperthreading to execute multiple threads concurrently.

One or more GPUs220of the system200may each comprise one or more circuits capable of acting as specialized processing components to handle computationally-intensive tasks, such as rendering graphics and performing complex mathematical calculations. GPUs220may be capable of parallel execution of general-purpose tasks alongside the CPUs204.

As noted above, a GPU220may comprise one or more streaming multiprocessors (SMs), CUs, or processors228, which may be responsible for executing instructions in parallel. Each SM, CU, or processor228of a GPU220may contain one or more processing cores or ALUs which may be capable of performing arithmetic and/or logical operations concurrently.

One, some, or all GPUs220of the system200may be capable of executing tasks such as scientific simulations, machine learning, and data analysis. For example, a GPU220of the system200may be designed for operation in workstation environments, such as for performing scientific simulations, executing and/or training machine learning models, performing data analysis, etc.

The GPU220may execute one or more kernels. Kernels executed by the GPU220may perform specific, parallelizable tasks on the GPU220. Such kernels may be written using GPU programming languages or frameworks, such as CUDA.

The interface216of the system200may comprise one or more circuits capable of connecting peripheral devices such as the NIC224, one or more GPUs220, and one or more CPUs204to a motherboard of the system200, as well as one or more devices used for system memory212. The interface216may comprise one or more high-speed lanes. Each lane may be, for example, a serial lane, and may consist of a pair of signaling wires for transmitting and/or receiving data. The interface216may be, for example, a PCIe bus.

The device(s) used for system memory212may include solid-state drives (SSDs), such as NVMe SSDs. The system memory212may be capable of providing fast and efficient data access and storage. Each of the CPU204, GPU220, and NIC224may be capable of sending data to and reading data from the system memory212via the interface216. Each of the CPU204, GPU220, and NIC224. Illustratively, but without limitation, the CPU204may have access to dedicated CPU memory208and the GPU220may have one or more devices dedicated to GPU memory232.

The disclosed systems and methods may be adaptable and usable for both systems with and without GPUs220. As described above, embodiments of the present disclosure may include a GPU220performing data packet preparation, scheduling, sending, and/or transmission. In some embodiments, the CPU204may instruct the GPU220(or multiple GPUs220) to perform various tasks. Such platforms may employ GPU accelerated signal processing, such as by using GDA-KI to enable the GPU220to prepare network work descriptors or WQEs and submit such descriptors to the NIC224.

Because the CPU204and GPU220may have different memory spaces, data that is processed by the GPU220is moved from the CPU204to the GPU220before the computation starts, and the results of the computation are moved back to the CPU204once processing has completed. The system memory212, on the other hand, represents global memory that is accessible to all threads as well as the host (e.g., the CPU204). Global memory is allocated and deallocated by the host and may be used to initialize the data that the GPU220will work on.

Referring now toFIG.3, additional details of a processing unit300interacting with an IO device304will be described in accordance with at least some embodiments of the present disclosure. A CPU204and/or GPU220may correspond to examples of a processing unit300. Other examples of a processing unit300include, without limitation, a DPU, a microprocessor, a collection of CPUs204, a collection of GPUs220, a collection of DPUs, a collection of microprocessors, and the like.

A NIC224may correspond to an example of an IO device304. It should be appreciated that embodiments of the present disclosure may apply to other queue-based producer-consumer models, such as the NVMe submission queue. In this context, an IO device304may correspond to a different type of device, such as a memory device.

The processing unit300is illustrated to include a plurality of SMs328. Within the context of the processing unit300, the basic unit of execution may be referred to as a warp. A warp is a collection of threads that are executed simultaneously by an SM328. Multiple warps can be executed on an SM328at once.

When a program on the processing unit300invokes a kernel grid, the blocks of the grid are enumerated and distributed to SMs328with available execution capacity. The threads of a thread block execute concurrently on one SM328, and multiple thread blocks can execute concurrently on one SM328. As thread blocks terminate, new blocks may be launched on the vacated SMs328.

The mapping between warps and thread blocks can affect the performance of the kernel. Mapping can be achieved by assigning identifiers (IDs) to threads (e.g., a thread ID) and by using an index of a thread. The index of a thread and its thread ID relate to each other as follows: for a 1-dimensional block, the thread index and thread ID are the same; for a 2-dimensional block, the thread index (x,y) has thread ID=x+yDx, for block size (Dx,Dy); for a 3-dimensional block, the thread index (x,y,x) has thread ID=x+yDx+zDxDy, for block size (Dx,Dy,Dz).

When a kernel is started, the number of blocks per grid and the number of threads per block are fixed (gridDim and blockDim). The processing unit300makes four pieces of information available to each thread: (1) the thread index (threadIdx); (2) The block index (blockIdx); (3) the size and shape of a block (blockDim); and (4) the size and shape of a grid (gridDim).

The processing unit300may transfer data312to the IO device304and/or receive data312from the IO device304using the WQ316, the DBR320, and a completion queue (CQ)324. The WQ316, DBR320, and CQ324may enable the SMs328of the processing unit300to interact directly with the IO device304.

WQs316may be used to hold WQEs, which represent operations to be performed. The WQ316may operate as a repository for WQEs. Each WQE in the WQs316may contain information about the operation such as the type of operation, the location of the data, and other control information. Thus, each WQ316consumes memory for every WQE it holds, impacting the overall memory consumption of a QP.

CQs324may be utilized to track a completion status of WQEs. When an operation associated with a WQE is completed, a completion event may be generated and placed into a CQ316. A QP may be associated with one or more CQs324. Therefore, for each QP, memory may be consumed for the storage of completion events within these CQs324.

DBRs320may be used as a notification mechanism when new WQEs are put on the WQs316or to solicit completion notifications from the CQs324. Each QP may be associated with a corresponding DBR.

In operation, the processing unit300may interact with the IO device304according to a series of operations (1)-(8). In a first operation (1), an application launches a kernel that produces data in the memory of the processing unit. The memory of the processing unit may include CPU memory208, GPU memory232, and/or system memory212. This operation may be viewed as an operation that locks a QP.

In a second operation (2), the application calls an operation (e.g., nvshmem_put) to communicate with another device (e.g., a peer computing system108a-d, a peer device112, IO device304, etc.). In this step, an SM328is used to create a work descriptor. The work descriptor (e.g., WQE), may be written directly to the WQ316buffer. As mentioned above, the WQ316may reside directly on memory of the processing unit300(e.g., as CPU memory208and/or GPU memory232). In some embodiments, this operation may be viewed as an operation in which one or a plurality of WQEs are created.

In a third operation (3), the SM328updates the DBR320buffer by creating a memory barrier. As noted above, the DBR320buffer may be on the memory of the processing unit300, thereby making the DBR320buffer directly accessible to the SM328.

In a fourth operation (4), the SM328notifies the IO device304of the WQE(s). This notification may be achieved by writing to the DB308register of the IO device304.

In a fifth operation (5), the IO device304reads the WQE(s) in the WQ316buffer. This operation may be executed using a remote direct memory access (RDMA), in which the IO device304is able to access information stored in the WQ316buffer without involvement by the SM328.

In a sixth operation (6), the IO device304reads that data312from the memory of the processing unit300. The data312may also be read using RDMA or a similar approach. The location of the data312may be obtained, at least in part, with data retrieved from the corresponding WQE.

In a seventh operation (7), the IO device304may transfer the data312to a remote node. For instance, the IO device304may transfer the data312to a peer computing system108a-d, a peer device112, or the like.

In an eighth operation (8), the IO device304may notify the processing unit300that the operation is completed by writing to the CQ324buffer. RDMA may also be used to write to the CQ324buffer. At this point, a QP may be unlocked.

With the basic operations in mind, improvements involving warp-level coalescing of WQE slot reservations and WQE creations as well as coalescing of memory barriers and DBR updates will now be described. Referring initially toFIG.4, details of achieving warp-level coalescing of WQE slot reservation and WQE creation will be described in accordance with at least some embodiments of the present disclosure.

The proposed solution aims at increasing the parallelism of the process described in connection withFIG.3for threads within the same CTA. Advantageously, the process described in connection withFIG.3does not necessarily have to be changed to support the improvements described herein. In this approach, the number of WQE submissions (e.g., WQE0-WQE 31) is reduced by electing a thread from those present in the warp to atomically update the QP head pointer to reserve space in the WQ buffer408(which may be similar or identical to WQ316) for the WQEs412from all present threads. Once the space has been reserved in the WQ buffer408, the threads can create WQEs412in parallel and a single submit thread404is selected to submit the WQEs412to the IO device304. The IO device304is unaware that the WQEs412were coalesced and processes the WQEs412in the normal fashion, as described in connection withFIG.3.

Further details of the WQE412coalescing may be understood with reference to the following pseudocode.

As shown above, it may not be required to coalesce WQE submissions, meaning that the process works at least as efficiently as the process ofFIG.3. However, when opportunities exist to coalesce two or more WQE submissions, then computational improvements can be made relative to the process ofFIG.3. As noted above, this process may be utilized by some or all of the threads within a warp.

With reference now toFIG.5, additional details related to coalescing WQE submissions from multiple submitter threads will be described. The process ofFIG.5may be used alone or in combination with the process depicted and descried in connection withFIG.4. Additional details of coalescing WQE submissions from multiple submitter threads may also be better understood with reference to the following pseudocode.

Code 2: Coalescing WQE submissions from multiple submitter threadssubmit_wqes(qp_t *qp, void *wqe_ptr, base_idx, num_wqes)1.new_idx = base_idx + num_wqes2.if (is_qp_shared_between_ctas(qp))a.// CUDA memory model requires us to call memorybarrier if we are using different CTAs. However,we don't have to from the GPU architecture'sperspective. So, we can change this checking tois_qp_shared_between_sm(qp) as well.b.Memory barrier // Push my WQE writes to the point ofconsistency before notifying other threads. This step isunnecessary if all threads belong to the same CTA.3.while (READ_ONCE(&submission_head) < base_idx)a.; // wait here until all prior WQEs from other threadsare ready4.if ((new_idx == READ_ONCE(&wqe_head)) || (new_idx −submission_tail >= threshold))// wqe_head is updated by item 4 of Code 1. This thread is stillthe submitter and proceeds to submit the pending WQEs.// We also check submission_tail and threshold and dosubmission even if there is another thread that can submit forus. The reason is that waiting for that thread (for coalescing)will add latency. It is worth regularly submitting WQEs to theNIC so that the NIC can start the processing rather thanwaiting for more potential coalescing.a.submission_tail := new_idxb.if (QP has DBR)i.Memory barrierii.Update DBRc.Memory barrierd.Write DB5.else// wqe_head advanced beyond new_idx, another thread willeventually enter submit_wqes. That thread can write DB and thatwill automatically submit the WQEs of this thread.6.WRITE_ONCE(&submission_head, new_idx);

The approach illustrated inFIG.5and highlighted in Code 2 handles scenarios where 1) a QP is dedicated to a CTA, and 2) a QP is shared among CTAs differently. A notable difference is the utilization of step 2.a when shared QPs are employed. From the perspective of the process, this difference is subtle. However, the difference delivers improved computational performance because the memory barrier is expensive compared to the rest of the post-send algorithm. Libraries that implement GDA-KI such as NVSHMEM have prior knowledge whether the QP is dedicated or shared. The libraries can choose the correct approach to gain maximum performance.

Step 4 of Code 2 also checks the “threshold” and does submission without waiting for further coalescing opportunities. The reasoning is that the submission latency can be very high when there are many threads producing WQEs412and entering Code 2. Without going to step 4.d, the IO device304could be idle since the IO device304does not normally read the WQ buffer408until it receives the DB signal. Accordingly, regularly submitting work to the IO device304is better than waiting for more coalescing. The approach illustrated in Code 2 andFIG.5provides a method using predetermined “threshold”, but it does not prevent use of a more sophisticated approach such as monitoring whether the IO device304is now idle or still busy.

Code 2 andFIG.5further illustrate the use of voting counters504. Use of voting counters504and atomicAdd plus counter checking helps to avoid locking QPs. In this way, multiple threads can create WQEs412in parallel. One thread is then elected to submit all created WQEs412to the IO device304. The other threads are thereby freed to exit the communication critical path, thereby enhancing performance.

The memory and/or storage devices of the system200may store instructions such as software, a program, an application, or other executable code for causing at least any of the CPU204, the GPU220, and the NIC224to perform, alone or in combination, any one or more of the methods discussed herein. The instructions may in some implementations reside, completely or partially, within at least one of the memory/storage devices illustrated inFIGS.1-5, or any suitable combination thereof.

In some embodiments, the electronic device(s), network(s), system(s), chip(s), circuit(s), or component(s), or portions or implementations thereof, ofFIGS.1-5, or some other figure herein, may be configured to perform one or more processes, techniques, or methods as described herein, or portions thereof. Such processes may be as depicted inFIGS.6-8and as described below.

Referring now toFIG.6, a first method600will be described in accordance with at least some embodiments of the present disclosure. The first method600may be performed alone or in combination with any other method described herein. Moreover, steps from other methods may be incorporated into the first method600, and vice versa. The first method600may correspond to a method of improving computational performance by coalescing WQE submissions.

The method600may start with a start operation (step604). Initiation of the method600may begin when one or more SMs328initiate the process of leveraging an IO device304. In some embodiments, the SMs328of the processing unit300may submit WQEs to the IO device304as part of leveraging the IO device304.

The method600may continue by electing a thread from among a plurality of threads to atomically update a queue head pointer (step608). Electing a single thread from among a plurality of threads helps to improve efficiencies associated with WQE submission. In particular, if a submitter thread were not elected and each thread were left to follow its own WQE submission process, either (1) the network QP would have to be locked, thereby limiting concurrency or (2) each thread would have to create its own QP, which may consume hundreds of GB of memory from the processing unit300. Selecting a single submitter thread for a plurality of threads helps to avoid these issues.

The method600may further continue by using the queue head pointer that was updated from the elected thread to reserve space in a plurality of memory registers in the WQ buffer418(step612). The number of memory registers reserved by the elected thread may be dictated by the number of threads in the warp (e.g., the number of threads having the WQE submissions coalesced with the elected submitter thread) and the size required for each thread. In some embodiments, the number of WQE slots is determined at the time of coalescing based on the number of threads and the size required for each thread's WQE412.

The method600continues with the elected submitter thread submitting the WQEs for all of the plurality of threads (step616). This approach frees up the non-elected threads to create WQEs concurrently.

Referring now toFIG.7, a second method700will be described in accordance with at least some embodiments of the present disclosure. The second method700may be performed alone or in combination with any other method described herein. Moreover, steps from other methods may be incorporated into the second method700, and vice versa. The second method700may correspond to a method of improving computational performance by coalescing WQE writes.

The method700begins by determining whether a QP is dedicated to a particular thread or is shared among a plurality of threads before allocating memory to the QP (step704). The method700may continue by determining that a QP is shared by two or more threads (step708).

When it is determined that the QP is shared by two or more threads, the method700may continue by checking whether QPs are shared between a common manager unit (step712). One example of a manager unit is the SM328.

The method700may further continue when it is determined that QPs are shared with a common manager unit (step716). In response to determining that two or more QPs are shared with a common manager unit, the method700continues by issuing a single WQE write back to the IO device304for the multiple WQE submissions (step720). Coalescing WQE writes in this way helps to reduce the number of overall WQE writes and may improve interactions between the processing unit300and IO device304.

Referring now toFIG.8, a third method800will be described in accordance with at least some embodiments of the present disclosure. The third method800may be performed alone or in combination with any other method described herein. Moreover, steps from other methods may be incorporated into the third method800, and vice versa. The third method800may correspond to a method of improving computational performance by enabling parallelism and also transferring ownership of submitting WQEs from one thread to another.

The method800may begin by electing a first threads to submit one or more WQE (step804). The first thread may be elected from a plurality of threads and the first thread may be elected to submit one, some, or all of the WQEs on behalf of the plurality of threads.

The method800may continue by transferring ownership of submitting the WQEs from the first thread to a second thread (step808). The second thread may also belong to the plurality of threads from which the first thread was originally elected. Alternatively or additionally, the second thread may not have belonged to the plurality of threads from which the first thread was originally elected. In other words, the thread that is initially elected to submit WQEs can change in response to additional threads arriving and inserting WQEs into the WQ. In some embodiments, it may be possible to avoid a lock by passing mutual exclusion directly from one thread (e.g., the first thread) to a new thread (e.g., the second thread). This transfer may be achieved through the submission_head parameter.

Even if it is possible to handoff ownership of the WQE submission from one thread to another, the originally-elected thread (e.g., the first thread) may still be enabled to submit pending WQEs to the IO device304in response to a given time period having elapsed (step812). In other words, an upper limit may be placed on how long a given WQE can be delayed to facilitate coalescing.

Aspects of the above systems and networking device include any one or more of the features as substantially disclosed herein in combination with any one or more other features as substantially disclosed herein.

Aspects of the above systems and networking device include any one of the aspects/features/embodiments in combination with any one or more other aspects/features/embodiments.

Aspects of the above systems and networking device include any use of any one or more of the aspects or features as disclosed herein.