Patent Description:
Computational accelerators are commonly used in offloading computation-intensive tasks from a central processing unit (CPU, also referred to as the host processor) of a host computer. Such accelerators typically comprise hardware logic that is dedicated to particular types of operations, such as cryptography or data compression, and can thus perform these operations much faster than software-driven computation by the CPU.

Methods for performing computational tasks using accelerators are known in the art. For example, <CIT> describes systems and methods for processing a non-volatile memory express over fabric (NVMe-oF) command at a Peripheral Component Interconnect Express (PCIe) attached accelerator device. Processing the NVMe-oF commands include receiving from a remote client, at a NVMe interface associated with the accelerator device, a Transport Control Protocol/Internet Protocol (TCP/IP)-encapsulated NVMe-oF command, and performing, at the accelerator device, functions associated with the NVMe-oF command that would otherwise be performed at a central processing unit (CPU).

<CIT> describes a packet processing apparatus that includes a first interface coupled to a host processor and a second interface configured to transmit and receive data packets to and from a packet communication network. A memory holds context information with respect to one or more flows of the data packets conveyed between the host processor and the network in accordance with a reliable transport protocol and with respect to encoding, in accordance with a session-layer protocol, of data records that are conveyed in the payloads of the data packets in the one or more flows. Processing circuitry, coupled between the first and second interfaces, transmits and receives the data packets and includes acceleration logic, which encodes and decodes the data records in accordance with the session-layer protocol using the context information while updating the context information in accordance with the serial numbers and the data records of the transmitted data packets.

<CIT> describes a method for facilitating remote direct memory access (RDMA) for microservers, which includes generating queue pair (QPs) in a memory of an input/output (I/O) adapter of a microserver chassis having a plurality of compute nodes executing thereon, the QPs being associated with a remote direct memory access (RDMA) connection between a first compute node and a second compute node in the microserver chassis, setting a flag in the QPs to indicate that the RDMA connection is local to the microserver chassis, and performing a loopback of RDMA packets within the I/O adapter from one memory region in the I/O adapter associated with the first compute node of the RDMA connection to another memory region in the I/O adapter associated with the second compute node of the RDMA connection.

In order to illustrate the invention, aspects and embodiments which may or may not fall within the scope of the claims are described herein.

An embodiment that is described herein provides a network node that includes a bus switching element, and a network adapter, a computational accelerator and a host, all coupled to communicate with each other via the bus switching element. The network adapter is configured to communicate with remote nodes over a communication network. The host is configured to establish a RDMA link between the accelerator and a RDMA endpoint by creating a Queue Pair (QP) to be used by the accelerator for communication with the RDMA endpoint via the RDMA link. The accelerator is configured to exchange data, via the network adapter, between a memory of the accelerator and a memory of the RDMA endpoint.

In some embodiments, the RDMA endpoint includes a client process running locally on the host, and the RDMA link includes at least the local client process, the accelerator, the PCIe switch, and the network adapter. The network adapter may be configured to read the data directly from the memory of the accelerator, to apply to the read data transport layer processing for producing packets for transmission, in response to detecting that the packets are destined to the network adapter, to loop the packets back to the network adapter, to recover the data from the looped back packets, and to write the recovered data directly to the memory of the host. Or, the network adapter may be configured to read the data directly from the memory of the host, to apply to the read data transport layer processing for producing packets for transmission, in response to detecting that the packets are destined to the network adapter, to loop the packets back to the network adapter, to recover the data from the looped back packets, and to write the recovered data directly to the memory of the accelerator.

In some embodiments, the RDMA endpoint is located on a remote node accessible over the communication network, and the RDMA link includes at least the accelerator, the PCIe switch, the network adapter, the communication network and the RDMA endpoint. The network adapter may be configured to receive from the remote node packets carrying the data, to apply to the received packets transport layer processing, to recover the data from the processed packets, and to write the recovered data directly to the memory of the accelerator. Or, the network adapter may be configured to read the data directly from the memory of the accelerator, to apply to the read data transport layer processing for producing packets for transmission, in response to detecting that the produced packets are destined to the remote node, to transmit the produced packets to the remote node via the communication network.

In some embodiments, the RDMA endpoint includes a remote client process running on a remote network node, and the host is configured to create the queue pair in response to receiving from the remote client process a request to setup the RDMA link between the remote client and the accelerator. In other embodiments, the RDMA endpoint includes a remote accelerator residing in a remote network node, and the accelerator is configured to exchange the data between the memory of the accelerator and a memory of the remote accelerator, using RDMA communication.

The accelerator may include a shared receive buffer including multiple receive buffers, and the host is configured to create the queue pair by creating a send queue in the memory of the accelerator, creating a shared receive queue in the memory of the host and posting receive requests in the shared receive queue before sending messages over RDMA to the accelerator, each receive request posted is associated with a respective receive buffer in the shared buffer of the accelerator.

There is additionally provided, in accordance with an embodiment that is described herein, a method, including, in a network node that includes a network adapter, an accelerator and a host, all coupled to communicate via a Peripheral Components Interconnect Express (PCIe) switch, and the network adapter communicates with remote nodes over a communication network, establishing, by the host, a RDMA link between the accelerator and the RDMA endpoint by creating a Queue Pair (QP) to be used by the accelerator for communication with the RDMA endpoint via a RDMA link. Data is exchanged by the accelerator, via the network adapter, between a memory of the accelerator and a memory of the RDMA endpoint.

In some embodiments, the RDMA endpoint comprises a client process running locally on the host, and wherein the RDMA link comprises at least the local client process, the accelerator, the PCIe switch, and the network adapter. Exchanging the data may comprise reading the data directly from the memory of the accelerator, applying to the read data transport layer processing for producing packets for transmission, in response to detecting that the packets are destined to the network adapter, looping the packets back to the network adapter, recovering the data from the looped back packets, and writing the recovered data directly to the memory of the host. Or, exchanging the data may comprise reading the data directly from the memory of the host, applying to the read data transport layer processing for producing packets for transmission, in response to detecting that the packets are destined to the network adapter, looping the packets back to the network adapter, recovering the data from the looped back packets, and writing the recovered data directly to the memory of the accelerator.

In some embodiments, the RDMA endpoint is located on a remote node accessible over the communication network, and wherein the RDMA link comprises at least the accelerator, the bus switching element, the network adapter, the communication network and the RDMA endpoint. Exchanging the data may comprise receiving from the remote node packets carrying the data, applying to the received packets transport layer processing, recovering the data from the processed packets, and writing the recovered data directly to the memory of the accelerator. Or, exchanging the data may comprise reading the data directly from the memory of the accelerator, applying to the read data transport layer processing for producing packets for transmission, in response to detecting that the produced packets are destined to the remote node, transmitting the produced packets to the remote node via the communication network.

In some embodiments, the RDMA endpoint comprises a remote client process running on a remote network node, and wherein creating the queue pair comprises creating the queue pair in response to receiving from the remote client process a request to setup the RDMA link between the remote client and the accelerator. In other embodiments, the RDMA endpoint comprises a remote accelerator residing in the remote node, and wherein exchanging the data comprises exchanging the data between the memory of the accelerator and a memory of the remote accelerator, using RDMA communication.

The accelerator may comprise a shared receive buffer comprising multiple receive buffers, wherein creating the queue pair comprises creating a send queue in the memory of the accelerator, creating a shared receive queue in the memory of the host and posting receive requests in the shared receive queue before sending messages over RDMA to the accelerator, wherein each receive request posted is associated with a respective receive buffer in the shared buffer of the accelerator.

These and other embodiments will be more fully understood from the following detailed description of the embodiments thereof, taken together with the drawings in which:.

Embodiments that are described herein provide systems and methods for efficient and scalable RDMA-based communication between a computational accelerator and local or remote host.

In various cloud and other networking applications such as disaggregation and distributed heterogenous computation, an accelerator in one host computer provides acceleration services to a remote host computer over a communication network.

An accelerator typically comprises a dedicated coprocessor, computational logic or an integrated circuit, designed to perform certain computational operations efficiently, e.g., in hardware. For high performance acceleration, large amounts of data typically need to be transferred efficiently and with low latency to and from a memory of the accelerator. The accelerator may provide acceleration services to a local and/or remote host(s). As an example, an accelerator may be assigned to accelerate a job together with other accelerators, in the same communication network.

The accelerator is typically accessed by the local host via a suitable bus such as a PCIe bus. In principle, accessing the accelerator may be mediated via the memory of the local host. This, however, would require the host processor coupled to the bus to handle data transfer operations, thus reducing the amount of computational resources remaining for the host to handle other tasks. In another approach, a full RDMA engine could be implemented in the accelerator. Implementing full RDMA transport layer functionality within the accelerator, however, is typically impractical due to high costs and limited hardware resources in the accelerator.

Network adapters typically implement various service types and transport operations, including Remote Direct Memory Access (RDMA) operations. An element that communicates using a RDMA protocol is also referred to herein as "RDMA endpoint. " A RDMA endpoint communicates with the transport layer of the communication network (e.g., an InfiniBand fabric or Ethernet) by manipulating a transport service instance, known as a Queue Pair (QP), made up of a send work queue (SQ) and a receive work queue (RQ). To send and receive messages over the communication network using a network adapter, a RDMA endpoint initiates work requests (WRs), which cause work items, called Work Queue Elements (WQEs), to be placed onto the appropriate work queues. In the present context and in the claims, a RDMA endpoint comprises an element that initiates WRs such as a client process running on a local or remote host, or any peripheral or coprocessor (e.g., an accelerator). The link established between two RDMA endpoints is also referred to herein as a "RDMA link.

A WQE typically has a data buffer (or buffers, e.g., in case of using a scatter gather list) associated with it, to be used for holding the data that is to be sent or received in executing the WQE. The network adapter executes the WQEs and thus communicates with the corresponding QP of the network adapter at the other end of the link.

A WQE corresponding to a Send Request (SR) specifies a source buffer containing a message to be transmitted. A WQE corresponding to a Read Request (RR) specifies a destination buffer for storing a received message. The completion of a given WR is typically indicated by placing a Completion Queue Entry (CQE) in an appropriate Completion Queue (CQ) accessible by the WR initiator. A CQE comprises a control message reporting the completion and outcome of executing a corresponding WR by the network adapter. For example, a CQE corresponding to a SR may report the locations to which data was scattered at the destination (an address in target memory), the size of the data transferred, a data verification result performed by the network adapter (e.g., CRC), timestamps and the like. A CQE may implicitly indicate that data buffered for sending is allowed to be overwritten, thus serving functionality of flow control.

In some disclosed embodiments, a network adapter such as a Network Interface Card (NIC) can access the accelerator memory and a host memory directly. Moreover, the network adapter implements RDMA transport layer functionality and applies Quality of Service (QoS) policies on behalf of the accelerator. The network adapter provides fast RDMA-based communication between the accelerator and client processes running on the local host or on remote hosts. RDMA communication between accelerators residing in different hosts is also supported. In the disclosed embodiments, messages that are exchanged between the accelerator and a remote node are not mediated via the host memory, which reduces the host bus bandwidth and the host memory bandwidth, and reduces communication overhead from the host.

The disclosed embodiments may be used, for example, for accessing remote accelerators, sharing accelerators among multiple users, and performing distributed computations in which multiple accelerators belonging to different host computers participate.

Consider a network node comprising a bus switching element (e.g., a PCIe switch), and further comprising a network adapter, an accelerator and a host, all coupled to communicate via the bus switching element. The network adapter is configured to communicate with remote nodes over a communication network. The accelerator is configured to exchange data, via the network adapter, between a memory of the accelerator and a memory of a RDMA endpoint. The host is configured to establish a RDMA link between the accelerator and the RDMA endpoint by creating a Queue Pair (QP) to be used by the accelerator for communication with the RDMA endpoint via the RDMA link.

In some embodiments, the RDMA endpoint comprises a client process running locally on the host, and the RDMA link comprises at least the local client process, the accelerator, the PCIe switch, and the network adapter. In these embodiments, the network adapter reads data directly from the memory of the accelerator (or from the memory of the host) and applies to the read data transport layer processing for producing packets for transmission. In response to detecting that the packets are destined to the same network adapter, the network adapter loops the packets back to itself, recovers the data from the looped back packets, and writes the recovered data directly to the memory of the host (or to the memory of the accelerator).

In some embodiments, the RDMA endpoint is located on a remote node accessible over the communication network, and the RDMA link comprises at least the accelerator, the PCIe switch, the network adapter, the communication network and the RDMA endpoint. In such embodiments, the network adapter receives from the remote node packets carrying data, applies to the received packets transport layer processing, recovers the data from the processed packets, and writes the recovered data directly to the memory of the accelerator. In the opposite direction, the network adapter reads data directly from the memory of the accelerator and applies to the read data transport layer processing for producing packets for transmission. In response to detecting that the produced packets are destined to the remote node, the network node transmits the produced packets to the remote node via the communication network.

In an embodiment, the RDMA endpoint comprises a remote client process running on a remote network node, and the host creates the queue pair in response to receiving from the remote client process a request to setup the RDMA link between the remote client and the accelerator. In another embodiment, the RDMA endpoint comprises a remote accelerator residing in the remote node, and the accelerator is configured to exchange data between the memory of the accelerator and a memory of the remote accelerator, using RDMA communication.

In some embodiments, the accelerator comprises a shared receive buffer comprising multiple receive buffers. In such embodiments, the host is configured to create the queue pair by creating a send queue in the memory of the accelerator, creating a shared receive queue in the memory of the host and posting receive requests in the shared receive queue before sending messages over RDMA to the accelerator. Each of the receive requests posted is associated with a respective receive buffer in the shared buffer of the accelerator. Using a shared buffer enables to reduce the memory footprint for receiving data in the accelerator and yet supporting large data bursts.

In the disclosed techniques the network adapter provides the accelerator with a message-based multi-channel interface for communicating with local and remote clients. The multi-channel interface requires only little hardware resources on the accelerator itself and is therefore highly scalable. The local and remote hosts are not confined to any specific software architecture. For example, Virtual Machine (VM), container, multi-process or any other suitable software architecture can be used.

By using the disclosed techniques, the overhead to the local host processor caused by handling communication with the local accelerator reduces, and therefore performance metrics such as memory bandwidth of the host memory and latency in communicating with the local accelerator improve significantly.

<FIG> is a block diagram that schematically illustrates a computing system <NUM> that supports accelerated computations, in accordance with an embodiment that is described herein.

In computing system <NUM>, a host computer <NUM> (also referred to as a host or a network node) communicates with other hosts <NUM>, via a communication network <NUM>. Communication network <NUM> may comprise any suitable communication network operating in accordance with any suitable communication protocols, such as an InfiniBand™ (IB) switch fabric or an Ethernet network.

Host computer <NUM> comprises a processor, in the form of a central processing unit (CPU) <NUM>, and a host memory <NUM> (also referred to as a system memory), which are connected by a suitable bus <NUM>. In the present example bus <NUM> comprises a Peripheral Component Interconnect Express (PCIe) bus. Host computer <NUM> comprises a network adapter <NUM>, such as an IB Host Channel Adapter (HCA) or a Network Interface Card (NIC), which is coupled to bus <NUM> via any suitable switching element <NUM>. In the present example, switching element <NUM> comprises a PCIe switch. In the description that follows it is assumed that network adapter <NUM> implements RDMA functionality, such as a "RDMA NIC.

Network adapter <NUM> comprises a network interface <NUM>, which is coupled to communication network <NUM>, and a host interface <NUM>, which connects via PCIe switch <NUM> and bus <NUM> to CPU <NUM> and host memory <NUM>. Packet processing circuitry <NUM>, coupled between network interface <NUM> and host interface <NUM>, generates outgoing packets for transmission over communication network <NUM> and processes incoming packets received from the communication network, as will be described below. Among other tasks, packet processing circuitry <NUM> handles transport layer packet processing. Network interface <NUM>, host interface <NUM> and packet processing circuitry <NUM> typically comprise dedicated hardware logic. Alternatively or additionally, at least some of the functions of packet processing circuitry <NUM> may be implemented in software on a suitable programmable processor.

Host computer <NUM> comprises a computational accelerator <NUM>, coupled to PCIe switch <NUM>. Computational accelerator <NUM> is also referred to as "accelerator" for brevity. Each of CPU <NUM> and network adapter <NUM> has access to accelerator <NUM> via PCIe switch <NUM>. Consequently, accelerator <NUM> may provide accelerated computation services to local CPU <NUM>, a remote host computer <NUM> via communication network <NUM>, or both.

In some embodiments, accelerator <NUM> and network adapter <NUM> are implemented on the same board. This, however, is not mandatory. In alternative embodiments, accelerator <NUM> and network adapter are implemented on separate boards that are inserted into different PCIe slots of host computer <NUM>.

Accelerator <NUM> comprises an accelerator memory <NUM> and an acceleration engine <NUM>. In accelerator memory <NUM>, one or more transmission (TX) buffers <NUM> and one or more reception (RX) buffers <NUM> are respectively used for storing data pending transmission to and data received from the local or a remote host. Accelerator <NUM> is configured to apply a predefined computational operation or function to data, e.g., in a RX buffer <NUM>, which produces data result, e.g., in a TX buffer <NUM>. Accelerator <NUM> may support one or more predefined computational operations such as, for example, cryptographic operations, data compression and decompression, mathematical and logical operations, or any other suitable operation.

Client processes <NUM>, (also referred to simply as "clients" for brevity) running on CPU <NUM>, such as processes generated by application software, communicate with clients <NUM> running on remote hosts <NUM> by means of QPs <NUM> on network adapter <NUM>. In some embodiments, each client <NUM> may be assigned multiple QPs <NUM>, which are used to communicate with different clients on various remote hosts <NUM>, using QPs <NUM>. In communicating using a given QP <NUM>, a client <NUM> to which that QP is assigned, posts WQEs to both the send queue and the receive queue of the QP.

In the disclosed embodiments, accelerator <NUM> communicates with local clients <NUM>, with remote clients <NUM> or both, using QP-based communication. To this end, accelerator <NUM> uses a dedicated QP for communicating with each respective client <NUM> or <NUM>. In some embodiments, each of the QPs that are assigned to accelerator <NUM> is split between accelerator <NUM> and CPU <NUM>. In the present example, the send queue parts (SQs <NUM>) of such QPs reside in the accelerator, and the receive queue parts of these QPs, however, are implemented in a Shared Receive Queue (SRQ) <NUM> in host <NUM>. Accelerator <NUM> consumes send requests (SRs) from SQs <NUM> to send data from TX buffers <NUM> and consumes receive requests (RRs) from SRQ <NUM> for receiving data into RX buffers <NUM>.

A driver program <NUM>, running on CPU <NUM>, manages the communication operations of clients <NUM> and accelerator <NUM> via network adapter <NUM>. For example, driver <NUM> allocates buffers for sending and receiving data carried, e.g., by packets <NUM> to and from remote clients <NUM> on peer devices (hosts <NUM>) using any of the participating QPs on host <NUM> and hosts <NUM>.

In some embodiments, SRQ <NUM> comprises a cyclic queue. In such embodiments, driver <NUM> posts WQEs in the form of receive requests (RRs) to SRQ <NUM> before communication starts. Each receive request in the SRQ is associated with a respective RX buffer <NUM>. In some embodiments, RX buffers <NUM> are divided into strides of a certain, uniform size, which may be set by driver <NUM>. In an embodiment, driver <NUM> posts the RRs to SRQ <NUM> once, e.g., at initialization, and the network adapter automatically re-posts consumed WQEs.

<FIG> are diagrams that schematically illustrate schemes for RDMA-based communication with an accelerator, in accordance with embodiments that are described herein.

<FIG> depicts a scheme, according to which a client <NUM> communicates with local accelerator <NUM> of host <NUM> via network adapter <NUM>. In an embodiment, driver <NUM> establishes a Reliable Connection (RC) transport between client <NUM> and accelerator <NUM>, by creating a RC QP <NUM> to be used by network adapter <NUM> at the client side, and another RC QP split between a SQ <NUM> and SRQ <NUM> (as described above) to be used by accelerator <NUM>. Driver <NUM> (of <FIG>) configures a loopback (using loopback logic - not shown) between a QP (SQ <NUM> and SRQ <NUM>) used for communication between the network adapter and the accelerator, and a QP (<NUM>) used for communication between the network adapter and the local client. The loopback logic is owned by driver <NUM>.

In some embodiments, a loopback between an accelerator QP and a local client QP (or a QP used for communication with a remote network node) is implemented by performing a send WQE from the accelerator QP and a receive WQE from the other QP, or performing a receive WQE from the accelerator QP and a send WQE from the other QP.

In sending data (e.g., a message) from a client <NUM> to accelerator <NUM>, network adapter <NUM> reads data directly from a buffer in host memory <NUM>, via PCIe switch <NUM>, using Direct Memory Access (DMA) methods. The network adapter packetizes the read data into packets for transmission and redirects the packets back to the network adapter. Network adapter <NUM> recovers the data from the redirected packets and writes the recovered data directly to a destination RX buffer <NUM> in accelerator <NUM> via PCIe switch <NUM>, using DMA methods.

In sending data (e.g., a message) from accelerator <NUM> to a client <NUM>, network adapter <NUM> reads data from a TX buffer <NUM> of the accelerator via PCIe switch <NUM> using DMA, packetizes the data for transmission and redirects the packets back to the network adapter. The network adapter recovers the data from the redirected packets and writes the recovered data to a specified buffer in host memory <NUM> via PCIe switch <NUM>.

In <FIG>, remote client <NUM> of remote host <NUM> communicates with accelerator <NUM> of host <NUM> via communication network <NUM> and network adapter <NUM>. Driver <NUM> establishes a reliable connection between client <NUM> and accelerator <NUM>, by creating a RC QP split between a SQ <NUM> and SRQ <NUM> (as described above) to be used by accelerator <NUM>. It is further assumed that client <NUM> on remote host <NUM> creates a RC QP <NUM> to be used by a network adapter at the remote client side. Driver <NUM> configures a loopback between a QP (SQ <NUM> and SRQ <NUM>) used for communication between the network adapter and the accelerator, and a QP used for communication with the remote network node using a QO <NUM>.

In sending data from remote client <NUM> to local accelerator <NUM>, network adapter <NUM> receives packets carrying data that were sent by the remote client over communication network <NUM>. Network adapter <NUM> recovers the data from the received packets and writes the data directly to a specified RX buffer <NUM> in accelerator <NUM> via PCIe switch <NUM>, using DMA.

In sending data from accelerator <NUM> to remote client <NUM>, network adapter <NUM> reads data directly from a TX buffer <NUM> via PCIe switch <NUM> using DMA. The network adapter packetizes the data and transmits the packets to communication network <NUM>. A network adapter at the remote host side receives the packets, recovers the data carried in these packets and writes the recovered data to a specified buffer in a memory of the remote host.

In <FIG> accelerator 50A of host <NUM> communicates using RDMA with accelerator 50B of remote host <NUM> via communication network <NUM> and network adapters 38A and 38B. Drivers <NUM> of hosts <NUM> and <NUM> establish a reliable connection between accelerators 50A and 50B, by creating respective RC QPs, split between SQ 84A and SRQ 86A to be used by accelerator 50A and split between SQ 84B and SRQ 86B to be used by accelerator 50B. Driver <NUM> in the local node configures a loopback between a QP (SQ 84A and SRQ 86A) used for communication between the network adapter and the accelerator, and a QP used for communication between the network adapter and the remote network node using a QP <NUM>. It is assumed that a similar loopback is also configured at the remote node.

In sending data (a message), e.g., from accelerator 50B to accelerator 50A, network adapter 38B reads data from a TX buffer <NUM> of accelerator 50B using DMA, packetizes the data into packets and transmits the packets to communication network <NUM>. Network adapter 38A receives the packets sent by the remote host over communication network <NUM>, recovers the data from the packets and writes the recovered data to a specified RX buffer <NUM> in accelerator 50A via PCIe switch <NUM>, using DMA.

The schemes in <FIG> above may be combined to create more complex configurations. For example, a first host may be connected via communication network <NUM> to second and third other hosts that each comprises a local accelerator (such as host <NUM>). The first host can communicate using RDMA with the accelerator of the second and with the accelerator of the third host, e.g., as described in <FIG>. Moreover, the accelerators of the second and third hosts can exchange messages with one another using RDMA-based communication as described in <FIG>. In some embodiments, using the above configuration, the first host can perform a chained accelerated operation efficiently. For example, the accelerator of the second host performs accelerated operation to data received from the first host to produce first result, and the accelerator of the third host performs an accelerated operation to the first result, sent by the second host, to produce a second result. The second host sends the second result to the first host. Note that the first host need not read the intermediate first result.

In some embodiments, network adapter <NUM> implements transport layer functionality on behalf of accelerator <NUM>. Such transport layer functionality comprises, for example, packetizing data into transport layer packets for transmission, and recovering data carried in transport layer packets received from communication network <NUM>.

Other transport layer tasks performed by network adapter <NUM> comprise handling reliable communication using packet ordering, sending acknowledgment messages notifying packets that were received successfully and retransmission of lost packets, verification of data received, e.g., using a Cyclic Redundancy Check (CRC) of a message content, managing flow control. Address translation, e.g., between addresses in the PCIe address space and the accelerator address space is typically performed as part of DMA operations.

In some embodiments, the PCIe link between accelerator <NUM> and network adapter <NUM> via PCIe switch <NUM> is configured to operate in a peer-to-peer (P2P) mode. This configuration allows efficient and fast transfer of data and control messages between accelerator <NUM> and network adapter <NUM> via PCIe switch <NUM>. Control messages comprise, for example, messages used for triggering Send and other RDMA operations, and messages used for notifying completion of WRs. Management operations such as, creation and teardown of various resources such as QPs, SRQ and CQ are carried out in software, e.g., by driver <NUM>.

In some embodiments, network adapter <NUM> handles Quality of Service (QoS) functionality. In an embodiment, driver <NUM> sets selected QoS policies specifying QoS-related parameters such as bandwidth budget, priorities among clients, bandwidth limitations and the like. Network adapter <NUM> enforces the QoS policies so that clients using accelerator <NUM> get respective portions of the available bandwidth budget in accordance with the QoS policies set by driver <NUM>.

In some embodiments, network adapter <NUM> may enforce a QoS policy by assigning (i) a maximum burst size for each client in each round, (ii) the round length and (iii) the number of slots that each client has in each round.

In some embodiments, network adapter <NUM> manages the runtime sharing of accelerator <NUM> resources and arbitrating among clients. Consider for example, two clients configured to share together a <NUM>% portion of the total accelerator bandwidth budget. Assuming an even sharing scheme, when both clients send data to the accelerator, each client gets <NUM>% of the accelerator bandwidth budget. When one of the clients, however, reduces its workload towards the accelerator, the other client may send data to the accelerator up to the <NUM>% bandwidth budget.

Communication with accelerator <NUM> may be carried out in various ways. In some embodiments, RX buffers <NUM> in accelerator <NUM> that are exposed to the PCIe bus are managed as a single striding Receive Memory Pool (RMP). Driver <NUM> initializes a SRQ <NUM> (in host <NUM>) and posts RRs to this SRQ, wherein each such RR points to a respective RX buffer <NUM>. Aspects of implementing a striding shared buffer are described, for example, in a <CIT>.

In an embodiment, accelerator <NUM> monitors memory utilization for each RR in the SRQ and notifies network adapter <NUM> when a used RX buffer <NUM> becomes available for receiving a subsequent message.

In an embodiment, accelerator <NUM> comprises a single Receive Completion Queue (RCQ). The network adapter notifies accelerator <NUM> that a received message has been written to a RX buffer <NUM> by writing a Completion Queue Entry (CQE) to this RCQ. The CQE specifies to which receive queue the message has been written. Note that accelerator <NUM> is triggered by the CQE and does not need to assemble PCIe transactions into a message, nor to verify the message content (e.g., based on a CRC of the message). Note that unlike conventional RDMA in which the host CPU receives a CQE by polling the completion queue, accelerator <NUM> receives a PCIe Transaction Layer Packet (TLP) containing the CQE, which triggers the operation with no further read or polling operation.

In some embodiments, for each client to which accelerator <NUM> sends messages, the accelerator has respective resources - a TX buffer <NUM>, a SQ <NUM> and a CQ. In some embodiments, to trigger transmission from a TX buffer to a given client, accelerator <NUM> posts a SR to the relevant SQ and writes a doorbell directly to network adapter <NUM> via PCIe switch <NUM> in a peer-to-peer mode. The doorbell signals the network adapter to read a SR from the SRQ. In alternative embodiments, multiple TX buffers <NUM> of accelerator <NUM> may be shared among multiple clients.

In some embodiments, accelerator <NUM> remaps addresses of accelerator memory <NUM> to be accessed by network adapter <NUM>. This allows exposing to the network adapter multiple message fragments as a contiguous range of memory addresses, which reduces communication overhead over PCIe. Specifically, in sending a fragmented message to the accelerator, the network adapter hides this from the accelerator by writing the fragmented message into the accelerator memory. In sending a fragmented message from the accelerator, the network adapter hides this from both the accelerator and the host.

In some situations, accelerator <NUM> may fail to process data received from the communication network sufficiently fast or may become blocked during transmission due to backpressure from the communication network. Limiting the transmission rate at the remote client side to prevent such occurrences may be carried out using flow control techniques. In an embodiment, the transport layer processing (e.g., as part of packet processing circuitry <NUM>) in network adapter <NUM> may apply flow control methods. For example, when the network adapter receives more data than it can handle in its pipeline, the network adapter may apply flow control by propagating backpressure to receive queues of the network adapter. This, however, can cause a head-of-line blocking among the accelerator channels sharing multiple RX buffers <NUM> because the bottleneck originates from the neck and blocks all the data before the data is split into the accelerator queues. Alternatively, flow control may be handled by host <NUM> at an application-layer level. In this embodiment, accelerator <NUM> exposes to the remote host (<NUM>) credits indicating the number of messages that the remote host is permitted to send to accelerator <NUM>. Credit counts may be updated e.g., explicitly by exchanging application-layer messages between the local and remote hosts, or implicitly by responding to remote requests. For example, accelerator <NUM> may send a message to a client indicating that the accelerator is ready for receiving data from that client.

Software elements in network node <NUM> may be configured in various way. In some embodiments, the host runs a kernel driver that exposes the resources of accelerator <NUM> to the user-space and enables network adapter <NUM> to access accelerator <NUM> using the PCIe peer-to-peer mode. In some embodiments, a daemon program in the user-space initiates the resources of the network adapter and accelerator and allows clients to establish connections. Note that some of the resources, e.g., SRQ <NUM> are global and therefore need to be created and initialized once after every reset event. Other resources such as QPs are allocated when a client establishes communication with the accelerator.

<FIG> is a flow chart that schematically illustrates a method for initializing an accelerator for RDMA-based communication, in accordance with an embodiment that is described herein. The method is typically executed once in response to a reset or powerup event.

The method begins at an acceleration configuration step <NUM>, at which driver <NUM> configures accelerator <NUM>. The configuration may comprise, for example, a name (address, ID) of the accelerator, accelerator capabilities, number of queues used, a QP number for each acceleration function, SRQ parameters, and the like. At a QP creation step <NUM>, driver <NUM> creates QPs to be used for communication by accelerator <NUM>. Specifically, driver <NUM> creates a SQs <NUM> within accelerator <NUM> and SRQ <NUM> within the host. SQs <NUM> and SRQ <NUM> will be used for sending data to and receiving data from local clients <NUM> and remote clients <NUM>. At a WQEs posting step <NUM>, driver <NUM> posts RR WQEs to SRQ <NUM>. Accelerator <NUM> will later receive data by network adapter <NUM> executing the RR WQEs in the SRQ sequentially and cyclically. Following step <NUM> the method terminates.

<FIG> is a flow chart that schematically illustrates a method for link establishment between accelerator and a remote network node, in accordance with an embodiment that is described herein.

The method begins at a channel establishment step <NUM>, at which local host <NUM> receives from a remote client <NUM> a control message to establish a RDMA link between accelerator <NUM> and the remote client. At a link establishment step <NUM>, driver <NUM> of local host <NUM> allocates a QP to be used by accelerator <NUM>. The SQ part of this QP resides on the accelerator and the RQ part is associated with SRQ <NUM> that was initialized at step <NUM>. It is assumed that the remote host allocates a corresponding QP to be used by the network adapter at the remote host side.

At a loopback configuration step <NUM>, driver <NUM> configures loopback logic in network adapter <NUM>. The loopback associates between a local QP used for communication between the network adapter and the accelerator, and another QP used for communication with a local client or with a remote network node. Following step <NUM>, the method terminates.

<FIG> is a flow chart that schematically illustrates a method for RDMA communication between an accelerator and a client on a remote host, in accordance with an embodiment that is described herein.

The method will be described with reference to remote host <NUM> communicating with accelerator <NUM> of local host <NUM> in <FIG>. It is assumed, that the method of <FIG> is executed after execution of the method of <FIG>, which means that accelerator <NUM> and the remote client are ready to communicate with one another.

The method splits into transmission and reception branches, which are typically executed in parallel.

In the transmission branch, the method begins at a preparation for transmission step <NUM>, with accelerator <NUM> placing data (e.g., a message) for transmission in a TX buffer <NUM>. At a WQE assembly step <NUM>, accelerator <NUM> assembles a SR that specifies transmission of the data placed in the TX buffer to the remote host. Note that the operation of assembling the SR at step <NUM> is much less complex than conventional assembling of a general RDMA WQE, because most of the SR information may be extracted from the data. The QP identifier may be extracted from a channel ID associated with the SR. Further at step <NUM>, the accelerator triggers network adapter <NUM> on local host <NUM> to consume the posted SR by writing a suitable control message to a doorbell on the network adapter via PCIe switch <NUM>. Doorbells in network adapter <NUM> may be implemented, for example, using registers, or any other suitable type of writable storage.

At a transmission step <NUM>, in response to the doorbell, network adapter <NUM> consumes the SR posted at step <NUM> and executes the SR by reading the data to be transmitted directly from Tx buffer using DMA methods. Network adapter <NUM> produces transport layer packets carrying the data and transmits the transport layer packets to remote host <NUM> over communication network <NUM>. At a completion step <NUM>, network adapter <NUM> posts a CQE to accelerator <NUM>, when the entire data (message) has been sent. The CQE indicates to accelerator <NUM> that the TX buffer recently used is now available for reuse.

In the reception method branch, the method begins at a packet reception step <NUM>, with network adapter <NUM> of local host <NUM> receiving transport layer packets from remote host <NUM>, over communication network <NUM> and consumes a RR from the SRQ specifying a target RX buffer in the accelerator for data carried in the received transport layer packets.

At a data recovery step <NUM>, network adapter <NUM> of local host <NUM> recovers the data from the transport layer packets, writes the recovered data directly (using DMA methods) to the RX buffer specified in the RR consumed at step <NUM>, and writes a CQE to accelerator <NUM> when all the data has been written successfully.

At an acceleration step <NUM>, accelerator <NUM> processes the data in the Rx buffer (by applying a specified function to the data) and notifies the network adapter when done. At a processing-completion step <NUM>, network adapter <NUM> sends a processing-completion notification to remote host <NUM> over communication network <NUM>.

Following each of steps <NUM> and <NUM>, the method loops back to steps <NUM> and <NUM> to perform further data transfer operations between accelerator <NUM> and remote host <NUM>.

Although the method of <FIG> refers to a client on a remote host, the accelerator need not be aware of whether the target client resides on remote host <NUM> and is accessible over communication network <NUM>, or the client runs on local host <NUM> and is accessible via PCIe switch <NUM>.

In the method of <FIG>, driver <NUM> establishes a RDMA link between accelerator <NUM> and a remote client <NUM>. Note that in embodiments in which the accelerator provides acceleration services to a local client, driver <NUM> similarly establishes a RDMA link between accelerator <NUM> and local client <NUM>.

<FIG> and <FIG> are diagrams that schematically illustrate elements and operations involved in RDMA communication between host <NUM> and local accelerator <NUM> via network adapter <NUM>, in accordance with embodiments that are described herein.

Each of <FIG> and <FIG> depicts host <NUM>, network adapter <NUM> and accelerator <NUM>, all coupled via PCIe switch <NUM>. Network adapter <NUM> and accelerator <NUM> communicate with one another in PCIe peer-to-peer mode.

In <FIG> and <FIG>, network adapter <NUM> comprises a DMA read module <NUM> (denoted DMA-R) for reading a message directly from a host buffer <NUM>. DMA-R <NUM> delivers data read from host buffer <NUM> to a transport layer transmitter <NUM> (denoted TR-TX). Network adapter <NUM> further comprises a transport layer receiver <NUM> (denoted TR-RX) coupled to a DMA write module <NUM> (denoted DMA-W) having direct access to a RX buffer <NUM> in accelerator <NUM>. Network adapter <NUM> further comprises one or more doorbells <NUM>. In some embodiments, TR-TX <NUM> and TR-RX <NUM> are part of packet processing circuitry <NUM>, and DMA-R <NUM> and DMA-W <NUM> are part of host interface <NUM>.

In <FIG>, a client <NUM> (not shown) on host <NUM> sends a message to local accelerator <NUM>. A sequence of numbered operations involved in this transaction are described herein. It is assumed that driver <NUM> has configured a loopback between RMP_RQ <NUM> and QP <NUM>.

In <FIG>, accelerator <NUM> sends a message to client <NUM> on local host <NUM>. A sequence of numbered operations involved in this transaction are described herein. It is assumed that driver <NUM> has configured a loopback between SQ <NUM> and QP <NUM>.

<FIG> and <FIG> are diagrams that schematically illustrate elements and operations involved in RDMA communication between remote host <NUM> and local accelerator <NUM> via network adapter <NUM>, in accordance with embodiments that are described herein.

In <FIG> and <FIG>, network adapter <NUM> comprises a DMA read module <NUM> (denoted DMA-R) for reading a message directly from a TX buffer <NUM>. DMA-R <NUM> delivers data read from TX buffer <NUM> to a transport layer transmitter <NUM> (denoted TR-TX). Network adapter <NUM> further comprises a transport layer receiver <NUM> (denoted TR-RX) coupled to a DMA write module <NUM> (denoted DMA-W) having direct access to a RX buffer <NUM> in accelerator <NUM>. In some embodiments, TR-TX <NUM> and TR-RX <NUM> are part of packet processing circuitry <NUM>, and DMA-R <NUM> and DMA-W <NUM> are part of host interface <NUM>. Note that DMA-R <NUM>, TR-TX <NUM>, TR-RX <NUM> and DMA-W <NUM> of <FIG> and <FIG> may be the same or different from those of <FIG> and <FIG> above.

In <FIG>, a client <NUM> on remote host <NUM> (not shown) sends a message to local accelerator <NUM>. A sequence of numbered operations involved in this transaction are described herein. It is assumed that driver <NUM> has configured a loopback between SRQ <NUM> and a QP used for communication with a remote network node. In <FIG>, TR-RX <NUM> receives transport layer packets from remote host <NUM> over communication network <NUM>. The sequence of operations thus comprises operations 4d, 4e, <NUM>, <NUM>, <NUM>, <NUM> and <NUM> that are essentially similar to same numbered operations in <FIG>. Note that operation number <NUM> in <FIG> differs from that of <FIG>. In <FIG>, network adapter <NUM> sends a CQE to remote host <NUM>, whereas in <FIG> to host <NUM>.

In <FIG>, accelerator <NUM> sends a message to client <NUM> on remote host <NUM>. A sequence of numbered operations involved in this transaction are described herein. It is assumed that driver <NUM> has configured a loopback between SQ_ <NUM> and a QP used for communication with a remote network node. In <FIG> TR-TX <NUM> sends the transport layer packets to remote host <NUM> over communication network <NUM>.

The sequence of operations thus includes operations <NUM>-4c, <NUM> and <NUM>, which are essentially similar to same numbered operations in <FIG>. Note that operations numbers 4c and <NUM> in <FIG> differ from those of <FIG>. In operation 4c of <FIG>, the transport layer packets produced by TR-RX <NUM> are transmitted to remote host <NUM> and are not looped back to the network adapter as in <FIG>. In addition, operation <NUM> of <FIG>, network adapter <NUM> receives a completion notification from remote host <NUM>, and not from host <NUM> as in <FIG>.

The configurations of computing system <NUM>, communication network <NUM>, network nodes <NUM> and <NUM>, including CPU <NUM>, network adapter <NUM>, accelerator <NUM> and bus switching element <NUM> of <FIG>, <FIG>, and <FIG> are given by way of example, which are chosen purely for the sake of conceptual clarity. In alternative embodiments, other suitable computing system, communication network, network nodes, CPU, network adapter, accelerator and bus switching element configurations can also be used.

Some elements of network node <NUM>, network adapter <NUM>, and accelerator <NUM> such as CPU <NUM>, host memory <NUM>, bus switching element <NUM>, packet processing circuitry <NUM> including TR-TX <NUM> and TRX-RX <NUM> and Host interface <NUM> including DMA-R <NUM> and DMA-W <NUM>, may be implemented in hardware, e.g., in one or more Application-Specific Integrated Circuits (ASICs) or Field-Programmable Gate Arrays (FPGAs). Additionally or alternatively, some elements of CPU <NUM>, packet processing circuitry <NUM>, host interface <NUM> and accelerator engine <NUM> can be implemented using software, or using a combination of hardware and software elements.

In some embodiments, some of the functions of each of CPU <NUM>, packet processing circuitry <NUM>, host interface <NUM> and accelerator engine <NUM> may be carried out by a general-purpose processor, which is programmed in software to carry out the functions described herein. The software may be downloaded to the relevant processor in electronic form, over a network, for example, or it may, alternatively or additionally, be provided and/or stored on non-transitory tangible media, such as magnetic, optical, or electronic memory.

Each of host memory <NUM> and accelerator memory <NUM> may comprise any suitable type of storage such as, for example, a Random Access Memory (RAM). Elements that are not necessary for understanding the principles of the present disclosure, such as various interfaces, addressing circuits, timing and sequencing circuits and debugging circuits, have been omitted from the figure for clarity.

The embodiments described above are given by way of example, and other suitable embodiments can also be used. For example, although the embodiments above refer mainly to RDMA in IB networks, the embodiments are applicable similarly to RDMA in other networks such as, for example Ethernet networks using a suitable networking protocol such as RDMA over Converged Ethernet (RoCE) or IP networks using the iWARP protocol.

Although in the embodiments above the accelerator is assumed to be implemented in hardware, this is not mandatory. In alternative embodiments, a software-based accelerator can also be used for providing accelerated computations by exchanging messages over RDMA.

Although the embodiments described herein mainly address RDMA-based communication between an accelerator and local or remote client, the methods and systems described herein can also be used in other applications, such as in performing a complex accelerated function, e.g., in an Artificial Intelligence (AI) engine. For example, the complex function can be divided among multiple chained accelerators that each performs part of the calculation and provides its output to a subsequent accelerator or to the final destination.

It will be appreciated that the embodiments described above are cited by way of example, and that the following claims are not limited to what has been particularly shown and described hereinabove.

It will be understood that the invention has been described above purely by way of example, and modifications of detail can be made within the scope of the claims.

Claim 1:
A network node (<NUM>), comprising:
A bus switching element (<NUM>);
a network adapter (<NUM>), a computational accelerator (<NUM>) and a host (<NUM>), all coupled to communicate with each other via the bus switching element, wherein:
the network adapter is configured to communicate with remote nodes (<NUM>) over a communication network (<NUM>);
the host is configured to establish a Remote Direct Memory Access, RDMA, link between the computational accelerator and a RDMA endpoint by creating a Queue Pair, QP, (<NUM>) to be used by the computational accelerator for communication with the RDMA endpoint via the RDMA link; and
the computational accelerator is configured to exchange data, via the network adapter, between a memory (<NUM>) of the computational accelerator and a memory of the RDMA endpoint.