Patent ID: 12242753

DETAILED DESCRIPTION

Embodiments of methods and apparatus for reduced network load with combined PUT or GET and receiver-managed offset are described herein. In the following description, numerous specific details are set forth to provide a thorough understanding of embodiments of the invention. One skilled in the relevant art will recognize, however, that the invention can be practiced without one or more of the specific details, or with other methods, components, materials, etc. In other instances, well-known structures, materials, or operations are not shown or described in detail to avoid obscuring aspects of the invention.

Reference throughout this specification to “one embodiment” or “an embodiment” means that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment of the present invention. Thus, the appearances of the phrases “in one embodiment” or “in an embodiment” in various places throughout this specification are not necessarily all referring to the same embodiment. Furthermore, the particular features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

For clarity, individual components in the Figures herein may also be referred to by their labels in the Figures, rather than by a particular reference number. Additionally, reference numbers referring to a particular type of component (as opposed to a particular component) may be shown with a reference number followed by “(typ)” meaning “typical.” It will be understood that the configuration of these components will be typical of similar components that may exist but are not shown in the drawing Figures for simplicity and clarity or otherwise similar components that are not labeled with separate reference numbers. Conversely, “(typ)” is not to be construed as meaning the component, element, etc. is typically used for its disclosed function, implement, purpose, etc.

Under a receiver-managed offset (RMO), the receiver keeps state indicating where to access memory (e.g., where to write for a PUT and where to read for a GET). Under a two-message RMO PUT, a sender sends a first message to allocate or “reserve” space on the receiver, then a second message to send data bytes into that reservation. In pseudocode this looks like:

1// setup2frontier = ...3barrier( )4// inner loop5for i in 1..N:6data, len, receiver = work(i, ...)7where = message_atomic_fetch_add(&frontier, len,receiver)8message_put(dst=where, src=data, len, receiver)

In the foregoing pseudocode, dst is the destination address, src is the source address of the data, and len is the length of the data (e.g., in bytes).

A limitation of this approach is that two messages are needed: one for the sender to request a receiver-side fetch-add to reserve space, then a second message to send the payload bytes. This has several down-sides. First, two messages are needed to send one payload. Ideally, just one message per payload should be needed. Second, small messages tend to be less efficient than large messages, and the fetch-add messages are small messages. Third, the message_put( ) cannot be started until where is known. In turn, the sender is idle waiting for the fetch-add result—thus the fetch-add blocks both the send( ) and also further work( ).

A model of RMO employing two messages is shown inFIG.1. The participants are an initiator100that initiates the data transfer and a target102that is where the data is targeted to be transferred. Generally, each of initiator100and102may be referred to as nodes, which may comprise physical entities or virtual entities. Under the embodiments illustrated herein, the nodes comprise physical entities such as compute platforms (aka compute nodes).

Initiator100includes multiple processing elements (PEs)104, multiple memories106(or otherwise memory partitioned into multiple memory regions), and a network interface controller (NIC)108, which is representative of various types of network interfaces, network adaptors, host controller adaptors (for InfiniBand), etc. Target102has a similar configuration including a NIC110, multiple memories112, and multiple processing elements114. Target102further is depicted as including indicators115, which are used to determine where (e.g., at which starting address or block) the data are to be PUT (written to or merged) or GET (read from) on target102.

Under a two-message RMO PUT, initiator100sends a first reserve “message”117to target102, which is used to reserve space in a memory buffer in memory112on target102. This results in an update in a data structure115, such as a list or circular buffer of indicators. Subsequently, initiator100sends a payload message119to target102containing the data that are to be written to the reserved space in the memory buffer. Upon receipt of payload message119, target102looks up the indicator in data structure115to determine where in memory to “update.”. For an RMO PUT, an update is often “write message bytes to memory”; however, the update could be a merge. Under one RMO scheme, the indicators in data structure115comprise addresses. Alternatively, the indicators may comprise counters.

Indicators can generally be multiple indicators per target resource—e.g., multiple per PE. Resources can be logical—e.g., per virtual address space. There may also be separate indicator groups per resource, e.g., distinct indicators for distinct PEs.

Under the pseudocode presented above, ‘&frontier’ is a memory address. An operation that increments a value according to ‘&frontier’ needs to locate the value, typically using an associative address match to find the value in a cache.” This results in an address match overhead.

In accordance with an aspect of the embodiments herein, a design is provided where a counter is managed in terms of a general “key.” For example, in one embodiment the key is a small integer. A small integer can be used to directly index a table, saving address match overhead. At the same time, the index is small and so can be conveyed in a message more efficiently than a full address—e.g., one byte instead of four to eight bytes typically needed for a full address. This can improve messaging efficiency.

FIG.2depicts an example of a single message RMO PUT. Under this approach, initiator100sends a message118comprising a collection of values to target102. Upon receipt of message118, target102looks up the current indicator in a data structure115used to store indicators (e.g., a table or a circular FIFO (first-in first-out)) to determine where in memory to update. Following the memory update, the indicator is updated if needed (e.g., incremented to next entry in the list) so the following messages can write to different locations. If necessary (current buffer is full or would overflow if written to), the indicator update may identify a new buffer into which the data are to be written. The RMO operations may also generate values (e.g., interrupt, flag, etc.) to inform PEs114that a message is updated memory on target102.

FIG.2further shows an exemplary data structure comprising a table200in which indicators are stored. Under this example, each PE is allocated a respective row in table200. Each cell202in table200is a memory address or offset, or other indicia that may be used to determine the starting address at where to access the memory on the target. For example, a block offset might be used, where it is known in advance that data are transferred in blocks or chunks having a fixed size.

An indicator table may also be indexed by small numbers (e.g., PE #and per-PE RMO #). Using an indexed table allows fast look-up vs. match-based look-up.

FIG.3shows examples of indicator table placement. In some instance, an indicator table may be entirely in the NIC, such as shown under the NIC-only target102(similar to shown inFIGS.1and2above). The NIC may also cache parts of an indicator table from memory, as shown for a cached target102a. In this example, an indicator table300is stored in memory302, while an indicator table cache304is implemented on NIC110.

FIG.4shows examples of using indexing, associative, and indirect implementations for locating indicators. Under the indexing implementation, and N-dimensional array (N=2 in this example) is mapped on a 1-D array400. The 1-D array entries comprise order pair402, where the first value identifies the PE and the second value identifies the RMOnum. In one embodiment, the index is a bitwise concatenation of fields; thus the index can be done in a single cycle. However, this is merely exemplary, as other indexing schemes may be used.

With many counters or when using dynamic counter assignment, it may be desirable to do an associative lookup. Under the associative scheme, a table or list404comprise key-value pairs is used, with the first column containing the keys and the second column containing the counter values. The key itself may be a concatenation of PEnum, RMOnum, e.g., “key”=<PEnum, RMOnum, . . . > or “key”=<RMOnum, PEnum, . . . >, which may be much smaller than a general address. This approach is often faster than full VADDR (virtual address) associative lookup.

In some implementations, it may be desirable to have several different <a, b, c> values map to a single shared counter. This is illustrated in the indirect mapping. An indirect table406includes cells408containing index values (i0, i1, etc.) that are used to map to a second 1-D table410containing counter values that is indexed by its row number. The index value in indirect table406is used to locate the counter value in table410. This may be used with either an array-index (as shown) or associative approaches.

This approach is illustrated in the following pseudocode, which includes an integer rmo value that is used as an rmo_key:

1int rmo = rmo_initialize(&frontier, ...)2for i in 1..N:3data, len, PE_tag = work(i, ...)4message_put_rmo(rmo_key=rmo, src=data, len, PE_tag)

In this and the following pseudocode examples, PE_tag is used to identify the PE associated with the data buffer in which data are to be written, merged, or read. Under some implementations, PE_tag includes a node ID+a PE ID, such as a PE number. Under other implementations, PE_tag is a PE ID (e.g., PE number).

If work( ) can return different data buffers on each iteration, then a non-blocking variant put_rmo_nb( ) can be used so computation and communication can be fully overlapped. In pseudocode:

1int rmo = rmo_initialize(&frontier, ...)2for i in 1..N:3data, len, PE_tag = work(i, ...)4message_put_rmo_nb(rmo_key=rmo, src=data, len, PE_tag)5drain( ) // complete all in-flight messages

A push( ) message operation may be used to send data from an initiator to a target. Another common single-sided message operation is get( ) which reads data from the target and returns the data to the initiator. In one embodiment, push( ) and get( ) are extended to support a fetching operation get_rmo( ) that moves data from target to initiator. In pseudocode:

1int rmo = rmo_initialize(&frontier, ...)2for i in 1..N:3buffer, len, PE_tag = work0(i, ...)4message_get_rmo(dst=buffer, src_key=rmo, len, PE_tag)5work1(i, buffer, len)

The message_get_rmo message passes a destination buffer along with a src_key comprising the integer rmo value. If separate buffers are provided, a non-blocking operation is supported. In pseudocode:

1int rmo = rmo_initialize(&frontier, ...)2for i in 1..N:3buffer[i], len, PE_tag = work0(i, ...)4message_get_rmo_nb(dst=buffer[i], src_key=rmo, len,PE_tag)5drain( ) // complete all in-flight messages; ensure buffer[ ] data is ready6for i in 1..N:7work1(i, buffer[i], len)

As schematic example of the get_rmo( ) process is shown inFIG.5, which depicts an initiator100and a target102having a similar configuration to that shown in the foregoing figures. As depicted by a first operation ‘1’, initiator102submits get RMO operation via a message_get_rmo message500with a desired destination buffer, RMO key, buffer length, and target PE. During the second operation ‘2’ an RMO lookup of initiator table116occurs at target102given the initiator's key. As depicted by a third operation ‘3’, target102locates and transfers the appropriate source buffer in memory502to NIC110. NIC110then sends a message containing the payload504that is delivered to the initiator destination buffer in memory506, as depicted by a fourth operation ‘4’.

FIG.6shows a variant of a get_rmo( ) process that is used for payload delivery at the initiator. Optionally, the approaches inFIGS.5and6may be combined. As depicted by like reference numbers inFIGS.5and6the operations are similar up to payload504being delivered to NIC108of initiator100. In this case, NIC108uses the initiator's RMO key to lookup the corresponding initiator and locate the appropriate destination buffer in memory506, as depicted by the fifth operation ‘5’.

Another message that may employ the techniques disclosed herein is an UPDATE message. The UPDATE message takes the message bytes and merges them with memory contents (updates memory) under some operation OP. For example, given LEN words and I in [0 . . . LEN), then DST[I]=OP(PAYLOAD[I], DST[I]). Where OP may be ADD, MUL, MIN, MAX, etc. In one respect, a PUT is effectively an UPDATE message where OP is just DST[I]=PAYLOAD[I]−writes DST[I] without reading it. For PUT and GET, the write to memory can generally be an UPDATE rather than a simple write; the RMO operation described here applies to both memory writes and UPDATE.

In recent years, OpenSHMEM has been extended to support two new programming constructs: teams and contexts. A “team” is a collaborating subset of tasks that can span the network. A “context” is an abstraction of a communication channel that can be named explicitly by a program in order to optimize the management of network resources. Other messaging systems have similar concepts, such as MPI communicators and Portals constructs.

In accordance with another aspect of some embodiments, an RMO interface that operates using teams and contexts is provided. Specifically, a team and context are created and associated in the usual way, then associated with the RMO by passing the team identifier to the RMO initialization and extending the message_put_rmo( ) call to take the associated context. In pseudocode:

1shmem_team_t rmo_team = ... // prior art2shmem_ctx_t rmo_ctx // prior art3shmem_team_create_ctx(rmo_team, ...options..., &rmo_ctx) // prior art4rmo = rmo_initialize(rmo_team, &frontier, ...) // disclosed5for i = 1..N:6data, len, PE_tag = work(i, ...)7// disclosed:8message_put_rmo(rmo_ctx, rmo_key=rmo, src=data, len,PE_tag)

RMO objects can be associated with PE-groups, or teams. For example, teams/RMOs may be bound to certain processors and/or have affinity to separate/dedicated memory spaces.

An example of a multi-PE/Teams-based RMO is shown inFIG.7, which depicts transfers between an initiator compute node700and a target compute node702. Initiator compute node700includes a Team ‘A’ of PE's704configured to access RMO memory (A)706and a Team ‘B’ of PE's708configured to access RMO memory (B)710. RMO memory (A)706and RMO memory (B)710are coupled to a NIC712.

Target compute node702includes a Team ‘A’ of PE's714configured to access RMO memory (A)716and a Team ‘B’ of PE's718configured to access RMO memory (B)720. Target compute node702also includes a NIC722and an RMO table724.

At target node702, RMO table724contains the rmo-key mappings to the applicable buffers in RMO memory (A)716and RMO memory (B)720.

FIGS.8aand8billustrate examples of RMOs and associated communication channels. InFIG.8a, one logical RMO “gadget” is shared across logical channels. Initiator800is coupled to target802via multiple logical channels820. Initiator800includes a plurality of PEs804enabled to access memory806and coupled to NIC808. Target802includes a NIC810, a plurality of PEs812enabled to access memory814. NIC810includes an RMO gadget816and an RMO table818.

Under the embodiment inFIG.8bthere is one logical RMO gadget816per logical channel820. Alternate embodiments may implement multiple RMO tables818(as shown), or may implement a multi-dimension table/data structure. Such RMO tables and data structures may also be sparse.

Applications to Nodes

The techniques disclosed herein provide advantages for large-scale networks, and also for small-scale “in-node networks” (INNs) used to connect tens, hundreds, or thousands, etc., of microprocessors in a socket or on a single board. Historically, compute nodes employ a cache-coherent domain. However, with increasing core/XPU counts, cache coherency cost grows. For example, coherency support physical structure size/cost can grow. Message delivery may require more hops, which results in more cost for coherency protocol “extra” messages. Additionally physical scaling may require multiple dies linked together, but link bandwidth/latency may between dies may be worse than in-die.

The foregoing issues may be addressed using modified protocols that employ more efficient communication patterns. These modified protocols avoid using space in physical structures and employ protocols with fewer hops. On such modified protocol is MOVPUT (U.S. Pat. No. 10,606,755), which supports core-to-core data transfers. MOVPUT can be started by an initiator as soon as data is ready, and which thus allows the target to read the data directly from its own cache and without incurring delays.

As INNs grow to connect hundreds of cores, the “fetch on demand” behavior of cache protocols, such as MESI (modified/exclusive/shared/invalid) and similar protocols for example, scale poorly for some workloads, hence the motivation for MOVPUT. However, using MOVPUT effectively for a wider range of workloads can reintroduce the fetch-add/send( ) pattern, where send( ) is implemented using MOVPUT, but the fetch-add causes the atomic operation's cache line to “bounce” from core to core.

FIG.9shows a node902in which in-node RMO is implemented, according to one embodiment. Node900includes a NIC902that accesses an RMO table904. Node900includes two CPUs or Other Processing Units (collectively termed XPUs)906and908. XPUs may include but are not limited to one or more of Graphic Processor Units (GPUs) or General Purpose GPUs (GP-GPUs), a Tensor Processing Unit (TPU), Data Processor Units (DPUs), Infrastructure Processing Units (IPUs), Artificial Intelligence (AI) processors or AI inference units and/or other accelerators, FPGAs and/or other programmable logic (used for compute purposes), etc. While some of the diagrams herein show the use of CPUs, this is merely exemplary and non-limiting. Generally, any type of XPU may be used in place of a CPU in the illustrated embodiments. Moreover, as used in the following claims, the terms “processor,” “processor unit,” and “processing units” are used to generically cover CPUs and various forms of XPUs.

Each of CPUs/XPUs906and908have a similar structure including multiple PEs910, an agent912and an RMO table914. CPU/XPU906is coupled to memory916and918, while CPU/XPU908is coupled to memory920and922. CPU/XPU906and908are communicatively coupled via an interconnect924, where interconnect924may be a physical interconnect or may be logical. When a CPU or XPU is implemented as a “socket,” interconnect924may comprise a socket-to-socket interconnect. When both CPU/XPU906and908are implemented in separate dies on the same substrate, interconnect924may comprise an inter-die interconnect.

In addition to the architecture shown for node900, a node may employ an array of processing elements, such as but not limited to an array of cores or an array of core “tiles.” Under various configurations, agents may be associated with individual cores, groups of cores, and/or groups of core tiles.

For node900, NIC902uses RMO table904when the node is acting as a target for a given data transaction. Node900may also operator as an initiator. For data transactions between CPU/XPU906and908, an agent912employs it associated RMO table914in a similar manner to the NICs in the embodiments discussed above. In one embodiment, an agent912and RMO table914are implemented in an interface for interconnect924.

Using the disclosed approach, the fetch-add and MOVPUT may be bundled, so that the MOVPUT payload is sent to the scalable atomics unit and then forwarded directly to the target cache. This is called AMOPUT (Atomic Memory Operation—PUT). In some embodiments, a socket will be built as a multi-chip module with several compute dies and at least one scalable atomics unit per die, with the target cache and scalable atomics unit being co-located on the same die. In turn, AMOPUT reduces the number of die or other communication domain crossings. Further, in multi-chip modules, the message cost between dies is much higher than the message cost within dies, so AMOPUT message reduction can have a benefit larger than simply the raw reduction in message count, by reducing the number of expensive/cross-die and/or cross-domain messages.

A node may include a “scalable atomics” unit for high-performance arithmetic on values shared among cores/PEs. An RMO agent may also be implemented using scalable atomics. Examples of scalable atomics units employing RMO agents are shown inFIGS.10a,10b, and10c.

Under the independent embodiment ofFIG.10a, a node1000aincludes multiple PEs1002coupled to a scalable atomics (SA) unit1004and an RMO agent1006with an RMO table1008, and memory1010and1012. In this embodiment, SA unit1004and RMO agent1006operate independently.

Under the remote embodiment ofFIG.10b, a node1000bincludes a scalable atomics unit1014with a traffic table1016, and an RMO agent1018. This embodiment re-uses an existing scalable atomics unit but adds traffic between SA and RMO agents in traffic table1016that can interfere with other traffic.

Under the integrated embodiment ofFIG.10c, a node1000cincludes an integrated scalable atomics unit with RMO agent1020and an RMO table1022. This embodiment can further reduce overhead both through integration and by not generating interfering traffic.

Memory Locality Improvements

A streamlined implementation can also improve target-side cache and memory locality. In the prior art using separate fetch-add and send( ), the target-side ADD and write of payload bytes are separated in time due to message latency. A set of initiators spanning near and far nodes may request fetch-add operations. The operations execute at the target in the order {A, B, C, D, . . . }. This causes memory to be allocated at the target in the order {A, B, C, D, . . . }.

However, round-trip message delay for A is generally different than for B. For example, A may be from a distant node and so has more speed-of-light delays, traverses more physical links and buffers, encounters more congestion points, and may run closer to the edge of congestion control. In other words, A's fetch-add may execute first at the target, but can have a long delay for the result to return to the initiator, and for the following message_send(dst=A, . . . ) to arrive at the target. In turn, messages may arrive at the target in some arbitrary “shuffle” of the original requests, e.g., {D, B, Z, M, . . . , A, . . . }. In high-scale systems, each target can receive hundreds of thousands of send( ) operations from distinct initiators, and re-ordering is thus frequent.

Reordering can lead to at least two problems. First, cache locality is reduced. With small messages, the message_send( ) payloads may be written to some kind of cache or write-combining structure. However, with poor locality, the benefit of these caching structures is reduced. Second, memory locality is reduced. Several common memory types (DRAM, Optane™) are organized as groups of bits often called “pages” (but entirely different than virtual memory pages), for example 512 bytes per “page”. Write and read bursts within a page may be significantly faster than reads and writes that span pages.

Consider 64-byte messages and 512-byte pages: an out-of-order delay of just 512/64=8 messages leads to excess page open/close costs, hurting bandwidth. At-scale systems with tens or hundreds of thousands of distinct initiators, can suffer excess open/close costs on a majority of transfers.

The approaches described and illustrated herein can use table lookups and fixed assignments in order to avoid matching and caching. In turn, message payloads can be written into sequential memory locations, giving better cache and memory locality.

Example NIC

An exemplary system architecture for a NIC1100is shown inFIG.11. NIC1100includes a NIC system board1102on which a NIC Chip/SoC1104, Dynamic Random Access Memory (DRAM)1106and Static Random Access Memory (SRAM)1108are mounted. Under various embodiments. NIC system board1102is representative of an Ethernet controller card, a daughter board, a multi-chip module board or substrate, or it may be part of a computer system board, such as a main board or motherboard for a computer server. NIC Chip/SoC1104is representative of Ethernet processing and/or control unit, and may be embodied in various forms, including as an Ethernet controller chip or a network processor unit (NPU). In addition to Ethernet, NIC1100is generally representative of a network interface, network adaptor, host control adaptor, etc., the implements one or more associated network protocols.

In the illustrated embodiment, NIC Chip/SoC1104includes an instruction store1110, a NIC processor1111including multiple cores1112, an SRAM controller1114, a DRAM controller1116, a Write DMA block1118, a Read DMA block1120, a PCIe interface1122, an optional TCAM (ternary content-addressable memory)1123, a scratch memory1124, a hash unit1126, Serializer/Deserializers (SerDes)1128and1130, and PHY interfaces1132and1134. Each of the components is interconnected to one or more other components via applicable interconnect structure and logic that is collectively depicted as an internal interconnect cloud1135.

Instruction store1110includes various instructions that are executed by cores1112, including Flow Classification instructions1113, Packet Decode instructions1115, RMO logic instructions1117, TCP logic instructions1119, and optional teams logic instructions1121. Under one embodiment, various packet processing operations are performed using a pipelined architecture. As an alternative, the combination of cores1112and instruction store1110may be implemented using embedded programmable logic, such as via a Field Programmable Gate Arrays (FPGA) or the like (not shown).

In one embodiment, instruction store1110is implemented as an on-chip store, such as depicted inFIG.11. Optionally, a portion or all of the instructions depicted in instruction store1110may be stored in SRAM1108and accessed using SRAM controller1114via an interface1138. SRAM1108may also be used for storing selected data and/or instructions relating to packet processing operations and instructions for implementing the algorithms described herein. For example, all or a portion of RMO tables and other related data structures may be stored in SRAM1108.

Memory in DRAM1106is used for transmit (TX) queues/buffers1125and receive (RX) queues/buffers1127and is accessed using DRAM controller1116via an interface1140. DRAM1106may also be used for storing other data structures relating to packet handling operations. In some embodiments, all or a portion of RMO tables and other related data structures are stored in DRAM1106. Write DMA block1118and Read DMA block1120are respectively configured to support DMA Write and Read operations to support DMA operations between data in DRAM1106and SRAM1108and host (e.g., compute node) memory (e.g., the memories shown in the embodiments above). In the illustrated embodiment, DMA communication between DRAM1106and a compute node is facilitated over PCIe interface1122via a PCIe link1142coupled to a PCIe interconnect or PCIe expansion slot1144, enabling DMA Write and Read transfers between DRAM1106and compute node or host memory for a host1146using the PCIe protocol.

In addition to PCIe, other interconnect technologies and protocols may be used. For example, these include but are not limited to Computer Express Link (CXL), InfiniBand, and Omni-Path.

Scratch memory1124and hash unit1126are illustrative of components employed by NICs for facilitating scratch memory and hashing operations relating to packet processing. For example, as described above a hash operation may be implemented for deriving flow IDs and for packet identification. In addition, a hash unit may be configured to support crypto-accelerator operations.

PHYs1132and1134facilitate Physical layer operations for the NIC, and operate as a bridge between the digital domain employed by the NIC logic and components and the analog domain employed for transmitting data via electrical, optical or wired signals. For example, in the illustrated embodiment ofFIG.11, each of PHYs1132and1134is coupled to a pair of I/O ports configured to send electrical signals over a wire or optical cable such as a high-speed Ethernet cable. Optical and wireless signal embodiments would employ additional circuitry and interfaces for facilitating connection via optical and wireless signals (not shown). In conjunction with PHY operations, SerDes1128and1130are used to serialize output packet streams and deserialize inbound packet streams.

Generally, a NIC may be configured to store routing data for facilitating packet identification and flow classification, including forwarding filters and rules either locally or using a memory-mapped IO (MMIO) address space in system or host memory. When stored locally, this routing data may be stored in either DRAM1106or SRAM1108. Routing data stored in a MMIO address space may be accessed by NIC1100via Read and Write DMA operations. Generally, setting up MMIO address space mapping may be facilitated by a NIC device driver in coordination with the operating system. The NIC device driver may also be configured to enable instructions in instruction store1110to be updated via the operating system. Optionally, the instructions in instruction store may comprise firmware instructions that are stored in non-volatile memory, such as Flash memory, which may either be integrated on NIC Chip/SoC1104or mounted to NIC system board1102(not shown).

As an option to using DRAM1106or SRAM1108, flow rules1118may be implemented in hardware-based logic such as a FPGA or other programmable logic device coupled to NIC processor1111. Hash unit1126may be implemented in the same hardware-based logic as that used for flow rules1118. Flow rules1118may also be implemented using TCAM1123.

NIC processor1111may employ any suitable processor architecture in current use or developed in the future. In one embodiment, the processor architecture is an Intel® x86 architecture, an IA-32 architecture or an IA-64 architecture. In one embodiment, the NIC processor architecture is an ARM®-based architecture.

Example IPU/SmartNIC

Aspects of the embodiments disclosed herein may be implemented in an Infrastructure Processor Unit, which may also be called a SmartNIC.FIG.12shows an example IPU1200, according to one embodiment. IPU1200includes multiple components that are coupled to a circuit board1201. The components include an FPGA1202that may be programmed to implement various logic described herein. Generally, an FPGA may access data stored in one or more memory devices, such as depicted by memory devices1204and1206. As described below, various types of memory devices may be used, including but not limited to DDR4 and DDR5 DIMMS (Dual Inline Memory Modules). The FPGA may also include onboard memory1208in which data may be stored.

In the illustrated embodiment, IPU1200includes a NIC chip1209with four network ports1210, respectively labeled Port 1, Port 2, Port 3, and Port 4. Data can be transferred between NIC chip1209and FPGA1202using separate links per network port1210or using a multiplexed interconnect. In one embodiment, NIC chip1209employs a 40 GB/s MAC, and each of the four network ports1210is a 10 GB/s port. In other embodiments, NIC chip1209may employ a MAC with other bandwidths. Also, the illustrated use of four ports is merely exemplary and non-limiting, as a IPU may have various numbers of network ports. In some embodiments, an IPU may include multiple NIC chips.

IPU1200further includes a CPU1212flash memory1214, a baseboard management controller (BMC)1216, and a USB module1218. CPU1212may be used to execute embedded software/firmware or the like. Flash memory1214may be used to store firmware and/or other instructions and data in a non-volatile manner. Other software may be loaded over a network coupled to one or more of the NIC ports.

In the illustrated embodiment, FPGA1202has a PCIe interface that is connected to a PCIe edge connector configured to be installed in a PCIe expansion slot. In one embodiment, the PCIe interface comprises an 8 lane (8×) PCIe interface1222. Other PCIe interface lane widths may be used in other embodiments, including 16 lane (16×) PCIe interfaces.

In some embodiments, a portion of the FPGA circuitry is programmed to implement RMO logic1117and/or Teams Logic1121. Optionally, similar logic may be implemented via execution of associated software/firmware on CPU1212or in NIC chip1209. Other logic and operations described in the foregoing embodiments may be implemented using one or more of FPGA1202, CPU1212, and NIC chip1209. FPGA circuitry on FPGA1202and/or execution of embedded software/firmware on CPU1212may also be used to implement/execute operators.

The memories illustrated in the Figures herein are logical representations of memory implemented via one or more physical memory devices. Such memory devices may include volatile memory, non-volatile memory, and hybrid memory devices.

Volatile memory is memory whose state (and therefore the data stored in it) is indeterminate if power is interrupted to the device. Dynamic volatile memory requires refreshing the data stored in the device to maintain state. One example of dynamic volatile memory incudes DRAM (Dynamic Random Access Memory), or some variant such as Synchronous DRAM (SDRAM). A memory subsystem as described herein can be compatible with a number of memory technologies, such as DDR3 (Double Data Rate version 3, original release by JEDEC (Joint Electronic Device Engineering Council) on Jun. 27, 2007). DDR4 (DDR version 4, initial specification published in September 2012 by JEDEC), DDR4E (DDR version 4), LPDDR3 (Low Power DDR version 3, JESD209-3B, August 2013 by JEDEC), LPDDR4) LPDDR version 4, JESD209-4, originally published by JEDEC in August 2014), WIO2 (Wide Input/Output version 2, JESD229-2 originally published by JEDEC in August 2014, HBM (High Bandwidth Memory, JESD325, originally published by JEDEC in October 2013, DDR5 (DDR version 5), LPDDR5, HBM2E, HBM3, and HBM-PIM, or others or combinations of memory technologies, and technologies based on derivatives or extensions of such specifications. The JEDEC standards are available at www.jedec.org.

A non-volatile memory (NVM) device is a memory whose state is determinate even if power is interrupted to the device. In one embodiment, the NVM device can comprise a block addressable memory device, such as NAND technologies, or more specifically, multi-threshold level NAND flash memory (for example, Single-Level Cell (“SLC”), Multi-Level Cell (“MLC”), Tri-Level Cell (“TLC”), Quad-Level Cell (“QLC”), Penta-Level Cell (PLC) or some other NAND). A NVM device can also include a byte-addressable write-in-place three dimensional crosspoint memory device, or other byte addressable write-in-place NVM devices (also referred to as persistent memory), such as single or multi-level Phase Change Memory (PCM) or phase change memory with a switch (PCMS), NVM devices that use chalcogenide phase change material (for example, chalcogenide glass), resistive memory including metal oxide base, oxygen vacancy base and Conductive Bridge Random Access Memory (CB-RAM), nanowire memory, ferroelectric random access memory (FeRAM, FRAM), magneto resistive random access memory (MRAM) that incorporates memristor technology, spin transfer torque (STT)-MRAM, a spintronic magnetic junction memory based device, a magnetic tunneling junction (MTJ) based device, a DW (Domain Wall) and SOT (Spin Orbit Transfer) based device, a thyristor based memory device, or a combination of any of the above, or other memory.

Although some embodiments have been described in reference to particular implementations, other implementations are possible according to some embodiments. Additionally, the arrangement and/or order of elements or other features illustrated in the drawings and/or described herein need not be arranged in the particular way illustrated and described. Many other arrangements are possible according to some embodiments.

In each system shown in a figure, the elements in some cases may each have a same reference number or a different reference number to suggest that the elements represented could be different and/or similar. However, an element may be flexible enough to have different implementations and work with some or all of the systems shown or described herein. The various elements shown in the figures may be the same or different. Which one is referred to as a first element and which is called a second element is arbitrary.

In the description and claims, the terms “coupled” and “connected,” along with their derivatives, may be used. It should be understood that these terms are not intended as synonyms for each other. Rather, in particular embodiments, “connected” may be used to indicate that two or more elements are in direct physical or electrical contact with each other. “Coupled” may mean that two or more elements are in direct physical or electrical contact. However, “coupled” may also mean that two or more elements are not in direct contact with each other, but yet still co-operate or interact with each other. Additionally, “communicatively coupled” means that two or more elements that may or may not be in direct contact with each other, are enabled to communicate with each other. For example, if component A is connected to component B, which in turn is connected to component C, component A may be communicatively coupled to component C using component B as an intermediary component.

An embodiment is an implementation or example of the inventions. Reference in the specification to “an embodiment,” “one embodiment,” “some embodiments,” or “other embodiments” means that a particular feature, structure, or characteristic described in connection with the embodiments is included in at least some embodiments, but not necessarily all embodiments, of the inventions. The various appearances “an embodiment,” “one embodiment,” or “some embodiments” are not necessarily all referring to the same embodiments.

Not all components, features, structures, characteristics, etc. described and illustrated herein need be included in a particular embodiment or embodiments. If the specification states a component, feature, structure, or characteristic “may”, “might”, “can” or “could” be included, for example, that particular component, feature, structure, or characteristic is not required to be included. If the specification or claim refers to “a” or “an” element, that does not mean there is only one of the element. If the specification or claims refer to “an additional” element, that does not preclude there being more than one of the additional element.

As discussed above, various aspects of the embodiments herein may be facilitated by corresponding software and/or firmware components and applications, such as software and/or firmware executed by an embedded processor or the like. Thus, embodiments of this invention may be used as or to support a software program, software modules, firmware, and/or distributed software executed upon some form of processor, processing core or embedded logic or a virtual machine running on a processor or core or otherwise implemented or realized upon or within a non-transitory computer-readable or machine-readable storage medium. A non-transitory computer-readable or machine-readable storage medium includes any mechanism for storing or transmitting information in a form readable by a machine (e.g., a computer). For example, a non-transitory computer-readable or machine-readable storage medium includes any mechanism that provides (e.g., stores and/or transmits) information in a form accessible by a computer or computing machine (e.g., computing device, electronic system, etc.), such as recordable/non-recordable media (e.g., read only memory (ROM), random access memory (RAM), magnetic disk storage media, optical storage media, flash memory devices, etc.). The content may be directly executable (“object” or “executable” form), source code, or difference code (“delta” or “patch” code). A non-transitory computer-readable or machine-readable storage medium may also include a storage or database from which content can be downloaded. The non-transitory computer-readable or machine-readable storage medium may also include a device or product having content stored thereon at a time of sale or delivery. Thus, delivering a device with stored content, or offering content for download over a communication medium may be understood as providing an article of manufacture comprising a non-transitory computer-readable or machine-readable storage medium with such content described herein.

The operations and functions performed by various components described herein may be implemented by software running on a processing element, via embedded hardware or the like, or any combination of hardware and software. Such components may be implemented as software modules, hardware modules, special-purpose hardware (e.g., application specific hardware, ASICs, DSPs, etc.), embedded controllers, hardwired circuitry, hardware logic, etc. Software content (e.g., data, instructions, configuration information, etc.) may be provided via an article of manufacture including non-transitory computer-readable or machine-readable storage medium, which provides content that represents instructions that can be executed. The content may result in a computer performing various functions/operations described herein.

As used herein, a list of items joined by the term “at least one of” can mean any combination of the listed terms. For example, the phrase “at least one of A, B or C” can mean A; B; C; A and B; A and C; B and C; or A, B and C.

The above description of illustrated embodiments of the invention, including what is described in the Abstract, is not intended to be exhaustive or to limit the invention to the precise forms disclosed. While specific embodiments of, and examples for, the invention are described herein for illustrative purposes, various equivalent modifications are possible within the scope of the invention, as those skilled in the relevant art will recognize.

These modifications can be made to the invention in light of the above detailed description. The terms used in the following claims should not be construed to limit the invention to the specific embodiments disclosed in the specification and the drawings. Rather, the scope of the invention is to be determined entirely by the following claims, which are to be construed in accordance with established doctrines of claim interpretation.