NETWORK INTERFACE DEVICE FEEDBACK FOR ADAPTIVE AND FAILOVER MULTIPATH ROUTING

Examples described herein relate to a network interface device comprising: circuitry, when operational, to: in response to congestion related to a link, cause transmission of link event information to at least one sender of packets to the link, wherein the link event information is to identify congestion information of at least one link other than the link.

BACKGROUND

Multi-path communication is utilized in networking to provide multiple options for network traffic to reach a destination from a sender. One approach to end-to-end multipath communication establishes multiple flows between two communication endpoints that leverage hop-by-hop and/or flow-based load-balancing (or other schemes) to ensure the flows use different and potentially non-overlapping paths of one or more links through the network to the destination. A sender device sends data through a first flow, while the second flow is kept active in case a link on the path of the first flow fails or is overloaded.

Some Clos architectures utilize hop-by-hop flow-based load-balancing. Schemes such as equal-cost multipath (ECMP), or other schemes, can be utilized whereby, as an example using ECMP, hashing of selected packet header fields (e.g., 5-tuple-based for IPv4 or flow label for IPv6) can be used to distribute traffic load over multiple links to attempt to provide that packets of a same flow take a same path and attempt to provide in-order delivery of packets in flows.

DETAILED DESCRIPTION

FIG.1shows an example of a sender process utilizing multiple parallel network paths through a network. For example, flow S1can utilize a primary path and flow S2can utilize a backup or redundant path. Where there are multiple paths from sender to receiver, multiple links can be available through Switch1and Switch2. The sender process sends a traffic flow on one or more of these paths. If an issue on one of the used paths occurs, such as congestion, a different path can be chosen from a pre-established set of paths so that packet traffic is either sent on a single path or sent among multiple paths. Multiple links can be provided by port3(Prt3) of Switch1connected to port1(Prt1) of Switch2and port4(Prt4) of Switch1connected to port2(Prt2) of Switch2. Switch1and switch2can be connected through two parallel links over which the traffic can be distributed.

One way to assign flows to links is flow hashing whereby hash values of a set of fields of the packet header (e.g., the 5-tuple for IPv4) can be used to assign the flow to a link. Assignment is consistent for a given 5-tuple and can enforce in-order delivery of packets of a flow, while being stateless.

In this example, other sender node sends packets associated with Flow O1through Switch1's Prt3to Switch2's Prt1to Receiver Node. Flow O1uses the same link as that of Flow S1. Port Prt3can receive packets from one or more queues associated with different traffic classes. Both flows O1and S1together have a higher arrival rate than the Prt3's link capacity and the depth of the queue on this port grows. At Switch1, application of scalable reliable datagram (SRD) (e.g., L. Shalev, H. Ayoub, N. Bshara, and E. Sabbag, “A Cloud-Optimized Transport Protocol for Elastic and Scalable HPC,” in IEEE Micro, vol. 40, no. 6, pp. 67-73, 1 November-December 2020) can detect this queue build-up after feedback from the Receiver Node. However, at this point in time, the queue buildup can be significant and could have led to packet drop(s) for Prt 3 on Switch1. Using SRD or other mechanism, flows S1and S2can be mapped to two different links from Switch1to Switch2. Sender Node can transfer the Sender Process' traffic load over Flow S1and flow S2can be kept and maintained as a backup in case the primary flow's path through the network fails or is congested. A path can include one or more links between ports of devices.

Switch1moving one of the two flows to a different link can cause packets to be enqueued in a congested queue and later packets to be enqueue in an uncongested queue. Thereby, later packets could overtake earlier packets of the same flow, potentially breaking in-order delivery of packets.

Some transport layer protocols (e.g., SRD, Transmission Control Protocol (TCP), Remote Direct Memory Access (RDMA) over Converged Ethernet (RoCEv2), etc.) provide in-band end-to-end congestion control mechanisms to react to congestion in the network whereby a receiver node indicates congestion at an intermediate node to a sender node. By design, these protocols react over a timespan of multiple end-to-end round-trip times (RTTs). While this is sufficient for many situations, in case of high-performance RDMA networking, significant queue build-up and potentially packet loss can occur.

At least to reduce a time for a sender to identify and potentially reduce an impact of congestion on packet transmission, technologies described herein provide back-to-sender (BTS) signals to transmit network telemetry from forwarding devices (e.g., switch, router) in the network to indicate information on a congested link and at least one other parallel link. In some examples, as described herein, BTS signals can include telemetry information concerning a currently used queue, port, and/or link and also telemetry information concerning at least one other parallel or multi-path queue, port, or link. BTS signals can include information about congestion events at the switch where the event occurred and be provided by a packet processing fast-path such as a programmable packet processing pipeline of the switch, as described herein. A sender can reduce impact of congestion on packet transmission by pause/resume flow control, path control or change, congestion control, and so forth adaptive to the type and location of the signaled event, as described herein. A sender reaction policy can change currently used path based on failure or congestion at a parallel link, or others. The BTS signals can lower reaction delay to events occurring in the network and feedback delay can be less than 1 RTT.

FIG.2depicts a system and operation. Sender Node200sets up two flows S1and S2to send traffic to Receiver Node. Sender Node200can select one or more paths for transmitted packets in-network hop-by-hop using a variety of techniques such as n-tuple hashing (where n is an integer), or explicitly establishing tunnels (e.g., Multiprotocol Label Switching (MPLS) Label Distribution Protocol (LDP), Segment Routing over IPv6 dataplane (SRv6) source routing, virtual local area network (VLAN)-based network slices, technologies described in Mudigonda, Jayaram, et al., “Spain: Cots data-center ethernet for multipathing over arbitrary topologies,” NSDI. Vol. 10. 2010 (hereafter “SPAIN”), and so forth. In Switch1, the flows can be assigned to Port3(Prt3) and Port4(Prt4).

In some examples, Sender Node200transmits a packet load for sender process202using Flow S1and maintains Flow S2as a backup path. A flow can include a sequence of packets transferred between two endpoints, generally representing a single session using a known protocol. Accordingly, a flow can be identified by a set of defined tuples and, for purposes of routing or directing a flow through links, a flow is identified by the two tuples that identify the endpoints, e.g., the source and destination addresses. In other examples, for purposes of routing or directing a flow through links, a flow can be identified by n-tuples. For content-based services (e.g., load balancer, firewall, intrusion detection system, etc.), flows can be differentiated at a finer granularity by using n-tuples (e.g., source address, destination address, IP protocol, transport layer source port, and destination port, or other protocol specific information, such as destination queue pair (QP) identifiers (IDs) for RoCEv2). A packet in a flow is expected to have the same set of tuples in the packet header. At least for ECMP, SRD, or others, a packet flow to be controlled can be identified by a combination of tuples (e.g., Ethernet type field, source and/or destination IP address, source and/or destination User Datagram Protocol (UDP) ports, source/destination TCP ports, or any other header field) and a unique source and destination queue pair (QP) number or identifier. A packet may be used herein to refer to various formatted collections of bits that may be sent across a network, such as Ethernet frames, IP packets, TCP segments, UDP datagrams, etc. Also, as used in this document, references to L2, L3, L4, and L7 layers (layer 2, layer 3, layer 4, and layer 7) are references respectively to the second data link layer, the third network layer, the fourth transport layer, and the seventh application layer of the OSI (Open System Interconnection) layer model.

For VLAN-based mechanisms, the VLAN could establish a path and identify a flow. For Multi-protocol label switching (MPLS), a label-switched path can be established and identify a flow, for source-routing-based mechanisms, such as segment routing each packets can include one or more hops to pass to the destination. In the segment routing case, source routing can be combined with n-tuple hashing to identify a flow. An example of segment routing is described at least in Internet Engineering Task Force (IETF) Segment Routing Architecture Request for Comments (RFC) 8402 (2018).

A path can represent a sequence of nodes, switches, and links that packets of a flow traverse through a network. A parallel link can represent one link of multiple links that connect a pair forwarding node to one or more forwarding nodes in a network through which packets can reach the same receiver node. Packets of parallel links can traverse the same path or different paths to the receiver node. For example, packets of flow S1can traverse same or different switches to Receiver Node250as switches used to forward packets of flow S2to Receiver Node250. End-to-end multipath communication can include two endpoints that establish multiple flows between them that are designed to take separate paths through the network.

Sender Node200can include a host system and a network interface device. At least one example of host system is described at least with respect toFIG.7or8. At least one example of a network interface device includes one or more of: a network interface controller (NIC), a remote direct memory access (RDMA)-enabled NIC, SmartNIC, router, switch, forwarding element, infrastructure processing unit (IPU), data processing unit (DPU), or network-attached appliance. At least one example of network interface device is described with respect toFIGS.5,6,7, and/or8. Similarly, Sender Node210can include a host system and network interface device. Similarly, Receiver Node250can include a host system and network interface device. At least one example of Switch1and Switch2is described with respect toFIGS.5,6,7, and/or8.

When sender node210transmits flow O1to queue for Port3, or for other causes, queue for Port3can start to build up and becomes congested, at (1), congestion can be detected on Switch1. For example, congestion can arise from one or more of the following cases:Case A: Transient link overload of link that is part of a group of parallel linksCase B: Link failure of link that is part of a group of parallel linksCase C: Link closing in on transmission capacityCase D: Link in group of parallel links has spare capacity while another link in the group of parallel links is overloaded

In the example ofFIG.2, a set of pre-established paths can be used by Sender Node200to transmit packets to Receiver Node250. However, other manners of setting a path from Sender Node200to Receiver Node250can occur, such as segment routing using source routing where Sender Node200can specify a link to use between switch1and switch2. As a reaction to a congested path, Sender Node200could add a specifier to select a specific link between switch1and switch2.

In response to a BTS indicating congestion, Sender Node200can utilize a backup flow S2for flow S1. Sender Node200can select a link to send backup flow S2based on BTS information received from switch. Sender Node200can utilize one or multiple different flows to transmit packets to a destination. For example, different flows can be identified by different tuples, different VLAN tags, segment routing design, or others. Sender node200can determine a path of packets of a flow to a destination or one or more intermediary nodes can decide next link of path to the destination.

In some examples, a packet processing pipeline of switch such as described with respect toFIG.6can be used to detect cases A-D at (1) and cause transmission of at least one BTS message at (2) and (4). The content of BTS message sent by switch nodes can depend on a multipath mechanism used by Sender Node200as well as the specific network imbalance that causes the BTS response.

For case A, at (1), there is a detection of a congested queue that corresponds to a link in a group of multiple links from Switch1to Switch2. In this example, the congested queue is for port3. For example, a queue can be identified as congested based on a queue depth of the queue (e.g., number of bytes or number of packets) and a threshold level. Based on a determination that the congested queue corresponds to a link in a group of multiple links from Switch1to Switch2, at (2), Switch1can utilize BTS to signal sender node200that flows S1and S2arriving in the queue of Port3that congestion is occurring as well as information about at least one parallel link (e.g., existence of at least one parallel link, congestion information, or other information described herein). Various examples of information conveyed in a BTS are described herein.

For case B (e.g., link failure of link that is part of a group of parallel links), at (1), there is a detection of a link failure in a group of multiple links from Switch1to Switch2. A link can include a physical connection between switches or a switch and a network interface device. Link failure can include inability of a link to transmit packets, transmission rate of packets or bits being at or below a threshold level, or error rate of transmitted packets meeting or exceeding a threshold level. Based on a determination that the failed link corresponds to a link in a group of multiple links from Switch1to Switch2, at (2), Switch1can utilize BTS to signal sender node200that flows S1and S2arriving in the queue of Port3that congestion is occurring and information about at least one other parallel link, as described herein. Various examples of information conveyed in a BTS are described herein.

For case C (e.g., link closing in on transmission capacity), at (1), there is a detection of link, in a group of multiple links from Switch1to Switch2, closing or arriving at transmission capacity. Transmission capacity can be set at a threshold bandwidth such as a threshold level of packets per second or bits per second that is less than or at a peak throughput rate for a link. Based on a determination that the link is closing in on transmission capacity corresponds to a link in a group of multiple links from Switch1to Switch2, at (2), Switch1can utilize BTS to signal sender node200that flows S1and S2arriving in the queue of Port3that congestion is occurring and information about at least one other parallel link, as described herein. Various examples of information conveyed in a BTS are described herein.

For case D (e.g., link in group of parallel links has spare capacity while another link in the group of parallel links is overloaded), at (1), there is a detection that a link in a group of parallel links has spare capacity while another link is overloaded in a group of multiple links from Switch1to Switch2. A link can be overloaded where a transmission rate through the link is less than or at a throughput rate for a link identified as overloaded. Spare capacity on another link can be identified based on the other link being capable of providing a transmission rate at or above a throughput rate (e.g., packets per second or bits per second). Based on determination that the link in group of parallel links has spare capacity while another link that is overloaded corresponds to a link in a group of multiple links from Switch1to Switch2, at (2), Switch1can utilize BTS to signal sender node200that flows S1and S2arriving in the queue of Port3that congestion is occurring. Various examples of information conveyed in a BTS are described herein.

Depending on a case (e.g., case A, B, C, or D) and information conveyed in a BTS, operation (3) can include a reaction performed by Sender Node200to receipt of the BTS. The BTS message can arrive at Sender Node200earlier than provided by transport-protocol-based in-band mechanisms such as latency measurements (used by SRD for example) because switch sends BTS to Sender Node200instead of to Receiver Node250for forwarding to Sender Node200. In some examples, Sender Node200can shift load from Flow S1to Flow S2and assign traffic to Flow S2until the congestion condition is resolved. However, other example reactions by sender node are described herein.

At (3), based on a received BTS message, Sender Node200network interface device can include circuitry to determine to change a path or utilize multiple paths. In some examples, based on a received BTS message, in addition, or alternatively, an operating system (OS) executed by a host system of Sender Node200can determine to change a path or utilize multiple paths. For example, adjusting transmission of packets can include Sender Node200causing Switch1to use one or more other links (e.g., at least port4(Prt4)), instead of Prt3, to transmit packets from Sender Node200via Switch2to Receiver Node250. For example, adjusting transmission of packets can include the Sender Node200causing Switch1using the congested link and one or more other links (e.g., at least port4(Prt4)) to transmit packets from Sender Node200via Switch2to Receiver Node250.

Sender Node200can cause Switch1to use Prt3and Prt4or another port or merely Prt4or another port by various technologies. For example, when SRD is used with ECMP and n-tuple flow hashing, Sender Node200can choose to send traffic from sender process202using Flow S2(FIG.2). In case a multipath approach with no pre-established alternative flows is used, Sender Node200can modify a field of the known n-tuple used to calculate the per-flow hash on Switch1. In some examples, the field includes a field that stores the source port number. A change can cause the flow to choose a different path through the network because of the change in outcome of the n-tuple hash calculation. In some cases, this new path will not include the congested link. If the new path still includes the congested link, Sender Node200can choose a different n-tuple field modification. If the BTS message includes information of the hash values to avoid and the hash-values to prefer, Sender Node200can try multiple values for the modification of the field of the n-tuples until the preferred hash value output is achieved.

In a case that a source-routing approach such as Segment Routing is used and the BTS message includes information on the Segment Routing adjacency of Switch1to avoid, instead of using the Receiver Node250as destination in the segment stack (e.g., segment stack: [Node:250]), Sender Node200can specify the new link to use in a segment stack (e.g., segment stack: [Node: Switch1, Adjacency: Port4, Node:250]. In case a VLAN or label-switched approach is used, Sender Node200can either leverage topology information from the path-establishment to select a path that does not include the congested link or just a random path which may or may not include the congested link.

At (4), Switch1can send a BTS message to at least one other sender node210. The BTS message can be a copy of BTS message that sent in (2) or can be different. At (5), the at least one sender node210can perform a reaction to the BTS message. Example reactions by other sender nodes210are described herein.

In (2) and (4), for a node or parallel link group, information in a BTS sent by Switch1can be supplemented by forwarding elements with forward status for nodes/parallel links on BTS path. For example, the BTS packet can include multipath status (e.g., bifurcations, nodes involved) of the forward path while traveling backwards to sender node200to allow sender node200to perform a reaction to decide which bifurcation in the multipath communication to avoid specifically. For example, in (2) and (4), examples of data to transmit is listed in the information column below.

FIGS.3A and3Bprovide examples triggers at congested node (e.g., Switch1) and sender node reaction to receipt of BTS information.FIG.3Adepicts an example of cases, example switch congestion or link state triggers, example BTS information sent to one or more senders, and example reaction of the one or more senders. For example, for a case A of parallel link overload, a trigger event at a switch can include a queue depth level being met or exceeded. At least for case A and potentially one or more other cases, the switch can send BTS information to one or more sender network interface devices. For example, BTS information can include one or more of: parallel link available (e.g., one or more other link is available, particular link identifier(s), or indicate a flow hash(es) to select link for ECMP), Flow ID, event information (e.g., pause time, congested queue depth, and/or queue depth gradient (e.g., an amount or percentage change in queue depth over a time interval)), node and link ID of location of congestion event, or others. BTS information can include congestion information collected on a path from switch to sender node at one or more hops of a path from switch to sender network interface device.

At least for case A and potentially one or more other cases, in response to receipt of the BTS information, the one or more sender network interface devices can perform one or more remedial actions. Remedial actions can include one or more of: move load to different flow or path; select new path(s) as well as new connection(s) that do not include congested link, resume sending on a congested path after congestion clears; allocate packets transmitted on first path or paths to different flow(s)/path(s) for a specified time and resume sending on a first path or paths after the specified time elapses; allocate packets transmitted on first path or paths to different flow(s)/path(s) for a specified time, continue in-band probing on the first path or paths and resume sending on first path or paths after the specified time elapses and in-band probe suggests that congestion is resolved; or allocate packets transmitted on first path or paths to different path or paths as well as new connection or connections that do not include congested link based on load information collected between the switch and the sender node path into account.

For example, for a case B of link failure, a trigger event at a switch can include a port status is failed or down. At least for case B and potentially one or more other cases, the switch can send BTS information to one or more sender network interface devices. For example, BTS information can include one or more of: parallel link available (e.g., one or more other link is available, particular link identifier(s), or indicate a flow hash(es) to select link for ECMP), Flow ID, event information (e.g., link failed), node and link ID of location of congestion event, or others. BTS information can include congestion information collected on a path from switch to sender node at one or more hops of a path from switch to sender network interface device. Note that multipath communications can be conducted using ECMP with or without SRD, or other techniques can be used such as segment routing, SPAIN, etc.

At least for case B and potentially one or more other cases, in response to receipt of the BTS information, the one or more sender network interface devices can perform one or more remedial actions. Remedial actions can include one or more of: move load to different flow or path; select new path(s) as well as new connection(s) that do not include congested link; or allocate packets transmitted on first path or paths to different path or paths as well as new connection or connections that do not include congested link based on load information collected between the switch and the sender node path into account.

For example, for a case C of a link reaching or approaching a capacity limit, a trigger event at a switch can include residual capacity is reducing toward 0% or other non-zero value. At least for case C and potentially one or more other cases, the switch can send BTS information to one or more sender network interface devices. For example, BTS information can include one or more of: parallel link available (e.g., one or more other link is available, particular link identifier(s), or indicate a flow hash(es) to select link for ECMP), Flow ID, event information (e.g., do not increase load on link), node and link ID of location of congestion event, or others. BTS information can include congestion information collected on a path from switch to sender node at one or more hops of a path from switch to sender network interface device.

At least for case C and potentially one or more other cases, in response to receipt of the BTS information, the one or more sender network interface devices can perform one or more remedial actions. Remedial actions can include one or more of: do not add new load to flow/path, indicate that a new connection does not use the congested link, or increase in-band probing resolution to improve load control on path.

For example, for a case D of spare capacity in a link, a trigger event at a switch can include residual capacity is increasing to 100% or other non-zero value. At least for case D and potentially one or more other cases, the switch can send BTS information to one or more sender network interface devices. For example, BTS information can include one or more of: parallel link available (e.g., one or more other link is available, particular link identifier(s), or indicate a flow hash(es) to select link for ECMP), Flow ID, node and link ID of location of congestion event, event information (e.g., available capacity), or others. BTS information can include congestion information collected on a path from switch to sender node at one or more hops of a path from switch to sender network interface device.

At least for case D and potentially one or more other cases, in response to receipt of the BTS information, the one or more sender network interface devices can perform one or more remedial actions. Remedial actions can include one or more of: increase load to flow/path or consider available capacity on link for new flow/path selection.

FIG.3Bdepicts an example of various combination cases. For example, for cases A and D, a trigger event at a switch can include reaching queue depth threshold and residual capacity on one or more parallel links is at or above a threshold. At least for cases A and D and potentially one or more other cases, the switch can send BTS information to one or more sender network interface devices. For example, BTS information can include one or more of: parallel link available (e.g., one or more other link is available, particular link identifier(s), or indicate a flow hash(es) to select link for ECMP), Flow ID, node and link IDs of location of congestion event, event information (e.g., pause time and/or congested queue depth for case A links or available link capacity for case D links), or others.

At least for cases A and D and potentially one or more other cases, in response to receipt of the BTS information, the one or more sender network interface devices can perform one or more remedial actions. Remedial actions can include moving packet traffic from paths that use the case A links to paths that use the case D links.

For example, for cases B and D, a trigger event at a switch can include link failure and residual capacity on one or more parallel links is at or above a threshold. At least for cases B and D and potentially one or more other cases, the switch can send BTS information to one or more sender network interface devices. For example, BTS information can include one or more of: parallel link available (e.g., one or more other link is available, particular link identifier(s), or indicate a flow hash(es) to select link for ECMP), Flow ID, node and link IDs of location of congestion event, event information (e.g., link failed for case B links or available link capacity for case D links), or others.

At least for cases B and D and potentially one or more other cases, in response to receipt of the BTS information, the one or more sender network interface devices can perform one or more remedial actions. Remedial actions can include moving packet traffic from paths that use the case B links (e.g., failed links) to different flow or path where the different flow or path as well as new connection do not include congested link, considering the available capacity on the case D links.

For example, for cases C and D, a trigger event at a switch can include residual capacity on a link decreasing towards 0% but a parallel ink has residual capacity that is at or above a threshold. At least for cases C and D and potentially one or more other cases, the switch can send BTS information to one or more sender network interface devices. For example, BTS information can include one or more of: parallel link available (e.g., one or more other link is available, particular link identifier(s), or indicate a flow hash(es) to select link for ECMP), Flow ID, node and link IDs of location of congestion event, event information (e.g., stop load increase on case C links and available link capacity for case D links), or others.

At least for cases C and D and potentially one or more other cases, in response to receipt of the BTS information, the one or more sender network interface devices can perform one or more remedial actions. Remedial actions can include for case C links, if sending rate increases, move load to one or more other path considering available capacity on case D links or if new data is to be sent, new paths avoid case C links (e.g., link closing in on capacity limit) and instead use other paths, considering available capacity on case D links (e.g., spare capacity).

Note that while examples are described with respect to BTS, information conveyed in BTS can be sent in packet headers, packet payloads, or manners described with respect to High Precision Congestion Control (HPCC) (e.g., “HPCC: High Precision Congestion Control,” SIGCOMM (2019)), or in-network telemetry (INT) (e.g., Internet Engineering Task Force (IETF) draft-kumar-ippm-ifa-01, “Inband Flow Analyzer” (February 2019)) to convey precise link load information.

FIG.4Adepicts an example process. The process can be performed by a switch in some examples. At402, a switch can receive packets for different links for forwarding. In some examples, the different links can be associated with redundant paths for packets from a sender node. At404, based on a trigger condition, the switch can indicate event information to at least one sender node. The sender node can include a sender of packets using one or more of the different, redundant links. Various examples of trigger conditions include detection of congestion of a queue that stores packets of a link, link failure, failure of a port that transmits packets of a link, capacity of a link increasing, or capacity of a link decreasing to zero or a threshold. Various examples of sender reactions are described herein.

FIG.4Bdepicts an example process. The process can be performed by a network interface device in some examples. At450, the network interface device can receive event information. For example, the event information can be transmitted by a switch based on a trigger condition. The network interface device can transmit packets on a link with one or more backup links available for use. At452, the network interface device can adjust one or more links used for transmission of packets between a switch that detected and reported the congestion and another switch or another network interface device based on event information. For example, adjusting transmission of packets can include the switch that detected and reported congestion using the congested link and potentially selecting one or more other links to transmit packets. For example, adjusting transmission of packets can include the switch that detected and reported congestion switching from using the congested link to using one or more other links to transmit packets. Various examples of remedial actions are described herein.

FIG.5depicts an example network interface device. In some examples, processors504and/or FPGAs540can be configured to perform path selection for packets based on event information received from a switch, as described herein. Some examples of network interface500are part of an Infrastructure Processing Unit (IPU) or data processing unit (DPU) or utilized by an IPU or DPU. An XPU or xPU can refer at least to an IPU, DPU, graphics processing unit (GPU), general purpose GPU (GPGPU), or other processing units (e.g., accelerator devices). An IPU or DPU can include a network interface with one or more programmable pipelines or fixed function processors to perform offload of operations that could have been performed by a CPU. The IPU or DPU can include one or more memory devices. In some examples, the IPU or DPU can perform virtual switch operations, manage storage transactions (e.g., compression, cryptography, virtualization), and manage operations performed on other IPUs, DPUs, servers, or devices. In some examples, BTS information can be processed by a processor (e.g., CPU, GPU, accelerator, packet processing pipeline, or others).

Network interface500can include transceiver502, processors504, transmit queue506, receive queue508, memory510, and bus interface512, and DMA engine552. Transceiver502can be capable of receiving and transmitting packets in conformance with the applicable protocols such as Ethernet as described in IEEE 802.3, although other protocols may be used. Transceiver502can receive and transmit packets from and to a network via a network medium (not depicted). Transceiver502can include PHY circuitry514and media access control (MAC) circuitry516. PHY circuitry514can include encoding and decoding circuitry (not shown) to encode and decode data packets according to applicable physical layer specifications or standards. MAC circuitry516can be configured to perform MAC address filtering on received packets, process MAC headers of received packets by verifying data integrity, remove preambles and padding, and provide packet content for processing by higher layers. MAC circuitry516can be configured to assemble data to be transmitted into packets, which include destination and source addresses along with network control information and error detection hash values.

Processors504can be one or more of: combination of: a processor, core, graphics processing unit (GPU), field programmable gate array (FPGA), application specific integrated circuit (ASIC), or other programmable hardware device that allow programming of network interface500. For example, a “smart network interface” or SmartNIC can provide packet processing capabilities in the network interface using processors504.

Processors504can include a programmable processing pipeline that is programmable by Programming Protocol-independent Packet Processors (P4), Software for Open Networking in the Cloud (SONiC), C, Python, Broadcom Network Programming Language (NPL), NVIDIA® CUDA®, NVIDIA® DOCA™, Infrastructure Programmer Development Kit (IPDK), or x86 compatible executable binaries or other executable binaries. A programmable processing pipeline can include one or more match-action units (MAUs) that can schedule packets for transmission using one or multiple granularity lists, as described herein. Processors, FPGAs, other specialized processors, controllers, devices, and/or circuits can be used utilized for packet processing or packet modification. Ternary content-addressable memory (TCAM) can be used for parallel match-action or look-up operations on packet header content. Processors504and/or FPGAs540can be configured to perform event detection and action.

Packet allocator524can provide distribution of received packets for processing by multiple CPUs or cores using receive side scaling (RSS). When packet allocator524uses RSS, packet allocator524can calculate a hash or make another determination based on contents of a received packet to determine which CPU or core is to process a packet.

Interrupt coalesce522can perform interrupt moderation whereby network interface interrupt coalesce522waits for multiple packets to arrive, or for a time-out to expire, before generating an interrupt to host system to process received packet(s). Receive Segment Coalescing (RSC) can be performed by network interface500whereby portions of incoming packets are combined into segments of a packet. Network interface500provides this coalesced packet to an application.

Direct memory access (DMA) engine552can copy a packet header, packet payload, and/or descriptor directly from host memory to the network interface or vice versa, instead of copying the packet to an intermediate buffer at the host and then using another copy operation from the intermediate buffer to the destination buffer.

Memory510can be any type of volatile or non-volatile memory device and can store any queue or instructions used to program network interface500. Transmit traffic manager can schedule transmission of packets from transmit queue506. Transmit queue506can include data or references to data for transmission by network interface. Receive queue508can include data or references to data that was received by network interface from a network. Descriptor queues520can include descriptors that reference data or packets in transmit queue506or receive queue508. Bus interface512can provide an interface with host device (not depicted). For example, bus interface512can be compatible with or based at least in part on PCI, PCIe, PCI-x, Serial ATA, and/or USB (although other interconnection standards may be used), or proprietary variations thereof.

FIG.6depicts an example switch. Various device and processor resources in the switch can be programmed to perform identification of link conditions, event detection, and event indication to a sender node, as described herein. Switch can receive a single packet from the source and sends one copy to one of the recipients. Switch604can route packets or frames of any format or in accordance with any specification from any port602-0to602-X to any of ports606-0to606-Y (or vice versa). Any of ports602-0to602-X can be connected to a network of one or more interconnected devices. Similarly, any of ports606-0to606-Y can be connected to a network of one or more interconnected devices.

In some examples, switch fabric610can provide routing of packets from one or more ingress ports for processing prior to egress from switch604. Switch fabric610can be implemented as one or more multi-hop topologies, where example topologies include torus, butterflies, buffered multi-stage, etc., or shared memory switch fabric (SMSF), among other implementations. SMSF can be any switch fabric connected to ingress ports and all egress ports in the switch, where ingress subsystems write (store) packet segments into the fabric's memory, while the egress subsystems read (fetch) packet segments from the fabric's memory.

Memory608can be configured to store packets received at ports prior to egress from one or more ports. Packet processing pipelines612can determine which port to transfer packets or frames to using a table that maps packet characteristics with an associated output port. Packet processing pipelines612can be configured to perform match-action on received packets to identify packet processing rules and next hops using information stored in a ternary content-addressable memory (TCAM) tables or exact match tables in some embodiments. For example, match-action tables or circuitry can be used whereby a hash of a portion of a packet is used as an index to find an entry. Packet processing pipelines612can implement access control list (ACL) or packet drops due to queue overflow.

For example, packet processing pipelines612can be configured to perform detection of events and indication of events to a sender node, as described herein. Configuration of operation of packet processing pipelines612, including its data plane, can be programmed using example programming languages and manners described herein. Processors616and FPGAs618can be utilized for packet processing or modification. In some examples, processors616can execute a virtual switch to provide virtual machine-to-virtual machine communications for virtual machines (or containers or other virtual execution environments) in a same server or among different servers.

FIG.7depicts an example system. Components of system700(e.g., processor710, graphics740, accelerators742, memory730, storage784, network interface750, and so forth) can be utilized to select a path for packets sent by network interface750or configure network interface750to select a path for transmitted packets. System700includes processor710, which provides processing, operation management, and execution of instructions for system700. Processor710can include any type of microprocessor, central processing unit (CPU), graphics processing unit (GPU), processing core, or other processing hardware to provide processing for system700, or a combination of processors. Processor710controls the overall operation of system700, and can be or include, one or more programmable general-purpose or special-purpose microprocessors, digital signal processors (DSPs), programmable controllers, application specific integrated circuits (ASICs), programmable logic devices (PLDs), or the like, or a combination of such devices.

In one example, system700includes interface712coupled to processor710, which can represent a higher speed interface or a high throughput interface for system components that needs higher bandwidth connections, such as memory subsystem720or graphics interface components740, or accelerators742. Interface712represents an interface circuit, which can be a standalone component or integrated onto a processor die. Where present, graphics interface740interfaces to graphics components for providing a visual display to a user of system700. In one example, graphics interface740can drive a high definition (HD) display that provides an output to a user. High definition can refer to a display having a pixel density of approximately 100 PPI (pixels per inch) or greater and can include formats such as full HD (e.g., 1080p), retina displays, 4K (ultra-high definition or UHD), or others. In one example, the display can include a touchscreen display. In one example, graphics interface740generates a display based on data stored in memory730or based on operations executed by processor710or both. In one example, graphics interface740generates a display based on data stored in memory730or based on operations executed by processor710or both.

Memory subsystem720represents the main memory of system700and provides storage for code to be executed by processor710, or data values to be used in executing a routine. Memory subsystem720can include one or more memory devices730such as read-only memory (ROM), flash memory, one or more varieties of random access memory (RAM) such as DRAM, or other memory devices, or a combination of such devices. Memory730stores and hosts, among other things, operating system (OS)732to provide a software platform for execution of instructions in system700. Additionally, applications734can execute on the software platform of OS732from memory730. Applications734represent programs that have their own operational logic to perform execution of one or more functions. Processes736represent agents or routines that provide auxiliary functions to OS732or one or more applications734or a combination. OS732, applications734, and processes736provide software logic to provide functions for system700. In one example, memory subsystem720includes memory controller722, which is a memory controller to generate and issue commands to memory730. It will be understood that memory controller722could be a physical part of processor710or a physical part of interface712. For example, memory controller722can be an integrated memory controller, integrated onto a circuit with processor710.

In one example, system700includes interface714, which can be coupled to interface712. In one example, interface714represents an interface circuit, which can include standalone components and integrated circuitry. In one example, multiple user interface components or peripheral components, or both, couple to interface714. Network interface750provides system700the ability to communicate with remote devices (e.g., servers or other computing devices) over one or more networks. Network interface750can include an Ethernet adapter, wireless interconnection components, cellular network interconnection components, USB (universal serial bus), or other wired or wireless standards-based or proprietary interfaces. Network interface750can transmit data to a device that is in the same data center or rack or a remote device, which can include sending data stored in memory.

In some examples, network interface device750can refer to one or more of: a network interface controller (NIC), a remote direct memory access (RDMA)-enabled NIC, SmartNIC, router, switch, forwarding element, infrastructure processing unit (IPU), data processing unit (DPU), or network-attached appliance. Some examples of network interface750are part of an Infrastructure Processing Unit (IPU) or data processing unit (DPU) or utilized by an IPU or DPU. An XPU or xPU can refer at least to an IPU, DPU, GPU, GPGPU, or other processing units (e.g., accelerator devices). An IPU or DPU can include a network interface with one or more programmable pipelines or fixed function processors to perform offload of operations that could have been performed by a CPU. A programmable pipeline can be programmed using one or more of: P4, SONiC, C, Python, Broadcom Network Programming Language (NPL), NVIDIA® CUDA®, NVIDIA® DOCA™, Infrastructure Programmer Development Kit (IPDK), or x86 compatible executable binaries or other executable binaries.

In some examples, OS732or a driver for network interface device750can select a path for packets based on event information of configure network interface750to select a path for packets based on event information.

In one example, system700includes one or more input/output (I/O) interface(s)760. I/O interface760can include one or more interface components through which a user interacts with system700(e.g., audio, alphanumeric, tactile/touch, or other interfacing). Peripheral interface770can include any hardware interface not specifically mentioned above. Peripherals refer generally to devices that connect dependently to system700. A dependent connection is one where system700provides the software platform or hardware platform or both on which operation executes, and with which a user interacts.

In one example, system700includes storage subsystem780to store data in a nonvolatile manner. In one example, in certain system implementations, at least certain components of storage780can overlap with components of memory subsystem720. Storage subsystem780includes storage device(s)784, which can be or include any conventional medium for storing large amounts of data in a nonvolatile manner, such as one or more magnetic, solid state, or optical based disks, or a combination. Storage784holds code or instructions and data786in a persistent state (e.g., the value is retained despite interruption of power to system700). Storage784can be generically considered to be a “memory,” although memory730is typically the executing or operating memory to provide instructions to processor710. Whereas storage784is nonvolatile, memory730can include volatile memory (e.g., the value or state of the data is indeterminate if power is interrupted to system700). In one example, storage subsystem780includes controller782to interface with storage784. In one example controller782is a physical part of interface714or processor710or can include circuits or logic in both processor710and interface714.

A volatile memory is memory whose state (and therefore the data stored in it) is indeterminate if power is interrupted to the device. Dynamic volatile memory uses refreshing the data stored in the device to maintain state. One example of dynamic volatile memory includes DRAM (Dynamic Random Access Memory), or some variant such as Synchronous DRAM (SDRAM). An example of a volatile memory include a cache. A memory subsystem as described herein may be compatible with a number of memory technologies, such as those consistent with specifications from JEDEC (Joint Electronic Device Engineering Council) or others or combinations of memory technologies, and technologies based on derivatives or extensions of such specifications.

A non-volatile memory (NVM) device is a memory whose state is determinate even if power is interrupted to the device. In one embodiment, the NVM device can comprise a block addressable memory device, such as NAND technologies, or more specifically, multi-threshold level NAND flash memory (for example, Single-Level Cell (“SLC”), Multi-Level Cell (“MLC”), Quad-Level Cell (“QLC”), Tri-Level Cell (“TLC”), or some other NAND). A NVM device can also comprise a byte-addressable write-in-place three dimensional cross point memory device, or other byte addressable write-in-place NVM device (also referred to as persistent memory), such as single or multi-level Phase Change Memory (PCM) or phase change memory with a switch (PCMS), Intel® Optane™ memory, NVM devices that use chalcogenide phase change material (for example, chalcogenide glass), a combination of one or more of the above, or other memory.

Communications between devices can take place using a network that provides die-to-die communications; chip-to-chip communications; circuit board-to-circuit board communications; and/or package-to-package communications. A die-to-die communications can be consistent with Embedded Multi-Die Interconnect Bridge (EMIB) or utilize an interposer.

FIG.8depicts an example system. In this system, IPU800manages performance of one or more processes using one or more of processors806, processors810, accelerators820, memory pool830, or servers840-0to840-N, where N is an integer of 1 or more. In some examples, processors806of IPU800can execute one or more processes, applications, VMs, containers, microservices, and so forth that request performance of workloads by one or more of: processors810, accelerators820, memory pool830, and/or servers840-0to840-N. IPU800can utilize network interface802or one or more device interfaces to communicate with processors810, accelerators820, memory pool830, and/or servers840-0to840-N. IPU800can utilize programmable pipeline804to process packets that are to be transmitted from network interface802or packets received from network interface802. Programmable pipeline804and/or processors806can be configured to perform identification of link conditions and event notification or response to event notification, as described herein.

In some examples, network interface and other embodiments described herein can be used in connection with a base station (e.g., 3G, 4G, 5G and so forth), macro base station (e.g., 5G networks), picostation (e.g., an IEEE 802.11 compatible access point), nanostation (e.g., for Point-to-MultiPoint (PtMP) applications), micro data center, on-premise data centers, off-premise data centers, edge network elements, fog network elements, and/or hybrid data centers (e.g., data center that use virtualization, serverless computing systems (e.g., Amazon Web Services (AWS) Lambda), content delivery networks (CDN), cloud and software-defined networking to deliver application workloads across physical data centers and distributed multi-cloud environments).

Some examples may be implemented using or as an article of manufacture or at least one computer-readable medium. A computer-readable medium may include a non-transitory storage medium to store logic. In some examples, the non-transitory storage medium may include one or more types of computer-readable storage media capable of storing electronic data, including volatile memory or non-volatile memory, removable or non-removable memory, erasable or non-erasable memory, writable or re-writable memory, and so forth. In some examples, the logic may include various software elements, such as software components, programs, applications, computer programs, application programs, system programs, machine programs, operating system software, middleware, firmware, software modules, routines, subroutines, functions, methods, procedures, software interfaces, API, instruction sets, computing code, computer code, code segments, computer code segments, words, values, symbols, or any combination thereof.

Example 1 includes one or more examples, and includes an apparatus that includes a network interface device comprising: circuitry, when operational, to: in response to congestion related to a link, cause transmission of link event information to at least one sender of packets to the link, wherein the link event information is to identify congestion information of at least one link other than the link.

Example 2 includes one or more examples, wherein the congestion related to a link comprises one or more of: link overload of a link that is part of the group of multiple links, link failure of a link that is part of the group of multiple links, link approaching transmission capacity, link in the group of multiple links having spare capacity while another link in the group of multiple links is overloaded, or a combination thereof.

Example 3 includes one or more examples, wherein the link event information comprises one or more of: backup link available, flow identifier, node and link identifier of event location, or event information.

Example 4 includes one or more examples, wherein the link event information comprises one or more of: transmission pause time, congested queue depth, indication of link failure, indication to pause packet transmission rate increase, or indication of available capacity on one or more links.

Example 5 includes one or more examples, wherein the network interface device comprises circuitry that is to determine a link in a group of multiple links to not select for packet transmission in response to receipt of the link event information.

Example 6 includes one or more examples, wherein the network interface device comprises circuitry that, in response to receipt of the link event information, is to perform an action comprising one or more of: select another link in the group of multiple links for packet transmission, select another link in the group of multiple links for packet transmission for a specified time, or select another link in the group of multiple links for packet transmission and revert to packet transmission over a former link.

Example 7 includes one or more examples, and includes a server communicatively coupled to the network interface device, wherein the server is to execute an operating system (OS) to specify an action that the network interface device is to perform in response to the link event information.

Example 8 includes one or more examples, wherein the network interface device comprises one or more of: a network interface controller (NIC), a remote direct memory access (RDMA)-enabled NIC, SmartNIC, router, switch, forwarding element, infrastructure processing unit (IPU), data processing unit (DPU), or network-attached appliance.

Example 9 includes one or more examples, wherein the circuitry comprises a programmable packet processing pipeline.

Example 10 includes one or more examples, and includes a non-transitory computer-readable medium comprising instructions stored thereon, that if executed by one or more processors, cause the one or more processors to: configure a network interface device to: in response to receipt of link event information, perform at least one action, wherein the link event information comprises congestion information of a link of a switch to which the network interface device transmits packets and congestion information of at least one other link of the switch.

Example 11 includes one or more examples, wherein the link event information is transmitted by at least one switch based on a link state trigger.

Example 12 includes one or more examples, wherein the link state trigger comprises one or more of: link overload of a link that is part of the group of multiple links, link failure of a link that is part of the group of multiple links, link approaching transmission capacity, link in the group of multiple links having spare capacity while another link in the group of multiple links is overloaded, or a combination thereof.

Example 13 includes one or more examples, wherein the link event information comprises one or more of: parallel link available, flow identifier, node and link identifier of event location, or event information, transmission pause time, congested queue depth, indication of link failure, indication to pause packet transmission rate increase, or indication of available capacity on one or more links.

Example 14 includes one or more examples, wherein the at least one action comprises one or more of: determine a link in a group of multiple links to not select for packet transmission, select another link in the group of multiple links for packet transmission, select another link in the group of multiple links for packet transmission for a specified time, or select another link in the group of multiple links for packet transmission and revert to packet transmission over a former link.

Example 15 includes one or more examples, wherein the network interface device comprises one or more of: a network interface controller (NIC), a remote direct memory access (RDMA)-enabled NIC, SmartNIC, router, switch, forwarding element, infrastructure processing unit (IPU), data processing unit (DPU), or network-attached appliance.

Example 16 includes one or more examples, and includes a method comprising: in a network interface device: adjusting a path of packet transmission among multiple enabled paths based on link event information received from at least one switch, wherein the link event information comprises congestion information of at least two link of the at least one switch.

Example 17 includes one or more examples, wherein the link event information is transmitted from at least one switch based on a link state trigger.

Example 18 includes one or more examples, wherein the link state trigger comprises one or more of: link overload of a link that is part of the group of multiple links, link failure of a link that is part of the group of multiple links, link approaching transmission capacity, link in the group of multiple links having spare capacity while another link in the group of multiple links is overloaded, or a combination thereof.

Example 19 includes one or more examples, wherein the link event information comprises one or more of: parallel link available, flow identifier, node and link identifier of event location, or event information, transmission pause time, congested queue depth, indication of link failure, indication to pause packet transmission rate increase, or indication of available capacity on one or more links.

Example 20 includes one or more examples, wherein the adjusting a path of packet transmission among multiple paths comprises one or more of: determine a link in a group of multiple links to not select for packet transmission, select another link in the group of multiple links for packet transmission, select another link in the group of multiple links for packet transmission for a specified time, or select another link in the group of multiple links for packet transmission and revert to packet transmission over a former link.