Abstract:
Congestion detection is implemented on the ingress direction, egress direction, or both directions in a network interface controller. Congestion notification is facilitated between a source and a destination through a network interface controller. In accordance with an aspect, a NIC is configured to label ingress packets—coming from the network—in relation to the presence of congestion. In accordance with another aspect, the NIC is configured to label egress packets—going to the network—in relation to the presence of congestion. A full offload NIC is equipped to operate on the packet labels in the presence of congestion and to react appropriately if it is on the receiving end, sending end, or both.

Description:
TECHNICAL FIELD 
     The present invention relates to congestion control in a packet switched network. Congestion is a common problem in packet switched networks, which typically lack admission control mechanisms that would prevent packets from entering an oversubscribed network. 
     BACKGROUND 
     In the context of Internet Protocol (IP) networks, congestion control is handled at the transport layer, e.g. by the Transmission Control Protocol (TCP). TCP congestion control mostly relies on an implicit congestion indication in the form of lost packets. Mechanisms for explicit congestion notification (ECN) have been proposed and standardized, requiring intermediate packet switches (routers) to explicitly mark bits in the IP header of a packet indicating the incidence of congestion on the path of the packet. The notifications are reflected back by the receiver in the headers of acknowledgment or other packets flowing in the opposite direction. ECN has seen limited deployment and in some networks has shown benefits compared to packet loss and implicit congestion detection. However, there are no mechanisms indicating congestion at the receiving and/or sending node. A window flow control mechanism is available in TCP and is limited to controlling buffer use at the receiving end. Moreover, it is often the case that flow control windows are sized without relation to the available resources at the receiver, such as memory, in part to improve performance when multiple connections are established, but the different connections are active at different times. 
     SUMMARY 
     A Network Interface Controller (NIC)—which may be, for example, network interface circuitry, such as on a PCI card connected to a host computer via a PCI host bus—typically includes receive functionality used to couple the host computer to a packet network through at least one interface, usually called a port. NIC circuitry has been an area of rapid development as advanced packet processing functionality and protocol offload have become requirements for so called “smart NICs”. 
     It has become common for receive functionality of NIC cards to parse packet headers and perform operations such as checksum checking computations, filter lookups and modifications such as VLAN tag removal and insertion. Additionally, the receive functionality in some NICs implement protocol offload such as TCP/IP, iSCSI, RDMA and other protocols. 
     Advances in network connectivity speed increases and system traffic generation and processing capacity are not coupled, and occur as the different relevant technologies develop. For example, Ethernet speeds typically increase in factors of 10× every few years. In contrast, system components, such as host bus, CPU speeds or memory subsystem bandwidth have different increase steps, varying from 2× every few years for the first, to 10% per year for CPUs and somewhat less for memory. 
     A mismatch may exist between the ingress rate of packets from the network at the receive functionality and the rate at which the host computer is accepting and/or can accept packets on the host bus from the NIC. This mismatch may be temporary, caused by load on the host system processors, or in general can be caused by the offered load exceeding the capacity of the host system. Receive functionality of a NIC can be equipped to detect and indicate such conditions, which remain otherwise mostly unbeknownst to network routers and switches. 
     In an Ethernet packet switched network, a “PAUSE” mechanism is available to request an immediate neighbor of a node to pause transmission when the node detects congestion. However, this action is limited in scope to the link level, and does not affect the sending rate of the source node, if that node is not the immediate neighbor but is instead connected to the destination through one or more switches. Instead, end-to-end congestion avoidance and control mechanisms are preferred, such as the ones implemented by the Transmission Control Protocol (TCP). 
     In accordance with an aspect of the invention, congestion detection and explicit notification mechanisms are extended to the receive functionality of the NIC, which is equipped to mark congestion notification bits similarly to intermediate routers. Furthermore, smart NICs which implement receive protocol offload, can themselves reflect the indication back to the sender in packets the NICs generate according to the protocol requirements. 
     In accordance with another aspect of the invention, congestion detection and notification are extended to transmit functionality of the NIC, whereby the NIC can detect temporary and/or long term mismatch in the traffic sending rate of the node and the traffic sinking rate of its network port. 
     Independently, server virtualization has been an area of active development, leading to significant virtualization related features being implemented in a NIC. Some NICs have thus incorporated virtual bridges for switching traffic between two virtual machines residing on the host system. In accordance with an aspect of the invention, receive and transmit functionality in a virtual bridge may provide congestion detection and notification capabilities even for connections that are facilitated by the NIC between functions executing on the same host. 
     ECN has been standardized in the context of TCP/IP networks for more than a decade, but has seen limited deployment and use, perhaps due to the heterogeneity of the network and the large number of entities managing it. However, there is potential for ECN gaining acceptance in the context of datacenters, which combine a large network infrastructure, where congestion may be a significant concern, with a single management authority. 
     Packet switched networks typically lack admission control mechanisms that operate based on congestion in the network. 
     In accordance with an aspect of the current invention, receive and/or transmit functionality of network interface circuitry (“NIC”) is equipped with ingress and/or egress congestion detection functionality, respectively. In the presence of ingress congestion, the NIC may pass an indication of the congestion (notification) to the receiving protocol stack. In the presence of egress congestion, the NIC may pass an indication of the congestion (notification) to the sending protocol stack. At least one of the receiving protocol stack and the sending protocol stack may be operating on a host coupled to the network by the NIC or operating in the NIC itself (protocol offload). 
     For example, the notification may be carried in the appropriate fields of packets of a protocol stack when relevant, or as sideband information. For example, in a TCP/IP-compliant network, the notification can be carried in the ECN bits as specified by the relevant standards. 
     A NIC that implements packet switching functionality can therefore provide an indication of congestion even for traffic that is confined to the NIC itself, such as in virtualized configurations, where the NIC switches packets between virtual machines executing on the host system. 
     A NIC which implements a network stack (protocol offload NIC) can furthermore act on congestion detection and perform the required congestion avoidance and control actions. In a TCP/IP network, the receiving action may include reflecting the detected congestion appropriately to the sending side. The sending action may include reducing sending speed. 
     A NIC which implements packet switching functionality and protocol offload can thus be extended to implement both sides of an explicit congestion notification capable network node, such as in virtualized configurations, where the NIC switches packets between virtual machines executing on the host system. 
     A NIC which, in addition to providing an interface between a network and a host, implements packet switching functionality where the NIC switches packets between network ports may be configured to mark the packets flowing through according to congestion detection, or other criteria. 
    
    
     
       BRIEF DESCRIPTION OF FIGURES 
         FIG. 1  is a block diagram illustrating a host system with a network adapter configured to detect and indicate congestion. 
         FIG. 2  is a block diagram illustrating how a network adaptor configured to detect and indicate congestion operates as a virtual network adaptor. 
         FIG. 3  more specifically shows how the network adaptor may detect congestion, in relation to a processing engine and switch functionality of the network adaptor. 
         FIG. 4  is a flowchart illustrating an example of the congestion detection and indication process that may be followed. 
         FIG. 5  is a block diagram illustrating a host system with a protocol processing network adapter configured to detect congestion and perform related protocol processing. 
     
    
    
     DETAILED DESCRIPTION 
     We now describe a novel application of the packet processing capability of a NIC, equipped with a congestion detection facility. Referring to  FIG. 1 , a NIC  101  is coupled to a host system  102 , using an attachment bus  103 . The host system  102  communicates via a network  104  to a peer  105 . A portion or all of the NIC  101 , perhaps also comprising circuitry on the host system  102 , may be considered an interface  106  between the network  104  and the host system  102 . The interface  106  may comprise an ingress interface and/or an egress interface. 
     The NIC  101  can detect congestion on the ingress portion of the interface  106  such as by monitoring the occupancy of relevant state elements, like header or payload FIFOs, or general usage of NIC resources for packet ingress. For example, a payload or header FIFO occupancy threshold can be programmed, such that when the FIFO occupancy exceeds said threshold, the FIFO is considered congested. In another example, a “freelist” of host based receive packet buffers is maintained, and the number of available buffers monitored. When the number of available buffers drops below a certain threshold, the receiving host is considered to be congested. As another example, the NIC  101  may also detect congestion through monitoring the internal state of busses and connections. Based thereon, the NIC  101  may maintain an ingress congestion state that is indicative of the detected ingress congestion, such as indicative of detected congestion episodes. (It is noted that, by “connection,” this is not meant to imply that a connection state is necessarily present and/or being maintained at both ends of the “connection.”) 
     According to the maintained congestion state, the NIC  101  may mark packets received during congestion detected episodes. The marking can be passed as sideband (out-of-band) information to the receiving host  102  to be associated with a received packet (e.g., as part of a receive descriptor associated with the packet), or a flow to which the packet belongs, or through setting or clearing appropriate bits in the packet headers. Modifying bits in the packet headers may require the NIC to adjust integrity check values that cover said bits. 
     The congestion notification markings are then processed by the host  102  according to a congestion protocol  107 . Typically, based on the processing the congestion notification markings, the congestion protocol  107  causes one or more congestion indication to be communicated back to a peer  105 , using congestion experienced indications. The double arrow  108  indicates this bidirectional passing of congestion information—from the NIC  101  to the host  102 , and from the host  102  to the peer  105 . 
     The NIC  101  may also (or instead) detect congestion on the egress interface when processing packets for transmission. Congestion detection may be based on monitoring the state of NIC resources in the transmit direction, similarly to the receive congestion detection. It may include monitoring the network (PAUSE) state of the egress interface. The NIC can, for example, mark the packets in appropriate fields in the headers or communicate the notification back to the host system to be passed to the congestion indication processing  107 . The notification can be associated with a packet by including identification information from the packet, such as appropriate network headers, or a flow information if a flow has been associated with the packet or, in general, an identifier associated with the packet. Alternatively, the NIC may mark the packets by modifying relevant bits in the packet headers to indicate congestion experienced. Modifying bits in the headers may require the NIC to recompute or adjust check values that cover the bits. 
     In accordance with another aspect, as shown in  FIG. 2 , a NIC  201  is coupled to a host system  202 , using an attachment bus  203 . The host system  202  communicates via a network  204  to a peer  205 . A portion or all of the NIC  201 , perhaps also comprising circuitry on the host system  202 , may be considered an interface  206  between the network  204  and the host system  202 . The interface  206  may comprise an ingress interface and/or an egress interface. 
     Like the  FIG. 1  arrangement, the NIC  201  can detect congestion on the ingress portion of the interface  206  and congestion on the egress portion of the interface  206 . Congestion notification markings may then be processed by the host  202  according to a congestion protocol  207 . 
     In addition, the host  202  may be configured to operate multiple virtual machines or guest functions. The NIC  201  may operate as a “virtual network”  214  between the virtual machines such as guest VM 1   208   a  and guest VM 2   208   b . Furthermore, similar to the description above with respect to  FIG. 1 , the NIC  201  may be configured to detect congestion in the virtual network  214 , and indicate the detected congestion such as in an inner IP header of VXLAN/GRE/tunnel traffic. 
     A NIC may also implement a function to switch packets between two entities on the host side, or between an ingress port and an egress port. Packets flowing through the switching functionality may be marked by the NIC according to the presence of congestion. It may be beneficial to implement marking as part of general traffic management (e.g., the congestion marking may not be based on congestion detection exclusively, but may also be based on desired traffic management).  FIG. 3  illustrates a NIC  300  that includes a host interface  301  and a network interface  303 . The NIC  300  includes processing engine functionality  312  to handle processing of ingress and egress data to a host via the host interface  301  including, for example, implementing the offloading of connection endpoint protocol processing from the host. The NIC  300  may also include switch processing functionality  314  to handle switching of packets. The processing engine functionality  312  and the switch processing functionality  314  may actually be handled by the same processing pipeline. 
     As indicated in  FIG. 3  receive congestion detection  302   a  for the processing engine functionality  312  may be the same as the receive congestion detection  302   b  for the switch processing functionality  314 ; and transmission congestion detection  304   a  for the processing engine functionality  312  may be the same as the transmit congestion detection  304   b  for the switch processing functionality  314 . Furthermore, congestion information may be passed by the processing engine functionality  312  between the receive path and transmit path, including in the case the offloading of connection endpoint protocol processing from the host to the NIC  300 . See, for example, arrows  306   a  and  306   b  in  FIG. 3 , which are indicative of such passing of congestion information. As discussed above relative to  FIG. 3 , the host interface  301  may be utilized to pass congestion indication information as part of packets provided to the host and/or as sideband information. 
       FIG. 4  is a flowchart illustrating a method in accordance with an aspect of the invention. At  402 , the network interface circuitry processes packets. At  404 , it is determined if, based on processing the packets, there is congestion relative to the packets. At  406 , if congestion has been detected, then the congestion is indicated. At  408 , the congestion indication is provided in correlation to the packets for which congestion has been detected. For example, the congestion experienced indication may be provided to the host correlated to the ingress packets, such as including the congestion experienced indication correlated to a flow to which the ingress packets belong. This may include, for example, providing the congestion experienced indication as at least one of information from headers of the ingress packets or a flow identification obtained by the network interface circuitry using a lookup. 
       FIG. 5  is a diagram illustrating a NIC  501  connecting a host system  502  to a peer  503  through a network  504 . The NIC performs receive and transmit network protocol processing, such as TCP/IP offload. The protocol processing is conceptually categorized into transmit  505  and receive  506  processing. The NIC is configured to detect congestion in the receive direction, the transmit direction, or both. A receive “congestion detected” indication may be passed to the transmit processing for appropriate action, e.g. applying a label or setting relevant bits on transmitted packets. A transmit “congestion detected” indication may be passed to the congestion algorithm processing, similarly to relevant indications received on ingress packets. The passing of “congesting detected” indications just described is indicated in  FIG. 5  by arrow  508 .