Packet flow control

A flow control method according to one embodiment may include transmitting a first plurality of packets from a transmitting node to a receiving node at an initial packet rate, and transmitting a second plurality packets from the transmitting node to the receiving node at a congested packet rate less than the initial packet rate in response to a signal from the receiving node representative of a congestion condition at the receiving node. Of course, many alternatives, variations, and modifications are possible without departing from this embodiment.

BACKGROUND

A variety of computer nodes may communicate with each other via a variety of communication links. Each node may function as a transmitting (source) and receiving (destination) device in order to exchange data and/or commands with each other using different communication protocols. Data and/or commands may be divided by the communication protocol into smaller packets of information for more efficient routing. A plurality of packets may be received and processed at the receiving node. As the amount of traffic increases, a congestion condition may occur at the receiving node.

When the congestion condition is encountered, some communication protocols specify that the receiving node send a pause type command back to the transmitting node. Upon receipt of the pause command, the transmitting node pauses or stops the transmission of any additional packets to the receiving node. The transmitting node may not send any additional packets to the receiving node until it receives another command from the receiving node indicating the congestion condition has ended. Alternatively, the transmitting node may wait a particular time interval before sending additional packets to the receiving node.

Such a stop and start method of handling congestion conditions suffers from several drawbacks. First, this method does not readily permit finer control over the bandwidth utilization of the communication link utilized by the transmitting and receiving node. This may create a larger latency variation for high priority traffic. Second, during persistent congestion conditions a plurality of pause type commands would be sent from the receiving node to the transmitting node resulting in poor bandwidth utilization of an already congested communication link. Third, the pause type command does not specify a quantity of available bandwidth for a given receiving node. Fourth, the pause type command may only be generated as a last resort resulting in an excessive amount of packets being dropped at the receiving node before the transmitting node stops sending additional packets. Fifth, longer latencies may develop if a lower amount of dropped packets are to be achieved as the transmitting node may spend more time in a pause mode not sending any packets.

DETAILED DESCRIPTION

The present disclosure will be described herein in connection with various embodiments and systems. Those skilled in the art will recognize that the features and advantages of the present disclosure may be implemented in a variety of configurations. It is to be understood, therefore, that the embodiments described herein are presented by way of illustration, not of limitation.

FIG. 1illustrates a system100including a transmitting node102and receiving node104communicating via a communication link105. The transmitting and receiving nodes102,104may represent any variety of computer nodes which may include, for example, one or more personal computers, server systems, switches, circuit boards, etc. The communication link105may also be a direct link between the transmitting and receiving node in a contained network. The transmitting node102, receiving node104, and communication link105may also comprise a local area network (LAN), wide area network (WAN) and/or storage area network (SAN). The communication link105may be a wireless link.

The transmitting node102may communicate data and/or commands to the receiving node104via the communication link105consistent with a variety of communication protocols. One such communication protocol may be an Ethernet protocol. The Ethernet protocol may comply or be compatible with the Ethernet standard published by the Institute of Electrical and Electronics Engineers (IEEE) titled the IEEE 802.3 Standard, published in March, 2002 and/or later versions of this standard. Such data and/or commands may be parsed into packets consistent with the communication protocol for more efficient routing.

A plurality of packet106,108. . .110may be transmitted by the transmitting node102to the receiving node104at an initial packet rate. The receiving node104may be able to detect a congestion condition. As used herein, a “congestion condition” may be an excessive accumulation of packets at the receiving node. Such a congestion condition may be detected in a variety of ways including a particular buffer of the receiving node that stores at least a portion of the incoming packets reaching a full threshold level. The transmitting node102may respond to the congestion condition detected at the receiving node by transmitting additional packets at a congested packet rate less than the initial packet rate. One way to control the rate of packets transmitted by the transmitting node to the receiving node is to control a length of an inter-packet gap (IPG), e.g., IPGs120,122disposed between the packets106,108. . .110.

FIG. 2illustrates an embodiment of controlling the packet rate from the transmitting node102to the receiving node104ofFIG. 1by controlling the IPG between packets in response to a congestion condition at the receiving node104. Initially, at condition C1no congestion is detected by the receiving node104. Under such a condition, the transmitting node102may transmit a plurality of packets at an initial packet rate. An IPG having a minimum size as determined by the communication protocol being utilized may be disposed between each packet. For instance, packets1,2, and3may be transmitted at an initial packet rate having a minimum IPG or IPG1 disposed between each packet. The packets1,2, and3may comply with the Ethernet communication protocol and, as such, the minimum IPG or IPG1 may be 12 bytes or 96 bits.

Communication between the transmitting and receiving node may continue at the initial packet rate until a congestion condition is detected at the receiving node104at Condition C2. Again, one way of detecting the congestion condition is for a particular buffer of the receiving node that stores at least a portion of the incoming packets to reach a full threshold level. In response to detection of the congestion condition, the receiving node may transmit a signal to the transmitting node instructing the transmitting node to slow the rate of packets. The signal may include packet x205having instructions to increase the current IPG1 level to a particular IPG2 level in order to effectively slow the rate of incoming packets to the receiving node104.

In response to the signal from the receiving node, e.g., packet x205, the transmitting node may increase the IPG disposed between packets. The transmitting node may therefore still transmit packets to the receiving node, yet at a slower packet rate compared to the initial packet rate with no congestion condition. For instance, packets4,5, and6may have IPG2 disposed between the packets, where IPG2>IPG1. The length of IPG2 compared to IPG1 may be selected in response to the severity of the congestion condition. In general, for a severe congestion condition, IPG2 may be selected to result in a large differential between IPG2 and IPG1 to considerably slow the packet rate. For a less severe congestion condition, IPG2 may be selected to result in a comparatively smaller differential between IPG2 and IPG1 than for the severe congestion condition to more slightly slow the packet rate. Hence, the packet rate may advantageously be finely tuned or controlled by controlling the length of IPG2.

At condition C3, the receiving node may detect an end of the data congestion condition. In one embodiment, this may be detected by the receiving node when the level of data in the receive buffer reaches a low threshold level. After detection of the end of the data congestion condition, the receiving node may send another signal, e.g., packet y, instructing the transmitting node to decrease IPG2 back to IPG1 in order to increase the packet rate back to the initial packet rate. In response, the transmitting node may now send additional packets at the faster initial packet rate. For example, packets7,8, and9may now have IPG1 disposed between each packet.

FIG. 3illustrates one embodiment205aof packet x205ofFIG. 2that may be sent from the receiving node to the transmitting node in response to detection of a congestion condition. The packet205amay include a destination address field302, a source address field304, a type field306, an Opcode field308, an IPG_Step field310, a priority field312, a pad314, and a cyclic redundancy check316. The destination address field302may specify the destination address of the packet, e.g., the transmitting node102. The destination address field may be 01—80_C2—00-00-01 which is a known Ethernet Media Access Control (MAC) address that is used for flow control. This allows the destination node to treat the packet specifically for flow control actions. This destination address field may be similar to the flow control functionality specified in the IEEE 802.3x standard published in May, 1997. The addition of the Opcode field308may enable continued transmission of packets at a slower rate rather than a PAUSE mechanism as detailed in the IEEE 802.3x standard. The destination address field302may require 6 bytes of a minimum sized 64 byte packet size for an Ethernet packet. The source address field304may specify the source address of the packet, e.g., the receiving node104. The source address field304may also require 6 bytes. The type field306may specify the type of packet such as a flow control type packet.

The packet205amay be an Ethernet flow control packet including additional fields such as the Opcode field308, the IPG_Step field310, and the PriMask312field. The Opcode field308may specify the type of flow control request. The type of flow control request may include a type (0x0001) specifying continued transmission of additional packets at a slower packet rate. The IPG_Step field310may have a plurality of steps, e.g., steps1-8, specifying a quantity to increase the IPG. The IPG_Step may be selected in response to the severity of the congestion condition detected. For instance, the IPG_Step may be set to a larger step, e.g., step8, in response to a severe congestion condition and may be set to a smaller step, e.g., step1, in response to a mild congestion condition. Hence, the packet rate may be finely tuned or controlled by controlling the length of IPG via the IPG_Step field310.

The priority (Primask) field312may be utilized to control specific priority traffic. Different communication protocols may have different levels of priority traffic and, in general, higher priority traffic may be favored over lower priority traffic. For instance, the Ethernet communication protocol may have eight levels of priority traffic. The priority field312may specify to increase the IPG on the lower priority traffic. Therefore, the congestion condition may be relieved by increasing the IPG on the lower priority traffic without increasing the IPG of the higher priority traffic. The pad field314may be utilized to pad the length of the packet205aso that the packet achieves a minimum packet length. For instance, given a minimum packet length of 64 bytes and the sum of all the other fields302,304,306,308,310,312, and316totaling 22 bytes, the pad314may be 42 bytes. Finally, error detection codes such as the cyclic redundancy check (CRC) may be included in the packet205a.

FIG. 4illustrates how the priority level field312of the packet205aofFIG. 3may be utilized to sequentially increase the IPG of lower to higher priority traffic.FIG. 4illustrates a first402and a second404plurality of lower priority packets having an increased IPG or IPG2 disposed between each packet. The IPG2 level for each plurality of packets402,404may have been set by respective packets consistent with packet205aspecifying a particular IPG_step in field310for the particular priority level of the plurality of packets402,404. The plurality of higher priority packets406may still have a minimum IPG or IPG1.

If the congestion condition at the receiving node continues, the receiving node may instruct a continually increasing priority of level of traffic to increase its IPG. Given the particulars of the congestion condition and the amount of lower priority traffic contributing to this condition, it is possible to slow the rate of the lower priority traffic only while concurrently maintaining a higher packet rate for the higher priority traffic as illustrated inFIG. 4.

As earlier detailed, the transmitting and receiving nodes102,104may be a variety of nodes.FIG. 5illustrates one embodiment where the transmitting and receiving nodes may be circuit boards and a switch508. The circuit boards502,503may be coupled via insertion into associated slots of the chassis500to a backplane506. The backplane506may have the switch508to facilitate communication between, at least, the two circuit boards502,503. Only two circuit boards502,503are illustrated inFIG. 5for clarity although the chassis500may accept any plurality of circuit boards, e.g.,14in one embodiment.

In one embodiment, the chassis500may be an Advanced Telecommunications Computing Architecture (Advanced TCA or ATCA) chassis, complying with or compatible with PCI Industrial Computer Manufacturers Group (PCIMG) rev. 3.0, Advanced Telecommunications Computing Architecture (ATCA), published Dec. 30, 2002. According to this embodiment, the circuit boards502,503disposed within the chassis500may be ATCA boards, also referred to as ATCA blades or blades herein. The ATCA blades upon proper insertion into associated slots in the chassis500may be mechanically coupled to the chassis500and electrically coupled to the backplane506. The backplane506may be the primary interconnect board of the chassis500and provide a number of connections for each ATCA blade such as a connection to a power riser board and a data riser board.

FIG. 6illustrates the two blades502,503and the switch508ofFIG. 5. Each blade502,503may be coupled to the switch508via an associated port602,604. For simplicity only two blades502,503and one switch508are illustrated although any plurality of blades and switch combinations may be utilized. The switch508may capable of detecting a congestion condition at each port602,604and may detect a congestion condition at one port602but not another port604. The switch508may then control the flow of packets to the congested port602from blade502to a congested packet rate, e.g., by increasing the IPG between packets. At the same time, the switch508may enable a maximum packet arrival rate at the non-congested port604coupled to the other blade503.

FIG. 7illustrates one embodiment502aof the blade502ofFIGS. 5 and 6. The blade502amay include a host processor712, a bus722, a chipset714, system memory721, a card slot730, and a network interface card (NIC)740. The host processor712may include one or more processors known in the art such as an Intel® XEON processor commercially available from the Assignee of the subject application. The bus722may include various bus types to transfer data and commands. For instance, the bus722may comply with the Peripheral Component Interconnect (PCI) Express™ Base Specification Revision 1.0, published Jul. 22, 2002, available from the PCI Special Interest Group, Portland, Oreg., U.S.A. (hereinafter referred to as a “PCI Express™ bus”). The bus722may alternatively comply with the PCI-X Specification Rev. 1.0a, Jul. 24, 2000, available from the aforesaid PCI Special Interest Group, Portland, Oreg., U.S.A. (hereinafter referred to as a “PCI-X bus”).

The chipset714may include a host bridge/hub system (not shown) that couples the processor712and system memory721to each other and to the bus722. The chipset714may include one or more integrated circuit chips, such as those selected from integrated circuit chipsets commercially available from the Assignee of the subject application (e.g., graphics memory and I/O controller hub chipsets), although other integrated circuit chips may also, or alternatively be used. System memory721may include one or more machine readable storage media such as random-access memory (RAM), dynamic RAM (DRAM), static RAM (SRAM), magnetic disk, and/or any other device that can store information.

When the NIC740is properly inserted into the slot730, connectors734and737become electrically and mechanically coupled to each other. When connectors734and737are so coupled to each other, the NIC740becomes electrically coupled to bus722and may exchange data and/or commands with system memory721and host processor712via bus722and chipset714.

Alternatively, without departing from this embodiment, the operative circuitry of the NIC740may be included in other structures, systems, and/or devices. These other structures, systems, and/or devices may be, for example, in the blade502aand coupled to the bus722. These other structures, systems, and/or devices may also be, for example, comprised in chipset714. The NIC740may act as an intermediary between the blade502aand other nodes to permit communication to and from the blade502aand other nodes. One such node may be the switch508. Communication may take place using any variety of communication protocols such as the Ethernet communication protocol.

The NIC740may include an integrated circuit (IC)760. The IC760may include protocol engine circuitry having a MAC layer. As used herein, an “integrated circuit” or IC means a semiconductor device and/or microelectronic device, such as, for example, a semiconductor integrated circuit chip. As used herein, “circuitry” may comprise, for example, singly or in any combination, hardwired circuitry, programmable circuitry, state machine circuitry, and/or firmware that stores instructions executed by programmable circuitry. The MAC layer, which again may be part of the IC760, may assemble packets for transmission by assembling a data portion of the packet with a header portion of the packet. To increase the IPG between packets, the MAC layer may hold the header for a longer period of time before assembling the header and data portion of the packet for transmission.

The blade502amay also include any variety of machine readable media such as system memory721. Machine readable program instructions may be stored in any variety of such machine readable media so that when the instructions are executed by a machine, e.g., by the processor712in one instance, or circuitry in another instance, it may result in the machine performing operations described herein.

FIG. 8illustrates one embodiment508aof the switch508ofFIGS. 5 and 6. The switch508amay include a buffer802, control pipeline circuitry804, transmit queue block circuitry808, memory controller803, and packet memory806. The control pipeline circuitry804may further include parser circuitry812, address resolution unit circuitry814, address memory816, and apply rules circuitry818. The switch508amay receive a plurality of packets at various ports. Only one packet870is illustrated inFIG. 8for clarity. Each packet870may have a header portion872and data portion874. The header portion872may include information such as source and destination computer nodes. The data portion874may include any variety of data being transferred from one computer node to another.

The header portion of each packet, e.g., header portion872of packet870, may be passed to the buffer802. The buffer802may be a first-in first-out (FIFO) buffer in one embodiment. In response to a level of data in the buffer802relative to one or more threshold levels, a congestion condition may be detected by the switch508a. The header portion of each received packet may then be passed from the buffer802to the control pipeline circuitry804. The control pipeline circuitry804may perform a variety of operations on the header of received packets. Parser circuitry812may parse received headers into associated fields, e.g., source address fields and destination address fields. Address resolution unit circuitry814may perform associated lookups such as source, destination, and rule lookups.

The address resolution unit circuitry814accordingly may accesses address memory816to perform such lookups. Apply rules circuitry818may apply rules that were obtained from address resolution unit circuitry814. The apply rules circuitry818may also form a transmit queue entry in the transmit queue block circuitry808for each packet which may then by queued into the appropriate port queue with the transmit queue block circuitry808. When a packet is being transmitted from the switch508a, the header portion for each packet may be obtained from the transmit queue block circuitry808and the data portion for each packet may be obtained from packet memory806by memory controller803and transmitted out the appropriate port. The memory controller803and transmit queue block circuitry808may be part of the MAC layer of the switch. To control the IPG between packets, the MAC layer may hold the header for a particular time interval before assembling the header and data portion of the packet for transmission. To increase the IPG, the MAC layer may hold the packet for a longer period of time before transmission.

FIG. 9is a flow chart of operations900consistent with an embodiment. Operation902may include transmitting a first plurality of packets from a transmitting node to a receiving node at an initial packet rate. Operation904may include transmitting a second plurality packets from the transmitting node to the receiving node at a congested packet rate less than the initial packet rate in response to a signal from the receiving node representative of a congestion condition at the receiving node.

It will be appreciated that the functionality described for all the embodiments described herein, may be implemented using hardware, firmware, software, or a combination thereof.

Thus one embodiment may comprise an article. The article may comprise a storage medium having stored therein instructions that when executed by a machine result in the following: transmitting a first plurality of packets to a receiving node at an initial packet rate; and transmitting a second plurality packets to the receiving node at a congested packet rate less than the initial packet rate in response to a signal from the receiving node representative of a congestion condition at the receiving node.

Another embodiment may comprise a chassis. The chassis may comprise a backplane having a switch, and a circuit board accepted in a slot of the chassis and coupled to the backplane. The circuit board may be capable of communicating with the switch. The circuit board may further be capable of transmitting a first plurality of packets to the switch. The circuit board may further be capable of receiving a signal from the switch representative of a congestion condition at the switch. The circuit board may further be capable of transmitting a second plurality of packets to the switch at a congested packet rate less than the initial packet rate in response to the signal from the switch.

Another embodiment may comprise a chassis. The chassis may comprise a backplane having a switch, and a circuit board accepted in a slot of the chassis and coupled to the backplane. The circuit board may comprise a network interface card. The network interface card may be capable of communicating with the switch. The network interface card may further be capable of transmitting a first plurality of packets to the switch. The network interface card may further be capable of receiving a signal from the switch representative of a congestion condition at the switch. The network interface card may further be capable of transmitting a second plurality of packets to the switch at a congested packet rate less than the initial packet rate in response to the signal from the switch.

Yet another embodiment may comprise an Advanced Telecommunications Computing Architecture (ATCA) chassis. The ATCA chassis may comprise at least one ATCA circuit board accepted in a slot of the ATCA chassis, and a backplane having a switch. The switch may have at least one port capable of communicating with the at least one ATCA circuit board. The at least one ATCA circuit board may be capable of transmitting a first plurality of packets to the switch via the at least one port at an initial packet rate. The switch may be capable of detecting a congestion condition at the at least one port and sending a signal to the at least one ATCA board representative of the congestion condition. The at least one ATCA circuit board may be capable of transmitting a second plurality of packets to the switch at a congested packet rate less than the initial packet rate in response to the signal from the switch. The least one ATCA board may be capable of controlling a length of an inter-packet gap disposed between at least two of the second plurality of packets to change the initial packet rate to the congested packet rate. The first and second plurality of packets may comply with an Ethernet communication protocol, and the inter-packet gap may be greater than 12 bytes.

Advantageously, in these embodiments, a transmitting node may continue to transmit packets to a receiving node despite a congestion condition at the receiving node. Additional packets may be transmitted to the receiving node at a congested packet rate less than an initial packet rate when no congestion is detected at the receiving node. The congested packet rate may advantageously be finely tuned or controlled by controlling the length of the IPG between packets. As opposed to a conventional start and stop method of flow control, this method may enable a more granular control of the bandwidth available on a given communication link, and may also enable a lower packet drop rate and shorter latency. This method is beneficial in a variety of transmitting and receiving node environments and particularly in a contained network environment where the transmitting node and receiving node are located in relative proximity to each other such as an ATCA chassis for modular servers.

Controlling the IPG of a particular priority traffic flow enables further refinement and control over packet flow. Different packet rates for differing priority level packets may be concurrently achieved. This is a vast improvement over a conventional flow control method that stops all traffic, regardless of its priority level, upon detection of a congestion condition. Accordingly, the link utilization of the communication link between the transmitting and receiving node is also improved compared to conventional start and stop flow control methods.

Various ports of a switch may also be adapted to treat congestion conditions individually so that a port detecting a congestion condition may control a packet rate to its port, while other non-congested ports of the switch may continue to receive packets at a maximum packet arrival rate. The IPG may also be dynamically controlled in response to varying traffic patterns and internal resources.