Source: https://patents.google.com/patent/US20030195983
Timestamp: 2018-04-21 01:40:04
Document Index: 287823380

Matched Legal Cases: ['arts 0', 'art 0', 'art 1', 'art 1', 'art 2', 'art 3']

US20030195983A1 - Network congestion management using aggressive timers - Google Patents
Network congestion management using aggressive timers Download PDF
US20030195983A1
US20030195983A1 US10442401 US44240103A US2003195983A1 US 20030195983 A1 US20030195983 A1 US 20030195983A1 US 10442401 US10442401 US 10442401 US 44240103 A US44240103 A US 44240103A US 2003195983 A1 US2003195983 A1 US 2003195983A1
US10442401
A network system includes links and end stations coupled between the links. Types of end stations include endnodes which originate or consume frames and routing devices which route frames between the links. At least one end station includes an aggressive timer adapted to respond to an occurrence of at least one condition of delayed frame transmission progress, to provide a frame delay indication when the at least one condition exists for a duration that exceeds a variable timing threshold. The variable timing threshold is configurable based on at least one network system attribute.
This patent application is a Continuation-in-part of U.S. patent application Ser. No. 09/578,019, entitled “RELIABLE MULTICAST,” filed May 24, 2000, and having Attorney Docket No. HP PDNO 10991834-2, which is herein incorporated by reference. U.S. patent application Ser. No. 09/578,019 is a Continuation-in-Part Application of U.S. patent application Ser. No. 09/578,155, filed May 23, 2000, entitled “RELIABLE DATAGRAM” having Attorney Docket No. HP PDNO 10991833-1 which is herein incorporated by reference. U.S. patent application Ser. No. 09/578,019 also claimed the benefit of the filing date of U.S. Provisional Patent Applications Serial No. 60/135,664, filed May 24, 1999 and having Attorney Docket No. HP PDNO 10991654-1; and Ser. No. 60/154,150, filed Sep. 15, 1999 and having Attorney Docket No. HP PDNO 10992562-1, both of which are herein incorporated by reference.
The present invention generally relates to communication in network systems and more particularly to network system congestion management.
A traditional network system, such as a computer system, has an implicit ability to communicate between its own local processors and from the local processors to its own I/O adapters and the devices attached to its I/O adapters. Traditionally, processors communicate with other processors, memory, and other devices via processor-memory buses. I/O adapters communicate via buses attached to processor-memory buses. The processors and I/O adapters on a first computer system are typically not directly accessible to other processors and I/O adapters located on a second computer system.
In conventional distributed computer systems, distributed processes, which are on different nodes in the distributed computer system, typically employ transport services to communicate. A source process on a first node communicates messages to a destination process on a second node via a transport service. A message is herein defined to be an application-defined unit of data exchange, which is a primitive unit of communication between cooperating sequential processes. Messages are typically packetized into frames for communication on an underlying communication services/fabrics. A frame is herein defined to be one unit of data encapsulated by a physical network protocol header and/or trailer.
Certain conventional distributed computer systems employ various means for addressing network traffic congestion. One known method is to drop frames that fail to make forward progress over a defined period of time. One realization of the method involves the use of forward progress timers implemented to prevent deadlock, or abnormal congestion, of a transmission port. Forward progress timers are typically implemented to expire 100-500 milliseconds after their initiation. Thus, relative to network transmission rates, a forward progress timer is quite coarse, as its purpose is to relieve severe deadlock.
On expiration of the forward progress timer, one or more frames are typically purged from the port. For example, the current stalled frame can be dropped from the queue. Alternatively, all packets in a queue targeting the congested port can be dropped. Typically, the sender of a dropped frame will become aware of the transmission failure via a protocol layer, such as by nonreceipt of an acknowledgment frame (ACK), and will respond to the inferred congestion assumed to be the cause of the transmission failure. Example sender responses include re-sending of the frame, and reducing of the transmission rate.
In practice, a plurality of senders are often targeting a congestion-prone port. As a result, frame dropping at times of congestion can lead to a problem of synchronization, where sources simultaneously respond to perceived congestion by reducing their respective transmission rates. As a result, the previously-congested port becomes under-utilized for some time. Hence, a cycle of congestion and under-utilization can develop, resulting in reduced overall throughput and efficiency.
One solution to the problem of synchronization is the use of random frame dropping, whereby only randomly-selected frames are dropped upon the expiration of a forward progress timer. Senders of the randomly-dropped frames perceive the failure of transmission as congestion, and respond to the perceived congestion. Consequently, the random frame dropping can dampen the aggregate global response to a port's congestion, resulting in greater port utilization, throughput, and efficiency.
A further improvement involves weighted random algorithms, which treat frames of a higher priority more favorably while employing randomization to avoid synchronization. Random and weighted-random-based techniques have been implemented in early detection schemes, such as disclosed in Weighted Random Early Detection on the Cisco 12000 Series Router, available at http://www.cisco.com/univercd/cc/td/doc/product/software/ios112/ios112p/gsr/w red_gs.pdf. Early detection is premised on anticipating congestion and responding before it becomes severe. To address the problem of a forward progress timer's coarse granularity, some anticipatory methods use a queue size measuring technique for estimating a congestion level. One drawback of such queue size-based approaches to congestion management, is their inability to provide a time-based service guarantee for high priority frames that require a minimum-delay service priority, such as multimedia or real-time applications.
Network utilization and associated congestion continues to grow as a result of the increasing scale of distributed computer systems, variety of network elements, and increasingly-complex applications. For example, the expanding use of multimedia and real-time applications, and their associated bandwidth requirements, is fueling the need for specialized congestion management techniques and policies.
For reasons stated above and for other reasons presented in the Description of the Preferred Embodiments section of the present specification, there is a need for a congestion management technique that improves network utilization, throughput, and efficiency.
One aspect of the present invention provides a network system having links and end stations coupled between the links. Types of end stations include endnodes which originate or consume frames and routing devices which route frames between the links. At least one end station includes an aggressive timer adapted to respond to an occurrence of at least one condition of delayed frame transmission progress, to provide a frame delay indication when the at least one condition exists for a duration that exceeds a variable timing threshold. The variable timing threshold is configurable based on at least one network system attribute.
[0014]FIG. 1 is a diagram of a distributed computer system.
[0015]FIG. 2 is a diagram of an example host processor node for the computer system of FIG. 1.
[0016]FIG. 3 is a diagram of a portion of a distributed computer system employing a reliable connection service to communicate between distributed processes.
[0017]FIG. 4 is a diagram of a portion of distributed computer system employing a reliable datagram service to communicate between distributed processes.
[0018]FIG. 5 is a diagram of an example host processor node for operation in a distributed computer system.
[0019]FIG. 6 is a diagram of a portion of a distributed computer system illustrating subnets in the distributed computer system.
[0020]FIG. 7 is a diagram of a switch for use in a distributed computer system.
[0021]FIG. 8 is a diagram of a portion of a distributed computer system.
[0022]FIG. 9A is a diagram of a work queue element (WQE) for operation in the distributed computer system of FIG. 8.
[0023]FIG. 9B is a diagram of the packetization process of a message created by the WQE of FIG. 9A into frames and flits.
[0024]FIG. 10A is a diagram of a message being transmitted with a reliable transport service illustrating frame transactions.
[0025]FIG. 10B is a diagram illustrating a reliable transport service illustrating flit transactions associated with the frame transactions of FIG. 10A.
[0026]FIG. 11 is a diagram of a layered architecture.
[0027]FIG. 12 is a diagram illustrating a method for recognizing and managing network system congestion using congestion detection, reporting, and responding mechanisms.
[0028]FIG. 13A is a simplified diagram illustrating one embodiment of a network system that includes an end station comprising an aggressive timer according to one embodiment of the present invention.
[0029]FIG. 13B is a diagram illustrating an exemplary end station of FIG. 13A that includes an aggressive timer adapted to monitor the end station's port according to one embodiment of the present invention.
[0030]FIG. 13C is a diagram of an exemplary end station of FIG. 13A that runs an application program according to one embodiment of the present invention.
[0031]FIG. 13D is a diagram of an exemplary end station of FIG. 13A that includes a traffic congestion manager according to the present invention.
[0032]FIG. 14 is a diagram of an example of one embodiment of a routing element having a network traffic congestion manager utilizing aggressive timers.
[0033]FIG. 15 is a flow diagram illustrating an aggressive timer's and traffic congestion manager's operation in the exemplary routing element of FIG. 14.
[0034]FIG. 16 is a diagram of one embodiment of an end station utilizing an aggressive timer and a forward progress timer.
One aspect of the present invention is directed to a method and apparatus providing an aggressive timer-based congestion manager. This aspect facilitates the application of various criteria for selective treatment of frames based on frame or network system attributes as part of congestion detection, reporting, and/or response mechanisms. In one embodiment, the congestion manager supports the enforcement of policies set by network fabric management. In one embodiment, the congestion manager is employed on a network entity, including a network fabric element or end station, such as a switch, router, or endnode.
An example embodiment of a distributed computer system is illustrated generally at 30 in FIG. 1. Distributed computer system 30 is provided merely for illustrative purposes, and the embodiments of the present invention described below can be implemented on network systems of numerous other types and configurations. For example, network systems implementing the present invention can range from a small server with one processor and a few input/output (I/O) adapters to massively parallel supercomputer systems with hundreds or thousands of processors and thousands of I/O adapters.
Furthermore, the present invention can be implemented in an infrastructure of remote computer systems connected by an internet or intranet.
Distributed computer system 30 includes a system area network (SAN) 32 which is a high-bandwidth, low-latency network interconnecting nodes within distributed computer system 30. A node is herein defined to be any device attached to one or more links of a network and forming the origin and/or destination of messages within the network. In the example distributed computer system 30, nodes include host processors 34 a-34 d; redundant array independent disk (RAID) subsystem 33; and I/O adapters 35 a and 35 b. The nodes illustrated in FIG. 1 are for illustrative purposes only, as SAN 32 can connect any number and any type of independent processor nodes, I/O adapter nodes, and I/O device nodes. Any one of the nodes can function as an endnode, which is herein defined to be a device that originates or finally consumes messages or frames in the distributed computer system.
A message is herein defined to be an application-defined unit of data exchange, which is a primitive unit of communication between cooperating sequential processes. A frame is herein defined to be one unit of data encapsulated by a physical network protocol header and/or trailer. The header generally provides control and routing information for directing the frame through SAN 32. The trailer generally contains control and cyclic redundancy check (CRC) data for ensuring frames are not delivered with corrupted contents.
SAN 32 is the communications and management infrastructure supporting both I/O and interprocess communication (IPC) within distributed computer system 30. SAN 32 includes a switched communications fabric (SAN FABRIC) allowing many devices to concurrently transfer data with high-bandwidth and low latency in a secure, remotely managed environment. Endnodes can communicate over multiple ports and utilize multiple paths 30 through the SAN fabric. The multiple ports and paths through SAN 32 can be employed for fault tolerance and increased bandwidth data transfers.
In distributed computer system 30, host processor nodes 34 a-34 d and RAID subsystem node 33 include at least one system area network interface controller (SANIC) 42. In one embodiment, each SANIC 42 is an endpoint that implements the SAN 32 interface in sufficient detail to source or sink frames transmitted on the SAN fabric. The SANICs 42 provide an interface to the host processors and I/O devices. In one embodiment the SANIC is implemented in hardware. In this SANIC hardware implementation, the SANIC hardware offloads much of CPU and I/O adapter communication overhead. This hardware implementation of the SANIC also permits multiple concurrent communications over a switched network without the traditional overhead associated with communicating protocols. In one embodiment, SAN 32 provides the I/O and IPC clients of distributed computer system 30 zero processor-copy data transfers without involving the operating system kernel process, and employs hardware to provide reliable, fault tolerant communications.
The host processors 34 a-34 d include central processing units (CPUs) 44 and memory 46.
Channel semantics and memory semantics are typically both necessary for I/O and IPC. A typical I/O operation employs a combination of channel and memory semantics. In an illustrative example I/O operation of distributed computer system 30, host processor 34 a initiates an I/O operation by using channel semantics to send a disk write command to I/O adapter 35 b. I/O adapter 35 b examines the command and uses memory semantics to read the data buffer directly from the memory space of host processor 34 a. After the data buffer is read, I/O adapter 35 b employs channel semantics to push an I/O completion message back to host processor 34 a.
In one embodiment, distributed computer system 30 performs operations that employ virtual addresses and virtual memory protection mechanisms to ensure correct and proper access to all memory. In one embodiment, applications running in distributed computed system 30 are not required to use physical addressing for any operations.
An example host processor node 34 is generally illustrated in FIG. 2. Host processor node 34 includes a process A indicated at 60 and a process B indicated at 62. Host processor node 34 includes SANIC 42. Host processor node 34 also includes queue pairs (QPs) 64 a and 64 b which provide communication between process 60 and SANIC 42. Host processor node 34 also includes QP 64 c which provides communication between process 62 and SANIC 42. A single SANIC, such as SANIC 42 in a host processor 34, can support thousands of QPs. By contrast, a SAN interface in an I/O adapter 35 typically supports less than ten QPs.
In one embodiment, receive work queues 68 only support one type of WQE, which is referred to as a receive buffer WQE. The receive buffer WQE provides a channel semantic operation describing a local buffer into which incoming send messages are written. The receive buffer WQE includes a scatter list describing several virtually contiguous local buffers. An incoming send message is written to these buffers. The buffer virtual addresses are in the address space of the process that created the local QP.
The reliable connection service of distributed computer system 100 associates a local QP with one and only one remote QP. Thus, QP 116 is connected to QP 122 via a non-sharable resource connection 128 having a non-sharable resource connection 128 a from send work queue 116 a to receive work queue 122 b and a non-sharable resource connection 128 b from send work queue 122 a to receive work queue 116 b. QP 118 is connected to QP 124 via a non-sharable resource connection 130 having a non-sharable resource connection 130 a from send work queue 118 a to receive work queue 124 b and a non-sharable resource connection 130 b from send work queue 124 a to receive work queue 118 b. QP 120 is connected to QP 126 via a non-sharable resource connection 132 having a non-sharable resource connection 132 a from send work queue 120 a to receive work queue 126 b and a non-sharable resource connection 132 b from send work queue 126 a to receive work queue 120 b.
A send buffer WQE placed on one QP in a reliable connection service causes data to be written into the receive buffer of the connected QP. RDMA operations operate on the address space of the connected QP.
The reliable connection service requires a process to create a QP for each process which is to communicate with over the SAN fabric. Thus, if each of N host processor nodes contain M processes, and all M processes on each node wish to communicate with all the processes on all the other nodes, each host processor node requires M2×(N−1) QPs. Moreover, a process can connect a QP to another QP on the same SANIC.
A portion of a distributed computer system employing a reliable datagram service to communicate between distributed processes is illustrated generally at 150 in FIG. 4. Distributed computer system 150 includes a host processor node 152, a host processor node 154, and a host processor node 156. Host processor node 152 includes a process A indicated at 158. Host processor node, 154 includes a process B indicated at 160 and a process C indicated at 162. Host processor node 156 includes a process D indicated at 164.
For example, a reliable datagram service 174 couples QP 166 to QPs 168, 170, and 172. Specifically, reliable datagram service 174 couples send work queue 166 a to receive work queues 168 b, 170 b, and 172 b. Reliable datagram service 174 also couples send work queues 168 a, 170 a, and 172 a to receive work queue 166 b.
The reliable datagram service permits a client process of one QP to communicate with any other QP on any other remote node. At a receive work queue, the reliable datagram service permits incoming messages from any send work queue on any other remote node.
The reliable datagram service greatly improves scalability because the reliable datagram service is connectionless. Therefore, an endnode with a fixed number of QPs can communicate with far more processes and endnodes with a reliable datagram service than with a reliable connection transport service. For example, if each of N host processor nodes contain M processes, and all M processes on each node wish to communicate with all the processes on all the other nodes, the reliable connection service requires M2×(N−1) QPs on each node. By comparison, the connectionless reliable datagram service only requires M QPs+(N−1) EE contexts on each node for exactly the same communications.
An example host processor node is generally illustrated at 200 in FIG. 5. Host processor node 200 includes a process A indicated at 202, a process B indicated at 204, and a process C indicated at 206. Host processor 200 includes a SANIC 208 and a SANIC 210. As discussed above, a host processor endnode or an I/O adapter endnode can have one or more SANICs. SANIC 208 includes a SAN link level engine (LLE) 216 for communicating with SAN fabric 224 via link 217 and an LLE 218 for communicating with SAN fabric 224 via link 219. SANIC 210 includes an LLE 220 for communicating with SAN fabric 224 via link 221 and an LLE 222 for communicating with SAN fabric 224 via link 223. SANIC 208 communicates with process A indicated at 202 via QPs 212 a and 212 b. SANIC 208 communicates with process B indicated at 204 via QPs 212 c-212 n. Thus, SANIC 208 includes N QPs for communicating with processes A and B. SANIC 210 includes QPs 214 a and 214 b for communicating with process B indicated at 204. SANIC 210 includes QPs 214 c-214 n for communicating with process C indicated at 206. Thus, SANIC 210 includes N QPs for communicating with processes B and C.
A portion of a distributed computer system is generally illustrated at 250 in/FIG. 6. Distributed computer system 250 includes a subnet A indicated at 252 and a subnet B indicated at 254. Subnet A indicated at 252 includes a host processor node 256 and a host processor node 258. Subnet B indicated at 254 includes a host processor node 260 and host processor node 262. Subnet A indicated at 252 includes switches 264 a-264 c. Subnet B indicated at 254 includes switches 266 a-266 c. Each subnet within distributed computer system 250 is connected to other subnets with routers. For example, subnet A indicated at 252 includes routers 268 a and 268 b which are coupled to routers 270 a and 270 b of subnet B indicated at 254. In one example embodiment, a subnet has up to 216 endnodes, switches, and routers.
An example embodiment of a switch is generally illustrated at 280 in FIG. 7. Each I/O path on a switch or router has an LLE. For example, switch 280 includes LLEs 282 a-282 h for communicating respectively with links 284 a-284 h.
The naming scheme for switches and routers is similar to the above-described naming scheme for endnodes. The following is an example switch and router naming scheme for identifying switches and routers in the SAN fabric. A switch name identifies each switch or group of switches packaged and managed together. Thus, there is a single switch name for each switch or group of switches packaged and managed together.
[0124]FIGS. 8, 9A, 9B, 10A, and 10B together illustrate example request and acknowledgment transactions. In FIG. 8, a portion of a distributed computer system is generally illustrated at 300. Distributed computer system 300 includes a host processor node 302 and a host processor node 304. Host processor node 302 includes a SANIC 306. Host processor node 304 includes a SANIC 308. Distributed computer system 300 includes a SAN fabric 309 which includes a switch 310 and a switch 312. SAN fabric 309 includes a link 314 coupling SANIC 306 to switch 310; a link 316 coupling switch 310 to switch 312; and a link 318 coupling SANIC 308 to switch 312.
In the example transactions, host processor node 302 includes a client process A indicated at 320. Host processor node 304 includes a client process B indicated at 322. Client process 320 interacts with SANIC hardware 306 through QP 324. Client process 322 interacts with SANIC hardware 308 through QP 326. QP 324 and 326 are software data structures. QP 324 includes send work queue 324 a and receive work queue 324 b. QP 326 includes send work queue 326 a and receive work queue 326 b.
Process 320 initiates a message request by posting WQEs to send work queue 324 a. Such a WQE is illustrated at 330 in FIG. 9A. The message request of client process 320 is referenced by a gather list 332 contained in send WQE 330. Each entry in gather list 332 points to a virtually contiguous buffer in the local memory space containing a part of the message, such as indicated by virtual contiguous buffers 334 a-334 d, which respectively hold message 0, parts 0, 1, 2, and 3.
Referring to FIG. 9B, hardware in SANIC 306 reads WQE 330 and packetizes the message stored in virtual contiguous buffers 334 a-334 d into frames and flits. As illustrated in FIG. 9B, all of message 0, part 0 and a portion of message 0, part 1 are packetized into frame 0, indicated at 336 a. The rest of message 0, part 1 and all of message 0, part 2, and all of message 0, part 3 are packetized into frame 1, indicated at 336 b. Frame 0 indicated at 336 a includes network header 338 a and transport header 340 a. Frame 1 indicated at 336 b includes network header 338 b and transport header 340 b.
As indicated in FIG. 9B, frame 0 indicated at 336 a is partitioned into flits 0-3, indicated respectively at 342 a-342 d. Frame 1 indicated at 336 b is partitioned into flits 4-7 indicated respectively at 342 e-342 h. Flits 342 a through 342 h respectively include flit headers 344 a-344 h.
Frames are routed through the SAN fabric, and for reliable transfer services, are acknowledged by the final destination endnode. If not successfully acknowledged, the frame is retransmitted by the source endnode. Frames are generated by source endnodes and consumed by destination endnodes. The switches and routers in the SAN fabric neither generate nor consume frames.
Flits are the smallest unit of flow control in the network. Flits are generated and consumed at each end of a physical link. Flits are acknowledged at the receiving end of each link and are retransmitted in response to an error. For example, controlling retransmission or abortion of flit transmission can be accomplished with each sender maintaining a per-port link retry timer to monitor the time between flit transmission and receipt acknowledgment. If the link retry timer expires, the sender attempts to retry transmission of the outstanding flit. A second timer, called a link kill timer, is active while the sender is operating in a retry mode. On expiration of the link kill timer, transmission is aborted.
[0133]FIG. 10B illustrates the flits associated with the request frames 336 and acknowledgment frames 346 illustrated in FIG. 10A passing between the host processor endnodes 302 and 304 and the switches 310 and 312. As illustrated in FIG. 10B, an ACK frame fits inside one flit. In one embodiment, one acknowledgment flit acknowledges several flits.
As illustrated in FIG. 10B, flits 342 a-h are transmitted from SANIC 306 to switch 310. Switch 310 consumes flits 342 a-h at its input port, creates flits 348 a-h at its output port corresponding to flits 342 a-h, and transmits flits 348 a-h to switch 312. Switch 312 consumes flits 348 a-h at its input port, creates flits 350 a-h at its output port corresponding to flits 348 a-h, and transmits flits 350 a-h to SANIC 308. SANIC 308 consumes flits 350 a-h at its input port. An acknowledgment flit is transmitted from switch 310 to SANIC 306 to acknowledge the receipt of flits 342 a-h. An acknowledgment flit 354 is transmitted from switch 312 to switch 310 to acknowledge the receipt of flits 348 a-h. An acknowledgment flit 356 is transmitted from SANIC 308 to switch 312 to acknowledge the receipt of flits 350 a-h.
Acknowledgment frame 346 a fits inside of flit 358 which is transmitted from SANIC 308 to switch 312. Switch 312 consumes flits 358 at its input port, creates flit 360 corresponding to flit 358 at its output port, and transmits flit 360 to switch 310. Switch 310 consumes flit 360 at its input port, creates flit 362 corresponding to flit 360 at its output port, and transmits flit 362 to SANIC 306. SANIC 306 consumes flit 362 at its input port. Similarly, SANIC 308 transmits acknowledgment frame 346 b in flit 364 to switch 312. Switch 312 creates flit 366 corresponding to flit 364, and transmits flit 366 to switch 310. Switch 310 creates flit 368 corresponding to flit 366, and transmits flit 368 to SANIC 306.
If a network is properly provisioned and the topology is well understood and controlled, network traffic will make forward progress under most operating workloads (albeit at reduced operating efficiency) assuming that network end nodes are consuming their inbound frames, and thus generally avoiding back pressure. However, at very high speeds, or in a network with variable speed link elements and/or topologies, the overall network efficiency and application effective throughput can drop to the point where the network appears to be down, even while network traffic is not in a condition of deadlock per se.
An example embodiment of a method for recognizing and managing network system congestion using congestion detection, reporting, and responding mechanisms is illustrated in FIG. 12. The method involves monitoring network traffic conditions for signs of inefficiency, or network bandwidth underutilization. At 435, monitored network conditions 436 are tested for the presence of certain conditions indicative of delayed frame transmission progress, defined at 437. The result of the testing at 435 is indicated at 438. At 439, based on test result 438, the duration of monitored conditions 436 is timed and compared to a configured timing threshold 440. At 441, an indication is provided if, at 439, the measured duration of monitored conditions 436 exceeds configured timing threshold 440. A response 442 is made based on indication 441 and on response configuration 443.
One embodiment of measurement mechanism 439 is implemented in a network end station, such as an end node or routing device, that includes one or more aggressive timers for indicating a delay in frame transmission. An aggressive timer is herein defined as a timing device or mechanism that monitors the transmission of information in a network system or network system element, and can be configured to provide an indication of whether and when the transmission of information fails to meet at least one variable timing threshold. An aggressive timer can be implemented in hardware or software.
One embodiment of a network system is illustrated generally at 449 in FIG. 13A. Network system 449 includes an end station 450 comprising aggressive timer 452. End station 450 interfaces with other end stations of SAN 451 through links 484 a and 484 b. Network traffic internal to end station 450 is indicated at 454. Aggressive timer 452 monitors network traffic 454 via a monitoring mechanism 456.
Aggressive timer 452's variable timing threshold is configurable and re-configurable as a function of network system attributes, as indicated at 453. An attribute of an entity is herein defined as including, but not limited to, at least one aspect, characteristic, condition, configuration, essence, parameter, property, quality, setting, or status, of the entity to which it refers. Aggressive timer 452 sufficiently monitors network traffic to permit its variable timing threshold to be configured to detect or anticipate the onset of congestion. Aggressive timer 452's variable timing threshold is configurable and re-configurable to accommodate different network circumstances, such as variations in network system attributes, including frame attributes and operating conditions within the network system.
The variable timing threshold of the aggressive timer 452 refers to a duration of time against which one or more defined conditions of delayed frame transmission progress can be compared. The conditions of delayed frame transmission progress can be defined in a number of ways according to network traffic management policy. For example, in one embodiment, an expiration of a link retry timer is a condition of delayed frame transmission progress. In one embodiment, the variable timing threshold is varied from one configuration to another by setting a starting count, ending count and/or counting rate. In one embodiment, the variable timing threshold is varied by adjusting a timing interval having a configurable ending time relative to a starting time. In one embodiment, the variable timing threshold is configurable to enable or disable the aggressive timer. When the variable timing threshold is exceeded, numerous suitable responses can be taken as part of enforcing a congestion management policy.
[0157]FIG. 13B illustrates an exemplary end station 450 according to one embodiment of the present invention, which includes an aggressive timer 452. The Aggressive timer 452 is adapted to monitor one or more ports 455 of end station 450. In one embodiment, aggressive timer 452 measures whether the transmission of a frame has stalled for a time that exceeds the variable timing threshold. In one embodiment, a variable timing threshold's duration is only slightly longer than the time needed for transmitting a given frame out of a port 455 under normal operating conditions. Such a short time limit facilitates fast recognition of transmission delay, which permits anticipation of network congestion. In one embodiment, the aggressive timer has microsecond granularity.
In one embodiment, aggressive timer 452 is implemented in end station 450 as a count-down timer set to a specific counting value and counting rate, which together, represent the configured variable timing threshold. In this embodiment, count down aggressive timer 452 counts down during one or more defined conditions of delayed frame transmission progress, such as a period of transmission delay associated with a port 455. In one form of this embodiment, the frame transmission delay is defined by an indication of the output frame's failure to make forward progress. If the frame makes forward progress, count-down aggressive timer 452 is reset. In another form of this embodiment, the count-down aggressive timer 452 is set to count while any part of a frame remains in a transmit register, regardless of forward progress status.
In one embodiment, the variable timing threshold is configured by an appropriate authority, such as network fabric management. In an alternative embodiment, the variable timing threshold is dynamically configurable, either manually or by program, based on potentially changing circumstances or decision-making criteria relating to the network system. Examples of potentially changing network circumstances include network system attributes, such as a congestion level of an end station's port, a policy for managing network congestion, or various attributes of a frame to be transmitted.
An exemplary potentially changing decision-making criteria relating to the network system, is a congestion management policy instituted by network fabric management. Network circumstances and decision-making criteria can also be interrelated, such that a congestion management policy, for example, can change in response to changing network system conditions. The dynamic characteristics of embodiments of the present invention can be employed to implement potentially changing network decision-making criteria, measuring potentially changing network circumstances, or both.
One embodiment of aggressive timer 452 is configurable based on one or more network system attributes. In this embodiment, configuration of the variable timing threshold can be made based on one or more predetermined network system attributes and/or based on measured network system status. Predetermined network system attributes herein refers to information characterizing at least one aspect of the network system and its contents that was determined at some time before the period surrounding the setting of the variable timing threshold, or over a period of time prior to, and potentially including, the decision-making period. Measured network system status, on the other hand, refers to at least one aspect of the network system and its contents that is determined in the period surrounding the setting of the variable timing threshold. The variable timing threshold can be dynamically configurable in embodiments of aggressive timer 452 having a variable timing threshold based on predetermined or measured network system attributes.
In some embodiments, aggressive timer 452 is configurable based on one or more attributes of at least one port 455. In one such embodiment, the aggressive timer 452's variable timing threshold is adjustable as a function of a measured presence of back pressure from a port. Back pressure herein refers to network operation that is indicative of insufficient buffer space. In another such embodiment, the variable timing threshold is based on historical data of a port's back pressure occurrences over a period of time. In one example operation of end station 450 of FIG. 13B, port 455 a tends to be congested, and end station 450 accordingly institutes a more stringent variable timing threshold for congestion management.
In one embodiment, the variable timing threshold is configurable based on the type of workload of applications utilizing the network system. FIG. 13C illustrates an end station 450, which runs an application 458. Aggressive timer 452's variable timing threshold is configurable based on the type of workload of application 458. Workload herein refers to the information exchanged over the network for a given application. Example types of workloads include digital video information for video applications, and file transfer protocol (FTP). Since video workloads are less tolerant of inconsistent transmission rates, an exemplary congestion management policy can impose variable timing thresholds with shorter limits for video applications than for web page browsing applications. In one embodiment, a network system which predominately carries video application data, comprises end stations 450 with aggressive timers 452 configured with stringent variable timing thresholds for early detection and prevention of congestion.
In one embodiment, the variable timing threshold is configurable based on an amount of time one or more frames fails to make forward progress. In one embodiment, end station 450 employs aggressive timer 452 to measure congestion experienced by a first frame targeting a particular port 455. In one example of this embodiment, the aggressive timer 452 is set to expire after a time period that is approximately the time needed to transmit the first frame to its targeted port 455 in the absence of congestion. In one embodiment, end station 450 is further configured to track the number of frame delay indications of aggressive timer 452, as applied to the first frame. If the frame is delayed due to congestion, aggressive timer 452 provides at least one frame delay indication. In this embodiment, the tracked number of frame delay indications represents an amount of time during which the frame fails to make forward progress. In this embodiment, the variable timing threshold for identifying a level of congestion that requires a response, is represented by a number of aggressive timer 452 expirations. The variable timing threshold for a second frame targeting the same port 455 can be adjusted based on the number.
In one embodiment, the variable timing threshold is configurable based on the type of network system architecture. In some embodiments, an end station 450 transmitting a frame employs predetermined information about at least one end station along the frame's intended transmission path to configure the variable timing threshold for the frame. In one such embodiment, the predetermined information includes the at least one end station's role in overall system performance. For example, if a first frame's routing path includes a switch that is a major network hub and potential bottleneck, the variable timing threshold for the first frame can be set to be more stringent.
In another embodiment, the variable timing threshold is configurable based on historical data of the congestion status of the end stations along a frame's routing path. In another embodiment, the variable timing threshold is based on a measured congestion status of an end station along the routing path. In another embodiment, the variable timer threshold is configurable based on a predetermined or measured transmission bandwidth of at least one downstream end station. The bandwidth might be restricted due to the end stations' capacity or congestion status. In an example of such an embodiment, a variable timing threshold of an aggressive timer 452 used with a port 455 is a function of the associated link hop speed.
In some embodiments, the variable timing threshold is based on a frame attribute. In one such embodiment, the frame upon which the threshold is based is examined to ascertain its relevant attributes. In this embodiment, the end station 450 includes a hardware or software mechanism for examining a frame's protocol header and/or trailer. In another embodiment, the variable timing threshold is based on the size of a frame. In one form of this embodiment, the aggressive timer 452 is configured to a limit that is proportional to the size of the frame in the transmit register of port 455. A limit that is only slightly greater than the time for transmitting the frame, provides a high sensitivity for detecting transmission delay.
In another embodiment, the variable timing threshold is configurable based on the output frame's information type, such as whether the frame carries data or control information. In one such embodiment, the information type is determined using a mechanism for parsing out a message's opcode. Depending on a policy for managing congestion in a distributed computer system, control frames can be given a higher or lower priority of service than data-bearing frames, such that the aggressive timer 452 is configured to allow higher priority frames more time to make forward progress before a congestion management response is taken with respect to the frames.
In some embodiments, the variable timing threshold is dynamically configurable based on a frame's source or destination end station. In one such embodiment, the aggressive timer 452 is configured based on the frame's final destination end station. In this embodiment, the end station 450 transmitting the frame examines the frame to determine its final destination, and configures the variable timing threshold of aggressive timer 452 accordingly. In one example embodiment, a frame addressed to an inherently slower device, such as a disk drive, is subject to a more stringent timing threshold (i.e., a shorter time limit) than a frame addressed to a faster device, such as a video output device. In this embodiment, the transmitting end station 450 has a mechanism to recognize the final destination end station type with respect to its role on system performance. Therefore, an exemplary transmitting end station, utilizing aggressive timing configured based on final frame destination, can enforce a priority policy that gives service precedence to frames destined for end station types that have more time-critical roles.
In one embodiment, the variable timing threshold is based on the source end station of a given frame. Network system policy may have assigned a higher or lower priority for frames originating from particular end nodes. Thus, an exemplary transmitting end station 450 along the frame's routing path examines the frame header to determine the originating end node, looks up the variable timing threshold for the end node, and applies the variable timing threshold as a time limit for the aggressive timer 452.
In another embodiment where the variable timing threshold for an aggressive timer is configurable based on one or more frame attributes, the frame being transported is examined for its group identification, which is indicative of the type of communication service of the frame. Example communication service types include, but are not limited to: multicasting, unicasting, and broadcasting. An exemplary embodiment of this type implements one or more policies for congestion management or priority servicing, where the policy dictates various priorities for respective communication services. In one such embodiment, the variable timing threshold is applied frame-by-frame according to each frame's communication service. For example, in a network system where a policy provides higher priority to multicast transmissions, end stations 450 along a routing path for a given multicast frame, apply less stringent timing thresholds (longer aggressive timer limits) to the multicast frame.
In one type of embodiment, the variable timing threshold is configurable based on a frame's assigned service level. In one such embodiment, the end station 450 examines the frame to be transmitted to determine its service level flag values. Depending on the frame's priority, the variable timing threshold is configured to be more or less stringent. In an exemplary embodiment, an aggressive timing limit for a low-priority frame is set to a lower, more stringent level, to cause the low-priority frame to not be transmitted in favor of preserving transmission bandwidth for higher-priority frames.
Upon expiration of the aggressive timer 452, an exemplary end station 450 takes one or more actions in response thereto. In one embodiment, as illustrated in FIG. 13D, the end station 450 includes a traffic congestion manager 460 for responding to the aggressive timer 452's indication of its expiration. Traffic congestion manager 460's response is denoted at 462; aggressive timer 452's indication is denoted at 464. The traffic congestion manager 460 can be realized in hardware or software. In one configuration, a traffic congestion manager and at least one aggressive timer are realized as a single functional unit 466, which performs the functions of both the aggressive timer and traffic congestion manager.
In some embodiments, the traffic congestion manager 460 is configurable to respond to the aggressive timer 452's indication in a number of ways. In one such embodiment, the traffic congestion manager responds to an expiration of the aggressive timer 452 by dropping at least one frame. In one exemplary embodiment, the traffic congestion manager 460 drops the frame currently being transmitted. In another exemplary embodiment, the traffic congestion manager 460 drops frames randomly, or by a weighted random algorithm. In another exemplary embodiment, the traffic congestion manager 460 drops one or more frames in a port's transmission buffer based on the frames' attributes. For example, traffic congestion manager 460 drops all frames in the output buffer targeting the same intermediate or final destination end station that the current frame targets when the aggressive timer 452 measuring the current frame's transmission progress expires. In another example, the traffic congestion manager 460 clears the entire transmit buffer of the port in response to aggressive timer 452's expiration. In one type of embodiment, the frame dropping is performed over a certain period of time in order to affect future frames as well as the current frame.
In an embodiment of a traffic congestion manager 460, upon expiration of the aggressive timer 452, congestion manager 460 truncates the current frame by discarding the frame's untransmitted flits. In another embodiment, the traffic congestion manager 460 responds to an expiration of aggressive timer 452 by generating one or more reporting frames to be used by one or more end stations or fabric management for congestion management purposes. An exemplary reporting frame includes data characterizing the nature of the aggressive timer's expiration, or circumstances surrounding the aggressive timer's expiration. In one embodiment, the reporting frame has data containing information about the delayed frame during the transmission of which the aggressive timer 452 expired, such as the frame's size, destination, or service level.
In another type of embodiment, the traffic congestion manager 460 responds to aggressive timer 452's expiration by tagging the current frame with information indicative of the congestion experienced by the frame. In one such embodiment, the current frame and all subsequent frames thereto are tagged for a period of time. Tagging can thus be used as a means for communicating traffic congestion information to other end stations in the network. The other end stations can include fabric management agents, neighboring routing elements, and source endnodes.
In another type of embodiment, the traffic congestion manager 460 responds to the aggressive timer 452's expiration by logging the expiration occurrence and its surrounding circumstances. One such embodiment of traffic congestion manager 460 maintains a log of aggressive timer 452 expirations, wherein the log contains information useful for characterizing the end station 450's congestion status over a period time. In one embodiment, this characterization is employed to re-configure at least one aggressive timer 452's variable timing threshold.
In one type of embodiment, the traffic congestion manager 460 is dynamically configurable. In one such embodiment, the response to the aggressive timer 452 is selectable by network system or fabric management. In other embodiments, the response is dynamically configurable based on predetermined or measured changing circumstances, as described above for the dynamically configurable aggressive timer.
In another embodiment, the response 462 is configurable based on one or more frame delay indications 464 provided by the aggressive timer 452. For example, in an implementation where the aggressive timer is configured to indicate multiple levels of delay (such as with a plurality of frame transmission delay indications), the traffic congestion manager 460 can take different responses as the frame delay continues. The responses 462 can progressively increase in their efficacy as a measured delay becomes longer. In an exemplary configuration, the initial response 462 is a type of event logging; the next response 462 is communication of congestion status (such as frame tagging or sending congestion management packets); the next response 462 is a type of selective frame dropping; finally, the most drastic response 462 to aggressive timer expiration is a flushing of all buffered frames for a period of time.
[0180]FIG. 14 illustrates an exemplary routing element 500 interfacing with SAN fabric 509 through a link indicated generally at 584. Routing element 500 includes ports 502 a, 502 b and 502 c. Ports 502 connect via links 584 a, 584 b and 584 c, respectively, with SAN fabric 509. Each port 502 has a receive register 504 and transmit queue 506. Transmit queues 506 include head-of-line transmit registers 508, and back-of-line queue input registers 509. Receive registers 504 and transmit registers 508 connect to link 284.
Bus 510 provides an interface for receive registers 504 and queue input registers 509 to exchange frames within the routing element 500. Receive registers 504 interface with a bus 510 that facilitates communication with the two transmit queues 506 from the two other ports. Each queue input register 509 interfaces with the two receive registers 504 from the two other ports via bus 510.
To illustrate a switching operation of routing element 500, consider an exemplary incoming frame to routing element 500 arriving to port 502 a, and which is to be transmitted out through port 502 c. The incoming frame arrives to port 502 a's receive register 504 a via link 584. The frame then passes into transmit queue 506 c through queue input register 509 c via bus 510. As transmit queue 506 c transmits previous frames out through transmit register 508 c, the frame advances until it reaches transmit register 508 c. Finally, the frame is transmitted from transmit register 508 c via link 584 into SAN fabric 309 on its way to its final destination.
Exemplary routing element 500 also includes per-port aggressive timers 512 a, 512 b and 512 c, and traffic congestion manager 516. Aggressive timers 512 and traffic congestion manager 516 are components of a local network traffic congestion management system for implementing traffic congestion management policies. Each aggressive timer 512 monitors the transmission status 514 of each corresponding transmission register 508. Transmission status 514 is indicative of frame transmission progress, such as whether flits of the transmitted frame are being sent. In another configuration, transmission status 514 is simply an indicator of when a new frame enters transmit register 508. Timers 512 also monitor transmission demand 515 for each corresponding port 502. The transmission demand 515 of each port 502 is indicative of each port's pending outgoing traffic volume. For example, transmission demand 515 provides a signal whenever the corresponding queue input register 509 receives a new frame.
Each aggressive timer 512 measures the presence of any transmission delay of frames in its corresponding transmit register 508. The measurement is based on one or more variable timing threshold(s) with which each aggressive timer 512 is configured. As discussed above, the variable timing thresholds themselves can be based on a variety of attributes, parameters, or attributes relating to the network system, including one or more frames. In the present example, a variable timing threshold is provided to each aggressive timer 512 by traffic congestion manager 516 via a timer configuration signal, indicated at 522. Timer configuration signal 522 can be supplied continuously, periodically, or occasionally to dynamically configure the aggressive timers 512. Each aggressive timer 512 can have a unique configuration signal, as indicated at 522 a, 522 b and 522 c.
If a measured transmission delay exceeds the configured variable timing threshold of an aggressive timer 512, the aggressive timer 512 will provide a timer expiration indication 518 to traffic congestion manager 516. Traffic congestion manager 516 can then take an appropriate response according to the traffic management policy instituted. Responses include a variety of actions, some of which are described above. One such action is dropping one or more frames from a congested port 502. Frames can be dropped selectively, based on one or more frame attributes. In the present example, a clear register signal is illustrated for each port at 520.
[0186]FIG. 15 illustrates an example operation of the aggressive timer 512 of one port 502 of routing element 500. As indicated at 600, aggressive timer 512 monitors transmission demand 515 of transmit queue 506. As indicated at 602, aggressive timer 512 monitors transmission status 514. Under non-congested, or unexceptional operating conditions, frames are either making forward progress or not arriving into transmit queue 506. Therefore, under unexceptional circumstances, aggressive timer 512 remains in a reset state or is in a state receptive to configuration signal 522, as indicated at 604.
If an exception occurs at a time when frames are arriving and when the output frame in transmit register 508 does not make forward progress, then aggressive timer 512 starts counting at 606. At 608, an inquiry and decision is made as to whether aggressive timer 512 has expired. If aggressive timer 512 has not expired and the exceptional conditions remain, then aggressive timer 512 continues applying the configured variable timing threshold, as indicated at 606. If the aggressive timer 512 has not expired but the exceptional circumstances no longer exist, then aggressive timer 512 is rest at 604 and the normal non-exceptional operating mode described above resumes. If, however, aggressive timer 512 expires, then traffic congestion manager 516 takes an appropriate action and response to the aggressive timer 512′ expiration, such as dropping one or more frames from transmit queue 506 and/or receive register 504. Aggressive timer 512 is then reset and/or reconfigured, as indicated at 604.
[0188]FIG. 16 illustrates an exemplary embodiment of an end station 750 having an aggressive timer 752 and a forward progress timer 770. End station 750 transmits or receives network traffic 754 via links 784 a and 784 b.
Aggressive timer 752 provides timewise monitoring of network traffic 754 via a monitoring mechanism 756, and according to aggressive timer configuration 753. Upon expiration, aggressive timer 752 provides a timer expiration indication 764. Traffic congestion manager 760 receives expiration indication 764, and provides a selected response 762 according to network traffic management policy for aggressively-timed network traffic. Forward progress timer 770 monitors network traffic 754 for an occurrence of a severe congestion indication 772, and according to preselected timing criteria. Forward progress timer 770 provides a response action 774 according to network traffic management policy for forward progress-timed network traffic.
Forward progress timer 770 is generally employed to facilitate a solution for severe, abnormal congestion. Forward progress timer 770 receives severe congestion indication 772 when end station 750 cannot send a frame after a long time, such as for tens or hundreds of milliseconds. In response, forward progress timer 770 takes one or more actions 774 for relieving the congestion.
Conversely, aggressive timer 752 is generally employed to facilitate implementation of a congestion management policy by recognizing and responding to early indications of congestion or bandwidth underutilization. Thus, exemplary aggressive timer 752 is implemented with finer granularity than the forward progress timer 770. In one embodiment, the aggressive timer 752 has a counting rate that is a multiple of forward progress timer 770's counting rate. In one embodiment, aggressive timer 752 has microsecond resolution.
The following Pseudo-Code I demonstrates the timing and congestion management functionality of an exemplary end station having a forward progress timer (FT) and an aggressive timer (AT).
PSEUDO-CODE I
while (frames are arriving to buffer) {
if (output port frame cannot make forward progress) {
start AT on output port
start FT on output port
reset AT and FT
Exception Processing:
if (AT expires and FT has not) {
reset AT
respond according to congestion management policy
} else if (FT expires) {
discard frames for a period of time defined by architecture
Update link-level flow control as required
Although specific embodiments have been illustrated and described herein for purposes of description of the preferred embodiment, it will be appreciated by those of ordinary skill in the art that a wide variety of alternate and/or equivalent implementations calculated to achieve the same purposes may be substituted for the specific embodiments shown and described without departing from the scope of the present invention. Those with skill in the chemical, mechanical, electro-mechanical, electrical, and computer arts will readily appreciate that the present invention may be implemented in a very wide variety of embodiments. This application is intended to cover any combinations, adaptations or variations of the preferred embodiments discussed herein. Unless otherwise described, no single embodiment is exclusive of any other described embodiment. Therefore, it is manifestly intended that this invention be limited only by the claims and the equivalents thereof.
end stations coupled between the links, wherein types of end stations include endnodes which originate or consume frames and routing devices which route frames between the links, wherein at least one end station includes:
an aggressive timer adapted to respond to an occurrence of at least one condition of delayed frame transmission progress to provide a first frame delay indication when the at least one condition exists for a duration that exceeds a variable timing threshold, wherein the variable timing threshold is configurable based on at least one network system attribute.
2. The network system of claim 1, wherein the variable timing threshold is configurable by adjusting a timing rate of the aggressive timer.
3. The network system of claim 1, wherein the variable timing threshold is configurable by adjusting a timing interval that includes a configurable ending time relative to a starting time.
4. The network system of claim 1, wherein the aggressive timer is configurable to be disabled.
5. The network system of claim 1, wherein the at least one network system attribute includes a traffic congestion management policy.
6. The network system of claim 1, wherein the at least one network system attribute includes an end station's at least one attribute.
7. The network system of claim 1, wherein the at least one network system attribute includes at least one attribute of the network system's architecture.
8. The network system of claim 1, wherein the at least one network system attribute includes at least one frame attribute.
9. The network system of claim 8 wherein the at least one frame attribute includes the frame size.
10. The network system of claim 8 wherein the frame attribute includes the type of information carried by the frame.
11. The network system of claim 8 wherein the at least one frame attribute includes a frame header attribute.
12. The network system of claim 8 wherein the at least one frame attribute includes an assigned service level of frame transmission.
13. The network system of claim 1 wherein the at least one network system attribute includes at least one intended destination end station for the frame.
14. The network system of claim 1 wherein the at least one network system attribute includes a type of transport service used for frame transmission.
15. The network system of claim 1 wherein the at least one network system attribute includes at least one predetermined network system attribute.
16. The network system of claim 1 wherein the at least one network system attribute includes a measured network system status.
17. The network system of claim 1 wherein the at least one network system attribute includes at least one frame transmission path's bandwidth attributes.
18. The network system of claim 17 wherein the bandwidth attributes include a frame's next link hop.
19. The network system of claim 1 wherein the at least one end station includes a port, and wherein the at least one network system attribute includes at least one attribute of the port.
20. The network system of claim 19 wherein the at least one port attribute includes back pressure from the port.
21. The network system of claim 19 wherein the at least one port attribute includes a congestion status history of the port.
22. The network system of claim 19 wherein the at least one port attribute includes transmission demands of the port.
23. The network system of claim 1 wherein the at least one network system attribute includes a type of workload of at least one application running on at least one end station.
24. The network system of claim 1, wherein the variable timing threshold is configurable based on at least one previous frame delay indication.
25. The network system of claim 1, wherein the at least one end station further comprises:
a forward progress timer adapted to provide a second frame delay indication when at least one delay in frame transmission exceeds a preset timing threshold that is greater than the variable timing threshold.
26. The network system of claim 1 wherein the aggressive timer is configured to count at a microsecond granularity.
27. The network system of claim 1, further comprising at least one traffic congestion manager adapted to respond to the first frame delay indication.
28. The network system of claim 27 wherein the at least one traffic congestion manager's response includes dropping at least one frame.
29. The network system of claim 28 wherein the frame dropping is based on at least one type of randomizing means for selecting frames to be dropped.
30. The network system of claim 28 wherein the frame dropping is based on applying at least one criterion for discriminating among frames based on frame attributes.
31. The network system of claim 28 wherein the at least one traffic congestion manager's response includes dropping a set of frames, all of which have at least one common attribute.
32. The network system of claim 27 wherein the at least one traffic congestion manager's response includes truncating at least one frame.
33. The network system of claim 27 wherein the at least one traffic congestion manager's response includes communicating traffic congestion information to at least one other end station.
34. The network system of claim 33 wherein the at least one traffic congestion manager's response includes generating at least one congestion reporting frame.
35. The network system of claim 33 wherein the at least one traffic congestion manager's response includes tagging at least one frame with information relating to congestion status of the at least one frame's routing path.
36. The network system of claim 27 wherein the at least one traffic congestion manager's response includes storing a record that is representative of the first frame delay indication.
37. The network system of claim 27 wherein the at least one traffic congestion manager is dynamically configurable based on at least one network system attribute.
38. The network system of claim 37 wherein the at least one traffic congestion manager is dynamically configurable based on the first frame delay indication.
39. The network system of claim 27 wherein the at least one traffic congestion manager is included in the at least one end station.
a forward progress timer adapted to provide a first frame delay indication when at least one delay in frame transmission exceeds a first timing threshold; and
an aggressive timer adapted to respond to an occurrence of at least one condition of delayed frame transmission progress, to provide a second frame delay indication when the at least one condition exists for a duration that exceeds a second timing threshold, wherein the second timing threshold is configurable based on at least one network system attribute.
41. The network system of claim 40 wherein the first timing threshold is greater than the variable timing threshold.
42. The network system of claim 40 wherein the configurable second timing threshold is variable in duration.
43. The network system of claim 40 wherein the forward progress timer is configured to count at a first counting rate, and the aggressive timer is configured to count at a second counting rate that is a multiple of the first counting rate.
44. The network system of claim 40 wherein the aggressive timer is configured to count at a microsecond granularity.
45. The network system of claim 40, wherein the at least one network system attribute includes at least one frame attribute.
46. The network system of claim 40, wherein the at least one end station includes a port, and wherein the at least one network system attribute includes at least one attribute of the port.
47. The network system of claim 40 wherein the at least one network system attribute includes a traffic congestion management policy.
48. The network system of claim 40 wherein at least one end station is configured to respond to at least one of the first and second frame delay indications.
49. The network system of claim 48 wherein the at least one end station comprises:
a traffic congestion manager configured to respond to the second frame delay indication.
50. The network system of claim 49 wherein the traffic congestion manager is configurable based on at least one network system attribute.
51. An end station comprising:
an aggressive timer adapted to monitor network traffic, and respond to an occurrence of at least one condition of delayed frame transmission progress to provide a first frame delay indication when the at least one condition exists for a duration that exceeds a variable timing threshold, wherein the variable timing threshold is configurable based on at least one network system attribute.
52. The end station of claim 51, further comprising at least one routing device.
53. The end station of claim 52 wherein the at least one routing device includes at least one switch.
54. The end station of claim 52 wherein the at least routing device includes at least one router.
55. The end station of claim 51, further comprising at least one processor endnode.
56. The end station of claim 51, wherein the monitored network traffic is monitored within the end station.
57. The end station of claim 51, wherein the at least one network system attribute includes at least one frame attribute.
58. The end station of claim 51, further comprising a port, wherein the at least one network system attribute includes at least one attribute of the port.
59. The end station of claim 51, wherein the at least one network system attribute includes a traffic congestion management policy.
60. The end station of claim 51, wherein the end station is configured to generate an indication upon the aggressive timer's recognition of the potential condition of network bandwidth underutilization.
61. The end station of claim 51, wherein the end station is configured to respond to the aggressive timer's recognition of the potential condition of network bandwidth underutilization.
62. The end station of claim 61 further comprising:
a traffic congestion manager, wherein the traffic congestion manager is configured to render the response.
63. The end station of claim 62 wherein the traffic congestion manager is configurable based on at least one network system attribute.
64. A method of detecting potential network system bandwidth underutilization, the method comprising:
configuring a timing threshold that defines when a continuous existence of at least one condition of delayed frame transmission progress constitutes an underutilization of network system bandwidth, wherein the configuring is based on at least one network system attribute; and
monitoring transmission progress of a frame, wherein the monitoring includes:
responding to any indicated existence of the at least one condition of delayed frame transmission progress;
timing the duration of the existence of the at least one condition; and
comparing the measured duration against the configured timing threshold.
65. The method of claim 64, wherein the responding to any indicated existence of the at least one condition of delayed frame transmission progress, includes initiating the timing.
providing a frame delay indication when the measured duration exceeds the timing threshold.
managing network traffic congestion when the measured duration exceeds the timing threshold.
68. The method of claim 67 wherein the managing is based on a network traffic congestion management policy.
69. The method of claim 64 wherein the at least one network system attribute includes a traffic congestion management policy.
70. The method of claim 64 wherein the at least one network system attribute includes an end station's at least one attribute.
71. The method of claim 70 wherein the at end station's least one attribute includes at least one port attribute.
72. The method of claim 64 wherein the at least one network system attribute includes at least one frame attribute.
US10442401 1999-05-24 2003-05-21 Network congestion management using aggressive timers Abandoned US20030195983A1 (en)
US13566499 true 1999-05-24 1999-05-24
US15415099 true 1999-09-15 1999-09-15
US09578155 US7318102B1 (en) 1999-05-24 2000-05-23 Reliable datagram
US09578019 US7346699B1 (en) 1999-05-24 2000-05-24 Reliable multicast
US10442401 US20030195983A1 (en) 1999-05-24 2003-05-21 Network congestion management using aggressive timers
US09578019 Continuation-In-Part US7346699B1 (en) 1999-05-24 2000-05-24 Reliable multicast
US20030195983A1 true true US20030195983A1 (en) 2003-10-16
ID=28795179
US10442401 Abandoned US20030195983A1 (en) 1999-05-24 2003-05-21 Network congestion management using aggressive timers
US (1) US20030195983A1 (en)
US20040184481A1 (en) * 2003-03-19 2004-09-23 Sansung Electronics Co., Ltd. Medium access control protocol layer module of a mobile station in a mobile ad hoc network and method for transmitting/receiving frames of the module
US20060047867A1 (en) * 2004-08-30 2006-03-02 International Business Machines Corporation Method, system, and storage medium for providing queue pairs for I/O adapters
US7023811B2 (en) * 2001-01-17 2006-04-04 Intel Corporation Switched fabric network and method of mapping nodes using batch requests
WO2006057730A3 (en) * 2004-10-22 2007-03-08 Davide Bergamasco Network device architecture for consolidating input/output and reducing latency
US20080091868A1 (en) * 2006-10-17 2008-04-17 Shay Mizrachi Method and System for Delayed Completion Coalescing
US20080104659A1 (en) * 2006-10-27 2008-05-01 Matthew James West Prioritized real-time data transmission
US7430615B2 (en) 2004-08-30 2008-09-30 International Business Machines Corporation RDMA server (OSI) global TCE tables
US7480298B2 (en) 2004-08-30 2009-01-20 International Business Machines Corporation Lazy deregistration of user virtual machine to adapter protocol virtual offsets
US7522597B2 (en) 2004-08-30 2009-04-21 International Business Machines Corporation Interface internet protocol fragmentation of large broadcast packets in an environment with an unaccommodating maximum transfer unit
US7813369B2 (en) 2004-08-30 2010-10-12 International Business Machines Corporation Half RDMA and half FIFO operations
US20100293275A1 (en) * 2009-05-12 2010-11-18 Qualcomm, Incorporated Method and apparatus for managing congestion in a wireless system
US20110264758A1 (en) * 2004-04-21 2011-10-27 Solarflare Communications, Inc. User-level stack
US8050260B1 (en) * 2007-01-30 2011-11-01 Qlogic, Corporation Method and system for load balancing in infiniband switches and networks
US20120230212A1 (en) * 2011-03-09 2012-09-13 Kaplan Laurence S Congestion detection in a network interconnect
US20130135992A1 (en) * 2010-05-14 2013-05-30 Telefonica, S.A. Method and system for managing high capacity traffic offload in an ip network nucleus in the transport layer
US20140317165A1 (en) * 2013-04-23 2014-10-23 Cisco Technology, Inc. Direct data placement over user datagram protocol in a network environment
US9264338B1 (en) * 2013-04-08 2016-02-16 Sprint Communications Company L.P. Detecting upset conditions in application instances
US20160056886A1 (en) * 2012-02-22 2016-02-25 Nippon Telegraph And Telephone Corporation Multi-lane transmission device and multi-lane transmission method
WO2016105446A1 (en) * 2014-12-27 2016-06-30 Intel Corporation Technologies for medium grained adaptive routing in high-performance network fabrics
US9419912B2 (en) 2014-02-11 2016-08-16 International Business Machines Corporation Selective underflow protection in a network switch
US5914936A (en) * 1996-05-16 1999-06-22 Hitachi, Ltd. ATM exchange performing traffic flow control
US5995488A (en) * 1996-10-08 1999-11-30 Advanced Micro Devices, Inc. Method and apparatus for regulating data flow in networks
US6046983A (en) * 1995-08-02 2000-04-04 Nippon Telegraph And Telephone Corporation Dynamic rate control system
US6061820A (en) * 1994-12-28 2000-05-09 Kabushiki Kaisha Toshiba Scheme for error control on ATM adaptation layer in ATM networks
US6438138B1 (en) * 1997-10-01 2002-08-20 Nec Corporation Buffer controller incorporated in asynchronous transfer mode network for changing transmission cell rate depending on duration of congestion and method for controlling thereof
US8612536B2 (en) * 2004-04-21 2013-12-17 Solarflare Communications, Inc. User-level stack
US20090125604A1 (en) * 2004-08-30 2009-05-14 International Business Machines Corporation Third party, broadcast, multicast and conditional rdma operations
US8055818B2 (en) * 2004-08-30 2011-11-08 International Business Machines Corporation Low latency queue pairs for I/O adapters
US7953085B2 (en) 2004-08-30 2011-05-31 International Business Machines Corporation Third party, broadcast, multicast and conditional RDMA operations
US8761020B1 (en) * 2007-01-30 2014-06-24 Intel Corporation Method and system for load balancing in InfiniBand switches and networks
US9391899B2 (en) * 2011-03-09 2016-07-12 Cray Inc. Congestion detection in a network interconnect
US20150139024A1 (en) * 2011-03-09 2015-05-21 Cray Inc. Congestion detection in a network interconnect
US9674092B2 (en) 2011-03-09 2017-06-06 Cray Inc. Congestion abatement in a network interconnect
US9674091B2 (en) 2011-03-09 2017-06-06 Cray Inc. Congestion causation in a network interconnect
US9491101B2 (en) 2011-03-09 2016-11-08 Cray Inc. Congestion abatement in a network interconnect
US20160261339A1 (en) * 2012-02-22 2016-09-08 Nippon Telegraph And Telephone Corporation Multi-lane transmission device and multi-lane transmission method
US9307053B2 (en) * 2013-04-23 2016-04-05 Cisco Technology, Inc. Direct data placement over user datagram protocol in a network environment
US9444758B2 (en) 2014-02-11 2016-09-13 International Business Machines Corporation Selective underflow protection in a network switch
Vasudevan et al. 2009 Safe and effective fine-grained TCP retransmissions for datacenter communication
US6272522B1 (en) 2001-08-07 Computer data packet switching and load balancing system using a general-purpose multiprocessor architecture
Phanishayee et al. 2008 Measurement and Analysis of TCP Throughput Collapse in Cluster-based Storage Systems.
US7327680B1 (en) 2008-02-05 Methods and apparatus for network congestion control
US20040098473A1 (en) 2004-05-20 System, method and computer program product for monitoring and controlling network connections from a supervisory operating system
Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:KRAUSE, MICHAEL R.;REEL/FRAME:013961/0427