Abstract:
A method to dynamically allocate credits for a particular port to port link based on measured link distance during the initial interswitch link configuration process. An apparatus implementing such method or a software upgrade to retrofit existing switches.

Description:
CROSS-REFERENCE TO RELATED APPLICATIONS  
       [0001]    This application is related to and incorporates by reference, U.S. patent application Ser. No. 09/872,412, entitled “Link Trunking and Measuring Link Latency in Fibre Channel Fabric” by David C. Banks; Kreg A. Martin; Shunjia Yu; Jieming Zhu and Kevan K. Kwong, filed Jun. 01, 2001; Ser. No. 10/207,541 entitled “Credit Sharing for Fibre Channel Links with Multiple Virtual Channels” by Kreg A, Martin and David C. Banks, filed Jul. 29, 2002; Ser. No. 10/207,361 entitled “Cascade Credit Sharing for Fibre Channel Links” by Kreg A. Martin and Shahe H. Krakirian, filed Jul. 29, 2002; and Ser. No. 10/205,793 entitled “Method and Apparatus for Round trip Delay Measurement in a Bi-directional, Point-to-Point, Serial Data Channel” by Farid A. Yazdy and Kreg A. Martin, filed Jul. 26, 2002. 
     
    
     
       BACKGROUND OF THE INVENTION  
         [0002]    1. Field of the Invention  
           [0003]    This invention relates generally to network switching devices and more particularly to Fibre Channel switching devices and the dynamic credit allocation for a port based on the port-to-port link distance.  
           [0004]    2. Description of the Related Art  
           [0005]    The Fibre Channel family of standards (developed by the American National Standards Institute (ANSI)) defines a high speed communications interface for the transfer of large amounts of data between a variety of hardware systems such as personal computers, workstations, mainframes, supercomputers, storage devices and servers that have Fibre Channel interfaces. Use of Fibre Channel is proliferating in client/server applications which demand high bandwidth and low latency I/O such as mass storage, medical and scientific imaging, multimedia communication, transaction processing, distributed computing and distributed database processing applications. U.S. Pat. No. 6,160,813 to Banks et al. discloses one of the state of the art Fibre Channel switch systems, which is hereby incorporated by reference.  
           [0006]    One or more interconnected switches form a network, called a fabric, which other devices, such as mass storage devices, servers or workstations, can be connected to. Any devices connecting to a fabric can communicate with any other devices connected to the fabric. A direct connection between two devices is a link. An interface on a device for connecting another device is a port. A non-switch device connecting to a fabric is a node on the network or fabric. A port on a non-switch and non-hub device is an N-port. A port on a switch may be an E-port, for connection to another switch port, an F-port, for connection to an N-port, an FL port for connection to an FC-AL loop or any combination of the above. A link between two switches is an inter-switch link (ISL).  
           [0007]    Each port has a transmitter and a receiver. The transmitter sends out information and the receiver receives incoming information. There are buffer memories associated with each port, either the transmitter or the receiver, to temporarily store the information in transit, before the information is confirmed to be transmitted towards its destination by a switch, or to be stored or used by a device at its destination. The buffer memory can be in the actual port or, preferably, may be centralized, as shown in U.S. Pat. No. 6,160,813. The buffer memory is broken down into units. One unit of buffer memory, which can hold one frame, is represented by one buffer-to-buffer credit or one credit. A frame is a unit of information transmitted, which comprises a header portion and a payload portion. The header portion identifies the frame, including a Source Identification (SID) and a Destination Identification (DID). The payload portion contains the data being transmitted. A frame may be 2112 data bytes long, which, plus header, CRC, EOF totals 2148 bytes.  
           [0008]    In the prior art, a receiver on a port is allocated a fixed amount of buffer space to temporarily store received frames, represented by a fixed number of buffer-to-buffer credits. The receiver controls the allocation of the buffer space. At the initial configuration when two switches connect, the receivers advertise to the transmitters the amount of buffer space represented by the number of credits available. The transmitters initialize their credit counters to the number of credits advertised by the receivers. Both the transmitting port and receiving port keep track of the use of the buffer space using the number of credits and credit counters. Each time a frame is received by the receiver, the frame is stored in a buffer space and the number of credits residing in the receiver is increased by one. The transmitting port keeps track of this by reducing its transmitter credit counter, which indicates how many more frames can be sent, while the receiver increments its receiver credit counter, which indicates how many frames are stored in the buffer space. Once the frame is confirmed to have been retransmitted by a transmitter on the receiving switch, or used by a device, then the buffer space is free to be used to store a new frame. At that time, a credit is returned by a transmitter on the receiving port to a receiver on the transmitting port and the receiver credit counter in the receiving port is decreased by one. When the transmitting port receives the credit, the transmitter credit counter in the transmitting port is increased by one.  
           [0009]    Even though frames travel through the fiber optics at the speed of light, it still takes time for frames to move from one device to another. It also takes time for a device to receive a frame; process it or retransmit it; and then return a credit, i.e. a confirmation of receipt, back to the transmitting port. It takes some more time for the credit traveling through the optical fiber to reach the transmitting port. During the turn-around time between when the transmitting port sends out a frame and the transmitting port receives a credit, the transmitting port may have sent out a number of frames at its transmitting speed if the transmitting port has available credits. When the transmitting port has at least a minimum number of credits to allow the transmitting port to continue transmitting until it receives credits back from the receiving port, the effective frame transmission rate is the highest. If the transmitting port does not have that minimum number of credits, then it has to temporarily stop sending frames when all the credits are used and wait until the credits return. Due to this stoppage, the effective frame transmission rate may be substantially lower than the actual transmission rate. That minimum number of credits depends on the turn-around time and the frame transmitting speed. The longer the transmission line, or the faster the transmitting speed, the more frames that may be in transit. At a fixed transmitter speed, the more credits a port can have, the longer the transmission distance can be while the link still maintains the full effective transmitter speed.  
           [0010]    In the prior art, the number of credits allocated to a port is fixed, but the distance between ports may be different. For long distant links, there may not be enough credits to sustain the full speed of the transmitters. For shorter distance links, there may be more credits than necessary such that some of the buffer space or credits are wasted. Even for Fibre Channel switches where the buffer memory are centrally allocated and controlled, the amount of credits or buffer space allocated to each port is still fixed. To alleviate such problem, in some other prior art Fibre Channel switches, an installer of the physical port to port link may manually configure the buffer space dedicated for a particular port depending on the distance between the port to port link. There may be discreet distant levels that an installer can select, such as 5, 10, 50, 100 kilometers. Still many times these levels are too far higher than the actual link distance deployed by the Fibre Channel network installer. For a given configuration, a particular distance level setting may be wasteful because it will over-commit the buffer credits based on next higher level. With the advancing in the speed of Fibre Channel switches and the distance of the Fibre Channel links, the demand for buffer space or credits is increasing. The inflexibility of buffer space or credits allocation becomes increasingly costly.  
           [0011]    Therefore, it is desirable to have a method to match the credit demand for a port connecting to a port to port link to the available credits on a Fibre Channel switch. It is also desirable to have devices to implement such a method.  
         BRIEF SUMMARY OF THE INVENTION  
         [0012]    The present invention utilizes a dynamic link distance configuration to allocate credits for a particular port to port link. During the initial interswitch link configuration process, the actual distance between the two connecting port is measured and the demand for credits is calculated. Such amount of credits is allocated to this link. The present invention can be implemented in the hardware of a Fibre Channel switch itself or as software upgrades to retrofit existing Fibre Channel switches, assuming the switch includes the link timer hardware. 
       
    
    
     BRIEF DESCRIPTION OF THE DRAWINGS  
       [0013]    A better understanding of the invention can be had when the following detailed description of the preferred embodiments is considered in conjunction with the following drawings, in which:  
         [0014]    [0014]FIG. 1 is a block diagram of a typical fabric with connecting devices.  
         [0015]    [0015]FIG. 2 is a block diagram of a two switch, four device network showing the interconnections, where the buffer memory for each port is centrally allocated and controlled on the switches.  
         [0016]    [0016]FIG. 3 is a block diagram of a two switch, four device network showing the interconnections similar to the ones shown in FIG. 2, except that the buffer memory allocated to each port is dynamically allocated during the initial port to port configuration.  
         [0017]    [0017]FIG. 4 shows a portion of the port to port initial configuration, where the distance between the ports is measured and the amount of necessary credits is determined. 
     
    
     DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENT  
       [0018]    [0018]FIG. 1 depicts a typical Storage Area Network (SAN)  20  utilizing a Fibre Channel network. A fabric  32  may comprise one or more switches  30 . Three switches  30  are shown. Many devices or nodes, such as a general storage device  24 , a server  26 , database storage devices  28  and a loop  22  (itself comprised of devices, not shown) are connected to the fabric  32 . Any devices on the fabric  32  can communicate to any other devices on the fabric  32 .  
         [0019]    [0019]FIG. 2 is a block diagram showing more details of several particular interconnections in a portion of a SAN  40 . Two switches  72  and  74  are shown, together with two servers  52 ,  54  and two storage devices  62 ,  64 . The N-ports  102 ,  104 ,  106  and  108  on servers  52  and  54 , and storage devices  62  and  64 , and their corresponding F-ports  66 ,  68 ,  76  and  78  on the switches  72  and  74  are linked by links  82 ,  84 ,  86  and  88 . The two switches  72  and  74  have one E-port each,  44  and  46 , which are connected through ISL  90 . Each switch may also have a central memory  56  and  58  in switches  72  and  74 , for buffering information in transit and a control module (not shown) for managing the information flow, such as directing flows of information and managing the buffer space. Each port has a transmitter (T) and a receiver (R). The available buffer spaces in the central memory are allocated among the ports on the switch.  
         [0020]    The available buffer space for each port can reside on the individual port or can reside centrally in the switch. Either way, the buffer space allocated to each port on the switch is fixed at a single amount by the switch manufacturer or at very few discreet levels that can be selected by the installer. In FIG. 2, each port  44 ,  66 ,  45 , and  68  on switch  72  are allocated one quarter of the central memory  56 . Buffer memory  144 ,  166 ,  145  and  168  are dedicated to port  44 ,  66 ,  45 , and  68 . Similarly, a quarter of the central buffer space  58 , i.e.,  147 ,  176 ,  146  and  178  are allocated to port  47 ,  76 ,  46 , and  78  on switch  74 . Buffer space  145  and  147  which are allocated to port  45  and  47  are wasted because neither of those two ports is in use in this network.  
         [0021]    [0021]FIG. 3 shows the same network configuration as in FIG. 2 except the buffer space allocations for the ports have changed. In this embodiment of the present invention, the allocation of buffer space to the ports is allocated dynamically based on the actual distance between the ports. The port  45  on switch  72  and port  47  on switch  74  are not in use, therefore, those two ports are not allocated any buffer space. The link between ports  44  and  46  is longer and also the demand for credits is more between those two ports, therefore the buffer space allocated to these two ports on each switch is bigger. The buffer spaces  244 ,  266 ,  245  and  268  allocated to ports  44 ,  66 ,  45 , and  68  on switch  72  are different and are based on the needs for each port. The buffer space  245  allocated to port  45  is at its minimum. Similarly, buffer spaces  246 ,  276 ,  247 , and  278  allocated to ports  46 ,  76 ,  47 , and  78  on switch  74  are of various sizes, and buffer space  245  is at its minimum. The actual determination of the buffer space or credit for each port will be described later.  
         [0022]    [0022]FIG. 4 shows a portion of the initial configuration that measures the distance between the two ports and determines the amount of credits or buffer space needed for the link. The direction from top to bottom on FIG. 4 is the direction of time. As shown in FIG. 4, there may be other processes before and after the measurement of distance and the determination of the credit need during the initial configuration process. When port  44  and port  46  physically connect, the initial configuration starts. After the initial device identification and some other configuration, the measurement of the distance between the two ports and buffer space determination can begin. Each port on any devices on a Fibre Channel network has its own unique identification. In this case, the world-wide number (WWN) may be used as such device identification. At the initial device identification, WWN of each port is exchanged.  
         [0023]    Continue referring FIG. 4, at time point  302 , switch  72  uses port  44  to send an ELP or Extended Link Parameters  1 LS packet to port  46  of switch  74 . At the earlier or later time  404 , switch  74  may use port  46  to also send an ELP packet to port  44 . This packet is to determine which port has a higher WWN, and may be used to determine which port will act as the initiator port or target port. An initiator port is the port that measures and determines the distance and buffer space for the link. A target port returns signals coming from the initiator port, but itself does not determine the link distance or the credit demand. It is immaterial which port acts as the initiator port or the target port, so long the two ports agree to the roles they play in the distance measurement. In the preferred embodiment, the ports use WWN to determine which one is the initiator port. In the preferred embodiment, the initiator port is the port having the higher WWN during the port initialization and the target port is the port with the lower WWN during the port initialization. In the example shown in FIG. 4, port  44  has a higher WWN than port  46 , so port  44  is the initiator port and port  46  is the target port in this connection.  
         [0024]    The next several exchanges of signals confirm that port  44  has a higher WWN then port  46 . At time  304 , port  44  receives the ELP packet from port  46  and at time  306  switch  72  sends out an ELPRJT or ELP Reject packet back to port  46  which receives the ELPRJT at time  406 , meaning “port 46 has a lower WWN than port 44.” Similarly, port  46  at time  402  receives the ELP packet from port  44  and switch  74  responds at time  408  an ELPACC or ELP Accept packet back to port  44 , which receives the ELPACC packet at time  308 , meaning “port 44 has a higher WWN than port 46.” 
         [0025]    A port on a switch which supports dynamically allocating the amount of credits for the port is referred to as supporting the LD mode of operation. For a port to port to operate in the LD mode, both ports on the link must support the LD mode, i.e. participate in the distance measurement and credits determination. It is not essential to make the inquiry, but the inquiry makes sure that the time/distance measurement can be used. So one of the ports inquires of the other port and confirms that both ports support the LD mode and both are using the same protocol in carrying out the distance measurement and credit determination. This is illustrated in the next few exchanges of signals in FIG. 4. Switch  72  uses port  44  at time  310  to send an ELP packet to port  46  to inquire of port  46  whether port  46  would support the LD mode, which is the dynamic long distance configuration mode. At time  410 , port  46  receives the ELP packet and at time  412  switch  74  responds with an ELPACC packet back to port  44 , which port  44  receives at time  312 . The ELPACC packet confirms that port  46  supports the LD mode. In the preferred embodiment, as indicated earlier, WWN is used to determine the initiator/target ports and that the port having the higher WWN is the initiator port. It is obvious that any identifiers that may be used in place of WWN and any unique sequence, such as numerical or alphabetical may be used to determine which port is the initiator port and which port is the target port. Once ELP and ELPACC packets are exchanged, both ports know that they will follow the same protocol, i.e. LD mode, to measure the distance and each one knows the exact action it will take under the protocol.  
         [0026]    After port  46  responds with the ELPACC packet, it is ready to return the next packet from port  44 , which will be a Mark primitive, a unique packet used to measure time elapsed in the link. Return here means that port  46  will respond to a packet coming from port  44  immediately after port  46  receives the signal without any processing in the port  46 . The Mark primitive is essentially a light pulse and port  46  is acting as a mirror. The port  44  emits a light pulse (Mark primitive). The light pulse (Mark primitive) is returned or reflected by port  46  and received by port  44 . Once port  44  receives the ELPACC packet at time  312 , the switch  72  knows that port  46  supports the LD mode and it is ready to do the distance measurement. The above preparatory steps are useful and are implemented in some embodiments of the current invention, but they are not essential.  
         [0027]    In the following steps, time/distance is measured. At time  314 , switch  72  uses port  44  to send out the Mark primitive to port  46  and at the same time starts a timer to measure the time elapsed between now and when the Mark primitive return packet comes back from port  46 . Port  46  will receive the Mark primitive at time  414  and immediately return it back to port  44  (i.e. reflect the light pulse back). The port  44  receives the Mark primitive at time  316  and stops the timer immediately. The time difference between time  316  and  314  is the time that the Mark primitive takes to travel a round trip between the two ports  44  and  46 . The packet is traveling at a substantially known portion of the speed of light in the link so the distance can be calculated by multiplying the time period between time  316  and  314  by the speed of light in the fabric and other values. By time  320 , the distance is determined and the number of buffer credits can be determined by a formula depending on the distance.  
         [0028]    The link distance can be calculated as follows: 
         DISTANCE=(LRT_LATENCY*3)/(20 *RI )  Eq.(1) 
         [0029]    Where DISTANCE is the link distance in km;  
         [0030]    LRT_LATENCY is the time lapse between time 316 and 314, in microseconds  
         [0031]    RI is the refractive index of glass (worst case)  
         [0032]    For a given link distance, buffer credits required are about 1 per km for 2 Gbit/sec links and 0.5 per km for 1 Gbit/sec links, approximately based on the equation below: 
         Number of credits=((20 *RI *DISTANCE)/3)*(100/MAX_FRAME_SIZE)*LINE_RATE+2  Eq. (2) 
         [0033]    Where MAX_FRAME_SIZE is the max frame size in FC, usually 2148 bytes  
         [0034]    LINE_RATE is the raw data rate of link, in Gb/s.  
         [0035]    The extra 2 credits in Eq.(2) are the usual amount of credit allowance for the latency within a switch for processing a signal/frame and/or other delays. The latency within a switch is the time between a port receiving a frame/signal, processing it, and responding or retransmitting the frame/signal. This latency within a switch is switch dependent and could be different for different switches.  
         [0036]    When distance information is not needed, Eq.(1) and Eq.(2) can be combined to get the direct relationship between time latency and the desired amount of credits: 
         Number of credits=(LRT_LATENCY*LINE_RATE*100)/MAX_FRAME_SIZE+2  Eq.(3) 
         [0037]    LRT_LATENCY is the time lapse between timer stops and starts, in microsecond  
         [0038]    LINE_RATE is the raw data rate of the link in Gb/sec,  
         [0039]    MAX_FRAME_SIZE is the maximum size of a frame transmitted by the ports, usually 2148 bytes.  
         [0040]    In one of the embodiments, the timer is a counter, counting at a rate of about 106.25 MHz. The maximum value of the counter is 524287 (0x 7ffff). The time-out is triggered when the timer/counter exceeds 524287, or about 5 ms. When a time-out occurs, a message is generated indicating that the current measurement has failed. Generally another attempt is made under such conditions until a pre-determined number of attempts have been made, at which time a message is generated indicating failure of the measurement process. Table 1 shows the typical link latency and corresponding distance and buffer credit allocation.  
                                             TABLE 1                       Link round trip       Buffer credits   Buffer credits       latency   Link distance   (# of frames) 2G   (# of       (micro second)   (km)   link   frames) 1G link                                1000   100   100   50       800   80   80   40       600   60   60   30       400   40   40   20       200   20   20   10       100   10   10   5                  
 
         [0041]    The number of credits according to Eq.(2) is the desired amount of credits for a port at distance. In the case that there are not enough credits on the switch to satisfy the demands from all ports, then only a portion of the demand for some ports can be satisfied. There are various ways to pick and choose, i.e. priority schemes. One priority scheme can be the first serve scheme: the available credits are used to satisfy the first connected port. The left over credits will be used to satisfy the demand of next connected port, and so on until all the credits are allocated. Then the remaining ports on the switch are no longer useable, i.e. cannot be used to connect to other ports. Another priority scheme can be an even distribution scheme: when the total demand for credits from all connected ports is more than the total available credits, then the same percentage of demanded credits is allocated (satisfied) for all ports, so that the total allocated credits is equal to the total available credits. This way, all the ports on a switch are useable, albeit at a lower effective speed due to the lack of credits. A formula for performing this even distribution scheme is: 
           Nai=Ni *(Total number of credits on switch)/sum (Number of credits of all ports), 
         [0042]    where Nai is the amount of credits reallocated to port i,  
         [0043]    Ni is the amount of credits allocated to port i by the equation above, and  
         [0044]    Sum (number of credits of all ports) is the sum of credits allocated to all ports as in the equation above.  
         [0045]    After the switch  72  has determined the number of credits needed for the link  90 , the switch  72  uses port  44  to send an ECP or Extended Credit Parameters ILS packet to port  46  at time  318 . This is a new ILS, which is preferred over using an ELP to allow simplified ELP processing, but an ELP can be used if desired. The ECP packet contains the needed credit information so that switch  74  can set up port  46  like switch  72  will set up port  44 . The switch  74  replies with an ECPACC or ECP Accept packet at time  416  to indicate that the credit information will be used to set up port  46 . When switch  72  receives the ECPACC packet at time  320  at port  44 , the credit information is applied to port  44 . If the switch  74  replies with an ECPRJT or ECP Reject packet, the credit information is not applied to either port  44  or port  46 , but the credit allocation is determined by other means.  
         [0046]    After the ECP packet handshake, the remaining configuration operations occur.  
         [0047]    The present invention may be implemented in many ways. For example, the present invention may be implemented in a software format, as a machine readable, machine executable program. The software program executing the present invention can be loaded into a processor or control module on a switch, or a buffer credit management module on a switch, during a power up initialization or a later set-up. This presumes that the hardware portions of the buffer credit logic are sufficiently programmable or are made sufficiently programmable to handle the changing credit allocations.  
         [0048]    The present invention may also be implemented in hardware format, as a new switch having the new credit sharing scheme built into the hardware.  
         [0049]    While illustrative embodiments of the invention have been illustrated and described, it will be appreciated that various changes can be made therein without departing from the spirit and scope of the invention.  
         [0050]    Titles and subtitles used in the text are intended only as focal points and an organization tool. These titles are not intended to specifically describe the applicable discussion or imply any limitation of that discussion.