Patent Document

BACKGROUND OF THE INVENTION 
     1. Field of the Invention 
     The present invention relates generally to storage area networks. 
     2. Description of the Related Art 
     Fibre Channel has been a preferred protocol for data center storage for many years and continues to be so. This is true despite some architectural problems with Fibre Channel. One architectural problem is a limited number of domains, commonly equated to switches. For one fabric Fibre Channel has a theoretical maximum of 239 domains, each capable of theoretically 256 areas, each with a theoretical limit of 255 devices. However, other aspects of the protocol put a much lower practical limit on the size of a fabric. One such aspect relates to operations that must occur when a switch is added to or removed from the fabric. The operations are so time and processor intensive that usually a fabric has many fewer domains for stability purposes. 
     This smaller practical domain limit and a practice of dedicating a domain to a switch results in a maximum fabric size much less than currently desired in modern large data centers. One solution to this problem has been the use of Fibre Channel routers. Basically routers connect two different fabrics but prevent the two fabrics from merging, as would occur under normal Fibre Channel procedures. Using routers each fabric can be kept at a reasonable size and yet the total number of devices on the overall network can reach much higher levels. 
     While routers have allowed a large increase in overall network size, because of other Fibre Channel fundamental characteristics, even a router topology becomes a limiting factor in network size. Certain Fibre Channel packets, specifically certain extended link service (ELS) requests (REQs) and responses (RSPs) contain device addresses in the payload of the packet as well as the header. One characteristic of routers is that the routers translate device addresses at each router location. This is because each node device on a fabric can only use fabric local addresses but this would result in many address conflicts if the packet is just provided unchanged to another fabric. So the router performs address translations for each packet. For headers this translation can be setup to be performed almost entirely in switch ASIC hardware but payload address translation cannot be automated in a similar manner. Thus each packet that carries addresses in the payload must be handled by a router processor using firmware. Thus, for a router, each ELS REQ and RSP packet that contains an address in the header must be trapped and handled by the router processor. This slows down operations and may lead to a performance limitation, which then turns into a network size limitation. 
     One characteristic of Fibre Channel that has led to its continued success is the reliability of the protocol. This underlying reliability is often increased by providing multiple paths for all routes. When this multipath approach is applied to routers between fabrics, it complicates the handling of the ELS REQ and RSP packets. Because an ELS RSP packet may travel a different route back to the source than was traveled by the ELS REQ packet, each router in the multiple paths must be aware of any needed translations. Therefore when a given router receives an ELS REQ packet that will have an ELS RSP packet that contains an address, information of that ELS REQ packet must be provided to all routers that might handle the ELS RSP packet, that is, all multipath routers. This requires additional packets be communicated between the routers themselves to maintain state. Then when the ELS RSP packet is received and has been modified as needed, the receiving router must inform all of the other routers that the ELS RSP packet has been processed so that the sequence can be removed from state memory. So yet another inter-router communication must occur. These inter-router communications are all handled by the router processors, so they further exacerbate performance issues of the processors, as well as slow down operations due to wait times for router responses before the actual packets can be forwarded. 
     Therefore, while routers have allowed much larger networks to be developed practically, the networks are again at the limits of growth, in part due to limitations of router processors. 
     SUMMARY OF THE INVENTION 
     In networks according to the present invention handling of ELS REQ and RSP packets that contain addresses in the payload is shifted to the edge fabric switches connected to the node devices issuing and receiving the ELS REQ packet, the ingress and egress switches. This allows the above described ELS REQ and RSP packet payload operations to be removed from the tasks handled by the router processor. As this removes a processing burden from the router processors, those router processors are free to handle other normal operations, thus allowing more processor bandwidth to be provided to those other operations, which allows further growth of the network as one limitation has been removed. By moving the operations to the ingress and egress switches, the need to replicate or provide commands between switches or routers is avoided as there are no redundant paths at that point. Further, by having the operations done at the ingress and egress switches there is also no consolidating factor of multiple flows from multiple node devices, just the flows from the directly attached node devices. This minimizes the impact on the ingress and egress switches as well. 
    
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
       The present invention has other advantages and features which will be more readily apparent from the following detailed description of the invention and the appended claims, when taken in conjunction with the accompanying drawings, in which: 
         FIG. 1  illustrates an exemplary network according to the prior art. 
         FIG. 2  illustrates an exemplary network according to the prior art. 
         FIG. 3  is a flow chart of the operations of the networks of  FIGS. 1 and 2 . 
         FIG. 4  illustrates an exemplary network according to the present invention. 
         FIG. 5  is a flow chart of the operations of the network of  FIG. 4 . 
         FIG. 6  is a block diagram of an exemplary switch or router according to the present invention. 
     
    
    
     DETAILED DESCRIPTION 
     Referring now to  FIG. 1 , a network  100  illustrating flows according to the prior art is shown. Network  100  includes a first fabric  102  and a second fabric  104 . The second first fabric  102  is formed by three switches  106 A,  106 B and  106 C. A host or server node device  108  is connected to switch  106 A. Fabric  104  is formed by three switches  112 A,  112 B and  112 C. A disk storage unit  114  is connected to switch  112 A while a tape storage unit  116  is connected to switch  112 B. Switch  106 A is also connected to a first router  110 A. Router  110 A is also connected to switch  112 A. Switch  112 C is connected to a second router  110 B which is also connected to switch  106 C. 
     By the operation of the routers  110 A and  110 B phantom devices appear in fabrics  102  and  104 . Disk storage unit  114 ′ appears to be connected to phantom domain created by the router  110 A, the phantom domain appearing as part of the first fabric  102 . To simplify  FIG. 2 , the disk storage unit  1114 ′ is shown connected to switch  106 A, as any access from the first fabric  102  must go through switch  106 A to access the disk storage unit  114 ′. A phantom tape storage unit  116 ′ appears to be connected to a phantom domain created by router  110 B, the phantom domain appearing as a part of the second fabric  104 , while a phantom host  108 ′ appears to be connected to a different phantom domain created by router  110 B, the phantom domain appearing as a part of the second fabric  104 . Again for simplicity the tape storage unit  116 ′ is shown connected to switch  106 C and host  108 ′ is shown connected to switch  112 C. 
       FIG. 1  illustrates the flow of an ELS REQ packet from host  108  to disk tape storage unit  114 . The ELS request packet is transmitted from host  108  and passes through switch  106 A enroute to router  110 A. Router  110 A traps the ELS REQ packet as modifications are necessary. After completing the modifications to the frame, the router  110 A sends a command to router  110 B so that router  110 B can place the ELS REQ packet in a context to allow trapping of the ELS RSP packet if it passes through router  110 B. After the acknowledgement for the command is received from router  110 B, router  110 A transmits the modified ELS REQ packet which is received at switch  112 A and forwarded to disk storage unit  114 . 
     Disk storage unit  114  performs the desired operation and provides an ELS RSP packet which travels through switch  112 A and switch  112 C to router  110 B. As router  110 B has formed a trap for this ELS RSP packet, the ELS RSP packet is provided to the router  110 B processor where the payload is modified. The modified packet is then provided out of the router  110 B to switch  106 C which provides the ELS RSP packet to switch  106 A which provides it to host  108 , thus completing the ELS operation. The router  110 B sends a command to router  110 A to delete the context for the ELS REQ/RSP operation as the ELS RSP packet has been received and modified. 
     As can be seen there are many operations required by the routers  110 A and  110 B which use router processor resources and delay processing of the ELS packets. For example, communication between the two routers  110 A and  110 B must occur at least to set up the context in router  110 B. The CPU-based processing is also done in the routers  110 A and  110 B and therefore as the number of ELS REQ and ELS RSP packets increases, the workload on the router  110 A,  110 B processors increases as described in the background. Ultimately this workload of the processors begins to limit the size of a network that can be handled by the routers  110 A and  110 B, thus artificially limiting the size of the network  100 . 
     It is understood that a simple network with only two fabrics, two routers and a few devices is illustrated in the Figures to simplify explanation of the prior art and operations according to the present invention. It is understood that in a conventional or actual embodiment there would be numerous hosts switches and storage units involved, enough so that throughput of the routers  110 A and  110 B would be a limiting factor in the size of the network. 
       FIG. 2  illustrates the simpler flow where the ELS REQ and ELS RSP packets do not contain a device address in the payload. The ELS REQ packet is issued from host  108  travels through switch  106 A and arrives at router  110 A. As no changes are necessary the router  110 A simply forwards the packet to switch  112 A and then to disk storage unit  114 . The ELS RSP packet is provided from the disk storage unit  114  through the switch  112 A to the switch  112 C and then to the router  110 B. As no context is set up in router  110 B, all ELS RSP packets must be trapped for handling by the processor of the router  110 B. As no changes are necessary in this scenario, the ELS RSP packet simply transfers through router  110 B to switch  106 C and then switch  106 A and finally to host  108 . By contrasting the flow of  FIG. 2  with the flow of  FIG. 1  the additional workload on the routers  110 A and  110 B can be understood. 
     The operations of  FIGS. 1 and 2  are provided in the flowchart of  FIG. 3 . In step  300  the router  110 A receives the ELS REQ packet provided from the host  108 . In step  302  the router  110 A determines if the ELS REQ packet or ELS RSP packet payload will be changed due to the presence of device addresses. This is done by trapping for particular ELS operation codes in the payload of the packet by trap logic contained in the router  110 A. If changes are necessary, the ELS REQ packet is trapped and modified by the router  110 A processor in step  304  and the relevant information is staged to router  110 B if the ELS RSP packet also requires changes. After the modification or if no changes are required in step  306 , the ELS REQ packet is forwarded by the router  110 A to the disk storage unit  114 . 
     For the ELS RSP packet, in step  310  the router  110 B receives the ELS RSP packet from switch  112 C. In step  312  the router  110 B traps the ELS RSP packet as it has been indicated based on the modification staging and context provided by the router  110 A in step  304 . If in step  314  the ELS RSP packet is a match, then in step  316  the ELS RSP packet payload is modified as necessary. After step  316  or if there was no match in step  314 , the ELS RSP packet is forwarded by router  110 B to the host  108  in step  318 . In step  320  the router  110 B provides the ELS delete message to router  110 A. 
     Operation of a preferred embodiment according to the present invention is illustrated in  FIGS. 4 and 5 .  FIG. 4  is the same network topology and components as shown in  FIG. 1  except the initial numerals are changed from a one to a four. Thus it is network  400 , fabric  402 , fabric  404  and so on. Further illustrated in  FIG. 4  are relevant portions of the header and payload of the ELS REQ and ELS RSP packets of interest. The host  408  provides an ELS REQ packet to ingress switch  406 A. Switch  406 A analyzes the ELS REQ packet to determine if changes are necessary to addresses in the payload of the ELS RSP packet. If so, a trap is set to handle the ELS RSP packet but no payload modifications are performed in switch  406 A on the ELS REQ packet. The ELS REQ packet is forwarded to the router  410 A where the router hardware automatically changes the header addresses from addresses of fabric  402 , indicated by DID 1  and SID 1 , to addresses of fabric  404 , indicated by the DID 2  and SID 2 . Therefore the packet that is transmitted from router  410 A has a header addressed DID 2  and SID 2  but the payload still contains the DID 1  information as the packet is not trapped for handling by the router  410 A. Upon receipt by the switch  412 A, the egress switch in the illustrated embodiment, the switch  412 A analyzes the packet and determines it is an ELS REQ packet with an address in the payload and therefore traps and modifies the address in the payload as indicated by the address changing to DID 2 . The modified packet is then forwarded to the targeted disk storage unit  414 . 
     Completing operation, the disk storage unit  414  provides an ELS RSP packet to switch  412 A which simply passes the ELS RSP packet through even though an address is present in the payload that must be changed. Switch  412 B passes the ELS RSP packet to router  410 B, which performs the header address translation as illustrated and provides the header translated packet to switch  406 C. Switch  406 C provides the packet to switch  406 A, the egress switch, which concludes that this is the ELS RSP packet to the previous ELS REQ packet and therefore traps the ELS RSP packet to the switch processor for modification. The switch  406 A processor modifies the ELS RSP packet payload to indicate the proper destination address, in the example DID 1 . This is done by having the switch processor review the header destination address and copy the header destination address into the payload address location. The packet is then forwarded to the host  408 . 
     As seen, there are no operations in the routers  410 A or  410 B that are performed by the router processors, only the conventional header translations which are performed by the router hardware in the preferred embodiment. This removes the processing required for the ELS REQ and ELS RSP packets by the router processors. This reduced workload for these two packet types allows the router processor bandwidth to be provided to and used by other router operations, which effectively allows the router to scale to a larger network level. As the operations of modifying the packet are performed as necessary by the ingress and egress switches, the actual modification workload is minimized and not concentrated in any particular device but only handled by the switches that are actually connected to devices that are issuing or receiving the respective ELS REQ and ELS RSP packets. 
     Also shown in  FIG. 4  for illustration are the address changes which are performed on ELS REQ and ELS RSP packets that contain two addresses in the payload. Effectively the relevant switches simply change both addresses in the payload. Reviewing the packet received by switch  412 A, it is noted that the packet contains the proper addresses in the header, DID 2  and SID 2  in the illustration, and the improper addresses, DID 1  and SID 1 , in the payload. By referencing the proper two addresses from the header, the switch  412 A simply places those values into the payload and then provides the packet to the disk storage unit  414 . 
     Operation according of  FIG. 4  and according to the present invention is illustrated in  FIG. 5 . In step  500  the ingress switch  406 A in fabric  402  receives an ELS REQ packet from the host  408 . In step  501  the switch  406 A determines that the packet is destined to the translate domain of the router  410 A. According to Fibre Channel standards, a router provides two levels of virtual domains at a connected port. The first level is a front domain and the second level is a translate domain. More on this operation and architecture can be illustrated by reviewing the FC-IFR Rev. 1.06 specification, especially Section 4.4. By determining that the destination address is the translate domain, this indicates that the packet is being addressed to a phantom device, such as phantom disk storage unit  414 ′, and therefore modifications may need to be performed. In step  502  the switch  406  determines if the ELS RSP payload will need to be changed or modified. If so, in step  504  an entry into the switch ASIC contained inside the switch  406  is made to trap the ELS RSP packet on its return. If not to the translate domain in step  501  or if no changes are needed in step  502  or after step  504 , the ELS REQ packet is forwarded to the router  410 A in step  506 . In step  508  the router  410 A forwards the ELS REQ packet to switch  412 A in fabric  404 . As switch  412 A is the egress switch for this particular packet as switch  412 A is connected to disk storage unit  414 , the ELS REQ packet is received at switch  412 A in step  510  and trapped to the switch processor. In step  512  the switch  412 A determines if the ELS REQ packet is from the translate domain provided by the router  410 A for fabric  404 . If so, in step  514  a determination is made whether the ELS REQ packet requires modification. If so, in step  516  the processor or CPU in switch  412 A modifies the payload address in the ELS REQ packet as illustrated in  FIG. 4 . If the packet is not from the translate domain in step  512  or is not required to be modified in step  514  or the edge switch has completed modification in step  516 , the request phase operations complete. 
     In step  520  the ELS RSP packet is forwarded by switches  412 A and  412 B to router  410 B. It is noted that no operations are performed in ingress switch  412 A regarding the ELS RSP packet. In step  522  the ELS RSP packet is forwarded by the router  410 B to fabric  402 , specifically switch  406 C, which then provides the ELS RSP packet to the switch  406 A, the egress switch for the ELS RSP packet. In step  524  the switch ASIC traps the ELS RSP packet based on the trap set in step  504 . In step  526  the switch processor modifies the address or addresses in the packet payload to provide the right addresses. If the ELS RSP packet is not trapped in step  524  or after step  526  the switch  406 A forwards the ELS RSP packet in step  528  to host  408  to complete the ELS operation. 
       FIG. 6  is a block diagram of an exemplary router or switch  698 . A control processor  690  is connected to a router or switch ASIC  695 . The ASIC  695  is connected to media interfaces  680  which are connected to ports  682 . Generally the control processor  690  configures the ASIC  695  and handles higher level router or switch operations, such as the name server, routing table setup, and the like. The ASIC  695  handles general high speed inline or in-band operations, such as switching, routing and frame header translation. The control processor  690  is connected to flash memory  665  or the like to hold the software and programs for the higher level router or switch operations and initialization such as performed in  FIGS. 3 and 5 ; to random access memory (RAM)  670  for working memory, such as the name server and router tables; and to an Ethernet PHY  685  and serial interface  675  for out-of-band management. 
     The ASIC  695  has four basic modules, port groups  635 , a frame data storage system  630 , a control subsystem  625  and a system interface  640 . The port groups  635  perform the lowest level of packet transmission and reception. Generally, frames are received from a media interface  680  and provided to the frame data storage system  630 . Further, frames are received from the frame data storage system  630  and provided to the media interface  680  for transmission out of port  682 . The frame data storage system  630  includes a set of transmit/receive FIFOs  632 , which interface with the port groups  635 , and a frame memory  634 , which stores the received frames and frames to be transmitted. The frame data storage system  630  provides initial portions of each frame, typically the frame header and a payload header for FCP frames, to the control subsystem  625 . The control subsystem  625  has the translate  626 , router  627 , filter  628  and queuing  629  blocks. The translate block  626  examines the frame header and performs any necessary address translations, such as those that happen in a router where packet header addresses must be changed. There can be various embodiments of the translation block  626 , with examples of translation operation provided in U.S. Pat. No. 7,752,361 and U.S. Pat. No. 7,120,728, both of which are incorporated herein by reference in their entirety. The router block  627  examines the frame header and selects the desired output port for the frame. The filter block  628  examines the frame header, and the payload header in some cases, to determine if the frame should be transmitted. The queuing block  629  schedules the frames for transmission based on various factors including quality of service, priority and the like. 
     Therefore by removing ELS REQ and ELS REP packet payload address translation duties from the routers and moving the duties to the ingress and/or egress switches, the processing demands on the router processor are significantly reduced. As the processing demands are significantly reduced, this allows increased size for the network as a whole as the router processor can do increased numbers of other router tasks. 
     The above description is illustrative and not restrictive. Many variations of the invention will become apparent to those skilled in the art upon review of this disclosure. The scope of the invention should therefore be determined not with reference to the above description, but instead with reference to the appended claims along with their full scope of equivalents.

Technology Category: 5