Abstract:
A method of initializing a Fibre Channel over Ethernet (FCoE) link between a Fibre Channel over Ethernet node (ENode) and Fibre Channel Forwarders (FCFs) in a network by discovering the presence of FCFs available to the ENode and other FCFs, assigning second MAC addresses to the ENode corresponding to each available FCF, and converting from a management and initialization process using the first MAC address to normal Fibre Channel operation using the second MAC addresses. The implementations described herein may additionally distinguish management and initialization processes from normal Fibre Channel operation using separate Ethertypes.

Description:
CROSS-REFERENCE TO RELATED APPLICATIONS 
       [0001]    The present application claims benefit of U.S. Provisional Patent Application No. 61/014,413, filed Dec. 17, 2007 and entitled “Address Assignment in Fibre Channel Over Ethernet Environments,” which is specifically incorporated by reference for all that it discloses and teaches. 
     
    
     BACKGROUND 
       [0002]    A storage area network (SAN) may be implemented as a high-speed, special purpose network that interconnects different kinds of data storage devices with associated data s on behalf of a large network of users. Typically, a storage area network includes high performance switches as part of the overall network of computing resources for an. The storage area network is usually clustered in close geographical proximity to other computing resources, such as computers, but may also extend to remote locations for and archival storage using wide area network carrier technologies. Fibre Channel networking is typically used in SANs although other communications technologies may also be employed, including Ethernet and IP-based storage networking standards (e.g., iSCSI, FCIP (Fibre Channel over IP), etc.). 
         [0003]    As used herein, the term “Fibre Channel” refers to the Fibre Channel family of standards (developed by the American National Standards Institute (ANSI)) and other related and draft standards. In general, Fibre Channel defines a transmission medium based on a high speed communications interface for the transfer of large amounts of data via connections between varieties of hardware devices. 
         [0004]    In a typical SAN, one or more Fibre Channel switches are used to communicatively connect one or more server devices with one or more data storage devices. Such switches generally support a high performance switching fabric and provide a number of communication ports for connecting to other switches, servers, storage devices, or other SAN devices. Other high performance fabrics may employ different fabric technologies, such as Infiniband. 
         [0005]    Other networking technologies, such as Ethernet, may also be employed in communicating between computing and networking devices. However, these networking technologies do not work seamlessly with high performance networks, such as a Fibre Channel network. 
       SUMMARY 
       [0006]    Implementations described and claimed herein address the foregoing problems by providing a method of initializing a FCoE link between a host or other device and Fibre Channel Forwarders (FCFs) in a network by discovering the presence of Fibre Channel Forwarders available to the host or other device, optionally assigning second MAC addresses to the host or other device corresponding to each available Fibre Channel Switch Port, and converting from a management and initialization process using the first MAC address to normal Fibre Channel operation using the second MAC addresses. The implementations described herein may additionally distinguish management and initialization processes from normal Fibre Channel operation using separate Ethertypes. 
         [0007]    Other implementations are also described and recited herein. 
     
    
     
       BRIEF DESCRIPTIONS OF THE DRAWINGS 
         [0008]      FIG. 1  illustrates an exemplary computing and storage framework including a local area network (LAN) and a storage area network (SAN). 
           [0009]      FIG. 2  illustrates an exemplary FCoE node (ENode) solicitation operation in a FCoE Initialization Protocol (FIP) Discovery phase. 
           [0010]      FIG. 3  illustrates an exemplary Fibre Channel Forwarder (FCF) solicitation operation in a FIP Discovery phase. 
           [0011]      FIG. 4  illustrates an exemplary FCF solicited advertisement operation in a FIP Discovery phase. 
           [0012]      FIG. 5  illustrates an exemplary MAC Address Assignment operation in a FIP MAC Address Assignment phase. 
       
    
    
     DETAILED DESCRIPTION 
       [0013]    Two conflicting mechanisms are considered for managing the MAC addresses of devices coupled to a high performance network. One, referred to as Server Provided MAC Addresses (SPMA), uses the MAC address offered by the host or other device as the only address which can be used to access the host or other device. The second, referred to as Fabric Provided MAC Addresses (FPMA) forces each of the available Fibre Channel Switch ports to select a unique MAC address derived from an identifier unique to a host or other device. The granted MAC address may be derived to meet the requirements of the attached host and/or Fibre Channel Switch port. This MAC address is then assigned and communicated to the host or other device. The host or other device then uses its granted address for Fibre Channel communication in violation of the Ethernet standards, while retaining its initial MAC address for further initialization and control functions. 
         [0014]    The two aforementioned methods are incompatible and non-negotiable, creating market confusion and inhibiting the adoption of either solution. The method herein disclosed overcomes this problem and allows a simple and smooth selection of appropriate MAC addresses without modifying any Fibre Channel or Ethernet standards. 
         [0015]      FIG. 1  illustrates an exemplary computing and storage framework including a local area network (LAN)  100  and a storage area network (SAN)  102 . A local area network (LAN)  100  provides communication connectivity among multiple devices, such as a workstation  116  and hosts  114 . Connectivity within the LAN  100  is provided by switches  106 ,  107 , and  108 . The LAN  100  is presumed to be the network for a relevant enterprise with a number of different segments, although any LAN configuration may be employed. 
         [0016]    A storage area network (SAN)  102  resides within the LAN  100  and provides communication connectivity, routing, and other SAN functionality among hosts  110  and storage units  112 . The SAN  102  includes a number of switches, such as switches  101 ,  103 , and  104 . Such switches  103  and  104  may be configured as a set of blade components inserted into a chassis, as rackable or stackable modules, or as other device structures. In one implementation, the chassis has a back plane or mid-plane into which the various blade components, such as switching blades and control processor blades, may be inserted. 
         [0017]    Two of the switches, i.e., switches  103  and  104 , of the SAN  102  are connected within the LAN  102  via a switch  106 . The switch  106  is also used to join other segments of the LAN  100 , as represented by the other switches  107  and  108 , which are also shown in the LAN  100 . In addition, a series of hosts  110  are connected to various switches  104  in the SAN  102 . Likewise storage units, such as described storage units  112 , are connected also to various switches  104  in the SAN  102 . 
         [0018]    Generally, a developing standard called Fibre Channel over Ethernet (FCoE) allows Fibre Channel (FC) frames to be transmitted and received over an Ethernet network. In one implementation, a standard FC frame is equipped with a specified FCoE header and embedded within an Ethernet frame for communication through the Ethernet network. When an FCoE frame is transmitted through the Ethernet network and reaches a properly equipped FC switch at the boundary of a SAN, the FC switch strips off the Ethernet and FCoE portions of the frame and forwards the embedded FC frame through the SAN. Likewise, when a standard FC frame is transmitted through the SAN and reaches a properly equipped FC switch at the boundary of the SAN and an Ethernet network, the FC switch adds an FCoE header and an Ethernet header (with appropriate synchronization fields) to the FC frame and forwards the newly-enhanced FCoE frame to the Ethernet network. 
         [0019]    The Ethernet header of the FCoE frame includes source and destination L2 (layer-2) addresses, such as MAC addresses, which the Ethernet network uses to direct the frame to its intended destination. For example, hosts and other devices on the Ethernet network can receive the FCoE frame if they are configured to receive frames having the MAC address in the destination field of the Ethernet header. Typically, each host or other device maintains a list of MAC addresses it is configured to receive. Such MAC addresses may be uni-cast addresses, multi-cast addresses, virtual addresses, etc. 
         [0020]    In addition, each host or other device also has at least one MAC address that it inserts into the source L2 address field of any frame it transmits. The source address allows a receiving device to determine the sender of a frame and, therefore, destination address to which any reply should be sent. Many hardware host bus adapters, software applications, and operating systems choose to define the MAC address that is to be used as the preferred destination address for Ethernet frames directed to the host bus adapter and as the source address for Ethernet frames transmitted from the host bus adapter. Other host bus adapters, software applications, and operating systems choose to accept a MAC address provided by the SAN switch in order to encode information otherwise provided by the SAN switch. Such encoded information may include the Fibre Channel Destination Identifier (FC_ID) of frames transmitted by the SAN switch to that host adapter. 
         [0021]    In one embodiment, a method of initializing an FCoE link between a host or other device (collectively an FCoE Node or ENode) and Fibre Channel Forwarders (FCFs) can be divided into three distinct phases: Discovery, MAC Address Assignment, and Fibre Channel Operation. The Discovery phase may include both solicitation and advertisement operations. 
         [0022]    To avoid interference with highly optimized FCoE operational paths, frames transmitted in the Discovery and MAC Address Assignment phases of the FCoE Initialization Protocol may use an ethertype separate from the FCoE Etherype to perform the FCoE Initialization Protocol. In an implementation, frames transmitted in the Discovery and MAC Address Assignment phases may be transmitted in the FIP Ethertype. 
         [0023]      FIG. 2  illustrates an exemplary FCoE node (ENode) solicitation operation in a FCoE Initialization Protocol (FIP) Discovery phase. In a communications network  200 , an Ethernet Bridge  202  couples Fibre Channel Forwarders  204  and  206  (e.g, Fibre Channel switching elements with one or more Lossless Ethernet MACs (FCF-MACs), each coupled with an FCoE Controller) on the edge of a SAN with hosts  208  and  210 . In one implementation, the Ethernet Bridge  202  is a Lossless Ethernet Bridge. In another implementation, the Ethernet Bridge  202  may include capabilities of congestion management and baby jumbo frame support. Congestion management may include one or more of the following: Pause, Per Priority Pause, and/or additional congestion management support protocol. 
         [0024]    The FCFs  204  and  206  are coupled to other switching elements within the SAN, which is capable of providing networked connectivity with other hosts and data storage devices. In the illustrated implementation, each host or ENode supports Fibre Channel over Ethernet (FCoE) and is capable of maintaining one or more virtual links, such as a virtual N_PORT (VN_PORT) to virtual F_PORT (VF_PORT) link, as designated by the FCoE Link End Point (FCoE_LEP) icons and VN_PORT icons shown at  212 . Each ENode has an ENode MAC address, which is the MAC address used by the ENode during FIP. 
         [0025]    Each FCF has at least one Lossless Ethernet MAC (FCF-MAC) that is coupled with a FCoE controller function. Each FCF-MAC supports the instantiation of virtual E_PORTs (VE_PORTs) and/or VF_PORTS, and the FCF-MACs are referred to as VE_PORT capable FCF-MACs and VF_PORT capable FCF-MACs, respectively. As such, the hosts  208  and  210  are capable of accessing the storage devices of the SAN through the bridge  202  and one or more of the FCFs  204  and  206  using an FCoE protocol. 
         [0026]    In one implementation, as an initial operation, the ENode MAC of the host  210  transmits via transmitter a multi-cast Discovery Solicitation frame to all VF_PORT capable FCF-MACs (shown in  FIG. 2  as transmissions  215 ) to request that the VF_PORT capable FCF-MACs reply with a solicited Discovery Advertisement. However, it should be understood that the Discovery Solicitation frame may also be uni-cast (addressed to a specific FCF-MAC). The Discovery Solicitation frame includes a MAC address field in the MAC address descriptor, and the MAC address field indicates the MAC address to be used for subsequent solicited Discovery Advertisements from VF_PORT capable FCF-MACs. The Discovery Solicitation frame may also include additional information, such as a name identifier identifying the ENode, and/or a descriptor indicating the maximum FCoE payload size that the ENode MAC is able to receive. 
         [0027]    Discovery Solicitation frames may be used by the ENode to identify a list of connected FCFs. Any Discovery Solicitations received by ENodes are discarded. 
         [0028]      FIG. 3  illustrates an exemplary Fibre Channel Forwarder (FCF) solicitation operation in a FIP Discovery phase. In a communications network  300 , an Ethernet Bridge  302  couples Fibre Channel Forwarders (FCFs)  304 ,  306 , and  309  on the edge of a SAN with hosts  308  and  310 . In one implementation, the Ethernet Bridge  302  may be a Lossless Ethernet Bridge. In another implementation, the Ethernet Bridge  302  may include capabilities of congestion management and baby jumbo frame support. Congestion management may include one or more of the following: Pause, Per Priority Pause, and/or additional congestion management protocol support. 
         [0029]    The FCFs  304 ,  306 , and  309  are coupled to other switching elements within the SAN, which is capable of providing networked connectivity with other hosts and data storage devices. In the illustrated implementation, each host or ENode supports Fibre Channel over Ethernet (FCoE) and is capable of maintaining one or more virtual links, such as a virtual N_PORT (VN_PORT) to virtual F_PORT (VF_PORT) link. Each ENode has an ENode MAC address, which is the MAC address used by the ENode during FIP. 
         [0030]    Each FCF has at least one Lossless Ethernet MAC (FCF-MAC) that is coupled with a FCoE controller function. Each FCF-MAC supports the instantiation of virtual E_PORTs (VE_PORTs) and/or VF_PORTS, and the FCF-MACs are referred to as VE_PORT capable FCF-MACs and VF_PORT capable FCF-MACs, respectively. As such, the hosts  308  and  310  are capable of accessing the storage devices of the SAN through the bridge  302  and one or more of the FCFs  304 ,  306 , and  309  using an FCoE protocol. 
         [0031]    In one implementation, as an initial operation, a VE_PORT capable FCF-MAC, such as the FCF-MAC of FCF  309 , may transmit via a transmitter a multi-cast Discovery Solicitation frame to all VE_PORT capable FCF-MACs (shown in  FIG. 3  as transmissions  315 ) to request that the VE_PORT capable FCF-MACs reply with a solicited Discovery Advertisement. However, it should be understood that the Discovery Solicitation frame may also be uni-cast (addressed to a specific FCF-MAC). The Discovery Solicitation frame includes a MAC address field in the MAC address descriptor, and the MAC address field indicates the MAC address to be used for subsequent solicited Discovery Advertisements from VE_PORT capable FCF-MACs. The Discovery Solicitation frame may also include additional information, such as a name identifier identifying the Switch_Name of the FCF, and/or a descriptor indicating the maximum FCoE payload size that the FCF-MAC is able to receive. 
         [0032]    In an implementation, for FCF-MACs that support FPMA, the FC-MAP field in the FC-MAP descriptor shall be set to the FC-MAP field in the FC-MAP descriptor shall be set to the FC-MAP value the FCF-MAC is using. If the FC-MAP value is not administratively configured, then the FC-MAP value shall be set to DEFAULT_FC-MAP. For FCF-MACs that only support SPMA, the FC-MAP field in the FC-MAP descriptor is reserved. 
         [0033]    After receiving a valid Discovery Solicitation originated by an FCF (i.e., the F bit is set to one), an FCF may perform verification steps. For example, the FCF may compare the name identifier field value in the Discovery Solicitation to ensure that it is different than the Switch_Name of the recipient FCF. Further, the FCF may evaluate the FC-MAP field in the Discovery Solicitation to determine if the FC-MAP field value is zero or is the same as the FC-MAP value of the recipient FCF. If any verification steps fail, the FCF will discard the Discovery Solicitation. 
         [0034]      FIG. 4  illustrates an exemplary FCF advertisement operation in a FIP Discovery phase. In a communications network  400 , an Ethernet Bridge  402  couples Fibre Channel Forwarders  404  and  406  on the edge of a SAN with hosts  408  and  410 . In one implementation, each of the FCFs  404  and  406  respond to a Discovery Solicitation from a host/ENode  410  with a solicited Discovery Advertisement (shown in  FIG. 3  as transmissions  415  and  420 ) addressed to host/ENode  410 . Each of Discovery Advertisements  415  and  420  includes an FCF-MAC address corresponding to the FCF-MAC transmitting the Discovery Advertisement in a MAC address field in a MAC address descriptor. A solicited Discovery Advertisement should be transmitted to a MAC address field value specified in a MAC address descriptor in the Discovery Solicitation received by the FCF. In some implementations, Discovery Advertisements transmitted in response to a multi-cast Discovery Solicitation may be delayed by random time intervals to reduce network congestion. 
         [0035]    In the case where an FCF receives Discovery Solicitations from the same source on multiple FCF-MACs, a separate solicited Discovery Advertisement is transmitted in response to each Discovery Solicitation received. 
         [0036]    While  FIG. 4  illustrates the transmission of uni-cast Discovery Advertisements transmitted in response to received Discovery Solicitations, it should be understood that VF_PORT capable FCF-MACs and VE_PORT capable FCF-MACs may also transmit unsolicited Discovery Advertisements at random intervals. More specifically, VF_Port capable FCF-MACs may periodically transmit unsolicited Discovery Advertisements to all ENode MACs on the network, and VE_PORT capable FCF-MACs may periodically transmit unsolicited Discovery Advertisements to all FCF-MACs on the network. 
         [0037]    Receipt of Discovery Advertisements, whether solicited or unsolicited, allows ENodes and FCF-MACs to compile and store a list of FCFs available and accessible via the network. For example, upon receipt of a Discovery Advertisement, an ENode FCoE Controller creates an entry per FCF-MAC in an internal FCF list. Each entry in the FCF list includes an indicator that the maximum FCoE size is verified, and an indicator of whether or not the FCF is able to accept additional login requests. FCoE Controllers for VE_PORT capable FCF-MACs create similar internal FCF lists including, for example, an indicator that the maximum FCoE Size is verified, and an indicator of whether or not the FCF is able to accept additional requests such as FIP exchange link parameter requests. 
         [0038]      FIG. 5  illustrates an exemplary MAC Address Assignment operation in a FIP MAC Address Assignment phase. In a communications network  500 , an Ethernet Bridge  502  couples Fibre Channel Forwarders  504  and  506  (e.g, Fibre Channel switching elements with one or more Lossless Ethernet MACs (FCF-MACs), each coupled with an FCoE Controller)on the edge of a SAN with hosts  508  and  510 . After Fibre Channel Forwarders  504  and  506  send Discovery Advertisements to the host/ENode  510 , the host/ENode  510  sends uni-cast MAC Address Request transmissions  515  and  520  to FCFs  504  and  506 , respectively. In response to the MAC Address Request transmissions, each of FCFS  504  and  506  responds with a uni-cast MAC Address Grant transmitted to the host/ENode  510 . The MAC address to be used by the ENode may be included in the address grant frame, and may be the same MAC address used by the Discovery process, or may be a different MAC address, consistent with the requirement to avoid duplicated MAC addresses. 
         [0039]    In an implementation, for example where SPMA is supported, the MAC address to be used by the ENode is included in the MAC Address Request, and the FCF grants the MAC address requested by the ENode. In another implementation, for example where FPMA is supported, the FCF grants a MAC address to the ENode that is developed by the FCF for the convenience of the internal circuitry of the FCF. Where FPMA is supported, the MAC address granted to the ENode may be based on a Fibre Channel Destination Identifier (FC_ID) that will be assigned in a subsequent login. 
         [0040]    After receiving the MAC Address Grant, ENode  510  may transmit a uni-cast confirmation to the FCF that granted the MAC address. The uni-cast confirmation indicates that the MAC Address Grant has been received and accepted the granted MAC address for use in communications with that FCF. This confirmation, however, is optional and may not be transmitted. 
         [0041]    If the FCF does not receive the uni-cast confirmation, it interprets this as an indication that the address assignment failed (e.g., the ENode did not accept the granted MAC address). If the ENode actually transmitted the uni-cast confirmation but the Fibre Channel Switch did not receive it, then the discrepancy may be reconciled during a subsequent login operation. 
         [0042]    After the ENode  510  is granted a MAC address, which may or may not be the same as the original MAC address, the new MAC address of the ENode may be used to perform fabric login (FLOGI) and all subsequent Fibre Channel operations in the normal manner according to Fibre Channel standards. More specifically, using the new MAC address, the ENode may transmit a FCoE Fabric Login Request to the FCF that granted the new MAC address. In response to the FLOGI Request, the FCF may transmit a FCoE Fabric Login Response to the ENode, granting the ENode an FC_ID for use with the newly granted MAC address. Once the ENode obtains an FC_ID, normal Fibre Channel operations can proceed in accordance with Fibre Channel standards. 
         [0043]    In another implementation, the assignment of a MAC address to the ENode and the FCoE Fabric Login process may occur in a single-frame sequence, called an FIP Fabric Login (FLOGI) Request. The FIP FLOGI Request can reduce the number of frames transmitted between the ENode and the FCF and streamlined the login and FC_ID assignment process. In an alternative implementation, the ENode may request a MAC address by including a proposed MAC address in the FCoE Fabric Login request transmitted to the FCF. This eliminates the need for the ENode to transmit separate MAC Address Requests and FCoE Fabric Login Requests to the FCF. Further, a new MAC address is granted to ENode in the same frame as the FCoE Fabric Login Response, which includes the FC_ID granted to the ENode by the FCF. Thus, fewer transmissions between the ENode and the FCF are required to allow normal Fibre Channel operations in accordance with Fibre Channel standards. 
         [0044]    In an implementation, the FIP FLOGI Request transmitted by the ENode includes an indication of the addressing mode(s) (i.e., FPMA, SPMA, or both) supported by that ENode. If an ENode only supports SPMA or supports both SPMA and FPMA, the ENode sets the MAC address field in the MAC address descriptor to the MAC address proposed in the FIP FLOGI Request for subsequent FCoE frames. If an ENode supports only FPMA, the ENode sets the MAC address field in the MAC address descriptor to the MAC Address proposed in the FIP FLOGI Request for subsequent FCoE frames, or to all zeroes to indicate that no MAC address is proposed. 
         [0045]    In an implementation, if an ENode only supports SPMA, the MAC address specified in the FIP FLOGI Request is returned in the FLOGI Response and is used as the VN_PORT MAC address for all subsequent FCoE frames. If an ENode supports only FPMA, the MAC address specified in the FIP FLOGI Response frame shall be used as the VN_Port MAC address for all subsequent FCoE frames, and the MAC address assigned will be a properly formed FPMA MAC address. In this case, the assigned MAC address should be the MAC address proposed in the FIP FLOGI Request, if the proposed MAC address is a properly formed FPMA MAC address. A properly formed FPMA MAC address is one in which the 24 most significant bits equal the Fabric&#39;s FC-MAP value and the least significant 24 bits equal the N_Port_ID assigned to the VN_PORT by the FCF, thus guaranteeing that the FPMAs are unique within the Fabric. 
         [0046]    In the case that an ENode supports both FPMA and SPMA, the assigned MAC address may be either the MAC address specified in the FIP FLOGI Request or a properly formed FPMA MAC address assigned by the FCF. In the case where both the FCF and the ENode support both SPMA and FPMA, the address assigned by the FCF may be of ether form. 
         [0047]    In an implementation FIP FLOGI Requests from ENodes are rejected by FCFs if the ENode support only SPMA and propose a MAC address that is not a uni-cast address. FIP FLOGI Requests are also rejected if the addressing mode (i.e., SPMA or FPMA) is not supported by the FCF. 
         [0048]    In the case where the FCF supports both FPMA and SPMA and the ENode supports only SPMA, the FCF can reject FIP FLOGI requests that contain a proposed MAC address in which the 24 most significant bits match the FC-MAP in use by the FCF. 
         [0049]    The embodiments of the invention described herein are implemented as logical steps in one or more computer systems. The logical operations of the present invention are implemented (1) as a sequence of processor-implemented steps executing in one or more computer systems and (2) as interconnected machine or circuit modules within one or more computer systems. The implementation is a matter of choice, dependent on the performance requirements of the computer system implementing the invention. Accordingly, the logical operations making up the embodiments of the invention described herein are referred to variously as operations, steps, objects, or modules. Furthermore, it should be understood that logical operations may be performed in any order, unless explicitly claimed otherwise or a specific order is inherently necessitated by the claim language. 
         [0050]    The above specification, examples and data provide a complete description of the structure and use of exemplary embodiments of the invention. Since many embodiments of the invention can be made without departing from the spirit and scope of the invention, the invention resides in the claims hereinafter appended. Furthermore, structural features of the different embodiments may be combined in yet another embodiment without departing from the recited claims.