Patent Publication Number: US-9838315-B2

Title: Stretched subnet routing

Description:
FIELD OF THE INVENTION 
     The present invention generally relates to routing to a remote host in an exterior network fabric. 
     BACKGROUND OF THE INVENTION 
     Interconnected data centers, such as those implemented using Data Center Interconnect (DCI) from Cisco Systems, Inc., are typically configured with a distributed anycast gateway and an enhanced forwarding mode that can be enabled for any subnet. Such a configuration facilitates routing both intra and inter-subnet traffic in generally the same consistent manner to optimize forwarding within a network fabric. The configuration also provides localization of the broadcast domain and prevents flooding within the fabric. 
     Multi-Protocol Border Gateway Protocol (MP-BGP) is typically used as the control protocol of choice to distribute host addresses within a given data center fabric. Consequently, traffic can be optimally forwarded to the appropriate top of rack (ToR) switch below which the destination host resides. The host route distribution also restricts a ToR switch or “leaf” to proxying for only “known” remote destinations. 
    
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
       The present invention will be understood and appreciated more fully from the following detailed description, taken in conjunction with the drawings in which: 
         FIG. 1  is a simplified pictorial illustration of two exemplary data centers, constructed and operative in accordance with embodiments disclosed herein, and connected via an inter-DC (data center) core; 
         FIG. 2  is a schematic illustration of an exemplary ToR switch from the data centers of  FIG. 1 ; 
         FIG. 3  is a schematic illustration of an exemplary border leaf from the data centers of  FIG. 1 ; 
         FIG. 4  is an exemplary process performed by the border leaf of  FIG. 3 ; and 
         FIG. 5  is an exemplary process performed by the ToR switch of  FIG. 2 . 
     
    
    
     DESCRIPTION OF EXAMPLE EMBODIMENTS 
     Overview 
     A method for improving routing for a stretched subnet includes receiving a first communication on a border leaf of the stretched subnet, where the border leaf is a top of rack (ToR) switch configured to facilitate connectivity between an internal data center fabric and at least one external site associated with the stretched subnet, based on routing information received with the received communication, identifying a source address for the received communication as either from within the internal data center fabric or from the at least one external site, and if the source address is from the external site, storing an abbreviated route based on the source address in at least one hardware table, where the abbreviated route is a route to the at least one external site, and upon subsequent receipt of a second communication to be forwarded to the source address, forwarding the second communication in accordance with the abbreviated route. 
     A method for improving routing for a stretched subnet includes on a ToR switch in an internal data center fabric of the stretched subnet, receiving routing information for a destination device, where the routing information comprises a destination address and an internal/external indication for the destination device, where the internal/external indication indicates whether the destination device is in the internal data center fabric or in an external data center fabric of the stretched subnet, and if the destination device is in the external data center fabric, storing an abbreviated route based on the destination address in at least one hardware table, where the abbreviated route is a route to the external data center fabric, and upon receipt of a communication to be forwarded to the destination address, forwarding the communication in accordance with the abbreviated route. 
     A method for increasing the number of destination devices in a stretched subnet includes in at least one hardware table on a ToR switch in a network fabric in the stretched subnet, defining a single abbreviated address for at least one external site in the stretched subnet, and routing all communications to each of the destination devices in the at least one external site according to the single abbreviated address. 
     DETAILED DESCRIPTION OF EXAMPLE EMBODIMENTS 
     Interconnected data centers provide flexibility in the form of stretched virtual routing and forwarding instances (vrfs), virtual fabrics and/or stretched subnets/vlans across multiple data center sites. 
     It is not uncommon for a data center customer or tenant to have multiple vrfs. These vrfs may have workloads that reside in different data center sites, thereby resulting in a layer-2 broadcast domain with a stretched subnet/vlan/segment that spans multiple data center sites. Existing enhanced forwarding semantics do address this issue, and provide host-based forwarding for any traffic (within or across subnets) irrespective of whether a destination of the traffic lies within the originating data center or in a second, target data center site. For example, Exterior Border Gateway Protocol (eBGP) running on the “border-leafs” (i.e., ToRs that connect with entities outside of the source fabric) is typically used to advertise host routes between different data center sites. For stretched subnets, the remote host routes are distributed from eBGP into iBGP (Interior Border Gateway Protocol), thereby resulting in the remote host routes being advertised throughout the fabric to all ToRs/leafs. 
     Typically, the gateway on the ToR switch is implemented in the form of an integrated-routing and bridging (IRB) interface, similar to a switch virtual interface (SVI). This implementation results in the installation of a subnet route with a glean adjacency in the FIB (Forwarding Information Base) LPM (longest prefix match) hardware table on the internal ToRs. With stretched subnets spanning multiple data centers, the host routes (with masks of /32 or /128 for IPv4 and IPv6, respectively) are also installed in a FIB hardware table, such as the HRT (host routing table) table, to ensure that traffic is destined to the right ToR switch below which the destination resides. Therefore, if a subnet is stretched across multiple data center sites, there may be a de facto requirement that every known host route from all of the involved data center fabrics is installed in the FIB hardware in each site. 
     Accordingly, in order to provide interconnectivity between multiple data centers, the number of entries in the FIB tables of each of the participating ToRs is effectively equivalent to the total number of known host destinations in the stretched subnet. It will therefore be appreciated that the maximum number of host destinations that may be supported by a stretched subnet is limited by the number of available entries in the participating ToR switches. For example, the FIB on a Cisco Nexus 6000 switch, has approximately 96,000 usable entries. Accordingly, a stretch subnet using a Nexus 6000 may not support more than a total of 96,000 host destinations across the interconnected data centers. It will be appreciated by one of ordinary skill that the embodiments described herein may support any other suitable ToR switch as well. It will similarly be appreciated by one of ordinary skill in the art that data centers are typically equipped with relatively inexpensive ToR switches in order to reduce costs. 
     In accordance with embodiments described herein, a stretch subnet may be configured to support routing for a total number of host destinations in excess of the number of entries in the FIB hardware tables. Reference is now made to  FIG. 1  which illustrates an exemplary stretched subnet  100 , constructed and operative in accordance with embodiments described herein to more optimally utilize the FIB resources on the internal ToR switches/leafs for cross-fabric stretched subnets while still providing optimized traffic forwarding for both within and cross fabric traffic. 
     Stretched subnet  100  comprises data center fabrics  10 A and  10 B, connected via inter data center core  60 . Data center fabrics  10  may be implemented, for example, using Layer-2 DCI. Common technologies used to implement Layer-2 DCI are VPLS, OTV, Layer-2 LISP, etc. Inter data center core  60  may be implemented, for example, using Layer-3 DCI. Common technologies used for Layer-3 DCI are MPLS, Layer-3 LISP, etc. 
     Each data center fabric  10  comprises a multiplicity of ToR switches  30 , each comprising a ToR gateway  35  to communicate with other ToR switches  30  via route reflectors  20 . At least one ToR switch in each data center fabric  10  is configured as a border leaf  40  in communication with inter data center core  60  via edge router  50 . Accordingly, a host in data center fabric  10 A may communicate with a host in data center fabric  10 B by first forwarding a communication through its associated ToR switch  30  via ToR gateway  30  and a route reflector  20  to border leaf  40 A. The communication is then forwarded through inter data center core  60  by way of edge routers  50  to border leaf  40 B in data center fabric  10 B. The communication is then forwarded in similar manner to the destination host behind an associated ToR switch. It will be appreciated that border-leafs  40  and edge routers  50  may be combined into single integrated physical components such as, for example, a Cisco Nexus 7000 switch; they are depicted as separate entities in  FIG. 1  to clarify the different functionalities that each may provide. 
     It will be appreciated by a person of ordinary skill in the art, that as per the configuration of stretched subnet  100 , all communication from anywhere in data center  10 A to anywhere in data center fabric  10 B is routed through border leaf  40 B. Therefore, in accordance with embodiments described herein, the original routing from the source fabric, i.e., data center fabric  10 A, may be reduced to the route to the entry point to the target fabric, border leaf  40 B in data center fabric  10 B. Once the communication is received by border leaf  40 B it may be routed to its intended destination. Accordingly, all of the destination hosts in data center fabric  10 A may be represented in the FIB hardware tables in ToRs  30  in data center fabric  10 B as a single entry detailing the route to data center fabric  10 B. 
     Reference is now made also to  FIGS. 2 and 3  which respectively illustrate an exemplary ToR switch  30  and an exemplary border leaf  40 , constructed and operative in accordance with embodiments described herein. ToR  30  comprises processor  31 , FIB  32 , routing module  34 , gateway  35  and routing information base (RIB)  36 . Routing module  34  is an application implemented in either software, hardware, or a combination thereof. Processor  31  is operative to at least execute routing module  34  to at least update and maintain FIB  32  and RIB  36 . FIB  32  is a hardware construct that, as described hereinabove, is used for routing to destinations both internal and external to a data center fabric  10 . FIB  32  may comprise a multiplicity of FIB tables  33  to be used when routing communications in the source data center fabric  10 . For example, FIB table  33 A may be the longest prefix match (LPM) routing table; FIB table  33 B may be the host routing table (HRT). RIB  36  is a software version of FIB  32  that may therefore not suffer from the same size limitations as FIB  32 . Gateway  35  is operative to provide communications with other devices such as route reflectors  20  and/or local hosts under ToR switch  30 . 
     Border leaf  40  comprises processor  41 , FIB  42 , route collection module  44 , gateway  45 , and RIB  46 . Accordingly, processor  31  FIB  32 , gateway  35  and routing information base (RIB)  36  may provide generally similar functionality as processor  41  FIB  42 , gateway  45  and RIB  46 , respectively. However, as will be described hereinbelow, routing module  34  and route collection module  44  may be configured to deliver slightly different functionality. 
     Reference is now made to  FIG. 4  which illustrates a FIB/RIB update process  200 , constructed and operative in accordance with embodiments described herein. Process  200  is executed by route collection module  44  on border leaf  40 . Route collection module  44  may receive (step  210 ) new routing information, i.e., a route which is currently not represented in FIB  42  and/or RIB  46 . The routing information may accompany a communication that is received by border leaf  40  via gateway  45 . It will be appreciated that the new routing information may represent either a source device for whose IP address there is no entry in FIB  42  and/or RIB  46 , or a new IP address associated with a previously “known” device with new routing information, i.e., the source device has moved and/or there have been changes in the intervening route. 
     Based on the received routing information, route collection module  44  may determine whether or not the associated device is from an external data center fabric  10  (step  220 ). If so, then route collection module  44  may define the “site of origin” (SOO) as “external”. Otherwise (step  230 ) route collection module  44  may define the “site of origin” (SOO) as “internal”. Route collection module  44  then updates FIB  42  and RIB  46  with the new routing information and the SOO as per the IP address for the source device. It will be appreciated by one of ordinary skill in the art that the use of the SOO to indicate internal/external status may be exemplary; the embodiments described herein may support other functionality for indicating internal/external destinations. For example, an AS PATH tag may also be used in a similar manner. 
     Route collection module  44  then propagates (step  250 ) the updates from step  240  via gateway  45  to internal ToR switches  30 . i.e., ToR switch  30  in the same data center fabric  10 . 
     Reference is now made also to  FIG. 5  which illustrates a FIB/RIB update process  300 , constructed and operative in accordance with embodiments described herein. Process  300  is performed by routing module  34  on ToR switch  30 . Routing module  34  may receive (step  310 ) new routing information, i.e., a route which is currently not represented in FIB  32  and/or RIB  36 . The new routing information may, for example, be received as a product of propagation from border leaf  40  as per step  250  in process  200 . 
     Routing module  34  may check the designation for SOO (step  320 ). If the SOO is set as “Internal”, routing module  34  updates FIB tables  33  ( FIG. 2 ) with the full route to be associated with the received IP address. Otherwise, routing module  34  may update FIB tables  33  with an abbreviated route to border leaf  40  to be associated with an identifier for the external SOO. Routing module  34  then updates RIB  46  with the new routing information as per the IP address for the source device. It will be appreciated by one of ordinary skill in the art, that routing module  34  may be configured to use other designations for step  320 . For example, AS PATH may also be used in a similar manner. 
     It will be appreciated by a person of ordinary skill in the art, that after updating FIB tables  33  as per process  300 , the use of entries in FIB tables  33  may be optimized such that, instead of a single entry for destination host, a single entry may be used for all external destination fabrics. For example, as will be appreciated by a person of ordinary skill in the art, a given border leaf  40 , such as border leaf  40 A in  FIG. 1 , may connect a network fabric, such as network fabric  10 A with multiple network fabrics  10  via edge routers  50  and inter-DC core  60 . Accordingly, if, per the limitation of 96,000 entries as discussed hereinabove, there had previously been a limit of 96,000 possible destination hosts, the embodiments described herein may support a limit of 96,000 possible destination hosts in a virtually unlimited number of fabrics. 
     It will similarly be appreciated that with the semantics of enhanced forwarding, as described hereinabove, all inter and intra-subnet traffic may be handled in generally the same manner via routing. The embodiments described herein therefore generally support the following rules: 
     A subnet glean entry for enhanced forwarding enabled subnets may not need to be installed on the internal leafs. There may be no need for Address Resolution Protocol (ARP) generation for unknown hosts with enhanced forwarding. The vrf default route on all the internal leafs points to one or more border leafs that are the transit to traffic destined to hosts in other data center sites (see  FIG. 1 ). 
     For non-stretched subnets hosted in other data center sites, the corresponding subnet prefixes do not need to be advertised inside the fabric. However, for stretched subnets, since a leaf may only proxy on behalf of known destinations, host routes in remote data center sites must still be advertised to the internal leafs via iBGP. The host routes in other data center sites may not be installed in the FIB hardware table of the internal leafs. These host routes may be distinguished based on that fact that they are tagged with a different fabric site-of-origin (SOO) tags. Note that traffic destined to these hosts may be routed via the vrf default route and is forwarded to one of the border-leafs. 
     On the border leafs themselves, the subnet prefixes corresponding to the enhanced forwarding subnets may be installed as glean to ensure that routed traffic destined to unknown destinations is appropriately handled (for example generation of ICMP unreachable messages etc.). 
     It will therefore be appreciated by one of ordinary skill in the art that cross-fabric routes for enhanced forwarding enabled subnets may consume minimal FIB hardware resources on the internal leafs. 
     It will also be appreciated by one of ordinary skill in the art that the glean subnet prefix route may still need to be installed in the FIB LPM for ARP generation for discovery of silent hosts for subnets which have traditional forwarding enabled. Accordingly, for these across fabric subnets, it may still be necessary to install cross-fabric host routes in the FIB hardware tables on the internal leafs to ensure that traffic to these hosts is correctly forwarded to the border-leaf. 
     It is appreciated that software components of the present invention may, if desired, be implemented in ROM (read only memory) form. The software components may, generally, be implemented in hardware, if desired, using conventional techniques. It is further appreciated that the software components may be instantiated, for example: as a computer program product or on a tangible medium. In some cases, it may be possible to instantiate the software components as a signal interpretable by an appropriate computer, although such an instantiation may be excluded in certain embodiments of the present invention. 
     It is appreciated that various features of the invention which are, for clarity, described in the contexts of separate embodiments may also be provided in combination in a single embodiment. Conversely, various features of the invention which are, for brevity, described in the context of a single embodiment may also be provided separately or in any suitable subcombination. 
     It will be appreciated by persons skilled in the art that the present invention is not limited by what has been particularly shown and described hereinabove. Rather the scope of the invention is defined by the appended claims and equivalents thereof: