Embodiments are directed to facilitate multiple tunnels to be reachable via inter-chassis ling from virtual link trunking (VLT) peers and also to avoid traffic tromboning with an optimal path to all next hops. During initialization, a default egress object is created through an ICL with the next hop defined as VLT peer2 by setting the MAC address to be the MAC address of the VLT peer2, and the VLAN to be any one of the L3 spanned VLANs. When any of the VXLAN tunnels are resolved through ICL LAG, the process uses the default egress object created on ICL. Using the default egress object created on ICL, any traffic from VLT peer1 intended to go through ICL is terminated on VLT peer2 and will get routed again to their respective next hops. This creates a single egress object to the VLT peer and allowing multiple VXLAN tunnels to be reachable through ICL with optimal path towards all next hops.

TECHNICAL FIELD

Embodiments are generally directed to virtual extensible LAN (VXLAN) networks, and more specifically to increasing connections between routers in VXLAN networks.

BACKGROUND

VXLAN (virtual extensible LAN) is a virtualization technology was developed to address the scalability problems associated with large cloud computing systems. VXLAN provides mechanisms to aggregate and tunnel multiple layer 2 Ethernet sub-networks across a layer 3 infrastructure. For example, VXLAN may be used to connect two or more layer 2 network domains and make them look like a common layer 2 domain. This allows virtual machines on different networks to communicate as if they were in the same layer 2 subnet. Technically, VXLAN uses a VLAN-like encapsulation technique to encapsulate MAC-based OSI layer 2 Ethernet frames within layer 3 UDP packets. VXLAN endpoints, which terminate VXLAN tunnels and may be both virtual or physical switch ports, are known as VXLAN tunnel endpoints (VTEPs). Networking devices generally process VXLAN traffic transparently. That is, IP encapsulated traffic is switched or routed the same as any IP traffic. The VXLAN gateways (VTEPs) provide the encapsulating/de-encapsulating services central to VXLAN. VTEPS can be virtual bridges in the hypervisor, VXLAN aware VM applications or VXLAN capable switching hardware.

The VXLAN specification was developed by Arista, Broadcom, Intel, VMware, and others to improve scaling in virtualized data centers, among other applications. Routers and switches made by Broadcom are thus ubiquitous in VXLAN systems. In such networks, egress port to next hop mapping (EGR_PORT_TO_NHI_MAPPING) is one-to-one for every egress port, and hence only one next hop index can be mapped to an egress port. In a virtual link trunking (VLT) topology, when network port is a VLT lag (link aggregation), VLT peers will be in the same broadcast LAN and it is not possible to reach multiple remote VTEPs from a VLT peer through inter-chassis link (ICL), due to a particular restriction imposed or associated with Broadcom routers. This limitation causes certain bottleneck conditions that can greatly reduce network performance. In an embodiment, ICL is a link standard that provides a dedicated blade for switch ports for end devices in a network fabric by transporting traffic between chassis over dedicated high-speed links.

One such affect is traffic tromboning where traffic between a branch user and an Internet-based site may be backhauled over a corporate WAN, through a data center, then “tromboned” through to its Internet destination, then back to that data center, and finally is sent back over the corporate WAN to the original site. Such tromboning effects can add significant amounts of latency (e.g., plus 30 to 80 milliseconds of access latency for branch users) that can greatly effect network performance.

It is advantageous, therefore, to facilitate multiple tunnels to be reachable via ICL from VLT peers and also to avoid traffic tromboning with optimal path to all next hops.

DETAILED DESCRIPTION

Although embodiments are described in relation to a VXLAN-based network, certain described methods may involve automated techniques in a distributed system, such as a very large-scale wide area network (WAN), metropolitan area network (MAN), or cloud based network system, however, those skilled in the art will appreciate that embodiments are not limited thereto, and may include smaller-scale networks, such as LANs (local area networks). Thus, aspects of the one or more embodiments described herein may be implemented on one or more computers executing software instructions, and the computers may be networked in a client-server arrangement or similar distributed computer network.

FIG. 1illustrates a VXLAN network100that implements one or more embodiments of an inter-chassis link VXLAN tunnel connectivity, under some embodiments. In system100, multiple host clusters102a-dencompass small virtualized environments of VMs. In a multi-tenant cloud architecture, these clusters can be coupled through a combination of layer 2 (L2) and layer 3 (L3) devices and networks. In general, layer 2 is the data link where data packets are encoded and decoded into bits. The switches and links of the L2 networks forward all traffic, so that anything transmitted by one device is forwarded to all devices. The layer 3 networks provide switching and routing technologies, creating logical paths, known as virtual circuits, for transmitting data from node to node. Routing and forwarding are functions of this layer, as well as addressing, internetworking, error handling, congestion control and packet sequencing. The layer 3 protocols reduce overall traffic levels by allowing users to divide networks into smaller parts and restrict broadcasts to only that sub-network.

Each cluster102inFIG. 1may be a Virtual Data Center based on a hypervisor, such as Hyper-V from VMware. System100also includes Top-of-Rack (TOR) switches that connects physical servers that host the VMs, that can each belong to at least one virtual LAN.

The clusters102thus contain a number of VMs or groups of VMs that are provisioned to perform certain tasks, such as to serve as backup targets in a data replication environment. In such an application, target VMs may be organized into one or more virtual centers representing a physical or virtual network of many virtual machines (VMs), such as on the order of thousands of VMs each. The VMs serve as target storage devices for data backed up from one or more data sources that utilize networked accessed storage devices. The data sourced by the data source may be any appropriate data, such as database data that is part of a database management system. In this case, the data may reside on one or more hard drives and may be stored in the database in a variety of formats, such as XML (Extensible Markup Language) databases.

A network server computer may be coupled directly or indirectly to the target VMs and to the data source through a central network, which may be a cloud network, LAN, WAN or other appropriate network. This network provides connectivity to the various systems, components, and resources of system100, and may be implemented using protocols such as Transmission Control Protocol (TCP) and/or Internet Protocol (IP), well known in the relevant arts. In a distributed network environment, network may represent a cloud-based network environment in which applications, servers and data are maintained and provided through a centralized cloud-computing platform. In an embodiment, system100represents a multi-tenant network in which a server computer runs a single instance of a program serving multiple clients (tenants) in which the program is designed to virtually partition its data so that each client works with its own customized virtual application, with each VM representing virtual clients that may be supported by one or more servers within each VM, or other type of centralized network server.

In an embodiment, system100implements the VXLAN architecture to aggregate and tunnel multiple layer 2 networks or sub-networks across an infrastructure. The VXLAN base case is to connect two or more layer three network domains and make them look like a common layer 2 domain. This allows virtual machines on different networks (e.g.,102aand102c) to communicate as if they were in the same layer 2 sub-network. In general, the networking devices ofFIG. 1process VXLAN traffic transparently, such that IP encapsulated traffic is switched or routed the same as any IP traffic. VXLAN gateways, also called Virtual Tunnel End Points (VTEP), provide the encapsulating/de-encapsulating services central to VXLAN. VTEPs can be implemented as virtual bridges in the hypervisor, VXLAN-aware VM applications or VXLAN capable switching hardware.

As stated above, in Broadcom router-based networks, egress port to next hop mapping is one-to-one for every egress port and hence only one next hop index can be mapped to an egress port. In a VLT topology, this means that when a network port is a link aggregated (VLT LAG), VLT peers and TOR switches will be in the same broadcast LAN and it is not possible to reach multiple remote VTEPs from a VLT peer through ICL, due to one-to-one mapping restriction. This same problem can also be seen with non-VLT ports also, such as when multiple remote VTEPs are reachable through ICL, but with different next hops.

FIG. 2illustrates a Broadcom router-based network prior to implementation of a multiple tunneling process. In diagram200, VXLAN G/W2 (gateway 2) and G/W3 (gateway 3) are reachable from VXLAN G/W1 via a L2 switch206through two different next hops NH1 and NH2 using the same egress port P1 of G/W1. Since the routers202and204are in the same broadcast LAN, two different next hops should be reachable through the same port.

Certain Ethernet switches, such as the Broadcom T2, Tomahawk and TH+ model switches have a restriction with respect to supporting multiple next hops via the same outgoing port for VXLAN encapsulation. For system200, either router-1 can be chosen as next hop or router-2 can be chosen as next hop, but not both at the same time via the single outgoing network port P1. If router-1 is chosen, then VXLAN encapsulated traffic originated at VXLAN G/W 1 for VXLAN G/W 3 will be tromboned at router-1 back to the L2 switch206and will be L2 forwarded to the intended next hop router-2. If a route to the destination IP does not exist in router-1, then the packet can get dropped as well.

Embodiments of a multiple tunneling process described herein facilitate multiple tunnels to be reachable via ICL from VLT peers and also to avoid traffic tromboning with an optimal path to all next hops. This is accomplished through a process that, during initialization (i.e., once the VLT peers are up and system MAC addresses are exchanged) creates a default egress object through an ICL with next hop defined as VLT peer2. This is done by setting the MAC address to be the MAC address of the VLT peer2, and the VLAN to be any one of the L3 spanned VLANs. When any of the VXLAN tunnels are resolved through ICL LAG, the process always use the default egress object created on ICL. Using the default egress object created on ICL, any traffic from VLT peer1 intended to go through ICL is terminated on VLT peer2 and will get routed again to their respective next hops. The process thus creates a single egress object to the VLT peer and allowing multiple VXLAN tunnels to be reachable through ICL with optimal path towards all next hops.

This solution greatly alleviates previous issues of traffic tromboning where packets will take the sub-optimal path, such as in a case where a VLT topology has two TOR switches and each one is a VTEP in the context of VXLAN. In this case, from VLT peer1, next-hop towards VTEP1 is programmed for ICL ports, and to reach VTEP2, traffic from VLT peer1 might take an indirect path (VLT peer1 to VLT peer2 to VTEP1 to VLT peer2 to VTEP2), whereas packet could have directly taken path towards VTEP2 from VLT peer 2, thus avoiding VTEP1.

The problem being overcome (e.g., the Broadcom restriction) involves supporting only one next hop via a network port in a broadcast LAN for reaching remote VTEPs. That is:[DVP→Ingress L3 Next Hop→Port→Port Mapping table (Key=Port)→Egress L3 Next Hop (Has DA Mac, Port, L3 Interface)]

The Port Mapping table (having port as the key and pointing to egress L3 next hop) is the bottleneck in the VXLAN data path. In a VXLAN VLT, the native traffic sent over ICL is only locally switched and never VXLAN encapsulated. The reason for this is that there is no way to determine whether the traffic was received in the peer-side via network port or access-port. If it was received via the network port, then the traffic can be sent back to originating VTEP itself resulting in loop.

In an embodiment of the VXLAN multiple tunnel process, the issues presented by system200ofFIG. 2are overcome in that tromboning of VXLAN traffic is avoided by always choosing peer-VLT as the next hop for VXLAN data path, when ICL is an outgoing port due to half-VLT going down (while Route/ARP computation would have actually computed a different next hop). When the local exit is down, an ICL backup path is enabled using protection logic. There is no need to rewrite individual L2 MAC addresses. The process also avoids unnecessary forwarding of VXLAN encapsulating packet back to host in case of unknown L2 traffic handling at peer VLT device; and avoids black holing of traffic when the destination route is unknown at the intermediate next hop router.

FIG. 3is a diagram that illustrates a VXLAN VLT multi-tunneling process, under some embodiments. System300ofFIG. 3includes two remote VTEPs302and304(denoted respectively VTEP2 and VTEP3) routed through router-1306and router-2308to VLT peer-1 and peer-2 and to host (DC1). The host310sends native unicast/broadcast traffic towards VLT peer-1 on VLAN x. VLAN x is mapped to VNID1000, for example. On one or more half-VLT uplinks' failure on peer-1 (e.g., a LACP timeout and ungrouping), no local exit available at VLT peer-1 for packets whose data path uplink to next hop router is down, and hence ICL is the only path to reach the end destination.

Router-1 and router-2 are next hop routers reachable via the same ICL port-channel. Due to BCM restriction only one of the next hop routers can be programmed in the hardware. To overcome this, peer-1 must pick a single next hop, in the VXLAN forwarding path related next hop entry, to be able to reach IP2 as well as IP3 (there could be more than two destinations if more uplinks are down). For packets meant for VTEP4, nothing changes, and the next hop info that the corresponding DVP (distance vector protocol) points to peer-1 continues to be the uplink port and MAC M3.

For packets meant for VTEP2 and VTEP3, once the uplink goes down, the next hop data pointed to by the corresponding DVPs is modified to contain the ICL port and the VLT peer MAC. Broadcom switches have a protection logic that makes this specific modification rapid, and leads to very minimal traffic loss upon half-VLT uplink failure. In an example embodiment, suppose peer-1 picks either router-1 or router-2 as the next hop (e.g., router-1). In this case: VLT peer-1 does the VXLAN encapsulation and sends it towards the next hop router; the outer header will have the destination MAC as M1 and the outer destination IP will have IP3. This is shown inFIG. 3as flow point311. The VXLAN encapsulation packet gets L2 switched at VLT peer-2 towards router-1, as shown in flow point312. Router-1 then routes the VXLAN encapsulation packet by looking at the outer destination IP by rewriting the destination MAC of outer header as M2. The packet is tromboned and comes back to VLT peer-2. VLT peer-2 then switches the VXLAN encapsulation packet towards router-2, as shown in flow point313. If, in case VLT peer-2 does not have the L2 MAC for router-2 at that instant, then the VXLAN encapsulation packet goes back to the originating host as well. Router-2 then forwards the packet towards Remote VTEP-3, as shown in flow point314. There is a theoretical possibility of router-1 to not have a route for IP3 as well.

In an embodiment, the multi-tunneling process sets peer-2's MAC as the next hop MAC, for all next hops reachable via ICL post uplinks' failure. VLT peers typically have a L3 adjacency, so peer-2 would be able to route directly to router-2, and the packet would be on the way to the target VTEP. The possibility of packet looping back (momentary flooding), getting black-holed (e.g., router-1 not having a route entry) or getting tromboned (packets get delivered but in a circuitous path), are all avoided in this process.

FIG. 4illustrates configuring an egress object created on ICL as a protection next hop for VLT LAGs, under some embodiments. As shown inFIG. 4, in case of VLT LAGs, the process configures the egress object created on ICL as a protection next hop for VLT LAGS. When there is a local failure in a VLT LAG, a next hop created on ICL can be triggered using protection switching logic402.

As described above, embodiments are directed to VXLAN tunneling process that may be implemented as a computer implemented software process, or as a hardware component, or both. As such, it may be an executable module executed by the one or more computers in the network, or it may be embodied as a hardware component or circuit provided in the system. The network environment ofFIG. 1may comprise any number of individual client-server networks coupled over the Internet or similar large-scale network or portion thereof. Each node in the network(s) comprises a computing device capable of executing software code to perform the processing steps described herein.

FIG. 5is a block diagram of a computer system used to execute one or more software components of a system for VXLAN tunneling, under some embodiments. The computer system1000includes a monitor1011, keyboard1016, and mass storage devices1022. Computer system1000further includes subsystems such as central processor1010, system memory1015, input/output (I/O) controller1021, display adapter1025, serial or universal serial bus (USB) port1030, network interface1035, and speaker1040. The system may also be used with computer systems with additional or fewer subsystems. For example, a computer system could include more than one processor1010(i.e., a multiprocessor system) or a system may include a cache memory.

Arrows such as1045represent the system bus architecture of computer system1000. However, these arrows are illustrative of any interconnection scheme serving to link the subsystems. For example, speaker1040could be connected to the other subsystems through a port or have an internal direct connection to central processor1010. The processor may include multiple processors or a multicore processor, which may permit parallel processing of information. Computer system1000shown inFIG. 5is an example of a computer system suitable for use with the present system. Other configurations of subsystems suitable for use with the present invention will be readily apparent to one of ordinary skill in the art.

Computer software products may be written in any of various suitable programming languages. The computer software product may be an independent application with data input and data display modules. Alternatively, the computer software products may be classes that may be instantiated as distributed objects. The computer software products may also be component software. An operating system for the system may be one of the Microsoft Windows®. family of systems (e.g., Windows Server), Linux, Mac OS X, IRIX32, or IRIX64. Other operating systems may be used. Microsoft Windows is a trademark of Microsoft Corporation.

Although certain embodiments have been described and illustrated with respect to certain example network topographies and node names and configurations, it should be understood that embodiments are not so limited, and any practical network topography is possible, and node names and configurations may be used. Likewise, certain specific programming syntax and data structures are provided herein. Such examples are intended to be for illustration only, and embodiments are not so limited. Any appropriate alternative language or programming convention may be used by those of ordinary skill in the art to achieve the functionality described.

For the sake of clarity, the processes and methods herein have been illustrated with a specific flow, but it should be understood that other sequences may be possible and that some may be performed in parallel, without departing from the spirit of the invention. Additionally, steps may be subdivided or combined. As disclosed herein, software written in accordance with the present invention may be stored in some form of computer-readable medium, such as memory or CD-ROM, or transmitted over a network, and executed by a processor. More than one computer may be used, such as by using multiple computers in a parallel or load-sharing arrangement or distributing tasks across multiple computers such that, as a whole, they perform the functions of the components identified herein; i.e. they take the place of a single computer. Various functions described above may be performed by a single process or groups of processes, on a single computer or distributed over several computers. Processes may invoke other processes to handle certain tasks. A single storage device may be used, or several may be used to take the place of a single storage device.

All references cited herein are intended to be incorporated by reference. While one or more implementations have been described by way of example and in terms of the specific embodiments, it is to be understood that one or more implementations are not limited to the disclosed embodiments. To the contrary, it is intended to cover various modifications and similar arrangements as would be apparent to those skilled in the art. Therefore, the scope of the appended claims should be accorded the broadest interpretation so as to encompass all such modifications and similar arrangements.