Packet recirculation for tunneling encapsulation

Techniques for operating a network device for multiple packet encapsulation for different tunnels are provided. In some embodiments, the network device may receive an original packet on an ingress port, the original packet being received from a first host and addressed to a second host; encapsulate the original packet in a first tunnel packet for a first tunnel; recirculate the first packet through a loopback port; encapsulate the recirculated packet in a second tunnel packet for a second tunnel; and egress the packet encapsulated for the second tunnel. The switch may further add a first tunnel header to the original packet to encapsulate the first packet and add a second tunnel header to the recirculated packet to encapsulate the recirculated packet.

BACKGROUND

Data packets use networking protocols to get to their destinations. However, not all networks support all protocols. Consider a wide area network (WAN) connecting Location A and Location B. Location A and Location B each have networks that use a protocol such as IPv6. However, the network between Location A and Location B uses another version of the Internet Protocol, such as IPv4. In this example, by encapsulating IPv6 packets inside IPv4 packets, IPv6 can be used at Locations A and B, while still sending data directly between Locations A and B. Encapsulating packets within other packets is called “tunneling.” Tunnels are a way to transport data across a network using protocols that are not supported by that network.

DETAILED DESCRIPTION

Overview

The present disclosure describes systems and techniques for operating a network device (e.g., switch, router, and the like) to perform multiple packet encapsulation for different tunnels. To send packets in one tunnel (tunnel1) through another tunnel (tunnel2), an originating network device may encapsulate the packets twice. In the first pass through the network device's forwarding pipeline, the packet may be encapsulated for tunnel1. In the second pass through the network device's forwarding pipeline, the packet may be encapsulated for tunnel2.

An example configuration may be referred to as VXLAN over GRE, where tunnel1is Virtual Extensible LAN (VXLAN) and tunnel2is Generic Routing Encapsulation (GRE). Consider two hosts (host1and host2) that communicate with each other through a VXLAN tunnel. Host1is connected to switch1and host2is connected to switch2. Switch1and switch2are VXLAN tunnel endpoints, the VXLAN tunnel originating at one side and terminating at the other side. Typically, they are reached through an Internet Protocol (IP) fabric. In this example, switch1does not have a direct way to reach switch2over VXLAN. However, switch2may be reached by switch1via a GRE tunnel. Between switch1and switch2is switch3. Switch1may reach switch3through a GRE tunnel. Switch3may then reach switch2through the VXLAN tunnel.

Switch1receives a packet from host1and the packet is addressed to host2. The packet processor in switch1encapsulates the packet in a VXLAN packet. The VXLAN packet is looped back into the packet processor of switch1, where the VXLAN packet is encapsulated in a GRE packet. The GRE packet egresses switch1and goes to switch3through a GRE tunnel. Switch3decapsulates the VXLAN packet from the GRE packet. The VXLAN packet egresses switch3and goes to switch2through the VXLAN tunnel. Switch2decapsulates the original packet from the VXLAN packet and forwards the original packet to host2.

GRE over VXLAN may also be performed using these techniques. More generally, this double encapsulation is applicable to other tunneling technologies.

Network Topology

FIG.1illustrates topology100A for a data network according to some embodiments. Topology100A may include host_1110A, switch_1120A, network130A, switch_2140A, and host_2150A. Network130A may include one or more switches, such as switch_A132A through switch_N134A. Network130A may be a segmented routing over Internet Protocol (IP) (layer 3) network.

Host_1110A and host_2150A may be computing devices, such as servers, desktop computers, laptop computers, tablet computers, smartphones, and the like. Host_1110A and host_2150A may communicates with switch_1120A and switch_2140A, respectively, directly or through one or more intermediate network devices (not shown). Hosts communicate with each other over one or more networks, such as network130A. The networks may include nodes such as switch_1120A, switch_A132A through switch_N134A, and switch_2140A. Although the nodes depicted inFIG.1are labeled as switches, they may be other network devices, such as routers and the like. Switch_1120A, switch_A132A through switch_N134A, and switch_2140A may be embodiments of network device600inFIG.6. Switch_A132A represents a network device in network130A that communicates with switch_1120A. There may be one or more network devices (not shown) between switch_A132A and switch_N134A. Switch_N134A may communicate with switch_2140A directly or through one or more intermediate network devices (not shown).

Suppose host_1110A communicates with host_2150A using a first tunneling protocol (through tunnel X160A), such as Virtual Extensible LAN (VXLAN), Generic Routing Encapsulation (GRE), Multiprotocol Label Switching (MPLS), IPinIP, and the like. VXLAN is an encapsulation protocol that uses tunneling to stretch layer 2 connections over an underlying layer 3 network. GRE is a protocol for encapsulating data packets that use one routing protocol inside the data packets of another protocol. A data packet may be a formatted unit of data carried by a data network. A data packet may include control information and user data (payload).

In this example, host_1110A communicates with switch_1120A, and host_2150A communicates with switch_2140A. Switch_1120A and switch_2140A are endpoints for tunnel X160A. Accordingly, switch_1120A and switch_2140A may have endpoint addresses associated with the tunneling protocol used by tunnel X160A. Switch_1120A may encapsulate data packets from host_1110A for communication through tunnel X160A. Switch_2140A may decapsulate data packets for host_2150A received through tunnel X160A. Switch_1may communicate with switch_2through network130A.

Suppose further that switch_A132A does not recognize switch_2's140A endpoint address for tunnel X160A. A forwarding table in switch_A132A may not be programmed with switch_2's140A endpoint address for tunnel X160A. This may be, for example, because switch_2140A does not advertise its endpoint address for tunnel X160A on network130A.

However, switch_N134A may recognize switch_2's140A endpoint address for tunnel X160A. In addition, switch_1120A may reach switch_N134A through a second tunneling protocol (through tunnel Y170A). The second tunneling protocol may be GRE, VXLAN, Multiprotocol Label Switching (MPLS), IPinIP, and the like. Typically, tunnel X160A and tunnel Y170A use different tunneling protocols. Here, switch_1120A may also be an endpoint for tunnel Y170A, and switch_N may be an endpoint for tunnel Y170A. Switch_A132A may recognize switch_N's134A endpoint address for tunnel Y170A.

In this example, switch_1120A may encapsulate a data packet from host_1110A to host_2150A twice: once for tunnel X160A and then for tunnel Y170A. Since the data packet encapsulated for tunnel X160A is further encapsulated for tunnel Y170A, tunnel X160A may be said to be “over” tunnel Y170A (“tunnel X over tunnel Y”). By way of example and not limitation, “VXLAN over GRE” and “GRE over VXLAN” are described in further detail below. The twice-encapsulated data packet may go from switch_1120A to switch_N134A through tunnel Y170A. Switch_N134A may decapsulate the data packet, leaving the data packet encapsulated for tunnel X160A, and forward the data packet to switch_2140A. Switch_2140A may decapsulate the data packet and forward the data packet to host_2150A.

Tunnel Y170A is depicted above tunnel X160A to convey that packets for tunnel X160A are transported over/via tunnel Y170A. A header for tunnel Y170A protocol may be the outermost header in the data packet and forwarding lookups will be performed on outer tunnel Y170A protocol header on intermediate switches switch_A132A through switch_N134A.

Network Device

FIGS.2A and2Billustrate switch_1120B according to some embodiments. The following description ofFIGS.2A and2Bis made with reference toFIG.1. Switch_1120E may be an embodiment of switch_1120A. Switch_1120E may comprise control plane210and data plane230. Control plane210may include CPU220which may be an embodiment of management CPU108described inFIG.6. Data plane230may include forwarding application specific integrated circuit (ASIC)240, ingress ports2601-260X, and egress ports2701-270Y. ASIC240may be an embodiment of packet processor112a-112pdescribed inFIG.6. ASIC240may comprise forwarding pipeline242and loopback port244.

Switch_1120E may receive data packet110E from host_1110A. Data packet110E may be addressed to host_2150A. Data packet110E may enter switch_1120E through ingress port2601and go to forwarding pipeline242(path282). Forwarding pipeline242may look up host_2150A in a forwarding table (not shown) and determine that host_2150A may be reached through tunnel X160A. Forwarding pipeline242may encapsulate data packet110B according to the protocol for tunnel X160A. The forwarding table defines how a data packet will be forwarded out of a network device. The forwarding table may match data packet header fields, such as the IP destination address, and when a match occurs, forward the frame to a specified egress port (e.g., of egress ports2701-270Y).

ASIC240's forwarding table may be programmed to take into account that switch_1120A does not have a direct route to switch_2140A. The route to switch_2140A may be through tunnel Y170A. Since control plane210knows that switch_2140A is reachable via tunnel Y170A, control plane210programs forwarding pipeline242so that the destination port is loopback port244and the data packet is recirculated. The packet encapsulated for tunnel X160A may re-enter forwarding pipeline242through loopback port244(paths284and286). Loopback port244provides a path for packets to be processed by forwarding pipeline242more than once. As shown, loopback port244may be an internal loopback provided by ASIC240. For example, ASIC240may have one or more dedicated internal loopback ports with unique port numbers. When a data packet is directed to a dedicated internal loopback port, the data packet may enter forwarding pipeline242again. Here, the recirculated packet may stay in ASIC240and not go to one of egress ports2701-270Y. Loopback port244may also be one of egress ports2701-270Y. In other words, some of egress ports2701-270Yare dedicated external loopback ports. A data packet sent to a dedicated external loopback port may be returned to forwarding pipeline242. For example, the dedicated external loopback port may be configured to return the data packet within switch_1120B to forwarding pipeline242, may have a special adapter/plug/cable to send the data packet going out from an external loopback port back into switch_1120E through one of ingress ports2601-260X, and the like. Going to forwarding pipeline242again may be referred to as packet recirculation. The second time through forwarding pipeline242, the packet encapsulated for tunnel X160A may be further encapsulated according to the protocol for tunnel Y170A. The twice-encapsulated data packet (data packet132B) egresses switch_1120E through egress port2701(path288)

Packet Recirculation Workflow

FIG.3illustrates workflow300for packet recirculation according to some embodiments. Workflow300may be performed by switch_1120B. Description of workflow300will be made with reference toFIGS.1,2A and2B.

Workflow300may commence at step310, where a switch_1120B receives data packet110B, referred to as the original packet. For example, data packet110E may ingress switch_1120E through ingress port2601and go to forwarding pipeline242(path282). At step320, packet110E is encapsulated for the first tunnel. For example, forwarding pipeline242may encapsulate packet110E according to the protocol for tunnel X160A.

At step330, the once-encapsulated packet (for the first tunnel) is recirculated back to forwarding pipeline242using loopback244(paths284and286). At step340, the once-encapsulated data packet may be encapsulated again in forwarding pipeline242, this time for the second tunnel. For example, forwarding pipeline242may encapsulate the data packet according to the protocol for tunnel Y170A.

At step350, the twice-encapsulated data packet egresses switch_1120E to the next hop for the second tunnel. For example, data packet132E egresses through egress port2701to switch_A132A.

VXLAN Over GRE

FIG.4Aillustrates communications path100C for VXLAN over GRE according to some embodiments. Communications path100C and its constituents may be an embodiment of topology100A and its constituents. Switch_1120C may be an embodiment of switch_1120B. Moreover, switch_A132C through switch_N134C and switch_2140C may each have at least some of the characteristics of switch_1120B. Communications path100C may include host_1110C, switch_1120C, network130C, switch_2140C, and host_2150C. Network130C may include switch_A132C through switch_N134C.

Switch_1120C may be a VXLAN Tunnel End Point (VTEP), VTEP1, with an IP address of 100.1.1.1. Switch_1120C may also be a GRE endpoint with an IP address of 162.1.1.161. Switch-N134C may be a GRE endpoint with an IP address of 180.1.1.10. Switch_2140C may be a VTEP, VTEP2, with an IP address of 200.1.1.1. Host_1110C may have an IP address of 10.1.1.1 and host_2150C may have an IP address of 10.1.1.2.

Host_1110C may send a data packet, addressed to host_2150C, to switch_1120C. Switch_1120C may determine that host_2150C is behind a VXLAN tunnel with a destination IP address of 200.1.1.1, which is VTEP2. Switch_1120C may encapsulate the data packet for VXLAN. Here, switch_1120C adds a VXLAN header to the packet with an outer destination IP address of 200.1.1.1 and an outer destination Media Access Controller (MAC) address of Switch_1's120C MAC address. Switch_1120C may also determine that switch_1120C does not have a direct route to reach outer destination IP address 200.1.1.1 and that address 200.1.1.1 is reached through a GRE tunnel. In other words, switch_1120C may resolve that outer destination IP address of 200.1.1.1 for VXLAN tunnel160C will go through GRE tunnel170C. The forwarding tables in ASIC240may be programmed/configured such that first route/forwarding lookup for host_2150C will produce loopback port244as the outgoing port. This resolution may be performed in control plane210and forwarding pipeline242is programmed/configured accordingly.

The VXLAN encapsulated data packet may be looped back and go through forwarding pipeline242of switch_1120C again. The second pass through the forwarding pipeline may encapsulate the data packet for GRE (add a GRE header). The twice-encapsulated packet egresses switch_1120C to switch_A132C.

The twice-encapsulated packet may proceed through GRE tunnel170C over network130C until it reaches the GRE endpoint (switch_N134C). Switch_N134C may decapsulate the GRE encapsulated data packet, restoring the VXLAN encapsulated data packet. The VXLAN encapsulated data packet may proceed through VXLAN tunnel160C to switch_2140C. Switch_2140C may decapsulate the VXLAN encapsulated data packet and forward the decapsulated data packet to host_2150C.

A reverse path from host_2150C to host_1110C may be as follows. Switch_2140C may receive a data packet from host_2150C. Switch_2140C may encapsulate the data packet with a VXLAN header and send the VXLAN encapsulated data packet to switch_N134C. Switch_N may further encapsulate the packet with a GRE header and send it to switch_1120C. Switch_1120C may receive the twice-encapsulated data packet.

Analyzing the outer data packet header, switch_1120may see the packet is addressed to its own MAC address as the destination MAC address and to its GRE endpoint address as the destination IP address. Switch_1120C may decapsulate the GRE encapsulated data packet, restoring the VXLAN encapsulated data packet. The VXLAN encapsulated data packet may be recirculated. During the second pass through the forwarding pipeline, the VXLAN encapsulated data packet may be decapsulated, based on the inner destination MAC address and destination IP address. Switch_1120C may forward the data packet to host_1110C.

FIG.4Billustrates control information of a data packet after each pass through forwarding pipeline242of switch_1120C, in accordance with various embodiments. The data packet payload is not depicted. The IP addresses shown inFIG.4Bcorrespond to the IP addresses shown inFIG.4A. Header410may be the data packet header received from host_1110C. Header410may include destination address412, source address414, source IP address416, and destination IP address418.

During the first pass through switch_1's120C forwarding pipeline, a VXLAN header may be added, resulting in header420. The VXLAN header may include destination address422, source address424, source IP address426, and destination IP address428.

During the second pass through switch_1's120C forwarding pipeline, a GRE header may be added, resulting in header430. The GRE header may include source IP address436and destination IP address438. Destination address432may also be added.

FIG.4Cillustrates configuration information470for VTEP switch_1110C and configuration information480for VTEP switch_2140C, according to some embodiments. Configuration information470may set up a VXLAN tunnel; specify that VTEP2 is reachable over a GRE tunnel, and set up the GRE tunnel. Configuration information480may set up a VXLAN tunnel. For example, GRE tunnel destination 180.1.1.10 reachability may be either statically configured or dynamically learned through routing protocols.

GRE Over VXLAN

FIG.5Aillustrates communications path100D for GRE over VXLAN according to some embodiments. Communications path100D and its constituents may be an embodiment of topology100A and its constituents. Switch_1120D may be an embodiment of switch_1120B. Moreover, switch_A132D through switch_N134D and switch_2140D may each have at least some of the characteristics of switch_1120B. Communications path100D may include host_1110D, switch_1120D, network130D, switch_2140D, and host_2150D. Network130D may include switch_A132D through switch_N134D.

By way of non-limiting example, switch_1120D may be in a headquarters and switch_2140D in a branch office. There may be a point-to-point connection established between the headquarters and branch office using a GRE tunnel. Suppose switch_1120D cannot reach the GRE endpoint at switch_2140D, but can reach switch_N134D over a VXLAN tunnel. Switch_N134D can reach switch_2140D over a GRE tunnel.

Switch_1120D may be a GRE endpoint with an IP address of 162.1.1.161. Switch_1120D may also be a VTEP, VTEP1, with an IP address of 100.1.1.1. Switch-N134D may be a VTEP, VTEP2, with an IP address of 200.1.1.1. Switch_2140D may be a GRE endpoint with an IP address of 180.1.1.10. Host_1110D may have an IP address of 10.1.1.2 and host_2150D may have an IP address of 4.4.4.4.

Host_1110D and host_2150D are in different subnet domains. When host_1110D sends a data packet with a destination IP address of 4.4.4.4, the destination address may be switch_1's120D MAC address. Since the data packet's destination address is switch_1's120D MAC address, switch_1120D may lookup destination IP address 4.4.4.4 in its forwarding table. Switch_1120D may determine that host_2150D is reachable through a GRE tunnel and encapsulates the data packet for GRE (add a GRE header). Switch_1120D may also update the layer 2 Ethernet header destination address with the next hop's (switch_A132D) MAC address and source address with switch_1's120D MAC address.

The GRE encapsulated data packet may be looped back and go through forwarding pipeline242of switch_1120D again. During the second pass through forwarding pipeline242, the packet's destination address is not switch_l's120D MAC address, so forwarding pipeline242may not perform layer 3 route lookup. Instead, forwarding pipeline242may perform layer 2 forwarding lookup. The layer 2 forwarding lookup may determine that the destination IP address of 180.1.1.10 and destination address of the next-hop switch's MAC address are reachable through VTEP2's IP address 200.1.1.1 (VXLAN tunnel170D). Switch_1120D may encapsulate the data packet for VXLAN (add a VXLAN header). The twice-encapsulated packet egresses switch_1120D to switch_A132D.

The twice-encapsulated packet may proceed through VXLAN tunnel170D over network130D until it reaches VTEP2 (switch_N134D). Switch_N134D may decapsulate the VXLAN encapsulated data packet, restoring the GRE encapsulated data packet. The GRE encapsulated data packet may proceed through GRE tunnel160D to switch_2140D. Switch_2140D may decapsulate the GRE encapsulated data packet and forward the decapsulated data packet to host_2150D.

A reverse path from host_2150D to host_1110D may be as follows. Switch_2140D may receive a data packet from host_2150D. Switch_2140D may encapsulate the data packet with a GRE header and send the GRE encapsulated data packet to switch_N134D. Switch_N may further encapsulate the packet with a VXLAN header and send it to switch_1120D. Switch_1120D may receive the twice-encapsulated data packet.

Analyzing the outer data packet header, switch_1120D may see the packet is addressed to its own MAC address as the destination MAC address and to its GRE endpoint address as the destination IP address. Switch_1120D may decapsulate the VXLAN encapsulated data packet, restoring the GRE encapsulated data packet. The GRE encapsulated data packet may be recirculated. During the second pass through the forwarding pipeline, the GRE encapsulated data packet may be decapsulated, based on the inner destination MAC address and destination IP address. Switch_1120D may forward the data packet to host_1110D.

FIG.5Billustrates control information of a data packet after each pass through forwarding pipeline242of switch_1120D. The data packet payload is not depicted. The IP addresses shown inFIG.5Bcorrespond to the IP address shown inFIG.5A. Header510may be the data packet header received from host_1110D. Header510may include destination address512, source address514, source IP address516, and destination IP address518.

During the first pass through switch_1's120D forwarding pipeline, the packet may be encapsulated for GRE (a GRE header is added), resulting in header520. The GRE header may include source IP address526and destination IP address528. Destination address522and source address524may also be added.

During the second pass through switch_1's120D forwarding pipeline242, a VXLAN header may be added, resulting in header530. The VXLAN header may include destination address532, source address534, source IP address536, and destination IP address538.

FIG.5Cillustrates configuration information570for GRE endpoint switch_1110dand configuration information580for GRE endpoint switch_2140D according to various embodiments. Configuration information570may set up a GRE tunnel; specify that the GRE endpoint is reachable over a VXLAN tunnel, and set the VXLAN tunnel. Configuration information580may set up a GRE tunnel.

FIG.5Dillustrates an example packet dump590in accordance with some embodiments. For example, packet dump590shows the addition of two tunnel headers by switch_1120D.

VXLAN and GRE tunneling are provided above by way of example and not limitation. Other tunneling protocols may be used.

Network Device

FIG.6depicts an example of a network device600in accordance with some embodiments of the present disclosure. In some embodiments, network device600can be a switch. As shown, network device600includes a management module102, an internal fabric module104, and a number of I/O modules106a-106p. Management module102includes the control plane (also referred to as control layer or simply the CPU) of network device600and can include one or more management CPUs108for managing and controlling operation of network device600in accordance with the present disclosure. Each management CPU108can be a general-purpose processor, such as an Intel®/AMD® x86 or ARM® microprocessor, that operates under the control of software stored in a memory, such as random access memory (RAM)126. Control plane refers to all the functions and processes that determine which path to use, such as routing protocols, spanning tree, and the like.

Internal fabric module104and I/O modules106a-106pcollectively represent the data plane of network device600(also referred to as data layer, forwarding plane, etc.). Internal fabric module104is configured to interconnect the various other modules of network device600. Each I/O module106a-106pincludes one or more input/output ports110a-110pthat are used by network device600to send and receive network packets. Input/output ports110a-110pare also known as ingress/egress ports. Each I/O module106a-106pcan also include a packet processor112a-112p. Each packet processor112a-112pcan comprise a forwarding hardware component (e.g., application specific integrated circuit (ASIC), field programmable array (FPGA), digital processing unit, graphics coprocessors, content-addressable memory, and the like) configured to make wire speed decisions on how to handle incoming (ingress) and outgoing (egress) network packets. In accordance with some embodiments some aspects of the present disclosure can be performed wholly within the data plane.

Management module102includes one or more management CPUs108that communicate with storage subsystem120via bus subsystem130. Other subsystems, such as a network interface subsystem (not shown inFIG.1), may be on bus subsystem130. Storage subsystem120includes memory subsystem122and file/disk storage subsystem128represent non-transitory computer-readable storage media that can store program code and/or data, which when executed by one or more management CPUs108, can cause one or more management CPUs108to perform operations in accordance with embodiments of the present disclosure.

Memory subsystem122includes a number of memories including main RAM126for storage of instructions and data during program execution and read-only memory (ROM)124in which fixed instructions are stored. File storage subsystem128can provide persistent (i.e., non-volatile) storage for program and data files, and can include a magnetic or solid-state hard disk drive, and/or other types of storage media known in the art.

One or more management CPUs108can run a network operating system stored in storage subsystem120. A network operating system is a specialized operating system for network device600(e.g., a router, switch, firewall, and the like). For example, the network operating system may be Arista Extensible Operating System (EOS), which is a fully programmable and highly modular, Linux-based network operating system. Other network operating systems may be used.

Bus subsystem130can provide a mechanism for letting the various components and subsystems of management module102communicate with each other as intended. Although bus subsystem130is shown schematically as a single bus, alternative embodiments of the bus subsystem can utilize multiple busses.