Patent Description:
With the development of technologies for data center deployments, there have emerged a new set of requirements, such as multihoming and redundancy, multicast optimization, provisioning simplicity, flow-based load balancing, and multipathing. These requirements are difficult to fulfill for current layer-<NUM> VPN (L2VPN) solutions such as virtual private local area network (LAN) service (VPLS). To meet these requirements, the Internet engineering task force (IETF) has proposed, in request for comments (RFC) <NUM>, a new VPN solution called Ethernet VPN (EVPN). It is a border gateway protocol (BGP) multi-protocol label switching (MPLS)-based solution.

In a particular EVPN instance, a provider edge (PE) device uses a provider tunnel (P-tunnel) to send broadcast, unknown unicast or multicast (BUM) traffic to other PE devices. The P-tunnel could be implemented by ingress replication or point to multipoint (P2MP) tunnels. For ingress replication, a given BUM packet is sent from a single ingress PE device to other PE devices. For P2MP tunnels, the transport technology could be resource reservation protocol-traffic engineering (RSVP-TE) or multicast label distribution protocol (mLDP) to create a multicast tree.

Further background information can be found in European patent application <CIT>.

As mentioned above, the existing flooding mechanisms in EVPN include ingress replication and P2MP tunnels. For ingress replication, the bandwidth efficiency is low since an ingress PE device has to replicate a BUM packet multiple times and send each one of the multiple copies to a corresponding egress PE device.

For P2MP tunnels, mandatory end-to-end multicast support is required for transport infrastructure. However, the support is often not available, which mostly happens on traverse networks. Furthermore, P2MP is complex in operation and has a high maintaining cost. No matter RSVP-TE/mLDP or other multicast protocols are employed, it is a heavy task to establish and maintain a multicast tree. In addition, P2MP has a fixed tree topology since the multicast tree established depends on the multicast protocol used. Thus, it is difficult to tune based on traffic engineering policy at running time.

The present disclosure proposes an improved forwarding solution for EVPN. Hereinafter, the solution will be described in detail with reference to <FIG>.

<FIG> is a schematic diagram showing a communication system into which an embodiment of the disclosure is applicable. The communication system may be used to implement an EVPN. As shown, the communication system comprises four provider edge (PE) devices <NUM>-<NUM>, <NUM>-<NUM>, <NUM>-<NUM> and <NUM>-<NUM> (simply denoted as PE1, PE2, PE3 and PE4), an Internet protocol (IP)/multi-protocol label switching (MPLS) network <NUM>, and two provider (P) devices <NUM>-<NUM> and <NUM>-<NUM> (simply denoted as P1 and P2). Although four PE devices and two P devices are shown in <FIG>, there may be more or less PE devices or P devices. Note that customer edge (CE) devices are omitted in <FIG> for brevity.

The CE device enables a terminal device to connect to the IP/MPLS network <NUM>. The terminal device may be, for example, a mobile phone, a pad computer, a laptop computer, a desktop computer, or any other devices with wired and/or wireless communication capability. The CE device may be, for example, a router, a switch, a gateway, a modem, a firewall, a network interface controller (NIC), a hub, a bridge, or any other type of data transfer device. The PE device <NUM>-<NUM>/<NUM>-<NUM>/<NUM>-<NUM>/<NUM>-<NUM> is an edge node of the IP/MPLS network <NUM> and functions as an edge device responsible for providing users with EVPN services. The PE device <NUM>-<NUM>/<NUM>-<NUM>/<NUM>-<NUM>/<NUM>-<NUM> may be, for example, a router, a switch, a gateway, a modem, a firewall, an NIC, a hub, a bridge, or any other type of data transfer device.

Each CE device may be either connected to one PE device, or multihomed to two or more PE devices via an Ethernet segment which comprises links between the CE device and each of the two or more PE devices. The Ethernet segment can be identified by an Ethernet segment identifier (ESI). The ESI may be manually configured or automatically derived. Once the ESI for the Ethernet segment is assigned for the CE device, it may be advertised by the two or more PE devices through Ethernet Segment Route defined in RFC <NUM> and related protocols. Thus, the two or more PE devices can automatically discover that they are all connected to the same Ethernet segment.

The IP/MPLS network <NUM> can route and/or forward traffic provided via the EVPN. The IP/MPLS network <NUM> may be, for example, an IP based network, or an MPLS based network, or a combination thereof. The P device <NUM>-<NUM>/<NUM>-<NUM> operates inside the IP/MPLS network <NUM> and does not directly interface to any CE device. The P device <NUM>-<NUM>/<NUM>-<NUM> may be, for example, a router, a switch, a gateway, a modem, a firewall, an NIC, a hub, a bridge, or any other type of data transfer device.

As an exemplary example, it is assumed that an enterprise has CE devices (for example, application servers) deployed in multiple data centers at different locations to communicate with each other within the same L2VPN. As the data centers are interconnected through transport networks such as IP/MPLS, then, the EVPN may be used to accommodate L2VPN services over the transport networks connecting to these data centers. These CE devices located in different data centers can be considered as belonging to the same EVPN instance. There may be multiple EVPN instances in the data centers.

<FIG> is a flowchart illustrating a method implemented at a controlling device according to an embodiment of the disclosure. For example, the controlling device may be implemented as a software defined networking (SDN) controller. As shown in <FIG> which may facilitate understanding of this method, the controlling device (e.g., SDN controller) may be applied into the environment as shown in <FIG>. It should be noted that due to some characteristics such as the capabilities of communicating with and controlling various network devices, as well as knowing various configuration information of the corresponding network, the SDN controller is suitable for implementing the controlling device of the present disclosure. However, those skilled in the art will understand that the controlling device may also be implemented in any other suitable ways as long as the steps described below regarding the controlling device can be performed.

At step <NUM>, the controlling device determines a first P device which is to handle BUM traffic from a first PE device in an EVPN instance, on behalf of remaining PE devices in the same EVPN instance. The first PE device may be any one of the PE devices in the same EVPN instance. In the exemplary example shown in <FIG>, it is assumed that PE1, PE2, PE3 and PE4 belong to the same EVPN instance and the first PE device corresponds to PE1. Then, the remaining PE devices correspond to PE2, PE3 and PE4.

At step <NUM>, the controlling device performs route reflection from the remaining PE devices to the first PE device such that the first PE device knows that BUM traffic needs to be forwarded to the first P device. The controlling device acts as a BGP route reflector to perform the route reflection.

For example, step <NUM> may be implemented as steps <NUM>-<NUM> of <FIG>. Step <NUM> is implemented as steps <NUM>, <NUM> and <NUM> of <FIG>. Specifically, at step <NUM>, routes related to BUM traffic handling are received from the PE devices in the EVPN instance, as shown in <FIG>. The route from a PE device may be, for example, an Inclusive Multicast Ethernet Tag Route which includes MPLS Label field in PMSI Tunnel Attribute, Originating Router's IP Address field and Next Hop field, as defined in RFC <NUM> and related protocols. The MPLS Label field is allocated by the PE device.

At step <NUM>, a provider tunnel (P-tunnel) from the first PE device to the remaining PE devices is determined based at least on the received routes related to BUM traffic handling. Specifically, by using these routes, a tree having the first PE device as the root and the remaining PE devices as the leaves may be determined as the P-tunnel.

Optionally, for optimization of the determined tree, the determination may be based on the transport topology among the PE devices in the EVPN instance. Further, optionally, the determination may be based on traffic engineering policy/status. For example, if the use of a portion of the determined tree should be avoided according to the traffic engineering policy/status, this portion may be removed from the determined tree. Thus, P-tunnel planning is not limited to or fixed to specific multicast protocols and thus can be more flexible.

At step <NUM>, the first P device is selected from the P-tunnel. For example, a P device located at the downstream of the first PE device and the upstream of the remaining PE devices (e.g., at the first branch in the downstream direction of the first PE device) may be selected from the determined tree as the first P device. In the example shown in <FIG>, it is assumed that the first P device corresponds to P1.

At step <NUM>, the routes from the remaining PE devices are modified such that upon receipt of the modified routes, the first PE device knows that BUM traffic needs to be forwarded to the first P device. The Inclusive Multicast Ethernet Tag Route, step <NUM> is implemented as steps <NUM>-<NUM> or may be implemented as steps <NUM>-<NUM> of <FIG>, depending on the types of the EVPN and the P-tunnel between the first PE and P devices. Specifically, when the P-tunnel between the first PE and P devices is implemented through P2MP tunnels, since the ESI label is upstream assigned, steps <NUM>-<NUM> are not needed.

When the P-tunnel between the first PE and P devices is implemented through ingress replication, for example, when it is a multipoint to point (MP2P)/point to point (P2P) tunnel, the implementation of step <NUM> depends on the type of the EVPN. If the EVPN is an MPLS-based VPN defined in RFC7432, such as MPLS over generic routing encapsulation (MPLSoGRE), steps <NUM>-<NUM> are needed. If the EVPN is a provider backbone bridging (PBB)-based VPN defined in RFC7623, or a virtual extensible LAN (VXLAN)/network virtualization using generic routing encapsulation (NVGRE) based VPN defined in draft-ietf-bess-evpn-overlay-<NUM>, steps <NUM>-<NUM> are not needed.

At step <NUM>, the Originating Router's IP Address field is set to an IP address of the controlling device, as shown in <FIG>. This can inform the first PE device that this route comes from the controlling device which acts as a route reflector. At step <NUM>, the Next Hop field is set to an IP address of the first P device, as shown in <FIG>. This can inform the first PE device that it has a neighbor, i.e. the first P device. At step <NUM>, the MPLS Label field in PMSI Tunnel Attribute allocated by the remaining PE device is replaced with a corresponding value allocated by the first P device, as shown in <FIG>. This MPLS label may also be referred to as VPN label herein. This modification can inform the first PE device that its neighbor has the indicated VPN label. In this way, all the routes from the remaining PE devices can be modified to inform the first PE device of the same information that it has a neighbor, i.e. the first P device.

As described above, if the P-tunnel between the first PE and P devices is implemented through ingress replication and the EVPN is an MPLS-based VPN, steps <NUM>-<NUM> are needed. At step <NUM>, the controlling device generates Ethernet A-D per Ethernet Segment Routes including ESIs of the remaining PE devices. Specifically, when the controlling device receives an Ethernet A-D per Ethernet Segment Route from each remaining PE device, in addition to reflecting this Ethernet A-D per Ethernet Segment Route without changes, the controlling device also generates a corresponding Ethernet A-D per Ethernet Segment Route including the ESI(s) of the remaining PE device. The Ethernet A-D per Ethernet Segment Route may be generated according to RFC <NUM> and related protocols.

At step <NUM>, the controlling device sends the generated Ethernet A-D per Ethernet Segment Routes to the first PE device. As a result, from the viewpoint of the first PE device, all the ESIs of the remaining PE devices are attached to the first P device. For example, as shown in <FIG>, the ESIs (A, B, C, D) of the remaining PE devices (PE2, PE3 and PE4) are all attached to P1. It should be noted that if the P-tunnel between the first PE and P devices is implemented through ingress replication and the EVPN is a PBB-EVPN, VXLAN or NVGRE based VPN, the same effect can be achieved without steps <NUM>-<NUM>.

At step <NUM>, the modified routes are sent to the first PE device, as shown in <FIG>. As described above, all the routes from the remaining PE devices can be modified to inform the first PE device of the same information that it has a neighbor, i.e. the first P device. Thus, from the viewpoint of the first PE device, the P-tunnel is only from the first PE device to the first P device. This P-tunnel may be implemented through ingress replication or P2MP tunnels.

In the case of ingress replication, the P-tunnel (MP2P/P2P tunnel) may be created through various techniques such as LDP, Segment Routing and so on. Note that if an MP2P/P2P tunnel has been established between the first PE and P devices, the MP2P/P2P tunnel may be used directly without the creating process. Assume that the ESI-label and the VPN label assigned by the first P device to the first PE device are ESI-A and VPN1 respectively, and the label switched path (LSP) label assigned to the first PE device is LSP1. Then, when a BUM packet needs to be forwarded, the payload of the packet may be encapsulated as shown in <FIG>. The encapsulated packet may be sent to the first P device via the MP2P/P2P tunnel.

In the case of P2MP tunnels, the P2MP tunnel may be created through various techniques such as RSVP-TE, mLDP and so on. Then, when a BUM packet needs to be forwarded, the payload of the packet may be encapsulated. The encapsulation may be similar to the case of ingress replication except that the ESI label and the VPN label are upstream assigned from the first PE device to the first P device. The encapsulated packet may be sent to the first P device via the P2MP tunnel.

Referring back to <FIG>, at step <NUM>, the controlling device configures the first P device such that upon receipt of BUM traffic from the first PE device, the first P device can forward the BUM traffic to the remaining PE devices. In this way, the first P device can act as a replication point for the first PE device. For example, step <NUM> may be implemented as steps <NUM>-<NUM> of <FIG>.

At step <NUM>, the controlling device generates a first forwarding rule that instructs the first P device to, upon receipt of BUM traffic from the first PE device, forward the BUM traffic to the remaining PE devices. For explaining the first forwarding rule, take <FIG> as an example. Assume that an MP2P/P2P tunnel has been established between P1 and each of PE2, PE3 and PE4. Specifically, for the MP2P/P2P tunnel between P1 and PE2, the VPN label and the LSP label are VPN2 and LSP2 respectively. For the MP2P/P2P tunnel between P1 and PE3, the VPN label and the LSP label are VPN3 and LSP3 respectively. For the MP2P/P2P tunnel between P1 and PE4, the VPN label and the LSP label are VPN4 and LSP4 respectively. Furthermore, assume that the controlling device configures the PE devices such that they assign a same ESI label for a same ESI, as shown at block <NUM>. This may be done through various southbound protocols such as NETCONF protocol. Thus, the same ESI label, ESI-A is assigned to P1 from PE2, PE3 and PE4.

Then, once a BUM packet is received on P1, the LSP1 will be popped as the tunnel is terminated. Thus, the label VPN1 appears as the topmost label in the packet. When MPLS based forwarding behavior is employed, the first forwarding rule may be expressed as:.

As shown above, to support split horizon mechanism defined in RFC <NUM> and related protocols, when performing the forwarding, the first P device may always encapsulate the ESI label of the first PE device into the packet forwarded out, no matter the ESI is attached to the first P device or not. It should be noted that the same ESI label (ESI-A) is merely an optional configuration for the purpose of simplifying the forwarding behavior. Optionally, for the same purpose, the controlling device may configure the PE devices such that they assign a same MPLS Label field in PMSI Tunnel Attribute for a same EVPN instance, as shown at block <NUM>. This may also be done through various southbound protocols such as NETCONF protocol. For the case of P2MP tunnels, the first forwarding rule may be generated similarly except that the ESI label and the VPN label are upstream assigned.

At step <NUM>, the first forwarding rule is sent to the first P device, as shown in <FIG>. The first forwarding rule may be sent through various southbound protocols such as network configuration (NETCONF) protocol. As a result, when the first P device receives a BUM packet from the first PE device, the first P device can forward the BUM packet to the remaining PE devices according to the first forwarding rule. This forwarding may be implemented through MP2P/P2P tunnels and/or P2MP tunnels. If the tunnels are not available, they may be created as described above.

In summary, in the solution shown in <FIG>, a controlling device (e.g., an SDN controller) is introduced to EVPN deployments. Under the help of the controlling device, each PE device considers that the P-tunnel is just between one P node and itself. The P node is on behalf of other PE devices in the same EVPN instance. As a result, flexible traffic replication happening inside the network fabric can be accommodated, without functional changes on existing PE devices. Since all controls are located on the controlling device, simple operations and low cost maintaining can be achieved.

<FIG> is a flowchart illustrating a method implemented at a controlling device according to another embodiment of the disclosure. At step <NUM>, the controlling device selects a second P device from the P-tunnel. The P-tunnel may be determined in step <NUM>. For example, a P device located at the downstream of the first P device and the upstream of one or more of the remaining PE devices may be selected from the determined tree as the second P device.

At step <NUM>, the controlling device configures the second P device such that upon receipt of BUM traffic from the first P device, the second P device can forward the BUM traffic to the one or more of the remaining PE devices. For example, step <NUM> may be implemented as steps <NUM>-<NUM>. At step <NUM>, the controlling device determines a second forwarding rule that instructs the second P device to, upon receipt of BUM traffic from the first P device, forward the BUM traffic to the one or more of the remaining PE devices. At step <NUM>, the second forwarding rule is sent to the second P device. Steps <NUM>-<NUM> may be similar to steps <NUM>-<NUM> and thus their detailed description is omitted here. Due to the introduction of the second P device, the P-tunnel planning can be more flexible.

<FIG> is a flowchart illustrating a method implemented at a provider device according to an embodiment of the disclosure. At step <NUM>, the P device receives configuration information from a controlling device in the EVPN. The configuration information instructs the P device to, upon receipt of BUM traffic from a first PE device in an EVPN instance, forward the BUM traffic to one or more of remaining PE devices in the same EVPN instance. The configuration information may be, for example, a forwarding rule which may be implemented as described above with respect to step <NUM>.

At step <NUM>, upon receipt of a packet of BUM traffic from the first PE device, the P device forwards the packet according to the configuration information. As described above, step <NUM> may include step <NUM> in which an ESI label is encapsulated into every packet forwarded out. The forwarding of the BUM packet has been described above and thus its details are omitted here.

It should be noted that the P device described with reference to <FIG> covers the first P device and the second P device described with reference to <FIG> and <FIG>, since the expressions "receive from" and "forward to" may refer to direct and/or indirect communication(s). Specifically, the first P device receives the BUM traffic directly from the first PE device, and forwards the BUM traffic to the remaining PE devices directly or indirectly via additional P device(s). The second P device receives the BUM traffic from the first PE device indirectly via the first P device, and forwards the BUM traffic to one or more of the remaining PE devices directly or indirectly via additional P device(s). It should also be noted that in the figures described above, two steps shown in succession may, in fact, be executed substantially concurrently, or the steps may sometimes be executed in the reverse order, depending upon the functionality involved.

<FIG> is a block diagram showing an apparatus suitable for use in practicing some embodiments of the disclosure. For example, any one of the controlling device and the P device described above may be implemented through the apparatus <NUM>. As shown, the apparatus <NUM> may include a processor <NUM>, a memory <NUM> that stores a program, and a communication interface <NUM> for communicating data with other external devices through wired and/or wireless communication.

The memory <NUM> may be of any type suitable to the local technical environment and may be implemented using any suitable data storage technology, such as semiconductor based memory devices, flash memory, magnetic memory devices and systems, optical memory devices and systems, fixed memory and removable memory. The processor <NUM> may be of any type suitable to the local technical environment, and may include one or more of general purpose computers, special purpose computers, microprocessors, digital signal processors (DSPs) and processors based on multi-core processor architectures, as non-limiting examples.

<FIG> is a block diagram showing a controlling device according to another embodiment of the disclosure. As shown, the controlling device <NUM> comprises a determination module <NUM>, a route reflection module <NUM> and a configuration module <NUM>. The determination module <NUM> is configured to determine a first P device which is to handle BUM traffic from a first PE device in an EVPN instance, on behalf of remaining PE devices in the same EVPN instance. The route reflection module <NUM> is configured to perform route reflection from the remaining PE devices to the first PE device such that the first PE device knows that BUM traffic needs to be forwarded to the first P device. The configuration module <NUM> is configured to configure the first P device such that upon receipt of BUM traffic from the first PE device, the first P device can forward the BUM traffic to the remaining PE devices.

For example, the determination module <NUM> may be configured to perform steps <NUM>-<NUM> of <FIG> and optionally step <NUM> of <FIG>. The route reflection module <NUM> may be configured to perform steps <NUM>, <NUM> and <NUM> of <FIG>. The configuration module <NUM> may be configured to perform steps <NUM>-<NUM> of <FIG> and optionally step <NUM> of <FIG>.

Similar to the controlling device, as another embodiment, the P device may comprise a reception module and a forwarding module. The reception module is configured to receive configuration information from a controlling device in the EVPN. The configuration information instructs the P device to, upon receipt of BUM traffic from a first PE device in an EVPN instance, forward the BUM traffic to one or more of remaining PE devices in the same EVPN instance. The forwarding module is configured to, upon receipt of a packet of BUM traffic from the first PE device, forward the packet according to the configuration information. For example, the forwarding module may be configured to perform step <NUM> of <FIG>.

<FIG> is a block diagram showing a controlling device according to another embodiment of the disclosure. As shown, the controlling device described above may be distributed on a first sub-controlling device and a second sub-controlling device. The first sub-controlling device is configured to determine the first P device as described above. The second sub-controlling device is configured to perform the route reflection and configure the first P device as described above. The first sub-controlling device may be the determination module <NUM> described above. It may be run at server node(s). Optionally, considering load balancing, the determination module <NUM> may also be implemented as several instances, each of which serves for one or more EVPN instances. The second sub-controlling device may be an SDN controller including the route reflection module <NUM> and the configuration module <NUM>.

As still another embodiment, for example, when the EVPN spans across multiple areas, the controlling device described above may be distributed on a plurality of sub-controlling devices each of which is connected to part of the PE devices in the EVPN instance. The plurality of sub-controlling devices may be configured to work together for the PE devices in the EVPN instance. The EVPN routes may be exchanged among these sub-controlling devices through BGP.

Claim 1:
A method implemented at a controlling device in an Ethernet virtual private network, EVPN, the method comprising:
determining (<NUM>) a first provider, P, device which is to handle broadcast, unknown unicast or multicast, BUM, traffic from a first provider edge, PE, device in an EVPN instance, on behalf of remaining PE devices in the same EVPN instance;
performing (<NUM>) route reflection from the remaining PE devices to the first PE device such that the first PE device knows that BUM traffic needs to be forwarded to the first P device; and
configuring (<NUM>) the first P device such that upon receipt of BUM traffic from the first PE device, the first P device can forward the BUM traffic to the remaining PE devices
wherein performing (<NUM>) the route reflection comprises:
receiving (<NUM>), from the PE devices in the EVPN instance, routes related to BUM traffic handling;
modifying (<NUM>) the routes from the remaining PE devices such that upon receipt of the modified routes, the first PE device knows that BUM traffic needs to be forwarded to the first P device; and
sending (<NUM>) the modified routes to the first PE device; and
wherein the route from a remaining PE device is an Inclusive Multicast Ethernet Tag Route; and
wherein modifying (<NUM>) the route comprises:
setting (<NUM>) Originating Router's IP Address field to an IP address of the controlling device;
setting (<NUM>) Next Hop field to an IP address of the first P device; and
replacing (<NUM>) Multi-Protocol Label Switching, MPLS, Label field in PMSI Tunnel Attribute allocated by the remaining PE device, with a corresponding value allocated by the first P device, and
wherein the controlling device act as a Border Gateway Protocol, BGP, route reflector adapted to perform the route reflection.