Patent Description:
Software-defined wide area networks (SD-WANs) represent the application of software-defined networking (SDN) principles to WAN connections, such as connections to cellular networks, the Internet, and Multiprotocol Label Switching (MPLS) networks. The power of SD-WAN is the ability to provide consistent service level agreement (SLA) for important application traffic transparently across various underlying tunnels of varying transport quality and allow for seamless tunnel selection based on tunnel performance characteristics that can match application SLAs.

<CIT> describes a network system for improving network communication performance is provided. The system include at least one client site network component implemented at least at a first client site, the client site network component bonding or aggregating one or more diverse network connections so as to configure a bonded connection that has increased throughput, at least one network server component configured to connect to the client site network component using the bonded connection, the network server component automatically terminating the bonded connection and passing data traffic to the at least one network, a virtual control plane interface at the at least one network server component, and a cloud network controller configured to manage the data traffic, wherein the cloud network controller is operable to configure the virtual control plane interface to provide a priority queue for the data traffic from or to a plurality of client site network components.

<CIT> describes techniques for specifying a backend virtual network for a service load balancer. An orchestrator is configured to receive a service definition for a service implemented by load balancing service traffic for the service among a plurality of backend virtual execution elements, wherein the service definition specifies a first virtual network to use as a backend virtual network for the service, to instantiate, in a selected one of the computing devices, a backend virtual execution element for the service, and to configure, based on the service definition specifying the first virtual network to use as the backend virtual network for the service, a network controller for the virtualized computing infrastructure to configure a load balancer to load balance service traffic to a first virtual network interface, of the backend virtual element, for the first virtual network.

The detailed description set forth below is intended as a description of various configurations of embodiments and is not intended to represent the only configurations in which the subject matter of this disclosure can be practiced. The detailed description includes specific details for the purpose of providing a more thorough understanding of the subject matter of this disclosure. However, it will be clear and apparent that the subject matter of this disclosure is not limited to the specific details set forth herein and may be practiced without these details. In some instances, structures and components are shown in block diagram form in order to avoid obscuring the concepts of the subject matter of this disclosure.

A method for horizontally scaling a network configuration is described in detail below. Entities may exist across different geographical regions, countries, and even continents. In some cases, business entities may want to centralize control plane aspects of their network. Accordingly, methods, systems, and techniques for horizontally a scaling software defined wide area network (SD-WAN) are described below. In some examples, IP addresses of a network controller appliances from a central location are provided over a single network connection to network edge devices in different regions.

Systems, methods, and computer-readable media are provided for scaling a network across different regions or domains. According to at least one example, a method is provided for scaling a network. The method includes: identifying a first connection with a network orchestrator during establishment of a second connection with the network orchestrator from a network controller; establishing a sibling session that links the second connection and the first connection at a control plane; inserting a sibling data message that identifies the sibling session into control messages sent; receiving a message from the network orchestrator over the second connection, the message including an address of the network controller associated with the second connection; and transmitting the second address of the network controller over the first connection to the network orchestrator. The network orchestrator receives the second address of the network controller and associates the first connection and the second connection as a single logical connection. The network orchestrator is configured to transmit the first address and the second address during setup of network edge devices.

In another example, an apparatus for scaling a network across different regions or domains is provided that includes a memory (e.g., configured to store data, such as virtual content data, one or more images, etc.) and one or more processors (e.g., implemented in circuitry) coupled to the memory. The one or more processors are configured to and can: identify a first connection with a network orchestrator during establishment of a second connection with the network orchestrator from a network controller; establish a sibling session that links the second connection and the first connection at a control plane; insert a sibling data message that identifies the sibling session into control messages sent; receive a message from the network orchestrator over the second connection, the message including an address of the network controller associated with the second connection; and transmit the second address of the network controller over the first connection to the network orchestrator. The network orchestrator receives the second address of the network controller and associates the first connection and the second connection as a single logical connection. The network orchestrator is configured to transmit the first address and the second address during setup of network edge devices.

In another example, a non-transitory computer-readable medium is provided that has stored thereon instructions that, when executed by one or more processors, cause the one or more processors to: identify a first connection with a network orchestrator during establishment of a second connection with the network orchestrator from a network controller; establish a sibling session that links the second connection and the first connection at a control plane; insert a sibling data message that identifies the sibling session into control messages sent; receive a message from the network orchestrator over the second connection, the message including an address of the network controller associated with the second connection; and transmit the second address of the network controller over the first connection to the network orchestrator. The network orchestrator receives the second address of the network controller and associates the first connection and the second connection as a single logical connection. The network orchestrator is configured to transmit the first address and the second address during setup of network edge devices.

In some examples, the sibling session is identified based on identifying a universal unique identifier (UUID) of the network orchestrator.

In some examples, the network orchestrator receives the sibling data message and associates the first connection and the second connection as a single logical connection.

In some examples, identification of the first connection occurs during a process challenge phase of a datagram transport layer security (DTLS) session initialization.

In some examples, the network orchestrator informs a network edge device of the second address of the network controller using a first connection.

In some examples, informing the network edge device of the second address of the network controller over the first connection precludes leakage of routes associated with network address translation.

In some examples, the network controller is located in a first region and the network orchestrator is located in a second region that is different from the first region.

In some examples, a network address translator converts IP addresses associated with the second region into IP addresses associated with the first region.

In some examples, the network controller is connected with a second network orchestrator is a different region.

In some examples, the first connection is an IPv4 connection and the second connection is an IPv6 connection, or the first connection is an IPv6 connection and the second connection is an IPv4 connection.

Disclosed are systems, apparatuses, methods, and computer-readable media for managing networks. According to at least one example, a method is provided for connecting to a network controller across different regions. The method includes: receiving a message including a sibling data message that indicates a first connection is related to a second connection, wherein the message is received when the first connection and the second connection are established with a network controller; transmitting a message to the network controller over the second connection, the message including a second address of the network controller associated with the second connection; receiving the address of the network controller associated with the second connection over the first connection from the network controller; linking the first connection and the second connection as a single logical connection; in response to receiving a request to connect a network edge device, transmitting a message to the network edge device over a first connection, the message identifying a first address of the network controller associated with the first connection and the second address.

In another example, an apparatus for connecting to a network controller across different regions is provided that includes a memory (e.g., configured to store data, such as virtual content data, one or more images, etc.) and one or more processors (e.g., implemented in circuitry) coupled to the memory. The one or more processors are configured to and can: receive a message including a sibling data message that indicates a first connection is related to a second connection, wherein the message is received when the first connection and the second connection are established with a network controller; transmit a message to the network controller over the second connection, the message including a second address of the network controller associated with the second connection; receive the address of the network controller associated with the second connection over the first connection from the network controller; link the first connection and the second connection as a single logical connection; in response to receiving a request to connect a network edge device, transmit a message to the network edge device over a first connection, the message identifying a first address of the network controller associated with the first connection and the second address.

In another example, a non-transitory computer-readable medium is provided that has stored thereon instructions that, when executed by one or more processors, cause the one or more processors to: receive a message including a sibling data message that indicates a first connection is related to a second connection, wherein the message is received when the first connection and the second connection are established with a network controller; transmit a message to the network controller over the second connection, the message including a second address of the network controller associated with the second connection; receive the address of the network controller associated with the second connection over the first connection from the network controller; link the first connection and the second connection as a single logical connection; in response to receiving a request to connect a network edge device, transmit a message to the network edge device over a first connection, the message identifying a first address of the network controller associated with the first connection and the second address.

In some examples, one or more of the methods, apparatuses, and computer-readable medium described above further comprise analyzing each connection of each network controller managed by a network orchestrator that is associated with a first type of address and analyzing each connection of each network controller that is associated with a second type of address and that is not linked to a connection associated with the first type of address.

Disclosed are systems, apparatuses, methods, and computer-readable media for managing networks. According to at least one example, a method is provided for controlling network edge devices in different regions. The method includes: a transceiver; a processor configured to execute instructions and cause the processor to: identifying a first connection with a network orchestrator during establishment of a second connection with the network orchestrator from the network controller, establishing a sibling session that links the second connection and the first connection at a control plane, inserting a sibling data message that identifies the sibling session into control messages, receiving a message from the network orchestrator over the second connection, the message including an address of the network controller associated with the second connection, transmitting the second address of the network controller over the first connection to the network orchestrator. The network orchestrator receives the second address of the network controller and associates the first connection and the second connection as a single logical connection. The network orchestrator is also configured to transmit the first address and the second address during setup of network edge devices.

In another example, an apparatus for controlling network edge devices in different regions is provided that includes a memory (e.g., configured to store data, such as virtual content data, one or more images, etc.) and one or more processors (e.g., implemented in circuitry) coupled to the memory. The one or more processors are configured to and can: a transceiver; a processor configured to execute instructions and cause the processor to: identify a first connection with a network orchestrator during establishment of a second connection with the network orchestrator from the network controller, establish a sibling session that links the second connection and the first connection at a control plane, insert a sibling data message that identifies the sibling session into control messages, receive a message from the network orchestrator over the second connection, the message including an address of the network controller associated with the second connection, transmit the second address of the network controller over the first connection to the network orchestrator. The network orchestrator receives the second address of the network controller and associates the first connection and the second connection as a single logical connection. The network orchestrator is also configured to transmit the first address and the second address during setup of network edge devices.

In another example, a non-transitory computer-readable medium is provided that has stored thereon instructions that, when executed by one or more processors, cause the one or more processors to: a transceiver; a processor configured to execute instructions and cause the processor to: identify a first connection with a network orchestrator during establishment of a second connection with the network orchestrator from the network controller, establish a sibling session that links the second connection and the first connection at a control plane, insert a sibling data message that identifies the sibling session into control messages, receive a message from the network orchestrator over the second connection, the message including an address of the network controller associated with the second connection, transmit the second address of the network controller over the first connection to the network orchestrator. The network orchestrator receives the second address of the network controller and associates the first connection and the second connection as a single logical connection. The network orchestrator is also configured to transmit the first address and the second address during setup of network edge devices.

In some examples, the sibling session is identified based on identifying a UUID of the network orchestrator.

In some examples, identification of the first connection occurs during a process challenge phase of a DTLS session initialization.

As noted above, a business entity may want to horizontally scale a network across different geographical regions while centralizing of the management of the network. In some cases, this network may be associated with a different domain (e.g., different network providers) and each domain would have different internet protocol (IP) addresses. Existing management solutions for different domains exist and require additional devices to be present in each domain to implement control plane functions. This causes the business entity to add hardware and complexity to the management of their network. Moreover, an entity has a combination of IPv4 and IPv6 capable equipment, and the mixture of IP addressing adds additional configuration complexity, as well as security issues (e.g., route leakage). Network providers may require that customers do not leak any routes to prevent various malicious network attacks (e.g., man in the middle, etc.).

A method is disclosed in detail below for horizontally scaling a network across different domains. As described with reference to <FIG>, a network controller appliance is configured to provide an IPv4 address and an IPv6 address (e.g., IPv4 connection or IPv6 connection) to network manager appliance at each domain or region, thereby allowing the network manager appliance or provide the IPv4 or IPv6 to the network edge devices that are managed by the single network connection. According, the network edge device can select and connect to the network controller appliance using a suitable interface from different domains or region. The business entity can thereby horizontally scale network locations across different domains and different geographical regions while providing a centralized control plane for managing devices of the network.

<FIG> illustrates an example of a network architecture <NUM> for implementing aspects of the present technology. An example of an implementation of the network architecture <NUM> is the Cisco® SD-WAN architecture. However, one of ordinary skill in the art will understand that, for the network architecture <NUM> and any other system discussed in the present disclosure, there can be additional or fewer component in similar or alternative configurations. The illustrations and examples provided in the present disclosure are for conciseness and clarity. Other embodiments may include different numbers and/or types of elements but one of ordinary skill the art will appreciate that such variations do not depart from the scope of the present disclosure.

In this example, the network architecture <NUM> can comprise an orchestration plane <NUM>, a management plane <NUM>, a control plane <NUM>, and a data plane <NUM>. The orchestration plane <NUM> can assist in the automatic on-boarding of edge network devices <NUM> (e.g., switches, routers, etc.) in an overlay network. The orchestration plane <NUM> can include one or more physical or virtual network orchestrator appliances <NUM>. The network orchestrator appliance(s) <NUM> can perform the initial authentication of the edge network devices <NUM> and orchestrate connectivity between devices of the control plane <NUM> and the data plane <NUM>. In some embodiments, the network orchestrator appliance(s) <NUM> can also enable communication of devices located behind Network Address Translation (NAT). In some embodiments, physical or virtual Cisco® SD-WAN vBond appliances can operate as the network orchestrator appliance(s) <NUM>.

The management plane <NUM> can be responsible for central configuration and monitoring of a network. The management plane <NUM> can include one or more physical or virtual network management appliances <NUM>. In some embodiments, the network management appliance(s) <NUM> can provide centralized management of the network via a graphical user interface to enable a user to monitor, configure, and maintain the edge network devices <NUM> and links (e.g., Internet transport network <NUM>, MPLS network <NUM>, <NUM>/LTE network <NUM>) in an underlay and overlay network. The network management appliance(s) <NUM> can support multi-tenancy and enable centralized management of logically isolated networks associated with different entities (e.g., enterprises, divisions within enterprises, groups within divisions, etc.). Alternatively or in addition, the network management appliance(s) <NUM> can be a dedicated network management system for a single entity. In some embodiments, physical or virtual Cisco® SD-WAN vManage appliances can operate as the network management appliance(s) <NUM>.

The control plane <NUM> can build and maintain a network topology and make decisions on where traffic flows. The control plane <NUM> can include one or more physical or virtual network controller appliance(s) <NUM>. The network controller appliance(s) <NUM> can establish secure connections to each network device <NUM> and distribute route and policy information via a control plane protocol (e.g., Overlay Management Protocol (OMP) (discussed in further detail below), Open Shortest Path First (OSPF), Intermediate System to Intermediate System (IS-IS), Border Gateway Protocol (BGP), Protocol-Independent Multicast (PIM), Internet Group Management Protocol (IGMP), Internet Control Message Protocol (ICMP), Address Resolution Protocol (ARP), Bidirectional Forwarding Detection (BFD), Link Aggregation Control Protocol (LACP), etc.). In some embodiments, the network controller appliance(s) <NUM> can operate as route reflectors. The network controller appliance(s) <NUM> can also orchestrate secure connectivity in the data plane <NUM> between and among the edge network devices <NUM>. For example, in some embodiments, the network controller appliance(s) <NUM> can distribute crypto key information among the edge network device(s) <NUM>. This can allow the network to support a secure network protocol or application (e.g., Internet Protocol Security (IPSec), Transport Layer Security (TLS), Secure Shell (SSH), etc.) without Internet Key Exchange (IKE) and enable scalability of the network. In some embodiments, physical or virtual Cisco® SD-WAN vSmart controllers can operate as the network controller appliance(s) <NUM>.

The data plane <NUM> can be responsible for forwarding packets based on decisions from the control plane <NUM>. The data plane <NUM> can include the edge network devices <NUM>, which can be physical or virtual network devices. The edge network devices <NUM> can operate at the edges various network environments of an organization, such as in one or more data centers or colocation centers <NUM>, campus networks <NUM>, branch office networks <NUM>, home office networks <NUM>, and so forth, or in the cloud (e.g., Infrastructure as a Service (IaaS), Platform as a Service (PaaS), SaaS, and other cloud service provider networks). The edge network devices <NUM> can provide secure data plane connectivity among sites over one or more WAN transports, such as via one or more Internet transport networks <NUM> (e.g., Digital Subscriber Line (DSL), cable, etc.), MPLS networks <NUM> (or other private packet-switched network (e.g., Metro Ethernet, Frame Relay, Asynchronous Transfer Mode (ATM), etc.), mobile networks <NUM> (e.g., <NUM>, <NUM>/LTE, <NUM>, etc.), or other WAN technology (e.g., Synchronous Optical Networking (SONET), Synchronous Digital Hierarchy (SDH), Dense Wavelength Division Multiplexing (DWDM), or other fiber-optic technology; leased lines (e.g., T1/E1, T3/E3, etc.); Public Switched Telephone Network (PSTN), Integrated Services Digital Network (ISDN), or other private circuit-switched network; small aperture terminal (VSAT) or other satellite network; etc.). The edge network devices <NUM> can be responsible for traffic forwarding, security, encryption, quality of service (QoS), and routing (e.g., BGP, OSPF, etc.), among other tasks. In some embodiments, physical or virtual Cisco® SD-WAN vEdge routers can operate as the edge network devices <NUM>.

<FIG> illustrates an example of a network topology <NUM> for showing various aspects of the network architecture <NUM>. The network topology <NUM> can include a management network <NUM>, a pair of network sites 204A and 204B (collectively, <NUM>) (e.g., the data center(s) <NUM>, the campus network(s) <NUM>, the branch office network(s) <NUM>, the home office network(s) <NUM>, cloud service provider network(s), etc.), and a pair of Internet transport networks 160A and 160B (collectively, <NUM>). The management network <NUM> can include one or more network orchestrator appliances <NUM>, one or more network management appliance <NUM>, and one or more network controller appliances <NUM>. Although the management network <NUM> is shown as a single network in this example, one of ordinary skill in the art will understand that each element of the management network <NUM> can be distributed across any number of networks and/or be co-located with the sites <NUM>. In this example, each element of the management network <NUM> can be reached through either transport network 160A or 160B.

Each site can include one or more endpoints <NUM> connected to one or more site network devices <NUM>. The endpoints <NUM> can include general purpose computing devices (e.g., servers, workstations, desktop computers, etc.), mobile computing devices (e.g., laptops, tablets, mobile phones, etc.), wearable devices (e.g., watches, glasses or other head-mounted displays (HMDs), ear devices, etc.), and so forth. The endpoints <NUM> can also include Internet of Things (IoT) devices or equipment, such as agricultural equipment (e.g., livestock tracking and management systems, watering devices, unmanned aerial vehicles (UAVs), etc.); connected cars and other vehicles; smart home sensors and devices (e.g., alarm systems, security cameras, lighting, appliances, media players, HVAC equipment, utility meters, windows, automatic doors, door bells, locks, etc.); office equipment (e.g., desktop phones, copiers, fax machines, etc.); healthcare devices (e.g., pacemakers, biometric sensors, medical equipment, etc.); industrial equipment (e.g., robots, factory machinery, construction equipment, industrial sensors, etc.); retail equipment (e.g., vending machines, point of sale (POS) devices, Radio Frequency Identification (RFID) tags, etc.); smart city devices (e.g., street lamps, parking meters, waste management sensors, etc.); transportation and logistical equipment (e.g., turnstiles, rental car trackers, navigational devices, inventory monitors, etc.); and so forth.

The site network devices <NUM> can include physical or virtual switches, routers, and other network devices. Although the site 204A is shown including a pair of site network devices and the site 204B is shown including a single site network device in this example, the site network devices <NUM> can comprise any number of network devices in any network topology, including multi-tier (e.g., core, distribution, and access tiers), spine-and-leaf, mesh, tree, bus, hub and spoke, and so forth. For example, in some embodiments, one or more data center networks may implement the Cisco® Application Centric Infrastructure (ACI) architecture and/or one or more campus networks may implement the Cisco® Software Defined Access (SD-Access or SDA) architecture. The site network devices <NUM> can connect the endpoints <NUM> to one or more edge network devices <NUM>, and the edge network devices <NUM> can be used to directly connect to the transport networks <NUM>.

In some embodiments, "color" can be used to identify an individual WAN transport network, and different WAN transport networks may be assigned different colors (e.g., mpls, private1, biz-internet, metro-ethernet, lte, etc.). In this example, the network topology <NUM> can utilize a color called "biz-internet" for the Internet transport network 160A and a color called "public-internet" for the Internet transport network 160B.

In some embodiments, each edge network device <NUM> can form a Datagram Transport Layer Security (DTLS) or TLS control connection to the network controller appliance(s) <NUM> and connect to any network control appliance <NUM> over each transport network <NUM>. In some embodiments, the edge network devices <NUM> can also securely connect to edge network devices in other sites via IPSec tunnels. In some embodiments, the BFD protocol may be used within each of these tunnels to detect loss, latency, jitter, and path failures.

On the edge network devices <NUM>, color can be used help to identify or distinguish an individual WAN transport tunnel (e.g., no same color may be used twice on a single edge network device). Colors by themselves can also have significance. For example, the colors metro-ethernet, mpls, and private1, private2, private3, private4, private5, and private6 may be considered private colors, which can be used for private networks or in places where there is no NAT addressing of the transport IP endpoints (e.g., because there may be no NAT between two endpoints of the same color). When the edge network devices <NUM> use a private color, they may attempt to build IPSec tunnels to other edge network devices using native, private, underlay IP addresses. The public colors can include <NUM>, biz, internet, blue, bronze, custom <NUM>, custom2, custom3, default, gold, green, lte, public-internet, red, and silver. The public colors may be used by the edge network devices <NUM> to build tunnels to post-NAT IP addresses (if there is NAT involved). If the edge network devices <NUM> use private colors and need NAT to communicate to other private colors, the carrier setting in the configuration can dictate whether the edge network devices <NUM> use private or public IP addresses. Using this setting, two private colors can establish a session when one or both are using NAT.

<FIG> illustrates an example of a diagram <NUM> showing the operation of OMP, which may be used in some embodiments to manage an overlay of a network (e.g., the network architecture <NUM>). In this example, OMP messages 302A and 302B (collectively, <NUM>) may be transmitted back and forth between the network controller appliance <NUM> and the edge network devices 142A and 142B, respectively, where control plane information, such as route prefixes, next-hop routes, crypto keys, policy information, and so forth, can be exchanged over respective secure DTLS or TLS connections 304A and 304B. The network controller appliance <NUM> can operate similarly to a route reflector. For example, the network controller appliance <NUM> can receive routes from the edge network devices <NUM>, process and apply any policies to them, and advertise routes to other edge network devices <NUM> in the overlay. If there is no policy defined, the edge network devices <NUM> may behave in a manner similar to a full mesh topology, where each edge network device <NUM> can connect directly to another edge network device <NUM> at another site and receive full routing information from each site.

OMP can advertise three types of routes:.

In the example of <FIG>, OMP is shown running over the DTLS/TLS tunnels <NUM> established between the edge network devices <NUM> and the network controller appliance <NUM>. In addition, the diagram <NUM> shows an IPSec tunnel 306A established between TLOC 308A and 308C over the WAN transport network 160A and an IPSec tunnel 306B established between TLOC 308B and TLOC 308D over the WAN transport network 160B. Once the IPSec tunnels 306A and 306B are established, BFD can be enabled across each of them.

<FIG> illustrates an example of a diagram <NUM> showing the operation of VPNs, which may be used in some embodiments to provide segmentation for a network (e.g., the network architecture <NUM>). VPNs can be isolated from one another and can have their own forwarding tables. An interface or sub-interface can be explicitly configured under a single VPN and may not be part of more than one VPN. Labels may be used in OMP route attributes and in the packet encapsulation, which can identify the VPN to which a packet belongs. The VPN number can be a four-byte integer with a value from <NUM> to <NUM>. In some embodiments, the network orchestrator appliance(s) <NUM>, network management appliance(s) <NUM>, network controller appliance(s) <NUM>, and/or edge network device(s) <NUM> can each include a transport VPN <NUM> (e.g., VPN number <NUM>) and a management VPN <NUM> (e.g., VPN number <NUM>). The transport VPN <NUM> can include one or more physical or virtual network interfaces (e.g., network interfaces 410A and 410B) that respectively connect to WAN transport networks (e.g., the MPLS network <NUM> and the Internet transport network <NUM>). Secure DTLS/TLS connections to the network controller appliance(s) <NUM> or between the network controller appliance(s) <NUM> and the network orchestrator appliance(s) <NUM> can be initiated from the transport VPN <NUM>. In addition, static or default routes or a dynamic routing protocol can be configured inside the transport VPN <NUM> to get appropriate next-hop information so that the control plane <NUM> may be established and IPSec tunnels <NUM> (not shown) can connect to remote sites.

The management VPN <NUM> can carry out-of-band management traffic to and from the network orchestrator appliance(s) <NUM>, network management appliance(s) <NUM>, network controller appliance(s) <NUM>, and/or edge network device(s) <NUM> over a network interface 410C. In some embodiments, the management VPN <NUM> may not be carried across the overlay network.

In addition to the transport VPN <NUM> and the management VPN <NUM>, the network orchestrator appliance(s) <NUM>, network management appliance(s) <NUM>, network controller appliance(s) <NUM>, or edge network device(s) <NUM> can also include one or more service-side VPNs <NUM>. The service-side VPN <NUM> can include one or more physical or virtual network interfaces (e.g., network interfaces 410D and 410E) that connect to one or more local-site networks <NUM> and carry user data traffic. The service-side VPN(s) <NUM> can be enabled for features such as OSPF or BGP, Virtual Router Redundancy Protocol (VRRP), QoS, traffic shaping, policing, and so forth. In some embodiments, user traffic can be directed over IPSec tunnels to other sites by redistributing OMP routes received from the network controller appliance(s) <NUM> at the site <NUM> into the service-side VPN routing protocol. In turn, routes from the local site <NUM> can be advertised to other sites by advertising the service VPN routes into the OMP routing protocol, which can be sent to the network controller appliance(s) <NUM> and redistributed to other edge network devices <NUM> in the network. Although the network interfaces 410A-E (collectively, <NUM>) are shown to be physical interfaces in this example, one of ordinary skill in the art will appreciate that the interfaces <NUM> in the transport and service VPNs can also be sub-interfaces instead.

<FIG> illustrates a network configuration that implements a control plane across different domains. In some instances, a network provider may want to horizontally scale network configurations across different geographical regions while centralizing of the management of the network. In this example, the network <NUM> includes a data center <NUM> that includes the management plane and the control plane functions. The data center <NUM> includes at least one network management appliance <NUM> that manages at least one network controller appliance <NUM>.

The network <NUM> is geographically separated into a first region <NUM> and a second region <NUM>, and either region can include the data center <NUM>. For example, the first region <NUM> and second region <NUM> are distinct geographical regions and can also be distinct network operators having different domains. For example, a first network operator may operate and manage the first region <NUM> and a second, different network operator may operate and manage the second region <NUM>. In any event, the data center <NUM> can be associated with either of the different regions and is illustrated separately for clarity.

In this example, the edge network devices <NUM> are managed by physical or virtual network orchestrator appliances <NUM> in each corresponding region. While management and controller functions can be incorporated into each different geographical regions, this would require additional devices (e.g., network controller appliance <NUM>) to be incorporated into that region. To that end, the network operator may prefer to keep management plane and control plane functions at a single location, and then horizontally scale out additional network capacity in different regions without incurring the extra costs associated with additional network devices.

Each connection, which runs as a DTLS tunnel, is established after device authentication succeeds, and carries the encrypted payload between the network controller appliance <NUM> and the edge network device <NUM>. This payload consists of route information necessary for the network controller appliance <NUM> to determine the network topology, and then to calculate the best routes to network destinations and distribute this route information to the edge network device <NUM>. The DTLS connection between a network controller appliance <NUM> and the edge network device <NUM> is a static connection. The network controller appliance <NUM> has no direct peering relationships with any devices that the edge network device <NUM> is connected to on the service side.

This type of solution would require NAT to translate between addresses of one domain (e.g., first region <NUM>) and another domain (e.g., second region <NUM>). In this example, data center <NUM> is presumed to be disposed within the first region <NUM> and would therefore would not require NAT for the first domain. However, a NAT <NUM> is required to communicate between the first region <NUM>, which includes data center <NUM>, and the second region <NUM>. The NAT <NUM> translates addresses within the second region into addresses associated with the first region to allow edge network devices <NUM> to communicate with the network controller appliance <NUM>. While the NAT <NUM> illustrated in <FIG> is not specifically located in any region, the NAT <NUM> can be configured in the first region <NUM>, the second region <NUM>, or between the different regions.

The various network devices illustrated in <FIG> can be capable of IPv4 and/or IPv6 communication. A device that includes both IPv4 and IPv6 interfaces is referred to as a dual-stack configuration. However, a network controller appliance <NUM> may only be capable of serving a single interface (e.g., IPv4 or IPv6) of each client device. As a result, IPv4 or IPv6 routes can be leaked using the NAT <NUM>. In some cases, a network operator may require that no routes are leaked to prevent malicious man-in-the-middle attacks and other security precautions. For example, there can be security and/or regional regulatory considerations that prevent routes from being leaked. To that end, this solution in <FIG> requires additional configuration to prevent IPv4 or IPv6 routes from leaking.

<FIG> illustrates an example sequence diagram <NUM> to prevent IPv4 or IPv6 route leakage using the network configuration illustrated in <FIG>. As will be described below, different IP addresses of a network controller appliance <NUM> will be relayed over a single connection (e.g., IPv4 or IPv6) to prevent route leakage. The address information will be relayed from a network controller appliance <NUM> to a network orchestrator appliance <NUM>, which will provide that information to an edge network device <NUM>.

After a network controller appliance <NUM> has configured a first connection (e.g., an IPv4 connection) with a network orchestrator appliance <NUM>, the network controller appliance <NUM> may request setup of an additional DTLS connection for a second address (e.g., an IPv6 address). During the DTLS challenge phase, the network controller appliance <NUM> may identify that a connection (e.g., the IPv4 connection) exists and identifies that the IPv4 connection is a sibling session at block <NUM>. The network controller appliance <NUM> thereby determines to insert a sibling session field (e.g., a type length value (TLV)) into control messages that are transmitted to the network orchestrator appliance <NUM> at block <NUM>.

The network orchestrator appliance <NUM> receives a control message, analyzes the control message, and identifies the sibling session field. The network orchestrator appliance <NUM> transmits an IPv6 address of the network controller appliance <NUM> as perceived by the network orchestrator appliance <NUM> to the network controller appliance <NUM> at block <NUM>. Accordingly, the IPv6 address that is transmitted is the address that is perceived by the network orchestrator appliance <NUM> because the network controller appliance <NUM> is located behind a NAT.

The network controller appliance <NUM> identifies the corresponding IPv4 session and returns the IPv6 address of the network controller appliance <NUM> to the network orchestrator appliance <NUM> at block <NUM> using the IPv4 connection. At block <NUM>, the network orchestrator appliance <NUM> and the network controller appliance <NUM> associate the IPv4 address and the IPv6 address of the network controller appliance <NUM>. In some examples, the IPv6 address of the network controller appliance <NUM> can be associated in the network orchestrator appliance with a Boolean value that indicates that the IPv6 address is related to another IPv4 address, but does not have to explicitly identify which IPv4 address. As will be described below, this Boolean value will allow the network orchestrator appliance to understand that this IPv6 address is associated with another address and can be skipped. In other examples, the IPv6 address and the IPv4 address can be expressly linked via a pointer or some other data structure.

At block <NUM>, an edge network device <NUM> may be activated and may request identification network controller appliances from the network orchestrator appliance <NUM>. This may occur when an edge network device <NUM> is activated (e.g., booted, restarted, etc.), and requests identification of the network controller appliances to identify a network controller appliance to handle control communication via the Overlay Management Protocol (OMP).

At block <NUM>, the network orchestrator appliance <NUM> searches for network controller appliances to identify to the edge network device <NUM>. During the search, the network orchestrator appliance <NUM> only considers each network controller appliance a single time, even if the network controller appliance is a dual-stack device and includes both an IPv4 address and an IPv6 address. That is, the network orchestrator appliance <NUM> prevents double counting of the network controller appliances.

The network orchestrator appliance <NUM> sends a response identifying at least one network controller appliance. When the network controller appliance includes an IPv4 address and an IPv6 address, the response identifies both the IPv4 address and the IPv6 address at block <NUM>. In this example, communication with the network controller appliance <NUM> is restricted in the IPv6 domain. Therefore, the network orchestrator appliance <NUM> transmits the message identifying both IPv4 and IPv6 addresses using an IPv4 connection at block <NUM>.

Consequently, the edge network device <NUM> receives both the IPv4 address and the IPv6 address of the network controller appliance <NUM> and is able to configure a suitable network connection with the network controller appliance <NUM>. That is, the edge network device <NUM> receives identification of IPv4 and IPv6 addresses and can select the appropriate connection with the network controller appliance. Accordingly, because the data center <NUM> is presumed to be located in the first region <NUM>, the devices located in second region <NUM> would be able to connect to the network devices in the first region <NUM> for control plane and management plane functions. This configuration allows horizontal scaling of additional domains and geographical regions and prevents leaking of routes, while allowing the network edge devices <NUM> to configure the optimal connection to the network controller appliances <NUM>.

While the examples described above in <FIG> and <FIG> are described to constrain messages to the IPv4 domain, the descriptions are equally applicable to IPv6 and constraining communications in the IPv4 domain. For example, the IPv4 addresses of the network controller appliances <NUM> can be transmitted to the network orchestrator appliances over the IPv6 connection to prevent IPv4 route leakage.

<FIG> illustrates an example method <NUM> for a network controller appliance. Although the example method <NUM> depicts a particular sequence of operations, the sequence may be altered without departing from the scope of the present disclosure. For example, some of the operations depicted may be performed in parallel or in a different sequence that does not materially affect the function of the method <NUM>. In other examples, different components of an example device or system that implements the method <NUM> may perform functions at substantially the same time or in a specific sequence.

According to some examples, the method <NUM> includes, when a second connection is requested with the network orchestrator, identifying a first connection with the network orchestrator at block <NUM>. In some examples, the identification of the first connection occurs during a challenge phase of a DTLS session initialization between the network orchestrator and the network controller that are located in different regions. As an example, the processor <NUM> illustrated in <FIG> may identify a first connection with the network orchestrator during establishment of a second connection with the network orchestrator from the network controller.

For purposes of explanation of this example, the first connection will be presumed to be an IPv4 connection for discussion and the second connection will be presumed to be an IPv6 connection. However, the first connection can be an IPv6 connection and the second connection can be an IPv4 connection.

Because the network orchestrator and the network controller are located in different regions, the network orchestrator and the network controller may be associated with different domains and a network address translator may be implemented to translate the IP addresses. In some examples, as described below with reference to <FIG>, the network controller can be connected to multiple network orchestrators.

According to some examples, the method <NUM> includes establishing a sibling session that links the IPv6 connection and the IPv4 connection at a control plane at block <NUM>. By linking the sessions, this may preclude double counting of the network connection by the network controller and network orchestrator, which can prevent a device from finding an optimal network connection. To that end, the processor <NUM> may use a universal unique identifier (UUID) of the network orchestrator to find and establish a sibling session that links the IPv6 connection and the IPv4 connection at a control plane.

According to some examples, the method <NUM> includes inserting a sibling data field that identifies the sibling session into control messages at block <NUM>. For example, the processor <NUM> may insert a sibling data message that identifies the sibling session into control messages. The network orchestrator receives the sibling data message and associates the IPv4 connection and the IPv6 connection as a single logical connection.

In response to the control message, the method <NUM> may receive a message from the network orchestrator over the IPv6 connection that includes an address of the network controller associated with the IPv6 connection at block <NUM>. The received address is the address of the network controller that is perceived by the network orchestrator (i.e., the translated IP address). Notably, the network controller is unaware of its perceived public IP addresses and therefore must receive the address from an external source. Therefore, the processor <NUM> may receive a message from the network orchestrator over the IPv6 connection that includes the address of the network controller associated with the IPv6 connection.

According to some examples, the method <NUM> includes transmitting the address of the network controller over a different connection to the network orchestrator at block <NUM>. For example, the processor <NUM> may transmit the IPv6 address of the network controller over the IPv4 connection to the network orchestrator. The network orchestrator receives the IPv6 address of the network controller and associates the IPv4 connection and the IPv6 connection as a single logical connection. The network orchestrator is configured to transmit the IPv4 address and the IPv6 address during setup of edge network devices using a single interface, and the edge network devices can select the ideal interface for the network connection. However, in some example, the network orchestrator informs a network edge device of the IPv6 address of the network controller using an IPv4 connection.

Accordingly, the method <NUM> allows the network orchestrator to provide both IPv4 and IPv6 addresses to the network edge devices using a single conenction. In this example, informing the network edge device of the IPv6 address of the network controller over the IPv4 connection precludes leakage of routes associated with network address translation because communications with a network edge device will constrain communications with the network controller to a single interface.

<FIG> illustrates an example method <NUM> for a network orchestrator. Although the example method <NUM> depicts a particular sequence of operations, the sequence may be altered without departing from the scope of the present disclosure. For example, some of the operations depicted may be performed in parallel or in a different sequence that does not materially affect the function of the method <NUM>. In other examples, different components of an example device or system that implements the method <NUM> may perform functions at substantially the same time or in a specific sequence.

According to some examples, the method <NUM> includes receiving a message including a sibling data field that indicates the first connection is related to the second connection at block <NUM>. The sibling data field is a TLV that is inserted by the network controller and indicates that an existing connection with the network controller exists. For example, the processor <NUM> illustrated in <FIG> may, while creating an IPv6 session, receive a message including a sibling data message that indicates an IPv4 connection is related to the IPv6 connection.

According to some examples, the method <NUM> includes transmitting a message to the network controller over the second connection at block <NUM>. This message may include a second address of the network controller that is associated with the second connection. For example, presuming that an IPv4 session exists, the processor <NUM> may transmit a message to the network controller over the IPv6 connection that includes an IPv6 address of the network controller that is perceived by the network orchestrator. As noted above, the network controller is unaware of its IPv6 address in different regions due to NAT and therefore the network orchestrator provides the IPv6 NAT address of the network controller to the network controller.

According to some examples, the method <NUM> includes receiving the address of the network controller associated with the second connection over the first connection from the network controller at block <NUM>. For example, to cause the network orchestrator to link the IPv4 and IPv6 sessions, the processor <NUM> may receive the IPv6 NAT address of the network controller over the IPv4 connection from the network controller.

According to some examples, the method <NUM> links the first connection and the second connection as a single logical connection at block <NUM>. For example, the processor <NUM> may link the IPv4 connection and the IPv6 connection as a single logical connection.

According to some examples, the network orchestrator may receive a request to connect a network edge device. Accordingly, the processor <NUM> of the network orchestrator may search for network controllers to identify to the edge network device. Accordingly, the network orchestrator may analyze each edge network device based on the IPv4 address. However, the network orchestrator may analyze each network edge devices based on the IPv6 address when the IPv6 address and IPv4 address are not linked. That is, the network orchestrator analyzes each network controller a single time.

The method <NUM> may further include, in response to receiving the request to connect a network edge device, transmitting a message to the network edge device over a IPv4 connection at block <NUM>. The message may identify a first address of the network controller associated with the first connection and the second address. For example, the processor <NUM> may, in response to receiving a request to connect a network edge device, transmit a message to the network edge device over an IPv4 connection that identifies the IPv4 address and the IPv6 address of the network controller.

<FIG> illustrates an example method <NUM> of a network edge device for connecting to a network controller. Although the example method <NUM> depicts a particular sequence of operations, the sequence may be altered without departing from the scope of the present disclosure. For example, some of the operations depicted may be performed in parallel or in a different sequence that does not materially affect the function of the method <NUM>. In other examples, different components of an example device or system that implements the method <NUM> may perform functions at substantially the same time or in a specific sequence.

According to some examples, the method <NUM> includes transmitting a request to a network orchestrator for an address of a network controller to manage the network edge device over a first connection at block <NUM>. For example, the processor <NUM> may transmit a request to a network orchestrator for an address of a network controller to manage the network edge device over an IPv4 connection.

According to some examples, the method <NUM> includes receiving a message identifying a first address of the network controller and an second network address of the network controller at block <NUM>. For example, the processor <NUM> may receive a message identifying an IPv4 address of the network controller and an IPv6 network address of the network controller.

According to some examples, the method <NUM> includes determining to connect to a network controller via the first address or the second address at block <NUM>. For example, the processor <NUM> may determine to connect to the network controller via an IPv4 address or an IPv6 address.

<FIG>, <FIG> are block diagrams illustrating communications of the network in different regions or domains. In particular, <FIG> illustrates a network controller <NUM> and a network orchestrator <NUM> are in communication over an IPv4 connection and a IPv6 connection. Although not shown, the network controller <NUM> and network orchestrator <NUM> are located in different geographical regions and associated with different domains. The IPv6 connection is translated using a NAT <NUM> to prevent route leakage. The network controller <NUM> and the network orchestrator <NUM> exchange UUIDs over the IPv4 connection at step <NUM> and UUIDs over the IPv6 connection at step <NUM>.

<FIG> illustrates that a sibling session <NUM> is formed that causes the IPv4 and IPv6 connections to be treated as a single logical connection.

<FIG> illustrates that that the network orchestrator transmits the IPv6 address of the network controller <NUM>, as perceived by the network orchestrator <NUM> at interface If<NUM>, is transmitted to the network controller <NUM> at step <NUM>.

<FIG> illustrates that the IPv6 address of the network controller <NUM>, as perceived by the network orchestrator <NUM>, is transmitted to the network orchestrator <NUM> over the IPv4 connection at step <NUM>. Once received by the network orchestrator <NUM>, the network orchestrator <NUM> associates the IPv4 and IPv6 addresses and treats the IPv4 and IPv6 connections as a single logical connection.

<FIG> illustrates that an edge device <NUM> requests a network controller from the network orchestrator <NUM>. At step <NUM>, the network orchestrator sends the IPv4 and IPv6 addresses of the network controller <NUM> over an IPv4 connection to the edge device. The edge device <NUM> may communicate with the network controller <NUM> using an IPv4 connection or using an IPv6 connection with the NAT <NUM>.

<FIG> illustrates that a second network orchestrator <NUM>, a second NAT <NUM>, and a second edge device <NUM> can be implemented in a different region or different domain to horizontally scale the network while centralizing network management functions. That is, the network controller <NUM> can be implemented to control both edge device <NUM> and edge device <NUM>, even though both are in different domains.

<FIG> shows an example of computing system <NUM>, which can be for example any computing device making up network orchestrator appliance <NUM>, network controller appliance <NUM>, edge network device <NUM>, or any component thereof in which the components of the system are in communication with each other using connection <NUM>. Connection <NUM> can be a physical connection via a bus, or a direct connection into processor <NUM>, such as in a chipset architecture. Connection <NUM> can also be a virtual connection, networked connection, or logical connection.

In some embodiments computing system <NUM> is a distributed system in which the functions described in this disclosure can be distributed within a datacenter, multiple datacenters, a peer network, etc. In some embodiments, one or more of the described system components represents many such components each performing some or all of the function for which the component is described. In some embodiments, the components can be physical or virtual devices.

Example system <NUM> includes at least one processing unit (CPU or processor) <NUM> and connection <NUM> that couples various system components including system memory <NUM>, such as read only memory (ROM) <NUM> and random access memory (RAM) <NUM> to processor <NUM>. Computing system <NUM> can include a cache of high-speed memory <NUM> connected directly with, in close proximity to, or integrated as part of processor <NUM>.

Processor <NUM> can include any general purpose processor and a hardware service or software service, such as services <NUM>, <NUM>, and <NUM> stored in storage device <NUM>, configured to control processor <NUM> as well as a special-purpose processor where software instructions are incorporated into the actual processor design. Processor <NUM> may essentially be a completely self-contained computing system, containing multiple cores or processors, a bus, memory controller, cache, etc. A multi-core processor may be symmetric or asymmetric.

To enable user interaction, computing system <NUM> includes an input device <NUM>, which can represent any number of input mechanisms, such as a microphone for speech, a touch-sensitive screen for gesture or graphical input, keyboard, mouse, motion input, speech, etc. Computing system <NUM> can also include output device <NUM>, which can be one or more of a number of output mechanisms known to those of skill in the art. In some instances, multimodal systems can enable a user to provide multiple types of input/output to communicate with computing system <NUM>. Computing system <NUM> can include communications interface <NUM> (e.g., a transceiver), which can generally govern and manage the user input and system output. There is no restriction on operating on any particular hardware arrangement and therefore the basic features here may easily be substituted for improved hardware or firmware arrangements as they are developed.

Storage device <NUM> can be a non-volatile memory device and can be a hard disk or other types of computer readable media which can store data that are accessible by a computer, such as magnetic cassettes, flash memory cards, solid state memory devices, digital versatile disks, cartridges, random access memories (RAMs), read only memory (ROM), and/or some combination of these devices.

The storage device <NUM> can include software services, servers, services, etc., that when the code that defines such software is executed by the processor <NUM>, it causes the system to perform a function. In some embodiments, a hardware service that performs a particular function can include the software component stored in a computer-readable medium in connection with the necessary hardware components, such as processor <NUM>, connection <NUM>, output device <NUM>, etc., to carry out the function.

Any of the steps, operations, functions, or processes described herein may be performed or implemented by a combination of hardware and software services or services, alone or in combination with other devices. In some embodiments, a service can be software that resides in memory of a client device and/or one or more servers of a content management system and perform one or more functions when a processor executes the software associated with the service. In some embodiments, a service is a program, or a collection of programs that carry out a specific function. In some embodiments, a service can be considered a server. The memory can be a non-transitory computer-readable medium.

Examples of computer-readable media that may be used to store instructions, information used, and/or information created during methods according to described examples include magnetic or optical disks, solid state memory devices, flash memory, USB devices provided with non-volatile memory, networked storage devices, and so on.

Typical examples of such form factors include servers, laptops, smart phones, small form factor personal computers, personal digital assistants, and so on.

Claim 1:
A method (<NUM>), comprising:
identifying (<NUM>) a first connection with a network orchestrator during establishment of a second connection with the network orchestrator from a network controller;
establishing (<NUM>) a sibling session that links the second connection and the first connection at a control plane;
inserting (<NUM>) a sibling data message that identifies the sibling session into control messages sent;
receiving (<NUM>) a message from the network orchestrator over the second connection, the message including an address of the network controller associated with the second connection; and
transmitting (<NUM>) the second address of the network controller over the first connection to the network orchestrator, wherein the network orchestrator receives the second address of the network controller and associates the first connection and the second connection as a single logical connection,
wherein the network orchestrator is configured to transmit the first address and the second address during setup of network edge devices.