Patent Description:
<CIT> relates to 'Validation of layer <NUM> using virtual routing forwarding containers in a network'. <CIT> relates to an 'SDN-based API controller'.

In a typical cloud data center environment, a large collection of interconnected servers often provide computing and/or storage capacity to run various applications. For example, a data center may comprise a facility that hosts applications and services for subscribers, i.e., customers of data center. The data center may, for example, host all of the infrastructure equipment, such as networking and storage systems, redundant power supplies, and environmental controls. In a typical data center, clusters of storage systems and application servers are interconnected via high-speed switch fabric provided by one or more tiers of physical network switches and routers. More sophisticated data centers provide infrastructure spread throughout the world with subscriber support equipment located in various physical hosting facilities.

A cloud computing infrastructure that manages deployment and infrastructure for application execution may involve two main roles: (<NUM>) orchestration-for automating deployment, scaling, and operations of applications across clusters of hosts and providing computing infrastructure, which may include virtual machines (VMs) or container-centric computing infrastructure; and (<NUM>) network management-for creating virtual networks in the network infrastructure to enable communication among applications running on virtual execution environments, such as containers or VMs, as well as among applications running on legacy (e.g., physical) environments. Software-defined networking contributes to network management.

Multi-cloud environment refers to the use of multiple clouds for computing and storage services. An enterprise may utilize an on-premise computing and/or storage service (e.g., on-premises cloud), and one or more off-premise clouds such as those hosted by third-party providers. Examples of the clouds include private, public, or hybrid public/private clouds that allow for ease of scalability while allowing different levels of control and security. An enterprise may utilize one or more of private, public, or hybrid public/private clouds based on the types of applications that are executed and other needs of the enterprise.

This disclosure describes techniques for configuring software defined network (SDN) controllers within different cloud computing domains and, in particular, a multi-cluster controller that operates and presents, in some examples, a single interface for seamlessly controlling and configuring SDN controllers in different cloud computing domains. In some examples, the techniques include the multi-cluster command controller that operates to transparently proxy configuration requests, issued by one or more users or administrators, to service provided by SDN controllers (referred to herein as endpoints) across a plurality of clusters within a network. In some examples, such techniques may include use of a proxy system that receives configuration requests from administers, parses a given configuration request to identify a cluster and a particular service offered by the SDN controller of the cluster, i.e., the endpoint of the SDN controller, to which the configurations are to be applied, and routes information about the configuration request to the appropriate endpoint. Such techniques may further include appropriately authenticating users, which may include storing, within the proxy system information about authentication credentials that may be associated with a user for a particular cluster or endpoint. Such techniques may also include dynamically maintaining a database of cluster objects and/or objects within a cluster as configurations involving endpoints and clusters are performed or as endpoints and clusters are otherwise managed.

The techniques described herein may provide certain technical advantages. For instance, a system that operates to proxy configuration traffic across any number of clusters may enable efficient multi-cluster configuration of endpoints and related objects, in some examples using only a single controller with a single set of authentication credentials for each user. Further, by including, within each configuration request, information (e.g., such as a prefix) that enables a proxy system to identify the endpoint that the configuration request pertains to, the proxy system may be able to efficiently route configuration requests to the appropriate endpoint. Further, by maintaining prefix and cluster information in a data store or a cache, a system that proxies requests across multiple clusters may operate with little or no additional latency as compared to directly configuring endpoints without a proxy.

In some examples, this disclosure describes operations performed by a computing system capable of communicating with a plurality of clusters in accordance with one or more aspects of this disclosure. In one specific example, this disclosure describes a method comprising receiving, by a computing system, a first request and a second request, wherein the computing system is capable of communicating with each of a plurality of clusters, each of the plurality of clusters including a plurality of endpoints; identifying, by the computing system and based on the first request, a first cluster associated with the first request; identifying, by the computing system and based on the first request, a first endpoint associated with the first request, wherein the first endpoint is within the first cluster; identifying, by the computing system and based on the second request, a second cluster associated with the second request; identifying, by the computing system and based on the second request, a second endpoint associated with the second request, wherein the second endpoint is within the second cluster; communicating, by the computing system, with the first endpoint to perform a first configuration operation specified by the first request; communicating, by the computing system, with the second endpoint to perform a second configuration operation specified by the second request; and updating, by the computing system, a data store to include information about the first configuration operation and the second configuration operation.

In another specific example, this disclosure describes a system comprising a plurality of clusters, each of the plurality of clusters including a plurality of configurable endpoints; a storage system; and processing circuitry having access to the storage system and capable of communicating with each of the plurality of configurable endpoints, wherein the processing circuitry is configured to: receive a plurality of requests, each specifying a configuration operation; identify, for each of the requests, a configuration cluster from among the plurality of clusters; identify, for each of the requests, a configuration endpoint within the configuration cluster; perform, for each of the requests, the specified configuration operation by communicating with the identified configuration endpoint within the identified configuration cluster so that performing the configurations operations associated with the plurality of requests includes performing configuration operations across each of the plurality of clusters; and update, for each of the requests, the storage system to include information about performing the specified configuration operation.

In another example, this disclosure describes a computer-readable medium comprising instructions for causing processing circuitry to perform operations comprising: receiving a plurality of requests, each specifying a configuration operation, identifying, for each of the requests, a configuration cluster from among the plurality of clusters, identifying, for each of the requests, a configuration endpoint of the respective SDN controller within the configuration cluster, performing, for each of the requests, the specified configuration operation by communicating with the identified configuration endpoint within the identified configuration cluster so that performing the configurations operations associated with the plurality of requests includes performing configuration operations across each of the plurality of clusters, and updating, for each of the requests, the storage system to include information about performing the specified configuration operation.

The foregoing is a simplified summary to provide background for some aspects of the disclosure, and is neither intended to identify key or critical elements of the disclosure nor to delineate or limit the scope of the disclosure. Instead, the foregoing merely presents some concepts in a simplified form as a prelude to the description below.

<FIG> is a conceptual diagram illustrating an example network in which multiple clusters may be configured, in accordance with one or more aspects of the present disclosure. The example of <FIG> illustrates a computing system or controller <NUM> interacting with one or more of software defined networks (SDNs) arranged as cloud-computing cluster 130A, cluster 130B, and cluster 130C (collectively, "clusters <NUM>," and representing any number of clusters). Each of cloud-computing clusters <NUM> is implemented by computing infrastructure that may be virtualized to support one or services implemented by the cluster. For instance, one or more of clusters <NUM> may be provisioned on a plurality of servers hosted on a network (e.g., Internet) to store, manage, and process data, or perform other functions.

In some examples, one or more of clusters <NUM> may be on-premises of an enterprise, where some or all of other clusters <NUM> are remote. In other examples, some or all of clusters <NUM> may be remote from the enterprise. Further, in some examples, clusters <NUM> may all be included within a single data center. In still other examples, each of clusters <NUM> may be deployed within its own data center, or possibly, one or more of clusters <NUM> may span multiple data centers or geographic regions.

In the example of <FIG>, controller <NUM> may receive configuration requests from a computing device operated by administrator <NUM> (or other appropriately authorized user) communicating with controller <NUM> either directly or over a network. <FIG> includes a plurality of software defined network controllers, referred to herein as virtual network controller 136A, 136B, and 136C (collectively, "virtual network controllers <NUM>") each within clusters 130A, 130B, and cluster 130C, respectively. Each of virtual network controllers <NUM> configure aspects of their respective cluster <NUM>, and may be implemented through a computing device and/or processing circuitry, whether physical or virtual. Further example details of a VNC <NUM> operating as a software defined network controller to configure overlay and/or underlay network elements within a computing domain are described in <CIT>, <CIT>, and <CIT>.

In some examples, each of virtual network controllers 136A may include or be implemented by one or more configurable services referred to herein as endpoints of the SDN controller. Virtual network controller 136A in the example of <FIG> is shown implemented by, composed by, or including endpoint 137A-<NUM> through and endpoint 137A-N (collectively "endpoints 137A" or "endpoints <NUM>" and representing any number of endpoints). Although not specifically shown in <FIG>, virtual network controller 136B may also be implemented by or composed by a number of endpoints (e.g., endpoint 137B-<NUM> through endpoint 137B-N, or collectively, "endpoints 137B"). Similarly, virtual network controller 136C may be implemented with using a number of endpoints (e.g., endpoint 137C-<NUM> through endpoint 137C-N, or collectively "endpoints 137C").

In each of clusters <NUM>, endpoints <NUM> may represent a different service offered or performed by the respective VNC of that cluster <NUM>. In some examples, each of endpoints <NUM> may be configurable through an API (application programming interface) exposed by the corresponding endpoint <NUM>. Endpoints <NUM> may provide any of a number of different types of services for managing an overlay and/or underlay network of the respective cloud-computing domain <NUM>, including authentication (e.g., OpenStack's Keystone service), image management (e.g., OpenStack's Glance service), storage (e.g., OpenStack's Swift service), analytics, telemetry, or other services, each provided through one or more endpoints <NUM>. In some examples, each of endpoints <NUM> of VNC 136A within cluster 130A (or within clusters <NUM> generally) operates as a different service that can be configured, such as a different process, virtual machine, container, or the like, for implementing the functions of the SDN controller. Each of clusters <NUM> further include a corresponding network <NUM> and any number of servers (e.g., servers 34A, 34B, and 34C) for providing compute resources. In general, each of components illustrated in <FIG> (e.g., computing systems <NUM>, clusters <NUM>, virtual network controllers <NUM> within each of <NUM>, and servers <NUM> within each of clusters <NUM>) may communicate over one or more networks, which may be or include the internet or any public or private communications network or other network. Such networks may include one or more of networks <NUM> within clusters <NUM>.

To enable configuration of aspects of virtual network controller 136A (or any of endpoints 137A included within virtual network controller 136A), virtual network controller 136A exposes an API that may be accessible (e.g., through a web browser interface) to an authenticated administrator (e.g., administrator <NUM>) operating a client computing device. In some examples, each of endpoints 137A within virtual network controller 136A may expose its own API to enable configuration of the service corresponding to that endpoint 137A. Administrator <NUM> may also separately configure virtual network controller 136B or aspects of any of endpoints 137B by using a client computing device to authenticate and then access an API exposed by virtual network controller 136B or any of endpoints 137B. Similarly, administrator <NUM> may also separately configure aspects of virtual network controller 136C or any of endpoints 137C by authenticating and accessing an API exposes by virtual network controller 136C or any of endpoints 137C.

Rather than managing and configuring each of virtual network controllers <NUM> (or endpoints <NUM>) separately, controller <NUM> may, as described herein, enable an administrator to manage and/or configure involving any of virtual network controllers <NUM> or endpoints <NUM> from a centralized device, or from a single point of contact. In some examples, controller <NUM> may serve as a dynamic proxy that provides a single point of contact to manage aspects of multiple clusters <NUM>. Controller <NUM> may be included within cluster 130A (as shown in <FIG>), but in other examples, controller <NUM> may be located elsewhere, within another one of clusters <NUM>, distributed across multiple clusters <NUM>, or outside of all clusters <NUM>. As further described herein, administrator <NUM> may manage one or more of clusters <NUM> by issuing configuration requests to controller <NUM>, and controller <NUM> may proxy the requests to one or more of clusters <NUM>, where the configurations are performed. One or more systems included within each of clusters <NUM> may respond to or otherwise communicate with controller <NUM>, and controller <NUM> may use information derived from those communications to generate a user interface for presentation to administrator <NUM> (i.e., to a computing device operated by administrator <NUM>). In addition, controller <NUM> may operate dynamically by detecting or sensing configuration changes involving one or more clusters <NUM>, and updating a data store of information about each of clusters <NUM>. In some examples, a cache may be used for storing some of the information included within the data store, to thereby reduce latency that might otherwise arise when performing configurations through controller <NUM>, rather than directly through one or more of virtual network controllers <NUM>.

In accordance with one or more aspects of the present disclosure, controller <NUM> may manage or configure one or more aspects of one or more clusters <NUM>. For instance, in an example that can be described with reference to <FIG>, controller <NUM> detects input from a computing device operated by administrator <NUM> and determines that administrator <NUM> is an authenticated user. Controller <NUM> detects further input and determines that the input corresponds to a request to configure one or more aspects of virtual network controller 136A within cluster 130A. Specifically, controller <NUM> determines that the input includes information identifying cluster 130A and endpoint 137A-<NUM> within cluster 130A and an indication of the configuration operation to be performed on endpoint 137A-<NUM>. Controller <NUM> communicates with endpoint 137A-<NUM> to perform the configuration operation specified by the input or otherwise manage endpoint 137A-<NUM>. In some examples, the configuration operation may involve management of an existing one of clusters <NUM>. In other examples, the configuration operation may involve creating a new cluster and associated endpoints within that new cluster.

In some examples, controller <NUM> accesses, upon receiving a configuration request, a data store (not shown in <FIG>) that includes information about clusters <NUM> and endpoints <NUM>. Controller <NUM> may use information accessed within the data store to identify the specific cluster <NUM> and/or endpoint <NUM> to be configured, and also to route the configuration request to the appropriate endpoint <NUM> and the appropriate cluster <NUM>. Controller <NUM> may update the data store as endpoints <NUM> are managed or as configurations are performed. Controller <NUM> may also update the data store when controller <NUM> otherwise detects configurations being performed, thereby dynamically updating the data store. Controller <NUM> may also maintain a cache of information from the data store (e.g., as a key-value store of endpoint information) to enable controller <NUM> to quickly identify the appropriate endpoint <NUM> and cluster <NUM> for a given configuration request.

Through techniques in accordance with one or more aspects of the present disclosure, such as by implementing controller <NUM> as a proxy for configuring clusters <NUM>, network <NUM> may enable configuration of multiple clusters <NUM> through a single controller, and using a single set of authentication credentials. Such an implementation may result in a more efficient way of configuring multiple clusters <NUM> because administering multiple clusters <NUM> may be performed without accessing multiple systems independently.

Further, by dynamically maintaining information about multiple clusters in a data store included within controller <NUM>, controller <NUM> may efficiently identify, for a given configuration request received from administrator <NUM>, which of endpoints <NUM> across multiple clusters <NUM> are being managed. By identifying the appropriate endpoint <NUM> associated with a given configuration request, controller <NUM> may efficiently route the configuration request to the appropriate cluster <NUM> and the appropriate endpoint <NUM> within that cluster <NUM>. Further, by caching information about endpoints <NUM>, controller <NUM> may perform techniques described herein while introducing little or no latency.

<FIG> is a block diagram illustrating an example network that dynamically proxies configuration requests to one or more clusters in a multi-cluster environment, in accordance with one or more aspects of the present disclosure. Network <NUM> of <FIG> may be described as an example or alternative implementation of network <NUM> of <FIG>. One or more aspects of <FIG> may be described herein within the context of <FIG>.

In <FIG>, and as in <FIG>, network <NUM> includes a computing system or controller <NUM> interacting with one or more of clusters <NUM> (i.e., clusters 130A, 130B, 130C). In the example of <FIG>, cluster 130C is illustrated with a dotted line to indicate that it is described herein as a cluster that may be instantiated or brought online as a result of operations performed by controller <NUM>, as further described below. Included within each of clusters <NUM> are virtual network controllers <NUM> (e.g., virtual network controller 136A within cluster 130A) and one or more networks <NUM>, each supported by a plurality of servers <NUM> (e.g., servers 34A through 34N). Each of virtual network controllers <NUM> includes, as described in connection with <FIG>, one or more endpoints <NUM> (e.g., virtual network controller 136A includes or is composed of endpoints 137A-<NUM> through 137A-N).

In general, each of clusters <NUM>, as well as the components included with each of clusters <NUM>, may correspond to like-numbered elements of <FIG>. Such devices, systems, and/or components may be implemented in a manner consistent with the description of the corresponding system provided in connection with <FIG>, although in some examples such systems may involve alternative implementations with more, fewer, and/or different capabilities. In general, systems, devices, components, user interface elements, and other items in Figures herein may correspond to like-numbered systems, devices, components, and items illustrated in other Figures, and may be described in a manner consistent with the description provided in connection with other Figures. For ease of illustration, a limited number of clusters <NUM>, endpoints <NUM>, systems and/or components within clusters <NUM>, administrators <NUM>, computing systems <NUM>, and other components are illustrated in <FIG>, although techniques in accordance with one or more aspects of the present disclosure may be performed with many more of such systems.

Controller <NUM> may be implemented as any suitable computing system, such as one or more server computers, workstations, mainframes, appliances, cloud computing systems, and/or other computing systems that may be capable of performing operations and/or functions described in accordance with one or more aspects of the present disclosure. In some examples, controller <NUM> represents a cloud computing system, server farm, and/or server cluster (or portion thereof) that provides services to client devices and other devices or systems. In other examples, controller <NUM> may represent or be implemented through one or more virtualized compute instances (e.g., virtual machines, containers) of a data center, cloud computing system, server farm, and/or server cluster.

In the example of <FIG>, controller <NUM> may include power source <NUM>, one or more processors <NUM>, one or more communication units <NUM>, one or more input devices <NUM>, one or more output devices <NUM>, and one or more storage devices <NUM>. Storage devices <NUM> may include authentication module <NUM>, authentication data <NUM>, API module <NUM>, user interface module <NUM>, data store <NUM>, and cache <NUM>. One or more of the devices, modules, storage areas, or other components of controller <NUM> may be interconnected to enable inter-component communications (physically, communicatively, and/or operatively). In some examples, such connectivity may be provided by through communication channels (e.g., communication channels <NUM>), a system bus, a network connection, an inter-process communication data structure, or any other method for communicating data.

Power source <NUM> may provide power to one or more components of controller <NUM>. Power source <NUM> may receive power from the primary alternating current (AC) power supply in a building, home, or other location. In other examples, power source <NUM> may be a battery or a device that supplies direct current (DC). In still further examples, controller <NUM> and/or power source <NUM> may receive power from another source. One or more of the devices or components illustrated within controller <NUM> may be connected to power source <NUM>, and/or may receive power from power source <NUM>. Power source <NUM> may have intelligent power management or consumption capabilities, and such features may be controlled, accessed, or adjusted by one or more modules of controller <NUM> and/or by one or more processors <NUM> to intelligently consume, allocate, supply, or otherwise manage power.

One or more processors <NUM> of controller <NUM> may implement functionality and/or execute instructions associated with controller <NUM> or associated with one or more modules illustrated herein and/or described below. One or more processors <NUM> may be, may be part of, and/or may include processing circuitry that performs operations in accordance with one or more aspects of the present disclosure. Examples of processors <NUM> include microprocessors, application processors, display controllers, auxiliary processors, one or more sensor hubs, and any other hardware configured to function as a processor, a processing unit, or a processing device. Central monitoring system <NUM> may use one or more processors <NUM> to perform operations in accordance with one or more aspects of the present disclosure using software, hardware, firmware, or a mixture of hardware, software, and firmware residing in and/or executing at controller <NUM>.

One or more communication units <NUM> of controller <NUM> may communicate with devices external to controller <NUM> by transmitting and/or receiving data, and may operate, in some respects, as both an input device and an output device. In some examples, communication unit <NUM> may communicate with other devices over a network. In other examples, communication units <NUM> may send and/or receive radio signals on a radio network such as a cellular radio network. In other examples, communication units <NUM> of controller <NUM> may transmit and/or receive satellite signals on a satellite network such as a Global Positioning System (GPS) network. Examples of communication units <NUM> include a network interface card (e.g. such as an Ethernet card), an optical transceiver, a radio frequency transceiver, a GPS receiver, or any other type of device that can send and/or receive information. Other examples of communication units <NUM> may include devices capable of communicating over Bluetooth®, GPS, NFC, ZigBee, and cellular networks (e.g., <NUM>, <NUM>, <NUM>), and Wi-Fi® radios found in mobile devices as well as Universal Serial Bus (USB) controllers and the like. Such communications may adhere to, implement, or abide by appropriate protocols, including Transmission Control Protocol/Internet Protocol (TCP/IP), Ethernet, Bluetooth, NFC, or other technologies or protocols.

One or more input devices <NUM> may represent any input devices of controller <NUM> not otherwise separately described herein. One or more input devices <NUM> may generate, receive, and/or process input from any type of device capable of detecting input from a human or machine. For example, one or more input devices <NUM> may generate, receive, and/or process input in the form of electrical, physical, audio, image, and/or visual input (e.g., peripheral device, keyboard, microphone, camera).

One or more output devices <NUM> may represent any output devices of controller <NUM> not otherwise separately described herein. One or more output devices <NUM> may generate, receive, and/or process output from any type of device capable of outputting information to a human or machine. For example, one or more output devices <NUM> may generate, receive, and/or process output in the form of electrical and/or physical output (e.g., peripheral device, actuator).

One or more storage devices <NUM> within controller <NUM> may store information for processing during operation of controller <NUM>. Storage devices <NUM> may store program instructions and/or data associated with one or more of the modules described in accordance with one or more aspects of this disclosure. One or more processors <NUM> and one or more storage devices <NUM> may provide an operating environment or platform for such modules, which may be implemented as software, but may in some examples include any combination of hardware, firmware, and software. One or more processors <NUM> may execute instructions and one or more storage devices <NUM> may store instructions and/or data of one or more modules. The combination of processors <NUM> and storage devices <NUM> may retrieve, store, and/or execute the instructions and/or data of one or more applications, modules, or software. Processors <NUM> and/or storage devices <NUM> may also be operably coupled to one or more other software and/or hardware components, including, but not limited to, one or more of the components of controller <NUM> and/or one or more devices or systems illustrated as being connected to controller <NUM>.

In some examples, one or more storage devices <NUM> are temporary memories, meaning that a primary purpose of the one or more storage devices is not long-term storage. Storage devices <NUM> of controller <NUM> may be configured for short-term storage of information as volatile memory and therefore not retain stored contents if deactivated. Examples of volatile memories include random access memories (RAM), dynamic random access memories (DRAM), static random access memories (SRAM), and other forms of volatile memories known in the art. Storage devices <NUM>, in some examples, also include one or more computer-readable storage media. Storage devices <NUM> may be configured to store larger amounts of information than volatile memory. Storage devices <NUM> may further be configured for long-term storage of information as non-volatile memory space and retain information after activate/off cycles. Examples of non-volatile memories include magnetic hard disks, optical discs, Flash memories, or forms of electrically programmable memories (EPROM) or electrically erasable and programmable (EEPROM) memories.

Authentication module <NUM> may perform functions relating to processing authentication credentials and authenticating users. Authentication module <NUM> may authenticate users to enable such users to access, manage, or configure specific clusters <NUM> or may also authenticate users to access, manage, or configure services or endpoints across multiple clusters, thereby enabling multi-cluster management of endpoints. Authentication module <NUM> may manage authentication data <NUM> and/or may store information to and access information from authentication data <NUM>. Authentication data <NUM> may include information derived from information received in communications with administrator <NUM> or with one or more of clusters <NUM>.

API module <NUM> may perform functions relating to performing multi-cluster management or configuration of one or more endpoints <NUM> within clusters <NUM>. API module <NUM> may process requests <NUM> and identify one or more endpoints to configure or manage and how such endpoints are to be configured or managed. API module <NUM> may access data store <NUM> and/or cache <NUM> to identify a public or private URL for an endpoint to be configured. API module <NUM> may cause communication unit <NUM> to communicate with clusters <NUM> to create one or more new clusters <NUM> or to configure one or more aspects (e.g., endpoints) within new or existing clusters <NUM>. API module <NUM> may generate and/or process REST API calls. For instance, API module <NUM> may process REST API calls received by controller <NUM> from administrator <NUM>, and may generate REST API calls that controller <NUM> communicates to one or more endpoints <NUM> within clusters <NUM>. API module <NUM> may receive information from and output information to one or more other modules, and may otherwise interact with and/or operate in conjunction with one or more other modules of controller <NUM>. In some examples, functions performed by API module <NUM> could be performed by software or by a hardware device executing software. In other examples, functions performed by API module <NUM> may be implemented primarily or partially through hardware.

User interface module <NUM> may perform functions relating to generating graphical user interfaces (or other types of user interfaces) for presentation at a computing device operated by one or more administrators <NUM>. For instance, user interface module <NUM> may generate data underlying authentication web pages. User interface module <NUM> may also generate data underlying web pages that present display objects that management, in a multi-cluster fashion, of clusters <NUM> or endpoints <NUM> within clusters <NUM>. Such user interfaces may have form similar to user interfaces <NUM> illustrated in <FIG>.

Data store <NUM> may represent any suitable data structure or storage medium for storing information related to endpoints within a cluster. Data store <NUM> may store information about endpoint types and other information used to configure endpoints <NUM> or to report information about current or available configurations of one or more endpoints <NUM>. In some examples, data store <NUM> may include a relational database and/or table for a SQL database (e.g., a PostgreSQL database) having the form illustrated in <FIG>. The information stored in data store <NUM> may be searchable and/or categorized such that one or more modules within controller <NUM> may provide an input requesting information from data store <NUM>, and in response to the input, receive information stored within data store <NUM>. Data store <NUM> may be primarily maintained by API module <NUM>.

Cache <NUM> may represent any suitable data store for storing subsets of data from data store <NUM>. Typically, cache <NUM> is smaller than data store <NUM> and has a faster access time than data store <NUM>, thereby enabling faster access to information that is stored in cache <NUM>. In some examples, cache <NUM> may be implemented as a key-value store that uses prefix <NUM> as a key for identifying an endpoint associated with a configuration request. Cache <NUM> may have a form similar to that of <FIG>, where a universally unique identifier ("UUID") and a prefix are used as a key to identify a private URL for an endpoint specified by a configuration request. Cache <NUM> may be created or updated by API module <NUM> when one or more new endpoints <NUM> are instantiated or brought online, or when detecting configurations to one or more endpoints <NUM>.

In the example of <FIG>, and in accordance with one or more aspects of the present disclosure, controller <NUM> may authenticate administrator <NUM>. For instance, in an example that can be described with reference to <FIG>, communication unit <NUM> of controller <NUM> detects input and outputs to authentication module <NUM> information about the input. Authentication module <NUM> determines that the input corresponds to a request to authenticate a user (e.g., administrator <NUM>). Authentication module <NUM> outputs information to user interface module <NUM>. User interface module <NUM> generates a user interface with a username and password prompt. User interface module <NUM> causes communication unit <NUM> to output information to administrator <NUM> (or a computing device operated by administrator <NUM>), such as over a network. The information controller <NUM> outputs to administrator <NUM> is sufficient to generate a username/password user interface, and upon receiving the information, a computing device operated by administrator <NUM> presents the user interface (e.g., at a display). Communication unit <NUM> thereafter detects input and outputs information about the input to authentication module <NUM>. Authentication module <NUM> determines, based on the information about the input, that the input corresponds to valid authentication credentials from administrator <NUM>.

Controller <NUM> may receive further input identifying a cluster and endpoint. For instance, with reference to <FIG>, communication unit <NUM> of controller <NUM> detects input and outputs to API module <NUM> information about the input. API module <NUM> analyzes the input and determines that the input corresponds to a request to configure or manage one or more aspects of clusters <NUM>. In the example of <FIG>, the input corresponds to request <NUM>. API module <NUM> further determines that request <NUM> corresponds to a request to configure endpoint 137A-<NUM> in cluster 130A. In some examples, request <NUM> may correspond to or include a REST API request generated by a computing device operated by administrator <NUM> and communicated to controller <NUM> over a network. In such an example, request <NUM> may have a form similar to the REST API call illustrated in <FIG>.

To identify endpoint 137A-<NUM>, API module <NUM> may extract, from request <NUM>, prefix <NUM> and identifier <NUM>. Identifier <NUM> may be a UUID associated with, and identifying, cluster 130A (in the example being described, identifier <NUM> identifies cluster 130A). Prefix <NUM> may be information specifying one or more of endpoints <NUM> within cluster 130A to be configured (in the example being described, prefix <NUM> identifies endpoint 137A-<NUM>). In some examples, a URL for an endpoint may have the form "http://<endpointListenIP>:<endpointListenPort>", where "endpointListenIP" is the IP address that the endpoint uses to listen for configuration requests or management communications, and where "endpointListenPort" is the port that the endpoint uses at that IP address to listen for configuration requests and/or management communications. Accordingly, a public and/or private URL for an endpoint that implements an OpenStack Keystone authentication service will have the form "http://<KeystoneListenIP>:<KeystoneListenPort>" where "KeystoneListenIP" is the IP address of the Keystone service endpoint, and the "KeystoneListenPort" is the port at the KeystoneListenIP where requests relating to the Keystone service are received. Endpoint services include analytics services, configuration services, and other services; such services may include those sometimes referred to as nodejs, telemetry, swift, glance, compute, baremetal, as well as other custom endpoint services.

After identifying the endpoint and cluster associated with request <NUM>, controller <NUM> may configure endpoint 137A-<NUM> within cluster 130A. For instance, again referring to <FIG> and after receiving request <NUM>, API module <NUM> outputs, to authentication module <NUM>, a request for authentication information associated endpoint 137A-<NUM> within cluster 130A. Authentication module <NUM> access authentication data <NUM> and accesses authentication credentials (e.g., a username and password combination) for authenticated administrator <NUM>. API module <NUM> identifies, by accessing <NUM> and/or cache <NUM>, a URL/port combination for endpoint 137A-<NUM>. Authentication module <NUM> causes communication unit <NUM> to securely output the authentication credentials to cluster 130A, and specifically, to endpoint 137A-<NUM> within cluster 130A. Endpoint 137A-<NUM> determines that the authentication credentials are valid. API module <NUM> causes communication unit <NUM> to further communicate with endpoint 137A-<NUM> to perform the configurations specified in request <NUM>. In some examples, the configurations may include modifications made to existing endpoints <NUM>, or addition or removal of one or more endpoints <NUM>.

In other examples, the configurations may include the addition or removal of one or more endpoints <NUM> within 130A. In such an example, controller <NUM> may communicate with virtual network controller 136A to invoke services provided by an API exposed by virtual network controller 136A. Such services may enable controller <NUM> (or other authenticated devices) to add, remove, or otherwise configure one or more endpoints <NUM> within cluster 130A.

Controller <NUM> may update data store <NUM> to reflect configuration changes associated with cluster 130A. For instance, in the example of <FIG>, API module <NUM> outputs, to data store <NUM>, information about the configurations performed within cluster 130A. Data store <NUM> stores the information. In some examples, API module <NUM> (or data store <NUM>) may also update cache <NUM>, which may be implemented as an in-memory key-value endpoint store, to reflect changes to any changes to the endpoints as a result of the configurations performed within cluster 130A. In some examples, such changes may include new addresses, prefixes, or other information associated with endpoints within cluster 130A, or may include changes to reflect removal of one or more endpoints within cluster 130A.

In addition to configuring aspects of existing clusters <NUM>, controller <NUM> may also create one or more new clusters, such as cluster 130C (illustrated as a dotted line in <FIG>). In one example, controller <NUM> may receive a request to create cluster 130C. For instance, in the example of <FIG>, controller <NUM> receives input that API module <NUM> determines corresponds to a request (e.g., from administrator <NUM>) to create new cluster 130C. API module <NUM> creates an object within data store <NUM> to correspond to cluster 130C. API module <NUM> further creates one or more objects within data store <NUM> to correspond to endpoints <NUM> within cluster 130C. API module <NUM> may create new routes for each of the new endpoints <NUM> within cluster 130C, and store associated information within data store <NUM>. API module <NUM> may cause communication unit <NUM> to communicate with one or more of virtual network controllers <NUM> to provision new cluster 130C and otherwise instantiate objects and/or systems within new cluster <NUM>.

In some examples, API module <NUM> may also update cache <NUM> to include at least a subset of the information stored within data store <NUM>. By doing so, when a new configuration or management request is received by controller <NUM>, controller <NUM> may process the request by accessing information about the endpoint <NUM> specified in the request without accessing data store <NUM>, thereby enabling low-latency access (i.e., through <NUM>) to information otherwise accessible through data store <NUM>. Cache <NUM> may, in some examples, enable controller <NUM> to serve as a proxy between administrator <NUM> and clusters <NUM> with little or no additional latency.

In some examples, to create cluster 130C, API module <NUM> causes communication unit <NUM> to communicate with one or more of virtual network controllers <NUM> to invoke services provided by virtual network controllers <NUM> for creating and establishing new cluster 130C and endpoints <NUM> included within new cluster 130C. In other examples, API module <NUM> may cause communication unit <NUM> to communicate with another system or higher-level service (not shown) that provides the capability for creating and/or establishing new cluster 130C and the endpoints 137C included within new cluster 130C. In still other examples, administrator <NUM> may use another tool to create and configure cluster 130C or to configure aspects of other clusters <NUM>. In such an example, controller <NUM> may communicate with each of clusters <NUM> to determine any changes, additions, removals, or other modifications to clusters <NUM>, and update data store <NUM> to reflect such changes. Alternatively, or in addition, controller <NUM> may receive input (e.g., from administrator <NUM>) about changes that have been made or will be made to clusters <NUM> using a tool other than controller <NUM>, and in that example, controller <NUM> may also update data store <NUM> to reflect such changes. Accordingly, controller <NUM> may operate dynamically to detect changes to any of clusters <NUM> (including additional clusters <NUM>), and update, often automatically, data store <NUM> and/or cache <NUM>.

After creating new cluster 130C, controller <NUM> may thereafter configure one or more endpoints 137C within new cluster 130C. For instance, still referring to <FIG>, controller <NUM> may detect input that API module <NUM> determines corresponds to a request, from administrator <NUM>, to configure one or more endpoints 137C within new cluster 130C. As previously described with respect to request <NUM>, the request may include prefix <NUM> and identifier <NUM>, with identifier <NUM> identifying cluster 130C and prefix <NUM> identifying which of endpoints 137C within cluster 130C to configure. API module <NUM> causes authentication module <NUM> to access authentication information for administrator <NUM> for one or more endpoints 137C within cluster 130C. API module <NUM> uses the authentication information to cause communication unit <NUM> to communicate with one or more endpoints 137C within cluster 130C and authenticate controller <NUM> to enable configurations within cluster 130C. API module <NUM> further causes communication unit <NUM> to output a configuration request (e.g., in the form of a REST API call) to one or more endpoints 137C within cluster 130C. In some examples, the configuration request may be a REST API call having a form similar to that illustrated in <FIG>. One or more endpoints 137C within 130C perform the requested configurations after receiving communications from controller <NUM>. In connection with the configurations, API module <NUM> updates data stores <NUM> and cache <NUM> to include any new information about endpoints 137C within 130C that result from the configurations performed with respect to cluster 130C.

Modules illustrated in <FIG> (e.g., navigation module <NUM>, communication module <NUM>, analysis module <NUM>, user interface module <NUM>, recovery module <NUM>, and transaction module <NUM>) and/or illustrated or described elsewhere in this disclosure may perform operations described using software, hardware, firmware, or a mixture of hardware, software, and firmware residing in and/or executing at one or more computing devices. For example, a computing device may execute one or more of such modules with multiple processors or multiple devices. A computing device may execute one or more of such modules as a virtual machine executing on underlying hardware. One or more of such modules may execute as one or more services of an operating system or computing platform. One or more of such modules may execute as one or more executable programs at an application layer of a computing platform. In other examples, functionality provided by a module could be implemented by a dedicated hardware device.

Although certain modules, data stores, components, programs, executables, data items, functional units, and/or other items included within one or more storage devices may be illustrated separately, one or more of such items could be combined and operate as a single module, component, program, executable, data item, or functional unit. For example, one or more modules or data stores may be combined or partially combined so that they operate or provide functionality as a single module. Further, one or more modules may interact with and/or operate in conjunction with one another so that, for example, one module acts as a service or an extension of another module. Also, each module, data store, component, program, executable, data item, functional unit, or other item illustrated within a storage device may include multiple components, sub-components, modules, sub-modules, data stores, and/or other components or modules or data stores not illustrated.

Further, each module, data store, component, program, executable, data item, functional unit, or other item illustrated within a storage device may be implemented in various ways. For example, each module, data store, component, program, executable, data item, functional unit, or other item illustrated within a storage device may be implemented as a downloadable or preinstalled application or "app. " In other examples, each module, data store, component, program, executable, data item, functional unit, or other item illustrated within a storage device may be implemented as part of an operating system executed on a computing device.

<FIG> is a block diagram illustrating an example multi-cluster or multi-cloud network having multiple data centers, in accordance with one or more aspects of the present disclosure. Network <NUM> of <FIG> may be described as an example or alternative implementation of network <NUM> of <FIG> or <FIG>. As in <FIG> and <FIG>, many of the components illustrated in <FIG> may correspond to like-numbered elements previously described in connection with <FIG> and <FIG>. In general, such like-numbered systems, devices, components, and items illustrated in <FIG> may be described in a manner consistent with the description provided in connection with <FIG> and <FIG>, although in some examples such systems, devices, components, and items may involve alternative implementations with more, fewer, and/or different capabilities.

<FIG> illustrates data centers 32A-32X, which house servers that form respective ones of clusters <NUM>. As one example, data center 32A houses servers 34A-34N that may be configured to provide the infrastructure for clusters 130A. The other data centers <NUM> may be substantially similar to data center 32A, but may house servers for other clusters <NUM>. Also, one of data centers <NUM> may house servers for multiple clusters <NUM>, or alternatively, one of clusters <NUM> may span multiple data centers <NUM>.

In the example illustrated in <FIG>, data centers 32A-32X (collectively, "data centers <NUM>") are interconnected with one another and with customer networks associated with customers <NUM> via a service provider network <NUM>. In general, each data center 32A provides an operating environment for applications and services for customers <NUM> coupled to the data center by service provider network <NUM>. Data centers <NUM> may, for example, host infrastructure equipment, such as networking and storage systems, redundant power supplies, and environmental controls. Service provider network <NUM> may be coupled to one or more networks administered by other providers, and may thus form part of a large-scale public network infrastructure, e.g., the Internet.

In some examples, each of data centers <NUM> may represent one of many geographically distributed network data centers. As illustrated in the example of <FIG>, each of data centers <NUM> may represent a facility that provides network services for customers <NUM>. Customers <NUM> may be collective categories such as enterprises and governments or individuals. For example, a network data center may host a virtual computing environment (e.g., cloud) that provides web services for several enterprises and end users. Other example services may include data storage, virtual private networks, traffic engineering, file service, data mining, scientific- or super-computing, and so on. In some examples, each of data centers <NUM> may be individual network servers, network peers, or otherwise.

In the illustrated example, each of data centers <NUM> includes a set of storage systems and application servers 34A-34N (herein, "servers <NUM>") interconnected via high-speed switch fabric provided by one or more tiers of physical network switches and routers, including a set of interconnected top-of-rack (TOR) switches 40A-40N (collectively, "TOR switches <NUM>") coupled to a distribution layer of chassis switches 42A-42Y (collectively, "chassis switches <NUM>"). Although not shown, each of data centers <NUM> may also include, for example, one or more non-edge switches, routers, hubs, gateways, security devices such as firewalls, intrusion detection, and/or intrusion prevention devices, servers, computer terminals, laptops, printers, databases, wireless mobile devices such as cellular phones or personal digital assistants, wireless access points, bridges, cable modems, application accelerators, or other network devices.

In the example illustrated in <FIG>, TOR switches <NUM> and chassis switches <NUM> provide servers <NUM> with redundant (multi-homed) connectivity to IP fabric <NUM> and service provider network <NUM>. Chassis switches <NUM> aggregate traffic flows and provides high-speed connectivity between TOR switches <NUM>. TOR switches <NUM> may be network devices that provide layer two (e.g., MAC) and/or layer <NUM> (e.g., IP) routing and/or switching functionality. TOR switches <NUM> and chassis switches <NUM> may each include one or more processors and a memory, and that are capable of executing one or more software processes. Chassis switches <NUM> are coupled to IP fabric <NUM>, which performs layer <NUM> routing to route network traffic between data centers <NUM> and customers <NUM> by service provider network <NUM>.

In the example illustrated in <FIG>, data center 32A is configured to provide the infrastructure for cluster 130A. For example, servers 34A-34N may be configured to execute virtualized machines (VMs), containers or other virtualized executional elements to support the operation of cluster 130A. Moreover, in the example of <FIG>, virtual network controller 136A is part of cluster 130A. Accordingly, servers 34A-34N may be configured to support the operation of virtual network controller 136A. Further, in some examples, controller <NUM> may be implemented as part of cluster 130A; accordingly, servers 34A to 34N may be configured to support the operation of controller <NUM>.

As illustrated in <FIG>, servers 34A through 34N execute VMs 50A through 50N. In the example illustrated, VMs 50A and 54N may together provide one or more virtualized machine on which virtual network controller 136A can execute and perform operations consistent with those described herein (e.g., provide a controller for endpoint configuration, provide route propagation, security, application deployment, and configuration within clusters 130A with, potentially, a single pane of glass interface). For instance, in some examples, each of endpoint services 137A-<NUM> through 137A-N may execute on a virtual machine in server 34A. As labeled in <FIG>, VM 50A executing on server 34A may provide an execution environment for execution of endpoint 137A-<NUM>, and VM 54N executing on server 34A may execute on endpoint 137A-N. Such virtualized machines on which endpoint 137A-<NUM> and 137A-N may execute and perform endpoint operations consistent with described elsewhere herein. Such services may include authentication (e.g., OpenStack's Keystone service), image management (e.g., OpenStack's Glance service), storage (e.g., OpenStack's Swift service), analytics, telemetry, or other services.

Similarly, servers 34B through 34N execute VMs 50B through 50N and VMs 54B through 54N. In the example illustrated, such VMs may together provide an execution environment and computing infrastructure for customer or tenant applications deployed within data center 32A. Although a specific allocation and arrangement of execution environments for components of controller <NUM> and endpoints <NUM> is illustrated in <FIG>, in other examples, a different arrangement may be used, and may span multiple data centers.

In general, VMs 50A through 50N and VMs 54A through 54N execute on processing circuitry of respective servers 34A, 34B, and 34N. VMs 50A, 50B, 50N, 54A, 54B, and 54N are illustrated merely to assist with understanding and should not be considered as limiting. For example, controller <NUM> may be configured to spin up and spin down virtual machines across or within servers <NUM> as needed to support the operations of 130A, virtual network controller 136A, any of endpoints 137A, and/or controller <NUM>. However, the example techniques are not so limited, and in some examples, controller 136A and/or controller <NUM> may be configured to determine resources within data center 32A that are to be utilized (e.g., how many VMs are spun up or spun down) for cluster 130A. Moreover, in some examples, controller <NUM> and/or virtual network controller 136A may be configured to determine resources within the other data centers <NUM> that are to be utilized (e.g., how many VMs are spun up or spun down) for the other clusters <NUM>.

Virtual network controller 136A provide a logically and in some cases physically centralized controller for facilitating operation of one or more virtual networks within each of data centers <NUM>, such as data center 32A. In some examples, controller <NUM> and/or virtual network controller 136A may operate in response to configuration input received from network administrator <NUM>. Moreover, as illustrated, in this example, administrator <NUM> may be tasked with providing configuration information so that controller <NUM> and/or virtual network controller 136A can perform the example operations described in this disclosure. Administrator <NUM> may represent an operator, developer, or application deployment specialist that uses a common interface to create and deploy virtual computing environment topologies to controller <NUM> for provisioning within the computing infrastructure.

In some examples, the traffic between any two network devices, such as between network devices within IP fabric <NUM> (not shown), between servers <NUM>, and customers <NUM>, or between servers <NUM>, for example, can traverse the physical network using many different paths. A packet flow (or "flow") can be defined by the five values used in a header of a packet, or "five-tuple," i.e., the protocol, source IP address, destination IP address, source port and destination port that are used to route packets through the physical network. For example, the protocol specifies the communications protocol, such as TCP or UDP, and source port and destination port refer to source and destination ports of the connection. The flow within data center 32A is one example of a flow. Another example of a flow is the flow of data between clusters <NUM>.

A set of one or more packet data units (PDUs) that include a packet header specifying a particular five-tuple represent a flow. Flows may be broadly classified using any parameter of a PDU, such as source and destination data link (e.g., MAC) and network (e.g., IP) addresses, a Virtual Local Area Network (VLAN) tag, transport layer information, a Multiprotocol Label Switching (MPLS) or Generalized MPLS (GMPLS) label, and an ingress port of a network device receiving the flow. For example, a flow may be all PDUs transmitted in a Transmission Control Protocol (TCP) connection, all PDUs sourced by a particular MAC address or IP address, all PDUs having the same VLAN tag, or all PDUs received at the same switch port. A flow may be additionally or alternatively defined by an Application Identifier (AppID) that is determined by a virtual router agent or other entity that identifies, e.g., using a port and protocol list or deep packet inspection (DPI), a type of service or application associated with the flow in that the flow transports application data for the type of service or application.

In the example of <FIG>, and in accordance with one or more aspects of the present disclosure, controller <NUM> may configure one or more aspects of one or more clusters <NUM>. For instance, with reference to <FIG>, controller <NUM> detects input from a computing device operated by administrator <NUM> and determines that administrator <NUM> is an authenticated user. Controller <NUM> detects further input and determines that the input corresponds to a request to configure one or more aspects of virtual network controllers 136A within cluster 130A. Specifically, controller <NUM> determines that the input includes information identifying cluster 130A and endpoint 137A-<NUM> within cluster 130A. Controller <NUM> uses the input to communicate with VMs 54A through 54N, which implement endpoint 137A-<NUM>, to perform the configurations specified by the input. Such configurations may involve changing the configuration of endpoint 137A-<NUM>, for example. In other examples, such configurations may include instantiating and/or creating an additional endpoint (e.g., endpoint 137A-<NUM>), which may be implemented through an additional set of virtual machines hosted on servers <NUM>.

<FIG> is a conceptual illustrations of an example database table that may be used to store information about endpoint configurations, in accordance with one or more aspects of the present disclosure. The table illustrated in <FIG> illustrates a number of sample columns that may be implemented in a SQL database performing the operations described herein as being performed by data store <NUM> of <FIG>. Also illustrated are sample data types associated with each listed column, and whether each corresponding column may contain null values.

<FIG> are conceptual illustrations of a table of object identifiers, endpoint prefixes, object types, and corresponding URLs, in accordance with one or more aspect of the present disclosure. <FIG> shows a universally unique identifier ("UUID") and a prefix are used as a key to identify a private URL and port value for an endpoint specified by a configuration request. Information shown in <FIG> may form the basis for a key-value store (e.g., cache <NUM>) as described in connection with <FIG>. <FIG> shows a table of prefixes for a cluster object, along with the UUID for that cluster object. In the specific example of <FIG>, the cluster object has a "contrail-cluster" type, and the listed prefixes are a set of prefixes associated with the contrail cluster identified by the UUID listed for that contrail cluster (i.e., in the center column).

<FIG> is an example REST API call that may be received by an example computing system that serves as a configuration proxy, in accordance with one or more aspects of the present disclosure. For instance, with reference to <FIG>, the REST API call of <FIG> may correspond to request <NUM>, and may represent a project-scoped token request sent by a computing device operated by administrator <NUM> to controller <NUM>. Controller <NUM> receives the REST API call and uses the "x-cluster-id" in the HEADER along with prefix "keystone" to look up the keystone service endpoint. Controller <NUM> then routes the request to the specific keystone service.

<FIG> is an example REST API call that may be initiated by an example computing system to an endpoint for the purpose of configuring that endpoint, in accordance with one or more aspects of the present disclosure. <FIG> illustrates a proxy request, pe. For instance, again referring to <FIG>, <FIG> shows a proxy request, to a "nodejs" endpoint. To perform the request, controller <NUM> parses the cluster-id and prefix from the URL with in the REST API call. Controller <NUM> uses the cluster-id (ac28718E-63FIG. <NUM>-4Dae-907F-ba459C883D26) and prefix (nodejs) to look up the endpoint private URL of the endpoint to be configured. Once controller <NUM> determines that private URL, controller <NUM> routes the request to the appropriate nodejs service.

<FIG> are conceptual diagrams illustrating example user interfaces presented by a user interface device, in accordance with one or more aspects of the present disclosure. Although the user interfaces illustrated in <FIG> are shown as graphical user interfaces, other types of interfaces may be presented in other examples, including a text-based user interface, a console or command-based user interface, a voice prompt user interface, or any other appropriate user interface. One or more aspects of the user interfaces illustrated in <FIG> may be described herein within the context of network <NUM> and/or controller <NUM> of <FIG>.

In some examples, and with reference to <FIG>, one or more of the user interfaces illustrated in <FIG> may be presented by a computing device operated by administrator <NUM>. For instance, user interface module <NUM> of controller <NUM> may, in response to input received from a computing device operated by administrator <NUM>, generate data sufficient for the computing device operated by administrator <NUM> to generate and display a user interface. User interface module <NUM> may output the data (i.e., a "user interface") over a network for display at the computing device operated by administrator <NUM>. That computing device may detect interactions with the user interface (e.g., mouse movements, keystrokes, touch input) and output information about the input over the network to controller <NUM>. Controller <NUM> may update or generate new data sufficient to generate further user interfaces. Controller <NUM> and the computing device operated by administrator <NUM> may continue to communicate, with the result being that multiple user interfaces, of the type illustrated in <FIG>, are presented for display, viewing, and interaction by administrator <NUM>.

<FIG> illustrates user interface 700A, implemented as a web page that may enable administrator <NUM> to authenticate with controller <NUM>. In some examples, user interface <NUM> may present a drop-down control enabling a user (e.g., administrator <NUM>) to select credentials associated with a cluster, and (after authenticating) use those credentials to view information about a cluster (e.g., view clusters within network <NUM>). Alternatively, or in addition, a user may select credentials associated with controller <NUM> (rather than credentials associated with any particular cluster), and use those credentials to view information about multiple endpoints across multiple clusters.

<FIG> illustrates a user interface (i.e., web page) presenting a multiple-cluster view. In the example of <FIG>, user interface 700B presents a list of clusters within network <NUM>. In the particular example shown in <FIG>, only a single cluster, "AIO," is listed, but in other examples many more clusters may be listed. Status or other information about each cluster may also be displayed within user interface 700B.

<FIG> illustrates a user interface that describes a sequence of steps that may be taken to create or instantiate a new cluster. Within the view shown in <FIG>, user interface 700C lists a number of servers associated with a selected cluster. A user may interact with user interface 700C to show other information about the selected cluster, including "Credentials," "Key pairs," and "Node profiles.

<FIG> illustrates a user interface that presents further information about a specific selected cluster. In <FIG>, user interface 700D presents including information about the number of different types of nodes (or endpoints) that are included in the cluster (the quantity of compute nodes, control nodes, analytics nodes, config nodes, database nodes). Further information, including analytics information is presented within user interface 700D. In some examples, the information shown in user interface 700D is available to a user that has authenticated using credentials associated with a specific cluster <NUM> (as opposed to credentials associated with controller <NUM>), but information about multiple clusters might not be available to such a user, without further authentication.

<FIG> illustrates a user interface that lists some or all of the endpoints included within a given cluster. In <FIG>, user interface 700E presents a list of endpoint prefixes for a selected cluster, along with the private and public URLs associated with each endpoint. Other information may also be provided, including, information about capabilities for each of the endpoints. In the example shown in <FIG>, an "Enable Proxy" column is shown in the "Endpoints" tab of user interface 700E.

<FIG> is a flow diagram illustrating an example process for performing endpoint configuration or management tasks in accordance with one or more aspects of the present disclosure. The process of <FIG> is illustrated from three different perspectives: operations performed by an example proxy controller <NUM> (left-hand column to the left of dashed line), operations performed by a first example endpoint (middle column between dashed lines), and operations performed by an example a second example endpoint (right-hand column to the right of dashed line).

In the example of <FIG>, the illustrated process may be performed by network <NUM> in the context illustrated in <FIG>. In particular, the proxy computing system (left column of <FIG>) may correspond to controller <NUM> of <FIG>. The first example endpoint (middle column of <FIG>) may correspond to endpoint 137A-<NUM> within cluster 130A of <FIG>. Similarly, the second example endpoint (right column) may correspond to an endpoint within cluster 130B, which although not specifically shown within <FIG>, may have a corresponding reference numeral of endpoint 137B-<NUM>. In other examples, different operations may be performed, or operations described in <FIG> as being performed by a particular component, module, system, and/or device may be performed by one or more other components, modules, systems, and/or devices. Further, in other examples, operations described in connection with <FIG> may be performed in a difference sequence, merged, omitted, or may encompass additional operations not specifically illustrated or described even where such operations are shown performed by more than one component, module, system, and/or device.

In the process illustrated in <FIG>, and in accordance with one or more aspects of the present disclosure, controller <NUM> may receive a configuration request (<NUM>). For instance, in an example that can be described with reference to <FIG>, communication unit <NUM> of controller <NUM> detects input and outputs an indication of input to API module <NUM>. API module <NUM> determines that the input corresponds to request <NUM> from a client device operated by administrator <NUM>.

Controller <NUM> may identify a configuration endpoint specified by the request (<NUM>). For instance, continuing with the example being described with reference to <FIG>, API module <NUM> parses request <NUM> to determine prefix <NUM> and identifier <NUM>. API module <NUM> determines that request <NUM> includes information about a configuration or management operation to be performed on a specific cluster. API module <NUM> further determines that identifier <NUM>, included within request <NUM>, identifies cluster 130A. API module <NUM> also determines that prefix <NUM>, also included within request <NUM>, identifies endpoint 137A-<NUM> within cluster 130A.

Controller <NUM> may proxy the request to the identified endpoint (<NUM>). For instance, again with reference to <FIG>, API module <NUM> accesses data store <NUM> and/or cache <NUM> to determine a URL for endpoint 137A-<NUM>. Authentication module <NUM> of controller <NUM> causes communication unit <NUM> to output a signal over a network destined for endpoint 137A-<NUM>. Authentication module <NUM> causes controller <NUM> to further communicate with endpoint 137A-<NUM> to authenticate administrator <NUM>. Authentication module <NUM> outputs information to API module <NUM>, indicating that administrator <NUM> has been authenticated to manage endpoint 137A-<NUM>. API module <NUM> causes communication unit <NUM> to output to a signal specifying a configuration operation to be performed by endpoint 137A-<NUM>. In some examples, the signal corresponds to a REST API call generated by API module <NUM> of controller <NUM>.

Endpoint 137A-<NUM> may receive the configuration request (<NUM>). For instance, in <FIG>, endpoint 137A-<NUM> detects input that it determines corresponds to the signal output by API module <NUM> of controller <NUM>. In some examples, the signal received by endpoint 137A-<NUM> may correspond to the REST API call generated by API module <NUM>.

Endpoint 137A-<NUM> may perform a configuration operation (<NUM>). For instance, referring again to <FIG>, endpoint 137A-<NUM> performs one or more configuration or management operations specified by the REST API call. Endpoint 137A-<NUM> may output information about the configurations back to controller <NUM>.

Controller <NUM> may update a database to reflect configuration changes (<NUM>). For instance, referring again to <FIG>, controller <NUM> may receive information from endpoint 137A-<NUM> about the configurations performed at endpoint 137A-<NUM>. API module <NUM> of controller <NUM> may update data store <NUM> to include information about the configurations, and API module <NUM> may also update cache <NUM> to include at least some of the information stored in data store <NUM>. Although controller <NUM> may receive configuration information directly from endpoint 137A-<NUM>, controller <NUM> may acquire such information in another way. For example, controller <NUM> may sense or detect configuration or other operations being made to one or more endpoints within any of clusters <NUM>. Upon sensing or detecting such operations, controller <NUM> may update data store <NUM> to reflect information about such operations. Accordingly, controller <NUM> may dynamically update data store <NUM> by using information that it has used to modify one or more endpoints <NUM>, by receiving configuration information from one or more endpoints <NUM>, by otherwise detecting or sensing information about configuration changes, or in another way.

As described herein, controller <NUM> is not limited to performing configuration operations for only one cluster. Instead, controller <NUM> may in some examples serve as a central proxy for routing configuration requests to multiple clusters. In particular, controller <NUM> may route configuration requests to multiple endpoints within multiple clusters. Accordingly, blocks <NUM>' and <NUM>' (drawn with dotted lines) are intended to illustrate that some configuration requests may be routed to endpoints in clusters other than cluster 130A. In particular, endpoint 137B-<NUM> within cluster 130B may receive the configuration request (<NUM>'). Endpoint 137B-<NUM> may perform the configuration operation specified in the configuration request (<NUM>'). In such an example, controller <NUM> may update data store <NUM> to reflect changes to the configuration of endpoint 137B-<NUM>.

Therefore, from one perspective there have been described techniques for configuring software defined network (SDN) controllers within different cloud computing domains and, in particular, a multi-cluster controller that operates and presents, in some examples, a single interface for seamlessly controlling and configuring SDN controllers in different cloud computing domains. In one example, this disclosure describes a system that includes a plurality of clusters, each of the plurality of clusters including a plurality of configurable endpoints; a storage system; and processing circuitry having access to the storage system and capable of communicating with each of the plurality of configurable endpoints. In some examples, the processing circuitry is configured to receive a plurality of requests, each specifying a configuration operation, identify, for each of the requests, a configuration cluster and a configuration endpoint within the configuration cluster, and perform, for each of the requests, the specified configuration operation.

For processes, apparatuses, and other examples or illustrations described herein, including in any flowcharts or flow diagrams, certain operations, acts, steps, or events included in any of the techniques described herein can be performed in a different sequence, may be added, merged, or left out altogether (e.g., not all described acts or events are necessary for the practice of the techniques). Moreover, in certain examples, operations, acts, steps, or events may be performed concurrently, e.g., through multi-threaded processing, interrupt processing, or multiple processors, rather than sequentially. Further certain operations, acts, steps, or events may be performed automatically even if not specifically identified as being performed automatically. Also, certain operations, acts, steps, or events described as being performed automatically may be alternatively not performed automatically, but rather, such operations, acts, steps, or events may be, in some examples, performed in response to input or another event.

For ease of illustration, only a limited number of devices (e.g., computing systems <NUM>, virtual network controllers <NUM>, endpoints <NUM>, networks <NUM>, servers <NUM>, as well as others) are shown within the Figures and/or in other illustrations referenced herein. However, techniques in accordance with one or more aspects of the present disclosure may be performed with many more of such systems, components, devices, modules, and/or other items, and collective references to such systems, components, devices, modules, and/or other items may represent any number of such systems, components, devices, modules, and/or other items.

The Figures included herein each illustrate at least one example implementation of an aspect of this disclosure. The scope of this disclosure is not, however, limited to such implementations. Accordingly, other example or alternative implementations of systems, methods or techniques described herein, beyond those illustrated in the Figures, may be appropriate in other instances. Such implementations may include a subset of the devices and/or components included in the Figures and/or may include additional devices and/or components not shown in the Figures.

The detailed description set forth above is intended as a description of various configurations and is not intended to represent the only configurations in which the concepts described herein may be practiced. The detailed description includes specific details for the purpose of providing a sufficient understanding of the various concepts. However, these concepts may be practiced without these specific details. In some instances, well-known structures and components are shown in block diagram form in the referenced figures in order to avoid obscuring such concepts.

Accordingly, although one or more implementations of various systems, devices, and/or components may be described with reference to specific Figures, such systems, devices, and/or components may be implemented in a number of different ways. For instance, one or more devices illustrated in the Figures herein (e.g., <FIG> and/or <FIG>) as separate devices may alternatively be implemented as a single device; one or more components illustrated as separate components may alternatively be implemented as a single component. Also, in some examples, one or more devices illustrated in the Figures herein as a single device may alternatively be implemented as multiple devices; one or more components illustrated as a single component may alternatively be implemented as multiple components. Each of such multiple devices and/or components may be directly coupled via wired or wireless communication and/or remotely coupled via one or more networks. Also, one or more devices or components that may be illustrated in various Figures herein may alternatively be implemented as part of another device or component not shown in such Figures. In this and other ways, some of the functions described herein may be performed via distributed processing by two or more devices or components.

Further, certain operations, techniques, features, and/or functions may be described herein as being performed by specific components, devices, and/or modules. In other examples, such operations, techniques, features, and/or functions may be performed by different components, devices, or modules. Accordingly, some operations, techniques, features, and/or functions that may be described herein as being attributed to one or more components, devices, or modules may, in other examples, be attributed to other components, devices, and/or modules, even if not specifically described herein in such a manner.

Although specific advantages have been identified in connection with descriptions of some examples, various other examples may include some, none, or all of the enumerated advantages. Other advantages, technical or otherwise, may become apparent to one of ordinary skill in the art from the present disclosure. Further, although specific examples have been disclosed herein, aspects of this disclosure may be implemented using any number of techniques, whether currently known or not, and accordingly, the present disclosure is not limited to the examples specifically described and/or illustrated in this disclosure.

If implemented in software, the functions may be stored, as one or more instructions or code, on and/or transmitted over a computer-readable medium and executed by a hardware-based processing unit. Computer-readable media may include computer-readable storage media, which corresponds to a tangible medium such as data storage media, or communication media including any medium that facilitates transfer of a computer program from one place to another (e.g., pursuant to a communication protocol). In this manner, computer-readable media generally may correspond to (<NUM>) tangible computer-readable storage media, which is non-transitory or (<NUM>) a communication medium such as a signal or carrier wave.

By way of example, and not limitation, such computer-readable storage media can include RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage, or other magnetic storage devices, flash memory, or any other medium that can be used to store desired program code in the form of instructions or data structures and that can be accessed by a computer. Disk and disc, as used, includes compact disc (CD), laser disc, optical disc, digital versatile disc (DVD), floppy disk and Blu-ray disc, where disks usually reproduce data magnetically, while discs reproduce data optically with lasers.

Accordingly, the terms "processor" or "processing circuitry" as used herein may each refer to any of the foregoing structure or any other structure suitable for implementation of the techniques described. In addition, in some examples, the functionality described may be provided within dedicated hardware and/or software modules.

Claim 1:
A system comprising:
a plurality of software-defined networking, SDN, computing clusters (130A, 130B, 130C), each of the plurality of clusters comprising a different SDN network (<NUM>) having a respective SDN network controller (136A, 136B, 136C) operable to configure an overlay network within the SDN network for communication between applications executing within the SDN network, each of the SDN controllers including a plurality of configurable endpoints (137A-<NUM>, 137A-<NUM>, 137A-N) that each provide a different service of the respective SDN controller; and
a command controller (<NUM>) comprising:
a storage system (<NUM>); and
processing circuitry having access to the storage system and capable of communicating with each of the plurality of configurable endpoints of each of the SDN controllers, wherein the processing circuitry is configured to:
receive a plurality of requests (<NUM>), each specifying a configuration operation, identify, for each of the requests, a configuration cluster from among the plurality of clusters,
identify, for each of the requests, a configuration endpoint of the respective SDN controller within the configuration cluster,
perform, for each of the requests, the specified configuration operation by communicating with the identified configuration endpoint within the identified configuration cluster so that performing the configurations operations associated with the plurality of requests includes performing configuration operations across each of the plurality of clusters, and
update, for each of the requests, the storage system to include information about performing the specified configuration operation.