Patent Description:
A computer network is a collection of computers and other hardware interconnected by communication channels that allow sharing of resources and information. Communication protocols define the rules and data formats for exchanging information in a computer network.

Distributed computing involves multiple computing devices organized to cooperatively perform a particular application. For example, a computationally expensive task may be split into subtasks to be performed in parallel by the computing devices in the distributed network, allowing the task to be completed faster. Distributing computing may also involve fragmenting a data set and storing it across multiple storage devices. Distributed computing may also involve the multiple computing devices handling individual requests from clients, such as requests for data received over the Internet. Providing computing services over the Internet using such a distributed approach is generally referred to as "cloud computing. " <CIT> describes secure integration of hybrid clouds with enterprise networks. <CIT> describes adaptive gossip protocol. <CIT> describes selecting services from multiple cloud vendors.

Techniques for implementing and managing a hybrid cloud computing network including virtual and physical nodes distributed across multiple cloud service providers and different networks are described. Dependent claims define preferred embodiments.

Other features, aspects and potential advantages will be apparent from the accompanying description and figures.

Cloud service providers provide the ability for customers to implement distributed applications without having to manage any physical hardware components themselves. Such cloud computing networks may allow customers to allocate and deallocate virtual processing resources (virtual nodes), such as virtual machine instances, operating-system-level virtual instances in frameworks like DOCKER, etc., programmatically using an application programming interface (API). Examples of these kinds of systems include MICROSOFT AZURE, AMAZON WEB SERVICES (AWS), VERIZON CLOUD, and GOOGLE CLOUD. One limitation of these services is a lack of integration between the different offerings. For example, a virtual node running in MICROSOFT AZURE cannot interoperate with one running in AWS without significant application level logic implemented by the customer in the node itself. Similarly, virtual nodes running in one of these cloud systems cannot interoperate with physical computing devices (physical nodes) outside the cloud system, such as a physical computing device residing on a customer's private network. Such a limitation can lead to inefficiencies such as backhauling data from one cloud system to another so the data can be processed by an appropriate node.

The present application describes techniques for implementing and managing a hybrid cloud network that can include virtual nodes from multiple different cloud service providers, as well as physical or virtual nodes hosted on a customer's private network. The techniques described herein can be used to organize different types of virtual and physical processing resources hosted on different networks into a hybrid cloud computing network that seamlessly utilizes these processing resources. Also described herein is a synchronization technique by which the nodes of the hybrid cloud network synchronize data, state information, and configuration information between one another in a peer-to-peer fashion such that the central management component need only coordinate the connections between the nodes at the outset.

These techniques may provide several advantages. In cases where a customer's data is already running through the network of a particular cloud service provider, such as where the cloud service provider is also the customer's internet service provider (ISP), the present techniques allow the data to be processed by a node on the cloud service provider's network rather than being backhauled to a central server or other location outside of the provider's network for processing. Avoiding this data backhaul may reduce network costs and increase efficiency. Further, the ability to process the customer's data using a node at the cloud service provider may allow the customer's data to be segregated from the data of other customers for security purposes, rather than backhauled to a central location and processed by a shared resource. This may be accomplished spawning virtual nodes specifically for the particular customer, and configuring the nodes to only process network traffic for that particular customer.

The present approach may also offer the ability to leverage globally available cloud infrastructure to service mobile users as they travel abroad, and may improve speed and performance by servicing remote users using a cloud node that is geographically nearby. In addition, the system may provide a configurable upgrade policy that allows even globally distributed organizations to control when upgrades occur in the cloud, including configuring on-demand upgrades and different upgrade schedules, depending on geographic location.

<FIG> is a block diagram of an example computer system <NUM> for implementing and managing a hybrid cloud computing network. As shown the computer system <NUM> includes a network manager <NUM> connected with cloud provider networks <NUM> and <NUM>, and a customer network <NUM>. The network manager <NUM> is in communication with nodes <NUM>, <NUM>, and <NUM> located on the various networks as shown. Node management traffic <NUM> is transmitted between the network manger <NUM> and the nodes <NUM>, <NUM>, and <NUM>. Node sync traffic <NUM> is transmitted between the nodes <NUM>, <NUM>, and <NUM>. Clients <NUM>, <NUM> are connected to the customer network <NUM> and the cloud service provider <NUM>, as shown. Nodes <NUM>, <NUM>, and <NUM> are organized into cluster <NUM>.

The network manager <NUM> oversees and manages the nodes <NUM>, <NUM>, <NUM> to coordinate them into a hybrid cloud network. As shown, two-way node management traffic <NUM> is exchanged between the network manager and the nodes <NUM>, <NUM>, <NUM>. In some cases, this node management traffic may include initial configuration messages to initialize a new node (e.g., calls to the particular cloud provider's API) and configure it for processing. The node management traffic <NUM> may also include heartbeat message (e.g., status indications) from the nodes <NUM>, <NUM>, <NUM> indicating their current status. In some cases, such as a case where the network manager <NUM> is prevented from connecting to a particular node by a firewall, the network manager <NUM> may respond to a heartbeat sent by a particular node with configuration messages.

As shown, the network manager <NUM> groups the nodes <NUM>, <NUM>, <NUM> into cluster <NUM>. A cluster is a grouping of nodes that are configured identically and are of the same type (different node types are discussed below). The nodes in the cluster <NUM> communicate with one another to synchronize various data, such as configuration data, usage data, analysis results data, program quarantine data, and other types of data. This synchronization process is discussed in greater detail in the description of <FIG> below.

In some cases, the nodes <NUM>, <NUM>, and <NUM> may analyze the network traffic received from clients (e.g., <NUM>, <NUM>), and forward the traffic onto the intended destination, such as a website or other resource on the Internet. The network traffic received from the clients <NUM>, <NUM> may include traffic using different communications protocols, such as, for example, Hypertext Transfer Protocol (HTTP), Domain Name System (DNS) protocol, File Transfer Protocol (FTP), or other protocols. In some cases, the nodes <NUM>, <NUM>, and <NUM> may also receive and process network traffic sent from resources on the external network to the clients <NUM>, <NUM>, such as webpages, files, or other data sent from servers on the Internet in response to requests by the clients.

In some cases, the network traffic sent from the clients <NUM>, <NUM> to the cloud provider <NUM> and the customer network <NUM> may be encrypted, such as, for example, using Hypertext Transfer Protocol Secure (HTTPS), Internet Protocol Security (IPSec) tunnels or other Virtual Private Network (VPN) techniques, Layer <NUM> Medium Access Control (MAC) Address redirection, Generic Routing Encapsulation (GRE), Web Cache Communication Protocol (WCCP), or other techniques. In some cases, the clients <NUM>, <NUM> may include a software agent executing locally to forward the network traffic to the appropriate network. The cloud provider network <NUM> and the customer network <NUM> may also receive a copy or mirror of the network traffic from the clients <NUM>, <NUM> for processing.

Each of the cloud provider networks <NUM>, <NUM> and the customer network <NUM> may be a globally or regionally distributed network, with the nodes and other components of the system located across different geographic areas and connected by high-speed communications networks, such as, for example, optical networks, wireless networks, satellite networks, or other types of networks. In some cases, the components may be connected at least partially over the Internet. The networks connecting the components may utilize different protocols or technologies at different layers in the Open Systems Interconnection (OSI) model, including transport layer technologies such as Ethernet, Asynchronous Transfer Mode (ATM), or Synchronous Optical Networking (SONET), and network layer technologies such as Internet Protocol (IP), Transmission Control Protocol (TCP), or Universal Datagram Protocol (UDP). The components of the cloud system <NUM> may communicate over these networks using application layer communications protocols, such as, for example, HTTP, FTP, Simple Object Access Protocol (SOAP), Remote Procedure Call (RPC), or using other proprietary or public protocols for application programming interfaces (APIs).

The clients <NUM>, <NUM> may be computing devices such as PCs, laptops, tablets, telephones, servers, routers, storage devices or other network enabled computing devices. The clients <NUM>, <NUM> may be computing devices owned or controlled by the customer or and cloud provider, respectively, and may be used by employees of each. In some cases, the clients <NUM>, <NUM> may not be owned or controlled by the customers, such as in the case the network is a bring your own device (BYOD) network, or an access network such as an Internet service provider (ISP) network.

The system <NUM> includes nodes <NUM>, <NUM>, <NUM>. As described above, nodes are resources within the system <NUM> configured to process data, such as network traffic received from clients <NUM>, <NUM>. The system <NUM> may include different types of nodes, such as, for example, web security nodes, reporting nodes, and sandbox nodes. The different types of nodes within the system <NUM> may be configured to perform different functions.

For example, web security nodes may be configured to analyze received network traffic and apply network policies to the traffic, such as by selectively blocking, allowing, filtering, or performing other actions on the traffic based on the configuration attribute set by the particular customer to which the particular node is assigned. For example, web security nodes may filter requests for content from the clients <NUM>, <NUM>, and/or content sent from external resources to the clients <NUM>, <NUM>. Content matching certain parameters specified by the customer may be filtered, such as, for example, requests to certain domain names or Universal Resource Locators (URLs), requests for or responses including specific file types, traffic formatted according to certain protocols, traffic from certain users or clients, or other parameters. The web security nodes may also identify and log (e.g., store with a reporting node) particular network events, including actual or suspected malware intrusions, actual or suspected network breaches, visits by clients to malicious, unsafe, or inappropriate websites, downloads of malicious, unapproved, or unlicensed software by clients, or other events. The web security nodes may also identify and store behavioral data, such as client or user network activity, network flows, or other data. In some cases, the web security nodes may be configured to provide proxy service to clients of an assigned customer by forwarding requests received from the clients to appropriate external resources, and forwarding responses from the resources back to the clients. Such forwarding may be selective based on the filtering functionality discussed above.

Reporting nodes may be configured to store network traffic and/or results of analysis by other nodes, and to produce reports based on the stored data for presentation to users or administrators of the system <NUM>. The reports may include, but are not limited to, drill down reports allowing network activity to be viewed at both specific and high levels, event logs showing network traffic or other events matching particular criteria, real-time dashboards providing views of the current state of a customer's network traffic, incident response dashboards for monitoring issues with the customers network traffic, and other reports.

Sandbox nodes may be configured to execute malicious or potentially malicious software programs in a virtual environment to allow the behavior of the programs to be analyzed without adverse effects to other computing devices external to the sandbox. In some cases, the malicious software programs may be identified by a web security node such as in a response from an external resource to request from a client. In addition to blocking the download of the malicious software program, the web security node may provide the identified malicious software program to sandbox node for execution and analysis.

The system <NUM> may include other types of nodes. A risk assessment node may calculate a risk score for identified security events (e.g., intrusions, data exfiltration, denial of service attacks, or other events) in order to allow prioritization of the events based on a level of risk, which may facilitate planning of a remedy or response by the effected organization. For example, the risk assessment node may assign a higher risk score to a data exfiltration involving malicious removal of sensitive data from customer network <NUM>, and assign a lower risk score to an intrusion on the customer network <NUM> that did not access any sensitive data. Such a risk score may be generated based on network traffic received from the clients <NUM>, or based on data generated or stored by other nodes in the system <NUM>.

A log indexer node may organize data stored by a reporting node in a specific way to allow it to be accessed quickly, such as by another node within the system <NUM>, or by a user or administrator of the cloud system <NUM> through a user interface.

As previously discussed, the nodes of the system <NUM> may be physical computing devices (physical nodes) or virtual machine instances within a virtual machine environments executed by physical computing devices (virtual nodes). A virtual node may also be a containerized program executing an operating-system-level virtualization architecture such as DOCKER. The system <NUM> may include both physical nodes and virtual nodes.

<FIG> is a block diagram of an example computer system <NUM> for synchronizing data between different nodes of the hybrid cloud computing network. As shown, the passive node sync traffic <NUM> is exchanged between master node <NUM> and the slave node <NUM> via the firewall <NUM>. Active node sync traffic <NUM> is exchanged between the slave node <NUM> and the master node <NUM>. In some cases, the slave nodes <NUM> and <NUM> may also exchange passive node sync traffic between one another.

The master node <NUM> ensures that data that needs to be synchronized is kept up to dates on all nodes in a cluster. In some cases, the master node <NUM> may be designated by the network manager <NUM>. The master node <NUM> may also be chosen by the nodes of a cluster themselves, such as by random or pseudo-random selection or based on an objective criteria such as average network latency. The master node <NUM> may be either a virtual or physical node and can reside in any of the cloud provider and customer networks that make up the system <NUM>.

As shown, the passive node sync traffic <NUM> is exchanged between master node <NUM> and the slave node <NUM> via the firewall <NUM>. This passive exchange involves the slave node first sending an outbound request to the master node <NUM>. Because the slave node <NUM> has initiated the connection to master node <NUM>, the firewall <NUM> will, in most cases, allow a response from master node <NUM> to reach slave node <NUM>. Firewalls are generally configured to block unsolicited connections from external devices unless a specific rule is configured to allow them (e.g., on a particular port). Thus, this passive synchronization mechanism allows the node synchronization to take place even if a firewall is present. In some cases, the network manager <NUM> may analyze network traffic (e.g., heartbeats) received from the slave node <NUM> (or other information) to determine that the slave node <NUM> is behind a firewall. In such a case, the network manager <NUM> will configure the slave node <NUM> to utilize passive node synchronization, and configure all other nodes to communicate with slave node <NUM> in this way. For a node that is not behind a firewall (e.g., slave node <NUM>), the network manager <NUM> will configure the node for active node synchronization in which other nodes can freely connect to the node in order to synchronize.

<FIG> is an example user interface <NUM> for configuring various features of the hybrid cloud computing network. The user interface <NUM> may be presented to a user, such as through a web browser, and may receive input from the user, for example in the form of keystrokes or mouse clicks. The user interface <NUM> includes an array of visual tiles (e.g., <NUM>, <NUM>) each associated with the particular function of the cloud computing system. Before accessing user interface <NUM>, the user may have provided login credentials to a multi-tenant authentication system, and a system that presents the user interface <NUM> may itself be multi-tenant. When the user activates one of the visual tiles, a request to a node associated with the particular function denoted by the tile is generated. This request is sent to a single-tenant node assigned to the customer with which the user is associated. The single-tenant node may respond with a subsequent user interface (e.g., a webpage to be rendered in the user's browser) allowing the user to access or change data associated with the particular customer. For example, when a user associated with a customer A clicks on the web security tile <NUM>, a request may be sent to a web security node assigned to customer A. The web security node may respond to the user with a webpage including configuration or other data associated with customer A. If a user from another customer clicks on the web security tile <NUM>, a request would be generated to a different web security node associated with that customer. In this way, a global, multi-tenant user interface may be implemented to service multiple customers of the cloud computing system, while requests involving customer data are still handled by single-tenant nodes dedicated to that particular customer.

<FIG> is a flow chart showing a process <NUM> for implementing and managing a hybrid cloud computing network. At <NUM>, a first node is configured to participate in a node cluster, wherein the first node is hosted by a first cloud service provider, and wherein participating in the node cluster includes performing one or more processing actions specific to the node cluster on data received by the node. In some cases, the first node is a virtual machine instance, and configuring the first node includes calling an application programming interface (API) provided by the first cloud service provider to create the virtual machine instance. In some implementations, the one or more processing actions specific to the node cluster may include performing threat analysis and selectively performing corrective action in response to receiving network traffic from a network client.

In some implementations, the network client is a client of the first cloud service provider, and the first node is configured to perform threat analysis and selectively perform corrective action on network traffic received from the network client. In some cases, the network client is a client of the second cloud service provider, and the second node is configured to perform threat analysis and selectively perform corrective action on network traffic received from the network client. In some cases, threat analysis may include analyzing traffic sent and received by the network client for patterns indicative of viruses or malware, executing downloaded programs in a sandbox environment, or other types of analysis. In some cases, corrective action may include blocking a request form the client, blocking a response to the client, disconnecting the client from the network, performing a virus scan on the client, or other corrective actions.

In some cases, a third node is configured to participate in the node cluster, and the third node is a physical computing device hosted in a private network. In some implementations a status indication from the third node over the public network. In response to receiving the status indication, a passive synchronization mechanism is determined for the third node based on its location on a private network. In response to determining the passive synchronization mechanism, the passive synchronization mechanism is transmitted to the first node over the network. In some cases, the passive synchronization mechanism is configured to allow the third node to obtain synchronization information from other nodes in the cluster without accepting inbound connections from the public network.

At <NUM>, a second node is configured to participate in the node cluster, wherein the second node is hosted by a second cloud service provider different than the first cloud service provider. At <NUM>, a status indication is received from the first node over a network. At <NUM>, in response to receiving the status indication, a synchronization mechanism is determined for the first node based on a network configuration of the first node, wherein the determined synchronization mechanism is configured to allow the first node to acquire synchronization data from other nodes in the node cluster. At <NUM>, in response to determining the synchronization mechanism, the determined synchronization mechanism is transmitted to the first node over the network.

<FIG> is a block diagram of computing devices <NUM>, <NUM> that may be used to implement the systems and methods described in this document, as either a client or as a server or plurality of servers. Computing device <NUM> is intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. Computing device <NUM> is intended to represent various forms of mobile devices, such as personal digital assistants, cellular telephones, smartphones, and other similar computing devices. Additionally computing device <NUM> or <NUM> can include Universal Serial Bus (USB) flash drives. The USB flash drives may store operating systems and other applications. The USB flash drives can include input/output components, such as a wireless transmitter or USB connector that may be inserted into a USB port of another computing device.

Additionally, the processor may be implemented using any of a number of architectures. For example, the processor <NUM> may be a CISC (Complex Instruction Set Computers) processor, a RISC (Reduced Instruction Set Computer) processor, or a MISC (Minimal Instruction Set Computer) processor.

The information carrier is a computer- or machine-readable medium, such as the memory <NUM>, expansion memory <NUM>, or memory on processor <NUM> that may be received, for example, over transceiver <NUM> or external interface <NUM>.

As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, apparatus and/or device (e.g., magnetic discs, optical disks, memory, Programmable Logic Devices (PLDs)) used to provide machine instructions and/or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal.

Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), peer-to-peer networks (having ad-hoc or static members), grid computing infrastructures, and the Internet.

Claim 1:
A computer-implemented method comprising:
configuring, by a network manager (<NUM>) managing nodes in a node cluster (<NUM>), a first node to participate in the node cluster (<NUM>), wherein the first node is hosted by a first cloud service provider, and wherein participating in the node cluster includes performing one or more processing actions specific to the node cluster on data received by the first node;
configuring, by the network manager (<NUM>), a second node to participate in the node cluster, wherein the second node is hosted by a second cloud service provider different than the first cloud service provider;
designating, by the network manager (<NUM>), the first node hosted by the first cloud service provider as a master node (<NUM>), wherein the second node hosted by the second cloud service provider is designed as a slave node (<NUM>), and wherein the master node (<NUM>) is configured to synchronize data on nodes of the node cluster;
analyzing, by the network manager (<NUM>), network traffic received from the slave node hosted by the second cloud service provider to determine that the slave node is behind a firewall; and
in response to determining that the slave node is behind a firewall,
configuring, by the network manager (<NUM>), the slave node for a passive synchronization mechanism, wherein the passive synchronization mechanism is configured to allow the slave node to acquire synchronization data from the master node (<NUM>) in the node cluster via a firewall, wherein the slave node is configured to send an outbound request to the master node (<NUM>) before establishing a connection with the master node (<NUM>) through the firewall to acquire the synchronization data according to the passive synchronization mechanism,
configuring, by the network manager (<NUM>), other nodes in the node cluster to communicate with the slave node according to the passive synchronization mechanism, and
configuring, by the network manager (<NUM>), an active synchronization mechanism for another node in the node cluster that is not behind a firewall for exchange of data with the master node.