SECURE SERVICE ACCESS WITH MULTI-CLUSTER NETWORK POLICY

Techniques associated with exchanging data between clusters are disclosed. A data packet can be received from a first pod in a first cluster of a cluster set that targets a second pod or service in a second cluster of the cluster set. A label identity is determined for the first pod from a table of pods and label identities. The label identity for the first pod is added in a virtual network identifier field of a data packet header. The data packet is communicated from a first virtual switch to the second cluster through a tunnel interface and gateway node. Upon receipt of the data packet, the label identity is extracted from the data packet header, and an ingress rule associated with the label identity can be determined. Access to the second pod is controlled based on the rule.

BACKGROUND

Software defined networking (SDN) involves a plurality of hosts in communication over a physical network infrastructure of a data center (e.g., an on-premise data center or a cloud data center). The physical network to which the plurality of physical hosts is connected may be referred to as an underlay network. Each host has one or more virtualized endpoints, such as virtual machines (VMs), containers, Docker containers, data compute nodes, isolated user space instances, namespace containers, and/or other virtual computing instances (VCIs), that are connected to, and may communicate over, logical overlay networks. For example, the VMs and/or containers running on the hosts may communicate with each other using an overlay network established by hosts using a tunneling protocol.

A container is a package that relies on virtual isolation to deploy and run applications that access a shared operating system (OS) kernel. Containerized applications, also referred to as containerized workloads, can include a collection of one or more related applications packaged into one or more groups of containers, referred to as pods.

Containerized workloads may run in conjunction with a container orchestration platform that automates much of the operational effort required to run containers with workloads and services. This operational effort includes a wide range of things needed to manage a container's lifecycle, including, but not limited to, provisioning, deployment, scaling (e.g., up and down), networking, and load balancing. Kubernetes® (K8S®) software is an example open-source container orchestration platform that automates the operation of such containerized workloads. A container orchestration platform may manage one or more clusters, such as a K8S cluster, including a set of nodes that run containerized applications.

As part of an SDN, any arbitrary set of VCIs in a data center may be placed in communication across a logical Layer 2 (L2) overlay network by connecting them to a logical switch. A logical switch is an abstraction of a physical switch collectively implemented by a set of virtual switches on each node (e.g., host machine or VM) with a VCI connected to the logical switch. The virtual switch on each node operates as a managed edge switch implemented in software by a hypervisor or operating system (OS) on each node. Virtual switches provide packet forwarding and networking capabilities to VCIs running on the node. In particular, each virtual switch uses hardware-based switching techniques to connect and transmit data between VCIs on a same node or different nodes.

A pod may be deployed on a single VM or a physical machine. The single VM or physical machine running a pod may be referred to as a node running the pod. From a network standpoint, containers within a pod share the same network namespace, meaning they share the same internet protocol (IP) address or IP addresses associated with the pod.

A network plugin, such as a container networking interface (CNI) plugin, may be used to create virtual network interface(s) usable by the pods for communicating on respective logical networks of the SDN infrastructure in a data center. In particular, the network plugin may be a runtime executable that configures a network interface, referred to as a pod interface, into a container network namespace. The network plugin is further configured to assign a network address (e.g., an IP address) to each created network interface (e.g., for each pod) and may also add routes relevant to the interface. Pods can communicate with each other using their respective IP addresses. For example, packets sent from a source pod to a destination pod may include a source IP address of the source pod and a destination IP address of the destination pod so that the packets are appropriately routed over a network from the source pod to the destination pod.

Communication between pods of a node may be accomplished through use of virtual switches implemented in nodes. Each virtual switch may include one or more virtual ports (Vports) that provide logical connection points between pods. For example, a pod interface of a first pod and a pod interface of a second pod may connect to Vport(s) provided by the virtual switch(es) of their respective nodes to allow for communication between the first and second pods. In this context, “connect to” refers to the capability of conveying network traffic, such as individual network packets or packet descriptors, pointers, or identifiers, between components to effectuate a virtual data path between software components.

Within a single cluster, the container orchestration platform supports network plugins for cluster networking, with such network plugins mainly focusing on pods and services within the single cluster. A service is an abstraction to expose an application running on a set of pods as a network service. While a client may make a request for the service, the request may be load balanced to different instances of the application (i.e., different pods). However, many Cloud providers operate multiple clusters in multiple regions or availability zones and run replicas of the same applications in several clusters.

SUMMARY

One or more embodiments of a method for exchanging data between member clusters comprises receiving a data packet from a first pod in a first cluster of a cluster set through a pod interface, in which the data packet targets a second pod in a second cluster of the cluster set, determining a label identity for the first pod from a table of pods and label identities, adding the label identity for the first pod in a virtual network identifier field of the data packet header, and communicating the data packet from a first virtual switch to the second cluster through a tunnel interface and gateway node. The method may further comprise receiving the data packet in a second virtual switch of the second cluster through a second gateway node and second tunnel interface of the second cluster, extracting the label identity from the data packet, determining an ingress rule associated with the label identity, and controlling access to the second pod based on the rule.

Further embodiments include one or more non-transitory computer-readable storage media storing instructions that, when executed by one or more processors of a computer system, cause the computer system to perform the method set forth above, and a computer system including at least one processor and memory configured to carry out the method set forth above.

To facilitate understanding, identical reference numerals have been used, where possible, to designate identical elements that are common to the figures. It is contemplated that elements disclosed in one embodiment may be beneficially utilized in other embodiments without specific recitation.

DETAILED DESCRIPTION

A network policy can be defined and enforced for a single cluster. A network policy is a set of rules that define how network traffic is allowed to flow and can be utilized to enforce security and control access to network resources. Traffic flow between pods and services within a cluster can be controlled in certain instances. For example, an administrator can define a network policy that specifies which pods can communicate with each other and which cannot. Further, a network policy can specify a set of ingress and egress rules that control traffic coming into a pod or service (e.g., ingress) and traffic leaving a pod or service (e.g., egress).

Techniques exist that enable applications to communicate with each other across clusters beyond the communication occurring in a single cluster, such that pods and services are accessible across clusters. A controller of each cluster may select one or more nodes (e.g., a plurality of nodes) as a gateway for the cluster. Each gateway in each cluster forms a tunnel with gateways of each other cluster. The tunnels may be formed using any suitable tunneling protocol (e.g., GENEVE, VXLAN, GRE, STT, L2TP). Accordingly, the gateways of each cluster can communicate with one another over the formed tunnels. Each node within each cluster is further configured to route traffic for a destination to another cluster, referred to as cross-cluster traffic, through the gateway of the cluster. A first gateway of the source node tunnels the traffic to a second gateway of the destination node. The second gateway of the destination node then routes the traffic to the destination node. A cluster set includes a plurality of member clusters, including pods or services that can communicate with each other through network tunnel connections between the gateways of the member clusters.

Techniques described herein pertain to extending network policy support beyond a single cluster to multiple cluster network traffic. A stretch or cross-cluster network policy (referred to herein as a network policy) can specify rules enforced regarding traffic flow between pods in different clusters. A network policy can be specified for different scopes, such as cluster and cluster set, in certain embodiments. A cluster scope can pertain to a traditional single cluster, and the cluster set scope can correspond to a group of clusters. In certain embodiments, a unique label identity can be determined for pods to match cross-cluster traffic accurately. The unique label identity can be generated from a normalized label string associated with a pod that combines pod labels and labels of respective namespaces in certain embodiments. Rules derived from a high-level network policy can be specified with respect to label identities. The rules and label identities can be distributed to cluster members through import from a cluster leader. Any packet flowing across cluster boundaries can carry the label identity of an initiating pod, such as in a virtual network identifier (VNI) field of the packet header. After a data packet reaches a target cluster, the label identity can be extracted and utilized to determine and enforce any rules associated with the label identity to permit or deny access to a destination pod.

FIG.1depicts examples of physical and virtual network components in a networking environment100where embodiments of the subject disclosure may be implemented.

Networking environment100includes a data center101. Data center101includes one or more hosts102, a management network192, a data network170, a network controller174, a network manager176, and a container control plane178including a multi-cluster controller180. Data network170and management network192may be implemented as separate physical networks or as separate virtual local area networks (VLANs) on the same physical network.

Host(s)102may be communicatively connected to data network170and management network192. Data network170and management network192are also referred to as physical or “underlay” networks, and may be separate physical networks or the same physical network as discussed. As used herein, the term “underlay” may be synonymous with “physical” and refers to physical components of networking environment100. As used herein, the term “overlay” may be used synonymously with “logical” and refers to the logical network implemented at least partially within networking environment100.

Host(s)102may be geographically co-located servers on the same rack or different racks in any arbitrary location in the data center. Host(s)102may be configured to provide a virtualization layer, also referred to as a hypervisor106, that abstracts processor, memory, storage, and networking resources of a hardware platform into multiple VMs1041-104X(collectively referred to herein as “VMs104” and individually referred to herein as “VM104”).

Host(s)102may be constructed on a server-grade hardware platform108, such as an x86 architecture platform. Hardware platform108of a host102may include components of a computing device such as one or more processors (CPUs)116, system memory118, one or more network interfaces (e.g., physical network interface cards (PNICs)120), storage122, and other components (not shown). A CPU116is configured to execute instructions, for example, executable instructions that perform one or more operations described herein and that may be stored in the memory and storage system. The network interface(s) enable host102to communicate with other devices through a physical network, such as management network192and data network170.

In certain aspects, hypervisor106implements one or more logical switches as a virtual switch140. Any arbitrary set of VMs in a datacenter may be placed in communication across a logical Layer 2 (L2) overlay network by connecting them to a logical switch. A logical switch is an abstraction of a physical switch that is collectively implemented by a set of virtual switches on each host that has a VM connected to the logical switch. The virtual switch on each host operates as a managed edge switch implemented in software by a hypervisor on each host. Virtual switches provide packet forwarding and networking capabilities to VMs running on the host. In particular, each virtual switch uses hardware-based switching techniques to connect and transmit data between VMs on a same host or different hosts.

Virtual switch140may be attached to a default port group defined by a network manager that provides network connectivity to host102and VMs104on host102. Port groups include subsets of virtual ports (“Vports”) of a virtual switch, each port group having a set of logical rules according to a policy configured for the port group. Each port group may comprise a set of Vports associated with one or more virtual switches on one or more hosts102. Ports associated with a port group may be attached to a common VLAN according to the IEEE 802.1Q specification to isolate the broadcast domain.

A virtual switch140may be a virtual distributed switch (VDS). In this case, each host102may implement a separate virtual switch corresponding to the VDS, but the virtual switches140at each host102may be managed like a single virtual distributed switch (not shown) across the hosts102.

Each of VMs104running on host102may include virtual interfaces, often referred to as virtual network interface cards (VNICs), such as VNICs146, which are responsible for exchanging packets between VMs104and hypervisor106. VNICs146can connect to Vports144, provided by virtual switch140. Virtual switch140also has Vport(s)142connected to PNIC(s)120, allowing VMs104to communicate with virtual or physical computing devices outside of host102through data network170or management network192.

Each VM104may also implement a virtual switch148for forwarding ingress packets to various entities running within the VM104. Such virtual switch148may run on a guest OS138of the VM104, instead of being implemented by a hypervisor, and may be programmed, for example, by agent110running on guest OS138of the VM104. For example, the various entities running within each VM104may include pods154including containers130. Depending on the embodiment, the virtual switch148may be configured with Open vSwitch (OVS), an open-source project to implement virtual switches to enable network automation while supporting standard management interfaces and protocols.

In particular, each VM104implements a virtual hardware platform that supports the installation of a guest OS138, which is capable of executing one or more applications. Guest OS138may be a standard commodity operating system. Examples of a guest OS include Microsoft Windows®, Linux®, or the like.

Each VM104may include a container engine136installed therein and running as a guest application under the control of guest OS138. Container engine136is a process that enables the deployment and management of virtual instances (referred to interchangeably herein as “containers”) by providing a layer of OS-level virtualization on guest OS138within VM104or an OS of host102. Containers130are software instances that enable virtualization at the OS level. With containerization, the kernel of guest OS138, or an OS of host102if the containers are directly deployed on the OS of host102, is configured to provide multiple isolated user-space instances, referred to as containers. Containers130appear as unique servers from the standpoint of an end user that communicates with each of containers130. However, from the standpoint of the OS on which the containers execute, the containers are user processes that are scheduled and dispatched by the OS.

Containers130encapsulate an application, such as application132, as a single executable software package that bundles application code with all the related configuration files, libraries, and dependencies required to run. Application132may be any software program, such as a word processing program or a gaming server.

Data center101includes a container control plane178. In certain aspects, the container control plane178may be a computer program that resides and executes in one or more central servers, which may reside inside or outside the data center101, or alternatively, may run in one or more VMs104on one or more hosts102. A user can deploy containers130through container control plane178. Container control plane178is an orchestration control plane, such as Kubernetes®, to deploy and manage applications or services thereof on nodes, such as hosts102or VMs104, of a node cluster, using containers130. For example, Kubernetes may deploy containerized applications as containers130and a container control plane178on a cluster of nodes. The container control plane178, for each cluster of nodes, manages the computation, storage, and memory resources to run containers130. Further, the container control plane178may support the deployment and management of applications (or services) on the cluster using containers130. In some cases, the container control plane178deploys applications as pods154of containers130running on hosts102, either within VMs104or directly on an OS of the host102. Other types of container-based clusters based on container technology, such as Docker® clusters, may also be considered. Though certain aspects are discussed with pods154running in a VM as a node, and container engine136, agent110, and virtual switch148running on guest OS138of VM104, the techniques discussed herein are also applicable to pods154running directly on an OS of host102as a node. For example, host102may not include hypervisor106, and may instead include a standard operating system. Further, agent110and container engine136may then run on the OS of host102.

Further, MC (multi-cluster) controller180can be included within or otherwise communicatively coupled with the container control plane178. The MC controller180is configured to connect multiple clusters together and support communications between pods running in different clusters. The MC controller can be configured to permit administrators to define network policies for traffic within a cluster. Moreover, the MC controller180can be configured to support an extended or stretch network policy, as described further herein, to allow administrators to specify cross-cluster network policies. In accordance with certain embodiments, the MC controller180can implement all portions of Antrea® or an Antrea® controller, where Antrea® is an open-source networking and security solution for clusters.

For packets to be forwarded to and received by pods154and their containers130running in a first VM1041, each of the pods154may be set up with a network interface, such as a pod interface165. The pod interface165is associated with an IP address, such that the pod154, and each container130within the pod154, is addressable by the IP address. Accordingly, after each pod154is created, network plugin124is configured to set up networking for the newly created pod154, enabling the new containers130of the pod154to send and receive traffic. As shown, pod interface1651is configured for and attached to a pod1541. Other pod interfaces, such as pod interface1652, may be configured for and attached to different, existing pods154.

The network plugin124may include a set of modules that execute on each node to provide networking and security functionality for the pods. In addition, an agent110may execute on each VM104(i) to configure the forwarding element and (ii) to handle troubleshooting requests. In addition, MC controller180may provide configuration data (e.g., forwarding information, network policy to be enforced) to agents110, which use this configuration data to configure the forwarding elements (e.g., virtual switches) on their respective VMs104, also referred to as nodes104. Agent110may further be configured to forward node104or cluster information. In certain embodiments, VM104can correspond to one of a plurality of clusters in a cluster set that is either a member cluster or a leader cluster.

Data center101includes a network management plane and a network control plane. The management plane and control plane each may be implemented as single entities (e.g., applications running on a physical or virtual compute instance) or as distributed or clustered applications or components. In alternative aspects, a combined manager/controller application, server cluster, or distributed application may implement both management and control functions. In the embodiment shown, network manager176at least in part implements the network management plane, and network controller174and container control plane178in part implement the network control plane.

The network control plane is a component of software defined network (SDN) infrastructure and determines the logical overlay network topology and maintains information about network entities such as logical switches, logical routers, and endpoints. The logical topology information is translated by the control plane into physical network configuration data that is then communicated to network elements of host(s)102. Network controller174generally represents a network control plane that implements software defined networks, e.g., logical overlay networks, within data center101. Network controller174may be one of multiple network controllers executing on various hosts in the data center that together implement the functions of the network control plane in a distributed manner. Network controller174may be a computer program that resides and executes in a server in data center101, external to data center101(e.g., such as in a public cloud) or, alternatively, network controller174may run as a virtual appliance (e.g., a VM) in one of hosts102. Network controller174collects and distributes information about the network from and to endpoints in the network. Network controller174may communicate with hosts102via management network192, such as through control plane protocols. In certain aspects, network controller174implements a central control plane (CCP) that interacts and cooperates with local control plane components, e.g., agents, running on hosts102in conjunction with hypervisors106.

Network manager176is a computer program that executes in a server in networking environment100, or alternatively, network manager176may run in a VM104, e.g., in one of hosts102. Network manager176communicates with host(s)102via management network192. Network manager176may receive network configuration input from a user, such as an administrator, or an automated orchestration platform (not shown) and generate desired state data that specifies logical overlay network configurations. For example, a logical network configuration may define connections between VCIs and logical ports of logical switches. Network manager176is configured to receive inputs from an administrator or other entity, e.g., via a web interface or application programming interface (API), and carry out administrative tasks for data center101, including centralized network management and providing an aggregated system view for a user.

An example container-based cluster for running containerized workloads is illustrated inFIG.2. It should be noted that the block diagram ofFIG.2is a logical representation of a container-based cluster and does not show where the various components are implemented and run on physical systems. While the example container-based cluster shown inFIG.2is a Kubernetes (K8S) cluster200, in other examples, the container-based cluster may be another type based on container technology, such as Docker® clusters.

When Kubernetes is used to deploy applications, a cluster, such as a single Kubernetes cluster200, is formed from a combination of worker nodes104and a control plane178. Though worker nodes104are shown as VMs104ofFIG.1, as discussed, the worker nodes104instead may be physical machines. In certain aspects, components of container control plane178run on VMs or physical machines. Worker nodes104are managed by control plane178, which manages the computation, storage, and memory resources to run all worker nodes104. Though pods154of containers130are shown running on cluster200, the pods may not be considered part of the cluster infrastructure but rather as containerized workloads running on cluster200.

Each worker node104, or worker compute machine, includes a kubelet210, which is an agent that ensures that one or more pods154run in the worker node104according to a defined specification for the pods, such as defined in a workload definition manifest. Each pod154may include one or more containers130. The worker nodes104can execute various applications and software processes using container130. Further, each worker node104includes a kube proxy220. Kube proxy220is a Kubernetes network proxy that maintains network rules on worker nodes104. These network rules allow network communication to pods154from network sessions inside or outside the Kubernetes cluster200.

Control plane178includes components such as an application programming interface (API) server240, a cluster store (etcd)250, a controller260, MC controller180, and a scheduler270. Components of the control plane178make global decisions about the Kubernetes cluster200(e.g., scheduling), as well as detect and respond to cluster events (e.g., starting up a new pod154when a workload deployment's replicas field is unsatisfied).

API server240operates as a gateway to Kubernetes cluster200. As such, a command line interface, web user interface, users, or services communicate with Kubernetes cluster200through API server240. One example of a Kubernetes API server240is kube-apiserver, which kube-apiserver is designed to scale horizontally—that is, this component scales by deploying more instances. Several instances of kube-apiserver may be run, and traffic may be balanced between those instances.

Cluster store (etcd)250is a data store, such as a consistent and highly-available key-value store, used as a backing store for data of the Kubernetes cluster200. In accordance with certain embodiments, a network policy and/or rules derived from the network policy can be stored in cluster store250. As discussed later herein, generated label identifiers can also be saved in cluster store250.

Controller260is a control plane178component that runs and manages controller processes in Kubernetes cluster200. For example, control plane178may have (e.g., four) control loops called controller processes that watch the state of cluster200and try to modify the current state of cluster200to match an intended state of cluster200. In certain aspects, controller processes of controller260are configured to monitor external storage for changes to the state of cluster200.

The MC controller180is configured to enable data flow between different clusters. Furthermore, the MC controller180can include functionality that allows administrators to define network policies that specify how traffic should be permitted or blocked between pods and services in the same cluster and across multiple clusters. In accordance with certain embodiments, The MC controller180check a label identity registry for all clusters and translate a high-level network policy (e.g., specified with label selectors) into data plane rules written with respect to label identities for enforcement. Though shown as separate, in certain aspects, MC controller180functionality may be part of controller260.

Scheduler270is a control plane178component configured to allocate new pods154to worker nodes104. Additionally, scheduler270may be configured to distribute resources and/or workloads across worker nodes104. Resources may refer to processor resources, memory resources, networking resources, and/or the like. Scheduler270may watch worker nodes104for how well each worker node104handles its workload and match available resources to the worker nodes104. Scheduler270may then schedule newly created containers130to one or more worker nodes104.

In other words, control plane178manages and controls components of a cluster. Control plane178handles most, if not all, operations within the Kubernetes cluster200, and its components define and control cluster configuration and state data. Control plane178configures and runs the deployment, management, and maintenance of the containerized applications.

FIG.3depicts a resource exchange pipeline300in accordance with an example embodiment. Three clusters are depicted: cluster A310A, cluster B310B, and cluster C310C(collectively referred to as clusters310). The clusters310can comprise a cluster set that is a group of clusters with a high degree of mutual trust that share services amongst themselves and work together as a single system.

An MC controller can be configured to synchronize services across clusters310and makes the services available for cross-cluster service discovery and connectivity. In accordance with certain embodiments, MC controllers can be decentralized and run in each cluster of a cluster set with two different roles: leader cluster and member clusters. As illustrated, cluster A310Aand cluster B310Bare member clusters, and cluster C310Cis the leader cluster. Further, each of the clusters310includes a respective API server240A,240B, and240Cand MC controller180A,180B, and180C.

The leader cluster310Cis configured to act as the control plane for the entire cluster set to facilitate the distribution of resource exporting and importing among clusters. The leader cluster310C(which can also be a member cluster) can also enable initially declaring a cluster set and generating secret tokens to be distributed to potential member clusters. With the generated tokens, clusters can join the cluster set by securely connecting to the leader cluster API server240C.

Resources can be exchanged by members of the cluster set through a resource exchange pipeline. In accordance with certain embodiments, two custom resources can traverse the resource pipeline: export and import. Export encapsulates information regarding a resource, such as type and specification of a resource being exported. Import aggregates exported resources from different clusters and computes a final payload to be imported into each cluster. To implement a resource exchange pipeline, a common area is introduced where resources declared for export can be accessed by all members by resource imports.

The leader cluster310Cserves as the common area in the cluster set. Member clusters can monitor import events in the common area storage330through the API server240Cin the leader cluster310Cand reconcile to in-cluster resources, such as service and network policy, to match the desired state specified by a resource import. The MC controller running in each member cluster can also be responsible for creating resource exports for any resources marked for export.

Multiple resources can be enclosed into resource exports and imports for specific purposes, including service and endpoints, cluster information, cluster network policy, and label identity. As per network policy, in-cluster network policies can be replicated to peer clusters when an administrator creates a resource export including a desired network policy. A network policy can be created in the leader cluster310C, which can be distributed to member clusters (or a subset based on filter criteria). The imported policy can then be applied to individual clusters as if the policy was created in-cluster locally. The network policy can be created declaratively, making it effortless for an administrator of a multi-cluster deployment to define a consistent security posture across all clusters without additional tooling. Declarative policy specification is especially useful for ensuring namespaces are isolated across all clusters in a cluster set by default. Concerning label identity, a custom resource can exist for identifying unique pod labels. Each cluster's MC controller can export their own label identities for use in cross-cluster traffic policy enforcement.

In addition to policy replication, the MC controllers can enforce network policies on cross-cluster traffic. In certain embodiments, network policy features enable restriction of pod egress traffic to backends of a multi-cluster service regardless of whether they are on the same cluster as the source pod or a different cluster. However, enforcing policy on ingress traffic is a problem, as cross-cluster packets are often subject to source network address translation (SNAT), which modifies the source internet protocol (IP) addresses of hosts or nodes, making it difficult to apply IP-based source matching. Further, even if the original IP address is not changed, many workload pod IP addresses and labels must be synchronized among the entire cluster set to match cross-cluster label selectors. This synchronization process can significantly impact network bandwidth and the overall performance of the solution, especially given the ephemeral nature of pods.

These challenges and performance issues are overcome by using a label identity to match cross-cluster traffic accurately. In certain embodiments, member clusters can generate a normalized string for all pods, such as by combining pod labels and labels of respective namespaces. The normalized string is exported to the leader cluster310Cthrough the resource exchange pipeline. Label generator320is configured to generate a unique label identity for each unique normalized label string in the cluster set. All member clusters can then import all label identities to ensure they are synchronized across the cluster set. The label identity can be included with any data packet flowing in the cluster set to enable precise ingress cross-cluster packet matching.

FIG.4depicts an example method400of resource exchange between clusters. In block410, a resource export is detected from a cluster for a resource marked for export. InFIG.3, cluster A310A, a local resource can be marked for export by an administrator of cluster A310Athrough the API server240A. By marking a local resource for export, the administrator provides permission for the local resource to be transmitted outside the cluster A310Ato another cluster. The MC controller180Acan identify the resource marked for export and trigger export of the local resource to the leader cluster C310C. In block420, method400performs processing of the resource export. The processing can involve resource-particular computations and filtering. In certain embodiments, the processing can comprise generating unique labels from exported label strings by the label generator320ofFIG.3. In block430, method400publishes or otherwise makes the resource available for import by other clusters. For instance, cluster B310Bcan monitor the common area of leader cluster C for resources and import the resources as local resources to cluster B310Bthrough the MC controller180Band API server240B. In accordance with one particular embodiment, a network policy that controls intra-cluster traffic, inter-cluster traffic, or both can be specified by an administrator of the leader cluster C310Cand imported into cluster B310B(as well as cluster A310A) as a local network policy for enforcement.

FIG.5is a flow chart diagram of an example label identifier generation and distribution method500. Under certain embodiments, the method500can be implemented by the label generator320in conjunction with MC controller180Cand API server240Cof leader cluster C310CofFIG.3. In block510, the method500receives a normalized label string for a pod from a member cluster that combines pod labels and respective namespaces.

In block520, method500generates a unique label identity for the pod based on the received string. In accordance with one embodiment, the label identity can be calculated as follows: “‘ns’+labels.FormatLabels (podNamespaceLabels)+‘&pod’+labels.FormatLabels (podLa bels)” wherein “ns” and “&pod” are text that serves to delineate namespace and pod portions of the label identity and “FormatLabels” is a function that determines and returns namespace and pod labels. An example label identity may appear as follows: “ns: kubernetes.io/metadata.name=us=west-, purpose-test&pod: app=client.” In certain embodiments, namespace labels are included in situations where policies utilize namespace selectors in addition to pod selectors to select ingress peers across clusters.

In block530, method500publishes the label identity for import by other clusters. To enable replication of label identities in a cluster set so that each cluster knows what label identities match a policy, mechanisms to export and import these identities can be employed. In accordance with certain embodiments, custom resource definitions (CRD) are specified for exporting and importing label identities as follows. A reconciler may also be added to the MC controller to monitor pod and namespace create, read, update, and delete (CRUD) events and update all label identities in a cluster into a resource export object of type “LabelIdentities.”

In certain embodiments, another reconciler can be added to the leader cluster C310C, which monitors export of type “LabelIdentities” from all member clusters and assigns an identifier for each unique label identity in the cluster set. Certain embodiments can include creating a custom resource definition (CRD) object of type “LabelIdentityImport” for each label identity and identifier pair. The label generator320in the leader cluster C310Ccan translate “n” “ResourceImport” objects into “k” (number of unique label identities) “LabelIdentityImport” objects specified below:

FIG.6depicts cross-cluster traffic and network policy enforcement. There are two clusters: cluster X610Xand cluster Y610Y. Each cluster includes a corresponding regular node (regular node620Xand regular node620Y) and gateway node (gateway node680Xand gateway node680Y). The regular nodes620Xand620Yperform computation tasks and can communicate with other nodes in a cluster. The gateway nodes680Xand680Yenable communication outside the cluster by serving as a bridge between an internal cluster and another cluster. The regular nodes620Xand620Yalso include respective virtual switches650Xand650Y, which enable network communication between pods within a node and between a pod and external pods or services. Regular node620Xincludes pod X630Xthat interfaces with the virtual switch650Xby way of a pod interface640X. The virtual switch650Xincludes a classifier table660to look up a label identifier for a target of a cross-cluster communication. The tunnel interface670Xcan be a virtual network interface that creates secure connections between two or more nodes. The gateway node680Xenables cross-cluster communication. The gateway node680Yis configured to receive communications from other gateways and pass the communication to the tunnel interface670Yand the virtual switch650Y. The virtual switch can include a rule table662associated with or more network policies. A rule can be looked up in rule table662with the label identity associated with the source of the communication. The rule can specify that the communication be blocked or denied. Alternatively, the rule can indicate that the communication is allowed or permitted, which can then result in passing the communication to pod Y630Ythrough the pod interface640Yif pod Y630Yis the communication destination.

FIG.7is a flow chart diagram of an example method700of cross-cluster communication. Method700can be employed in conjunction with components associated with cross-cluster communication inFIG.6.

In block710, method700receives a data packet from a first pod in a first cluster targeting a second pod in a second cluster. InFIG.6, pod X630Xin cluster X610Xcan send a data packet to pod Y630Yin cluster Y610Y. In one embodiment, the virtual switch650Xcan receive the data packet from the pod X630Xthrough the pod interface640X.

In block720, method700determines a label identity. In accordance with certain embodiments, the classifier table660ofFIG.6can be utilized to look up a label identifier for the pod X630X, for instance, based on pod labels or namespace.

In block730, method700adds the label identity to the data packet header (e.g., tun_id). Any packet flowing between cluster boundaries can carry the label identifier of the initiating pod in the virtual network identifier (VNI) field of its header in some embodiments. The data packet with the label identity can be transmitted through the tunnel interface670Xand gateway node680Xto cluster Y610YinFIG.6.

In block740, method700receives the data packet in the second cluster. For instance, the data packet can be received by gateway node680Y. Subsequently, the data packet can be received by the regular node620Yand the virtual switch650Ythrough the tunnel interface670YinFIG.6.

In block750, method700extracts the label identity from the data packet. In certain embodiments, method700can extract the label identity from a VNI field in a header of the data packet.

In block760, method700identifies and applies zero or more policy rules based on the label identity. A network policy can be specified with respect to the label identity. Accordingly, in certain embodiments, zero or more rules can be identified from rule table662inFIG.6with a lookup of the label identity. If a rule blocks access to pod Y630Ybased on the label identity of the sending pod X630X, method700can terminate. Alternatively, if there is no rule or a rule that provides permission to communicate with pod Y630Y, then the data packet can be routed to pod Y630Ythrough the pod interface640Y, and method700can subsequently terminate.

What follows is an example of a multi-cluster network policy including an ingress rule that may be specified by a cluster set administrator.

The ingress rule specifies pods that are allowed to communicate with pods with an application label of “db.” All pods in the namespace “product-us-west” from all clusters in the cluster set are selected, and if the namespace exists in that cluster whose labels match an application of “client,” the pods that match the label are allowed to communicate with the application “db.” All other pods are not permitted to communicate with the application “db.” Here, the scope is set as cluster set as opposed to cluster to indicate that the policy applies to multiple clusters in a cluster set and not a single cluster.

Although not present in the example, the network policy can include, the policy can include additional rules that have the same destination and matching condition as original rules but use an unknown label, in accordance with certain embodiments. A label may be unknown due to a change pod label update or addition of a new pod. The network policy can control data packets with a normal label identifier (e.g., same format, similarity match that satisfies a threshold) and drop packets with unknown label identifiers. In this way, a preexisting pod need not lose connection awaiting a label identity update.

It should be understood that, for any process described herein, there may be additional or fewer steps performed in similar or alternative orders, or in parallel, within the scope of the various embodiments, consistent with the teachings herein, unless otherwise stated.

One or more embodiments may be implemented as one or more computer programs or as one or more computer program modules embodied in one or more computer-readable media. The term computer-readable medium refers to any data storage device that can store data that can thereafter be input to a computer system-computer readable media may be based on any existing or subsequently developed technology for embodying computer programs in a manner that enables them to be read by a computer. Examples of a computer-readable medium include a hard drive, network attached storage (NAS), read-only memory, random-access memory (e.g., a flash memory device), a CD (Compact Discs)—CD-ROM, a CD-R, or a CD-RW, a DVD (Digital Versatile Disc), a magnetic tape, and other optical and non-optical data storage devices. The computer-readable medium can also be distributed over a network-coupled computer system so that the computer-readable code is stored and executed in a distributed fashion.

Although one or more embodiments have been described in some detail for clarity of understanding, it will be apparent that certain changes and modifications may be made within the scope of the claims. Accordingly, the described embodiments are to be considered as illustrative and not restrictive, and the scope of the claims is not to be limited to details given herein, but may be modified within the scope and equivalents of the claims. In the claims, elements or steps do not imply any particular order of operation, unless explicitly stated in the claims.

In accordance with the various embodiments, virtualization systems may be implemented as hosted embodiments, non-hosted embodiments, or embodiments that tend to blur distinctions between the two are all envisioned. Furthermore, various virtualization operations may be wholly or partially implemented in hardware. For example, a hardware implementation may employ a look-up table to modify storage access requests to secure non-disk data.