System and method for providing services for guest container-based clusters without direct access to resource provider

A computing system and method for providing a service for a guest container-based cluster of the computing system utilizes a translated service software object created in a supervisor container-based cluster of the computing system to create at least one resource to support the service requested for the guest container-based cluster. The translated service software object is created in response to a service software object being created in the guest container-based cluster. In response to the translated service software object being created in the supervisor container-based cluster, at least one service-related software object is then created in the supervisor container-based cluster that corresponds to the translated service software object. Provider-specific instructions are then sent to a resource provider from the supervisor container-based cluster to create at least one resource associated with the at least one service-related software object to support the requested service for the guest container-based cluster.

BACKGROUND

Kubernetes® software is an open-source container orchestration software that automates the operation of containerized cloud applications. Kubernetes software can support data center outsourcing to public cloud service providers or can be used for web hosting at scale. Server and mobile applications with complex custom code can be deployed using Kubernetes software on commodity hardware to lower the costs on application provisioning with public cloud hosts and to optimize software development processes. Thus, Kubernetes software is rapidly emerging as a mainstream platform for container orchestration in both on-premise and cloud environments.

As increasing amount of workloads are being deployed with Kubernetes software, there are concerns that some communications within a Kubernetes environment may be exposed security risks. Thus, there is a need to minimize security risks in Kubernetes environments.

SUMMARY

A computing system and method for providing a service for a guest container-based cluster of the computing system utilizes a translated service software object created in a supervisor container-based cluster of the computing system to create at least one resource to support the service requested for the guest container-based cluster. The translated service software object is created in response to a service software object being created in the guest container-based cluster. In response to the translated service software object being created in the supervisor container-based cluster, at least one service-related software object is then created in the supervisor container-based cluster that corresponds to the translated service software object. Provider-specific instructions are then sent to a resource provider from the supervisor container-based cluster to create at least one resource associated with the at least one service-related software object to support the requested service for the guest container-based cluster.

A computer-implemented method for providing a service for a guest container-based cluster in a computing system in accordance with an embodiment of the invention comprises receiving a request for the service at the guest container-based cluster; in response to the request, creating a service software object corresponding to the service in the guest container-based cluster; in response to the service software object being created in the guest container-based cluster, creating a translated service software object in a supervisor container-based cluster of the computing system; in response to the translated service software object being created in the supervisor container-based cluster, creating at least one service-related software object in the supervisor container-based cluster that corresponds to the translated service software object; and in response to the at least one service-related software object being created in the supervisor container-based cluster, sending provider-specific instructions to a resource provider from the supervisor container-based cluster to create at least one resource associated with the at least one service-related software object to support the requested service for the guest container-based cluster. In some embodiments, the steps of this method are performed when program instructions contained in a non-transitory computer-readable storage medium are executed by one or more processors.

A computing system in accordance with an embodiment of the invention comprises memory and at least one processor configured to receive a request for a service at a guest container-based cluster of the computing system; in response to the request, create a service software object corresponding to the service in the guest container-based cluster; in response to the service software object being created in the guest container-based cluster, create a translated service software object in a supervisor container-based cluster of the computing system; in response to the translated service software object being created in the supervisor container-based cluster, create at least one service-related software object in the supervisor container-based cluster that corresponds to the translated service software object; and in response to the at least one service-related software object being created in the supervisor container-based cluster, send provider-specific instructions to a resource provider from the supervisor container-based cluster to create at least one resource associated with the at least one service-related software object to support the requested service for the guest container-based cluster.

DETAILED DESCRIPTION

Turning now toFIG. 1, a computing system100in accordance with an embodiment of the invention is illustrated. As shown inFIG. 1, the computing system100includes a supervisor container-based cluster102and a number of guest container-based clusters104, which are supported by hardware resources106and a virtualization infrastructure108. The supervisor and guest container-based clusters102and104are computing environments that offer compute, storage and network as resources for hosting or deployment of services or applications. In an embodiment, the supervisor and guest container-based clusters102and104may be Kubernetes® clusters. However, in other embodiments, the supervisor and guest container-based clusters102and104may be another type of container-based cluster based on container technology, such as Docker® clusters. As explained below, the computing system100uses a technique to avoid a requirement for the guest container-based clusters104to directly access the virtualization infrastructure108when services that need external resources are requested for the guest container-based clusters to reduce security exposure of the virtualization infrastructure, which is part of the management network of the computing system.

The hardware resources106of the computing system100include host computers (hosts)110, physical storage resources112and physical network resources114. These hardware resources may be provided by a cloud provider if the supervisor and guest container-based clusters102and104are deployed in a public cloud. Alternatively, these hardware resources may be part of an on-premises data center. Each of the hosts includes hardware components commonly found on a server grade computer, such as CPU, memory, network interface card and one or more storage devices. In addition, each host includes a virtualization layer that abstracts processor, memory, storage, and networking resources of the host's hardware into virtual machines that run concurrently on the host. In an embodiment, the virtual machines run on top of a hypervisor that enables sharing of the hardware resources of the host by the virtual machines. One example of a hypervisor may be used in the hosts is a VMware ESXi™ hypervisor provided as part of the VMware vSphere® solution made commercially available from VMware, Inc. The hypervisor of each host may run on top of the operating system of the host or directly on hardware components of the host.

The physical storage resources112may include the storage devices of the hosts110and/or other storage devices, which may be part of a storage system, such as a physical or virtual storage area network (SAN) or a network-attached storage (NAS).

The physical network resources114may include physical switches, physical routers and/or physical gateways. The physical switches provide connectivity between various components in the computing system100. The physical routers perform packet forwarding functions to ensure data are transmitted to their intended destinations. The physical gateways serve as junctions between the computing system100and other networks, such as other data centers and the Internet.

The virtualization infrastructure108of the computing system100provides virtualization to provision virtual compute, storage and network resources for the supervisor and guest container-based clusters102and104from the hardware resources106. In the illustrated embodiment, the virtualization infrastructure includes a virtualization manager and a software-defined network (SDN) manager. However, in other embodiments, the virtualization infrastructure may include other components that support the supervisor and guest container-based clusters.

The virtualization manager116is configured to carry out administrative tasks for the computing system100, including managing the hosts110, managing various virtual computing instances (VCIs), such as virtual machines and containers running on the hosts, provisioning new VCIs, migrating VCIs from one host to another host, and load balancing between the hosts. One example of the virtualization manager116is the VMware vCenter Server® product that is available from VMware, Inc.

The SDN manager118is configured to provide a graphical user interface (GUI) and REpresentational State Transfer (REST) application programming interfaces (APIs) for creating, configuring, and monitoring SDN components and resources, such as logical switches, edge services gateways, load balancers, virtual servers, server pools and pool members. Thus, the SDN manager can be viewed as a resource provider that can provide various resources for different components of the computing system100, such as the supervisor and guest container-based clusters102and104. The SDN manager allows configuration and orchestration of logical network components for logical switching and routing, networking and edge services, load balancing, and security services and distributed firewall (DFW). In some embodiments, the SDN manager is further configured to provide these functionalities in a public cloud computing environment. One example of the SDN manager is the NSX® manager of VMware NSX-T product that is available from VMware, Inc.

The virtualization manager116and the SDN manager118provide support for the supervisor container-based cluster102to be created and execute supervisory functionalities, as explained below. The supervisor container-based cluster includes multiple containers120that run various software processes or applications. A container is a package that relies on virtual isolation to deploy and run applications that access a shared operating system (OS) kernel. An example of a container is the container created using a Docker engine made available by Docker, Inc.

The supervisor container-based cluster102is configured to create and manage one or more guest container-based clusters104, which also includes containers120, using compute, network and storage resources available to the supervisor container-based cluster. Since guest container-based clusters are created by the supervisor container-based clusters, each of the guest container-based clusters can be viewed as a child container-based cluster and the supervisor container-based cluster can be viewed as the parent container-based cluster of that guest container-based cluster.

In some embodiments, the supervisor and guest container-based clusters102and104are Kubernetes® clusters that are deployed using containers running on virtual machines, which are hosted in a logical cluster of the host computers110. As used herein, a virtual machine is an emulation of a physical computer system in the form of a software computer that, like a physical computer, can run an operating system and applications, including containers. A virtual machine may be comprised of a set of specification and configuration files and is backed by the physical resources of the physical host computer. A virtual machine may have virtual devices that provide the same functionality as physical hardware and have additional benefits in terms of portability, manageability, and security. An example of a virtual machine is the virtual machine created using VMware vSphere® solution made commercially available from VMware, Inc of Palo Alto, Calif. A host computer200with containers running in one or more virtual machines in accordance with an embodiment of the invention, which is representative of the host computers110in the computing system100is illustrated inFIG. 2.

As shown inFIG. 2, the host computer is configured to support a number of virtual machines (VMs)220-1,220-2. . .220-x(where x is a positive integer). The number of VMs supported by the host computer can be anywhere from one to more than one hundred. The exact number of VMs supported by the host computer is only limited by the physical resources of the host computer. The VMs share at least some of the hardware resources of the host computer, which include system memory222, one or more processors224, a storage interface226, and a network interface228. The system memory222, which may be random access memory (RAM), is the primary memory of the host computer. The processor224can be any type of a processor, such as a central processing unit (CPU) commonly found in a server. The storage interface226is an interface that allows that host computer to communicate with a storage (not shown). As an example, the storage interface may be a host bus adapter or a network file system interface. The network interface228is an interface that allows the host computer to communicate with other devices and systems through one or more networks. As an example, the network interface may be a network interface controller (NIC).

In the illustrated embodiment, the VMs220-1,220-2. . .220-xrun on “top” of a hypervisor230, which is a software interface layer that, using virtualization technology, enables sharing of the hardware resources of the host computer200by the VMs. However, in other embodiments, one or more of the VMs can be nested, i.e., a VM running in another VM. Any computer virtualization architecture can be implemented. For example, the hypervisor may run on top of the host computer's operating system or directly on hardware of the host computer. With the support of the hypervisor, the VMs provide isolated execution spaces for one or more containers.

The VMs220-1,220-2. . .220-x, as well as containers232running in the VMs, are able to communicate with each other using an internal software OSI Layer 2 switch (not shown) and with other computer systems or components connected to the host computer via a network using the network interface228of the host computer200. In addition, the VMs, as well as the containers running in the VMs, are able to access storage using the storage interface226of the host computer.

Turning now toFIG. 3, an exemplary Kubernetes® (K8S®) cluster300in accordance with an embodiment of the invention, which can be deployed as the supervisor container-based cluster102or one of the guest container-based clusters104of the computing system100, is illustrated. As shown inFIG. 3, the K8S cluster300includes a master node302and a number of worker nodes304. Each worker node304, or worker compute machine, includes a kublet306that runs one or more pods308in the worker node. Each pod308may include one or more containers310. The worker nodes can be used to execute various applications and software processes using the containers. The master node302includes an API server312and a controller manager314. The API server312operates as a gateway to the K8S cluster300, which can be accessed by users, automation and components in the K8S cluster. The controller manager314operates to monitor the API server312, and execute various operations in response to certain events. If the K8S cluster300is a guest container-based cluster, the controller manager314may execute certain operations when services are requested for the cluster, as described below.

The master node302may include other components, such as a kube-proxy316, a scheduler318, a container runtime interface (CRI)320, a container network interface (CNI)322and a container storage interface (CSI)324, which are well-known components of a K8S cluster. The kube-proxy316is a Kubernetes network proxy that proxies User Datagram Protocol (UDP), Transmission Control Protocol (TCP) and Stream Control Transmission Protocol (SCTP), provides load balancing and can be used to reach services. There may be kube-proxy in each of the worker nodes. The scheduler318operates to make sure that the pods are matched to the nodes in the K8S cluster300so that the kublets306can run them. The CRI320, CNI322and CSI324provide compute, network and storage resources for the containers310in the K8S cluster.

If the K8S cluster300is a supervisor container-based cluster, the K8S cluster has additional components, which include at least a guest cluster controller326, a workload control plane (WCP) controller328, a service controller330and a container plugin332, as shown inFIG. 3. These components enable the K8S cluster300to execute various operations as the supervisor container-based cluster to create and manage one or more guest container-based clusters104, and to enable various capabilities for the guest container-based clusters.

The guest cluster controller326operates to manage the lifecycle of guest container-based clusters created by the K8S cluster300as the supervisor container-based cluster. Thus, the guest cluster controller can create, manage and delete guest container-based clusters. In an embodiment, the guest cluster controller may be a Tanzu Kubernetes cluster controller provided by a vSphere® with Kubernetes product, which is available from VMware, Inc.

The workload control plane controller328operates to enable various capabilities for the guest container-based clusters created through the K8S cluster300as the supervisor container-based cluster. In particular, the workload control plane controller can create translated service software objects, for example, virtual machine service software objects, to enable capabilities for the guest container-based clusters. A virtual machine service software object is a K8s Custom Resource created in the supervisor cluster K8s API to represent an LB Service that is backed by a pool of VM endpoints. In the computing system100, one VM service is created for each k8s “Service of loadbalancer type” of the guest cluster, with the endpoints to be all VMs (i.e., VM IP addresses) in the guest cluster with node port (NodePort) number allocated for the guest cluster service. In an embodiment, the guest cluster controller may be a Cluster API for Workload Control Plane (CAPW) controller provided by a vSphere with Kubernetes product, which is available from VMware, Inc.

The service controller330operates to monitor the service-related software objects, for example, virtual machine service software objects, created by the workload control plane controller328and uses the specifications defined in the virtual machine service software objects to create and configure various VMs. In particular, the service controller may create various software objects for services that correspond to the virtual machine service software objects created by the workload control plane controller. As an example, with respect to a virtual machine service software objects for a load balancer service, the service controller may create (1) a load balancer object, (2) a load balancer type service object and (3) one or more endpoints, which includes parameters and definitions of resources that are needed to created. In an embodiment, the service controller may be a virtual machine operator (VMOP) provided by a vSphere with Kubernetes product, which is available from VMware, Inc.

The container plugin332is a controller that watches for certain software objects created in the K8S cluster300functioning as the supervisor container-based cluster and orchestrates the creation of corresponding resources by an external resource provider, which may be the SDN manager118in the computing system100. The corresponding resources are dependent on the resource provider. Thus, the container plugin is designed or programmed to send provider-specific instructions to a particular resource provider that can create the needed resources. In order to send the correct provider-specific instructions for the software objects created in the supervisor container-based cluster, the container plugin maps each of the software objects to one or more provider-specific resources that need to be created by the resource provider. Once the provider-specific resources are determined, appropriate instructions can be sent to the resource provider so that the provider-specific resources can be created by resource provider. As an example, with respect to software objects related to a load balancer service, the container plugin will send provider-specific instructions to create (1) load balancer (e.g., NSX-T load balancer) and T1 gateway (e.g., NSX-T T1 gateway) that are mapped to a load balancer object, (2) a virtual server (e.g., NSX-T virtual server) and a server pool (e.g., NSX-T server pool) that are mapped to a load balancer type service object and (3) static pool members that are mapped one or more endpoints. In an embodiment, the container plugin may be a NSX container plugin (NCP) provided by a vSphere with Kubernetes product, which is available from VMware, Inc. Thus, in this embodiment, the container plugin is designed or programmed to interface with an NSX manager. However, in other embodiments, the container plugin is designed or programmed to interface with another resource provider, such as Avi Network or HAProxy.

When a service, such as a load balancer service, that requires the creation of one or more external resources, such as an external load balancer and other network resources, is requested for a guest container-based cluster, there needs to be a communication mechanism between the guest container-based cluster and the resource provider, such as the SDN manager118of the virtual infrastructure108. However, if the communication mechanism provides a direct access from the guest container-based cluster to the resource provider, this introduces a security vulnerability to the computing system100. This is due to the fact that the network of the guest container-based clusters is less secure than the management network of the virtual infrastructure to which the virtualization manager and the SDN manager are connected. Thus, rather than a direct access to the virtual infrastructure, the computing system uses an indirect access mechanism to the virtual infrastructure through the supervisor container-based cluster when resources are required in response to requested services for the guest container-based clusters, as explained below.

Turning now toFIG. 4, components of the computing system100are shown to illustrate the process of providing service, which needs one or more external resources to be created by a resource provider, e.g., the SDN manager118, for a guest container-based cluster without direct access to the resource provider by the guest container-based cluster. InFIG. 4, only one of the guest container-based clusters104, the supervisor container-based cluster102and the SDN manager118of the computing system100are shown. In order to not obscure the inventive features of the computing system, the supervisor container-based cluster102is illustrated with only the API server312, the service controller330and the container plugin332, while the guest container-based cluster104is illustrated with only the API server312and the controller manager314.

When a user, such as a developer, wants to create a service that requires one or more external resources for the guest container-based cluster104, a software object for the service is created in the guest container-based cluster via the API server312of the guest container-based cluster in response to a request for the service made by the user. As an example, if the user wants to create a load balancer type service for the guest container-based cluster, a load balancer type service software object is created in the guest container-based cluster via the API server of the guest container-based cluster.

The creation of the software object for the service is detected by the controller manager314of the guest container-based cluster104, which monitors the API server312of the guest container-based cluster for certain events in the guest container-based cluster, such as the creation of software objects in the guest container-based cluster. When the creation of the software object for the service is detected, a translated service object is created in the supervisor container-based cluster102by the controller manager314of the guest container-based cluster104via the API server312of the supervisor container-based cluster102. In a particular embodiment, the translated service object is a virtual machine service object for the requested service that corresponds to the software object for the service created in the guest container-based cluster.

The creation of the translated service object in the supervisor container-based cluster102is then detected by the service controller330in the supervisor container-based cluster102, which monitors the API server312of the supervisor container-based cluster102for the certain events in the supervisor container-based cluster, such as the creation of particular service software objects in the supervisor container-based cluster. When the creation of the translated service software object is detected, one or more service-related software objects that corresponds to the translated service software object are created in the supervisor container-based cluster by the service controller330. In an embodiment, the service-related software objects that are created include (1) a load balancer object, (2) a load balancer type service object and (3) one or more endpoints.

The creation of the service-related software objects in the supervisor container-based cluster102is then detected by the container plugin332, which monitors the API server312of the supervisor container-based cluster for the certain events in the supervisor container-based cluster, such as the creation of service-related software objects in the supervisor container-based cluster. When the creation of the service-related software objects is detected, the service-related software objects are mapped to provider-specific resources that are needed for the requested service by the container plugin. In an embodiment, (1) the load balancer object is mapped to a load balancer (e.g., NSX-T load balancer) and a T1 gateway (e.g., NSX-T T1 gateway), (2) the load balancer type service object is mapped to a virtual server (e.g., NSX-T virtual server) and a server pool (e.g., NSX-T server pool), and (3) the endpoints are mapped to static pool members. After the resources have been determined by the container plugin, provider-specific instructions are sent to the SDN manager118, which is the resource provider in the illustrated embodiment, to create the provider-specific resources that are needed for the requested service. All the necessary configurations for the guest container-based cluster to enable the request service using the resources created by the resource provider are taken care by the supervisor container-based cluster.

In this fashion, there is no direct access to the SDN manager118by the guest container-based cluster104to create the resources needed for the request service for the guest container-based cluster. Rather, the supervisor container-based cluster102is used to translate the requested service for the container-based cluster and access the SDN manager118. Thus, the supervisor container-based cluster functions as an intermediary mechanism between the guest container-based cluster and the SDN manager when services are quested for the guest container-based cluster, which significantly reduces security vulnerability of the management network. This process of providing service for guest container-based clusters using the supervisor container-based cluster is further explained below by describing the operation of the computing system100for such a process.

An operation of the computing system100for providing a service for a guest container-based cluster104of the computing system in accordance with an embodiment of the invention is described with reference to a process flow diagram ofFIG. 5. The service for the guest container-based cluster is any service for a container-based cluster that requires one or more external resources to be created by a resource provider, such as the SDN manager118of the virtualization infrastructure108in the computing system. In this description, the service being requested is a load balancer and the guest container-based cluster is a Kubernetes cluster. However, the service being requested can be any service for a container-based cluster and the guest container-based cluster can be any type of a container-based cluster, such as a Docker cluster.

The operation begins at step502, where a request for a load balancer service is received at the guest container-based cluster104. In an embodiment, the request is received by the API server312in the guest container-based cluster from a user interface, where a user can make the service request. In one implementation, the request may be made by setting the service type to “load balancer” in a service configuration file for the guest container-based cluster.

Next, at step504, in response to the received service request, a service software object corresponding to the requested service is created by the API server312in the guest container-based cluster104. In the embodiment in which the guest container-based cluster104is a Kubernetes cluster, the service software object created is a REST object that targets a set of pods in the guest container-based cluster.

Next, at step506, the creation of the service software object in the guest container-based cluster104is detected by the controller manager314in the guest container-based cluster. In an embodiment, the event of creating the service software object is notified to the controller manager314by the API server312in the guest container-based cluster.

Next, at step508, in response to the detection of the service software object being created in the guest container-based cluster104, a translated service software object corresponding to the service software object created in the guest container-based cluster is created in the supervisor container-based cluster102by the controller manager314in the guest container-based cluster104via the API server312in the supervisor container-based cluster102. The translated service software object created in the supervisor container-based cluster is a representation of the service software object created in the guest container-based cluster. Thus, it can be viewed that the service software object created in the guest container-based cluster is translated into a different service software object created in the supervisor container-based cluster. In an embodiment, the translated service software object created in the supervisor container-based cluster is a virtual machine service object.

Next, at step510, the creation of the translated service software object in the supervisor container-based cluster102is detected by the service controller330in the supervisor container-based cluster. In an embodiment, the event of creating the translated service software object is notified to the service controller330by the API server312in the supervisor container-based cluster. In this embodiment, the service controller may be subscribed with the API server312in the supervisor container-based cluster to receive certain type of events, such as creation of virtual machine services.

Next, at step512, in response to the detection of the translated service software object being created in the supervisor container-based cluster102, at least one service-related software object is created in the supervisor container-based cluster that corresponds to the translated service software object created in the supervisor container-based cluster by the service controller330in the supervisor container-based cluster. In an embodiment for a load balancer virtual machine service, three service-related software objects are created by the service controller330. The three provider-specific software objects are (1) a load balancer software object, (2) a load balancer type service and (3) endpoints.

Next, at step514, the creation of the service-related software objects in the supervisor container-based cluster102is detected by the container plugin332in the supervisor container-based cluster. In an embodiment, the API server312in the supervisor container-based cluster is continuously monitored by the container plugin to detect certain type of events, such as creation events of service-related software objects in the supervisor container-based cluster.

Next, at step516, in response to the detection of the service-related software objects being created in the supervisor container-based cluster102, each service-related software object is mapped to one or more provider-specific resources that need to be created by the resource provider, e.g., the SDN manager118, for that service-related software object. In an embodiment, the following mapping is used by the container plugin332to map the service-related software objects to corresponding provider-specific resources:(1) a load balancer software object↔a load balancer (e.g., an NSX-T load balancer) and a T1 gateway (e.g., an NSX-T T1 gateway)(2) a load balancer type service↔a virtual server (e.g., an NSX-T virtual server) and a server pool (e.g., an NSX-T server pool)(3) endpoints↔static pool members

Next, at step518, provider-specific instructions are sent from the container plugin332in the supervisor container-based cluster102to the resource provider, e.g., to create the mapped resources. In an embodiment, the instructions are sent to the SDN manager of the virtual infrastructure.

Next, at step520, in response to the instructions, the resources are created by the resource provider, e.g., the SDN manager118, to support the requested service made for the guest container-based cluster. In the embodiment in which the three types of provider-specific software objects (a load balancer software object, a load balancer type service and endpoints) are created in the supervisor container-based cluster, the following resources are created by the resource provider, e.g., the SDN manager118: a load balancer (e.g., an NSX-T load balancer), a T1 gateway (e.g., an NSX-T T1 gateway), a virtual server (e.g., an NSX-T virtual server), a server pool (e.g., an NSX-T server pool) and static pool members.

A computer-implemented method for providing a service for a guest container-based cluster in a computing system in accordance with an embodiment of the invention is described with reference to a flow diagram ofFIG. 6. At block602, a request for the service is received at the guest container-based cluster. At block604, in response to the request, a service software object corresponding to the service is created in the guest container-based cluster. At block606, in response to the service software object being created in the guest container-based cluster, a translated service software object is created in a supervisor container-based cluster of the computing system. At block608, in response to the translated service software object being created in the supervisor container-based cluster, at least one service-related software object is created in the supervisor container-based cluster that corresponds to the translated service software object. At block610, in response to the at least one service-related software object being created in the supervisor container-based cluster, provider-specific instructions are sent to a resource provider from supervisor container-based cluster to create at least one resource associated with the at least one service-related software object to support the requested service for the guest container-based cluster.