Patent Description:
Different types of virtualization have gained traction with the widespread adaptation of cloud computing. Today, there are multiple technologies available for virtualization, but two significantly different approaches are virtual machines and containers.

With a traditional virtual machine, a whole computer environment is virtualized, including the operating system kernel and applications. Virtual machines are executed and managed by hypervisors. Multiple mature hypervisors exist today including Quick Emulator (QEMU)-Kernel-based Virtual Machine (KVM), VMWare, Hyper-V and Xen. For virtual machines, the operating system kernel can be different compared to the kernel of the physical machine (referred also as host machine). Modern Central Processing Units (CPUs) have specialized instruction sets and execution rings to optimize the performance of virtual machines. In general, virtual machines achieve a high level of separation from the host machine and other virtual machines executed on the same host machine. The virtual machine world is not standardized in general, the different hypervisors support multiple image formats, and in general it is possible to convert one image format to another image format, making it possible to run the same image under different hypervisors. There are also solutions for building multi-hypervisor environments (e.g. an OpenStack cloud with multiple hypervisors).

With a traditional container, the virtualization is achieved in the kernel by using namespaces for logical separation (e.g. Process identifications (PIDs) of other processes, networking of other processes) and cgroups for physical resource (e.g. CPU share) separation. Containers in general are believed to be less secure than virtual machines because e.g. kernel bugs can lead to privilege escalations.

The container world has recently started to move towards standardization led by the Open Container Initiative (OCI). As of today two specifications exist: the runtime specification and the bundle specification. These OCI specifications make it possible for multiple runtimes to be developed that can execute the same container images.

Since the publication of the standard, multiple compatible runtimes have been open sourced (e.g. runC, KataContainers, crun, gVisor). In practice, the OCI specifications enable simply changing the container runtime engine under Docker. Due to the OCI specifications it also became possible to blend the virtual machine and container worlds. An example is Kata Containers which, despite its name, launches lightweight virtual machines when used as a container runtime under Docker.

For both virtual machines and containers, the different virtualization engines (hypervisors/container runtimes) may provide different performance and sometimes feature characteristics. For example, if we compare container runtimes some of them may provide faster container startup (e.g. gVisor), while others may bring significant security improvements (e.g. KataContainers). Regarding virtual machines, it has been reported recently that the different hypervisors have different energy efficiencies whilst running the same virtual machines.

While existing solutions support multiple virtualization engines (either virtual machine or container), the selection between such virtualisation engines is only available manually. For example, for Docker the -runtime flag can be used when calling the docker run command. Pouch containers from Alibaba Cloud also openly supports multiple container runtimes with manual selection. For OpenStack the hypervisor_type tag can be used to determine which hypervisor to use in a multi-hypervisor environment.

Such manual selection can result in: suboptimal performance for applications, for example for some applications the startup time may be the most important performance characteristic; suboptimal resource utilization for the provider, for example using a runtime with lower memory footprint may be more beneficial; insufficient security level, for example, sacrificing performance to reach high security level for some applications in a container environment may be desirable.

<CIT> discloses a dynamic cloud stack tuning system.

<CIT> discloses a non-transitory computer-readable storage device which stores instructions that, when executed on a computing system, cause the computing system to receive a request for creating a new software container and determine that characteristics of the new software container match a co-tenant policy of an existing software container on a server.

Aspects of the invention are set out in the independent claims appended hereto.

For a better understanding of the embodiments of the present disclosure, and to show how it may be put into effect, reference will now be made, by way of example only, to the accompanying drawings, in which:.

Embodiments described herein provide dynamic and automatic selection of a virtualization engine technology for cloud providers. A combination of heuristics and configurations may be used to select the appropriate virtualization engine to execute a particular application image. A user may also influence the selection decision with hints or recommendations provided together with the application image deployment request. This dynamic selection makes it possible to meet requirements, for example relating to performance, security and cost for each application image individually in the same cloud environment. Subsequent deployments of the same application image may also be executed using different virtualization engines in order to find the best option. Automatic defense solutions may also be enabled against malicious user applications.

<FIG> illustrates a block diagram of a logical architecture of a system according to some embodiments.

The system <NUM> comprises an application deployment request block <NUM>. The application deployment request comprises an identification of an application image being request for deployment, or the application image <NUM> itself. The application deployment request <NUM> may also comprise an indication of a user requirement <NUM> associated with the virtualisation engine to be selected to execute the application image. An application image may comprise the information necessary to execute the application in a container. An application image may also be defined as a binary comprising requirements for running a single container. An application image may also comprise metadata describing the applications needs and capabilities. A virtual machine (VM) may comprise a software implementation of a machine (i.e. a computer) that executes programs like a physical machine. An application image of a virtual machine may comprise a copy of the application, which may contain an operating system (OS), data files, and applications (similarly to a personal computer).

The system <NUM> further comprises a selection engine <NUM>. The selection engine <NUM> may be configured to determine, based on at least the identification of the application image, a virtualisation engine to execute the application image.

The selection engine <NUM> receives the application deployment request <NUM>. Upon receiving the application deployment request <NUM>, the selection engine <NUM>, evaluates one or more heuristics to decide which virtualization engine (i.e. engine A, engine B etc.) <NUM>, <NUM>, should be used for the given application deployment request <NUM>. (For clarity only two virtualisation engines are illustrated, although it will be appreciated that there may be any number of possible virtualisation engines). For these heuristics, the selection engine <NUM> may use available virtualization engine characteristics, any user requirements in the application deployment request <NUM>, previous execution statistics, and/or provide requirements. After selecting the appropriate virtualization engine, an execution engine <NUM> may be instructed to deploy the application image using the selected virtualization engine.

The execution engine <NUM> may receive the application image from the selection engine <NUM>, or from a separate engine.

<FIG> illustrates a system level architecture according to some embodiments. The User <NUM> represents the entity who submits the application deployment request <NUM> as illustrated in <FIG>, to the system, which in this example comprises a cloud implemented system <NUM>.

The cloud system <NUM> comprises a control node <NUM> that comprises selection engine <NUM> as described with reference to <FIG>. The control node <NUM> may also comprise an application image store <NUM>.

For example, the application identification forming part of the application deployment request may comprise a reference to an application image in the application image store <NUM>.

After the selection engine <NUM> selects the virtualization engine, <NUM>, <NUM>, <NUM> or <NUM>, to use to execute the application image, the application image is deployed on one of the compute nodes <NUM> or <NUM> of the cloud system <NUM>. As the figure implies, it is possible that not all compute nodes support all Virtualization engines. In this example, virtualisation engine A is supported by both compute node <NUM> and <NUM>, however, virtualisation engine B is only supported by compute node <NUM>, and virtualisation engine C is only supported by compute node <NUM>. The control node <NUM> may therefore keep track of the availability of the different virtualization engines on the different compute nodes.

<FIG> illustrates a method performed by a selection engine, for example selection engine <NUM>, for selecting a first virtualisation engine to execute an application deployment request.

In step <NUM>, the selection engine receives the application deployment request comprising an identification of an application image. As noted in <FIG>, the application deployment request may also comprise one or more user requirements.

User requirements may comprise, for example, one or more of the following:.

In step <NUM>, the selection engine may optionally receive one or more provider requirements. For example, a given application image may require different amount of resources (e.g. memory, CPU or power in general) when executed with different virtualization engines. From the provider's perspective it may be beneficial to use the runtime with the lowest resource requirements (making it possible to run more applications on the same hardware). It may be also possible that application A requires less resources with virtualization engine A, while application B operates more cost effectively with virtualization engine B. The selection engine may record such information about previous executions of each application and make decisions accordingly (e.g. by trying virtualisation engines A, B and C for a given application image, then selecting B as the best option).

In step <NUM>, the selection engine may optionally retrieve the application image associated with the identification of the application image. For example, the application image may be retrieved from the application image store as illustrated in <FIG>. This step may be performed in examples where the selection engine is responsible for passing the application image itself to the execution engine for execution by the selected virtualisation engine. However, it will be appreciated that in some embodiments, different nodes in the system may be responsible for passing the application image to the appropriate engine, and the selection engine may simply select which virtualisation engine is to be used, and may provide the appropriate notification of the selection to other nodes in the system where required.

In step <NUM>, the selection engine may optionally obtain previous execution statistics relating to the execution of the application image by each of a plurality of virtualisation engines that are available to execute the application image. These previous execution statistics may be utilised to determine which of the plurality of virtualisation engines should be selected to execute the application image. The previous execution statistics may be stored by the selection engine <NUM>.

In step <NUM>, the selection engine may optionally obtain information relating to each of the plurality of virtualisation engines. Again this information may be utilised to determine which of the plurality of virtualisation engines should be selected to execute the application image. The information relating to each of the plurality of virtualisation engines may again be stored by the selection engine.

In step <NUM>, the selection engine selects the first virtualisation engine from the plurality of virtualisation engines based on a plurality of respective values of at least one characteristic associated with execution of the application image by each of the plurality of virtualisation engines. For example, the at least one characteristic associated with execution of the application image be each of the plurality of virtualisation engines may comprise one or more of: a security level, a start-up time for the application image; a memory accessing time for the application image; a CPU efficiency; a presence or lack of a feature in the virtualisation engine; and a number of previous executions of the application image by the virtualisation engine.

The respective values of the at least one characteristic may therefore be as follows:.

It will be appreciated that the respective values of the at least one characteristic may have been obtained by the selection engine in steps <NUM> ad <NUM>. For example, the information relating to the start-up time of a particular application image on a particular virtualisation engine may form part of the previous execution statistics obtained in step <NUM>. Similarly, the presence or lack of a GPU in a particular virtualisation engine may form part of the information relating to each of the plurality of virtualisation engines received in step <NUM>.

Furthermore, the respective values of the characteristics may actually fall on a scale between the two extremes. For example, application image from users may be certified, and so have a "medium" security, or images from well-known users can be treated as trusted and so have a "high" security. Application images from users that are not certified may then have a "low" security.

In some embodiments, to perform step <NUM>, the selection engine may prioritise the plurality of respective values of a first characteristic over the plurality of respective values of a second characteristic. For example, the selection engine may be configured to prioritise security over other characteristics when selecting the first virtualisation engine. In this example therefore, when selecting the first virtualisation engine, the selection engine may priorities the value associated with the security characteristic for virtualisation engines over the values associated with other characteristics.

In step <NUM>, the selection engine may select an appropriate compute node for execution of the application image on the first virtualisation engine. This step may be performed where scenarios as in <FIG> exist, and not all virtualisation engines are supported by all compute nodes.

In step <NUM>, the selection engine initiates execution of the application image by the first virtualisation engine. In some examples, step <NUM> comprises the selection engine transmitting the application image to the compute node for execution by the first virtualisation engine. In other embodiments, the selection engine initiates the transfer of the application image by a different network node.

<FIG> illustrates an example of step <NUM> in <FIG> in more detail.

In step <NUM>, the selection engine determines a first requirement and a second requirement. The first requirement may then be treated with higher priority than the second requirement. It will be appreciated that there may be any number of requirements which may be prioritised differently. The first requirement and the second requirements may be determined based on the received application identification, any received user requirements in the application deployment request in step <NUM>, and any received provider requirements in step <NUM>.

For example, in a generic cloud environment, users may provide any application image. Such custom application images may contain malicious applications that can target unpublished security issues (e.g. for privilege escalation). To guarantee high level of security, such images can be treated as untrusted, and executed with a virtualization engine that may provide lower performance, but higher isolation guarantees.

On the contrary, cloud providers may offer pre-built applications (e.g. specific data stores) to customers. As these images are created by the provider and most likely only accessible via well-defined APIs, they can be treated as trusted images. These trusted images can be executed with virtualization engines that offer less isolation guarantees but higher performance for example.

The selection engine may therefore be configured to, based on the identification of the application image, determine whether the application image is a user provided application image or a pre-built application image. Responsive to the application image comprising a user provided application image, the selection engine may be configured to set the first requirement as a security requirement. For example, the first requirement may stipulate that the value for the security characteristic associated with the first virtualisation engine must be "high".

In contrast, responsive to the application image comprising a pre-built application image, the selection engine may be configured to look to a different characteristic to determine the first requirements, as security is not considered as high a priority. For example, the selection engine may have received one or both of a user requirement and a provider requirement. The selection engine may be configured to set one of the user requirement or the provider requirement as the first requirement. Whether the selection engine is configured to select the user requirement or provider requirement first may be preconfigured in the selection engine, or may be determined based on the received requirements.

Responsive to the first requirement being a security requirement, the selection engine may be configured to set one of the user requirement or the provider requirement as the second requirement. The selection engine may then be configured to set the other one of the user requirement or the provider requirement as a third requirement. In some examples, the security requirement may be handled with the highest priority, followed by the user requirement and the provider requirement.

It will be appreciated that the order and number of different requirements may be configured differently in different implementations of embodiments described herein. In particular, there may be more than one user requirement that is set as a numbered requirement. In the example illustrated in <FIG>, two requirements are determined. The first and second requirement may be determined as described above.

In step <NUM>, the selection engine determines if there is available information for a respective value of the first and second characteristics associated with execution of the application image by a third virtualisation engine.

The first characteristic is based on the first requirement, and the second characteristic is based on the second requirement. In an example, the first requirement stipulates that the selected virtualisation engine should maximize the number of served requests per second, and so the first characteristic comprises the number of severed requests per second.

Similarly, in an example, the second requirement stipulates that the selected virtualisation engine should minimise the memory usage per application image, so the second characteristic comprises memory usage per application image. In this example, the second requirement may comprise a provider requirement as the goal of the provider may be to minimize the memory usage per application to enable running more application instances on the same amount of hardware.

Responsive to a lack of available information for a respective value of the first characteristic or the second characteristic associated with execution of the application image by any of the virtualisation engines (for example a third virtualisation engine), the method passes to step <NUM>.

In step <NUM>, the selection engine initiates execution of the application image on each virtualisation engine for which there is a lack of available information (for example, the third virtualisation engine) to obtain the respective value of the first and/or second characteristic associated with execution of the application image by the third virtualisation engine. In other words, the selection engine performs test runs of the application image on any virtualisation engines for which information is missing, in order to obtain the missing information for the first or second characteristics. In some examples, the execution engine may report the respective value of the first and/or second characteristic back to the selection engine. In other examples, the application reports the respective value of the first and/or second characteristic back to the selection engine based on internal measurements. Following collection of the missing information, the method passes to step <NUM>.

Responsive to there being available information for both the first and second characteristic for the plurality of virtualisation engines, the method passes to step <NUM>.

In step <NUM>, the selection engine selects a subset of the plurality of virtualisation engines based on the plurality of respective values of the first characteristic. For example, the selection engine may allocate each of the plurality of virtualisation engines whose associated value of the first characteristic for execution of the application image meets the first requirement to the subset of the plurality of virtualisation engines. In the example, where the first requirement stipulates that the first virtualisation engine should maximize the number of served requests per second therefore, the selection engine may put any virtualisation engine whose number of served requests per second when running the application image is a maximum value into the subset.

Responsive to the subset comprising more than one virtualisation engine, the method passes to step <NUM>.

In step <NUM>, the selection engine selects the first virtualisation engine from the subset based on the plurality of respective values of the second characteristic. The selection engine is configured to select a virtualisation engine from the subset whose associated value of the second characteristic for execution of the application image meets the second requirement as the first virtualisation engine. In other words, in the example where the second requirement stipulates that the selected virtualisation engine should minimise the memory usage per application image, the selection engine may select the first virtualisation engine from the subset as the virtualisation engine that has the minimum memory usage per application image.

Returning to step <NUM>, responsive to the subset comprising only one virtualisation engine, the method passes to step <NUM>, in which the selection engine, selects the one virtualisation engine in the subset as the first virtualisation engine.

It will be appreciated that, in general, the selection engine may be implemented such that the selection of the first virtualization engine is made using a weighted result from the different input sources (i.e. the user requirements, provider requirements etc,). For example, multiple heuristics may be evaluated, each resulting in a preferred (ordered) list of virtualization engines. The previous execution statistics can be used as input for some of the heuristics (e.g. performance measurements). Finally, the user requirements can also result in a preference order of the available virtualization engines that may be weighted in during the final selection.

For example, in some embodiments, the selection engine may be configured to: apply a first weighting value to the plurality of respective values of the first characteristic to generate a plurality of weighted first values; apply a second weighting value to the plurality of respective values of the second characteristic to generate a plurality of weighted second values; determine a plurality of combined values based on the plurality of weighted first values and the plurality of weighted second values, wherein each combined value is associated with execution of the application image by one of the plurality of virtualisation engines; and select the first virtualisation engine based on the plurality of combined values. To perform these embodiments, the qualitative values of some of the proposed characteristics (for example, "high", "medium" and "low" for security) may have to be converted into quantitative equivalents for the purposes of generating the weighted sum.

In some embodiments, after a first virtualisation has been selected to execute an application image, some received input may cause the selection engine to re-select a new virtualisation engine to execute the application.

For example, in some embodiments, responsive to the respective value of the first characteristic for the first virtualisation engine changing such that the respective value no longer meets the first requirement, the selection engine may be configured to remove the first virtualisation engine from the subset of the plurality of virtualisation engines.

The selection engine may then reselect a second virtualisation engine from the subset of the plurality of virtualisation engines; and may initiate execution of the application image by the second virtualisation engine.

<FIG> illustrates an example of a method in the selection engine performed in response to a security issue associated with the first virtualization engine.

System security issues may be published or shared with cloud providers first before publication. Such issues may comprise for example privilege escalation possibilities in the Linux Kernel. A large amount of such issues may impact only a subset of the available virtualization engines in the environment of the cloud provider. In this scenario a restrictive heuristic can be implemented that forces most or all of the applications to be executed on runtimes that are not impacted by the security issue, until it becomes fixed and the Compute Nodes get patched.

In step <NUM>, the selection engine receives an indication of a security issue associated with the first virtualisation engine. For example, security issues discovered about virtualisation engines may be published online. These issues may be automatically retrieved or received by the selection engine.

Responsive to receiving the indication of a security issue associated with the first virtualisation engine, in step <NUM> the selection engine, selects a second virtualisation engine from the plurality of virtualisation engines based on the plurality of respective values of the at least one characteristic associated with execution of the application image by each of the plurality of virtualisation engines. In other words, as the selection engine has been notified about a security issue with the first virtualisation engine.

For example, as described above, the selection engine may remove the first virtualisation engine from the subset determined as described with reference to <FIG>, and may select a second virtualisation engine from the remaining virtualisation engines in the subset.

In step <NUM>, the selection engine initiates execution of the application image by the second virtualisation engine. In particular, if multiple applications are affected by the security issue, for example if multiple application images are being run by the affected virtualisation engine, then the selection engine may gradually re-deploy the application images.

<FIG> illustrates a method, in a selection engine, of performing measurements, for example as described in step <NUM> of <FIG>.

In step <NUM>, the selection engine schedules execution of the given application image virtualisation engine combination.

In step <NUM>, the selection engine initiates execution of the application image by the virtualisation engine and directs some user load towards the application image.

In step <NUM>, the selection engine allows the application image to run for a duration of t seconds. During this time, the application or the execution image, may report the respective values of the characteristics to the selection engine.

In step <NUM>, the selection engine records the respective values of the characteristics received in step <NUM> for future deployments and also if a virtualization engine is found to perform better than the previously declared best option, the gradual re-deployment of the application image may be triggered as a final step of this measurement process.

Embodiments described herein may be realized used the OCI runtimes as described above. Essentially, according to the specifications the same container bundle can be executed with different container runtimes. Using the described embodiments, the selection between the runtimes for each bundle deployment can be automatized. The different runtimes provide different characteristics:
For example, runC may be the default container runtime that is the most mature, reliable and provides stable performance. RunC uses namespaces resulting in less separation compared to other runtime options. RunC requires a larger amount of memory per container compared to other options, thus having larger cost for the provider.

Crun is an experimental re-implementation of runC providing a x3 faster startup time. Crun is not mature, thus may contain bugs, but if startup time is a key requirement, it can be a viable selection. Security wise, it is the same as runC.

KataContainers uses lightweight virtual machines. With this significantly different approach, this technology is the most secure OCI compatible runtime. KataContainers starts x10 slower than runC containers though and provides slightly lower compute performance due to the nature of the used virtualization technique.

gVisor provides a different approach for containerization by using a user space kernel for system call interception. While gVisor containers start x3 faster than runC, they lag behind in some performance benchmarks due to the overhead system call interception. The same system call interception approach though gives a significant security advantage.

Cloud solutions like OpenStack may support multi-hypervisor environments. Embodiments described herein may also be used with virtual machines as well. In an example using hypervisors, different solutions may support different, or only partially overlapping, sets of image formats. As a result, a user supplied application image may need to be converted to a different image format before it can be executed with different hypervisors. In this example, the hypervisors provide different performance characteristics, while security-wise the solutions may be similar. Table <NUM> illustrates an example performance comparison for <NUM> well-known hypervisors: XenServer, VMWare vSphere, Hyper-V, Qemu+KVM In particular, table <NUM> summarizes the determined ranking for the <NUM> main characteristics.

<FIG> illustrates a selection engine <NUM> comprising processing circuitry (or logic) <NUM>. The processing circuitry <NUM> controls the operation of the selection engine <NUM> and may implement the method described herein in relation to a selection engine <NUM>. The processing circuitry <NUM> may comprise one or more processors, processing units, multi-core processors or modules that are configured or programmed to control the selection engine <NUM> in the manner described herein. In particular implementations, the processing circuitry <NUM> may comprise a plurality of software and/or hardware modules that may each be configured to perform, or are for performing, individual or multiple steps of the method described herein in relation to the selection engine <NUM>.

Briefly, the processing circuitry <NUM> of the selection engine <NUM> may be configured to: receive the application deployment request comprising an identification of an application image; select the first virtualisation engine from a plurality of virtualisation engines based on a plurality of respective values of at least one characteristic associated with execution of the application image by each of the plurality of virtualisation engines; initiate execution of the application image by the first virtualisation engine.

In some embodiments, the selection engine <NUM> may optionally comprise a communications interface <NUM>. The communications interface <NUM> of the selection engine <NUM> may be for use in communicating with other nodes, such as other virtual nodes. For example, the communications interface <NUM> of the selection engine <NUM> may be configured to transmit to and/or receive from other nodes requests, resources, information, data, signals, or similar. The processing circuitry <NUM> of selection engine <NUM> may be configured to control the communications interface <NUM> of the selection engine <NUM> to transmit to and/or receive from other nodes requests, resources, information, data, signals, or similar.

Optionally, the selection engine <NUM> may comprise a memory <NUM>. In some embodiments, the memory <NUM> of the selection engine <NUM> may be configured to store program code that can be executed by the processing circuitry <NUM> of the selection engine <NUM> to perform the method described herein in relation to the selection engine <NUM>. Alternatively or in addition, the memory <NUM> of the selection engine <NUM>, can be configured to store any requests, resources, information, data, signals, or similar that are described herein. The processing circuitry <NUM> of the selection engine <NUM> may be configured to control the memory <NUM> of the selection engine <NUM> to store any requests, resources, information, data, signals, or similar that are described herein.

Claim 1:
A method performed by a selection engine for selecting a first virtualisation engine to execute an application deployment request (<NUM>), the method comprising:
receiving (<NUM>) the application deployment request (<NUM>) comprising an identification of an application image (<NUM>);
selecting (<NUM>) the first virtualisation engine from a plurality of virtualisation engines based on a plurality of respective values of at least one characteristic associated with execution of the application image by each of the plurality of virtualisation engines; and
initiating (<NUM>) execution of the application image by the first virtualisation engine; wherein the at least one characteristic comprises a first characteristic and a second characteristic, characterized in that selecting the first virtualisation engine comprises:
allocating (<NUM>) each of the plurality of virtualisation engines whose associated value of the first characteristic for execution of the application image meets a first requirement to a subset of the plurality of virtualisation engines; and
responsive to the subset comprising more than one virtualisation engine, selecting (<NUM>) a virtualisation engine from the subset whose associated value of the second characteristic for execution of the application image meets a second requirement as the first virtualisation engine.