Patent Description:
<CIT> describes a computer-implemented method for performing live migrations of software containers that may include (i) identifying a request to migrate a software container from a source computing system to a target computing system while a process executes within the software container, (ii) creating a checkpoint of the process in execution (iii) transferring the checkpoint to the target computing system, (iv) updating the checkpoint recurrently by recurrently creating an incremental checkpoint of the process and merging the incremental checkpoint into the checkpoint, (v) predicting, before updating the checkpoint with an iteration of the incremental checkpoint and based on a size of the iteration of the incremental checkpoint, that finalizing a migration of the software container to the target computing system would meet a predetermined time objective, and (vi) finalizing the migration of the software container to the target computing system.

<CIT> describes that it may be determined that a cloud desktop should be migrated from a current region. A destination region to which the cloud desktop will be migrated can be identified. A data volume of the cloud desktop may be copied from the current region to the destination region. The data volume at the current region and the data volume at the destination region may be maintained in sync during the copying. Upon completion of the copying, a current user session associated with the cloud desktop at the current region may be frozen, a current memory and processor state of the current user session may be copied to the destination region, and a second cloud desktop instance at the destination region may be started using the copied data volume and current memory and processor state. The current user session may be connected to the second cloud desktop instance.

Though both virtual machines and containers can be used as virtualization techniques to accommodate compute, communications, or other types of computing services, these virtualization techniques can have different characteristics. For instance, virtual machines can incur a significantly more resource overhead than containers. A virtual machine typically has an operating system, a full set of files and directory structures, a unique configuration, virtual memory, and applications, all of which can amount to tens of gigabytes in size. In contrast, containers (e.g., Docker-based containers) are software packages that provide a certain amount of facilities a software application or service needs to run, such as code, runtime, tools, system libraries, etc. Containers may share resources, for example an operating system kernel, files, folders, configuration, provisioning, storage, and network. This enables efficient use of host computing resources and lowers administrative costs. Containers typically have a much lower memory and disk footprints than virtual machines (e.g. megabytes instead of gigabytes in size).

Under certain scenarios, laptops, tablets, smartphones, or other types of computing devices may benefit from flexibilities of switching between using local and remote computing resources while sandboxing. For example, a computer (e.g., a laptop) can use local computing resources when the computer is in "airplane mode," out of range of a local computer network, or otherwise without network connectivity. In another example, a computer may use remote computing resources when performing compute-intensive tasks such as compiling code or running other calculation-heavy workloads. In another example, a remote embedded device may resort to local processing when partitioned from a network, however may automatically switch to remote computing resources when network connection is reestablished. In a further example, a computer may have a low battery level and need to use remote resources to conserve power and/or maintain an online presence.

Under at least some of the foregoing scenarios, containerization can help switching between using local and remote resources by abstracting a workload from underlying hardware and/or software. While migration of virtual machines is well understood, migrating a containerized workload may be difficult. Unlike virtual machines that contain a complete operating system, containers share resources from an underlying operating system kernel. Sometimes, containers and one or more applications executing in the containers, as well as an underlying operating system may have changed over time. Such changes can be a result of applying a software patch to the operating system and/or application for fixing security or functional issues, installing an extension of the containers or the one or more applications to add functionality, or other suitable modifications. In another example, tracking various operating system handles associated with the containers and the one or more applications can also be difficult because operating systems and/or software drivers typically do not provide such functionalities.

The disclosed technology is directed to implementing a synchronization engine for facilitating efficient migration of containers between source and destination computing devices. A computing device (e.g., a laptop, referred to below as "local device") determines whether a source container hosted on the local device (referred to as "local container") is to be migrated to a destination device such as a remote server located in a datacenter. Example criteria for the determination can be based on a power level, a current status of network connectivity, a current computation load, or other suitable parameters of the local device. In other examples, the criteria can also be based on user input, device or sensor input, migration schedule, or other suitable conditions. In a further example, a resource (e.g. a printer) is not locally available, and the workload is moved to a location (e.g., the datacenter) where that resource is available. In another example, a host detects audit log tampering and evacuates a sensitive workload to a safe remote location.

Upon determining that the local container is to be migrated, the synchronization engine on the local device is configured to transmit a migration request with suitable credentials to a cloud computing system indicating that a migration is about to occur. In response, the cloud computing system can be configured to authenticate the migration request based on the credentials and then select a remote server to instantiate a new container on the remote server (referred to as "remote container"). In certain implementations, the migration request can also contain data identifying version, build, or other suitable types of parameter of an operating system on the local device. The remote server (or a hypervisor thereon) instantiates a virtual machine having an operating system with compatible and/or the same version, build, or other types of parameter as that in the migration request.

The remote server is then configured to instantiate the remote container using pre-distributed or shared container images. Such container images can include digital data representing a complete file system having file folders, files, applications, configuration or other suitable information of the container. The container images can be identified using versions, builds, hash values, series numbers, or other suitable identifiers. In other implementations, the remote server can receive a copy of the container image from the local device with or without requesting the local device for the copy of the container image.

In certain embodiments, a template is used to represent a container and associated configuration parameters. Example configuration parameters can include one or more parameters of memory configuration, security settings, network settings, graphics settings, device settings. In some embodiments, this template may have a pre-defined schema that is shared with computers that support container migration. This template may then be synchronized between computers either on a schedule, or as a part of container migration.

In certain embodiments, the synchronization engine of the local device can also transmit an initial memory snapshot of the local container to the cloud computing systems along with the migration request or upon authorization to migrate the local container. The initial memory snapshot can include data of memory state, memory state, shell projection handles, or other suitable information. In other embodiments, instead of receiving the initial memory snapshot from the local device, a base memory snapshot corresponding to the local container, for instance, during an initial boot of the local container, can be pre-distributed or shared with the remote server. Upon obtaining the initial memory snapshot, the remote server instantiates the remote container using the initial memory snapshot or applying the initial memory snapshot to an already instantiated remote container.

Subsequently, the synchronization engine of the local device can be configured to generate and transmit to the remote server, one or more differences from the initial memory snapshot of the local container using a copy-on-write, a memory partition, or other suitable techniques. In response, the remote server can update the instantiated remote container with the received differences of the memory snapshot. In certain embodiments, upon completions of instantiating the remote container, the remote server can be configured to signal the local device that a migration of the local container to the cloud computing system is ready to occur. During such memory update/sharing, the images of the operating system on both the local device and the remote server need to be the same version. Additionally, memory fix-ups may be performed because library files in the operating systems may not be loaded at the same location on the local device and the remote server.

The remote server may provide a user of the local device certain choices as to when to finalize the migration, or proceed based on other suitable conditions. During migration, the remote server may also force one or more processes executing on the local device to sleep or quiesce prior to the migration. Once migration of the local container to the remote container is completed, display output at the local device is switched from the local container to the remote container, for instance, Remote Desktop Protocol (RDP), an X Window system, or other suitable techniques.

During the migration process, handles to local operating system resources for the local container can be tracked. A "handle" generally refers to an abstract reference to a computing resource. For example, handles can be used when an application references blocks of memory or objects managed by an operating system or a database. An example handle can be an opaque identifier or a pointer that allows access to further information regarding the computing resource. In one embodiment, a handle table (or other suitable types of data structure) can be implemented on both the local device and the remote server to ensure handles of the local container are also opened on the remote server. This may be achieved by synchronizing the handle tables wholesale or on a per-handle basis. In some implementations, the handle table may also be cached. For example, new handles on the remote server may result in a cache miss and be remotely fetched from the local device and migrated only as-needed. In further embodiments, the handles can also be stored in a linked list, a tree, or other suitable types of data structure.

The disclosed technology thus facilitates efficient switching between using local resources and using remote resources migration via containerization. By tracking the memory state of a migrated container, the container can be migrated from a local device to a remote server, or vice versa, in a seamless manner. As such, users can readily switching between using local resources and using remote resources with little or no interruption. Though the migration process described above in the context of migrating the local container as a source container from the local device to the remote server, in other embodiments, the migration process may also be used to migrate the remote container as the source container from the remote server to the local device in the reverse direction.

Certain embodiments of systems, devices, components, modules, routines, data structures, and processes for container migration to/from datacenters or other suitable computing facilities are described below. In the following description, specific details of components are included to provide a thorough understanding of certain embodiments of the disclosed technology. A person skilled in the relevant art will also understand that the technology can have additional embodiments. The technology can also be practiced without several of the details of the embodiments described below with reference to <FIG>.

As used herein, the term a "computing facility" generally refers to an interconnected computer network having a plurality of network nodes that connect a plurality of servers or hosts to one another or to external networks (e.g., the Internet). The term "network node" generally refers to a network device. Example network nodes include routers, switches, hubs, bridges, load balancers, security gateways, or firewalls. A "host" generally refers to a physical local device configured to implement, for instance, one or more virtual machines or other suitable virtualized components. For example, a host can include a remote server or remote device having a hypervisor configured to support one or more virtual machines, containers, or other suitable types of virtual components.

A computer network can be conceptually divided into an overlay network implemented over an underlay network. An "overlay network" generally refers to an abstracted network implemented over and operating on top of an underlay network. The underlay network can include multiple physical network nodes interconnected with one another. An overlay network can include one or more virtual networks. A "virtual network" generally refers to an abstraction of a portion of the underlay network in the overlay network. A virtual network can include one or more virtual end points referred to as "tenant sites" individually used by a user or "tenant" to access the virtual network and associated computing, storage, or other suitable resources. A tenant site can host one or more tenant end points ("TEPs"), for example, virtual machines. The virtual networks can interconnect multiple TEPs on different hosts. Virtual network nodes in the overlay network can be connected to one another by virtual links individually corresponding to one or more network routes along one or more physical network nodes in the underlay network.

Also used herein, the term "container" generally refers to a software package that contains a piece of software (e.g., an application) in a complete filesystem having codes (e.g., executable instructions), a runtime environment, system tools, system libraries, or other suitable components sufficient to execute the piece of software. Containers running on a single server or virtual machine can all share the same operating system kernel and can make efficient use of system memory or virtual memory. A container can have similar resource isolation and allocation benefits as virtual machines. However, a different architectural approach allows containers to be much more portable and efficient than virtual machines. For example, a virtual machine typically includes one or more applications, necessary binaries and libraries of the applications, and an entire operating system. In contrast, a container can include an application and all of its dependencies, but shares an operating system kernel with other containers on the same host. As such, containers can be more resource efficient and flexible than virtual machines. One example container is a Windows Server container by Microsoft Corporation of Redmond, Washington. Another example container is a Linux container or LXC. Docker is a popular mechanism to package and deliver containers, provided by Docker, Inc. of San Francisco, California.

Under certain scenarios, a container is switched between using local and remote resources. While migration of virtual machines is well understood, migrating a container workload may be difficult. Unlike virtual machines that contain a complete operating system, containers share resources from an underlying operating system kernel. Sometimes, containers, one or more applications executing in the containers, as well as an underlying operating system may have changed over time. Such changes can be a result of applying a software patch to the operating system and/or application for fixing security or functional issues, installing an extension of the containers or the one or more applications to add functionality, or other suitable modifications. In another example, tracking various operating system handles associated with the containers and the one or more applications can also be difficult because operating systems and/or software drivers typically do not provide such functionalities. The disclosed technology is directed to implementing a synchronization engine for facilitating efficiently migration of containers between local and remote computing devices, as described in more detail below with reference to <FIG>.

<FIG> is a schematic diagram illustrating a distributed computing system <NUM> implementing efficient container migration in accordance with embodiments of the disclosed technology. As shown in <FIG>, the distributed computing system <NUM> can include an underlay network <NUM> interconnecting a plurality of local devices <NUM> (shown as first, second, and third local devices 103a-103c, respectively) of corresponding users <NUM> (shown as first, second, and third user 101a-101c, respectively), and a computing facility <NUM>. Even though particular components are shown in <FIG>, in other embodiments, the distributed computing system <NUM> can also include additional and/or different constituents. For example, the distributed computing system <NUM> can include network storage devices, utility infrastructures, and/or other suitable components in addition to or in lieu of those shown in <FIG>.

The local devices <NUM> can each include a computing device that facilitates corresponding users <NUM> to access cloud services provided by the remote servers <NUM> via the underlay network <NUM>. For example, in the illustrated embodiment, the local devices <NUM> individually include a desktop computer. In other embodiments, the local devices <NUM> can also include laptop computers, tablet computers, smartphones, or other suitable computing devices. Even though three users <NUM> are shown in <FIG> for illustration purposes, in other embodiments, the distributed computing system <NUM> can facilitate any suitable number of users <NUM> to access suitable types of computing services provided by the remote servers <NUM>.

As shown in <FIG>, the underlay network <NUM> can include one or more physical network devices <NUM> that interconnect the local devices <NUM> and the computing facility <NUM>. Examples of the network devices <NUM> can include routers, switches, firewalls, load balancers, or other suitable network components. Even though particular connection scheme is shown in <FIG> for illustration purposes, in other embodiments, the network devices <NUM> can be operatively coupled in a hierarchical, flat, "mesh," or other suitable topologies.

The computing facility <NUM> can include a management controller <NUM> and a plurality of remote servers <NUM> operatively coupled to one another by the network devices <NUM>. In certain embodiments, the remote servers <NUM> can individually include a physical server or a computing blade having several physical servers. In other embodiments, the remote servers <NUM> can also include one or more physical servers with multiple processor cores, or other suitable types of computing devices. In any of the foregoing embodiments, the remote servers <NUM> can individually include one or more non-volatile memories (shown as NVMs <NUM> in <FIG>).

The remote servers <NUM> can be organized into racks, availability zones, groups, sets, computing clusters, or other suitable divisions. For example, in the illustrated embodiment of <FIG>, the remote servers <NUM> are grouped into three computing clusters <NUM> (shown individually as first, second, and third computing clusters 105a-105c, respectively), which are operatively coupled to corresponding network devices <NUM> in the underlay network <NUM>. Even though three computing clusters <NUM> are shown in <FIG> for illustration purposes, in other embodiments, the computing facility <NUM> can include one, two, eight, sixteen, or any other suitable numbers of computing clusters <NUM> with similar or different components and/or configurations.

Each cluster <NUM> can also include a cluster controller <NUM> configured to monitor status and manage operations of the remote servers <NUM> in the corresponding computing cluster <NUM>. For example, the cluster controller <NUM> can monitor whether a remote server <NUM> or components thereof has failed. In response to detecting a failure of the remote server <NUM> or components thereof, the cluster controller <NUM> can attempt to remedy the detected failure by, for instance, migrating virtual machines and/or containers hosted on the failed remote server <NUM> to other remote servers <NUM> in the same cluster <NUM>, restarting the failed remote server <NUM>, replacing hardware components of the failed remote server <NUM>, and/or perform other suitable operations. Though the cluster controllers <NUM> are shown as separate physical servers in <FIG>, in other embodiments, the cluster controllers <NUM> can also include computing services provided by one or more of the remote servers <NUM> in corresponding computing clusters <NUM>.

The management controller <NUM> can be configured to monitor, control, or otherwise manage operations of the computing clusters <NUM>. For example, in certain embodiments, the management controller <NUM> can include a fabric controller configured to manage processing, storage, communications, or other suitable types of hardware resources in the computing clusters <NUM> for hosting desired computing services. In other embodiments, the management controller <NUM> can also include a datacenter controller, application delivery controller, or other suitable types of controller. In the illustrated embodiment, the management controller <NUM> is shown as being separate from the computing clusters <NUM>. In other embodiments, the management controller <NUM> can include one or more remote servers <NUM> in the computing clusters <NUM>. In further embodiments, the management controller <NUM> can include software services hosted on one or more of the remote servers <NUM> in the computing clusters <NUM>.

In operation, the users <NUM> can request various computing services (e.g., deployment of a site) via, for example, user portals <NUM> presented on corresponding local devices <NUM>. In response, the management controller <NUM> can allocate one or more remote servers <NUM> or other computing resources (e.g., one or more remote servers <NUM>) to execute suitable instructions to provide the requested computing services. For example, the users <NUM> can request, via a corresponding local device <NUM> to migrate a local container (not shown) currently executing on the local device <NUM> to one or more remote servers <NUM> in the computing facility <NUM>, as described in more detail below with reference to <FIG>. In some embodiments, the request for computing resources may be automated by software running on local device <NUM>, user portals <NUM>, or other entities.

In some embodiments, the remote server <NUM> may be selected based on certain attributes of the remote servers <NUM>. Example attributes can include cost, location, network latency, availability, security guarantees, government data policy, and other suitable characteristics. In some embodiments, the local container 122a running on the local device <NUM> may be synchronized with multiple remote servers <NUM> to achieve high availability, optimal location, and/or other suitable objectives.

<FIG> is a schematic diagram illustrating certain hardware/software components of the distributed computing system <NUM> of <FIG> in accordance with embodiments of the disclosed technology. In <FIG> and in other Figures herein, individual software components, objects, classes, modules, and routines may be a computer program, procedure, or process written as source code in C, C++, C#, Java, and/or other suitable programming languages. A component may include, without limitation, one or more modules, objects, classes, routines, properties, processes, threads, executables, libraries, or other components. Components may be in source or binary form. Components may include aspects of source code before compilation (e.g., classes, properties, procedures, routines), compiled binary units (e.g., libraries, executables), or artifacts instantiated and used at runtime (e.g., objects, processes, threads). In certain embodiments, the various components and modules described below can be implemented with actors. In other embodiments, generation of the application and/or related services can also be implemented using monolithic applications, multi-tiered applications, or other suitable components.

Components within a system can take different forms within the system. As one example, a system comprising a first component, a second component and a third component can, without limitation, encompass a system that has the first component being a property in source code, the second component being a binary compiled library, and the third component being a thread created at runtime. The computer program, procedure, or process may be compiled into object, intermediate, or machine code and presented for execution by one or more processors of a personal computer, a network server, a laptop computer, a smartphone, and/or other suitable computing devices. Equally, components may include hardware circuitry.

A person of ordinary skill in the art would recognize that hardware may be considered fossilized software, and software may be considered liquefied hardware. As just one example, software instructions in a component may be burned to a Programmable Logic Array circuit, or may be designed as a hardware circuit with appropriate integrated circuits. Equally, hardware may be emulated by software. Various implementations of source, intermediate, and/or object code and associated data may be stored in a computer memory that includes read-only memory, random-access memory, magnetic disk storage media, optical storage media, flash memory devices, and/or other suitable computer readable storage media excluding propagated signals.

As shown in <FIG>, the first server 106a and the second server 106b can each include a processor <NUM>, a memory <NUM>, an input/output component <NUM>, and one or more non-volatile memories <NUM> operatively coupled to one another. The processor <NUM> can include a microprocessor, a field-programmable gate array, and/or other suitable logic devices. The memory <NUM> can include volatile and/or nonvolatile media (e.g., ROM; RAM, NVRAM, magnetic disk storage media; optical storage media; flash memory devices, and/or other suitable storage media) and/or other types of computer-readable storage media configured to store data received from, as well as instructions for, the processor <NUM> (e.g., instructions for performing the methods discussed below with reference to Figures 5A-<NUM>). The input/output component <NUM> can include a network interface card or other suitable types of input/output devices configured to accept input from and provide output to an operator and/or an automated software controller (not shown).

The memory <NUM> of the first and second remote servers 106a and 106b can include instructions executable by the corresponding processors <NUM> to cause the individual remote servers <NUM> to provide a hypervisor <NUM> (identified individually as first and second hypervisors 140a and 140b) and other suitable virtual components such as virtual network interface card, virtual switches, etc. (not shown). The hypervisors <NUM> can individually be configured to initiate, monitor, terminate, and/or otherwise locally manage a host <NUM> and one or more virtual machines <NUM> (or containers) organized into tenant sites <NUM>. For example, as shown in <FIG>, the first server 106a can provide a first hypervisor 140a that manages first and second tenant sites 142a and 142b, respectively, for the same or different tenants or users <NUM> (<FIG>). The second server 106b can provide a second hypervisor 140b that manages first and second tenant sites 142a' and 142b', respectively.

The hypervisors <NUM> can be software, firmware, or hardware components. The tenant sites <NUM> can each include multiple virtual machines <NUM> or other suitable tenant instances for a tenant. For example, the first server 106a and the second server 106b can both host the tenant site 142a and 142a' for a first user 101a (<FIG>). The first server 106a and the second server 106b can both host the tenant site 142b and 142b' for a second user 101b (<FIG>). Each virtual machine <NUM> can be executing a corresponding operating system, middleware, and/or applications.

Also shown in <FIG>, the distributed computing system <NUM> can include one or more virtual networks <NUM> that interconnect the tenant sites 142a and 142b across multiple remote servers <NUM>. For example, a first virtual network 142a interconnects the first tenant sites 142a and 142a' at the first server 106a and the second server 106b. A second virtual network 146b interconnects the second tenant sites 142b and 142b' at the first server 106a and the second server 106b. Even though a single virtual network <NUM> is shown as corresponding to one tenant site <NUM>, in other embodiments, multiple virtual networks <NUM> (not shown) may be configured to correspond to a single tenant site <NUM>.

The virtual machines <NUM> on the virtual networks <NUM> can communicate with one another via the underlay network <NUM> (<FIG>) even though the virtual machines <NUM> are located on different remote servers <NUM>. Communications of each of the virtual networks <NUM> can be isolated from other virtual networks <NUM>. In certain embodiments, communications can be allowed to cross from one virtual network <NUM> to another through a security gateway or otherwise in a controlled fashion. A virtual network address can correspond to one of the virtual machine <NUM> in a virtual network <NUM>. Thus, different virtual networks <NUM> can use one or more virtual network addresses that are the same. Example virtual network addresses can include IP addresses, MAC addresses, and/or other suitable addresses.

As shown in <FIG>, the hypervisor <NUM> and/or the host <NUM> can assign one or more of the non-volatile memories <NUM> to be accessed by a virtual machine <NUM> via a PCIe bus. For example, the first server 106a can assign a non-volatile memory <NUM> to the virtual machine <NUM>'. The second server 106b can assign another non-volatile memory <NUM> to the virtual machine <NUM>". As alluded to above, the servers <NUM> can utilize one or more of the virtual machines <NUM> to facilitate efficient container migration from the local devices <NUM> (<FIG>) and vice versa, as described in more detail below with reference to <FIG>.

<FIG> are block diagrams illustrating certain hardware/software components of a local device <NUM> and a remote server <NUM> during certain operational stages migrating a local container 122a from the local device <NUM> to the remote server <NUM> in accordance with embodiments of the disclosed technology. Though the operations are described in the context of migrating the local container 122a (as a source container) from the local device <NUM> as a source device to be instantiated as the remote container 122b (as a destination container) on the remote server <NUM> as the destination device, in certain implementations, at least some operations described herein are equally applicable to migrating the remote container 122b from the remote server <NUM> as a source device to the local device <NUM> as a destination device. In addition, even though the migration process is described below as being triggered by a request <NUM> from the user <NUM>, in other implementations, the migration process can be initiated based on a current operating state of the local device <NUM>. Example operating states can include a power level, a status of network connectivity, a processor usage, a level of system storage, security status, attestation status, or other suitable operating parameters.

In certain embodiments, the local device <NUM> can include a data store 110a containing one or more container images 114a and a processor (not shown) configured to execute suitable instructions of one or more of the container images 114a to provide the local container 122a, for instance, as a container runtime. As used herein, a "runtime" generally refers to a computing environment in which a piece of software runs (e.g. programs loaded into memory, instructions scheduled and executed with a processor). In other embodiments, the container images 114a can also be available from a remote, removable, or other suitable types of storage location. As shown in <FIG>, the local container 122a can also be allocated to and allowed to access a container memory 124a during execution of the local container 122a and/or one or more applications (not shown) in the local container 122a. As such, the container memory 124a can be configured to contain memory state of the local container 122a and/or one or more applications executing in the local container 122a. Though the local container 122a is shown in <FIG> as being hosted by the local device <NUM> directly, in some embodiments, the local device <NUM> can also execute suitable instructions to provide a virtual machine (not shown) to host the local container 122a.

In certain implementations, the local device <NUM> can also include an output display <NUM> configured to surface execution results <NUM> of the local container 122a to the user <NUM>. In certain embodiments, the output display <NUM> can include a user interface element (e.g., a graphical user interface) surfaced on an output component (e.g., a LED screen) of the local device <NUM>. In other embodiments, the output display <NUM> can include other suitable hardware/software components. Various protocols can be used to surface the result <NUM> on the output display <NUM>. One suitable example protocol is Remote Desktop Protocol (RDP). In some embodiments, the protocol simply reconnects the remote container 122b to the output display <NUM>. In other embodiments, the protocol and container migration can be integrated and orchestrated. A make before break or other similar types of technique may be applied to reduce the interruption to output display <NUM>. In some embodiments, the user <NUM> may have a seamless experience when accessing output display <NUM> because the remote container 122b quickly instantiates remotely. In some embodiments, the user <NUM> may be notified of a failure or delay. In some embodiments in which the container migration is automatic, the user <NUM> may be notified of the change. In other implementations, the local device <NUM> may not include an output display <NUM>, for instance, when the local device <NUM> is an embedded device. Instead, execution results from the local or remote container 122a and 122b can be used to actuate, transmit, or perform other suitable operations.

As shown in <FIG>, each of the local device <NUM> and the remote server <NUM> can also implement a synchronization engine <NUM> (shown as local and remote synchronization engines 150a and 150b, respectively) configured to facilitate container migration between the local device <NUM> and the remote server <NUM>. The synchronization engine <NUM> can include an interface component <NUM>, a memory component <NUM>, and a control component <NUM> operatively coupled to one another. Even though the functionalities of various components of the synchronization engines <NUM> are described below in the context of migrating the local container 122a to the remote server <NUM>, similar components of the synchronization engines 150a and 150b can have similar or the same functionalities when migrating a remote container 122b (shown in <FIG>) from the remote server <NUM> to the local device <NUM>.

The interface components <NUM> of the synchronization engines <NUM> can be configured to communicate with each other via a suitable synchronization protocol and transport used to synchronize state between the local device <NUM> and the remote server <NUM>. In certain embodiments, the individual interface components <NUM> can include a network interface card and corresponding software drivers. In other embodiments, the interface components <NUM> can include other suitable hardware/software components.

The interface component <NUM> can also be configured to transmit or relay a migration request <NUM> between the local device <NUM> and the remote server <NUM>. For instance, as shown in <FIG>, upon receiving the request <NUM> from the user <NUM>, the interface component <NUM> at the local device <NUM> can relay the request <NUM> to the remote server <NUM> along with, for example, credentials <NUM> for validating the user <NUM>, device information <NUM> containing version, build, or other suitable parameters of the local device <NUM>, and/or other suitable information. In other examples, the credentials <NUM> and/or the device information <NUM> can be pre-distributed or shared between the local device <NUM> and the remote server <NUM>. In some embodiments, the device information <NUM> may contain a configuration template that contains certain aspects of the container configuration. In some embodiments, the device information <NUM> may contain one or more of an operating system version, a list of applied patches, driver version information, firmware version information, and/or other suitable information. In some embodiments, the request <NUM> may be automatically generated by a component (e.g., a container engine) on local device <NUM> instead of being initiated by the user <NUM>.

Upon receiving the request <NUM>, the remote server <NUM> can be configured to validate the user <NUM> based on the received or pre-distributed credentials <NUM>. Upon validating the user <NUM>, the control component <NUM> at the remote server <NUM> is configured to instantiate a virtual machine <NUM> based on the received or pre-distributed device information <NUM>. As such, the virtual machine <NUM> is provided with an operating system kernel that is compatible with or the same as that of the local device <NUM>.

The memory components <NUM> at the local device <NUM> can be configured to synchronize both an image and memory state for a migrated container. In one embodiment, as shown in <FIG>, upon receiving the request <NUM> to migrate the local container 122a to the remote server <NUM>, the memory component <NUM> can be configured to generate a container image <NUM> of the local container 122a. The container image <NUM> can include digital data representing a filesystem, one or more applications executing in the local container, and configuration parameters thereof. Example techniques for generating the container image <NUM> can include executing a "commit" or "file management" command for Docker. Upon receiving a user request <NUM>, the memory component <NUM> can also be configured to capture, from the container memory 124a, a memory snapshot <NUM> containing a memory state of the local container 122a and the one or more applications executing in the local container 122a. In other embodiments, the container image <NUM> can be pre-distributed to the remote server <NUM>. As such, instead of receiving the container image <NUM> from the local device <NUM>, the remote server <NUM> can be configured to retrieve a copy of the container image <NUM> from the data store 110b based on, for instance, the device information <NUM>.

In certain implementations, execution of the local container 122a can then be paused subsequent to capturing the memory snapshot <NUM> while the memory component <NUM> can be configured to transmit the generated image and the captured memory snapshot <NUM> to the remote server <NUM>. Upon receiving the container image <NUM> and the memory snapshot <NUM>, the remote server <NUM> instantiates a remote container 122b based on the container image <NUM> and the memory snapshot <NUM>. Subsequently, the remote server <NUM> continues executing the one or more applications in the remote container 122b and provides a result <NUM>' of such execution to the output display <NUM> at the local device <NUM>.

In certain implementations, the local container 122a has shared memory with the host (e.g., the local device <NUM> or a virtual machine thereon). When shared memory is mapped into the local container 122a, the memory snapshot <NUM> proceeds and the memory map information can also be captured and shared with the remote server <NUM> via the device information <NUM> or a similar mechanism. When the remote container 122b on the remote server <NUM> is instantiated, the same memory map can be created as a part of this instantiation.

In another implementation, execution of the local container 122a can continue after capturing the memory snapshot <NUM>, as shown in <FIG>. Subsequent to transmitting the initial memory snapshot <NUM> as shown in <FIG>, the memory component <NUM> can be configured to capture additional memory snapshots and generate one or more memory difference <NUM>' based on the subsequently captured memory snapshots relative to a prior one. The memory component <NUM> can then be configured to transmit the memory difference <NUM>' to the remote server <NUM> continuously, periodically, or in other suitable manners. Upon receiving the memory difference <NUM>', the control component <NUM> at the remote server <NUM> can be configured to apply the received memory difference to the instantiated remote container 122b. The control component <NUM> can then transmit, via the interface component <NUM>, a query <NUM> to the local device <NUM> inquiring whether any additional memory difference <NUM>' is still present on the local device <NUM>.

As shown in <FIG>, upon receiving a response <NUM> from the local device <NUM> indicating that no more memory difference <NUM>' is present on the local device <NUM>, the remote server <NUM> continues executing the one or more applications in the remote container 122b and provides the execution result <NUM>' to the output display <NUM> of the local device <NUM>. In certain embodiments, before the output display <NUM> is switched from the local container 122a to the remote container 122b, the remote server <NUM> can also optionally transmit a pause command <NUM> to the local device <NUM>. In response to the pause command <NUM>, the control component <NUM> at the local device <NUM> can be configured to pause the execution of the local container 122a and switch the output display <NUM> to the remote container 122b. In other embodiments, switching the output display <NUM> can be performed without pausing the local container 122a.

Even though the migration process described above with reference to <FIG> involves capturing a memory snapshot of the local container 122a and instantiating the remote container 122b based thereon, in other embodiments, the memory snapshot <NUM> can also be pre-distributed between the local device <NUM> and the remote server <NUM>. For example, as shown in <FIG>, both container images <NUM> and memory snapshots <NUM> corresponding to the container images <NUM> can be pre-distributed between the local device <NUM> and the remote server <NUM>. In one example, the pre-distributed memory snapshot <NUM> can include a captured memory state of the local or remote container <NUM> during a boot up operation. In other examples, the pre-distributed memory snapshot <NUM> can include a captured memory state of the local or remote container <NUM> during other suitable execution points.

As such, during the migration process, the remote server <NUM> can be configured to retrieve both the container image <NUM> and an initial memory snapshot <NUM> from the data store 110b. The memory component <NUM> at the local device <NUM> can then be configured to generate and transmit to the remote server <NUM> one or more memory differences <NUM>' to be applied to the instantiated remote container 122b, as described above with reference to <FIG>. Subsequently, the remote server <NUM> can be configured to transmit the query <NUM> to the local device <NUM>, and switch the output display <NUM> to the remote container 122b upon receiving the response <NUM>, as described above with reference to <FIG>.

During the migration process described above with reference to <FIG>, handles to local operating system resources for the local container 122a can be tracked. A "handle" generally refers to an abstract reference to a computing resource. For example, handles can be used when an application references blocks of memory or objects managed by an operating system or a database. An example handle can be an opaque identifier or a pointer that allows access to further information regarding the computing resource. In one embodiment, a handle table can be implemented on both the local device <NUM> and the remote server <NUM> to ensure handles of the local container are also opened on the remote server. This may be achieved by synchronizing the handle tables wholesale, or on a per-handle basis. In some implementations, the handle table may also be cached. For example, new handles on the remote server <NUM> may result in a cache miss and be remotely fetched from the local device <NUM> and migrated only as-needed.

The disclosed technology thus facilitates efficient switching between using local resources and using remote resources migration via containerization. By tracking the memory state of a migrated local container 122a (or a remote container 122b), the container <NUM> can be migrated from the local device <NUM> to the remote server <NUM>, or vice versa, in a seamless manner. As such, users <NUM> can readily switching between using local resources and using remote resources with little or no interruption.

Several embodiments of the disclosed technology can also implement embodiments described above to migrate containers running on a remote server <NUM> to a local device <NUM>. For users who have multiple local devices <NUM> (e.g. a desktop computer, a laptop computer, a phone, etc.), attributes of such local devices <NUM> may be shared as a part of the device information <NUM>, and thus enabling the synchronization engine 150b to target the appropriate local device <NUM>. In one embodiment, user presence may be used to determine an interactive application scenario may be migrated to a laptop computer that the user is currently using. In another embodiment, compute resources may be used to determine a compute intensive application may be migrated to a desktop computer.

<FIG> is a flowchart illustrating a process <NUM> of container migration in accordance with embodiments of the disclosed technology. Even though the process <NUM> is described in relation to the distributed computing system <NUM> of <FIG> and <FIG>, in other embodiments, the process <NUM> can also be implemented in other suitable systems.

As shown in <FIG>, the process <NUM> includes receiving a request to migrate a container from a source device to a destination device at stage <NUM>. In one embodiment, the container can be migrated from a local device <NUM> (<FIG>) to a remote server <NUM> (<FIG>). In another embodiment, the container can be migrated from the remote server <NUM> to the local device <NUM>. In a further embodiment, the container can be migrated from one remote server <NUM> to another in the computing facility <NUM> (<FIG>).

The process <NUM> then include a stage <NUM> in which a virtual machine is started on the destination device, as described above with reference to <FIG>. In some embodiments, operations at stage <NUM> may also include allocating additional system resources such as memory, devices, etc. Such just-in-time resource allocation may be determined by the device information <NUM> or other indicators. In other embodiments, the operation at stage <NUM> can be omitted such that the container is migrated directly to the destination device.

The process <NUM> then includes instantiating a remote container at stage <NUM>. The remote container <NUM> is instantiated based on a copy of captured current image and memory snapshot of the container on the source device. At least one of the image or memory snapshot of the container can be pre-distributed between the source and the destination devices. Example operations of instantiating the remote container are described in more detail below with reference to <FIG> and <FIG>. The process <NUM> then includes a decision stage <NUM> to determine whether instantiating the remote container is complete. In response to determining that instantiating the remote container is complete, the process <NUM> includes switching a display output on the source device to the remote container at stage <NUM>, as described above with reference to <FIG>.

<FIG> is a flowchart illustrating example operations of instantiating the remote container. As shown in <FIG>, the operations include receiving a copy of a captured image and memory snapshot of a container to be migrated at stage <NUM>. The operations can then include allocating various resources (e.g., system or virtual memory) to the remote container at stage <NUM>. The operations can then include booting the remote container using the received coy of the captured image and memory snapshot at stage <NUM>.

<FIG> is a flowchart illustrating additional example operations of instantiating the remote container. As shown in <FIG>, the operations can include receiving a container identifier of the container to be migrated at stage <NUM>. Such a container identifier can include a serial number or other suitable types of identifier. The operations can then include retrieving at least one of pre-distributed image or memory snapshot of the container based on the container identifier at stage <NUM>. The operations can then include booting the remote container <NUM> using the retrieved image and/or memory snapshot at stage <NUM>.

The operations can then include a decision stage <NUM> to determine whether any differential memory snapshots of the container exist at the source device. In response to determining that additional differential memory snapshots of the container exist at the source device, the operations can proceed to receiving and applying the additional differential memory snapshots to the remote container at stage <NUM> before revert back to determining whether further differential memory snapshots of the container exist at the source device. In certain implementations, the operation at stage <NUM> can also include updating the pre-distributed image and/or memory snapshot of the container based on the additional differential memory snapshot. In response to determining that no differential memory snapshots of the container exist at the source device, the operations can proceed to an optional stage <NUM> of instructing the source device to pause execution of the container and then indicating that instantiation of the remote container is complete at stage <NUM>.

<FIG> is a flowchart illustrating example operations of synchronizing handles between a source device and a destination device during migration of a container from the source device to the destination device. As shown in <FIG>, the operations can optionally include partially or completely synchronizing a handle table containing entries identifying various handles between the source and destination devices at stage <NUM>. The operations can then include a decision stage <NUM> to determine whether a new handle corresponds to an entry in the synchronized handle table. In response to determining that the new handle does not correspond to an entry in the synchronized handle table, the operations can include retrieving information regarding the new handle from the source device at stage <NUM>. In response to determining that the new handle does correspond to an entry in the synchronized handle table, the operations can include retrieving information regarding the new handle from the handle table at stage <NUM>.

<FIG> is a computing device <NUM> suitable for certain components of the distributed computing system <NUM> in <FIG>. For example, the computing device <NUM> can be suitable for the local devices <NUM> or the remote servers <NUM> of <FIG>. In a very basic configuration <NUM>, the computing device <NUM> can include one or more processors <NUM> and a system memory <NUM>. A memory bus <NUM> can be used for communicating between processor <NUM> and system memory <NUM>.

Depending on the desired configuration, the processor <NUM> can be of any type including but not limited to a microprocessor (µP), a microcontroller (µC), a digital signal processor (DSP), or any combination thereof. The processor <NUM> can include one more levels of caching, such as a level-one cache <NUM> and a level-two cache <NUM>, a processor core <NUM>, and registers <NUM>. An example processor core <NUM> can include an arithmetic logic unit (ALU), a floating point unit (FPU), a digital signal processing core (DSP Core), or any combination thereof. An example memory controller <NUM> can also be used with processor <NUM>, or in some implementations memory controller <NUM> can be an internal part of processor <NUM>.

Depending on the desired configuration, the system memory <NUM> can be of any type including but not limited to volatile memory (such as RAM), non-volatile memory (such as ROM, flash memory, etc.) or any combination thereof. The system memory <NUM> can include an operating system <NUM>, one or more applications <NUM>, and program data <NUM>. As shown in <FIG>, the operating system <NUM> can include a hypervisor <NUM> for managing one or more virtual machines <NUM>. This described basic configuration <NUM> is illustrated in <FIG> by those components within the inner dashed line.

The computing device <NUM> can have additional features or functionality, and additional interfaces to facilitate communications between basic configuration <NUM> and any other devices and interfaces. For example, a bus/interface controller <NUM> can be used to facilitate communications between the basic configuration <NUM> and one or more data storage devices <NUM> via a storage interface bus <NUM>. The data storage devices <NUM> can be removable storage devices <NUM>, non-removable storage devices <NUM>, or a combination thereof. Examples of removable storage and non-removable storage devices include magnetic disk devices such as flexible disk drives and hard-disk drives (HDD), optical disk drives such as compact disk (CD) drives or digital versatile disk (DVD) drives, solid state drives (SSD), and tape drives to name a few. Example computer storage media can include volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information, such as computer readable instructions, data structures, program modules, or other data. The term "computer readable storage media" or "computer readable storage device" excludes propagated signals and communication media.

The system memory <NUM>, removable storage devices <NUM>, and non-removable storage devices <NUM> are examples of computer readable storage media. Computer readable storage media include, but not limited to, RAM, ROM, NVRAM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other media which can be used to store the desired information and which can be accessed by computing device <NUM>. Any such computer readable storage media can be a part of computing device <NUM>. The term "computer readable storage medium" excludes propagated signals and communication media.

The computing device <NUM> can also include an interface bus <NUM> for facilitating communication from various interface devices (e.g., output devices <NUM>, peripheral interfaces <NUM>, and communication devices <NUM>) to the basic configuration <NUM> via bus/interface controller <NUM>. Example output devices <NUM> include a graphics processing unit <NUM> and an audio processing unit <NUM>, which can be configured to communicate to various external devices such as a display or speakers via one or more A/V ports <NUM>. Example peripheral interfaces <NUM> include a serial interface controller <NUM> or a parallel interface controller <NUM>, which can be configured to communicate with external devices such as input devices (e.g., keyboard, mouse, pen, voice input device, touch input device, etc.) or other peripheral devices (e.g., printer, scanner, etc.) via one or more I/O ports <NUM>. An example communication device <NUM> includes a network controller <NUM>, which can be arranged to facilitate communications with one or more other local devices <NUM> over a network communication link via one or more communication ports <NUM>.

The network communication link can be one example of a communication media. Communication media can typically be embodied by computer readable instructions, data structures, program modules, or other data in a modulated data signal, such as a carrier wave or other transport mechanism, and can include any information delivery media. A "modulated data signal" can be a signal that has one or more of its characteristics set or changed in such a manner as to encode information in the signal. By way of example, and not limitation, communication media can include wired media such as a wired network or direct-wired connection, and wireless media such as acoustic, radio frequency (RF), microwave, infrared (IR) and other wireless media. The term computer readable media as used herein can include both storage media and communication media.

The computing device <NUM> can be implemented as a portion of a small-form factor portable (or mobile) electronic device such as a cell phone, a personal data assistant (PDA), a personal media player device, a wireless web-watch device, a personal headset device, an application specific device, or a hybrid device that include any of the above functions. The computing device <NUM> can also be implemented as a personal computer including both laptop computer and non-laptop computer configurations.

Claim 1:
A method performed in a computing system (<NUM>) having a source device (<NUM>) interconnected to a destination device (<NUM>) by a computer network, the method comprising:
receiving, at the destination device (<NUM>), a request to migrate a source container (122A) currently executing on the source device (<NUM>) to the destination device (<NUM>), the source container (122A) including a software package having a software application in a filesystem sufficiently complete for execution of the software application in an operating system by a processor of the source device (<NUM>) to provide a display output of the software application; and
in response to the received request from the source device (<NUM>), at the destination device (<NUM>),
starting a virtual machine (<NUM>) having an operating system that is compatible with that of the source device (<NUM>);
instantiating, in the started virtual machine (<NUM>), a destination container (122B) using a copy of an image and a memory snapshot of the source container (122A) on the source device (<NUM>); and
upon completion of instantiating the destination container (122B) at the destination device (<NUM>):
continuing execution of the software application in the destination container (122B); and
transmitting, via the computer network, a remote display output of the software application from the destination container (122B) to the source device (<NUM>) to be surfaced on the source device (<NUM>).