Patent Description:
Broadly speaking, a cloud computing system includes two sections, a front end and a back end, that are in electronic communication with one another via the Internet. The front end includes the interface that users encounter through a client device. The back end includes the resources that deliver cloud-computing services, including processors, memory, storage, and networking hardware. The back end of a cloud computing system typically includes one or more data centers, which may be located in different geographical areas. Each data center typically includes a large number (e.g., hundreds or thousands) of computing devices, which may be referred to as host machines.

At least some of the services that are offered by a cloud computing service provider may utilize virtualization technologies that allow computing resources to be shared by multiple users. For example, virtualization technologies allow a single physical computing device to be shared among multiple users by providing each user with one or more virtual machines hosted by the single physical computing machine. Each such virtual machine may act as a distinct logical computing system, and the various virtual machines may be isolated from one another. As another example, virtualization technologies allow data storage hardware to be shared among multiple users by providing each user with a virtual data store. Each such virtual data store may act as a distinct logical data store, and the various virtual data stores may be isolated from one another.

Virtualization technologies may also be used in the context of computer networking. Network virtualization involves combining hardware and software network resources and network functionality into a software-based administrative entity, which may be referred to as a virtual network (VNET). A cloud computing service provider may enable users (e.g., customers) to create VNETs within a cloud computing system. The use of network virtualization technologies in the context of a cloud computing environment is sometimes referred to as software-defined networking.

Resources may be assigned to VNETs. In this context, the term "resource" may refer to any item that is capable of being managed by a cloud computing system. Some examples of resources include virtual machines, virtual data stores, databases, and web applications. The resources within a VNET may communicate with each other and with other entities that are accessible via the Internet.

Virtual machines (VMs) that are assigned to VNETs may be referred to herein as "VNET VMs. " Virtual machines that are not assigned to VNETs may be referred to herein as "non-VNET VMs. " In other words, a non-VNET VM has not been assigned to a VNET, whereas a VNET VM has been assigned to a VNET. Generally speaking, VNET VMs are able to communicate with other VNET VMs, and non-VNET VMs are able to communicate with other non-VNET VMs. Currently, however, VNET VMs are unable to communicate directly with non-VNET VMs (and vice versa). Benefits may be realized by techniques that facilitate such communication. Document <CIT> discloses a distributed routing domain wherein each user or tenant can deploy a multi-subnet routing topology in a network-virtualized datacenter. A virtualization module implements the distributed routing domain and enforces a multi-subnet routing topology in a distributed fashion without requiring a standalone physical router or VM router. The topology and the routing rules are distributed in a network virtualization module on each hypervisor host, and collectively realize the multi-subnet topology for a virtual network over any physical network topology.

In accordance with one aspect of the present disclosure, a method is disclosed that includes migrating a virtual machine to a virtual network (VNET). The virtual machine is one of a plurality of non-VNET virtual machines that are deployed in a cloud computing system. The virtual machine is associated with a physical internet protocol (IP) address. The method also includes assigning a VNET address to the virtual machine and causing the virtual machine to operate in a hybrid state. In the hybrid state the virtual machine uses the physical IP address to communicate with other non-VNET virtual machines. In the hybrid state the virtual machine uses the VNET address to communicate with other VNET virtual machines. The method also includes providing a network stack on a host machine with a first packet processing rule set and a second packet processing rule set. The virtual machine runs on the host machine. The first packet processing rule set is configured to process first data packets corresponding to a first address space that has been defined for the virtual network. The second packet processing rule set is configured to process second data packets corresponding to a second address space that is distinct from the first address space and that does not overlap with the first address space.

The migration of the virtual machine to the VNET may occur in such a way that the virtual machine does not lose connectivity with the other non-VNET virtual machines as the virtual machine is migrated to the virtual network.

The first packet processing rule set may include encapsulation rules that specify how encapsulation should be performed on the first data packets. The second packet processing rule set may permit the second data packets to be transmitted without encapsulation.

The method may further include migrating the plurality of non-VNET virtual machines to the virtual network and deploying a plurality of pure VNET virtual machines within the virtual network.

In accordance with another aspect of the present disclosure, a system, as defined in claim <NUM>, is disclosed that includes one or more processors, memory in electronic communication with the one or more processors, and instructions stored in the memory. The instructions are executable by the one or more processors to create a virtual network (VNET), migrate a non-VNET virtual machine to the VNET, and cause the non-VNET virtual machine to transition to a hybrid virtual machine that operates in a hybrid state. The hybrid virtual machine has both a physical internet protocol (IP) address and a VNET address. The hybrid virtual machine is configured to use the physical IP address to communicate with other non-VNET virtual machines. The hybrid virtual machine is configured to use the VNET address to communicate with other VNET virtual machines. The instructions are also executable by the one or more processors to provide a network stack with a first packet processing rule set and a second packet processing rule set. The first packet processing rule set is configured to process first data packets corresponding to a first address space that has been defined for the virtual network. The second packet processing rule set is configured to process second data packets corresponding to a second address space that is distinct from the first address space and that does not overlap with the first address space.

The hybrid virtual machine can maintain connectivity with the other non-VNET virtual machines during migration to the VNET.

The network stack may be included within a first host machine. The non-VNET virtual machine may run on the first host machine.

The network stack may be configured to receive a data packet that includes a destination address, compare the destination address to at least one of the first address space and the second address space, and select a rule set for processing the data packet based on the comparison.

The system may further include a VNET virtual machine that runs on a second host machine. The network stack may be configured to encapsulate a data packet that is destined for the VNET virtual machine to form an encapsulated data packet.

The encapsulated data packet may include a header and a payload. The header of the encapsulated data packet may include a header source address and a header destination address. The hybrid virtual machine may run on a first host machine. The header source address may include a first physical IP address that is associated with the first host machine. The header destination address may include a second physical IP address that is associated with the second host machine. The payload of the encapsulated data packet may include the data packet.

The system may further include additional instructions that are executable by the one or more processors to migrate a plurality of non-VNET virtual machines to the VNET. The plurality of non-VNET virtual machines may maintain connectivity with the other non-VNET virtual machines during migration to the VNET. The system may further include additional instructions that are executable by the one or more processors to deploy a plurality of pure VNET virtual machines within the virtual network subsequent to the migration.

Each non-VNET virtual machine may be identified by a unique physical IP address. The plurality of pure VNET virtual machines may not be individually associated with physical IP addresses.

In accordance with another aspect of the present disclosure, a computer-readable medium, as defined in claim <NUM>, is disclosed that includes instructions that are executable by one or more processors to cause a network stack within a host machine to receive a first data packet that includes a first source address and a first destination address. The first source address corresponds to a hybrid virtual machine that is part of a virtual network. The first destination address corresponds to a first destination virtual machine. The computer-readable medium also includes instructions that are executable by one or more processors to determine, based on the first destination address, that the first destination virtual machine belongs to the virtual network. The computer-readable medium also includes instructions that are executable by one or more processors to encapsulate the first data packet based on encapsulation rules that are specified in a first packet processing rule set and receive a second data packet that includes a second source address and a second destination address. The second source address corresponds to the hybrid virtual machine. The second destination address corresponds to a second destination virtual machine. The computer-readable medium also includes instructions that are executable by one or more processors to determine, based on the second destination address, that the second destination virtual machine does not belong to the virtual network. The computer-readable medium also includes instructions that are executable by one or more processors to cause the second data packet to be transmitted to the second destination virtual machine without encapsulation based on a second packet processing rule set that is different from the first packet processing rule set.

The first source address may include a virtual network address. The second source address may include a physical internet protocol (IP) address.

The computer-readable medium may further include additional instructions that are executable by the one or more processors to define a first address space for the virtual network and define a second address space for virtual machines outside of the virtual network. The second address space may be distinct from the first address space and may not overlap with the first address space.

Determining that the first destination virtual machine belongs to the virtual network may include determining that the first destination address is included within the first address space.

Determining that the second destination virtual machine does not belong to the virtual network may include determining that the second destination address is not included within the first address space.

Determining that the second destination virtual machine does not belong to the virtual network may include determining that the second destination address is included within the second address space.

Encapsulating the first data packet forms an encapsulated data packet that may include a header and a payload. The header of the encapsulated data packet may include a header source address and a header destination address. The header source address may include a first physical internet protocol (IP) address. The first physical IP address may be associated with a first host machine that comprises the hybrid virtual machine. The header destination address may include a second physical IP address. The second physical IP address may be associated with a second host machine that includes the second destination virtual machine. The payload of the encapsulated data packet may include the first data packet.

Additional features and advantages will be set forth in the description that follows. Features and advantages of the disclosure may be realized and obtained by means of the systems and methods that are particularly pointed out in the appended claims. Features of the present disclosure will become more fully apparent from the following description and appended claims, or may be learned by the practice of the disclosed subject matter as set forth hereinafter.

In order to describe the manner in which the above-recited and other features of the disclosure can be obtained, a more particular description will be rendered by reference to specific embodiments thereof which are illustrated in the appended drawings. For better understanding, the like elements have been designated by like reference numbers throughout the various accompanying figures. Understanding that the drawings depict some example embodiments, the embodiments will be described and explained with additional specificity and detail through the use of the accompanying drawings in which:.

As noted above, VNET VMs are currently unable to communicate directly with non-VNET VMs (and vice versa). The present disclosure describes techniques that facilitate such communication. More specifically, the present disclosure is related to a hybrid state for a virtual machine (VM) in a cloud computing system. The hybrid state enables a VM to communicate with both VNET VMs and non-VNET VMs.

In at least some implementations of the techniques disclosed herein, there may be a unique physical IP address associated with each non-VNET VM. In this context, the term "physical IP address" can refer to an IP address that is routable on a physical computer network. Non-VNET VMs can use physical IP addresses to communicate with each other.

For example, consider two non-VNET VMs: a first non-VNET VM and a second non-VNET VM. The first non-VNET VM can be associated with a first physical IP address, which may be referred to as PA1. The second non-VNET VM can be associated with a second physical IP address, which may be referred to as PA2. Suppose that the first non-VNET VM sends a data packet to the second non-VNET VM. In this example, the source address of the data packet would be PA1, and the destination address of the data packet would be PA2.

In at least some implementations of the techniques disclosed herein, a VNET VM can use a private virtual IP address for communication with other VNET VMs. This private virtual IP address may be referred to herein as a VNET address. VNET addresses can be assigned by customers of a cloud computing provider, in which case a VNET address may be referred to as a customer address (CA). The VNET address (or CA) is unique within the context of the VNET, but may not be unique outside of that context. VNET VMs can use VNET addresses to communicate with each other. In addition, encapsulation can be performed so that the physical IP addresses of the host machines on which the VNET VMs are running are also used to facilitate communication between VNET VMs.

For example, consider two VNET VMs: a first VNET VM and a second VNET VM. For purposes of the present example, it will be assumed that the first VNET VM is associated with a first VNET address (e.g., a first customer address), which may be referred to as CA1. It will also be assumed that the second VNET VM is associated with a second VNET address (e.g., a second customer address), which may be referred to as CA2. In addition, it will be assumed that the VNET VMs are running on different host machines. In particular, it will be assumed that the first VNET VM is running on a first host machine with a first physical IP address, which may be referred to as PA1. It will also be assumed that the second VNET VM is running on a second host machine with a second physical IP address, which may be referred to as PA2. Suppose that the first VNET VM sends a data packet to the second VNET VM. The first VNET VM would create a data packet with a source address of CA1 and a destination address of CA2. This data packet would be delivered to a network stack on the first host machine. This network stack would perform encapsulation to create an outer header for the data packet. Within the outer header, the source address would be PA1, and the destination address would be PA2.

As noted above, VNET VMs and non-VNET VMs are currently unable to directly communicate with each other. In other words, although communication between VNET VMs and non-VNET VMs can occur, such communication currently requires at least one intermediate entity (e.g., a load balancer) within the VNET that has a public IP address. A non-VNET VM can communicate with a VNET VM through such an intermediate entity. For example, a non-VNET VM could send a data packet to the intermediate entity, which could then deliver the data packet to the VNET VM. However, a non-VNET VM is currently unable to send a data packet directly to the VNET VM (or vice versa). For example, a non-VNET VM is currently not permitted to send a data packet that is addressed to the VNET address (e.g., the customer address) of the VNET VM. This is at least partially because the address spaces of VNET VMs and non-VNET VMs can overlap. In other words, there can be some overlap between the VNET addresses (e.g., customer addresses) that are associated with VNET VMs and the physical IP addresses that are associated with non-VNET VMs.

There are, however, various reasons why it can be desirable for VNET VMs and non-VNET VMs to be able to directly communicate with each other. As one example, consider a deployment of non-VNET VMs that is being migrated to a VNET. Because it is desirable for the migration to occur with minimal interruption (ideally no interruption) of service, the migration can occur on a gradual basis. For example, just one VM (or a few VMs) can be migrated at a time. This means that, within the deployment, there could be (i) non-VNET VMs that have been migrated to the VNET, (ii) non-VNET VMs that have not yet been migrated to the VNET, and (iii) newly created VMs within the VNET (which may be referred to as "pure" VNET VMs). It can be desirable for (i) to be able to communicate with (ii), and also for (i) to be able to communicate with (iii).

One aspect of the present disclosure makes such communication possible by creating a hybrid state for a VM that was initially created as a non-VNET VM but has been migrated to a VNET. A VM that is operating in the hybrid state can communicate with other VMs in the VNET as well as non-VNET VMs that have not yet been migrated to the VNET.

<FIG> illustrates an example of a deployment within a cloud computing system that includes three non-VNET VMs: a first non-VNET VM 102a, a second non-VNET VM 102b, and a third non-VNET VM 102c. The first non-VNET VM 102a includes a physical address (PA) 104a, the second non-VNET VM 102b includes a PA 104b, and the third non-VNET VM 102c includes a PA 104c.

For purposes of the present example, it will be assumed that all of the non-VNET VMs 102a-c shown in <FIG> are able to communicate with each other. The non-VNET VMs 102a-c use the PAs 104a-c for such communication. For example, suppose that the first VM 102a sends a data packet to the second VM 102b. The source address of that data packet would be the PA 104a of the first VM 102a, and the destination address of that data packet would be the PA 104b of the second VM 102b.

Referring to both <FIG>, suppose that this deployment of non-VNET VMs 102a-c is going to be migrated to a VNET <NUM>. This migration can occur gradually (e.g., one VM at a time). Suppose that the first non-VNET VM 102a is migrated first. <FIG> illustrates a point in time at which the first non-VNET VM 102a has been migrated to the VNET <NUM>, but the second non-VNET VM 102b and the third non-VNET VM 102c have not yet been migrated to the VNET <NUM>. The first non-VNET VM 102a shown in <FIG> has been changed to a hybrid VM 102a' in <FIG>. The hybrid VM 102a' operates in a hybrid state that enables the hybrid VM 102a' to continue to communicate with the second non-VNET VM 102b and the third non-VNET VM 102c while those VMs 102b-c remain outside of the VNET <NUM>. The hybrid state also allows the hybrid VM 102a' to communicate with other VNET VMs (i.e., other VMs within the VNET <NUM>), such as the VNET VM 116a shown in <FIG>.

In the hybrid state, the hybrid VM 102a' continues to use its physical address (the PA 104a) to communicate with the second non-VNET VM 102b and the third non-VNET VM 102c. However, the hybrid VM 102a' is assigned a VNET address 118a to use for communication with other VMs within the VNET <NUM>, such as the VNET VM 116a shown in <FIG>. The VNET address 118a is unique within the context of the VNET <NUM>, but may not be unique outside of that context. In some implementations, the VNET address 118a may be referred to as a customer address (CA), because the VNET address 118a can be assigned by a customer of a cloud computing provider that administers the cloud computing system.

A VNET VM that is assigned to the VNET <NUM> when the VNET VM is initially created (instead of being created outside of the VNET <NUM> and then migrated to the VNET <NUM>) may be referred to as a "pure" VNET VM. The VNET VM 116a shown in <FIG> can be a pure VNET VM. If the VNET VM 116a is a "pure" VNET VM, the VNET VM 116a is not assigned a physical IP address. Instead, the VNET VM 116a can be assigned a VNET address 120a that is unique within the context of the VNET <NUM> but may not be unique outside of that context. The hybrid VM 102a' uses its VNET address 118a to communicate with the VNET VM 116a.

Thus, while operating in the hybrid state, the hybrid VM 102a' can be configured to send data packets to (and receive data packets from) the non-VNET VMs 102b-c. The hybrid VM 102a' can also be configured to send data packets to (and receive data packets from) other VMs within the VNET <NUM>, such as the VNET VM 116a.

<FIG> illustrate an example showing how a data packet <NUM> can be sent from the hybrid VM 102a' to the VNET VM 116a. For purposes of the present example, it will be assumed that the hybrid VM 102a' and the VNET VM 116a are running on different host machines. In particular, it will be assumed that the hybrid VM 102a' is running on a first host machine 130a, and the VNET VM 116a is running on a second host machine 130b. The first host machine 130a has a physical IP address (PA) 131a, and the second host machine 130b has a different PA 131b.

Referring briefly to <FIG>, the data packet <NUM> includes a header <NUM> and a payload <NUM>. As indicated above, when operating in the hybrid state, the hybrid VM 102a' uses a VNET address 118a for communication with other VMs within the VNET <NUM>. Thus, when the hybrid VM 102a' creates a data packet <NUM> to send to the VNET VM 116a, the hybrid VM 102a' uses its VNET address 118a as the source address <NUM> in the header <NUM> of the data packet <NUM>. The hybrid VM 102a' uses the VNET address 120a of the VNET VM 116a as the destination address <NUM> in the header <NUM> of the data packet <NUM>.

Referring again to <FIG>, after the hybrid VM 102a' creates the data packet <NUM>, the data packet <NUM> is delivered to a network stack <NUM> on the first host machine 130a. As will be explained in greater detail below, the network stack <NUM> can be configured to process data packets that are destined for non-VNET VMs (e.g., non-VNET VMs 102b-c) differently from data packets that are destined for VNET VMs (e.g., VNET VM 116a). Because the data packet <NUM> shown in <FIG> is destined for a VNET VM 116a, the network stack <NUM> performs encapsulation to form an encapsulated data packet <NUM>. As shown in <FIG>, the encapsulated data packet <NUM> includes a header <NUM> and a payload <NUM> that includes the data packet <NUM>. The header <NUM> of the encapsulated data packet <NUM> includes a source address <NUM> and a destination address <NUM>. The source address <NUM> in the header <NUM> may be referred to herein as a header source address <NUM>, and the destination address <NUM> in the header <NUM> may be referred to herein as a header destination address <NUM>. In the depicted example, the header source address <NUM> is the physical IP address of the first host machine 130a (i.e., PA 131a), and the header destination address <NUM> is the physical IP address of the second host machine 130b (i.e., PA 131b).

The network stack <NUM> on the first host machine 130a causes the encapsulated data packet <NUM> to be transmitted over a physical communication medium <NUM> to the second host machine 130b. A network stack <NUM> on the second host machine 130b receives the encapsulated data packet <NUM>, strips away the header <NUM>, and delivers the data packet <NUM> to the VNET VM 116a.

In the example shown in <FIG>, it is assumed that the hybrid VM 102a' and the VNET VM 116a are running on different host machines 130a-b. However, this is not necessary. In an alternative example, the hybrid VM 102a' can send a data packet to a VNET VM that is running on the same host machine as the hybrid VM 102a' (e.g., the first host machine 130a in the example shown in <FIG>). In this case, the data packet can still be delivered to the network stack <NUM>. However, it would not be necessary for encapsulation to be performed. The network stack <NUM> could simply deliver the data packet to the intended VNET VM on the first host machine 130a.

<FIG> illustrates an example of a data packet <NUM> that can be sent from the hybrid VM 102a' to a non-VNET VM, such as the second non-VNET VM 102b. As indicated above, in the hybrid state the hybrid VM 102a' continues to use its physical address (the PA 104a) to communicate with the non-VNET VMs 102b-c. Thus, in the present example, the source address <NUM> of the data packet <NUM> is the PA 104a of the hybrid VM 102a', and the destination address <NUM> of the data packet <NUM> is the PA 104b of the second non-VNET VM 102b.

The hybrid VM 102a' and the second non-VNET VM 102b can be running on the same host machine or on different host machines. If the hybrid VM 102a' and the second non-VNET VM 102b are running on different host machines, the data packet <NUM> can traverse the same basic path as the data packet <NUM> shown in <FIG> (e.g., from the hybrid VM 102a' to the network stack <NUM> on the first host machine 130a, across the communication medium <NUM> to a network stack on the host machine on which the second non-VNET VM 102b is running). For the sake of simplicity, however, those details are omitted from <FIG>.

As indicated above, the deployment of non-VNET VMs 102a-c shown in <FIG> can be migrated to a VNET <NUM>. <FIG> illustrates various entities within the VNET <NUM> after this migration has occurred. The first non-VNET VM 102a, second non-VNET VM 102b, and third non-VNET VM 102c shown in <FIG> have been transitioned to a first hybrid VM 102a', second hybrid VM 102b', and a third hybrid VM 102c' in <FIG>. The hybrid VMs 102a', 102b', 102c', maintain their physical addresses (PA 104a, PA 104b, PA 104c). In addition, the hybrid VMs 102a', 102b', 102c' are assigned VNET addresses. In particular, the first hybrid VM 102a' is assigned a first VNET address 118a, the second hybrid VM 102b' is assigned a second VNET address 118b, and the third hybrid VM 102c' is assigned a third VNET address 118c.

<FIG> also shows the VNET with a plurality of VNET VMs, including a first VNET VM 116a and a second VNET VM 116b. The first VNET VM 116a is assigned a first VNET address 120a, and the second VNET VM 116b is assigned a second VNET address 120b. The VNET VMs 116a, 116b may be "pure" VNET VMs. In some embodiments, the VNET VMs 116a, 116b may be created after all of the non-VNET VMs 102a, 102b, 102c have been migrated to the VNET <NUM> and transitioned to hybrid VMs 102a', 102b', 102c'.

To make it possible for a particular VM to operate in the hybrid state, the host machine on which the VM is running can include a network stack that is configured to process data packets that are destined for non-VNET VMs differently from data packets that are destined for VNET VMs. To facilitate this, one or more rule sets can be configured in the network stack of the host machine on which the VM is running. In this context, the term "rule" can refer to one or more actions that are performed in response to one or more conditions being satisfied. The term "rule set" can refer to a single rule or a plurality of rules. In some implementations, the network stack can include at least two different rule sets: a first rule set for processing data packets that are sent to (or received from) VNET VMs, and a second rule set for processing data packets that are sent to (or received from) non-VNET VMs.

<FIG> illustrates an example of a network stack <NUM> that is configured to implement various packet processing rule sets. The network stack <NUM> is included on a host machine <NUM>. A hybrid VM <NUM> is running on the host machine <NUM>. The hybrid VM <NUM> belongs to a VNET and therefore has a VNET address <NUM>. The hybrid VM <NUM> uses the VNET address <NUM> for communicating with other VMs within the VNET. The hybrid VM <NUM> also includes a physical IP address (PA) <NUM> that it uses to communicate with non-VNET VMs.

The hybrid VM <NUM> creates a data packet <NUM> to be sent to another VM, which can be a non-VNET VM or a VNET VM. The network stack <NUM> receives the data packet <NUM> from the hybrid VM <NUM>. The network stack <NUM> includes a component that may be referred to herein as a packet classifier <NUM>. The packet classifier <NUM> is configured to determine whether the data packet <NUM> is going to be sent to a non-VNET VM or to a VNET VM. To make this determination, the packet classifier <NUM> evaluates the destination address that is included in the header of the data packet <NUM>. More specifically, the packet classifier <NUM> compares the destination address of the data packet <NUM> to an address space <NUM> corresponding to the VNET to which the hybrid VM <NUM> belongs, and selects a rule set for processing the data packet <NUM> based on the comparison.

More specifically, in the depicted example, the VNET addresses that are used for VNET VMs are distinct from the physical IP addresses that are used for non-VNET VMs. In other words, the address space <NUM> of VNET VMs is distinct from and does not overlap with the address space <NUM> of non-VNET VMs. Thus, by evaluating the destination address that is included in the header of the data packet <NUM>, the packet classifier <NUM> is able to determine whether the data packet <NUM> is being sent to a VNET VM or to a non-VNET VM. If the destination address of the data packet <NUM> falls within the address space <NUM> of VNET VMs, the packet classifier <NUM> determines that the data packet <NUM> is being sent to a VNET VM. If, however, the destination address of the data packet <NUM> falls within the address space <NUM> of non-VNET VMs, the packet classifier <NUM> determines that the data packet <NUM> is being sent to a non-VNET VM.

As noted above, data packets that are being sent to VNET VMs are processed differently than data packets that are being sent to non-VNET VMs. The network stack <NUM> is shown with a component that is configured to process data packets that are being sent to VNET VMs. This component may be referred to as a VNET packet processor 270a. The network stack <NUM> is also shown with a component that is configured to process data packets that are being sent to non-VNET VMs. This component may be referred to as a non-VNET packet processor 270b. The VNET packet processor 270a processes data packets in accordance with a rule set that may be referred to as a VNET packet processing rule set 272a. The non-VNET packet processor 270b processes data packets in accordance with a rule set that may be referred to as a non-VNET packet processing rule set 272b.

If the packet classifier <NUM> determines that the destination address of the data packet <NUM> falls within the VNET address space <NUM>, then the VNET packet processor 270a processes the data packet <NUM> based on the VNET packet processing rule set 272a. If, however, the packet classifier <NUM> determines that the destination address of the data packet <NUM> falls within the non-VNET address space <NUM>, then the non-VNET packet processor 270b processes the data packet <NUM> based on the non-VNET packet processing rule set 272b.

In some implementations, the VNET packet processing rule set 272a can include one or more rules <NUM> specifying how encapsulation should be performed on the data packet <NUM>. These rules <NUM> may be referred to herein as encapsulation rules <NUM>. Thus, if the packet classifier <NUM> determines that the destination address of the data packet <NUM> falls within the VNET address space <NUM> and the data packet <NUM> is processed in accordance with the VNET packet processing rule set 272a, the data packet <NUM> can be encapsulated in accordance with the encapsulation rules <NUM>. This results in the creation of an encapsulated data packet <NUM>. The encapsulated data packet <NUM> can be similar to the encapsulated data packet <NUM> shown in <FIG>. For example, the encapsulated data packet <NUM> can include a header that includes a source address and a destination address. The source address can be the physical IP address of the host machine <NUM> on which the hybrid VM <NUM> is running. The destination address can be the physical IP address of the host machine on which the destination VM (i.e., the VM to which the data packet <NUM> is being sent) is running. The data packet <NUM> created by the hybrid VM <NUM> can be included in the payload of the encapsulated data packet <NUM>.

In some implementations, the non-VNET packet processing rule set 272b does not include any encapsulation rules. In other words, the non-VNET packet processing rule set 272b can permit a data packet <NUM> to be transmitted to a destination VM without encapsulation. Thus, if the packet classifier <NUM> determines that the destination address of the data packet <NUM> falls within the non-VNET address space <NUM>, the unencapsulated data packet <NUM> can simply be sent to the destination VM.

<FIG> illustrates an example of a data packet 228a that can be sent by the hybrid VM <NUM> and processed by the network stack <NUM> in the example shown in <FIG>. The data packet 228a shown in <FIG> is destined for a VNET VM (i.e., for another VM within the VNET to which the hybrid VM <NUM> belongs). The data packet 228a includes a header 244a and a payload 250a. The header 244a includes a source address 232a and a destination address 234a.

The source address 232a corresponds to the hybrid VM <NUM>. As discussed above, the hybrid VM <NUM> includes two different addresses, a PA <NUM> and a VNET address <NUM>. Because the data packet 228a is being sent to a VNET VM, the source address 232a includes the VNET address <NUM> of the hybrid VM <NUM>.

The destination address 234a corresponds to the destination VM. Because the data packet 228a shown in <FIG> is destined for a VNET VM, the destination address 234a includes a VNET address <NUM> corresponding to the VNET VM.

The data packet 228a can be processed by the network stack <NUM> in the following manner. The packet classifier <NUM> can determine, based on the destination address 234a, that the destination VM is part of the same VNET to which the hybrid VM <NUM> belongs. More specifically, the packet classifier <NUM> can compare the destination address 234a (i.e., the VNET address <NUM>) to the VNET address space <NUM> and determine that the destination address 234a is included within the VNET address space <NUM>.

Based on determining that the destination VM is part of the same VNET to which the hybrid VM <NUM> belongs, the data packet 228a can be processed by the VNET packet processor 270a. More specifically, the VNET packet processor 270a can process the data packet 228a in accordance with the VNET packet processing rule set 272a. This can include encapsulating the data packet 228a in accordance with one or more encapsulation rules <NUM> to form an encapsulated data packet <NUM>.

<FIG> illustrates another example of a data packet 228b that can be sent by the hybrid VM <NUM> and processed by the network stack <NUM> in the example shown in <FIG>. The data packet 228b shown in <FIG> is destined for a non-VNET VM (i.e., for a VM that is not part of the VNET to which the hybrid VM <NUM> belongs). The data packet 228b includes a header 244b and a payload 250b. The header 244b includes a source address 232b and a destination address 234b.

The source address 232b corresponds to the hybrid VM <NUM>. Because the data packet 228b is being sent to a non-VNET VM, the source address 232b is the PA <NUM> of the hybrid VM <NUM>.

The destination address 234b corresponds to the destination VM. Because the data packet 228b shown in <FIG> is destined for a non-VNET VM, the destination address 234b includes a PA <NUM> corresponding to the non-VNET VM.

The data packet 228b can be processed by the network stack <NUM> in the following manner. The packet classifier <NUM> can determine, based on the destination address 234b, that the destination VM is not part of the same VNET to which the hybrid VM <NUM> belongs. For example, the packet classifier <NUM> can compare the destination address 234b (i.e., the PA <NUM>) to the VNET address space <NUM> and determine that the destination address 234b is not included within the VNET address space <NUM>. As another example, the packet classifier <NUM> can compare the destination address 234b to the non-VNET address space <NUM> and determine that the destination address 234b is included within the non-VNET address space <NUM>.

Based on determining that the destination VM is not part of the same VNET to which the hybrid VM <NUM> belongs, the data packet 228b can be processed by the non-VNET packet processor 270b. More specifically, the non-VNET packet processor 270b can process the data packet 228b in accordance with the non-VNET packet processing rule set 272b. Thus, the data packet 228b can be transmitted to the destination VM without encapsulation.

<FIG> illustrates an example of a method <NUM> for facilitating communication between non-VNET VMs 102a-c and VNET VMs 116a-b in accordance with the present disclosure. The method <NUM> can be implemented by one or more system-level entities within a cloud computing system, such as a fabricator and/or a data center controller. The method <NUM> will be described in relation to the examples shown in <FIG>and <FIG>.

The method <NUM> includes creating <NUM> a VNET <NUM> and migrating <NUM> a non-VNET VM 102a to the VNET <NUM>. The action of migrating <NUM> the non-VNET VM 102a to the VNET <NUM> can include assigning <NUM> a VNET address 118a to the VM 102a. The VNET address 118a enables the VM 102a to communicate with other VMs in the VNET <NUM> (e.g., the VNET VM 116a).

The method <NUM> also includes causing <NUM> the non-VNET VM 102a to transition to a hybrid VM 102a' that operates in a hybrid state. Advantageously, the non-VNET VM 102a/hybrid VM 102a' does not lose connectivity with other non-VNET VMs 102b-c during this transition. As discussed above, the non-VNET VM 102a is assigned a physical IP address (PA) 104a. In the hybrid state, the hybrid VM 102a' continues to use the PA 104a to communicate with the other non-VNET VMs 102b-c.

The method <NUM> also includes providing <NUM> a network stack <NUM> with a VNET packet processing rule set 272a for processing data packets that are being sent to VNET VMs, and a non-VNET packet processing rule set 272b for processing data packets that are being sent to non-VNET VMs. As described above, the VNET packet processing rule set 272a can be configured to process data packets corresponding to a VNET address space <NUM> that has been defined for the VNET <NUM>. The non-VNET packet processing rule set 272b can be configured to process data packets corresponding to a non-VNET address space <NUM> that is distinct from and does not overlap with the VNET address space <NUM>.

<FIG> illustrates another example of a method <NUM> for facilitating communication between non-VNET VMs 102a-c and VNET VMs 116a-b in accordance with the present disclosure. The method <NUM> can be implemented by one or more system-level entities within a cloud computing system, such as a fabricator and/or a data center controller. The method <NUM> will be described in relation to the examples shown in <FIG>and <FIG>.

The method <NUM> includes migrating <NUM> a plurality of non-VNET VMs 102a-c to a VNET <NUM> and causing the plurality of non-VNET VMs 102a-c to transition to hybrid VMs 102a'-c' that operate in a hybrid state. Some or all of the actions described above in connection with the method <NUM> shown in <FIG> may be performed for each of the non-VNET VMs 102a-c.

The method <NUM> also includes deploying <NUM> a plurality of "pure" VNET VMs 116a-b within the VNET <NUM>. As discussed above, the pure VNET VMs 116a-b can be assigned to the VNET <NUM> when the VNET VMs 116a-b are initially created (instead of being created outside of the VNET <NUM> and then migrated to the VNET <NUM>). In some embodiments, the pure VNET VMs 116a-b can be deployed after the plurality of non-VNET VMs 102a-c have been migrated to the VNET <NUM> and transitioned to hybrid VMs 102a'-c'.

<FIG> illustrates another example of a method <NUM> for facilitating communication between non-VNET VMs 102a-c and VNET VMs 116a-b in accordance with the present disclosure. The method <NUM> can be implemented by a network stack <NUM> running on a host machine <NUM> within a cloud computing system. The method <NUM> will be described in relation to the examples shown in <FIG> and <FIG>.

As a prerequisite to the method <NUM>, a VNET address space <NUM> and a non-VNET address space <NUM> can be defined. The VNET address space <NUM> and the non-VNET address space <NUM> can be defined so that they are distinct from and do not overlap with one another.

The method <NUM> includes receiving <NUM> a data packet <NUM>. The data packet <NUM> includes, among other things, a destination address. The destination address can be used to determine <NUM> whether the data packet <NUM> corresponds to the VNET address space <NUM>. For example, the destination address of the data packet <NUM> can be compared to the VNET address space <NUM> and/or to the non-VNET address space <NUM>.

If it is determined <NUM> that the data packet <NUM> corresponds to the VNET address space <NUM>, then the VNET packet processing rule set 272a can be selected <NUM> for processing the data packet <NUM>. If, however, it is determined <NUM> that the data packet <NUM> does not correspond to the VNET address space <NUM>, then the non-VNET packet processing rule set 272b can be selected <NUM> for processing the data packet <NUM>.

<FIG> illustrates an example of a method <NUM> that can be implemented by a network stack <NUM> on a host machine <NUM> that includes a hybrid VM <NUM>. The method <NUM> will be described in relation to the example shown in <FIG>.

The method <NUM> includes receiving <NUM> a first data packet 228a and determining <NUM>, based on the destination address 234a of the first data packet 228a, that the destination VM is part of the VNET to which the hybrid VM <NUM> belongs. In other words, the method <NUM> includes determining <NUM> that the intended recipient of the first data packet 228a is a VNET VM. In response to determining <NUM> that the destination VM is a VNET VM, the method <NUM> also includes processing <NUM> the first data packet 228a in accordance with a VNET packet processing rule set 272a.

The method <NUM> also includes receiving <NUM> a second data packet 228b and determining <NUM>, based on the destination address 234b of the second data packet 228b, that the destination VM is not part of the VNET to which the hybrid VM <NUM> belongs. In other words, the method <NUM> includes determining <NUM> that the intended recipient of the second data packet 228b is a non-VNET VM. In response to determining <NUM> that the destination VM is a non-VNET VM, the method <NUM> also includes processing <NUM> the second data packet 228b in accordance with a non-VNET packet processing rule set 272b.

<FIG> illustrates an example of certain components that can be utilized in a cloud computing system <NUM>. Broadly speaking, the cloud computing system <NUM> includes two sections, a front end and a back end, that are in electronic communication with one another via one or more computer networks <NUM> (typically including the Internet). The front end of the cloud computing system <NUM> includes an interface <NUM> that users encounter through a client device <NUM>. The back end of the cloud computing system <NUM> includes the resources that deliver cloud-computing services.

More specifically, the back end of the cloud computing system <NUM> includes a plurality of data centers <NUM>. A particular data center <NUM> includes a plurality of host machines 721a-n, including a first host machine 721a and an Nth host machine 721n. The host machines 721a-n can alternatively be referred to as servers. A data center management controller <NUM> performs management operations with respect to the host machines 721a-n. A load balancer <NUM> distributes requests and workloads over the host machines 721a-n to prevent a situation where a single host machine becomes overwhelmed and also to maximize available capacity and performance of the resources in the data center <NUM>. A plurality of routers/switches <NUM> support data traffic between the host machines 721a-n, and also between the data center <NUM> and external resources and users via the network(s) <NUM>.

The host machines 721a-n can be traditional standalone computing devices and/or they can be configured as individual blades in a rack of many server devices. The host machines 721a-n each have one or more input/output (I/O) connectors. In <FIG>, the first host machine 721a is shown with an I/O connector 729a, and the Nth host machine 721n is shown with an I/O connector 729n. The I/O connectors 729a-n enable the host machines 721a-n to be placed in electronic communication with each other and with other computing entities in the cloud computing system <NUM>, such as the data center management controller <NUM>.

The host machines 721a-n each include one or more processors, which may be referred to herein as host processors. In <FIG>, the first host machine 721a is shown with a set of one or more host processors 731a, and the Nth host machine 721n is shown with a set of one or more host processors 731n. The host processors 731a-n can be general purpose single- or multi-chip microprocessors (e.g., an Advanced RISC (Reduced Instruction Set Computer) Machine (ARM)), special purpose microprocessors (e.g., a digital signal processor (DSP)), microcontrollers, programmable gate arrays, and so forth, including combinations thereof. The host processors 731a-n can alternatively be referred to as central processing units (CPUs).

The host machines 721a-n each include storage (e.g., hard disk drives) and memory (e.g., RAM) that can be accessed and used by the host processors and VMs. In <FIG>, the first host machine 721a is shown with memory 771a-b and storage 773a-b, and the Nth host machine 721n is shown with memory 775a-b and storage 777a-b.

The host machines 721a-n each include an operating system (OS), which may be referred to herein as a host operating system (or host OS). In <FIG>, the first host machine 721a is shown with a host operating system 733a, and the Nth host machine 721n is shown with a host operating system 733n. The host operating systems 733a-n are executed by the host processors 731a-n, and they support multiple virtual machines. In <FIG>, the first host machine 721a is shown with a plurality of VMs including a first VM (VM1) 735a, a second VM (VM2) 735b, and an Nth VM (VMn) 735n. The Nth host machine 721n is also shown with a plurality of VMs including a first VM (VM1) 737a, a second VM (VM2) 737b, and an Nth VM (VMn) 737n.

Each VM can run its own operating system. <FIG> shows VM1 735a on the first host machine 721a running VM OS1 739a, VM2 735b on the first host machine 721a running VM OS2 739b, and VMn 735n on the first host machine 721a running VM OSn 739n. Similarly, <FIG> shows VM1 737a on the Nth host machine 721n running VM OS1 741a, VM2 737b on the Nth host machine 721n running VM OS2 741b, and VMn 737n on the Nth host machine 721n running VM OSn 741n.

In some implementations, the various VM operating systems running on a particular host machine can all be the same operating system. Alternatively, the various VM operating systems running on a particular host machine can include different operating systems. The VM operating systems can be, for example, different versions of the same operating system (e.g., different VMs can be running both current and legacy versions of the same operating system). Alternatively, the VM operating systems on a particular host machine can be provided by different manufacturers.

One or more applications can be running on each VM. <FIG> shows VM1 735a on the first host machine 721a running App1 751a and App2 751b, VM2 735b on the first host machine 721a running App1 753a and App2 753b, and VMn 735n on the first host machine 721a running App1 755a and App2 755b. Similarly, <FIG> shows VM1 737a on the Nth host machine 721n running App1 757a and App2 757b, VM2 737b on the Nth host machine 721n running App1 759a and App2 759b, and VMn 737n on the Nth host machine 721n running App1 761a and App2 761b.

The techniques disclosed herein can be implemented in hardware, software, firmware, or any combination thereof, unless specifically described as being implemented in a specific manner. Any features described as modules, components, or the like can also be implemented together in an integrated logic device or separately as discrete but interoperable logic devices. If implemented in software, the techniques can be realized at least in part by a non-transitory computer-readable medium having computer-executable instructions stored thereon that, when executed by at least one processor, perform some or all of the steps, operations, actions, or other functionality disclosed herein. The instructions can be organized into routines, programs, objects, components, data structures, etc., which can perform particular tasks and/or implement particular data types, and which can be combined or distributed as desired in various embodiments.

The term "processor" can refer to a general purpose single- or multi-chip microprocessor (e.g., an Advanced RISC (Reduced Instruction Set Computer) Machine (ARM)), a special purpose microprocessor (e.g., a digital signal processor (DSP)), a microcontroller, a programmable gate array, or the like. A processor can be a central processing unit (CPU). In some embodiments, a combination of processors (e.g., an ARM and DSP) could be used to implement some or all of the techniques disclosed herein.

The term "memory" can refer to any electronic component capable of storing electronic information. For example, memory can be embodied as random access memory (RAM), read-only memory (ROM), magnetic disk storage media, optical storage media, flash memory devices in RAM, on-board memory included with a processor, erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM) memory, registers, and so forth, including combinations thereof.

The steps, operations, and/or actions of the methods described herein can be interchanged with one another without departing from the scope of the claims. In other words, unless a specific order of steps, operations, and/or actions is required for proper functioning of the method that is being described, the order and/or use of specific steps, operations, and/or actions can be modified without departing from the scope of the claims.

The term "determining" (and grammatical variants thereof) can encompass a wide variety of actions. For example, "determining" can include calculating, computing, processing, deriving, investigating, looking up (e.g., looking up in a table, a database or another data structure), ascertaining and the like.

The terms "comprising," "including," and "having" are intended to be inclusive and mean that there can be additional elements other than the listed elements. For example, any element or feature described in relation to an embodiment herein can be combinable with any element or feature of any other embodiment described herein, where compatible.

Claim 1:
A system, comprising:
one or more processors;
memory in electronic communication with the one or more processors; and
instructions stored in the memory, the instructions being executable by the one or more processors to:
create a virtual network, VNET, (<NUM>);
migrate a non-VNET virtual machine to the VNET (<NUM>), wherein the non-VNET virtual machine is one of a plurality of non-VNET virtual machines that are deployed in a cloud computing system and the non-VNET virtual machine is associated a physical internet protocol IP, address;
cause the non-VNET virtual machine to transition to a hybrid virtual machine that operates in a hybrid state (<NUM>), wherein the hybrid virtual machine has both the physical internet protocol, IP,
address and a VNET address, wherein the hybrid virtual machine is configured to use the physical IP address to communicate with other non-VNET virtual machines, and wherein the hybrid virtual machine is configured to use the VNET address to communicate with other VNET virtual machines; and
provide a network stack with a first packet processing rule set and a second packet processing rule set (<NUM>), wherein the first packet processing rule set is configured to process first data packets from the hybrid virtual machine corresponding to a first address space that has been defined for the virtual network, wherein a destination address of the first data packet indicates that a destination virtual machine of the first data packet is a virtual machine that is part of the VNET to which the hybrid virtual machine belongs, and wherein the second packet processing rule set is configured to process second data packets corresponding to a second address space that is distinct from the first address space and that does not overlap with the first address space, wherein a destination address of the second data packet indicates that a destination virtual machine of the second data packet is a non-VNET virtual machine that is not part of the VNET to which the hybrid virtual machine belongs, wherein the first packet processing rule set comprises encapsulation rules that specify how encapsulation should be performed on the first data packets; and
the second packet processing rule set permits the second data packets to be transmitted without encapsulation;
wherein the network stack is included within a first host machine; and
the non-VNET virtual machine runs on the first host machine.