Patent Description:
Some computing systems may be implemented using so called virtual machine technology. In particular, a virtual machine runs on a host machine where the host machine's physical resources are used to back the virtual resources of the virtual machine. Thus, for example, processing power on a physical machine can be used to implement virtual processors on a virtual machine. Memory on a physical machine may be used to implement memory on a virtual machine. Disk resources on a host physical machine may be used to implement virtual storage on a virtual machine.

Traditional virtual machines have used block based storage, addressed using Logical Block Addressing (LBA) for their persistent data needs. For example, such block base storage may include various drives such as hard drives and solid state drives. These traditional virtual devices have aimed to emulate traditional block based devices. Now that there is a new mode of underlying storage, byte addressable persistent memory. It would be useful to make full use of the new byte addressable persistent memory in virtual machine environments.

<CIT> relates to a computer having a processor and a byte-addressable nonvolatile read-write main memory. The memory is partitioned into plural regions, each region having at least one defined operational property. At least one of the regions is a metadata region to store plural data sets. Each data set specifies a location in memory, and the at least one operational property, of a corresponding one of the regions. <NPL>, relates to extending commodity hypervisors to virtualize persistent memory (PM). A system, namely VPM, provides both full-virtualization as well as a para-virtualization interface that provide persistence hints to the hypervisor is implemented. By leveraging the fact that PM has similar characteristics with DRAM except for persistence, VPM supports transparent data migration by leveraging the two-dimensional paging (e.g., EPT) to adjust the mapping between guest PM to host physical memory (DRAM or PM).

It is the object of the present invention to provide an enhanced method for making data on a persistent memory device available to a virtual machine.

Traditional virtual machines have used block based storage, addressed using Logical Block Addressing (LBA) for their persistent data needs. These traditional virtual devices have aimed to emulate traditional block based devices, such as hard drives and/or solid state drives. Now that there is a new mode of underlying storage, byte addressable persistent memory, such as phase change memory, these types of devices are being emulated in virtual machines to take advantage of the high-speed and persistent nature of the storage.

Virtual persistent memory can be implemented as a virtual device that emulates physical persistent memory inside a virtual machine.

Some embodiments illustrated herein use files on an underlying physical storage substrate as a repository for the data presented to the virtual machine. These files provide benefits to the host administrator, just as the virtual disk files used for virtual block devices:.

For example, host administrators can move these files to different physical persistent memory devices on the same or different machines using standard copy mechanisms.

Alternatively or additionally, host administrators can create a template file and copy that template file to multiple virtual machines using standard copy mechanisms.

Alternatively or additionally, host administrators can convert between file formats that support block based access and those that support virtual persistent memory access.

Alternatively or additionally, host administrators can present the contents of a virtual persistent memory device as a block device on the host (e.g., a loopback mount).

Alternatively or additionally, host administrators can sparsely allocate these files. Data on the underlying host substrate is only consumed when necessary and the underlying physical storage substrate can be overprovisioned.

Alternatively or additionally, hosts can react to bad blocks in the underlying storage substrate and prevent guest access to these bad blocks.

Alternatively or additionally, host administrators can create trees of differencing disks that are presented as virtual persistent memory devices.

Additional details are now illustrated. Reference is now made to <FIG> which illustrates a host <NUM>. The host <NUM> is a physical machine which includes physical hardware and hosts one more virtual machines such as the virtual machine <NUM>-<NUM>. Note that while a single virtual machine <NUM>-<NUM> is illustrated in the example, it should be appreciated that typically, a host will host multiple virtual machines. Indeed, many of the advantages of virtual machine technology are achieved when a host machine uses the resources of the host machine to implement multiple virtual machines. In the example illustrated in <FIG>, the host <NUM> includes block based storage <NUM>, and byte addressable persistent memory <NUM>.

The block base storage <NUM> may be one or more block based devices, such as hard disks, solid state disks, and/or other block addressable storage devices. The block based storage <NUM> can be used to back certain elements of the virtual machine <NUM>-<NUM>. That is, the block based storage <NUM> can be used as the physical hardware to implement one or more virtual devices. For example, block based storage <NUM>-<NUM> on the virtual machine <NUM>-<NUM> can be backed by the block based storage <NUM>. Alternatively or additionally, the virtual machine <NUM>-<NUM> may have byte addressable storage <NUM>-<NUM>, such as memory type storage, which is backed by the blocked based storage <NUM>.

Additionally or alternatively, the virtual machine <NUM>-<NUM> may have block based storage <NUM>-<NUM> that is backed by the byte addressable persistent memory <NUM>. The virtual machine <NUM>-<NUM> may have byte addressable storage <NUM>-<NUM> that is backed by the byte addressable persistent memory <NUM>.

The byte addressable persistent memory <NUM> includes a Block Translation Table (BTT) <NUM>, which provides sector atomicity support for byte addressable storage. In particular, the byte addressable persistent memory <NUM> is able to be addressed at byte granularity. However, as noted above, the byte addressable persistent memory <NUM> may be used to back block based storage. Thus, it is possible for a block based access to fail where the block contains some valid data and some invalid data. The BTT <NUM> provides sector atomicity support so that applications that rely on block based accesses can function appropriately. The BTT <NUM> is an indirection table that re-maps blocks on a volume. It functions as a file system that only provides atomic sector updates.

Embodiments further implement a file translation table <NUM> which allows at least portions of the byte addressable persistent memory <NUM> to be read by the host <NUM> as a file such that the portions can be read and copied by the host <NUM> as if they were files. In particular, the file translation table <NUM> illustrated translates block offsets to virtual byte offsets and virtual byte offsets to file offsets needed to read and write data on the byte addressable persistent memory <NUM> as a file. Note that in some embodiments, a data structure may be maintained to translate virtual byte offsets to file offsets, and reuses the mapping in the BTT <NUM> for translating block offsets to virtual byte offsets.

These files are also specific to the persistent memory format of the byte addressable persistent memory <NUM>. A file used to present virtual persistent memory to a virtual machine cannot ordinarily be used to similarly present block based storage to a virtual machine. OS storage stacks on persistent memory devices support a compatibility mode to allow atomic block-sized transactions on these devices. They achieve this by putting down an industry standard BTT <NUM>, which describes logical block offsets <-> physical byte offsets. Ordinarily, this introduces a physical incompatibility between the persistent memory and block based storage stacks. However, using the file translation table described above, this can be overcome.

Embodiments illustrated herein take a file that was presented to a virtual machine and convert it to a format that can be presented to virtual machines as block storage, and vice versa using mapping such as that shown in the file translation table <NUM>. Embodiments thus alleviate this physical incompatibility.

Further, some embodiments are implemented securely by using a locked down usermode process to convert the contents of the file. Alternatively or additionally, embodiments may be implemented where this locked down process is used to convert existing block file format types among themselves, such as between dynamic and fixed files, and between VHD (Virtual Hard Disc) and VHDX (Virtual Hard Disk Extended) files. The locked down usermode process is an additional security measure aimed at protecting against attacks that attempt to leverage security flaws in the parsing code by presenting intentionally malformed file contents to the parsing process. These attacks typically attempt to run arbitrary code in the security context of the target process. If the process is locked down, however, even if an attacker manages to trigger the arbitrary code execution, such code would not be able to gain access to other resources on the host, or to produce adverse effects on the host. The parsing process is typically locked down by running it within a security context that has minimal privileges on the host (e.g. cannot access any other file beside the file being converted, cannot execute operations requiring admin privileges, cannot access network resources, etc.).

Embodiments may alternatively or additionally allow the host to loopback mount one of these files that has been formatted as a virtual persistent memory compatibility file. This allows host administrators to mount and read data off the file, even if it was presented as a virtual persistent memory device to a virtual machine.

The following now illustrates one example conversion process.

Some of the compatibility problems between the virtual hard disk persistent memory (VHDPMEM) file and a virtual hard disk block capable file (VHD or VHDX) are:.

Therefore, embodiments may implement the following, or a similar, conversion routine to convert VHDPMEM -> VHD(X):.

Embodiments may implement the following conversion routine to convert VHD(X) -> VHDPMEM:.

As discussed previously, some embodiments may implement a loopback mount of a filesystem. In particular, a loopback mounted filesystem is a virtual filesystem that provides an alternate path to an existing filesystem. Some embodiments illustrated herein facilitate the ability to implement a loopback mount using byte addressable persistent memory. The following illustrates one example of how loopback mount is implemented in this fashion.

Loopback mount on some operating system, such as Windows® available from Microsoft® Corporation of Redmond, Washington, requires a storage miniport driver <NUM> usable by a hypervisor that translates SCSI (or other appropriate protocol) requests directed to the virtual block addressable storage (e.g., block addressable storage <NUM>-<NUM>) to those understood by the underlying adapter for the hardware, e.g., in this example, the byte addressable persistent memory <NUM>. In Windows this is VHDMP, the VHD MiniPort which is a VHD parser and dependency provider. The example illustrated below uses the VHDMP, but is should be appreciated that in other embodiments, other appropriate miniport or other drivers may be alternatively or additionally used. Some embodiments include a filter <NUM> that sits on top of miniport driver <NUM> (or similar entity in other systems) that understands and performs translations of logical block addresses to actual virtual logical block devices (e.g., block addressable storage <NUM>-<NUM>), based on the BTT <NUM> data. Using these translations, the filter <NUM> can be used to enable loopback mount using the operating system's native capabilities for doing so, by resolving any addressing incompatibilities.

The following illustrates details with respect to sparse allocation, thin provisioning, and/or host memory protections. Thin provisioning (i.e., overprovisioning) and sparse allocation can be used in virtualization systems to over-allocate physical resources such that the virtualization system appears to have more physical resources than it actually has. That is, the virtualization system that is thinly provisioned will have more virtual resources at a point in time than it has actual physical resources. However, not all virtual resources will be used at the same time, such that the physical resources are sufficient to back all of the virtual resources.

The VHDPMEM file format, and other formats in other systems, natively supports having regions that are not yet allocated and assigned. Many hypervisors, such as Hyper-V available from Microsoft® Corporation of Redmond, Washington, support memory intercepts. That is, a hypervisor can intercept memory operations from a virtual machine operating system to ensure that that operations are handled appropriately on the underlying hardware of the host. With this foundation, embodiments can implement overprovisioning mechanisms. The following illustrates an example:.

A similar mechanism can be used to unmap sub-regions of the file when the host determines that memory errors have occurred.

Note that the host takes care to not map known-bad regions into the guest VM <NUM>-<NUM>.

Referring now to <FIG>, the following illustrates how embodiments implement differencing functionality using virtual persistent memory disks. One common virtual storage technique is to create trees of differencing disks. A differencing disk has a parent and contains only data that is different from the parent. Virtual persistent memory devices implement similar functionality by using memory intercepts on write. <FIG> illustrates a tree <NUM>-<NUM> in a disk <NUM>-<NUM> of a first byte addressable persistent memory <NUM>-<NUM> and a tree <NUM>-<NUM> in a disk <NUM>-<NUM> of a second byte addressable persistent memory <NUM>-<NUM>.

At VM power on time, the host <NUM> will traverse the tree of differencing disks to find the correct memory of a byte addressable persistent memory <NUM> to map into the guest VM <NUM>-<NUM> at any specific location. These maps, referred to generally as <NUM>, could be of various granularities (i.e. embodiments may take only a few pages from one virtual persistent memory device (e.g., byte addressable persistent memory <NUM>-<NUM>) and then many contiguous runs of pages from another virtual persistent memory device (e.g., byte addressable persistent memory <NUM>-<NUM>).

The host <NUM> adds one or more memory intercept triggers so that data could be updated on write into the correct location. Note that writes always go to the leaf of the tree. Parent nodes are left unchanged as they will be needed for other differencing disk operations.

Note that while <FIG> illustrates differencing disks on the same machine, it should be appreciated that embodiments may be implemented in a cloud environment with multiple machines. Thus, various embodiments can read and write data as files from and to different persistent memory storage devices on the same or different machines.

A number of methods and method acts are illustrated herein.

Referring now to <FIG>, a method <NUM> is illustrated. The method <NUM> may be practiced in a virtual machine environment. The virtual machine environment includes a persistent memory storage device. The persistent memory storage device has the ability to appear as a memory device having available memory to a virtual machine on a host and as a file to the host. The method includes acts for copying data stored in the persistent memory storage device for a virtual machine.

The method <NUM> includes the host reading data from the persistent memory storage device as file data (act <NUM>). For example, the host <NUM> may include a hypervisor that is configured to read data on the byte addressable persistent memory <NUM>. The host <NUM> may read data on the byte addressable persistent memory <NUM> as file data by using a translation such as the translation table <NUM>.

The method <NUM> further includes the host writing the data from the persistent memory storage device as file data (act <NUM>).

The method <NUM> may further include a virtual machine reading the written data as data stored in a persistent memory device. For example, the virtual machine <NUM>-<NUM> may read the data written as file data in a native format for the byte addressable persistent memory <NUM>.

The method <NUM> may further include a virtual machine reading the written data as block addressable data. Thus for example, as illustrated in figure one, the virtual machine <NUM>-<NUM> may be able to read data stored on the byte addressable persistent memory <NUM> as block addressable data. This allows the virtual machine <NUM>-<NUM> to implement the block based storage <NUM>-<NUM>.

The method <NUM> may be practiced where the host reads the data from the persistent memory storage device as file data as a result of translating at least one of a block offset or a virtual byte offset to a file offset. As noted previously herein, this may be accomplished using a file translation data structure such as the file translation table <NUM> illustrated above.

The method <NUM> may be practiced where the host reading data from the persistent memory storage device as file data and the host writing the data from the persistent memory storage device as file data is performed by a locked down usermode process. For example, as illustrated above, and locked down process <NUM> may be used to perform various will read and write actions.

Referring now to figure four, a method <NUM> is illustrated. The method <NUM> may be practiced in a virtual machine environment. The virtual machine environment includes a persistent memory storage device. The persistent memory storage device has the ability to appear as a memory device having available memory to a virtual machine on a host and a file to the host. The method includes acts for distributing common data across one or more virtual machines.

The method <NUM> includes the host accessing file data (act <NUM>).

The method <NUM> further includes the host writing the file data to one or more persistent memory storage devices as file data. This could be used, for example, for differencing disks as illustrated in the example shown in <FIG>. Alternatively or additionally, this may be used to copy file data when one disk is implemented per virtual machine.

The method <NUM> may further include a virtual machine reading the written data as data in a persistent memory device.

The method <NUM> may further include a virtual machine reading the written data as block addressable data.

The method <NUM> may further include comprising mapping at least one of a block offset or a virtual byte offset to a file offset as a result of the host writing the data from the persistent memory storage device as file data. For example, embodiments may generate portions of the file translation table <NUM> illustrated above.

The method <NUM> may be practiced where the host writing the data from the persistent memory storage device as file data comprises the host writing disk data to a tree for a differencing disk. In some such embodiments the method <NUM> may further include adding one or more memory intercept triggers on the disk data such that guest virtual machine requests to write to the data will be intercepted by a hypervisor such that the writes are written to a leaf of the tree rather than to an existing parent node.

The method <NUM> may be practiced where the host accessing file data and the host writing the file data to one or more persistent memory storage devices as file data is performed by a locked down usermode process.

The method <NUM> may further include implementing a loopback mount by using a filter to translate at least one of a block offset or a virtual byte offset to a file offset to enable a miniport driver coupled to the filter to access the one or more persistent memory storage devices.

The method <NUM> may be practiced where the host writing the file data to one or more persistent memory storage devices as file data is performed as part of a thin provisioning process. In some such embodiments, the method may further include processing a memory intercept and mapping a byte addressable persistent memory location to a guest virtual machine physical address.

The method <NUM> may be practiced where the host accessing file data and the host writing the file data to one or more persistent memory storage devices as file data is performed as part of an error handling operation. In some such embodiments, the method <NUM> may further include adding a memory intercept trigger for a file and subsequently performing the acts of the host accessing file data and the host writing the file data to one or more persistent memory storage devices as file data to move file data to a new location on the one or more persistent memory storage devices.

Physical computer-readable storage media includes RAM, ROM, EEPROM, CD-ROM or other optical disk storage (such as CDs, DVDs, etc), magnetic disk storage or other magnetic storage devices, or any other medium which can be used to store desired program code means in the form of computer-executable instructions or data structures and which can be accessed by a general purpose or special purpose computer.

When information is transferred or provided over a network or another communications connection (either hardwired, wireless, or a combination of hardwired or wireless) to a computer, the computer properly views the connection as a transmission medium. Transmissions media can include a network and/or data links which can be used to carry or desired program code means in the form of computer-executable instructions or data structures and which can be accessed by a general purpose or special purpose computer.

Claim 1:
A method of copying data stored in a byte addressable persistent memory storage device (<NUM>) for a virtual machine (<NUM>-<NUM>) on a host (<NUM>) in a virtual machine environment, the virtual machine environment comprising the byte addressable persistent memory storage device, the byte addressable persistent memory storage device having the ability to appear as a memory device (<NUM>-<NUM>, <NUM>-<NUM>) having available memory to the virtual machine and as a file to the host, the method comprising:
the host reading (<NUM>) data from the persistent memory storage device as file data from a virtual hard disk persistent memory file as a result of translating a block offset or a virtual byte offset to a file offset using a file translation table (<NUM>), wherein reading the data from the persistent memory storage device as file data includes reading the raw contents of the virtual hard disk persistent memory file and translating virtual byte offsets to logical block offsets in the file translation table by respecting a Block Translation Table (<NUM>), BTT, stored on the byte addressable persistent memory storage device, the BTT providing a mapping for translating block offsets to virtual byte offsets;
the host writing (<NUM>) the data from the persistent memory storage device as file data to a virtual hard disk block capable file; and
the virtual machine reading the written data as data stored as block addressable data to implement a block based virtual storage on the virtual machine using the byte addressable persistent memory storage as the physical hardware to implement the block based virtual storage on the virtual machine.