# Patent

## Title
Optimized copy of virtual machine storage files

## Abstract

A facility is described for optimizing the copying of virtual machine storage files. In various embodiments, the facility refrains from copying portions of the virtual machine storage file that do not contain any meaningful data for the purposes of the guest file system within the virtual machine storage file. In some embodiments, the facility refrains from copying portions of the file that are in use by the guest file system, but are of no practical value.

## Background
BACKGROUND
Virtual machines are commonly used by individuals, businesses, organizations, and other entities. A virtual machine is a self-contained, isolated software environment that runs its own operating system(s) and applications as if it were a physical computer. A virtual machine behaves like a physical computer, but comprises software components. Several virtual machines can all execute on the same physical computer. A physical computer includes one or more processors, computer readable media (e.g., memory), storage devices, input/output devices, etc.
A virtual machine comprises a collection of files, with the exact number being dependent upon the nature of the virtualization product being used. Irrespective of the virtualization product being used, all virtual machines have a file known as the “virtual machine storage file.” As an example, with VMWARE® products, this is commonly referred to as a VMDK file. With MICROSOFT® and CITRIX® products, this is instead referred to as a VHD file. These specific references to VMDK and VHD files are made only in the interest of clarity. One skilled in the art would recognize that other file types may also be used.
A single virtual machine may have more than one virtual machine storage file associated with it. A virtual machine storage file typically comprises at least one volume with a file system within it.
Some virtual machine storage files contain not just a guest file system, but also an operating system, referred to as the guest operating system. For example, a virtual machine running MICROSOFT EXCHANGE SERVER may have two virtual machine storage files—one containing a WINDOWS operating system plus MICROSOFT EXCHANGE software and a second one containing data for MICROSOFT EXCHANGE. The references to specific products made herein are simply in the interests of clarity.
System administrators sometimes need to copy virtual machine storage files from one location to another. Some illustrative, but by no means comprehensive set of examples are provided for clarification purposes. Virtualization enables easier load balancing, and load balancing requires moving a virtual machine from one physical host system to another. Virtual machine storage files are sometimes needed to be copied when virtual machine management software establishes a library, with the means to check virtual machines into the library, and check out virtual machines from the library. Each check in and checkout operation leads to copying of virtual machine storage files.
Programs and tools exist to copy files from one location to another. Some periodically create checkpoints and allow for the copy operation to be resumed from the checkpoint in case a previous copy operation was not completed. This feature is highly useful when copying large files across significant geographic distances. Some file-copy programs tune the size of the I/O to the state of the connection between the source and destination of the copy operation.



BRIEF DESCRIPTION OF THE DRAWINGS
FIG. 1 is a block diagram illustrating a virtual machine storage file layout in some embodiments.
FIG. 2 is a flow diagram illustrating logic employed by some embodiments of the facility.



DETAILED DESCRIPTION
A software and/or hardware facility (“the facility”) for optimizing the copying of virtual machine storage files is described. The term “source virtual machine storage file” or just “source” is used to refer to the file that is being copied. The term “destination” refers to the location where the file is being copied. The facility performs an optimized virtual machine storage file copy operation that takes into account what portions of the virtual machine storage file are relevant, and only copies those portions in various embodiments. The facility determines not only the nature of the file system contained within the virtual machine storage file, but also what parts of the virtual machine storage file contain meaningful data (as indicated by said file system).
FIG. 1 shows the layout of a typical virtual machine storage file. The file begins with an optional virtual machine storage file header 101, followed by some file system metadata 103. File systems typically divide the volume into entities referred to as logical clusters. The file system metadata 103 includes a bitmap describing which logical clusters are being used and which ones are free. FIG. 1 illustrates the logical cluster data blocks 105. Hereafter, this document refers to logical clusters using the term Lcn. The virtual machine storage file typically ends with a virtual machine storage file footer 107. To speed up operation and derive other benefits, the facility does not copy the Lcn data blocks indicated as free.
In some embodiments, the facility utilizes operating system application program interfaces (APIs) to access the virtual machine storage file as both a file and a volume. These APIs are provided by the operating system within which the facility is executing. For example, in MICROSOFT WINDOWS version 7 and MICROSOFT WINDOWS 2008 R2 operating systems, the facility utilizes the OpenVirtualDisk and AttachVirtualDisk APIs to access the virtual machine storage file as a read-only volume attached to the operating system within which the facility is executing. Thereafter, the facility can use operating system provided APIs such as FSCTL_GET_NTFS_DATA to determine useful information such as the storage allocation unit (also referred to as a cluster size) of the guest file system contained within the virtual machine storage file. Other examples of this include FSCTL_GET_VOLUME_BITMAP, which enables the facility to determine which clusters the guest file system is using, and IOCTL_VOLUME_LOGICAL_TO_PHYSICAL, which the facility uses to determine the first and last clusters of the guest file system located within the virtual machine storage file. In some embodiments, the facility always copies the portions of the virtual machine prior to the first cluster and after the last cluster. In other embodiments, alternate schemes may be employed. However the facility only copies Lcn data blocks if the Lcn bitmap indicates that particular portion is in use by the guest file system. (NOTE: In some embodiments, the facility fills the unused portions with zeros). In some embodiments, the facility fills the unused portions with zeros in the destination file.
In some embodiments, the facility utilizes its own API library to access the virtual machine storage file as both a file and a volume. This can be true when the facility executes in an operating system environment that does not provide APIs to access the volume(s) contained within the virtual machine storage file. Examples of such APIs include, but are not limited to OpenVirtualDisk and AttachVirtualDisk APIs in Microsoft Windows 7 and Microsoft Windows Server 2008 R2. Another example is when the facility is executing within an operating system that does not support the semantics of the guest file system contained within the virtual machine storage file.
In some embodiments, the facility ignores certain portions of the machine storage file that the guest file system indicates are in use. These portions pertain to files that are deemed non-essential. Examples include not copying the Lcn data blocks occupied by the hibernation file, and the paging file. (Note: these are simply examples, not a comprehensive list.) The exact details of what files are deemed non-essential depend upon the nature of the guest file system, and the intended use of the newly copied virtual machine file.
FIG. 2 is a flow diagram illustrating a routine performed by the facility in some embodiments. At block 202, the routine determines the nature of the file system contained within the virtual machine storage file. This is determined in a number of different possible ways. When the routine is executing on an operating system that provides the required APIs, the virtual machine storage file is mounted and accessed as a volume. In the MICROSOFT WINDOWS version 7 family of operating systems, this would be by using the OpenVirtualDisk and AttachVirtualDisk APIs. The result is that the facility can determine a handle to the mounted volume of interest.
Once the routine has obtained a handle to the volume of interest, it can determine the file system by using the FSCTL_FILESYSTEM_GET_STATISTICS or the FSCTL_QUERY_FILE_SYSTEM_RECOGNITION API. Some embodiments of the facility use alternative means such as a built in API library to determine the file system within the virtual machine storage file.
In some embodiments, the routine uses NTFS file system specific APIs. Different embodiments of the facility that can handle guest file systems other than NTFS use different but similar APIs. The references to NTFS are in the interest of clarity and by no means should be constructed to limit the facility to operate only on NTFS guest file systems.
At block 204, the facility uses API FSCTL_GET_NTFS_VOLUME_DATA to determine the file system cluster size, and also the number of logical clusters present within the guest file system. Logical clusters are an administrative unit that file systems typically divide the volume they cover into. Each logical cluster is X bytes in size, the size being determined when the volume is formatted with the file system. File systems also typically track the clusters to determine whether they are free or in use, by utilizing a data structure called an Lcn bitmap. Each bit in the bitmap represents one logical cluster data block in the volume.
At block 206, the facility determines the location of the first cluster data block within the virtual machine storage file. The facility accomplishes this by using the API IOCTL_VOLUME_LOGICAL_TO_PHYSICAL, after which it copies all of the data from the beginning of the virtual machine storage file to this location.
At block 208, the facility retrieves the Logical Cluster Number (Lcn) bit map, N bits at a time. This can be accomplished using the FSCTL_GET_VOLUME_BITMAP API.
At decision block 210, the facility determines whether it has received clusters. Once a set of N Lcn bitmaps are obtained, they are processed by the logic of block 212. To process, the facility determines if a cluster bitmap indicates the data is in use. If so, the corresponding data is copied from the source file to the destination file. However, if the bit indicates the cluster is not in use, the source and destination file pointers are advanced by the size of the cluster.
When the N obtained Lcn bitmap bits are all processed, the facility loops to the logic of block 208 to obtain another bitmap set containing data for N or fewer clusters.
When all the bitmap data for a volume has been obtained and processed, the facility continues at block 214. The facility copies all of the remaining contents of the source virtual machine storage file that are after the last Lcn data block, to the corresponding offset in the destination virtual machine storage file. This includes the virtual machine storage file footer 107. The routine then returns at block 216.
In various embodiments, when a virtual machine storage file contains multiple volumes (and file systems), the facility operates on the virtual machine storage file one file system volume at a time.
Thus, the facility treats the contents of the VHD file as a non opaque structure during the file copy operation AND while the VHD is not running.

## Claims

1. A method performed by a host computer system for performing an optimized copy of virtual machine storage files, the method comprising:
under control of a physical host computer system configured to execute virtual machines associated with virtual machine storage files, the physical host computer system having executing thereon a physical host operating system that provides one or more application programming interfaces (APIs):
mounting, as a volume, on the physical host computer system, a virtual machine storage file associated with one of the virtual machines;
determining a handle of the mounted volume via the one or more APIs of the physical host operating system;
wherein the virtual machine storage file comprises guest system metadata and a guest file system;
wherein the guest file system comprises a plurality of logical clusters having a file system cluster size expressible in bytes;
wherein the guest system metadata comprises a cluster bitmap, the cluster bitmap comprising information on whether each logical cluster of the guest file system is in use by the guest file system;
identifying the guest file system included in the virtual machine storage file via the handle and the one or more APIs of the physical host operating system;
obtaining, via the one or more APIs of the physical host operating system, the file system cluster size of the guest file system and a number of logical clusters in the plurality of logical clusters of the guest file system;
identifying, via the one or more APIs of the physical host operating system, a location within the virtual machine storage file of the first logical cluster of the guest file system;
copying, to a destination file, the virtual machine storage file from a beginning of the virtual machine storage file to the identified location of the first logical cluster of the guest file system;
accessing the cluster bitmap for information on whether each logical cluster in the guest file system is in use by the guest file system;
for each logical cluster of the plurality of logical clusters, determining, from the cluster bitmap, whether the logical cluster is in use by the guest file system;
for each logical cluster of the plurality of logical clusters, responsive to a determination that the logical cluster is not in use by the guest file system, advancing file pointer of the destination file by the file system cluster size without copying the logical cluster to the destination file; and
for each logical cluster of the plurality of logical clusters, responsive to a determination that the logical cluster is in use by the guest file system, copying the logical cluster to the destination file.

2. The method of claim 1 further comprising:
determining from the accessed cluster bitmap whether additional clusters remain to be processed; and
if additional clusters remain to be processed, further processing the cluster bitmap.

3. The method of claim 1 further comprising:
identifying a location within the virtual machine storage file of the last cluster of the guest file system; and
copying, to the destination file, the virtual machine storage file from the identified location of the last cluster of the guest file system to an end of the virtual machine storage file.

4. The method of claim 3 further comprising setting file meta data.

5. The method of claim 1 further comprising avoiding copying blocks occupied by a hibernation file or a paging file associated with the guest file system.

6. The method of claim 1 wherein the virtual machine storage file being copied comprises a guest operating system image associated with the guest file system.

7. The method of claim 1 wherein the virtual machine storage file being copied comprises the guest file system, but does not comprise a guest operating system image.

8. The method of claim 1 wherein the virtual machine storage file being copied comprises a single volume embedded within the virtual machine storage file.

9. The method of claim 1 wherein the virtual machine storage file being copied comprises multiple volumes embedded within the virtual machine storage file.

10. The method of claim 1 wherein the virtual machine storage file comprises a Microsoft specified VHD file format.

11. The method of claim 1 wherein the virtual machine storage file comprises a VMware specified VMDK file format.

12. The method of claim 1 wherein the virtual machine storage file is copied to multiple destination files.

13. The method of claim 1 further comprising:
determining, from the accessed cluster bitmap, clusters in the guest file system that are free; and
filling with zeroes portions of the destination file associated with the clusters determined to be free.

14. The method of claim 1 wherein accessing the virtual machine storage file comprises accessing the virtual machine storage file as a read-only volume attached to a host operating system of the host computer system.

15. A system for copying virtual machine storage files, the system comprising a physical host computer system configured to execute a virtual machine, the virtual machine associated with a guest operating system and a guest file system, the physical host computer system having executing thereon a physical host operating system that provides one or more application programming interfaces (APIs), the physical host computing system configured to implement a method comprising:
mounting, as a volume, on the physical host computer system, a virtual machine storage file associated with one of the virtual machines;
determining a handle of the mounted volume via the one or more APIs of the physical host operating system;
wherein the virtual machine storage file comprises guest system metadata and a guest file system;
wherein the guest file system comprises a plurality of logical clusters having a file system cluster size expressible in bytes;
wherein the guest system metadata comprises a cluster bitmap, the cluster bitmap comprising information on whether each logical cluster of the guest file system is in use by the guest file system;
identifying the guest file system included in the virtual machine storage file via the handle and the one or more APIs of the physical host operating system;
obtaining, via the one or more APIs of the physical host operating system, the file system cluster size of the guest file system and a number of logical clusters in the plurality of logical clusters of the guest file system;
identifying, via the one or more APIs of the physical host operating system, a location within the virtual machine storage file of the first logical cluster of the guest file system;
copying, to a destination file, the virtual machine storage file from a beginning of the virtual machine storage file to the identified location of the first logical cluster of the guest file system;
accessing the cluster bitmap for information on whether each logical cluster in the guest file system is in use by the guest file system;
for each logical cluster of the plurality of logical clusters, determining, from the cluster bitmap, whether the logical cluster is in use by the guest file system;
for each logical cluster of the plurality of logical clusters, responsive to a determination that the logical cluster is not in use by the guest file system, advancing file pointer of the destination file by the file system cluster size without copying the logical cluster to the destination file; and
for each logical cluster of the plurality of logical clusters, responsive to a determination that the logical cluster is in use by the guest file system, copying the logical cluster to the destination file.

16. A computer-program product comprising a non-transitory computer-usable medium having computer-readable program code embodied therein, the computer-readable program code adapted to be executed to implement a method, the method being performed by an agent installed in a computing environment on a computer system, the method comprising:
mounting, as a volume, on a physical host computer system, a virtual machine storage file associated with a virtual machine;
determining a handle of the mounted volume via one or more APIs provided by a physical host operating system executing on the physical host computer system;
wherein the virtual machine storage file comprises guest system metadata and a guest file system;
wherein the guest file system comprises a plurality of logical clusters having a file system cluster size expressible in bytes;
wherein the guest system metadata comprises a cluster bitmap, the cluster bitmap comprising information on whether each logical cluster of the guest file system is in use by the guest file system;
identifying the guest file system included in the virtual machine storage file via the handle and the one or more APIs of the physical host operating system;
obtaining, via the one or more APIs of the physical host operating system, the file system cluster size of the guest file system and a number of logical clusters in the plurality of logical clusters of the guest file system;
identifying, via the one or more APIs of the physical host operating system, a location within the virtual machine storage file of the first logical cluster of the guest file system;
copying, to a destination file, the virtual machine storage file from a beginning of the virtual machine storage file to the identified location of the first logical cluster of the guest file system;
accessing the cluster bitmap for information on whether each logical cluster in the guest file system is in use by the guest file system;
for each logical cluster of the plurality of logical clusters, determining, from the cluster bitmap, whether the logical cluster is in use by the guest file system;
for each logical cluster of the plurality of logical clusters, responsive to a determination that the logical cluster is not in use by the guest file system, advancing file pointer of the destination file by the file system cluster size without copying the logical cluster to the destination file; and
for each logical cluster of the plurality of logical clusters, responsive to a determination that the logical cluster is in use by the guest file system, copying the logical cluster to the destination file.