Filter appliance for object-based storage system

A framework for performing transformations of logical storage volumes in software is provided. This framework interposes on various operations that can be performed on a logical storage volume, such as input/output (IO) operations, via one or more filters, which may be implemented by an appliance that is inserted into the data path of the operations issued to the logical storage volume.

BACKGROUND

As computer systems scale to enterprise levels, particularly in the context of supporting large-scale data centers, the underlying data storage systems frequently employ a storage area network (SAN) or network attached storage (NAS). As is conventionally well appreciated, SAN or NAS provides a number of technical capabilities and operational benefits, fundamentally including virtualization of data storage devices, redundancy of physical devices with transparent fault-tolerant fail-over and fail-safe controls, geographically distributed and replicated storage, and centralized oversight and storage configuration management decoupled from client-centric computer systems management.

SCSI and other block protocol-based storage devices, such as a storage system30shown inFIG. 1A, utilize a storage system manager31, which represents one or more programmed storage processors, to aggregate the storage units or drives in the storage device and present them as one or more LUNs (Logical Unit Numbers)34each with a uniquely identifiable number. LUNs34are accessed by one or more computer systems10through a physical host bus adapter (HBA)11over a network20(e.g., Fibre Channel, etc.). Within computer system10and above HBA11, storage access abstractions are characteristically implemented through a series of software layers, beginning with a low-level device driver layer12and ending in an operating system specific file system layers15. Device driver layer12, which enables basic access to LUNs34, is typically specific to the communication protocol used by the storage system (e.g., SCSI, etc.). A data access layer13may be implemented above device driver layer12to support multipath consolidation of LUNs34visible through HBA11and other data access control and management functions. A logical volume manager14, typically implemented between data access layer13and conventional operating system file system layers15, supports volume-oriented virtualization and management of LUNs34that are accessible through HBA11. Multiple LUNs34can be gathered and managed together as a volume under the control of logical volume manager14for presentation to and use by file system layers15as a logical device.

FIG. 1Bis a block diagram of a conventional NAS or file-level based storage system40that is connected to one or more computer systems10via network interface cards (NIC)11′ over a network21(e.g., Ethernet). Storage system40includes a storage system manager41, which represents one or more programmed storage processors. Storage system manager41implements a file system45on top of physical, typically disk drive-based storage units, referred to inFIG. 1Bas spindles42, that reside in storage system40. From a logical perspective, each of these spindles can be thought of as a sequential array of fixed sized extents43. File system45abstracts away complexities of targeting read and write operations to addresses of the actual spindles and extents of the disk drives by exposing to connected computer systems, such as computer systems10, a namespace comprising directories and files that may be organized into file system level volumes44(hereinafter referred to as “FS volumes”) that are accessed through their respective mount points.

It has been recognized that the storage systems described above are not sufficiently scalable to meet the particular needs of virtualized computer systems. For example, a cluster of server machines may service as many as 10,000 virtual machines (VMs), each VM using a multiple number of “virtual disks” and a multiple number of “snapshots,” each which may be stored, for example, as a file on a particular LUN or FS volume. Even at a scaled down estimation of 2 virtual disks and 2 snapshots per VM, this amounts to 60,000 distinct disks for the storage system to support if VMs were directly connected to physical disks (i.e., 1 virtual disk or snapshot per physical disk). In addition, storage device and topology management at this scale are known to be difficult. As a result, the concept of datastores in which VMs are multiplexed onto a smaller set of physical storage entities (e.g., LUN-based VMFS clustered file systems or FS volumes), such as described in U.S. Pat. No. 7,849,098, entitled “Providing Multiple Concurrent Access to a File System,” incorporated by reference herein, was developed.

In conventional storage systems employing LUNs or FS volumes, workloads from multiple VMs are typically serviced by a single LUN or a single FS volume. As a result, resource demands from one VM workload will affect the service levels provided to another VM workload on the same LUN or FS volume. Efficiency measures for storage such as latency and input/output operations per second, or IOPS, thus vary depending on the number of workloads in a given LUN or FS volume and cannot be guaranteed. Consequently, storage policies for storage systems employing LUNs or FS volumes cannot be executed on a per-VM basis and service level agreement (SLA) guarantees cannot be given on a per-VM basis. In addition, data services provided by storage system vendors, such as snapshot, replication, encryption, and deduplication, are provided at a granularity of the LUNs or FS volumes, not at the granularity of a VM's virtual disk. As a result, snapshots can be created for the entire LUN or the entire FS volume using the data services provided by storage system vendors, but a snapshot for a single virtual disk of a VM cannot be created separately from the LUN or the file system in which the virtual disk is stored.

An object-based storage system disclosed in U.S. patent application Ser. No. 13/219,358, filed Aug. 26, 2011, incorporated by reference herein, provides a solution by exporting logical storage volumes that are provisioned as storage objects, referred to herein as “virtual volumes.” These storage objects are accessed on demand by connected computer systems using standard protocols, such as SCSI and NFS, through logical endpoints for the protocol traffic that are configured in the storage system. Logical storage volumes are created from one or more logical storage containers having an address space that maps to storage locations of the physical data storage units. The reliance on logical storage containers provide users of the object-based storage system with flexibility in designing their storage solutions, because a single logical storage container may span more than one physical storage system and logical storage containers of different customers can be provisioned from the same physical storage system with appropriate security settings.

SUMMARY

One or more embodiments disclosed herein provide a framework for performing transformations of logical storage volumes in software, by interposing on various operations, such as input/output (IO) operations, that can be performed on a logical storage volume. In one embodiment, one or more filters are implemented to interpose on the operations, and the filters are implemented by an appliance that is inserted into the data path of the operations issued to the logical storage volume.

A method of interposing a filter in an IO path between an issuer of IO operations and a logical storage volume, according to an embodiment, includes the steps of creating a filtered logical storage container having properties defined by the filter within a filter appliance, and logically moving the logical storage volume from its current, unfiltered logical storage container to the filtered logical storage container such that IO operations in the IO path are redirected to the filter appliance for processing by the filter appliance. The filter appliance may include functionalities for any type of filter, such as a cache filter, a compression/decompression filter, and an encryption/decryption filter.

A method of interposing on operations targeted for execution on a logical storage volume, according to an embodiment, includes the steps of receiving a read operation targeted for execution on a first logical storage volume, issuing a read operation targeted for execution on a second logical storage volume, in place of the read operation targeted for execution on the first logical storage volume, to obtain read data from the second logical storage volume, executing at least one filtering operation on the read data to generate transformed data, and returning the transformed data in response to the received read operation.

A method of interposing on operations targeted for execution on a logical storage volume, according to another embodiment, includes the steps of receiving a write operation targeted for execution on a first logical storage volume, executing at least one filtering operation on data referenced in the received write operation to generate transformed data, issuing a write operation targeted for execution on a second logical storage volume, in place of the write operation targeted for execution on the first logical storage volume, to write the transformed data to the second logical storage volume, and returning a write acknowledgment in response to the received write operation upon receiving confirmation that the write operation targeted for execution on the second logical storage volume has completed.

A computer system according to an embodiment includes virtual machines (VMs) running therein, wherein one of the VMs is issuing IO operations and another of the VMs is a filter appliance that is configured to: (i) receive a read/write operation targeted for execution on a first logical storage volume, (ii) transform the read/write operation into a set of different operations including a read/write operation targeted for execution on a second logical storage volume, (iii) issue the read/write operation targeted for execution on the second logical storage volume, and (iv) return an indication of completion of the received read/write operation.

Embodiments further include a non-transitory computer-readable storage medium storing instructions that when executed by a computer system cause the computer system to perform one of the methods set forth above.

DETAILED DESCRIPTION

FIGS. 2A and 2Bare block diagrams of a storage system cluster that implements virtual volumes, each of which may be transformed using the filter framework described herein. The storage system cluster includes one or more storage systems, e.g., storage systems1301and1302, which may be disk arrays, each having a plurality of data storage units (DSUs), one of which is labeled as141in the figures, and storage system managers131and132that control various operations of storage systems130. Two or more storage systems130may implement a distributed storage system manager135that controls the operations of the storage system cluster as if they were a single logical storage system. DSUs represent physical storage units, e.g., disk or flash based storage units such as rotating disks or solid state disks. The storage system cluster creates and exposes virtual volumes (vvols), as further detailed herein, to connected computer systems, such as computer systems1001and1002. Applications (e.g., VMs accessing their virtual disks, etc.) running in computer systems100access the vvols on demand using standard protocols, such as SCSI in the embodiment ofFIG. 2Aand NFS in the embodiment ofFIG. 2B, through logical endpoints for the SCSI or NFS protocol traffic, known as “protocol endpoints” (PEs), that are configured in storage systems130. For simplicity, computer systems100are shown to be directly connected to storage systems130. However, it should be understood that they may be connected to storage systems130through multiple paths and one or more of switches.

Distributed storage system manager135or a single storage system manager131or132may create vvols (e.g., upon request of a computer system100, etc.) from logical “storage containers,” which represent a logical aggregation of physical DSUs. In general, a storage container may span more than one storage system and many storage containers may be created by a single storage system manager or a distributed storage system manager. Similarly, a single storage system may contain many storage containers. InFIGS. 2A and 2B, storage container142Acreated by distributed storage system manager135is shown as spanning storage system1301and storage system1302, whereas storage container142Band storage container142Care shown as being contained within a single storage system (i.e., storage system1301and storage system1302, respectively). It should be recognized that, because a storage container can span more than one storage system, a storage system administrator can provision to its customers a storage capacity that exceeds the storage capacity of any one storage system. It should be further recognized that, because multiple storage containers can be created within a single storage system, the storage system administrator can provision storage to multiple customers using a single storage system.

In the embodiment ofFIG. 2A, each vvol is provisioned from a block based storage system. In the embodiment ofFIG. 2B, a NAS based storage system implements a file system145on top of DSUs141and each vvol is exposed to computer systems100as a file object within this file system. In addition, as will be described in further detail below, applications running on computer systems100access vvols for IO operations through PEs. For example, as illustrated in dashed lines inFIGS. 2A and 2B, vvol151and vvol152are accessible via PE161; vvol153and vvol155are accessible via PE162; vvol154is accessible via PE163and PE164; and vvol156is accessible via PE165. It should be recognized that vvols from multiple storage containers, such as vvol153in storage container142Aand vvol155in storage container142C, may be accessible via a single PE, such as PE162, at any given time. It should further be recognized that PEs, such as PE166, may exist in the absence of any vvols that are accessible via them.

In the embodiment ofFIG. 2A, storage systems130implement PEs as a special type of LUN using known methods for setting up LUNs. As with LUNs, a storage system130provides each PE a unique identifier known as a WWN (World Wide Name). In one embodiment, when creating the PEs, storage system130does not specify a size for the special LUN because the PEs described herein are not actual data containers. In one such embodiment, storage system130may assign a zero value or a very small value as the size of a PE-related LUN such that administrators can quickly identify PEs when requesting that a storage system provide a list of LUNs (e.g., traditional data LUNs and PE-related LUNs), as further discussed below. Similarly, storage system130may assign a LUN number greater than 255 as the identifying number for the LUN to the PEs to indicate, in a human-friendly way, that they are not data LUNs. As another way to distinguish between the PEs and LUNs, a PE bit may be added to the Extended Inquiry Data VPD page (page 86h). The PE bit is set to 1 when a LUN is a PE, and to 0 when it is a regular data LUN. In the embodiment ofFIG. 2B, the PEs are created in storage systems130using known methods for setting up mount points to FS volumes. Each PE that is created in the embodiment ofFIG. 2Bis identified uniquely by an IP address and file system path, also conventionally referred together as a “mount point.” However, unlike conventional mount points, the PEs are not associated with FS volumes.

FIG. 3is a block diagram of components of the storage system cluster ofFIG. 2A or 2Bfor managing virtual volumes according to an embodiment. The components include software modules of storage system managers131and132executing in storage systems130in one embodiment or software modules of distributed storage system manager135in another embodiment, namely an IO manager304, a volume manager306, a container manager308, and a data access layer310. In the descriptions of the embodiments herein, it should be understood that any actions taken by distributed storage system manager135may be taken by storage system manager131or storage system manager132depending on the embodiment.

In the example ofFIG. 3, distributed storage system manager135has created three storage containers SC1, SC2, and SC3 from DSUs141, each of which is shown to have spindle extents labeled P1 through Pn. In general, each storage container has a fixed physical size, and is associated with specific extents of DSUs. In the example shown inFIG. 3, distributed storage system manager135has access to a container database316that stores for each storage container, its container ID, physical layout information and some metadata. Container database316is managed and updated by a container manager308, which in one embodiment is a component of distributed storage system manager135. The container ID is a universally unique identifier that is given to the storage container when the storage container is created. Physical layout information consists of the spindle extents of DSUs141that are associated with the given storage container and stored as an ordered list of <system ID, DSU ID, extent number>. The metadata section may contain some common and some storage system vendor specific metadata. For example, the metadata section may contain the IDs of computer systems or applications or users that are permitted to access the storage container. As another example, the metadata section contains an allocation bitmap to denote which <system ID, DSU ID, extent number> extents of the storage container are already allocated to existing vvols and which ones are free. In one embodiment, a storage system administrator may create separate storage containers for different business units so that vvols of different business units are not provisioned from the same storage container. Other policies for segregating vvols may be applied. For example, a storage system administrator may adopt a policy that vvols of different customers of a cloud service are to be provisioned from different storage containers. Also, vvols may be grouped and provisioned from storage containers according to their required service levels. In addition, a storage system administrator may create, delete, and otherwise manage storage containers, such as defining the number of storage containers that can be created and setting the maximum physical size that can be set per storage container.

Also, in the example ofFIG. 3, distributed storage system manager135has provisioned (on behalf of requesting computer systems100) multiple vvols, each from a different storage container. In general, vvols may have a fixed physical size or may be thinly provisioned, and each vvol has a vvol ID, which is a universally unique identifier that is given to the vvol when the vvol is created. For each vvol, a vvol database314stores for each vvol, its vvol ID, the container ID of the storage container in which the vvol is created, and an ordered list of <offset, length> values within that storage container that comprise the address space of the vvol. Vvol database314is managed and updated by volume manager306, which in one embodiment, is a component of distributed storage system manager135. In one embodiment, vvol database314also stores a small amount of metadata about the vvol. This metadata is stored in vvol database314as a set of key-value pairs, and may be updated and queried by computer systems100via the out-of-band path at any time during the vvol's existence. Stored key-value pairs fall into three categories. The first category is: well-known keys—the definition of certain keys (and hence the interpretation of their values) are publicly available. One example is a key that corresponds to the virtual volume type (e.g., in virtual machine embodiments, whether the vvol contains a VM's metadata or a VM's data). Another example is the App ID, which is the ID of the application that stored data in the vvol. The second category is: computer system specific keys—the computer system or its management module stores certain keys and values as the virtual volume's metadata. The third category is: storage system vendor specific keys—these allow the storage system vendor to store certain keys associated with the virtual volume's metadata. One reason for a storage system vendor to use this key-value store for its metadata is that all of these keys are readily available to storage system vendor plug-ins and other extensions via the out-of-band channel for vvols. The store operations for key-value pairs are part of virtual volume creation and other processes, and thus the store operation should be reasonably fast. Storage systems are also configured to enable searches of virtual volumes based on exact matches to values provided on specific keys.

IO manager304is a software module (also, in certain embodiments, a component of distributed storage system manager135) that maintains a connection database312that stores currently valid IO connection paths between PEs and vvols. In the example shown inFIG. 3, seven currently valid IO sessions are shown. Each valid session has an associated PE ID, secondary level identifier (SLLID), vvol ID, and reference count (RefCnt) indicating the number of different applications that are performing IO through this IO session. The process of establishing a valid IO session between a PE and a vvol by distributed storage system manager135(e.g., on request by a computer system100) is referred to herein as a “bind” process. For each bind, distributed storage system manager135(e.g., via IO manager304) adds an entry to connection database312. The process of subsequently tearing down the IO session by distributed storage system manager135is referred to herein as an “unbind” process. For each unbind, distributed storage system manager135(e.g., via IO manager304) decrements the reference count of the IO session by one. When the reference count of an IO session is at zero, distributed storage system manager135(e.g., via IO manager304) may delete the entry for that IO connection path from connection database312. In one embodiment, the generation number is changed to a monotonically increasing number or a randomly generated number, when the reference count changes from 0 to 1 or vice versa. In another embodiment, the generation number is a randomly generated number and the RefCnt column is eliminated from connection database312, and for each bind, even when the bind request is to a vvol that is already bound, distributed storage system manager135(e.g., via IO manager304) adds an entry to connection database312.

In the storage system cluster ofFIG. 2A, IO manager304processes IO operations from computer systems100received through the PEs using connection database312. When an IO operation is received at one of the PEs, IO manager304parses the IO operation to identify the PE ID and the SLLID contained in the IO operation in order to determine a vvol for which the IO operation was intended. By accessing connection database314, IO manager304is then able to retrieve the vvol ID associated with the parsed PE ID and SLLID. InFIG. 3and subsequent figures, PE ID is shown as PE_A, PE_B, etc. for simplicity. In one embodiment, the actual PE IDs are the WWNs of the PEs. In addition, SLLID is shown as S0001, S0002, etc. The actual SLLIDs are generated by distributed storage system manager135as any unique number among SLLIDs associated with a given PE ID in connection database312. The mapping between the logical address space of the virtual volume having the vvol ID and the physical locations of DSUs141is carried out by volume manager306using vvol database314and by container manager308using container database316. Once the physical locations of DSUs141have been obtained, data access layer310(in one embodiment, also a component of distributed storage system manager135) performs the IO operation on these physical locations.

In the storage system cluster ofFIG. 2B, IO operations are received through the PEs and each such IO operation includes an NFS handle (or similar file system handle) to which the IO operation has been issued. In one embodiment, connection database312for such a system contains the IP address of the NFS interface of the storage system as the PE ID and the file system path as the SLLID. The SLLIDs are generated based on the location of the vvol in the file system145. The mapping between the logical address space of the vvol and the physical locations of DSUs141is carried out by volume manager306using vvol database314and by container manager308using container database316. Once the physical locations of DSUs141have been obtained, data access layer performs IO operation on these physical locations. It should be recognized that for a storage system ofFIG. 2B, container database312may contain an ordered list of file: <offset, length> entries in the Container Locations entry for a given vvol (i.e., a vvol can be comprised of multiple file segments that are stored in the file system145).

In one embodiment, connection database312is maintained in volatile memory while vvol database314and container database316are maintained in persistent storage, such as DSUs141. In other embodiments, all of the databases312,314,316may be maintained in persistent storage.

FIG. 4is a flow diagram of method steps410for creating a storage container. In one embodiment, these steps are carried out by storage system manager131, storage system manager132or distributed storage system manager135under control of a storage administrator. As noted above, a storage container represents a logical aggregation of physical DSUs and may span physical DSUs from more than one storage system. At step411, the storage administrator (via distributed storage system manager135, etc.) sets a physical capacity of a storage container. Within a cloud or data center, this physical capacity may, for example, represent the amount of physical storage that is leased by a customer. The flexibility provided by storage containers disclosed herein is that storage containers of different customers can be provisioned by a storage administrator from the same storage system and a storage container for a single customer can be provisioned from multiple storage systems, e.g., in cases where the physical capacity of any one storage device is not sufficient to meet the size requested by the customer, or in cases such as replication where the physical storage footprint of a vvol will naturally span multiple storage systems. At step412, the storage administrator sets permission levels for accessing the storage container. In a multi-tenant data center, for example, a customer may only access the storage container that has been leased to him or her. At step413, distributed storage system manager135generates a unique identifier for the storage container. Then, at step414, distributed storage system manager135(e.g., via container manager308in one embodiment) allocates free spindle extents of DSUs141to the storage container in sufficient quantities to meet the physical capacity set at step411. As noted above, in cases where the free space of any one storage system is not sufficient to meet the physical capacity, distributed storage system manager135may allocate spindle extents of DSUs141from multiple storage systems. After the partitions have been allocated, distributed storage system manager135(e.g., via container manager308) updates container database316with the unique container ID, an ordered list of <system number, DSU ID, extent number>, and context IDs of computer systems that are permitted to access the storage container.

According to embodiments described herein, storage capability profiles, e.g., SLAs or quality of service (QoS), may be configured by distributed storage system manager135(e.g., on behalf of requesting computer systems100) on a per vvol basis. Therefore, it is possible for vvols with different storage capability profiles to be part of the same storage container. In one embodiment, a system administrator defines a default storage capability profile (or a number of possible storage capability profiles) for newly created vvols at the time of creation of the storage container and stored in the metadata section of container database316. If a storage capability profile is not explicitly specified for a new vvol being created inside a storage container, the new vvol will inherit the default storage capability profile associated with the storage container.

FIG. 5Ais a block diagram of an embodiment of a computer system having a filter appliance and configured to implement virtual volumes hosted on a storage system cluster ofFIG. 2A. Computer system101may be constructed on a conventional, typically server-class, hardware platform500that includes one or more central processing units (CPU)501, memory502, one or more network interface cards (NIC)503, and one or more host bus adapters (HBA)504. HBA504enables computer system101to issue IO operations to virtual volumes through PEs configured in storage devices130. As further shown inFIG. 5A, operating system508is installed on top of hardware platform500and a number of applications5121-512Nand a filter appliance513are executed on top of operating system508. Examples of operating system508include any of the well-known commodity operating systems, such as Microsoft Windows®, Linux®, and the like.

According to embodiments described herein, each application512has one or more vvols associated therewith and issues IO operations to block device instances of the vvols created by operating system508pursuant to “CREATE DEVICE” calls by application512into operating system508. The association between block device names and vvol IDs are maintained in block device database533. IO operations from applications5122-512Nare received by a file system driver510, which converts them to block IO operations, and provides the block IO operations to a virtual IO interposer (VIOI)531. IO operations from application5121, on the other hand, are shown to bypass file system driver510and provided directly to VIOI531, signifying that application5121accesses its block device directly as a raw storage device, e.g., as a database disk, a log disk, a backup archive, and a content repository, in the manner described in U.S. Pat. No. 7,155,558 entitled “Providing Access to a Raw Data Storage Unit in a Computer System,” the entire contents of which are incorporated by reference herein.

Upon receipt of a block IO operation from one of applications512, VIOI531determines whether or not the block device name specified in the block IO operation is associated with a filtered vvol with reference to block device database533. In the example illustrated inFIG. 5A, the block device name “foo” is shown to be associated with a filtered vvol, vvol1-f (where the suffix “-f” is the nomenclature used herein to indicate that a certain vvol is a filtered vvol), whereas the block device names “dbase” and “log” are not associated with filtered vvols. If the block device name specified in the block IO operation is not associated with a filtered vvol, VIOI531passes the block IO operation to a virtual volume device driver532. In turn, virtual volume device driver532accesses block device database533to reference a mapping between the block device name specified in the IO operation and the PE ID (WWN of PE LUN) and SLLID that define the IO connection path to the vvol associated with the block device name. Other information that is stored in block device database533includes an active bit value for each block device that indicates whether or not the block device is active, and a CIF (commands-in-flight) value. An active bit of “1” signifies that IO operations can be issued to the block device. An active bit of “0” signifies that the block device is inactive and IO operations cannot be issued to the block device. The CIF value provides an indication of how many IO operations are in flight, i.e., issued but not completed. In the example shown herein, the block device, foo, is active, and has some commands-in-flight. The block device, dbase, is inactive with no outstanding commands. Finally, the block device, log, is active, but the application currently has no pending IO operations to the device.

If the block device name specified in the block IO operation received by VIOI531is associated with a filtered vvol, VIOI531redirects the block IO operation to filter appliance513for processing through a control interface511. Filter appliance513performs one or more filtering operations on the data referenced in the block IO operation. For reads, filter appliance513issues a block IO operation to virtual volume device driver532(identifying the block device name as “foo-f” in the example) and performs the filtering operations on the read data that is returned, and then passes the filtered read data back to VIOI531through control interface511. For writes, filter appliance513transforms the write data that is referenced in the block IO operation by performing filtering operations thereon and issues a block IO operation to virtual volume device driver532(identifying the block device name as “foo-f” in the example) to write the transformed data. Then, upon receipt of a write acknowledgement, filter appliance513passes the write acknowledgement to VIOI531through control interface511. For both reads and writes, upon receipt of the block IO operation from filter appliance513, virtual volume device driver532handles the block IO operation in the same manner as described above when the block IO operation was directly passed to virtual volume device driver532. Also, the target vvol is the same. The difference is in the data that is returned to the application as read data (filtered data is returned) and written as write data (filtered data is written).

The functions of control interface511are implemented with memory that is shared between filter appliance513and OS508(in particular, VIOI531). Unidirectional queues are configured in the shared memory and each queue has a producer and a consumer. For reads, VIOI531is the producer and filter appliance513is the consumer. For writes, filter appliance513is the producer and VIOI531is the consumer.

Virtual volume device driver532issues raw block-level IO operations (as received from VIOI531or filter appliance513) to data access layer540. Data access layer540includes device access layer534, which applies command queuing and scheduling policies to the raw block-level IO operations, and device driver536for HBA504which formats the raw block-level IO operations in a protocol-compliant format and sends them to HBA504for forwarding to the PEs via an in-band path. In the embodiment where SCSI protocol is used, the vvol information is encoded in the SCSI LUN data field, which is an 8-byte structure, as specified in SAM-5 (SCSI Architecture Model-5). The PE ID is encoded in the first 2 bytes, which is conventionally used for the LUN ID, and the vvol information, in particular the SLLID, is encoded in the SCSI second level LUN ID, utilizing (a portion of) the remaining 6 bytes.

As further shown inFIG. 5A, data access layer540also includes an error handling unit542for handling IO errors that are received through the in-band path from the storage system. In one embodiment, the IO errors received by error handling unit542are propagated through the PEs by IO manager304. Examples of IO error classes include path errors between computer system101and the PEs, PE errors, and vvol errors. The error handling unit542classifies all detected errors into aforementioned classes. When a path error to a PE is encountered and another path to the PE exists, data access layer540transmits the IO operation along a different path to the PE. When the IO error is a PE error, error handing unit542updates block device database533to indicate an error condition for each block device issuing IO operations through the PE. When the IO error is a vvol error, error handing unit542updates block device database533to indicate an error condition for each block device associated with the vvol. Error handing unit542may also issue an alarm or system event so that further IO operations to block devices having the error condition will be rejected.

FIG. 5Bis a block diagram of the computer system ofFIG. 5Athat has been configured to interface with the storage system cluster ofFIG. 2Binstead of the storage system cluster ofFIG. 2A. In this embodiment, data access layer540includes an NFS client545and a device driver546for NIC503. NFS client545maps the block device name to a PE ID (IP address of NAS storage system) and a SLLID which is a NFS file handle corresponding to the block device. This mapping is stored in block device database533as shown inFIG. 5B. It should be noted that the Active and CIF columns are still present but not illustrated in the block device database533shown inFIG. 5B. NFS client545also translates the raw block-level IO operations received from virtual volume device driver532to NFS file-based IO operations. Device driver546for NIC503then formats the NFS file-based IO operations in a protocol-compliant format and sends them to NIC503, along with the NFS handle, for forwarding to one of the PEs.

FIG. 5Cis a block diagram of another embodiment of a computer system having a filter appliance and configured to implement virtual volumes. In this embodiment, computer system102is configured with virtualization software, shown herein as hypervisor560. Hypervisor560is installed on top of hardware platform550, which includes CPU551, memory552, NIC553, and HBA554, and supports a virtual machine execution space570within which multiple virtual machines (VMs)5711-571Nand a filter appliance VM583may be concurrently instantiated and executed. In one or more embodiments, hypervisor560and virtual machines571,583are implemented using the VMware vSphere® product distributed by VMware, Inc. of Palo Alto, Calif. Each virtual machine571implements a virtual hardware platform573that supports the installation of a guest operating system (OS)572which is capable of executing applications579. Examples of a guest OS572include any of the well-known commodity operating systems, such as Microsoft Windows, Linux, and the like. In each instance, guest OS572includes a native file system layer (not shown inFIG. 5C), for example, either an NTFS or an ext3FS type file system layer. These file system layers interface with virtual hardware platforms573to access, from the perspective of guest OS572, a data storage HBA, which in reality, is virtual HBA574implemented by virtual hardware platform573that provides the appearance of disk storage support (in reality, virtual disks or virtual disks575A-575X) to enable execution of guest OS572. In certain embodiments, virtual disks575A-575Xmay appear to support, from the perspective of guest OS572, the SCSI standard for connecting to the virtual machine or any other appropriate hardware connection interface standard known to those with ordinary skill in the art, including IDE, ATA, and ATAPI. Although, from the perspective of guest OS572, file system calls initiated by such guest OS572to implement file system-related data transfer and control operations appear to be routed to virtual disks575A-575Xfor final execution, in reality, such calls are processed and passed through virtual HBA574to adjunct virtual machine monitors (VMM)5611-561Nthat implement the virtual system support needed to coordinate operation with hypervisor560. In particular, HBA emulator562functionally enables the data transfer and control operations to be correctly handled by hypervisor560which ultimately passes such operations through its various layers to HBA554that connect to storage systems130.

According to embodiments described herein, each VM571has one or more vvols associated therewith and issues IO operations to block device instances of the vvols created by hypervisor560pursuant to “CREATE DEVICE” calls by VM571into hypervisor560. The association between block device names and vvol IDs are maintained in block device database580. IO operations from VMs5712-571Nare received by a SCSI virtualization layer563, which converts them into file IO operations understood by a virtual machine file system (VMFS) driver564. VMFS driver564then converts the file IO operations to block IO operations, and provides the block IO operations to a virtual IO interposer (VIOI)581. IO operations from VM5711, on the other hand, are shown to bypass VMFS driver564and provided directly to VIOI581, signifying that VM5711accesses its block device directly as a raw storage device, e.g., as a database disk, a log disk, a backup archive, and a content repository, in the manner described in U.S. Pat. No. 7,155,558.

Upon receipt of a block IO operation from one of VMs571, VIOI581determines whether or not the block device name specified in the block IO operation is associated with a filtered vvol with reference to block device database580. In the example illustrated inFIG. 5C, the block device name “vmdk” is shown to be associated with a filtered vvol, vvol12-f, whereas the block device names “dbase,” “log,” and “snapn” are not associated with filtered vvols. If the block device name specified in the block IO operation is not associated with a filtered vvol, VIOI581passes the block IO operation to a virtual volume device driver565. In turn, virtual volume device driver565accesses block device database580to reference a mapping between the block device name specified in the IO operation and the PE ID (WWN of PE LUN) and SLLID that define the IO connection path to the vvol associated with the block device name. Other information that is stored in block device database580includes an active bit value for each block device that indicates whether or not the block device is active, and a CIF value.

If the block device name specified in the block IO operation received by VIOI581is associated with a filtered vvol, VIOI581redirects the block IO operation to filter appliance VM583for processing through a virtual machine control interface (VMCI)582. Filter appliance VM583performs one or more filtering operations on the data referenced in the block IO operation. For reads, filter appliance VM583issues a block IO operation to virtual volume device driver565(identifying the block device name as “vmdk-f” in the example) and performs the filtering operations on the read data that is returned, and then passes the filtered read data back to VIOI581through control interface582. For writes, filter appliance VM583transforms the write data that is referenced in the block IO operation by performing filtering operations thereon and issues a block IO operation to virtual volume device driver565(identifying the block device name as “vmdk-f” in the example) to write the transformed data. Then, upon receipt of a write acknowledgement, filter appliance VM583passes the write acknowledgement to VIOI581through control interface582. For both reads and writes, upon receipt of the block IO operation from filter appliance VM583, virtual volume device driver565handles the block IO operation in the same manner as described above when the block IO operation was directly passed to virtual volume device driver565. Also, the target vvol is the same. The difference is in the data that is returned to the VM as read data (filtered data is returned) and written as write data (filtered data is written).

The functions of VMCI582are implemented with memory that is shared between filter appliance VM583and hypervisor560(in particular, VIOI581). Unidirectional queues are configured in the shared memory and each queue has a producer and a consumer. For reads, VIOI581is the producer and filter appliance VM583is the consumer. For writes, filter appliance VM583is the producer and VIOI581is the consumer.

Virtual volume device driver565issues raw block-level IO operations (as received from VIOI581or filter appliance VM583) to data access layer566. Data access layer566includes device access layer567, which applies command queuing and scheduling policies to the raw block-level IO operations, and device driver568for HBA554which formats the raw block-level IO operations in a protocol-compliant format and sends them to HBA554for forwarding to the PEs via an in-band path. In the embodiment where SCSI protocol is used, the vvol information is encoded in the SCSI LUN data field, which is an 8-byte structure, as specified in SAM-5 (SCSI Architecture Model-5). The PE ID is encoded in the first 2 bytes, which is conventionally used for the LUN ID, and the vvol information, in particular the SLLID, is encoded in the SCSI second level LUN ID, utilizing (a portion of) the remaining 6 bytes. As further shown inFIG. 5C, data access layer566also includes an error handling unit569, which functions in the same manner as error handling unit542.

FIG. 5Dis a block diagram of the computer system ofFIG. 5Cthat has been configured to interface with the storage system cluster ofFIG. 2Binstead of the storage system cluster ofFIG. 2A. In this embodiment, data access layer566includes an NFS client585and a device driver586for NIC553. NFS client585maps the block device name to a PE ID (IP address) and SLLID (NFS file handle) corresponding to the block device. This mapping is stored in block device database580as shown inFIG. 5D. It should be noted that the Active and CIF columns are still present but not illustrated in the block device database580shown inFIG. 5D. NFS client585also translates the raw block-level IO operations received from virtual volume device driver565to NFS file-based IO operations. Device driver586for NIC553then formats the NFS file-based IO operations in a protocol-compliant format and sends them to NIC553, along with the NFS handle, for forwarding to one of the PEs.

It should be recognized that the various terms, layers and categorizations used to describe the components inFIGS. 5A-5Dmay be referred to differently without departing from their functionality or the spirit or scope of the invention. For example, VMM561may be considered separate virtualization components between VM571and hypervisor560(which, in such a conception, may itself be considered a virtualization “kernel” component) since there exists a separate VMM for each instantiated VM. Alternatively, each VMM561may be considered to be a component of its corresponding virtual machine since such VMM includes the hardware emulation components for the virtual machine. In such an alternative conception, for example, the conceptual layer described as virtual hardware platform573may be merged with and into VMM561such that virtual host bus adapter574is removed fromFIGS. 5C and 5D(i.e., since its functionality is effectuated by host bus adapter emulator562).

As shown inFIG. 6, VIOI531has several components, including a master VIOI control601, a control channel602, and one or more data channels603. Master VIOI control601is responsible for registering filter appliance513. When a new filter appliance is deployed in computer system101, master VIOI control601registers the new filter appliance in a filter registry that is maintained in OS508. Control channel602provides a communications link to a corresponding control channel612in filter appliance513. Data channels603/613between VIOI531and filter appliance513are established using control channel602/612, and IO data are transmitted over data channels603/613. For reads, data read from a virtual volume are processed in sequence (upwards through the stack) by filters6211. . .621Nand the processed data are transmitted over data channels603/613. For writes, IO data received over data channels603/613are processed in sequence (downwards through the stack) by filters6211. . .621Nand the processed IO data are written to a virtual volume. In one embodiment, filter6211is a compression/decompression filter and filter6212is a cache filter. In another embodiment, filter6211is an encryption/decryption filter and filter6212is a cache filter. In further embodiments, other types of filters may be implemented and in any number including 1. VIOI581is configured in a similar manner as VIOI531to register and communicate with filter appliance VM583, and the details thereof are not described herein.

In the embodiments ofFIGS. 5C and 5D, filter appliance VM583, when successfully added to a host computer, is automatically powered on. When filter appliance VM583successfully boots, a storage rescan is performed on the host computer to discover storage containers accessible by the host computer including a filtered storage container that filter appliance VM583exports. The process of attaching a filter a VM so as to implement a new storage policy for the VM is carried out by logically moving all of the vvols associated with the VM to the filtered storage container exported by filter appliance VM583. This logical move will cause all IO operations targeted for the moved vvols to flow through filter appliance VM583for filter processing by updating block device database in the manner described above for block device database533and block device database580. The detach process is the reverse of the attach process. Given a VM associated with one or more vvols in the filtered storage container exported by filter appliance VM583, when the storage policy for the VM no longer requires the filter processing by filter appliance VM583, the vvols are logically moved from the filtered storage container exported by filter appliance VM583to an unfiltered storage container and the block device database is updated accordingly, as a result of which IO operations targeted for the moved vvols no longer flow through filter appliance VM583.

Interposition of the filter in the IO data path described above allows a third party to design and implement any number of filters in a flexible manner. When multiple filters are provided, they can be stacked one above another in such a way that an upper filter gets to do its IO transformations before passing the transformed IO to the next lower filter in the stack. The completions will appropriately be called in the reverse order.

FIGS. 7 and 8are flow diagrams that each illustrate a method of interposing on an IO operation targeted for virtual volumes, as carried out by a VIOI (e.g., VIOI531or VIOI581) and a filter appliance (e.g., filter appliance513or filter appliance VM583).FIG. 7illustrates such a method for read IO operations, andFIG. 8illustrates such a method for write IO operations. In addition, the filter appliance is assumed to implement multiple filters where the bottom filter is a cache filter.

At step710, the VIOI receives a read IO operation that targets a first vvol. The first vvol may be part of an unfiltered storage container or may be a filtered vvol that was created as a result of a vvol being migrated from an unfiltered storage container to a filtered storage container. At step712, the VIOI determines whether the first vvol is part of an unfiltered storage container or a filtered storage container. If the first vvol is part of an unfiltered storage container, step714is executed where the VIOI passes down its storage stack (e.g., to virtual volume device driver532or565) a read IO operation targeting the first vvol. In response to this, read data is returned, and at step716, the VIOI passes this read data back to the issuer of the read IO operation at step710.

At step712, if the VIOI determines that the first vvol is part of a filtered storage container, the filter appliance at step720issues a read IO operation targeting a second vvol, which is the vvol that was moved from an unfiltered storage container to the filtered storage container. For example, if vvol1 was migrated from an unfiltered storage container to a filtered storage container and the filtered version of vvol1 is known as vvol1-f, any read IO operation targeting vvol1-f would cause a read operation targeting vvol1 to be issued at step720. At step722, the filter appliance performs a first filter operation. It should be recognized that, for reads, the first filter is at the bottom of the filter stack (in this example, the cache filter). Therefore, the first filter operation is determining whether the read data is already cached. If so, the filter appliance terminates the issued read IO operation at step724. If not, the filter appliance waits for read completion. Once the read has completed and the filter appliance obtains the read data from the second vvol at step730, the filter appliance performs further filtering processing by applying the remaining filters on the read data one-by-one, working up the filter stack (step732). After the top filter has been applied to the read data, the filter appliance at step734returns the filtered data as the read data to the issuer of the read IO operation at step710.

At step810, the VIOI receives a write IO operation that targets a first vvol. The first vvol may be part of an unfiltered storage container or may be a filtered vvol that was created as a result of a vvol being migrated from an unfiltered storage container to a filtered storage container. At step812, the VIOI determines whether the first vvol is part of an unfiltered storage container or a filtered storage container. If the first vvol is part of an unfiltered storage container, step814is executed where the VIOI passes down its storage stack (e.g., to virtual volume device driver532or565) a write IO operation targeting the first vvol. In response to this, a write acknowledgement is returned, and at step816, the VIOI passes this write acknowledgement back to the issuer of the write IO operation at step810.

At step812, if the VIOI determines that the first vvol is part of a filtered storage container, the filter appliance begins filter processing on the write data referenced in the write IO operation by applying the filters to the write data one-by-one, working down the filter stack (step820). In the example given herein, the bottom filter is the cache filter, which determines at step822whether the write data transformed by the upper filters is cached. If not, the filter appliance at step824issues a write IO operation to write the write data transformed by the upper filters to a second vvol, which is the vvol that was moved from an unfiltered storage container to the filtered storage container (e.g., vvol1 in the method described above in conjunction withFIG. 7). After step824, the filter appliance waits for a write acknowledgement. Upon receiving it, the filter appliance at step826returns the write acknowledgement to the issuer of the write IO operation at step810. If at step822, the cache filter determines that the write data transformed by the upper filters is cached, step824is skipped and step826is executed.

Although one or more embodiments have been described in some detail for clarity of understanding, it will be apparent that certain changes and modifications may be made within the scope of the claims. For example, SCSI is employed as the protocol for SAN devices and NFS is used as the protocol for NAS devices. Any alternative to the SCSI protocol may be used, such as Fibre Channel, and any alternative to the NFS protocol may be used, such as CIFS (Common Internet File System) protocol. Accordingly, the described embodiments are to be considered as illustrative and not restrictive, and the scope of the claims is not to be limited to details given herein, but may be modified within the scope and equivalents of the claims. In the claims, elements and/or steps do not imply any particular order of operation, unless explicitly stated in the claims.

In addition, while described virtualization methods have generally assumed that virtual machines present interfaces consistent with a particular hardware system, the methods described may be used in conjunction with virtualizations that do not correspond directly to any particular hardware system. Virtualization systems in accordance with the various embodiments, implemented as hosted embodiments, non-hosted embodiments, or as embodiments that tend to blur distinctions between the two, are all envisioned. Furthermore, various virtualization operations may be wholly or partially implemented in hardware. For example, a hardware implementation may employ a look-up table for modification of storage access requests to secure non-disk data.