Abstract:
A storage virtualization system that follows a four-layer hierarchy model, which facilitates the ability to create storage policies to automate complex storage management issues, is provided. The four-layers are a disk pool, Redundant Arrays of Independent Disks (RAID arrays), storage pools and a virtual pool of Virtual Disks (Vdisks). The storage virtualization system creates virtual storage arrays from the RAID arrays and assigns these arrays to storage pools in which all of the arrays have identical RAID levels and underlying chunk sizes representing in abstraction very large arrays. Virtual disks are then created from these pools wherein the abstraction of a storage pool makes it possible to create storage policies for the automatic expansion of virtual disks as they fill with user files.

Description:
CROSS-REFERENCE TO RELATED APPLICATIONS  
       [0001]     This application claims priority to U.S. Provisional Application No. 60/604,195, filed on Aug. 25, 2004, entitled Storage Virtualization, the disclosure of which is hereby incorporated by reference in its entirety. Additionally, the entire disclosures of the present assignee&#39;s following U.S. Provisional Application No. 60/604,359, entitled Remote Replication, filed on the same date as the present application is incorporated herein by reference in its entirety 
     
    
     BACKGROUND  
       [0002]     1. Field of the Invention  
         [0003]     The present invention relates to systems and methods for managing virtual disk storage provided to host computer systems.  
         [0004]     2. Description of Related Art  
         [0005]     Virtual disk storage is relatively new. Typically, virtual disks are created, presented to host computer systems and their capacity is obtained from physical storage resources in, for example, a storage area network.  
         [0006]     In storage area network management, for example, there are a number of challenges facing the industry. For example, in complex multi-vendor, multi-platform environments, storage network management is limited by the methods and capabilities of individual device managers. Without common application languages, customers are greatly limited in their ability to manage a variety of products from a common interface. For instance, a single enterprise may have NT, SOLARIS, AIX, HP-UX and/or other operating systems spread across a network. To that end, the Storage Networking Industry Association (SNIA) has created work groups to address storage management integration. There remains a significant need for improved management systems that can, among other things, facilitate storage area network management.  
         [0007]     While various systems and methods for managing array controllers and other isolated storage subsystems are known, there remains a need for effective systems and methods for representing and managing virtual disks in various systems, such as for example, in storage area networks.  
       SUMMARY  
       [0008]     A storage virtualization system that follows a four-layer hierarchy model, which facilitates the ability to create storage policies to automate complex storage management issues, is provided. The four-layers are a disk pool, Redundant Arrays of Independent Disks (RAID arrays), storage pools and a virtual pool of Virtual Disks (Vdisks). The storage virtualization system creates virtual storage arrays from the RAID arrays and assigns these arrays to storage pools in which all of the arrays have identical RAID levels and underlying chunk sizes representing in abstraction very large arrays. Virtual disks are then created from these pools wherein the abstraction of a storage pool makes it possible to create storage policies for the automatic expansion of virtual disks as they fill with user files. 
     
    
     BRIEF DESCRIPTION OF THE DRAWINGS  
       [0009]      FIG. 1  is a schematic illustration of a storage virtualization system;  
         [0010]      FIG. 2  is a schematic illustration of a virtual disk copy system;  
         [0011]      FIG. 3  is a block diagram of the storage virtualization system;  
         [0012]      FIG. 4  is a schematic illustration of multiple storage pools;  
         [0013]      FIG. 5  is a diagram illustrating a layout of a storage area disk;  
         [0014]      FIG. 6  is a schematic illustration of a virtual disk&#39;s volume access and usage bitmap;  
         [0015]      FIG. 7  is a block diagram illustrating a virtual disk&#39;s storage allocation and address mapping;  
         [0016]      FIG. 8  is a flowchart for Logical Unit number (LUN) mapping;  
         [0017]      FIG. 9  is a flowchart for a procedure of storage allocation during creation of a virtual disk;  
         [0018]      FIG. 10  is a block diagram illustrating an example of Local Unit number (LUN) mapping;  
         [0019]      FIG. 11  is a flowchart for Local Unit number (LUN) masking (access control);  
         [0020]      FIG. 12  is a schematic illustration of Logical Unit (LUN) number mapping and masking;  
         [0021]      FIG. 13  is a table depicting operating system partition and file system interface; and  
         [0022]      FIG. 14  is a flowchart for a procedure of storage allocation when growing a virtual disk. 
     
    
     DETAILED DESCRIPTION  
       [0023]     The key to realizing the benefits of networked storage and enabling users to effectively take advantage of their network storage resources and infrastructure is storage management software that includes virtualization capability. Referring to  FIG. 1  there is shown a schematic illustration of a storage virtualization system  20  that follows a four-layer hierarchy model, which facilitates the ability to create storage policies to automate complex storage management issues. As shown in  FIG. 1  the four-layers are a disk pool  22 , Redundant Arrays of Independent Disks (RAID arrays)  24 , storage pools  26  and a virtual pool of Virtual Disks (Vdisks)  28 .  
         [0024]     The storage virtualization system  20  allows any server or host  32  to see a large repository of available data through by example a fiber channel fabric  30  as though it was directly attached. It allows users to add storage and to dynamically manage storage resources as virtual storage pools instead of managing individual physical disks. The storage virtualization system  20  features enable virtual volumes to be created, expanded, deleted, moved or selectively presented regardless of the underlying storage subsystem. It simplifies storage provisioning thus reducing administrative overhead. Referring to  FIG. 2  the storage virtualization system  20  enables IT professionals to easily expand or create a virtual disk on a per file system basis. If an attached server requires additional storage space, either an existing virtual disk  34  can be expanded, or an additional virtual disk  36  can be created and assigned to the server. The process of adding or expanding virtual disk volumes is non-disruptive with no system downtime.  
         [0025]     Turning now to  FIG. 3  there is shown a block diagram of the storage virtualization system  20  wherein a volume manager or storage area network file system (hereinafter referred to as SANfs)  38  is the foundation of the storage virtualization system  20  and data service. SANfs  38  may be built onto any raw storage devices (eg, RAID storage or hard drive) to provide storage provisioning and advanced data management. The process of creating virtual storage volumes or a storage pool  26  begins with the creation of RAID arrays. These arrays may be formatted as RAID level 0, 1, 3, 4, 5, or 10 (0+1). Referring to  FIG. 4  there is shown a schematic illustration of multiple storage pools  26   a ,  26   b  through  26   n . A storage pool  26  is defined as a concatenation of RAID storage and/or other external storage unit&#39;s  24   a ,  24   b  through  24   n . Each storage pool  26  shares a central cache  40 , boosting the overall host I/O performance. There are 64 terabytes of cache address space allocated to each storage pool  26 , thus each storage pool  26  can dynamically expand up to 64 terabytes. External Storage, such as a hard drive, RAID storage  24 , or any 3 rd  party storage unit, may be added into a storage pool  26  for capacity expansion without interrupting on-going I/O.  
         [0026]     A diagram illustrating a layout of a SANfs  38  on a storage pool  26  is shown in  FIG. 5 . Each storage pool  26  has its own SANfs  48  created for virtualization and data service management  20 . As shown in the diagram each SANfs  48  has a super block  42 , an allocation bitmap  44 , a vnode table  46 , Pad 0   74 , GUI data  78 , payload chunks  52  in predefined size of 512 MB or more and Pad 1   76  ending in an application-defined metadata area  50 . The super block  42  holds SANfs  48  parameters and layout map with its content loaded into memory for quick reference. Therefore the super block  42  contains file system parameters that are used to construct the sanfs layout and vnode table  46 . Most of the parameters are set by the SANfs  38  creation utility based on external storage information. All number values in the super block and vnode are in little endian. The same operating code can handle multiple SANfs  38  with different parameters based on their super block  42  content. The allocation bitmap  44  records free and used chunks in a SANfs  48  wherein one bit represents one chunk. The chunk size is the minimum allocation size in a SANfs  48  with the chunk sizes itself a SANfs parameter. Therefore a SANfs with a chunk size of 512 MB may manage up to two (2) TB capacity (512*8*512 MB) and for a chunk size of two (2) GB, the SANfs  38  may manage up to eight (8) TB capacity (512*8*2 GB.) SANfs  48  may resize online by adjusting the allocation bitmap  44  and super block parameters  42  wherein each SANfs  38  may present up to 512 volumes.  
         [0027]     The allocation bitmap  44  is always 512 bytes in size. The allocation bitmap  44  is used to monitor the amount of free space currently on a storage pool  26 . The free space is monitored in chucks of 512 MB. The maximum number of chunks is 4096, with chunk size of 16 GB, it manages up to 64TB storage. The bitmap  44  is constantly updated to reflect the space that has been allocated or freed on a storage pool. The vnode table  46  is used to record and manage virtual disks or volumes that have been created on a storage pool and is the central metadata repository for the volumes. There are 512 vnodes  28  in a vnode table  46  wherein each vnode is 4 KB in size (8 blocks), thus a vnode table is 512×4 KB in size (4096 blocks). The Pad 0   74  locations is reserved for future use with pad 1   76  and the sanfs metadata backup area  50  being used as data chunk during storage pool  26  expansions. The metadata backup area  50  is always stored at the end of a storage pool  26 . A sanfs expansion utility program relocates the metadata backup  50  to the end, and re-calculates the size of pad 1   76  and the last_data_blk  80 . Lastly, the metadata backup area  50  is comprised of the super block  42 , allocation bitmap  44 , and the vnode table  46 . Thus, two copies of the metadata are maintained, one at the beginning and one at the end of a storage pool  26 . The metadata can be recovered if one copy is lost or corrupted.  
         [0028]     Referring to  FIG. 6  there is shown a schematic illustrating a virtual disk volume access  80  and usage bitmap  82 . A volume  34  is a logical storage container, which may span multiple SANfs chunks, continuously or discretely. Referring to  FIG. 3 , the servers or hosts  32  see the storage virtualization volumes as physical storage devices. A volume  34  may grow or shrink online, though the volume shrink is normally disabled. The volume structure and properties are described by Vnode  26  and stored in the SANfs  38  Vnode table area  46 . Each volume  34  may be accessed on two controllers  84  and  86  at specified ports as a single image. This allows for I/O path redundancy. Turning to  FIG. 6  each volume  34  has a reserved 64 MB area at the beginning to store volume specific metadata, such as the volume&#39;s usage bitmap  82 . Each volume  34  has the usage bitmap  82  to record if an area in its payload data has ever been written. A volume&#39;s payload data is virtually partitioned into 1 MB chunks  88  numbered as chunk  0  . . . N−1. If there is a write to chunk m, then the bit m in the usage bitmap  82  will be set. The volume usage bitmap facilitates fast data copy during volume mirroring and replication, i.e., only used data chunks in the source volume need to be copied.  
         [0029]     Referring to  FIG. 7  there is shown a block diagram illustrating a virtual disk&#39;s storage allocation and address mapping. Volume storage allocation uses extent-based capacity management where an extent  92  is defined as a group of physically continuous chunks in a SANfs. Each vdisk  34  has an extent table  90  stored in its Vnode  28  to record volume storage allocation and direct vdisk  34  accesses to the storage pools  26  access. Vdisk storage allocation utilizes an extent-based capacity management scheme to obtain large continuous chunks for a vdisk and decrease SANfs fragment. A vdisk may have multiple extents. A Vnode  28  and its in-core structure have following functional components: volume properties, such as size, type, serial number, internal LUN, and host interfaces to define the volume presentation to host and the extent allocation table  90  to map logical block address to physical block address. A vdisk  34  may have multiple extents  92 .  
         [0030]     Referring once again to  FIG. 3  the Host  32  IO requests and internal volume manipulation are handled by the IO manager  56  utilizing the storage virtualization system  20 . The IO manager  56  initiates data movement based on the volume type and its associated data services. The volume type includes: normal volume, local mirror volume, snapshot volume and remote replication volume. The data services associated with a normal volume includes local mirror  62 , snapshot  64 , remote replication  66 , volume copy  68  and volume rollback  70 . For a Host  32  IO to a normal volume operation, the IO manager  56  translates the Host  32  IO logical address into the SANfs  38  physical address. As the SANfs  38  minimum extent size is 512 MB, most of the host IO will reside in one extent and the IO manager  56  only needs to initiate one physical IO to the extent  92 . For the cross-extent host IO, the IO manager  56  will initiate two physical IOs to the two extents. Given the fact that most volumes have only one extent  92  and the cross-extent host IO is rare, the IO translation overhead is trivial. There is almost no performance penalty in the virtualization layer.  
         [0031]     For a write to normal volume with local mirror  62  attached operations, the IO manger  56  will also copy the write data to the local mirror volume. As the copy happens inside the cache  40 , for burst-write, the cost is just an extra memory move. For a write to normal volume with remote replication  66  attached operations, the IO manager  56  will also send the write data to the replication channels. In synchronized replication mode, the IO manager  56  will wait the write ACK from remote site before acknowledging the Host  32  the write completion, thus incurring larger latency. In asynchronized replication mode, the IO manager  56  will acknowledge host the write complication once the data has been written to the local volume, and schedule the actual replication process into background.  
         [0032]     For a write to normal volume with snapshot  64  attached operations, the snapshot  64  uses the copy-on-write (COW) technique to instantly create snapshot with adaptive and automatic storage allocation. The initial COW storage allocated is about 5% to 10% of the source volume capacity. When COW data grows to exceed the current COW storage capacity, the IO manager  56  will automatically allocate more SANfs  38  chunks to the COW storage. For this kind of write, the IO manager  56  will first do the copy-on-write data movement if needed, then move the write data to the source volume. For Data movement during volume copy  68  operations, a volume copy operation is used to clone volume locally or to remote sites. Any type of volumes may be cloned. For example, by cloning a snapshot volume, a full set of point in time (PIT) data will be generated for testing or achieving purpose. During the volume clone process, the IO manager  56  reads from the source volume and writes to the destination volume. Lastly, for data movement during volume rollback  70  operations, when a source volume has snapshots, or suspended local mirror  62  or remote replication  66 , a user may choose the volume rollback operation to bring back the source volume content to a previous state. During the rollback operation, the IO manager  56  selectively reads the data from the reference volume and patch to the source volume.  
         [0033]     Referring back to  FIG. 3 , the Logical Unit numbering (LUN) mapping and masking  58  occurs just below the Host  32  level and offers volume presentation and access control. The storage virtualization system  20  may present up to 128 volumes per host port to the storage clients. Each volume is assigned an unique internal LUN number, called ilun (0 . . . 127), per host interface. The LUN mapping  58  allows a Host  32  to see a volume at the host designated LUN address (called hlun). A Host is identified by its HBA&#39;s WWN, called hWWN. The SANfs maintains the LUN mapping table per host port.  FIG. 10  is a block diagram illustrating an example of Local Unit number (LUN) mapping illustrating a table  144  having three components and two keys. The three components are hWWN, hlun, ilun. KEY h  is generated by hashing the related hWWN and hlun together. KEY i  is generated by hashing the related hWWN and ilun together.  
         [0034]      FIG. 8  is a flowchart for Logical Unit number (LUN) mapping  58  wherein when an request  94  comes in it always carries the hWWN and hlun to tell from which host this IO comes from and at what LUN address. The LUN mapping code calculates the key from the incoming hWWN and hlun by the same hash function, and looks up  96  the LUN mapping table in the following sequences: 
        1. If the key matches a KEY h  in the table  144  (LMAP T 1 ), direct the IO request to the volume whose internal LUN has the value of the associated ilun  98 , otherwise go to 2.     2. If the key matches a KEY i  in the table  146  (LMAP T 2 ), reject the IO request, otherwise go to 3.     3. Direct the IO request to the volume whose internal LUN equals to the hlun  102 . This means there is no LUN mapping on the &lt;hWWN, hlun&gt;.          
         [0038]     For example, with LUN mapping properly set up, Host A  162  can view volume  0  to volume  5  as LUN  0  to LUN  5 , Host B  164  can view volume  6  to volume  10  also as LUN  0  to LUN  5  instead of as LUN  6  to LUN  10 . LUN masking controls which hosts can see a volume  160 . Each volume can store up to 64 host HBA WWNs, from which the accesses are allowed. When LUN masking is turned on, only those IO requests from the specified hosts will be honored. As shown in the flowchart of  FIG. 8 , path A is for normal LUN mapping access. Path C is to block access to a vdisk which has a LUN mapping address different from the hLUN  94  and path B is for access without LUN mapping  108 .  
         [0039]      FIG. 9  is a flowchart for a procedure of storage allocation during creation of a virtual disk wherein a request to create a vdisk of X GB on SANfs Y  108 . If X&gt;free space on Y  112  then the creation failed  110 . If not then retrieve the allocation bitmap of SANfs Y  114  and scan the bitmap from the beginning to find the first free extent, Z GB in size  116 . If X&lt;=Z  118  then allocate this extent with X GB capacity to the vdisk and update allocation bitmap  124  and the creation was a success  126 . If X=&gt;Z then check to see if X&lt;=8*Z  120  and if yes allocate this extent with Z GB capacity to the vdisk, and update allocation bitmap  122 . Perform the operation X=X−Z  130  and continue to search the bitmap to find the next free extent  134 . If X=&gt;8*Z then this extent is too small for the vdisk and continue to search next free extent  132 . Was a free extent found  136 . If yes, assume Z GB is the size of this extent  140  and go to step  118 . If no, cannot create the vdisk and release previous allocated extents  138  wherein the expansion failed  142 .  
         [0040]      FIG. 10  is a block diagram illustrating an example of Local Unit number (LUN) mapping interface. This interface is shared by all vdisks on a storage enclosure to present a vdisk to a host at user specified LUN address. This user specified LUN address is called hLUN. The storage virtualization system may present one vdisk to multiple hosts at different or same hLUNs and also enforces that one host can only access a vdisk through an unique hLUN on that host. Each vdisk has an unique internal LUN address. This internal LUN address per vdisk is called iLUN. The LUN presentation function is to direct an IO request of &lt;WWN, hLUN&gt; to a corresponding vdisk of iLUN. &lt;WWN, HLUN&gt; represents an IO request from a host with WWN to this host perceived LUN address of hLUN. There are two tables to facilitate the LUN presentation, also known as LUN mapping. This first table is called LMAP T 1   144 , and the second table is called LMAP T 2   146 , as shown in figure x. The LMAP T 1   144  table stores user specified LUN mapping parameters, i.e., the content of LMAP T 1   144  is from user input. The LMAP T 2   146  is deduced from LMAP T 1   144 . As LUN mapping translation occurs for every I/O request, a hash function is used for quick lookup on LMAP T 1   144  and LMAP T 2   146 . The hash key for LMAP T 1   144  is &lt;wwn, hlun&gt;, so is &lt;wwn, ilun&gt; for LMAP T 2   146 .  
         [0041]      FIG. 11  is a flowchart for a procedure of LUN masking (access control). This interface enforces the LUN access control to allow on specified hosts to access a vdisk. A host is represented by the WWNs of its fibre channel adapters. The vnode interface can store up to 64 WWNs to support access control up to 64 hosts. The access control can be turned on and off per vdisk. If a vdisk&#39;s control is off, any host can access the vdisk. Referring to  FIG. 11  the I/O request to vdisk X from host Y of WWNi  148 . Check the X&#39;s access control  150 . If the X&#39;s access control is not on then grant access  152 . If the X&#39;s access control is on then check  156  if WWNi is in X&#39;s WWN table and if it is grant access  158  and if not deny access  154 .  
         [0042]      FIG. 12  is a schematic illustration of Logical Unit (LUN) number mapping and masking. The LUN Access Control Interface  161  controls which hosts  162  and  164  for example may access the which volumes  160 . The host is represented by the WWNs of its fibre channel adapters. Access control can be turned on and off per volume. If access control is turned off, all hosts can access the volume  160 . Referring to  FIG. 13  there is shown a table  166  depicting operating system (OS) partition and file system interface. The storage virtualization system can detect if OS partitions  168  exist on a vdisk by scanning the front area of the vdisk. If OS partitions  168  are detected, it will scan each partition to collect file system information  170  on a partition. The collected partition and file system information is stored in the vnode&#39;s file system interface as depicted in table  166 . Up to eight partitions per vdisk may be supported. A warning threshold  180  is provided which is a user specified percentage of file system used space over its total capacity  176 . Once the threshold  180  is exceeded, the storage virtualization system will notify the user to grow the vdisk and file system capacity. Date services can operate on a specific partition by using the partition start address  172  and partition length  174 .  
         [0043]     Referring now to  FIG. 14  there is shown a flowchart for a procedure of translating host IO request to physical storage. First, a Host request access (Read/Write) is received with X blocks starting at block number Y on a vdisk  182 . Then find on which extent(s) the stripe &lt;Y . . . Y+X−1&gt; resides by lookup on the extent table  184  to find the containing extent  186 . If no extent is found then the translation failed and access is denied  188 . If only one extent  190  is found wherein this stripe wholly resides, say it&#39;s Ei  192 . Then set Yp=Y+pool_start_address of Ei, wherein Yp is Ei&#39;s start address on the pool  196  and access the physical stripe on the pool as &lt;Yp . . . Yp+X−1&gt; 198 . The translation is now done  204 . If more than one extent is found  190  then this stripe overrides two extents, say they are Ei and Ej and assume X1 blocks resides in Ei, X2 blocks in Ej, X=X1+× 194 . Then set Yp=Y+pool_start_address of Ei and Yq=pool_start_address of Ej, wherein Yp is Ei&#39;s start address on the pool and Yq is Ej&#39;s start address on the pool  200 . Next, access the physical stripes on the pool as &lt;Yp . . . Yp+X1−1&gt; and &lt;Yq . . . Yq+X2−1&gt; 202  and the translation is done.  
         [0044]     As described above SAN servers share the virtualized storage pool that is presented by storage virtualization. Data is not restricted to a certain hard disk—it can reside in any virtual drive. Through the SANfs software, an IT administrator can easily and efficiently allocate the right amount of storage to each server (LUN masking) based on the needs of users and applications. The virtualization system may also present a virtual disk that is mapped to a host LUN or a server (LUN mapping). Virtualization system storage allocation is a flexible, intelligent, and non-disruptive storage provisioning process. Under the control of storage virtualization, storage resources are consolidated, optimized and used to their fullest extent versus traditional non-SAN environments which only utilize about half of their available storage capacity. Consolidation of storage resources also results in reduced costs in overhead, allowing effective data storage management with less manpower.