System and method for pre-provisioning storage in a networked environment

System and method for pre-provisioning data storage in a network storage environment. Embodiments may pre-provision more storage than needed and make the spare storage available to two or more hosts in the storage network. Spare storage may be pre-provisioned as part of a pool or pools, and any one of the hosts on the storage network may claim spare storage out of the pool(s) to which it has access on an as-needed basis. Embodiments remove the data center's change control process from the critical path in provisioning additional storage, and do not result in the generation of I/O errors on writes to storage if the pool of spare storage available to a host is exhausted. In one embodiment, a coordinating service on the storage network may coordinate access to the pool of spare storage by the hosts on the storage network.

BACKGROUND OF THE INVENTION

1. Field of the Invention

This invention is related to the field of computer systems and, more particularly, to network data storage systems.

2. Description of the Related Art

In the past, large organizations relied heavily on parallel SCSI technology to provide the performance required for their enterprise data storage needs. More recently, organizations are recognizing that the restrictions imposed by SCSI architecture may be too costly for SCSI to continue as a viable solution. Such restrictions include the following:SCSI disk arrays must be located no more than 25 meters from the host server;The parallel SCSI bus is susceptible to data errors resulting from slight timing discrepancies or improper port termination; andSCSI array servicing frequently requires downtime for every disk in the array.

One solution has been to create technology that enables storage arrays to reside directly on the network, where disk accesses may be made directly rather than through the server's SCSI connection. This network-attached storage (NAS) model eliminates SCSI's restrictive cable distance, signal timing, and termination requirements. However, it adds a significant load to the network, which frequently is already starved for bandwidth. Gigabit Ethernet technology only alleviates this bottleneck for the short term, so a more elegant solution is desirable.

The storage area network (SAN) model places storage on its own dedicated network, removing data storage from both the server-to-disk SCSI bus and the main user network. This dedicated network most commonly uses Fibre Channel technology, a versatile, high-speed transport. The SAN includes one or more hosts that provide a point of interface with LAN users, as well as (in the case of large SANs) one or more fabric switches, SAN hubs and other devices to accommodate a large number of storage devices. The hardware (e.g. fabric switches, hubs, bridges, routers, cables, etc.) that connects workstations and servers to storage devices in a SAN is referred to as a “fabric.” The SAN fabric may enable server-to-storage device connectivity through Fibre Channel switching technology to a wide range of servers and storage devices.

SAN management conventionally includes provisioning of and control over host access to individual LUNs (Logical Unit Numbers) within an array or other collection of potentially heterogeneous storage devices. SANs may include storage devices and systems from various storage array providers, for example Hitachi Data Systems, Hewlett Packard and EMC. Ensuring that SAN applications have the required storage resources may include providing secure storage from storage devices (e.g. disk arrays, tape backup devices, etc.) to hosts within the SAN.

A LUN is the SCSI (Small Computer System Interface) identifier of a logical unit within a target, the system component that receives a SCSI I/O command. A logical unit is an entity within a SCSI target that executes I/O commands. SCSI I/O commands are sent to a target and executed by a logical unit within that target. A SCSI physical disk may have a single logical unit, or alternatively may have more than one logical unit. Tape drives and array controllers may incorporate multiple logical units to which I/O commands can be addressed. Each logical unit exported by an array controller corresponds to a virtual disk.

LUN binding refers to the creation of access paths between an addressable unit (which may also be referred to as an AddrUnit, an AU, a unit, a volume, a logical unit, a logical disk, or a logical device) within a disk array and a port on the array. Masking, or LUN masking, may be used to refer to the process of enabling access to a particular Addressable Unit (AU) of storage for a host on the SAN.

FIG. 1illustrates LUN binding. In the LUN binding process, an AU108is bound to a specified array port106(e.g. array port106A or106B) in a specified storage device104(e.g. a storage system/disk array). This results in the creation of a LUN102. AUs108A,108B,108C, and108D are storage volumes built out of one or more physical discs within the storage device104. Array ports106A and106B may be connected to a SAN fabric, and function as SCSI targets behind which the AUs108bound to those ports1066are visible.

“LUN” is the term for the access path itself between an AU and an array port, so LUN binding is actually the process of creating LUNs. However, a LUN is also frequently identified with the AU behind it and treated as though it had the properties of that AU. For the sake of convenience, a LUN may be thought of as being the equivalent of the AU it represents. Note, however, that two different LUNs may represent two different paths to a single volume. A LUN may be bound to one or more array ports. A LUN may be bound to multiple array ports, for example, for failover, switching from one array port to another array port if a problem occurs.

FIG. 2illustrates LUN masking in a SAN. LUN masking is a security operation that indicates that a particular host120(e.g. host120A or120B), HBA (Host Bus Adapter)122(e.g. HBA122A or122B), or HBA port124(e.g. HBA port124A or124B) is able to communicate with a particular LUN102. In the LUN masking process, a bound AU108(e.g. AU108A,108B,108C or108D) may be masked to a specified HBA port124, HBA122, or host120(e.g. all HBAs on the host) through a specified array port106in a specified storage device104. When an array LUN102is masked, an entry is added to the Access Control List (ACL)110(e.g. ACL110A,110B,110C,110D, or110E) for that LUN102. Each ACL110includes the World Wide Name of each HBA port124that has permission to use that access path—that is, to access that AU108through the particular array port106represented by the LUN102.

LUN masking may be thought of as the removal of a mask between an AU and a host to allow the host to communicate with the LUN. The default behavior of the storage device may be to prohibit all access to LUNs unless a host has explicit permission to view the LUNs. The default behavior may depend on the array model and, in some cases, the software used to create the AU.

The storage management process of allocating units of storage (e.g., AUs on storage devices) to storage consumers (e.g., host systems, or hosts) in a storage network may be referred to as storage provisioning. In a storage network, one or more hosts are each provisioned (granted access to) portions of the storage to meet their current storage requirements. However, storage requirements may, and in many storage networks tend to, increase over time. Rather than purchasing and installing storage on an as-needed basis, a conventional practice in storage management is to pre-provision storage, to purchase and install extra storage in advance to meet anticipated increases in storage requirements of a particular host for at least the near future.

There are conventional mechanisms for provisioning units of extra storage to hosts in a storage network to meet the storage requirements of the hosts as they increase to require more storage. One conventional provisioning mechanism is to purchase and install one or more storage devices (e.g., disk arrays) that provide the required storage as well as extra storage, create a set of LUNs for accessing AUs of the storage to meet the current storage requirements, and allocate the LUNs to the host system. When a host needs additional storage, one or more allocated LUNs may then be used for accessing one or more spare AUs on the storage allocated to the host. Note that, in order for the host to gain access to the newly-allocated storage, the SAN may have to be reconfigured, and the host itself may have to be reconfigured and rebooted. Alternatively, one or more spare LUNs may be preconfigured for accessing the AUs of the extra storage. When a host needs more storage, the storage network and/or the host itself may need to be reconfigured to give the host access to one or more of the spare LUNs.

Storage networks may be fragile, and such a dynamic reconfiguration of the SAN may cause problems elsewhere. Reconfiguring the SAN to give a host access to a LUN may involve configuration of ports on the storage device, switches in between the host and the storage device, and the host itself. The host system may need to be reconfigured and/or rebooted to see the newly-allocated LUN(s). Because of these and other issues, many data centers do not allow this mechanism of dynamic reconfiguration of storage networks. In some data centers, requests for additional storage are required to go through a change control process that may take days or even weeks. Reconfiguration of the storage network to allocate units of spare storage to hosts may be limited by the change control process to periodic maintenance or change windows. This makes it difficult and time-consuming to obtain additional storage when needed, and has led to other provisioning solutions such as thin provisioning.

Another conventional provisioning mechanism is referred to as thin provisioning. Thin provisioning is a provisioning mechanism in which virtual LUNs are over-provisioned to the hosts. In thin provisioning, more storage is provisioned through the virtual LUNs than is actually available. The actual extra storage that is available to be allocated to the virtual LUNs may be referred to as the backing store. For example, in thin provisioning, five hosts may each be assigned virtual LUNs for accessing a terabyte of storage. Thus, five terabytes of storage have been assigned, but the backing store may actually only have one terabyte of actual storage space available. As storage is consumed by the hosts, the array allocates the backing store to the virtual LUNs when needed. Thin provisioning is an “allocate on demand” mechanism. Using thin provisioning, the SAN typically does not have to be reconfigured for the hosts to access additional storage. Thin provisioning thus may help avoid at least some of the configuration problems encountered with other provisioning mechanisms as described above.

However, in a storage network using thin provisioning, if the array ever runs out of real storage (the backing store), the storage network may start failing I/O write operations, and hosts and applications on hosts using that array are typically not prepared to handle these types of I/O errors gracefully. As storage is allocated from the backing store, new storage may have to be installed in the storage network. If the backing store is exhausted, the hosts would think they have storage space available that is not actually available. I/O writes may be blocked or I/O errors may be generated when the hosts attempt to write to the storage. Many systems and applications do not expect to get I/O errors on writes to the storage that they think they actually have, and there are generally no mechanisms for gracefully recovering from these types of errors on writes to storage. A host may have the storage mirrored, and so may try to write to the other mirror, which also does not have storage, so that I/O attempt fails as well.

As an example, a typical file system may attempt to write some data, and get an I/O error on the write if the backing store has been exhausted. For example, the data may be a critical piece of file system metadata. After receiving the I/O error on the write, the file system may not know what is actually on the disk; the file system data may all be corrupted.

As another example, if a database writes into its log and attempts to write back into its table spaces, and then receives I/O errors because backing store has been exhausted, then the database may crash and corrupt itself. Databases are typically not prepared to deal with I/O errors on writes to storage. The database application may record the I/O errors, eventually filling up its error log, and then just halt. This is a difficult type of failure from which to recover.

SUMMARY

Embodiments of a system and method for pre-provisioning data storage in a network storage environment are described. In embodiments of the pre-provisioning mechanism, one or more pools of spare storage are created, and the storage network may be configured so that access to the spare storage is shared among multiple systems (e.g., hosts). When a host needs additional storage, one or more units (e.g., Logical Unit Numbers (LUNs)) of spare storage to which the host has access may be selected from one of the spare pools. The selected storage may then removed from the pool for exclusive access by the host so that other hosts that are configured on the storage network to access that storage cannot use it. In one embodiment, a change control request may be generated to reconfigure the storage network to remove access to units of storage to which a host has obtained exclusive access from any other hosts with access to the selected storage. Since the selected storage has been removed from the pool, other hosts that have access to the storage may not obtain exclusive access to the selected storage while the change control request is pending.

Embodiments of the pre-provisioning mechanism may pre-provision more storage than needed and make the spare storage available to two or more hosts in the storage network. Spare storage may be pre-provisioned as part of a pool or pools, and any one of the hosts on the storage network may claim spare storage out of the pool(s) to which it has access on an as-needed basis. Note that a host does not have to claim an entire storage device or disk from the pool, but may claim one or more portions, or AUs, of a storage device or disk. Using embodiments, spare storage may be made more widely available to hosts on the storage network than in conventional storage networks.

In one embodiment, one or more pools of spare LUNs may be pre-provisioned to host systems or servers on a storage network. In pre-provisioning, each LUN may be masked to two or more of the hosts, giving each of the hosts access to the LUN. When a unit of additional storage (represented by a LUN) is needed by a host, the host may obtain a spare LUN to which the host has access from one of the pools of spare LUNs, the spare LUN may be removed from the pool, and the host may begin using the storage represented by the LUN. A change control request may then be generated to reconfigure the storage network during a change window to disallow access to that LUN from the other hosts that have access to the LUN.

Embodiments of the pre-provisioning mechanism remove the data center's change control process from the critical path in provisioning additional storage. There is no need to wait for a “change window” to reconfigure the storage network to grant access to additional storage. Pre-provisioned LUNs may be removed from the pools at any time, as needed, for use by the hosts, while reconfiguration of the storage network to remove access to the removed LUNs from other hosts may be scheduled at the data center's convenience. Unlike thin provisioning, embodiments of the pre-provisioning mechanism do not result in the generation of I/O errors on writes to storage if the pool of spare storage available to a host is exhausted. If the pool of spare storage available to a host are exhausted, and an application on the host requires more storage, the error generated would be an allocation error, and not a write error.

In one embodiment, a coordinating service on the storage network may coordinate access to the pool of spare storage by the hosts on the storage network. Each host in the storage network that may make use of the pool of spare storage may be configured to communicate with the coordinating service to obtain spare storage from the pool(s). In one embodiment, there may be a central coordinating service or management server for each pool of spare LUNs that hosts with access to that pool may communicate with to gain exclusive access to a LUN in the pool. Alternatively, there may be a single coordinating service that manages access by hosts to all pools of spare LUNs in a storage network.

DETAILED DESCRIPTION OF EMBODIMENTS

Embodiments of a system and method for pre-provisioning data storage in a network storage environment are described. In embodiments of the pre-provisioning mechanism, one or more pools of spare storage may be created, and the storage network may be configured so that access to the spare storage may be shared among multiple systems (e.g., hosts). When a host needs additional storage, one or more units (e.g., Logical Unit Numbers (LUNs)) of unused storage to which the host has access may be selected from one of the spare pools. The selected storage may then be removed from the pool for exclusive access by the host. In one embodiment, a change control request may be submitted to reconfigure the storage network to remove access to the selected storage from any other hosts with access to the selected storage. Since the selected storage has been removed from the pool, other hosts that have access to the storage may not select or use the selected storage while the change control request is pending.

Embodiments of the pre-provisioning mechanism may pre-provision more storage than needed and make the spare storage available to two or more hosts in the storage network. Spare storage may be pre-provisioned as part of a pool or pools, and any one of the hosts on the storage network may claim spare storage out of the pool(s) to which it has access on an as-needed basis. Note that a host does not have to claim an entire storage device or disk from the pool, but may claim one or more portions, or AUs, of a storage device or disk. Using embodiments, spare storage may be made more widely available to hosts on the storage network through the pool(s) of spare storage than in conventional storage networks.

In one embodiment, one or more pools of spare LUNs may be pre-provisioned to host systems or servers on a storage network. In pre-provisioning, each LUN may be masked to two or more of the hosts, giving each of the hosts access to the LUN. When a unit of additional storage (represented by a LUN) is needed by a host, the host may obtain a spare LUN to which the host has access from one of the pools of spare LUNs, the spare LUN may be removed from the pool, and the host may then begin using the storage represented by the LUN. A change control request may be generated to reconfigure the storage network during a change window to disallow access to that LUN from the other hosts that have access to the LUN.

While embodiments of the pre-provisioning mechanism are generally described herein as being used by hosts on a storage network to obtain additional storage from one or more pools of pre-provisioned LUNs, note that various applications on hosts or other systems that use the storage network may generate demand for additional storage that may be fulfilled through the pre-provisioning mechanism. Each host may host one or more instances of various applications, services, processes, etc. that may consume storage space in the storage network. In addition, other systems, such as client systems, may also include applications, services, processes, etc. that may consume storage space in the storage network. Any of these consumers of storage space may require additional storage on the storage network, and the need for additional storage may be met from pools of spare LUNs according to embodiments of the pre-provisioning mechanism as described herein.

Embodiments of the pre-provisioning mechanism remove the data center's change control process from the critical path in provisioning additional storage. There is no need to wait for a “change window” to reconfigure the storage network to grant access to additional storage. Pre-provisioned LUNs may be removed from the pools at any time, as needed, for use by the hosts, while reconfiguration of the storage network to remove access to the removed LUNs from other hosts may be scheduled at the data center's convenience.

Unlike thin provisioning, embodiments of the pre-provisioning mechanism do not result in the generation of I/O errors on writes to storage if the pool of spare storage available to a host is exhausted. If the pool of spare storage available to a host are exhausted, and an application on the host requires more storage, the error generated would be an allocation error, and not a write error. In other words, the error would be generated when a request for more storage was made and not when a write to storage was made. Allocation errors typically do not cause severe problems for applications that are difficult to recover from as do I/O write errors. Applications are typically configured to handle failures on storage allocation gracefully.

In one embodiment of the pre-provisioning mechanism, coordinating software, which may be referred to herein as a coordinating service, on a host or hosts or elsewhere on the storage network may coordinate access to the pool of spare storage by the hosts on the storage network. Each host in the storage network that may make use of the pool of spare storage is configured to communicate with the coordinating service. If a host needs to acquire more storage, the host communicates with the coordinating service to obtain spare storage from the pool(s). In one embodiment, there may be a central coordinating service or management server for each pool of spare LUNs that hosts with access to that pool may communicate with to gain exclusive access to a LUN in the pool. Alternatively, there may be a single coordinating service that manages access by hosts to all pools of spare LUNs in a storage network. In one embodiment, a LUN to which a host has been granted exclusive access may be “marked” with an indication that the LUN has been assigned so that no other hosts with access to the LUN are allowed to use the LUN. In one embodiment, a form of device reservation such as SCSI-3 persistent reservation may be used to mark the LUN. In one embodiment, a label may be written on the LUN to mark the LUN. Other embodiments may use other mechanisms to mark the LUN, or combinations thereof. As mentioned above, a change control request for removal of access to the LUN by other hosts may be submitted, and the removal of access to the LUN may then be performed during a “change window” for the storage network in the data center or at some other time. In one embodiment, a change control request may be automatically generated by the coordinating service when a host requests and is granted exclusive access to a LUN in a pool.

The following is an example of an embodiment of the pre-provisioning mechanism managing spare storage for applications in a storage network. A volume manager is used as an example of an application that may consume storage. A volume manager may include functionality to virtualize physical storage accessible by applications running on host systems. As used herein, the term “volume manager” broadly refers to host software that selects and combines storage space (e.g., LUNs) from more than one physical storage device into a logical volume. A volume manager may support various specific storage management functionality, such as various levels of RAID functionality, including data mirroring and striping. Conventionally, a volume manager is configured with a dedicated pool of storage for its exclusive use. This dedicated pool of storage is a subset of all the storage available on the storage network. In addition, the dedicated pool of storage may be a subset of all the storage that the system (e.g., host) on which the volume manager is implemented can “see”; e.g., the storage that the system has been configured to access through the fabric of the storage network. Other applications on the system or on other systems may also be configured with their own dedicated pools of storage on the storage network. Some of the storage on the storage network is typically “spare” storage that is not allocated to any particular application, and at least some of this spare storage may not be configured to be accessed by systems (e.g., hosts) on the storage network. In addition, some of the storage that is accessible by the system on which the volume manager is implemented may be spare storage. As the volume manager consumes storage space, it uses storage from its dedicated pool. If the volume manager needs additional storage, it cannot use storage on the storage network that is not in its dedicated pool. Some of that other storage may be unused “spare” storage that is accessible by the system, but the volume manager does not know what the storage not in its dedicated pool may be being used for, so cannot determine what of that storage is used and what is unused. In addition, as previously mentioned, some spare storage on the storage network may not be configured for access by the system on which the volume manager is implemented.

In one embodiment of the pre-provisioning mechanism, at least some of the spare storage on the storage network may be placed into one or more pools of spare storage, with each pool including units of storage (e.g., LUNs) each pre-provisioned for access by two or more systems (e.g., hosts) on the storage network. Thus, two or more systems on the storage network, and applications on those systems, may be pre-provisioned to access the LUNs in the pool(s) of spare storage. In one embodiment, one or more coordinating services may be implemented on one or more of the hosts on the storage network. For this example, for simplicity, it will be assumed that there is one coordinating service and one pool of spare storage (LUNs) access to which is controlled through the coordinating service. The coordinating service may track which LUNs in the pool of spare storage are available, which systems are configured to access which LUNs, and which LUNs have been assigned to particular systems and are thus not available to other systems.

The volume manager may be configured to communicate with the coordinating service. Note that the volume manager may be initially configured to use a dedicated pool of storage. If the volume manager needs additional storage beyond what is available in its dedicated pool, the volume manager requests one or more additional units of storage (LUNs) from the coordinating service. To meet the request, the coordinating service may determine which of the LUNs in the pool of spare storage are configured for access by the system on which the volume manager is implemented, and which of those LUNs are available to meet the request (i.e., which of those LUNs have not been previously granted to meet other requests, and thus are no longer available). The coordinating service may then inform the volume manager as to which LUNs in the pool that the volume manager may use. The coordinating service may then “mark” the LUNs as having been granted for exclusive access to the volume manager so that the LUNs are not available to other applications and systems that have access to them. In one embodiment, a form of device reservation such as SCSI-3 persistent reservation may be used to mark the LUN. In one embodiment, a label may be written on the LUN to mark the LUN. Other embodiments may use other mechanisms to mark the LUN, or combinations thereof. In one embodiment, a change control request may be generated so that access to the LUNs to which the volume manager has been granted can be removed from other systems by reconfiguring the storage network, for example at the next change window. In one embodiment, the change control request may be automatically generated.

In the above example, an embodiment is described where an application (e.g., a volume manager) interacts directly with the coordinating service to gain exclusive access to one or more units of spare storage in the pool. In another embodiment, instead of having an application directly communicate with the coordinating service to obtain units of spare storage from the pool, when an application (e.g., a volume manager) needs additional storage, an administrator of the system may interact with the coordinating service through a user interface (e.g. a command line or graphical user interface) to inform the coordinating service as to what units of storage (e.g., LUNs) the system is configured to access, and to request one or more of the units of storage (e.g., LUNs) be assigned for exclusive access by the application. The coordinating service may determine which of the LUNs accessible by the system (and in the pool of spare storage) are “free”, i.e. have not been given to another system or application for exclusive access, and return this information to the system administrator. The volume manager may then be configured to use this additional storage. The coordinating service records that the LUNs assigned to the volume manager are no longer available to be assigned to other applications or systems. Again, a change control request may be generated so that the storage network can be reconfigured to remove access to the LUNs from other systems, for example at a next change window.

In one embodiment, a combination of the two methods described above may be used. In other words, some applications may be configured to request exclusive access to spare LUNs from a pool by directly communicating with the coordinating service, while other applications may gain exclusive access to spare LUNs from the pool through interaction of a system administrator with the coordinating service. Note that other embodiments may use other mechanisms for gaining exclusive access to spare LUNs in a pool.

In embodiments, the coordinating service may be configured to handle situations where two or more potentially conflicting requests for spare LUNs are received. The coordinating service coordinates access to the pool of spare storage among all the systems that are configured to access the storage, and provides a handshake/lock mechanism to prevent conflicts in accessing spare storage in the pool. If, for example, there are three different requests made at the same time by systems that are configured to access overlapping units of storage, the coordinating service selects and provides three different units of storage from the pool to satisfy the requests, if available. Once the coordinating service provides a unit of storage from the pool to meet a request, it “marks” the unit of storage so that the unit of storage is no longer available to be selected and provided to meet other requests.

Embodiments of the pre-provisioning mechanism essentially create a layer where a pool of spare storage (LUNs) is configured for access by multiple hosts. All hosts in the storage network may be pre-provisioned with access to at least some of the units of storage in the pool(s). If hosts need access to additional storage, the hosts do a “handshake” through the coordinating service so that two hosts do not gain exclusive access to, and start to use, the same units of storage. Unlike some conventional systems, the storage network does not have to be reconfigured for a host to gain access to additional units of storage; the storage available to application(s) on the host may be dynamically increased.

Note that it is still possible to exhaust the storage available to a host in the pool. However, unlike thin provisioning, if storage accessible by a host is exhausted, an error may be generated on a storage allocation request rather than on an I/O write. For example, a database running on a host may need to grow its storage. The database expects that if it needs to create a new table space or grow an existing table space, it might not have the disk space, and databases are generally configured to gracefully handle those situations. However, databases do not typically expect, and are not typically configured to gracefully handle, failures on writes to disk space that they think they have available. Thus, embodiments of the pre-provisioning mechanism move potential failures due to lack of disk space back from the write frame to the allocation frame where applications generally expect that they might encounter a failure, and are generally configured to gracefully handle the allocation failure.

FIG. 3illustrates an exemplary storage network implementing a pre-provisioning mechanism according to one embodiment. For one embodiment, SAN may be described as a high-speed, special-purpose network that interconnects storage devices204(e.g. storage devices204A and204B) with associated data servers (e.g. hosts220A,220B, and220C) on behalf of a larger network of users. A SAN may employ Fibre Channel technology. A SAN may include one or more hosts220(e.g. hosts220A,220B, and220C), one or more storage devices204(e.g. storage devices204A and204B), and one or more SAN fabrics230. A SAN may also include one or more administration systems (not shown). One or more end-user platforms (not shown) may access the SAN, typically via a LAN or WAN connection, such as network306, to one or more of the hosts220. Hosts220may also be interconnected via one or more networks306.

Host systems220may be any of various types of devices, including, but not limited to, a personal computer system, desktop computer, laptop or notebook computer, mainframe computer system, workstation, network appliance, network computer, Internet appliance, or other suitable device. Hosts220may run any of a variety of operating systems, including, but not limited to: Solaris 2.6, 7, 8, 9, etc.; Linux; AIX; HP-UX 11.0b, 11i, etc.; Microsoft Windows NT 4.0 (Server and Enterprise Server) and Microsoft Windows 2000 (Server, Advanced Server and Datacenter Editions). Each host system220may include one or more host bus adapters (HBAs)222which each provide host adapter ports for coupling to the SAN. Each host220is typically connected to the fabric230via one or more HBA222ports. SAN fabric230may enable server-to-storage device connectivity through Fibre Channel switching technology. SAN fabric230hardware may include one or more switches232, bridges234, hubs236, or other devices238such as routers, as well as the interconnecting cables (for Fibre Channel SANs, fibre optic cables). Hosts220may each include one or more SAN applications (not shown).

Storage devices204may include, but are not limited to, RAID (Redundant Array of Independent Disks) systems, disk arrays, JBODs (Just a Bunch Of Disks, used to refer to disks that are not configured according to RAID), tape devices, and optical storage devices. Each storage device204(e.g. storage devices204A and204B) may include one or more addressable storage units (AUs) and one or more array ports for coupling to the SAN. AUs are storage volumes built out of one or more physical discs within the storage device204. Array ports may be connected to a SAN fabric, and function as SCSI targets behind which the AUs bound to those ports are visible. In the LUN binding process, an AU is bound to a specified array port in a specified storage device204(e.g. a storage system/disk array)). This results in the creation of a LUN202. “LUN” is the term for the access path itself between an AU and an array port. Note, however, that a LUN202is also frequently identified with the AU behind it and treated as though it had the properties of that AU. For the sake of convenience, a LUN202may be thought of as being the equivalent of the AU it represents.

When the storage network is first configured, and/or when new storage devices are added or existing storage devices are expanded, LUNs202for accessing AUs in each storage device may be created. During initial configuration or when adding or expanding storage devices, some of the LUNs202may be specifically assigned to particular hosts220for use. In this example, LUNs202A on storage device204A and LUNs202B on storage device204B may each be assigned to one of hosts220A,220B and220C. Typically, assignment includes configuring the storage network so that a particular array port of a LUN202is configured to communicate with a particular HBA port on a particular HBA222on one of the hosts220through SAN fabric230. Various components of the SAN fabric230, such as switches232, may be involved in the configuration.

In embodiments of the pre-provisioning mechanism, one or more spare LUNS212on each storage device may be created and placed into pools210of pre-provisioned LUNs. In this example, pool210A includes spare LUNs212A on storage device204A, and pool210B includes spare LUNs212B on storage device204B. Each spare LUN212in a pool may be pre-configured, or “masked”, for access by two or more of the hosts220on the storage network. When additional storage (represented by a spare LUN212) is needed by a host220, for example host220A, the host220A may select a spare LUN212to which the host has been preconfigured to access from one of the pools210of spare LUNs, for example from pool210A, remove the spare LUN212from the pool210A, and then begin using the storage represented by the spare LUN212. In one embodiment, “removing” a LUN212from a pool210may include marking the LUN212in the pool with an indication so that other hosts220will not access, or will not be granted exclusive access to, the LUN212. Note that other hosts220may initially be still configured to access the LUN212through an HBA222port, but are prevented from actually using the LUN212once host220A claims exclusive access to the LUN212. A change control request may then be generated to reconfigure the storage network, for example during a change window, to disallow access to the LUN212from the other hosts220that have access to the LUN212. Essentially, when reconfigured, the spare LUN212to which host220A has been granted exclusive access becomes one of LUNs202. All access to the LUN212by other hosts is removed, and the LUN212is no longer part of a pool210of spare LUNs.

In one embodiment, one or more of hosts220, or alternatively one or more other servers or hosts on the storage network, may include a coordinating service (not shown) through which hosts220may request exclusive access to spare LUNs212in pools210. In one embodiment, the coordinating service(s) may maintain information on the spare LUNs212in pools210, may receive requests from hosts220for one or more spare LUNs212from pools210, may select one or more spare LUNs212from pools210pre-provisioned for access by a requesting host220to meet a request, may inform the requesting host220as to which LUN(s)212the host220has been granted exclusive access, and may update its pool information to indicate that the host220has been granted exclusive access to the LUN(s)212so that no other hosts220are granted access to the LUN(s)212. A change control request may be generated so that the storage network may be reconfigured, for example during a change window, to remove access to the LUN(s)212by all hosts220except the host220to which exclusive access has been granted. In one embodiment, the coordinating service may automatically generate the change control request. In one embodiment, communications between the hosts220and the coordinating service may be via network240.

FIGS. 4A through 4Cillustrate an exemplary storage network in which a pool of spare LUNs is pre-provisioned to two or more host systems according to one embodiment. InFIG. 4A, three hosts220(hosts220A,220B and220C) have each been configured for exclusive access to a LUN202on storage device(s)204(LUNs202A,202B and202C, respectively) via the HBAs222of the hosts220. The connections from the HBAs220through SAN fabric230to LUNs202are represented by bolded lines. In addition, three spare LUNS212on storage device(s)204(spare LUNs212A,212B and212C) have been pre-provisioned to hosts220in pool210. Each spare LUN212in pool210has been configured for access by each host220via the HBA222of the host220. The pre-provisioned connections from the HBAs220through SAN fabric230to spare LUNs212are represented by the thinner, non-bolded lines. Spare LUNs212represent extra storage space on storage device(s)204that has been pre-provisioned for access by each of the hosts220on an as-needed basis. Initially, none of the hosts220is using the spare storage represented by the spare LUNs212in pool210.

At some point, one of the hosts220may require additional storage. InFIG. 4B, host220A requires additional storage, and selects spare LUN212A which host220A has been preconfigured to access from pool210to meet its storage requirements, LUN212A may be removed from the pool210, and then host220A may begin using the storage represented by the spare LUN212A. In one embodiment, “removing” LUN212A from pool210may include marking the LUN212A in the pool with an indication so that other hosts220will not access, or will not be granted exclusive access to, the LUN212A. Note that, as indicated inFIG. 4B, other hosts220may initially be still configured to access the LUN212A, but are prevented from actually using the LUN212A once host220A claims exclusive access to the LUN212A. A change control request may then be generated to reconfigure the storage network, for example during a change window, to disallow access to the LUN212A from the other hosts220that have access to the LUN212A. Essentially, when reconfigured, the spare LUN212A to which host220A has been granted exclusive access becomes one of LUNs202. All access to the LUN212A by other hosts is removed, and the LUN212A is no longer part of pool210.FIG. 4Cillustrates the exemplary storage network ofFIGS. 4A and 4Bafter reconfiguration to remove access to spare LUN212A by hosts220B and220C. Note that spare LUN212A has now become LUN202D, and is no longer indicated as being in pool210. Also note that access paths through SAN fabric230from hosts220B and220C to LUN202D (formerly spare LUN212A) have been removed.

In embodiments, spare LUNs212may be added to pool210. For example, it may be necessary or desirable to add additional storage, and spare LUNs212, if pool210is exhausted or nearly exhausted. The data center may, for example, set a policy that pool210should include at least three spare LUNs212, or alternatively at least a certain amount of storage represented by the spare LUNs, to meet potential storage requirements for the storage network over a period. To add spare LUNs212, additional LUNs in storage device(s)204may be created and added to the pool if there are AUs that have not been configured for access via a LUN. Alternatively, additional storage devices may be added to the storage network, one or more LUNs may be created for at least some of the AUs in the storage devices, and the LUNs may be configured and added as spare LUNs212in the pool210. Typically, but not necessarily, configuring and adding additional LUNs may be done during a change window or other “down time”. In one embodiment, if a host220no longer needs a LUN202to which it has exclusive access, the storage network may be reconfigured (for example, during a change window) to move the LUN202to pool210of spare LUNs212.

In one embodiment, one of hosts220, or alternatively some other server or host on the storage network, may include a coordinating service through which hosts220may request exclusive access to a spare LUN212in a pool210.FIG. 5illustrates an exemplary storage network implementing a pre-provisioning mechanism including a coordinating service according to one embodiment. In this example, one or more storage devices in the storage network are represented as storage250for simplicity. Pools210represent one or more pools of spare LUNs in storage250that are pre-provisioned to hosts220. Host220B implements an instance of coordinating service260.

In one embodiment, the coordinating service260may maintain information262on the spare LUNs in pool(s)210. If a host220needs additional storage, the host220may communicate with the coordinating service260to request a spare LUN from a pool210to which the host has access. The coordinating service260may select a spare LUN212from a pool210to meet the request, and may inform the requesting host220that the host220has been granted exclusive access to the LUN. The coordinating service260may update its pool information262to indicate that the host220has been granted exclusive access to the LUN212so that no other hosts220are granted access to the LUN. A change control request may be generated so that the storage network may be reconfigured, for example during a change window, to remove access to the LUN by all hosts220except the host220to which exclusive access has been granted. In one embodiment, the coordinating service260may automatically generate the change control request. In one embodiment, communications between the hosts220and the coordinating service260may be via a network240.

Note that while inFIG. 5coordinating service260is illustrated as being implemented on host220B that is one of the hosts220that is configured to access LUNs on storage250, in one embodiment a coordinating service260may be implemented on a separate system, such as a SAN management server. In one embodiment, coordinating service260may be implemented as part of a SAN management service. In one embodiment, each pool210of spare LUNs in a storage network may have a unique instance of a coordinating service260that controls access to spare LUNs in the pool210by hosts220. In one embodiment, there may be two or more instances of coordinating service260in a storage network, with each instance of coordinating service260being responsible for managing access to one or more pools210of spare LUNs.

In one embodiment, rather than having a central coordinating service260that manages access by the hosts to spare storage in the pool(s), a coordinating mechanism may be implemented as a distributed or clustered application on two or more systems cooperating in a peer-to-peer relationship to coordinate access to the spare storage in the pools. For example, if there are ten hosts sharing a pool of spare storage, the hosts may all run a clustered application to coordinate access to the spare storage in the pool in a peer-to-peer relationship. In implementations with many systems having access to a pool of spare storage, for example 1000 systems that have access to the pool in an IP network that does not impose a connection limit as does Fibre Channel, such a peer-to-peer implementation of a coordinating mechanism may be impractical. For such implementations, there may be a centralized coordinating service on the network, for example implemented on a cluster of two or more highly available hosts, that serve as “gatekeepers” that coordinate access to the spare storage in the pool(s) rather than having all of the systems running a peer-to-peer type application.

FIG. 6andFIG. 7illustrate exemplary configurations for pools of spare storage according to embodiments, and are not intended to be limiting. Note thatFIGS. 4A through 4Cillustrate spare LUNs in one pool. In embodiments, spare LUNs may be organized into one or more pools. In bothFIGS. 6 and 7, spare LUNs are illustrated as organized into two pools for simplicity. Note that the discussion ofFIGS. 6 and 7may be extended to implementations where spare LUNs are organized into more than two pools.

InFIG. 6, the spare LUNs212on storage device(s)204are shown divided equally into two pools210via SAN fabric230. Hosts220are also divided, with some hosts220being configured, or pre-provisioned, to access LUNs212only in one pool210, and other hosts220being configured, or pre-provisioned, to access LUNs212only in the other pool210. In this example, hosts220A and220B are configured to access LUNs212A,212B and212C in pool210A, and hosts220C and220D are configured to access LUNs212D,212E and212F in pool210B. In this example, each host220is configured to access all the LUNs212in its associated pool210. Note that this arrangement is exemplary, and that a host220may be configured to access only a subset of the LUNs212in its associated pool210, and/or may be configured to access LUNs212in two or more pools.

The organization of the pools210and hosts220inFIG. 6, where the hosts220are split among two or more discrete pools210, gives each host220access to as many LUNs212as are in the pool210to which the host is associated or assigned. If there are N LUNs in a pool210, then a host assigned to that pool210and that pool only may be configured to access at most N LUNs. InFIG. 6, each host212is configured to access three LUNs212and only three LUNs.

FIG. 7illustrates a different organization for pools210and hosts220that may provide a host220access to more LUNs212than the organization illustrated inFIG. 6, and that thus may be more robust than splitting hosts220among two or more discrete pools210. In this example, the spare LUNs212on storage device(s)204are again shown divided equally into two pools210via SAN fabric230. However, instead of splitting hosts220among the two pools210, non-overlapping sets of hosts220are configured to access sets of LUNs212from the two pools210. Thus, each host220may be configured to access more than the number of LUNs212in any single pool210. This makes it less likely that spare storage in any one pool210will be exhausted.

InFIG. 7, each host220is configured to access four LUNs212from the two pools, with each host220configured to access two LUNs212from each pool210of three LUNs212. In this example, host220A is configured to access LUNs212A and212B in pool210A, and LUNs212D and212E in pool210B. Host220B is configured to access LUNs212B and212C in pool210A, and LUNs212D and212F in pool210B. Host220C is configured to access LUNs212A and212B in pool210A, and LUNs212D and212F in pool210B. Host220D is configured to access LUNs212A and212C in pool210A, and LUNs212E and212F in pool210B. Note again that this is exemplary, and that other assignments of hosts220to LUNs212may be made. For clarity:

Fibre Channel (FC) is a common connection/communications medium used to connect storage to hosts in storage networks. FC has a commonly implemented connection limit of 32 logins to any target. In a storage network that uses FC, there cannot be more than 32 HBA ports, or systems (hosts), configured to access the same LUN. Thus, in FC implementations, or in other implementations using storage protocols with connection limits, depending on the number of hosts, the pre-provisioned spare storage (LUNs) may be divided into one or more pools. For example, in a FC storage network with 40 hosts and N spare LUNs, the N spare LUNs may be divided into one or more pools. With one pool, each of the 40 hosts may be configured to access the N spare LUNs in the pool (depending on available HBA ports on the hosts), but each LUN may be configured for access by at most 32 of the hosts. With two pools, 20 of the hosts may be assigned to one pool, the other 20 assigned to the other pool, and each host may be configured to access all of the LUNs in its assigned pool, as in the example illustrated inFIG. 6. Alternatively, the hosts may be assigned to LUNs in both pools, as in the example illustrated inFIG. 7. Note that the N spare LUNs may be divided into three or more pools, with hosts either assigned to a particular pool or configured to access LUNs in one or more of the pools.

Hosts are configured, or masked, to access particular spare LUNs; a host has access to LUNs which it is configured to access, and thus is “assigned” to one or more pools that include spare LUNs to which it has access. Once the spare LUNs are assigned to hosts and thus divided into one or more pools, any host that has access to a particular LUN in a pool may request and be granted exclusive access to the LUN. Other hosts that have been configured to access the spare LUN may then be blocked from gaining exclusive access to the LUN. As described above, in one embodiment, a coordinating service may act as a coordinator between hosts configured to access a pool of spare LUNs. A change control request may be generated, and during a change window the storage network may be reconfigured so that other hosts no longer have access to that spare LUN.

During a change window, the storage network may also be reconfigured to refresh the pools of spare LUNs by adding new spare LUNs and/or by “balancing” the pools by reconfiguring various hosts to access different spare LUNs in the same or other pools or removing access by certain hosts to certain spare LUNs, effectively moving spare LUNs to other pools and/or moving hosts to other pools. Note that a host does not have to wait for a change window to gain access to and start using needed storage. A host may gain exclusive access to spare storage at any time; at a change window, the storage network may be reconfigured to remove access to LUNs that have been claimed for exclusive access, refresh the pools, balance the pools, or even add or remove pools.

Note that, in the above example where 40 hosts are split with 20 of the hosts configured to access the spare LUNs in one pool and the other 20 hosts configured to access the spare LUNs in another pool, or in general in any implementation of the pre-provisioning mechanism, it is possible to exhaust the spare LUNs in a pool if the hosts consume all of the spare LUNs. If this happens, errors may be generated on storage allocations and not on I/O write operations, as might happen in storage networks that use thin provisioning. As noted, storage allocation errors are less likely to cause serious problems for applications than are errors on I/O write operations.

Various methods may be used to divide the spare LUNs into pools. As mentioned, N spare LUNs may be divided into M pools (e.g., two pools), and the hosts on the storage network may be divided more-or-less equally among the pools, with each host assigned to only one pool. Knowledge of the storage demand of various hosts may be applied to modify this method, for example by assigning hosts that tend to use more storage to larger pools, and hosts that tend to use less storage to smaller pools.

Alternatively, N spare LUNs may be divided into a set of M pools, with the hosts having access to spare LUNs in each of the M pools, and a non-overlapping set of hosts having access to each pool. Using this method, each host may be able to see more spare storage than in the method described above that splits hosts among a set of discrete pools. Each host will be able to access more spare storage because each host has access to more than one pool. Using this method, it may be less likely that one set of hosts will exhaust a particular pool.

Note that the above methods of dividing hosts among pools of spare storage are exemplary, and that a variety of other methods may be used to determine how large to make each pool, how much spare storage to assign to each host, how to divide the hosts among the pools, and how to overlap the pools for optimal robustness.

For an implementation where the storage protocol imposes a connection limit, and thus the spare storage may be distributed among two or more pools, a storage service may be implemented for each pool of spare storage. Alternatively, there may be one central storage service, implemented on one or more systems, that manages access by the hosts to all the pools of spare storage. As another alternative, each host or system may include an implementation of a distributed storage management application that cooperate in a peer-to-peer relationship to coordinate access to the pool(s) of spare storage.

Storage Protocols without Connection Limits

Some storage/network protocols, such as IP, that may be used in storage networks to connect hosts to storage may not have a connection limit as does FC. In storage networks that use a storage protocol without a connection limit, spare storage may be configured into one pool that may be larger than a pool would typically be in an FC implementation, where the connection limit may necessitate breaking the spare storage into two or more smaller pools. On an IP network, for example, there is no connection limit; it is possible to have 1000 or more hosts all “seeing” the same storage. Thus, in an IP network, it is possible to have 1000 or more hosts sharing one pool of spare storage. Note that, even in such a storage network, embodiments allow the spare storage to be broken into two or more pools.

As noted above, in implementations with many systems having access to a pool of spare storage, for example 1000 systems that have access to the pool in an IP network that does not impose a connection limit as does Fibre Channel, there may be a centralized coordinating service on the network, for example implemented on a cluster of two or more highly available hosts, that serve as “gatekeepers” that coordinate access to the spare storage in the pool(s) rather than having all of the systems running a peer-to-peer type application.

FIG. 8is a flowchart illustrating a method of pre-provisioning spare storage to hosts in a storage network according to one embodiment. As indicated at300, spare LUNs in a storage system may be pre-provisioned to hosts on a storage network so that each spare LUN is accessible to two or more of the hosts through the storage network fabric. The spare LUNs may be divided into one or more pools according to one of the configurations for pools of spare storage as previously described herein.

At some point, a host on the storage network may need additional storage. As indicated at302, the host may request and obtain exclusive access to one of the spare LUNs to which it has access through the storage network fabric. The spare LUN may then be “removed” from the pool of spare LUNs so that other hosts cannot obtain exclusive access to the LUN. In one embodiment, to obtain exclusive access to a spare LUN, the host may communicate with a coordinating service to request a spare LUN. The coordinating service may determine if there is a spare LUN available that is accessible to the host and that another host has not obtained exclusive access to. In one embodiment, the coordinating service may maintain pool information in which the status of spare LUNs in the pool(s) of spare LUNs is recorded and updated when necessary. If there is a spare LUN available, the coordinating service may then inform the requesting host that it has obtained a selected spare LUN from the pool(s) for exclusive access. The host may then begin using the storage unit represented by the LUN. If there is no spare LUN available to meet the host's request, then the coordinating service may inform the host that no spare LUN is available for exclusive access. Note that errors generated on the host by failure to obtain a spare LUN would be on storage allocation and not on I/O writes.

As indicated at304, a recordation that the host has obtained exclusive access to the LUN may be made so that other hosts that are configured to access the LUN cannot obtain exclusive access to the LUN. In one embodiment, a coordinating service that coordinates access to the spare LUNs in the pool(s) may record this information in pool information that it maintains. In one embodiment, a form of device reservation such as SCSI-3 persistent reservation may be used to record the LUN assignment. Other embodiments may use other mechanisms, or combinations thereof, to record the LUN assignment.

As indicated at306, a change control request may be generated to request reconfiguration of the storage network to remove access to the LUN to which the host has obtained exclusive access from the other hosts that have access to the LUN. In one embodiment, an administrator may generate the change control request. In another embodiment, a coordinating service may automatically generate the change control request. As indicated at308, the storage network may then be reconfigured during a change window in accordance with the change control request. Note that the pool(s) of spare LUNs may be replenished with additional spare LUNs and/or balanced during a change window if necessary.

FIG. 9illustrates an exemplary storage network implementing an embodiment of the pre-provisioning mechanism and including a system including software implementing a coordinating service according to one embodiment. System750may be any of various types of devices, including, but not limited to, a personal computer system, desktop computer, laptop or notebook computer, server, mainframe computer system, workstation, network computer, or other suitable device. System750may include at least one processor752. The processor752may be coupled to a memory754. Memory754is representative of various types of possible memory media, also referred to as “computer readable media.” Hard disk storage, floppy disk storage, removable disk storage, flash memory and random access memory (RAM) are examples of memory media. The terms “memory” and “memory medium” may include an installation medium, e.g., a CD-ROM or floppy disk, a computer system memory such as DRAM, SRAM, EDO RAM, SDRAM, DDR SDRAM, Rambus RAM, etc., or a non-volatile memory such as a magnetic media, e.g., a hard drive or optical storage. The memory medium may include other types of memory as well, or combinations thereof.

System750may couple, for example over a wired or wireless network or networks (e.g. network740), to one or more other devices, such as hosts720A through720D, via one or more wired or wireless network interfaces. System750may also couple, over a network, storage network, or by some other type of connection such as direct connection, to one or more storage devices704, including storage devices on which units of storage (e.g., LUNs712in pool710) are provisioned for one or more hosts720.

Hosts720may couple to the storages devices704via a storage network “fabric”700. This coupling of hosts720to storage devices704via an infrastructure (storage network fabric700) may be referred to as a “storage network”. A storage network may, for example, be a Storage Area Network (SAN), a LAN with Network-Attached Storage (NAS), or any network capable of coupling storage devices to hosts720. The storage devices704on a storage network may include any of one or more types of storage devices, and may include physical and/or logical devices. A storage device704may be any type of computer-accessible medium capable of storing data including, but not limited to: storage media or memory media such as magnetic or optical media, stand-alone disks, RAID systems, JBODs, any of one or more types of backup devices, including, but not limited to, various types of tape devices and optical storage devices, CD-ROM (CD, CD-R, or CD-RW), volatile or non-volatile media such as RAM (e.g. SDRAM, DDR, RDRAM, SRAM, etc.), ROM, Flash Memory, MEMS, etc.

System750may include, in memory754, an implementation of a coordinating service760. In one embodiment, the coordinating service760may maintain information762on the spare LUNs in pool710. If a host720needs additional storage, the host720may communicate with the coordinating service760to request exclusive access to a spare LUN from pool710to which the host has access. The coordinating service760may select a spare LUN712from pool710to meet the request, and may inform the requesting host720that the host720has been granted exclusive access to the LUN. The coordinating service760may update its pool information762to indicate that the host720has been granted exclusive access to the LUN712so that no other hosts720are granted access to the LUN. A change control request may be generated so that the storage network may be reconfigured, for example during a change window, to remove access to the LUN by all hosts720except the host720to which exclusive access has been granted. In one embodiment, the coordinating service760may automatically generate the change control request. In one embodiment, communications between the hosts720and the coordinating service760may be via a network740.

Note that the configuration illustrated inFIG. 9is an exemplary implementation of a storage network implementing an embodiment of the pre-provisioning mechanism and including a coordinating service and is not intended to be limiting. Embodiments of the pre-provisioning mechanism as described herein may be implemented in other configurations of systems and storage environments.

CONCLUSION