High-speed data transfer in a storage virtualization controller

A storage virtualization controller for transferring data between a host and a storage device at a wire-speed data transfer rate. A downstream processing element adapted for connection to the storage device is configurable coupled to an upstream processing element adapted for connection to the host. A central processing element coupled to the upstream processing element grants permission to the upstream processing element to transfer the data at the wire-speed rate without further involvement by the central processing element.

BACKGROUND OF THE INVENTION

Storage area networks, also known as SANs, facilitate sharing of storage devices with one or more different host server computer systems and applications. Fibre channel switches (FCSs) can connect host servers with storage devices creating a high speed switching fabric. Requests to access data pass over this switching fabric and onto the correct storage devices through logic built into the FCS devices. Host servers connected to the switching fabric can quickly and efficiently share blocks of data stored on the various storage devices connected to the switching fabric.

Storage devices can share their storage resources over the switching fabric using several different techniques. For example, storage resources can be shared using storage controllers that perform storage virtualization. This technique can make one or more physical storage devices, such as disks, which comprise a number of logical units (sometimes referred to as “physical LUNs”) appear as a single virtual logical unit or multiple virtual logical units, also known as VLUNs. By hiding the details of the numerous physical storage devices, a storage virtualization controller advantageously simplifies storage management between a host and the storage devices. In particular, the technique enables centralized management and maintenance of the storage devices without involvement from the host server.

Performing storage virtualization is a sophisticated process. By way of comparison, a fibre channel switch does relatively little processing on the various command and data frames which pass through it on the network. But a storage virtualization controller must perform a much greater amount of processing than a fabric channel switch in order to convert the requested virtual storage operation to a physical storage operation on the proper storage device or devices.

In many instances it is advantageous to place the storage virtualization controller in the middle of the fabric, with the host servers and controllers arranged at the outer edges of the fabric. Such an arrangement is generally referred to as a symmetric, in-band, or in-the-data-path configuration. However, this configuration is generally problematic if the controller cannot operate at the specified data rate of the fabric. In the case of fibre channel, for example, this data rate is at least 1 gigabit per second. If the controller is not capable of operating at the specified data rate, traffic on the network will be slowed down and the overall throughput and latency deleteriously reduced.

SUMMARY OF THE INVENTION

In one embodiment, the present invention provides a storage virtualization controller for transferring data between a host and a storage device at a wire-speed data transfer rate. The controller includes a downstream processing element adapted to connect to a storage device, and an upstream processing element adapted to connect to a host. The upstream processing element is further adapted to configurably couple to the downstream processing element. The controller also includes a central processing element coupled to the upstream processing element. The central processing element grants permission to the upstream processing element to transfer the data through the downstream processing element without further involvement by the central processing element.

The present invention may also be implemented as a method of accessing a virtual logical unit in a storage area network. In the method, an upstream processing element receives a request from a host to access the virtual logical unit. A storage device associated with the virtual logical unit, and a downstream processing element associated with the storage device, are identified. Permission for the downstream processing element to perform the access request is obtained from a central processing element by the upstream processing element. After permission is granted, data is transferred between the upstream processing element and the downstream processing element at substantially a rated speed of the storage area network.

DESCRIPTION OF THE PREFERRED EMBODIMENT

Referring now to the drawings, there is illustrated an embodiment of a storage virtualization controller constructed in accordance with the present invention which can transfer data between a host, such as a server, and a storage device at a wire-speed data transfer rate. The host can be connected to an upstream processing element (UPE), and the storage device to a downstream processing element (DPE), of the controller. In operation, a central processing element (CPE) of the controller grants permission to the UPE to transfer the data between the host and the storage device through the UPE and the DPE without any further involvement by the CPE. One such controller is a virtual storage exchange (VSX) device designed by Confluence Networks, Incorporated of Milpitas, Calif. (VSX is a trademark of Confluence Networks, Incorporated).

As best understood with reference to the exemplary configuration ofFIG. 1, a storage area network (SAN)100may include one or more SAN switch fabrics, such as fabrics104,105. Fabric104is connected to hosts102, while fabric105is connected to storage devices106. At least one storage virtualization controller126is inserted in the midst of SAN100, and connected to both fabrics104,105to form a symmetric, in-band storage virtualization configuration. In an in-band configuration, communications between server devices102and storage devices106pass through controller126for performing data transfer in accordance with the present invention.

Host servers102are generally communicatively coupled (through fabric104) via links150to individual UPEs of controller126. In an alternate configuration, one or more host servers may be directly coupled to controller126, instead of through fabric104. Controller126includes at least one UPE for each server102(such as host servers108,110,112,114) connected to the controller126. As will be discussed subsequently in greater detail, storage virtualization controller126appears as a virtual logical unit (VLUN) to each host server.

Storage devices106are communicatively coupled (through fabric105) via links152to individual DPEs of controller126. In an alternate configuration, one or more storage devices may be directly coupled to controller126, instead of through fabric105. Controller126includes at least one DPE for each storage device106(such as storage devices130,132,134,136,138) connected to the controller126. Controller126appears as an initiator to each storage device106.

Considering now the virtualization of storage provided by an embodiment of the present invention, and with reference to the exemplary SAN200ofFIG. 2, a storage virtualization controller202has been configured to provide four virtual logical units214,216,218,220associated with hosts204-210. In the general case, a VLUN includes N “slices” of data from M physical storage devices, where a data “slice” is a range of data blocks. In operation, a host requests to read or write a block of data from or to a VLUN. In this exemplary configuration, host1204is associated with VLUN1214; host2205, host3206, and host4207are associated with VLUN2216; host5208and host6209are associated with VLUN3218, and host7210is associated with VLUN4220. A host204-210accesses its associated VLUN by sending commands to the storage virtualization controller202to read and write virtual data blocks in the VLUN. Controller202maps the virtual data blocks to physical data blocks on individual ones of the storage devices232,234,236, according to a preconfigured mapping arrangement. Controller202then communicates the commands and transfers the data blocks to and from the appropriate ones of the storage devices232,234,236. Each storage device232,234,236can include one or more physical LUNs; for example, storage device1232has two physical LUNs, LUN1A222and LUN1B223.

To illustrate further the mapping of virtual data blocks to physical data blocks, all the virtual data blocks of VLUN1214are mapped to a portion224aof the physical data blocks LUN2224of storage device234. Since VLUN2216requires more physical data blocks than any individual storage device232,234,236has available, one portion216aof VLUN2216is mapped to the physical data blocks of LUN1A222of storage device232, and the remaining portion216bof VLUN2216is mapped to a portion226aof the physical data blocks of LUN3226of storage device236. One portion218aof VLUN3218is mapped to a portion224bof LUN2224of storage device234, and the other portion218bof VLUN3218is mapped to a portion226bof LUN3226of storage device236. It can be seen with regard to VLUN3that such a mapping arrangement allows data block fragments of various storage devices to be grouped together into a VLUN, thus advantageously maximizing utilization of the physical data blocks of the storage devices. All the data blocks of VLUN4220are mapped to LUN1B223of storage device232.

While the above-described exemplary mapping illustrates the concatenation of data block segments on multiple storage devices into a single VLUN, it should be noted that other mapping schemes, including but not limited to striping and replication, can also be utilized by the controller202to form a VLUN. Additionally, the storage devices232,234,236may be heterogeneous; that is, they may be from different manufacturers or of different models, and may have different storage sizes, capabilities, architectures, and the like. Similarly, the hosts204-210may also be heterogeneous; they may be from different manufacturers or of different models, and may have different processors, operating systems, networking software, applications software, capabilities, architectures, and the like.

It can be seen from the above-described exemplary mapping arrangement that different VLUNs may contend for access to the same storage device. For example, VLUN2216and VLUN4220may contend for access to storage device1232; VLUN1214and VLUN3218may contend for access to storage device2234; and VLUN2216and VLUN3218may contend for access to storage device3236. A storage virtualization controller according to an embodiment of the present invention performs the mappings and resolves access contention, while allowing data transfers between the host and the storage device to occur at wire-speed.

Considering now the storage virtualization controller302in greater detail, and with reference to the SAN300ofFIG. 3, the controller302includes at least one upstream processing element (two UPEs304,306are shown for clarity) adapted for connection to a corresponding one of the hosts308,310for transferring data between the corresponding host and its VLUN. The controller302also includes at least one downstream processing element (three DPEs312,314,316are shown for clarity) each adapted for connection to a corresponding one of the storage devices318,320,322. Each storage device318,320,322is representative of at least a portion of a VLUN. A selected one of the UPEs (in this example, UPE304) is configurably and intermittently coupleable for a period of time to an associated one of the DPEs (in this example, DPE312) to form a data channel324. The coupling is typically exclusive, such that only the coupled host can access the coupled storage device during that period. The period of exclusivity typically corresponds to the time required to transfer of a slice of data between the corresponding host and the corresponding storage device. The channel324is operable at a wire-speed data transfer rate to transfer blocks of data between host308connected to the channel324and storage device318connected to the channel324.

The term “wire-speed” means that the data transfers occur at a rate that matches the data rate of the link to which the storage virtualization controller302is connected and does not result in adverse degradation of the link. The ability of the controller302to perform wire-speed transfers is of particular importance when the controller302is used in a high-speed network such as, but not limited to, a fibre channel-based version of SAN100previously discussed with reference toFIG. 1. Fibre channel networks typically operate at a data rate of at least one gigabit per second , with some versions that are available at the present time operating at a two gigabit per second data rate. In a symmetric, in-band configuration where a substantial amount of SAN traffic flows through the controller302, even a small delay caused by the controller302on each data transaction will accumulate to form a substantial aggregate delay and resulting slowdown of the network. The controller302of the present invention performs such wire-speed transfers and avoids degradation of network operation. The details of how this wire-speed transmission is accomplished in the storage virtualization controller302will be discussed subsequently with regard to the interactions between a UPE, a DPE, and a central processing element (CPE), such as CPE326. CPE326regulates the establishment of the channel by performing contention resolution and granting the exclusive permission to the UPE to transfer data with the DPE, but does not otherwise participate in the data transfers between the UPE and the DPE. An individual CPE may manage one or more VLUNs. For instance, in the example SAN300ofFIG. 3, CPE326manages the VLUNs associated with host1308(which is linked to CPE326via UPE1304) and host M310(linked to CPE326via UPE M306). Controller302may include more than one CPE, such as CPE327; each CPE may manage a different set of VLUNs via connections to additional UPEs and hosts (not shown). However, for clarity, the discussion of the present invention will only involve the operation of a single CPE326.

Before considering the various elements of the storage virtualization controller302in further detail, it is useful to discuss the format and protocol of the storage requests that are sent over SAN300from a host to a storage device through the controller302. Many storage devices frequently utilize the Small Computer System Interface (SCSI) protocol to read and write the bytes, blocks, frames, and other organizational data structures used for storing and retrieving information. Hosts access a VLUN using these storage devices via some embodiment of SCSI commands; for example, layer4of Fibre Channel protocol. However, it should be noted that the present invention is not limited to storage devices or network commands that use SCSI protocol.

Storage requests may include command frames, data frames, and status frames. The controller302processes command frames only from hosts, although it may send command frames to storage devices as part of processing the command from the host. A storage device never sends command frames to the controller302, but only sends data and status frames. A data frame can come from either host (in case of a write operation) or the storage device (in case of a read operation).

In many cases one or more command frames is followed by a large number of data frames. Command frames for read and write operations include an identifier that indicates the VLUN that data will be read from or written to. A command frame containing a request, for example, to read or write a 50 kB block of data from or to a particular VLUN may then be followed by 25 continuously-received data frames each containing 2 kB of the data. Since data frames start coming into the controller302only after the controller has processed the command frame and sent a go-ahead indicator to the host or storage device that is the originator of the data frames, there is no danger of data loss or exponential delay growth if the processing of a command frame is not done at wire-speed; the host or the storage device will not send more frames until the go-ahead is received. However, data frames flow into the controller302continuously once the controller gives the go-ahead. If a data frame is not processed completely before the next one comes in, the queuing delays will grow continuously, consuming buffers and other resources. In the worst case, the system could run out of resources if heavy traffic persists for some time.

Considering now in further detail the downstream processing elements, each DPE312,314,316can be connected to a corresponding storage device318,320,322. The connection is typically made between a port (not shown) on the controller302that is connected to the DPE312,314,316and a corresponding port (not shown) on the storage device318,320,322. Each DPE312,314,316functions as an initiator to its connected storage device318,320,322, and transfers commands, data, and status to and from the storage device318,320,322typically according to SCSI protocol.

Considering now in further detail the upstream processing elements, each UPE304,306can be connected to a corresponding host308,310. The connection is typically made between a port (not shown) on the controller302that is connected to the UPE304,306and a corresponding port (not shown) on the host308,310. Each UPE appears as a VLUN to its connected host308,310, and transfers commands, data, and status to and from the host308,310typically according to an embodiment of SCSI commands such as layer 4 of Fibre Channel protocol.

Commands sent from a host (for example, host308) to a UPE (for example, UPE304) are received by a command validator330. The command validator330validates the command frames to form validated commands, and routes the validated commands for execution. One class of commands is routed to the CPE326as will be discussed subsequently, while another class of commands is routed to an upstream command processor332. Validation includes verifying that the host has access rights to a particular VLUN; for example, a particular host may have only read access to the VLUN, and write commands would result in an error condition.

The UPE304also includes a virtual logical unit map334that identifies the storage device and the DPE associated with the VLUN. The command validator330uses the map334to identify the DPE (for example, DPE314) that is associated with the storage device (for example, storage device320) corresponding to the VLUN. The upstream command processor332can then be coupled to the identified DPE. The map334is typically a data structure (for example, a tree or table) that contains data that describes which slices of LUNs on which storage devices comprise the VLUN. The map334is preconfigured by a storage configuration module350, as will be discussed subsequently in greater detail.

In processing a command frame for a read, write, or status command, the upstream command processor332seeks permission from the CPE326to gain exclusive access to the storage device. Once permission is received, the UPE304may engage in wire-speed communications with the selected DPE314corresponding to the data block. The upstream command processor332configures a data frame exchanger336to communicate with the selected DPE314such that subsequently-received data frames associated with the command frame get transferred between the UPE304and the DPE314at wire-speed. The upstream command processor332may inform the CPE326of the status of the execution of commands. All the interactions between the UPE304and the CPE326will be discussed subsequently in greater detail.

Considering now in further detail the central processing element326, the CPE326is coupled to at least one UPE (FIG. 3illustrates two UPEs304,306as coupled to CPE326). CPE326manages access to the VLUNs of the hosts308,310connected to UPEs304,306. The coupling of UPEs to CPE326is preconfigured by storage configuration module350, as will be discussed subsequently in greater detail.

To arbitrate access by a host to a storage device, the CPE326utilizes a permission processor340. The permission processor340receives requests for a period of exclusive access to the storage device from the UPEs304,306that the CPE326manages. In processing such a request, the permission processor340accesses a virtual logical unit state store342that maintains a current state of each VLUN that the CPE326manages. Based at least in part on the current state of the relevant VLUN, the permission processor340may grant permission, deny permission, or defer permission. If permission is granted, the UPE may proceed with wire-speed data transfer, as has been explained above. Thus the CPE326performs contention resolution for the controller302.

Permission is typically granted if the storage device is on-line and no other UPE currently has exclusive access to the storage device. Permission is typically denied if the storage device is off-line, is being formatted, or has been reserved for an unknown time period. Permission is typically deferred for later execution if another UPE currently has exclusive access to the storage device. If permission is deferred, the CPE maintains and manages a queue of access requests, and will inform the proper UPE when its request is granted.

The CPE326also includes a virtual logical unit status manager344that receives status information for commands executed by the upstream command processor332. The VLUN status manager344updates the current state of the VLUN in the VLUN state store342based on the received status information.

As mentioned previously, the UPE passes a second class of commands to the CPE326for processing. This processing is performed by a central command processor346. Some of the commands in this second class include SCSI commands that directly affect or are affected by the VLUN state, such as Reserve, Release, Test Unit Ready, Start Unit, Stop Unit, Request Sense, and the like. Others of the commands in the second class include task management commands that affect the permission granting process or a pending queue, such as Abort Task Set, Clear Task Set, LU Reset, Target Reset, and the like. If any of the second class of commands results in a change of VLUN state, the central command processor346updates the current state of the VLUN in the VLUN state store342accordingly.

As referred to previously, the storage virtualization controller302also typically includes a storage configuration module350. A user352may interact with the controller (either directly through a user interface provided by the storage configuration module350, or through an intermediary system) to define the mapping of VLUNs to LUNs on storage devices318,320,322, and further to DPEs312,314,316. The configuration may be in accordance with user-defined profiles, and can implement desired storage topologies such as mirroring, striping, replication, clustering, and the like. The resulting configuration for each VLUN of a host308,310is stored in the VLUN map334of the UPE304,306connected to the host.

It should be noted that the various processing elements (CPE, DPE, UPE) of the storage virtualization controller302can be implemented using a variety of technologies. In some implementations, each element may include a separate processor, with processing logic implemented either in firmware, software, or hardware. In other implementations, multiple elements may be implemented as separate processes performed by a single processor through techniques such as multitasking. In still other implementations, one or more custom ASICs may implement the elements.

Another embodiment of the present invention, as best understood with reference toFIG. 4, is a method400for accessing a virtual logical unit in a storage area network that may be implemented by the processors of the storage virtualization controller302. Alternatively, the method400may be considered as a flowchart of at least a portion of the operation of the storage virtualization controller302. At402, an upstream processing element receives an access request for the virtual logical unit. The access request is received from a host connected to the upstream processing element, and includes a command frame. At404, a storage device associated with at least a portion of the virtual logical unit is identified by accessing a predetermined virtual logical unit map. At406a downstream processing element associated with the storage device is identified by accessing a predetermined virtual logical unit map. At408, permission for the downstream processing element to perform the access request is requested from a central processing element by the upstream processing element. If permission is denied (“Denied” branch of410), the method400concludes without transferring the data. If permission is obtained (“Granted” branch of410), then, at411, the downstream processing element performs the access request on the storage device, typically by executing at least one command in the command frame on the storage device. Commands, as will be discussed subsequently, may include read data and write data commands. Then, at412, data associated with the access request, such as the data to be read or written, is transferred between the upstream processing element and the downstream processing element at substantially a rated speed of the storage area network after the permission is obtained, and the method400concludes. If permission is deferred (“Deferred” branch of410), then, at414, the access request is placed in a queue to wait for permission to be granted; when permission is received, the method continues at411as previously described.

Considering now in further detail the step at408of obtaining permission, and with reference toFIG. 5, at502a permission request for access to the downstream processing element is sent from the upstream processing element to the central processing element. At504, the central processing element determines whether to grant permission. At506, a permission response is sent from the central processing element to the upstream processing element, and the obtaining permission step408concludes.

Considering now in further detail the step at504of determining whether to grant permission, and with reference toFIG. 6, at602a current state of the storage device, the downstream processing element, or both is checked in order to determine whether the storage device and the downstream processing element are available for access by the upstream processing element. If the checked device or element is not available (“Unavailable” branch of604), then at606the permission is set to “Denied”, and the step504concludes. If the resource is not presently available due to its usage by another UPE, but will become available when that UPE (and perhaps other UPEs with higher priority in a queue) finishes with the usage (“Soon” branch of604), then at612the permission is set to “Deferred”, and the step504concludes. If the resource is available (“Immediate” branch of604), then at608the current state of the resource is set to indicate its usage by the UPE. At610, the permission is set to “Denied”, and the step504concludes.

A further embodiment of the present invention, as best understood with reference toFIG. 7, is a method700for accessing a virtual logical unit in a storage area network that may be implemented by the processors of the storage virtualization controller302. Alternatively, the method700may be considered as a flowchart of the operation of the storage virtualization controller302. The method begins at702by receiving at a first upstream processing element a first request from a first host to access a first virtual logical unit. At704, the first request is processed so as to associate a storage device and a downstream processing element with the first virtual logical unit. At706, permission for the first upstream processing element to exclusively transfer data with the downstream processing element at substantially a rated speed of the storage area network is requested from the central processing element. At708, permission is granted by the central processing element. At710, a second request from a second host to access a second virtual logical unit is received at a second upstream processing element. At712, the second request is processed so as to associate the storage device and the downstream processing element with the second virtual logical unit, setting up a resource contention situation where both the first UPE and the second UPE both require access to the same DPE. At714, permission for the second upstream processing element to exclusively transfer data with the downstream processing element at substantially the rated speed of the storage area network is requested from the central processing element. At716, the central processing element withholds permission at least until such time as the first upstream processing element concludes the exclusive data transfer, thus resolving the contention situation.

Considering now in further detail the processing of a read command by the storage virtualization controller, and with reference to the flow diagram ofFIG. 8where “( )” indicates the order of processing, a Read Command (1) is sent from the host to the UPE. The UPE sends a Command Request (2) to the CPE. When access permission is granted, the CPE sends a Command Acknowledgment (3) to the UPE. The UPE then sends the Read Command (4) to the DPE, which passes the Read Command (5) to the storage device. When the storage device is ready, it returns Data (6) to the DPE, which passes the Data (7) to the UPE, which in turn passes the Data (8) to the host. The transfer of data from the storage device to the DPE to the UPE to the host occurs at wire speed according to the present invention. The storage device also returns Status (9) to the DPE, which passes the Status (10) to the UPE, which in turn passes the Status (11) to the host. The host then sends Status Acknowledgment (12) to the UPE, and the UPE sends Command Done (13) to the DPE.

Considering now in further detail the processing of a write command by the storage virtualization controller, and with reference toFIG. 9where “( )” indicates the order of processing, a Write Command (1) is sent from the host to the UPE. The UPE sends a Command Request (2) to the CPE. When access permission is granted, the CPE sends a Command Acknowledgment (3) to the UPE. The UPE then sends the Write Command (4) to the DPE, which passes the Write Command (5) to the storage device. When the storage device is ready to accept data, it sends Xfer Ready (6) to the DPE, which passes Xfer Ready (7) to the UPE, which in turn passes Xfer Ready (8) to the host. The host then sends Data (9) to the UPE, which passes the Data (10) to the DPE, which in turn passes Data (11) to the storage device. The transfer of data from the host to the UPE to the DPE to the storage device occurs at wire speed according to the present invention. The storage device provides Status (12) indicative of the transfer to the DPE, which passes Status (13) to the UPE, which in turn passes Status (14) to the host. The host then sends Status Acknowledgment (15) to the UPE, and the UPE sends Command Done (16) to the DPE.

Considering now in further detail the rejection of a command by the storage virtualization controller, and with reference toFIG. 10where “( )” indicates the order of processing, a Command (1) is sent from the host to the UPE. The UPE sends a Command Request (2) to the CPE. When access permission is denied, the CPE sends a Command Not Acknowledged response (3) to the UPE. The UPE then sends Status (4) indicative of the command rejection to the host. The host then sends Status Acknowledgment (5) to the UPE, and the UPE sends Command Done (6) to the DPE.

From the foregoing it will be appreciated that the storage virtualization controller, system, and methods provided by the present invention represent a significant advance in the art. Although several specific embodiments of the invention have been described and illustrated, the invention is not limited to the specific methods, forms, or arrangements of parts so described and illustrated. For example, the invention is not limited to storage systems that use SCSI storage devices, nor to networks utilizing fibre channel protocol. This description of the invention should be understood to include all novel and non-obvious combinations of elements described herein, and claims may be presented in this or a later application to any novel and non-obvious combination of these elements. The foregoing embodiments are illustrative, and no single feature or element is essential to all possible combinations that may be claimed in this or a later application. The invention is not limited to the above-described implementations, but instead is defined by the appended claims in light of their full scope of equivalents. Where the claims recite “a” or “a first” element of the equivalent thereof, such claims should be understood to include incorporation of one or more such elements, neither requiring nor excluding two or more such elements.