Storage area network with target side recognition and routing table upload

A network of data processors for providing access to a resource, such as implemented a Storage Area Network that uses iSCSI and Microsoft MPIO-based network communication protocols. In preferred embodiments, the system or method uses (a) target-side consideration of MPIO disk structures, such as by having iSCSI initiators in from iSCSI targets via an iSCSI session object that is settable by a service action; and/or (b) uploading of routing tables from iSCSI targets to iSCSI initiator(s), such as to a Device Specific Module (DSM). Implementations may also involve, in a preferred embodiment: (1) requesting that the contents of the routing table be provided from the iSCSI target side storage servers to the iSCSI initiators, and using this as one of the factors influencing path selection as performed in the MPIO DSM on the initiator side; and/or (2) transmitting information from the iSCSI initiator to the iSCSI target storage server identifying connections as belonging to the same MPIO disk, and then receiving back from the iSCSI target storage server, information about the connections that should be created, and further then having a connection load balancer on the storage server handle those connections as related connections that need to be assigned each to a different Network Interface (NIC) on the storage servers.

BACKGROUND OF THE INVENTION

This invention relates to networked data storage and in particular to target side recognition and host page routing table update, as adapted for use with Multi-Path Input Output (MPIO) or similar protocols.

As computer users rely more and more on access to remote stored information such as for databases, web applications, e-commerce, online transaction processing, and the like, the amount of remotely located information that needs to be managed and stored increases as well. The management of this information continues to provide challenges to information system administrators.

While standard disk servers are quite capable of storing data, their capacity is limited. They can also become a bottleneck if too many users try to access the same storage device at the same time. In one traditional approach to solving this problem, a large number of storage devices are attached to a single network server. A network administrator can then create a “farm” of such servers for a given application.

However, as server farms increase in size, and as companies rely more heavily on data-intensive applications, even this storage model is not completely useful. Recently, a number of vendors have collaborated to develop so-called Storage Area Network (SAN) technology. SANs provide more options for network storage, including much faster access than peripheral devices that operate as Network Attached Storage (NAS). SANs further provide flexibility to create separate networks to handle large volumes of data.

A SAN is a high speed special purpose network that interconnects different kinds of data storage devices with associated data servers on behalf of a larger network of users. A SAN can be clustered in close proximity to other computing resources, but may also extend to remote locations using wide area network technologies such as Asynchronous Transfer Mode (ATM), Fiber Channel, Internet Small Computer Systems Interface (iSCSI), and the like.

SANs typically support disk mirroring, backup and restore, archival and retrieval of archived data, data migration from one storage device to another, and the sharing of data among different servers. SANs can also incorporate subnetworks within network attached storage systems.

Certain improvements have been developed to SANs such as those described in U.S. Patent Publication No. 2004/0143637, entitled “Adaptive Storage Block Data Distribution”, hereby incorporated by reference in its entirety. That design allows a storage area network administrator to control how data is stored and managed without requiring a gateway to monitor all incoming request traffic. The specific systems described in that patent application provide block level data storage services in an iSCSI environment, for example. Such a service may be employed to partition the available data storage across a plurality of equivalent servers (also called “targets” in iSCSI terminology). Each of the equivalent servers presents a similar interface to a client (called “initiators” in iSCSI terminology) and each equivalent server presents the same response to the same request from the client. Each server can also therefore communicate with other servers to notify each other when responsibility for certain data blocks is moved from one server to another server.

To maintain an understanding of the location of different data blocks across different partitions and across the different disk volumes maintained by the data block storage system, each equivalent server maintains a routing table. To this end, each equivalent server executes a routing table process that tracks the different data blocks being stored on the data block storage system and the particular equivalent server that is responsible for each data block. Changes to the routing table can be propagated by the routing table process. More particularly, since the location of the data blocks may change from time to time, routing tables need to be updated to reflect these changes.

In that system, the routing table thus shows which parts of a given volume (which page) reside on which server; each of the servers is considered a member of a particular group of servers. However, the routing tables do not necessarily list all members of a group, and tracking group membership isn't a function of the routing tables.

“MPIO” as implemented in certain versions of Microsoft™ Windows has recently emerged as a solution for providing higher availability and higher performance storage area connectivity, with the so-called multi-path I/O protocols. Multi-path Input/Output (MPIO) is a server, network, and storage configuration that allows for multiple physical connections between a host computer and target storage array. The benefit of this configuration is increased availability via multiple redundant communication paths, and improved performance by simultaneously making available for use all available paths.

Using MPIO, storage arrays can support multiple network connections and networks can be configured to support multiple physical connections. Host systems can also have multiple initiator processes. MPIO can work in many standard operating system configurations such as stand-alone systems and clusters.

Implementation of MPIO is somewhat complex, however, in that multi-path software, initiators, network elements and storage array software must all work together to provide a robust solution. A deficiency in any component can cause the solution not to meet requirements for high availability, quick recovery, and high performance operation.

MPIO solutions have historically been implemented such that each storage array vendor has a custom software solution. This creates a configuration management burden on the end user in the event that the end user desires to deploy different solutions.

Microsoft has recently developed its own Multi-Path I/O framework for use within Windows™ environments. The Microsoft MPIO architecture allows multiple storage systems to share the same MPIO solution. This framework allows, for example, multiple physical device objects to be joined together into a single functional device object called an “MPIO disk”. This framework, however, requires storage vendors to create a Device Specific Module (DSM) for each product in order to “claim” physical objects as a single functional object.

In one standard MPIO solution, the server (the “iSCSI target”) is capable of recognizing such MPIO disk structures. This is supported by having the DSM change iSCSI port names such as, for example, using vendor specific SCSI op codes so that all connections of an MPIO disk share a common SCSI port name.

However, there are at least several problems even with this approach:

1. This information alone cannot be used to provide load balancing among multiple physical connections. Thus, it is not possible to spread connections from a single MPIO disk over all volume members, nor is it possible to even locate two connections on the same interface of a single volume member. In other words, an attempt to load balance cannot take into account a situation where ideally one would spread the set of connections across other available disk volumes. In the event of a network failure, multiple connections might be on a common physical path that has failed, but this path cannot be deleted. It would be better to spread the available connections away from one another across all available physical paths.

2. In addition, the standard DSM does not always receive notification from the higher level MPIO process that an iSCSI connection has been reestablished. In this scenario, iSCSI reservations will not work correctly, which means that clusters will also not operate correctly.

SUMMARY OF THE INVENTION

The apparatuses and methods described herein include data operating systems that manage requests for a plurality of clients for access to a set of resources. In one embodiment, the systems comprise a plurality of servers wherein the set of resources is partitioned across the plurality of servers. The servers execute a load monitor process that is capable of communicating with the other load monitor processes for generating a measure of the client load on the overall system and the client load on each of the respective servers. The server also exchanges available path information with the client processes. More specifically, a client process also provides the server process with information concerning the connections that are servicing Multi-Path objects, so that the server can take this information into account when providing path information back to the clients.

In a preferred embodiment, such as implemented in a Storage Area Network that uses iSCSI and Microsoft MPIO-based network communication protocols, this generally includes:

(a) target-side (server) consideration of MPIO disk structures, such as by informing the target side of MPIO disk structure information via an iSCSI session object that is settable by a service action; and/or

(b) uploading of routing tables from the iSCSI target side to the iSCSI initiator side, such as to a Device Specific Module (DSM) on the iSCSI initiator side.

Implementations may also involve, in preferred embodiments:

(1) requesting that the contents of the routing table be provided from the iSCSI target side storage servers to the iSCSI initiators, and using this as one of the factors influencing path selection as performed in the MPIO DSM on the initiator side; and/or

(2) transmitting, from the iSCSI initiator to the iSCSI target storage server, information that identifies connections as belonging to the same MPIO disk, and then receiving back from the iSCSI target storage server information about the connections that should be created, and further then having a connection load balancer on the storage server handle those connections as related connections that need to each be assigned to a different Network Interface (NIC) on the storage servers.

Further implementation details and the advantages of these features are discussed below.

DETAILED DESCRIPTION OF THE INVENTION

FIG. 1illustrates an example system in which the invention may be implemented. In general, the system may be a Storage Area Network (SAN) such as that described in U.S. Patent Publication 2004/0030755 entitled “Transparent Request Routing for a Partitioned Application Server”, filed Aug. 12, 2003 by Koning, et al.; U.S. Patent Publication 2004/0143637 entitled “Adaptive Storage Block Data Distribution”, filed Jan. 20, 2003 by Koning, et al.; and/or U.S. Patent Publication 2004/0215792 entitled “Client Load Distribution”, filed Jan. 21, 2004 by Koning, et al., each of which are hereby incorporated by reference in their entirety.

Referring toFIG. 1particularly, one or more clients12-1,12-2, . . . ,12-c(collectively referred to as clients12) are connected via a network14, such as the Internet, an intranet, a Wide Area Network (WAN) or Local Area Network (LAN), or by other connection, to servers16-1,16-2,16-3, . . . ,16-sthat are part of a server group16.

The clients12can be any suitable data processing system such as a Personal Computer (PC), a workstation, a handheld computing device, a wireless communication device, or any other such device equipped with a network client process capable of accessing and interacting with the server group16to exchange information with the server group16. The network client12may be any client application that allows the user to exchange data with the server.

Multiple network interfaces are preferably provided with each client12and server16such that multiple communication paths20-pexist between them; these multiple paths will be described in greater detail below.

The clients12and the server group16may use unsecured communication paths for accessing services at the remote server group16. To add security to such an arrangement, the clients and servers can employ any known security group mechanism, such as a Netscape Secured Socket Layer (SSL).

In a preferred embodiment, the hosts and targets rely on storage network communication protocols, preferably iSCSI running over TCP/IP, that support multiple communication paths20-p. Thus, each client12preferably also includes a multipath process such as Microsoft™ Multi-Path Input Output (MPIO)42, a Device Specific Module (DSM) process44, an iSCSI initiator process46, and a network stack such as provided by one or more TCP/IP network interfaces (NICs)48-1, . . . ,48-m. The clients12will also sometimes be referred to herein as iSCSI initiators, and the servers will sometimes be referred to as iSCSI targets.

Each server16-1,16-12and16-3may comprise a commercially available server platform, such as a Sun Sparc™ system running a version of the Unix operating system. Each server16may also include other software components that extend their operation to accomplish the transactions described herein, and the architecture of the servers may vary according to the application.

For example, each server16may have built-in extensions, typically referred to as modules, to allow the servers16to perform the operations described below. Or servers16may have access to a directory of executable files, each of which may be employed for performing the operations, or parts of the operations described below. Certain such modules of interest to the present invention, as depicted inFIG. 1, are an iSCSI target process28-3, and one or more TCP/IP network interface modules (NICs)26-3.

Further, in other embodiments, the servers16-1,16-2and16-3may employ a software architecture that builds certain of the processes described below into the server's operating system, into a device driver, or into a software process that operates on a peripheral device, such as a tape library, a RAID storage system or some other device. In any case, it will be understood by those of ordinary skill in the art that the systems and methods described herein may be realized through many different embodiments, and practices, and that the particular embodiment and practice employed will vary as a function of the application of interest and all these embodiments and practices fall within the scope hereof.

In operation, the clients12will have need to access the resources partitioned across the server group16. Accordingly, each of the clients12will send requests to the server group16. In a typical operation, one of the clients12-2will contact one of the servers, for example server16-2, in the group16to access a resource, such as a data block, page, file, database, application, or other resource. The contacted server16-2itself may not hold or have control over the entire requested resource. To address this, the server group16is configured to make the partitioned resources available to the client12. For illustration, the diagram shows two resources, one resource18that is partitioned over all three servers, servers16-1,16-2,16-3, and another resource17that is partitioned over only two servers16-1,16-2. In the example application of the server group16being a block data storage system, each resource17and18may be a partitioned block data volume. The resources spread over the several servers16can be directories, individual files within a directory, or even blocks within a file.

Other partitioned services can be contemplated. For example, it may be possible to partition a database in an analogous fashion or to provide a distributed file system, or a distributed or partitioned server that supports applications being delivered over the Internet. In general, the approach can be applied to any service where a client request can be interpreted as a request for a piece of the total resource, and operations on the pieces do not require global coordination among all the pieces.

In the embodiment ofFIG. 1, the server group16provides a block data storage service that may operate as a Storage Area Network (SAN) comprised of a plurality of equivalent servers, servers16-1,16-2and16-3. Each of the servers16-1,16-2and16-3may support one or more portions of the partitioned block data volumes18and17. In the depicted system10, there are two data volumes and three servers, however there is no specific limit on the number of servers. Similarly, there is no specific limit on the number of resources or data volumes. Moreover, each data volume may be contained entirely on a single server, or it may be partitioned over several servers, either all of the servers in the server group, or a subset of the server group. In practice, there may of course be limits due to implementation considerations, for example the amount of memory available in the servers16-1,16-2and16-3or the computational limitations of the servers16-1,16-2and16-3. Moreover, the grouping itself, i.e., deciding which servers will comprise a group16, may in one embodiment comprise an administrative decision. In a typical scenario, a group might at first contain only a few servers, perhaps only one. The system would add servers to a group as needed to obtain the level of service required. Increasing servers creates more space (memory, disk storage) for resources that are stored, more CPU processing capacity to act on the client requests, and more network capacity (network interfaces) to carry the requests and responses from and to the clients. It will be appreciated by those of skill in the art that the systems described herein are readily scaled to address increased client demands by adding additional servers into the group16.

II. Load Distribution in General

The clients typically act independently, and as such, the client load placed on the server group16will vary over time. As client demand for access to resources varies, the system can redistribute client load to take better advantage of the available resources in server group16. To this end, in one embodiment the system10comprises a plurality of equivalent servers. As client requests are delivered to the equivalent servers, the equivalent servers coordinate among themselves to generate a measure of system load and to generate a measure of the client load of each of the equivalent servers. In a preferred practice, this coordinating is done in a manner that is transparent to the clients12, so that the clients12see only requests and responses traveling between the clients12and server group16.

In a preferred embodiment, a given client12connecting to a particular server16-1will see the resources managed by the entire server group16as if the group were a single server16-1. The client12is not aware that the server group16is actually constructed out of a potentially large number of servers16-1,16-2, . . .16-s, nor is it aware of the partitioning of the block data volumes17,18over the several servers16-1,16-2, . . .16-s. As a result, the exact number of servers and the manner in which resources are partitioned among the servers16may be changed without affecting the network environment seen by the client12.

Continuing to referring toFIG. 1, in the partitioned server group16, any volume may be spread over any number of servers within the group16. In the example mentioned above, one volume17(resource1) is spread over two servers16-1,16-2, whereas another volume18(resource2) may be spread over three servers16-1,16-2,16-3. Advantageously, the respective volumes are arranged in fixed-size groups of blocks, also referred to as “pages,” wherein an exemplary page contains 8192 blocks. Other suitable page sizes may be employed.

Each server in the group16preferably contains a corresponding routing table15for each volume, with the routing table15identifying the server on which a specific page of a specific volume can be found. For example, when the server16-1receives a request from a client12for partitioned “volume18, block 93847”, the server16-1calculates the page number (page number11in this example for a page size of 8192) by dividing the requested block number by the page size and looks up, in its respective routing table15-1, the number of the server that contains the requested page number11.

If server16-3contains page number11, the request is forwarded to server16-3, which reads the data and returns the data to the server16-1. Server16-1then sends the requested data to the client12. In other words, the response is always returned to the client12via the same server16-1that received the request from the client12. It is therefore transparent to the client12to which server16-1,16-2,16-3it is actually connected. Instead, the client only “sees” the server group16and requests the resources of the server group16.

It should be noted here that the routing of client requests is done separately for each request. This allows portions of a resource to exist at different servers. It also allows resources, or portions thereof, to be moved while the client is connected to the server group16. To accomplish that, the routing tables15-1,15-2,15-3, . . . ,15-sare updated as necessary and subsequent client requests are forwarded to the new server now responsible for handling that request.

At least for a given resource17, the routing tables15are identical. For example, entries in routing tables15-1and15-2for servers16-1and16-2are identical for resource17.

This system is therefore different from a “redirect” mechanism, wherein a server16determines that it is unable to handle requests from a client, and redirects the client to the server that can do so. The client then would have to establish a new connection to another server. Since establishing a connection is relatively inefficient, the redirect mechanism is ill-suited for handling frequent requests.

As shown inFIG. 1, an example server16-3also includes one or more network interfaces (NICs)26-3-1,26-3-2, . . .26-3-q, an iSCSI target process28-3, a request handling process40-3, a load monitor process22-3, and client distribution process30-3. The request handling process40-3handles client requests in the partitioned server16environment by checking if the requested resource is present at an initial server that received the request from the client12. It then examines its routing table15-3, to determine at which server the requested resource is located. If the requested resource is present at the initial server16-3, the initial server16-3returns the requested resource to the client12. Conversely, if the requested resource is not present at the initial server16-3, the initial server will use data from its routing table15-3to determine which server actually holds the resource requested by the client. The request is then forwarded to the server that holds the requested resource, which then returns the requested resource to the initial server. The process request handling process40-3then has the initial server forward the requested resource to the client12.

Turning now toFIG. 2, one particular embodiment of a block data service system10is shown in more detail. Specifically,FIG. 2depicts a system10wherein the client12communicates with the server group16. The server group16includes three servers, server16-1,16-2and16-3. Each server includes a routing table depicted as routing tables15-1,15-2and15-3(which correspond to routing tables15inFIG. 1). In addition to the routing tables, each of the equivalent servers16-1,16-2and16-3include a load monitor process,22-1,22-2and22-3respectively.

Each of the servers16-1,16-2,16-3communicate with other servers in their respective group16, such as to share routing tables15-1,15-2and15-3. As described above, the routing tables15track which of the individual equivalent servers is responsible for a particular resource maintained by the server group16. In the embodiment shown, the server group16may be a SAN, or part of a SAN, wherein each of the equivalent servers16-1,16-2and16-3has an individual IP address that may be employed by a client12for accessing that particular equivalent server on the SAN.

As further described above, each of the equivalent servers16-1,16-2and16-3is capable of providing the same response to the same request from a client12. To that end, the routing tables of the individual equivalent16-1,16-2and16-3coordinate with each other to provide a global database of the different resources. In exemplary embodiments, the servers include data blocks, pages, or other organizations of data blocks, and the individual equivalent servers that are responsible for those respective data blocks, pages, files or other storage organization.

FIG. 3depicts an example routing table15. Each routing table15in the server group16, such as table15-1, includes an identifier (Server ID) for each of the equivalent servers16-1,16-2and16-3that support the partitioned data block storage service. Additionally, each of the routing tables15includes information that identifies those data blocks and pages associated with each of the respective equivalent servers. In the embodiment depicted byFIG. 3, the equivalent servers support two partitioned volumes. A first one of the volumes, Volume18, is distributed or partitioned across all three equivalent servers16-1,16-2and16-3. The second partitioned volume, Volume17, is partitioned across two of the equivalent servers, servers16-2and16-3respectively.

The routing tables15may be employed by the system10to balance client load across the available servers. Specifically load monitor processes22-1,22-2and22-3each observe the request patterns arriving at their respective equivalent servers to determine to determine whether patterns or requests from clients12are being forwarded to the SAN and whether these patterns can be served more efficiently or reliably by a different arrangement of client connections to the several servers. In one embodiment, the load monitor processes22-1,22-2and22-3monitor client requests coming to their respective equivalent servers.

In one embodiment, the load monitor processes22each build a table representative of the different requests that have been seen by the individual request monitor processes. Each of the load monitor processes22-1,22-2and22-3are capable of communicating between themselves for the purpose of building a global database of requests seen by each of the equivalent servers. Accordingly, in this embodiment, each of the load monitor processes is capable of integrating request data from each of the equivalent servers16-1,16-2and16-3to generate a global request database representative of the request traffic seen by the entire block data storage system10. In one embodiment, this global request database is made available to the client distribution processes30-1,30-2and30-3for their use in determining whether a more efficient or reliable arrangement of client connections is available.

FIG. 2also illustrates that the server group16may be capable of redistributing client load by having client12-3, which was originally connected to (communicating with) server16-1, redistributed to server16-2. To this end,FIG. 2depicts an initial condition wherein the server16-1is communicating with clients12-1,12-2, and12-3. This is depicted by the bidirectional arrows coupling the server16-1to the respective clients12-1,12-2, and12-3. As further shown in an initial condition, clients12-4and12E are communicating with server16-3and no client (during the initial condition) is communicating with server16-2. Accordingly, during this initial condition, server16-1is servicing requests from three clients, clients12-1,12-2, and12-3. Server16-2is not servicing or responding to requests from any of the clients.

Accordingly, in this initial condition the server group16may determine that server16-1is overly burdened or asset constrained. This determination may result from an analysis that server16-1is overly utilized given the assets it has available. For example, it could be that the server16-1has limited memory and that the requests being generated by clients12-1,12-2, and12-3have overburdened the memory assets available to server16-1. Thus, server16-1may be responding to client requests at a level of performance that is below an acceptable limit. Alternatively, it may be determined that server16-1, although performing and responding to client requests at an acceptable level, is overly burdened with respect to the client load (or bandwidth) being carried by server16-2. Accordingly, the client distribution process30of the server group16may make a determination that overall efficiency may be improved by redistributing client load from its initial condition to one wherein server16-2services requests from client12-3.

Considerations that drive the connection load balancing decision may vary. One example may be the desire to reduce routing: for example, if one server is the destination of a significantly larger fraction of requests than the others on which portions of the resource (e.g., volume) resides, it may be advantageous to move the connection to that server. Another example may be to balance server communications load: if the total communications load on a server is substantially greater than that on some other server, it may be useful to move some of the connections from the highly loaded server to the lightly loaded one, and balancing of resource access load (e.g., disk I/O load)—as in preceding example but for disk I/O load rather than communication load. This is an optimization process that involves multiple dimensions, and the specific decisions made for a given set of measurements may depend on administrative policies, historical data about client activity, the capabilities of the various servers and network components, etc.

To this end,FIG. 2depicts this redistribution of client load by illustrating a connection325(depicted by a dotted bi-directional arrow) between client12-3and server16-2. It will be understood that after redistribution of the client load, the communication path between the client12-3and server16-1may terminate.

Balancing of client load is also applicable to new connections from new clients. When a client12-6determines that it needs to access the resources provided by server group16, it establishes an initial connection to that group16. This connection will terminate at one of the servers16-1,16-2, or16-3. Since the group appears as a single system to the client, it will not be aware of the distinction between the addresses for16-1,16-2, and16-3, and therefore the choice of connection endpoint may be random, round robin, or fixed, it will not be responsive to the current load patterns among the servers in group16.

When this initial client connection is received, the receiving server can at that time make a client load balancing decision. If this is done, the result may be that a more appropriate server is chosen to terminate the new connection, and the client connection is moved accordingly. The load balancing decision in this case may be based on the general level of loading at the various servers, the specific category of resource requested by the client12-6when it established the connection, historic data available to the load monitors in the server group16relating to previous access patterns from server12-6, policy parameters established by the administrator of server group16, etc.

Another consideration in handling initial client connections is the distribution of the requested resource. As stated earlier, a given resource may be distributed over a proper subset of the server group. If so, it may happen that the server initially picked by client12-6for its connection serves no part of the requested resource. While it is possible to accept such a connection, it is not a particularly efficient arrangement because in that case all requests from the client, not merely a fraction of them, will require forwarding. For this reason it is useful to choose the server for the initial client connection only from among the subset of servers in server group16that actually serve at least some portion of the resource requested by new client12-6.

This decision can be made efficiently by the introduction of a second routing database. The first routing database15described earlier specifies the precise location of each separately moveable portion of the resource of interest. Copies of that first routing database's need to be available at each server that terminates a client connection on which that client is requesting access to the resource in question. However, the second “connection balancing” routing database simply states for a given resource (as a whole) which servers among those in server group16currently provide some portion of that resource. For example, a connection balancing routing database describing the resource arrangement shown inFIG. 1consists of two entries: one for resource17that lists servers16-2and16-3, and one for resource18that lists servers16-1,16-2, and16-3. The servers that participate in the volume have the entire routing table for that volume as shown onFIG. 3, while non-participating servers have only the bottom section of the depicted routing table. In a partitioning of the two volumes as shown inFIG. 1, this means that server16-1has the bottom left table shown inFIG. 3and both of the tables on the right side of the figure, while servers16-1and16-2have all four tables depicted.

III. Connection Load Balancing with Target Side Recognition of MPIO and Routing Table Upload

As alluded to previously in connection withFIG. 1, the client12may preferably be an iSCSI compatible host40running Microsoft's MPIO42, a DSM44, and iSCSI initiator46processes. MPIO solutions use redundant physical network components—such as multiple network interfaces26, cables, and switches—to create multiple logical and physical paths between each client12(host) and the servers (storage devices)16. For example, a given server16-3may actually have three NICs26-3-1,26-3-3and26-3-3. In addition to being used for further load balancing, in the event that one or more component fails thereby causing the path to fail, logic implemented by the MPIO module42uses an alternate I/O path so that applications can still access their data. Multiple physical network connections20-1,20-2. . . ,20-nare therefore provided between a given host40(on client12-1) and a given server16-2.

The additional connections can be provided in a secure environment in various ways. In a preferred embodiment, this can be done using the Diffie-Hellman (DH) algorithm, avoiding the need to manage host passwords and the like. For example, once the user logs in using host-side security protocols, the additional connections are installed as secure channels using known DH protocols in IPsec. Thus an ideal solution to routing requests also distributes the requests over the available multiple MPIO connections as well.

I. Target Side Recognition of MPIO Disks

It is therefore desirable to provide an enhancement to previous MPIO solutions by including a session object in a process running in an iSCSI initiator12. The session object can carry identifiers for MPIO structures to the iSCSI targets16. In one embodiment the MPIO identifiers can be settable by a new service action of a iSCSI vendor specific op code.

In a preferred embodiment, informing the target side of the MPIO disk structure can involve cooperation between various modules on the client (iSCSI initiator)12, such as the kernel DSM44and a specific Windows service (EHCM49), and other modules on the server (iSCSI target16), specifically a Connection Load Balancer (CLB)33portion of client distribution process30-3. In particular, a dialog is established between the host EHCM process and the CLB in the target. Thus, the target need not automatically “recognize” MPIO disk structures, but rather, it is preferably specifically informed of the existence of MPIO disk structures. In one implementation, this can occur as follows.

The EHCM module49server in the client sends a job request,(IOCTL_EQLDSM_EHCM_WORK_RQST ioctl)
to the DSM44on the host. The response tells the EHCM49what type of job is required. With each job request response, the DSM kernel can also return the time at which the EHCM should send the next request.

Two jobs on the client side are relevant here:

EqlDsmIoctlJob::CONNECTION_SETUP and

each of which are now described in more detail in connection withFIG. 4.

Step401: Additional MPIO and iSCSI information is obtained from various userspace sources.

Step402: Using a particular service action of a vendor-specific opcode (such as op code20h), any necessary corrections are made to the DSM endpoint information (for example, if iSCSI has moved a connection).

Step403: All the information is then sent to the target16(the CLB33).

Step404: The target16determines an optimal MPIO configuration and sends its recommendations back to the client12.

Step405: The EHCM49(which is in client12userspace, of course) removes and adds connections as appropriate.

Step410: A table upload request is transmitted to the target.

Step411: Corrections are then made to the DSM endpoint information as required (as above).

Step412: The entire routing table is then read (e.g., in 64 KB chunks).

Step413: The routing table is downloaded to the DSM44using the ioctlIOCTL_EQLDSM_ROUTING_TABLE_UPLOAD.

FIG. 5illustrates more detail of the corresponding response process on the target side, which is to return TCP/IP endpoints of connections to be used by the iSCSI initiator. Initiated by a service action on a vendor specific op code, a first step500is to determine which servers16are holding part of the requested resource, such as by consulting the routing table(s)15. Next, in step502, a determination is made as to which NICs26-3are actually present on those corresponding servers16. In step504, any of those NICs not presently operating, or not available for assignment to the initiator in question (according to the CLB process33-3) are omitted from the list. Finally, in step406, the resulting list is returned to the initiator12.

By returning a list of connections (TCP/IP address pairs) that should be established, then there is an implication back to the initiator12that any other existing connections should be terminated.

In this way, the initiator12is informed of allowable connection information, without having to be informed of the information in tables15that the servers16use among themselves. The initiator12can then use whatever algorithm is suitable (e.g., round robin, least-queue-depth, lowest-latency, etc.) to manage the connections it has been told about.

FIG. 6shows the result. Here, a given iSCSI initiator12can now make use of four (4) physical NICs without knowing further details of the available three (3) servers in the group16at the iSCSI target that each have three (3) available NICs, for a total of nine (9) physical connections there. The MPIO process at the initiator12can now simply balance the load among these available connections of which it has been informed.

It can now be understood that the following advantages are provided by this implementation.

1. Information about MPIO disks can now be used by the connection load balancer (CLB33) component, which is logically a function of the group16. Thus, when the connection process queries the network protocol stack26on each given storage server16for information, it can return [connection, MPIO identifier] pairs. Since the EHCM49now transmits the MPIO configuration information directly to the CL33, the rest of the system need not be aware of MPIO identifier information. In other words, the connection load balancer is now also aware that a set of connections are related and that they should be spread across several available physical connections20, away from each other. This can be implemented by making the connections go to different storage arrays (different servers16) or to different NICs26servicing a given server16.

2. When a connection is closed by initiator12, it can now notify other sessions that correspond to the same MPIO disk and those sessions can return a vendor specific check condition back to the DSM44. This provides the DSM44with a reliable notification mechanism for connection changes.

II. Uploading of Routing Table

The DSM model, as specified in Microsoft's MPIO, allows a hardware vendor to perform device specific load balancing. With the prior art, this can entail, for example, routing a iSCSI command to the least busy NIC26. In that case, one thing that can be done is to route commands directly to the server16where the requested data is located, rather than forcing the iSCSI target side (that is, the servers16) to do the routing among themselves as explained above.

This approach requires uploading a routing table to the DSM44process in the host, preferably as demanded by the DSM44. This can be periodic or whenever changes are made. This is an inexpensive process since the routing table is typically quite small. With this approach, the DSM44does not need to actually balance the connection since it does not have necessary per-interface information. The DSM44can now, create at least as many connections as there are volume members. It is therefore always possible to route client side but determine the best target side endpoint.

Once a connection is established, the DSM44can use a service action (vendor specific op code20hand service action03has described below) to assess the situation, but that may involve target side routing. There may be load balancing considerations or network considerations, i.e., sub-network considerations, that are more important than doing all routing on the client12.

The new target functionality can be implemented as three new service actions for vender specific SCSI op code20h, as sent from the initiator (DSM44) to the iSCSI targets. The new target functionality can thus define three opcode20hservice actions as:

Service Action Assignments

This is the standard service action implemented for op code20h. It sends a message to the volume leader, advising it of a name change and then updates the session is iSCSI port name. This service action can remain unchanged for backward compatibility, although it is no longer used.

In response to this service action the following actions are then taken.

1. Information concerning the connections being used for a particular MPIO object is conveyed to the target. This can be conveyed as a list of connections. However, in an optional configuration, rather than provide such a list, a Globally Unique Identifier (GUID) can be defined and then subsequently used for each MPIO object.

2. Standard service action0is then run. This involves sending a message to the volume leader to let it update its tables so that it will not orphan any reservations already held by that session.

This service action also has no input parameters. The response data is the routing table. In one embodiment, this may be a message that begins with a header giving the number of volume pages, the volume members and the storage array identifier of each volume member, with the header being followed by bit fields for each volume member. However, other formats are possible.

This service action has no input. It returns a universal storage array identifier (UUID) of the volume member on which the connection is terminated.

In addition, the following two request/response messages are also provided between processes:

CDHKeyExchReq/CDHKeyExchRsp; and

The first of these message pairs is used to dynamically create temporary security/login (CHAP) credentials, which are used to securely “clone” the initial iSCSI connection created directly by the user (using Diffie Hellman as alluded to above).

The second of the message pairs is how the host transmits MPIO configuration information to the target (i.e., to the CLB33process), and then subsequently receives instructions on how to optimally set up the MPIO connection mesh.

By sending updated routing tables to the host, the host DSM44can then take into account routing considerations when performing path selection.

Also, this process informs the host12of the number of storage servers16involved in providing a volume17or18(this is part of the routing data). This allows the host12to open at least that many connections. When it does that, the storage group recognizes that these connections all come from the same host12and it can now spread them over the storage servers rather than simply doing the connection load balancing separately for each connection.

More particularly, the connections are all marked as belonging to the MPIO disk, such as via the information conveyed in service action1. This allows the connection load balancer33to know that they are related, and to do the balancing differently than it would otherwise. Normally, the connection load balancer33, when presented with a set of connections to handle, would place each connection individually, since it doesn't know of or assume any relationship among the connections. As a result, depending on the current load patterns in the group, several connections may share resources (such as storage server NICs).

However, when the connection load balancer33is handling a set of connections all of which relate to a single MPIO disk, it can now avoid assigning them to the same NIC. This ensures that the multiple MPIO connections will not in fact share a path, which would defeat the fault tolerance and performance enhancing benefits MPIO sets is out to deliver. The MPIO client is told by the storage server (by the connection balancer) how many connections it should create, which takes into account issues such as failed interfaces. As a result it should normally be the case that the client will open as many connections as can be spread over the available interfaces and no more than that—and the connection load balancer33can indeed spread them that way because it knows that they all relate to the same MPIO disk.