Disk-less quorum device for a clustered storage system

A disk-less quorum device in a clustered storage system includes non-volatile memory to store status information regarding the cluster and each storage controller in the cluster. The quorum device maintains a bitmap, shared by the controllers in the cluster, in the non-volatile memory. The bitmap indicates the status of a write operation to any data block or parity block. A “dirty” data unit in the bitmap indicates that a write operation has been submitted but is not yet finished. Upon submitting a write request (to update a data block or a parity block) to the storage facility, a controller sets the corresponding data unit “dirty” in the bitmap. After receiving an acknowledgement from the storage facility indicating that the operation has been completed, the controller clears the corresponding data unit. If a controller fails during a write operation, another controller can use the bitmap to re-establish data consistency.

FIELD OF THE INVENTION

At least one embodiment of the present invention pertains to network data storage systems, and more particularly, to a clustered data storage system.

BACKGROUND

In the field of network data storage, a clustered data storage system links multiple controllers to provide redundancy of storage access.FIG. 1shows an example of a clustered data storage system. As shown, the system includes multiple controllers1. Each controller1is coupled locally to a storage facility2. The storage facility2is managed by each of the controllers1. The storage facility2may be, for example, one or more conventional magnetic disks, optical disks such as CD-ROM or DVD based storage, magneto-optical (MO) storage, or any other type of non-volatile storage devices suitable for storing large quantities of data. The storage facility2can be organized as one or more Redundant Array of Independent Disks (RAID) groups, in which case each controller1accesses the storage facility2using an appropriate RAID method.

A controller1receives and responds to various read and write requests from a host (not shown inFIG. 1), relating to volumes, Logical Unit Numbers (LUNs), files, and/or other logical containers of data stored in (or to be stored in) the storage facility2. If one of the controllers1fails, another controller1can take over for the failed controller1without sacrificing performance.

Each of the controllers1is also coupled to a quorum device4via a network3, which may operate based on a conventional protocol such as InfiniBand or Fibre Channel, Internet Protocol (IP), or other protocol(s). A quorum device, such as quorum device4inFIG. 1, is a device that stores state information regarding the cluster and each controller1in the cluster, including identification of nodes in the cluster, which nodes are active (or should be active), which nodes are not active, etc. In a conventional technique, the quorum device4is implemented with a disk-based storage device (“disk”). Data stored on the disk is frequently updated to reflect the current status of each controller1. However, each access to a quorum disk is time consuming, such that accesses to the disk decrease the clustered storage system's throughput and performance.

In addition, in one known technique a clustered storage system such as shown inFIG. 1uses non-volatile random access memory (NVRAM) to ensure data integrity in the event of a failure of one or more controllers1. Each controller1has an NVRAM to store a log of each write request and the associated write data received at the controller1from a host. Such log data, which is sometimes called “NVLog”, is also transmitted by the controller1that creates it to one or more other controllers1in the cluster. Thus, each controller1has a local copy of the NVLog of another controller1. Therefore, if one of the controller1fails before completion of a particular write request, a different controller1can re-submit the write request to ensure data integrity, by using its local copy of the failed controller's NVLog.

However, transmitting each write request between the controllers1in order to share NVLogs incurs substantial network traffic, thus decreasing the clustered storage system's throughput and performance. Even in the case of a very simple clustered system that has only two controllers, a first controller's sending a write request and data to the other controller doubles the load on the first controller in connection with that write request. In a larger cluster, the load would increase geometrically with each additional controller to which the first controller needs to send its received write requests and data.

SUMMARY OF THE INVENTION

One aspect of the invention is a method of responding to a failure in a clustered storage system which includes a plurality of storage controllers configured as failover partners, where the failure has caused a write operation by a first storage controller to fail to complete, and the write operation was initiated in response to a write request from a host, the write request requesting that a set of data be written. The method includes completing the write operation from a second storage controller of the plurality of storage controllers without using any log of the set of data and without requiring any communication from the first storage controller to the second storage controller.

Another aspect of the invention is a storage controller and/or other apparatus that can perform the method.

Yet another aspect of the invention is a quorum device for use in a clustered storage system, the quorum device including a memory and a network interface. The memory is to store a data structure corresponding to a storage device in the clustered storage system, where the data structure includes a separate data unit for each of a plurality of storage blocks in the storage device. Each data unit indicates whether a write operation is pending for the corresponding storage block. The apparatus receives updates of the data structure, through the network interface, from a plurality of external storage controllers that can access the storage device.

Still another aspect of the invention is a method corresponding to the functionality of said quorum device.

Other aspects of the invention will be apparent from the accompanying figures and from the detailed description which follows.

DETAILED DESCRIPTION

A method and system of using disk-less quorum device in a clustered storage system are described. References in this specification to “an embodiment”, “one embodiment”, or the like, mean that the particular feature, structure or characteristic being described is included in at least one embodiment of the present invention. Occurrences of such phrases in this specification do not necessarily all refer to the same embodiment.

The present invention includes a technique of using a disk-less quorum device for a clustered storage system. In one embodiment, the disk-less quorum device includes non-volatile memory to store not only the usual state information regarding the cluster and each controller in the cluster, but also write status information for individual blocks in a storage facility. This approach eliminates the need to maintain NVLogs (logs which store the write data) and eliminates the need to communicate write requests and data between controllers in a cluster (either directly or indirectly).

In particular, the disk-less quorum device stores in NVRAM a bitmap shared by all of the controllers in a cluster, which indicates the status of write operations to data blocks and parity blocks. In one embodiment, the bitmap includes a separate data unit (e.g., a bit or a predetermined number of bits) for each data block and each parity block in the storage system and for the currency of data/parity. Each such data unit indicates whether a write operation is pending for the corresponding block. Upon submitting a write request to a storage facility to update a data block or a parity block, a controller sets the corresponding data unit in the bitmap in the quorum device (sometimes called setting the data unit “dirty”). After receiving an acknowledgement from the storage facility that the write operation has been successfully completed, the controller clears that data unit (sometimes called setting the data unit “clean”). What is meant by “completed” or “completion” of a write request in this context is that the data has been successfully written to the storage facility. If a controller fails during a write operation, another controller can use the information in the bitmap to re-establish data consistency.

This technique has several advantages over the prior approaches described above. For example, because the disk-less quorum device is physically separate from the controllers and the storage facilities, it is less likely to be affected if a controller or storage device fails. If any particular controller needs to be replaced, the pending write status information will not be lost. Further, in contrast with the prior technique of maintaining NVLogs and communicating write requests between controllers, this technique requires no NVLogs (log of write data) be maintained at all, while providing at least the same level of failure protection. And, because write requests and data do not need to be communicated between controllers in a cluster, much less data needs to be transferred between devices, thereby reducing network bandwidth consumption and processing load on the controllers. Further, the disk-less quorum device is implemented with NVRAM, which in general is much faster at handling input (I/O) operations than disk devices.

FIG. 2illustrates a clustered storage system, according to an embodiment of the present invention. As shown, the clustered storage system includes a number of controllers1. Each of the controllers1can be, for example, a storage server, such as a file server, a block-level storage server, or a combination thereof. Each controller1is coupled locally to each of a plurality of storage facilities2(e.g., disks, RAID groups or other type of storage devices). A controller1receives and responds to various read and write requests from a particular host5or set of hosts5, relating to volumes, LUNs (Logical Unit Numbers), files, and/or other data units stored in (or to be stored in) the storage facilities2. Each storage facility2may be, for example, one or more conventional magnetic disks, optical disks such as CD-ROM or DVD based storage, magneto-optical (MO) storage, or any other type of non-volatile storage devices suitable for storing large quantities of data. Each storage facility2can be organized as one or more RAID groups, in which case each controller1accesses a storage facility2using an appropriate RAID method.

Each of the controllers1is coupled to a quorum device24via a storage network3(e.g., InfiniBand, Fibre Channel, etc.). Each controller1can use a conventional protocol to communicate with the quorum device24, such as Fibre Channel Protocol (FCP), Internet Small Computer System Interface (iSCSI), Internet FCP (iFCP), InfiniBand, or any other protocol(s) suitable for storage access. In one embodiment, a quorum device24is disk-less, such that its internal storage space is implemented, for example, using NVRAM. Data stored in a quorum device24can be, therefore, accessed much faster than a disk-based quorum device.

A quorum device24may include an internal memory controller (not shown), through which controllers1can access the internal NVRAM of the quorum device24. In other embodiments, however, a quorum device24does not include an internal memory controller. In such an embodiment, the memory (e.g., NVRAM) within the quorum device24may be directly accessible to external devices such as the controllers1in the cluster, by using, for example, remote direct memory access (RDMA).

In one embodiment, the clustered storage system has two quorum devices24, i.e., a primary quorum device and backup quorum device, as shown inFIG. 2, with mirrored NVRAM. In this embodiment, the primary quorum device may be normally active while the backup quorum device is normally passive. In that case, the backup quorum device stores exactly the same data as the primary quorum device in their respective NVRAMs. If the primary quorum device fails, the backup quorum device provides the ability to recover the data in the failed quorum device.

Alternatively, this configuration might not use mirrored NVRAM, in which case the controllers1need to update the state of all requests in the backup quorum device in case the active quorum device fails. Note that a fallback option for a controller1in the event both (all) quorum devices24fail is to use the storage2as disk-based quorum.

A quorum device24stores vital data, including information indicating the state of each controller1in the cluster. In one embodiment, a quorum device24also includes and maintains a bitmap indicating each and every pending, executing, and/or confirmed write request received at any of the controllers1from any of the hosts5, to keep track of the state of the write operations. This bitmap can be used for recovery in the event of a failure.

A quorum device24can be accessed from more than one controller1. Because the quorum device24is external to the controllers1, adding or removing a controller1from the cluster only requires simple reconfiguration of the quorum device24(or possibly the storage facility2, depending on the implementation, e.g., maximum number of controllers1, etc.). The storage space of the quorum device24may be partitioned based on any of various criteria, such as controller, storage device, device access, and/or failover capability.

FIG. 3illustrates an example of a quorum device according to an embodiment of the present invention, including a partitioning of the quorum device's storage space. In the illustrated embodiment, the quorum device includes NVRAM31and a network interface34through which external storage controllers1can access the NVRAM31. The network interface34may be, for example, any conventional type of network interface, such as an Ethernet adapter, Fibre Channel adapter, InfiniBand adapter, or the like.

The storage space300of the quorum device24is implemented within NVRAM31in the quorum device24. To facilitate description it is assumed that the clustered storage system has three controllers1and three storage facilities2. Hence, as shown, the quorum device's storage space300is divided into four sections301,302,303, and304. Each of the sections301-303is reserved for a different one of the three storage facilities2. The cluster details section304is reserved for storing data regarding the state of the cluster, including the state of each controller1. Each of the sections301-303stores a bitmap305indicating, for each block of the corresponding storage facility2, whether a write request is pending or done. In one embodiment, each bitmap305is variable in size, but the size is the same for all bitmaps305in the system. Each bitmap305includes information indicating the current owning controller1and state information regarding the write requests from that controller.

In one embodiment, each bitmap305includes a separate data unit for each data block and each parity block in the associated storage facility and information regarding the relative currency of data and parity. The data units in the bitmap may each be a single bit or a predetermined number of bits, for example. To facilitate explanation, the term “bit” is used henceforth in this description to refer to each such data unit; however, it should be understood that the term “bit” is intended to mean a data unit of any length.

FIG. 4illustrates an example of the cluster details section304. As shown, section304includes a cluster ID field310to record the identifier (ID) of the cluster. The cluster online field311stores the time the cluster was brought online. The epoch312is a field which is incremented every time the state of the cluster changes. A change may be, for example, bringing the cluster online, joining or leaving of a controller1, etc. Change of epoch313records the last point in time the epoch312was changed. Member list314stores IDs of a list of controllers1and the location within section304storing each controller's status information. For example, section315stores status information of Controller—1.

In further detail, section315includes a controller ID field320, a current update field321, an epoch field322, and a joined field323. The current update field321is the time when this controller most recently updated its own status. This field is used to compare against the other time values to determine whether this controller does periodic updates. If a controller1fails, it will cease to update its current update field321in the quorum device31. Accordingly, if the controller has not updated its status for a certain amount of time, the quorum device will declare this controller non-operational and update the deadzone field316, which is described below. The epoch field322is compared with the cluster's epoch field312to make sure that the two epochs' values are the same. If the values are different, that means the controller has not been updated to get the most recent status information regarding the cluster. Joined field323stores the time the controller joined the cluster. Additional fields may also be present in section315, depending on type of implementation.

The deadzone field316is a field that includes an identifier of any controller that is to be disabled from the cluster due to, for example, faulty or offensive behavior of the controller, loss of connectivity, missing periodic updates, etc. When the quorum device determines that the current update field321of a controller1has not been updated for more than a specified period of time, it will update the deadzone field316to identify that controller, resulting in the controller being logically removed from the cluster. The deadzone field316may also include a code indicating the reason. Controllers1check this field periodically and on startup, in response to cluster changes, etc., to make sure they are allowed to stay online.

FIG. 5illustrates an example of the details of a bitmap305shown inFIG. 3.FIG. 5only shows a portion of the bitmap305. The illustrated example of the bitmap305is based on a configuration in which a RAID-6 parity scheme is used to organize the data/parity blocks and each stripe includes six data blocks, D1, D2, . . . , D6. Note, however, that this particular configuration is discussed here only for purposes of illustration.

RAID-6 uses two independent parity blocks, P1and P2, for each stripe, which are calculated based on the very same blocks but using different algorithms: For example, P1may be calculated as an exclusive OR (XOR) while P2is calculated as a Reed-Solomon (RS) encoding. Note, however, that the technique being introduced here can also be used with redundancy schemes other than RAID-6, such as one in which each stripe has only a single parity block, or some other number of parity blocks. Hence, the bitmap305will have a single field or a number of fields analogous to P1and P2, depending on the redundancy scheme being used. Furthermore, computation of parity based on XOR and/or RS encoding is just an example; the particular algorithm by which parity is calculated is not germane to the technique being introduced here.

In the embodiment illustrated inFIG. 5, for each number N of stripes in a given storage facility2, the bitmap305includes a header section400and N stripe information sections430. The header section400records information such as the owner of the N stripes of data blocks, for the purpose of determining whether a write request from a controller1(shown inFIG. 2) should be authorized (e.g., to prevent concurrent access to the storage facility2by different controllers). Each stripe information section430represents a stripe of a RAID array. Each of the fields404-409represents a data block in a stripe. Field410represents a first parity block P1of the stripe and the field411represents a second parity block P2of the stripe. In one embodiment, each of the fields404-411comprises a single bit. A “1” value of one of the fields404-409indicates that a write operation on the corresponding data block is pending. A “1” value of P1or P2indicates the corresponding parity block is being updated. In one embodiment, P1is computed as an XOR of the data blocks in the stripe (D1, D2, D3, D4, D5, D6) and P2is computed as a Reed-Solomon encoding of those data blocks. Upon changing of any of the data blocks D1-D6, P1and P2will be recomputed.

In one embodiment, the D time indicator field412indicates the time a data block of the corresponding stripe is updated; the P1time indicator field413indicates the time P1is updated; and the P2time indicator field414indicates the time P2is updated. These time indicator fields412-414are collectively called “currency information” herein. By examining this currency information, a controller can determine the order in which a data block, P1, and P2were updated for a given stripe in response to a write request. This information can be used in the event of a failure or an incomplete write, to determine whether data and/or parity are correct as stored on disk. For example, if a failure occurs while processing multiple write requests to the same stripe, and at least one write but not all of them has been acknowledged to the host before the failure occurred, then some data in the stripe may be dirty; therefore, it is necessary to determine which is most current on disk, data or parity, to reestablish data consistency.

Of course, bitmaps used for this purpose do not have to have this particular format. For example, depending on the implementation, a bitmap305may include a greater or smaller number of bits to track state. Also, note that not all values in the bitmap are sent by a controller1; some are updated by the quorum device24to reflect the currency of each block, as noted above.

In another embodiment, the time indicators412-414in the bitmap305are replaced, for each stripe, by a set of counters which indicate the order in which the blocks in the stripe were written. More specifically, the currency information may include a separate counter field for each data block and each parity block in each stripe, where the value of each counter indicates the order in which the corresponding block was written for that stripe. Therefore, the number of bits needed to implement each counter depends on the total number of blocks per stripe (including data and parity blocks). For a stripe with eight total blocks, e.g., six data blocks and two parity blocks as described above, a three-bit counter field is required for each block, to represent eight possible sequence values. The counter fields for each stripe are reset/rewritten any time a block in the stripe is rewritten.

In yet another embodiment, instead of using time indicators412-414or counters as currency information, a separate set of “sub-bitmaps” is included in (or associated with) the main bitmap305for each stripe, for that purpose. More specifically, for each stripe, a separate sub-bitmap is provided for each parity block in the stripe. Hence, if each stripe includes two parity blocks, then the bitmap305will have two such sub-bitmaps for each stripe. Each sub-bitmap includes a separate value (field) for each data block in the stripe, to indicate whether the data block or the parity block is more current. The sub-bitmap, therefore, essentially provides a pointer to the last updated block(s) in the stripe.

For example, if each stripe has M data blocks, D1through DM, and two parity blocks P1and P2, then for each stripe the bitmap305can include (or be associated with) two sub-bitmaps as follows:

where the value of Pi/Dj indicates whether Pi has been updated to reflect the last write to Dj (i=1, 2, . . . , M and j=1 or 2 in this example). For example, the value of P2/D1indicates whether P2has been updated to reflect the last write of D1, and so forth.

Each such value (P1/D1, etc.) can be a single bit. Each such value is set at the beginning of any write operation that affects the corresponding data block and is cleared only when the corresponding parity block (e.g., P1or P2) has been updated to reflect the write to the data block. A value of “0” can be used to indicate that the parity block was updated when the data block was last written (i.e., parity is “clean”) while a “1” represents that the parity block has not been updated when the data block was last written (parity is “dirty”). This information can be used in the event of a failure or incomplete write, to determine whether data and/or parity are correct as stored on disk.

For example, assume a write to a single data block, D3, is to be performed in a given stripe. In that case, the fields in the main bitmap305corresponding to D3, P1and P2all will initially be marked dirty. In addition, the values P1/D3and P2/D3for that stripe will also initially be marked dirty.

If the write to D3and the associated update of P1succeed but the update of P2fails, then D3and P1will be marked clean in the main bitmap305, but P2will remain dirty. In addition, the value P1/D3will be cleared to “0” to indicate that P1has been updated to reflect the write to D3; however, the value P2/D3will still be at “1”, indicating that P2has not been updated to reflect the write to D3. Recovery from this type of situation is discussed below in connection withFIG. 7.

FIG. 6is a flow diagram illustrating a normal process by which a controller1performs a write operation, including updating information maintained in the quorum device24, according to an embodiment of the present invention. At step601, the controller1receives a write request from a host5. At step602, the controller1determines whether the host5has authorization to access the target block(s). In one embodiment, the controller1maintains an access control list (ACL) for this purpose. The ACL specifies whether a host5has the right to access various data blocks. Further, in one embodiment the controller1can access the header400of the bitmap305(shown inFIG. 5) to determine whether the requesting host5has access rights. If the requesting host5has access rights, the process goes to step602.

At step603the controller1breaks the write request into one or more block-level write requests (assuming the request was not already in such format) and then processes each of the block-level write requests one by one, according to steps604through614. In other embodiments, the write request from the host5may already specify a particular target block.

The controller1first selects one of the block-level write requests to process at step604. At step605, the controller1then updates the bitmap (e.g., bitmap305shown inFIG. 5) in the quorum device24to indicate that a write operation on the target block is pending. In one embodiment, the controller1sets the particular bit (D1, D2. . . , D6) which corresponds to the target block to “1”. Next, at step606, the controller1updates the bitmap to indicate that an update of the parity block P1is pending. In one embodiment, the controller1sets the bit corresponding to P1in the bitmap to “1”. At step607, the controller1then updates the bitmap to indicate that an update on the parity block P2is pending, e.g., by setting the bit corresponding to P2in the bitmap to “1”. At step608, the controller1sends the new (updated) data block to the storage facility2. After receiving the new data block, the storage facility2stores the new data block and returns an acknowledgement to the controller1. At step609, the controller1computes the parity block P1based on the new data block. At step610, the controller1sends the new parity block P1to the storage facility. At step611, the controller1computes the parity block P2based on the new data block. At step612, the controller1sends the new parity block P2to the storage facility2. In one embodiment, P1is calculated as an XOR operation on the data blocks of the corresponding stripe, whereas P2is calculated by using Reed-Solomon operation on the data blocks of the corresponding stripe.

Assuming neither controller1nor the storage facility2fails, then after the storage facility2performs the update of the data block D, the parity block P1, and the parity block P2, the storage facility2sends acknowledgements of those actions back to the controller1. Thus, in the normal course of operation of a storage system such as shown inFIG. 2, at step613, the controller1receives acknowledgements from the storage facility1indicating the success of the updates. Then at step614, the controller1updates the bitmap in the quorum device24to indicate that the block-level write request has completed. In one embodiment, the controller1does this by clearing the bits corresponding to D, P1and P2in the bitmap (i.e., sets them “clean”). The controller1then repeats steps604through614for another one of the block-level write requests until (step615) all of the block-level write requests have been completed.

Finally, after all of the block-level write requests have been completed, at step616the controller1sends an acknowledgement (“ACK”) to the host5that initiated the write request, indicating completion of the host-initiated write request.

Note that in contrast with the prior technique which used NVLogs in each controller, with this technique no write requests or actual data need to be communicated between controllers to provide redundancy. Rather, simple bitmap updates are communicated between the affected controller1and the quorum device24, which involves much less data transfer. No communication is required between any of the storage controllers to maintain the bitmap or to recover from a write failure. Consequently, no matter how many redundant controllers are added to a cluster, the amount of inter-node data traffic does not increase.

Consider the following stripe as an example (note that physical arrangement of blocks may vary on storage devices):

where each Di is a data block (i=1, 2, . . . 6) and each Pj is a parity block (j=1 or 2). If D1is updated in response to a write request, then P1and P2need to be updated also. If all blocks are updated in sequence, then an acknowledgement is sent by the controller1to the host that initiated the write request. If for some reason (e.g., a controller failure) D1is updated, but not one or both of the parity blocks, then the parity blocks can be recalculated based on the data blocks. Similarly, if one or both of the parity blocks are updated but D1is not, then D1can be reconstructed based on the updated parity block or blocks. In any case, an acknowledgement (“ACK”) can be sent to the initiating host as soon as enough blocks in the stripe have been updated so that any remaining blocks in the stripe for which updates are still pending can be recovered (i.e., by parity based reconstruction).

If a controller1fails while performing a write operation, such that the write fails to complete (i.e., not all target blocks get successfully updated in a storage facility2), then a different controller can take over to complete the write operation. In certain situations, however, it is necessary to determine for a given stripe which is more current, the data or parity. For example, this knowledge is important if a failure occurs while processing multiple write requests to the same stripe, and at least one write but not all of them has been acknowledged to the host before the failure occurred, then some data in the stripe may be dirty. In that event, it is necessary to determine which is most current on disk, data or parity, to reestablish data consistency.

FIG. 7is a flow diagram illustrating a failure recovery process, by which a controller1can reestablish data consistency (or verify it, as the case may be) after another controller has failed. To simplify description, the process is described with respect to only a single data block, D, and two parity blocks, P1and P2.

At step701, the controller accesses the relevant bitmap305stored in the quorum device24to determine the status of the relevant stripe. In particular, the controller checks the status of the fields corresponding to D, P1and P2in the bitmap and the corresponding currency information relating to those blocks. As described above, the currency information may be in the form of time indicator fields412-414(FIG. 5), counters, or sub-bitmaps, for example.

If D, P1and P2are all indicated as clean, then the controller does nothing (step702), since this means that the write was completed and has been acknowledged to the host. If D is indicated as clean but P1and/or P2are dirty (i.e., P1and/or P2has not been updated to reflect the write to D), then the controller reconstructs the dirty parity block or blocks from the data blocks in the stripe at step703. The controller then marks P1and P2as clean at step704.

If D is indicated as dirty and P1and/or P2is clean (i.e., P1and/or P2is current, but D is not), then at step705the controller reconstructs the data block D from whichever parity block is clean (up-to-date). This situation most likely will occur during a failure of one disk in the storage facility. The controller then marks D as clean at step706.

If D, P1and P2are all indicated as dirty (which means that the write was never acknowledged to the host), then at step707the controller will check D against P1and P2for consistency. If all of these blocks are consistent with each other (step708), then the controller simply marks D, P1and P2a clean at step710. If any of D, P1or P2is inconsistent with any of the others, then D is assumed to be correct, so the controller reconstructs P1and P2from D at step709, and then marks D, P1and P2a clean at step710.

FIG. 8is a high level block diagram of a processing system that can be used to implement a quorum device24, a controller1or a host5. Certain standard and well-known components which are not germane to the present invention are not shown.

In the illustrated embodiment, the processing system includes one or more processors801coupled to a bus system803. The bus system803inFIG. 8is an abstraction that represents any one or more separate physical buses and/or point-to-point connections, connected by appropriate bridges, adapters and/or controllers. The bus system803, therefore, may include, for example, a system bus, a form of Peripheral Component Interconnect (PCI) bus, HyperTransport or industry standard architecture (ISA) bus, small computer system interface (SCSI) bus, universal serial bus (USB), or Institute of Electrical and Electronics Engineers (IEEE) standard 1394 bus (sometimes referred to as “Firewire”).

In the case of a quorum device24, the processor(s)801may be (or may include) a memory controller. In the case of a controller1or host5, the processor(s)801are the central processing unit (CPU) of the processing system and, thus, control the overall operation of processing system. In certain embodiments, the processor(s)801accomplish this by executing software stored in memory802. A processor801may be, or may include, one or more programmable general-purpose or special-purpose microprocessors, digital signal processors (DSPs), programmable controllers, application specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), programmable logic devices (PLDs), or the like, or a combination of such devices.

Note that in another embodiment, a quorum device24does not include its own internal processor(s)801. In such an embodiment, the memory (e.g., NVRAM) within the quorum device24may be directly accessible to external devices such as the controllers1in the cluster, by using, for example, RDMA.

The processing system also includes memory802coupled to the bus system803. The memory802represents any form of random access memory (RAM), read-only memory (ROM), flash memory, or a combination thereof. Memory802stores software/firmware instructions and/or data804, such as instructions and/or data to implement the techniques introduced above, including (in the case of a quorum device24) the boot code of the quorum device24.

Also connected to the processors801through the bus system803are a non-volatile storage device805and a network adapter807. The non-volatile storage device805may be or include any conventional medium for storing data in a non-volatile manner, such as one or more disks. In a quorum device24, the non-volatile storage device805may be or may include NVRAM to store the bitmap and cluster state information as described above (seeFIG. 3). A quorum device24may also include other forms of non-volatile memory, such as flash and/or disk, to be used to store initialization code and/or information that may need to be saved in the event of a reboot.

The network adapter807provides the processing system with the ability to communicate with other devices over a network and may be, for example, an Ethernet adapter, a Fibre Channel adapter, InfiniBand adapter, or the like.

In the case of a controller1, the processing system may also include a storage adapter (not shown), which allows the processing system to access external storage devices, such as a storage device2.

Thus, a method and system of using disk-less quorum device in a clustered storage system have been described.

Software to implement the technique introduced here may be stored on a machine-readable medium. A “machine-accessible medium”, as the term is used herein, includes any mechanism that provides (i.e., stores and/or transmits) information in a form accessible by a machine (e.g., a computer, network device, personal digital assistant (PDA), manufacturing tool, any device with a set of one or more processors, etc.). For example, a machine-accessible medium includes recordable/non-recordable media (e.g., read-only memory (ROM); random access memory (RAM); magnetic disk storage media; optical storage media; flash memory devices; etc.), etc.

“Logic”, as is used herein, may include, for example, software, hardware and/or combinations of hardware and software.