Abstract:
In some embodiments, storage devices, such as a storage drive or a storage node, in an array of storage devices may be reintroduced into the array of storage devices after a period of temporary unavailability without fully rebuilding the entire previously unavailable storage device.

Description:
LIMITED COPYRIGHT AUTHORIZATION 
       [0001]    A portion of disclosure of this patent document includes material which is subject to copyright protection. The copyright owner has no objection to the facsimile reproduction by anyone of the patent document or the patent disclosure as it appears in the Patent and Trademark Office patent file or records, but otherwise reserves all copyrights whatsoever. 
       FIELD OF THE INVENTION 
       [0002]    This invention relates generally to storage devices, and more specifically to managing storage devices in a computer system. 
       BACKGROUND 
       [0003]    In recent years, the amount of data stored digitally on computer storage devices has increased dramatically. To accommodate increasing data storage needs, larger capacity storage devices have been developed. Typically, these storage devices are a single magnetic storage disk. Unfortunately, multiple concurrent access requests to a single storage drive can slow data reads and writes to a single drive system. One response to this problem has been to connect a plurality of storage devices to form a storage node. On storage nodes, data may be distributed over several storage disks. For example, a read operation for a file distributed over several storage drives may be faster than for a file located on a single drive because a distributed system permits parallel read requests for smaller portions of the file. Another response has been to connect a plurality of storage nodes to form a storage system of even larger capacity, referred to as a “cluster.” 
         [0004]    One problem associated with distributed systems is drive failure and data loss. Though read and write access times tend to decrease as the number of storage devices in a system increase, the chances of storage device failures also increase as the number of storage devices increases. Thus, a distributed system is vulnerable to both temporary and permanent unavailability of storage devices. 
         [0005]    When a storage device, for example, either a storage drive or a storage node, becomes unavailable, storage systems have to remove the storage device from the system and fully reconstruct the devices. As storage devices become increasingly larger, the amount of time required to fully reconstruct an unavailable storage device increases correspondingly, which affects response time and further exacerbates the risk of permanent data loss due to multiple device failures. 
       SUMMARY OF THE INVENTION 
       [0006]    Because of the foregoing challenges and limitations, there is a need to provide a system that manages a set storage devices even if one or more of the storage devices becomes unavailable. 
         [0007]    In one embodiment, a method for managing unavailable storage devices comprises detecting that a troubled storage device is unavailable, wherein a data set is stored on the troubled storage device, responding to a read or write request for data at least a portion of the data set while the troubled storage device is unavailable, and detecting that the troubled storage device is available and providing access to the data set stored on the troubled storage device without full reconstruction of the troubled storage device. 
         [0008]    In another embodiment, a storage system for managing unavailable storage devices comprises a first storage device configured to respond to a read or write request for at least a portion of the data set after the first storage device returns from an unavailable state without full reconstruction of the first storage device. In one embodiment, the storage system further comprises at least one operational storage device configured to store a representation of at least a portion of the data set and provide access to the representation of at least a portion of the data set if the first storage device is unavailable. 
         [0009]    In a further embodiment, a storage system for managing storage devices comprises a plurality of storage devices configured to store data distributed among at least two of the plurality of storage devices. In one embodiment, the storage system is further configured such that if one or more of the plurality of storage devices becomes unavailable and then becomes available again, the data is available after the one or more of the plurality of storage devices becomes available again. 
         [0010]    For purposes of this summary, certain aspects, advantages, and novel features of the invention are described herein. It is to be understood that not necessarily all such advantages may be achieved in accordance with any particular embodiment of the invention. Thus, for example, those skilled in the art will recognize that the invention may be embodied or carried out in a manner that achieves one advantage or group of advantages as taught herein without necessarily achieving other advantages as may be taught or suggested herein. 
     
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
         [0011]      FIG. 1  illustrates one embodiment of a storage device. 
           [0012]      FIGS. 2A ,  2 B,  2 C,  2 D, and  2 E illustrate one embodiment of an example scenario where one of a set of drives goes down and then returns. 
           [0013]      FIGS. 3A ,  3 B, and  3 C illustrate one embodiment of an example scenario of a write journal when a drive goes down and then returns. 
           [0014]      FIG. 4  illustrates one embodiment of a flowchart of operations for a read. 
           [0015]      FIG. 5  illustrates one embodiment of a flowchart of operations for a write. 
           [0016]      FIG. 6  illustrates one embodiment of a flowchart of operations for a journal flush. 
           [0017]      FIG. 7  illustrates one embodiment of connections of storage nodes in one embodiment of a distributed file system. 
           [0018]      FIG. 8A  illustrates one embodiment of data stored in storage nodes in one embodiment of a distributed system. 
           [0019]      FIG. 8B  illustrates one embodiment of data stored in storage nodes in one embodiment of a distributed system wherein two storage drives are unavailable. 
           [0020]      FIG. 9  illustrates one embodiment of a map data structure for storing locations of file data. 
           [0021]      FIG. 10  illustrates one embodiment of a map data structure for storing data regarding file metadata. 
       
    
    
     DETAILED DESCRIPTION OF PREFERRED EMBODIMENTS 
       [0022]    Systems and methods which represent exemplary embodiments of the invention will now be described with reference to the drawings. Variations to the systems and methods which represent other embodiments will also be described. The terminology used in the description presented herein is not intended to be interpreted in any limited or restrictive manner, simply because it is being utilized in conjunction with a detailed description of certain specific embodiments. Furthermore, embodiments of the invention may include several novel features, no single one of which is solely responsible for its desirable attributes or which is essential to practicing the systems and methods described herein. 
       I. Overview 
       [0023]    In one embodiment, the storage system provides access to data stored on a set of storage devices even when one of the storage devices is unavailable. While the storage device is unavailable, the storage system reconstructs the requested data and stores data targeted for the unavailable drive in a new location. Even though unavailable, the storage device does not have to be fully reconstructed and replaced, but can return to the storage system once it becomes available. Thus, in such embodiments, access to data on the storage devices continues with out significant interruption. 
         [0024]    As used herein, the term “storage device” generally refers to a device configured to store data, including, for example, a storage drive, such as a single hard drive in an array of hard drives or in a storage node or an array of storage nodes, where each of the storage nodes may comprise multiple hard drives. 
         [0025]    In one embodiment, a user or client device communicates with a storage system comprising one or more storage devices. In one embodiment, sets of data stored on the storage system (generically referred to herein as “data sets” or “files”) are striped, or distributed, across two or more of the storage devices, such as across two or more storage drives or two or more storage nodes. In one embodiment, files are divided into stripes of two or more data blocks and striping involves storing data blocks of a file on two or more storage devices. For example, if a file comprises two data blocks, a first data block of the file may be stored on a first storage device and a second data block of the file may be stored on a second storage device. A map data structure stores information on where the data is stored. 
         [0026]    In addition to storing the data blocks of files on the storage devices, some embodiments may also store data protection data associated with the data. One example of data protection data is parity data, however, there are many other types of data protection data as discussed in further detail below. Those of ordinary skill in the art will recognize that parity data can be used to reconstruct portions of data that has been corrupted or is otherwise unavailable. In one embodiment, parity data is calculated by XORing two or more bits of data for which parity protection is desired. For example, if four data bits store the values 0110, the parity bit is equal to 0 XOR 1 XOR 1 XOR 0. Thus, the parity bit is 0. This parity bit may then be stored on a storage device and, if any one of the data bits later become lost or unavailable, the lost or unavailable bit can be reconstructed by XORing the remaining bits with the parity bit. With reference to the above-noted data block 0110, if bit one is unavailable (01X0), then bit  1  can be reconstructed using the logical equation 0 (Parity bit) XOR 0 XOR 1 XOR 0 to determine that unavailable bit one is 1. In other embodiments, other parity, error correction, accuracy, or data protection schemes may be used. The map data structure also store information on where the data protection data is stored. 
         [0027]    In the embodiment, if one of the storage devices is unavailable, the storage system may use the data protection data to reconstruct the missing data. In addition, the storage system may use the map data structure to track current locations of write data intended for an unavailable storage device, but stored on another storage device. 
       II. Storage System 
       [0028]      FIG. 1  illustrates one embodiment of a storage node  100  used to store data on a set of storage devices. In the embodiment of  FIG. 1 , the storage node  100  comprises multiple storage devices  130 ,  140 ,  150 ,  160  that are each coupled to a bus  170 . An input/output interface  120  is coupled to the bus  170  and is configured to receive and transmit data to and from the storage node  100 . The storage node  100  further comprises a controller  110  that is coupled to the bus  170  so that the controller is in communication with other components in the storage node  100 . In one embodiment, the controller  110  manages the operations of the devices  130 ,  140 ,  150 ,  160  as read and write requests are received, such as, for example, from a user. 
         [0029]    A. Storage Devices 
         [0030]    In the exemplary storage node  100 , each of the storage devices  130 ,  140 ,  150 ,  160  comprises a hard drive. However, it is recognized that the storage devices  130 ,  140 ,  150 ,  160  may include one or more drives, nodes, disks, clusters, objects, drive partitions, virtual volumes, volumes, drive slices, containers, and so forth. Moreover, the storage devices may be implemented using a variety of products that are well known in the art, such as, for example, ATA100 devices, SCSI devices, and so forth. In addition, the size of the storage devices may be the same size or may be of two or more different sizes. 
         [0031]    B. Request Module 
         [0032]    In one embodiment, the storage node  100  also includes a request module  180  for handling requests to read data from the storage devices  130 ,  140 ,  150 ,  160  as well as requests to write data to the storage devices  130 ,  140 ,  150 ,  160 . The storage node  100  may also include other modules, such as a reconstruction module for starting the reconstruction of one or more unavailable and/or failed storage devices  130 ,  140 ,  150 ,  160 . The storage node  100  may also include a restriper module that scans an unavailable storage devices, identifies data stored in the unavailable storage devices and begins moving the data to one or more available storage devices. The storage node  100  may also include a collector module that frees data that is no longer referenced due to writes while a drive was unavailable. 
         [0033]    In general, the word module, as used herein, refers to logic embodied in hardware or firmware, or to a collection of software instructions, possibly having entry and exit points, written in a programming language, such as, for example, C or C++. A software module may be compiled and linked into an executable program, installed in a dynamic link library, or may be written in an interpreted programming language such as, for example, BASIC, Perl, or Python. It will be appreciated that software modules may be callable from other modules or from themselves, and/or may be invoked in response to detected events or interrupts. Software instructions may be embedded in firmware, such as an EPROM. It will be further appreciated that hardware modules may be comprised of connected logic units, such as gates and flip-flops, and/or may be comprised of programmable units, such as programmable gate arrays or processors. The modules described herein are preferably implemented as software modules, but may be represented in hardware or firmware. Moreover, although in some embodiments a module may be separately compiled, in other embodiments a module may represent a subset of instructions of a separately compiled program, and may not have an interface available to other logical program units. 
         [0034]    C. Group Protocol Module 
         [0035]    In some embodiments, the storage node  100  also includes a group protocol module  195 . The group protocol module  195  maintains information regarding the storage devices that are available to the storage system for read and/or write access. In one embodiment, the group protocol module  195  communicates with the storage devices  130 ,  140 ,  150 ,  160  and indicates, for example, the current operational state (for example, available, unavailable, up, down, dead) and/or how much space is available on the each device. In one embodiment, the group protocol module  195  comprises information regarding the availability of devices on other storage nodes. In one embodiment, when a device becomes unavailable, the group protocol module  195  notifies other storage nodes in the storage system  200 . Similarly, when a previously unavailable device becomes available again, the group protocol module  195  communicates to the information nodes in the storage system. 
         [0036]    D. Journal 
         [0037]    In some embodiments, the storage node  100  also includes a journal  190 , which may comprise one or more memory devices, such as NVRAM, flash ROM, or EEPROM, and/or a hard drive. The journal  190  is configured to store data that is intended to be stored on a device, and may or may not store other data. In an advantageous embodiment, the journal  190  is persistent such that it does not lose data when power to the storage node  100  is lost or interrupted. Thus, in the event of failure of node  100  and/or one or more of the storage devices  130 ,  140 ,  150 ,  160 , recovery actions can be taken when power is regained or the storage node  100  reboots to ensure that transactions that were in progress prior to the failure are either completed or are aborted. If an unavailable device does not return to service, for example, the unavailable device is permanently unavailable and the information stored on the journal  190  may be transferred to other devices in the storage node  100  or, alternatively, transferred to storage devices in other storage nodes. 
         [0038]    In some embodiments, the journal  190  is implemented as a non-linear journal. Embodiments of a non-linear journal suitable for storing write data are disclosed in U.S. patent application Ser. No. 11/506,597, entitled “Systems And Methods For Providing Nonlinear Journaling,” U.S. patent application Ser. No. 11/507,073, entitled “Systems And Methods For Providing Nonlinear Journaling,”, U.S. patent application Ser. No. 11/507,070, entitled “Systems And Methods For Providing Nonlinear Journaling,” and Ser. No. 11/507,076, entitled “Systems And Methods For Allowing Incremental Journaling,” all filed on Aug. 8, 2006, and all of which are hereby incorporated herein by reference in their entirety. 
         [0039]    It is also recognized that in some embodiments, the storage system is implemented without using a journal. In such embodiments, the data may be synchronously written to disk during the write, and/or the data may be written, for example, to a persistent write-back cache that saves the data until the storage device becomes available. 
         [0040]    E. System Information 
         [0041]    The storage node  100  may run on a variety of computer systems such as, for example, a computer, a server, a smart storage unit, and so forth. In one embodiment, the computer may be a general purpose computer using one or more microprocessors, such as, for example, an Intel® Pentium® processor, an Intel® Pentium® II processor, an Intel® Pentium® Pro processor, an Intel® Pentium® IV processor, an Intel® Pentium® D processor, an Intel® Core™ processor, an xx86 processor, an 8051 processor, a MIPS processor, a Power PC processor, a SPARC processor, an Alpha processor, and so forth. The computer may run a variety of operating systems that perform standard operating system functions such as, for example, opening, reading, writing, and closing a file. It is recognized that other operating systems may be used, such as, for example, Microsoft® Windows® 3.X, Microsoft® Windows 98, Microsoft® Windows® 2000, Microsoft® Windows® NT, Microsoft® Windows® CE, Microsoft® Windows® ME, Microsoft® Windows® XP, Palm Pilot OS, Apple® MacOS®, Disk Operating System (DOS), UNIX, IRIX, Solaris, SunOS, FreeBSD, Linux®, or IBM® OS/2® operating systems. 
         [0042]    F. Files 
         [0043]    As used herein, a file is a collection of data stored in one logical unit that is associated with one or more filenames. For example, the filename “test.txt” may be associated with a file that comprises data representing text characters. The data blocks of the file may be stored at sequential locations on a storage device or, alternatively, portions of the data blocks may be fragmented such that the data blocks are not in one sequential portion on the storage device. In an embodiment where file striping is used, such as in a RAID 5 storage system, for example, data blocks of a file may be stored on multiple storage devices. For example, in a RAID 5 system, data blocks are interleaved across multiple storage devices within an array of storage devices. The stripe width is the size of the data block stored on a single device before moving on to the next device in the device array. On the last device in the device array, redundancy information is stored, rather than data blocks of the file. The redundancy information in RAID 5 is the parity of the previous interleaved data blocks. The process repeats for other data blocks of the file, except that the device that includes the parity data rotates from device to device in the array for each stripe. It is recognized that a variety of striping techniques may be used. 
         [0044]    G. Data Protection 
         [0045]    In some embodiments the storage system may utilize one or more types of data protection. For example, the storage system may implement one or more error correcting codes. These codes include a code “in which each data signal conforms to specific rules of construction so that departures from this construction in the received signal can generally be automatically detected and corrected. It is used in computer data storage, for example in dynamic RAM, and in data transmission.” (http://en.wikipedia.org/wiki/Error_correcting_code). Examples of error correction code include, but are not limited to, Hamming code, Reed-Solomon code, Reed-Muller code, Binary Golay code, convolutional code, and turbo code. In some embodiments, the simplest error correcting codes can correct single-bit errors and detect double-bit errors, and other codes can detect or correct multi-bit errors. 
         [0046]    In addition, the error correction code may include forward error correction, erasure code, fountain code, parity protection, and so forth. “Forward error correction (FEC) is a system of error control for data transmission, whereby the sender adds redundant to its messages, which allows the receiver to detect and correct errors (within some bound) without the need to ask the sender for additional data.” (http://en.wikipedia.org/wiki/forward_error_correction). Fountain codes, also known as rateless erasure codes, are “a class of erasure codes with the property that a potentially limitless sequence of encoding symbols can be generated from a given set of source symbols such that the original source symbols can be recovered from any subset of the encoding symbols of size equal to or only slightly larger than the number of source symbols.” (http://en.wikipedia.org/wiki/Fountain_code). “An erasure code transforms a message of n blocks into a message with &gt;n blocks such that the original message can be recovered from a subset of those blocks” such that the “fraction of the blocks required is called the rate, denoted r (http://en.wikipedia.org/wiki/Erasure_code). “Optimal erasure codes produce n/r blocks where any n blocks is sufficient to recover the original message.” (http://en.wikipedia.org/wiki/Erasure_code). “Unfortunately optimal codes are costly (in terms of memory usage, CPU time or both) when n is large, and so near optimal erasure codes are often used,” and “[t]hese require (1+ε)n blocks to recover the message. Reducing ε can be done at the cost of CPU time.” (http://en.wikipedia.org/wiki/Erasure_code). 
         [0047]    The data protection may include other error correction methods, such as, for example, Network Appliance&#39;s RAID double parity methods, which includes storing data in horizontal rows, calculating parity for data in the row, and storing the parity in a separate row parity disks along with other double parity methods, diagonal parity methods, and so forth. 
         [0048]    In another embodiment, odd parity may be used such that an additional NOT logical operator is applied after XORing data bits in order to determine the unavailable bit. Those of skill in the art will appreciate that there are other parity schemes that may be used in striping data and recovering lost data in a storage system. Any suitable scheme may be used in conjunction with the systems and methods described herein. 
       III. Example Scenario of a Down Drive 
       [0049]    For purposes of illustration, an example scenario of a set of drives will be discussed wherein one of the drives becomes unavailable while the storage system is receiving read and write requests. This example scenario is just one of many possible scenarios and is meant only to illustrate some embodiments of the storage system. 
         [0050]    A. Data Map 
         [0051]      FIG. 2A  illustrates an example scenario where one of a set of drives goes down and then returns to the storage system. The storage system includes five drives, Drive  0 , Drive  1 , Drive  2 , Drive  3 , and Drive  4 . The storage system stores a set of data d 0 , d 1 , d 2 , d 3 , d 4 , and d 5  wherein the data is protected using different types of parity protection. Data d 0 , d 1 , and d 2  are protected using 3+1 parity protection, where p 0 (d 0 −d 2 ) is the related parity data. Data d 3  and d 4  are protected using 2+2 parity protection, where p 0 (d 3 −d 4 ) and p 1 (d 3 −d 4 ) are the related parity data. Data d 5  is protected using 2× mirroring or 1+1 parity, where p 0 (d 5 ) is the related parity data. The storage system also includes a map data structure that stores the locations of the data and the parity data. As set forth in the map and as shown in the drives, d 0  is stored on Drive  0  at location  0 , d 1  is stored on Drive  1  at location  0 , d 2  is stored on Drive  2  at location  3 , d 3  is stored on Drive  0  at location  1 , d 4  is stored on Drive  1  at location  1 , d 5  is stored on Drive  2  at location  2 , p 0 (d 0 −d 2 ) is stored on Drive  3  at location  0 , p 0 (d 3 −d 4 ) is stored on Drive  3  at location  3 , p 1 (d 3 −d 4 ) is stored on Drive  2  at location  1 , and p 0 (d 5 ) is stored on Drive  3  at location  2 . 
         [0052]    In  FIG. 2B , Drive  1  becomes unavailable, such as, for example, because the connection to Drive  1  is unplugged. If the storage system receives a read request for d 1 , then the storage system will read d 0  from Drive  0 , d 2  from Drive  2  and p 0 (d 0 −d 2 ) from Drive  3  and then reconstruct d 1  and return d 1 . 
         [0053]    In  FIG. 2C , Drive  1  becomes available, such as, for example, the connection to Drive  1  is plugged back in. The storage system is the same as before Drive  1  became unavailable. Moreover, even though Drive  1  became unavailable, Drive  1  did not have to be removed from the storage system and fully recreated. Instead, once it became available, it was integrated back into the storage system and made available. 
         [0054]    In  FIG. 2D , Drive  1  becomes unavailable, and the storage system receives a write request for d 0 , d 1 , and d 2 . The storage system determines whether all of the data locations for d 0 , d 1 , d 2 , and their corresponding parity data p 0 (d 0 −d 2 ) are available. Because Drive  1  is not available for d 1 , then the storage system decides to store d 1  on Drive  4  at location  0 , which maintains the data protection by not having d 1  on the same drive as the other data or parity data. Then the storage system updates the map so that the location for d 1  is Drive  4 , location  0  as shown in the map for  FIG. 2D . The storage system then writes the data blocks d 0  to Drive  0 , location  0 , d 1  to Drive  4 , location  0 , and d 2  to Drive  2 , location  3 ; computes the parity data p 0 (d 0 −d 2 ), and stores the parity data p 0 (d 0 −d 2 ) on Drive  3 , location  0 . 
         [0055]    In  FIG. 2E , Drive  1  becomes available and the location of the data that was moved from Drive  1  while it was unavailable remains stored on the newly assigned drive and appropriately reference in the map. In addition, data that was not written while Drive  1  was not moved and remains on Drive  1  and is now accessible on Drive  1 , such as, for example, d 4 . Again, even though Drive  1  became unavailable, Drive  1  did not have to be removed from the storage system and fully recreated. Instead, once it became available, it was integrated back into the storage system and made available. 
         [0056]    It is recognized that in some embodiments, after the storage system recognizes that a drive is unavailable, the storage system may begin to move the data from the unavailable drive to another drive so that in case the drive becomes permanently unavailable the migration process has already begun, but if the drive becomes available, the data that has not been moved remains on the now available drive. It is also recognized that the example scenario of  FIGS. 2A ,  2 B,  2 C,  2 D, and  2 E are meant only to illustrate embodiments of a storage system and not to limit the scope of the invention. 
         [0057]    B. Journal 
         [0058]    In some embodiments, the storage system includes a journal for storing write transactions for the drives. In some circumstances, the actual writes to the disks of the drives d 0  not occur right away. Accordingly, after a write request is processed, the data is stored in the journal until the journal is flushed. When the journal is flushed, it writes the data to the available disks. However, if a drive is not available, the data can remain in the journal until the drive becomes available and at that time it is written to the drives disk. The data can remain in the journal until the drive becomes available or the drive becomes permanently unavailable wherein the data is then removed from the journal. In other systems, once a drive is marked as unavailable, all data stored in the journal for that drive is deleted and the drive is recreated even if a drive is only down for a very short time period and fully functional when it returns and becomes available. 
         [0059]      FIGS. 3A ,  3 B, and  3 C illustrate one embodiment of an example scenario of a write journal when a drive becomes unavailable and then becomes available. In  FIG. 3A , all of the drives are available so their status is set to UP. The storage system then receives a request to write d 4  on Drive  1  at location  1  with a new data value. The storage system stores d 4  in the journal associating it with Drive  1  and waits for the journal to be flushed. In  FIG. 3B , Drive  1  goes becomes unavailable and the status is set to DOWN. The journal is flushed, but because Drive  1  is DOWN, d 4  is kept in the journal. In  FIG. 3C , Drive  1  becomes available and the status is set to UP. When the journal is flushed, d 4  is written to Drive  1  and removed from the journal. 
         [0060]    Again, even though Drive  1  became unavailable, the data destined for Drive  1  did not have to be deleted from the journal. Instead, once Drive  1  became available, it was integrated back into the system and the data was properly stored on the disk of Drive  1 . 
         [0061]    It is recognized that the journal can be implemented in many different ways and that the storage system may not include a journal as set forth above. This example scenario is meant only to illustrate some embodiments of a storage system and not to limit the scope of the invention. 
       IV. Read Request 
       [0062]      FIG. 4  illustrates one embodiment of a flowchart of operations for processing a read request. Beginning at a start state  410 , the read request process  400  proceeds to the next state and receives a read request  420 . The read request  420  may be for one or more blocks of data. The read request process  400  then determines whether all data blocks are available  430 . If all data blocks are available, the read request process  400  reads the data  440 . If all data blocks are not available, then the read request process  400  reads the available data and if possible reconstructs the missing data using the data protection data  450 . Next, the read request process  400  returns the data blocks (from the read and/or the reconstruction) or an error message if the read and/or the reconstruction failed  460  and proceeds to an end state  470 . 
         [0063]    In one embodiment, the reconstruction may fail if, for example, there is not enough data protection data to reconstruct the unavailable data blocks, such as, for example, if the parity is 4+1 and two of the data blocks are unavailable. 
         [0064]    While  FIG. 4  illustrates one embodiment of processing a read request, it is recognized that a variety of embodiments may be used. For example, the read request process  400  may read the available data and then determine whether all data blocks are available. Moreover, depending on the embodiment, certain of the blocks described in the figure above may be removed, others may be added, and the sequence may be altered. 
       V. Write Request 
       [0065]      FIG. 5  illustrates one embodiment of a flowchart of operations for performing a write request. Beginning at a start state  510 , the write request process  500  proceeds to the next state and receives a write request  520 . Proceeding to the next state, the write request process  500  determines whether the devices on which the data blocks and parity blocks are to be stored are available  530 . The write request may, for example, check the map data structure entries for each of the data blocks and parity blocks to determine the devices on which they will be stored, whereas in other embodiments, the drives on which they will be stored are provided to the write request process  500 . Moreover, to determine whether a device is available, the write request process  500  may check the group management protocol data that indicates the states of the devices. If there is more than one device not available, the write request process  500  determines new locations for the data and/or parity blocks  540  and updates the metadata to correspond to the new locations  550 . Next, the write request process  500  writes the data blocks to the appropriate devices  560 , writes the parity data  570 , and returns to an end state  580 . 
         [0066]    In one embodiment, the write request process  500  may fail and/or return an error if, for example, there is not enough room to store the data and/or parity blocks on other devices, such as, for example, if the parity is 4+1, there are six drives and two of the drives are unavailable. In such a scenario, the write request process  500  may return an error and/or may store the data in any available space, but return a message that some of the data is stored without the requested data protection. 
         [0067]    While  FIG. 5  illustrates one embodiment of processing a write request, it is recognized that a variety of embodiments may be used. For example, the write request process  500  may compute the data protection data or the data protection data may be received by the write request process  500 . Moreover, depending on the embodiment, certain of the blocks described in the figure above may be removed, others may be added, and the sequence may be altered. 
         [0068]    As discussed in detail below, the storage system may be implemented as part of a distributed file system. In one embodiment of a distributed file system, the write request also checks to see if all copies of the metadata storing the locations of the data is stored on available nodes. One embodiment of pseudocode for implementing a write request process is as follows: 
         [0000]    
       
         
               
               
             
           
               
                   
                   
               
             
             
               
                   
                 Write( ) { 
               
               
                   
                   If (not all inodes available) { 
               
               
                   
                     Re-allocate missing inodes on available drives 
               
               
                   
                     Write new copies of the inode 
               
               
                   
                     Update lin tree to point to new inodes 
               
               
                   
                   } 
               
               
                   
                   If (not all data and parity blocks available) { 
               
               
                   
                     Re-allocate missing data and parity blocks on available 
               
               
                   
                      drives 
               
               
                   
                     Update file metatree to point to new blocks 
               
               
                   
                   } 
               
               
                   
                   For all data blocks b { 
               
               
                   
                     Write_block_to_journal (b) 
               
               
                   
                   } 
               
               
                   
                   For all parity blocks b { 
               
               
                   
                     Write_block_to_journal (b) 
               
               
                   
                   } 
               
               
                   
                 } 
               
               
                   
                   
               
             
          
         
       
     
         [0069]    In one embodiment, the inodes store metadata about files, and the LIN tree stores location data for the inodes. While the above psuedocode represents one embodiment of an implementation of a write process for one embodiment of a distributed file system, it is recognized that the write process may be implemented in a variety of ways and is not limited to the exemplary embodiment above. 
       VI. Journal Flush 
       [0070]    As noted above, in some embodiments, when a write request is processed, the data to be stored to a disk is stored in a journal until the write to the disk until after the write has occurred.  FIG. 6  illustrates one embodiment of a flowchart of operations for a journal flush. Beginning at a start state  610 , the journal flush process  600  proceeds to the next state and for all devices d  620 , the journal flush process  600  determines whether the devices is UP, DOWN or DEAD  630 . If the device is UP, the journal flush process  600  flushes the blocks for that device to the device&#39;s disk  640 . If the device is DOWN, the journal flush process  600  leaves the blocks for that device in the journal  650 . If the device is DEAD, the journal flush process  600  discards the blocks in the journal that device  660 . Once the devices d have been reviewed  670 , the journal flush process  600  proceeds to an end state  680 . 
         [0071]    While  FIG. 6  illustrates one embodiment of flushing the journal, it is recognized that a variety of embodiments may be used. For example, the flush journal process  600  may review more than one device d at a time. In addition, if a device is DEAD, the flush journal process  600  may send the blocks to a process that is handling the reconstruction of the DEAD drive. Moreover, depending on the embodiment, certain of the blocks described in the figure above may be removed, others may be added, and the sequence may be altered. 
         [0072]    One embodiment of pseudocode for implementing a journal flush process is as follows: 
         [0000]    
       
         
               
               
             
           
               
                   
                   
               
             
             
               
                   
                 Flush_journal( ) { 
               
               
                   
                   For all drives d { 
               
               
                   
                     If (d is down) { 
               
               
                   
                       Leave blocks in the journal 
               
               
                   
                     } else if (d is up) { 
               
               
                   
                       Flush blocks to disk 
               
               
                   
                       When disk returns success, 
               
               
                   
                       discard blocks from the journal 
               
               
                   
                     } else if (d is dead) { 
               
               
                   
                       Discard blocks in the journal 
               
               
                   
                     } 
               
               
                   
                   } 
               
               
                   
                 } 
               
               
                   
                   
               
             
          
         
       
     
         [0073]    While the above psuedocode represents one embodiment of an implementation of a journal flush process, it is recognized that the journal flush process may be implemented in a variety of ways and is not limited to the exemplary embodiment above. 
       VII. Distributed System Embodiments 
       [0074]    For purposes of illustration, some embodiments will now be described in the context of a distributed system such as, for example a distributed file system. Embodiments of a distributed file system suitable for accommodating reverse lookup requests are disclosed in U.S. patent application Ser. No. 10/007,003, entitled, “Systems And Methods For Providing A Distributed File System Utilizing Metadata To Track Information About Data Stored Throughout The System,” filed Nov. 9, 2001 which claims priority to Application No. 60/309,803, entitled “Systems And Methods For Providing A Distributed File System Utilizing Metadata To Track Information About Data Stored Throughout The System,” filed Aug. 3, 2001, U.S. Pat. No. 7,156,524 entitled “Systems and Methods for Providing A Distributed File System Incorporating a Virtual Hot Spare,” filed Oct. 25, 2002, and U.S. patent application Ser. No. 10/714,326 entitled “Systems And Methods For Restriping Files In A Distributed File System,” filed Nov. 14, 2003, which claims priority to Application No. 60/426,464, entitled “Systems And Methods For Restriping Files In A Distributed File System,” filed Nov. 14, 2002, all of which are hereby incorporated herein by reference in their entirety. 
         [0075]    In one embodiment of a distributed file system, metadata structures, also referred to as inodes, are used to represent and manipulate the files and directories within the system. An inode is a data structure that describes a file or directory and may be stored in a variety of locations including on a storage device. 
         [0076]    A directory, similar to a file, is a collection of data stored in one unit under a directory name. A directory, however, is a specialized collection of data regarding elements in a file system. In one embodiment, a file system is organized in a tree-like structure. Directories are organized like the branches of trees. Directories may begin with a root directory and/or may include other branching directories. Files resemble the leaves or the fruit of the tree. Although in the illustrated embodiment an inode represents either a file or a directory, in other embodiments, an inode may include metadata for other elements in a distributed file system, in other distributed systems, in other file systems, or other systems. In some embodiments files d 0  not branch, while in other embodiments files may branch. 
         [0077]      FIG. 7  illustrates an exemplary distributed system  700  comprising storage nodes  710 ,  720 ,  730 ,  740  and users  750 ,  760  that are in data communication via a communication medium  770 . The communication medium  770  may comprise one or more wired and/or wireless networks of any type, such as SANs, LANs, WANs, MANs, and/or the Internet. In other embodiments, the distributed system  700  may be comprised of hard-wired connections between the storage nodes  710 ,  720 ,  730 ,  740 , or any combination of communication types known to one of ordinary skill in the art. 
         [0078]    In the embodiment of  FIG. 7 , the users  750 ,  760  may request data via any of the storage nodes  710 ,  720 ,  730 ,  740  via the communication medium  770 . The users  750 ,  760  may comprise a personal computer, a mainframe terminal, PDA, cell phone, laptop, a client application, or any device that accesses a storage device in order to read and/or write data. 
         [0079]      FIG. 8A  illustrates a storage system  700  wherein data is stored on each of four storage nodes  710 ,  720 ,  730 ,  740 , where each of the storage nodes comprises multiple storage devices, such as multiple hard drives. For example, storage node  710  comprises hard drives  802 ,  804 ,  806 , and  808 . Where the example embodiment shows the same number of devices for each node, in other embodiments, each node could have different numbers of drives. 
         [0080]      FIG. 8B  illustrates  FIG. 8C  also illustrates one embodiment of data stored on storage drives and storage nodes in one embodiment of a distributed system wherein two storage drives are unavailable. If the storage system determines that device  816  is unavailable, one or more of the data blocks on the unavailable device  816  can be moved to other devices that have available storage space, such as, for example, if the distributed system  800  receives a write request to write data on an unavailable device  816 . In one embodiment, the distributed system  800  is a distributed file system where the metadata inode and the data sets are files. 
         [0081]    A. Embodiments of Mapping Structures 
         [0082]      FIG. 9  illustrates one embodiment of a map structure  900  used to store location data about data sets stored on one or more storage devices. The map structure  900  stores the location of the data blocks of the data set and data protection blocks of the data set. For example, nodes  960  store data indicating the location of the first stripe of data from  FIG. 2E . The first stripe of data includes d 0 , d 1 , d 2 , and p 0 (d 0 −d 2 ). In this embodiment, the data is indexed in a b-tree map structure  900  using the offset into the data set of the first block in the stripe. It is recognized, however, that a variety of map structures  900  may be used and/or different map structures  900  may be used for data protection and data. Node  970  stores the locations of the second stripe of data. The second stripe of data includes d 3 , d 4 , p 0 (d 3 −d 4 ), and p 1 (d 3 −d 4 ). Node  980  stores the locations of the third stripe of data. The third stripe of data includes d 5  and p 0 (d 5 ). The leaf nodes  960 ,  970 ,  980 ,  990 ,  992 ,  994  store data indicating the location of the data stripes. The leaf nodes  960 ,  970 ,  980 ,  990 ,  992 ,  994  are associated with parent nodes  930 ,  940   950 . The parent nodes  930 ,  940 ,  950  are associated with a root node  920 . In the exemplary map structure  900 , all copies of the superblocks related to the data set reference the root nodes  920  of the map structure  900 . 
         [0083]    In one embodiment, when a read or write request is received by the storage system, the map structure  900  is traversed in order to find the location of the requested data. For example, as indicated in leaf node  980 , the data block d 5  is stored on Drive  2 , location  2  and the related parity data, p 0  is stored on Drive  3 , location  2 . Thus, when the storage system receives a request for the data, the map structure  900  is traversed beginning at superblock  910 , continuing to root node  920  and to node  940 , and ending at node  980  where the location data for d 5  is located. More particularly, the node  940  comprises an entry, 6, which may be referred to as a key. If the requested data is less than 6, the location data is stored off of the first branch of the node, for example, node  980 ; if the requested data is greater than or equal to 6, then the location data is stored off of the right branch of node  940 . A similar process is performed in order to traverse from one of nodes  920  or  940  to one of the leaf nodes. 
         [0084]    If the storage device storing the data for d 5  is unavailable, the data blocks stored on the unavailable storage device may be migrated to another storage device. When this occurs, the map structure  900  is updated to indicate the new location of the data blocks in order to allow the data blocks to be accessed. In addition, if the device storing the data for node  980   b , for example, is unavailable, a copy of node  980   b  is made and stored on an available node, and the same goes for the nodes  940  and  920 . Systems and methods for traversing the map structure to check to see whether the nodes are available are disclosed in U.S. patent application Ser. No. 11/262,308 and U.S. Provisional Application Nos. 60/623,846 and 60/628,527 referenced below. 
         [0085]    In one embodiment, the map structure  900  is a file map structure that stores the locations of the file data and the parity data of a file. The superblocks are the inodes for the file. 
         [0086]      FIG. 10  illustrates one embodiment of a map structure  1000  used to store data on a distributed file system. More particularly, the map structure  1000  illustrates nodes that may be used in an index tree that maps the locations of inodes on a distributed file system using the unique identifier of the inodes, also referred to as a LIN tree. For example, metadata nodes  1035 ,  1040 ,  1050 , and  1055  store data indicating the location of the file index, or inode, corresponding to the particular .txt files noted in the Figure. As illustrated in  FIG. 10 , the leaf nodes  1035 ,  1040  are associated with a parent node  1030  and the leaf nodes  1050 ,  1055  are associated with a parent node  1045 . Each of the parent nodes  1030 ,  1045  are associated with a root node  1025 . In the exemplary map structure  1000 , four superblocks  1005 ,  1010 ,  1015 ,  1020 , are illustrated, where each superblock may be stored on a different node in a storage system. The superblocks each include references to each copy of the root node  1025  that may be stored on multiple devices. In one embodiment, multiple copies of each node are stored on various devices of a distributed storage system. U.S. patent application Ser. No. 11/255,818 entitled “Systems and Methods for Maintaining Distributed Data,” filed Oct. 21, 2005, which is hereby incorporated by reference in its entirety, describes additional exemplary methods of map of data and directory information in a file system. 
         [0087]    In one embodiment, in operation, when a read or write request is received by the storage system, the index structure is traversed in order to find the metadata node for the requested file. For example, as indicated in leaf node  1035 , the file “K_file.txt” has an index of 8. Thus, when the storage system receives a request for the file associated with an index of 8, the map structure  1000  is traversed, beginning at a superblock  1005 ,  1010 ,  1015 ,  1020 , continuing to node  1025 , then continuing to node  1030 , and ending at node  1035 , where the metadata node for the file associated with index 8 is located. More particularly, the node  1025  comprises an entry,  20 , which may be referred to as a key. If the requested file&#39;s index is less than or equal to 20, the files inode location is stored off of the first branch of the node, for example, node  1030 ; if the requested file&#39;s index is greater than 20, then the file&#39;s inode location is stored off of the second branch of the tree, for example, node  1045 . A similar process is performed in order to traverse from one of nodes  1030  or  1045  to one of the leaf nodes comprising the location of the files inode. 
         [0088]    Similar to the discussion above, if any of the nodes, including parent nodes, root nodes and superblocks, are stored on an unavailable device, references to the nodes on the unavailable devices should be updated to point to the new location of the index data previously stored on the unavailable nodes. 
         [0089]    The embodiment of  FIG. 10  illustrates a scenario when the metadata node device storing leaf node  1035   a , node  1030  and an inode  1025   a  are unavailable. Thus, when one of the inode files for the file “K_file.txt” is moved to another device, metadata nodes  1035   a  and  1035   b  are updated to reflect the new location of the inode file. The system may then determine that one of the metadata files, for example, node  1035   a , is stored on an unavailable device, and so metadata node  1035   b  is copied to become new node  1035   a  and new node  1035   a  is stored on an extant device. The system then updates the nodes  1030   a  and  1030   b  to reference the newly stored node  1035   a . The system may then determine that node  1030   a  is stored on an unavailable device, and so node  1030   b  is copied to become new node  1030   a , and new node  1030   a  is stored on an extant device. The system then updates nodes  1025   a  and  1025   b  to reference the newly stored  1030   a . Because nodes  1025   a  and  1025   b  are on available devices, no additional updating is needed. Accordingly, nodes  1135   a ,  1130   a ,  1130   b ,  1125   a , and  1125   b  are updated (as indicated by the dotted lines). 
         [0090]    In one embodiment, more than one copy of each index and leaf node is stored in the distributed file system so that if one of the devices fails, the index data will still be available. In one embodiment, the distributed file system uses a process that restores copies of the index and leaf nodes of the map data structures  900 ,  1000  if one of the copies is stored on an unavailable device. 
         [0091]    As used herein, data structures are collections of associated data elements, such as a group or set of variables or parameters. In one embodiment a structure may be implemented as a C-language “struct.” One skilled in the art will appreciate that many suitable data structures may be used. 
         [0092]    Embodiments of systems and methods for restoring metadata and data that is stored on nodes or drives that are unavailable and for updating the map data structure are disclosed in U.S. patent application Ser. No. 11/255,337, entitled “Systems And Methods For Accessing And Updating Distributed Data,” filed on Oct. 21, 2005, U.S. patent application Ser. No. 11/262,308, entitled “Distributed System With Asynchronous Execution Systems And Methods,” filed on Oct. 28, 2005, which claims priority to U.S. Provisional Appl. No. 60/623,846, entitled “Distributed System With Asynchronous Execution Systems And Methods,” filed on Oct. 29, 2004, and U.S. Provisional Appl. No. 60/628,527, entitled “Distributed System With Asynchronous Execution Systems And Methods,” filed on Nov. 15, 2004, and Patent Appl. No. 10,714,326, entitled “Systems and Methods for Restriping Files In A Distributed System,” filed on Nov. 14, 2003, which claims priority to U.S. Provisional Appl. No. 60/426,464, entitled “Systems and Methods for Restriping Files In A Distributed System,” filed on Nov. 14, 2002, all of which are hereby incorporated herein by reference in their entirety. 
         [0093]    B. Group Management Protocol 
         [0094]    In some embodiments, a group management protocol (“GMP”) is used to maintain a view of the nodes and/or drives available to the distributed file system. The GMP communicates which storage devices, for example, storage nodes and storage drives, are available to the storage system, their current operational state (for example, available, unavailable, up, down, dead) and how much space is available on the each device. The GMP sends a notification when a storage devices is unavailable, when it becomes available again, and/or when it becomes is permanently unavailable. The storage system uses information from the GMP to determine which storage devices are available for reading and writing after receiving a read or write request. 
         [0095]    On embodiment of a set of pseudocode for a GMP is set forth as follows: 
         [0000]    
       
         
               
             
           
               
                   
               
             
             
               
                 If (receive an error from the drive on a write) { 
               
               
                  Send notice that drive is about to go down 
               
               
                  Mark drive as down on the participant side 
               
               
                  Execute a GMP transaction to inform the rest of the cluster the drive 
               
               
                   is down 
               
               
                    Broadcast that we want to bring drive down (GMP prepare 
               
               
                     message) 
               
               
                    Receive an OK from all nodes (GMP prepared message) 
               
               
                    Broadcast that they should take the drive down (GMP 
               
               
                     commit message) 
               
               
                    Each initiator updates their map that the drive is down 
               
               
                 } 
               
               
                   
               
             
          
         
       
     
         [0096]    While the above pseudocode represents one embodiment of an implementation of a GMP, it is recognized that the GMP may be implemented in a variety of ways and is not limited to the exemplary embodiment above. Moreover the GMP may be used in conjunction with other protocols for coordinating activities among multiple nodes and/or systems. Embodiments of a protocol for coordinating activities among nodes are disclosed in U.S. patent application Ser. No. 11/262,306, entitled “Non-Blocking Commit Protocol Systems And Methods,” filed Oct. 28, 2005, which claims priority to U.S. Provisional Appl. No. 60/623,843, entitled “Non-Blocking Commit Protocol Systems And Methods,” filed Oct. 29, 2004, and U.S. patent application Ser. No. 11/449,153, entitled “Non-Blocking Commit Protocol Systems And Methods,” filed Jun. 8, 2006, all of which are hereby incorporated herein by reference in their entirety. 
         [0097]    Some of the figures and descriptions relate to an embodiment of the invention wherein the environment is that of a distributed file system, the present invention is not limited by the type of environment in which the systems and methods are used, however, and the systems and methods may be used in other environments, such as, for example, other file systems, other distributed systems, non-distributed systems, the Internet, the World Wide Web, a private network for a hospital, a broadcast network for a government agency, an internal network of a corporate enterprise, an intranet, a local area network, a wide area network, a wired network, a wireless network, a system area network, and so forth. It is also recognized that in other embodiments, the systems and methods described herein may be implemented as a single module and/or implemented in conjunction with a variety of other modules and the like. 
       VIII. Other Embodiments 
       [0098]    While certain embodiments of the invention have been described, these embodiments have been presented by way of example only, and are not intended to limit the scope of the present invention. The above-mentioned alternatives are examples of other embodiments, and they d 0  not limit the scope of the invention. It is recognized that a variety of data structures with various fields and data sets may be used. In addition, other embodiments of the flow charts may be used. 
         [0099]    It is also recognized that the term “remote” may include data, objects, devices, components, and/or modules not stored locally, that are or are not accessible via the local bus or data stored locally and that is “virtually remote.” Thus, remote data may include a device which is physically stored in the same room and connected to the user&#39;s device via a network. In other situations, a remote device may also be located in a separate geographic area, such as, for example, in a different location, country, and so forth. 
         [0100]    Moreover, while the description details certain embodiments of the invention, it will be appreciated that no matter how detailed the foregoing appears in text, the invention can be practiced in many ways. As is also stated above, it should be noted that the use of particular terminology when describing certain features or aspects of the invention should not be taken to imply that the terminology is being re-defined herein to be restricted to including any specific characteristics of the features or aspects of the invention with which that terminology is associated. The scope of the invention should therefore be construed in accordance with the appended claims and any equivalents thereof.