Source: http://www.google.com/patents/US20020161846?dq=5987118
Timestamp: 2015-11-28 19:46:30
Document Index: 233712182

Matched Legal Cases: ['Application No. 60', 'Application No. 60', 'Application No. 60', 'Application No. 60', 'Application No. 60', 'Application No. 60', 'Application No. 60', 'Application No. 60']

Patent US20020161846 - Data path controller architecture - Google PatentsSearch Images Maps Play YouTube News Gmail Drive More »Sign inAdvanced Patent SearchPatentsA data path controller architecture is disclosed. The architecture includes a network interface for communicating with one or more clients, a storage interface for communicating with one or more disk drives. A data engine configured to communicate with the storage interface to receive file data from...http://www.google.com/patents/US20020161846?utm_source=gb-gplus-sharePatent US20020161846 - Data path controller architectureAdvanced Patent SearchPublication numberUS20020161846 A1Publication typeApplicationApplication numberUS 10/060,956Publication dateOct 31, 2002Filing dateJan 29, 2002Priority dateJan 29, 2001Also published asUS6745286, US6754773, US6775792, US6871295, US20020156973, US20020156974, US20020156975, US20020161850, US20020161973, US20020165942, US20020166026, US20020166079, US20020174295, US20020174296, US20020178162Publication number060956, 10060956, US 2002/0161846 A1, US 2002/161846 A1, US 20020161846 A1, US 20020161846A1, US 2002161846 A1, US 2002161846A1, US-A1-20020161846, US-A1-2002161846, US2002/0161846A1, US2002/161846A1, US20020161846 A1, US20020161846A1, US2002161846 A1, US2002161846A1InventorsThomas Ulrich, James Schweitzer, Gregory Bolstad, Jay Randall, John Staub, Wilbur Priester, David Barry, Leonard Olsen, Danny Lam, Ronald GodshalkOriginal AssigneeUlrich Thomas R., Schweitzer James R., Bolstad Gregory D., Randall Jay G., Staub John R., Priester Wilbur George, Barry David H., Olsen Leonard D., Danny Lam, Godshalk Ronald K.Export CitationBiBTeX, EndNote, RefManPatent Citations (3), Referenced by (74), Classifications (28), Legal Events (10) External Links: USPTO, USPTO Assignment, EspacenetData path controller architecture
US 20020161846 A1Abstract
A data path controller architecture is disclosed. The architecture includes a network interface for communicating with one or more clients, a storage interface for communicating with one or more disk drives. A data engine configured to communicate with the storage interface to receive file data from the one or more disk drives. The data engine communicates with the network interface to send file data to the one or more clients. A CPU is configured to queue transaction requests for the data engine in response to file requests from the clients. The data engine receives file data in response to at least a portion of the transaction requests. The data engine sends file data to the one or more clients in response to the transaction requests. Images(47) Claims(52)
What is claimed is: 1. A file server comprising: a network interface for communicating with one or more clients; a storage interface for communicating with one or more disk drives; a data engine configured to communicate with said storage interface to receive file data from said one or more disk drives, said data engine further configured to communicate with said network interface to send file data to said one or more clients; and a CPU configured to queue transaction requests for said data engine in response to file requests from said one or more clients, said data engine configured to receive file data in response to at least a portion of said transaction requests, said data engine configured to send file data to said one or more clients in response to at least a portion of said transaction requests. 2. The file server of claim 1, wherein said network interface comprises one or more Fibre Channel interfaces. 3. The file server of claim 1, wherein said storage interface comprises one or more Fibre Channel interfaces. 4. The file server of claim 1, wherein said storage interface comprises one or more SCSI interfaces. 5. The file server of claim 1, wherein said storage interface comprises one or more IDE interfaces. 6. The file server of claim 1, wherein at least one of said storage interface and said network interface comprises an InfiniBand interface. 7. The file server of claim 1, further comprising one or more data caches operably connected to said data engine. 8. The file server of claim 1, said data engine further configured to generate parity. 9. The file server of claim 1, wherein said data engine is further configured to generate exclusive-or parity. 10. The file server of claim 1, wherein said data engine is further configured to compute exclusive-or parity for distributed parity groups. 11. The file server of claim 1, wherein said data engine is further configured to regenerate lost data at least in part from parity data. 12. The file server of claim 1, wherein said transaction requests comprise storage transaction requests and network transaction requests, said storage transaction requests queued to said storage interface and said network transaction requests queued to said network interface. 13. The file server of claim 1, wherein each of said transaction requests comprises an opcode. 14. The file server of claim 13, wherein said opcode comprises a code to specify at least one of a read from cache, a write to cache, an XOR write to cache, a write to a first cache with an XOR write to a second cache, and a write to said first cache and said second cache. 15. The file server of claim 1, wherein said CPU communicates with said storage interface using a PCI bus. 16. The file server of claim 15, wherein said CPU queues transactions to said storage interface. 17. The file server of claim 1, wherein said CPU communicates with said storage interface using a first PCI bus and said CPU communicates with said network interface using a second PCI bus. 18. The file server of claim 17, wherein said CPU queues network transactions to said network interface, said network transactions comprising data flow between at least one of said clients and at least one data cache operably connected to said data engine. 19. The file server of claim 17, wherein said CPU queues storage transactions to said storage interface, said storage transactions comprising data flow between at least one of said disk drives and at least one data cache operably connected to said data engine. 20. The file server of claim 19, wherein said storage transaction further comprises at least one parity operation. 21. The file server of claim 1, further comprising a metadata cache operably connected to said CPU. 22. The file server of claim 21, wherein said CPU manages metadata stored in said metadata cache, said metadata comprising directory information that describes a directory structure of at least a portion of a network file system. 23. The file server of claim 21, wherein said CPU manages metadata, said metadata configured to describe a directory structure of a portion of a distributed file system, said metadata comprising location information for files catalogued in said directory structure. 24. The file server of claim 23, wherein said location information comprises server identifiers that identify respective servers for accessing files catalogued in said directory structure. 25. The file server of claim 1, further comprising a metadata cache, said CPU managing metadata stored in said metadata cache, wherein said metadata identifies data blocks stored on one or more of said disk drives and corresponding parity blocks stored on one or more of said disk drives. 26. The file server of claim 25, wherein said metadata identifies parity groups, said parity groups comprising a plurality of information blocks, said information blocks comprising one or more data blocks, said information blocks further comprising a parity block, each of said information blocks stored on a different disk drive. 27. The file server of claim 26, wherein a size of a first parity group is independent of a size of a second parity group. 28. A method of providing file services, comprising: receiving a file request from a client, said request received by a first processing module; accessing metadata to locate file data corresponding to said file request, said metadata stored in a metadata cache provided to said first processing module; queuing at least one storage transaction requests to a storage interface, said storage transaction request queued by said first processing module; storing disk data retrieved as a result of said storage transaction request in a data cache operably connected to a data engine, said data engine operating asynchronously with respect to said first processing module; queuing one or more network transaction requests to a network interface, said network transaction request queued by said first processing module upon completion of said at least one storage transaction request; and sending file data from said data cache to said client according to said network transaction request, wherein said sending operation is performed asynchronously, with respect to said first processing module, by said data engine and said network interface. 29. The method of claim 28, wherein said network interface comprises one or more Fibre Channel interfaces. 30. The method of claim 28, wherein said storage interface comprises one or more Fibre Channel interfaces. 31. The method of claim 28, wherein said storage interface comprises one or more SCSI interfaces. 32. The method of claim 28, wherein said storage interface comprises one or more IDE interfaces. 33. The method of claim 28, wherein at least one of said storage interface and said network interface comprises an InfiniBand interface. 34. The method of claim 28, wherein said data engine computes parity. 35. The method of claim 28, further comprising re-generating lost data using parity processing in said data engine. 36. The method of claim 35, wherein said parity processing comprises exclusive-or parity processing. 37. The method of claim 28, wherein said storage transaction request comprises an opcode. 38. The method of claim 37, wherein said opcode comprises a code to specify at least one of a read from cache, a write to cache, an XOR write to cache, a write to a first cache with an XOR write to a second cache, and a write to said first cache and said second cache. 39. The method of claim 28, wherein said first processing module communicates with said storage interface using a PCI bus. 40. The method of claim 28, wherein said first processing module communicates with said storage interface using a first PCI bus and said first processing module communicates with said network interface using a second PCI bus. 41. The method of claim 28, wherein a storage transaction comprises at least one parity operation. 42. The method of claim 28, further comprising modifying metadata stored in said metadata cache, said metadata comprising directory information that describes a directory structure of at least a portion of a network file system. 43. The method of claim 28, further comprising modifying metadata stored in said metadata cache, said metadata comprising information for locating data stored in files one or more storage devices. 44. The method of claim 43, wherein said information for locating data comprises one or more server identifiers. 45. The method of claim 28, wherein said metadata identifies parity groups, said parity groups comprising data blocks stored on one or more disk drives and corresponding parity blocks stored on another disk drive, said metadata comprising information regarding a disk location for each of said data blocks and for said parity block. 46. The method of claim 28, wherein said metadata identifies parity groups, said parity groups comprising a plurality of information blocks, said information blocks comprising one or more data blocks, said information blocks further comprising a parity block, each of said information blocks stored on a different disk drive, said first processing unit assigning disk locations for said information blocks, said data engine accessing said data blocks and generating said parity block. 47. The method of claim 46, wherein a size of a first parity group is independent of a size of a second parity group. 48. An apparatus comprising: a network interface for communicating with one or more clients; a storage interface for communicating with one or more disk drives; means for receiving file requests from said clients, managing file system metadata, queuing network transaction requests to said network interface, and queuing storage transaction requests to said storage interface; and means for receiving received data from said storage interface and storing at least a portion of said received data in a data cache according to a first address word containing a first opcode provided from at least one of said queued storage transaction requests, and for sending file data from said data cache to said network interface according to a second address word containing a second opcode provided from at least one of said queued network transaction requests. 49. The apparatus of claim 48, said means for receiving received data further computing parity data from said received data. 50. A method of providing file services, comprising: receiving a write request from a client, said write request received by a first processing module; accessing metadata to locate a file corresponding to said write request, said metadata stored in a metadata cache provided to said first processing module; queuing a network transaction request to a network interface, said network transaction request queued by said first processing module to retrieve, from said client, data to be written to said file; storing disk data retrieved as a result of said network transaction request in a data cache operably connected to a data engine, said data engine operating asynchronously with respect to said first processing module; queuing at least one storage transaction request to a storage interface, said storage transaction request queued by said first processing module upon completion of said network transaction request; and sending file data from said data cache to said storage interface according to said storage transaction request, wherein said sending operation is performed by said data engine and said network interface asynchronously with respect to said first processing module. 51. An apparatus comprising: a network interface for communicating with one or more clients; a storage interface for communicating with one or more disk drives; means for receiving write requests from said clients, managing file system metadata, queuing network transaction requests to said network interface, and queuing storage transaction requests to said storage interface; and means for receiving write data from said network interface and storing at least a portion of said write data in a data cache according to a first address word containing a first opcode provided from at least one of said queued network transaction requests, and for sending file data from said data cache to said storage interface according to a second address word containing a second opcode provided from at least one of said queued storage transaction requests. 52. The apparatus of claim 51, said means for receiving write data computing parity data from said write data.
REFERENCE TO RELATED APPLICATIONS [0001] The present application claims priority benefit under 35 U.S.C. �119(e) from all of the following U.S. Provisional Applications, the contents of which are hereby incorporated by reference in their entirety: [0002] U.S. Provisional Application No. 60/264671, filed Jan. 29, 2001, titled “DYNAMICALLY DISTRIBUTED FILE SYSTEM”; [0003] U.S. Provisional Application No. 60/264694, filed Jan. 29, 2001, titled “A DATA PATH ACCELERATOR ASIC FOR HIGH PERFORMANCE STORAGE SYSTEMS”; [0004] U.S. Provisional Application No. 60/264672, filed Jan. 29, 2001, titled “INTEGRATED FILE SYSTEM/PARITY DATA PROTECTION”; [0005] U.S. Provisional Application No. 60/264673, filed Jan. 29, 2001, titled “DISTRIBUTED PARITY DATA PROTECTION”; [0006] U.S. Provisional Application No. 60/264670, filed Jan. 29, 2001, titled “AUTOMATIC IDENTIFICATION AND UTILIZATION OF RESOURCES IN A DISTRIBUTED FILE SERVER”; [0007] U.S. Provisional Application No. 60/264669, filed Jan. 29, 2001, titled “DATA FLOW CONTROLLER ARCHITECTURE FOR HIGH PERFORMANCE STORAGE SYSTEMS”; [0008] U.S. Provisional Application No. 60/264668, filed Jan. 29, 2001, titled “ADAPTIVE LOAD BALANCING FOR A DISTRIBUTED FILE SERVER”; and [0009] U.S. Provisional Application No. 60/302424, filed Jun. 29, 2001, titled “DYNAMICALLY DISTRIBUTED FILE SYSTEM”.
BACKGROUND OF THE INVENTION [0010] 1. Field of the Invention [0011] This invention relates to the field of data storage and management. More particularly, this invention relates to high-performance mass storage systems and methods for data storage, backup, and recovery. [0012] 2. Description of the Related Art [0013] In modem computer systems, collections of data are usually organized and stored as files. A file system allows users to organize, access, and manipulate these files and also performs administrative tasks such as communicating with physical storage components and recovering from failure. The demand for file systems that provide high-speed, reliable, concurrent access to vast amounts of data for large numbers of users has been steadily increasing, in recent years. Often such systems use a Redundant Array of Independent Disks (RAID) technology, which distributes the data across multiple disk drives, but provides an interface that appears to users as one, unified disk drive system, identified by a single drive letter. In a RAID system that includes more than one array of disks, each array is often identified by a unique drive letter, and in order to access a given file, a user must correctly identify the drive letter for the disk array on which the file resides. Any transfer of files from one disk array to another and any addition of new disk arrays to the system must be made known to users so that they can continue to correctly access the files. [0014] RAID systems effectively speed up access to data over single-disk systems, and they allow for the regeneration of data lost due to a disk failure. However, they do so by rigidly prescribing the configuration of system hardware and the block size and location of data stored on the disks. Demands for increases in storage capacity that are transparent to the users or for hardware upgrades that lack conformity with existing system hardware cannot be accommodated, especially while the system is in use. In addition, such systems commonly suffer from the problem of data fragmentation, and they lack the flexibility necessary to intelligently optimize use of their storage resources. [0015] RAID systems are designed to provide high-capacity data storage with built-in reliability mechanisms able to automatically reconstruct and restore saved data in the event of a hardware failure or data corruption. In conventional RAID technology, techniques including spanning, mirroring, and duplexing are used to create a data storage device from a plurality of smaller single disk drives with improved reliability and storage capacity over conventional disk systems. RAID systems generally incorporate a degree of redundancy into the storage mechanism to permit saved data to be reconstructed in the event of single (or sometimes double) disk failure within the disk array. Saved data is further stored in a predefined manner that is dependent on a fixed algorithm to distribute the information across the drives of the array. The manner of data distribution and data redundancy within the disk array impacts the performance and usability of the storage system and may result in substantial tradeoffs between performance, reliability, and flexibility. [0016] A number of RAID configurations have been proposed to map data across the disks of the disk array. Some of the more commonly recognized configurations include RAID-1, RAID-2, RAID-3, RAID-4, and RAID-5. [0017] In most RAID systems, data is sequentially stored in data stripes and a parity block is created for each data stripe. The parity block contains information derived from the sequence and composition of the data stored in the associated data stripe. RAID arrays can reconstruct information stored in a particular data stripe using the parity information, however, this configuration imposes the requirement that records span across all drives in the array resulting in a small stripe size relative to the stored record size. [0018] [0018]FIG. 21 illustrates the data mapping approach used in many conventional RAID storage device implementations. Although the diagram corresponds most closely to RAID-3 or RAID-4 mapping schemas, other RAID configurations are organized in a similar manner. As previously indicated, each RAID configuration uses a striped disk array 2110 that logically combines two or more disk drives 2115 into a single storage unit. The storage space of each drive 2115 is organized by partitioning the space on the drives into stripes 2120 that are interleaved so that the available storage space is distributed evenly across each drive. [0019] Information or files are stored on the disk array 2110. Typically, the writing of data to the disks occurs in a parallel manner to improve performance. A parity block is constructed by performing a logical operation (exclusive OR) on the corresponding blocks of the data stripe to create a new block of data representative of the result of the logical operation. The result is termed a parity block and is written to a separate area 2130 within the disk array. In the event of data corruption within a particular disk of the array 10, the parity information is used to reconstruct the data using the information stored in the parity block in conjunction with the remaining non-corrupted data blocks. [0020] In the RAID architecture, multiple disks a typically mapped to a single ‘virtual disk’. Consecutive blocks of the virtual disk are mapped by a strictly defined algorithm to a set of physical disks with no file level awareness. When the RAID system is used to host a conventional file system, it is the file system that maps files to the virtual disk blocks where they may be mapped in a sequential or non-sequential order in a RAID stripe. The RAID stripe may contain data from a single file or data from multiple files if the files are small or the file system is highly fragmented. [0021] The aforementioned RAID architecture suffers from a number of drawbacks that limit its flexibility and scalability for use in reliable storage systems. One problem with existing RAID systems is that the data striping is designed to be used in conjunction with disks of the same size. Each stripe occupies a fixed amount of disk space and the total number of stripes allowed in the RAID system is limited by the capacity of the smallest disk in the array. Any additional space that may be present on drives having a capacity larger than the smallest drive goes unused as the RAID system lacks the ability to use the additional space. This further presents a problem in upgrading the storage capacity of the RAID system, as all of the drives in the array must be replaced with larger capacity drives if additional storage space is desired. Therefore, existing RAID systems are inflexible in terms of their drive composition, increasing the cost and inconvenience to maintain and upgrade the storage system. [0022] A further problem with conventional RAID arrays resides in the rigid organization of data on the disks of the RAID array. As previously described, this organization typically does not use available disk space in an efficient manner. These systems further utilize a single fixed block size to store data which is implemented with the restriction of sequential file storage along each disk stripe. Data storage in this manner is typically inefficient as regions or gaps of disk space may go unused due to the file organization restrictions. Furthermore, the fixed block size of the RAID array is not able to distinguish between large files, which benefit from larger block size, and smaller files, which benefit from smaller block size for more efficient storage and reduced wasted space. [0023] Although conventional RAID configurations are characterized as being fault-tolerant, this capability is typically limited to single disk failures. Should more than one (or two) disk fail or become inoperable within the RAID array before it can be replaced or repaired there is the potential for data loss. This problem again arises from the rigid structure of data storage within the array that utilizes sequential data striping. This problem is further exacerbated by the lack of ability of the RAID system to flexibly redistribute data to other disk areas to compensate for drive faults. Thus, when one drive becomes inoperable within the array, the likelihood of data loss increases significantly until the drive is replaced resulting in increased maintenance and monitoring requirements when using conventional RAID systems. [0024] With respect to conventional data storage systems or other computer networks, conventional load balancing includes a variety of drawbacks. For example, decisions relating to load balancing are typically centralized in one governing process, one or more system administrators, or combinations thereof. Accordingly, such systems have a single point of failure, such as the governing process or the system administrator. Moreover, load balancing occurs only when the centralized process or system administrator can organize performance data, make a decision, and then transmit that decision throughout the data storage system or computer network. This often means that the such load balancing can be slow to react, difficult to optimize for a particular server, and difficult to scale as the available resources expand or contract. In addition, conventional load balancing typically is limited to balancing processing and communications activity between servers only. SUMMARY OF THE INVENTION [0025] The present invention solves these and other problems by providing a dynamically distributed file system that accommodates current demands for high capacity, throughput, and reliability, while presenting to the users a single-file-system interface that appears to include every file in the system on a single server or drive. In this way, the file system is free to flexibly, transparently, and on-the-fly distribute and augment physical storage of the files in any manner that suits its needs, across disk drives, and across servers, and users can freely access any file without having specific knowledge of the files current physical location. [0026] One embodiment includes a storage device and architecture which possesses features such as transparent scalability where disks of non-identical capacity can be fully-utilized without the “dead-space” restrictions associated with conventional disk arrays. In one embodiment a flexible storage space allocation system handles storing large and small file types to improve disk space utilization. In another embodiment an improved method for maintaining data integrity overcomes the single drive (or double) fault limitation of conventional systems in order to increase storage reliability while at the same time reducing maintenance and monitoring requirements. [0027] In one embodiment, distributed parity groups (DPG) are integrated into the distributed file storage system technology. This architecture provides capabilities for optimizing the use of disk resources by moving frequently and infrequently accessed data blocks between drives so as to maximize the throughput and capacity utilization of each drive. [0028] In one embodiment, the architecture supports incorporation of new disk drives without significant reconfiguration or modification of the exiting distributed file storage system to provide improved reliability, flexibility, and scalability. Additionally, the architecture permits the removal of arbitrary disk drives from the distributed file storage system and automatically redistributes the contents of these drives to other available drives as necessary. [0029] The distributed file storage system can proactively position objects for initial load balancing, such as, for example, to determine where to place a particular new object. Additionally, the distributed file storage system can continue to proactively position objects, thereby accomplishing active load balancing for the existing objects throughout the system. According to one embodiment, one or more filters may be applied during initial and/or active load balancing to ensure one or a small set of objects are not frequently transferred, or churned, throughout the resources of the system. [0030] As used herein, load balancing can include, among other things, capacity balancing, throughput balancing, or both. Capacity balancing seeks balance in storage, such as the number of objects, the number of Megabytes, or the like, stored on particular resources within the distributed file storage system. Throughput balancing seeks balance in the number of transactions processed, such as, the number of transactions per second, the number of Megabytes per second, or the like, handled by particular resources within the distributed file storage system. According to one embodiment, the distributed file storage system can position objects to balance capacity, throughput, or both, between objects on a resource, between resources, between the servers of a cluster of resources, between the servers of other clusters of resources, or the like. [0031] The distributed file storage system can comprise resources, such as servers or clusters, which can seek to balance the loading across the system by reviewing a collection of load balancing data from itself, one or more of the other servers in the system, or the like. The load balancing data can include object file statistics, server profiles, predicted file accesses, or the like. A proactive object positioner associated with a particular server can use the load balancing data to generate an object positioning plan designed to move objects, replicate objects, or both, across other resources in the system. Then, using the object positioning plan, the resource or other resources within the distributed file storage system can execute the plan in an efficient manner. [0032] According to one embodiment, each server pushes objects defined by that server's respective portion of the object positioning plan to the other servers in the distributed file storage system. By employing the servers to individually push objects based on the results of their object positioning plan, the distributed file storage system provides a server-, process-, and administrator-independent approach to object positioning, and thus load balancing, within the distributed file storage system. [0033] In one embodiment, the network file storage system includes a first file server operably connected to a network fabric; a second file server operably connected to the network fabric; first file system information loaded on the first file server; and second file system information loaded on the second file server, the first file system information and the second file system information configured to allow a client computer operably connected to the network fabric to locate files stored by the first file server and files stored by the second file server without prior knowledge as to which file server stores the files. In one embodiment, the first file system information includes directory information that describes a directory structure of a portion of the network file system whose directories are stored on the first file server, the directory information includes location information for a first file, the location information includes a server id that identifies at least the first file server or the second file server. [0034] In one embodiment, the network file storage system loads first file system metadata on a first file server operably connected to a network fabric; loads second file system metadata on a second file server connected to the network fabric, the first file system metadata and the second file system metadata include information to allow a client computer operably connected to the network fabric to locate a file stored by the first file server or stored by the second file server without prior knowledge as to which file server stores the file. [0035] In one embodiment, the network file storage system performs a file handle lookup on a computer network file system by: sending a root-directory lookup request to a first file server operably connected to a network fabric; receiving a first lookup response from the first file server, the first lookup response includes a server id of a second file server connected to the network fabric; sending a directory lookup request to the second file server; and receiving a file handle from the second file server. [0036] In one embodiment, the network file storage system allocates space by: receiving a file allocation request in a first file server, the first file server owning a parent directory that is to contain a new file, the file allocation request includes a file handle of the parent directory; determining a selected file server from a plurality of file servers; sending a file allocation request from the first server to the selected server; creating metadata entries for the new file in file system data managed by the selected file server; generating a file handle for the new file; sending the file handle to the first file server; and creating a directory entry for the new file in the parent directory. [0037] In one embodiment, the network file storage system includes: a first file server operably connected to a network fabric; a second file server operably connected to the network fabric; first file system information loaded on the first file server; and second file system information loaded on the second file server, the first file system information and the second file system information configured to allow a client computer operably connected to the network fabric to locate files owned by the first file server and files owned by the second file server without prior knowledge as to which file server owns the files, the first file server configured to mirror at least a portion of the files owned by the second file server, the first file server configured to store information sufficient to regenerate the second file system information, and the second file server configured to store information sufficient to regenerate the first file system information. [0038] In one embodiment, the network file storage system: loads first file system metadata on a first file server operably connected to a network fabric; loads second file system metadata on a second file server connected to the network fabric, the first file system metadata and the second file system metadata include information to allow a client computer operably connected to the network fabric to locate a file stored by the first file server or stored by the second file server without prior knowledge as to which file server stores the file; maintains information on the second file server to enable the second file server to reconstruct an information content of the first file system metadata; and maintains information on the first file server to enable the first file server to reconstruct an information content of the second file system metadata. [0039] In one embodiment the computer network file storage system is fault-tolerant and includes: a first file server operably connected to a network fabric; a second file server operably connected to the network fabric; a first disk array operably coupled to the first file server and to the second file server; a second disk array operably coupled to the first file server and to the second file server; first file system information loaded on the first file server, the first file system information including a first intent log of proposed changes to the first metadata; second file system information loaded on the second file server, the second file system information including a second intent log of proposed changes to the second metadata, the first file server having a copy of the second intent log, the second file server maintaining a copy of the first intent log, thereby allowing the first file server to access files on the second disk array in the event of a failure of the second file server. [0040] In one embodiment, a distributed file storage system provides hot-swapping of file servers by: loading first file system metadata on a first file server operably connected to a network fabric, the first file system operably connected to a first disk drive and a second disk drive; loading second file system metadata on a second file server connected to the network fabric, the second file system operably connected to the first disk drive and to the second disk drive; copying a first intent log from the first file server to a backup intent log on the second file server, the first intent log providing information regarding future changes to information stored on the first disk drive; and using the backup intent log to allow the second file server to make changes to the information stored on the first disk drive. [0041] In one embodiment, a distributed file storage system includes: a first file server operably connected to a network fabric; a file system includes first file system information loaded on the first file server, the file system configured to create second file system information on a second file server that comes online sometime after the first file server has begun servicing file requests, the file system configured to allow a requester to locate files stored by the first file server and files stored by the second file server without prior knowledge as to which file server stores the files. [0042] In one embodiment, a distributed file storage system adds servers during ongoing file system operations by: loading first file system metadata on a first file server operably connected to a network fabric; creating at least one new file on a second file server that comes online while the first file server is servicing file requests, the at least one new file created in response to a request issued to the first file server, the distributed file system configured to allow a requester to locate files stored by the first file server and files stored by the second file server without prior knowledge as to which file server stores the files. [0043] In one embodiment, a distributed file storage system includes: first metadata managed primarily by a first file server operably connected to a network fabric, the first metadata includes first file location information, the first file location information includes at least one server id; and second metadata managed primarily by a second file server operably connected to the network fabric, the second metadata includes second file location information, the second file location information includes at least one server identifier, the first metadata and the second metadata configured to allow a requestor to locate files stored by the first file server and files stored by the second file server in a directory structure that spans the first file server and the second file server. [0044] In one embodiment, a distributed file storage system stores data by: creating first file system metadata on a first file server operably connected to a network fabric, the first file system metadata describing at least files and directories stored by the first file server; creating second file system metadata on a second file server connected to the network fabric, the second file system metadata describing at least files and directories stored by the second file server, the first file system metadata and the second file system metadata includes directory information that spans the first file server and the second file server, the directory information configured to allow a requestor to find a location of a first file catalogued in the directory information without prior knowledge as to a server location of the first file. [0045] In one embodiment, a distributed file storage system balances the loading of servers and the capacity of drives associated with the servers, the file system includes: a first disk drive including a first unused capacity; a second disk drive including a second unused capacity, wherein the second unused capacity is smaller than the first unused capacity; a first server configured to fill requests from clients through access to at least the first disk drive; and a second server configured to fill requests from clients through access to at least the second disk drive, and configured to select an infrequently accessed file from the second disk drive and push the infrequently accessed files to the first disk drive, thereby improving a balance of unused capacity between the first and second disk drives without substantially affecting a loading for each of the first and second servers. [0046] In one embodiment, a distributed file storage system includes: a first file server operably connected to a network fabric; a second file server operably connected to the network fabric; first file system information loaded on the first file server; and second file system information loaded on the second file server, the first file system information and the second file system information configured to allow a client computer operably connected to the network fabric to locate files stored by the first file server and files stored by the second file server without prior knowledge as to which file server stores the files. [0047] In one embodiment, a data engine offloads data transfer operations from a server CPU. In one embodiment, the server CPU queues data operations to the data engine. [0048] In one embodiment, a distributed file storage system includes: a plurality of disk drives for storing parity groups, each parity group includes storage blocks, the storage blocks includes one or more data blocks and a parity block associated with the one or more data blocks, each of the storage blocks stored on a separate disk drive such that no two storage blocks from a given parity set reside on the same disk drive, wherein file system metadata includes information to describe the number of data blocks in one or more parity groups. [0049] In one embodiment, a distributed file storage system stores data by: determining a size of a parity group in response to a write request, the size describing a number of data blocks in the parity group; arranging at least a portion of data from the write request according to the data blocks; computing a parity block for the parity group; storing each of the data blocks on a separate disk drive such that no two data blocks from the parity group reside on the same disk drive; and storing each the parity block on a separate disk drive that does not contain any of the data blocks. [0050] In one embodiment, a distributed file storage system includes: a plurality of disk drives for storing parity groups, each parity group includes storage blocks, the storage blocks includes one or more data blocks and a parity block associated with the one or more data blocks, each of the storage blocks stored on a separate disk drive such that no two storage blocks from a given parity set reside on the same disk drive; a redistribution module to dynamically redistribute parity groups by combining some parity groups to improve storage efficiency. [0051] In one embodiment, a distributed file storage system stores data by: determining a size of a parity group in response to a write request, the size describing a number of data blocks in the parity group; arranging at least a portion of data from the write request according to the data blocks; computing a parity block for the parity group; storing each of the data blocks on a separate disk drive such that no two data blocks from the parity group reside on the same disk drive; storing the parity block on a separate disk drive that does not contain any of the data blocks; and redistributing the parity groups to improve storage efficiency. [0052] In one embodiment, a distributed file storage system includes: a plurality of disk drives for storing parity groups, each parity group includes storage blocks, the storage blocks includes one or more data blocks and a parity block associated with the one or more data blocks, each of the storage blocks stored on a separate disk drive such that no two storage blocks from a given parity set reside on the same disk drive; and a recovery module to dynamically recover data lost when at least a portion of one disk drive in the plurality of disk drives becomes unavailable, the recovery module configured to produce a reconstructed block by using information in the remaining storage blocks of a parity set corresponding to an unavailable storage block, the recovery module further configured to split the parity group corresponding to an unavailable storage block into two parity groups if the parity group corresponding to an unavailable storage block spanned all of the drives in the plurality of disk drives.