Method and system for a fast full style system check using multithreaded read ahead

A method for file system checking in a storage device. The method includes executing a computer system having a plurality microprocessor cores, initiating a file system check operation by using a file system check agent that execute on the computer system and accesses a storage device, and validating a plurality of meta-data structures of the file system. The method further includes dividing and allocating the metadata structures among a plurality of worker threads. For each worker thread, data corresponding to the metadata structures is processed using a read ahead operation. file system check is processed to completion, wherein the read ahead operation feeds data corresponding to the metadata structures to each of the plurality of worker threads in parallel.

FIELD OF THE INVENTION

The present invention is generally related to digital computer systems.

BACKGROUND OF THE INVENTION

Information drives business. Companies today rely to an unprecedented extent on online, frequently accessed, constantly changing data to run their businesses. Unplanned events that inhibit the availability of this data can seriously damage business operations. Additionally, any permanent data loss, from natural disaster or any other source, will likely have serious negative consequences for the continued viability of a business. Therefore, when disaster strikes, companies must be prepared to eliminate or minimize data loss, and recover quickly with useable data.

Companies have come to rely upon high-availability clusters to provide the most critical services and to store their most critical data. In general, there are different types of clusters, such as, for example, compute clusters, storage clusters, scalable clusters, and the like. High-availability clusters (also known as HA Clusters or Failover Clusters) are computer clusters that are implemented primarily for the purpose of providing high availability of services which the cluster provides. They operate by having redundant computers or nodes which are then used to provide service when system components fail. Normally, if a server with a particular application crashes, the application will be unavailable until someone fixes the crashed server. HA clustering remedies this situation by detecting hardware/software faults, and immediately restarting the application on another system without requiring administrative intervention, a process known as Failover. As part of this process, clustering software may configure the node before starting the application on it. For example, appropriate file systems may need to be imported and mounted, network hardware may have to be configured, and some supporting applications may need to be running as well.

A volume manager is often used to manage large-scale centralized computer storage systems. However, problems exist where, in such systems, the file systems grow large enough to become multi-terabyte filesystems. Companies that have created multi-terabyte file systems often complain that full file system check of the file system results in hours or even days of down time. This delay is so severe that companies often decided to size the file system based on the amount of time it would take to full file system check on that file system.

SUMMARY OF THE INVENTION

In one embodiment, the present invention comprises a computer implemented method for file system checking in a storage device. The method includes executing a computer system having a plurality microprocessor cores, initiating a file system check operation by using a file system check agent that execute on the computer system and accesses a storage device, and validating a plurality of meta-data structures of the file system. The method further includes dividing and allocating the metadata structures among a plurality of worker threads. For each worker thread, data corresponding to the metadata structures is processed using a read ahead operation. file system check is processed to completion, wherein the read ahead operation feeds data corresponding to the metadata structures to each of the plurality of worker threads in parallel.

In one embodiment, the plurality of worker threads comprises file system checker threads.

In one embodiment, the plurality of worker threads comprises read ahead component threads.

In one embodiment, the file system check validates whether data is listed with a properly connecting parent directory.

In one embodiment, the read ahead operation utilizes a parallel I/O capability of the storage device.

In one embodiment, the plurality of worker threads execute in parallel on the plurality of microprocessor cores of the computer system.

In one embodiment, the metadata structures comprise a plurality of inodes.

In one embodiment, the present invention is implemented as a computer readable storage medium having stored thereon, computer executable instructions that, if executed by a computer system cause the computer system to perform a fast file system check method. The method includes executing a computer system having a plurality microprocessor cores, initiating a file system check operation by using a file system check agent that execute on the computer system and accesses a storage device, and validating a plurality of meta-data structures of the file system. The method further includes dividing and allocating the metadata structures among a plurality of worker threads. For each worker thread, data corresponding to the metadata structures is processed using a read ahead operation. file system check is processed to completion, wherein the read ahead operation feeds data corresponding to the metadata structures to each of the plurality of worker threads in parallel.

In one embodiment, the plurality of worker threads comprises file system checker threads.

In one embodiment, the plurality of worker threads comprises read ahead component threads.

In one embodiment, the file system check validates whether data is listed with a properly connecting parent directory.

In one embodiment, the read ahead operation utilizes a parallel I/O capability of the storage device.

In one embodiment, the plurality of worker threads execute in parallel on the plurality of microprocessor cores of the computer system.

In one embodiment, the metadata structures comprise a plurality of inodes.

In one embodiment, the present invention is implemented as a computer system having a multicore microprocessor coupled to a memory, the memory having computer readable instructions which when executed by the microprocessor cause the computer system to perform a fast file system check method. The method includes executing a computer system having a plurality microprocessor cores, initiating a file system check operation by using a file system check agent that execute on the computer system and accesses a storage device, and validating a plurality of meta-data structures of the file system. The method further includes dividing and allocating the metadata structures among a plurality of worker threads. For each worker thread, data corresponding to the metadata structures is processed using a read ahead operation. File system check is processed to completion, wherein the read ahead operation feeds data corresponding to the metadata structures to each of the plurality of worker threads in parallel.

In one embodiment, the plurality of worker threads comprises file system checker threads.

In one embodiment, the plurality of worker threads comprises read ahead component threads.

In one embodiment, the file system check validates whether data is listed with a properly connecting parent directory.

In one embodiment the read ahead operation utilizes a parallel I/O capability of the storage device.

In one embodiment the plurality of worker threads execute in parallel on the plurality of microprocessor cores of the computer system.

DETAILED DESCRIPTION OF THE INVENTION

Although the present invention has been described in connection with one embodiment, the invention is not intended to be limited to the specific forms set forth herein. On the contrary, it is intended to cover such alternatives, modifications, and equivalents as can be reasonably included within the scope of the invention as defined by the appended claims.

In the following detailed description, numerous specific details such as specific method orders, structures, elements, and connections have been set forth. It is to be understood however that these and other specific details need not be utilized to practice embodiments of the present invention. In other circumstances, well-known structures, elements, or connections have been omitted, or have not been described in particular detail in order to avoid unnecessarily obscuring this description.

References within the specification to “one embodiment” or “an embodiment” are intended to indicate that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment of the present invention. The appearance of the phrase “in one embodiment” in various places within the specification are not necessarily all referring to the same embodiment, nor are separate or alternative embodiments mutually exclusive of other embodiments. Moreover, various features are described which may be exhibited by some embodiments and not by others. Similarly, various requirements are described which may be requirements for some embodiments but not other embodiments.

FIG. 1is a block diagram depicting a network architecture100in which client systems110,120and130, as well as storage servers140A and140B (e.g., it will and any of which can be implemented using computer system200), are coupled to a network150in accordance with one embodiment of the present invention. A storage server140A is further depicted as having storage devices160A(1)-(N) directly attached, and storage server140B is depicted with storage devices160B(1)-(N) directly attached. Storage servers140A and140B are also connected to a SAN fabric170, although connection to a storage area network is not required for operation of the disclosure. SAN fabric170supports access to storage devices180(1)-(N) by storage servers140A and140B, and so by client systems110,120and130via network150. SAN fabric170also supports access to a storage array190, which is also shown as an example of a specific storage device. Certain functionality of embodiments of the present invention are implemented via a file system check agent195, which is shown as instantiated on the server140B.

Client computer systems110,120and130are communicatively coupled via the network150. Client systems110,120and130are able to access information on data server140A or140B using, for example, a web browser or other client software (not shown). Such a client allows client systems110,120and130to access data hosted by storage server140A or140B or one of storage devices160A(1)-(N),160B(1)-(N),180(1)-(N) or storage array190. Additionally,FIG. 1depicts the use of a network such as the Internet for exchanging data, but the present disclosure is not limited to the Internet or any particular network-based environment. It should be noted that although two data server nodes140A and140B are shown, embodiments of the present invention can function with highly scalable distributed computer systems having hundreds of nodes, or more (e.g., cloud computing system, etc.).

FIG. 2shows a diagram depicting an exemplary high availability cluster-based computing system200in accordance with one embodiment of the present invention. As illustrated inFIG. 2, 4 nodes are shown coupled to a common storage device210. In one embodiment, the nodes on each host are an instantiation comprising an application, a file system check agent, a network file system (NFS) and a cluster file system (CFS). Hence, the computer system200comprises a four node cluster. Each of the constituent nodes is shown connected to the storage device210. The nodes of the cluster communicate with one another via the local area network201. These local area networks can in turn be connected to one or more wide area networks (e.g., the Internet). Although the embodiment is described in the context of a cluster, it should be appreciated that embodiments of the present invention can readily scale and function in distributed computer systems and/or cloud based computer systems having hundreds or thousands of nodes.

FIG. 2shows the manner in which the common storage device can support the mounting and dismounting of the various file systems to which it provides data storage services. The storage device210is typically equipped to provide high availability and redundancy while maintaining flexibility of the storage services provided to the various file systems. The storage device210typically includes a disk array that functions by providing storage space for mounting file systems and for providing high availability and disaster recovery type functionality for the file systems.

FIG. 3shows a first exemplary implementation of a storage device in accordance with one embodiment of the present invention. As depicted inFIG. 3, the 4 nodes of the cluster300are coupled to node310which hosts a volume manager and the file system check agent. The volume manager functions with a plurality of coupled LUNs (e.g., disk drives, solid-state drives, or the like) to provide volumes (e.g., data storage) which can be used to mount the plurality of network file systems and cluster file systems.

As depicted inFIG. 3, the 4 nodes of the cluster300are coupled to a node310which hosts a volume manager and the file system check agent. However, in theFIG. 3embodiment, the volume manager functions with a plurality of LUNs (e.g., that can be created by the storage array210). For example, the storage device210can be used to provide LUNs which can then be used to mount the plurality of network file systems and cluster file systems and the like.

FIG. 4shows a diagram of an exemplary file system check agent400in accordance with one embodiment of the present invention. As depicted inFIG. 4, the agent400includes a plurality of parallel file system checkers401-404and a corresponding plurality of metadata read ahead components411-414.

Embodiments of the present invention overcome the problems that plague conventional full file system checks on terabyte size file systems. Embodiments of the present invention advantageously utilize the plurality of file system checkers and the plurality of metadata read ahead components to create multiple threads that can perform file system check functions on multiple different parts of a file system in parallel. Additionally, the multiple metadata read ahead components advantageously initiate meta-data read on a per thread basis. These features provided by the functionality of embodiments of the present invention can typically result in a 90% improvement in full file system check execution time in comparison to the conventional solution. Thus, embodiments of the present invention enable customers to create multi-terabyte file systems without fear that a full file system check of the file system would result in hours or even days of down time. This factor enables customers to size their file systems in accordance with their needs, as opposed to sizing the file system based on the amount of time it would take to perform a full file system check on it.

FIG. 5shows a diagram of the parallel file system checkers401-404and the metadata read ahead components411-414as they form threads that execute in parallel on for cores of a microprocessor of a computer system in accordance with one embodiment of the present invention. As described above, the file system check agent advantageously utilizes the plurality of parallel file system checkers and the plurality of parallel data ahead components to create multiple threads. These multiple threads can perform file system check functions on multiple different parts of a file system in parallel. Hence, the parallel file system checkers401-404run in parallel on respective cores1through4. In theFIG. 5embodiment, the multiple metadata read ahead components execute with their corresponding parallel file system checker on a respective core.

FIG. 6shows a diagram of the parallel file system checkers401-404and the metadata read ahead components411-414as they form threads that execute in parallel on for cores of a microprocessor of a computer system in accordance with one embodiment of the present invention. As described above, the file system check agent advantageously utilizes the plurality of parallel file system checkers and the plurality of parallel data ahead components to create multiple threads. However, in theFIG. 6embodiment, the multiple metadata read ahead components and the multiple parallel file system checkers each execute with their own respective core. This feature greatly increases the number of parallel threads that can perform file system check functions on multiple different parts of a file system in parallel, thereby generating a large increase in performance.

FIG. 7shows a flowchart of the steps of a process700for performing a fast file system check in accordance with one embodiment of the present invention.

As described above, embodiments of the present invention implement a file system check agent that creates a multithreaded execution environment that scans various metadata structures, validates them, corrects them if required and repairs the file system. This multithreaded approach advantageously utilizes a multicore system CPU and advantageously utilizes the ability of the storage subsystem (e.g., storage subsystem210) to service multiple input outputs concurrently.

The file system check agent creates multiple threads parallel file system checkers to scan various parts of a file s system in parallel. Additionally, as described above, each file system checker thread triggers it's corresponding data read-ahead components to make maximum use of storage array I/O throughput and to make sure the file system checker threads undergo as little blocked on I/O as possible. This feature can typically allow a full file system check to becomes approximately twice to 10 times as fast (e.g., or more) in comparison to a conventional file system check approach.

Process700begins in step701, where the basic metadata structures of a file system like super block and metadata inodes (e.g., structural file set inodes) are validated.

In step702, the file system check agent creates multiple parallel file system checker threads to scan various parts of the file system in parallel.

In step703, the total number of inodes (e.g., the metadata descriptors that identify attributes of the various data blocks) in the file system are now divided and initiates read ahead allocated between worked threads and each worker thread reads a block worth of it's own inodes and initiates read-ahead on a few blocks of the remaining blocks of it's list of inodes to be validated.

In step704, as an inode is taken for validation by a thread, a read-ahead is initiated on other inode related metadata that comprises the inode (e.g., like block-maps, inode maps, etc) as that related metadata becomes available. In one embodiment, a read-ahead is initiated to check whether the inode is properly connected to a parent directory, or the like. In such an embodiment, as a block map of the inode becomes available, a read-ahead is also issued on the extent maps (or allocation unit maps) comprising the inode and other related metadata of the inode. In this manner, embodiments of the present invention take advantage of the fact that as metadata becomes available, read-aheads can be issued on related metadata is in fact referenced by the metadata just read in.

Subsequently, in step705, various checks like duplicate block map check, directory connectivity check, unreferenced inode check, and the like, are split between worker threads with proper locking when accessing shared data structures.

FIG. 8depicts a block diagram of a computer system810suitable for implementing certain functionality in accordance with one embodiment of the present invention. Computer system810includes a bus812which interconnects major subsystems of computer system810, such as a central processor814, a system memory817(typically RAM, but which may also include ROM, flash RAM, or the like), an input/output controller818, an external audio device, such as a speaker system820via an audio output interface822, an external device, such as a display screen824via display adapter826, serial ports828and830, a keyboard832(interfaced with a keyboard controller833), a storage interface834, a floppy disk drive837operative to receive a floppy disk838, a host bus adapter (HBA) interface card835A operative to connect with a Fiber Channel network890, a host bus adapter (HBA) interface card835B operative to connect to a SCSI bus839, and an optical disk drive840operative to receive an optical disk842. Also included are a mouse846(or other point-and-click device, coupled to bus812via serial port828), a modem847(coupled to bus812via serial port830), and a network interface848(coupled directly to bus812).

Bus812allows data communication between central processor814and system memory817, which may include read-only memory (ROM) or flash memory (neither shown), and random access memory (RAM) (not shown), as previously noted. The RAM is generally the main memory into which the operating system and application programs are loaded. In theFIG. 8embodiment, the system memory817instantiates a reclamation manager module850which implements the continuous reclamation functionality described above. The ROM or flash memory can contain, among other code, the Basic Input-Output system (BIOS) which controls basic hardware operation such as the interaction with peripheral components. Applications resident with computer system810are generally stored on and accessed via a computer readable medium, such as a hard disk drive (e.g., fixed disk844), an optical drive (e.g., optical drive840), a floppy disk unit837, or other storage medium. Additionally, applications can be in the form of electronic signals modulated in accordance with the application and data communication technology when accessed via network modem847or interface848.

Storage interface834, as with the other storage interfaces of computer system810, can connect to a standard computer readable medium for storage and/or retrieval of information, such as a fixed disk drive844. Fixed disk drive844may be a part of computer system810or may be separate and accessed through other interface systems. Modem847may provide a direct connection to a remote server via a telephone link or to the Internet via an internet service provider (ISP). Network interface848may provide a direct connection to a remote server via a direct network link to the Internet via a POP (point of presence). Network interface848may provide such connection using wireless techniques, including digital cellular telephone connection, Cellular Digital Packet Data (CDPD) connection, digital satellite data connection or the like.

Many other devices or subsystems (not shown) may be connected in a similar manner (e.g., document scanners, digital cameras and so on). Conversely, all of the devices shown inFIG. 8need not be present to practice the present disclosure. The devices and subsystems can be interconnected in different ways from that shown inFIG. 8. The operation of a computer system such as that shown inFIG. 8is readily known in the art and is not discussed in detail in this application. Code to implement the present disclosure can be stored in computer-readable storage media such as one or more of system memory817, fixed disk844, optical disk842, or floppy disk838. The operating system provided on computer system810may be MS-DOS®, MS-WINDOWS®, OS/2®, UNIX®, Linux®, or another known operating system.

FIG. 9shows an exemplary computer system900according to one embodiment. Computer system900depicts the components of a basic computer system providing the execution environment for certain hardware-based and software-based functionality for the above described embodiments. For example, computer system900can be a system upon which the one or more software agents (e.g., file system check agent400fromFIG. 4) are instantiated. Computer system900can be implemented as, for example, a server computer system, workstation computer system, desktop computer system, or laptop computer system. Similarly, computer system900can be implemented as a handheld device. Computer system900typically includes at least some form of computer readable media (e.g., computer readable storage medium901). Computer readable media can be a number of different types of available media that can be accessed by computer system900and can include, but is not limited to, computer storage media.

In its most basic configuration, computer system900typically includes processing unit903and memory901. Depending on the exact configuration and type of computer system900that is used, memory901can be volatile (e.g., such as DRAM, etc.)901a, non-volatile901b(e.g., such as ROM, flash memory, etc.) or some combination of the two. Similarly, the memory901can comprise other devices besides solid-state devices, such as, for example, magnetic disk-based media, optical media, or the like.

Additionally, computer system900can include other mass storage systems (e.g., removable905and/or non-removable907) such as magnetic or optical disks or tape. Similarly, computer system900can include input devices909and/or output devices911(e.g., such as a display). Computer system900can further include network connections913to other devices, computers, networks, servers, etc. using either wired or wireless media. As all of these devices are well known in the art, they need not be discussed in detail.

It should further be noted, that the computer system900can have some, most, or all of its functionality supplanted by a distributed computer system having a large number of dispersed computing nodes, such as would be the case where the functionality of the computer system900is partly or wholly executed using a cloud computing environment.

It should further be noted, that the computer system900can have some, most, or all of its functionality supplanted by a distributed computer system having a large number of dispersed computing nodes, such as would be the case where the functionality of the computer system900is partly or wholly executed using a cloud computing environment.