Systems and methods for preventing input/output performance decrease after disk failure in a distributed file system

In accordance with embodiments of the present disclosure, a method may include receiving from a plurality of data nodes of a distributed file system an indication of whether a fault condition exists with respect to a storage resource of the respective data node. The method may also include receiving an input/output request for a storage resource of a particular data node from a host information handling system communicatively coupled to the distributed file system. The method may further include, responsive to the input/output request, directing the input/output request to the particular data node if no fault condition exists with respect to storage resources of the particular data node and directing the input/output request to another data node of the distributed file system if a fault condition exists with respect to one or more storage resources of the particular data node.

TECHNICAL FIELD

The present disclosure relates in general to information handling systems, and more particularly to management of a distributed file system.

BACKGROUND

Dedicated storage solutions are commonplace in the market, particularly in the implementation of data centers. Such storage solutions may be in the form of network-based solutions which are often implemented as or part of a storage area network (SAN) employing Internet Small Computer System Interface (iSCSI), Fibre Channel, or other suitable communications standards. In some instances, a distributed storage system may be used as a storage solution. In a distributed file system, data may be spread across multiple storage nodes, which may allow for redundancy and increased performance.

When a disk within a storage node of a distributed file system fails, the distributed file system is typically rebuilt to reconstruct or recover the data of the failed disk, such rebuild being enabled by redundant data stored on the distributed file system. However, during such rebuild process, input/output (I/O) performance of client information handling systems attempting to access storage nodes may be degraded. In addition, the rebuild process may take significant amounts of time, especially when storage nodes are serving intensive I/O requests.

SUMMARY

In accordance with the teachings of the present disclosure, the disadvantages and problems associated with traditional approaches to file system management have been reduced or eliminated.

In accordance with embodiments of the present disclosure, a method may include receiving from a plurality of data nodes of a distributed file system an indication of whether a fault condition exists with respect to a storage resource of the respective data node. The method may also include receiving an input/output request for a storage resource of a particular data node from a host information handling system communicatively coupled to the distributed file system. The method may further include, responsive to the input/output request, directing the input/output request to the particular data node if no fault condition exists with respect to storage resources of the particular data node and directing the input/output request to another data node of the distributed file system if a fault condition exists with respect to one or more storage resources of the particular data node.

In accordance with these and other embodiments of the present disclosure, an information handling system may include a processor and a computer-readable medium having stored thereon a program of instructions. The program of instructions may be configured to, when read and executed by the processor receive from a plurality of data nodes of a distributed file system an indication of whether a fault condition exists with respect to a storage resource of the respective data node, receive an input/output request for a storage resource of a particular data node from a host information handling system communicatively coupled to the distributed file system, and, responsive to the input/output request, direct the input/output request to the particular data node if no fault condition exists with respect to storage resources of the particular data node, and direct the input/output request to another data node of the distributed file system if a fault condition exists with respect to one or more storage resources of the particular data node.

In accordance with these and other embodiments of the present disclosure, an article of manufacture may include a non-transitory computer readable medium and computer-executable instructions carried on the computer readable medium. The instructions may be readable by a processor and, when read and executed, may causing the processor to receive from a plurality of data nodes of a distributed file system an indication of whether a fault condition exists with respect to a storage resource of the respective data node, receive an input/output request for a storage resource of a particular data node from a host information handling system communicatively coupled to the distributed file system and responsive to the input/output request, direct the input/output request to the particular data node if no fault condition exists with respect to storage resources of the particular data node and direct the input/output request to another data node of the distributed file system if a fault condition exists with respect to one or more storage resources of the particular data node.

Technical advantages will be apparent to those of ordinary skill in the art in view of the following specification, claims, and drawings.

DETAILED DESCRIPTION

Preferred embodiments and their advantages are best understood by reference toFIGS. 1-4, wherein like numbers are used to indicate like and corresponding parts.

For the purposes of this disclosure, information handling resources may broadly refer to any component system, device or apparatus of an information handling system, including without limitation processors, service processors, BIOSs, buses, memories, I/O devices and/or interfaces, storage resources, network interfaces, motherboards, and/or any other components and/or elements of an information handling system.

An information handling system may include or may be coupled to an array of physical storage resources. The array of physical storage resources may include a plurality of physical storage resources, and may be operable to perform one or more input and/or output storage operations, and/or may be structured to provide redundancy. In operation, one or more physical storage resources disposed in an array of physical storage resources may appear to an operating system as a single logical storage array.

In certain embodiments, an array of physical storage resources may be implemented as a Redundant Array of Independent Disks (also referred to as a Redundant Array of Inexpensive Disks or a RAID). RAID implementations may employ a number of techniques to provide for redundancy, including striping, mirroring, and/or parity generation/checking. As known in the art, RAIDs may be implemented according to numerous RAID levels, including without limitation, standard RAID levels (e.g., RAID 0, RAID 1, RAID 3, RAID 4, RAID 5, and RAID 6), nested RAID levels (e.g., RAID 01, RAID 03, RAID 10, RAID 30, RAID 50, RAID 51, RAID 53, RAID 60, RAID 100), non-standard RAID levels, or others.

FIG. 1illustrates a block diagram of an example storage system100, in accordance with certain embodiments of the present disclosure. As depicted inFIG. 1, system100may include one or more hosts102and a distributed file system110communicatively coupled to hosts102via a network108.

A host102may comprise an information handling system. A host102may generally be operable to receive data from and/or communicate data to one or more storage resources114via network108. In certain embodiments, host102may be a server. In another embodiment, host102may be a dedicated storage system such as, for example, a network attached storage (NAS) system responsible for operating on the data in a storage array (e.g., a distributed file system110comprising storage resources114) and sending and receiving data from hosts coupled to the storage system. As depicted inFIG. 1, a host102may include a processor103and a memory104communicatively coupled to processor103.

A processor103may include any system, device, or apparatus configured to interpret and/or execute program instructions and/or process data, and may include, without limitation a microprocessor, microcontroller, digital signal processor (DSP), application specific integrated circuit (ASIC), or any other digital or analog circuitry configured to interpret and/or execute program instructions and/or process data. In some embodiments, a processor103may interpret and/or execute program instructions and/or process data stored in an associated memory104, stored in distributed file system110, and/or another component of a host102and/or system100.

A memory104may be communicatively coupled to an associated processor103and may include any system, device, or apparatus configured to retain program instructions and/or data for a period of time (e.g., computer-readable media). A memory104may include RAM, EEPROM, a PCMCIA card, flash memory, magnetic storage, opto-magnetic storage, or any suitable selection and/or array of volatile or non-volatile memory that retains data after power to a host102is turned off.

In addition to a processor103and a memory104, a host102may include one or more other information handling resources. An information handling resource may include any component system, device or apparatus of an information handling system, including without limitation a processor (e.g., processor103), bus, memory (e.g., memory104), input-output device and/or interface, storage resource (e.g., hard disk drives), network interface, electro-mechanical device (e.g., fan), display, power supply, and/or any portion thereof. An information handling resource may comprise any suitable package or form factor, including without limitation an integrated circuit package or a printed circuit board having mounted thereon one or more integrated circuits.

Network108may be a network and/or fabric configured to communicatively couple hosts102to each other and to distributed file system110. In certain embodiments, network108may include a communication infrastructure, which provides physical connections, and a management layer, which organizes the physical connections of hosts102, data nodes112, and other devices coupled to network108. Network108may be implemented as, or may be a part of, a storage area network (SAN), personal area network (PAN), local area network (LAN), a metropolitan area network (MAN), a wide area network (WAN), a wireless local area network (WLAN), a virtual private network (VPN), an intranet, the Internet or any other appropriate architecture or system that facilitates the communication of signals, data and/or messages (generally referred to as data). Network108may transmit data using any storage and/or communication protocol, including without limitation, Fibre Channel, Fibre Channel over Ethernet (FCoE), Small Computer System Interface (SCSI), Internet SCSI (iSCSI), Frame Relay, Ethernet Asynchronous Transfer Mode (ATM), Internet protocol (IP), or other packet-based protocol, and/or any combination thereof. Network108and its various components may be implemented using hardware, software, or any combination thereof.

Distributed file system110may comprise a plurality of data nodes112for storing data. In some embodiments, distributed file system110may be configured to logically appear to each host102as a single logical storage unit. In other embodiments, distributed file system110may be configured such that portions thereof each appear to hosts102as multiple logical storage units. The various data nodes112may comprise part of one or more RAIDs and/or other suitable redundant storage array(s). As shown inFIG. 1, distributed file system110may include a name node106and a plurality of data nodes112. In some embodiments, distributed file system110may comprise a Hadoop distributed file system.

A name node106may comprise an information handling system (e.g., a server) configured to manage a namespace of distributed file system110and regulate access by hosts102to files stored on data nodes112. Accordingly, name node106may include a processor and a memory embodying instructions for performing the functionality of name node106.

A data node112may comprise an information handling system (e.g., a server) configured to manage storage resources114integral to or attached to the data node. Thus, distributed file system110may operate in a master/slave architecture, whereby name node106is the master and data nodes112are the slaves. A data node112may include a processor and a memory embodying instructions for performing the functionality of the data node112.

Storage resources114may include hard disk drives, magnetic tape libraries, optical disk drives, magneto-optical disk drives, compact disk drives, compact disk arrays, disk array controllers, and/or any computer-readable medium operable to store data. In some embodiments, storage resources114may form all or part of a redundant storage array.

In operation, distributed file system110exposes a file system namespace and allows user data from hosts102to be stored in files. A file to be stored on distributed file system110may be split into one or more blocks and such blocks may be stored in a set of data nodes112. Name node106may execute file system namespace operations including opening, closing, and renaming files and directories, and may also determine mapping of blocks to data nodes112. Data nodes112may be configured to serve read and write requests from hosts102, and may be configured to perform block creation, deletion, and replication upon instruction from name node106.

In addition, name node106and data nodes112may be configured to manage file system I/O in the event of a failure (e.g., disk drive failure) of a storage resource114of a data node112in order to reduce or eliminate degraded I/O performance during rebuild of such failure. Such functionality is illustrated below with reference toFIGS. 2 and 3.

FIG. 2illustrates a flow chart of an example method200for managing a storage resource failure by a data node112, in accordance with the present disclosure. According to some embodiments, method200may begin at step202. As noted above, teachings of the present disclosure may be implemented in a variety of configurations of storage system100. As such, the preferred initialization point for method200and the order of the steps comprising method200may depend on the implementation chosen.

At step202, a disk status detection monitor may execute on each data node112. The disk status detection monitor may monitor storage resources114of a data node112for failure. In the event of such a failure, method200may proceed to step204. Otherwise, method200may remain at step202.

At step204, in response to a failure of a component storage resource114, a data node112may report a fault to name node106.

At step206, disk status detection monitor may continue to monitor storage resources114of a data node112for clearance of all failures (e.g., which may occur after a failed storage resource114is rebuilt). In the event of such all failures being cleared, method200may proceed to step208. Otherwise, method200may remain at step206.

At step208, in response to all failures of storage resources114of a data node112being cleared, data node112may communicate a message to name node106that no faults exist at data node112. After completion of step208, method200may proceed again to step202.

AlthoughFIG. 2discloses a particular number of steps to be taken with respect to method200, method200may be executed with greater or lesser steps than those depicted inFIG. 2. In addition, althoughFIG. 2discloses a certain order of steps to be taken with respect to method200, the steps comprising method200may be completed in any suitable order.

Method200may be implemented using storage system100or any other system operable to implement method200. In certain embodiments, method200may be implemented partially or fully in software and/or firmware embodied in computer-readable media.

FIG. 3illustrates a flow chart of an example method300for managing read I/O by name node106, in accordance with the present disclosure. According to some embodiments, method300may begin at step302. As noted above, teachings of the present disclosure may be implemented in a variety of configurations of storage system100. As such, the preferred initialization point for method300and the order of the steps comprising method300may depend on the implementation chosen.

At step302, name node106may receive a request from a host102to read data from a particular data node112. At step304, name node106may determine whether the particular data node112is in a fault condition. Such determination may be made based on fault messages received from such data node112(seeFIG. 2and description thereof). If a fault condition exists at such particular data node112, method300may proceed to step306. Otherwise, method300may proceed to step308.

At step306, in response to a determination that the particular data node112is in a fault condition, name node106may instruct the host102requesting the read I/O to access the read data from another data node112at which the data requested from the particular data node112is replicated. After completion of step306, method300may proceed again to step302.

At step308, in response to a determination that the particular data node112is in a fault condition, name node106may instruct the host102requesting the read I/O to access the read data from the particular data node112. After completion of step308, method300may proceed again to step302.

AlthoughFIG. 3discloses a particular number of steps to be taken with respect to method300, method300may be executed with greater or lesser steps than those depicted inFIG. 3. In addition, althoughFIG. 3discloses a certain order of steps to be taken with respect to method300, the steps comprising method300may be completed in any suitable order.

Method300may be implemented using storage system100or any other system operable to implement method300. In certain embodiments, method300may be implemented partially or fully in software and/or firmware embodied in computer-readable media.

FIG. 4illustrates a flow chart of an example method400for managing write I/O by name node106, in accordance with the present disclosure. According to some embodiments, method400may begin at step402. As noted above, teachings of the present disclosure may be implemented in a variety of configurations of storage system100. As such, the preferred initialization point for method400and the order of the steps comprising method400may depend on the implementation chosen.

At step402, name node106may receive a request from a host102to write data to a particular data node112. At step404, name node106may determine whether the particular data node112is in a fault condition. Such determination may be made based on fault messages received from such data node112(seeFIG. 2and description thereof). If a fault condition exists at such particular data node112, method400may proceed to step406. Otherwise, method400may proceed to step408.

At step406, in response to a determination that the particular data node112is in a fault condition, name node106may instruct the host102requesting the write I/O to write the I/O data to another data node112at which the data requested from the particular data node112is replicated. After completion of step406, method400may proceed again to step402.

At step408, in response to a determination that the particular data node112is in a fault condition, name node106may instruct the host102requesting the write I/O to write the I/O data to the particular data node112. After completion of step408, method400may proceed again to step402.

AlthoughFIG. 4discloses a particular number of steps to be taken with respect to method400, method400may be executed with greater or lesser steps than those depicted inFIG. 4. In addition, althoughFIG. 4discloses a certain order of steps to be taken with respect to method400, the steps comprising method400may be completed in any suitable order.

Method400may be implemented using storage system100or any other system operable to implement method400. In certain embodiments, method400may be implemented partially or fully in software and/or firmware embodied in computer-readable media.

Although the present disclosure has been described in detail, it should be understood that various changes, substitutions, and alterations can be made hereto without departing from the spirit and the scope of the disclosure as defined by the appended claims.